I am using a 12x12 numpy matrix, and I am getting "shape too large to be a matrix." My best guess is that numpy "kron" function is making trouble.
Here's my code:
a = np.matrix("0 1 0; 0 0 1; 0 0 0 ")
a_dag = np.matrix("0 0 0; 1 0 0 ; 0 1 0")
Sp = np.matrix("0 1; 0 0")
Sm = np.matrix("0 0; 1 0")
...
119 H_I1 = (np.exp(1j*(phi-omega*t))*kron(np.eye(3),Sp,np.eye(2))
120 +np.exp(-1j*(phi-omega*t))*kron(np.eye(3),Sm,np.eye(2)))
121 H_I2 = kron(a,Sp,np.eye(2)) + kron(a_dag,Sm,np.eye(2))
Here's the error:
Traceback (most recent call last):
File "/home/fyodr/qc_final.py", line 121, in <module>
H_I2 = kron(a,Sp,np.eye(2)) + kron(a_dag,Sm,np.eye(2))
File "/home/fyodr/qc_final.py", line 70, in kron
return np.kron(m[0],kron(m[1:]))
File "/usr/lib/python2.7/dist-packages/numpy/lib/shape_base.py", line 754, in kron
result = wrapper(result)
File "/usr/lib/python2.7/dist-packages/numpy/matrixlib/defmatrix.py", line 303, in __array_finalize__
raise ValueError("shape too large to be a matrix.")
ValueError: shape too large to be a matrix.
Thanks!
EDIT: I defined kron as
def kron(*m):
if len(m) == 1:
return m
else :
return np.kron(m[0],kron(m[1:]))
If np.kron were computing a regular kronecker product, then this should not be a problem.
As I commented, your kron with 3 arguments is unknown. But if it produces a 3d array as some stage, it could produce your error.
In [264]: np.kron(a.A, np.ones((3,3,3))).shape
Out[264]: (3, 9, 9)
A 2d array with a 3d returns a 3d array. But if a is a np.matrix it tries to convert that to a matrix resulting in the error. np.matrix is always 2d.
In [265]: np.kron(a, np.ones((3,3,3))).shape
---------------------------------------------------------------------------
....
ValueError: shape too large to be a matrix.
Experienced numpy users don't use np.matrix unless we really need its features, and can live with its drawbacks.
With the kron that you added, the recursive step does:
In [270]: m = (a, Sp, np.eye(2))
In [271]: kron(m[1:])
Out[271]:
((matrix([[0, 1],
[0, 0]]), array([[ 1., 0.],
[ 0., 1.]])),)
In [272]: np.array(_)
Out[272]:
array([[[[ 0., 1.],
[ 0., 0.]],
[[ 1., 0.],
[ 0., 1.]]]])
In [273]: _.shape
Out[273]: (1, 2, 2, 2)
For 2 items, your kron returns a nested tuple of arrays. np.kron applies a np.asanyarray(b) to that 2nd argument, which results in a 4d array.
Applying your kron to full *m, but turning the matrices into arrays:
In [275]: kron(a.A, Sp.A, np.eye(2))
Out[275]:
array([[[[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]],
[[ 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1., 0.],
[ 0., 0., 0., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]]]])
In [276]: _.shape
Out[276]: (1, 2, 6, 6)
Did you even test the kron function by itself? It should have been debugged before use in a more complicated task.
Related
I know "*variable_name" assists in packing and unpacking.
But how does variable_name.shape work? Unable to visualize why the second dimension is squeezed out when prefixing with ""?
print("top_class.shape {}".format(top_class.shape))
top_class.shape torch.Size([64, 1])
print("*top_class.shape {}".format(*top_class.shape))
*top_class.shape 64
for numpy.array that is extensively used in math-related and image processing programs, .shape describes the size of the array for all existing dimensions:
>>> import numpy as np
>>> a = np.zeros((3,3,3))
>>> a
array([[[ 0., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 0.]],
[[ 0., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 0.]],
[[ 0., 0., 0.],
[ 0., 0., 0.],
[ 0., 0., 0.]]])
>>> a.shape
(3, 3, 3)
>>>
The asterisk "unpacks" the tuple into several separate arguments, in your case (64,1) becomes 64, 1, so only the first one get printed because there's only one format specification.
A is an numpy array with shape (6, 8)
I want:
x_id = np.array([0, 3])
y_id = np.array([1, 3, 4, 7])
A[ [x_id, y_id] += 1 # this doesn't actually work.
Tricks like ::2 won't work because the indices do not increase regularly.
I don't want to use extra memory to repeat [0, 3] and make a new array [0, 3, 0, 3] because that is slow.
The indices for the two dimensions do not have equal length.
which is equivalent to:
A[0, 1] += 1
A[3, 3] += 1
A[0, 4] += 1
A[3, 7] += 1
Can numpy do something like this?
Update:
Not sure if broadcast_to or stride_tricks is faster than nested python loops. (Repeat NumPy array without replicating data?)
You can convert y_id to a 2d array with the 2nd dimension the same as x_id, and then the two indices will be automatically broadcasted due to the dimension difference:
x_id = np.array([0, 3])
y_id = np.array([1, 3, 4, 7])
A = np.zeros((6,8))
A[x_id, y_id.reshape(-1, x_id.size)] += 1
A
array([[ 0., 1., 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0.]])
I am trying to store matrices into an array, however when I append the matrix, it would get every element and output just an 1 dimensional array.
Example Code:
matrix_array= np.array([])
for y in y_label:
matrix_array= np.append(matrix_array, np.identity(3))
Clearly np.append is the wrong tool for the job:
In [144]: np.append(np.array([]), np.identity(3))
Out[144]: array([ 1., 0., 0., 0., 1., 0., 0., 0., 1.])
From its docs:
If axis is not specified, values can be any shape and will be
flattened before use.
With list append
In [153]: alist=[]
In [154]: for y in [1,2]:
...: alist.append(np.identity(3))
...:
In [155]: alist
Out[155]:
[array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]]), array([[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]])]
In [156]: np.array(alist)
Out[156]:
array([[[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]],
[[ 1., 0., 0.],
[ 0., 1., 0.],
[ 0., 0., 1.]]])
In [157]: _.shape
Out[157]: (2, 3, 3)
I am using h5py to build a dataset. Since I want to store arrays with different #of rows dimension, I use the h5py special_type vlen. However, I experience behavior I can't explain, maybe you can me help in understanding what is happening:
>>>> import h5py
>>>> import numpy as np
>>>> fp = h5py.File(datasource_fname, mode='w')
>>>> dt = h5py.special_dtype(vlen=np.dtype('float32'))
>>>> train_targets = fp.create_dataset('target_sequence', shape=(9549, 5,), dtype=dt)
>>>> test
Out[130]:
array([[ 0., 1., 1., 1., 0., 1., 1., 0., 1., 0., 0.],
[ 1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1.]])
>>>> train_targets[0] = test
>>>> train_targets[0]
Out[138]:
array([ array([ 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 1.], dtype=float32),
array([ 1., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0.], dtype=float32),
array([ 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0.], dtype=float32),
array([ 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0.], dtype=float32),
array([ 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)], dtype=object)
I do expect the train_targets[0] to be of this shape, however I can't recognize the rows in my array. They seem to be totally jumbled about, however it is consistent. By which I mean that every time I try the above code, train_targets[0] looks the same.
To clarify: the first element in my train_targets, in this case test, has shape (5,11), however the second element might be of shape (5,38) which is why I use vlen.
Thank you for your help
Mat
I think
train_targets[0] = test
has stored your (11,5) array as an F ordered array in a row of train_targets. According to the (9549,5) shape, that's a row of 5 elements. And since it is vlen, each element is a 1d array of length 11.
That's what you get back in train_targets[0] - an array of 5 arrays, each shape (11,), with values taken from test (order F).
So I think there are 2 issues - what a 2d shape means, and what vlen allows.
My version of h5py is pre v2.3, so I only get string vlen. But I suspect your problem may be that vlen only works with 1d arrays, an extension, so to speak, of byte strings.
Does the 5 in shape=(9549, 5,) have anything to do with 5 in the test.shape? I don't think it does, at least not as numpy and h5py see it.
When I make a file following the string vlen example:
>>> f = h5py.File('foo.hdf5')
>>> dt = h5py.special_dtype(vlen=str)
>>> ds = f.create_dataset('VLDS', (100,100), dtype=dt)
and then do:
ds[0]='this one string'
and look at ds[0], I get an object array with 100 elements, each being this string. That is, I've set a whole row of ds.
ds[0,0]='another'
is the correct way to set just one element.
vlen is 'variable length', not 'variable shape'. While the https://www.hdfgroup.org/HDF5/doc/TechNotes/VLTypes.html documentation is not entirely clear on this, I think you can store 1d arrays with shape (11,) and (38,) with vlen, but not 2d ones.
Actually, train_targets output is reproduced with:
In [54]: test1=np.empty((5,),dtype=object)
In [55]: for i in range(5):
test1[i]=test.T.flatten()[i:i+11]
It's 11 values taken from the transpose (F order), but shifted for each sub array.
I have an NxN matrix filled with zeros. Now I want to add to the matrix, say, n ones and m twos to random places. I.e. I want to create a matrix where there is some fixed amount of a given number at random places and possibly a fixed amount of some other given number in random places. How do I do this?
In Matlab I would do this by making a random permutation of the matrix indices with randperm() and then filling the n first indices given by randperm of the matrix with ones and m next with twos.
You can use numpy.random.shuffle to randomly permute an array in-place.
>>> import numpy as np
>>> X = np.zeros(N * N)
>>> X[:n] = 1
>>> X[n:n+m] = 2
>>> np.random.shuffle(X)
>>> X = X.reshape((N, N))
Would numpy.random.permutation be what you are looking for?
You can do something like this:
In [9]: a=numpy.zeros(100)
In [10]: p=numpy.random.permutation(100)
In [11]: a[p[:10]]=1
In [12]: a[p[10:20]]=2
In [13]: a.reshape(10,10)
Out[13]:
array([[ 0., 1., 0., 0., 0., 2., 0., 1., 0., 0.],
[ 0., 0., 1., 0., 0., 0., 0., 0., 2., 0.],
[ 0., 2., 0., 0., 0., 0., 2., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 2., 0., 2., 1., 1., 0.],
[ 0., 0., 0., 0., 1., 0., 2., 0., 0., 0.],
[ 0., 2., 0., 2., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 2., 0., 0., 0., 1., 0.]])
Here we create a random permutation, then set the first 10 indices taken from the permutation in a to 1, then the next 10 indices to 2.
To generate the indices of the elements for where to add ones and twos, what about this?
# assuming N, n and m exist.
In [1]: import random
In [3]: indices = [(m, n) for m in range(N) for n in range(N)]
In [4]: random_indices = random.sample(indices, n + m)
In [5]: ones = random_indices[:n]
In [6]: twos = random_indices[n:]
Corrected as commented by Petr Viktorin in order not to have overlapping indexes in ones and twos.
An alternate way to generate the indices:
In [7]: import itertools
In [8]: indices = list(itertools.product(range(N), range(N)))