If I run:
x = np.zeros(6)
y = np.zeros([7, 6])
z = y * x
Then everything is fine, and there are no Python errors.
However, I am using a Python module (call if foo) containing a function (call if bar), which returns a 7x6 NumPy array. It has the same shape as y above, and the same data type (float64). But when I run the following:
x = np.zeros(6)
y = foo.bar()
z = y * x
I get the following error:
ValueError: shapes (7,6) and (1,6) not aligned: 6 (dim 1) != 1 (dim 0)
But as far as I can tell, y is exactly the same format in these two examples, with the same shape and data type. What's causing this error, and why is it not caused in the first example?
In [446]: x = np.zeros(6)
...: y = np.zeros([7, 6])
...: z = y * x
In [447]: z.shape
Out[447]: (7, 6)
Here we are doing element-wise multiplication, a (7,6) with a (6,). By broadcasting the (6,) becomes (1,6) and then (7,6) to match y.
Evidently in the foo.bar case, y is np.matrix subclass:
In [454]: y1 = np.matrix(y)
In [455]: y1*x
---...
219 # This promotes 1-D vectors to row vectors
--> 220 return N.dot(self, asmatrix(other))
...
ValueError: shapes (7,6) and (1,6) not aligned: 6 (dim 1) != 1 (dim 0)
Note the different display for y1:
In [456]: y1
Out[456]:
matrix([[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.]])
With np.matrix * is defined as np.dot, the matrix product. The x is also converted np.matrix, producing a (1,6) matrix. The error message follows from the definition of matrix multiplication.
np.multiply can be used force the element-wise multiplication. Note the class of the result:
In [458]: np.multiply(y1,x)
Out[458]:
matrix([[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.]])
Because of confusions like this np.matrix is being discouraged.
I don't know which version you're running, but I am running version 1.16.3 on Python 3.
It seems to me you're defining your x differently than in your example snippet. You seem to be defining it as a 6x1 matrix instead of a "vector", which is considered on Numpy to only have one dimension. Try multiplying y by np.zeros([6,1]) and you'll see an error.
Bottom line is:
Use the .shape property a lot when debugging, it's very useful when you're doing matrix multiplication.
On np, multiplication between a matrix and a one dimensional array is carried differently than multiplications between 2 matrices.
Related
I want to create 3D array with a dimension which depends on the length of a 2D list
Method 1 :
the length of list_1 is 2 and each element has the same length
list_1 = [[1,2,3,4,5], [1,2,3,4,5]]
array_1 = np.zeros((2,len(list_1), 114))
the shape of array_1 is (2,2,114)
method 2 :
the length of list_2 is also 2 but elements do not have the same length
list_2 = [[1,2,3,4,5],[1,2,3,4]]
array_2 = [np.zeros((len(list_2[i]),114)) for i in range(len(list_2))]
array_2 = np.array(list_2, dtype=object)
In this case the shape of array_2 is (2,)
Does someone know the reason ? I do not understand why I do not get the same shape.
Is there a way to get the same shape ?
You made a list of 2 arrays, that differ in the number of rows:
In [130]: [np.zeros((len(list_2[i]), 3)) for i in range(len(list_2))]
Out[130]:
[array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]]),
array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]])]
np.array cannot combine those into one 3d array, so it makes a shape 2 array, containing those same 2 arrays:
In [131]: np.array(_, object)
Out[131]:
array([array([[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.],
[0., 0., 0.]]),
array([[0., 0., 0.],
[0., 0., 0.]
[0., 0., 0.],
[0., 0., 0.]])], dtype=object)
If you don't include the object dtype it warns about making a "ragged array".
Using len(list_1) for the first case, and len(list_2[i]) in the second, are two very different situations. In one you are using length of the list itself, and the the length of the sublists.
If you truncated the sublists to the same length:
In [137]: np.array([np.zeros((len(list_2[i][:4]),114)) for i in range(len(list_2))],object).shape
Out[137]: (2, 4, 114)
The subarrays now have the same shape (4,114), and can be combined into one 3d array.
I want to create a matrix with Numpy, but I need to add every element by its row and column indices.
for example:
my_matrix = np.matrix(np.zeros((5, 5)))
my_matrix.insert(row_index=2, column_index=1, value=10)
output:
matrix([[0., 0., 0., 0., 0.],
[10., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])
How can I do that?
Do you want to add or insert values?
The add function that you mentioned is used as an element-wise addition.
Example:
np.add([1, 2], [2, 3])
Out[41]: array([3, 5])
If you really want to create a matrix a matrix by inserting values with column and row indices, create the matrix first and insert your values afterwards.
number_rows = 10
number_cols = 20
arr = np.empty((number_rows, number_cols))
arr[2, 1] = 10
The use of np.matrix is discouraged, if not actually deprecated. It is rarely needed, except for some backward compatibility cases.
In [1]: arr = np.zeros((5,5))
In [2]: arr
Out[2]:
array([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])
In [3]: mat = np.matrix(arr)
In [4]: mat
Out[4]:
matrix([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])
Indexing one row of arr produces a 1d array
In [5]: arr[2]
Out[5]: array([0., 0., 0., 0., 0.])
Indexing one row of mat produces a 2d matrix, with shape (1,5)
In [6]: mat[2]
Out[6]: matrix([[0., 0., 0., 0., 0.]])
We can access an element in the 1d array:
In [7]: arr[2][1]
Out[7]: 0.0
but this indexing of the mat tries to access a row, and gives an error:
In [8]: mat[2][1]
---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
<ipython-input-8-212ad5378f8e> in <module>
----> 1 mat[2][1]
...
IndexError: index 1 is out of bounds for axis 0 with size 1
In both cases it is better to access an element with the tuple syntax, rather than the chained one:
In [9]: arr[2,1]
Out[9]: 0.0
In [10]: mat[2,1]
Out[10]: 0.0
This indexing also works for setting values. But try to avoid iterating to set individual values. Try to find ways of creating the whole array with the desired values directly, with whole array methods, not iteration.
Can someone explain that code for me?
def gradientDescent(X, y, theta, alpha, iters):
temp = np.matrix(np.zeros(theta.shape))
parameters = int(theta.ravel().shape[1])
cost = np.zeros(iters)
for i in range(iters):
error = (X * theta.T) - y
for j in range(parameters):
term = np.multiply(error, X[:,j])
temp[0,j] = theta[0,j] - ((alpha / len(X)) * np.sum(term))
theta = temp
cost[i] = computeCost(X, y, theta)
return theta, cost
Evaluate it step by step on a sample:
In [13]: np.matrix(np.zeros((3,4)))
Out[13]:
matrix([[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]])
In [14]: _.ravel()
Out[14]: matrix([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
In [15]: _.shape
Out[15]: (1, 12)
In [16]: _[1]
Out[16]: 12
A np.matrix is always 2d, even when ravelled.
If we used an array, rather than matrix:
In [17]: np.zeros((3,4))
Out[17]:
array([[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]])
In [18]: _.ravel()
Out[18]: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
In [19]: _.shape
Out[19]: (12,)
In [20]: _[0]
Out[20]: 12
The loop requires that X, theta and temp all have the same 2nd dimension. I also think theta must be a (1,n) matrix to start with. Otherwise this ravel parameters would be too large. But in that case the ravel isn't needed in the first place.
I am using a 12x12 numpy matrix, and I am getting "shape too large to be a matrix." My best guess is that numpy "kron" function is making trouble.
Here's my code:
a = np.matrix("0 1 0; 0 0 1; 0 0 0 ")
a_dag = np.matrix("0 0 0; 1 0 0 ; 0 1 0")
Sp = np.matrix("0 1; 0 0")
Sm = np.matrix("0 0; 1 0")
...
119 H_I1 = (np.exp(1j*(phi-omega*t))*kron(np.eye(3),Sp,np.eye(2))
120 +np.exp(-1j*(phi-omega*t))*kron(np.eye(3),Sm,np.eye(2)))
121 H_I2 = kron(a,Sp,np.eye(2)) + kron(a_dag,Sm,np.eye(2))
Here's the error:
Traceback (most recent call last):
File "/home/fyodr/qc_final.py", line 121, in <module>
H_I2 = kron(a,Sp,np.eye(2)) + kron(a_dag,Sm,np.eye(2))
File "/home/fyodr/qc_final.py", line 70, in kron
return np.kron(m[0],kron(m[1:]))
File "/usr/lib/python2.7/dist-packages/numpy/lib/shape_base.py", line 754, in kron
result = wrapper(result)
File "/usr/lib/python2.7/dist-packages/numpy/matrixlib/defmatrix.py", line 303, in __array_finalize__
raise ValueError("shape too large to be a matrix.")
ValueError: shape too large to be a matrix.
Thanks!
EDIT: I defined kron as
def kron(*m):
if len(m) == 1:
return m
else :
return np.kron(m[0],kron(m[1:]))
If np.kron were computing a regular kronecker product, then this should not be a problem.
As I commented, your kron with 3 arguments is unknown. But if it produces a 3d array as some stage, it could produce your error.
In [264]: np.kron(a.A, np.ones((3,3,3))).shape
Out[264]: (3, 9, 9)
A 2d array with a 3d returns a 3d array. But if a is a np.matrix it tries to convert that to a matrix resulting in the error. np.matrix is always 2d.
In [265]: np.kron(a, np.ones((3,3,3))).shape
---------------------------------------------------------------------------
....
ValueError: shape too large to be a matrix.
Experienced numpy users don't use np.matrix unless we really need its features, and can live with its drawbacks.
With the kron that you added, the recursive step does:
In [270]: m = (a, Sp, np.eye(2))
In [271]: kron(m[1:])
Out[271]:
((matrix([[0, 1],
[0, 0]]), array([[ 1., 0.],
[ 0., 1.]])),)
In [272]: np.array(_)
Out[272]:
array([[[[ 0., 1.],
[ 0., 0.]],
[[ 1., 0.],
[ 0., 1.]]]])
In [273]: _.shape
Out[273]: (1, 2, 2, 2)
For 2 items, your kron returns a nested tuple of arrays. np.kron applies a np.asanyarray(b) to that 2nd argument, which results in a 4d array.
Applying your kron to full *m, but turning the matrices into arrays:
In [275]: kron(a.A, Sp.A, np.eye(2))
Out[275]:
array([[[[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]],
[[ 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 1., 0., 0.],
[ 0., 0., 0., 0., 1., 0.],
[ 0., 0., 0., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0.]]]])
In [276]: _.shape
Out[276]: (1, 2, 6, 6)
Did you even test the kron function by itself? It should have been debugged before use in a more complicated task.
I am using h5py to build a dataset. Since I want to store arrays with different #of rows dimension, I use the h5py special_type vlen. However, I experience behavior I can't explain, maybe you can me help in understanding what is happening:
>>>> import h5py
>>>> import numpy as np
>>>> fp = h5py.File(datasource_fname, mode='w')
>>>> dt = h5py.special_dtype(vlen=np.dtype('float32'))
>>>> train_targets = fp.create_dataset('target_sequence', shape=(9549, 5,), dtype=dt)
>>>> test
Out[130]:
array([[ 0., 1., 1., 1., 0., 1., 1., 0., 1., 0., 0.],
[ 1., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1.]])
>>>> train_targets[0] = test
>>>> train_targets[0]
Out[138]:
array([ array([ 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 1.], dtype=float32),
array([ 1., 0., 0., 0., 1., 0., 0., 0., 0., 1., 0.], dtype=float32),
array([ 0., 0., 0., 1., 0., 0., 0., 0., 1., 0., 0.], dtype=float32),
array([ 0., 0., 1., 0., 0., 0., 0., 1., 0., 0., 0.], dtype=float32),
array([ 0., 1., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)], dtype=object)
I do expect the train_targets[0] to be of this shape, however I can't recognize the rows in my array. They seem to be totally jumbled about, however it is consistent. By which I mean that every time I try the above code, train_targets[0] looks the same.
To clarify: the first element in my train_targets, in this case test, has shape (5,11), however the second element might be of shape (5,38) which is why I use vlen.
Thank you for your help
Mat
I think
train_targets[0] = test
has stored your (11,5) array as an F ordered array in a row of train_targets. According to the (9549,5) shape, that's a row of 5 elements. And since it is vlen, each element is a 1d array of length 11.
That's what you get back in train_targets[0] - an array of 5 arrays, each shape (11,), with values taken from test (order F).
So I think there are 2 issues - what a 2d shape means, and what vlen allows.
My version of h5py is pre v2.3, so I only get string vlen. But I suspect your problem may be that vlen only works with 1d arrays, an extension, so to speak, of byte strings.
Does the 5 in shape=(9549, 5,) have anything to do with 5 in the test.shape? I don't think it does, at least not as numpy and h5py see it.
When I make a file following the string vlen example:
>>> f = h5py.File('foo.hdf5')
>>> dt = h5py.special_dtype(vlen=str)
>>> ds = f.create_dataset('VLDS', (100,100), dtype=dt)
and then do:
ds[0]='this one string'
and look at ds[0], I get an object array with 100 elements, each being this string. That is, I've set a whole row of ds.
ds[0,0]='another'
is the correct way to set just one element.
vlen is 'variable length', not 'variable shape'. While the https://www.hdfgroup.org/HDF5/doc/TechNotes/VLTypes.html documentation is not entirely clear on this, I think you can store 1d arrays with shape (11,) and (38,) with vlen, but not 2d ones.
Actually, train_targets output is reproduced with:
In [54]: test1=np.empty((5,),dtype=object)
In [55]: for i in range(5):
test1[i]=test.T.flatten()[i:i+11]
It's 11 values taken from the transpose (F order), but shifted for each sub array.