Very often, arrays are squeezed with np.squeeze(). In the documentation, it says
Remove single-dimensional entries from the shape of a.
However I'm still wondering: Why are zero and nondimensional entries in the shape of a? Or to put it differently: Why do both a.shape = (2,1) and (2,) exist?
Besides the mathematical differences between the two things, there is the issue of predictability. If your suggestion was followed, you could at no point rely on the dimension of your array. So any expression of the form my_array[x,y] would need to be replaced by something that first checks if my_array is actually two-dimensional and did not have an implicit squeeze at some point. This would probably obfuscate code far more than the occasional squeeze, which does a clearly specified thing.
Actually, it might even be very hard to tell, which axis has been removed, leading to a whole host of new problems.
In the spirit of The Zen of Python, also Explicit is better than implicit, we can also say that we should prefer explicit squeeze to implicit array conversion.
This helps you get rid of useless one dimension arrays like using
[7,8,9] instead of [[[7,8,9]]]
or [[1,2,3],[4,5,6]] instead of [[[[1,2,3],[4,5,6]]]].
Check this link from tutorials point for example.
One example of the importance is when multiplying arrays. Two 2-dimensional arrays will multiply each value at a time
e.g.
>>> x = np.ones((2, 1))*2
>>> y = np.ones((2, 1))*3
>>> x.shape
(2,1)
>>> x*y
array([[ 6.],
[ 6.]])
If you multiply a 1d array by a 2d array then the behaviour is different
>>> z = np.ones((2,))*3
>>> x*z
array([[ 6., 6.],
[ 6., 6.]])
Secondly, you also might want to squeeze the earlier dimensions e.g. a.shape = (1,2,2) to a.shape = (2,2)
When you squeeze a (2,1) array, you get (2,) which works as both (2,1) and (1,2):
>>> a = np.ones(2)
>>> a.shape
(2,)
>>> a.T.shape
(2,)
>>> X = np.ones((2,2))*2
>>> np.dot(a,X)
[4. 4.]
>>> np.dot(X,a)
[4. 4.]
This cannot happen with a (2,1) array:
>>> b = np.ones((2,1))
>>> np.dot(b,X)
Traceback (most recent call last):
ValueError: shapes (2,1) and (2,2) not aligned: 1 (dim 1) != 2 (dim 0)
Related
When I multiply a NxN numpy matrix by a N elements numpy array I get an error saying that the shapes are not aligned.
from numpy import matrix,ones,eye
A = matrix(eye(3))
b = ones(3)
A*b
ValueError: shapes (3,3) and (1,3) not aligned: 3 (dim 1) != 1 (dim 0)
Also trasposing the vector does not solve the issue.
A*b.T
ValueError: shapes (3,3) and (1,3) not aligned: 3 (dim 1) != 1 (dim 0)
This make sense as numpy does not distinguish between column and row vectors so b.T is equal to b.
How can I perform a simple matrix-vector multiplication?
(Don't use np.matrix, it's deprecated. Instead just use 2D arrays for linear algebra.)
Use the matrix multiplication operator #:
In [177]: from numpy import ones,eye
...: A = eye(3)
...: b = ones(3)
...: A # b
Out[177]: array([1., 1., 1.])
Because A is a matrix, Python calls A's __mul__ method to compute A*b, with b as its argument. That is, it calls A.__mul__(b). A numpy matrix insists on making everything a 2-d matrix, so it converts b to a matrix before performing the matrix multiplication. When b is converted to a numpy matrix, the result has shape (1, 3):
In [248]: b
Out[248]: array([1., 1., 1.])
In [249]: np.matrix(b).shape
Out[249]: (1, 3)
Then __mul__ complains that the shapes are not aligned, because A has shape (3, 3) and the converted b has shape (1, 3).
One way to fix this is to ensure that b is 2-d with shape (3, 1) before doing the multiplication:
In [250]: b = ones((3, 1))
In [251]: A * b
Out[251]:
matrix([[1.],
[1.],
[1.]])
In the long term, though, it would be better to modify your code to not use matrix at all, as mentioned by #w-m.
The problem arises from the fact that the operator '*' is performing element-wise multiplication and NOT matrix multiplication as you intend. In python 3 this can be done using the '#' operator, as suggested by w-m. In python 2, however, you must use.
import numpy as np
result = np.dot(A,b)
The '*' operator will try to multiple every element of A with the corresponding element of b. If these are not the same shape you will get an error, which is what you see.
EDIT: I misunderstood OP's question. '*' will work for matrix multiplication if both objects are matrices. However, np.ones(3) produces a numpy array, which is not a numpy matrix object, so it doesn't work and tries to do element-wise multiplication.
if b becomes:
b = np.matrix((1,1,1)).T
then the code will work. It should also be noted that np.dot(A,b) will work even if both A and b are matrix objects, numpy arrays, or a mix of the two, making it the most general solution.
I've seen others post on this, but it's not clear to me if there's a better solution. I've got a 2D NumPy array, and I'd like to append a column to it. For example:
import numpy as np
A = np.array([[2., 3.],[-1., -2.]])
e = np.ones(2)
print(A)
print(e)
B = np.hstack((A,e.reshape((2,1))))
print(B)
does exactly what I want. But is there a way to avoid this clunky use of reshape?
If you want to avoid using reshape then you have to be appending a column of the right dimensions:
e = np.ones((2, 1))
B = np.hstack((A,e))
Note the modification to the call to ones. The reason you have to use reshape at the moment is that numpy does not regard an array of dimension 2 to be the same as an array of dimension (2, 1). The second is a 2D array where the size of one of the dimensions is 1.
My nomination for a direct solution is
np.concatenate((A, e[:, None]), axis=1)
The [:,None] turns e into a (2,1) which can be joined to the (2,2) to produce a (2,3). Reshape does the same, but isn't as syntactically pretty.
Solutions using hstack, vstack, and c_ do the same thing but hide one or more details.
In this case I think column_stack hides the most details.
np.column_stack((A, e))
Under the covers this does:
np.concatenate((A, np.array(e, copy=False, ndmin=2).T), axis=1)
That np.array(... ndmin=2).T is yet another way of doing the reshape.
There are many solutions. I like np.c_ which treats 1d inputs as columns (hence c) resulting in a concise, clutter-free, easy to read:
np.c_[A, e]
# array([[ 2., 3., 1.],
# [-1., -2., 1.]])
As Tim B says, to hstack you need a (2,1) array. Alternatively (keeping your e as a one-dimensional array), vstack to the transpose, and take the transpose:
In [11]: np.vstack((A.T, e)).T
Out[11]:
array([[ 2., 3., 1.],
[-1., -2., 1.]])
I have the following problem with shape of ndarray:
out.shape = (20,)
reference.shape = (20,0)
norm = [out[i] / np.sum(out[i]) for i in range(len(out))]
# norm is a list now so I convert it to ndarray:
norm_array = np.array((norm))
norm_array.shape = (20,30)
# error: operands could not be broadcast together with shapes (20,30) (20,)
diff = np.fabs(norm_array - reference)
How can I change shape of norm_array from (20,30) into (20,) or reference to (20,30), so I can substract them?
EDIT: Can someone explain me, why they have different shape, if I can access both single elements with norm_array[0][0] and reference[0][0] ?
I am not sure what you are trying to do exactly, but here is some information on numpy arrays.
A 1-d numpy array is a row vector with a shape that is a single-valued tuple:
>>> np.array([1,2,3]).shape
(3,)
You can create multidimensional arrays by passing in nested lists. Each sub-list is a 1-d row vector of length 1, and there are 3 of them.
>>> np.array([[1],[2],[3]]).shape
(3,1)
Here is the weird part. You can create the same array, but leave the lists empty. You end up with 3 row vectors of length 0.
>>> np.array([[],[],[]]).shape
(3,0)
This is what you have for you reference array, an array with structure but no values. This brings me back to my original point:
You can't subtract an empty array.
If I make 2 arrays with the shapes you describe, I get an error
In [1856]: norm_array=np.ones((20,30))
In [1857]: reference=np.ones((20,0))
In [1858]: norm_array-reference
...
ValueError: operands could not be broadcast together with shapes (20,30) (20,0)
But it's different from yours. But if I change the shape of reference, the error messages match.
In [1859]: reference=np.ones((20,))
In [1860]: norm_array-reference
...
ValueError: operands could not be broadcast together with shapes (20,30) (20,)
So your (20,0) is wrong. I don't know if you mistyped something or not.
But if I make reference 2d with 1 in the last dimension, broadcasting works, producing a difference that matches (20,30) in shape:
In [1861]: reference=np.ones((20,1))
In [1862]: norm_array-reference
If reference = np.zeros((20,)), then I could use reference[:,None] to add that singleton last dimension.
If reference is (20,), you can't do reference[0][0]. reference[0][0] only works with 2d arrays with at least 1 in the last dim. reference[0,0] is the preferred way of indexing a single element of a 2d array.
So far this is normal array dimensions and broadcasting; something you'll learn with use.
===============
I'm puzzled about the shape of out. If it is (20,), how does norm_array end up as (20,30). out must consist of 20 arrays or lists, each of which has 30 elements.
If out was 2d array, we could normalize without iteration
In [1869]: out=np.arange(12).reshape(3,4)
with the list comprehension:
In [1872]: [out[i]/np.sum(out[i]) for i in range(out.shape[0])]
Out[1872]:
[array([ 0. , 0.16666667, 0.33333333, 0.5 ]),
array([ 0.18181818, 0.22727273, 0.27272727, 0.31818182]),
array([ 0.21052632, 0.23684211, 0.26315789, 0.28947368])]
In [1873]: np.array(_) # and to array
Out[1873]:
array([[ 0. , 0.16666667, 0.33333333, 0.5 ],
[ 0.18181818, 0.22727273, 0.27272727, 0.31818182],
[ 0.21052632, 0.23684211, 0.26315789, 0.28947368]])
Instead take row sums, and tell it to keep it 2d for ease of further use
In [1876]: out.sum(axis=1,keepdims=True)
Out[1876]:
array([[ 6],
[22],
[38]])
now divide
In [1877]: out/out.sum(axis=1,keepdims=True)
Out[1877]:
array([[ 0. , 0.16666667, 0.33333333, 0.5 ],
[ 0.18181818, 0.22727273, 0.27272727, 0.31818182],
[ 0.21052632, 0.23684211, 0.26315789, 0.28947368]])
I'm trying to understand how numpy's broadcasting affects the output of np.allclose.
>>> np.allclose([], [1.])
True
I don't see why that works, but this does not:
>>> np.allclose([], [1., 2.])
ValueError: operands could not be broadcast together with shapes (0,) (2,)
What are the rules here? I can't finding anything in the numpy docs regarding empty arrays.
Broadcasting rules apply to addition as well,
In [7]: np.array([])+np.array([1.])
Out[7]: array([], dtype=float64)
In [8]: np.array([])+np.array([1.,2.])
....
ValueError: operands could not be broadcast together with shapes (0,) (2,)
Let's look at the shapes.
In [9]: np.array([]).shape,np.array([1.]).shape,np.array([1,2]).shape
Out[9]: ((0,), (1,), (2,))
(0,) and (1,) - the (1,) can be adjusted to match the shape of the other array. A 1 dimension can be adjusted to match the other array, from example increased from 1 to 3. But here it was (apparently) adjusted from 1 to 0. I don't usually work with arrays with a 0 dimension, but this looks like a proper generalization of higher dimensions.
Try (0,) and (1,1). The result is (1,0):
In [10]: np.array([])+np.array([[1.]])
Out[10]: array([], shape=(1, 0), dtype=float64)
(0,), (1,1) => (1,0),(1,1) => (1,0)
As for the 2nd case with shapes (0,) and (2,); there isn't any size 1 dimension to adjust, hence the error.
Shapes (0,) and (2,1) do broadcast (to (2,0)):
In [12]: np.array([])+np.array([[1.,2]]).T
Out[12]: array([], shape=(2, 0), dtype=float64)
Broadcasting doesn't affect np.allclose in any other way than it affects any other function.
As in the comment by #cel, [1.] is of dimension 1 and so can be broadcasted to any other dimension, including 0. On the other hand [1., 2.] is of dimension 2 and thus cannot be broadcasted.
Now why allclose([],[1.]) == True? This actually makes sense: it means that all elements in [] are close to 1.. The opposite would mean that there is at least one element in [] which is not close to 1. which is obviously False since there are no elements at all in [].
Another way to think about it is to ask yourself how you would actually code allclose():
def allclose(array, target=1.):
for x in array:
if not isclose(x, target):
return False
return True
This would return True when called with [].
A really stupid question, but I could not figure the right way..
A is a 2 by 2 matrix, and B is a 2 by 1 matrix.
In a 10 iterations loop, B_new=A*B. B_new is 2 by 1.
Save B_new to an output matrix B_final after each iteration. So in the end, B_final is 2 by 10.
However, I have problem of adding B to B_new in a loop. Below is my code, can anyone give me some suggestions?
import numpy as np
a=np.ones(shape=(2,2))
b=np.ones(shape=(2,1))
c_final=np.zeros(shape=(2,10))
for i in range(0,10):
c=np.dot(a,b)
b=c
c_final[:,i]=c
Here is the error message:
c_final[:,i]=c
ValueError: output operand requires a reduction, but reduction is not enabled
The error you're seeing is because when numpy broadcasts c_final[:,i] and np.dot(a,b) together it produces an array with shape (2,2), which then can't be assigned to c_final[:,i] since it has a shape of (2,1). I think it's much clearer if you just play around with it in the interpreter:
>>> import numpy as np
>>> a = np.ones((2,2))
>>> b = np.ones((2,1))
>>> c_final = np.zeros((2,10))
>>> np.dot(a,b)
array([[ 2.],
[ 2.]])
>>> np.dot(a,b).shape
(2, 1)
>>> c_final[:,0]
array([ 0., 0.])
>>> c_final[:,0].shape
(2,)
>>> np.broadcast(c_final[:,0],np.dot(a,b)).shape
(2, 2)
The way around this is to flatten np.dot(a,b) by using np.squeeze or something similar so that when they are broadcast together they produce a 2 element array. For example:
>>> c_final[:,0] = np.dot(a,b).squeeze()
You're not alone in finding the error message unhelpful. Someone filed a ticket about this about a year ago.