Consider the following exercise in Numpy array broadcasting.
import numpy as np
v = np.array([[1.0, 2.0]]).T # column array
A2 = np.random.randn(2,10) # 2D array
A3 = np.random.randn(2,10,10) # 3D
v * A2 # works great
# causes error:
v * A3 # error
I know the Numpy rules for broadcasting, and I'm familiar with bsxfun functionality in Matlab. I understand why attempting to broadcast a (2,1) array into a (2,N,N) array fails, and that I have to reshape the (2,1) array into a (2,1,1) array before this broadcasting goes through.
My question is: is there any way to tell Python to automatically pad the dimensionality of an array when it attempts to broadcast, without me having to specifically tell it the necessary dimension?
I don't want to explicitly couple the (2,1) vector with the multidimensional array it's going to be broadcast against---otherwise I could do something stupid and absurdly ugly like mult_v_A = lambda v,A: v.reshape([v.size] + [1]*(A.ndim-1)) * A. I don't know ahead of time if the "A" array will be 2D or 3D or N-D.
Matlab's bsxfun broadcasting functionality implicitly pads the dimensions as needed, so I'm hoping there's something I could do in Python.
It's ugly, but this will work:
(v.T * A3.T).T
If you don't give it any arguments, transposing reverses the shape tuple, so you can now rely on the broadcasting rules to do their magic. The last transpose returns everything to the right order.
NumPy broadcasting adds additional axes on the left.
So if you arrange your arrays so the shared axes are on the right and the broadcastable axes are on the left, then you can use broadcasting with no problem:
import numpy as np
v = np.array([[1.0, 2.0]]) # shape (1, 2)
A2 = np.random.randn(10,2) # shape (10, 2)
A3 = np.random.randn(10,10,2) # shape (10, 10, 2)
v * A2 # shape (10, 2)
v * A3 # shape (10, 10, 2)
Related
Can anyone explain the condition in which one would require to reshape along axis=0? Please see the example below, with a given numpy array:
a=np.array([1,2,3,4,5,6])
[1,2,3,4,5,6]
(Reshaping follows below)
a1 = np.expand_dims(a, axis=0)
[[1,2,3,4,5,6]]
The expansion typically happens when we are using a function, which performs operations on an (m, n) array, to process a special case where m = 1.
If the shape of the given data is (n,) we have to expand_dims along the first axis so that the shape is (1, n).
Some functions are nice enough to take special care of the (n,) situation. But sometimes we have to do the conversion, (n,) → (1, n), ourselves.
I'm currently learning about broadcasting in Numpy and in the book I'm reading (Python for Data Analysis by Wes McKinney the author has mentioned the following example to "demean" a two-dimensional array:
import numpy as np
arr = np.random.randn(4, 3)
print(arr.mean(0))
demeaned = arr - arr.mean(0)
print(demeaned)
print(demeand.mean(0))
Which effectively causes the array demeaned to have a mean of 0.
I had the idea to apply this to an image-like, three-dimensional array:
import numpy as np
arr = np.random.randint(0, 256, (400,400,3))
demeaned = arr - arr.mean(2)
Which of course failed, because according to the broadcasting rule, the trailing dimensions have to match, and that's not the case here:
print(arr.shape) # (400, 400, 3)
print(arr.mean(2).shape) # (400, 400)
Now, i have gotten it to work mostly, by substracting the mean from every single index in the third dimension of the array:
demeaned = np.ones(arr.shape)
for i in range(3):
demeaned[...,i] = arr[...,i] - means
print(demeaned.mean(0))
At this point, the returned values are very close to zero and i think, that's a precision error. Am i actually right with this thought or is there another caveat, that i missed?
Also, this doesn't seam to be the cleanest, most 'numpy'-way to achieve what i wanted to achieve. Is there a function or a principle that i can make use of to improve the code?
As of numpy version 1.7.0, np.mean, and several other functions, accept a tuple in their axis parameter. This means that you can perform the operation on the planes of the image all at once:
m = arr.mean(axis=(0, 1))
This mean will have shape (3,), with one element for each plane of the image.
If you want to subtract the means of each pixel individually, you have to remember that broadcasting aligns shape tuples on the right edge. That means that you need to insert an extra dimension:
n = arr.mean(axis=2)
n = n.reshape(*n.shape, 1)
Or
n = arr.mean(axis=2)[..., None]
Try np.apply_along_axis().
np.apply_along_axis(lambda x: x - np.mean(x), 2, arr)
Output: you get the array of the same shape where each cell is demeaned in the dimension you want (the second parameter, here it is 2).
When I do p = np.zeros((3,1)) I get a matrix in the shape (3, 1).
Sometimes when I am working with NumPy arrays that are nx1, however, I get that their shape is (3,).
How can I make these (3,) shaped arrays into (3,1)?
i.e. here is a minimum runnable program:
a = np.random.randn(3)
>>a.shape
(3,)
I want it to be (3,1). I know I could just call with arguments 3,1 but this is just an example, sometimes I can't control the generative process but only manipulate the output.
Just check the shape and add another axis if needed:
if len(a.shape) == 1:
a = a[..., np.newaxis]
# or this, if you need more generality:
a = a.reshape(a.shape + (1,) * (desired_dimensions - len(a.shape)))
There's an np.atleast_2d function, but it would produce a 1-by-3 array instead of 3-by-1.
Consider the following simple example:
X = numpy.zeros([10, 4]) # 2D array
x = numpy.arange(0,10) # 1D array
X[:,0] = x # WORKS
X[:,0:1] = x # returns ERROR:
# ValueError: could not broadcast input array from shape (10) into shape (10,1)
X[:,0:1] = (x.reshape(-1, 1)) # WORKS
Can someone explain why numpy has vectors of shape (N,) rather than (N,1) ?
What is the best way to do the casting from 1D array into 2D array?
Why do I need this?
Because I have a code which inserts result x into a 2D array X and the size of x changes from time to time so I have X[:, idx1:idx2] = x which works if x is 2D too but not if x is 1D.
Do you really need to be able to handle both 1D and 2D inputs with the same function? If you know the input is going to be 1D, use
X[:, i] = x
If you know the input is going to be 2D, use
X[:, start:end] = x
If you don't know the input dimensions, I recommend switching between one line or the other with an if, though there might be some indexing trick I'm not aware of that would handle both identically.
Your x has shape (N,) rather than shape (N, 1) (or (1, N)) because numpy isn't built for just matrix math. ndarrays are n-dimensional; they support efficient, consistent vectorized operations for any non-negative number of dimensions (including 0). While this may occasionally make matrix operations a bit less concise (especially in the case of dot for matrix multiplication), it produces more generally applicable code for when your data is naturally 1-dimensional or 3-, 4-, or n-dimensional.
I think you have the answer already included in your question. Numpy allows the arrays be of any dimensionality (while afaik Matlab prefers two dimensions where possible), so you need to be correct with this (and always distinguish between (n,) and (n,1)). By giving one number as one of the indices (like 0 in 3rd row), you reduce the dimensionality by one. By giving a range as one of the indices (like 0:1 in 4th row), you don't reduce the dimensionality.
Line 3 makes perfect sense for me and I would assign to the 2-D array this way.
Here are two tricks that make the code a little shorter.
X = numpy.zeros([10, 4]) # 2D array
x = numpy.arange(0,10) # 1D array
X.T[:1, :] = x
X[:, 2:3] = x[:, None]
I generally use MATLAB and Octave, and i recently switching to python numpy.
In numpy when I define an array like this
>>> a = np.array([[2,3],[4,5]])
it works great and size of the array is
>>> a.shape
(2, 2)
which is also same as MATLAB
But when i extract the first entire column and see the size
>>> b = a[:,0]
>>> b.shape
(2,)
I get size (2,), what is this? I expect the size to be (2,1). Perhaps i misunderstood the basic concept. Can anyone make me clear about this??
A 1D numpy array* is literally 1D - it has no size in any second dimension, whereas in MATLAB, a '1D' array is actually 2D, with a size of 1 in its second dimension.
If you want your array to have size 1 in its second dimension you can use its .reshape() method:
a = np.zeros(5,)
print(a.shape)
# (5,)
# explicitly reshape to (5, 1)
print(a.reshape(5, 1).shape)
# (5, 1)
# or use -1 in the first dimension, so that its size in that dimension is
# inferred from its total length
print(a.reshape(-1, 1).shape)
# (5, 1)
Edit
As Akavall pointed out, I should also mention np.newaxis as another method for adding a new axis to an array. Although I personally find it a bit less intuitive, one advantage of np.newaxis over .reshape() is that it allows you to add multiple new axes in an arbitrary order without explicitly specifying the shape of the output array, which is not possible with the .reshape(-1, ...) trick:
a = np.zeros((3, 4, 5))
print(a[np.newaxis, :, np.newaxis, ..., np.newaxis].shape)
# (1, 3, 1, 4, 5, 1)
np.newaxis is just an alias of None, so you could do the same thing a bit more compactly using a[None, :, None, ..., None].
* An np.matrix, on the other hand, is always 2D, and will give you the indexing behavior you are familiar with from MATLAB:
a = np.matrix([[2, 3], [4, 5]])
print(a[:, 0].shape)
# (2, 1)
For more info on the differences between arrays and matrices, see here.
Typing help(np.shape) gives some insight in to what is going on here. For starters, you can get the output you expect by typing:
b = np.array([a[:,0]])
Basically numpy defines things a little differently than MATLAB. In the numpy environment, a vector only has one dimension, and an array is a vector of vectors, so it can have more. In your first example, your array is a vector of two vectors, i.e.:
a = np.array([[vec1], [vec2]])
So a has two dimensions, and in your example the number of elements in both dimensions is the same, 2. Your array is therefore 2 by 2. When you take a slice out of this, you are reducing the number of dimensions that you have by one. In other words, you are taking a vector out of your array, and that vector only has one dimension, which also has 2 elements, but that's it. Your vector is now 2 by _. There is nothing in the second spot because the vector is not defined there.
You could think of it in terms of spaces too. Your first array is in the space R^(2x2) and your second vector is in the space R^(2). This means that the array is defined on a different (and bigger) space than the vector.
That was a lot to basically say that you took a slice out of your array, and unlike MATLAB, numpy does not represent vectors (1 dimensional) in the same way as it does arrays (2 or more dimensions).