Can someone explain exactly what the axis parameter in NumPy does?
I am terribly confused.
I'm trying to use the function myArray.sum(axis=num)
At first I thought if the array is itself 3 dimensions, axis=0 will return three elements, consisting of the sum of all nested items in that same position. If each dimension contained five dimensions, I expected axis=1 to return a result of five items, and so on.
However this is not the case, and the documentation does not do a good job helping me out (they use a 3x3x3 array so it's hard to tell what's happening)
Here's what I did:
>>> e
array([[[1, 0],
[0, 0]],
[[1, 1],
[1, 0]],
[[1, 0],
[0, 1]]])
>>> e.sum(axis = 0)
array([[3, 1],
[1, 1]])
>>> e.sum(axis=1)
array([[1, 0],
[2, 1],
[1, 1]])
>>> e.sum(axis=2)
array([[1, 0],
[2, 1],
[1, 1]])
>>>
Clearly the result is not intuitive.
Clearly,
e.shape == (3, 2, 2)
Sum over an axis is a reduction operation so the specified axis disappears. Hence,
e.sum(axis=0).shape == (2, 2)
e.sum(axis=1).shape == (3, 2)
e.sum(axis=2).shape == (3, 2)
Intuitively, we are "squashing" the array along the chosen axis, and summing the numbers that get squashed together.
To understand the axis intuitively, refer the picture below (source: Physics Dept, Cornell Uni)
The shape of the (boolean) array in the above figure is shape=(8, 3). ndarray.shape will return a tuple where the entries correspond to the length of the particular dimension. In our example, 8 corresponds to length of axis 0 whereas 3 corresponds to length of axis 1.
If someone need this visual description:
There are good answers for visualization however it might help to think purely from analytical perspective.
You can create array of arbitrary dimension with numpy.
For example, here's a 5-dimension array:
>>> a = np.random.rand(2, 3, 4, 5, 6)
>>> a.shape
(2, 3, 4, 5, 6)
You can access any element of this array by specifying indices. For example, here's the first element of this array:
>>> a[0, 0, 0, 0, 0]
0.0038908603263844155
Now if you take out one of the dimensions, you get number of elements in that dimension:
>>> a[0, 0, :, 0, 0]
array([0.00389086, 0.27394775, 0.26565889, 0.62125279])
When you apply a function like sum with axis parameter, that dimension gets eliminated and array of dimension less than original gets created. For each cell in new array, the operator will get list of elements and apply the reduction function to get a scaler.
>>> np.sum(a, axis=2).shape
(2, 3, 5, 6)
Now you can check that the first element of this array is sum of above elements:
>>> np.sum(a, axis=2)[0, 0, 0, 0]
1.1647502999560164
>>> a[0, 0, :, 0, 0].sum()
1.1647502999560164
The axis=None has special meaning to flatten out the array and apply function on all numbers.
Now you can think about more complex cases where axis is not just number but a tuple:
>>> np.sum(a, axis=(2,3)).shape
(2, 3, 6)
Note that we use same technique to figure out how this reduction was done:
>>> np.sum(a, axis=(2,3))[0,0,0]
7.889432081931909
>>> a[0, 0, :, :, 0].sum()
7.88943208193191
You can also use same reasoning for adding dimension in array instead of reducing dimension:
>>> x = np.random.rand(3, 4)
>>> y = np.random.rand(3, 4)
# New dimension is created on specified axis
>>> np.stack([x, y], axis=2).shape
(3, 4, 2)
>>> np.stack([x, y], axis=0).shape
(2, 3, 4)
# To retrieve item i in stack set i in that axis
Hope this gives you generic and full understanding of this important parameter.
Some answers are too specific or do not address the main source of confusion. This answer attempts to provide a more general but simple explanation of the concept, with a simple example.
The main source of confusion is related to expressions such as "Axis along which the means are computed", which is the documentation of the argument axis of the numpy.mean function. What the heck does "along which" even mean here? "Along which" essentially means that you will sum the rows (and divide by the number of rows, given that we are computing the mean), if the axis is 0, and the columns, if the axis is 1. In the case of axis is 0 (or 1), the rows can be scalars or vectors or even other multi-dimensional arrays.
In [1]: import numpy as np
In [2]: a=np.array([[1, 2], [3, 4]])
In [3]: a
Out[3]:
array([[1, 2],
[3, 4]])
In [4]: np.mean(a, axis=0)
Out[4]: array([2., 3.])
In [5]: np.mean(a, axis=1)
Out[5]: array([1.5, 3.5])
So, in the example above, np.mean(a, axis=0) returns array([2., 3.]) because (1 + 3)/2 = 2 and (2 + 4)/2 = 3. It returns an array of two numbers because it returns the mean of the rows for each column (and there are two columns).
Both 1st and 2nd reply is great for understanding ndarray concept in numpy. I am giving a simple example.
And according to this image by #debaonline4u
https://i.stack.imgur.com/O5hBF.jpg
Suppose , you have an 2D array -
[1, 2, 3]
[4, 5, 6]
In, numpy format it will be -
c = np.array([[1, 2, 3],
[4, 5, 6]])
Now,
c.ndim = 2 (rows/axis=0)
c.shape = (2,3) (axis0, axis1)
c.sum(axis=0) = [1+4, 2+5, 3+6] = [5, 7, 9] (sum of the 1st elements of each rows, so along axis0)
c.sum(axis=1) = [1+2+3, 4+5+6] = [6, 15] (sum of the elements in a row, so along axis1)
So for your 3D array,
Related
I have a numpy array, a, of shape (2, 5, 2). I need to get 2 elements from the array at locations [0, 1, 0] and [1, 3, 0]. To retrieve these elements, I have an array, b, with entries [1, 3]. When I use a[:,b], I get 2x2 matrix (as expected). How can I change the indexing to return the required vector instead? Following is an MWE.
import numpy as np
a = np.random.randn(2, 5, 2)
b = np.random.randint(0, 5, 2)
c = a[:, b]
c.shape # prints (2, 2), while I only need a[0, b[0]] and a[1, b[1]].
PS: This problem can be solved using a for loop. However in my actual code the index is in the order of 1000 and therefore not feasible.
As pointed out in the comments, one can do c = a[np.arange(a.shape[0]), b] to obtain the required result.
I am trying to get a good understanding on broadcasting rules in numpy, but I have noticed I firstly need to get a good understanding on what 1-dimensional numpy array is. I found multiple sources saying that 1-dimensional numpy array is neither a horizontal or vertical vector. From that I'd expect that it behaves differently depending on an operation done and other component of the operation. But I can't really find a case when 1-dimensional array would behave like a column vector. For example:
a = np.arange(3)
b = np.arange(3)[:, np.newaxis]
a + b
array([[0, 1, 2],
[1, 2, 3],
[2, 3, 4]])
which indicates that a behaves like a horizontal vector. On the other hand, if we add it to horizontal vector b:
a = np.arange(3)
b = np.arange(3)[np.newaxis, :]
a + b
array([[0, 1, 4]])
a still behaves like a horizontal vector. On the other hand a seems to be indifferent to transformation with .T. So my question is - does 1-dimensional numpy arrays always mimic the horizontal vector behaviour? If not, what are the cases when they behave like standard vertical vector?
What you just came across is known as right align property of numpy arrays. When you have a vector of shape (n, ) and some other array of shape (a, b, c, d, ..., z) then numpy will always try to broadcast the vector to shape (1, 1, ...., n) and finally check if n is broadcastable with z (in other words, z is a multiple of n).
Now, if you don't want the behaviour, you will have to tell numpy explicitly, how do you want to broadcast with the other array with which you are operating by adding axis to the vector using np.newaxis. You can also use the function np.broadcast_arrays to get the broadcasted arrays.
For example,
import numpy as np
a = np.array([1, 2, 3])
b = np.eye(3)
# broadcasts a to shape (1, 3) first
# adds the vector a to rows of b
# [[1, 0, 0] [[1, 2, 3]
# [0, 1, 0] + [1, 2, 3]
# [0, 0, 1]] [1, 2, 3]]
print(a + b)
# Tell numpy explicitly, how you want
# your vector to be broadcasted
# Now, a is first broadcasted to shape (3, 1)
# and the vector a is added to the columns of b
# [[1, 0, 0] [[1, 1, 1]
# [0, 1, 0] + [2, 2, 2]
# [0, 0, 1]] [3, 3, 3]]
print(b + a[np.newaxis, :])
Let's say I have a numpy.ndarray with shape (2,3,2) as below,
arr = np.array([[[1,3], [2,5], [1,2]],[[3,3], [6,5], [5,2]]])
I want to reshape it in such a way that:
arr.shape == (2,3)
arr == [[(1,3), (2,5), (1,2)],[(3,3), (6,5), (5,2)]]
and
each value of arr is a size 2 tuple
The reason I want to do this is that I want to take the minimum along axis 0 of the 3dimensional array, but I want to preserve the value that the min of the rows in paired with.
arr = np.array(
[[[1, 4],
[2, 1],
[5, 2]],
[[3, 3],
[6, 5],
[1, 7]]])
print(np.min(arr, axis=0))
>>> [[1,3],
[2,1],
[1,2]]
>>>Should be
[[1,4],
[2,1],
[1,7]]
If the array contained tuples, it would be 2 dimensional, and the comparison operator for minimize would still function correctly,
so I would get the correct result. But I haven't found any way to do this besides iterating over the arrays, which is inefficient and obvious in implementation.
Is it possible to perform this conversion efficiently in numpy?
Don't use tuples at all - just view it as a structured array, which supports the lexical comparison you're after:
a = np.array([[[1,3], [2,5], [1,2]],[[3,3], [6,5], [5,2]]])
a_pairs = a.view([('f0', a.dtype), ('f1', a.dtype)]).squeeze(axis=-1)
min_pair = np.partition(a_pairs, 0, axis=0)[0] # min doesn't work on structured types :(
array([(1, 4), (2, 1), (1, 7)],
dtype=[('f0', '<i4'), ('f1', '<i4')])
First, let's find out which pairs to take:
first_eq = arr[0,:,0] == arr[1,:,0]
which_compare = np.where(first_eq, 1, 0)[0]
winner = arr[:,:,which_compare].argmin(axis=0)
Here, first_eq is True where the first elements match, so we would need to compare the second elements. It's [False, False, False] in your example. which_compare then is [0, 0, 0] (because the first element of each pair is what we will compare). Finally, winner tells us which of the two pairs to choose along the second axis. It is [0, 0, 1].
The last step is to extract the winners:
arr[winner, np.arange(arr.shape[1])]
That is, take the winner (0 or 1) at each point along the second axis.
Here's one way -
# Get each row being fused with scaling based on scale being decided
# based off the max values from the second col. Get argmin indices.
idx = (arr[...,1] + arr[...,0]*(arr[...,1].max()+1)).argmin(0)
# Finally use advanced-indexing to get those rows off array
out = arr[idx, np.arange(arr.shape[1])]
Sample run -
In [692]: arr
Out[692]:
array([[[3, 4],
[2, 1],
[5, 2]],
[[3, 3],
[6, 5],
[5, 1]]])
In [693]: out
Out[693]:
array([[3, 3],
[2, 1],
[5, 1]])
I am trying to get the indices that would sort each column of an array using the function argsort. However, it keeps simply returning zeros instead of the true indices. For example:
x = np.matrix([[5, 2, 6], [3, 4, 1]])
print(x)
print(x[:,0])
print(x[:,1])
print(x[:,2])
print(x[:,0].argsort())
print(x[:,1].argsort())
print(x[:,2].argsort())
I am expecting this to return three arrays. [1 0], [0 1] and [1 0] denoting the indices of each column if it were sorted, however, instead I get three arrays that all contain zeros.
Any help much appreciated!
Indexing a matrix with a slice always returns another 2-d matrix. (This behavior is not the same as for a regular numpy array.) See, for example, the output of x[:,0]:
In [133]: x[:,0]
Out[133]:
matrix([[5],
[3]])
x[:,0] is a matrix with shape (2, 1).
To argsort the first (and only) column of that matrix, you have to tell argsort to use the first axis:
In [135]: x[:,0].argsort(axis=0)
Out[135]:
matrix([[1],
[0]])
The default (axis=-1) is to use the last axis, and since the rows in that matrix have length 1, the result when axis is not given is a column of zeros.
By the way, you can do all the columns at once:
In [138]: x
Out[138]:
matrix([[5, 2, 6],
[3, 4, 1]])
In [139]: x.argsort(axis=0)
Out[139]:
matrix([[1, 0, 1],
[0, 1, 0]])
I have a tensor with the shape (4, 3, 20). When I do X[:, 0, :].shape I get (4, 20). When I do X[:, [0,2,0,1], :].shape I get (4, 4, 20).
What I have is a list of indexes representing the second dimension of my tensor. I want to get a two-dimensional matrix like I get when I do X[:, 0, :] but I have different indexes for the second dimension instead of only one. How do I do that?
Your question is unclear, but I'll make a guess
In [58]: X=np.arange(24).reshape(4,3,2)
In [59]: X[range(4),[0,2,0,1],:]
Out[59]:
array([[ 0, 1],
[10, 11],
[12, 13],
[20, 21]])
This picks row 0 from the 1st plane; row 2 from the 2nd, etc. The result has the same shape as X[:,0,:], but values are pulled from different 1st dimension planes.
In [61]: X[:,0,:]
Out[61]:
array([[ 0, 1], # same
[ 6, 7],
[12, 13], # same
[18, 19]])
I think you are looking for np.squeeze. So, for cases when the indexing list, say L has just one element and upon indexing the input array with it would result in a 3D array with a singleton second dimension (dimension of length 1), would result in a 2D output with that squeez-ing. For L with more than one element, the indexing would result in a 3D array without any singleton dimension and thus, no change with that squeez-ing and hence the desired output. Thus, the solution with it would be -
np.squeeze(X[:,L,:])
Sample run to test out shapes on a random array -
In [25]: A = np.random.rand(4,3,20)
In [26]: L = [0]
In [27]: np.squeeze(A[:,L,:]).shape
Out[27]: (4, 20)
In [28]: L = [0,2,0,1]
In [29]: np.squeeze(A[:,L,:]).shape
Out[29]: (4, 4, 20)