I am wondering why in numpy there are one dimensional array of dimension (length, 1) and also one dimensional array of dimension (length, ) w/o a second value.
I am running into this quite frequently, e.g. when using np.concatenate() which then requires a reshape step beforehand (or I could directly use hstack/vstack).
I can't think of a reason why this behavior is desirable. Can someone explain?
Edit:
It was suggested by one of the comments that my question is a possible duplicate. I am more interested in the underlying working logic of Numpy and not that there is a distinction between 1d and 2d arrays which I think is the point of the mentioned thread.
The data of a ndarray is stored as a 1d buffer - just a block of memory. The multidimensional nature of the array is produced by the shape and strides attributes, and the code that uses them.
The numpy developers chose to allow for an arbitrary number of dimensions, so the shape and strides are represented as tuples of any length, including 0 and 1.
In contrast MATLAB was built around FORTRAN programs that were developed for matrix operations. In the early days everything in MATLAB was a 2d matrix. Around 2000 (v3.5) it was generalized to allow more than 2d, but never less. The numpy np.matrix still follows that old 2d MATLAB constraint.
If you come from a MATLAB world you are used to these 2 dimensions, and the distinction between a row vector and column vector. But in math and physics that isn't influenced by MATLAB, a vector is a 1d array. Python lists are inherently 1d, as are c arrays. To get 2d you have to have lists of lists or arrays of pointers to arrays, with x[1][2] style of indexing.
Look at the shape and strides of this array and its variants:
In [48]: x=np.arange(10)
In [49]: x.shape
Out[49]: (10,)
In [50]: x.strides
Out[50]: (4,)
In [51]: x1=x.reshape(10,1)
In [52]: x1.shape
Out[52]: (10, 1)
In [53]: x1.strides
Out[53]: (4, 4)
In [54]: x2=np.concatenate((x1,x1),axis=1)
In [55]: x2.shape
Out[55]: (10, 2)
In [56]: x2.strides
Out[56]: (8, 4)
MATLAB adds new dimensions at the end. It orders its values like a order='F' array, and can readily change a (n,1) matrix to a (n,1,1,1). numpy is default order='C', and readily expands an array dimension at the start. Understanding this is essential when taking advantage of broadcasting.
Thus x1 + x is a (10,1)+(10,) => (10,1)+(1,10) => (10,10)
Because of broadcasting a (n,) array is more like a (1,n) one than a (n,1) one. A 1d array is more like a row matrix than a column one.
In [64]: np.matrix(x)
Out[64]: matrix([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
In [65]: _.shape
Out[65]: (1, 10)
The point with concatenate is that it requires matching dimensions. It does not use broadcasting to adjust dimensions. There are a bunch of stack functions that ease this constraint, but they do so by adjusting the dimensions before using concatenate. Look at their code (readable Python).
So a proficient numpy user needs to be comfortable with that generalized shape tuple, including the empty () (0d array), (n,) 1d, and up. For more advanced stuff understanding strides helps as well (look for example at the strides and shape of a transpose).
Much of it is a matter of syntax. This tuple (x) isn't a tuple at all (just a redundancy). (x,), however, is.
The difference between (x,) and (x,1) goes even further. You can take a look into the examples of previous questions like this. Quoting the example from it, this is an 1D numpy array:
>>> np.array([1, 2, 3]).shape
(3,)
But this one is 2D:
>>> np.array([[1, 2, 3]]).shape
(1, 3)
Reshape does not make a copy unless it needs to so it should be safe to use.
Related
Can anyone explain the condition in which one would require to reshape along axis=0? Please see the example below, with a given numpy array:
a=np.array([1,2,3,4,5,6])
[1,2,3,4,5,6]
(Reshaping follows below)
a1 = np.expand_dims(a, axis=0)
[[1,2,3,4,5,6]]
The expansion typically happens when we are using a function, which performs operations on an (m, n) array, to process a special case where m = 1.
If the shape of the given data is (n,) we have to expand_dims along the first axis so that the shape is (1, n).
Some functions are nice enough to take special care of the (n,) situation. But sometimes we have to do the conversion, (n,) → (1, n), ourselves.
Assume we want to compute the dot product of a matrix and a column vector:
So in Numpy/Python here we go:
a=numpy.asarray([[1,2,3], [4,5,6], [7,8,9]])
b=numpy.asarray([[2],[1],[3]])
a.dot(b)
Results in:
array([[13],
[31],
[49]])
So far, so good, however why is this also working?
b=numpy.asarray([2,1,3])
a.dot(b)
Results in:
array([13, 31, 49])
I would expect that [2,1,3] is a row vector (which requires a transpose to apply the dot product), but Numpy seems to see arrays by default as column vectors (in case of matrix multiplication)?
How does this work?
EDIT:
And why is:
b=numpy.asarray([2,1,3])
b.transpose()==b
So the matrix dot vector array does work (so then it sees it as a column vector), however other operations (transpose) does not work. This is not really consistent design isn't it?
Let's first understand how the dot operation is defined in numpy.
(Leaving broadcasting rules out of the discussion, for simplicity) you can perform dot(A,B) if the last dimension of A (i.e. A.shape[-1]) is the same as the next-to-last dimension of B (i.e. B.shape[-2]) if B.ndim>=2, and simply the dimension of B if B.ndim==1.
In other words, if A.shape=(N1,...,Nk,X) and B.shape=(M1,...,M(j-1),X,Mj) (note the common X). The resulting array will have the shape (N1,...,Nk,M1,...,Mj) (note that X was dropped).
Or, if A.shape=(N1,...,Nk,X) and B.shape=(X,). The resulting array will have the shape (N1,...,Nk) (note that X was dropped).
Your examples work because they satisfy the rules (the first example satisfies the first, the second satisfies the second):
a=numpy.asarray([[1,2,3], [4,5,6], [7,8,9]])
b=numpy.asarray([[2],[1],[3]])
a.shape, b.shape, '->', a.dot(b).shape # X=3
=> ((3, 3), (3, 1), '->', (3, 1))
b=numpy.asarray([2,1,3])
a.shape, b.shape, '->', a.dot(b).shape # X=3
=> ((3, 3), (3,), '->', (3,))
My recommendation is that, when using numpy, don't think in terms of "row/column vectors", and if possible don't think in terms of "vectors" at all, but in terms of "an array with shape S". This means that both row vectors and column vectors are simply "1dim arrays". As far as numpy is concerned, they are one and the same.
This should also make it clear why in your case b.transponse() is the same as b. b being a 1dim array, when transposed, remains a 1dim array. Transpose doesn't affect 1dim arrays.
I have a 'row' vector cast as a numpy ndarray. I would simply like to make it a 'column' vector (I don't care too much about the type as long as it is compatible with matplotlib). Here is an example of what I'm trying:
import numpy as np
a = np.ndarray(shape=(1,4), dtype=float, order='F')
print(a.shape)
a.T #I think this performs the transpose?
print(a.shape)
The output looks like this:
(1, 4)
(1, 4)
I was hoping to get:
(1, 4)
(4, 1)
Can someone point me in the right direction? I have seen that the transpose in numpy doesn't do anything to a 1D array. But is this a 1D array?
Transposing an array does not happen in place. Writing a.T creates a view of the transpose of the array a, but this view is then lost immediately since no variable is assigned to it. a remains unchanged.
You need to write a = a.T to bind the name a to the transpose:
>>> a = a.T
>>> a.shape
(4, 1)
In your example a is indeed a 2D array. Transposing a 1D array (with shape (n,)) does not change that array at all.
you can alter the shape 'in place' which will be the same as a.T for (1,4) but see the comment by Mr E whether it's needed. i.e.
...
print(a.shape)
a.shape = (4, 1)
print(a.shape)
You probably don't want or need the singular dimension, unless you are trying to force a broadcasting operation.
Link
You can treat rank-1 arrays as either row or column vectors. dot(A,v)
treats v as a column vector, while dot(v,A) treats v as a row vector.
This can save you having to type a lot of transposes.
Consider the following simple example:
X = numpy.zeros([10, 4]) # 2D array
x = numpy.arange(0,10) # 1D array
X[:,0] = x # WORKS
X[:,0:1] = x # returns ERROR:
# ValueError: could not broadcast input array from shape (10) into shape (10,1)
X[:,0:1] = (x.reshape(-1, 1)) # WORKS
Can someone explain why numpy has vectors of shape (N,) rather than (N,1) ?
What is the best way to do the casting from 1D array into 2D array?
Why do I need this?
Because I have a code which inserts result x into a 2D array X and the size of x changes from time to time so I have X[:, idx1:idx2] = x which works if x is 2D too but not if x is 1D.
Do you really need to be able to handle both 1D and 2D inputs with the same function? If you know the input is going to be 1D, use
X[:, i] = x
If you know the input is going to be 2D, use
X[:, start:end] = x
If you don't know the input dimensions, I recommend switching between one line or the other with an if, though there might be some indexing trick I'm not aware of that would handle both identically.
Your x has shape (N,) rather than shape (N, 1) (or (1, N)) because numpy isn't built for just matrix math. ndarrays are n-dimensional; they support efficient, consistent vectorized operations for any non-negative number of dimensions (including 0). While this may occasionally make matrix operations a bit less concise (especially in the case of dot for matrix multiplication), it produces more generally applicable code for when your data is naturally 1-dimensional or 3-, 4-, or n-dimensional.
I think you have the answer already included in your question. Numpy allows the arrays be of any dimensionality (while afaik Matlab prefers two dimensions where possible), so you need to be correct with this (and always distinguish between (n,) and (n,1)). By giving one number as one of the indices (like 0 in 3rd row), you reduce the dimensionality by one. By giving a range as one of the indices (like 0:1 in 4th row), you don't reduce the dimensionality.
Line 3 makes perfect sense for me and I would assign to the 2-D array this way.
Here are two tricks that make the code a little shorter.
X = numpy.zeros([10, 4]) # 2D array
x = numpy.arange(0,10) # 1D array
X.T[:1, :] = x
X[:, 2:3] = x[:, None]
I generally use MATLAB and Octave, and i recently switching to python numpy.
In numpy when I define an array like this
>>> a = np.array([[2,3],[4,5]])
it works great and size of the array is
>>> a.shape
(2, 2)
which is also same as MATLAB
But when i extract the first entire column and see the size
>>> b = a[:,0]
>>> b.shape
(2,)
I get size (2,), what is this? I expect the size to be (2,1). Perhaps i misunderstood the basic concept. Can anyone make me clear about this??
A 1D numpy array* is literally 1D - it has no size in any second dimension, whereas in MATLAB, a '1D' array is actually 2D, with a size of 1 in its second dimension.
If you want your array to have size 1 in its second dimension you can use its .reshape() method:
a = np.zeros(5,)
print(a.shape)
# (5,)
# explicitly reshape to (5, 1)
print(a.reshape(5, 1).shape)
# (5, 1)
# or use -1 in the first dimension, so that its size in that dimension is
# inferred from its total length
print(a.reshape(-1, 1).shape)
# (5, 1)
Edit
As Akavall pointed out, I should also mention np.newaxis as another method for adding a new axis to an array. Although I personally find it a bit less intuitive, one advantage of np.newaxis over .reshape() is that it allows you to add multiple new axes in an arbitrary order without explicitly specifying the shape of the output array, which is not possible with the .reshape(-1, ...) trick:
a = np.zeros((3, 4, 5))
print(a[np.newaxis, :, np.newaxis, ..., np.newaxis].shape)
# (1, 3, 1, 4, 5, 1)
np.newaxis is just an alias of None, so you could do the same thing a bit more compactly using a[None, :, None, ..., None].
* An np.matrix, on the other hand, is always 2D, and will give you the indexing behavior you are familiar with from MATLAB:
a = np.matrix([[2, 3], [4, 5]])
print(a[:, 0].shape)
# (2, 1)
For more info on the differences between arrays and matrices, see here.
Typing help(np.shape) gives some insight in to what is going on here. For starters, you can get the output you expect by typing:
b = np.array([a[:,0]])
Basically numpy defines things a little differently than MATLAB. In the numpy environment, a vector only has one dimension, and an array is a vector of vectors, so it can have more. In your first example, your array is a vector of two vectors, i.e.:
a = np.array([[vec1], [vec2]])
So a has two dimensions, and in your example the number of elements in both dimensions is the same, 2. Your array is therefore 2 by 2. When you take a slice out of this, you are reducing the number of dimensions that you have by one. In other words, you are taking a vector out of your array, and that vector only has one dimension, which also has 2 elements, but that's it. Your vector is now 2 by _. There is nothing in the second spot because the vector is not defined there.
You could think of it in terms of spaces too. Your first array is in the space R^(2x2) and your second vector is in the space R^(2). This means that the array is defined on a different (and bigger) space than the vector.
That was a lot to basically say that you took a slice out of your array, and unlike MATLAB, numpy does not represent vectors (1 dimensional) in the same way as it does arrays (2 or more dimensions).