I am working on an AI project of my own where I append the result(a list) of each layer( which can vary in size) to a list. With lists this worked fine, but I transitioned this to numpy arrays for scalability and I couldn't not get this done. Here is what i want to do.
a = np.array([[1,2,3],[4,5,6]])
b= np.array([7,8])
I want to make
a = np.array([[1,2,3],[4,5,6],[7,8]])
I have tried append and concatenate but those seemed to fail, giving an error that they must be of the same size. Your help is appreciated.
Assuming you want to append b to a as an additional column, the following takes advantage of numpy.c_:
import numpy as np
new_a = np.c_[a,b]
print(new_a)
# array([[1, 2, 3, 7],
# [4, 5, 6, 8]])
Otherwise, be careful with numpy.array objects as the shape of them matters!
As what coldspeed mentioned in the comment for the question, a = np.array([[1,2,3],[4,5,6],[7,8]]) does not generate a compact scalar dtype array but instead generates an array of lists.
Using a.dtype will return the data type of your array's elements.
a = np.array([[1,2,3],[4,5,6]]) returns int32
a = np.array([[1,2,3],[4,5,6],[7,8]]) returns object
Aside from that, the functions Append and Concatenate require that your arrays are of the same length in the dimension you are joining them. Perhaps what you could do is to pad array b with 'nan' to fit the required length:b = np.array([[7,8,np.nan]]) which will give you
[[ 1. 2. 3.]
[ 4. 5. 6.]
[ 7. 8. nan]]
As your number of layers increases, you might need to check for the length of b and pad array a instead.
If your intention is to add on array b as a column, then HAL 9001's answer would be better.
Related
Why can't I get the transpose when of alpha but I can get it for beta? What do the additional [] do?
alpha = np.array([1,2,3,4])
alpha.shape
alpha.T.shape
beta = np.array([[1,2,3,4]])
beta.shape
beta.T.shape
From the documention (link):
Transposing a 1-D array returns an unchanged view of the original array.
The array [1,2,3,4] is 1-D while the array [[1,2,3,4]] is a 1x4 2-D array.
The second pair of bracket indicates that it is a 2D array, so with such and array the transposed array is different from the first array (since the transpose switches the 2 dimensions). However if the array is only 1D the transpose doesn't change anything and the resulting array is equal to the starting one.
alpha is a 1D array, the transpose is itself.
beta is a 2D array, so you can transform (1,n) to (n,1).
To do the same with alpha, you need to add a dimension, you don't need to transpose it:
alpha[:, None]
alpha is a 1D array with shape (4,). The transpose is just alpha again, i.e. alpha == alpha.T.
beta is a 2D array with shape (1,4). It's a single row, but it has two dimensions. Its transpose looks like a single column with shape (4,1).
When I arrived at the programming language world, having come from the "math side of the business" this also seemed strange to me. After giving some thought to it I realized that from a programming perspective they are different. Have a look at the following list:
a = [1,2,3,4,5]
This is a 1D structure. This is so, because to get back the values 1,2,3,4 and 5 you just need to assign one address value. 3 would be returned if you issued the command a[2] for instance.
Now take a look at this list:
b = [[ 1, 2, 3, 4, 5],
[11, 22, 33, 44, 55]]
To get back the 11 for instance you would need two positional numbers, 1 because 11 is located in the 2nd list and 0 because in the second list it is located in the first position. In other words b[1,0] gives back to you 11.
Now comes the trick part. Look at this third list:
c = [ [ 100, 200, 300, 400, 500] ]
If you look carefully each number requires 2 positional numbers to be taken back from the list. 300 for instance requires 0 because it is located in the first (and only) list and 2 because it is the third element of the first list. c[0,2] gets you back 300.
This list can be transposed because it has two dimensions and the transposition operation is something that switches the positional arguments. So c.T would give you back a list whose shape would be [5,1], since c has a [1,5] shape.
Get back to list a. There you have a list with only one positional number. That list has a shape of [5] only, so there´s no second positional argument to the transposition operation to work with. Therefore it remains [5] and if you try a.T you get back a.
Got it?
Best regards,
Gustavo,
I'm reading the documentation on np.stack:
Join a sequence of arrays along a new axis.
output: ndarray
So np.stack is going to take, say, 2 numpy array and return what? It will return a new array, which contains a, um, sequence of arrays?
I can't visualize what an array consisting of a sequence of arrays is, so how about I run a little experiment:
import numpy as np
from random import randint
arrays = [2.5 * np.random.randn(1,2)+ 3 for _ in range(1,3)]
arrays = [a.astype(int) for a in arrays]
arrays
This gives me:
[array([[1, 2]]), array([[2, 3]])]
Then,
np.stack(arrays, axis=0)
gives
array([[[1, 2]],
[[2, 3]]])
Pretending for a second that the printout is not basically unreadable (10 square brackets, really?), I see what appears to be 2 arrays, in an array, in a ordered sequence. I guess the documentation is correct, but I still have no mental visualization of what this object looks like.
Maybe I should look at the dimensions:
np.stack(arrays, axis=0).shape
gives
(2, 1, 2)
So we have two rows, one column, and two layers of this? Isn't that one array?
My questions are:
What exactly is a 'sequence of arrays' and how does an array possess a notion of order, as does a sequence by definition?
Why would anyone ever want a 'sequence of arrays' anyway, whatever that is, as opposed to concatenating multiple arrays into one (as the .shape implies this really is anyways)?
Why did they call this function "stack" and how does the colloquial use of this word attempt to be helpful?
Thanks.
EDIT too many good answers...having trouble selecting one for the checkmark...
What exactly is a 'sequence of arrays' and how does an array possess a notion of order, as does a sequence by definition?
A sequence is an abstract-data type that, as you intuited, is an ordered collection of items. In Python, a sequence can be assumed to implement __getitem__ and __len__, that is, it supports bracketed indexing, e.g. seq[0] or seq[1], and has a len. It also implements __contains__, __iter__, __reversed__, index, and count. Built-in sequence types include list, tuple, str, bytes, range and memoryview. A numpy.ndarray is a sequence.
Why would anyone ever want a 'sequence of arrays' anyway, whatever that is, as opposed to concatenating multiple arrays into one (as the .shape implies this really is anyways)?
The documentation is letting you know that the function accepts any sequence of arrays. A multidimensional array is a sequence of arrays, itself, or you can pass a list or tuple of arrays you want to "stack" on top (or up against) each other. The function returns a numpy.ndarray, which is any numpy array (n-dimensional array). It is a slightly different operation than concatenate. See below.
Why did they call this function "stack" and how does the colloquial use of this word attempt to be helpful?
Because it stacks stuff together. According to the docs for np.stack, it "Join[s] a sequence of arrays along a new axis.", np.concatenate on the other hand: "Join[s] a sequence of arrays along an existing axis."
Looking at the example in the docs is helpful.
>>> a = np.array([1, 2, 3])
>>> b = np.array([2, 3, 4])
>>> np.stack((a, b), axis=0)
array([[1, 2, 3],
[2, 3, 4]])
>>> np.stack((a, b), axis=1)
array([[1, 2],
[2, 3],
[3, 4]])
>>>
np.concatenate does something different:
>>> np.concatenate((a,b))
array([1, 2, 3, 2, 3, 4])
It is one of many related stacking, concatenating, appending operations on np.arrays that come built-in.
I think your problem starts with np.random.randn(1,2) possibly not giving you what you expect. This is going to give you a (1,2) array.
I helps to think of this as a nested list. The "outer" list has one item, and this item is the "inner" list of two items (in fact this is the exact way it is represented inside the array() wrapper).
Now you make a list of two of these arrays. These brackets are outside the array wrapper so it is just a list. the np.stack command, then, moves the brackets inside the wrapper, in a certain way according to the axis command. In this case axis=0, and the number of elements in the 0 axis becomes the number of items in the outer list. The other two dimensions move over, and the shape becomes (2,1,2)
As a list, this would be a list of two items, each item being a list of a single item, this single item being a list of two numbers.
There are many different ways to arrange these arrays, other than stack. The other major one is np.concatenate, which will let you join them along an existing axis (axis=0 will have an output of shape (2,2), while axis=1 will have shape (1,4)) stack is for when you want a new axis to join them along.
In Python parlance, "sequence" is just a thing that can contain other things, has a sense of "order", and can optionally indexed like seq[i]. Python is duck-typed so anything behaving like a sequence can be called one. It can be a list or a tuple, for example. In this case the ordering is definitely important.
The sequence can be built up piece-by-piece, as is common in programming. For example you may download some data using a RESTful API, build an array from each response, and append each one to a list. After you're done with requesting, you can use the accumulated "sequence of arrays" for your computation later.
If you look at the examples on that page it's pretty clear. You have a list of arrays, let's say, l = [a, b] where a and b are arrays. What stack(l) does is to create a new array with a "on top of" b, if the shapes are compatible. Of course there's also concatenate but it isn't the same. stack creates a new dimension, but concatenate connects along the axis you specify, as the name suggests.
In [1]: import numpy as np
# Ten pieces of 3x4 paper, contained in a list, or "sequence"
In [2]: arrays = [np.random.randn(3, 4) for _ in xrange(10)]
# Stack the paper so you get a paperstack with thickness 10 ;)
In [3]: np.stack(arrays, axis=0).shape
Out[3]: (10, 3, 4)
# Join each piece's long side with the next one, so you get a larger piece of paper
In [4]: np.concatenate(arrays, axis=0).shape
Out[4]: (30, 4)
From the release note:
The new function np.stack provides a general interface for joining a sequence of arrays along a new axis, complementing np.concatenate for joining along an existing axis.
Numpy arrays:
import numpy
a = np.array([1,2,3])
a.shape gives (3,) -> indicates a 1D array
b = np.array([2, 3, 4])
np.stack((a,b),axis=0) -->Stack the two arrays row-wise which means just put the first arrow on top of the second array
np.stack((a,b),axis=1) -->columns wise stack
Now answering your questions in the same order:
A sequence of arrays in the above example is a and b , a and b together form a sequence of arrays.
The reason this function is useful is - Imagine if you are trying to build a matrix where each row (array) is obtained from some special function specific to that row number , np.stack allows you to stack each array and create a 2D array with ease.
In the true sense , stack means placing things over something. In this , you are just adding an array on top of another array.
Hope this helps!
Edit - Based on a comment-
The difference between concatenate and stack (apart from functionality)-
Concat is more like merge
a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6]])
np.stack((a,b),axis=0) ->error
np.concatenate((a,b),axis=0) -->no error
np.stack is just concatenate on a new axis.
Your arrays is a list of 2 (1,2) shaped arrays. It is a sequence.
In [325]: arrays = [np.array([[1, 2]]), np.array([[2, 3]])]
In [326]: arrays
Out[326]: [array([[1, 2]]), array([[2, 3]])]
In [327]: arrays[0].shape
Out[327]: (1, 2)
In [328]: np.array(arrays)
Out[328]:
array([[[1, 2]],
[[2, 3]]])
In [329]: _.shape
Out[329]: (2, 1, 2)
That's the same as if we took np.array of this list of lists
np.array([[[1, 2]],[[2, 3]]])
np.stack with the default axis=0 does the same thing:
In [332]: np.stack(arrays).shape
Out[332]: (2, 1, 2)
It joins the 2 (1,2) arrays on a new axis (dimension). It might be clearer if arrays contained 3 arrays, producing a (3,1,2) array.
np.stack give more flexibility, allowing us to join the arrays on other new axes (I'll flag that with '):
In [335]: np.stack(arrays, axis=0).shape
Out[335]: (2', 1, 2)
In [336]: np.stack(arrays, axis=1).shape
Out[336]: (1, 2', 2)
In [337]: np.stack(arrays, axis=2).shape
Out[337]: (1, 2, 2')
Regarding the name, it's a spin off of hstack, vstack, and column_stack, which have been part of numpy for a long time. Those are all specialized applications of np.concatenate. And if you look at the code, np.source(np.stack), you'll see that np.stack is also just an application of concatenate.
The name is not any more exotic than a 'stack of papers'.
Your arrays list contains 2' (1,2) shaped arrays. np.stack adds a dimension to each (with reshape). For the default axis=0, it reshapes them to (1',1,2). Then it does a concatenate on the first axis, resulting (2',1,2) shape.
I'm having trouble figuring out how to create a 10x1 numpy array with the number 5 in the first 3 elements and the other 7 elements with the number 0. Any thoughts on how to do this efficiently?
Simplest would seem to be:
import numpy as np
the_array = np.array([5]*3 + [0]*7)
Does this simple approach present some specific disadvantage for your purposes?
Of course there are many alternatives, such as
the_array = np.zeros((10,))
the_array[:3] = 5
If you need to repeat this specific operation a huge number of times, so small differences in speed matter, you could benchmark various approaches to see where a nanosecond or so might be saved. But the premise is so unlikely I would not suggest doing that for this specific question, even though I'm a big fan of timeit:-).
I think the way proposed by Alex Martelli is the clearest but here's another alternative using np.repeat which can be quite useful for constructing arrays with repeating values:
>>> np.repeat([5, 0], [3, 7])
array([5, 5, 5, 0, 0, 0, 0, 0, 0, 0])
So here, a list of values [5, 0] is passed in along with a list of repeats [3, 7]. In the returned NumPy array the first element of the values list, 5, is repeated 3 times and the second element 0 is repeated 7 times.
Just do the following.
import numpy as np
arr = np.zeros(10)
arr[:3] = 5
Both Alex and ajcr have useful answers but one thing to keep in mind is what your expected data type needs are.
np.zeros for example will cast a float whereas the other two answers will cast an int.
You can of course recast by using the 'astype' method:
https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.astype.html
import numpy as np
np.repeat([5,0],[3,7]).reshape(10,1)
import numpy as np
array = np.zeros(10)
print("An array of zeros : ")
print (array)
array = np.ones(10)
print("An array of ones : ")
print (array)
array = np.ones(10)*5
print("An array of fives : ")
print (array)
I have troubles understanding the behaviour of numpy masked array.
Here is the snippet that puzzles me for two reasons:
arr = numpy.ma.array([(1,2),(3,4)],dtype=[("toto","int"),("titi","int")])
arr[0][0] = numpy.ma.masked
when doing this nothing happens, no mask is applied on the element [0][0]
changing the data to [[1,2],[3,4]] (instead of [(1,2),(3,4)]), I get the following error:
TypeError: expected a readable buffer object
It seems that I misunderstood completely how to setup (and use) masked array.
Could you tell me what is wrong with this code ?
thanks
EDIT: without specifying the dtypes, it works like expected
The purpose of a masked array is to tell for any operation that some elements of the array are invalid to be used, i.e. masked.
For example, you have an array:
a = np.array([[2, 1000], [3, 1000]])
And you want to ignore any operations with the elements >100. You create a masked array like:
b = np.ma.array(a, mask=(a>100))
You can perform some operations in both arrays to see the differences:
a.sum()
# 2005
b.sum()
# 5
a.prod()
# 6000000
b.prod()
# 6
As you see, the masked items are ignored...
how can I change the values of the diagonal of a matrix in numpy?
I checked Numpy modify ndarray diagonal, but the function there is not implemented in numpy v 1.3.0.
lets say we have a np.array X and I want to set all values of the diagonal to 0.
Did you try numpy.fill_diagonal? See the following answer and this discussion. Or the following from the documentation (although currently broken):
http://docs.scipy.org/doc/numpy/reference/generated/numpy.fill_diagonal.html
If you're using a version of numpy that doesn't have fill_diagonal (the right way to set the diagonal to a constant) or diag_indices_from, you can do this pretty easily with array slicing:
# assuming a 2d square array
n = mat.shape[0]
mat[range(n), range(n)] = 0
This is much faster than an explicit loop in Python, because the looping happens in C and is potentially vectorized.
One nice thing about this is that you can also fill a diagonal with a list of elements, rather than a constant value (like diagflat, but for modifying an existing matrix rather than making a new one). For example, this will set the diagonal of your matrix to 0, 1, 2, ...:
# again assuming 2d square array
n = mat.shape[0]
mat[range(n), range(n)] = range(n)
If you need to support more array shapes, this is more complicated (which is why fill_diagonal is nice...):
m[list(zip(*map(range, m.shape)))] = 0
(The list call is only necessary in Python 3, where zip returns an iterator.)
You can use numpy.diag_indices_from() to get the indices of the diagonal elements of your array. Then set the value of those indices.
X[np.diag_indices_from(X)] = 0.
Example:
>>> import numpy as np
>>> X = np.random.rand(5, 5)
>>> print(X)
[[0.59480384 0.20133725 0.59147423 0.22640441 0.40898203]
[0.65230581 0.57055258 0.97009881 0.58535275 0.32036626]
[0.71524332 0.73424734 0.92461381 0.38704119 0.08147428]
[0.18931865 0.97366736 0.11482649 0.82793141 0.13347333]
[0.47402986 0.73329347 0.18892479 0.11883424 0.78718883]]
>>> X[np.diag_indices_from(X)] = 0
>>> print(X)
[[0. 0.20133725 0.59147423 0.22640441 0.40898203]
[0.65230581 0. 0.97009881 0.58535275 0.32036626]
[0.71524332 0.73424734 0. 0.38704119 0.08147428]
[0.18931865 0.97366736 0.11482649 0. 0.13347333]
[0.47402986 0.73329347 0.18892479 0.11883424 0. ]]
Here's another good way to do this. If you want a one-dimensional view of the array's main diagonal use:
A.ravel()[:A.shape[1]**2:A.shape[1]+1]
For the i'th superdiagonal use:
A.ravel()[i:max(0,A.shape[1]-i)*A.shape[1]:A.shape[1]+1]
For the i'th subdiagonal use:
A.ravel()[A.shape[1]*i:A.shape[1]*(i+A.shape[1]):A.shape[1]+1]
Or in general, for the i'th diagonal where the main diagonal is 0, the subdiagonals are negative and the superdiagonals are positive, use:
A.ravel()[max(i,-A.shape[1]*i):max(0,(A.shape[1]-i))*A.shape[1]:A.shape[1]+1]
These are views and not copies, so they will run faster for extracting a diagonal, but any changes made to the new array object will apply to the original array.
On my machine these run faster than the fill_diagonal function when setting the main diagonal to a constant, but that may not always be the case. They can also be used to assign an array of values to a diagonal instead of just a constant.
Notes: for small arrays it may be faster to use the flat attribute of the NumPy array.
If speed is a major issue it could be worth it to make A.shape[1] a local variable.
Also, if the array is not contiguous, ravel() will return a copy, so, in order to assign values to a strided slice, it will be necessary to creatively slice the original array used to generate the strided slice (if it is contiguous) or to use the flat attribute.
Also, it was originally planned that in NumPy 1.10 and later the 'diagonal' method of arrays will return a view instead of a copy.
That change hasn't yet been made though, but hopefully at some point this trick to get a view will no longer be necessary.
See http://docs.scipy.org/doc/numpy-dev/reference/generated/numpy.diagonal.html
def replaceDiagonal(matrix, replacementList):
for i in range(len(replacementList)):
matrix[i][i] = replacementList[i]
Where size is n in an n x n matrix.
>>> a = numpy.random.rand(2,2)
>>> a
array([[ 0.41668355, 0.07982691],
[ 0.60790982, 0.0314224 ]])
>>> a - numpy.diag(numpy.diag(a))
array([[ 0. , 0.07982691],
[ 0.60790982, 0. ]])
You can do the following.
Assuming your matrix is 4 * 4 matrix.
indices_diagonal = np.diag_indices(4)
yourarray[indices_diagonal] = Val