Situation
I have objects that have attributes which are represented by numpy arrays:
>> obj = numpy.array([1, 2, 3])
where 1, 2, 3 are the attributes' values.
I'm about to write a few methods that should work equally on both a single object and a group of objects. A group of objects is represented by a 2D numpy array:
>>> group = numpy.array([[11, 21, 31],
... [12, 22, 32],
... [13, 23, 33]])
where the first digit indicates the object and the second digit indicates the attribute. That is 12 is attribute 2 of object 1 and 21 is attribute 1 of object 2.
Why this way and not transposed? Because I want the array indices to correspond to the attributes. That is object_or_group[0] should yield the first attribute either as a single number or as a numpy array, so it can be used for further computations.
Alright, so when I want to compute the dot product for example this works out of the box:
>>> obj = numpy.array([1, 2, 3])
>>> obj.dot(object_or_group)
What doesn't work is element-wise addition.
Input:
>>> group
array([[1, 2, 3],
[4, 5, 6]])
>>> obj
array([10, 20])
The resulting array should be the sum of the first element of group and obj and similar for the second element:
>>> result = numpy.array([group[0] + obj[0],
... group[1] + obj[1]])
>>> result
array([[11, 12, 13],
[24, 25, 26]])
However:
>>> group + obj
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (2,3) (2,)
Which makes sense considering numpy's broadcasting rules.
It seems that there is no numpy function which performs an addition (or equivalently the broadcasting) along a specified axis. While I could use
>>> (group.T + obj).T
array([[11, 12, 13],
[24, 25, 26]])
this feels very cumbersome (and if, instead of a group, I consider a single object this feels wrong indeed). Especially because numpy covered each and every corner case for its usage, I have the feeling that I might have gotten something conceptually wrong here.
To sum it up
Similarly to
>>> obj1
array([1, 2])
>>> obj2
array([10, 20])
>>> obj1 + obj2
array([11, 22])
(which performs an element-wise - or attribute-wise - addition) I want to do the same for groups of objects:
>>> group
array([[1, 2, 3],
[4, 5, 6]])
while the layout of such a 2D group array must be such that the single objects are listed along the 2nd axis (axis=1) in order to be able to request a certain attribute (or many) via normal indexing: obj[0] and group[0] should both yield the first attribute(s).
what you want to do seems to work with this simple code !!
>>> m
array([[1, 2, 3],
[4, 5, 6]])
>>> g = np.array([10,20])
>>> m + g[ : , None]
array([[11, 12, 13],
[24, 25, 26]])
You appear to be confused about which dimension of the matrix is an object and which is an attirbute, as evidenced by the changing object size in your examples. In fact, it it the fact that you are swapping dimensions to match that changing size that is throwing you off. You are also using the unfortunate example of a 3x3 group for your dot product, which is further throwing off your explanation.
In the examples below, objects will be three-element vectors, i.e., they will have three attributes each. The example group will have consistently two rows, meaning two objects in it, and three columns, because objects have three attributes.
The first row of the group, group[0], a.k.a. group[0, :], will be the first object in the group. The first column, group[:, 0] will be the first attribute.
Here are a couple of sample objects and groups to illustrate the points that follow:
>>> obj1 = np.array([1, 2, 3])
>>> obj2 = np.array([4, 5, 6])
>>> group1 = np.array([[7, 8, 9],
[0, 1, 2]])
>>> group2 = np.array([[3, 4, 5]])
Addition will work out of the box because of broadcasting now:
>>> obj1 + obj2
array([5, 7, 9])
>>> group1 + obj1
array([[ 8, 10, 12],
[ 1, 3, 5]])
As you can see, corresponding attributes are getting added just fine. You can even add together groups, but only if they are the same size or if one of them only contains a single object:
>>> group1 + group2
array([[10, 12, 14],
[ 3, 5, 7]])
>>> group1 + group1
array([[14, 16, 18],
[ 0, 2, 4]])
The same will be true for all the binary elementwise operators: *, -, /, np.bitwise_and, etc.
The only remaining question is how to make dot products not care if they are operating on a matrix or a vector. It just so happens that dot products don't care. Your common dimension is always the number of attributes, so the second operand (the multiplier) needs to be transposed so that the number of columns becomes the number of rows. np.dot(x1, x2.T), or equivalently x1.dot(x2.T) will work correctly whether x1 and x2 are groups or objects:
>>> obj1.dot(obj2.T)
32
>>> obj1.dot(group1.T)
array([50, 8])
>>> group1.dot(obj1.T)
array([50, 8])
You can use either np.atleast_1d or np.atleast_2d to always coerce the result into a particular shape so you don't end up with a scalar like the obj1.dot(obj2.T) case. I would recommend the latter, so you always have a consistent number of dimensions regardless of the inputs:
>>> np.atleast_2d(obj1.dot(obj2.T))
array([[32]])
>>> np.atleast_2d(obj1.dot(group1.T))
array([[50, 8]])
Just keep in mind that the dimensions of the dot product will be the the number of objects in the first operand by the number of objects in the second operand (everything will be treated as a group). The attributes will get multiplied and summed together. Whether or not that has a valid interpretation for your purposes is entirely for you to decide.
UPDATE
The only remaining problem at this point is attribute access. As stated above obj1[0] and group1[0] mean very different things. There are three ways to reconcile this difference, listed in the order that I personally prefer them, with 1 being the most preferable:
Use the Ellipsis indexing object to get the last index instead of the first
>>> obj1[..., 0]
array([1])
>>> group1[..., 0]
array([7, 0])
This is the most efficient way since it does not make any copies, just does a normal index on the original arrays. As you can see, there will be no difference between the result from a single object (1D array) and a group with only one object in it (2D array).
Make all your objects 2D. As you pointed out yourself, this can be done with a decorator, and/or using np.atleast_2d. Personally, I would prefer having the convenience of using 1D arrays as single objects without having to wrap them in 2D.
Always access attributes via a transpose:
>>> obj1.T[0]
1
>>> group1.T[0]
array([7, 0])
While this is functionally equivalent to #1, it is clunky and unsightly by comparison, in addition to doing something very different under-the-hood. This approach at the very least creates a new view of the underlying array, and may run the risk of making unnecessary copies in certain cases if the group arrays are not laid out just right. I would not recommend this approach even if it does solve the problem if uniform access.
Related
I came across a code snippet where I could not understand two of the statements, though I could see the end result of each.
I will create a variable before giving the statements:
train = np.random.random((10,100))
One of them read as :
train = train[:-1, 1:-1]
What does this slicing mean? How to read this? I know that that -1 in slicing denotes from the back. But I cannot understand this.
Another statement read as follows:
la = [0.2**(7-j) for j in range(1,t+1)]
np.array(la)[:,None]
What does slicing with None as in [:,None] mean?
For the above two statements, along with how each statement is read, it will be helpful to have an alternative method along, so that I understand it better.
One of Python's strengths is its uniform application of straightforward principles. Numpy indexing, like all indexing in Python, passes a single argument to the indexed object's (i.e., the array's) __getitem__ method, and numpy arrays were one of the primary justifications for the slicing mechanism (or at least one of its very early uses).
When I'm trying to understand new behaviours I like to start with a concrete and comprehensible example, so rather than 10x100 random values I'll start with a one-dimensional 4-element vector and work up to 3x4, which should be big enough to understand what's going on.
simple = np.array([1, 2, 3, 4])
train = np.array([[1, 2, 3, 4],
[5, 6, 7, 8],
[9, 10, 11, 12]])
The interpreter shows these as
array([1, 2, 3, 4])
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12]])
The expression simple[x] is equivalent to (which is to say the interpreter ends up executing) simple.__getitem__(x) under the hood - note this call takes a single argument.
The numpy array's __getitem__ method implements indexing with an integer very simply: it selects a single element from the first dimension. So simple[1] is 2, and train[1] is array([5, 6, 7, 8]).
When __getitem__ receives a tuple as an argument (which is how Python's syntax interprets expressions like array[x, y, z]) it applies each element of the tuple as an index to successive dimensions of the indexed object. So result = train[1, 2] is equivalent (conceptually - the code is more complex in implementation) to
temp = train[1] # i.e. train.__getitem__(1)
result = temp[2] # i.e. temp.__getitem__(2)
and sure enough we find that result comes out at 7. You could think of array[x, y, z] as equivalent to array[x][y][z].
Now we can add slicing to the mix. Expressions containing a colon can be regarded as slice literals (I haven't seen a better name for them), and the interpreter creates slice objects for them. As the documentation notes, a slice object is mostly a container for three values, start, stop and slice, and it's up to each object's __getitem__ method how it interprets them. You might find this question helpful to understand slicing further.
With what you now know, you should be able to understand the answer to your first question.
result = train[:-1, 1:-1]
will call train.__getitem__ with a two-element tuple of slices. This is equivalent to
temp = train[:-1]
result = temp[..., 1:-1]
The first statement can be read as "set temp to all but the last row of train", and the second as "set result to all but the first and last columns of temp". train[:-1] is
array([[1, 2, 3, 4],
[5, 6, 7, 8]])
and applying the [1:-1] subscripting to the second dimension of that array gives
array([[2, 3],
[6, 7]])
The ellipsis on the first dimension of the temp subscript says "pass everything," so the subscript expression[...]can be considered equivalent to[:]. As far as theNonevalues are concerned, a slice has a maximum of three data points: _start_, _stop_ and _step_. ANonevalue for any of these gives the default value, which is0for _start_, the length of the indexed object for _stop_, and1for _step. Sox[None:None:None]is equivalent tox[0:len(x):1]which is equivalent tox[::]`.
With this knowledge under your belt you should stand a bit more chance of understanding what's going on.
I was surprised that numpy.split yields a list and not an array. I would have thought it would be better to return an array, since numpy has put a lot of work into making arrays more useful than lists. Can anyone justify numpy returning a list instead of an array? Why would that be a better programming decision for the numpy developers to have made?
A comment pointed out that if the slit is uneven, the result can't be a array, at least not one that has the same dtype. At best it would be an object dtype.
But lets consider the case of equal length subarrays:
In [124]: x = np.arange(10)
In [125]: np.split(x,2)
Out[125]: [array([0, 1, 2, 3, 4]), array([5, 6, 7, 8, 9])]
In [126]: np.array(_) # make an array from that
Out[126]:
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
But we can get the same array without split - just reshape:
In [127]: x.reshape(2,-1)
Out[127]:
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
Now look at the code for split. It just passes the task to array_split. Ignoring the details about alternative axes, it just does
sub_arys = []
for i in range(Nsections):
# st and end from `div_points
sub_arys.append(sary[st:end])
return sub_arys
In other words, it just steps through array and returns successive slices. Those (often) are views of the original.
So split is not that sophisticate a function. You could generate such a list of subarrays yourself without a lot of numpy expertise.
Another point. Documentation notes that split can be reversed with an appropriate stack. concatenate (and family) takes a list of arrays. If give an array of arrays, or a higher dim array, it effectively iterates on the first dimension, e.g. concatenate(arr) => concatenate(list(arr)).
Actually you are right it returns a list
import numpy as np
a=np.random.randint(1,30,(2,2))
b=np.hsplit(a,2)
type(b)
it will return type(b) as list so, there is nothing wrong in the documentation, i also first thought that the documentation is wrong it doesn't return a array, but when i checked
type(b[0])
type(b[1])
it returned type as ndarray.
it means it returns a list of ndarrary's.
I ran across something that seemed to me like inconsistent behavior in Numpy slices. Specifically, please consider the following example:
import numpy as np
a = np.arange(9).reshape(3,3) # a 2d numpy array
y = np.array([1,2,2]) # vector that will be used to index the array
b = a[np.arange(len(a)),y] # a vector (what I want)
c = a[:,y] # a matrix ??
I wanted to obtain a vector such that the i-th element is a[i,y[i]]. I tried two things (b and c above) and was surprised that b and c are not the same... in fact one is a vector and the other is a matrix! I was under the impression that : was shorthand for "all elements" but apparently the meaning is somewhat more subtle.
After trial and error I somewhat understand the difference now (b == np.diag(c)), but would appreciate clarification on why they are different, what exactly using : implies, and how to understand when to use either case.
Thanks!
It's hard to understand advanced indexing (with lists or arrays) without understanding broadcasting.
In [487]: a=np.arange(9).reshape(3,3)
In [488]: idx = np.array([1,2,2])
Index with a (3,) and (3,) producing shape (3,) result:
In [489]: a[np.arange(3),idx]
Out[489]: array([1, 5, 8])
Index with (3,1) and (3,), result is (3,3)
In [490]: a[np.arange(3)[:,None],idx]
Out[490]:
array([[1, 2, 2],
[4, 5, 5],
[7, 8, 8]])
The slice : does basically the same thing. There are subtle differences, but here it's the same.
In [491]: a[:,idx]
Out[491]:
array([[1, 2, 2],
[4, 5, 5],
[7, 8, 8]])
ix_ does the same thing, converting the (3,) & (3,) to (3,1) and (1,3):
In [492]: np.ix_(np.arange(3),idx)
Out[492]:
(array([[0],
[1],
[2]]), array([[1, 2, 2]]))
A broadcasted sum might help visualize the two cases:
In [495]: np.arange(3)*10+idx
Out[495]: array([ 1, 12, 22])
In [496]: np.sum(np.ix_(np.arange(3)*10,idx),axis=0)
Out[496]:
array([[ 1, 2, 2],
[11, 12, 12],
[21, 22, 22]])
When you pass
np.arange(len(a)), y
You can view the result as being all the indexed pairs for the zipped elements you passed. In this case, indexing by np.arange(len(a)) and y
np.arange(len(a))
# [0, 1, 2]
y
# [1, 2, 2]
effectively takes elements: (0, 1), (1, 2), and (2, 2).
print(a[0, 1], a[1, 2], a[2, 2]) # 0th, 1st, 2nd elements from each indexer
# 1 5 8
In the second case, you take the entire slice along the first dimension. (Nothing before the colon.) So this is all elements along the 0th axis. You then specify with y that you want the 1st, 2nd, and 2nd element along each row. (0-indexed.)
As you pointed out, it may seem a bit unintuitive that the results are different given that the individual elements of the slice are equivalent:
a[:] == a[np.arange(len(a))]
and
a[:y] == a[:y]
However, NumPy advanced indexing cares what type of data structure you pass when indexing (tuples, integers, etc). Things can become hairy very quickly.
The detail behind that is this: first consider all NumPy indexing to be of the general form x[obj], where obj is the evaluation of whatever you passed. How NumPy "behaves" depends on what type of object obj is:
Advanced indexing is triggered when the selection object, obj, is a
non-tuple sequence object, an ndarray (of data type integer or bool),
or a tuple with at least one sequence object or ndarray (of data type
integer or bool).
...
The definition of advanced indexing means that x[(1,2,3),] is
fundamentally different than x[(1,2,3)]. The latter is equivalent to
x[1,2,3] which will trigger basic selection while the former will
trigger advanced indexing. Be sure to understand why this occurs.
In your first case, obj = np.arange(len(a)),y, a tuple that fits the bill in bold above. This triggers advanced indexing and forces the behavior described above.
As for the second case, [:,y]
When there is at least one slice (:), ellipsis (...) or np.newaxis in
the index (or the array has more dimensions than there are advanced
indexes), then the behaviour can be more complicated. It is like
concatenating the indexing result for each advanced index element.
Demonstrated:
# Concatenate the indexing result for each advanced index element.
np.vstack((a[0, y], a[1, y], a[2, y]))
Let's say I have a numpy array of rgb-imagetype looking like this:
d = [ [ [0, 1, 2], [3, 4, 5], [6 ,7 ,8] ],
[ [9, 10, 11], [12, 13, 14], [15, 16 ,17] ],
[ [18,19, 20], [21, 22, 23], [24, 25 ,26] ] ]
I select a few random r/g or b pixels using random
import random
r = random.sample(range(1, len(d)*len(d[0])*3), 3)
# for example r = [25, 4, 15]
How can I then select the data I want?
Like I want 25th value in array d for the first r_value = 25 which corresponds to d[2][2][1], because it is the 25th value.
What you want to do is index it as a flat or 1d array. There are various ways of doing this. ravel and reshape(-1) create 1d views, flatten() creates a 1d copy.
The most efficient is the flat iterator (an attribute, not method):
In [347]: d.flat[25]
Out[347]: 25
(it can be used in an assignment as well, eg. d.flat[25]=0.
In [341]: idx = [25, 4, 15]
In [343]: d.flat[idx]
Out[343]: array([25, 4, 15])
To find out what the 3d index is, there's utility, unravel_index (and a corresponding ravel_multi_index)
In [344]: fidx=np.unravel_index(idx,d.shape)
In [345]: fidx
Out[345]:
(array([2, 0, 1], dtype=int32),
array([2, 1, 2], dtype=int32),
array([1, 1, 0], dtype=int32))
In [346]: d[fidx]
Out[346]: array([25, 4, 15])
This a tuple, the index for one element is read 'down', e.g. (2,2,1).
On a large array, flat indexing is actually a bit faster:
In [362]: dl=np.ones((100,100,100))
In [363]: idx=np.arange(0,1000000,40)
In [364]: fidx=np.unravel_index(idx,dl.shape)
In [365]: timeit x=dl[fidx]
1000 loops, best of 3: 447 µs per loop
In [366]: timeit x=dl.flat[idx]
1000 loops, best of 3: 312 µs per loop
If you are going to inspect/alter the array linearly on a frequent basis, you can construct a linear view:
d_lin = d.reshape(-1) # construct a 1d view
d_lin[25] # access the 25-th element
Or putting it all in a one-liner:
d.reshape(-1)[25] # construct a 1d view
You can from now on access (and modify) the elements in d_view as a 1d array. So you access the 25-th value with d_lin[25]. You don't have to construct a new view each time you want to access/modify an element: simply reuse the d_lin view.
Furthermore the order of flattening can be specified (order='C' (C-like), order='F' (Fortran-like) or order='A' (Fortran-wise if contiguous in meory, C-like otherwise)). order='F' means that we first iterate over the greatest dimension.
The advantage of a view (but this can also lead to unintended behavior), is that if you assign a new value through d_lin, like d_lin[25] = 3, it will alter the original matrix.
Alternatives are .flat, or np.ravel. So the following are somewhat equivalent:
d.reshape(-1)[25]
np.ravel(d)[25]
d.flat[25]
There are however some differences between the reshape(..) and ravel(..) approach against the flat approach. The most important one is the fact that d.flat does not create a full view. Indeed if we for instance want to pass the view to another function that expects a numpy array, then it will crash, for example:
>>> d.flat.sum()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'numpy.flatiter' object has no attribute 'sum'
>>> d.reshape(-1).sum()
351
>>> np.ravel(d).sum()
351
This is not per se a problem. If we want to limit the number of tools (for instance as a protection mechanism), then this will actually give us a bit more security (although we can still set elements elements in bulk, and call np.sum(..) on an flatiter object).
You can use numpy.flatten method, like this:
a = np.array(d)
d[25] # print 26
Assuming you do not want to flatten your array:
if you know the size of the sublists beforehand, you can easily calculate it out. In your example, every element of the main list is a list of exactly 3 elements, each element containing 3. So to access n you could do something like
i = n//9
j = (n%9)//3
k = (n%3)
element = d[i][j][k]
For n=25, you'd get i = 2, j = 2, k = 1, just as you wanted.
In python2, you can (and have to) use the normal / operator instead of //
If you only care on the value, you can flatten your array, and access it directly as
val = d.flatten()[r]
If you really want the index corresponding to the flattened index, you need to something like this:
ix_2 = r % d.shape[2]
helper_2 = (r - ix_2) / d.shape[2]
ix_1 = helper_2 % d.shape[1]
helper_1 = (helper_2 - ix_1) / d.shape[1]
ix_0 = helper_1 % d.shape[0]
val = d[ix_0, ix_1, ix_2]
I'm really confused by the index logic of numpy arrays with several dimensions. Here is an example:
import numpy as np
A = np.arange(18).reshape(3,2,3)
[[[ 0, 1, 2],
[ 3, 4, 5]],
[[ 6, 7, 8],
[ 9, 10, 11]],
[[12, 13, 14],
[15, 16, 17]]])
this gives me an array of shape (3,2,3), call them (x,y,z) for sake of argument. Now I want an array B with the elements from A corresponding to x = 0,2 y =0,1 and z = 1,2. Like
array([[[ 1, 2],
[4, 5]],
[[13, 14],
[16, 17]]])
Naively I thought that
B=A[[0,2],[0,1],[1,2]]
would do the job. But it gives
array([ 2, 104])
and does not work.
A[[0,2],:,:][:,:,[1,2]]
does the job. But I still wonder whats wrong with my first try. And what is the best way to do what I want to do?
There are two types of indexing in NumPy basic and advanced. Basic indexing uses tuples of slices for indexing, and does not copy the array, but rather creates a view with adjusted strides. Advanced indexing in contrast also uses lists or arrays of indices and copies the array.
Your first attempt
B = A[[0, 2], [0, 1], [1, 2]]
uses advanced indexing. In advanced indexing, all index lists are first broadcasted to the same shape, and this shape is used for the output array. In this case, they already have the same shape, so the broadcasting does not do anything. The output array will also have this shape of two entries. The first entry of the output array is obtained by using all first indices of the three lists, and the second by using all second indices:
B = numpy.array([A[0, 0, 1], A[2, 1, 2]])
Your second approach
B = A[[0,2],:,:][:,:,[1,2]]
does work, but it is inefficient. It uses advanced indexing twice, so your data will be copied twice.
To get what you actually want with advanced indexing, you can use
A[np.ix_([0,2],[0,1],[1,2])]
as pointed out by nikow. This will copy the data only once.
In your example, you can get away without copying the data at all, using only basic indexing:
B = A[::2, :, 1:2]
I recommend the following advanced tutorial, which explains the various indexing methods: NumPy MedKit
Once you understand the powerful ways to index arrays (and how they can be combined) it will make sense. If your first try was valid then this would collide with some of the other indexing techniques (reducing your options in other use cases).
In your example you can exploit that the third index covers a continuous range:
A[[0,2],:,1:]
You could also use
A[np.ix_([0,2],[0,1],[1,2])]
which is handy in more general cases, when the latter indices are not continuous. np.ix_ simply constructs three index arrays.
As Sven pointed out in his answer, there is a more efficient way in this specific case (using a view instead of a copied version).
Edit: As pointed out by Sven my answer contained some errors, which I have removed. I still think that his answer is better, but unfortunately I can't delete mine now.
A[(0,2),:,1:]
If you wanted
array([[[ 1, 2],
[ 4, 5]],
[[13, 14],
[16, 17]]])
A[indices you want,rows you want, col you want]