I know how to take x[:,:,:,:,j,:] (which takes the jth slice of dimension 4).
Is there a way to do the same thing if the dimension is known at runtime, and is not a known constant?
One option to do so is to construct the slicing programatically:
slicing = (slice(None),) * 4 + (j,) + (slice(None),)
An alternative is to use numpy.take() or ndarray.take():
>>> a = numpy.array([[1, 2], [3, 4]])
>>> a.take((1,), axis=0)
array([[3, 4]])
>>> a.take((1,), axis=1)
array([[2],
[4]])
You can use the slice function and call it with the appropriate variable list during runtime as follows:
# Store the variables that represent the slice in a list/tuple
# Make a slice with the unzipped tuple using the slice() command
# Use the slice on your array
Example:
>>> from numpy import *
>>> a = (1, 2, 3)
>>> b = arange(27).reshape(3, 3, 3)
>>> s = slice(*a)
>>> b[s]
array([[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]]])
This is the same as:
>>> b[1:2:3]
array([[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]]])
Finally, the equivalent of not specifying anything between 2 : in the usual notation is to put None in those places in the tuple you create.
If everything is decided at runtime, you could do:
# Define the data (this could be measured at runtime)
data_shape = (3, 5, 7, 11, 13)
print('data_shape = {}'.format(data_shape))
# Pick which index to slice from which dimension (could also be decided at runtime)
slice_dim = len(data_shape)/2
slice_index = data_shape[slice_dim]/2
print('slice_dim = {} (data_shape[{}] = {}), slice_index = {}'.format(slice_dim, slice_dim, data_shape[slice_dim], slice_index))
# Make a data set for testing
data = arange(product(data_shape)).reshape(*data_shape)
# Slice the data
s = [slice_index if a == slice_dim else slice(None) for a in range(len(data_shape))]
d = data[s]
print('shape(data[s]) = {}, s = {}'.format(shape(d), s))
Although this is longer than ndarray.take(), it will work if slice_index = None, as in the case where the array happens to have so few dimensions that you don't actually want to slice it (but you don't know you don't want to slice it ahead of time).
You can also use ellipsis to replace the repeating colons.
See an answer to How do you use the ellipsis slicing syntax in Python? for an example.
Related
Is there a batch way to replace several particular values in a pytorch tensor at once without a for loop?
Example:
old_values = torch.Tensor([1, 2, 3, 4, 5, 5, 2, 3, 3, 2])
old_new_value = [[2,22], [3,33], [6, 66]]
old_new_value = [[2,22], [3,33], [6, 66]], which means 2 should be replaced by 22, and 3 should be replaced by 33 and 6 to 66
Can I have an efficient way to achieve the following end_result?
end_result = torch.Tensor([1, 22, 33, 4, 5, 5, 22, 33, 33, 22])
Note that old_values is not unique. Also, it is possible that old_new_value has a pair here(6, 66) that does not exist in the old_values.
Also, the old_new_values includes unique rows,
If you don't have any duplicate elements in your input tensor, here's one straightforward way using masking and value assignment using basic indexing. (I'll assume that the data type of the input tensor is int. But, you can simply adapt this code in a straightforward manner to other dtypes). Below is a reproducible illustration, with explanations interspersed in inline comments.
# input tensors to work with
In [75]: old_values
Out[75]: tensor([1, 2, 3, 4, 5], dtype=torch.int32)
In [77]: old_new_value
Out[77]:
tensor([[ 2, 22],
[ 3, 33]], dtype=torch.int32)
# generate a boolean mask using the values that need to be replaced (i.e. 2 & 3)
In [78]: boolean_mask = (old_values == old_new_value[:, :1]).sum(dim=0).bool()
In [79]: boolean_mask
Out[79]: tensor([False, True, True, False, False])
# assign the new values by basic indexing
In [80]: old_values[boolean_mask] = old_new_value[:, 1:].squeeze()
# sanity check!
In [81]: old_values
Out[81]: tensor([ 1, 22, 33, 4, 5], dtype=torch.int32)
A small note on efficiency: Throughout the whole process, we never made any copy of the data (i.e. we operate only on new views by massaging the shapes according to our needs). Therefore, the runtime would be blazing fast.
Not sure anyone still cares about this, but just in case, here is a solution that also works when old_values is not unique:
mask = old_values == old_new_value[:, :1]
new_values = (1 - mask.sum(dim=0)) * old_values + (mask * old_new_value[:,1:]).sum(dim=0)
Masking works as in #kmario23's solution, but the mask is multiplied with the new values and sum-reduced to end up with the new values at all the right replacement positions. The negative mask is applied to the old values to use those at all other positions. Then both masked tensors are summed to obtain the desired result.
Fancy Indexing vs Views in Numpy
In an answer to this equation: is is explained that different idioms will produce different results.
Using the idiom where fancy indexing is to chose the values and said values are set to a new value in the same line means that the values in the original object will be changed in place.
However the final example below:
https://scipy-cookbook.readthedocs.io/items/ViewsVsCopies.html
"A final exercise"
The example appears to use the same idiom:
a[x, :][:, y] = 100
but it still produces a different result depending on whether x is a slice or a fancy index (see below):
a = np.arange(12).reshape(3,4)
ifancy = [0,2]
islice = slice(0,3,2)
a[islice, :][:, ifancy] = 100
a
#array([[100, 1, 100, 3],
# [ 4, 5, 6, 7],
# [100, 9, 100, 11]])
a = np.arange(12).reshape(3,4)
ifancy = [0,2]
islice = slice(0,3,2)
a[ifancy, :][:, islice] = 100 # note that ifancy and islice are interchanged here
>>> a
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
My intuition is that if the first set of fancy indexes is a slice it treats the object like a view and therefore the values in the orignal object are changed.
Whereas in the second case the first set of fancy indexes is itself a fancy index so it treats the object as a fancy index creating a copy of the original object. This then means that the original object is not changed when the values of the copy object are changed.
Is my intuition correct?
The example hints that one should think of the sqeuence of getitem and setitem can someone explain it to my properly in theis way?
Python evaluates each set of [] separately. a[x, :][:, y] = 100 is 2 operations.
temp = a[x,:] # getitem step
temp[:,y] = 100 # setitem step
Whether the 2nd line ends up modifying a depends on whether temp is a view or copy.
Remember, numpy is an addon to Python. It does not modify basic Python syntax or interpretation.
Let's say I have a numpy array of rgb-imagetype looking like this:
d = [ [ [0, 1, 2], [3, 4, 5], [6 ,7 ,8] ],
[ [9, 10, 11], [12, 13, 14], [15, 16 ,17] ],
[ [18,19, 20], [21, 22, 23], [24, 25 ,26] ] ]
I select a few random r/g or b pixels using random
import random
r = random.sample(range(1, len(d)*len(d[0])*3), 3)
# for example r = [25, 4, 15]
How can I then select the data I want?
Like I want 25th value in array d for the first r_value = 25 which corresponds to d[2][2][1], because it is the 25th value.
What you want to do is index it as a flat or 1d array. There are various ways of doing this. ravel and reshape(-1) create 1d views, flatten() creates a 1d copy.
The most efficient is the flat iterator (an attribute, not method):
In [347]: d.flat[25]
Out[347]: 25
(it can be used in an assignment as well, eg. d.flat[25]=0.
In [341]: idx = [25, 4, 15]
In [343]: d.flat[idx]
Out[343]: array([25, 4, 15])
To find out what the 3d index is, there's utility, unravel_index (and a corresponding ravel_multi_index)
In [344]: fidx=np.unravel_index(idx,d.shape)
In [345]: fidx
Out[345]:
(array([2, 0, 1], dtype=int32),
array([2, 1, 2], dtype=int32),
array([1, 1, 0], dtype=int32))
In [346]: d[fidx]
Out[346]: array([25, 4, 15])
This a tuple, the index for one element is read 'down', e.g. (2,2,1).
On a large array, flat indexing is actually a bit faster:
In [362]: dl=np.ones((100,100,100))
In [363]: idx=np.arange(0,1000000,40)
In [364]: fidx=np.unravel_index(idx,dl.shape)
In [365]: timeit x=dl[fidx]
1000 loops, best of 3: 447 µs per loop
In [366]: timeit x=dl.flat[idx]
1000 loops, best of 3: 312 µs per loop
If you are going to inspect/alter the array linearly on a frequent basis, you can construct a linear view:
d_lin = d.reshape(-1) # construct a 1d view
d_lin[25] # access the 25-th element
Or putting it all in a one-liner:
d.reshape(-1)[25] # construct a 1d view
You can from now on access (and modify) the elements in d_view as a 1d array. So you access the 25-th value with d_lin[25]. You don't have to construct a new view each time you want to access/modify an element: simply reuse the d_lin view.
Furthermore the order of flattening can be specified (order='C' (C-like), order='F' (Fortran-like) or order='A' (Fortran-wise if contiguous in meory, C-like otherwise)). order='F' means that we first iterate over the greatest dimension.
The advantage of a view (but this can also lead to unintended behavior), is that if you assign a new value through d_lin, like d_lin[25] = 3, it will alter the original matrix.
Alternatives are .flat, or np.ravel. So the following are somewhat equivalent:
d.reshape(-1)[25]
np.ravel(d)[25]
d.flat[25]
There are however some differences between the reshape(..) and ravel(..) approach against the flat approach. The most important one is the fact that d.flat does not create a full view. Indeed if we for instance want to pass the view to another function that expects a numpy array, then it will crash, for example:
>>> d.flat.sum()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'numpy.flatiter' object has no attribute 'sum'
>>> d.reshape(-1).sum()
351
>>> np.ravel(d).sum()
351
This is not per se a problem. If we want to limit the number of tools (for instance as a protection mechanism), then this will actually give us a bit more security (although we can still set elements elements in bulk, and call np.sum(..) on an flatiter object).
You can use numpy.flatten method, like this:
a = np.array(d)
d[25] # print 26
Assuming you do not want to flatten your array:
if you know the size of the sublists beforehand, you can easily calculate it out. In your example, every element of the main list is a list of exactly 3 elements, each element containing 3. So to access n you could do something like
i = n//9
j = (n%9)//3
k = (n%3)
element = d[i][j][k]
For n=25, you'd get i = 2, j = 2, k = 1, just as you wanted.
In python2, you can (and have to) use the normal / operator instead of //
If you only care on the value, you can flatten your array, and access it directly as
val = d.flatten()[r]
If you really want the index corresponding to the flattened index, you need to something like this:
ix_2 = r % d.shape[2]
helper_2 = (r - ix_2) / d.shape[2]
ix_1 = helper_2 % d.shape[1]
helper_1 = (helper_2 - ix_1) / d.shape[1]
ix_0 = helper_1 % d.shape[0]
val = d[ix_0, ix_1, ix_2]
Situation
I have objects that have attributes which are represented by numpy arrays:
>> obj = numpy.array([1, 2, 3])
where 1, 2, 3 are the attributes' values.
I'm about to write a few methods that should work equally on both a single object and a group of objects. A group of objects is represented by a 2D numpy array:
>>> group = numpy.array([[11, 21, 31],
... [12, 22, 32],
... [13, 23, 33]])
where the first digit indicates the object and the second digit indicates the attribute. That is 12 is attribute 2 of object 1 and 21 is attribute 1 of object 2.
Why this way and not transposed? Because I want the array indices to correspond to the attributes. That is object_or_group[0] should yield the first attribute either as a single number or as a numpy array, so it can be used for further computations.
Alright, so when I want to compute the dot product for example this works out of the box:
>>> obj = numpy.array([1, 2, 3])
>>> obj.dot(object_or_group)
What doesn't work is element-wise addition.
Input:
>>> group
array([[1, 2, 3],
[4, 5, 6]])
>>> obj
array([10, 20])
The resulting array should be the sum of the first element of group and obj and similar for the second element:
>>> result = numpy.array([group[0] + obj[0],
... group[1] + obj[1]])
>>> result
array([[11, 12, 13],
[24, 25, 26]])
However:
>>> group + obj
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: operands could not be broadcast together with shapes (2,3) (2,)
Which makes sense considering numpy's broadcasting rules.
It seems that there is no numpy function which performs an addition (or equivalently the broadcasting) along a specified axis. While I could use
>>> (group.T + obj).T
array([[11, 12, 13],
[24, 25, 26]])
this feels very cumbersome (and if, instead of a group, I consider a single object this feels wrong indeed). Especially because numpy covered each and every corner case for its usage, I have the feeling that I might have gotten something conceptually wrong here.
To sum it up
Similarly to
>>> obj1
array([1, 2])
>>> obj2
array([10, 20])
>>> obj1 + obj2
array([11, 22])
(which performs an element-wise - or attribute-wise - addition) I want to do the same for groups of objects:
>>> group
array([[1, 2, 3],
[4, 5, 6]])
while the layout of such a 2D group array must be such that the single objects are listed along the 2nd axis (axis=1) in order to be able to request a certain attribute (or many) via normal indexing: obj[0] and group[0] should both yield the first attribute(s).
what you want to do seems to work with this simple code !!
>>> m
array([[1, 2, 3],
[4, 5, 6]])
>>> g = np.array([10,20])
>>> m + g[ : , None]
array([[11, 12, 13],
[24, 25, 26]])
You appear to be confused about which dimension of the matrix is an object and which is an attirbute, as evidenced by the changing object size in your examples. In fact, it it the fact that you are swapping dimensions to match that changing size that is throwing you off. You are also using the unfortunate example of a 3x3 group for your dot product, which is further throwing off your explanation.
In the examples below, objects will be three-element vectors, i.e., they will have three attributes each. The example group will have consistently two rows, meaning two objects in it, and three columns, because objects have three attributes.
The first row of the group, group[0], a.k.a. group[0, :], will be the first object in the group. The first column, group[:, 0] will be the first attribute.
Here are a couple of sample objects and groups to illustrate the points that follow:
>>> obj1 = np.array([1, 2, 3])
>>> obj2 = np.array([4, 5, 6])
>>> group1 = np.array([[7, 8, 9],
[0, 1, 2]])
>>> group2 = np.array([[3, 4, 5]])
Addition will work out of the box because of broadcasting now:
>>> obj1 + obj2
array([5, 7, 9])
>>> group1 + obj1
array([[ 8, 10, 12],
[ 1, 3, 5]])
As you can see, corresponding attributes are getting added just fine. You can even add together groups, but only if they are the same size or if one of them only contains a single object:
>>> group1 + group2
array([[10, 12, 14],
[ 3, 5, 7]])
>>> group1 + group1
array([[14, 16, 18],
[ 0, 2, 4]])
The same will be true for all the binary elementwise operators: *, -, /, np.bitwise_and, etc.
The only remaining question is how to make dot products not care if they are operating on a matrix or a vector. It just so happens that dot products don't care. Your common dimension is always the number of attributes, so the second operand (the multiplier) needs to be transposed so that the number of columns becomes the number of rows. np.dot(x1, x2.T), or equivalently x1.dot(x2.T) will work correctly whether x1 and x2 are groups or objects:
>>> obj1.dot(obj2.T)
32
>>> obj1.dot(group1.T)
array([50, 8])
>>> group1.dot(obj1.T)
array([50, 8])
You can use either np.atleast_1d or np.atleast_2d to always coerce the result into a particular shape so you don't end up with a scalar like the obj1.dot(obj2.T) case. I would recommend the latter, so you always have a consistent number of dimensions regardless of the inputs:
>>> np.atleast_2d(obj1.dot(obj2.T))
array([[32]])
>>> np.atleast_2d(obj1.dot(group1.T))
array([[50, 8]])
Just keep in mind that the dimensions of the dot product will be the the number of objects in the first operand by the number of objects in the second operand (everything will be treated as a group). The attributes will get multiplied and summed together. Whether or not that has a valid interpretation for your purposes is entirely for you to decide.
UPDATE
The only remaining problem at this point is attribute access. As stated above obj1[0] and group1[0] mean very different things. There are three ways to reconcile this difference, listed in the order that I personally prefer them, with 1 being the most preferable:
Use the Ellipsis indexing object to get the last index instead of the first
>>> obj1[..., 0]
array([1])
>>> group1[..., 0]
array([7, 0])
This is the most efficient way since it does not make any copies, just does a normal index on the original arrays. As you can see, there will be no difference between the result from a single object (1D array) and a group with only one object in it (2D array).
Make all your objects 2D. As you pointed out yourself, this can be done with a decorator, and/or using np.atleast_2d. Personally, I would prefer having the convenience of using 1D arrays as single objects without having to wrap them in 2D.
Always access attributes via a transpose:
>>> obj1.T[0]
1
>>> group1.T[0]
array([7, 0])
While this is functionally equivalent to #1, it is clunky and unsightly by comparison, in addition to doing something very different under-the-hood. This approach at the very least creates a new view of the underlying array, and may run the risk of making unnecessary copies in certain cases if the group arrays are not laid out just right. I would not recommend this approach even if it does solve the problem if uniform access.
As far as I can tell, this is not officially not possible, but is there a "trick" to access arbitrary non-sequential elements of a list by slicing?
For example:
>>> L = range(0,101,10)
>>> L
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
Now I want to be able to do
a,b = L[2,5]
so that a == 20 and b == 50
One way besides two statements would be something silly like:
a,b = L[2:6:3][:2]
But that doesn't scale at all to irregular intervals.
Maybe with list comprehension using the indices I want?
[L[x] for x in [2,5]]
I would love to know what is recommended for this common problem.
Probably the closest to what you are looking for is itemgetter (or look here for Python 2 docs):
>>> L = list(range(0, 101, 10)) # works in Python 2 or 3
>>> L
[0, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]
>>> from operator import itemgetter
>>> itemgetter(2, 5)(L)
(20, 50)
If you can use numpy, you can do just that:
>>> import numpy
>>> the_list = numpy.array(range(0,101,10))
>>> the_indices = [2,5,7]
>>> the_subset = the_list[the_indices]
>>> print the_subset, type(the_subset)
[20 50 70] <type 'numpy.ndarray'>
>>> print list(the_subset)
[20, 50, 70]
numpy.array is very similar to list, just that it supports more operation, such as mathematical operations and also arbitrary index selection like we see here.
Just for completeness, the method from the original question is pretty simple. You would want to wrap it in a function if L is a function itself, or assign the function result to a variable beforehand, so it doesn't get called repeatedly:
[L[x] for x in [2,5]]
Of course it would also work for a string...
["ABCDEF"[x] for x in [2,0,1]]
['C', 'A', 'B']
Something like this?
def select(lst, *indices):
return (lst[i] for i in indices)
Usage:
>>> def select(lst, *indices):
... return (lst[i] for i in indices)
...
>>> L = range(0,101,10)
>>> a, b = select(L, 2, 5)
>>> a, b
(20, 50)
The way the function works is by returning a generator object which can be iterated over similarly to any kind of Python sequence.
As #justhalf noted in the comments, your call syntax can be changed by the way you define the function parameters.
def select(lst, indices):
return (lst[i] for i in indices)
And then you could call the function with:
select(L, [2, 5])
or any list of your choice.
Update: I now recommend using operator.itemgetter instead unless you really need the lazy evaluation feature of generators. See John Y's answer.
None of the other answers will work for multidimensional object slicing. IMHO this is the most general solution (uses numpy):
numpy.ix_ allows you to select arbitrary indices in all dimensions of an array simultaneously.
e.g.:
>>> a = np.arange(10).reshape(2, 5) # create an array
>>> a
array([[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]])
>>> ixgrid = np.ix_([0, 1], [2, 4]) # create the slice-like grid
>>> ixgrid
(array([[0],
[1]]), array([[2, 4]]))
>>> a[ixgrid] # use the grid to slice a
array([[2, 4],
[7, 9]])