Clean array indexing with arrays in numpy - python

I'm running into this problem over and over again, and can't seem to find a clean solution for this. So I'm trying to index an array with another array. I have a 2d numpy array. And a 1d numpy array with the same length as the 1st dimension of the 2d array I'm trying to index and the elements represent the indices of the columns I try to extract:
import numpy as np
A = np.random.rand((5,3))
B = np.asarray([2,1,2,0,1])
The behaviour that I want is extracting for all rows the corresponding column in array B. This could be done by
C = A[np.arange(A.shape[0]),B]
But I can imagine that there is a better way to get this behaviour. Using a : as indexing the first row gives the wrong behaviour.
If there is a cleaner way of doing this that would be great. I'm really used to this array indexing from Matlab, but maybe there is no equivalent in numpy. Using boolean indices is of course an option, but that also requires converting arrays all the time.

I think what you look after is np.choose(B,A.T) :
In [125]: A
Out[125]:
array([[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11],
[12, 13, 14]])
In [126]: B = np.asarray([2,1,2,0,1])
In [127]: np.choose(B,A.T)
Out[127]: array([ 2, 4, 8, 9, 13])

In [60]: A = np.arange(1,16).reshape(5,3)
In [61]: B = np.array([2,1,2,0,1])
In [62]: C = A[np.arange(A.shape[0]),B]
In [63]: C
Out[63]: array([ 3, 5, 9, 10, 14])
In Octave
>> A = reshape(1:15, 3,5).';
>> B = [3,2,3,1,2];
>> A
A =
1 2 3
4 5 6
7 8 9
10 11 12
13 14 15
>> A(:,B)
ans =
3 2 3 1 2
6 5 6 4 5
9 8 9 7 8
12 11 12 10 11
15 14 15 13 14
This is the same as numpy:
In [65]: A[:,B]
Out[65]:
array([[ 3, 2, 3, 1, 2],
[ 6, 5, 6, 4, 5],
[ 9, 8, 9, 7, 8],
[12, 11, 12, 10, 11],
[15, 14, 15, 13, 14]])
You imply that there's something clean for indexing one item from each column in MATLAB/Octave, but I missing it. I used to work a lot in that language, but I've gotten out of practice.
sub2ind does the job:
>> sub2ind([3,5],B, 1:5)
ans =
3 5 9 10 14
>> A.'(:)(sub2ind([3,5],B,1:5))
ans =
3
5
9
10
14
(The 'F' vs 'C' ordering is complicating my comparison)
numpy has a similar ravel_multi_index:
In [69]: np.ravel_multi_index((np.arange(5),B),(5,3))
Out[69]: array([ 2, 4, 8, 9, 13], dtype=int32)
In [71]: A.flat[_]
Out[71]: array([ 3, 5, 9, 10, 14])

Related

Extract sub-array from 2D array using logical indexing - python

I am trying to extract a sub-array using logical indexes as,
a = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]])
a
Out[45]:
array([[ 1, 2, 3, 4],
[ 5, 6, 7, 8],
[ 9, 10, 11, 12],
[13, 14, 15, 16]])
b = np.array([False, True, False, True])
a[b, b]
Out[49]: array([ 6, 16])
python evaluates the logical indexes in b per element of a. However in matlab you can do something like
>> a = [1 2 3 4; 5 6 7 8; 9 10 11 12; 13 14 15 16]
a =
1 2 3 4
5 6 7 8
9 10 11 12
13 14 15 16
>> b = [2 4]
b =
2 4
>> a(b, b)
ans =
6 8
14 16
how can I achieve the same result in python without doing,
c = a[:, b]
c[b,:]
Out[51]:
array([[ 6, 8],
[14, 16]])
Numpy supports logical indexing, though it is a little different than what you are familiar in MATLAB. To get the results you want you can do the following:
a[b][:,b] # first brackets isolates the rows, second brackets isolate the columns
Out[27]:
array([[ 6, 8],
[14, 16]])
The more "numpy" method will be understood after you will understand what happend in your case.
b = np.array([False, True, False, True]) is similar to b=np.array([1,3]) and will be easier for me to explain. When writing a[[1,3],[1,3]] what happens is that numpy crates a (2,1) shape array, and places a[1,1] in the [0] location and a[3,3] in the second location. To create an output of shape (2,2), the indexing must have the same dimensionality. Therefore, the following will get your result:
a[[[1,1],[3,3]],[[1,3],[1,3]]]
Out[28]:
array([[ 6, 8],
[14, 16]])
Explanation:
The indexing arrays are:
temp_rows = np.array([[1,1],
[3,3]])
temp_cols = np.array([[1,3],
[1,3])
both arrays have dimensions of (2,2) and therefore, numpy will create an output of shape (2,2). Then, it places a[1,1] in location [0,0], a[1,3] in [0,1], a[3,1] in location [1,0] and a[3,3] in location [1,1]. This can be expanded to any shape but for your purposes, you wanted a shape of (2,2)
After figuring this out, you can make things even simpler by utilizing the fact you if you insert a (2,1) array in the 1st dimension and a (1,2) array in the 2nd dimension, numpy will perform the broadcasting, similar to the MATLAB operation. This means that by using:
temp_rows = np.array([[1],[3]])
temp_cols = np.array([1,3])
you can do:
a[[[1],[3]], [1,3])
Out[29]:
array([[ 6, 8],
[14, 16]])
You could use np.ix_ here.
a[np.ix_(b, b)]
# array([[ 6, 8],
# [14, 16]])
Output returned by np.ix_
>>> np.ix_(b, b)
(array([[1],
[3]]),
array([[1, 3]]))
You could make use of a outer product of the b vector. The new dimesion you can obtain from the number of True values using a sum.
import numpy as np
a = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]])
b = np.array([False, True, False, True])
#
M = np.outer(b, b)
new_dim = b.sum()
new_shape = (new_dim, new_dim)
selection = a[M].reshape(new_shape)
The result looks like
[[ 6 8]
[14 16]]

Python realization of Matlab 'Outer product'

I am trying to rewrite the following snippet of Matlab code about outer product of matrices into python code,
function Y = matlab_outer_product(X,x)
A = reshape(X, [size(X) ones(1,ndims(x))]);
B = reshape(x, [ones(1,ndims(X)) size(x)]);
Y = squeeze(bsxfun(#times,A,B));
end
My one-to-one translation of this to python code is as following (considering how the shape of numpy array and matlab matrices are arranged),
def python_outer_product(X, x):
X_shape = list(X.shape)
x_shape = list(x.shape)
A = X.reshape(*list(np.ones(np.ndim(x),dtype=int)),*X_shape)
B = x.reshape(*x_shape,*list(np.ones(np.ndim(X),dtype=int)))
Y = A*B
return Y.squeeze()
Then trying the inputs, for instance,
matlab_outer_product([1,2],[[3,4];[5,6]])
python_out_product(np.array([[1,2]], np.array([[3,4],[5,6]])))
The outputs don't quite match. In matlab, it outputs
output(:,:,1) = [[3,5];[6,10]]
output(:,:,2) = [[4,6];[8,12]]
In python, it outputs
output = array([
[[ 3, 6],
[ 4, 8]],
[[ 5, 10],
[ 6, 12]]
])
They're almost identical, but not quite. I wonder what's wrong with code and how to change the python code to match with matlab output?
In full gory detail (since my MATLAB memory is old):
Octave
>> X = [1,2];
>> x = [[3,4];[5,6]];
>> A = reshape(X, [size(X) ones(1,ndims(x))]);
>> B = reshape(x, [ones(1,ndims(X)) size(x)]);
>> A
A =
1 2
>> B
B =
ans(:,:,1,1) = 3
ans(:,:,2,1) = 5
ans(:,:,1,2) = 4
ans(:,:,2,2) = 6
>> bsxfun(#times,A,B)
ans =
ans(:,:,1,1) =
3 6
ans(:,:,2,1) =
5 10
ans(:,:,1,2) =
4 8
ans(:,:,2,2) =
6 12
>> squeeze(bsxfun(#times,A,B))
ans =
ans(:,:,1) =
3 5
6 10
ans(:,:,2) =
4 6
8 12
You start with a (1,2) and (2,2), expand the second to (1,1,2,2). The bsxfun produces a (1,2,2,2) which is squeezed to (2,2,2).
A is X reshaped to [1 2 1 1], but the two outer size 1 dimensions are squeeze out, resulting in no change.
This MATLAB outter is a bit convoluted, using bsxfun to perform elementwise multiplication of (1,2,1,1) with (1,1,1,2). At least in Octave it's the same as
A.*B
In numpy
In [77]: X
Out[77]: array([[1, 2]]) # (1,2)
In [78]: x
Out[78]:
array([[3, 4], # (2,2)
[5, 6]])
Note that the MATLAB/Octave x when flattened has elements (3,5,4,6), while the numpy ravel is [3,4,5,6].
In numpy I can simply do:
In [79]: X[:,:,None,None]*x
Out[79]:
array([[[[ 3, 4], (1,2,2,2)
[ 5, 6]],
[[ 6, 8],
[10, 12]]]])
or without the extra size 1 dimension of X:
In [84]: (X[0,:,None,None]*x)
Out[84]:
array([[[ 3, 4],
[ 5, 6]],
[[ 6, 8],
[10, 12]]])
In [85]: (X[0,:,None,None]*x).ravel()
Out[85]: array([ 3, 4, 5, 6, 6, 8, 10, 12])
compare that with the Octave ravel
>> squeeze(bsxfun(#times,A,B))(:)'
ans =
3 6 5 10 4 8 6 12
We could add a transpose to the numpy
In [96]: (X[0,:,None,None]*x).transpose(2,1,0).ravel()
Out[96]: array([ 3, 6, 5, 10, 4, 8, 6, 12])
In [97]: (X[0,:,None,None]*x).transpose(2,1,0)
Out[97]:
array([[[ 3, 6],
[ 5, 10]],
[[ 4, 8],
[ 6, 12]]])
At least in numpy we can tweak the dimension order in lots of ways, so I won't try to suggest an optimal. I still think it's better to write code that's "natural" to numpy than to slavishly match the MATLAB order.
another try
I realized, above, that the MATLAB is just doing A*.B with
(1,2,1,1) arrays (1,1,1,2), where the extra 1's were added to "broadcast".
Using transpose to the same dimension outermost (leading in numpy)
In [5]: X = X.T; x = x.T
In [6]: X.shape
Out[6]: (2, 1)
In [7]: x.shape
Out[7]: (2, 2)
In [8]: x
Out[8]:
array([[3, 5],
[4, 6]])
In [9]: x.ravel()
Out[9]: array([3, 5, 4, 6]) # compare with MATLAB (:)'
Elementwise multiplication with the same dimension expansion:
In [10]: X[None,None,:,:]*x[:,:,None,None]
Out[10]:
array([[[[ 3],
[ 6]],
[[ 5],
[10]]],
[[[ 4],
[ 8]],
[[ 6],
[12]]]])
In [11]: _.shape
Out[11]: (2, 2, 2, 1) # compare with octave (1,2,2,2)
In [12]: __.squeeze()
Out[12]:
array([[[ 3, 6],
[ 5, 10]],
[[ 4, 8],
[ 6, 12]]])
the ravel is the same as Octave:
In [13]: ___.ravel()
Out[13]: array([ 3, 6, 5, 10, 4, 8, 6, 12])
expand_dims can be used instead of the indexing. Internally it uses reshape:
In [15]: np.expand_dims(X,(0,1)).shape
Out[15]: (1, 1, 2, 1)
In [16]: np.expand_dims(x,(2,3)).shape
Out[16]: (2, 2, 1, 1)

Filter a 2D numpy array from an array of values

Let's say I have a numpy array with the following shape :
nonSortedNonFiltered=np.array([[9,8,5,4,6,7,1,2,3],[1,3,2,6,4,5,7,9,8]])
I want to :
- Sort the array according to nonSortedNonFiltered[1]
- Filter the array according to nonSortedNonFiltered[0] and an array of values
I currently do the sorting with :
sortedNonFiltered=nonSortedNonFiltered[:,nonSortedNonFiltered[1].argsort()]
Which gives : np.array([[9 5 8 6 7 4 1 3 2],[1 2 3 4 5 6 7 8 9]])
Now I want to filter sortedNonFiltered from an array of values, for example :
sortedNonFiltered=np.array([[9 5 8 6 7 4 1 3 2],[1 2 3 4 5 6 7 8 9]])
listOfValues=np.array([8 6 5 2 1])
...Something here...
> np.array([5 8 6 1 2],[2 3 4 7 9]) #What I want to get in the end
Note : Each value in a column of my 2D array is exclusive.
You can use np.in1d to get a boolean mask and use it to filter columns in the sorted array, something like this -
output = sortedNonFiltered[:,np.in1d(sortedNonFiltered[0],listOfValues)]
Sample run -
In [76]: nonSortedNonFiltered
Out[76]:
array([[9, 8, 5, 4, 6, 7, 1, 2, 3],
[1, 3, 2, 6, 4, 5, 7, 9, 8]])
In [77]: sortedNonFiltered
Out[77]:
array([[9, 5, 8, 6, 7, 4, 1, 3, 2],
[1, 2, 3, 4, 5, 6, 7, 8, 9]])
In [78]: listOfValues
Out[78]: array([8, 6, 5, 2, 1])
In [79]: sortedNonFiltered[:,np.in1d(sortedNonFiltered[0],listOfValues)]
Out[79]:
array([[5, 8, 6, 1, 2],
[2, 3, 4, 7, 9]])

Numpy multiply multiple columns by scalar

This seems like a really simple question but I can't find a good answer anywhere. How might I multiply (in place) select columns (perhaps selected by a list) by a scalar using numpy?
E.g. Multiply columns 0 and 2 by 4
In: arr=([(1,2,3,5,6,7), (4,5,6,2,5,3), (7,8,9,2,5,9)])
Out: arr=([(4,2,12,5,6,7), (16,5,24,2,5,3), (28,8,36,2,5,9)])
Currently I am doing this in multiple steps but I feel like there must be a better way especially if the list gets larger. Current way:
arr['f0'] *= 4
arr['f2'] *= 4
You can use array slicing as follows for this -
In [10]: arr=([(1,2,3,5,6,7), (4,5,6,2,5,3), (7,8,9,2,5,9)])
In [11]: narr = np.array(arr)
In [13]: narr[:,(0,2)] = narr[:,(0,2)]*4
In [14]: narr
Out[14]:
array([[ 4, 2, 12, 5, 6, 7],
[16, 5, 24, 2, 5, 3],
[28, 8, 36, 2, 5, 9]])
An alternate solution is to create a class that inherits from np.ndarray and add a method to it to make the in-place mutation more intuitive.
Code:
import numpy as np
class A(np.ndarray):
def __new__(cls, a):
arr = np.asarray(a)
arr = arr.view(cls)
return arr
def mutate(self, col, k):
self[:,col] *= k
return self
a = A([(1,2,3,5,6,7), (4,5,6,2,5,3), (7,8,9,2,5,9)])
print a
print '---------------------'
a.mutate(0, 4)
a.mutate(2, 4)
print a
Result:
[[1 2 3 5 6 7]
[4 5 6 2 5 3]
[7 8 9 2 5 9]]
---------------------
[[ 4 2 12 5 6 7]
[16 5 24 2 5 3]
[28 8 36 2 5 9]]
You can use the following along with array slicing
arr=([(1,2,3,5,6,7), (4,5,6,2,5,3), (7,8,9,2,5,9)])
array = np.array(arr)
array[:,(0,2)]*=4
array
Out[10]:
array([[ 4, 2, 12, 5, 6, 7],
[16, 5, 24, 2, 5, 3],
[28, 8, 36, 2, 5, 9]])

Any function in numpy/pandas/python to search and replace

I have matrix of 4x4 like this
ds1=
4 13 6 9
7 12 5 7
7 0 4 22
9 8 12 0
and other file with two columns:
ds2 =
4 1
5 3
6 1
7 2
8 2
9 3
12 1
13 2
22 3
ds1 = ds1.apply(lambda x: ds2_mean[1] if [condition])
What condition to be added to compare and check that elements from ds1 and ds2 are equal?
I want col1 value from 2nd matrix to be replaced by col2 value in matrix 1, so resultant matrix should look like
1 2 1 3
2 1 3 2
2 0 1 3
3 2 1 0
please see Replacing mean value from one dataset to another this does not answer my question
If you are working with numpy arrays, you could do this -
# Make a copy of ds1 to initialize output array
out = ds1.copy()
# Find out the row indices in ds2 that have intersecting elements between
# its first column and ds1
_,C = np.where(ds1.ravel()[:,None] == ds2[:,0])
# New values taken from the second column of ds2 to be put in output
newvals = ds2[C,1]
# Valid positions in output array to be changed
valid = np.in1d(ds1.ravel(),ds2[:,0])
# Finally make the changes to get desired output
out.ravel()[valid] = newvals
Sample input, output -
In [79]: ds1
Out[79]:
array([[ 4, 13, 6, 9],
[ 7, 12, 5, 7],
[ 7, 0, 4, 22],
[ 9, 8, 12, 0]])
In [80]: ds2
Out[80]:
array([[ 4, 1],
[ 5, 3],
[ 6, 1],
[ 7, 2],
[ 8, 2],
[ 9, 3],
[12, 1],
[13, 2],
[22, 3]])
In [81]: out
Out[81]:
array([[1, 2, 1, 3],
[2, 1, 3, 2],
[2, 0, 1, 3],
[3, 2, 1, 0]])
Here is another solution. Using DataFrame.replace() function.
df1.replace(to_replace= df2[0].tolist(), value= df2[1].tolist, inplace=True)

Categories