I have some physical simulation code, written in python and using numpy/scipy. Profiling the code shows that 38% of the CPU time is spent in a single doubly nested for loop - this seems excessive, so I've been trying to cut it down.
The goal of the loop is to create an array of indices, showing which elements of a 1D array the elements of a 2D array are equal to.
indices[i,j] = where(1D_array == 2D_array[i,j])
As an example, if 1D_array = [7.2, 2.5, 3.9] and
2D_array = [[7.2, 2.5]
[3.9, 7.2]]
We should have
indices = [[0, 1]
[2, 0]]
I currently have this implemented as
for i in range(ni):
for j in range(nj):
out[i, j] = (1D_array - 2D_array[i, j]).argmin()
The argmin is needed as I'm dealing with floating point numbers, and so the equality is not necessarily exact. I know that every number in the 1D array is unique, and that every element in the 2D array has a match, so this approach gives the correct result.
Is there any way of eliminating the double for loop?
Note:
I need the index array to perform the following operation:
f = complex_function(1D_array)
output = f[indices]
This is faster than the alternative, as the 2D array has a size of NxN compared with 1xN for the 1D array, and the 2D array has many repeated values. If anyone can suggest a different way of arriving at the same output without going through an index array, that could also be a solution
In pure Python you can do this using a dictionary in O(N) time, the only time penalty is going to be the Python loop involved:
>>> arr1 = np.array([7.2, 2.5, 3.9])
>>> arr2 = np.array([[7.2, 2.5], [3.9, 7.2]])
>>> indices = dict(np.hstack((arr1[:, None], np.arange(3)[:, None])))
>>> np.fromiter((indices[item] for item in arr2.ravel()), dtype=arr2.dtype).reshape(arr2.shape)
array([[ 0., 1.],
[ 2., 0.]])
The dictionary method that some others have suggest might work, but it requires that you know ahead of time that every element in your target array (the 2d array) has an exact match in your search array (your 1d array). Even when this should be true in principle, you still have to deal with floating point precision issues, for example try this .1 * 3 == .3.
Another approach is to use numpy's searchsorted function. searchsorted takes a sorted 1d search array and any traget array then finds the closest elements in the search array for every item in the target array. I've adapted this answer for your situation, take a look at it for a description of how the find_closest function works.
import numpy as np
def find_closest(A, target):
order = A.argsort()
A = A[order]
idx = A.searchsorted(target)
idx = np.clip(idx, 1, len(A)-1)
left = A[idx-1]
right = A[idx]
idx -= target - left < right - target
return order[idx]
array1d = np.array([7.2, 2.5, 3.9])
array2d = np.array([[7.2, 2.5],
[3.9, 7.2]])
indices = find_closest(array1d, array2d)
print(indices)
# [[0 1]
# [2 0]]
To get rid of the two Python for loops, you can do all of the equality comparisons "in one go" by adding new axes to the arrays (making them broadcastable with each other).
Bear in mind that this produces a new array containing len(arr1)*len(arr2) values. If this is a very big number, this approach could be infeasible depending on the limitations of your memory. Otherwise, it should be reasonably quick:
>>> (arr1[:,np.newaxis] == arr2[:,np.newaxis]).argmax(axis=1)
array([[0, 1],
[2, 0]], dtype=int32)
If you need to get the index of the closest matching value in arr1 instead, use:
np.abs(arr1[:,np.newaxis] - arr2[:,np.newaxis]).argmin(axis=1)
Related
Example of what I want to do:
import numpy as np
values = np.array([7, 7, 5, 2, 3, 9])
indices = np.array([
np.array([3,5]),
np.array([4]),
np.array([1,2,3])
])
>>> values[indices]
array([
array([2,9]),
array([3]),
array([7,5,2]),
])
Is it possible to achieve this using vectorization?
Right now I'm doing it with a for loop, but it can get slow.
Thanks!
We could concatenate the indices, index into values with those and finally split back -
idx = np.concatenate(indices)
all_out = values[idx]
lens = list(map(len,indices))
ssidx = np.r_[0,lens].cumsum()
out = [all_out[i:j] for (i,j) in zip(ssidx[:-1],ssidx[1:])]
For completeness, here's the straight-forward indexing based version -
[values[i] for i in indices]
So, with the proposed method we are making use of slicing and hence reducing per-iteration workload. As such, alongwith the step to get idx that needs concatenation of all indices in the proposed one, it makes sense for the case with small indexing arrays in indices.
When dealing with a 3-dimensional matrix "M" of dimensions (A, B, C), one can index M using 2 vectors X with elements in [0, A) and Y with elements in [0, B) of the same dimension D.
More specifically, I understand that when writing
M[X,Y,:]
we are taking, for each "i" in D,
M[X[i], Y[i], :],
thus producing a DxC matrix in the end.
Now suppose
X is a numpy array of dim U, same concept as before
this time Y is a matrix UxL, where each row correspond to a Boolean numpy array
(a mask)
and look at the following code
for u in U:
my_matrix[Y[u], X[u], :] += 1 # Y[u] is the mask that selects specific elements of the first dimension
I would like to write the same code without the for loop. Something like this
np.add.at(my_matrix, (Y, X), 1) # i use numpy.ufunc.at since same elements could occur multiple times in X or Y.
which unfortunately returns the following error
IndexError: boolean index did not match indexed array along dimension 0; dimension is L but corresponding boolean dimension is 1
This issue can also be found when performing assignment
for u in U:
a_matrix[u, Y[u], :] = my_matrix[Y[u], X[u], :]
Do you know how I can address this problem(s) in an elegant way?
The straightforward way of simply using the usual nd-array-shaped fancy indexing is unlikely to work for your problem. Here's why I'm saying this: Y has boolean rows which tell you which indices to take along the first dimension. So Y[0] and Y[1] might have a different amount of True elements, thus the rows of Y would slice subarrays with varying length along the first dimension. In other words, your array-shaped indices can't be translated to a rectangular subarray.
But if you think about what your indexing arrays mean, there's a way out. The rows of Y exactly tell you which elements to modify. If we clutter all the indices to a huge collection of 1d fancy indices, we can pinpoint each (x,y) point along the first dimension which we want to index.
In particular, consider the following example (sorely missing from your question, by the way):
A = np.arange(4*3*2).reshape(4,3,2)
Y = np.array([[True,False,False,True],
[True,True,True,False],
[True,False,False,True]])
X = np.array([2,1,2])
A is shape (4,3,2), Y is shape (3,4) (and the first and last rows are identical on purpose), X is shape (3,)` (and the first and last elements are identical on purpose). Let's turn the boolean indices into a collection of linear indices:
U,inds = Y.nonzero()
#U: array([0, 0, 1, 1, 1, 2, 2])
#inds: array([0, 3, 0, 1, 2, 0, 3])
As you can see, U are the row indices of each True element in Y. These are the indices that give the correspondence between rows of Y and elements of X. The second array, inds are the actual linear indices (for a given row) along the first dimension.
We're almost done, all we need is to pair up the elements of inds with the corresponding index from X for the second dimension. This is actually pretty easy: we just need to index X with U.
So all in all, the following two are equivalent looping and fancy-indexing solutions to the same problem:
B = A.copy()
for u in range(X.size):
A[Y[u],X[u],:] += 1
U,inds = Y.nonzero()
np.add.at(B,(inds,X[U]),1)
A is modified with a loop, B is modified using np.add.at. We can see that the two are equal:
>>> (A == B).all()
True
And if you take a look at the example, you can see that I intentionally duplicated the first and third bunch of indices. This demonstrates that np.add.at is working properly with these fancy indices, complete with accumulating indices that appear multiple times on input. (Printing B and comparing with the initial value of A you can see that the last items are incremented twice.)
Is there a more pythonic/numpythonic way to do some sort of nested/hierarchical slicing, i.e. a prettier version of this:
_sum = 0
for i in np.arange(n):
_sum += someFunc(A[i,:])
Basically I would like to map someFunc (which takes arrays of any shape and returns a number) over the rows and then sum the results.
I have been thinking about np.sum(someFunc(A[:,:])), but according to my understanding this will just map someFuncover the whole array.
If I understood correctly, you could use a list comprehension like this:
sum([someFunc(A[i:]) for i in np.arange(n)])
Define a function to count 1's in an array:
def foo(x):
return (x==1).sum()
and a 2d array:
In [431]: X=np.array([[1,0,2],[3,1,1],[0,2,3]])
I can apply it iteratively to rows
In [432]: [foo(i) for i in X] # iterate on 1st dimension
Out[432]: [1, 2, 0]
In [433]: [foo(X[i,:]) for i in range(3)]
Out[433]: [1, 2, 0]
and get the total count with sum (here the Python sum)
In [434]: sum([foo(X[i,:]) for i in range(3)])
Out[434]: 3
As written foo gets the same thing with applied to the whole array
In [435]: foo(X)
Out[435]: 3
and for row counts, use the np.sum axis control:
In [440]: np.sum(X==1, axis=1)
Out[440]: array([1, 2, 0])
apply_along_axis can to the same sort of row iteration:
In [438]: np.apply_along_axis(foo,1,X)
Out[438]: array([1, 2, 0])
but for this it is overkill. It's more useful with 3d or larger arrays where it is awkward to iterate over all dimensions except the nth one. It's never faster than doing your own iteration.
It's clearly best if you can write the function to work on the whole array. But if you must iterate on rows, there aren't any magical solutions. vectorize and frompyfunc wrap functions that work with scalar values, not 1d arrays. Some row problems are solved by casting the rows as larger dtype objects (e.g. unique rows).
I got a very strange problem about the map function, it will increase a dimension automatically.
matrix = range(4)
matrix = numpy.reshape(matrix,(2,2))
vector = numpy.ones((1,2))
newMatrix = map(lambda line: line/vector, matrix)
np.shape(newMatrix) # I got (2,1,2)
I am confused, the matrix has the shape(2,2), but why after the map() function, the newMatrix has such a shape (2,1,2)? How can I fix with this problem?
I think what you are trying to do is simply newMatrix = matrix / vector. Remember that numpy performs element-wise operations. map is doing what it is defined to do, i.e. return a list after applying your function to each item in the iterator. So map operates on each row of your matrix at a time. You have two rows; thus, your new shape is 2 x 1 x 2.
This example may illustrate what is going on (I replaced your 'matrix', and 'vector' names with neutral variable names)
In [13]: x = np.arange(4).reshape(2,2)
In [14]: y=np.ones((1,2))
In [15]: list(map(lambda line:line/y, x))
Out[15]: [array([[ 0., 1.]]), array([[ 2., 3.]])]
Notice the 2 arrays have shape (1,2), which matches that of y. x[0,:]/y shows this as well. Wrap that list in np.array..., and you get a (2,1,2).
Notice what happens when I use a 1d array, z:
In [16]: z=np.ones((2,))
In [17]: list(map(lambda line:line/z, x))
Out[17]: [array([ 0., 1.]), array([ 2., 3.])]
I ran this sample in Python3, where map returns a generator. To get an array from that I have to use
np.array(list(map(...)))
I don't think I've seen the use of map with numpy arrays before. I'm a little surprised that in Python2 it returns an array, not just a list. A more common version of your iteration is to wrap a list comprehension in np.array...
np.array([line/y for line in x])
But as noted in the other answer, you don't need iteration for this simple case. x/y is sufficient. How to avoid iteration is a frequent SO question.
Say that I have 4 numpy arrays
[1,2,3]
[2,3,1]
[3,2,1]
[1,3,2]
In this case, I've determined [1,2,3] is the "minimum array" for my purposes, as it is one of two arrays with lowest value at index 0, and of those two arrays it has the the lowest index 1. If there were more arrays with similar values, I would need to compare the next index values, and so on.
How can I extract the array [1,2,3] in that same order from the pile?
How can I extend that to x arrays of size n?
Thanks
Using the python non-numpy .sort() or sorted() on a list of lists (not numpy arrays) automatically does this e.g.
a = [[1,2,3],[2,3,1],[3,2,1],[1,3,2]]
a.sort()
gives
[[1,2,3],[1,3,2],[2,3,1],[3,2,1]]
The numpy sort seems to only sort the subarrays recursively so it seems the best way would be to convert it to a python list first. Assuming you have an array of arrays you want to pick the minimum of you could get the minimum as
sorted(a.tolist())[0]
As someone pointed out you could also do min(a.tolist()) which uses the same type of comparisons as sort, and would be faster for large arrays (linear vs n log n asymptotic run time).
Here's an idea using numpy:
import numpy
a = numpy.array([[1,2,3],[2,3,1],[3,2,1],[1,3,2]])
col = 0
while a.shape[0] > 1:
b = numpy.argmin(a[:,col:], axis=1)
a = a[b == numpy.min(b)]
col += 1
print a
This checks column by column until only one row is left.
numpy's lexsort is close to what you want. It sorts on the last key first, but that's easy to get around:
>>> a = np.array([[1,2,3],[2,3,1],[3,2,1],[1,3,2]])
>>> order = np.lexsort(a[:, ::-1].T)
>>> order
array([0, 3, 1, 2])
>>> a[order]
array([[1, 2, 3],
[1, 3, 2],
[2, 3, 1],
[3, 2, 1]])