How can I remove loops in this simple matrix assignment in order to increase performance?
nk,ncol,nrow=index.shape
for kk in range(0,nk):
for ii in range(0,nrow):
for jj in range(0,ncol):
idx=index[kk][ii][jj]
counter[idx][ii][jj]+=1
I come from C++ and I am finding it difficult to adapt to numpy's functions to do some very basic matrix manipulation like this one. I think I have simplified it to a one dimensional loop, but this is still too slow for what I need and it seems to me that there is got to be a more direct way of doing it. Any suggestions? thanks
for kk in range(0,nk):
xx,yy = np.meshgrid(np.arange(ncol),np.arange(nrow))
counter[index[kk,:,:].flatten(),yy.flatten(),xx.flatten()]+=1
If I understand it correctly, you are looking for this:
uniq, counter = np.unique(index, return_counts=True, axis=0)
The uniq should give you unique set of x,ys (x,y will be flattened into a single array) and counter corresponding number of repetitions in the array index
EDIT:
Per OP's comment below:
xx,yy = np.meshgrid(np.arange(ncol),np.arange(nrow))
idx, counts = np.unique(np.vstack((index.flatten(),np.repeat(yy.flatten(),nk),np.repeat(xx.flatten(),nk))), return_counts=True,axis=1)
counter[tuple(idx)] = counts
Related
So I have 2d numpay array arr. It's a relatively big one: arr.shape = (2400, 60000)
What I'm currently doing is the following:
randomly (with replacement) select arr.shape[0] indices
access (row-wise) chosen indices of arr
calculating column-wise averages and selecting max value
I'm repeating it for k times
It looks sth like:
no_rows = arr.shape[0]
indicies = np.array(range(no_rows))
my_vals = []
for k in range(no_samples):
random_idxs = np.random.choice(indicies, size=no_rows, replace=True)
my_vals.append(
arr[random_idxs].mean(axis=0).max()
)
My problem is that is very slow. With my arr size, it takes ~3s for 1 loop. As I want a sample that is bigger than 1k - my current solution solution pretty bad (1k*~3s -> ~1h). I've profiled it and the bottleneck is accessing row based on indices. "mean" and "max" work fast. np.random.choice is also ok.
Do you see any area for improvement? A more efficient way of accessing indices or maybe better a faster approach that solves the problem without this?
What I tried so far:
numpy.take (slower)
numpy.ravel :
sth similar to:
random_idxs = np.random.choice(sample_idxs, size=sample_size, replace=True)
test = random_idxs.ravel()[arr.ravel()].reshape(arr.shape)
similar approach to current one but without loop. I created 3d arr and accessed rows across additional dimension in one go
Since advanced indexing will generate a copy, the program will allocate huge memory in arr[random_idxs].
So one of the most simple way to improve efficiency is that do things batch wise.
BATCH = 512
max(arr[random_idxs,i:i+BATCH].mean(axis=0).max() for i in range(0,arr.shape[1],BATCH))
This is not a general solution to the problem, but should make your specific problem much faster. Basically, arr.mean(axis=0).max() won't change, so why not take random samples from that array?
Something like:
mean_max = arr.mean(axis=0).max()
my_vals = np.array([np.random.choice(mean_max, size=len(mean_max), replace=True) for i in range(no_samples)])
You may even be able to do: my_vals = np.random.choice(mean_max, size=(no_samples, len(mean_max)), replace=True), but I'm not sure how, if at all, that would change your statistics.
I was trying to take advantage of the Broadcasting property of Python while replacing the for loop of this snippet:
import numpy as np
B = np.random.randn(10,1)
k = 25
for i in range(len(B)):
B[i][0]= B[i][0] + k
with this:
for i in range((lenB)):
B=B+k
I observed that I was getting different results. When I tried outside the loop, B = B+k, gave the same results as what I was expecting with B[i][0] = B[i][0] + k
Why is this so? Does Broadcasting follow different rules inside loops?
In your 2nd option you ment to do the following:
B=B+k
As you see, you don't need the for loop and it is MUCH faster than looping over the "vector" (numpy array).
It is some form of "Vectorization" calculation instead of iterative calculation which is better in terms of complexity and readability. Both will yield the same result.
You can see a lot of examples on Vectorization vs Iteration, including running time, here.
And you can see a great video of Andrew Ng going over numpy broadcasting property.
For an assignment I have to use different combinations of features belonging to some data, to evaluate a classification system. By features I mean measurements, e.g. height, weight, age, income. So for instance I want to see how well a classifier performs when given just the height and weight to work with, and then the height and age say. I not only want to be able to test what two features work best together, but also what 3 features work best together and would like to be able to generalise this to n features.
I've been attempting this using numpy's mgrid, to create n dimensional arrays, flattening them, and then making arrays that use the same elements from each array to create new ones. Tricky to explain so here is some code and psuedo code:
import numpy as np
def test_feature_combos(data, combinations):
dimensions = combinations.shape[0]
grid = np.empty(dimensions)
for i in xrange(dimensions):
grid[i] = combinations[i].flatten()
#The above code throws an error "setting an array element with a sequence" error which I understand, but this shows my approach.
**Pseudo code begin**
For each element of each element of this new array,
create a new array like so:
[[1,1,2,2],[1,2,1,2]] ---> [[1,1],[1,2],[2,1],[2,2]]
Call this new array combo_indices
Then choose the columns (features) from the data in a loop using:
new_data = data[:, combo_indices[j]]
combinations = np.mgrid[1:5,1:5]
test_feature_combos(data, combinations)
I concede that this approach means a lot of unnecessary combinations due to repeats, however I cannot even implement this so beggars can not be choosers.
Please can someone advise me on how I can either a) implement my approach or b) achieve this goal in a much more elegant way.
Thanks in advance, and let me know if any clarification needs to be made, this was tough to explain.
To generate all combinations of k elements drawn without replacement from a set of size n you can use itertools.combinations, e.g.:
idx = np.vstack(itertools.combinations(range(n), k)) # an (n, k) array of indices
For the special case where k=2 it's often faster to use the indices of the upper triangle of an n x n matrix, e.g.:
idx = np.vstack(np.triu_indices(n, 1)).T
I have a code in python with the following elements:
I have an intensities vector which is something like this:
array([ 1142., 1192., 1048., ..., 29., 18., 35.])
I have also an x vector which looks like this:
array([ 0, 1, 1, ..., 1060, 1060, 1061])
Then, I have the for loop where I fill another vector, radialDistribution like this:
for i in range(1000):
radialDistribution[i] = sum(intensities[np.where(x == i)]) / len(np.where(x == i)[0])
The problem is that it takes 20 second to complete it...therefore I want to vectorize it. But I am quite new with broadcasting in Numpy and didn't find so much out there...therefore I need your help.
I tried this, but didn't work:
i= np.ogrid[:1000]
intensities[i] = sum(sortedIntensities1D[np.where(sortedDists1D == i)]) / len(np.where(sortedDists1D == i)[0])
Could you help me just telling me where should I look to learn the vectorization procedures with Numpy?
Thanks in advance for your valuable help!
If your x vector has consecutive integers starting at 0, then you can simply do:
radialDistribution = np.bincount(x, weights=intensities) / np.bincount(x)
Here is my implementation of group_by functionality in numpy. It is conceptually similar to the pandas solution; except that this does not require pandas, and ought to become a part of the numpy core, in my opinion.
Using this functionality, your code would look like this:
radialDistribution = group_by(x).mean(intensities)
and would complete in notime.
Look also at the test_radial function defined at the end, which may come even closer to your endgoal.
Here's a method that uses broadcasting:
# arrays need to be at least 2D for broadcasting
x = np.atleast_2d(x)
# create vector of indices
i = np.atleast_2d(np.arange(x.size))
# do the vectorized calculation
bool_eq = (x == i.T)
totals = np.sum(np.where(bool_eq, intensities, 0), axis=1)
rD = totals / np.sum(bool_eq, axis=1)
This uses broadcasting two times: in the operation x == i.T and in the call to np.where. Unfortunately the code above is very slow, even slower than the original. The main bottleneck here is np.where, which we can speed up in this case by taking the product of the Boolean array and the intensities (also by broadcasting):
totals = np.sum(bool_eq*intensities, axis=1)
And this is essentially the same as a matrix-vector product, so we can write:
totals = np.dot(intensities, bool_eq.T)
The end result is a faster code than the original (at least until the memory use for the intermediary array becomes the limiting factor), but you're probably better off with an iterative approach, as suggested by one of the other answers.
Edit: making use of np.einsum was faster still (in my trial):
totals = np.einsum('ij,j', bool_eq, intensities)
Building on my itertools.groupby solution in https://stackoverflow.com/a/22265803/901925 here's a solution that works on 2 small arrays.
import numpy as np
import itertools
intensities = np.arange(12,dtype=float)
x=np.array([1,0,1,2,2,1,0,0,1,2,1,0]) # general, not sorted or consecutive
first a bincount solution, adjusted for nonconsecutive values
# using bincount
# if 'x' are not consecutive
J=np.bincount(x)>0
print np.bincount(x,weights=intensities)[J]/np.bincount(x)[J]
Now a groupby solution
# using groupby;
# sort if need
I=np.argsort(x)
x=x[I]
intensities=intensities[I]
# make a record array for use by groupby
xi=np.zeros(shape=x.shape, dtype=[('intensities',float),('x',int)])
xi['intensities']=intensities
xi['x']=x
g=itertools.groupby(xi, lambda z:z['x'])
xx=np.array([np.array([z[0] for z in y[1]]).mean() for y in g])
print xx
Here's a compact numpy solution, using the return_index option of np.unique, and np.split. x should be sorted. I'm not optimistic about the speed for large arrays, since there will be iteration in unique and split in addition to the comprehension.
[values, index] = np.unique(x, return_index=True)
[y.mean() for y in np.split(intensities, index[1:])]
My question is about a specific array operation that I want to express using numpy.
I have an array of floats w and an array of indices idx of the same length as w and I want to sum up all w with the same idx value and collect them in an array v.
As a loop, this looks like this:
for i, x in enumerate(w):
v[idx[i]] += x
Is there a way to do this with array operations?
My guess was v[idx] += w but that does not work, since idx contains the same index multiple times.
Thanks!
numpy.bincount was introduced for this purpose:
tmp = np.bincount(idx, w)
v[:len(tmp)] += tmp
I think as of 1.6 you can also pass a minlength to bincount.
This is a known behavior and, though somewhat unfortunate, does not have a numpy-level workaround. (bincount can be used for this if you twist its arm.) Doing the loop yourself is really your best bet.
Note that your code might have been a bit more clear without re-using the name w and without introducing another set of indices, like
for i, w_thing in zip(idx, w):
v[i] += w_thing
If you need to speed up this loop, you might have to drop down to C. Cython makes this relatively easy.