My question is about a specific array operation that I want to express using numpy.
I have an array of floats w and an array of indices idx of the same length as w and I want to sum up all w with the same idx value and collect them in an array v.
As a loop, this looks like this:
for i, x in enumerate(w):
v[idx[i]] += x
Is there a way to do this with array operations?
My guess was v[idx] += w but that does not work, since idx contains the same index multiple times.
Thanks!
numpy.bincount was introduced for this purpose:
tmp = np.bincount(idx, w)
v[:len(tmp)] += tmp
I think as of 1.6 you can also pass a minlength to bincount.
This is a known behavior and, though somewhat unfortunate, does not have a numpy-level workaround. (bincount can be used for this if you twist its arm.) Doing the loop yourself is really your best bet.
Note that your code might have been a bit more clear without re-using the name w and without introducing another set of indices, like
for i, w_thing in zip(idx, w):
v[i] += w_thing
If you need to speed up this loop, you might have to drop down to C. Cython makes this relatively easy.
Related
Be a an ndarray, e. g.:
a = np.random.randn(Size)
Where Size >> 1. Is it possible to define an array b s.t. its i-th element depends on all of the elements of a up to i (excluded or included is not the problem) without a for loop?
b[i] = function(a[:i])
So if function was simply np.sum(a[:i]) my desired output would be:
for i in range(1, Size):
b[i] = np.sum(a[:i])
The only solution I was able to think about was to write the corresponding C code and wrap it, but is there some python native solution to avoid it???
I stress that the sum is a mere ex., I'm lookin for a generalization to arbitrary function that can, howewver, be expressed elementwise by means of numpy mathematical function (np.exp() e.g.)
Many of the ufunc have an accumulate method. np.cumsum is basically np.add.accumulate. But if you can't use one of those, or some clever combination, and you still want speed, you will need to write some sort of compiled code. numba seems to be preferred tool these days.
In your example use just numpy cumsum operation https://numpy.org/doc/stable/reference/generated/numpy.cumsum.html.
Edit:
For example, if you create a = np.ones(10) with all values equal 1. Then b = np.cumsum(a) will contain [1 2 ... 10].
Or as you wanted:
for i in range(1, Size):
b[i] = np.sum(a[:i])
Also you can specify axis to apply cumsum to or maybe use numpy.cumprod (same operation but with product).
How can I remove loops in this simple matrix assignment in order to increase performance?
nk,ncol,nrow=index.shape
for kk in range(0,nk):
for ii in range(0,nrow):
for jj in range(0,ncol):
idx=index[kk][ii][jj]
counter[idx][ii][jj]+=1
I come from C++ and I am finding it difficult to adapt to numpy's functions to do some very basic matrix manipulation like this one. I think I have simplified it to a one dimensional loop, but this is still too slow for what I need and it seems to me that there is got to be a more direct way of doing it. Any suggestions? thanks
for kk in range(0,nk):
xx,yy = np.meshgrid(np.arange(ncol),np.arange(nrow))
counter[index[kk,:,:].flatten(),yy.flatten(),xx.flatten()]+=1
If I understand it correctly, you are looking for this:
uniq, counter = np.unique(index, return_counts=True, axis=0)
The uniq should give you unique set of x,ys (x,y will be flattened into a single array) and counter corresponding number of repetitions in the array index
EDIT:
Per OP's comment below:
xx,yy = np.meshgrid(np.arange(ncol),np.arange(nrow))
idx, counts = np.unique(np.vstack((index.flatten(),np.repeat(yy.flatten(),nk),np.repeat(xx.flatten(),nk))), return_counts=True,axis=1)
counter[tuple(idx)] = counts
I was unable to find anything describing how to do this, which leads to be believe I'm not doing this in the proper idiomatic Python way. Advice on the 'proper' Python way to do this would also be appreciated.
I have a bunch of variables for a datalogger I'm writing (arbitrary logging length, with a known maximum length). In MATLAB, I would initialize them all as 1-D arrays of zeros of length n, n bigger than the number of entries I would ever see, assign each individual element variable(measurement_no) = data_point in the logging loop, and trim off the extraneous zeros when the measurement was over. The initialization would look like this:
[dData gData cTotalEnergy cResFinal etc] = deal(zeros(n,1));
Is there a way to do this in Python/NumPy so I don't either have to put each variable on its own line:
dData = np.zeros(n)
gData = np.zeros(n)
etc.
I would also prefer not just make one big matrix, because keeping track of which column is which variable is unpleasant. Perhaps the solution is to make the (length x numvars) matrix, and assign the column slices out to individual variables?
EDIT: Assume I'm going to have a lot of vectors of the same length by the time this is over; e.g., my post-processing takes each log file, calculates a bunch of separate metrics (>50), stores them, and repeats until the logs are all processed. Then I generate histograms, means/maxes/sigmas/etc. for all the various metrics I computed. Since initializing 50+ vectors is clearly not easy in Python, what's the best (cleanest code and decent performance) way of doing this?
If you're really motivated to do this in a one-liner you could create an (n_vars, ...) array of zeros, then unpack it along the first dimension:
a, b, c = np.zeros((3, 5))
print(a is b)
# False
Another option is to use a list comprehension or a generator expression:
a, b, c = [np.zeros(5) for _ in range(3)] # list comprehension
d, e, f = (np.zeros(5) for _ in range(3)) # generator expression
print(a is b, d is e)
# False False
Be careful, though! You might think that using the * operator on a list or tuple containing your call to np.zeros() would achieve the same thing, but it doesn't:
h, i, j = (np.zeros(5),) * 3
print(h is i)
# True
This is because the expression inside the tuple gets evaluated first. np.zeros(5) therefore only gets called once, and each element in the repeated tuple ends up being a reference to the same array. This is the same reason why you can't just use a = b = c = np.zeros(5).
Unless you really need to assign a large number of empty array variables and you really care deeply about making your code compact (!), I would recommend initialising them on separate lines for readability.
Nothing wrong or un-Pythonic with
dData = np.zeros(n)
gData = np.zeros(n)
etc.
You could put them on one line, but there's no particular reason to do so.
dData, gData = np.zeros(n), np.zeros(n)
Don't try dData = gData = np.zeros(n), because a change to dData changes gData (they point to the same object). For the same reason you usually don't want to use x = y = [].
The deal in MATLAB is a convenience, but isn't magical. Here's how Octave implements it
function [varargout] = deal (varargin)
if (nargin == 0)
print_usage ();
elseif (nargin == 1 || nargin == nargout)
varargout(1:nargout) = varargin;
else
error ("deal: nargin > 1 and nargin != nargout");
endif
endfunction
In contrast to Python, in Octave (and presumably MATLAB)
one=two=three=zeros(1,3)
assigns different objects to the 3 variables.
Notice also how MATLAB talks about deal as a way of assigning contents of cells and structure arrays. http://www.mathworks.com/company/newsletters/articles/whats-the-big-deal.html
If you put your data in a collections.defaultdict you won't need to do any explicit initialization. Everything will be initialized the first time it is used.
import numpy as np
import collections
n = 100
data = collections.defaultdict(lambda: np.zeros(n))
for i in range(1, n):
data['g'][i] = data['d'][i - 1]
# ...
How about using map:
import numpy as np
n = 10 # Number of data points per array
m = 3 # Number of arrays being initialised
gData, pData, qData = map(np.zeros, [n] * m)
I have a very large 400x300x60x27 array (lets call it 'A'). I took the maximum values which is now a 400x300x60 array called 'B'. Basically I need to find the index in 'A' of each value in 'B'. I have converted them both to lists and set up a for loop to find the indices, but it takes an absurdly long time to get through it because there are over 7 million values. This is what I have:
B=np.zeros((400,300,60))
C=np.zeros((400*300*60))
B=np.amax(A,axis=3)
A=np.ravel(A)
A=A.tolist()
B=np.ravel(B)
B=B.tolist()
for i in range(0,400*300*60):
C[i]=A.index(B[i])
Is there a more efficient way to do this? Its taking hours and hours and the program is still stuck on the last line.
You don't need amax, you need argmax. In case of argmax, the array will only contain the indices rather than values, the computational efficiency of finding the values using indices are much better than vice versa.
So, I would recommend you to store only the indices. Before flattening the array.
instead of np.amax, run A.argmax, this will contain the indices.
But before you're flattening it to 1D, you will need to use a mapping function that causes the indices to 1D as well. This is probably a trivial problem, as you'd need to just use some basic operations to achieve this. But that would also consume some time as it needs to be executed quite some times. But it won't be a searching probem and would save you quite some time.
You are getting those argmax indices and because of the flattening, you are basically converting to linear index equivalents of those.
Thus, a solution would be to add in the proper offsets into the argmax indices in steps leveraging broadcasting at each one of them, like so -
m,n,r,s = A.shape
idx = A.argmax(axis=3)
idx += s*np.arange(r)
idx += r*s*np.arange(n)[:,None]
idx += n*r*s*np.arange(m)[:,None,None] # idx is your C output
Alternatively, a compact way to put it would be like so -
m,n,r,s = A.shape
I,J,K = np.ogrid[:m,:n,:r]
idx = n*r*s*I + r*s*J + s*K + A.argmax(axis=3)
If I am using the sparse.lil_matrix format, how can I remove a column from the matrix easily and efficiently?
Much simpler and faster. You might not even need the conversion to csr, but I just know for sure that it works with csr sparse matrices and converting between shouldn't be an issue.
from scipy import sparse
x_new = sparse.lil_matrix(sparse.csr_matrix(x)[:,col_list])
I've been wanting this myself and in truth there isn't a great built-in way to do it yet. Here's a way to do it. I chose to make a subclass of lil_matrix and add the remove_col function. If you want, you can instead add the removecol function to the lil_matrix class in your lib/site-packages/scipy/sparse/lil.py file. Here's the code:
from scipy import sparse
from bisect import bisect_left
class lil2(sparse.lil_matrix):
def removecol(self,j):
if j < 0:
j += self.shape[1]
if j < 0 or j >= self.shape[1]:
raise IndexError('column index out of bounds')
rows = self.rows
data = self.data
for i in xrange(self.shape[0]):
pos = bisect_left(rows[i], j)
if pos == len(rows[i]):
continue
elif rows[i][pos] == j:
rows[i].pop(pos)
data[i].pop(pos)
if pos == len(rows[i]):
continue
for pos2 in xrange(pos,len(rows[i])):
rows[i][pos2] -= 1
self._shape = (self._shape[0],self._shape[1]-1)
I have tried it out and don't see any bugs. I certainly think that it is better than slicing the column out, which just creates a new matrix as far as I know.
I decided to make a removerow function as well, but I don't think that it is as good as removecol. I'm limited by not being able to remove one row from an ndarray in the way that I would like. Here is removerow which can be added to the above class
def removerow(self,i):
if i < 0:
i += self.shape[0]
if i < 0 or i >= self.shape[0]:
raise IndexError('row index out of bounds')
self.rows = numpy.delete(self.rows,i,0)
self.data = numpy.delete(self.data,i,0)
self._shape = (self._shape[0]-1,self.shape[1])
Perhaps I should submit these functions to the Scipy repository.
For a sparse csr matrix (X) and a list of indices to drop (index_to_drop):
to_keep = list(set(xrange(X.shape[1]))-set(index_to_drop))
new_X = X[:,to_keep]
It is easy to convert lil_matrices to csr_matrices. Check tocsr() in lil_matrix documentation
Note however that going from csr to lil matrices using tolil() is expensive. So, this choice is good when you do not require to have your matrix in lil format.
I'm new to python so my answer is probably wrong, but I was wondering why something like the following won't be efficient?
Lets say your lil_matrix is called mat and that you want to remove the i-th column:
mat=hstack( [ mat[:,0:i] , mat[:,i+1:] ] )
Now the matrix will turn to a coo_matrix after that but you can turn it back to lil_matrix.
Ok, I understand that this will have to create the two matrices inside the hstack before it does the assignment to the mat variable so it would be like having the original matrix plus one more at the same time but I guess if the sparsity is big enough then I think there shouldn't be any memory problems (since memory (and time) is the whole reason of using sparse matrices).
def removecols(W, col_list):
if min(col_list) = W.shape[1]:
raise IndexError('column index out of bounds')
rows = W.rows
data = W.data
for i in xrange(M.shape[0]):
for j in col_list:
pos = bisect_left(rows[i], j)
if pos == len(rows[i]):
continue
elif rows[i][pos] == j:
rows[i].pop(pos)
data[i].pop(pos)
if pos == len(rows[i]):
continue
for pos2 in xrange(pos,len(rows[i])):
rows[i][pos2] -= 1
W._shape = (W._shape[0], W._shape[1]-len(col_list))
return W
Just rewrote your code to work with col_list as input - maybe this will be helpful for somebody.
By looking at the notes for each sparse matrix, specifically in our case is csc matrix it has the following advantages as listed in the documentation [1]
efficient arithmetic operations CSC + CSC, CSC * CSC, etc.
efficient column slicing
fast matrix vector products (CSR, BSR may be faster)
If you have the column indices you want to remove, just use slicing.
For removing rows use csr matrix since it is efficient in row slicing