Related
Gist
Basically I want to perform an increase in dimension of two axes on a n-dimensonal tensor.
For some reason this operation seems very slow on bigger tensors.
If someone can give me a reason or better method I'd be very happy.
Goal
Going from (4, 8, 8, 4, 4, 4, 4, 4, 16, 8, 4, 4, 1) to (4, 32, 8, 4, 4, 4, 4, 4, 4, 8, 4, 4, 1) takes roughly 170 second. I'd like to improve on that. Below is an example, finding the correct indices is not necessary here.
Example Code
Increase dimension (0,2) of tensor
tensor = np.arange(16).reshape(2,2,4,1)
I = np.identity(4)
I tried 3 different methods:
np.kron
indices = [1,3,0,2]
result = np.kron(
I, tensor.transpose(indices)
).transpose(np.argsort(indices))
print(result.shape) # should be (8,2,16,1)
manual stacking
col = []
for i in range(4):
row = [np.zeros_like(tensor)]*4
row[i]=tensor
col.append(a)
result = np.array(col).transpose(0,2,3,1,4,5).reshape(8,2,16,1)
print(result.shape) # should be (8,2,16,1)
np.einsum
result =np.einsum("ij, abcd -> iabjcd", I, tensor).reshape(8,2,16,1)
print(result.shape) # should be (8,2,16,1)
Results
On my machine they performed the following (on the big example with complex entries):
np.einsum ~ 170s
manual stacking ~ 185s
np.kron ~ 580s
As Jérôme pointed out:
all your operations seems to involve a transposition which is known to be very expensive on modern hardware.
I reworked my algorithm to not rely on the dimensional increase by doing certain preprocessing steps. This indeed speeds up the overall process substantially.
I'm working with wav files at the moment and want to split up the data array with all channels into a matrix, where every row is the specific channel data.
I have the data array like so, and wish the following matrix format:
data = [channel1_frame1, channel2_frame1, ..., channeln_frame1, channel1_frame2, ...] # integers
matrix = [[channel1_frame1, channel1_frame2, channel1_frame3,...],
[channel2_frame1, channel2_frame2, channel2_frame3,...]
...
[channeln_frame1, channeln_frame2, channeln_frame3,...]]
So while it is possible to use np.zeros to create the fitting matrix form and then use a python loop to basically say:
matrix = np.zeros(#channels, len(data)/#channels)
for index in range(#channels):
matrix[i] = data[i::#channels]
Can't I somehow do that with numpy alone? The performance gain might not be so high, especially because the amount of channels will be very low in most cases, but it kind of pisses me off that I've not been able to find a more "elegant" solution to this.
Thanks in advance if you have an idea,
have a nice weekend!
You could use numpy.reshape:
data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
matrix = np.reshape(2, 5) # Assume 2 channels and 5 frames
print(matrix)
This code outputs
array([[ 1, 2, 3, 4, 5],
[ 6, 7, 8, 9, 10]])
In your case you would use reshape(len(data)/#channels, #channels) on your data.
I'm currently having the following issue, given an array let's say for simplicity a 4 x 4 array (I'm actually working with 512 x 512 )
X = np.array([[3, 5, 2, 4],
[7, 6, 8, 8],
[1, 6, 7, 7],
[2, 1, 3, 4]])
I would like to loop/slide around the array in a way that I can save new arrays in the form
np.array([3,5],[7,6]), np.array([2,4], [8,8]), np.array([1,6],[2,1]), np.array ([7,7],[1,4]) and so on (Ideally that I could choose the step and the size of my "sliding" window). Also I would like to select these arrays according to some conditions , see below.
For the moment I managed to do almost everything by slicing (see code) my matrix. This gives the correct slicing with the step which I want, then by using the itertools module I can loop over all my list of lists, count the elements smaller than a certain value and save them.
What I cannot do is to link the indices between all these new lists together to the main matrix.
For this reason, I was thinking to move everything to numpy arrays which should be (in my understanding) much more efficient in terms of computing and I guess should solve my issue of indices. Now the only problem is that I don't know to solve this problem for an arbitrary n x n matrix.
X = np.array([[3, 5, 2, 4],
[7, 6, 8, 8],
[1, 6, 7, 7],
[2, 1, 3, 4]])
width = len(m[0])
height = len(m)
height = 2 # size of the matrix sliding
width = 2
for i in range(0, height - slice_y + 1,2):
for j in range(0, width - slice_x + 1,2):
Appended.append(
[
[m[a][b] for b in range(j, j + slice_x)]
for a in range(i, i + slice_y)
]
)
Ideally what I would like is for a general matrix N x N but for the moment also only like in the example to get the output as a form of arrays like:
np.array([3,5],[7,6]) . np.array ([2,4], [8,8]) , np.array ([1,6],[2,1]), np.array ([7,7],[1,4])
And let's say once found that e.g the array np.array([2,4], [8,8]) has two elements bigger than 7 and the sum is more than 20 to save the coordinates of this array with respect of my initial matrix. So saving the indices couples X[0][2], X[0][3], X[1][2], X[1][3] or at least the first X[0][2] so by knowing the step of my "window" I can access my subarrays by indexing my main matrix.
You can apparently slice numpy arrays directly
X = np.array([[3, 5, 2, 4],
[7, 6, 8, 8],
[1, 6, 7, 7],
[2, 1, 3, 4]])[0:2,0:2]
I would in your case generate a list of the indices of the edges of the submatrix you're going to use. Then use that to generate the list of submatricies then use that to generate a list of true or false values based on the submatricies. Then you can use that list of true/false values to prune your initial list of indicies. You could also do this without storing the submatricies at all.
indicies= [((i,i+s_width),(j,j+s_height)) for i in range(0,width-s_width) for j in range(0,height-s_height)]
I have need for hstacking multple arrays with with the same number of rows (although the number of rows is variable between uses) but different number of columns. However some of the arrays only have one column, eg.
array = np.array([1,2,3,4,5])
which gives
#array.shape = (5,)
but I'd like to have the shape recognized as a 2d array, eg.
#array.shape = (5,1)
So that hstack can actually combine them.
My current solution is:
array = np.atleast_2d([1,2,3,4,5]).T
#array.shape = (5,1)
So I was wondering, is there a better way to do this? Would
array = np.array([1,2,3,4,5]).reshape(len([1,2,3,4,5]), 1)
be better?
Note that my use of [1,2,3,4,5] is just a toy list to make the example concrete. In practice it will be a much larger list passed into a function as an argument. Thanks!
Check the code of hstack and vstack. One, or both of those, pass the arguments through atleast_nd. That is a perfectly acceptable way of reshaping an array.
Some other ways:
arr = np.array([1,2,3,4,5]).reshape(-1,1) # saves the use of len()
arr = np.array([1,2,3,4,5])[:,None] # adds a new dim at end
np.array([1,2,3],ndmin=2).T # used by column_stack
hstack and vstack transform their inputs with:
arrs = [atleast_1d(_m) for _m in tup]
[atleast_2d(_m) for _m in tup]
test data:
a1=np.arange(2)
a2=np.arange(10).reshape(2,5)
a3=np.arange(8).reshape(2,4)
np.hstack([a1.reshape(-1,1),a2,a3])
np.hstack([a1[:,None],a2,a3])
np.column_stack([a1,a2,a3])
result:
array([[0, 0, 1, 2, 3, 4, 0, 1, 2, 3],
[1, 5, 6, 7, 8, 9, 4, 5, 6, 7]])
If you don't know ahead of time which arrays are 1d, then column_stack is easiest to use. The others require a little function that tests for dimensionality before applying the reshaping.
Numpy: use reshape or newaxis to add dimensions
If I understand your intent correctly, you wish to convert an array of shape (N,) to an array of shape (N,1) so that you can apply np.hstack:
In [147]: np.hstack([np.atleast_2d([1,2,3,4,5]).T, np.atleast_2d([1,2,3,4,5]).T])
Out[147]:
array([[1, 1],
[2, 2],
[3, 3],
[4, 4],
[5, 5]])
In that case, you could use avoid reshaping the arrays and use np.column_stack instead:
In [151]: np.column_stack([[1,2,3,4,5], [1,2,3,4,5]])
Out[151]:
array([[1, 1],
[2, 2],
[3, 3],
[4, 4],
[5, 5]])
I followed Ludo's work and just changed the size of v from 5 to 10000. I ran the code on my PC and the result shows that atleast_2d seems to be a more efficient method in the larger scale case.
import numpy as np
import timeit
v = np.arange(10000)
print('atleast2d:',timeit.timeit(lambda:np.atleast_2d(v).T))
print('reshape:',timeit.timeit(lambda:np.array(v).reshape(-1,1))) # saves the use of len()
print('v[:,None]:', timeit.timeit(lambda:np.array(v)[:,None])) # adds a new dim at end
print('np.array(v,ndmin=2).T:', timeit.timeit(lambda:np.array(v,ndmin=2).T)) # used by column_stack
The result is:
atleast2d: 1.3809496470021259
reshape: 27.099974197000847
v[:,None]: 28.58291715100131
np.array(v,ndmin=2).T: 30.141663907001202
My suggestion is that use [:None] when dealing with a short vector and np.atleast_2d when your vector goes longer.
Just to add info on hpaulj's answer. I was curious about how fast were the four methods described. The winner is the method adding a column at the end of the 1d array.
Here is what I ran:
import numpy as np
import timeit
v = [1,2,3,4,5]
print('atleast2d:',timeit.timeit(lambda:np.atleast_2d(v).T))
print('reshape:',timeit.timeit(lambda:np.array(v).reshape(-1,1))) # saves the use of len()
print('v[:,None]:', timeit.timeit(lambda:np.array(v)[:,None])) # adds a new dim at end
print('np.array(v,ndmin=2).T:', timeit.timeit(lambda:np.array(v,ndmin=2).T)) # used by column_stack
And the results:
atleast2d: 4.455070924214851
reshape: 2.0535152913971615
v[:,None]: 1.8387219828073285
np.array(v,ndmin=2).T: 3.1735243063353664
I wanted to repeat the rows of a scipy csr sparse matrix, but when I tried to call numpy's repeat method, it simply treats the sparse matrix like an object, and would only repeat it as an object in an ndarray. I looked through the documentation, but I couldn't find any utility to repeats the rows of a scipy csr sparse matrix.
I wrote the following code that operates on the internal data, which seems to work
def csr_repeat(csr, repeats):
if isinstance(repeats, int):
repeats = np.repeat(repeats, csr.shape[0])
repeats = np.asarray(repeats)
rnnz = np.diff(csr.indptr)
ndata = rnnz.dot(repeats)
if ndata == 0:
return sparse.csr_matrix((np.sum(repeats), csr.shape[1]),
dtype=csr.dtype)
indmap = np.ones(ndata, dtype=np.int)
indmap[0] = 0
rnnz_ = np.repeat(rnnz, repeats)
indptr_ = rnnz_.cumsum()
mask = indptr_ < ndata
indmap -= np.int_(np.bincount(indptr_[mask],
weights=rnnz_[mask],
minlength=ndata))
jumps = (rnnz * repeats).cumsum()
mask = jumps < ndata
indmap += np.int_(np.bincount(jumps[mask],
weights=rnnz[mask],
minlength=ndata))
indmap = indmap.cumsum()
return sparse.csr_matrix((csr.data[indmap],
csr.indices[indmap],
np.r_[0, indptr_]),
shape=(np.sum(repeats), csr.shape[1]))
and be reasonably efficient, but I'd rather not monkey patch the class. Is there a better way to do this?
Edit
As I revisit this question, I wonder why I posted it in the first place. Almost everything I could think to do with the repeated matrix would be easier to do with the original matrix, and then apply the repetition afterwards. My assumption is that post repetition will always be the better way to approach this problem than any of the potential answers.
from scipy.sparse import csr_matrix
repeated_row_matrix = csr_matrix(np.ones([repeat_number,1])) * sparse_row
It's not surprising that np.repeat does not work. It delegates the action to the hardcoded a.repeat method, and failing that, first turns a into an array (object if needed).
In the linear algebra world where sparse code was developed, most of the assembly work was done on the row, col, data arrays BEFORE creating the sparse matrix. The focus was on efficient math operations, and not so much on adding/deleting/indexing rows and elements.
I haven't worked through your code, but I'm not surprised that a csr format matrix requires that much work.
I worked out a similar function for the lil format (working from lil.copy):
def lil_repeat(S, repeat):
# row repeat for lil sparse matrix
# test for lil type and/or convert
shape=list(S.shape)
if isinstance(repeat, int):
shape[0]=shape[0]*repeat
else:
shape[0]=sum(repeat)
shape = tuple(shape)
new = sparse.lil_matrix(shape, dtype=S.dtype)
new.data = S.data.repeat(repeat) # flat repeat
new.rows = S.rows.repeat(repeat)
return new
But it is also possible to repeat using indices. Both lil and csr support indexing that is close to that of regular numpy arrays (at least in new enough versions). Thus:
S = sparse.lil_matrix([[0,1,2],[0,0,0],[1,0,0]])
print S.A.repeat([1,2,3], axis=0)
print S.A[(0,1,1,2,2,2),:]
print lil_repeat(S,[1,2,3]).A
print S[(0,1,1,2,2,2),:].A
give the same result
and best of all?
print S[np.arange(3).repeat([1,2,3]),:].A
After someone posted a really clever response for how best to do this I revisited my original question, to see if there was an even better way. I I came up with one more way that has some pros and cons. Instead of repeating all of the data (as is done with the accepted answer), we can instead instruct scipy to reuse the data of the repeated rows, creating something akin to a view of the original sparse array (as you might do with broadcast_to). This can be done by simply tiling the indptr field.
repeated = sparse.csr_matrix((orig.data, orig.indices, np.tile(orig.indptr, repeat_num)))
This technique repeats the vector repeat_num times, while only modifying the the indptr. The downside is that due to the way the csr matrices encode data, instead of creating a matrix that's repeat_num x n in dimension, it creates one that's (2 * repeat_num - 1) x n where every odd row is 0. This shouldn't be too big of a deal as any operation will be quick given that each row is 0, and they should be pretty easy to slice out afterwards (with something like [::2]), but it's not ideal.
I think the marked answer is probably still the "best" way to do this.
One of the most efficient ways to repeat the sparse matrix would be the way OP suggested. I modified indptr so that it doesn't output rows of 0s.
## original sparse matrix
indptr = np.array([0, 2, 3, 6])
indices = np.array([0, 2, 2, 0, 1, 2])
data = np.array([1, 2, 3, 4, 5, 6])
x = scipy.sparse.csr_matrix((data, indices, indptr), shape=(3, 3))
x.toarray()
array([[1, 0, 2],
[0, 0, 3],
[4, 5, 6]])
To repeat this, you need to repeat data and indices, and you need to fix-up the indptr. This is not the most elegant way, but it works.
## repeated sparse matrix
repeat = 5
new_indptr = indptr
for r in range(1,repeat):
new_indptr = np.concatenate((new_indptr, new_indptr[-1]+indptr[1:]))
x = scipy.sparse.csr_matrix((np.tile(data,repeat), np.tile(indices,repeat), new_indptr))
x.toarray()
array([[1, 0, 2],
[0, 0, 3],
[4, 5, 6],
[1, 0, 2],
[0, 0, 3],
[4, 5, 6],
[1, 0, 2],
[0, 0, 3],
[4, 5, 6],
[1, 0, 2],
[0, 0, 3],
[4, 5, 6],
[1, 0, 2],
[0, 0, 3],
[4, 5, 6]])