We Keep Coding

Operations on very large numpy arrays of different dimensions: running waying too slow

Suppose I have two arrays:
import numpy as np
A = np.random.rand(500,4,200)
B = np.random.rand(100,100)
I would like to do the following operation (in index notation):
C_{ijkmn} = A_{ijk} - B_{mn}
The obvious way to do this is
C = A[:,:,:,None,None] - B[None,None,None,:,:] # has shape 500,4,200,100,100
But it crashes my kernel and takes up waaay too much memory. So my question is simply how can I go about performing this sort of operation more effeciently? Would numba help? Thanks so much!!

Welp, turns out the best thing to do here was to calculate a few examples of C and interpolate over fine grid. This seemed to work well! Just using a scipy function...

Why multiplication functions of scipy sparse and numpy arrays give different results?

I have two matrices in Python 2.7: one dense A_dense and the another sparse matrix A_sparse. I am interested in computing element-wise multiplication followed by sum. There are two ways to do it: use numpy's multiplication or scipy sparse multiplication. I expect them to give exactly same result with difference in execution time. But I find that they give different results for certain matrix sizes.
import numpy as np
from scipy import sparse
L=2000
np.random.seed(2)
rand_x=np.random.rand(L)
A_sparse_init=np.diag(rand_x, -1)+np.diag(rand_x, 1)
A_sparse=sparse.csr_matrix(A_sparse_init)
A_dense=np.random.rand(L+1,L+1)
print np.sum(A_sparse.multiply(A_dense))-np.sum(np.multiply(A_dense[A_sparse.nonzero()], A_sparse.data))
Output:
1.1368683772161603e-13
If I choose L=2001, then output is:
0.0
To check the size dependence of the difference using two different multiplication method, I wrote:
L=100
np.random.seed(2)
N_loop=100
multiply_diff_arr=np.zeros(N_loop)
for i in xrange(N_loop):
rand_x=np.random.rand(L)
A_sparse_init=np.diag(rand_x, -1)+np.diag(rand_x, 1)
A_sparse=sparse.csr_matrix(A_sparse_init)
A_dense=np.random.rand(L+1,L+1)
multiply_diff_arr[i]=np.sum(A_sparse.multiply(A_dense))-np.sum(np.multiply(A_dense[A_sparse.nonzero()], A_sparse.data))
L+=1
I got the following plot:
Can anyone help me understand what's happening? Don't we expect the difference between two methods to be at least 1e-18 rather than 1e-13?

I don't have a full answer, but this might help find the answer:
Under the hood, scipy.sparse will convert to coo format and do this:
ret = self.tocoo()
if self.shape == other.shape:
data = np.multiply(ret.data, other[ret.row, ret.col])
The question is then why these two operations give different results:
ret = A_sparse.tocoo()
c = np.multiply(ret.data, A_dense[ret.row, ret.col])
ret.data = c.view(type=np.ndarray)
c.sum() - ret.sum()
-1.1368683772161603e-13
Edit:
The difference stems from different defaults on which axis to add.reduce first.
E.g.:
A_sparse.multiply(A_dense).sum(axis=1).sum()
A_sparse.multiply(A_dense).sum(axis=0).sum()
Numpy defaults to 0 first.

Convert an existing NumPy array into a ctype array to be shared among multiprocessing

Let's say I have an existing array that we don't want to make any changes to, but like to be converted to a ctype array and be shared among all the multiprocessing later on.
The actual array I want to be shared is of shape 120,000 x 4, which is too large to type all out here, so let's pretend such an array is way smaller and looks like this:
import numpy as np
import multiprocessing as mp
import ctypes
array_from_data = np.array([[275,174,190],
[494, 2292, 9103],
[10389,284,28],
[193,746,293]])
I have read other posts that discuss the ctype array and multiprocessing, like this one. However, the answers are not quite the same as what I am looking for, because so far they are not exactly about converting an existing NumPy array.
My questions are the following:
1) How to do a simple conversion from an existing Numpy array to a ctype array?
2) How to make the array to be shared among all the multiprocessing in a simple fashion?
Thank you in advance.
EDIT: spellings and some clarifications on the actual array
EDIT2: Apparently the os itself affects how the multiprocessing will behave and I need to specify it: My os is Windows 10 64-bit.

The workaround I found months ago requires flattening the array into a 1-dimensional array first, even though I only understand half of what is under the hood.
The gist of the solution is to:
1) make a RawArray of the same size and same dtypes as the array we are trying to share
2) create a numpy array that uses the same memory location as the RawArray
3) fill in the elements to the newly created numpy array
Workaround:
import ctypes
import multiprocessing as mp
import numpy as np
array_from_data = np.array([[275,174,190],
[494, 2292, 9103],
[10389,284,28],
[193,746,293]])
flattened_array1 = array_from_data.flatten(order='C')
flattened_array2 = np.array([1,0,1,0,1]).astype(bool)
flattened_array3 = np.array([1,0,1,0,-10]).astype(np.float32)
array_shared_in_multiprocessing1 = mp.RawArray(ctypes.c_int32,len(flattened_array1))
temp1 = np.frombuffer(array_shared_in_multiprocessing1, dtype=np.int32)
temp1[:] = flattened_array1
array_shared_in_multiprocessing2 = mp.RawArray(ctypes.c_bool,len(flattened_array2))
temp2 = np.frombuffer(array_shared_in_multiprocessing2, dtype=bool)
temp2[:] = flattened_array2
array_shared_in_multiprocessing3 = mp.RawArray(ctypes.c_float,len(flattened_array3))
temp2 = np.frombuffer(array_shared_in_multiprocessing3, dtype=np.float32)
temp2[:] = flattened_array3

Matlab-Python translation error

Matlab Code:
AP(queryIdx) = diff([0;recall]')*prec
My python code:
AP[queryIdx] = np.dot(np.diff(np.concatenate(([[0]], recall), axis=0).transpose()),prec)
Variables:(Checked and am quite sure they are equivalent in python and in Matlab)
Recall: 1000x1 np array*
prec: 1000x1 np array
* prints out as [[.],.....,[.]]
Results:
Matlab: .1011
Python: 0.05263158
Only cause I can think of outside of the code is that python uses more
precision, but I doubt that would make such a large difference)
*Edit There was a problem with my prec variable. The above code worked

That code looks a bit messy. Try replacing it with this:
AP[queryIdx] = np.dot(np.diff(np.hstack([0, recall.ravel()])), prec.ravel())
In your post, you mentioned that you have a 1000 x 1 array for both recall and prec. This to me is interpreted as a 2D array with a singleton dimension: the second dimension. As such, you'd need to convert this back to a 1D array using ravel.
Now, np.hstack horizontally stacks 1D arrays together and so this will append a 0 at the front, then apply the diff operator, and the perform the dot product with prec.
One common gotcha that MATLAB coders have with numpy is the representation of 1D arrays in numpy. There is no such thing as the transpose of a 1D array. All numpy 1D arrays are row vectors. If you explicitly want to make the 1D array a column vector, you need to include an additional dimension and make the second dimension 1, then transpose it. Something like this:
r = v[:][None].T
In any case, let's verify the results:
MATLAB
>> recall = (1:1000).';
>> prec = (1000:-1:1).';
>> diff([0; recall].')*prec
ans =
500500
Python (IPython)
In [1]: import numpy as np
In [2]: recall = np.arange(1,1001)
In [3]: prec = np.arange(1000,0,-1)
In [4]: np.dot(np.diff(np.hstack([0, recall.ravel()])), prec.ravel())
Out[4]: 500500

root mean square in numpy and complications of matrix and arrays of numpy

Can anyone direct me to the section of numpy manual where i can get functions to accomplish root mean square calculations ...
(i know this can be accomplished using np.mean and np.abs .. isn't there a built in ..if no why?? .. just curious ..no offense)
can anyone explain the complications of matrix and arrays (just in the following case):
U is a matrix(T-by-N,or u say T cross N) , Ue is another matrix(T-by-N)
I define k as a numpy array
U[ind,:] is still matrix
in the following fashion
k = np.array(U[ind,:])
when I print k or type k in ipython
it displays following
K = array ([[2,.3 .....
......
9]])
You see the double square brackets (which makes it multi-dim i guess)
which gives it the shape = (1,N)
but I can't assign it to array defined in this way
l = np.zeros(N)
shape = (,N) or perhaps (N,) something like that
l[:] = k[:]
error:
matrix dimensions incompatible
Is there a way to accomplish the vector assignment which I intend to do ... Please don't tell me do this l = k (that defeats the purpose ... I get different errors in program .. I know the reasons ..If you need I may attach the piece of code)
writing a loop is the dumb way .. which I'm using for the time being ...
I hope I was able to explain .. the problems I'm facing ..
regards ...

For the RMS, I think this is the clearest:
from numpy import mean, sqrt, square, arange
a = arange(10) # For example
rms = sqrt(mean(square(a)))
The code reads like you say it: "root-mean-square".

For rms, the fastest expression I have found for small x.size (~ 1024) and real x is:
def rms(x):
return np.sqrt(x.dot(x)/x.size)
This seems to be around twice as fast as the linalg.norm version (ipython %timeit on a really old laptop).
If you want complex arrays handled more appropriately then this also would work:
def rms(x):
return np.sqrt(np.vdot(x, x)/x.size)
However, this version is nearly as slow as the norm version and only works for flat arrays.

For the RMS, how about
norm(V)/sqrt(V.size)

I don't know why it's not built in. I like
def rms(x, axis=None):
return sqrt(mean(x**2, axis=axis))
If you have nans in your data, you can do
def nanrms(x, axis=None):
return sqrt(nanmean(x**2, axis=axis))

Try this:
U = np.zeros((N,N))
ind = 1
k = np.zeros(N)
k[:] = U[ind,:]

I use this for RMS, all using NumPy, and let it also have an optional axis similar to other NumPy functions:
import numpy as np
rms = lambda V, axis=None: np.sqrt(np.mean(np.square(V), axis))

If you have complex vectors and are using pytorch, the vector norm is the fastest approach on CPU & GPU:
import torch
batch_size, length = 512, 4096
batch = torch.randn(batch_size, length, dtype=torch.complex64)
scale = 1 / torch.sqrt(torch.tensor(length))
rms_power = batch.norm(p=2, dim=-1, keepdim=True)
batch_rms = batch / (rms_power * scale)
Using batch vdot like goodboy's approach is 60% slower than above. Using naïve method similar to deprecated's approach is 85% slower than above.