We Keep Coding

Efficiently compute multiple dot products with numpy

I am trying to compute many dot products efficiently. This post gets really close to what I am trying to do, but I can't quite get it to work. I have a large list of matrices (a), and a list of vectors (b). I want to do a series of dot product operations between them. This is what works now:
import numpy as np
a # shape (15000,4,4)
b # shape (15000,4)
out = np.empty((15000,4))
for i in range(15000):
out[i] = np.dot(a[i],b[i])
All my attempts to adapt np.tensordot or np.einsum from the linked post have failed to give me what I want. If anyone sees how to do this I would really appreciate it.

Einstein summation works just fine:
>>> a = np.random.randn(100, 4, 4)
>>> b = np.random.randn(100, 4)
>>> foo = np.einsum('ijk,ik->ij', a, b)
>>> bar = np.zeros_like(foo)
>>> for i, (ai, bi) in enumerate(zip(a, b)):
bar[i] = np.dot(ai, bi)
>>> np.allclose(foo, bar)
True
To explain the summation a bit, note that you're contracting the last axis of b. So you can think about doing each inner product as if by np.einsum('jk,k->j', a[0], b[0]). But we're doing one for each element of a and b, thus the inclusion of the first axis, which is not contracted. Hence, ijk,ik->ij.

Apply a function to each row of a numpy matrix w.r.t. its index

I have a numpy matrix A of shape [n,m] and an array b of length n. What I need is to take sum of b[i] least elements of the i'th row of A.
So the code might look like this:
A = np.array([[1,2,3],
[4,5,6],
[7,8,9]])
b = np.array([2,3,1])
sums = magic_function() #sums = [3, 15, 7]
I've considered np.apply_along_axis() function but it seems that your function can only depend on the row itself in this case.

Vectorized approach making use of NumPy broadcasting to create the mask of valid ones along each row and then perform sum-reduction -
mask = b[:,None] > np.arange(A.shape[1])
out = (A*mask).sum(1)
Alternatively, with np.einsum to get the reduction -
out = np.einsum('ij,ij->i',A,mask)
We can also use np.matmul/# notation on Python 3.x -
out = (A[:,None] # mask[...,None]).squeeze()

What is the meaning of single quote(') in matlab, and how to change it to python

grad = (1/m * (h-y)' * X) + lambda * [0;theta(2:end)]'/m;
cost = 1/(m) * sum(-y .* log(h) - (1-y) .* log(1-h)) + lambda/m/2*sum(theta(2:end).^2);
How to change this two lines to python? I tried to use the zip to do the same job as '. But it shows the error.

Short answer:
The ' operator in MATLAB is the matrix (conjugate) transpose operator. It flips the matrix around dimensions and takes the complex conjugate of the matrix (the second part being what trips people up) The short answer is that the equivalent of a' in Python is np.atleast_2d(a).T.conj().
Slightly longer answer:
Don't use ' in MATLAB unless you really know what you are doing. Use .', which is the ordinary transpose. It is the equivalent of np.atleast_2d(a).T in Python (no conjugate). If you are sure that the a.ndim >= 2 in python, then you can just use a.T. If you are sure that a.ndim == 1 in Python, you can use a[None].T. If you are sure that a.ndim == 0 in Python then transposing is pointless so just do whatever you want.
Very Long Answer:
The basic idea about a transpose is that it flips an array or matrix around one dimension So consider this:
>> a=[1,2,3,4,5,6]
a =
1 2 3 4 5 6
>> a'
ans =
1
2
3
4
5
6
>> b=[1,2,3;4,5,6]
b =
1 2 3
4 5 6
>> b'
ans =
1 4
2 5
3 6
So it seems pretty clear, ' does a transpose. But that is deceiving:
c=[1j,2j,3j,4j,5j,6j]
c =
Columns 1 through 3
0.000000000000000 + 1.000000000000000i 0.000000000000000 + 2.000000000000000i 0.000000000000000 + 3.000000000000000i
Columns 4 through 6
0.000000000000000 + 4.000000000000000i 0.000000000000000 + 5.000000000000000i 0.000000000000000 + 6.000000000000000i
>> c'
ans =
0.000000000000000 - 1.000000000000000i
0.000000000000000 - 2.000000000000000i
0.000000000000000 - 3.000000000000000i
0.000000000000000 - 4.000000000000000i
0.000000000000000 - 5.000000000000000i
0.000000000000000 - 6.000000000000000i
Where did all those negatives come from? They weren't in the original array. The reason for this is described in the documentation. The ' operator in MATLAB isn't a normal transpose operator, the normal transpose operator is .'. The ' operator does a complex conjugate transpose. It does the transpose of the matrix and does the complex conjugate of that matrix.
The problem is that this is almost never what you actually want. It will result in code that seems to work as expected, but silently changes your FFT data, for example. So unless you are absolutely, positively sure your algorithm requires a complex conjugate transpose, use .'.
As for Python, the Python transpose operator is .T. So you consider this:
>>> a = np.array([[1, 2, 3, 4, 5, 6]])
>>> print(a)
[[1 2 3 4 5 6]]
>>> print(a.T)
[[1]
[2]
[3]
[4]
[5]
[6]]
>>> b = np.array([[1j, 2j, 3j, 4j, 5j, 6j]])
[[ 0.+1.j 0.+2.j 0.+3.j 0.+4.j 0.+5.j 0.+6.j]]
>>> (1j*np.ones((1,10))).T
[[ 0.+1.j]
[ 0.+2.j]
[ 0.+3.j]
[ 0.+4.j]
[ 0.+5.j]
[ 0.+6.j]]
Notice the lack of any negatives for the imaginary part. If you want to get the complex conjugate transpose, you need to use np.conj(a) or a.conj() to get the complex conjugate (either before or after doing the transpose). However, numpy has its own transpose pitfall:
>>> c = np.array([1, 2, 3, 4, 5, 6])
>>> print(c)
[1 2 3 4 5 6]
>>> print(c.T)
[1 2 3 4 5 6]
Huh? It didn't do anything. The reason is that np.array([1, 2, 3, 4, 5, 6]) creates a 1D array. A transpose is flipping the array along a particular dimension. That is meaningless when there is only one dimension, so the transpose doesn't do anything.
"But," you might object, "didn't a transpose of the 1D MATLAB matrix work?" The reason is more fundamental to how MATLAB and numpy store data. Consider Python:
>>> np.array([[1, 2, 3], [4, 5, 6]]).ndim
2
>>> np.array([1, 2, 3, 4, 5, 6]).ndim
1
>>> np.array(1).ndim
0
That seems reasonable. A 2D array has two dimensions, a 1D array has one dimension, and a scalar has zero dimensions. But try the same thing in MATLAB:
>> ndims([1,2,3;4,5,6])
ans =
2
>> ndims([1,2,3,4,5,6])
ans =
2
>> ndims(1)
ans =
2
Everything has 2 dimensions! MATLAB has no 1D or 0D data structures, everything in MATLAB must have at least 2 dimensions (although it may be possible to create your own effectively 1D or 0D class in MATLAB). So taking the transpose of your "1D" data structure in MATLAB worked becaused it wasn't actually 1D.
Both the conjugate transpose and the 1D transpose issues come down to the basic data type MATLB and numpy use. MATLAB uses matrices, which inherently are at least 2D. nump, on the other hand, uses arrays, which can have any number of dimensions. MATLAB matrices use matrix mathematics as their normal operations (so a * b in MATLAB is a matrix product) while Python arrays use element-by-element mathematics as their normal operators (so a * b is an element-by-element product, equivalent of a .* b in MATLAB). MATLAB has element-by-element operators, and numpy arrays have matrix operators (although no matrix transpose yet, though adding one is being considered), so this mostly applies to the default operations.
To avoid this issue in Python, there are several ways to get around it. Indexing with None in Python inserts additional dimensions. So for a 1D array a, a[None] will be a 2D array where the first dimension has a length of 1. If you don't know ahead of time what the dimensionality of your array is, you can use np.atleast_2d(a), which will make sure a has at least two dimensions. So 0D becomes 2D, 1D becomes 2D, 2D stays 2D, 3D stays 3D, 4D stays 4D, etc.
That being said, numpy has a matrix class that works the same as MATLAB's in all these regards (it even has a conjugate transpose operator, .H). Don't use it. The python community has standardized around arrays, since in practice that is almost always what you want. That means that most Python tools expect arrays, and many will either malfunction if given matrices or will convert them to arrays. So just use arrays.

The " ' " in Matlab is 'transpose' of a matrix. The numpy package is the fundamental package for scientific computing in python. numpy.transpose could be used to carry the same task out.
import numpy as np
matrix = np.arange(6).reshape((2,3))
This going to create a matrix with two rows and three columns as follows :
>>> array([[0, 1, 2],[3, 4, 5]])
Then the transpose is given as:
np.transpose (matrix)
>>> array([[0, 3],[1, 4],[2, 5]])
I hope it helps

Creating index array in numpy - eliminating double for loop

I have some physical simulation code, written in python and using numpy/scipy. Profiling the code shows that 38% of the CPU time is spent in a single doubly nested for loop - this seems excessive, so I've been trying to cut it down.
The goal of the loop is to create an array of indices, showing which elements of a 1D array the elements of a 2D array are equal to.
indices[i,j] = where(1D_array == 2D_array[i,j])
As an example, if 1D_array = [7.2, 2.5, 3.9] and
2D_array = [[7.2, 2.5]
[3.9, 7.2]]
We should have
indices = [[0, 1]
[2, 0]]
I currently have this implemented as
for i in range(ni):
for j in range(nj):
out[i, j] = (1D_array - 2D_array[i, j]).argmin()
The argmin is needed as I'm dealing with floating point numbers, and so the equality is not necessarily exact. I know that every number in the 1D array is unique, and that every element in the 2D array has a match, so this approach gives the correct result.
Is there any way of eliminating the double for loop?
Note:
I need the index array to perform the following operation:
f = complex_function(1D_array)
output = f[indices]
This is faster than the alternative, as the 2D array has a size of NxN compared with 1xN for the 1D array, and the 2D array has many repeated values. If anyone can suggest a different way of arriving at the same output without going through an index array, that could also be a solution

In pure Python you can do this using a dictionary in O(N) time, the only time penalty is going to be the Python loop involved:
>>> arr1 = np.array([7.2, 2.5, 3.9])
>>> arr2 = np.array([[7.2, 2.5], [3.9, 7.2]])
>>> indices = dict(np.hstack((arr1[:, None], np.arange(3)[:, None])))
>>> np.fromiter((indices[item] for item in arr2.ravel()), dtype=arr2.dtype).reshape(arr2.shape)
array([[ 0., 1.],
[ 2., 0.]])

The dictionary method that some others have suggest might work, but it requires that you know ahead of time that every element in your target array (the 2d array) has an exact match in your search array (your 1d array). Even when this should be true in principle, you still have to deal with floating point precision issues, for example try this .1 * 3 == .3.
Another approach is to use numpy's searchsorted function. searchsorted takes a sorted 1d search array and any traget array then finds the closest elements in the search array for every item in the target array. I've adapted this answer for your situation, take a look at it for a description of how the find_closest function works.
import numpy as np
def find_closest(A, target):
order = A.argsort()
A = A[order]
idx = A.searchsorted(target)
idx = np.clip(idx, 1, len(A)-1)
left = A[idx-1]
right = A[idx]
idx -= target - left < right - target
return order[idx]
array1d = np.array([7.2, 2.5, 3.9])
array2d = np.array([[7.2, 2.5],
[3.9, 7.2]])
indices = find_closest(array1d, array2d)
print(indices)
# [[0 1]
# [2 0]]

To get rid of the two Python for loops, you can do all of the equality comparisons "in one go" by adding new axes to the arrays (making them broadcastable with each other).
Bear in mind that this produces a new array containing len(arr1)*len(arr2) values. If this is a very big number, this approach could be infeasible depending on the limitations of your memory. Otherwise, it should be reasonably quick:
>>> (arr1[:,np.newaxis] == arr2[:,np.newaxis]).argmax(axis=1)
array([[0, 1],
[2, 0]], dtype=int32)
If you need to get the index of the closest matching value in arr1 instead, use:
np.abs(arr1[:,np.newaxis] - arr2[:,np.newaxis]).argmin(axis=1)

numpy element-wise multiplication of an array and a vector

I want to do something like this:
a = # multi-dimensional numpy array
ares = # multi-dim array, same shape as a
a.shape
>>> (45, 72, 37, 24) # the relevant point is that all dimension are different
v = # 1D numpy array, i.e. a vector
v.shape
>>> (37) # note that v has the same length as the 3rd dimension of a
for i in range(37):
ares[:,:,i,:] = a[:,:,i,:]*v[i]
I'm thinking there has to be a more compact way to do this with numpy, but I haven't figured it out. I guess I could replicate v and then calculate a*v, but I am guessing there is something better than that too. So I need to do element wise multiplication "over a given axis", so to speak. Anyone know how I can do this? Thanks. (BTW, I did find a close duplicate question, but because of the nature of the OP's particular problem there, the discussion was very short and got tracked into other issues.)

Here is one more:
b = a * v.reshape(-1, 1)
IMHO, this is more readable than transpose, einsum and maybe even v[:, None], but pick the one that suits your style.

You can automatically broadcast the vector against the outermost axis of an array. So, you can transpose the array to swap the axis you want to the outside, multiply, then transpose it back:
ares = (a.transpose(0,1,3,2) * v).transpose(0,1,3,2)

You can do this with Einstein summation notation using numpy's einsum function:
ares = np.einsum('ijkl,k->ijkl', a, v)

I tend to do something like
b = a * v[None, None, :, None]
where I think I'm officially supposed to write np.newaxis instead of None.
For example:
>>> import numpy as np
>>> a0 = np.random.random((45,72,37,24))
>>> a = a0.copy()
>>> v = np.random.random(37)
>>> for i in range(len(v)):
... a[:,:,i,:] *= v[i]
...
>>> b = a0 * v[None,None,:,None]
>>>
>>> np.allclose(a,b)
True