So I would like to apply a function over two 3d matrixes with numpy and I can't figure out how. I read about numpy.apply_over_axes() but can't make it work.
This is my code now:
c = np.random.beta(2,3,size=(2,80))
def my_func(a,b):
xi = np.matmul(b, c)
spe = np.power(a - xi, 2)
return spe.sum()
a = np.zeros(shape=(5,1000,80))
b = np.random.beta(2,3,size=(5,1000,2))
np.apply_over_axes(func=my_func,a=[a,b],axes=[0,0,0])
Which doesnt work and returns
could not broadcast input array from shape (5,1000,80) into shape (5,1000)
I will like to iterate though a and b and apply my_func to every vector of the 3rd dimension.
This would do the job but with normal for loops:
results = []
for i in range(len(a)): #5 Iterations
for j in range(len(a[i])): #1000 Iterations
results.append(my_func(a[i][j], b[i][j]))
I would like to obtain this results this but using numpy functions.
The contraction operation hidden in np.matmul(b, c) can be achieved through np.tensordot(b, c, axes=[2, 0]), where the [2, 0] indicates that the third axis in b is contracted with the first axis in c. That is, np.tensordot(b, c, axes=[2, 0]).shape is (5, 1000, 80). From there on, ordinary broadcasting applies, and your code boils down to
a = np.zeros(shape=(5, 1000, 80))
b = np.random.beta(2, 3, size=(5, 1000, 2))
c = np.random.beta(2, 3, size=(2, 80))
xi = np.tensordot(b, c, axes=[2, 0])
spe = np.power(a - xi, 2)
results2 = spe.sum(axis=2)
Let's check that this indeed matches what you get by simply using loops:
In [55]: results = np.array(results).reshape(5, 1000)
In [56]: np.allclose(results, results2)
Out[56]: True
Running your code in an ipython session:
In [88]: c = np.random.beta(2,3,size=(2,80))
...:
...: def my_func(a,b):
...: xi = np.matmul(b, c)
...:
...: spe = np.power(a - xi, 2)
...: return spe.sum()
...:
...: a = np.zeros(shape=(5,1000,80))
...: b = np.random.beta(2,3,size=(5,1000,2))
...:
...: np.apply_over_axes(func=my_func,a=[a,b],axes=[0,0,0])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-88-c5e5a66c9d0a> in <module>
10 b = np.random.beta(2,3,size=(5,1000,2))
11
---> 12 np.apply_over_axes(func=my_func,a=[a,b],axes=[0,0,0])
<__array_function__ internals> in apply_over_axes(*args, **kwargs)
/usr/local/lib/python3.6/dist-packages/numpy/lib/shape_base.py in apply_over_axes(func, a, axes)
485
486 """
--> 487 val = asarray(a)
488 N = a.ndim
489 if array(axes).ndim == 0:
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a, dtype, order)
83
84 """
---> 85 return array(a, dtype, copy=False, order=order)
86
87
ValueError: could not broadcast input array from shape (5,1000,80) into shape (5,1000)
You should have shown us the full error with traceback.
That traceback shows us that it is trying make one array from your list of two. Since the shapes don't match, it raises an error. With a different mismatch it would have created a (2,) object array, which would just move raise problems later:
In [89]: np.array([a,b])
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-89-964832cdcfcd> in <module>
----> 1 np.array([a,b])
ValueError: could not broadcast input array from shape (5,1000,80) into shape (5,1000)
But the problem is you did not read the docs, or take them seriously:
func : function
This function must take two arguments, `func(a, axis)`.
a : array_like
Input array.
axes : array_like
Axes over which `func` is applied; the elements must be integers.
a is supposed to be an array, not a list of two arrays. func is supposed to take an axis parameter, not another array. And I don't know what you are trying to do with [0,0,0]. For 3d arrays [0,1] might apply, but not a repeated 0.
your loop
With a bit better numpy style:
In [91]: results = []
...: for i in range(a.shape[0]): #5 Iterations
...: for j in range(a.shape[1]): #1000 Iterations
...: results.append(my_func(a[i,j], b[i,j]))
...:
In [92]: np.array(results).shape
Out[92]: (5000,)
rework my_func
To do this without the loops, we need to use whole-array functions within my_func. There isn't a numpy apply that compiles python code - you have to look at numba or cython for that.
xi=np.matmul(b,c). b is (5,1000,2), c is (2,80). matmul is happy doing a dot combining the last axis of b with the 2nd to the last of c.
In [93]: xi = np.matmul(b,c)
In [94]: xi.shape
Out[94]: (5, 1000, 80)
That matches a, so
In [97]: spe = np.power(a-xi,2)
In [98]: spe.shape
Out[98]: (5, 1000, 80)
then sum on that last axis:
In [99]: res = spe.sum(axis=2)
In [100]: res.shape
Out[100]: (5, 1000)
which matches your loop:
In [101]: np.allclose(res.ravel(), np.array(results))
Out[101]: True
Except for the last sum, your myfunc runs with the whole arrays.
In [103]: my_func(a,b)
Out[103]: 46883.49325596101
Related
I have to write a python function where i need to compute the vector
For A is n by n and xn is n by 1
r_n = Axn - (xn^TAxn)xn
Im using numpy but .T doesn't work on vectors and when I just do
r_n = A#xn - (xn#A#xn)#xn but xn#A#xn gives me a scaler.
I've tried changing the A with the xn but nothing seems to work.
Making a 3x1 numpy array like this...
import numpy as np
a = np.array([1, 2, 3])
...and then attempting to take its transpose like this...
a_transpose = a.T
...will, confusingly, return this:
# [1 2 3]
If you want to define a (column) vector whose transpose you can meaningfully take, and get a row vector in return, you need to define it like this:
a = np.reshape(np.array([1, 2, 3]), (3, 1))
print(a)
# [[1]
# [2]
# [3]]
a_transpose = a.T
print(a_transpose)
# [[1 2 3]]
If you want to define a 1 x n array whose transpose you can take to get an n x 1 array, you can do it like this:
a = np.array([[1, 2, 3]])
and then get its transpose by calling a.T.
If A is (n,n) and xn is (n,1):
A#xn - (xn#A#xn)#xn
(n,n)#(n,1) - ((n,1)#(n,n)#(n,1)) # (n,1)
(n,1) error (1 does not match n)
If xn#A#xn gives scalar that's because xn is (n,) shape; as per np.matmul docs that's a 2d with two 1d arrays
(n,)#(n,n)#(n,) => (n,)#(n,) -> scalar
I think you want
(1,n) # (n,n) # (n,1) => (1,1)
Come to think of it that (1,1) array should be same single values as the scalar.
Sample calculation; 1st with the (n,) shape:
In [6]: A = np.arange(1,10).reshape(3,3); x = np.arange(1,4)
In [7]: A#x
Out[7]: array([14, 32, 50]) # (3,3)#(3,)=>(3,)
In [8]: x#A#x # scalar
Out[8]: 228
In [9]: (x#A#x)#x
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
Cell In[9], line 1
----> 1 (x#A#x)#x
ValueError: matmul: Input operand 0 does not have enough dimensions (has 0, gufunc core with signature (n?,k),(k,m?)->(n?,m?) requires 1)
matmul does not like to work with scalars. But we can use np.dot instead, or simply multiply:
In [10]: (x#A#x)*x
Out[10]: array([228, 456, 684]) # (3,)
In [11]: A#x - (x#A#x)*x
Out[11]: array([-214, -424, -634])
Change the array to (3,1):
In [12]: xn = x[:,None]; xn.shape
Out[12]: (3, 1)
In [13]: A#xn - (xn.T#A#xn)*xn
Out[13]:
array([[-214],
[-424],
[-634]]) # same numbers but in (3,1) shape
I have a complicated linear system $y = Ax$ where I cannot specify the matrix A, I can however write a function that computes Ax, and I have made this into a linear operator.
I need to find $A^T$.
I have tried finding $A^T$ by hand but it is becoming tricky.
I found that scipy has a built in function .transpose(), I have tried using this with a simple example,
def mv(v):
return np.array([2*v[0]- v[1], 3*v[1]])
A = LinearOperator((2,2), matvec=mv)
C = A.transpose()
but then when I try to use this it doesn't seem to work. I tried comparing the results
A.matvec(np.ones(2))
array([1., 3.])
C.rmatvec(np.ones(2))
array([1., 3.])
but the results are the same? I'm not sure why this is, surely the second result should be [2, 2].
For a 2d array, M:
In [28]: M = np.array([[1,3,2],[2,1,2],[5,2,1]]); x=np.array([1,2,3])
The transpose lets us switch the order:
In [29]: M#x
Out[29]: array([13, 10, 12])
In [30]: x#M.T
Out[30]: array([13, 10, 12])
Is that what your C does? Implementing the rmatvec inplace of the A.matvec.
Trying to use matvec on C produces an error. C itself is a LinearOperator, one that (somehow) references the operations defined for A.
In [40]: C.matvec([1,1])
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
Cell In[40], line 1
----> 1 C.matvec([1,1])
File ~\miniconda3\lib\site-packages\scipy\sparse\linalg\_interface.py:232, in LinearOperator.matvec(self, x)
229 if x.shape != (N,) and x.shape != (N,1):
230 raise ValueError('dimension mismatch')
--> 232 y = self._matvec(x)
234 if isinstance(x, np.matrix):
235 y = asmatrix(y)
File ~\miniconda3\lib\site-packages\scipy\sparse\linalg\_interface.py:583, in _TransposedLinearOperator._matvec(self, x)
581 def _matvec(self, x):
582 # NB. np.conj works also on sparse matrices
--> 583 return np.conj(self.A._rmatvec(np.conj(x)))
File ~\miniconda3\lib\site-packages\scipy\sparse\linalg\_interface.py:535, in _CustomLinearOperator._rmatvec(self, x)
533 func = self.__rmatvec_impl
534 if func is None:
--> 535 raise NotImplementedError("rmatvec is not defined")
536 return self.__rmatvec_impl(x)
NotImplementedError: rmatvec is not defined
The key is that for C, matvec is implemented with a A.rmatvec:
np.conj(self.A._rmatvec(np.conj(x)))
I haven't worked a lot with this LinearOperator class, but I view it was an abstract class that can be used in iterative solvers in much the same as a 'conventional' 2d array, except that all it has to define is one or more operations like matvec. In your case it's the mv function.
From the LinearOpertor docs:
shape : tuple
Matrix dimensions (M, N).
matvec : callable f(v)
Returns returns A * v.
rmatvec : callable f(v)
Returns A^H * v, where A^H is the conjugate transpose of A.
contrived example
Define a rmatvec for A:
In [51]: def mv(v):
...: return np.array([2*v[0]- v[1], 3*v[1]])
...: def rmv(v):
...: return np.array([2*v[1]- v[0], 3*v[0]])
...: A = linalg.LinearOperator((2,2), matvec=mv, rmatvec=rmv)
...: C = A.transpose()
In [52]: A.matvec((1,6))
Out[52]: array([-4, 18])
In [53]: A.rmatvec((1,6))
Out[53]: array([11, 3])
In [54]: A.rmatvec((6,1))
Out[54]: array([-4, 18])
Notice how C just switches the roles of matvec and rmatvec:
In [55]: C.matvec((1,6))
Out[55]: array([11, 3])
In [56]: C.rmatvec((1,6))
Out[56]: array([-4, 18])
transpose will also change the shape of the operator. If A takes (2,) and returns a (3,), it has shape (3,2); C has shape (2,3), consistent with being a right-hand operator:
In [58]: def mv(v):
...: return np.array([2*v[0]- v[1], 3*v[1],0])
...: A = linalg.LinearOperator((3,2), matvec=mv)
...: C = A.transpose()
In [59]: A
Out[59]: <3x2 _CustomLinearOperator with dtype=float64>
In [60]: C
Out[60]: <2x3 _TransposedLinearOperator with dtype=float64>
To find the transpose of a function you would need to use automatic differentiation which python has in built tools for. I managed, in the end, to find the transpose of my linear operator by hand.
I am trying to append two 3D arrays in python and I thought I would be plain easy however despite trying multiple things it is not working.
Arr1 = np.empty(0,6,2)
print(Arr1.shape)
>> (0,6,2)
df = pd.DataFrame({'var1': np.arange(10), 'var2': np.arange(10), 'prob':
np.random.randint(0,10,10)})
xs = []
ys = []
for i in range(6,10):
xs.append(df[i-6:i][['var1', 'var2']].values)
ys.append(df.iloc[i]['prob'])
Arr2 = np.array(xs).reshape(-1,6,2)
print(Arr2.shape)
>> (4,6,2)
I am trying to append/merge Arr2 into Arr1 on axis=0. I tried following things but I keep getting errors.
try 1: x = np.append(Arr2 ,Arr1 ) -> no error but it gives one dimensional array as output
try 2: x = np.append(Arr2 ,Arr1 ,axis=0) -> does not work. gives error
try 3: x = np.append(Arr2 ,Arr1 ,axis=1) -> does not work. gives error
try 4: x = np.append(Arr2 ,Arr1 ,axis=2) -> does not work. gives error
try 2: x = x = np.stack([Arr2 ,Arr1 ]) -> does not work. gives error
I know I am missing the axis logic but would appreciate any help.
In [145]: Arr1 = np.empty((0,6,2))
In [146]: Arr2 = np.ones((4,6,2))
Works, same as Arr2:
In [148]: np.append(Arr2 ,Arr1 ,axis=0).shape
Out[148]: (4, 6, 2)
What's the point to doing this? I suspect you are intending to repeat this in a loop. :(
np.append is poorly named. It is not a list append clone. About the only place it's useful is adding a scalar to a 1d array. It saves the effort of making the scalar an array! That's all. All other uses are better done with concatenate - it lets you join a whole list of arrays with one call. np.append only lets you specify 2 at a time :(
Error:
In [149]: np.append(Arr2 ,Arr1 ,axis=1).shape
Traceback (most recent call last):
Input In [149] in <cell line: 1>
np.append(Arr2 ,Arr1 ,axis=1).shape
File <__array_function__ internals>:180 in append
File /usr/local/lib/python3.8/dist-packages/numpy/lib/function_base.py:5392 in append
return **concatenate((arr, values), axis=axis)**
File <__array_function__ internals>:180 in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 4 and the array at index 1 has size 0
np.append with axis is just a front end to concatenate. It sould be obvious that you can't join a (0,6,2) and (4,6,2) on the 6 - the 4 and 0 don't match!
In [150]: np.append(Arr2 ,Arr1 ,axis=2).shape
....
Same error - the 0 and 4 don't match
In [151]: np.stack((Arr2 ,Arr1)).shape
....
ValueError: all input arrays must have the same shape
stack is meant to join identically shaped arrays on a new axis.
In [152]: np.stack((Arr2 ,Arr2)).shape
Out[152]: (2, 4, 6, 2)
In [153]: np.stack((Arr2 ,Arr2),axis=2).shape
Out[153]: (4, 6, 2, 2)
edit
It may help to join several identical arrays on different axes. Note where the shape match the source, and where it's a multiple:
In [154]: arr = np.ones((2,3,4))
In [155]: np.concatenate((arr, arr, arr), axis=0).shape
Out[155]: (6, 3, 4) # 2*3
In [156]: np.concatenate((arr, arr, arr), axis=1).shape
Out[156]: (2, 9, 4) # 3*3
In [157]: np.concatenate((arr, arr, arr), axis=2).shape
Out[157]: (2, 3, 12) # 384
In [158]: np.stack((arr, arr, arr), axis=2).shape
Out[158]: (2, 3, 3, 4) # new size 3 axis
I'm trying to create K MxN matrices in Python, stored in a (M,N,K) numpy array, C, from two matrices, A and B, with shapes (K, M) and (K,N) respectively. The first matrix is computed as C0 = a0.T x b0, where a0 is the first row of A and b1 is the first row of B, the second matrix as C1 = a1.T x b0 and so on.
Right now I'm using a for loop to compute the matrices.
import numpy as np
A = np.random.random((10,800))
B = np.random.random((10,500))
C = np.zeros((800,500,10))
for k in range(10):
C[:,:,k] = A[k,:][:,None] # B[k,:][None,:]
Since the operations are independent, I was wondering if there was some pythonic way to avoid the for loop. Perhaps I can vectorize the code, but I fail to see how it could be done.
In [235]: A = np.random.random((10,800))
...: B = np.random.random((10,500))
...: C = np.zeros((800,500,10))
...: for k in range(10):
...: C[:,:,k] = A[k,:][:,None] # B[k,:][None,:]
...:
In [236]: C.shape
Out[236]: (800, 500, 10)
Batched matrix product, followed by transpose
In [237]: np.allclose((A[:,:,None]#B[:,None,:]).transpose(1,2,0), C)
Out[237]: True
But since the matrix product axis is size 1, and there's no other summation, broadcasted multiply is just as good:
In [238]: np.allclose((A[:,:,None]*B[:,None,:]).transpose(1,2,0), C)
Out[238]: True
Execution time is about the same
I've got a numpy array of row vectors of shape (n,3) and another numpy array of matrices of shape (n,3,3). I would like to multiply each of the n vectors with the corresponding matrix and return an array of shape (n,3) of the resulting vectors.
By now I've been using a for loop to iterate through the n vectors/matrices and do the multiplication item by item.
I would like to know if there's a more numpy-ish way of doing this. A way without the for loop that might even be faster.
//edit 1:
As requested, here's my loopy code (with n = 10):
arr_in = np.random.randn(10, 3)
matrices = np.random.randn(10, 3, 3)
for i in range(arr_in.shape[0]): # 10 iterations
arr_out[i] = np.asarray(np.dot(arr_in[i], matrices[i]))
That dot-product is essentially performing reduction along axis=1 of the two input arrays. The dimensions could be represented like so -
arr_in : n 3
matrices : n 3 3
So, one way to solve it would be to "push" the dimensions of arr_in to front by one axis/dimension, thus creating a singleton dimension at axis=2 in a 3D array version of it. Then, sum-reducing the elements along axis = 1 would give us the desired output. Let's show it -
arr_in : n [3] 1
matrices : n [3] 3
Now, this could be achieved through two ways.
1) With np.einsum -
np.einsum('ij,ijk->ik',arr_in,matrices)
2) With NumPy broadcasting -
(arr_in[...,None]*matrices).sum(1)
Runtime test and verify output (for einsum version) -
In [329]: def loop_based(arr_in,matrices):
...: arr_out = np.zeros((arr_in.shape[0], 3))
...: for i in range(arr_in.shape[0]):
...: arr_out[i] = np.dot(arr_in[i], matrices[i])
...: return arr_out
...:
...: def einsum_based(arr_in,matrices):
...: return np.einsum('ij,ijk->ik',arr_in,matrices)
...:
In [330]: # Inputs
...: N = 16935
...: arr_in = np.random.randn(N, 3)
...: matrices = np.random.randn(N, 3, 3)
...:
In [331]: np.allclose(einsum_based(arr_in,matrices),loop_based(arr_in,matrices))
Out[331]: True
In [332]: %timeit loop_based(arr_in,matrices)
10 loops, best of 3: 49.1 ms per loop
In [333]: %timeit einsum_based(arr_in,matrices)
1000 loops, best of 3: 714 µs per loop
You could use np.einsum. To get v.dot(M) for each vector-matrix pair, use np.einsum("...i,...ij", arr_in, matrices). To get M.dot(v) use np.einsum("...ij,...i", matrices, arr_in)