I am trying to matrix multiply a 2x2 matrix with a 2x1 matrix. Both matrices have entries which are linspaces such that the resulting 2x1 matrix gives me a value for each value of the linspace.
I get this dimensionality error however.
matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1 is different from 2)
For readability I am not posting the whole code but what's necessary.
I have also replaced linspace values with indicative text.
Matrix "L" is a result of other 2x2 multiplications which contain constants, thus no errors there.
The matrix B (2x2) gives the desired result, so the problem comes down to the multiplication between B and C.
import numpy as np
from sympy import *
# Defining range of values
z = np.linspace(initial, final, 10)
g = np.linspace(initial, final, 10)
y = np.linspace(initial, final, 10)
# Matrix operations
A = np.array([[1, z], [0, 1]], dtype=object)
B = np.matmul(L,A)
C = np.array([[y],[g]])
D = np.matmul(B, C)
print(total)
An alternative POV of what I am trying to do, is that for the matrix "B" when multiplied with the 2x1 "C" which contains unknowns, to calculate those unknowns "y" and "g"
Many thanks,
P.S; For an array "C" with single value entries, the multiplication runs as expected.
Edit; As per mozway's suggestion, I am providing the prints of array "A" and "M" which will make stuff clearer, but let M = B
In [66]: initial, final = 0,1
In [67]: z = np.linspace(initial,final,11)
In [68]: z
Out[68]: array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])
A is (2,2), but contains a mix of array and scalars
In [69]: A = np.array([[1,z],[0,1]], object)
In [70]: A
Out[70]:
array([[1,
array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])],
[0, 1]], dtype=object)
In [71]: A.shape
Out[71]: (2, 2)
Now make a (2,2) numeric array:
In [72]: L = np.eye(2)
In [75]: L[1,1] = 2
In [76]: np.matmul(L,A)
Out[76]:
array([[1.0,
array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])],
[0.0, array([2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])]],
dtype=object)
matmul does work with object dtype arrays, provided the elements implement the necessary + and *. The result is still (2,2), but the (1,1) term 2*z.
Now for the C:
In [77]: C = np.array([[z],[z]])
In [78]: C
Out[78]:
array([[[0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]],
[[0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]]])
In [79]: C.shape
Out[79]: (2, 1, 11)
This is float dtype, 3d array.
In [81]: B=Out[76]
In [82]: np.matmul(B,C)
Traceback (most recent call last):
File "<ipython-input-82-5eababb7341e>", line 1, in <module>
np.matmul(B,C)
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 1 is different from 2)
In [83]: B.shape
Out[83]: (2, 2)
In [84]: C.shape
Out[84]: (2, 1, 11)
There's a mismatch in shapes. But change C definition so it is a 2d array:
In [85]: C = np.array([z,z])
In [86]: C.shape
Out[86]: (2, 11)
In [87]: np.matmul(B,C)
Out[87]:
array([[array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]),
array([0.1 , 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2 ]),
...
array([1.8, 1.8, 1.8, 1.8, 1.8, 1.8, 1.8, 1.8, 1.8, 1.8, 1.8]),
array([2., 2., 2., 2., 2., 2., 2., 2., 2., 2., 2.])]],
dtype=object)
In [88]: _.shape
Out[88]: (2, 11)
Here the (2,2) B matmuls with (2,11) just fine producing (2,11). But each element is itself a (11,) array - because of the z used in defining A.
But you say you want a (2,1) C. To get that we have to use:
In [91]: C = np.empty((2,1), object)
In [93]: C[:,0]=[z,z]
In [94]: C
Out[94]:
array([[array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])],
[array([0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ])]],
dtype=object)
Be very careful when trying to create object dtype arrays. Things might not be what you expect.
Now matmul of (2,2) with (2,1) => (2,1), object dtype
In [95]: D = np.matmul(B,C)
In [96]: D.shape
Out[96]: (2, 1)
In [99]: D
Out[99]:
array([[array([0. , 0.11, 0.24, 0.39, 0.56, 0.75, 0.96, 1.19, 1.44, 1.71, 2. ])],
[array([0. , 0.2, 0.4, 0.6, 0.8, 1. , 1.2, 1.4, 1.6, 1.8, 2. ])]],
dtype=object)
Keep in mind that matmul is very fast with working with numeric dtype arrays. It does work with object dtype arrays, but speed is much slower, more like using list comprehensions.
Not sure what you're trying to do (you should provide a reproducible example, there are currently many missing variables), and the expected output.
Nevertheless, the definition of A is fundamentally wrong. I imagine you expect a 2x2 array, but as z is a (10,) shaped array, you will end up with A being a weird object array whose element (0,1) is an array.
This prevents you do to any further mathematical operation.
Related
I am having some difficulties achieving the following. Let's say I have two sets of data obtained from a test:
import numpy as np
a = np.array([[0.0, 1.0, 2.0, 3.0], [0.0, 2.0, 4.0, 6.0]]).T
b = np.array([[0.5, 1.5, 2.5, 3.5], [0.5, 1.5, 2.5, 3.5]]).T
where the data in the 0th column represents (in my case) displacement and the data in the 1th column represents the respective measured force values.
(Given data represents two lines with slopes of 2 and 1, both with a y-intercept of 0.)
Now I am trying to program a script that averages those two arrays despite the mismatched x-values, such that it will yield
c = [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5], [0.0, 0.75, 1.5,
2.25, 3.0, 3.75, 4.5, 5.25]]).T
(A line with a slope of 1.5 and a y-intercept of 0.)
I tried my best using slicing and linear interpolation, however it seems like I cannot get my head around it (I am a beginner).
I'd be very glad for any input and tips and hope the information I gave to you is sufficient!
Thanks in advance,
Robert
You can get the coefficients (slope and intercept) of each dataset, obtain the mean, and fit that data to a new array of x values.
Step by Step:
Fit deg-1 polynomial to each array a, and b using polyfit to get the coefficients of each (slope and intercept):
coef_a = np.polyfit(a[:,0], a[:,1], deg=1)
coef_b = np.polyfit(b[:,0], b[:,1], deg=1)
>>> coef_a
array([ 2.00000000e+00, 2.22044605e-16])
>>> coef_b
array([ 1.00000000e+00, 1.33226763e-15])
Get the mean of those coefficients to use as the coefficients of c:
coef_c = np.mean(np.stack([coef_a,coef_b]), axis=0)
>>> coef_c
array([ 1.50000000e+00, 7.77156117e-16])
Create new x-values for c using np.arange
c_x = np.arange(0,4,0.5)
>>> c_x
array([ 0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5])
use polyval to fit your new c coeficients to your new x values:
c_y = np.polyval(coef_c, c_x)
>>> c_y
array([ 7.77156117e-16, 7.50000000e-01, 1.50000000e+00,
2.25000000e+00, 3.00000000e+00, 3.75000000e+00,
4.50000000e+00, 5.25000000e+00])
Put your c_x and c_y values together using stack:
c = np.stack([c_x, c_y])
>>> c
array([[ 0.00000000e+00, 5.00000000e-01, 1.00000000e+00,
1.50000000e+00, 2.00000000e+00, 2.50000000e+00,
3.00000000e+00, 3.50000000e+00],
[ 7.77156117e-16, 7.50000000e-01, 1.50000000e+00,
2.25000000e+00, 3.00000000e+00, 3.75000000e+00,
4.50000000e+00, 5.25000000e+00]])
If you round that to 2 decimals, you'll see it's the same as your desired outcome:
>>> np.round(c, 2)
array([[ 0. , 0.5 , 1. , 1.5 , 2. , 2.5 , 3. , 3.5 ],
[ 0. , 0.75, 1.5 , 2.25, 3. , 3.75, 4.5 , 5.25]])
In a single statement:
c = np.stack([np.arange(0, 4, 0.5),
np.polyval(np.mean(np.stack([np.polyfit(a.T[0], a.T[1], 1),
np.polyfit(b.T[0], b.T[1], 1)]),
axis=0),
np.arange(0, 4, 0.5))])
>>> c
array([[ 0.00000000e+00, 5.00000000e-01, 1.00000000e+00,
1.50000000e+00, 2.00000000e+00, 2.50000000e+00,
3.00000000e+00, 3.50000000e+00],
[ 7.77156117e-16, 7.50000000e-01, 1.50000000e+00,
2.25000000e+00, 3.00000000e+00, 3.75000000e+00,
4.50000000e+00, 5.25000000e+00]])
I have a large matrix (236680*236680), and my pc does not have sufficient memory to read in the complete matrix so that I am thinking the Scipy sparse matrix. My goal is to multiply a generated matrix (not sparse) by np.eye(the number of observation)-np.ones(the number of observation)/the number of observation with a sparse matrix.
In Scipy, I use the following code, but the computation is still huge. My questions include:
to generate the first matrix, is there any other way to speed the process?
for the matrix multiplication, is there any way to reduce the memory usage, as the first matrix is not sparse?
-
from scipy.sparse import lil_matrix
fline=5
nn=1/fline
M=lil_matrix((fline,fline))
M.setdiag(values=1-nn,k=0)
for i in range(fline)[1:]:
M.setdiag(values=0-nn,k=i)
M.setdiag(values=0-nn,k=-i)
#the first matrix is:
array([[ 0.8, -0.2, -0.2, -0.2, -0.2],
[-0.2, 0.8, -0.2, -0.2, -0.2],
[-0.2, -0.2, 0.8, -0.2, -0.2],
[-0.2, -0.2, -0.2, 0.8, -0.2],
[-0.2, -0.2, -0.2, -0.2, 0.8]])
#the second matrix is:
array([[0., 0., 0., 1., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 1., 0.],
[0., 0., 0., 0., 0.],
[1., 0., 1., 0., 0.]])
a2=M.dot(B)
#the final expected results
array([[-0.2, 0. , -0.2, 0.6, 0. ],
[-0.2, 0. , -0.2, -0.4, 0. ],
[-0.2, 0. , -0.2, 0.6, 0. ],
[-0.2, 0. , -0.2, -0.4, 0. ],
[ 0.8, 0. , 0.8, -0.4, 0. ]])
Updated: is there any way to improve the speed of the cross product? Numpy dot and Scipy sparse dot functions are tested.
For the first problem: Mathematically,
arr1 = array([[ 0.8, -0.2, -0.2, -0.2, -0.2],
[-0.2, 0.8, -0.2, -0.2, -0.2],
[-0.2, -0.2, 0.8, -0.2, -0.2],
[-0.2, -0.2, -0.2, 0.8, -0.2],
[-0.2, -0.2, -0.2, -0.2, 0.8]])
is equivalent to
arr1 = -0.2 * [[1,1,1,1,1,], + 1
[1,1,1,1,1,], 1
[1,1,1,1,1,], 1
[1,1,1,1,1,], 1
[1,1,1,1,1,]] 1
= [1] [1, 1, 1, 1, 1] * 0.2 + 1
[1] 1
[1] 1
[1] 1
[1] 1
Thus, it can be generated using
-0.2 * np.outer([1,1,1,1,1], [1,1,1,1,1]) + scipy.sparse.identity(5)
For the second problem, let me abuse the notation
-0.2* [1] [1, 1, 1, 1, 1] # B + scipy.sparse.identity(5) # B
[1]
[1]
[1]
[1]
can be reduced to
np.outer([1, 1, 1, 1, 1], B.sum(axis=0)) * -0.2 + scipy.sparse.identity(5) # B
One needs not really compute np.outer([1, 1, 1, 1, 1], B.sum(axis=0)) as this would be a dense square matrix that the memory may not fit. (Note that the outer product is basically repeats B.sum(axis=0) in every row it contains.)
To recover the results in a memory efficient way, you only need to store B.sum(axis=0) and scipy.sparse.identity(5) # B .
Scipy sparse matrix is used, since one of the matrics is a sparse matrix and the cross product function in the sparse matrix is the fastest between Numpy and Scipy.
For the first question, #Tai's answer is the foundation, but I use numpy.full function (a little bit faster).
For the second question, dividing the whole matrix and save smaller computed matrices in files are used.
from scipy import sparse
from scipy.sparse import vstack
import h5sparse
import numpy as num
fline=236680
nn=1/fline; dd=1-nn; off=0-nn
div=int(fline/(61*10))
for i in range(61*10):
divM= num.full((fline, div), off) + sparse.identity(fline,format='csc')[:,0+div*i:div+div*i]
vs=[]
for j in range(divM.shape[1]):
divMB=csr_matrix(divM.T[j]).dot(weights)
vs.append(divMB)
divapp=vstack(vs)
if i ==0:
h5f = h5sparse.File("F:/dissertation/dallastest/temp/tt1.h5")
h5f.create_dataset('sparse/matrix', data=divapp, chunks=(389,),maxshape=(None,))
else:
h5f['sparse/matrix'].append(divapp)
I have an array of matrices in which I would like to multiply each matrix by a different number. I tried it this way:
>>> import numpy as np
>>> c = np.array([[[1, 2],[3, 4]],[[1, 2],[3, 4]]])
>>> d = np.array([0.1, 0.2])
>>> d*c
array([[[ 0.1, 0.4],
[ 0.3, 0.8]],
[[ 0.1, 0.4],
[ 0.3, 0.8]]])
While my intention is to get this result -
>>> np.array([d[0]*c[0], d[1]*c[1]])
array([[[ 0.1, 0.2],
[ 0.3, 0.4]],
[[ 0.2, 0.4],
[ 0.6, 0.8]]])
What is the NumPy'iest way to do it?
You need an extra couple of axes:
In [22]: d[:,None,None] * c
Out[22]:
array([[[ 0.1, 0.2],
[ 0.3, 0.4]],
[[ 0.2, 0.4],
[ 0.6, 0.8]]])
d[:,None,None] has shape (2,1,1) which is broadcast across your c array of shape (2,2,2) to multiply each block of c by the corresponding element of d.
You can do this with normal broadcasting:
>>> c*d[:, np.newaxis, np.newaxis]
array([[[ 0.1, 0.2],
[ 0.3, 0.4]],
[[ 0.2, 0.4],
[ 0.6, 0.8]]])
I want to combine two numpy arrays to produce an array with the largest values from each array.
import numpy as np
a = np.array([[ 0., 0., 0.5],
[ 0.1, 0.5, 0.5],
[ 0.1, 0., 0.]])
b = np.array([[ 0., 0., 0.0],
[ 0.5, 0.1, 0.5],
[ 0.5, 0.1, 0.]])
I would like to produce
array([[ 0., 0., 0.5],
[ 0.5, 0.5, 0.5],
[ 0.5, 0.1, 0.]])
I know you can do
a += b
which results in
array([[ 0. , 0. , 0.5],
[ 0.6, 0.6, 1. ],
[ 0.6, 0.1, 0. ]])
This is clearly not what I'm after. It seems like such an easy problem and I assume it most probably is.
You can use np.maximum to compute the element-wise maximum of the two arrays:
>>> np.maximum(a, b)
array([[ 0. , 0. , 0.5],
[ 0.5, 0.5, 0.5],
[ 0.5, 0.1, 0. ]])
This works with any two arrays, as long as they're the same shape or one can be broadcast to the shape of the other.
To modify the array a in-place, you can redirect the output of np.maximum back to a:
np.maximum(a, b, out=a)
There is also np.minimum for calculating the element-wise minimum of two arrays.
You are looking for the element-wise maximum.
Example:
>>> np.maximum([2, 3, 4], [1, 5, 2])
array([2, 5, 4])
http://docs.scipy.org/doc/numpy/reference/generated/numpy.maximum.html
inds = b > a
a[inds] = b[inds]
This modifies the original array a which is what += is doing in your example which may or may not be what you want.
I have a matrix
A = np.array([[0.2, 0.4, 0.6],
[0.5, 0.5, 0.5],
[0.6, 0.4, 0.2]])
I want a new matrix, where the value of the entry in row i and column j is the product of all the entries of the ith row of A, except for the cell of that row in the jth column.
array([[ 0.24, 0.12, 0.08],
[ 0.25, 0.25, 0.25],
[ 0.08, 0.12, 0.24]])
The solution that first occurred to me was
np.repeat(np.prod(A, 1, keepdims = True), 3, axis = 1) / A
But this only works so long as no entries have values zero.
Any thoughts? Thank you!
Edit: I have developed
B = np.zeros((3, 3))
for i in range(3):
for j in range(3):
B[i, j] = np.prod(i, A[[x for x in range(3) if x != j]])
but surely there is a more elegant way to accomplish this, which makes use of numpy's efficient C backend instead of inefficient python loops?
If you're willing to tolerate a single loop:
B = np.empty_like(A)
for col in range(A.shape[1]):
B[:,col] = np.prod(np.delete(A, col, 1), 1)
That computes what you need, a single column at a time. It is not as efficient as theoretically possible because np.delete() creates a copy; if you care a lot about memory allocation, use a mask instead:
B = np.empty_like(A)
mask = np.ones(A.shape[1], dtype=bool)
for col in range(A.shape[1]):
mask[col] = False
B[:,col] = np.prod(A[:,mask], 1)
mask[col] = True
A variation on your solution using repeat, uses [:,None].
np.prod(A,axis=1)[:,None]/A
My 1st stab at handling 0s is:
In [21]: B
array([[ 0.2, 0.4, 0.6],
[ 0. , 0.5, 0.5],
[ 0.6, 0.4, 0.2]])
In [22]: np.prod(B,axis=1)[:,None]/(B+np.where(B==0,1,0))
array([[ 0.24, 0.12, 0.08],
[ 0. , 0. , 0. ],
[ 0.08, 0.12, 0.24]])
But as the comment pointed out; the [0,1] cell should be 0.25.
This corrects that problem, but now has problems when there are multiple 0s in a row.
In [30]: I=B==0
In [31]: B1=B+np.where(I,1,0)
In [32]: B2=np.prod(B1,axis=1)[:,None]/B1
In [33]: B3=np.prod(B,axis=1)[:,None]/B1
In [34]: np.where(I,B2,B3)
Out[34]:
array([[ 0.24, 0.12, 0.08],
[ 0.25, 0. , 0. ],
[ 0.08, 0.12, 0.24]])
In [55]: C
array([[ 0.2, 0.4, 0.6],
[ 0. , 0.5, 0. ],
[ 0.6, 0.4, 0.2]])
In [64]: np.where(I,sum1[:,None],sum[:,None])/C1
array([[ 0.24, 0.12, 0.08],
[ 0.5 , 0. , 0.5 ],
[ 0.08, 0.12, 0.24]])
Blaz Bratanic's epsilon approach is the best non iterative solution (so far):
In [74]: np.prod(C+eps,axis=1)[:,None]/(C+eps)
A different solution iterating over the columns:
def paulj(A):
P = np.ones_like(A)
for i in range(1,A.shape[1]):
P *= np.roll(A, i, axis=1)
return P
In [130]: paulj(A)
array([[ 0.24, 0.12, 0.08],
[ 0.25, 0.25, 0.25],
[ 0.08, 0.12, 0.24]])
In [131]: paulj(B)
array([[ 0.24, 0.12, 0.08],
[ 0.25, 0. , 0. ],
[ 0.08, 0.12, 0.24]])
In [132]: paulj(C)
array([[ 0.24, 0.12, 0.08],
[ 0. , 0. , 0. ],
[ 0.08, 0.12, 0.24]])
I tried some timings on a large matrix
In [13]: A=np.random.randint(0,100,(1000,1000))*0.01
In [14]: timeit paulj(A)
1 loops, best of 3: 23.2 s per loop
In [15]: timeit blaz(A)
10 loops, best of 3: 80.7 ms per loop
In [16]: timeit zwinck1(A)
1 loops, best of 3: 15.3 s per loop
In [17]: timeit zwinck2(A)
1 loops, best of 3: 65.3 s per loop
The epsilon approximation is probably the best speed we can expect, but has some rounding issues. Having to iterate over many columns hurts the speed. I'm not sure why the np.prod(A[:,mask], 1) approach is slowest.
eeclo https://stackoverflow.com/a/22441825/901925 suggested using as_strided. Here's what I think he has in mind (adapted from an overlapping block question, https://stackoverflow.com/a/8070716/901925)
def strided(A):
h,w = A.shape
A2 = np.hstack([A,A])
x,y = A2.strides
strides = (y,x,y)
shape = (w, h, w-1)
blocks = np.lib.stride_tricks.as_strided(A2[:,1:], shape=shape, strides=strides)
P = blocks.prod(2).T # faster to prod on last dim
# alt: shape = (w-1, h, w), and P=blocks.prod(0)
return P
Timing for the (1000,1000) array is quite an improvement over the column iterations, though still much slower than the epsilon approach.
In [153]: timeit strided(A)
1 loops, best of 3: 2.51 s per loop
Another indexing approach, while relatively straight forward, is slower, and produces memory errors sooner.
def foo(A):
h,w = A.shape
I = (np.arange(w)[:,None]+np.arange(1,w))
I1 = np.array(I)%w
P = A[:,I1].prod(2)
return P
Im on the run, so I do not have time to work out this solution; but what id do is create a contiguous circular view over the last axis, by means of concatenating the array to itself along the last axis, and then use np.lib.index_tricks.as_strided to select the appropriate elements to take an np.prod over. No python loops, no numerical approximation.
edit: here you go:
import numpy as np
A = np.array([[0.2, 0.4, 0.6],
[0.5, 0.5, 0.5],
[0.5, 0.0, 0.5],
[0.6, 0.4, 0.2]])
B = np.concatenate((A,A),axis=1)
C = np.lib.index_tricks.as_strided(
B,
A.shape +A.shape[1:],
B.strides+B.strides[1:])
D = np.prod(C[...,1:], axis=-1)
print D
Note: this method is not ideal, as it is O(n^3). See my other posted solution, which is O(n^2)
If you are willing to tolerate small error you could use the solution you first proposed.
A += 1e-10
np.around(np.repeat(np.prod(A, 1, keepdims = True), 3, axis = 1) / A, 9)
Here is an O(n^2) method without python loops or numerical approximation:
def double_cumprod(A):
B = np.empty((A.shape[0],A.shape[1]+1),A.dtype)
B[:,0] = 1
B[:,1:] = A
L = np.cumprod(B, axis=1)
B[:,1:] = A[:,::-1]
R = np.cumprod(B, axis=1)[:,::-1]
return L[:,:-1] * R[:,1:]
Note: it appears to be about twice as slow as the numerical approximation method, which is in line with expectation.