Related
I keep getting a value error when I try to add the last step to my array x[i+1]=... What am I doing wrong here? allocated arrays with similar code and had no issues earlier. Ay help is appreciated.
Source Code:
import numpy as np
import matplotlib.pyplot as plt
#parameters
sigma=10.
rho=28.
beta=8/3.
ti=0.
tf=100
dt=0.01
#pre-allocation
x = np.zeros(tf)
y = np.zeros(tf)
z = np.zeros(tf)
#initial conditions
x[0]=1.
y[0]=1.
z[0]=1.
#functions
fx= lambda x: sigma*(y-x) #y too?
fy= lambda y: x*(rho-z)-y
fz= lambda z: x*y-(beta*z)
#euler-richardson
for i in np.arange(0,tf-1):
k1_x = fx(x[i])
k1_y = fy(y[i])
k1_z = fz(z[i])
k2_x = fx((x[i]+(0.5*k1_x))*dt) #maybe just dt?
k2_y = fy((y[i]+(0.5*k1_y))*dt)
k2_z = fz((z[i]+(0.5*k1_z))*dt)
x[i+1] = x[i] + k2_x
y[i+1] = y[i] + k2_y
z[i+1] = z[i] + k2_z
Error:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
TypeError: only size-1 arrays can be converted to Python scalars
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
Input In [10], in <cell line: 2>()
8 k2_y = fy((y[i]+(0.5*k1_y))*dt)
9 k2_z = fz((z[i]+(0.5*k1_z))*dt)
---> 11 x[i+1] = x[i] + k2_x
12 y[i+1] = y[i] + k2_y
13 z[i+1] = z[i] + k2_z
ValueError: setting an array element with a sequence.
Steps needed to prevent this error:
Easiest way to fix this problem is to use the data-type which support all type of data-type.
Second way to fix this problem is to match the default data-type of array and assigning value.
You need to pass the dtype of array as object, as we must have same dtype for element in the array while in the List we can have a different dtype. It will work fine, I have run your code, you need to change only these lines
#pre-allocation
x = np.zeros(tf, dtype=object)
y = np.zeros(tf, dtype=object)
z = np.zeros(tf, dtype=object)
This solution is helpful.
Let's try this in IPython to see what happens:
In [1]: import numpy as np
In [2]: sigma=10.
...: rho=28.
...: beta=8/3.
...: ti=0.
...: tf=100
...: dt=0.01
...: #pre-allocation
...: x = np.zeros(tf)
...: y = np.zeros(tf)
...: z = np.zeros(tf);
In [3]: #initial conditions
...: x[0]=1.
...: y[0]=1.
...: z[0]=1.
In [5]: # functions
...: fx= lambda x: sigma*(y-x)
...: fy= lambda y: x*(rho-z)-y
...: fz= lambda z: x*y-(beta*z);
Out[5]:
Personally, I find the usage of x, y and z as both global names and function parameters confusing.
Now we call one of your functions:
In [6]: fx(x[4])
Out[6]:
array([10., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0.])
Your functions only have one parameter. That means they will use the other referenced objects from the context they are called in.
In this case these are the numpy arrays x, y and z.
Hence the return value of an array.
Effectivtly, fx(x[4]) is the same as:
In [10]: sigma*(y-x[4])
Out[10]:
array([10., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0.])
I would like to compute the following with sympy:
Where I is a 3x3 identity matrix. The end use is to use this with symbolic matrices.
I have the following:
import sympy as sp
I = sp.eye(3)
Missing operations with sympy
With numpy I can just use the einsum function and have:
import numpy as np
I = np.eye(3)
Res = (np.einsum("ij,kl->ijkl", I, I)
+ np.einsum("ik,jl->ijkl", I, I)
+ np.einsum("il,jk->ijkl", I, I))
However, einsum will not accept sympy's objects for this operation.
How can I compute this with sympy?
While the notation makes the einsum expression convenient, it isn't a matrix-product. It's more like an extended outer product. There's no sum-of-products:
In [22]: I = np.eye(3)
...: Res = (
...: np.einsum("ij,kl->ijkl", I, I)
...: + np.einsum("ik,jl->ijkl", I, I)
...: + np.einsum("il,jk->ijkl", I, I)
...: )
In [23]: Res
Out[23]:
array([[[[3., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]],
[[0., 1., 0.],
[1., 0., 0.],
[0., 0., 0.]],
[[0., 0., 1.],
[0., 0., 0.],
[1., 0., 0.]]],
[[[0., 1., 0.],
[1., 0., 0.],
[0., 0., 0.]],
[[1., 0., 0.],
[0., 3., 0.],
[0., 0., 1.]],
[[0., 0., 0.],
[0., 0., 1.],
[0., 1., 0.]]],
[[[0., 0., 1.],
[0., 0., 0.],
[1., 0., 0.]],
[[0., 0., 0.],
[0., 0., 1.],
[0., 1., 0.]],
[[1., 0., 0.],
[0., 1., 0.],
[0., 0., 3.]]]])
In [24]: Res.shape
Out[24]: (3, 3, 3, 3)
In numpy we can use broadcasting to do the same thing:
In [25]: res1 = I[:,:,None,None]*I + I[:,None,:,None]*I[None,:,None,:]+I[:,None,None,:]*I[None,:,:,Non
...: e]
In [26]: res1.shape
Out[26]: (3, 3, 3, 3)
In [27]: np.allclose(Res, res1)
Out[27]: True
Your sp.eye produces a MutableDenseMatrix
https://docs.sympy.org/latest/modules/matrices/dense.html#sympy.matrices.dense.MutableDenseMatrix
Feel free to study its docs. My impression is that sympy matrices don't implement multidimensional arrays with anything like the power of numpy.
If I run:
x = np.zeros(6)
y = np.zeros([7, 6])
z = y * x
Then everything is fine, and there are no Python errors.
However, I am using a Python module (call if foo) containing a function (call if bar), which returns a 7x6 NumPy array. It has the same shape as y above, and the same data type (float64). But when I run the following:
x = np.zeros(6)
y = foo.bar()
z = y * x
I get the following error:
ValueError: shapes (7,6) and (1,6) not aligned: 6 (dim 1) != 1 (dim 0)
But as far as I can tell, y is exactly the same format in these two examples, with the same shape and data type. What's causing this error, and why is it not caused in the first example?
In [446]: x = np.zeros(6)
...: y = np.zeros([7, 6])
...: z = y * x
In [447]: z.shape
Out[447]: (7, 6)
Here we are doing element-wise multiplication, a (7,6) with a (6,). By broadcasting the (6,) becomes (1,6) and then (7,6) to match y.
Evidently in the foo.bar case, y is np.matrix subclass:
In [454]: y1 = np.matrix(y)
In [455]: y1*x
---...
219 # This promotes 1-D vectors to row vectors
--> 220 return N.dot(self, asmatrix(other))
...
ValueError: shapes (7,6) and (1,6) not aligned: 6 (dim 1) != 1 (dim 0)
Note the different display for y1:
In [456]: y1
Out[456]:
matrix([[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.]])
With np.matrix * is defined as np.dot, the matrix product. The x is also converted np.matrix, producing a (1,6) matrix. The error message follows from the definition of matrix multiplication.
np.multiply can be used force the element-wise multiplication. Note the class of the result:
In [458]: np.multiply(y1,x)
Out[458]:
matrix([[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.]])
Because of confusions like this np.matrix is being discouraged.
I don't know which version you're running, but I am running version 1.16.3 on Python 3.
It seems to me you're defining your x differently than in your example snippet. You seem to be defining it as a 6x1 matrix instead of a "vector", which is considered on Numpy to only have one dimension. Try multiplying y by np.zeros([6,1]) and you'll see an error.
Bottom line is:
Use the .shape property a lot when debugging, it's very useful when you're doing matrix multiplication.
On np, multiplication between a matrix and a one dimensional array is carried differently than multiplications between 2 matrices.
Is there a way I can allocate memory for scipy sparse matrix functions to process large datasets?
Specifically, I'm attempting to use Asymmetric Least Squares Smoothing (translated into python here and the original here) to perform a baseline correction on a large mass spec dataset (length of ~60,000).
The function (see below) uses the scipy.sparse matrix operations.
def baseline_als(y, lam, p, niter):
L = len(y)
D = sparse.csc_matrix(np.diff(np.eye(L), 2))
w = np.ones(L)
for i in xrange(niter):
W = sparse.spdiags(w, 0, L, L)
Z = W + lam * D.dot(D.transpose())
z = spsolve(Z, w*y)
w = p * (y > z) + (1-p) * (y < z)
return z
I have no problem when I pass data sets that are 10,000 or less in length:
baseline_als(np.ones(10000),100,0.1,10)
But when passing larger data sets, e.g.
baseline_als(np.ones(50000), 100, 0.1, 10)
I get a MemoryError, for the line
D = sparse.csc_matrix(np.diff(np.eye(L), 2))
Try changing
D = sparse.csc_matrix(np.diff(np.eye(L), 2))
to
diag = np.ones(L - 2)
D = sparse.spdiags([diag, -2*diag, diag], [0, -1, -2], L, L-2)
D will be a sparse matrix in DIAgonal format. If it turns out that being in CSC format is important, convert it using the tocsc() method:
D = sparse.spdiags([diag, -2*diag, diag], [0, -1, -2], L, L-2).tocsc()
The following example shows that the old and new versions generate the same matrix:
In [67]: from scipy import sparse
In [68]: L = 8
Original:
In [69]: D = sparse.csc_matrix(np.diff(np.eye(L), 2))
In [70]: D.A
Out[70]:
array([[ 1., 0., 0., 0., 0., 0.],
[-2., 1., 0., 0., 0., 0.],
[ 1., -2., 1., 0., 0., 0.],
[ 0., 1., -2., 1., 0., 0.],
[ 0., 0., 1., -2., 1., 0.],
[ 0., 0., 0., 1., -2., 1.],
[ 0., 0., 0., 0., 1., -2.],
[ 0., 0., 0., 0., 0., 1.]])
New version:
In [71]: diag = np.ones(L - 2)
In [72]: D = sparse.spdiags([diag, -2*diag, diag], [0, -1, -2], L, L-2)
In [73]: D.A
Out[73]:
array([[ 1., 0., 0., 0., 0., 0.],
[-2., 1., 0., 0., 0., 0.],
[ 1., -2., 1., 0., 0., 0.],
[ 0., 1., -2., 1., 0., 0.],
[ 0., 0., 1., -2., 1., 0.],
[ 0., 0., 0., 1., -2., 1.],
[ 0., 0., 0., 0., 1., -2.],
[ 0., 0., 0., 0., 0., 1.]])
I have an NxN matrix filled with zeros. Now I want to add to the matrix, say, n ones and m twos to random places. I.e. I want to create a matrix where there is some fixed amount of a given number at random places and possibly a fixed amount of some other given number in random places. How do I do this?
In Matlab I would do this by making a random permutation of the matrix indices with randperm() and then filling the n first indices given by randperm of the matrix with ones and m next with twos.
You can use numpy.random.shuffle to randomly permute an array in-place.
>>> import numpy as np
>>> X = np.zeros(N * N)
>>> X[:n] = 1
>>> X[n:n+m] = 2
>>> np.random.shuffle(X)
>>> X = X.reshape((N, N))
Would numpy.random.permutation be what you are looking for?
You can do something like this:
In [9]: a=numpy.zeros(100)
In [10]: p=numpy.random.permutation(100)
In [11]: a[p[:10]]=1
In [12]: a[p[10:20]]=2
In [13]: a.reshape(10,10)
Out[13]:
array([[ 0., 1., 0., 0., 0., 2., 0., 1., 0., 0.],
[ 0., 0., 1., 0., 0., 0., 0., 0., 2., 0.],
[ 0., 2., 0., 0., 0., 0., 2., 0., 0., 1.],
[ 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 2., 0., 2., 1., 1., 0.],
[ 0., 0., 0., 0., 1., 0., 2., 0., 0., 0.],
[ 0., 2., 0., 2., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
[ 1., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
[ 0., 0., 0., 0., 2., 0., 0., 0., 1., 0.]])
Here we create a random permutation, then set the first 10 indices taken from the permutation in a to 1, then the next 10 indices to 2.
To generate the indices of the elements for where to add ones and twos, what about this?
# assuming N, n and m exist.
In [1]: import random
In [3]: indices = [(m, n) for m in range(N) for n in range(N)]
In [4]: random_indices = random.sample(indices, n + m)
In [5]: ones = random_indices[:n]
In [6]: twos = random_indices[n:]
Corrected as commented by Petr Viktorin in order not to have overlapping indexes in ones and twos.
An alternate way to generate the indices:
In [7]: import itertools
In [8]: indices = list(itertools.product(range(N), range(N)))