I have a sparse matrix A (using scipy.sparse) and a vector b, and want to solve Ax = b for x. A has more rows than columns, so it appears to be overdetermined; however, the rows of A are linearly dependent, so that in actuality the row rank of A is equal to the number of columns. For example, A could be
A = np.array([[1., 1.], [-1., -1.], [1., 0.]])
while b is
b = np.array([0., 0., 1.])
The solution is then x = [1., -1.]. I'm wondering how to solve this system in Python, using the functions available in scipy.sparse.linalg. Thanks!
Is your system possibly underdetermined? If it is not, and there is actually a solution, then the least squares solution will be that solution, so you can try
from scipy.sparse.linalg import lsqr
return_values = lsqr(A, b)
x = return_values[0]
If your system is actually underdetermined, this should find you the minimum L2 norm solution. If it doesn't work, set the parameter damp to something very small (e.g. 1e-5).
If your system is exactly determined (i.e. A is of full rank) and has a solution, and your matrix A is tall, as you describe it, then you can find an equivalent system in the normal equations:
A.T.dot(A).dot(x) == A.T.dot(b)
has a unique solution in x. This is a square linear system and is thus solvable using linear system solvers such as scipy.sparse.linalg.spsolve
The formally correct way of solving your problem is to use SVD. You have a system of the form
A [MxN] * x [Nx1] = b [Mx1]
The SVD decomposes the matrix A into three others, so you get:
U [MxM] * S[MxN] * V[N*N] * x[Nx1] = b[Mx1]
The matrices U and V are both orthogonal (their inverse is their transpose), and S is a diagonal matrix. If we rewrite the above we get:
S[MxN] * V [N * N] * x[Nx1] = U.T [MxM] * b [Mx1]
If M > N then the matrix S will have its last M - N rows full of zeros, and if your system is truly determined, then U.T b should also have the last M - N rows zero. That means that you can solve your system as:
>>> a = np.array([[1., 1.], [-1., -1.], [1., 0.]])
>>> b = np.array([0., 0., 1.])
>>> u, s, v = np.linalg.svd(a)
>>> np.allclose(u.T.dot(b)[-m+n:], 0) #check system is not overdetermined
True
>>> np.linalg.solve(s[:, None] * v, u.T.dot(b)[:n])
array([ 1., -1.])
Related
I have the following simple python function that calculates the entropy of a single input X according to Shannon's Theory of Information:
import numpy as np
def entropy(X:'numpy array'):
_, frequencies = np.unique(X, return_counts=True)
probabilities = frequencies/X.shape[0]
return -np.sum(probabilities*np.log2(probabilities))
a = np.array([1., 1., 1., 3., 3., 2.])
b = np.array([1., 1., 1., 3., 3., 3.])
c = np.array([1., 1., 1., 1., 1., 1.])
print(f"entropy(a): {entropy(a)}")
print(f"entropy(b): {entropy(b)}")
print(f"entropy(c): {entropy(c)}")
With the output being the following:
entropy(a): 1.4591479170272446
entropy(b): 1.0
entropy(c): -0.0
However, I also need to calculate the derivative over dx:
d entropy / dx
This is not an easy task since the main formula
-np.sum(probabilities*np.log2(probabilities))
takes in probabilities, not x values, therefore it is not clear how to differentiate over dx.
Does anyone have an idea on how to do this?
One way to solve this is to use finite differences to compute the derivative numerically.
In this context, we can define a small constant to help us compute the numerical derivative. This function takes a one-argument function and computes its derivative for input x:
ε = 1e-12
def derivative(f, x):
return (f(x + ε) - f(x)) / ε
To make our work easier, let us define a function that computes the innermost operation of the entropy:
def inner(x):
return x * np.log2(x)
Recall that the derivative of the sum is the sum of derivatives. Therefore, the real derivative computation takes place in the inner function we just defined.
So, the numerical derivative of the entropy is:
def numerical_dentropy(X):
_, frequencies = np.unique(X, return_counts=True)
probabilities = frequencies / X.shape[0]
return -np.sum([derivative(inner, p) for p in probabilities])
Can we do better? Of course we can! The key insight here is the product rule: (f g)' = fg' + gf', where f=x and g=np.log2(x). (Also notice that d[log_a(x)]/dx = 1/(x ln(a)).)
So, the analytical entropy can be computed as:
import math
def dentropy(X):
_, frequencies = np.unique(X, return_counts=True)
probabilities = frequencies / X.shape[0]
return -np.sum([(1/math.log(2, math.e) + np.log2(p)) for p in probabilities])
Using the sample vectors for testing, we have:
a = np.array([1., 1., 1., 3., 3., 2.])
b = np.array([1., 1., 1., 3., 3., 3.])
c = np.array([1., 1., 1., 1., 1., 1.])
print(f"numerical d[entropy(a)]: {numerical_dentropy(a)}")
print(f"numerical d[entropy(b)]: {numerical_dentropy(b)}")
print(f"numerical d[entropy(c)]: {numerical_dentropy(c)}")
print(f"analytical d[entropy(a)]: {dentropy(a)}")
print(f"analytical d[entropy(b)]: {dentropy(b)}")
print(f"analytical d[entropy(c)]: {dentropy(c)}")
Which, when executed, gives us:
numerical d[entropy(a)]: 0.8417710972707937
numerical d[entropy(b)]: -0.8854028621385623
numerical d[entropy(c)]: -1.4428232973189605
analytical d[entropy(a)]: 0.8418398787754222
analytical d[entropy(b)]: -0.8853900817779268
analytical d[entropy(c)]: -1.4426950408889634
As a bonus, we can test whether this is correct with an automatic differentiation library:
import torch
a, b, c = torch.from_numpy(a), torch.from_numpy(b), torch.from_numpy(c)
def torch_entropy(X):
_, frequencies = torch.unique(X, return_counts=True)
frequencies = frequencies.type(torch.float32)
probabilities = frequencies / X.shape[0]
probabilities.requires_grad_(True)
return -(probabilities * torch.log2(probabilities)).sum(), probabilities
for v in a, b, c:
h, p = torch_entropy(v)
print(f'torch entropy: {h}')
h.backward()
print(f'torch derivative: {p.grad.sum()}')
Which gives us:
torch entropy: 1.4591479301452637
torch derivative: 0.8418397903442383
torch entropy: 1.0
torch derivative: -0.885390043258667
torch entropy: -0.0
torch derivative: -1.4426950216293335
I need a 2n x n matrix in NumPy consisting of the n x n identity matrix and the negative n x n identity matrix stacked on top of one another.
This was my original solution, which works fine.
def id_stack(n):
id_ = np.identity(n)
return np.vstack((id_, -id_))
id_stack(3)
# array([[ 1., 0., 0.],
# [ 0., 1., 0.],
# [ 0., 0., 1.],
# [-1., -0., -0.],
# [-0., -1., -0.],
# [-0., -0., -1.]])
Then I figured I could just set the diagonals instead and be faster like this, which also works.
def id_stack2(n):
full = np.zeros((2*n, n))
rng = np.arange(n)
full[rng, rng] = 1
full[rng + n, rng] = -1
return full
I was wondering if there is an even faster way of accomplishing this, maybe using some kind of stride tricks?
As you probably noticed from your own examples, allocating one big buffer and setting elements in it is generally faster than allocating two smaller buffers and a big buffer to copy them into.
The neat thing about numpy is that you can get views to the same buffer without allocating a new array. For example:
output = np.zeros((2 * n, n))
A useful view in this case is
flat = output.ravel()
You can set every n + 1st element to 1, starting from the first, for a total of n elements in the flattened view, and similar for -1. This requires only a simple indexing operation on the raveled view:
output[:n * n:n + 1] = 1
output[n * n::n + 1] = -1
This avoids creating the full range arrays, and triggering advanced indexing semantics, which are more memory intensive as well.
I am realizing Exponentiation of a matrix using FOR:
import numpy as np
fl=2
cl=2
fl2=fl
cl2=cl
M = random.random((fl,cl))
M2 = M
Result = np.zeros((fl,cl))
Temp = np.zeros((fl,cl))
itera = 2
print('Matriz A:\n',M)
print('Matriz AxA:\n',M2)
for i in range (0,itera):
for a in range(0,fl):
for b in range (0,cl):
Result[a,b]+=M[a,b]*M[a,b]
temp[a,b]=Result[a,b]
Res[a,k]=M[a,b]
print('Potencia:\n',temp)
print('Matriz:\n', Result)
The error is that it does not perform well the multiplication in Result[a,b]+=M[a,b]*M[a,b] and when I save it in a temporary matrix to multiply it with the original matrix, it does not make the next jump in for i in range (0,itera):
I know I can perform the function np.matmul
but I try to do it with the FOR loop
Example
You're looking for np.linalg.matrix_power.
If you're using numpy, don't use a for loop, use a vectorized operation.
arr = np.arange(16).reshape((4,4))
np.linalg.matrix_power(arr, 3)
array([[ 1680, 1940, 2200, 2460],
[ 4880, 5620, 6360, 7100],
[ 8080, 9300, 10520, 11740],
[11280, 12980, 14680, 16380]])
Which is the same as the explicit multiplication:
arr # arr # arr
>>> np.array_equal(arr # arr # arr, np.linalg.matrix_power(arr, 3))
True
Since you asked
If you really want a naive solution using loops, we can put together the pieces quite easily. First we need a way to actually multiple the matrices. There are options that beat n^3 complexity, this answer is not going to do that. Here is a basic matrix multiplication function:
def matmultiply(a, b):
res = np.zeros(a.shape)
size = a.shape[0]
for i in range(size):
for j in range(size):
for k in range(size):
res[i][j] += a[i][k] * b[k][j]
return res
Now you need an exponential function. This function takes a matrix and a power, and raises a matrix to that power.
def loopy_matrix_power(a, n):
res = np.identity(a.shape[0])
while n > 0:
if n % 2 == 0:
a = matmultiply(a, a)
n /= 2
else:
res = matmultiply(res, a)
n -= 1
return res
In action:
loopy_matrix_power(arr, 3)
array([[ 1680., 1940., 2200., 2460.],
[ 4880., 5620., 6360., 7100.],
[ 8080., 9300., 10520., 11740.],
[11280., 12980., 14680., 16380.]])
There are some problems here:
you do not reset the result matrix after multiplication is done, hence you keep adding more values; and
you never assign the result back to m to perform a next generation of multiplications.
Naive power implementation
I think it is also better to "encapsulate" matrix multiplication in a separate function, like:
def matmul(a1, a2):
m, ka = a1.shape
kb, n = a2.shape
if ka != kb:
raise ValueError()
res = np.zeros((m, n))
for i in range(m):
for j in range(n):
d = 0.0
for k in range(ka):
d += a1[i,k] * a2[k,j]
res[i, j] = d
return res
Then we can calculate the power of this matrix with:
m2 = m
for i in range(topow-1):
m = matmul(m, m2)
Note that we can not use m here as the only matrix. Since if we write m = matmul(m, m), then m is now m2. But that means that if we perform the multiplication a second time, we get m4 instead of m3.
This then produces the expected results:
>>> cross = np.array([[1,0,1],[0,1,0], [1,0,1]])
>>> matmul(cross, cross)
array([[2., 0., 2.],
[0., 1., 0.],
[2., 0., 2.]])
>>> matmul(cross, matmul(cross, cross))
array([[4., 0., 4.],
[0., 1., 0.],
[4., 0., 4.]])
>>> matmul(cross, matmul(cross, matmul(cross, cross)))
array([[8., 0., 8.],
[0., 1., 0.],
[8., 0., 8.]])
Logarithmic power multiplication
The above can calculate the Mn in O(n) (linear time), but we can do better, we can calculate this matrix in logarithmic time: we do this by looking if the power is 1, if it is, we simply return the matrix, if it is not, we check if the power is even, if it is even, we multiply the matrix with itself, and calculate the power of that matrix, but with the power divided by two, so M2 n=(M×M)n. If the power is odd, we do more or less the same, except that we multiply it with the original value for M: M2 n + 1=M×(M×M)n. Like:
def matpow(m, p):
if p <= 0:
raise ValueError()
if p == 1:
return m
elif p % 2 == 0: # even
return matpow(matmul(m, m), p // 2)
else: # odd
return matmul(m, matpow(matmul(m, m), p // 2))
The above can be written more elegantly, but I leave this as an exercise :).
Note however that using numpy arrays for scalar comuputations is typically less efficient than using the matrix multiplication (and other functions) numpy offers. These are optimized, and are not interpreted, and typically outperform Python equivalents significantly. Therefore I would really advice you to use these. The numpy functions are also tested, making it less likely that there are bugs in it.
I have a list of times (called times in my code, produced by the code suggested to me in the thread astropy.io fits efficient element access of a large table) and I want to do some statistical tests for periodicity, using Zn^2 and epoch folding tests. Some steps in the code take quite a while to run, and I am wondering if there is a faster way to do it. I have tried the equivalent map and lambda functions, but that takes even longer. My list of times has several hundred or maybe thousands of elements, depending on the dataset. Here is my code:
phase=[(x-mintime)*testfreq[m]-int((x-mintime)*testfreq[m]) for x in times]
# the above step takes 3 seconds for the dataset I am using for testing
# testfreq[m] is just one of several hundred frequencies I am testing
# times is of type numpy.ndarray
phasebin=[int(ph*numbins)for ph in phase]
# 1 second (numbins is 20)
powerarray=[phasebin.count(n) for n in range(0,numbins-1)]
# 0.3 seconds
poweravg=np.mean(powerarray)
chisq[m]=sum([(pow-poweravg)**2/poweravg for pow in powerarray])
# the above 2 steps are very quick
for n in range(0,maxn): # maxn is 3
cosparam=sum([(np.cos(2*np.pi*(n+1)*ph)) for ph in phase])
sinparam=sum([(np.sin(2*np.pi*(n+1)*ph)) for ph in phase])
# these steps each take 4 seconds
z2[m,n]=sum(z2[m,])+(cosparam**2+sinparam**2)/count
# this is quick (count is the number of times)
As this steps through several hundred frequencies on either side of frequencies identified through an FFT search, it takes a very long time to run. The same functionality in a lower level language runs much more quickly, but I need some of the Python modules for plotting, etc. I am hoping that Python can be persuaded to do some of the operations, particularly the phase, phasebin, powerarray, cosparam, and sinparam calculations, significantly faster, but I am not sure how to make this happen. Can anyone tell me how this can be done, or do I have to write and call functions in C or fortran? I know that this could be done in a few minutes e.g. in fortran, but this Python code takes hours as it is.
Thanks very much.
Instead of Python lists, you could use the numpy library, it is much faster for linear algebra type operations. For example to add two arrays in an element-wise fashion
>>> import numpy as np
>>> a = np.array([1,2,3,4,5])
>>> b = np.array([2,3,4,5,6])
>>> a + b
array([ 3, 5, 7, 9, 11])
Similarly, you can multiply arrays by scalars which multiplies each element as you'd expect
>>> 2 * a
array([ 2, 4, 6, 8, 10])
As far as speed, here is the Python list equivalent of adding two lists
>>> c = [1,2,3,4,5]
>>> d = [2,3,4,5,6]
>>> [i+j for i,j in zip(c,d)]
[3, 5, 7, 9, 11]
Then timing the two
>>> from timeit import timeit
>>> setup = '''
import numpy as np
a = np.array([1,2,3,4,5])
b = np.array([2,3,4,5,6])'''
>>> timeit('a+b', setup)
0.521275608325351
>>> setup = '''
c = [1,2,3,4,5]
d = [2,3,4,5,6]'''
>>> timeit('[i+j for i,j in zip(c,d)]', setup)
1.2781205834379108
In this small example numpy was more than twice as fast.
for loop substitute - operating on complete arrays
First multiply phase by 2*pi*n using broadcasting
phase = np.arange(10)
maxn = 3
ens = np.arange(1, maxn+1) # array([1, 2, 3])
two_pi_ens = 2*np.pi*ens
b = phase * two_pi_ens[:, np.newaxis]
b.shape is (3,10) one row for each value of range(1, maxn)
Take the cosine then sum to get the three cosine parameters
c = np.cos(b)
c_param = c.sum(axis = 1) # c_param.shape is 3
Take the sine then sum to get the three sine parameters
s = np.sin(b)
s_param = s.sum(axis = 1) # s_param.shape is 3
Sum of the squares divided by count
d = (np.square(c_param) + np.square(s_param)) / count
# d.shape is (3,)
Assign to z2
for n in range(maxn):
z2[m,n] = z2[m,:].sum() + d[n]
That loop is doing a cumulative sum. numpy ndarrays have a cumsum method.
If maxn is small (3 in your case) it may not be noticeably faster.
z2[m,:] += d
z2[m,:].cumsum(out = z2[m,:])
To illustrate:
>>> a = np.ones((3,3))
>>> a
array([[ 1., 1., 1.],
[ 1., 1., 1.],
[ 1., 1., 1.]])
>>> m = 1
>>> d = (1,2,3)
>>> a[m,:] += d
>>> a
array([[ 1., 1., 1.],
[ 2., 3., 4.],
[ 1., 1., 1.]])
>>> a[m,:].cumsum(out = a[m,:])
array([ 2., 5., 9.])
>>> a
array([[ 1., 1., 1.],
[ 2., 5., 9.],
[ 1., 1., 1.]])
>>>
Hi Everyone I am python newbie
I have to implement lasso L1 regression for a class assignment. This involves solving a quadratic equation involving block matrices.
minimize x^t * H * x + f^t * x
where x > 0
Where H is a 2 X 2 block matrix with each element being a k dimensional matrix and x and f being a 2 X 1 vectors each element being a k dimension vector.
I was thinking of using ndarrays.
Such that :
np.shape(H) = (2, 2, k, k)
np.shape(x) = (2, k)
But I figured out that np.dot(X, H) doesn't work here.
Is there an easy way to solve this problem? Thanks in advance.
First of all, I am convinced that converting to matrices will lead to more efficient computations. Stating that, if you consider your 2k x 2k matrix being a 2 x 2 matrix, then you operate in a tensor product of vector spaces, and have to use tensordot instead of dot.
Let give it a try, with k=5 for example:
>>> import numpy as np
>>> k = 5
Define our matrix a and vector x
>>> a = np.arange(1.*2*2*k*k).reshape(2,2,k,k)
>>> x = np.arange(1.*2*k).reshape(2,k)
>>> x
array([[ 0., 1., 2., 3., 4.],
[ 5., 6., 7., 8., 9.]])
now we can multipy our tensors. Be sure to choose right axes, I didn't tested following formula explicetely, and there might be an error
>>> result = np.tensordot(a,x,([1,3],[0,1]))
>>> result
array([[ 985., 1210., 1435., 1660., 1885.],
[ 3235., 3460., 3685., 3910., 4135.]])
>>> np.shape(result)
(2, 5)
np.einsum gives good control over which axes are summed.
np.einsum('ijkl,jk',H,x)
is one possible (generalized) dot product, (2,4) (first and last dim of H)
np.einsum('ijkl,jl',H,x)
is another. You need to be explicit - which dimensions of x go with which of H.