I am working on the implementation of a gauss elimination algorithm in python using numpy. While working on it, I have noticed a weird behaviour. Here goes what I've done so far:
def gauss_elimination(a, b):
n = len(b)
print(a)
for k in range(0, n-1):
for i in range(k+1, n):
lam = a[i, k] / a[k, k]
a[i, k:n] = a[i, k:n] - (lam * a[k, k:n])
b[i] = b[i] - lam * b[k]
print(a)
return b
With this code, considering the following arrays:
a = np.array([[4, -2, 1], [-2, 4, -2], [1, -2, 4]])
b = np.array([11, -16, 17])
The result will be:
array([ 11, -10, 10])
This is how the algorithm changed the array a:
array([[ 4, -2, 1],
[ 0, 3, -1],
[ 0, 0, 2]])
Which is wrong. For some reasong, the value in the second row and third column is -1, when it should -1.5. I've inserted some printing in the way to see what was actually happening, and for some reason numpy is truncating the result. This is the changed code:
def gauss_elimination(a, b):
n = len(b)
print(a)
for k in range(0, n-1):
for i in range(k+1, n):
lam = a[i, k] / a[k, k]
print(a[i, k:n])
print(lam * a[k, k:n])
print(a[i, k:n] - lam * a[k, k:n])
a[i, k:n] = a[i, k:n] - (lam * a[k, k:n])
b[i] = b[i] - lam * b[k]
print(a)
return b
And considering the same arrays that were defined a while back, the results will be:
[[ 4 -2 1]
[-2 4 -2]
[ 1 -2 4]]
[ 0. 3. -1.5] # This shows that the value is being calculated as presumed
[[ 4 -2 1]
[ 0 3 -1] # But when the object a is updated, the value is -1 and not -1.5
[ 1 -2 4]]
[ 0. -1.5 3.75]
[[ 4 -2 1]
[ 0 3 -1]
[ 0 -1 3]]
[0. 2.6666666666666665]
[[ 4 -2 1]
[ 0 3 -1]
[ 0 0 2]]
I am little bit confused. Perhaps I might have made a mistake, but the printing shows that everything is being calculated as it should be. Any tips?
The problem
Your array's dtype is "int32":
>>> import numpy as np
>>> x = np.array([1, 2, 3])
>>> x[1] / x[2]
0.6666666666666666
>>> x[0] = x[1] / x[2]
>>> x
array([0, 2, 3])
>>> x.dtype
>>> dtype('int32')
The "solution"
You can use dtype="float64" at instantiation to fix this issue:
>>> x = np.array([1, 2, 3], dtype="float64")
>>> x[0] = x[1] / x[2]
>>> x
array([0.66666667, 2. , 3. ])
Of course, you could also do this without having to explicitly specify the dtype by instantiating your array with floats in the first place:
>>> x = np.array([1., 2., 3.])
>>> x.dtype
dtype('float64')
However, whenever dealing with numerical methods in programming, you should be cognizant of the limitations of floating point arithmetic. Take a look at the note at the bottom of this answer for more details.
Explanation
NumPy arrays are homogeneous arrays, meaning that every element is of the same type. The dtype or "data type" determines how much memory each element in an array requires. This in turn dictates the memory footprint of an entire array in memory.
Because NumPy arrays are homogenenous, when an array is created it is stored in a contiguous chunk of memory which makes accessing elements very vast and easy since all elements take up the same number of bytes. This is part of the reason NumPy is so fast.
Just FYI, this is a very simplified explanation of what's going on here. Essentially, NumPy arrays are arrays of a single type ("int8", or "int32", or "float64", etc).
An operation like division on integers always results in a float, and since NumPy doesn't want to assume that you wanted to change the dtype of the entire array (which would require creating an entire new array in memory), it quietly maintains the dtype of the existing array.
Note
Anytime you're dealing with numerical computations, you need to be aware of the limitations of floating point arithmetic. Floating point arithmetic is prone to inaccuracy due to the nature of how these numbers are stored in memory.
In the case of solving systems of equations using linear algebra, even after solving for a particular vector x in Ax = b using a straight forward Gaussian elimination algorithm based on floating point arithmetic, the expression Ax will not necessarily be exactly equal to b. I recommend reading further: https://en.wikipedia.org/wiki/Floating-point_arithmetic
You've stumbled upon a discipline known as numerical analysis, by the way! More specifically numerical linear algebra. This is a deep topic. Gaussian elimination is a lovely algorithm, but I also hope this answer galvanizes future readers to look into more advanced algorithms for solving systems of equations.
Related
I found my program failed to overwrite an np.ndarray (the X variable) in the for loop by assignment statement like "X[i] = another np.ndarray with matched shape". I have no idea how this could happen...
Codes:
import numpy as np
def qr_tridiagonal(T: np.ndarray):
m, n = T.shape
X = T.copy()
Qt = np.identity(m)
for i in range(n-1):
ai = X[i, i]
ak = X[i+1, i]
c = ai/(ai**2 + ak**2)**.5
s = ak/(ai**2 + ak**2)**.5
# Givens rotation
tmp1 = c*X[i] + s*X[i+1]
tmp2 = c*X[i+1] - s*X[i]
print("tmp1 before:", tmp1)
print("X[i] before:", X[i])
X[i] = tmp1
X[i+1] = tmp2
print("tmp1 after:", tmp1)
print("X[i] after:", X[i])
print()
print(X)
return Qt.T, X
A = np.array([[1, 1, 0, 0], [1, 1, 1, 0], [0, 1, 1, 1], [0, 0, 1, 1]])
Q, R = qr_tridiagonal(A)
Output (the first 4 lines):
tmp1 before: [1.41421356 1.41421356 0.70710678 0. ]
X[i] before: [1 1 0 0]
tmp1 after: [1.41421356 1.41421356 0.70710678 0. ]
X[i] after: [1 1 0 0]
Though X[i] is assigned by tmp1, the values in the array X[i] or X[i, :] remain unchanged. Hope somebody help me out....
Other info: the above is a function to compute QR factorization for tridiagonal matrices using Givens Rotation.
I did check that assigning constant values to X[i] work, e.g. X[i] = 10 then the printed results fit this statement. But if X[i] = someArray then in my codes it would fail. I am not sure whether this is a particular issue triggered by the algorithm I was implementing in the above codes, because such scenarios never happen before.
I did try to install new environments using conda to make sure that my python is not problematic. The above strange outputs should be able to re-generate on other devices.
Many thanks to #hpaulj
It turns out to be a problem of datatype. The program is ok but the input datatype is int, which results in intermediate trancation errors.
A lesson learned: be aware of the dtype of np.ndarray!
I am writing a jury-rigged PyTorch version of scipy.linalg.toeplitz, which currently has the following form:
def toeplitz_torch(c, r=None):
c = torch.tensor(c).ravel()
if r is None:
r = torch.conj(c)
else:
r = torch.tensor(r).ravel()
# Flip c left to right.
idx = [i for i in range(c.size(0)-1, -1, -1)]
idx = torch.LongTensor(idx)
c = c.index_select(0, idx)
vals = torch.cat((c, r[1:]))
out_shp = len(c), len(r)
n = vals.stride(0)
return torch.as_strided(vals[len(c)-1:], size=out_shp, stride=(-n, n)).copy()
But torch.as_strided currently does not support negative strides. My function, therefore, throws the error:
RuntimeError: as_strided: Negative strides are not supported at the moment, got strides: [-1, 1].
My (perhaps incorrect) understanding of as_strided is that it inserts the values of the first argument into a new array whose size is specified by the second argument and it does so by linearly indexing those values in the original array and placing them at subscript-indexed strides given by the final argument.
Both the NumPy and PyTorch documentation concerning as_strided have scary warnings about using the function with "extreme care" and I don't understand this function fully, so I'd like to ask:
Is my understanding of as_strided correct?
Is there a simple way to rewrite this so negative strides work?
Will I be able to pass a gradient w.r.t c (or r) through toeplitz_torch?
> 1. Is my understanding of as_strided correct?
The stride is an interface for your tensor to access the underlying contiguous data buffer. It does not insert values, no copies of the values are done by torch.as_strided, the strides define the artificial layout of what we refer to as multi-dimensional array (in NumPy) or tensor (in PyTorch).
As Andreas K. puts it in another answer:
Strides are the number of bytes to jump over in the memory in order to get from one item to the next item along each direction/dimension of the array. In other words, it's the byte-separation between consecutive items for each dimension.
Please feel free to read the answers over there if you have some trouble with strides. Here we will take your example and look at how it is implemented with as_strided.
The example given by Scipy for linalg.toeplitz is the following:
>>> toeplitz([1,2,3], [1,4,5,6])
array([[1, 4, 5, 6],
[2, 1, 4, 5],
[3, 2, 1, 4]])
To do so they first construct the list of values (what we can refer to as the underlying values, not actually underlying data): vals which is constructed as [3 2 1 4 5 6], i.e. the Toeplitz column and row flattened.
Now notice the arguments passed to np.lib.stride_tricks.as_strided:
values: vals[len(c)-1:] notice the slice: the tensors show up smaller, yet the underlying values remain, and they correspond to those of vals. Go ahead and compare the two with storage_offset: it's just an offset of 2, the values are still there! How this works is that it essentially shifts the indices such that index=0 will refer to value 1, index=1 to 4, etc...
shape: given by the column/row inputs, here (3, 4). This is the shape of the resulting object.
strides: this is the most important piece: (-n, n), in this case (-1, 1)
The most intuitive thing to do with strides is to describe a mapping between the multi-dimensional space: (i, j) ∈ [0,3[ x [0,4[ and the flattened 1D space: k ∈ [0, 3*4[. Since the strides are equal to (-n, n) = (-1, 1), the mapping is -n*i + n*j = -1*i + 1*j = j-i. Mathematically you can describe your matrix as M[i, j] = F[j-i] where F is the flattened values vector [3 2 1 4 5 6].
For instance, let's try with i=1 and j=2. If you look at the Topleitz matrix above M[1, 2] = 4. Indeed F[k] = F[j-i] = F[1] = 4
If you look closely you will see the trick behind negative strides: they allow you to 'reference' to negative indices: for instance, if you take j=0 and i=2, then you see k=-2. Remember how vals was given with an offset of 2 by slicing vals[len(c)-1:]. If you look at its own underlying data storage it's still [3 2 1 4 5 6], but has an offset. The mapping for vals (in this case i: 1D -> k: 1D) would be M'[i] = F'[k] = F'[i+2] because of the offset. This means M'[-2] = F'[0] = 3.
In the above I defined M' as vals[len(c)-1:] which basically equivalent to the following tensor:
>>> torch.as_strided(vals, size=(len(vals)-2,), stride=(1,), storage_offset=2)
tensor([1, 4, 5, 6])
Similarly, I defined F' as the flattened vector of underlying values: [3 2 1 4 5 6].
The usage of strides is indeed a very clever way to define a Toeplitz matrix!
> 2. Is there a simple way to rewrite this so negative strides work?
The issue is, negative strides are not implemented in PyTorch... I don't believe there is a way around it with torch.as_strided, otherwise it would be rather easy to extend the current implementation and provide support for that feature.
There are however alternative ways to solve the problem. It is entirely possible to construct a Toeplitz matrix in PyTorch, but that won't be with torch.as_strided.
We will do the mapping ourselves: for each element of M indexed by (i, j), we will find out the corresponding index k which is simply j-i. This can be done with ease, first by gathering all (i, j) pairs from M:
>>> i, j = torch.ones(3, 4).nonzero().T
(tensor([0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2]),
tensor([0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]))
Now we essentially have k:
>>> j-i
tensor([ 0, 1, 2, 3, -1, 0, 1, 2, -2, -1, 0, 1])
We just need to construct a flattened tensor of all possible values from the row r and column c inputs. Negative indexed values (the content of c) are put last and flipped:
>>> values = torch.cat((r, c[1:].flip(0)))
tensor([1, 4, 5, 6, 3, 2])
Finally index values with k and reshape:
>>> values[j-i].reshape(3, 4)
tensor([[1, 4, 5, 6],
[2, 1, 4, 5],
[3, 2, 1, 4]])
To sum it up, my proposed implementation would be:
def toeplitz(c, r):
vals = torch.cat((r, c[1:].flip(0)))
shape = len(c), len(r)
i, j = torch.ones(*shape).nonzero().T
return vals[j-i].reshape(*shape)
> 3. Will I be able to pass a gradient w.r.t c (or r) through toeplitz_torch?
That's an interesting question because torch.as_strided doesn't have a backward function implemented. This means you wouldn't have been able to backpropagate to c and r! With the above method, however, which uses 'backward-compatible' builtins, the backward pass comes free of charge.
Notice the grad_fn on the output:
>>> toeplitz(torch.tensor([1.,2.,3.], requires_grad=True),
torch.tensor([1.,4.,5.,6.], requires_grad=True))
tensor([[1., 4., 5., 6.],
[2., 1., 4., 5.],
[3., 2., 1., 4.]], grad_fn=<ViewBackward>)
This was a quick draft (that did take a little while to write down), I will make some edits. If you have some questions or remarks, don't hesitate to comment! I would be interested in seeing other answers as I am not an expert with strides, this is just my take on the problem.
The question seems dummy, but I cannot get it right. The output cm1 is expected to be floats, but I only get zeros and ones.
import numpy as np
import scipy.spatial.distance
sim = scipy.spatial.distance.cosine
a = [2, 3, 1]
b = [3, 1, 2]
c = [1, 2, 6]
cm0 = np.array([a,b,c])
ca, cb, cc = 0.9, 0.7, 0.4
cr = np.array([ca, cb, cc])
cm1 = np.empty_like(cm0)
for i in range(3):
for j in range(3):
cm1[i,j] = cm0[i,j] * cr[i] * cr[j]
print(cm1)
And I get:
[[1 1 0]
[1 0 0]
[0 0 0]]
empty_like() matches the type of the given numpy array by default, as hpaulj suggested in the comments. In your case cm0 is of type integer.
The empty_like function accepts multiple arguments though, one of wich is dtype. Setting dtype to float should solve the problem:
cm1 = np.empty_like(cm0, dtype=float)
And also Python truncates floating point numbers at the decimal point when converting to integers. In your case, every multiplication done results in a number between 1.89 and 0.36, so flooring the results will result in 0s and 1s respectively.
As #hpaulj said in the comments section, the problem is using empty_like which will keep the cm0 dtype, to solve it try:
cm1 = np.empty_like(cm0, dtype=float)
I am looking into the NumPy correlation function
numpy.correlate(a, v, mode='valid')[source]
Cross-correlation of two 1-dimensional sequences.
This function computes the correlation as generally defined in signal processing texts:
c_{av}[k] = sum_n a[n+k] * conj(v[n])
Then for the example:
a = [1, 2, 3]
v = [0, 1, 0.5]
np.correlate([1, 2, 3], [0, 1, 0.5], "full")
array([ 0.5, 2. , 3.5, 3. , 0. ])
So the k in the output array is from 0 to 4 in this example. However, I am wondering how does a[n+k] is defined when (n+k) > 2 in this case?
Also, how is conjugate(v(n)) defined and how is each element in array computed?
The formula c_{av}[k] = sum_n a[n+k] * conj(v[n]) is a little misleading because k on the left is not necessarily the Python index of the output array. In the 'full' mode, the possible values of k are those for which there exists at least one n such that a[n+k] * conj(v[n]) is defined (that is, both n+k and n fall in the ranges of respective arrays).
In your examples, k in sum_n a[n+k] * conj(v[n]) can be -2, -1, 0, 1, 2. These generate 5 values that you see. For example, k being -2 results in a[2-2]*conj(v[2]) which is 0.5, and so on.
In general, the range of k in the 'full' mode is from 1-len(a) to len(v)-1 inclusive. So, if k is really understood as Python index, then the formula should be
c_{av}[k] = sum_n a[n+k+len(a)-1] * conj(v[n])
I'm almost exactly in a similar situation as the asker here over a year ago:
fast way to invert or dot kxnxn matrix
So I have a tensor with indices a[n,i,j] of dimensions (N,M,M) and I want to invert the M*M square matrix part for each n in N.
For example, suppose I have
In [1]: a = np.arange(12)
a.shape = (3,2,2)
a
Out[1]: array([[[ 0, 1],
[ 2, 3]],
[[ 4, 5],
[ 6, 7]],
[[ 8, 9],
[10, 11]]])
Then a for loop inversion would go like this:
In [2]: inv_a = np.zeros([3,2,2])
for m in xrange(0,3):
inv_a[m] = np.linalg.inv(a[m])
inv_a
Out[2]: array([[[-1.5, 0.5],
[ 1. , 0. ]],
[[-3.5, 2.5],
[ 3. , -2. ]],
[[-5.5, 4.5],
[ 5. , -4. ]]])
This will apparently be implemented in NumPy 2.0, according to this issue on github...
I guess I need to install the dev version as seberg noted in the github issue thread, but is there another way to do this in vectorized manner right now?
Update:
In NumPy 1.8 and later, the functions in numpy.linalg are generalized universal functions.
Meaning that you can now do something like this:
import numpy as np
a = np.random.rand(12, 3, 3)
np.linalg.inv(a)
This will invert each 3x3 array and return the result as a 12x3x3 array.
See the numpy 1.8 release notes.
Original Answer:
Since N is relatively small, how about we compute the LU decomposition manually for all the matrices at once.
This ensures that the for loops involved are relatively short.
Here's how this can be done with normal NumPy syntax:
import numpy as np
from numpy.random import rand
def pylu3d(A):
N = A.shape[1]
for j in xrange(N-1):
for i in xrange(j+1,N):
#change to L
A[:,i,j] /= A[:,j,j]
#change to U
A[:,i,j+1:] -= A[:,i,j:j+1] * A[:,j,j+1:]
def pylusolve(A, B):
N = A.shape[1]
for j in xrange(N-1):
for i in xrange(j+1,N):
B[:,i] -= A[:,i,j] * B[:,j]
for j in xrange(N-1,-1,-1):
B[:,j] /= A[:,j,j]
for i in xrange(j):
B[:,i] -= A[:,i,j] * B[:,j]
#usage
A = rand(1000000,3,3)
b = rand(3)
b = np.tile(b,(1000000,1))
pylu3d(A)
# A has been replaced with the LU decompositions
pylusolve(A, b)
# b has been replaced to the solutions of
# A[i] x = b[i] for each A[i] and b[i]
As I have written it, pylu3d modifies A in place to compute the LU decomposition.
After replacing each NxN matrix with its LU decomposition, pylusolve can be used to solve an MxN array b representing the right hand sides of your matrix systems.
It modifies b in place and does the proper back substitutions to solve the system.
As it is written, this implementation does not include pivoting, so it isn't numerically stable, but it should work well enough in most cases.
Depending on how your array is arranged in memory, it is probably still a good bit faster to use Cython.
Here are two Cython functions that do the same thing, but they iterate along M first.
It's not vectorized, but it is relatively fast.
from numpy cimport ndarray as ar
cimport cython
#cython.boundscheck(False)
#cython.wraparound(False)
def lu3d(ar[double,ndim=3] A):
cdef int n, i, j, k, N=A.shape[0], h=A.shape[1], w=A.shape[2]
for n in xrange(N):
for j in xrange(h-1):
for i in xrange(j+1,h):
#change to L
A[n,i,j] /= A[n,j,j]
#change to U
for k in xrange(j+1,w):
A[n,i,k] -= A[n,i,j] * A[n,j,k]
#cython.boundscheck(False)
#cython.wraparound(False)
def lusolve(ar[double,ndim=3] A, ar[double,ndim=2] b):
cdef int n, i, j, N=A.shape[0], h=A.shape[1]
for n in xrange(N):
for j in xrange(h-1):
for i in xrange(j+1,h):
b[n,i] -= A[n,i,j] * b[n,j]
for j in xrange(h-1,-1,-1):
b[n,j] /= A[n,j,j]
for i in xrange(j):
b[n,i] -= A[n,i,j] * b[n,j]
You could also try using Numba, though I couldn't get it to run as fast as Cython in this case.