I am trying to use broadcasting to speed up my numpy code. the real code has much larger arrays and loops through multiple times, but I think this snippet illustrates the issue.
import numpy as np
row = np.array([0,0,1,1,4])
dl_ddk = np.array([0,8,29,112,11])
change1 = np.zeros(5)
change2 = np.zeros(5)
for k in range(0, row.shape[0]):
i = row[k]
change1[i] += dl_ddk[k]
change2[row] += dl_ddk
print(change1)
print(change2)
change1 = [8, 141, 0, 0 11]
change2 = [8, 112, 0, 0 11]
I thought these two change arrays would be equals however, it seems that the broadcast operations += is overwriting rather than adding values. Is there a way to vectorize a loop in np with matrix referencing like this that will give the same results as change1?
You can use np.bincount() and use dl_ddk as the weights:
import numpy as np
row = np.array([0,0,1,1,4])
dl_ddk = np.array([0,8,29,112,11])
change1 = np.bincount(row, weights=dl_ddk)
print(change1)
# [ 8. 141. 0. 0. 11.]
The bit in the docs show using it in a way almost exactly like your problem:
If weights is specified the input array is weighted by it, i.e. if a
value n is found at position i, out[n] += weight[i] instead of out[n]
+= 1.
In [1]: row = np.array([0,0,1,1,4])
...: dl_ddk = np.array([0,8,29,112,11])
...: change1 = np.zeros(5)
...: change2 = np.zeros(5)
...: for k in range(0, row.shape[0]):
...: i = row[k]
...: change1[i] += dl_ddk[k]
...: change2[row] += dl_ddk
change2 does not match because of buffering. ufunc has added a at method to address this:
Performs unbuffered in place operation on operand 'a' for elements specified by 'indices'.
In [3]: change3 = np.zeros(5)
In [4]: np.add.at(change3, row, dl_ddk)
In [5]: change1
Out[5]: array([ 8., 141., 0., 0., 11.])
In [6]: change2
Out[6]: array([ 8., 112., 0., 0., 11.])
In [7]: change3
Out[7]: array([ 8., 141., 0., 0., 11.])
Related
how can I speed up the calculations for this equation using GPU or cuda as the file contains 30.000 points
points = pd.read_csv('file.dat', sep='\t', usecols=[0, 1])
d = pd.DataFrame(np.zeros((max_id, max_id)))
dis = sch.distance.pdist(points, 'euclidean')
n = 0
for i in range(max_id):
print(i)
for j in range(i + 1, max_id):
d.at[i, j] = dis[n]
d.at[j, i] = d.at[i, j]
n += 1
EDIT
i tried
points = genfromtxt(path, delimiter='\t', usecols=[0, 1])
points =torch.tensor(points)
d = pd.DataFrame(np.zeros((max_id, max_id)))
dis = torch.cdist(points)
but got
TypeError: cdist() missing 1 required positional argument: 'x2'
does that mean I need to read points or the two columns of points separately?
NumPy doesn't natively support GPUs. Though you can use some libraries which are friendly with numpy and supports GPU. One of the option like that would be to use PyTorch. torch.cdist would be one of the function you can look at (Then you don't need to organize it like that using for loops). Also there is torch.nn.functional.pdist. Note also that, you don't need to use for loop in second case. Once you get the output, you can just reshape it as per your need.
Sorry, numpy doesn't support GPU, but it's not impossible to speed up your program. It seems that you want to write the calculated distance value to the non diagonal position. I use the pure numpy array to show you how to speed up your program:
>>> max_id = 5
>>> d = np.zeros((max_id, max_id))
>>> dis = np.arange(max_id // 2 * (max_id - 1)) ** 2 # set the value at will
>>> d[np.triu_indices(max_id, k=1)] = dis
>>> d + d.T # d += d.T is better
array([[ 0., 0., 1., 4., 9.],
[ 0., 0., 16., 25., 36.],
[ 1., 16., 0., 49., 64.],
[ 4., 25., 49., 0., 81.],
[ 9., 36., 64., 81., 0.]])
After testing, when the max_id is 500, the accelerated program is more than ten times faster than the previous one:
>>> def loop(max_id):
... d = np.zeros((max_id, max_id))
... dis = np.arange(max_id // 2 * (max_id - 1)) ** 2
... n = 0
... for i in range(max_id):
... for j in range(i + 1, max_id):
... d[i, j] = d[j, i] = dis[n]
... n += 1
...
>>> def triu(max_id):
... d = np.zeros((max_id, max_id))
... dis = np.arange(max_id // 2 * (max_id - 1)) ** 2
... d[np.triu_indices(max_id, k=1)] = dis
... d += d.T
...
>>> timeit(lambda: loop(500), number=10)
0.546407600006205
>>> timeit(lambda: triu(500), number=10)
0.0350386000063736
If you want to do more complex loop operations, you can learn about numba.
Update:
It seems that the size of non diagonal block is larger than size of dis, you just need to slice the result of triu_indices:
dis_size = dis.size
i, j = np.triu_indices(max_id, k=1)
d[i[:dis_size], j[:dis_size]] = dis
I'm using a video processing tool that needs to input the processing data from each frame into an array.
for p in det.read(frame, fac):
point_values = np.array([])
for j, (x, y) in enumerate(p): #iteration through points
point_values = np.append(point_values,y)
point_values = np.append(point_values,x)
this code runs again each frame. I'm expecting "point_values = np.array([])" to reset the array and then start filling it again.
I'm not sure if my logic is wrong or is it a syntax issue.
Your code does:
In [77]: p = [(0,0),(0,2),(1,0),(1,2)]
In [78]: arr = np.array([])
In [79]: for j,(x,y) in enumerate(p):
...: arr = np.append(arr,y)
...: arr = np.append(arr,x)
...:
In [80]: arr
Out[80]: array([0., 0., 2., 0., 0., 1., 2., 1.])
No syntax error. The list equivalent is faster and cleaner:
In [85]: alist =[]
In [86]: for x,y in p: alist.extend((y,x))
In [87]: alist
Out[87]: [0, 0, 2, 0, 0, 1, 2, 1]
But you don't give any indication of how this action is supposed to fit within a larger context. You create a new point_values for each p, but then don't do anything with it.
This is one of the first things I try to code in python (and any programming language) and my first question here, so I hope I provide everything neccessary to help me.
I have upper triangular matrix and I need to solve system of equations Wx=y, where W (3x3 matrix) and y (vector) are given. I cannot use numpy.linalg functions, so I try to implement this, but backwards of course.
After several failed attempts, I limited my task to 3x3 matrix. Without loop, code looks like this:
x[0,2]=y[2]/W[2,2]
x[0,1]=(y[1]-W[1,2]*x[0,2])/W[1,1]
x[0,0]=(y[0]-W[0,2]*x[0,2]-W[0,1]*x[0,1])/W[0,0]
Now, every new sum contains more elements, which are schematic, but nevertheless need to be defined somehow. I suppose there must be sum function in numpy, but not linalg, which does such things, but I cannot find it.
My newest, partial "attempt" begins with something like this:
n=3
for k in range(n):
for i in range(n-k-1):
x[0,n-k-1]=y[n-k-1]/W[n-k-1,n-k-1]
Which, of course, contains only first element of each sum.
I would be thankful for any assistance.
Example I am working on:
y=np.array([ 0.80064077, 2.64300842, -0.74912957])
W=np.array([[6.244998,2.88230677,-5.44435723],[0.,2.94827198,2.26990852],[0.,0.,0.45441135]]
n=W.shape[1]
x=np.zeros((1,n), dtype=np.float)
Proper solution should look like:
[-2.30857143 2.16571429 -1.64857143]
Here's one approach to use generic n and with one-loop -
def one_loop(y, W, n):
out = np.zeros((1,n))
for i in range(n-1,-1,-1):
sums = (W[i,i+1:]*out[0,i+1:]).sum()
out[0,i] = (y[i] - sums)/W[i,i]
return out
For performance, we can replace that sum-reduction step with a dot-product. Thus, sums could be alternatively computed like so -
sums = W[i,i+1:].dot(x[0,i+1:])
Sample runs
1) n = 3 :
In [149]: y
Out[149]: array([ 5., 8., 7.])
In [150]: W
Out[150]:
array([[ 6., 6., 2.],
[ 3., 3., 3.],
[ 4., 8., 5.]])
In [151]: x = np.zeros((1,3))
...: x[0,2]=y[2]/W[2,2]
...: x[0,1]=(y[1]-W[1,2]*x[0,2])/W[1,1]
...: x[0,0]=(y[0]-W[0,2]*x[0,2]-W[0,1]*x[0,1])/W[0,0]
...:
In [152]: x
Out[152]: array([[-0.9 , 1.26666667, 1.4 ]])
In [154]: one_loop(y, W, n=3)
Out[154]: array([[-0.9 , 1.26666667, 1.4 ]])
2) n = 4 :
In [156]: y
Out[156]: array([ 5., 8., 7., 6.])
In [157]: W
Out[157]:
array([[ 6., 2., 3., 3.],
[ 3., 4., 8., 5.],
[ 8., 6., 6., 4.],
[ 8., 4., 2., 2.]])
In [158]: x = np.zeros((1,4))
...: x[0,3]=y[3]/W[3,3]
...: x[0,2]=(y[2]-W[2,3]*x[0,3])/W[2,2]
...: x[0,1]=(y[1]-W[1,3]*x[0,3]-W[1,2]*x[0,2])/W[1,1]
...: x[0,0]=(y[0]-W[0,3]*x[0,3]-W[0,2]*x[0,2]-W[0,1]*x[0,1])/W[0,0]
...:
In [159]: x
Out[159]: array([[-0.22222222, -0.08333333, -0.83333333, 3. ]])
In [160]: one_loop(y, W, n=4)
Out[160]: array([[-0.22222222, -0.08333333, -0.83333333, 3. ]])
One more take (now updated to the state-of-the-art provided by Divakar in another answer):
import numpy as np
y=np.array([ 0.80064077, 2.64300842, -0.74912957])
W=np.array([[6.244998,2.88230677,-5.44435723],[0.,2.94827198,2.26990852],[0.,0.,0.45441135]])
n=W.shape[1]
x=np.zeros((1,n), dtype=np.float)
for i in range(n-1, -1, -1):
x[0,i] = (y[i]-W[i,i+1:].dot(x[0,i+1:]))/W[i,i]
print(x)
gives:
[[-2.30857143 2.16571429 -1.64857143]]
My take
n=3
for k in range(n):
print("s=y[%d]"% (n-k-1))
s = y[n-k-1]
for i in range(0,k):
print("s - W[%d,%d]*x[0,%d]" % (n-k-1, n-i-1, n-i-1))
s = s - W[n-k-1,n-i-1]*x[0,n-i-1]
print("x[0,%d] = s/W[%d,%d]" % (n-k-1,n-k-1,n-k-1))
x[0,n-k-1] = s/W[n-k-1,n-k-1]
print(x)
and without print statements
n=3
for k in range(n):
s = y[n-k-1]
for i in range(0,k):
s = s - W[n-k-1,n-i-1]*x[0,n-i-1]
x[0,n-k-1] = s/W[n-k-1,n-k-1]
print(x)
Output
s=y[2]
x[0,2] = s/W[2,2]
s=y[1]
s - W[1,2]*x[0,2]
x[0,1] = s/W[1,1]
s=y[0]
s - W[0,2]*x[0,2]
s - W[0,1]*x[0,1]
x[0,0] = s/W[0,0]
[[-2.30857143 2.16571429 -1.64857143]]
I need obtain a "W" matrix of multiples matrix multiplications (all multiplications result in column vectors).
from numpy import matrix
from numpy import transpose
from numpy import matmul
from numpy import dot
# Iterative matrix multiplication
def iterativeMultiplication(X, Y):
W = [] # Matrix of matricial products
X = matrix(X) # same number of rows
Y = matrix(Y) # same number of rows
h = 0
while (h < X.shape[1]):
W.append([])
W[h] = dot(transpose(X), Y) # using "dot" function
h += 1
return W
But, unexpectedly, I obtain a list of objects with their respective data types.
X = [[0., 0., 1.], [1.,0.,0.], [2.,2.,2.], [2.,5.,4.]]
Y = [[-0.2], [1.1], [5.9], [12.3]] # Edit Y column
iterativeMultiplication( X, Y )
Results in:
[array([[37.5],[73.3],[60.8]]),
array([[37.5],[73.3],[60.8]]),
array([[37.5],[73.3],[60.8]])]
I need any method for obtain only the numerical values for the matrix conversion.
W = matrix(W) # Results in error
It is the same using "matmul" function. Thx for your time.
If you want to stack multiple matrices, you can use numpy.vstack:
W = numpy.vstack(W)
Edit: There seems to be a discrepancy between your function, X and Y versus the "result" list in your question. But based on your comments below, what you're actually looking for is numpy.hstack (horizontal stack) which will give you the desired 3x3 matrix based on your "result" list.
W = numpy.hstack(W)
Of course you are going to get a list. You initial W as a list, and append the same calculation to it 3 times.
But your 3 element arrays don't make sense with this data, array([[ 3.36877336],[ 3.97112615],[ 3.8092797 ]]).
If I make Xm=np.matrix(X), etc:
In [162]: Xm
Out[162]:
matrix([[ 0., 0., 1.],
[ 1., 0., 0.],
[ 2., 2., 2.],
[ 2., 5., 4.]])
In [163]: Ym
Out[163]:
matrix([[ 0.1, -0.2],
[ 0.9, 1.1],
[ 6.2, 5.9],
[ 11.9, 12.3]])
In [164]: Xm.T.dot(Ym)
Out[164]:
matrix([[ 37.1, 37.5],
[ 71.9, 73.3],
[ 60.1, 60.8]])
In [165]: Xm.T*Ym # matrix interprets * as .dot
Out[165]:
matrix([[ 37.1, 37.5],
[ 71.9, 73.3],
[ 60.1, 60.8]])
You need to edit the question, to have both valid Python code (missing def and :), and results that match the inputs.
===============
In [173]: Y = [[-0.2], [1.1], [5.9], [12.3]]
In [174]: Ym=np.matrix(Y)
Out[176]:
matrix([[ 37.5],
[ 73.3],
[ 60.8]])
=====================
This iteration is clumsy:
h = 0
while (h < X.shape[1]):
W.append([])
W[h] = dot(transpose(X), Y) # using "dot" function
h += 1
A more Pythonic approach
for h in range(X.shape[1]):
W.append(np.dot(...))
Or even
W = [np.dot(....) for h in range(X.shape[1])]
I have a list which consists of several numpy arrays with different shapes.
I want to reshape this list of arrays into a numpy vector and then change each element in the vector and then reshape it back to the original list of arrays.
For example:
input
[numpy.zeros((2,2)), numpy.ones((3,3))]
First
To vector
[0,0,0,0,1,1,1,1,1,1,1,1,1]
Second
every time change only one element. for example change the 1st element 0 to 2
[0,2,0,0,1,1,1,1,1,1,1,1,1]
Last
convert it back to
[array([[0,2],[0,0]]),array([[1,1,1],[1,1,1],[1,1,1]])]
Is there any fast implementation? Thanks very much.
It seems like converting to a list and back will be inefficient. Instead, why not figure out which array to index (and where) and then just update that index? e.g.
def change_element(arr1, arr2, ix, value):
which = ix >= arr1.size
arr = [arr1, arr2][which]
ix = ix - arr1.size if which else ix
arr.ravel()[ix] = value
And here's some example usage:
>>> arr1 = np.zeros((2, 2))
>>> arr2 = np.ones((3, 3))
>>> change_element(arr1, arr2, 1, 2)
>>> change_element(arr1, arr2, 6, 3.14)
>>> arr1
array([[ 0., 2.],
[ 0., 0.]])
>>> arr2
array([[ 1. , 1. , 3.14],
[ 1. , 1. , 1. ],
[ 1. , 1. , 1. ]])
>>> change_element(arr1, arr2, 7, 3.14)
>>> arr1
array([[ 0., 2.],
[ 0., 0.]])
>>> arr2
array([[ 1. , 1. , 3.14],
[ 3.14, 1. , 1. ],
[ 1. , 1. , 1. ]])
A few notes -- This updates the arrays in place. It doesn't create new arrays. If you really need to create new arrays, I suppose you could np.copy them and return. Also, this relies on the arrays sharing memory before and after the ravel. I don't remember the exact circumstances where ravel would return a new array rather than a view into the original array . . .
Generalizing to more arrays is actually quite easy. We just need to walk down the list of arrays and see if ix is less than the array size. If it is, we've found our array. If it isn't, we need to subtract the array's size from ix to represent the number of elements we've traversed thus far:
def change_element(arrays, ix, value):
for arr in arrays:
if ix < arr.size:
arr.ravel()[ix] = value
return
ix -= arr.size
And you can call this similar to before:
change_element([arr1, arr2], 6, 3.14159)
#mgilson probably has the best answer for you, but if you absolutely have to convert to a flat list first and then go back again (perhaps because you need to do something else with the flat list as well), then you can do this with list comprehensions:
lst = [numpy.zeros((2,4)), numpy.ones((3,3))]
tlist = [e for a in lst for e in a.ravel()]
tlist[1] = 2
i = 0
lst2 = []
dims = [a.shape for a in lst]
for n, m in dims:
lst2.append(np.array(tlist[i:i+n*m]).reshape(n,m))
i += n*m
lst2
[array([[ 0., 2.],
[ 0., 0.]]), array([[ 1., 1., 1.],
[ 1., 1., 1.],
[ 1., 1., 1.]])]
Of course, you lose the information about your array sizes when you flatten, so you need to store them somewhere (here, in dims).