Working in Python, I am doing some physics calculations over an NxM grid of values, where N goes from 1 to 3108 and M goes from 1 to 2304 (this corresponds to a large image). I need calculate a value at each and every point in this space, which totals ~ 7 million calculations. My current approach is painfully slow, and I am wondering if there is a way to complete this task and it not take hours...
My first approach was just to use nested for loops, but this seemed like the least efficient way to solve my problem. I have tried using NumPy's nditer and iterating over each axis individually, but I've read that it doesn't actually speed up my computations. Rather than looping through each axis individually, I also tried making a 3-D array and looping through the outer axis as shown in Brian's answer here How can I, in python, iterate over multiple 2d lists at once, cleanly? . Here is the current state of my code:
import numpy as np
x,y = np.linspace(1,3108,num=3108),np.linspace(1,2304,num=2304) # x&y dimensions of image
X,Y = np.meshgrid(x,y,indexing='ij')
all_coords = np.dstack((X,Y)) # moves to 3-D
all_coords = all_coords.astype(int) # sets coords to int
For reference, all_coords looks like this:
array([[[1.000e+00, 1.000e+00],
[1.000e+00, 2.000e+00],
[1.000e+00, 3.000e+00],
...,
[1.000e+00, 2.302e+03],
[1.000e+00, 2.303e+03],
[1.000e+00, 2.304e+03]],
[[2.000e+00, 1.000e+00],
[2.000e+00, 2.000e+00],
[2.000e+00, 3.000e+00],
...,
[2.000e+00, 2.302e+03],
[2.000e+00, 2.303e+03],
[2.000e+00, 2.304e+03]],
and so on. Back to my code...
'''
- below is a function that does a calculation on the full grid using the distance between x0,y0 and each point on the grid.
- the function takes x0,y0 and returns the calculated values across the grid
'''
def do_calc(x0,y0):
del_x, del_y = X-x0, Y-y0
np.seterr(divide='ignore', invalid='ignore')
dmx_ij = (del_x/((del_x**2)+(del_y**2))) # x component
dmy_ij = (del_y/((del_x**2)+(del_y**2))) # y component
return dmx_ij,dmy_ij
# now the actual loop
def do_loop():
dmx,dmy = 0,0
for pair in all_coords:
for xi,yi in pair:
DM = do_calc(xi,yi)
dmx,dmy = dmx+DM[0],dmy+DM[1]
return dmx,dmy
As you might see, this code takes an incredibly long time to run... If there is any way to modify my code such that it doesn't take hours to complete, I would be extremely interested in knowing how to do that. Thanks in advance for the help.
Here is a method that gives a 10,000x speedup at N=310, M=230. As the method scales better than the original code I'd expect a factor of more than a million at the full problem size.
The method exploits the shift invariance of the problem. For example, del_x**2 is essentially the same up to shift at each call of do_calc, so we compute it only once.
If the output of do_calc is weighted before summation the problem is no longer fully translation invariant, and this method doesn't work anymore. The result, however, can then be expressed in terms of linear convolution. At N=310, M=230 this still leaves us with a more than 1,000x speedup. And, again, this will be more at full problem size
Code for original problem
import numpy as np
#N, M = 3108, 2304
N, M = 310, 230
### OP's code
x,y = np.linspace(1,N,num=N),np.linspace(1,M,num=M) # x&y dimensions of image
X,Y = np.meshgrid(x,y,indexing='ij')
all_coords = np.dstack((X,Y)) # moves to 3-D
all_coords = all_coords.astype(int) # sets coords to int
'''
- below is a function that does a calculation on the full grid using the distance between x0,y0 and each point on the grid.
- the function takes x0,y0 and returns the calculated values across the grid
'''
def do_calc(x0,y0):
del_x, del_y = X-x0, Y-y0
np.seterr(divide='ignore', invalid='ignore')
dmx_ij = (del_x/((del_x**2)+(del_y**2))) # x component
dmy_ij = (del_y/((del_x**2)+(del_y**2))) # y component
return np.nan_to_num(dmx_ij), np.nan_to_num(dmy_ij)
# now the actual loop
def do_loop():
dmx,dmy = 0,0
for pair in all_coords:
for xi,yi in pair:
DM = do_calc(xi,yi)
dmx,dmy = dmx+DM[0],dmy+DM[1]
return dmx,dmy
from time import time
t = [time()]
### pp's code
x, y = np.ogrid[-N+1:N-1:2j*N - 1j, -M+1:M-1:2j*M - 1J]
den = x*x + y*y
den[N-1, M-1] = 1
xx = x / den
yy = y / den
for zz in xx, yy:
zz[N:] -= zz[:N-1]
zz[:, M:] -= zz[:, :M-1]
XX = xx.cumsum(0)[N-1:].cumsum(1)[:, M-1:]
YY = yy.cumsum(0)[N-1:].cumsum(1)[:, M-1:]
t.append(time())
### call OP's code for reference
X_OP, Y_OP = do_loop()
t.append(time())
# make sure results are equal
assert np.allclose(XX, X_OP)
assert np.allclose(YY, Y_OP)
print('pp {}\nOP {}'.format(*np.diff(t)))
Sample run:
pp 0.015251636505126953
OP 149.1642508506775
Code for weighted problem:
import numpy as np
#N, M = 3108, 2304
N, M = 310, 230
values = np.random.random((N, M))
x,y = np.linspace(1,N,num=N),np.linspace(1,M,num=M) # x&y dimensions of image
X,Y = np.meshgrid(x,y,indexing='ij')
all_coords = np.dstack((X,Y)) # moves to 3-D
all_coords = all_coords.astype(int) # sets coords to int
'''
- below is a function that does a calculation on the full grid using the distance between x0,y0 and each point on the grid.
- the function takes x0,y0 and returns the calculated values across the grid
'''
def do_calc(x0,y0, v):
del_x, del_y = X-x0, Y-y0
np.seterr(divide='ignore', invalid='ignore')
dmx_ij = (del_x/((del_x**2)+(del_y**2))) # x component
dmy_ij = (del_y/((del_x**2)+(del_y**2))) # y component
return v*np.nan_to_num(dmx_ij), v*np.nan_to_num(dmy_ij)
# now the actual loop
def do_loop():
dmx,dmy = 0,0
for pair, vv in zip(all_coords, values):
for (xi,yi), v in zip(pair, vv):
DM = do_calc(xi,yi, v)
dmx,dmy = dmx+DM[0],dmy+DM[1]
return dmx,dmy
from time import time
from scipy import signal
t = [time()]
x, y = np.ogrid[-N+1:N-1:2j*N - 1j, -M+1:M-1:2j*M - 1J]
den = x*x + y*y
den[N-1, M-1] = 1
xx = x / den
yy = y / den
XX, YY = (signal.fftconvolve(zz, values, 'valid') for zz in (xx, yy))
t.append(time())
X_OP, Y_OP = do_loop()
t.append(time())
assert np.allclose(XX, X_OP)
assert np.allclose(YY, Y_OP)
print('pp {}\nOP {}'.format(*np.diff(t)))
Sample run:
pp 0.12683939933776855
OP 158.35225439071655
I'm currently trying to plot a graph of iterations of a certain function in python. I have defined the function as stated below but I am unsure on how to plot the graph such that the y value is on the y axis and the iteration number is on the x axis.
So, I have tried using the plt.plot function with different values in as my x values but using logistic(4, 0.7) as the y value for the y axis.
def logistic(A, x):
y = A * x * (1 - x)
return y
But each return an error. Can anyone shed any light on this, I want to do a total of 1000 iterations.
I dont understand much what you are saying concerning x being number ofiteration while you are showing us function logistic(4, 0.7). As far as I know, iterations is integer, whole number. You cant iterate just halfly or partially
def logistic(A, x):
y = A * x * (1 - x)
return y
A = 1
x_vals = []
y_vals = []
for x in range(1,1000):
x_vals.append(x)
y_vals.append(logistic(A,x))
#plt.plot(x_vals,y_vals) # See every iteration
#plt.show()
plt.plot(x_vals,y_vals) # See all iterations at once
plt.show()
Ah, the logistic map. Are you trying to make a cobweb plot? If so, your error may be elsewhere. As others have mentioned, you should post the error message and your code, so we can better help you. However, based on what you've given us, you can use numpy.arrays to achieve your desired result.
import numpy as np
import matplotlib.pyplot as plt
start = 0
end = 1
num = 1000
# Create array of 'num' evenly spaced values between 'start' and 'end'
x = np.linspace(start, end, num)
# Initialize y array
y = np.zeros(len(x))
# Logistic function
def logistic(A, x):
y = A * x * (1 - x)
return y
# Add values to y array
for i in range(len(x)):
y[i] = logistic(4, x[i])
plt.plot(x,y)
plt.show()
However, with numpy.arrays, you can omit the for loop and just do
x = np.linspace(start, end, num)
y = logistic(4, x)
and you'll get the same result, but faster.
I'm trying to sum a two dimensional function using the array method, somehow, using a for loop is not outputting the correct answer. I want to find (in latex) $$\sum_{i=1}^{M}\sum_{j=1}^{M_2}\cos(i)\cos(j)$$ where according to Mathematica the answer when M=5 is 1.52725. According to the for loop:
def f(N):
s1=0;
for p1 in range(N):
for p2 in range(N):
s1+=np.cos(p1+1)*np.cos(p2+1)
return s1
print(f(4))
is 0.291927.
I have thus been trying to use some code of the form:
def f1(N):
mat3=np.zeros((N,N),np.complex)
for i in range(0,len(mat3)):
for j in range(0,len(mat3)):
mat3[i][j]=np.cos(i+1)*np.cos(j+1)
return sum(mat3)
which again
print(f1(4))
outputs 0.291927. Looking at the array we should find for each value of i and j a matrix of the form
mat3=[[np.cos(1)*np.cos(1),np.cos(2)*np.cos(1),...],[np.cos(2)*np.cos(1),...]...[np.cos(N+1)*np.cos(N+1)]]
so for N=4 we should have
mat3=[[np.cos(1)*np.cos(1) np.cos(2)*np.cos(1) ...] [np.cos(2)*np.cos(1) ...]...[... np.cos(5)*np.cos(5)]]
but what I actually get is the following
mat3=[[0.29192658+0.j 0.+0.j 0.+0.j ... 0.+0.j] ... [... 0.+0.j]]
or a matrix of all zeros apart from the mat3[0][0] element.
Does anybody know a correct way to do this and get the correct answer? I chose this as an example because the problem I'm trying to solve involves plotting a function which has been summed over two indices and the function that python outputs is not the same as Mathematica (i.e., a function of the form $$f(E)=\sum_{i=1}^{M}\sum_{j=1}^{M_2}F(i,j,E)$$).
The return statement is not indented correctly in your sample code. It returns immediately in the first loop iteration. Indent it on the function body instead, so that both for loops finish:
def f(N):
s1=0;
for p1 in range(N):
for p2 in range(N):
s1+=np.cos(p1+1)*np.cos(p2+1)
return s1
>>> print(f(5))
1.527247272700347
I have moved your code to a more numpy-ish version:
import numpy as np
N = 5
x = np.arange(N) + 1
y = np.arange(N) + 1
x = x.reshape((-1, 1))
y = y.reshape((1, -1))
mat = np.cos(x) * np.cos(y)
print(mat.sum()) # 1.5272472727003474
The trick here is to reshape x to a column and y to a row vector. If you multiply them, they are matched up like in your loop.
This should be more performant, since cos() is only called 2*N times. And it avoids loops (bad in python).
UPDATE (regarding your comment):
This pattern can be extended in any dimension. Basically, you get something like a crossproduct. Where every instance of x is matched up with every instance of y, z, u, k, ... Along the corresponding dimensions.
It's a bit confusing to describe, so here is some more code:
import numpy as np
N = 5
x = np.arange(N) + 1
y = np.arange(N) + 1
z = np.arange(N) + 1
x = x.reshape((-1, 1, 1))
y = y.reshape((1, -1, 1))
z = z.reshape((1, 1, -1))
mat = z**2 * np.cos(x) * np.cos(y)
# x along first axis
# y along second, z along third
# mat[0, 0, 0] == 1**2 * np.cos(1) * np.cos(1)
# mat[0, 4, 2] == 3**2 * np.cos(1) * np.cos(5)
If you use this for many dimensions, and big values for N, you will run into memory problems, though.
I have attempted to solve the following problem. I tried to solve it first with a set step size h using 0.1. However I need to change this in my code and use a for loop to loop through the values 0,1,..,20. I am a little confused how to do this problem but I was hoping to get some help with fixing the code I produced so far. Thanks!
import numpy as np
from math import sin
def derivative(func , x, h ):
for h in range(20):
return (func(x+h)-func(x))/h
def f(x):
return sin(x)
print(derivative(f, pi/4))
Gives the output
0.6706029729039897
MY EDIT:
def derivative(func , x, h ):
for h in range(20):
return (func(x+h)-func(x))/h
The exercise is asking you to compute the derivative using varying precision (represented using the variable h), and compare that to the exact/real derivative of the function.
Let h = 10 ^ -j, with j varying from 0 to 20. This means h will go (discretely) from 10⁻⁰ to 10⁻²⁰. You can use a for-loop and the range(...) function for that. Then pass that to the derivative function (to which you can a third parameter for the value of h)
def derivative(func, x, h):
return (func(x + h) - func(x)) / h
Next, you need to compare that to the exact derivative. The function f(x) = sin(x) has a known (exact) derivative which is cos(x). In math notation, d(sin x)/dx = cos x. This means that for any x, cos(x) will give you the exact derivative of sin at that x.
So you need to compare the result of the derivative(...) function to the value of cos(x). This will give you the difference. You can then use the basic Python function abs(x) to get the absolute value of that difference, which will give you the absolute difference, which is the desired result. Do that for each j from 0 to 20 and store the results somewhere, in an array or a dict.
from math import sin, cos, pi
x = pi / 4
diffs = {}
for j in range(21): # range is exclusive so range(21) will stop at 20
h = 10 ** -j
deriv = derivative(sin, x, h)
exact = cos(x)
diff = abs(deriv - exact)
diffs[h] = diff
Then, you can use pyplot's loglog function to plot those results on a graph, passing as X the range(...) result and as Y the array containing the results.
import matplotlib.pyplot as plt
ordered = sorted(diffs.items())
x, y = zip(*ordered)
plt.loglog(x, y)
I want to create a logarithmic function with base x then plot it: y=logx10.
So I use:
y= math.log(10,x)
but it returned an error said: only length-1 array can be converted to Python scalars.
So what is the correct way to create a log function with base x?
The simple way to get a "smoother" line is by increasing the number of points (i.e., make length bigger.)
Also, you likely want to sort your x list before calculating and plotting:
length = 100 # or higher
:
x = sorted([random.uniform(rand_min, rand_max) for r in xrange(length)])
y = [math.log(10, _x) for _x in x]
Since you want 2 lists of values (x, y), you will have to generate the x list first, and use it to generate the y list:
import math
import random
length = 10
rand_min = 0.02
rand_max = 0.91
x = [random.uniform(rand_min, rand_max) for r in xrange(length)]
y = [math.log(10, _x) for _x in x]
Here you have lists x and y, both of length length.