I have the following minimum code using scipy.interpolate.interp2d to do interpolation on 2d grid data.
import numpy as np
from scipy import interpolate
x = np.arange(-5.01, 5.01, 0.25)
y = np.arange(-5.01, 5.01, 0.25)
xx, yy = np.meshgrid(x, y)
z = np.sin(xx**2+yy**2)
f = interpolate.interp2d(x, y, z, kind='cubic')
Now f here can be used to evaluate other points. The problem is the points I want to evaluate are totally random points not forming a regular grid.
# Evaluate at point (x_new, y_new), in total 256*256 points
x_new = np.random.random(256*256)
y_new = np.random.random(256*256)
func(x_new, y_new)
This will cause a runtime error in my PC, it seems to treat x_new and y_new as mesh grid, generate a evaluation matrix 65536x65536, which is not my purpose.
RuntimeError: Cannot produce output of size 65536x65536 (size too large)
One way to get things done is to evaluate points one by one, using code:
z_new = np.array([f(i, j) for i, j in zip(x_new, y_new)])
However, it is slow!!!
%timeit z_new = np.array([f(i, j) for i, j in zip(x_new, y_new)])
1.26 s ± 46.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
Is there any faster way to evaluate random points?
Faster here I mean comparable with time below:
x_new = np.random.random(256)
y_new = np.random.random(256)
%timeit f(x_new, y_new)
Same 256*256 = 65536 evaluations, time for this in my PC:
1.21 ms ± 39.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
It does not have to be in comparable speed with 1.21ms, 121 ms is totally acceptable.
The function you are looking for is scipy.interpolate.RegularGridInterpolator
Given a set of points (x,y,z), where x & y are defined on a regular grid, it allows you to sample the z-value of intermediate (x,y) points. In your case, this would look as follows
import numpy as np
from scipy import interpolate
x = np.arange(-5.01, 5.01, 0.25)
y = np.arange(-5.01, 5.01, 0.25)
def f(x,y):
return np.sin(x**2+y**2)
z = f(*np.meshgrid(x, y, indexing='ij', sparse=True))
func = interpolate.RegularGridInterpolator((x,y), z)
x_new = np.random.random(256*256)
y_new = np.random.random(256*256)
xy_new = list(zip(x_new,y_new))
z_new = func(xy_new)func(xy_new)
For more details, see https://docs.scipy.org/doc/scipy-0.16.1/reference/generated/scipy.interpolate.RegularGridInterpolator.html
Related
I have a large collection (26,214,400 to be exact) of sets of data I want to perform a linear regressions on, i.e. each of the 26,214,400 data sets consists of n x values and n y values and I want to find y = m * x + b. For any set of points I can use sklearn or numpy.linalg.lstsq, something like:
A = np.vstack([x, np.ones(len(x))]).T
m, b = np.linalg.lstsq(A, y, rcond=None)[0]
Is there a way to set up the matrices such that I can avoid a python loop through 26,214,400 items? Or do I have to use a loop and would be better served using something like Numba?
I ended up going the numba route which yielded a ~20x speed up on my laptop, it used all my cores so I assume more CPUs the better. The answer looked something like
import numpy as np
from numpy.linalg import lstsq
import numba
#numba.jit(nogil=True, parallel=True)
def fit(XX, yy):
""""Fit a large set of points to a regression"""
assert XX.shape == yy.shape, "Inputs mismatched"
n_pnts, n_samples = XX.shape
scale = np.empty(n_pnts)
offset = np.empty(n_pnts)
for i in numba.prange(n_pnts):
X, y = XX[i], yy[i]
A = np.vstack((np.ones_like(X), X)).T
offset[i], scale[i] = lstsq(A, y)[0]
return offset, scale
Running it:
XX, yy = np.random.randn(2, 1000, 10)
offset, scale = fit(XX, yy)
%timeit offset, scale = fit(XX, yy)
1.87 ms ± 37.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
The non-jitted version has this timing:
41.7 ms ± 620 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
I need to calculate the distance between each pixcel and each centroid.
Arguments:
X (numpy array): PxD 1st set of data points (usually data points)
C (numpy array): KxD 2nd set of data points (usually cluster centroids points)
Returns:
dist: PxK numpy array position ij is the distance between the i-th point of the first set an the j-th point of the second set
def distance(X, C):
dist = numpy.empty((X.shape[0], C.shape[0]))
for i,x in enumerate(X):
for y,c in enumerate(C):
dist[i][y] = euclidean_dist(x,c)
return dist
def euclidean_dist(x, y):
x1, y1, z1 = x
x2, y2, z2 = y
return math.sqrt((x1-x2)**2 + (y1-y2)**2 + (z1-z2)**2)
If you can add scipy dependency, then this is already implemented in scipy.spatial.distance.cdist. Otherwise we can use numpy.broadcasting and numpy.linalg.norm:
Scipy Implemenation
from scipy.spatial import distance
distance.cdist(X, C, 'euclidean')
Numpy Implementation
import numpy as np
np.linalg.norm(X[:,None,:] - C, axis=2)
Performance
P = 100_000
K = 10_00
D = 3
X = np.random.randint(0,10, (P,D))
C = np.random.randint(0,10, (K,D))
%timeit distance.cdist(X, C, 'euclidean')
1.06 s ± 57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit np.linalg.norm(X[:,None,:] - C, axis=2)
15 s ± 2.18 s per loop (mean ± std. dev. of 7 runs, 1 loop each)
We can see that for large sizes of X and C scipy implementation is way faster.
I have an interpolated function of two Cartesian variables (created using RectBivariateSpline), and I'm looking for the fastest means of evaluating that function over a polar grid.
I've approached this problem by first defining spaces in r (the radial coordinate) and t (the angular coordinate), creating a meshgrid from these, converting this meshgrid to Cartesian coordinates, and then evaluating the function at each point by looping over the Cartesian meshgrid. The below code demonstrates this.
import numpy as np
import scipy as sp
from scipy.interpolate import RectBivariateSpline
# this shows the type of data/function I'm working with:
n = 1000
x = np.linspace(-10, 10, n)
y = np.linspace(-10, 10, n)
z = np.random.rand(n,n)
fun = RectBivariateSpline(x, y, z)
# this defines the polar grid and converts it to a Cartesian one:
nr = 1000
nt = 360
r = np.linspace(0, 10, nr)
t = np.linspace(0, 2*np.pi, nt)
R, T = np.meshgrid(r, t, indexing = 'ij')
kx = R*np.cos(T)
ky = R*np.sin(T)
# this evaluates the interpolated function over the converted grid:
eval = np.empty((nr, nt))
for i in range(0, nr):
for j in range(0, nt):
eval[i][j] = fun(kx[i][j], ky[i][j])
In this way, I get an array whose elements match up with the R and T arrays, where i corresponds to R, and j to T. This is important, because for each radius I need to sum the evaluated function over the angular coordinate.
This approach works, but is dreadfully slow... in reality I am working with much, much larger arrays than those here. I'm looking for a way to speed this up.
One thing I've noticed is that one can submit two 1D arrays to a 2-variable function and have returned a 2D array of the function evaluated at each possible combination of the input points. Because my function isn't a polar one, however, I can't just submit my radial and angular arrays to the function. Ideally there'd be a way of converting an interpolated function to accept polar arguments, but I don't think this is possible.
I should note further that there is no way I can define the function in terms of radial coordinates in the first place: the data I'm using is output from a discrete Fourier transform, which requires rectangularly-gridded data.
Any help would be appreciated!
By examining the __call__ method of RectBivariateSpline here, you can use the grid=False option to avoid the slow double loop here.
This alone provides an order of magnitude speed up on the example you gave. I would expect the speedup to be even better on larger data sets.
Also the answers are the same between the methods as expected.
import numpy as np
import scipy as sp
from scipy.interpolate import RectBivariateSpline
# this shows the type of data/function I'm working with:
n = 1000
x = np.linspace(-10, 10, n)
y = np.linspace(-10, 10, n)
z = np.random.rand(n,n)
fun = RectBivariateSpline(x, y, z)
# this defines the polar grid and converts it to a Cartesian one:
nr = 1000
nt = 360
r = np.linspace(0, 10, nr)
t = np.linspace(0, 2*np.pi, nt)
R, T = np.meshgrid(r, t, indexing = 'ij')
kx = R*np.cos(T)
ky = R*np.sin(T)
# this evaluates the interpolated function over the converted grid:
def evaluate_slow(kx, ky):
eval = np.empty((nr, nt))
for i in range(0, nr):
for j in range(0, nt):
eval[i][j] = fun(kx[i][j], ky[i][j])
return eval
def evaluate_fast(kx, ky):
eval = fun(kx.ravel(), ky.ravel(), grid=False)
return eval
%timeit evaluate_slow(kx, ky)
%timeit evaluate_fast(kx, ky)
eval1 = evaluate_slow(kx, ky)
eval2 = evaluate_fast(kx, ky)
print(np.all(np.isclose(eval1, eval2.reshape((nr, nt)))))
1.69 s ± 73.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
262 ms ± 2.86 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
True
I have a function which is of the form :
def f(x, y):
total = 0
u = np.zeros(10)
for i in range(0,10):
u[i] = x * i + y* i
if u[i] < 10:
print('do something')
total = total + u[i]
return total
this function when i try with a given x and y values works well.
f(3,4)
Out[49]: 63.0
I want to create a 3d contour plot using matplotlib. Tried with
x = np.linspace(-6, 6, 30)
y = np.linspace(-6, 6, 30)
X, Y = np.meshgrid(x, y)
Z = f(X, Y)
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(X, Y, Z, 50, cmap='binary')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z');
I had to create a mesh grid for 3d plot. I get an error when I try with this because of the loop in my function. I get an error
ValueError: setting an array element with a sequence.
How to plot 3d graphs if my function has a loop?
You need np.vectorize:
# same setup as above, then
Z = np.vectorize(f)(X, Y)
import pylab as plt
plt.imshow(Z, extent=[x[0], x[-1], y[0], y[-1]])
(I checked with imshow but contour3D will work too.)
np.vectorize will take a function that accepts scalar (non-array) arguments and magically loops over arrays. It's nominally equivalent to:
Z2 = np.array([f(xx, yy) for xx in x for yy in y]).reshape(X.shape)
print(np.abs(Z - Z2).max()) # should print 0
but faster: after I remove the print in f:
In [47]: %timeit Z = np.vectorize(f)(X, Y)
6 ms ± 339 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
In [48]: %timeit Z2 = np.array([f(xx, yy) for xx in x for yy in y]).reshape(X.shape)
13.7 ms ± 310 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
(I had to remove the prints for timing because printing is Very Slow.)
I have several thousand "observations". Each observation consists of location (x,y) and sensor reading (z), see example below.
I would like to fit a bi-linear surface to the x,y, and z data. I am currently doing it with the code-snippet from amroamroamro/gist:
def bi2Dlinter(xdata, ydata, zdata, gridrez):
X,Y = np.meshgrid(
np.linspace(min(x), max(x), endpoint=True, num=gridrez),
np.linspace(min(y), max(y), endpoint=True, num=gridrez))
A = np.c_[xdata, ydata, np.ones(len(zdata))]
C,_,_,_ = scipy.linalg.lstsq(A, zdata)
Z = C[0]*X + C[1]*Y + C[2]
return Z
My current approach is to cycle through the rows of the DataFrame. (This works great for 1000 observations but is not usable for larger data-sets.)
ZZ = []
for index, row in df2.iterrows():
x=row['x1'], row['x2'], row['x3'], row['x4'], row['x5']
y=row['y1'], row['y2'], row['y3'], row['y4'], row['y5']
z=row['z1'], row['z2'], row['z3'], row['z4'], row['z5']
ZZ.append(np.median(bi2Dlinter(x,y,z,gridrez)))
df2['ZZ']=ZZ
I would be surprised if there is not a more efficient way to do this.
Is there a way to vectorize the linear interpolation?
I put the code here which also generates dummy entries.
Thanks
Looping over DataFrames like this is generally not recommended. Instead you should opt to try and vectorize your code as much as possible.
First we create an array for your inputs
x_vals = df2[['x1','x2','x3','x4','x5']].values
y_vals = df2[['y1','y2','y3','y4','y5']].values
z_vals = df2[['z1','z2','z3','z4','z5']].values
Next we need to create a bi2Dlinter function that handles vector inputs, this involves changing linspace/meshgrid to work for an array and changing the least_squares function. Normally scipy.linalg functions work over an array but as far as I'm aware the .lstsq method doesn't. Instead we can use the .SVD to replicate the same functionality over an array.
def create_ranges(start, stop, N, endpoint=True):
if endpoint==1:
divisor = N-1
else:
divisor = N
steps = (1.0/divisor) * (stop - start)
return steps[:,None]*np.arange(N) + start[:,None]
def linspace_nd(x,y,gridrez):
a1 = create_ranges(x.min(axis=1), x.max(axis=1), N=gridrez, endpoint=True)
a2 = create_ranges(y.min(axis=1), y.max(axis=1), N=gridrez, endpoint=True)
out_shp = a1.shape + (a2.shape[1],)
Xout = np.broadcast_to(a1[:,None,:], out_shp)
Yout = np.broadcast_to(a2[:,:,None], out_shp)
return Xout, Yout
def stacked_lstsq(L, b, rcond=1e-10):
"""
Solve L x = b, via SVD least squares cutting of small singular values
L is an array of shape (..., M, N) and b of shape (..., M).
Returns x of shape (..., N)
"""
u, s, v = np.linalg.svd(L, full_matrices=False)
s_max = s.max(axis=-1, keepdims=True)
s_min = rcond*s_max
inv_s = np.zeros_like(s)
inv_s[s >= s_min] = 1/s[s>=s_min]
x = np.einsum('...ji,...j->...i', v,
inv_s * np.einsum('...ji,...j->...i', u, b.conj()))
return np.conj(x, x)
def vectorized_bi2Dlinter(x_vals, y_vals, z_vals, gridrez):
X,Y = linspace_nd(x_vals, y_vals, gridrez)
A = np.stack((x_vals,y_vals,np.ones_like(z_vals)), axis=2)
C = stacked_lstsq(A, z_vals)
n_bcast = C.shape[0]
return C.T[0].reshape((n_bcast,1,1))*X + C.T[1].reshape((n_bcast,1,1))*Y + C.T[2].reshape((n_bcast,1,1))
Upon testing this on data for n=10000 rows, the vectorized function was significantly faster.
%%timeit
ZZ = []
for index, row in df2.iterrows():
x=row['x1'], row['x2'], row['x3'], row['x4'], row['x5']
y=row['y1'], row['y2'], row['y3'], row['y4'], row['y5']
z=row['z1'], row['z2'], row['z3'], row['z4'], row['z5']
ZZ.append((bi2Dlinter(x,y,z,gridrez)))
df2['ZZ']=ZZ
Out: 5.52 s ± 17.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%%timeit
res = vectorized_bi2Dlinter(x_vals,y_vals,z_vals,gridrez)
Out: 74.6 ms ± 159 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
You should pay careful attention to whats going on in this vectorize function and familiarize yourself with broadcasting in numpy. I cannot take credit for the first three functions, instead I will link their answers from stack overflow for you to get an understanding.
Vectorized NumPy linspace for multiple start and stop values
how to solve many overdetermined systems of linear equations using vectorized codes?
How to use numpy.c_ properly for arrays