We Keep Coding

What is the best way/method to digitize the data of a 3D surface into a grid of pixels with smaller resolution in Python?

I want to digitize (= average out over cells) photon count data into pixels given by a grid that tells how they are aligned. The photon count data is stored in a 2D array. I want to split that data into cells, each of which would correspond to a pixel. The idea is basically the same as changing an HD image to a smaller resolution. I'd like to achieve this in Python.
The digitizing function I've written:
import numpy as np
def digitize(function_data, grid_shape):
"""
function_data = 2D array of function values of some 3D shape,
eg.: exp(-(x^2 + y^2 -> want to digitize this
grid_shape: an array of length 2 which contains the dimensions of the smaller resolution
"""
l = len(function_data)
pixel_len_x = int(l/grid_shape[0])
pixel_len_y = int(l/grid_shape[1])
digitized_data = np.empty((grid_shape[0], grid_shape[1]))
for i in range(grid_shape[0]): #row-index of pixel in smaller-resolution grid
for j in range(grid_shape[1]): #column-index of pixel in smaller-resolution grid
hd_pixel = []
for k in range(pixel_len_y):
hd_pixel.append(z_data[k][j:j*pixel_len_x])
hd_pixel = np.ravel(hd_pixel) #turns 2D array into 1D to be able to compute average
pixel_avg = np.average(hd_pixel)
digitized_data[i][j] = pixel_avg
return digitized_data
In theory, this function should do what I want to achieve, but when tested it doesn't yield the expected results. Either a completed version of my function or any other method that achieves my goal would be extremely helpful.

You could also use a interpolation function, if you can use SciPy. Here we use one of the gridded data interpolating functions, RectBivariateSpline to upsample your function, but you can find numerous examples on this and other sites.
import numpy as np
import matplotlib.pyplot as plt
from scipy.interpolate import RectBivariateSpline as rbs
# Sampling coordinates
x = np.linspace(-2,2,20)
y = np.linspace(-2,2,30)
# Your function
f = np.exp(-(x[:,None]**2 + y**2))
# Interpolator
interp = rbs(x, y, f)
# Higher resolution coordinates
x_hd = np.linspace(x.min(), x.max(), x.size * 5)
y_hd = np.linspace(y.min(), y.max(), y.size * 5)
# New higher res function
f_hd = interp(x_hd, y_hd, grid = True)
# Some plots
fig, ax = plt.subplots(ncols = 2)
ax[0].imshow(f)
ax[1].imshow(f_hd)

Using FFT for 3D array representation of 2D field

I need to obtain the fourier transform of a complex field. I'm using python.
My input is a 2D snapshot of the electric field in the xy-plane.
I currently have a 3D array F[x][y][z] where F[x][y][0] contains the real component and F[x][y]1 contains the complex component of the field.
My current code is very simple and does this:
result=np.fft.fftn(F)
result=np.fft.fftshift(result)
I have the following questions:
1) Does this correctly compute the fourier transform of the field, or should the field be entered as a 2D matrix with each element containing both the real and imaginary component instead?
2) I entered the complex component values of the field using the real multiple only (i.e if the complex value is 6i I entered 6), is this correct or should this be entered as a complex value instead (i.e. entered as '6j')?
3) As this is technically a 2D input field, should I use np.fft.fft2 instead? Doing this means the output is not centered in the middle.
4) The output does not look like what I'd expect the fourier transform of F to look like, and I'm unsure what I'm doing wrong.
Full example code:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import cm
x, y = np.meshgrid(np.linspace(-1,1,100), np.linspace(-1,1,100))
d = np.sqrt(x*x+y*y)
sigma, mu = .35, 0.0
g1 = np.exp(-( (d-mu)**2 / ( 2.0 * sigma**2 ) ) )
F=np.empty(shape=(300,300,2),dtype=complex)
for x in range(0,300):
for y in range(0,300):
if y<50 or x<100 or y>249 or x>199:
F[x][y][0]=g1[0][0]
F[x][y][1]=0j
elif y<150:
F[x][y][0]=g1[x-100][y-50]
F[x][y][1]=0j
else:
F[x][y][0]=g1[x-100][y-150]
F[x][y][1]=0j
F_2D=np.empty(shape=(300,300))
for x in range(0,300):
for y in range(0,300):
F_2D[x][y]=np.absolute(F[x][y][0])+np.absolute(F[x][y][1])
plt.imshow(F_2D)
plt.show()
result=np.fft.fftn(F)
result=np.fft.fftshift(result)
result_2D=np.empty(shape=(300,300))
for x in range(0,300):
for y in range(0,300):
result_2D[x][y]=np.absolute(result[x][y][0])+np.absolute(result[x][y][1])
plt.imshow(result_2D)
plt.show()
plotting F gives this:
With np.fft.fftn, the image shown at the end is:
And with np.fft.fft2:
Neither of these look like what I would expect the fourier transform of F to look like.

I add here another answer, suitable to the added code.
The answer is still np.fft.fft2(). Here's an example. I modified the code slightly. To verify that we need fft2 I discarded one of the blobs, and then we know that a single Gaussian blob should transform into a Gaussian blob (with a certain phase, that's not shown when plotting absolute value). I also decreased the standard deviation so that the frequency response will widen a little.
Code:
import numpy as np
import matplotlib.pyplot as plt
x, y = np.meshgrid(np.linspace(-1,1,100), np.linspace(-1,1,100))
d = np.sqrt(x**2+y**2)
sigma, mu = .1, 0.0
g1 = np.exp(-( (d-mu)**2 / ( 2.0 * sigma**2 ) ) )
N = 300
positions = [ [150,100] ]#, [150,200] ]
sz2 = [int(x/2) for x in g1.shape]
F_2D = np.zeros([N,N])
for x0,y0 in positions:
F_2D[ x0-sz2[0]: x0+sz2[0], y0-sz2[1]:y0+sz2[1] ] = g1 + 1j*0.
result = np.fft.fftshift(np.fft.fft2(F_2D))
plt.subplot(211); plt.imshow(F_2D)
plt.subplot(212); plt.imshow(np.absolute(result))
plt.title('$\sigma$=.1')
plt.show()
Result:
To get back to the original problem, we need only change
positions = [ [150,100] , [150,200] ]
and sigma=.35 instead of sigma=.1.

You should use complex numpy variables (by using 1j) and use fft2. For example:
N = 16
x0 = np.random.randn(N,N,2)
x = x0[:,:,0] + 1j*x0[:,:,1]
X = np.fft.fft2(x)
Using fftn on x0 will do a 3D FFT, and using fft will do vector-wise 1D FFT.

Basemap interpolation alternative - regridding data

I'm moving from basemap to cartopy given basemap is going to be phased out. I've previously used the basemap.interp functionality to interpolate data, e.g. say I have data at 1 degree resolution (180x360), I would run the following to interpolate to 0.5 degrees.
import numpy as np
from mpl_toolkits import basemap
Old_Lon = np.linspace(-180,180,360)
Old_Lat = np.linspace(-90,90,180)
New_Lon = np.linspace(-180,180,720)
New_Lat = np.linspace(-90,90,360)
New_Lon,New_Lat = np.meshgrid(New_Lon,New_Lat)
New_Data = basemap.interp(Old_Data,Old_Lon,Old_Lat,New_Lon,New_Lat,order=0)
order gives me options to choose from nearest neighbour, bi-linear etc. Is there an alternative that does this in as simple way? I've seen scipy has interpolation but I'm not sure how to apply it. Any help would be appreciated!

I eventually decided to take the raw code from Basemap and make it into a standalone function - I'll be recommending it to the cartopy guys to implement it as its a useful feature. Posting here as could be useful to someone else:
def Interp(datain,xin,yin,xout,yout,interpolation='NearestNeighbour'):
"""
Interpolates a 2D array onto a new grid (only works for linear grids),
with the Lat/Lon inputs of the old and new grid. Can perfom nearest
neighbour interpolation or bilinear interpolation (of order 1)'
This is an extract from the basemap module (truncated)
"""
# Mesh Coordinates so that they are both 2D arrays
xout,yout = np.meshgrid(xout,yout)
# compute grid coordinates of output grid.
delx = xin[1:]-xin[0:-1]
dely = yin[1:]-yin[0:-1]
xcoords = (len(xin)-1)*(xout-xin[0])/(xin[-1]-xin[0])
ycoords = (len(yin)-1)*(yout-yin[0])/(yin[-1]-yin[0])
xcoords = np.clip(xcoords,0,len(xin)-1)
ycoords = np.clip(ycoords,0,len(yin)-1)
# Interpolate to output grid using nearest neighbour
if interpolation == 'NearestNeighbour':
xcoordsi = np.around(xcoords).astype(np.int32)
ycoordsi = np.around(ycoords).astype(np.int32)
dataout = datain[ycoordsi,xcoordsi]
# Interpolate to output grid using bilinear interpolation.
elif interpolation == 'Bilinear':
xi = xcoords.astype(np.int32)
yi = ycoords.astype(np.int32)
xip1 = xi+1
yip1 = yi+1
xip1 = np.clip(xip1,0,len(xin)-1)
yip1 = np.clip(yip1,0,len(yin)-1)
delx = xcoords-xi.astype(np.float32)
dely = ycoords-yi.astype(np.float32)
dataout = (1.-delx)*(1.-dely)*datain[yi,xi] + \
delx*dely*datain[yip1,xip1] + \
(1.-delx)*dely*datain[yip1,xi] + \
delx*(1.-dely)*datain[yi,xip1]
return dataout
--

The SciPy interpolation routines return a function that you can call to perform an interpolation. For nearest neighbour interpolation on a regular grid, you can use scipy.interpolate.RegularGridInterpolator:
import numpy as np
from scipy.interpolate import RegularGridInterpolator
nearest_function = RegularGridInterpolator(
(old_lon, old_lat), old_data, method="nearest", bounds_error=False
)
new_data = np.array(
[[nearest_function([i, j]) for j in new_lat] for i in new_lon]
).squeeze()
That isn't perfect, though, because lon=175 are all fill values. (If I hadn't set bounds_error=False then you'd get an error there.) In that case, you need to ask how you want to wrap around the dateline. A straightforward solution would be to copy the lon=0 line to the end of the array and call it lon=180.
Should you want linear or higher order interpolation one day, which I'd recommend if your data are points rather than cells, you can use scipy.interpolate.RectBivariateSpline:
import numpy as np
from scipy.interpolate import RectBivariateSpline
old_step = 10
old_lon = np.arange(-180, 180, old_step)
old_lat = np.arange(-90, 90, old_step)
old_data = np.random.random((len(old_lon), len(old_lat)))
interp_function = RectBivariateSpline(old_lon, old_lat, old_data, kx=1, ky=1)
new_lon = np.arange(-180, 180, new_step)
new_lat = np.arange(-90, 90, new_step)
new_data = interp_function(new_lon, new_lat)

scipy splrep() with weights not fitting the given curve

Using scipy's splrep I can easily fit a test sinewave:
import numpy as np
from scipy.interpolate import splrep, splev
import matplotlib.pyplot as plt
plt.style.use("ggplot")
# Generate test sinewave
x = np.arange(0, 20, .1)
y = np.sin(x)
# Interpolate
tck = splrep(x, y)
x_spl = x + 0.05 # Just to show it wors
y_spl = splev(x_spl, tck)
plt.plot(x_spl, y_spl)
The splrep documentation states that the default value for the weight parameter is np.ones(len(x)). However, plotting this results in a totally different plot:
tck = splrep(x, y, w=np.ones(len(x_spl)))
y_spl = splev(x_spl, tck)
plt.plot(x_spl, y_spl)
The documentation also states that the smoothing condition s is different when a weight array is given - but even when setting s=len(x_spl) - np.sqrt(2*len(x_spl)) (the default value without a weight array) the result does not strictly correspond to the original curve as shown in the plot.
What do I need to change in the code listed above in order to make the interpolation with weight array (as listed above) output the same result as the interpolation without the weights?
I have tested this with scipy 0.17.0. Gist with a test IPython notebook

You only have to change one line of your code to get the identical output:
tck = splrep(x, y, w=np.ones(len(x_spl)))
should become
tck = splrep(x, y, w=np.ones(len(x_spl)), s=0)
So, the only difference is that you have to specify s instead of using the default one.
When you look at the source code of splrep you will see why that is necessary:
if w is None:
w = ones(m, float)
if s is None:
s = 0.0
else:
w = atleast_1d(w)
if s is None:
s = m - sqrt(2*m)
which means that, if neither weights nor s are provided, s is set to 0 and if you provide weights but no s then s = m - sqrt(2*m) where m = len(x).
So, in your example above you compare outputs with the same weights but with different s (which are 0 and m - sqrt(2*m), respectively).

Interpolation of curve

I have a code where a curve is generated using random values. and a Horizontal line which runs through it. The code is as follows.
import numpy as np
import matplotlib.pylab as pl
data = np.random.uniform(low=-1600, high=-550, size=(288,))
line = [-1290] * 288
pl.figure(figsize = (10,5))
pl.plot(data)
pl.plot(line)
Now I need to find the the coordinates for the all the points of intersections of the curve (data) and the line. The curve is made of linear segments that join neighboring points . And there are a lot of intersection points where the curve meets the line. any help would be appreciated. thank you!

I like the Shapely answer because Shapely is awesome, but you might not want that dependency. Here's a version of some code I use in signal processing adapted from this Gist by #endolith. It basically implements kazemakase's suggestion.
from matplotlib import mlab
def find_crossings(a, value):
# Normalize the 'signal' to zero.
sig = a - value
# Find all indices right before any crossing.
indices = mlab.find((sig[1:] >= 0) & (sig[:-1] < 0) | (sig[1:] < 0) & (sig[:-1] >= 0))
# Use linear interpolation to find intersample crossings.
return [i - sig[i] / (sig[i+1] - sig[i]) for i in indices]
This returns the indices (your x values) of where the curve crosses the value (-1290 in your case). You would call it like this:
find_crossings(data, -1290)
Here's what I get for 100 points:
x = find_crossings(data, -1290)
plt.figure(figsize=(10,5))
plt.plot(data)
plt.plot(line)
plt.scatter(x, [-1290 for p in x], color='red')
plt.show()

I think the curve, as you interpret it, does in fact follow an equation. In particular, it is made of linear segments that join neighboring points.
Here is what you can do:
find all pairs of neighbors where one lies above and the other below the line
for each pair find the intersection of the horizontal line with the line joining the points

Here is a solution that use shapely:
import numpy as np
import matplotlib.pylab as pl
np.random.seed(0)
data = np.random.uniform(low=-1600, high=-550, size=(50,))
line = [-1290] * len(data)
pl.figure(figsize = (10,5))
pl.plot(data)
pl.plot(line)
from shapely import geometry
line = geometry.LineString(np.c_[np.arange(len(data)), data])
hline = geometry.LineString([[-100, -1290], [1000, -1290]])
points = line.intersection(hline)
x = [p.x for p in points]
y = [p.y for p in points]
pl.plot(x, y, "o")
the output: