We Keep Coding

Average of a 3D numpy slice based on 2D arrays

I am trying to calculate the average of a 3D array between two indices on the 1st axis. The start and end indices vary from cell to cell and are represented by two separate 2D arrays that are the same shape as a slice of the 3D array.
I have managed to implement a piece of code that loops through the pixels of my 3D array, but this method is painfully slow in the case of my array with a shape of (70, 550, 350). Is there a way to vectorise the operation using numpy or xarray (the arrays are stored in an xarray dataset)?
Here is a snippet of what I would like to optimise:
# My 3D raster containing values; shape = (time, x, y)
values = np.random.rand(10, 55, 60)
# A 2D raster containing start indices for the averaging
start_index = np.random.randint(0, 4, size=(values.shape[1], values.shape[2]))
# A 2D raster containing end indices for the averaging
end_index = np.random.randint(5, 9, size=(values.shape[1], values.shape[2]))
# Initialise an array that will contain results
mean_array = np.zeros_like(values[0, :, :])
# Loop over 3D raster to calculate the average between indices on axis 0
for i in range(0, values.shape[1]):
for j in range(0, values.shape[2]):
mean_array[i, j] = np.mean(values[start_index[i, j]: end_index[i, j], i, j], axis=0)

One way to do this without loops is to zero-out the entries you don't want to use, compute the sum of the remaining items, then divide by the number of nonzero entries. For example:
i = np.arange(values.shape[0])[:, None, None]
mean_array_2 = np.where((i >= start_index) & (i < end_index), values, 0).sum(0) / (end_index - start_index)
np.allclose(mean_array, mean_array_2)
# True
Note that this assumes that the indices are in the range 0 <= i < values.shape[0]; if this is not the case you can use np.clip or other means to standardize the indices before computation.

Can someone explain to me how map_coordinates works on n dimensional datasets?

I have a 3D image that I'm trying to transform, with a known coordinate mapping. I'm trying to use map_coordinates but the scipy documentation only talks about mapping to a 1D vector leaving me rather confused.
The transformation is vectorized so I can give it a meshgrid of x y z indices and it produces a 3 x nx x ny x nz array, where the first index goes over the xyz components of the vector field and the others correspond directly to the meshgrid input dimensions
Now I just need to map the array elements of an output array to the corresponding pixels in the initial image.
I want to use map_coordinates but the format of the coordinates argument is not clear to me.
Can anyone give me an example of how I would create the coordinates array in this case?

Finally figure it out so figured I would leave this here
# transposed_frame is the 3d image that needs to be transformed (shape (632, 352, 35))
# Meshgrid of the matrix coordinates
x_grid, y_grid, z_grid = np.meshgrid(range(transposed_frame.shape[0]), range(transposed_frame.shape[1]), range(transposed_frame.shape[2]), indexing = 'ij')
# inverse transformation that needs to be applied to the image (shape (3, 632, 352, 35))
# the first dimension goes over the different components of the vector field x y z
# So transform[0, i, j, k] is the x coordinate of the vector field in the point [i, j, k]
transform = vectorized_interp_field(x_grid, y_grid, z_grid)
# Transforming the image through map_coordinates is then as simple as
inverse_transformed = map_coordinates(transposed_frame, transform)
The part I didn't understand about map_coordinates was exactly what form the mapping matrix was supposed to have for higher dimensional data.
It seems to work in general as follows
B = map_coordinates(A, mapping)
B[i, j, k] = A[mapping[0, i, j, k], mapping[1, i, j, k], mapping[2, i, j, k]]

Unravel 1D list back to 3D array

Basically, is there a way to transform a 1D list that has been "flattened" through the numpy.ravel() function back to it's original 3D form ? I know the dimensions, and one might ask why I just don't use the original 3D array in the first place, instead of converting it - but there reasons for that.
I just need to know if I can actually create the same 3D array from a 1D array that was created by using numpy.ravel() on the 3D array.
Basically the 3D array was created like this:
import numpy as np
nx = 50
ny = 40
nz = 150
x = np.linspace(1, 51, nx)
y = np.linspace(1, 41, ny)
z = np.linspace(1, 151, nz)
x_bc = x[:, np.newaxis, np.newaxis]
y_bc = y[np.newaxis, :, np.newaxis]
z_bc = z[np.newaxis, np.newaxis, :]
arr = x_bc + y_bc + z_bc
And nope, I can't just do this to get it back, since calculations has been done to it in the mean time, and then converted to a 1D array in the mean time as well. So the data in this array is not the same as the one I actually want to convert back.

Just reshape it back to the original shape?
raveled = np.ravel(arr)
new_arr = raveled.reshape(*arr.shape)

Does numpy.reshape do what you want?

Reshape from flattened indices in Python

I have an image of size M*N whose pixels coordinates has been flattened to a 1D array according to a space-filling curve (i.e. not a classical rasterization where I could have used reshape).
I thus process my 1D array (flattened image) and I then would like to reshape it to a M*N array (initial size).
So far, I have done this with a for-loop:
for i in range(img_flat.size):
img_res[x[i], y[i]] = img_flat[i]
x and y being the x and y pixels coordinates according to my path scan.
However, I am wondering how to do this in a unique line of code.

If x and y are numpy arrays of dimension 1 and lengths n, and img_flat also has length n img_res is a numpy array of dimension 2 (h, w) such that `h*w = n, then:
img_res[x, y] = img_flat
Should suffice

In fact, it was easy:
vec = np.arange(0, seg.size, dtype=np.uint)
img_res[x[vec], y[vec]] = seg[vec]

Slice subarray from numpy array by list of indices

I have a 2D numpy array input_array and two lists of indices (x_coords and y_coords). Id like to slice a 3x3 subarray for each x,y pair centered around the x,y coordinates. The end result will be an array of 3x3 subarrays where the number of subarrays is equal to the number of coordinate pairs I have.
Preferably by avoiding for loops. Currently I use a modification of game of life strides from the scipy cookbook:
http://wiki.scipy.org/Cookbook/GameOfLifeStrides
shape = (input_array.shape[0] - 2, input_array.shape[0] - 2, 3, 3)
strides = input_array.strides + input_array.strides
strided = np.lib.stride_trics.as_strided(input_array, shape=shape, strides=strides).\
reshape(shape[0]*shape[1], shape[2], shape[3])
This creates a view of the original array as a (flattened) array of all possible 3x3 subarrays. I then convert the x,y coordinate pairs to be able to select the subarrays I want from strided:
coords = x_coords - 1 + (y_coords - 1)*shape[1]
sub_arrays = strided[coords]
Although this works perfectly fine, I do feel it is a bit cumbersome. Is there a more direct approach to do this? Also, in the future I would like to extend this to the 3D case; slicing nx3x3 subarrays from a nxmxk array. It might also be possible using strides but so far I haven't been able to make it work in 3D

Here is a method that use array broadcast:
x = np.random.randint(1, 63, 10)
y = np.random.randint(1, 63, 10)
dy, dx = [grid.astype(int) for grid in np.mgrid[-1:1:3j, -1:1:3j]]
Y = dy[None, :, :] + y[:, None, None]
X = dx[None, :, :] + x[:, None, None]
then you can use a[Y, X] to select blocks from a. Here is an example code:
img = np.zeros((64, 64))
img[Y, X] = 1
Here is graph ploted by pyplot.imshow():

A very straight forward solution would be a list comprehension and itertools.product:
import itertools
sub_arrays = [input_array[x-1:x+2, y-1:y+2]
for x, y in itertools.product(x_coords, y_coords)]
This creates all possible tuples of coordinates and then slices the 3x3 arrays from the input_array.
But this is sort-of a for loop. And you will have to take care, that x_coords and y_coords are not on the border of the matrix.