I'm working with a triangulated mesh consisting of points 3 x n and triangles specified by the point indices 3 x m. I can easily plot that e.g. using mlab
mesh = mlab.triangular_mesh(p[0,:],p[1,:],p[2,:],t.T
I am also generating a mask masking points which are out of bounds or nan, so I have a mask the size of n. Now I want to mask the triangles which have a masked point. My solutions so far:
1: Use the mask to turn all masked points into nan, e.g.
p[mask] = nan
mlab then still shows nan (I would need to include a threshold filter...) and I actually don't want to mess with my data
2: Generating a triangle mask, which I started like this
def triangleMask(triangles, pointmask):
maskedTris = np.zeros((triangles.shape[1]), dtype=np.bool)
maskedIdx = np.nonzero(pointmask)[0]
for i,t in enumerate(triangles.T):
if (i%5000) == 0:
print('working it.:', i)
for p in t:
if p in maskedIdx:
maskedTris[i] = True
break
return maskedTris
This works, but is not fast. And in my case, n = 250.000 and m = 500.000, so "not fast" is quite a problem.
I know there's a mask keyword in mlab, but I cant get it to work. Masking only the points in the triangular_mesh call yields and error since t then refers to indices which are larger than the size of p.
So you have a points array of shape (3, n), a triangles array of shape (3, m) and a point_mask boolean array of shape (n,), and would like to create a triangle_mask of shape (m,) holding True at position j if any of the indices in triangles[:, j] corresponds to a True in point_mask. You can do that with a little bit of fancy indexing:
triangle_mask = np.any(point_mask[triangles], axis=0)
To understand what's going on, point_mask[triangles] creates a boolean array of shape (3, m), with the value at position (i, j) being point_mask[triangles[i, j]].
Related
I'm basically trying to take the weighted mean of a 3D dataset, but only on a filtered subset of the data, where the filter is based off of another (2D) array. The shape of the 2D data matches the first 2 dimensions of the 3D data, and is thus repeated for each slice in the 3rd dimension.
Something like:
import numpy as np
myarr = np.array([[[4,6,8],[9,3,2]],[[2,7,4],[3,8,6]],[[1,6,7],[7,8,3]]])
myarr2 = np.array([[7,3],[6,7],[2,6]])
weights = np.random.rand(3,2,3)
filtered = []
for k in range(len(myarr[0,0,:])):
temp1 = myarr[:,:,k]
temp2 = weights[:,:,k]
filtered.append(temp1[np.where(myarr2 > 5)]*temp2[np.where(myarr2 > 5)])
average = np.array(np.sum(filtered,1)/len(filtered[0]))
I am concerned about efficiency here. Is it possible to vectorize this so I don't need the loop, or are there other suggestions to make this more efficient?
The most glaring efficiency issue, even the loop aside, is that np.where(...) is being called multiple times inside the loop, on the same condition! You can just do this a single time beforehand. Moreover, there is no need for a loop. Your operation basically equates to:
mask = myarr2 > 5
average = (myarr[mask] * weights[mask]).mean(axis=0)
There is no need for an np.where either.
myarr2 is an array of shape (i, j) with same first two dims as myarr and weight, which have some shape (i, j, k).
So if there are n True elements in the boolean mask myarr2 > 5, you can apply it on your other arrays to obtain (n, k) elements (taking all elements along third axis, when there is a True at a certain [i, j] position).
I am trying to calculate the average of a 3D array between two indices on the 1st axis. The start and end indices vary from cell to cell and are represented by two separate 2D arrays that are the same shape as a slice of the 3D array.
I have managed to implement a piece of code that loops through the pixels of my 3D array, but this method is painfully slow in the case of my array with a shape of (70, 550, 350). Is there a way to vectorise the operation using numpy or xarray (the arrays are stored in an xarray dataset)?
Here is a snippet of what I would like to optimise:
# My 3D raster containing values; shape = (time, x, y)
values = np.random.rand(10, 55, 60)
# A 2D raster containing start indices for the averaging
start_index = np.random.randint(0, 4, size=(values.shape[1], values.shape[2]))
# A 2D raster containing end indices for the averaging
end_index = np.random.randint(5, 9, size=(values.shape[1], values.shape[2]))
# Initialise an array that will contain results
mean_array = np.zeros_like(values[0, :, :])
# Loop over 3D raster to calculate the average between indices on axis 0
for i in range(0, values.shape[1]):
for j in range(0, values.shape[2]):
mean_array[i, j] = np.mean(values[start_index[i, j]: end_index[i, j], i, j], axis=0)
One way to do this without loops is to zero-out the entries you don't want to use, compute the sum of the remaining items, then divide by the number of nonzero entries. For example:
i = np.arange(values.shape[0])[:, None, None]
mean_array_2 = np.where((i >= start_index) & (i < end_index), values, 0).sum(0) / (end_index - start_index)
np.allclose(mean_array, mean_array_2)
# True
Note that this assumes that the indices are in the range 0 <= i < values.shape[0]; if this is not the case you can use np.clip or other means to standardize the indices before computation.
I've an image of about 8000x9000 size as a numpy matrix. I also have a list of indices in a numpy 2xn matrix. These indices are fractional as well as may be out of image size. I need to interpolate the image and find the values for the given indices. If the indices fall outside, I need to return numpy.nan for them. Currently I'm doing it in for loop as below
def interpolate_image(image: numpy.ndarray, indices: numpy.ndarray) -> numpy.ndarray:
"""
:param image:
:param indices: 2xN matrix. 1st row is dim1 (rows) indices, 2nd row is dim2 (cols) indices
:return:
"""
# Todo: Vectorize this
M, N = image.shape
num_indices = indices.shape[1]
interpolated_image = numpy.zeros((1, num_indices))
for i in range(num_indices):
x, y = indices[:, i]
if (x < 0 or x > M - 1) or (y < 0 or y > N - 1):
interpolated_image[0, i] = numpy.nan
else:
# Todo: Do Bilinear Interpolation. For now nearest neighbor is implemented
interpolated_image[0, i] = image[int(round(x)), int(round(y))]
return interpolated_image
But the for loop is taking huge amount of time (as expected). How can I vectorize this? I found scipy.interpolate.interp2d, but I'm not able to use it. Can someone explain how to use this or any other method is also fine. I also found this, but again it is not according to my requirements. Given x and y indices, these generated interpolated matrices. I don't want that. For the given indices, I just want the interpolated values i.e. I need a vector output. Not a matrix.
I tried like this, but as said above, it gives a matrix output
f = interpolate.interp2d(numpy.arange(image.shape[0]), numpy.arange(image.shape[1]), image, kind='linear')
interp_image_vect = f(indices[:,0], indices[:,1])
RuntimeError: Cannot produce output of size 73156608x73156608 (size too large)
For now, I've implemented nearest-neighbor interpolation. scipy interp2d doesn't have nearest neighbor. It would be good if the library function as nearest neighbor (so I can compare). If not, then also fine.
It looks like scipy.interpolate.RectBivariateSpline will do the trick:
from scipy.interpolate import RectBivariateSpline
image = # as given
indices = # as given
spline = RectBivariateSpline(numpy.arange(M), numpy.arange(N), image)
interpolated = spline(indices[0], indices[1], grid=False)
This gets you the interpolated values, but it doesn't give you nan where you need it. You can get that with where:
nans = numpy.zeros(interpolated.shape) + numpy.nan
x_in_bounds = (0 <= indices[0]) & (indices[0] < M)
y_in_bounds = (0 <= indices[1]) & (indices[1] < N)
bounded = numpy.where(x_in_bounds & y_in_bounds, interpolated, nans)
I tested this with a 2624x2624 image and 100,000 points in indices and all told it took under a second.
I want to remove the locations of bad pixels in my coordinate- and disparity-arrays. Therefore I wrote some code but it feels a bit circuitous and a little too long for the task. The main idea behind the code is that I want all array entries removed that contain a disparity value of -17. The same should happen for my pixel coordinate arrays of my 2000x2000 image.
Here is my code using a mask and flattened arrays. ( in the end I want 3 arrays containing x, y and the disparity value sorted in the same order not containing the entries and coordinates of the bad pixels)
Thanks for any hints that improve this code!
#define 2000x2000 index arrays
xyIdx = np.mgrid[0:disp.shape[0],0:disp.shape[1]]
xIdx = xyIdx[1]
yIdx = xyIdx[0]
#flatten indice-arrays and the disparity-array
xIdxFlat = xIdx.flatten()
yIdxFlat = yIdx.flatten()
dispFlat = disp.flatten()
#create mask with all values = 1 'true'
mask = np.ones(dispFlat.shape, dtype='bool')
#create false entrys in the mask wherever the minimum disparity or better
#said a bad pixel is located
for x in range(0,len(dispFlat)):
if dispFlat[x] == -17:
mask[x] = False
#mask the arrays and remove the entries that belong to the bad pixels
xCoords = np.zeros((xIdxFlat[mask].size), dtype='float64')
xCoords[:] = xIdxFlat[mask]
yCoords = np.zeros((yIdxFlat[mask].size), dtype='float64')
yCoords[:] = yIdxFlat[mask]
dispPoints = np.zeros((dispFlat[mask].size), dtype='float64')
dispPoints[:] = dispFlat[mask]
Create a mask of valid ones !=-17. Use this mask to get the valid row, col indices, which would be the X-Y coordinates. Finally index into the input array with the mask or the row, col indices for the filtered data array. Thus, you won't need to do all of that flattening business.
Hence, the implementation would be -
mask = disp != -17
yCoords, xCoords = np.where(mask) # or np.nonzero(mask)
dispPoints = disp[yCoords, xCoords] # or disp[mask]
Say I have a 3d numpy array:
i, j, k = 10, 3, 4
arr = np.arange(120).reshape(i, j, k)
and a 2d boolean array:
mask = np.random.random((j, k)) > 0.5
n = mask.sum()
I want to be able to extract the 1d arrays from arr along its 1st dimension which correspond with the True values of mask. The result should have shape, (i, n). How could this be done?
I pulling up some old code and for some reason I was doing arr[mask] but this gives a shape of (n, k) (I'm not sure why) and a warning:
VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 10949 but corresponding boolean dimension is 11
Simply mask along the last two axes -
arr[:,mask]