We Keep Coding

Image processing convolution: Why do I slide my kernel from np.arange(pad, imgWidth+pad)?

I am trying to learn kernel convolution for image processing. Now, I understand the concept of kernel convolution, but I am a bit confused about code that I have found for it at https://www.pyimagesearch.com/2016/07/25/convolutions-with-opencv-and-python/
Specifically, I am confused about the bounds in the for loops and the location of the convolution output.
def convolve(image, kernel):
# grab the spatial dimensions of the image, along with
# the spatial dimensions of the kernel
(iH, iW) = image.shape[:2]
(kH, kW) = kernel.shape[:2]
# allocate memory for the output image, taking care to
# "pad" the borders of the input image so the spatial
# size (i.e., width and height) are not reduced
pad = (kW - 1) // 2
image = cv2.copyMakeBorder(image, pad, pad, pad, pad,
cv2.BORDER_REPLICATE)
output = np.zeros((iH, iW), dtype="float32")
# loop over the input image, "sliding" the kernel across
# each (x, y)-coordinate from left-to-right and top to
# bottom
#QUESTION 1 SECTION BEGIN
for y in np.arange(pad, iH + pad):
for x in np.arange(pad, iW + pad):
# extract the ROI of the image by extracting the
# *center* region of the current (x, y)-coordinates
# dimensions
roi = image[y - pad:y + pad + 1, x - pad:x + pad + 1]
#QUESTION 1 SECTION END
# perform the actual convolution by taking the
# element-wise multiplication between the ROI and
# the kernel, then summing the matrix
k = (roi * kernel).sum()
#QUESTION 2 SECTION BEGIN
# store the convolved value in the output (x,y)-
# coordinate of the output image
output[y - pad, x - pad] = k
#QUESTION 2 SECTION END
Question 1: Why is np.arange from pad to iH+pad, and not from pad to iH-pad ? I assume that we start from pad so that the center pixel in the region of interest is never on the edge of the image. However, I would think that going to iH+pad would overshoot and have the center pixel end up outside of image dimensions.
Question 2: This code has us store the output pixel at a location to the left and up from where I centered my convolution roi, no ? If so, could someone explain the logic behind doing this for me?
Thank you!

np.arange(pad, iH + pad) runs over iH pixels, which is the width of the original input image. The padded image has a width of iH + 2*pad, so this is running from pad pixels from the beginning to pad pixels from the end of an image column, such that one can index up to pad pixels in both directions without exiting the padded image.
Regarding your second question: the input image was padded, the indexing is into the padded image. image[pad,pad] obtains the top-left pixel of the original image before padding, and corresponds to output[0,0]. output is not padded.

Mapping tensor in pytorch

I have the following two tensors:
img is a RGB image of shape (224,224,3)
uvs is a tensor with same spacial size e.g. (224, 224, 2) that maps to coordinates (x,y). In other words it provides (x,y) coordinates for every pixel of the input image.
I want to create now a new output image tensor that contains on index (x,y) the value of the input image. So the output should be an image as well with the pixels rearranged according to the mapping tensor.
Small toy example:
img = [[c1,c2], [c3, c4]] where c is a RGB color [r, g, b]
uvs = [[[0,0], [1,1]],[[0,1], [1,0]]]
out = [[c1, c3], [c4, c2]]
How would one achieve such a thing in pytorch in a fast vectorized manner?

Try with:
out = img[idx[...,0], idx[...,1]]

I was able to solve it (with the help of Quang Hoang answer)
out[idx[...,0], idx[...,1]] = img

What you need is torch.nn.functional.grid_sample(). You can do something like this:
width, height, channels = (224, 224, 3)
# Note that the image is channel-first (CHW format). In this example, I'm using a float image, so the values must be in the range (0, 1).
img = torch.rand((channels, height, width))
# Create the indices of shape (224, 224, 2). Any other size would work too.
col_indices = torch.arange(width, dtype=torch.float32)
row_indices = torch.arange(height, dtype=torch.float32)
uvs = torch.stack(torch.meshgrid([col_indices, row_indices]), dim=-1)
# Transform the indices from pixel coordiantes to the to the range [-1, 1] such that:
# * top-left corner of the input = (-1, -1)
# * bottom-right corner of the input = (1, 1)
# This is required for grid_sample() to work properly.
uvs[..., 0] = (uvs[..., 0] / width) * 2 - 1
uvs[..., 1] = (uvs[..., 1] / height)* 2 - 1
# Do the "mapping" operation (this does a bilinear interpolation) using `uvs` coordinates.
# Note that grid_sample() requires a batch dimension, so need to use `unsqueeze()`, then
# get rid of it using squeeze().
mapped = torch.nn.functional.grid_sample(
img.unsqueeze(0),
uvs.unsqueeze(0),
mode='bilinear',
align_corners=True,
)
# The final image is in HWC format.
result = mapped.squeeze(0).permute(1, 2, 0)
Side note: I found your question by searching for a solution for a related problem I had for a while. While I was writing an answer to you question, I realized what bug was causing the the problem I was facing. By helping you I effectively helped my self, so I hope this helps you! :)

Most optimized way to filter patch positions in an image

So my problem is this: I have an RGB image as a numpy array of dimensions (4086, 2048, 3), I split this image dimension into 96x96 patches and get back the positions of these patches in a numpy array. I always get 96x96 patches in every case. If the dimensions of the image can't allow me to create "pure" 96x96 patches on the x or y axis I just add a left padding to it so the last patches overlap a bit with the patch before it.
Now with these positions in hand I want to get rid of all 96x96 patches for which the RGB value is 255 in all three channels for every pixel in the patch, in the fastest way possible and I want to get back all the patches positions which don't have this value.
I would like to know:
What is the fastest way to extract the 96x96 patches positions from the image dimension? (for now I have a for loop)
How can you get rid of pure white patches (with value 255 on the 3 channels) in most optimal way? (for now I have a for loop)
I have a lot of these images to process like that with images resolution going up to (39706, 94762, 3) so my "for loops" becomes quickly inefficient here. Thanks for your help! (I take solutions which make use of the GPU too)
Here is the pseudo code to give you an idea on how it's done for now:
patches = []
patch_y = 0
y_limit = False
slide_width = 4086
slide_height = 2048
# Lets imagine this image_slide has 96x96 patches which value is 255
image_slide = np.random.rand(slide_width, slide_height, 3)
while patch_y < slide_height:
patch_x = 0
x_limit = False
while patch_x < slide_width:
# Extract the patch at the given position and return it or return None if it's 3 RGB
# channels are 255
is_white = PatchExtractor.is_white(patch_x, patch_y, image_slide)
# Add the patches position to the list if it's not None (not white)
if not is_white:
patches.append((patch_x, patch_y))
if not x_limit and patch_x + crop_size > slide_width - crop_size:
patch_x = slide_width - crop_size
x_limit = True
else:
patch_x += crop_size
if not y_limit and patch_y + crop_size > slide_height - crop_size:
patch_y = slide_height - crop_size
y_limit = True
else:
patch_y += crop_size
return patches
Ideally, I would like to get my patches positions outside a "for loop" then once I have them I can test if they are white or not outside a for loop as well with the fewer possible calls to numpy (so the code is processed in the C layer of numpy and doesn't go back and forth to python)

As you suspected you can vectorize all of what you're doing. It takes roughly a small integer multiple of the memory need of your original image. The algorithm is quite straightforward: pad your image so that an integer number of patches fit in it, cut it up into patches, check if each patch is all white, keep the rest:
import numpy as np
# generate some dummy data and shapes
imsize = (1024, 2048)
patchsize = 96
image = np.random.randint(0, 256, size=imsize + (3,), dtype=np.uint8)
# seed some white patches: cut a square hole in the random noise
image[image.shape[0]//2:3*image.shape[0]//2, image.shape[1]//2:3*image.shape[1]//2] = 255
# pad the image to necessary size; memory imprint similar size as the input image
# white pad for simplicity for now
nx,ny = (np.ceil(dim/patchsize).astype(int) for dim in imsize) # number of patches
if imsize[0] % patchsize or imsize[1] % patchsize:
# we need to pad along at least one dimension
padded = np.pad(image, ((0, nx * patchsize - imsize[0]),
(0, ny * patchsize - imsize[1]), (0,0)),
mode='constant', constant_values=255)
else:
# no padding needed
padded = image
# reshape padded image according to patches; doesn't copy memory
patched = padded.reshape(nx, patchsize, ny, patchsize, 3).transpose(0, 2, 1, 3, 4)
# patched is shape (nx, ny, patchsize, patchsize, 3)
# appending .copy() as a last step to the above will copy memory but might speed up
# the next step; time it to find out
# check for white patches; memory imprint the same size as the padded image
filt = ~(patched == 255).all((2, 3, 4))
# filt is a bool, one for each patch that tells us if it's _not_ all white
# (i.e. we want to keep it)
patch_x,patch_y = filt.nonzero() # patch indices of non-whites from 0 to nx-1, 0 to ny-1
patch_pixel_x = patch_x * patchsize # proper pixel indices of each pixel
patch_pixel_y = patch_y * patchsize
patches = np.array([patch_pixel_x, patch_pixel_y]).T
# shape (npatch, 2) which is compatible with a list of tuples
# if you want the actual patches as well:
patch_images = patched[filt, ...]
# shape (npatch, patchsize, patchsize, 3),
# patch_images[i,...] is an image with patchsize * patchsize pixels
As you can see, in the above I used white padding to get a congruent padded image. I believe this is in line with the philosophy of what you're trying to do. If you want to replicate what you're doing in the loop exactly, you can pad your image manually using the overlapping pixels that you'd take into account near the edge. You'd need to allocate a padded image of the right size, then manually slice the overlapping pixels of the original image in order to set the edge pixels in the padded result.
Since you mentioned that your images are huge and consequently padding leads to far too much memory use, you can avoid padding with some elbow grease. You can use slices of your huge image (which doesn't create a copy), but then you have to manually handle the edges where you don't have full slices. Here's how:
def get_patches(img, patchsize):
"""Compute patches on an input image without padding: assume "congruent" patches
Returns an array shaped (npatch, 2) of patch pixel positions"""
mx,my = (val//patchsize for val in img.shape[:-1])
patched = img[:mx*patchsize, :my*patchsize, :].reshape(mx, patchsize, my, patchsize, 3)
filt = ~(patched == 255).all((1, 3, 4))
patch_x,patch_y = filt.nonzero() # patch indices of non-whites from 0 to nx-1, 0 to ny-1
patch_pixel_x = patch_x * patchsize # proper pixel indices of each pixel
patch_pixel_y = patch_y * patchsize
patches = np.stack([patch_pixel_x, patch_pixel_y], axis=-1)
return patches
# fix the patches that fit inside the image
patches = get_patches(image, patchsize)
# fix edge patches if necessary
all_patches = [patches]
if imsize[0] % patchsize:
# then we have edge patches along the first dim
tmp_patches = get_patches(image[-patchsize:, ...], patchsize)
# correct indices
all_patches.append(tmp_patches + [imsize[0] - patchsize, 0])
if imsize[1] % patchsize:
# same along second dim
tmp_patches = get_patches(image[:, -patchsize:, :], patchsize)
# correct indices
all_patches.append(tmp_patches + [0, imsize[1] - patchsize])
if imsize[0] % patchsize and imsize[1] % patchsize:
# then we have a corner patch we still have to fix
tmp_patches = get_patches(image[-patchsize:, -patchsize:, :], patchsize)
# correct indices
all_patches.append(tmp_patches + [imsize[0] - patchsize, imsize[1] - patchsize])
# gather all the patches into an array of shape (npatch, 2)
patches = np.vstack(all_patches)
# if you also want to grab the actual patch values without looping:
xw, yw = np.mgrid[:patchsize, :patchsize]
patch_images = image[patches[:,0,None,None] + xw, patches[:,1,None,None] + yw, :]
# shape (npatch, patchsize, patchsize, 3),
# patch_images[i,...] is an image with patchsize * patchsize pixels
This will also exactly replicate your looping code, since we're explicitly taking the edge patches such that they overlap with the previous patches (there's no spurious white padding). If you want to have the patches in a given order you'll have to sort them now, though.

How to reshape a 3D numpy array?

I have a list of numpy arrays which are actually input images to my CNN. However size of each of my image is not cosistent, and my CNN takes only images which are of dimension 224X224. How do I reshape each of my image into the given dimension?
print(train_images[key].reshape(224, 224,3))
gives me an output
ValueError: total size of new array must be unchanged
I would be very grateful if anybody could help me with this.

New array should have the same amount of values when you are reshaping. What you need is cropping the picture (if it is bigger than 224x224) and padding (if it is smaller than 224x224) or resizing on both occasions.
Cropping is simply slicing with correct indexes:
def crop(np_img, size):
v_start = round((np_img.shape[0] - size[0]) / 2)
h_start = round((np_img.shape[1] - size[1]) / 2)
return np_img[v_start:v_start+size[1], h_start:h_start+size[0],:]
Padding is slightly more complex, this will create a zeros array in desired shape and plug in the values of image inside:
def pad_image(np_img, size):
v_start = round((size[0] - np_img.shape[0]) / 2)
h_start = round((size[1] - np_img.shape[1]) / 2)
result = np.zeros(size)
result[v_start:v_start+np_img.shape[1], h_start:h_start+np_img.shape[0], :] = np_img
return result
You can also use np.pad function for it:
def pad_image(np_img, size):
v_dif = size[0] - np_img.shape[0]
h_dif = size[1] - np_img.shape[1]
return np.lib.pad(np_img, ((v_dif, 0), (h_dif, 0), (0, 0)), 'constant', constant_values=(0))
You may realize padding is a bit different in two functions, I didn't want to over complicate the problem and just padded top and left on the second function. Did the both sides in first one since it was easier to calculate.
And finally for resizing, you better use another library. You can use scipy.misc.imresize, its pretty straightforward. This should do it:
imresize(np_img, size)

Here are a few ways I know to achieve this:
Since you're using python, you can use cv2.resize(), to resize the image to 224x224. The problem here is going to be distortions.
Scale the image to adjust to one of the required sizes (W=224 or H=224) and trim off whatever is extra. There is a loss of information here.
If you have the larger image, and a bounding box, use some delta to bounding box to maintain the aspect ratio and then resize down to the required size.
When you reshape a numpy array, the produce of the dimensions must match. If not, it'll throw a ValueError as you've got. There's no solution using reshape to solve your problem, AFAIK.

The standard way is to resize the image such that the smaller side is equal to 224 and then crop the image to 224x224. Resizing the image to 224x224 may distort the image and can lead to erroneous training. For example, a circle might become an ellipse if the image is not a square. It is important to maintain the original aspect ratio.

How to crop same size image patches with different locations from a stack of images?

Suppose I have an ndarray imgs of shape ( num_images, 3, width, height ) that stores a stack of num_images RGB images all of the same size.
I would like to slice/crop from each image a patch of shape ( 3, pw, ph ) but the center location of the patch is different for each image and is given in centers array of shape (num_images, 2).
Is there a nice/pythonic way of slicing imgs to get patches (of shape (num_images,3,pw,ph)) each patch is centered around its corresponding centers?
for simplicity it is safe to assume all patches fall within image boundaries.

Proper slicing is out of the question, because you need to access the underlying data on irregular intervals. You could get the crops with a single "fancy indexing" operation, but you'll need a (very) large indexing array. Therefor I think using a loop is easier and faster.
Compare the following two functions:
def fancy_indexing(imgs, centers, pw, ph):
n = imgs.shape[0]
img_i, RGB, x, y = np.ogrid[:n, :3, :pw, :ph]
corners = centers - [pw//2, ph//2]
x_i = x + corners[:,0,None,None,None]
y_i = y + corners[:,1,None,None,None]
return imgs[img_i, RGB, x_i, y_i]
def just_a_loop(imgs, centers, pw, ph):
crops = np.empty(imgs.shape[:2]+(pw,ph), imgs.dtype)
for i, (x,y) in enumerate(centers):
crops[i] = imgs[i,:,x-pw//2:x+pw//2,y-ph//2:y+ph//2]
return crops