We Keep Coding

Raster and vector analysis for profile extraction in python

I have quite complex problem and I have two options to solve it.
For a multiline shapefile (river) I would like to get cross profiles and extract DEM values for the lines.
I was thinking 1: Create ortogonal lines at defined step:
#Define a shp for the output features. Add a new field called 'M100' where the z-value of the line is stored to uniquely identify each profile
layerOut = outShp.CreateLayer('line_utm_neu_perp', layerRef, osgeo.ogr.wkbLineString)
layerDefn = layerOut.GetLayerDefn() # gets parameters of the current shapefile
layerOut.CreateField(ogr.FieldDefn('M100', ogr.OFTReal))
# Calculate the number of profiles/perpendicular lines to generate
n_prof = int(geomIn.Length()/spc)
# Define rotation vectors
rot_anti = np.array([[0, -1], [1, 0]])
rot_clock = np.array([[0, 1], [-1, 0]])
# Start iterating along the line
for prof in range(1, n_prof):
# Get the start, mid and end points for this segment
seg_st = geomIn.GetPoint(prof-1) # (x, y, z)
seg_mid = geomIn.GetPoint(prof)
seg_end = geomIn.GetPoint(prof+1)
# Get a displacement vector for this segment
vec = np.array([[seg_end[0] - seg_st[0],], [seg_end[1] - seg_st[1],]])
# Rotate the vector 90 deg clockwise and 90 deg counter clockwise
vec_anti = np.dot(rot_anti, vec)
vec_clock = np.dot(rot_clock, vec)
# Normalise the perpendicular vectors
len_anti = ((vec_anti**2).sum())**0.5
vec_anti = vec_anti/len_anti
len_clock = ((vec_clock**2).sum())**0.5
vec_clock = vec_clock/len_clock
# Scale them up to the profile length
vec_anti = vec_anti*sect_len
vec_clock = vec_clock*sect_len
# Calculate displacements from midpoint
prof_st = (seg_mid[0] + float(vec_anti[0]), seg_mid[1] + float(vec_anti[1]))
prof_end = (seg_mid[0] + float(vec_clock[0]), seg_mid[1] + float(vec_clock[1]))
# Write to output
geomLine = ogr.Geometry(ogr.wkbLineString)
geomLine.AddPoint(prof_st[0],prof_st[1])
geomLine.AddPoint(prof_end[0],prof_end[1])
featureLine = ogr.Feature(layerDefn)
featureLine.SetGeometry(geomLine)
featureLine.SetFID(prof)
featureLine.SetField('M100',round(seg_mid[2],1))
layerOut.CreateFeature(featureLine)
Problem here is that it works on one line only and not on multiline.
2 option could be creating parallel lines with offset and extract values at the same distance from the start. But I tried it only once and it did not work on my objects.
z = shapefile.offset_curve(10.0,'left')
But here I do not know what object to pass in order to make it work. Also I was thinking about creating buffer and extracting values of raster.
I will be grateful for any suggestions.

Slicing 2D numpy array periodically

I have a numpy array of 300x300 where I want to keep all elements periodically. Specifically, for both axes I want to keep the first 5 elements, then discard 15, keep 5, discard 15, etc. This should result in an array of 75x75 elements. How can this be done?

You can created a 1D mask, that carries out the keep/discard function, and then repeat the mask and apply the mask to the array. Here is an example.
import numpy as np
size = 300
array = np.arange(size).reshape((size, 1)) * np.arange(size).reshape((1, size))
mask = np.concatenate((np.ones(5), np.zeros(15))).astype(bool)
period = len(mask)
mask = np.repeat(mask.reshape((1, period)), repeats=size // period, axis=0)
mask = np.concatenate(mask, axis=0)
result = array[mask][:, mask]
print(result.shape)

You can view the array as series of 20x20 blocks, of which you want to keep the upper-left 5x5 portion. Let's say you have
keep = 5
discard = 15
This only works if
assert all(s % (keep + discard) == 0 for s in arr.shape)
First compute the shape of the view and use it:
block = keep + discard
shape1 = (arr.shape[0] // block, block, arr.shape[1] // block, block)
view = arr.reshape(shape1)[:, :keep, :, :keep]
The following operation will create a copy of the data because the view creates a non-contiguous buffer:
shape2 = (shape1[0] * keep, shape1[2] * keep)
result = view.reshape(shape2)
You can compute shape1 and shape2 in a more general manner with something like
shape1 = tuple(
np.stack((np.array(arr.shape) // block,
np.full(arr.ndim, block)), -1).ravel())
shape2 = tuple(np.array(shape1[::2]) * keep)
I would recommend packaging this into a function.

Here is my first thought of a solution. Will update later if I think of one with fewer lines. This should work even if the input is not square:
output = []
for i in range(len(arr)):
tmp = []
if i % (15+5) < 5: # keep first 5, then discard next 15
for j in range(len(arr[i])):
if j % (15+5) < 5: # keep first 5, then discard next 15
tmp.append(arr[i,j])
output.append(tmp)
Update:
Building off of Yang's answer, here is another way which uses np.tile, which repeats an array a given number of times along each axis. This relies on the input array being square in dimension.
import numpy as np
# Define one instance of the keep/discard box
keep, discard = 5, 15
mask = np.concatenate([np.ones(keep), np.zeros(discard)])
mask_2d = mask.reshape((keep+discard,1)) * mask.reshape((1,keep+discard))
# Tile it out -- overshoot, then trim to match size
count = len(arr)//len(mask_2d) + 1
tiled = np.tile(mask_2d, [count,count]).astype('bool')
tiled = tiled[:len(arr), :len(arr)]
# Apply the mask to the input array
dim = sum(tiled[0])
output = arr[tiled].reshape((dim,dim))

Another option using meshgrid and a modulo:
# MyArray = 300x300 numpy array
r = np.r_[0:300] # A slide from 0->300
xv, yv = np.meshgrid(r, r) # x and y grid
mask = ((xv%20)<5) & ((yv%20)<5) # We create the boolean mask
result = MyArray[mask].reshape((75,75)) # We apply the mask and reshape the final output

torch.rfft - fft-based convolution creating different output than spatial convolution

I implemented FFT-based convolution in Pytorch and compared the result with spatial convolution via conv2d() function. The convolution filter used is an average filter. The conv2d() function produced smoothened output due to average filtering as expected but the fft-based convolution returned a more blurry output.
I have attached the code and outputs here -
spatial convolution -
from PIL import Image, ImageOps
import torch
from matplotlib import pyplot as plt
from torchvision.transforms import ToTensor
import torch.nn.functional as F
import numpy as np
im = Image.open("/kaggle/input/tiger.jpg")
im = im.resize((256,256))
gray_im = im.convert('L')
gray_im = ToTensor()(gray_im)
gray_im = gray_im.squeeze()
fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])
conv_gray_im = gray_im.unsqueeze(0).unsqueeze(0)
conv_fil = fil.unsqueeze(0).unsqueeze(0)
conv_op = F.conv2d(conv_gray_im,conv_fil)
conv_op = conv_op.squeeze()
plt.figure()
plt.imshow(conv_op, cmap='gray')
FFT-based convolution -
def fftshift(image):
sh = image.shape
x = np.arange(0, sh[2], 1)
y = np.arange(0, sh[3], 1)
xm, ym = np.meshgrid(x,y)
shifter = (-1)**(xm + ym)
shifter = torch.from_numpy(shifter)
return image*shifter
shift_im = fftshift(conv_gray_im)
padded_fil = F.pad(conv_fil, (0, gray_im.shape[0]-fil.shape[0], 0, gray_im.shape[1]-fil.shape[1]))
shift_fil = fftshift(padded_fil)
fft_shift_im = torch.rfft(shift_im, 2, onesided=False)
fft_shift_fil = torch.rfft(shift_fil, 2, onesided=False)
shift_prod = fft_shift_im*fft_shift_fil
shift_fft_conv = fftshift(torch.irfft(shift_prod, 2, onesided=False))
fft_op = shift_fft_conv.squeeze()
plt.figure('shifted fft')
plt.imshow(fft_op, cmap='gray')
original image -
spatial convolution output -
fft-based convolution output -
Could someone kindly explain the issue?

The main problem with your code is that Torch doesn't do complex numbers, the output of its FFT is a 3D array, with the 3rd dimension having two values, one for the real component and one for the imaginary. Consequently, the multiplication does not do a complex multiplication.
There currently is no complex multiplication defined in Torch (see this issue), we'll have to define our own.
A minor issue, but also important if you want to compare the two convolution operations, is the following:
The FFT takes the origin of its input in the first element (top-left pixel for an image). To avoid a shifted output, you need to generate a padded kernel where the origin of the kernel is the top-left pixel. This is quite tricky, actually...
Your current code:
fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])
conv_fil = fil.unsqueeze(0).unsqueeze(0)
padded_fil = F.pad(conv_fil, (0, gray_im.shape[0]-fil.shape[0], 0, gray_im.shape[1]-fil.shape[1]))
generates a padded kernel where the origin is in pixel (1,1), rather than (0,0). It needs to be shifted by one pixel in each direction. NumPy has a function roll that is useful for this, I don't know the Torch equivalent (I'm not at all familiar with Torch). This should work:
fil = torch.tensor([[1/9,1/9,1/9],[1/9,1/9,1/9],[1/9,1/9,1/9]])
padded_fil = fil.unsqueeze(0).unsqueeze(0).numpy()
padded_fil = np.pad(padded_fil, ((0, gray_im.shape[0]-fil.shape[0]), (0, gray_im.shape[1]-fil.shape[1])))
padded_fil = np.roll(padded_fil, -1, axis=(0, 1))
padded_fil = torch.from_numpy(padded_fil)
Finally, your fftshift function, applied to the spatial-domain image, causes the frequency-domain image (the result of the FFT applied to the image) to be shifted such that the origin is in the middle of the image, rather than the top-left. This shift is useful when looking at the output of the FFT, but is pointless when computing the convolution.
Putting these things together, the convolution is now:
def complex_multiplication(t1, t2):
real1, imag1 = t1[:,:,0], t1[:,:,1]
real2, imag2 = t2[:,:,0], t2[:,:,1]
return torch.stack([real1 * real2 - imag1 * imag2, real1 * imag2 + imag1 * real2], dim = -1)
fft_im = torch.rfft(gray_im, 2, onesided=False)
fft_fil = torch.rfft(padded_fil, 2, onesided=False)
fft_conv = torch.irfft(complex_multiplication(fft_im, fft_fil), 2, onesided=False)
Note that you can do one-sided FFTs to save a bit of computation time:
fft_im = torch.rfft(gray_im, 2, onesided=True)
fft_fil = torch.rfft(padded_fil, 2, onesided=True)
fft_conv = torch.irfft(complex_multiplication(fft_im, fft_fil), 2, onesided=True, signal_sizes=gray_im.shape)
Here the frequency domain is about half the size as in the full FFT, but it is only redundant parts that are left out. The result of the convolution is unchanged.

Plot a gamut in cie1931 colour space Python 2.7

Gamut I want to plot in CIE1931 space: https://www.google.co.uk/search?biw=1337&bih=1257&tbm=isch&sa=1&ei=9x3kW7rqBo3ygQb-8aWYBw&q=viewpixx+gamut&oq=viewpixx+gamut&gs_l=img.3...2319.2828.0.3036.5.5.0.0.0.0.76.270.5.5.0....0...1c.1.64.img..0.0.0....0.KT8w80tcZik#imgrc=77Ufw31S6UVlYM
I want to create a triangle plot of the ciexyY colours within the these coordinates: (.119,.113),(.162,.723),(.695,.304) as in the image - with a set luminance Y at 30.0.
I have created a 3d array of xy values between 0-1.
I then created a matrix with 1s inside the triangle and 0s outside the triangle.
I multiplied the triangle matrix by the xyY ndarray.
Then I looped through the xyY ndarray and converted xyY values to rgb, and displayed them.
The result is somewhat close but not correct. I think the error is in the last section when I convert to rgb, but I'm not sure why. This is the current image: https://imgur.com/a/7cWY0FI. Any recommendations would be really appreciated.
from __future__ import division
import numpy as np
from colormath.color_objects import sRGBColor, xyYColor
from colormath.color_conversions import convert_color
import matplotlib.pyplot as plt
def frange(x,y,jump):
while x < y:
yield x
x += jump
def onSameSide(p1,p2, A,B):
cp1 = np.cross(B-A, p1-A)
cp2 = np.cross(B-A, p2-A)
if(np.dot(cp1, cp2) >= 0):
return True
else:
return False
def isPointInTriangle(p,A,B,C):
if(onSameSide(p,A,B,C) and onSameSide(p,B,A,C) and onSameSide(p,C,A,B)):
return True
else:
return False
xlen = 400
ylen = 400
#CIExyY colour space
#Make an array (1,1,3) with each plane representing how x,y,Y vary in the coordinate space
ciexyY = np.zeros((3,xlen,ylen))
ciexyY[2,:,:]=30.0
for x in frange(0,1,1/xlen):
ciexyY[0,:,int(xlen*x)]=x
for y in frange(0,1,1/xlen):
ciexyY[1,int(ylen*y),:]=y
#coordinates from Viewpixx gamut, scaled up to 100
blue=np.array((.119,.113,30.0))
green=np.array((.162,.723,30.0))
red=np.array((.695,.304,30.0))
#scale up to size of image
blue = np.multiply(blue,xlen)
green = np.multiply(green,xlen)
red = np.multiply(red,xlen)
#make an array of zeros and ones to plot the shape of Viewpixx triangle
triangleZeros = np.zeros((xlen,ylen))
for x in frange(0,xlen,1):
for y in frange(0,ylen,1):
if(isPointInTriangle((x,y,0),blue,green,red)):
triangleZeros[x,y]=1
else:
triangleZeros[x,y]=0
#cieTriangle
cieTriangle = np.multiply(ciexyY,triangleZeros)
#convert cieTriangle xyY to rgb
rgbTriangle = np.zeros((3,xlen,ylen))
for x in frange(0,xlen,1):
for y in range(0,ylen,1):
xyYcolour = xyYColor(cieTriangle[0,x,y],cieTriangle[1,x,y],cieTriangle[2,x,y])
rgbColour = convert_color(xyYcolour,sRGBColor)
rgbTriangle[0,x,y] = rgbColour.rgb_r
rgbTriangle[1,x,y] = rgbColour.rgb_g
rgbTriangle[2,x,y] = rgbColour.rgb_b
rgbTriangle = np.transpose(rgbTriangle)
plt.imshow(rgbTriangle)
plt.show()

We have all the common Chromaticity Diagrams in Colour, I would recommend it over python-colormath because Colour is vectorised and thus much faster.
Do you have a render of your current image to share though?

from colour.plotting import plot_chromaticity_diagram_CIE1931
plot_chromaticity_diagram_CIE1931()

Shape recognition with numpy/scipy (perhaps watershed)

My goal is to trace drawings that have a lot of separate shapes in them and to split these shapes into individual images. It is black on white. I'm quite new to numpy,opencv&co - but here is my current thought:
scan for black pixels
black pixel found -> watershed
find watershed boundary (as polygon path)
continue searching, but ignore points within the already found boundaries
I'm not very good at these kind of things, is there a better way?
First I tried to find the rectangular bounding box of the watershed results (this is more or less a collage of examples):
from numpy import *
import numpy as np
from scipy import ndimage
np.set_printoptions(threshold=np.nan)
a = np.zeros((512, 512)).astype(np.uint8) #unsigned integer type needed by watershed
y, x = np.ogrid[0:512, 0:512]
m1 = ((y-200)**2 + (x-100)**2 < 30**2)
m2 = ((y-350)**2 + (x-400)**2 < 20**2)
m3 = ((y-260)**2 + (x-200)**2 < 20**2)
a[m1+m2+m3]=1
markers = np.zeros_like(a).astype(int16)
markers[0, 0] = 1
markers[200, 100] = 2
markers[350, 400] = 3
markers[260, 200] = 4
res = ndimage.watershed_ift(a.astype(uint8), markers)
unique(res)
B = argwhere(res.astype(uint8))
(ystart, xstart), (ystop, xstop) = B.min(0), B.max(0) + 1
tr = a[ystart:ystop, xstart:xstop]
print tr
Somehow, when I use the original array (a) then argwhere seems to work, but after the watershed (res) it just outputs the complete array again.
The next step could be to find the polygon path around the shape, but the bounding box would be great for now!
Please help!

#Hooked has already answered most of your question, but I was in the middle of writing this up when he answered, so I'll post it in the hopes that it's still useful...
You're trying to jump through a few too many hoops. You don't need watershed_ift.
You use scipy.ndimage.label to differentiate separate objects in a boolean array and scipy.ndimage.find_objects to find the bounding box of each object.
Let's break things down a bit.
import numpy as np
from scipy import ndimage
import matplotlib.pyplot as plt
def draw_circle(grid, x0, y0, radius):
ny, nx = grid.shape
y, x = np.ogrid[:ny, :nx]
dist = np.hypot(x - x0, y - y0)
grid[dist < radius] = True
return grid
# Generate 3 circles...
a = np.zeros((512, 512), dtype=np.bool)
draw_circle(a, 100, 200, 30)
draw_circle(a, 400, 350, 20)
draw_circle(a, 200, 260, 20)
# Label the objects in the array.
labels, numobjects = ndimage.label(a)
# Now find their bounding boxes (This will be a tuple of slice objects)
# You can use each one to directly index your data.
# E.g. a[slices[0]] gives you the original data within the bounding box of the
# first object.
slices = ndimage.find_objects(labels)
#-- Plotting... -------------------------------------
fig, ax = plt.subplots()
ax.imshow(a)
ax.set_title('Original Data')
fig, ax = plt.subplots()
ax.imshow(labels)
ax.set_title('Labeled objects')
fig, axes = plt.subplots(ncols=numobjects)
for ax, sli in zip(axes.flat, slices):
ax.imshow(labels[sli], vmin=0, vmax=numobjects)
tpl = 'BBox:\nymin:{0.start}, ymax:{0.stop}\nxmin:{1.start}, xmax:{1.stop}'
ax.set_title(tpl.format(*sli))
fig.suptitle('Individual Objects')
plt.show()
Hopefully that makes it a bit clearer how to find the bounding boxes of the objects.

Use the ndimage library from scipy. The function label places a unique tag on each block of pixels that are within a threshold. This identifies the unique clusters (shapes). Starting with your definition of a:
from scipy import ndimage
image_threshold = .5
label_array, n_features = ndimage.label(a>image_threshold)
# Plot the resulting shapes
import pylab as plt
plt.subplot(121)
plt.imshow(a)
plt.subplot(122)
plt.imshow(label_array)
plt.show()