We Keep Coding

Overlapping chunks in Xarray dataset for Kernel operations

I try to run a 9x9 pixel kernel across a large satellite image with a custom filter. One satellite scene has ~ 40 GB and to fit it into my RAM, I'm using xarrays options to chunk my dataset with dask.
My filter includes a check if the kernel is complete (i.e. not missing data at the edge of the image). In that case a NaN is returned to prevent a potential bias (and I don't really care about the edges). I now realized, that this introduces not only NaNs at the edges of the image (expected behaviour), but also along the edges of each chunk, because the chunks don't overlap. dask provides options to create chunks with an overlap, but are there any comparable capabilities in xarray? I found this issue, but it doesn't seem like there has been any progress in this regard.
Some sample code (shortened version of my original code):
import numpy as np
import numba
import math
import xarray as xr
#numba.jit("f4[:,:](f4[:,:],i4)", nopython = True)
def water_anomaly_filter(input_arr, window_size = 9):
# check if window size is odd
if window_size%2 == 0:
raise ValueError("Window size must be odd!")
# prepare an output array with NaNs and the same dtype as the input
output_arr = np.zeros_like(input_arr)
output_arr[:] = np.nan
# calculate how many pixels in x and y direction around the center pixel
# are in the kernel
pix_dist = math.floor(window_size/2-0.5)
# create a dummy weight matrix
weights = np.ones((window_size, window_size))
# get the shape of the input array
xn,yn = input_arr.shape
# iterate over the x axis
for x in range(xn):
# determine limits of the kernel in x direction
xmin = max(0, x - pix_dist)
xmax = min(xn, x + pix_dist+1)
# iterate over the y axis
for y in range(yn):
# determine limits of the kernel in y direction
ymin = max(0, y - pix_dist)
ymax = min(yn, y + pix_dist+1)
# extract data values inside the kernel
kernel = input_arr[xmin:xmax, ymin:ymax]
# if the kernel is complete (i.e. not at image edge...) and it
# is not all NaN
if kernel.shape == weights.shape and not np.isnan(kernel).all():
# apply the filter. In this example simply keep the original
# value
output_arr[x,y] = input_arr[x,y]
return output_arr
def run_water_anomaly_filter_xr(xds, var_prefix = "band",
window_size = 9):
variables = [x for x in list(xds.variables) if x.startswith(var_prefix)]
for var in variables[:2]:
xds[var].values = water_anomaly_filter(xds[var].values,
window_size = window_size)
return xds
def create_test_nc():
data = np.random.randn(1000, 1000).astype(np.float32)
rows = np.arange(54, 55, 0.001)
cols = np.arange(10, 11, 0.001)
ds = xr.Dataset(
data_vars=dict(
band_1=(["x", "y"], data)
),
coords=dict(
lon=(["x"], rows),
lat=(["y"], cols),
),
attrs=dict(description="Testdata"),
)
ds.to_netcdf("test.nc")
if __name__ == "__main__":
# if required, create test data
create_test_nc()
# import data
with xr.open_dataset("test.nc",
chunks = {"x": 50,
"y": 50},
) as xds:
xds_2 = xr.map_blocks(run_water_anomaly_filter_xr,
xds,
template = xds).compute()
xds_2["band_1"][:200,:200].plot()
This yields:
enter image description here
You can clearly see the rows and columns of NaNs along the edges of each chunk.
I'm happy for any suggestions. I would love to get the overlapping chunks (or any other solution) within xarray, but I'm also open for other solutions.

You can use Dask's map_blocks as follows:
arr = dask.array.map_overlap(
water_anomaly_filter, xds.band_1.data, dtype='f4', depth=4, window_size=9
).compute()
da = xr.DataArray(arr, dims=xds.band_1.dims, coords=xds.band_1.coords)
Note that you will likely want to tune depth and window_size for your specific application.

Cross-correlation between images

I'd like to compute the cross correlation using de Fast Fourier Transform, for cloud motion tracking following the steps of the image below.
def roi_image(image):
image = cv.imread(image, 0)
roi = image[700:900, 1900:2100]
return roi
def FouTransf(image):
img_f32 = np.float32(image)
d_ft = cv.dft(img_f32, flags = cv.DFT_COMPLEX_OUTPUT)
d_ft_shift = np.fft.fftshift(d_ft)
rows, cols = image.shape
opt_rows = cv.getOptimalDFTSize(rows)
opt_cols = cv.getOptimalDFTSize(cols)
opt_img = np.zeros((opt_rows, opt_cols))
opt_img[:rows, :cols] = image
crow, ccol = opt_rows / 2 , opt_cols / 2
mask = np.zeros((opt_rows, opt_cols, 2), np.uint8)
mask[int(crow-50):int(crow+50), int(ccol-50):int(ccol+50)] = 1
f_mask = d_ft_shift*mask
return f_mask
def inv_FouTransf(image):
f_ishift = np.fft.ifftshift(image)
img_back = cv.idft(f_ishift)
img_back = cv.magnitude(img_back[:, :, 0], img_back[:, :, 1])
return img_back
def rms(sigma):
rms = np.std(sigma)
return rms
# Step 1: Import images
a = roi_image(path_a)
b = roi_image(path_b)
# Step 2: Convert the image to frequency domain
G_t0 = FouTransf(a)
G_t0_conj = G_t0.conj()
G_t1 = FouTransf(b)
# Step 3: Compute C(m, v)
C = G_t0_conj * G_t1
# Step 4: Convert the image to space domain to obtain Cov (p, q)
c_w = inv_FouTransf(C)
# Step 5: Compute Cross correlation
R_pq = c_w / (rms(a) * rms(b))
I'm a little confused because I've never use that technique. ¿The application es accurate?
HINT: eq (1) is : R(p,q) = Cov(p,q) / (sigma_t0 * sigma_t1). If more information is required the paper is: "An Automated Techinique or Obtaining Cloud Motion from Geostatiory Satellite Data Using Cross Correlation".
I found this source but I don't know if does something I'm trying.

If you are trying to do something similar to cv2.matchTemplate(), a working python implementation of the Normalized Cross-Correlation (NCC) method can be found in this repository:
########################################################################################
# Author: Ujash Joshi, University of Toronto, 2017 #
# Based on Octave implementation by: Benjamin Eltzner, 2014 <b.eltzner#gmx.de> #
# Octave/Matlab normxcorr2 implementation in python 3.5 #
# Details: #
# Normalized cross-correlation. Similiar results upto 3 significant digits. #
# https://github.com/Sabrewarrior/normxcorr2-python/master/norxcorr2.py #
# http://lordsabre.blogspot.ca/2017/09/matlab-normxcorr2-implemented-in-python.html #
########################################################################################
import numpy as np
from scipy.signal import fftconvolve
def normxcorr2(template, image, mode="full"):
"""
Input arrays should be floating point numbers.
:param template: N-D array, of template or filter you are using for cross-correlation.
Must be less or equal dimensions to image.
Length of each dimension must be less than length of image.
:param image: N-D array
:param mode: Options, "full", "valid", "same"
full (Default): The output of fftconvolve is the full discrete linear convolution of the inputs.
Output size will be image size + 1/2 template size in each dimension.
valid: The output consists only of those elements that do not rely on the zero-padding.
same: The output is the same size as image, centered with respect to the ‘full’ output.
:return: N-D array of same dimensions as image. Size depends on mode parameter.
"""
# If this happens, it is probably a mistake
if np.ndim(template) > np.ndim(image) or \
len([i for i in range(np.ndim(template)) if template.shape[i] > image.shape[i]]) > 0:
print("normxcorr2: TEMPLATE larger than IMG. Arguments may be swapped.")
template = template - np.mean(template)
image = image - np.mean(image)
a1 = np.ones(template.shape)
# Faster to flip up down and left right then use fftconvolve instead of scipy's correlate
ar = np.flipud(np.fliplr(template))
out = fftconvolve(image, ar.conj(), mode=mode)
image = fftconvolve(np.square(image), a1, mode=mode) - \
np.square(fftconvolve(image, a1, mode=mode)) / (np.prod(template.shape))
# Remove small machine precision errors after subtraction
image[np.where(image < 0)] = 0
template = np.sum(np.square(template))
out = out / np.sqrt(image * template)
# Remove any divisions by 0 or very close to 0
out[np.where(np.logical_not(np.isfinite(out)))] = 0
return out
The returned object from normxcorr2() is the cross correlation matrix.

Plot a gamut in cie1931 colour space Python 2.7

Gamut I want to plot in CIE1931 space: https://www.google.co.uk/search?biw=1337&bih=1257&tbm=isch&sa=1&ei=9x3kW7rqBo3ygQb-8aWYBw&q=viewpixx+gamut&oq=viewpixx+gamut&gs_l=img.3...2319.2828.0.3036.5.5.0.0.0.0.76.270.5.5.0....0...1c.1.64.img..0.0.0....0.KT8w80tcZik#imgrc=77Ufw31S6UVlYM
I want to create a triangle plot of the ciexyY colours within the these coordinates: (.119,.113),(.162,.723),(.695,.304) as in the image - with a set luminance Y at 30.0.
I have created a 3d array of xy values between 0-1.
I then created a matrix with 1s inside the triangle and 0s outside the triangle.
I multiplied the triangle matrix by the xyY ndarray.
Then I looped through the xyY ndarray and converted xyY values to rgb, and displayed them.
The result is somewhat close but not correct. I think the error is in the last section when I convert to rgb, but I'm not sure why. This is the current image: https://imgur.com/a/7cWY0FI. Any recommendations would be really appreciated.
from __future__ import division
import numpy as np
from colormath.color_objects import sRGBColor, xyYColor
from colormath.color_conversions import convert_color
import matplotlib.pyplot as plt
def frange(x,y,jump):
while x < y:
yield x
x += jump
def onSameSide(p1,p2, A,B):
cp1 = np.cross(B-A, p1-A)
cp2 = np.cross(B-A, p2-A)
if(np.dot(cp1, cp2) >= 0):
return True
else:
return False
def isPointInTriangle(p,A,B,C):
if(onSameSide(p,A,B,C) and onSameSide(p,B,A,C) and onSameSide(p,C,A,B)):
return True
else:
return False
xlen = 400
ylen = 400
#CIExyY colour space
#Make an array (1,1,3) with each plane representing how x,y,Y vary in the coordinate space
ciexyY = np.zeros((3,xlen,ylen))
ciexyY[2,:,:]=30.0
for x in frange(0,1,1/xlen):
ciexyY[0,:,int(xlen*x)]=x
for y in frange(0,1,1/xlen):
ciexyY[1,int(ylen*y),:]=y
#coordinates from Viewpixx gamut, scaled up to 100
blue=np.array((.119,.113,30.0))
green=np.array((.162,.723,30.0))
red=np.array((.695,.304,30.0))
#scale up to size of image
blue = np.multiply(blue,xlen)
green = np.multiply(green,xlen)
red = np.multiply(red,xlen)
#make an array of zeros and ones to plot the shape of Viewpixx triangle
triangleZeros = np.zeros((xlen,ylen))
for x in frange(0,xlen,1):
for y in frange(0,ylen,1):
if(isPointInTriangle((x,y,0),blue,green,red)):
triangleZeros[x,y]=1
else:
triangleZeros[x,y]=0
#cieTriangle
cieTriangle = np.multiply(ciexyY,triangleZeros)
#convert cieTriangle xyY to rgb
rgbTriangle = np.zeros((3,xlen,ylen))
for x in frange(0,xlen,1):
for y in range(0,ylen,1):
xyYcolour = xyYColor(cieTriangle[0,x,y],cieTriangle[1,x,y],cieTriangle[2,x,y])
rgbColour = convert_color(xyYcolour,sRGBColor)
rgbTriangle[0,x,y] = rgbColour.rgb_r
rgbTriangle[1,x,y] = rgbColour.rgb_g
rgbTriangle[2,x,y] = rgbColour.rgb_b
rgbTriangle = np.transpose(rgbTriangle)
plt.imshow(rgbTriangle)
plt.show()

We have all the common Chromaticity Diagrams in Colour, I would recommend it over python-colormath because Colour is vectorised and thus much faster.
Do you have a render of your current image to share though?

from colour.plotting import plot_chromaticity_diagram_CIE1931
plot_chromaticity_diagram_CIE1931()

matplotlib argrelmax doesn't find all maxes

I have a project where I'm sampling analog data and attempting to analyze with matplotlib. Currently, my analog data source is a potentiometer hooked up to a microcontroller, but that's not really relevant to the issue. Here's my code
arrayFront = RunningMean(array(dataFront), 15)
arrayRear = RunningMean(array(dataRear), 15)
x = linspace(0, len(arrayFront), len(arrayFront)) # Generate x axis
y = linspace(0, len(arrayRear), len(arrayRear)) # Generate x axis
min_vals_front = scipy.signal.argrelmin(arrayFront, order=2)[0] # Min
min_vals_rear = scipy.signal.argrelmin(arrayRear, order=2)[0] # Min
max_vals_front = scipy.signal.argrelmax(arrayFront, order=2)[0] # Max
max_vals_rear = scipy.signal.argrelmax(arrayRear, order=2)[0] # Max
maxvalfront = max(arrayFront[max_vals_front])
maxvalrear = max(arrayRear[max_vals_rear])
minvalfront = min(arrayFront[min_vals_front])
minvalrear = min(arrayRear[min_vals_rear])
plot(x, arrayFront, label="Front Pressures")
plot(y, arrayRear, label="Rear Pressures")
plot(x[min_vals_front], arrayFront[min_vals_front], "x")
plot(x[max_vals_front], arrayFront[max_vals_front], "o")
plot(y[min_vals_rear], arrayRear[min_vals_rear], "x")
plot(y[max_vals_rear], arrayRear[max_vals_rear], "o")
xlim(-25, len(arrayFront) + 25)
ylim(-1000, 7000)
legend(loc='upper left')
show()
dataFront and dataRear are python lists that hold the sampled data from 2 potentiometers. RunningMean is a function that calls:
convolve(x, ones((N,)) / N, mode='valid')
The problem is that the argrelmax (and min) functions don't always find all the maxes and mins. Sometimes it doesn't find ANY max or mins, and that causes me problems in this block of code
maxvalfront = max(arrayFront[max_vals_front])
maxvalrear = max(arrayRear[max_vals_rear])
minvalfront = min(arrayFront[min_vals_front])
minvalrear = min(arrayRear[min_vals_rear])
because the [min_vals_(blank)] variables are empty. Does anyone have any idea what is happening here, and what I can do to fix the problem? Thanks in advance.
Here's one of graphs of data where not all the maxes and mins are found:

signal.argrelmin is a thin wrapper around signal.argrelextrema with comparator=np.less. np.less(a, b) returns the truth value of a < b element-wise. Notice that np.less requires a to be strictly less than b for it to be True.
Your data has the same minimum value at a lot of neighboring locations. At the local minima, the inequality between local minimum and its neighbors does not satisfy a strictly less than relationship; instead it only satisfies a strictly less than or equal to relationship.
Therefore, to find these extrema use signal.argrelmin with comparator=np.less_equal. For example, using a snippet from your data:
import numpy as np
from scipy import signal
arrayRear = np.array([-624.59309896, -624.59309896, -624.59309896,
-625., -625., -625.,])
print(signal.argrelmin(arrayRear, order=2)[0])
# []
print(signal.argrelextrema(arrayRear, np.less_equal)[0])
# [0 1 3 4 5]
print(signal.argrelextrema(arrayRear, np.less_equal, order=2)[0])
# [0 3 4 5]

Python 2D array -- How to plug in x and retrieve y value?

I have been looking for an answer since yesterday but no luck. So I have a 1D spectrum (.fits) file with flux value at each wavelength. I have converted them into a 2D array (x,y)=(wavelength, flux) and want to write a program which will return flux(y) at some assigned wavelengths(x). I have tried this:
#modules
import scipy
import numpy as np
import pyfits as pf
#Target Global Vaiables
hdulist_tg = pf.open('cutmask1-2.0001.fits')
hdr_tg = hdulist_tg[0].header
flux_tg = hdulist_tg[0].data
crval_tg = hdr_tg['CRVAL1'] #Starting wavelength
cdel_tg = hdr_tg['CDELT1'] #Wavelength axis width
wave_tg = crval_tg + np.arange(3183)*cdel_tg #Create an x-axis
wavelist = [6207,6315,6369,6438,6490,6565,6588]
wave_flux=[]
diff = 10
for wave in wave_tg:
for flux in flux_tg:
wave_flux.append((wave,flux))
for item in wave_flux:
wave = item[0]
flux = item[1]
#Where I got my actual wavelength that exists in wave_tg
diffmatch = np.abs(wave - wavelist[0])
if diffmatch < diff:
flux_wave = flux
diff = diffmatch
wavematch = wave
print wavelist[0],flux_wave,wavematch
but the program always return the same flux value even though the wavelength is different. Please help...

I would skip the creation of the two dimensional table altogether and just use interp:
fluxvalues = np.interp(wavelist, wave_tg, flux_tg)
For the file you posted, the code you posted doesn't work due to the hard-coded length of the wave_tg array. I would therefore recommend you rather use
wave_tg = crval_tg + np.arange(len(flux_tg))*cdel_tg
Also, for some reason it seems that the file you posted doesn't actually go up to the wavelengths you are looking up. You might need to check that you are calculating the corresponding wavelengths correctly or check that you are looking up the right wavelengths.

I've made some changes in your code:
using numpy ot create wave_flux as a ndarray using np.hstack(), np.repeat() and np.tile()
using fancy indexing to get the values matching your search
The resulting code is:
#modules
import scipy
import numpy as np
import pyfits as pf
#Target Global Vaiables
hdulist_tg = pf.open('cutmask1-2.0001.fits')
hdr_tg = hdulist_tg[0].header
flux_tg = hdulist_tg[0].data
crval_tg = hdr_tg['CRVAL1'] #Starting wavelength
cdel_tg = hdr_tg['CDELT1'] #Wavelength axis width
wave_tg = crval_tg + np.arange(3183)*cdel_tg #Create an x-axis
wavelist = [6207,6315,6369,6438,6490,6565,6588]
wave_flux = np.vstack(( np.repeat(wave_tg, len(flux_tg)),
np.tile(flux_tg, len(wave_tg)) )).transpose()
wave_ref = wavelist[0]
diff = 10
print wave_flux[ np.abs(wave_flux[:,0]-wave_ref) < diff ]
Which will return a sub-group of wave_flux with the wave values in column 0 and flux values in column 1:
[[ 6197.10300138 500.21020508]
[ 6197.10300138 523.24102783]
[ 6197.10300138 510.6390686 ]
...,
[ 6216.68436446 674.94732666]
[ 6216.68436446 684.74255371]
[ 6216.68436446 712.20098877]]