We Keep Coding

Linear regression for calibrating on a 2D grid - numpy

I am running a laser experiment where I'm trying to measure some features, but I also have to deal with a linear background. This background is both a constant (whatever light I measure when the laser is off) as well as a multiplicative scale factor. This background cannot be determined analytically, so I need to do an mx+b fit on the data. But, I need to do this on every point in the field of view.
The way I'd do it would be to take calibration images at a range of uniform brightnesses, and then run a regression, assigning a unique m_ij and b_ij to every point. I could probably do this in a for loop, but that seems like it'd be insanely slow for an image that's on the order of 1 mpx.
I found a solution here that used np.vander. I've tried using that, but (a) don't quite understand what I'm doing with it, and (b) it doesn't work with curve_fit. I could use np.linalg.lstsq, but it doesn't allow me to assign yerr corresponding to the noise of the images.
My current non-working example:
def fit_many_with_error(x, y, order=2, xerrs=None, yerrs=None):
'''
arguments:
x: [N]
y: [N x S]
where:
N - # of measurements per pixel
S - # pixels
returns [`order` x S]
'''
def f(x, m, b):
return m * x + b
A = np.vander(x, N=order)
B = np.vander(y, N=order)
params = curve_fit(f, A, B, sigma=None)
return params
params = fit_many_with_error(xvals, yvals)
Which gives me ValueError: operands could not be broadcast together with shapes (20,) (20,200,100)

Is there a python friendly function to calculate moving average for 3D data?

I am trying to create a smooth contour map by using a simple moving average filter. I have a .CSV that has three column x, y (which are poisition) and z which is the heat at a given x,y. Each column has 23,236 value. I have tried 1d moving average seperatly for x and then for y. Given the size of the data it has been neither fast nor effective at smoothing. I would appricate any help. the code below plots the contour map.
df = 'D:/F1_amp .csv'
df = pd.read_csv("F1_amp.csv");
Z = df.pivot_table(index='x', columns='y', values='z').T.values
X_unique = np.sort(df.x.unique())
Y_unique = np.sort(df.y.unique())
X, Y = np.meshgrid(X_unique, Y_unique)
pd.DataFrame(Z).round(3)
pd.DataFrame(X).round(3)
pd.DataFrame(Y).round(3)
plt.contourf(X, Y, Z, 20, cmap='hot')
plt.colorbar();

You can do this via Convolution. This approach is fast and easy to generalize.
For 2D convolution you could use scipy.signal.covolve2d(). You can vary the size and the values of the kernel, I used a constant 3x3 kernel as example.
import numpy as np
import scipy.signal as sg
kernel_shape = (3, 3)
kernel = np.full(kernel_shape, 1/np.prod(kernel_shape))
z = sg.convolve2d(z, kernel, mode='valid')

Calculate the length of an edge consisting of many pixel data

I have made a workflow code to detect the edges of a flame in an image. I could get the edge line. It consists of many pixel points stored in an array (data in my code). Now based on the data, I would like to calculate the length of the edge. The idea is to calculate the distance between every point in data and sum them all to get the length. I really stuck in making that. Please help me, many thanks.
Here is a processed image:
Here is the original image that converted to the processed image, I put in the code is to compare the result:
import cv2
import matplotlib.pyplot as plt
if __name__ == '__main__':
path = '1897_1.jpg' #processed image
pic = cv2.imread(path)
original = cv2.imread('1897_2.jpg') #original image
img2 = cv2.flip(original, 1)
b,g,r = cv2.split(pic)
img4 = cv2.flip(b, 1)
h,w = img4.shape
data = []
th_val = 20
for i in range(h):
for j in range(w):
val = img4[i, j]
if (val >= th_val):
data.append(j)
break
b1 = range(len(data))
b2 = len(data)
result = [b2]
print (b2)
plt.figure(figsize = (10, 8))
plt.subplot(121)
plt.imshow(img4)
plt.plot(data, b1)
plt.axis('off');
plt.subplot(122)
plt.plot(data, b1)
plt.imshow(img2)
plt.axis('off')

I came up with a very simple solution, it is far from optimal, but it works for this example, and it is a good starting point. Unfortunately, this solution is not optimal for the blue chanell, where the curve is not smooth, but it works for green and red chanells.
data contains width coordinates of the first red pixel overcoming threshold. So, all first pixels are separated by 1 pixel step on vertical axes and data[i+1] - data[i] on horizontal axes. These two values can be considered as two cathetus of the squeare triangle, and the hypothenuse is the distance we want to calculate. So, here is the solution:
length = 0
for i in range(0,len(data)-1):
cathetus = data[i+1]-data[i]
hypothenuse = (cathetus**2 + 1**2)**1/2
length += hypothenuse
print(length)
Update
I have came up with two solutions: a hardcoded one and one released in the form of the function. Let us start with the first one: mean is a rather good approximator for the signal + noise. In the situation, when you do not have very strong noise or missing data, you may use this approach. In the example below we select points with x in [1,2,3] then we calculate mean y for these points and assign mean to coordinate x=2. Next we select points x in [2,3,4] and so on. As a result, we obtain mean_data list with y coordinates and mean_x with x coordinates. We can calculate length with the approach described above. You may also increase the power of smoothing by averaging over 4 and more points from data.
mean_data = []
mean_x = range(1,len(data)-1)
for i in range(0,len(data)-2):
mean_d = (data[i] + data[i+1] + data[i+2])/3
mean_data.append(mean_d)
Another approach is to use smoothing tools from scipy package. One of them is described below. When calculating the length you will have to adjust to new x axes xnew.
from scipy.interpolate import spline
import numpy as np
#transform to np.arrays initial data
b1_ = np.array(b1)
data_ = np.array(data)
# create new x with more data points
xnew = np.linspace(b1_.min(),b1_.max(),50) #50 is a number of points in between
smoothed_data = spline(b1_,data_,xnew)

How to best perform a surface integral over 2D point data?

I have a data set of 363 x- by 190 y-points with an associated functional value that I would to integrate over multiple different subregions.. I've tried to create a SciPy interp2d function to integrate; however, creating that function even with linear interpolation has taken over 2 hours (and is not yet done).
What is a better approach to perform this task?
Some snippets below...
In the convert_RT_to_XY function below, imb/jmb are the r,theta mesh boundaries that I convert to Cartesian boundaries.
Later, in my code, I convert the mesh boundaries (imb/jmb) to mesh-center values (imm,jmm), convert to vectors (iX, iY), convert my function a vector (iZ), and then attempt to make my interpolation function.
# Convert R, T mesh vectors to X, Y mesh arrays.
def convert_RT_to_XY(imb, jmb):
R, T = np.meshgrid(imb,jmb)
X = R * np.cos(np.radians(T*360))
Y = R * np.sin(np.radians(T*360))
return(X, Y)
...
imm = imb[:-1]+np.divide(np.diff(imb),2)
jmm = jmb[:-1]+np.divide(np.diff(jmb),2)
iX, iY = convert_RT_to_XY(imm, jmm)
iX = np.ndarray.flatten(iX)
iY = np.ndarray.flatten(iY)
iZ = np.ndarray.flatten(plot_function)
f = interpolate.interp2d(iX, iY, iZ, kind='linear')
Ultimately, I want to perform:
result = dblquad(f, 10, 30,
lambda x: 10,
lambda x: 30))

Look into SciPy's RectBivariateSpline. If you are placing your data on a Cartesian grid anyway, it performs much faster than interp2D

given a pixel and gaussian filter, apply gaussian blur

I'm trying to apply a gaussian filter to an image. My filter is a n by n square maxtrix represented as list of list. I've converted my image into 3 separate 2d arrays of R, G and B so that calling B[x][y] would give me the blue value of that pixel at x,y.
I'm having a difficult time thinking of a way of applying the blur. For example, if I have the filter [[1,1,1],[1,1,1],[1,1,1]], the blurred result at B[x][y] should be
1*B[x-1][y-1] + 1*B[x-1][y] + 1*B[x-1][y+1] +
1*B[x][y-1] + 1*B[x][y] + 1* B[x][y+1] +
1*B[x+1][y-1] + 1*R[x+1][y] + 1*B[x+1][y+1]
I'm think of traversing each of the color 2d array (R, G, and B) using a nested loop, but I don't know what to do from that point since the value of x and y is changing at each location.
Given a red/ green/ blue pixel at x,y, how would I apply a gaussian filter of size n x n onto that pixel?

Start here: http://campar.in.tum.de/twiki/pub/Chair/TeachingSs04ImageSeg/VL_Segmentierung_kap8.pdf
Adjusting variables in the equation will greatly affect the sharpness of edges that you see.

You should make a copy of the image pixel plane to use as the unchanged original as you compute each new pixel:
def blur( P ):
T = list( P )
for x ...
P[x][y] = 1*T[x-1][y-1] + 1*T[x-1][y] + 1*T[x-1][y+1] +
1*T[x][y-1] + 1*T[x][y] + 1* T[x][y+1] +
1*T[x+1][y-1] + 1*T[x+1][y] + 1*T[x+1][y+1]

Pillow will do this for you. http://pillow.readthedocs.org/en/latest/reference/ImageFilter.html
If your goal is to get the gaussian blur done and not to do it yourself.
Also other image ops in there.