I have 3 images, with an applyied mean filter.
I0 beeing just the noise image, taken with the cap on.
I20 taken an image which only shows a 20% reflectance target
I90 an image showing only a 90% reflectance target for each pixel.
So rather than looping over each pixel and using polynomial fit (https://docs.scipy.org/doc/numpy/reference/generated/numpy.polyfit.html)
Where X = [I0(i), I20(i), I90(i)] and Y=[0,0.2,0.9]
and then applying the polyfit to get the parameters for each pixel,
is there a way to feed a X(i,3) and Y(i,3) into polyfit or something similar to get the same result but faster?
Thanks
Ben
If your goal is to vectorize polyfit then yes, this can be done but requires rewriting np.polyfit manually. Fortunately, it can be built on top of np.linalg.lstsq and the polynomial design matrix provided by np.vander. All in all, the routine looks like the following:
import numpy as np
def fit_many(x, y, order=2):
'''
arguments:
x: [N]
y: [N x S]
where:
N - # of measurements per pixel
S - # pixels
returns [`order` x S]
'''
A = np.vander(x, N=order)
return np.linalg.lstsq(A, y, rcond=None)[0]
And can be used like below
# measurement x values. I suppose those are your reflectances?
x = np.array([0, 1, 2])
y = np.array([ # a row per pixel
[-1, 0.2, 0.9],
[-.9, 0.1, 1.2],
]).T
params = fit_many(x, y)
import matplotlib.pyplot as plt
poly1 = np.poly1d(params[:, 0])
poly2 = np.poly1d(params[:, 1])
plt.plot(x, y[:, 0], 'bo')
plt.plot(x, poly1(x), 'b-')
plt.plot(x, y[:, 1], 'ro')
plt.plot(x, poly2(x), 'r-')
plt.show()
Keep in mind np.linalg.lstsq doesn't allow for dimensions higher than two, so you will have to reshape your 2d image into flattened versions, fit and convert back.
Related
I have a numpy array of shape x,y,z which represents z matrixes of x by y. I can slice each of the matrixes and then use clip with percentiles to filter out outliers:
mx = array[:, :, 0] # taking the first matrix
filtered_mx = np.clip(mx, np.percentile(mx, 1), np.percentile(mx, 99))
Is there some efficient way to do the same without doing it on a slice at a time?
You can pass arrays to np.clip, so it is possible to have different limits across the z dimension of mx:
import numpy as np
# Create random mx
x, y, z = 10, 11, 12
mx = np.random.random((x, y, z))
# Calculate the percentiles across the x and y dimension
perc01 = np.percentile(mx, 1, axis=(0, 1), keepdims=True)
perc99 = np.percentile(mx, 99, axis=(0, 1), keepdims=True)
# Clip array with different limits across the z dimension
filtered_mx = np.clip(mx, a_min=perc01, a_max=perc99)
I am trying to understand to the meaning of NFFT in numpy.fft.rfft. But I get confused why when NFFT included or not, the outputs get very different. Please see the example below.
numpy.fft.rfft([0, 1, 0, 0, 4.3, 3, 599], 8)
array([ 607.3 +0.j , -5.71421356+600.41421356j,
-594.7 -4.j , -2.88578644-597.58578644j,
599.3 +0.j ])
numpy.fft.rfft([0, 1, 0, 0, 4.3, 3, 599])
array([ 607.3 +0.j , 369.55215218+472.32571033j,
-133.53446083+578.34336489j, -539.66769135+261.30917157j])
The FFT is an efficient implementation of the Discrete Fourier Transform (DFT), which is a discrete function of frequency. It is also related to the Discrete-Time Fourier Transform (DTFT), itself a continuous function of frequency. More specifically, the DFT corresponds exactly to the DTFT evaluated at the discrete frequencies of the DFT.
In other words, when computing a Discrete Fourier Transform with numpy.fft.rfft, you are essentially sampling the DTFT function at discrete frequency points. You can see this by plotting transforms of different lengths on the same graph with the following:
import numpy as np
import matplotlib.pyplot as plt
x = [0, 1, 0, 0, 4.3, 3, 599]
# Compute the DTFT at a sufficiently large number of points using the explicit formula
N = 2048
f = np.linspace(0, 0.5, N)
dtft = np.zeros(len(f), dtype=np.complex128)
for n in range(0,len(x)):
dtft += x[n] * np.exp(-1j*2*np.pi*f*n)
# Compute the FFT without NFFT argument (NFFT defaults to the length of the input)
y1 = np.fft.rfft(x)
f1 = np.fft.rfftfreq(len(x))
# Compute the FFT with NFFT argument
N2 = 8
y2 = np.fft.rfft(x,N2)
f2 = np.fft.rfftfreq(N2)
# Plot results
plt.figure(1)
plt.subplot(2,1,1)
plt.plot(f, np.abs(dtft), label='DTFT')
plt.plot(f1, np.abs(y1), 'C1x', label='FFT N=7')
plt.plot(f2, np.abs(y2), 'C2s', label='FFT N=8')
plt.title('Magnitude')
plt.legend(loc='upper right')
plt.subplot(2,1,2)
plt.plot(f, np.angle(dtft), label='DTFT')
plt.plot(f1, np.angle(y1), 'C1x', label='FFT N=7')
plt.plot(f2, np.angle(y2), 'C2s', label='FFT N=8')
plt.title('Phase')
plt.legend(loc='upper right')
plt.show()
I have a problem showing data in a graph. The graph frame appears, but no graph is to be seen. Can you please help ?
I made sure that the dimension of the x axis and the data is the same ... I simply cannot find out why I do not get a graph in return.
Thank you very much in advance.
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import norm
n = 1000
theta = 0.8
d = np.sqrt(1-theta**2)
def p(x,y):
"Stochastic kernel for the TAR model"
return norm().pdf((y-theta*np.abs(x))/d)/d
Z = norm().rvs(n)
X = np.empty(n)
for t in range(n-1):
X[t+1] = theta*np.abs(X[t])+d*Z[t+1]
n = len(X)
X = X.reshape((n, 1))
ys = np.linspace(-3,3,200)
k = len(ys)
ys = ys.reshape((1,k))
v = p(X,ys)
kernel = np.mean(v, axis=0)
h = len(kernel)
kernel = kernel.reshape((1,h))
fig, ax = plt.subplots(figsize=(10,7))
ax.plot(ys,kernel, 'b-', lw=2,alpha=0.6, label='look ahead estimate')
plt.show()
The problem is, that through reshaping the two 1-dimensional arrays ys and kernel to a 1xk or 1xh array respectively you get 2-dimensional arrays, where the first dimension is 1. The plot function apparently only iterates through the first dimension, which is why the plot doesn't show anything.
I can think of two easy options to fix that:
Do not reshape the variables kernel and ys:
# ... continuing your code ...
ys = np.linspace(-3,3,200)
k = len(ys)
#ys = ys.reshape((1,k))
v = p(X,ys)
kernel = np.mean(v, axis=0)
h = len(kernel)
#kernel = kernel.reshape((1,h))
fig, ax = plt.subplots(figsize=(10,7))
ax.plot(ys,kernel, 'b-', lw=2,alpha=0.6, label='look ahead estimate')
plt.show()
Call your plot function like this:
ax.plot(ys[0],kernel[0], 'b-', lw=2, alpha=0.6, label='look ahead estimate')
I hope this solves your problem.
To understand why you still have to reshape X:
Let's first understand your function p(x,y) in terms of dimensions:
def p(x,y):
"Stochastic kernel for the TAR model"
"""If x is not reshaped, you substract two one-dimensional arrays from each other,
which have not the same dimensions (dim(x) == 1000, dim(y) == 200 in your case).
This throws an error.
If you reshape X before passing to this function, the y array is substracted
element-wise by each of the values of X, which gives you a matrix with dimension
dim(x) x dim(y).
"""
return norm().pdf((y-theta*np.abs(x))/d)/d
For illustration what happens here dimension-wise:
>>> X = np.array([[1], [2], [3], [4]])
>>> Y = np.array([1, 2, 3])
>>> Y-X
array([[ 0, 1, 2],
[-1, 0, 1],
[-2, -1, 0],
[-3, -2, -1]])
Now we take a look what happens with the matrix returned by p(x,y):
The calculation of the kernel with np.mean(v, axis=0), where v is the returned matrix from p(X,ys), works such, that np.mean iterates over the lines of the matrix v and calculates the mean value of each "line vector" in the matrix. This gives you an one dimensional array (dimension of ys) which you can plot over ys.
How could I smooth the x[1,3] and x[3,2] elements of the array,
x = np.array([[0,0,0,0,0],[0,0,0,1,0],[0,0,0,0,0],[0,0,1,0,0],[0,0,0,0,0]])
with two two-dimensional gaussian functions of width 1 and 2, respectively? In essence I need a function that allows me to smooth single "point like" array elements with gaussians of differing widths, such that I get an array with smoothly varying values.
I am a little confused with the question you asked and the comments you have posted. It seems to me that you want to use scipy.ndimage.filters.gaussian_filter but I don't understand what you mean by:
[...] gaussian functions with different sigma values to each pixel. [...]
In fact, since you use a 2-dimensional array x the gaussian filter will have 2 parameters. The rule is: one sigma value per dimension rather than one sigma value per pixel.
Here is a short example:
import matplotlib.pyplot as pl
import numpy as np
import scipy as sp
import scipy.ndimage
n = 200 # widht/height of the array
m = 1000 # number of points
sigma_y = 3.0
sigma_x = 2.0
# Create input array
x = np.zeros((n, n))
i = np.random.choice(range(0, n * n), size=m)
x[i / n, i % n] = 1.0
# Plot input array
pl.imshow(x, cmap='Blues', interpolation='nearest')
pl.xlabel("$x$")
pl.ylabel("$y$")
pl.savefig("array.png")
# Apply gaussian filter
sigma = [sigma_y, sigma_x]
y = sp.ndimage.filters.gaussian_filter(x, sigma, mode='constant')
# Display filtered array
pl.imshow(y, cmap='Blues', interpolation='nearest')
pl.xlabel("$x$")
pl.ylabel("$y$")
pl.title("$\sigma_x = " + str(sigma_x) + "\quad \sigma_y = " + str(sigma_y) + "$")
pl.savefig("smooth_array_" + str(sigma_x) + "_" + str(sigma_y) + ".png")
Here is the initial array:
Here are some results for different values of sigma_x and sigma_y:
This allows to properly account for the influence of the second parameter of scipy.ndimage.filters.gaussian_filter.
However, according to the previous quote, you might be more interested in the assigement of different weights to each pixel. In this case, scipy.ndimage.filters.convolve is the function you are looking for. Here is the corresponding example:
import matplotlib.pyplot as pl
import numpy as np
import scipy as sp
import scipy.ndimage
# Arbitrary weights
weights = np.array([[0, 0, 1, 0, 0],
[0, 2, 4, 2, 0],
[1, 4, 8, 4, 1],
[0, 2, 4, 2, 0],
[0, 0, 1, 0, 0]],
dtype=np.float)
weights = weights / np.sum(weights[:])
y = sp.ndimage.filters.convolve(x, weights, mode='constant')
# Display filtered array
pl.imshow(y, cmap='Blues', interpolation='nearest')
pl.xlabel("$x$")
pl.ylabel("$y$")
pl.savefig("smooth_array.png")
And the corresponding result:
I hope this will help you.
So I have a data set that is in the matrix form:
x1, Y1, VALUE1
x2, Y1, VALUE2
x3, Y1, VALUE3
x1, Y2, VALUE4
x2, Y2, VALUE5
x3, Y2, VALUE6
and so on. I get my contours properly except my x and y axes go from say 1, 2, 3...N. This is fine because it is representing pixels so isn't incorrect, but I would like to change the axes values from pixels to the actual units. I can't seem to find a way to instruct contour to allow me to add this.
bsquare=np.reshape(value,(x length,y length))
blue=contour(bsquare,colors='b')
plt.show()
where xlength and ylength are the number of points in either axis.
plt.contour can be given arrays X, Y, Z then it takes the Z as the contour values and the X and Y are used on their respective axes. Here is a script that first makes some data to play with, then gets into an array of the form you describe:
import matplotlib.pyplot as plt
import numpy as np
# Make some test data
nx = 2
ny = 3
x = np.linspace(0, 3, nx)
y = np.linspace(50, 55, ny)
X, Y = np.meshgrid(x, y)
Z = np.sin(X) + Y
# Now get it into the form you describe
data = [[[x[i], y[j], Z[j, i]] for i in range(nx)] for j in range(ny)]
data = np.array(data)
print data
>>>
[[[ 0. 50. 50. ]
[ 3. 50. 50.14112001]]
[[ 0. 52.5 52.5 ]
[ 3. 52.5 52.64112001]]
[[ 0. 55. 55. ]
[ 3. 55. 55.14112001]]]
Note I am using a numpy.array not just a normal list this is important in the next step. Lets split up that data as I presume you have done into the x and y values and the values themselves:
# Now extract the data
x_values = data[:, :, 0]
y_values = data[:, :, 1]
values = data[:, :, 2]
Now all of these things are nx, ny arrays, that is they have the same shape as your bsquare. You can check this by printing values.shape and changing the integers nx, ny. Now I will plot three things:
Firstly as you have done simply contour plot the values, this automatically adds the axes values
Secondly I plot using the arrays to give the correct scalings and
Finally I will plot the origin data set to show it properly recovers the data.
You will need to compare the axis values with where the fake data was created:
fig, axes = plt.subplots(ncols=3, figsize=(10, 2))
axes[0].contour(values)
axes[1].contour(x_values, y_values, values)
axes[2].contour(X, Y, Z)
How you implement this will largely depend on how you have imported your data. If you can simply turn it into a numpy.array() then I think this will solve your issue.
Hopefully I understood your problem correctly.