We Keep Coding

Coordinates Python

I am facing with the sorting airfoil coordinates. In particular given a set of coordinates, which are not sorted, I have to sorted them starting from the trailing edge upper surface. Here I report the code that I have developed but as you can see, the starting point do not match with what I suppose, moreover exist several oscillations as you can see in the reported figure (and a detail, in blue the starting point after the sort).
Can someone suggest me what I miss? How can I do?
Thanks you in advance.
def sort_airfoil(points):
x0 = np.mean(-points[:,1])
y0 = np.mean(points[:,2])
r = np.sqrt((-points[:,1]-x0)**2 + (points[:,2]-y0)**2)
tempx=-points[:,1]
xmax=np.max(tempx)
ind_max=np.where(tempx==xmax)
ymax=np.max(points[ind_max,2])
ind_max_t=np.where((tempx>0.95*xmax) & (tempx<xmax))
ymax_t=points[ind_max_t,2]
ymin=np.min(ymax_t)
indx_temp=np.where(points[:,2]==ymin)
xmin=np.max(tempx[indx_temp])
xmed=(xmin+xmax)/2
ymed=(ymin+ymax)/2
print(x0,y0)
print(xmin,ymin)
print((xmin+xmax)/2, (ymin+ymax)/2)
angle0=np.arctan2((ymed-y0),(xmed-x0))
print("angle", angle0)
angles = np.where((points[:,2]-y0) > 0, np.arccos((-points[:,1]-x0)/r), 2*np.pi-np.arccos((-points[:,1]-x0)/r))
angles=angles-angle0
for i in range(len(angles)):
if angles[i]<0:
angles[i]=angles[i]+2*np.pi
elif angles[i]>2*np.pi:
angles[i]=angles[i]-2*np.pi
mask = np.argsort(angles)
x_sorted = points[mask,1]
y_sorted = points[mask,2]
points_new=np.zeros([len(points), 3])
points_new[:,0]=points[:,0]
points_new[:,1]=x_sorted
points_new[:,2]=y_sorted
return points_new

The issue comes from the algorithm itself: it only work when the points form a convex polygon. However, the shape is concave.
More specifically, the first sorted points (and the last ones) form a zigzag-shaped lines because there is two sets of points (green arrows) interleaving with growing angles (red arrow) from the median point (red line).
Note the points are horizontally flipped on the gathered point from the question. Thus the points are sorted clockwise.
One simple solution is to split horizontally the shape in many set of point (eg. 10 set) so that each set form a convex shape. Then, the parts can be merged to form the final shape. The merge consists in finding the points at the "edge" of each locally-sorted set of points (parts) and reorder the partially sorted array of points consequently.
More specifically, the points of each part are split in 2 sub-sets: the upper ones and the lower ones. You can find them easily by selecting the 2 left-most points of a right part with the right-most points of a left part. The 2 top-most points needs to be connected each other and the same for the 2 bottom-most points. Thus, the sequence of the two upper sets of points needs to be reordered so they are contiguous and the same for the lower part.
Here is an example:
Note that if you are unsure about how to split the points in many parts so that each one form a convex-shaped sets of points, then you can: split the shape in n parts, check if the set of points form a convex shape by computing a convex hull (eg. using a Graham scan) and split evenly the parts that are concave (recursively). This is quite expensive, but more robust.

Measuring shift between two images along one direction only

I have to measure shifts between two monochromatic images.
These images are actually spectra before calibration, which are very noisy and full of unwanted features, but they basically look like following
I know that between different images, they have shifts along x-direction, but not along y-direction. And I want to know the amount of the shift along x-direction between them.
Luckily I found a function in skimage, register_translation, which can be used for arbitrary subpixel precision. But the problem is, I want to know shift along x-direction only, and I want resulting y-direction shift to be 0, but the program finds the shift to x and y at the same time, presumably along the direction perpendicular to the features. (marked as blue arrow in the figure)
So, I am wondering :
is there any function or package in python that measures the shift between two images along one direction only, or even with any prior knowledge?
what is a correct way of finding shifts between two noisy images? Would finding maximum cross-correlation value in FFT space would do the job?

Some simple maths should do in this situation if register_translation gives you the xy shift, be it in vector or component form. You can calculate the movement in x that would be required if the y shift was non-existent, which is what you want. I am travelling so unfortunately can't give you the graph right now, would recommend drawing the triangles out.
The extra x shift required (x_extra) is defined by:
x_extra = y * tan[arctan(y_shift/x_shift)]
Which is simplified to:
x_extra = y_shift^2 / x_shift
Therefore, the total shift in x is:
x_shift_total = x_shift + x_extra
Where the x_shift is given to you by register_translation.
If you then move imageA by x_shift_total, it should be aligned with imageB, assuming the x_shift given by register_translation is correct.
#jni I would be keen to implement this as an option in register_translation!

I'm not positive it will work, but: one of the benefits of open source is that you can look at the implementation details of register_translation, then try to adapt it to your case. In your case, I would replace the fftn with fftn(..., axis=1), so that you only compute the fft along the columns axis. Then, multiply the two FFT signals together (this is equivalent to the convolution of each line, as suggested by #CypherX). Finally, you have to find a way to "coalesce" the shifts found along each line into a single measurement. One idea would be to take each shift (the maximum along that line) and plot a histogram. One would hope that you get a sharp peak around the true x shift.
If it works, it would be a pretty great contribution to scikit-image to add an "axis" keyword argument to register_translation. You can read the how to contribute guide and propose a change accordingly!

Another, much faster and simpler, approach would be to calculate the horizontal profile at the same location in both images. That would give you a 1D profile for each image horizontally. Simple peak finding will then give you the location of the lines, and the difference between the peak indexes will tell you the shift solely in the x-axis.
I use this approach routinely to do shift detection similar to your problem, and it is very very fast, very simple, and very robust.
# pick a row to use
row = 10
x_profile1 = np.mean(image1[row, :], axis=0)
x_profiel2 = np.mean(image2[row, :], axis=0)
# 'get_peaks' is a function to return indices of found peaks - several
# around
peaks1 = get_peaks(x_profile1)
peaks2 = get_peaks(x_profile2)
x_shift = peaks1[0] - peaks2[0]

Method-1
You could use convolution between the two images to find where you get a maximum. You could envision this as sliding the non-shifted images over the shifted image from left to right, and the convolution will produce maxima corresponding to the scenario when the identical sections of each image lies on top of one-another. Take a look at scipy.ndimage.convolution and scipy.signal.convolve and see which one suits your needs better.
Method-2
On the other hand, you could take a horizontal slice from each image and find the position of the peaks (assuming black strips are 1's and white regions are 0's).
Calculate the centroids of these peaks in each image. Find the difference between the positions of these centroids and that is the shift your are looking for.
For robustness, you could then apply this to various rows of the image-pairs and the average of all the such differences would be a more statistically viable result for a measure of horizontal shift.

Simulate speakers around a sphere using superposition - speed improvments needed

Note: Drastic speed improvements since posting, see edits at bottom.
I have some working code by it over utilizes loops and I'm pretty sure there should be a faster way of doing it. The size of the output array ends up being pretty large and so when I try to make other arrays the same size of the output, I run out of memory rather quickly.
I am simulating many speakers placed around a sphere all pointing toward the center. I have a simulation of a single speaker and I would like to leverage this single simulation by using the principle of superposition. Basically I want to sum up rotated copies of the single transducer simulation to get my final result.
I have an axisymmetric simulation of acoustic pressure data in cylindrical coordinates ("polar_coord_r", "polar_coord_z"). The pressure field from the simulation is unique at each R and Z value and completely described by a 2D array ("P_real_RZ").
I want to sum together multiple, rotated copies of the this pressure field onto a 3D Cartesian output array. Each copy is rotated to a different location on the sphere. Currently, I am specifying the rotation with an x,y,z point because it allows me to do vector math (spherical coordinates wouldn't allow me to do this as elegantly). The output array is rather large (770 × 770 × 804).
I have some working code to get the output from a single copy of the speaker ("transducer"). It takes about 12 seconds for each slice so it would take over two hours to add each new speaker!! I want to have a dozen or so copies of the speaker so this will take way to long.
The code takes a slice with constant X and computes the R and Z positions at each location in the that slice. "r_distance" is a 2D array containing the radial distance from a line passing between the origin and a point ("point"). Similarlity, "z_distance" is a 2D array containing the distance along that same line.
To get the pressure for the slice, I find the indices of the closest matching "polar_coord_r" and "polar_coord_z" to the computed R distances and Z distances. I use these indices to find what value of pressure (from P_real_RZ) to place at each value in the output.
Some definitions:
xx, yy, and zz are 1D arrays of describing the slices through the output volume
XXX, YYY, and ZZZ are 3D arrays produced by numpy.meshgrid
point is a point which defines the direction that the speaker is rotated. Basically it's just a position vector of the speakers center.
P_real_RZ is a 2D array which specifies the real pressure at each unique R and Z value.
polar_coord_r and polar_coord_z are 1D arrays which define the unique values of R and Z on which P_real_RZ is defined.
current_transducer (only one so far represented in this code) is the pressure values computer for the current point.
output is the result from summing many speakers/transducers together.
Any suggestions to speed up this code is greatly appreciated.
Working loop:
for i, x in enumerate(xx):
# Creates a unit vector from origin to a point
vector = normalize(point)
# Create a slice of the cartesian space with constant X
xyz_slice = np.array([x*np.ones_like(XXX[i,:,:]), YYY[i,:,:], ZZZ[i,:,:]])
# Projects the position vector of each point of the slice onto the unit vector.
projection = np.array(list(map(np.dot, xyz_slice, vector )))
# Normalizes the projection which results in the Z distance along the line passing through the point
#z_distance = np.apply_along_axis(np.linalg.norm, 0, projection) # this is the slow bit
z_distance = np.linalg.norm(projection, axis=0) # I'm an idiot
# Uses vector math to determine the distance from the line
# Each point in the XYZ slice is the sum of vector along the line and the vector away from the line (radial vector).
# By extension the position of the xyz point minus the projection of the point against the unit vector, results in the radial vector
# Norm the radial vector to get the R value for everywhere in the slice
#r_distance = np.apply_along_axis(np.linalg.norm, 0, xyz_slice - projection) # this is the slow bit
r_distance = np.linalg.norm(xyz_slice - projection, axis=0) # I'm an idiot
# Map the pressure data to each point in the slice using the R and Z distance with the RZ pressure slice.
# look for a more efficient way to do this perhaps. currently takes about 12 seconds per slice
r_indices = r_map_v(r_distance) # 1.3 seconds by itself
z_indices = z_map_v(z_distance)
r_indices[r_indices>384] = 384 # find and remove indicies above the max for r_distance
z_indices[r_indices>803] = 803
current_transducer[i,:,:] = P_real_RZ[z_indices, r_indices]
# Sum the mapped pressure data into the output.
output += current_transducer
I have also tried to work with the simulation data in the form of a 3D Cartesian array. That is the pressure data from the simulation for all x, y, and z values the same size as the output.I can rotate this 3D array in one direction (not two rotations needed for speakers arranged on a sphere). This takes up waaaay too much memory and is still painfully slow. I end up getting memory errors with this approach.
Edit: I found a slightly simpler way to do it but it is still slow. I've updated the code above so that there are no longer nested loops.
I ran a line profiler and the slowest lines by far were the two containing np.apply_along_axis(). I'm afraid I might have to rethink how I do this completely.
Final Edit: I initially had a nested loop which I assumed to be the issue. I don't know what made me think I needed to use apply_along_axis with linalg.norm. In any case that was the issue.

I haven't looked for all the ways that you could optimize this code, but this issue jumped out at me: "I ran a line profiler and the slowest lines by far were the two containing np.apply_along_axis()." np.linalg.norm accepts an axis argument. You can replace the line
z_distance = np.apply_along_axis(np.linalg.norm, 0, projection)
with
z_distance = np.linalg.norm(projection, axis=0)
(and likewise for the other use of np.apply_along_axis and np.linalg.norm).
That should improve the performance a bit.

How to find a satellites passing rate over a geographical area?

I've god some satellite data which essentially is the geographical position of a satellite that circles the earth at a given time. This is data saved with a latitude, longitude and unixtime in a SQLite DB. This is retrieved as following:
latitudes = [] #Long list of latitudes
longitudes = [] #Long list of longitudes
unixtimes = [] #Long list of corresponding unixtimes
So, I'm interested to distinguish the latitude/longitude recordings for each time the satellite is over a fairly large geographical area (for each passing). However I'm unsure on how I would do this.
Now I've manually, by visual inspection of the plots of the position, found the first 'occurrence' of the satellite in that area, then I have found the next occurrence in the same way. The passing time is then the difference between each of those events. However, this passing time varies over time, so this method is not that accurate over time. An other problem is that it is dependent on the geographical positions, If I want the time of the first passing, and the passing time for any other geographical position I have to manually inspect once again. I've included my code. Note that the seq function is simply a function I've retrieved from SO that gives me the ability to iterate over non-integer increments.
def seq(start, end, step):
assert(step != 0)
sample_count = abs(end - start) / step
return itertools.islice(itertools.count(start, step), sample_count)
gridsize = 5 #Unit: degrees
upperleftlong = #Upper corner of geographical area
upperleftlat = #Upper corner of geographical area
lowerrightlong = #Lower corner of geographical area
lowerrightlat = #Lower corner of geographical area
passrate = 5500 #Time between passings in seconds
start = 1498902400 #Time of first passing
end = 1498905700 #Approximately passing length
numberofpassings = 600 #Number of passings that should be checked for
for p in range(0,numberofpassings+1):
start = 1398903400+passrate*p
end = 1398905400+passrate*p
for i in seq(lowerrightlat, upperleftlat+gridsize, gridsize):
for j in seq(upperleftlong, lowerrightlong+gridsize, gridsize):
positions = getPositionsFromDB(j,i,start,end,gridsize,databasepath, con)
So, does anyone have a clever way to signify passing rate, passing time and discover which geographical positions that belongs to each passing?
I'm working with Python and SQLite.

From the period of your satellite (5500 seconds), I fairly certain that your satellite is the Space Station. Very few other satellites are normally active at that low altitude (370 km) because of the low lifespan.
The Heavens-above site has many tools to predict sighting of the Space station (and others). Spot-the-station is dedicated and provides the predictions. Satellites calculations on-line is a large collections of tools which can be of help too.
If interested in the workings of such programs, an open source project, Predict, is available with source code.
Of course, Wikipedia has to be present, with a list of apps, and references to many libraries with tools for predictions.
Note: Integer increments are fine, but numpy can give you floating point increments if you use numpy.arange. This is much more flexible, and you can work with physical, non-scaled values, without the risk of running into integer overflows.

These spectrum bands used to be judged by eye, how to do it programmatically?

Operators used to examine the spectrum, knowing the location and width of each peak and judge the piece the spectrum belongs to. In the new way, the image is captured by a camera to a screen. And the width of each band must be computed programatically.
Old system: spectroscope -> human eye
New system: spectroscope -> camera -> program
What is a good method to compute the width of each band, given their approximate X-axis positions; given that this task used to be performed perfectly by eye, and must now be performed by program?
Sorry if I am short of details, but they are scarce.
Program listing that generated the previous graph; I hope it is relevant:
import Image
from scipy import *
from scipy.optimize import leastsq
# Load the picture with PIL, process if needed
pic = asarray(Image.open("spectrum.jpg"))
# Average the pixel values along vertical axis
pic_avg = pic.mean(axis=2)
projection = pic_avg.sum(axis=0)
# Set the min value to zero for a nice fit
projection /= projection.mean()
projection -= projection.min()
#print projection
# Fit function, two gaussians, adjust as needed
def fitfunc(p,x):
return p[0]*exp(-(x-p[1])**2/(2.0*p[2]**2)) + \
p[3]*exp(-(x-p[4])**2/(2.0*p[5]**2))
errfunc = lambda p, x, y: fitfunc(p,x)-y
# Use scipy to fit, p0 is inital guess
p0 = array([0,20,1,0,75,10])
X = xrange(len(projection))
p1, success = leastsq(errfunc, p0, args=(X,projection))
Y = fitfunc(p1,X)
# Output the result
print "Mean values at: ", p1[1], p1[4]
# Plot the result
from pylab import *
#subplot(211)
#imshow(pic)
#subplot(223)
#plot(projection)
#subplot(224)
#plot(X,Y,'r',lw=5)
#show()
subplot(311)
imshow(pic)
subplot(312)
plot(projection)
subplot(313)
plot(X,Y,'r',lw=5)
show()

Given an approximate starting point, you could use a simple algorithm that finds a local maxima closest to this point. Your fitting code may be doing that already (I wasn't sure whether you were using it successfully or not).
Here's some code that demonstrates simple peak finding from a user-given starting point:
#!/usr/bin/env python
from __future__ import division
import numpy as np
from matplotlib import pyplot as plt
# Sample data with two peaks: small one at t=0.4, large one at t=0.8
ts = np.arange(0, 1, 0.01)
xs = np.exp(-((ts-0.4)/0.1)**2) + 2*np.exp(-((ts-0.8)/0.1)**2)
# Say we have an approximate starting point of 0.35
start_point = 0.35
# Nearest index in "ts" to this starting point is...
start_index = np.argmin(np.abs(ts - start_point))
# Find the local maxima in our data by looking for a sign change in
# the first difference
# From http://stackoverflow.com/a/9667121/188535
maxes = (np.diff(np.sign(np.diff(xs))) < 0).nonzero()[0] + 1
# Find which of these peaks is closest to our starting point
index_of_peak = maxes[np.argmin(np.abs(maxes - start_index))]
print "Peak centre at: %.3f" % ts[index_of_peak]
# Quick plot showing the results: blue line is data, green dot is
# starting point, red dot is peak location
plt.plot(ts, xs, '-b')
plt.plot(ts[start_index], xs[start_index], 'og')
plt.plot(ts[index_of_peak], xs[index_of_peak], 'or')
plt.show()
This method will only work if the ascent up the peak is perfectly smooth from your starting point. If this needs to be more resilient to noise, I have not used it, but PyDSTool seems like it might help. This SciPy post details how to use it for detecting 1D peaks in a noisy data set.
So assume at this point you've found the centre of the peak. Now for the width: there are several methods you could use, but the easiest is probably the "full width at half maximum" (FWHM). Again, this is simple and therefore fragile. It will break for close double-peaks, or for noisy data.
The FWHM is exactly what its name suggests: you find the width of the peak were it's halfway to the maximum. Here's some code that does that (it just continues on from above):
# FWHM...
half_max = xs[index_of_peak]/2
# This finds where in the data we cross over the halfway point to our peak. Note
# that this is global, so we need an extra step to refine these results to find
# the closest crossovers to our peak.
# Same sign-change-in-first-diff technique as above
hm_left_indices = (np.diff(np.sign(np.diff(np.abs(xs[:index_of_peak] - half_max)))) > 0).nonzero()[0] + 1
# Add "index_of_peak" to result because we cut off the left side of the data!
hm_right_indices = (np.diff(np.sign(np.diff(np.abs(xs[index_of_peak:] - half_max)))) > 0).nonzero()[0] + 1 + index_of_peak
# Find closest half-max index to peak
hm_left_index = hm_left_indices[np.argmin(np.abs(hm_left_indices - index_of_peak))]
hm_right_index = hm_right_indices[np.argmin(np.abs(hm_right_indices - index_of_peak))]
# And the width is...
fwhm = ts[hm_right_index] - ts[hm_left_index]
print "Width: %.3f" % fwhm
# Plot to illustrate FWHM: blue line is data, red circle is peak, red line
# shows FWHM
plt.plot(ts, xs, '-b')
plt.plot(ts[index_of_peak], xs[index_of_peak], 'or')
plt.plot(
[ts[hm_left_index], ts[hm_right_index]],
[xs[hm_left_index], xs[hm_right_index]], '-r')
plt.show()
It doesn't have to be the full width at half maximum — as one commenter points out, you can try to figure out where your operators' normal threshold for peak detection is, and turn that into an algorithm for this step of the process.
A more robust way might be to fit a Gaussian curve (or your own model) to a subset of the data centred around the peak — say, from a local minima on one side to a local minima on the other — and use one of the parameters of that curve (eg. sigma) to calculate the width.
I realise this is a lot of code, but I've deliberately avoided factoring out the index-finding functions to "show my working" a bit more, and of course the plotting functions are there just to demonstrate.
Hopefully this gives you at least a good starting point to come up with something more suitable to your particular set.

Late to the party, but for anyone coming across this question in the future...
Eye movement data looks very similar to this; I'd base an approach off that used by Nystrom + Holmqvist, 2010. Smooth the data using a Savitsky-Golay filter (scipy.signal.savgol_filter in scipy v0.14+) to get rid of some of the low-level noise while keeping the large peaks intact - the authors recommend using an order of 2 and a window size of about twice the width of the smallest peak you want to be able to detect. You can find where the bands are by arbitrarily removing all values above a certain y value (set them to numpy.nan). Then take the (nan)mean and (nan)standard deviation of the remainder, and remove all values greater than the mean + [parameter]*std (I think they use 6 in the paper). Iterate until you're not removing any data points - but depending on your data, certain values of [parameter] may not stabilise. Then use numpy.isnan() to find events vs non-events, and numpy.diff() to find the start and end of each event (values of -1 and 1 respectively). To get even more accurate start and end points, you can scan along the data backward from each start and forward from each end to find the nearest local minimum which has value smaller than mean + [another parameter]*std (I think they use 3 in the paper). Then you just need to count the data points between each start and end.
This won't work for that double peak; you'd have to do some extrapolation for that.

The best method might be to statistically compare a bunch of methods with human results.
You would take a large variety data and a large variety of measurement estimates (widths at various thresholds, area above various thresholds, different threshold selection methods, 2nd moments, polynomial curve fits of various degrees, pattern matching, and etc.) and compare these estimates to human measurements of the same data set. Pick the estimate method that correlates best with expert human results. Or maybe pick several methods, the best one for each of various heights, for various separations from other peaks, and etc.