We Keep Coding

Using find_peaks in loop to find peak heights from data

This problem is about using scipy.signal.find_peaks for extracting mean peak height from data files efficiently. I am a beginner with Python (3.7), so I am not sure if I have written my code in the most optimal way, with regard to speed and code quality.
I have a set of measurement files containing one million data points (30MB) each. The graph of this data is a signal with peaks at regular intervals, and with noise. Also, the baseline and the amplitude of the signal vary across parts of the signal. I attached an image of an example. The signal can be much less clean.
My goal is to calculate the mean height of the peaks for each file. In order to do this, first I use find_peaks to locate all the peaks. Then, I loop over each peak location and detect the peak in a small interval around the peak, to make sure I get the local height of the peak.
I then put all these heights in numpy arrays and calculate their mean and standard deviation afterwards.
Here is a barebone version of my code, it is a bit long but I think that might also be because I am doing something wrong.
import numpy as np
from scipy.signal import find_peaks
# Allocate empty lists for values
mean_heights = []
std_heights = []
mean_baselines = []
std_baselines = []
temperatures = []
# Loop over several files, read them in and process data
for file in file_list:
temperatures.append(file)
# Load in data from a file of 30 MB
t_dat, x_dat = np.loadtxt(file, delimiter='\t', unpack=True)
# Find all peaks in this file
peaks, peak_properties = find_peaks(x_dat, prominence=prom, width=0)
# Calculate window size, make sure it is even
if round(len(t_dat)/len(peaks)) % 2 == 0:
n_points = len(t_dat) // len(peaks)
else:
n_points = len(t_dat) // len(peaks) + 1
t_slice = t_dat[-1] / len(t_dat)
# Allocate np arrays for storing heights
baseline_list = np.zeros(len(peaks) - 2)
height_list = np.zeros(len(peaks) - 2)
# Loop over all found peaks, and re-detect the peak in a window around the peak to be able
# to detect its local height without triggering to a baseline far away
for i in range(len(peaks) - 2):
# Making a window around a peak_properties
sub_t = t_dat[peaks[i+1] - n_points // 2: peaks[i+1] + n_points // 2]
sub_x = x_dat[peaks[i+1] - n_points // 2: peaks[i+1] + n_points // 2]
# Detect the peaks (2 version, specific to the application I have)
h_min = max(sub_x) - np.mean(sub_x)
_, baseline_props = find_peaks(
sub_x, prominence=h_min, distance=n_points - 1, width=0)
_, height_props = find_peaks(np.append(
min(sub_x) - 1, sub_x), prominence=h_min, distance=n_points - 1, width=0)
# Add the heights to the np arrays storing the heights
baseline_list[i] = baseline_props["prominences"]
height_list[i] = height_props["prominences"]
# Fill lists with values, taking the stdev and mean of the np arrays with the heights
mean_heights.append(np.mean(height_list))
std_heights.append(np.std(height_list))
mean_baselines.append(np.mean(baseline_list))
std_baselines.append(np.std(baseline_list))
It takes ~30 s to execute. Is this normal or too slow? If so, can it be optimised?

In the meantime I have improved the speed by getting rid of various inefficiencies, that I found by using the Python profiler. I will list the optimisations here ordered by significance for the speed:
Using pandas pd.read_csv() for I/O instead of np.loadtxt() cut off about 90% of the runtime. As also mentioned here, this saves a lot of time. This means changing this:
t_dat, x_dat = np.loadtxt(file, delimiter='\t', unpack=True)
to this:
data = pd.read_csv(file, delimiter = "\t", names=["t_dat", "x_dat"])
t_dat = data.values[:,0]
x_dat = data.values[:,1]
Removing redundant len() calls. I noticed that len() was called many times, and then noticed that this happened unnecessary. Changing this:
if round(len(t_dat) / len(peaks)) % 2 == 0:
n_points = int(len(t_dat) / len(peaks))
else:
n_points = int(len(t_dat) / len(peaks) + 1)
to this:
n_points = round(len(t_dat) / len(peaks))
if n_points % 2 != 0:
n_points += 1
proved to be also a significant improvement.
Lastly, a disproportionally component of the computational time (about 20%) was used by the built in Python functions min(), max() and sum(). Since I was using numpy arrays already, switching to the numpy equivalents for these functions, resulted in a 84% improvement on this part. This means for example changing max(sub_x) to sub_x.max().
These are all unrelated optimisations that I still think might be useful for Python beginner like myself, and they do help a lot.

Why is my image convolution function so slow?

I wasn't sure if I should post this on the machine learning board or this one, but I chose this one since my problem has more to do with optimization. I am trying to build a YOLO model from scratch in python, but each convolution operation takes 10 seconds. Clearly I am doing something wrong, as YOLO is supposed to be super fast (able to produce results real-time). I don't need the network to run real-time, but it will be a nightmare trying to train it if it takes several hours to run on one image. How could I optimize the code below? Apparently there is a lot of room for improvement.
Here is my convolution function:
def convolve(image, filter, stride, modifier):
new_image = np.zeros ([image.shape[0], _round((image.shape[1]-filter.shape[1])/stride)+1, _round((image.shape[2]-filter.shape[2])/stride)+1], float)
#convolve
for channel in range (0, image.shape[0]):
filterPositionX = 0
filterPositionY = 0
while filterPositionX < image.shape[1]-filter.shape[1]+1:
while filterPositionY < image.shape[2]-filter.shape[2]+1:
sum = 0
for i in range(0,filter.shape[1]):
for j in range(0,filter.shape[2]):
if filterPositionX+i<image.shape[1] and filterPositionY+j<image.shape[2]:
sum += image[channel][filterPositionX+i][filterPositionY+j]*filter[channel][i][j]
new_image[channel][int(filterPositionX/stride)][int(filterPositionY/stride)] = sum*modifier
filterPositionY += stride
filterPositionX += stride
filterPositionY = 0
#condense
condensed_new_image = np.zeros ([new_image.shape[1], new_image.shape[2]], float)
for i in range(0, new_image.shape[1]):
for j in range(0, new_image.shape[2]):
sum = 0
for channel in range (0, new_image.shape[0]):
sum += new_image[channel][i][j]
condensed_new_image[i][j] = sum
condensed_new_image = np.clip (condensed_new_image, 0, 255)
return condensed_new_image
Running the function on a 448x448 grayscale image with a 7x7 filter and a stride of 2 takes about 10 seconds. My computer has an i7 processor.

Why is it slow: Because the time complexity of the function you coded is
O(n*n*n*k*k), where image size is n*n and filter size is k*k
How to make it faster: Avoid loops and use matrix operations (vectorize). Matrix operations are parallelized.

Because it relays a lot in normal python code; i.e. your operations are performed element by element. You should vectorise them. Please take a look into this guide: https://wiseodd.github.io/techblog/2016/07/16/convnet-conv-layer/

Speeding up Fourier-related transform computations in python (OpenCV)

I have an image and I need to compute a fourier-related transform over it called Short Time Fourier Transform (for extra mathematical info check:http://en.wikipedia.org/wiki/Short-time_Fourier_transform).
In order to do that I need to :
(1) place a window at the starting pixel of the image (x,y)=(M/2,M/2)
(2) Truncate the image using this window
(3) Compute the FFT of the truncated image, save results.
(4) Incrementally slide the window to the right
(5) Go to step 3, until window reaches the end of the image
However I need to perform the aformentioned calculation in real time...
But it is rather slow !!!
Is there anyway to speed up the aformentioned process ??
I also include my code:
height, width = final_frame.shape
M=2
for j in range(M/2, height-M/2):
for i in range(M/2, width-M/2):
face_win=final_frame[j-M/2:j+M/2, i-M/2:i+M/2]
#these steps are perfomed in order to speed up the FFT calculation process
height_win, width_win = face_win.shape
fftheight=cv2.getOptimalDFTSize(height_win)
fftwidth=cv2.getOptimalDFTSize(width_win)
right = fftwidth - width_win
bottom = fftheight - height_win
bordertype = cv2.BORDER_CONSTANT
nimg = cv2.copyMakeBorder(face_win,0,bottom,0,right,bordertype, value = 0)
dft = cv2.dft(np.float32(face_win),flags = cv2.DFT_COMPLEX_OUTPUT)
dft_shift = np.fft.fftshift(dft)
magnitude_spectrum = 20*np.log(cv2.magnitude(dft_shift[:,:,0],dft_shift[:,:,1]))

Of course the bulk of your time is going to be spent in the FFT's and other transformation code, but I took a shot at easy optimizations of the other parts.
Changes
Frame size calculations are the same every loop so move them out (~nil improvement)
Type coercion from uint8 to float32 can be done once on the whole image rather than converting each frame. (small but measurable improvement)
If the window size is already the same as the optimal size (I guess it always will be if you keep M as a power of 2), then don't do the bordered copy. Just use the face_win view as-is. (small but measurable improvement)
Total improvement 26s --> 22s. Not much but there it is.
Standalone Code (just add 1024x768.jpg)
import time
import cv2
import numpy as np
# image loading for anybody else who wants to use this
final_frame = cv2.imread('1024x768.jpg')
final_frame = cv2.cvtColor(final_frame, cv2.COLOR_BGR2GRAY)
final_frame_f32 = final_frame.astype(np.float32) # moved out of the loop
# base data
M = 4
height, width = final_frame.shape
# various calculations moved out of the loop
m_half = M//2
height_win, width_win = [2 * m_half] * 2 # can you even use odd values for M?
fftheight = cv2.getOptimalDFTSize(height_win)
fftwidth = cv2.getOptimalDFTSize(width_win)
bordertype = cv2.BORDER_CONSTANT
right = fftwidth - width_win
bottom = fftheight - height_win
start = time.time()
for j in range(m_half, height-m_half):
for i in range(m_half, width-m_half):
face_win = final_frame_f32[j-m_half:j+m_half, i-m_half:i+m_half]
# only copy for border if necessary
if (fftheight, fftwidth) == (height_win, width_win):
nimg = face_win
else:
nimg = cv2.copyMakeBorder(face_win, 0, bottom, 0, right, bordertype, value=0)
dft = cv2.dft(nimg, flags=cv2.DFT_COMPLEX_OUTPUT)
dft_shift = np.fft.fftshift(dft)
magnitude_spectrum = 20 * np.log(cv2.magnitude(dft_shift[:, :, 0], dft_shift[:, :, 1]))
elapsed = time.time() - start
print elapsed
Bugs
I fixed these in the code above but I didn't edit your original since you may have intended it to be that way
you calculate nimg but then use the original face_win in the dft
to be explicit, I changed M/2 etc. to M//2

Adobe Photoshop-style posterization and OpenCV

It seems Adobe Photoshop does posterization by quantizing each color channel separately, based on the number of levels specified. So for example, if you specify 2 levels, then it will take the R value, and set it to 0 if your R value is less than 128 or 255 if your value is >= 128. It will do the same for G and B.
Is there an efficient way to do this in python with OpenCV besides iterating through each pixel and making that comparison and setting the value separately? Since an image in OpenCV 2.4 is a NumPy ndarray, is there perhaps an efficient way to do this calculation strictly through NumPy?

Your question specifically seems to be asking about a level of 2. But what about levels more than 2. So i have added a code below which can posterize for any level of color.
import numpy as np
import cv2
im = cv2.imread('messi5.jpg')
n = 2 # Number of levels of quantization
indices = np.arange(0,256) # List of all colors
divider = np.linspace(0,255,n+1)[1] # we get a divider
quantiz = np.int0(np.linspace(0,255,n)) # we get quantization colors
color_levels = np.clip(np.int0(indices/divider),0,n-1) # color levels 0,1,2..
palette = quantiz[color_levels] # Creating the palette
im2 = palette[im] # Applying palette on image
im2 = cv2.convertScaleAbs(im2) # Converting image back to uint8
cv2.imshow('im2',im2)
cv2.waitKey(0)
cv2.destroyAllWindows()
This code uses a method called palette method in Numpy which is really fast than iterating through the pixels. You can find more details how it can be used to speed up code here : Fast Array Manipulation in Numpy
Below are the results I obtained for different levels:
Original Image :
Level 2 :
Level 4 :
Level 8 :
And so on...

We can do this quite neatly using numpy, without having to worry about the channels at all!
import cv2
im = cv2.imread('1_tree_small.jpg')
im[im >= 128]= 255
im[im < 128] = 0
cv2.imwrite('out.jpg', im)
output:
input:

The coolest "posterization" I have seen uses
Mean Shift Segmentation. I used the code
from the author's GitHub repo to create the following image (you need to
uncomment line 27 of Maincpp.cpp to perform the segmentation step).

Use cv::LUT(). It is simplest and fastest way.
cv::Mat posterize(const cv::Mat &bgrmat, uint8_t lvls)
{
cv::Mat lookUpTable(1, 256, CV_8U);
uchar* p = lookUpTable.ptr();
float step = 255.0f / lvls;
for(int i = 0; i < 256; ++i)
p[i] = static_cast<uchar>(step * std::floor(i / step));
cv::Mat omat;
cv::LUT(bgrmat,lookUpTable,omat);
return omat;
}

Generalization for n levels of the answer from fraxel
import cv2 as cv
import matplotlib.pyplot as plt
im = cv.imread("Lenna.png")
n = 5
for i in range(n):
im[(im >= i*255/n) & (im < (i+1)*255/n)] = i*255/(n-1)
plt.imshow(cv.cvtColor(im, cv.COLOR_BGRA2RGB))
plt.show()
n = 2
n = 5

Numpy manipulating array of True values dependent on x/y index

So I have an array (it's large - 2048x2048), and I would like to do some element wise operations dependent on where they are. I'm very confused how to do this (I was told not to use for loops, and when I tried that my IDE froze and it was going really slow).
Onto the question:
h = aperatureimage
h[:,:] = 0
indices = np.where(aperatureimage>1)
for True in h:
h[index] = np.exp(1j*k*z)*np.exp(1j*k*(x**2+y**2)/(2*z))/(1j*wave*z)
So I have an index, which is (I'm assuming here) essentially a 'cropped' version of my larger aperatureimage array. *Note: Aperature image is a grayscale image converted to an array, it has a shape or text on it, and I would like to find all the 'white' regions of the aperature and perform my operation.
How can I access the individual x/y values of index which will allow me to perform my exponential operation? When I try index[:,None], leads to the program spitting out 'ValueError: broadcast dimensions too large'. I also get array is not broadcastable to correct shape. Any help would be appreciated!
One more clarification: x and y are the only values I would like to change (essentially the points in my array where there is white, z, k, and whatever else are defined previously).
EDIT:
I'm not sure the code I posted above is correct, it returns two empty arrays. When I do this though
index = (aperatureimage==1)
print len(index)
Actually, nothing I've done so far works correctly. I have a 2048x2048 image with a 128x128 white square in the middle of it. I would like to convert this image to an array, look through all the values and determine the index values (x,y) where the array is not black (I only have white/black, bilevel image didn't work for me). I would then like to take all the values (x,y) where the array is not 0, and multiply them by the h[index] value listed above.
I can post more information if necessary. If you can't tell, I'm stuck.
EDIT2: Here's some code that might help - I think I have the problem above solved (I can now access members of the array and perform operations on them). But - for some reason the Fx values in my for loop never increase, it loops Fy forever....
import sys, os
from scipy.signal import *
import numpy as np
import Image, ImageDraw, ImageFont, ImageOps, ImageEnhance, ImageColor
def createImage(aperature, type):
imsize = aperature*8
middle = imsize/2
im = Image.new("L", (imsize,imsize))
draw = ImageDraw.Draw(im)
box = ((middle-aperature/2, middle-aperature/2), (middle+aperature/2, middle+aperature/2))
import sys, os
from scipy.signal import *
import numpy as np
import Image, ImageDraw, ImageFont, ImageOps, ImageEnhance, ImageColor
def createImage(aperature, type):
imsize = aperature*8 #Add 0 padding to make it nice
middle = imsize/2 # The middle (physical 0) of our image will be the imagesize/2
im = Image.new("L", (imsize,imsize)) #Make a grayscale image with imsize*imsize pixels
draw = ImageDraw.Draw(im) #Create a new draw method
box = ((middle-aperature/2, middle-aperature/2), (middle+aperature/2, middle+aperature/2)) #Bounding box for aperature
if type == 'Rectangle':
draw.rectangle(box, fill = 'white') #Draw rectangle in the box and color it white
del draw
return im, middle
def Diffraction(aperaturediameter = 1, type = 'Rectangle', z = 2000000, wave = .001):
# Constants
deltaF = 1/8 # Image will be 8mm wide
z = 1/3.
wave = 0.001
k = 2*pi/wave
# Now let's get to work
aperature = aperaturediameter * 128 # Aperaturediameter (in mm) to some pixels
im, middle = createImage(aperature, type) #Create an image depending on type of aperature
aperaturearray = np.array(im) # Turn image into numpy array
# Fourier Transform of Aperature
Ta = np.fft.fftshift(np.fft.fft2(aperaturearray))/(len(aperaturearray))
# Transforming and calculating of Transfer Function Method
H = aperaturearray.copy() # Copy image so H (transfer function) has the same dimensions as aperaturearray
H[:,:] = 0 # Set H to 0
U = aperaturearray.copy()
U[:,:] = 0
index = np.nonzero(aperaturearray) # Find nonzero elements of aperaturearray
H[index[0],index[1]] = np.exp(1j*k*z)*np.exp(-1j*k*wave*z*((index[0]-middle)**2+(index[1]-middle)**2)) # Free space transfer for ap array
Utfm = abs(np.fft.fftshift(np.fft.ifft2(Ta*H))) # Compute intensity at distance z
# Fourier Integral Method
apindex = np.nonzero(aperaturearray)
U[index[0],index[1]] = aperaturearray[index[0],index[1]] * np.exp(1j*k*((index[0]-middle)**2+(index[1]-middle)**2)/(2*z))
Ufim = abs(np.fft.fftshift(np.fft.fft2(U))/len(U))
# Save image
fim = Image.fromarray(np.uint8(Ufim))
fim.save("PATH\Fim.jpg")
ftfm = Image.fromarray(np.uint8(Utfm))
ftfm.save("PATH\FTFM.jpg")
print "that may have worked..."
return
if __name__ == '__main__':
Diffraction()
You'll need numpy, scipy, and PIL to work with this code.
When I run this, it goes through the code, but there is no data in them (everything is black). Now I have a real problem here as I don't entirely understand the math I'm doing (this is for HW), and I don't have a firm grasp on Python.
U[index[0],index[1]] = aperaturearray[index[0],index[1]] * np.exp(1j*k*((index[0]-middle)**2+(index[1]-middle)**2)/(2*z))
Should that line work for performing elementwise calculations on my array?

Could you perhaps post a minimal, yet complete, example? One that we can copy/paste and run ourselves?
In the meantime, in the first two lines of your current example:
h = aperatureimage
h[:,:] = 0
you set both 'aperatureimage' and 'h' to 0. That's probably not what you intended. You might want to consider:
h = aperatureimage.copy()
This generates a copy of aperatureimage while your code simply points h to the same array as aperatureimage. So changing one changes the other.
Be aware, copying very large arrays might cost you more memory then you would prefer.
What I think you are trying to do is this:
import numpy as np
N = 2048
M = 64
a = np.zeros((N, N))
a[N/2-M:N/2+M,N/2-M:N/2+M]=1
x,y = np.meshgrid(np.linspace(0, 1, N), np.linspace(0, 1, N))
b = a.copy()
indices = np.where(a>0)
b[indices] = np.exp(x[indices]**2+y[indices]**2)
Or something similar. This, in any case, sets some values in 'b' based on the x/y coordinates where 'a' is bigger than 0. Try visualizing it with imshow. Good luck!
Concerning the edit
You should normalize your output so it fits in the 8 bit integer. Currently, one of your arrays has a maximum value much larger than 255 and one has a maximum much smaller. Try this instead:
fim = Image.fromarray(np.uint8(255*Ufim/np.amax(Ufim)))
fim.save("PATH\Fim.jpg")
ftfm = Image.fromarray(np.uint8(255*Utfm/np.amax(Utfm)))
ftfm.save("PATH\FTFM.jpg")
Also consider np.zeros_like() instead of copying and clearing H and U.
Finally, I personally very much like working with ipython when developing something like this. If you put the code in your Diffraction function in the top level of your script (in place of 'if __ name __ &c.'), then you can access the variables directly from ipython. A quick command like np.amax(Utfm) would show you that there are indeed values!=0. imshow() is always nice to look at matrices.