We Keep Coding

Finding the maximum values of a set of local maxima using matplotlib and numpy

I want to ask a question about finding the maxima of a set of peaks using matplotlib and numpy.
I have been given data containing peaks and asked to calculate the maxima of the set of peaks.
Below is a picture of the peaks.
I discovered the find_peaks method and attempted to solve the problem using this.
I wrote the following block of code in Jupyter:
%pylab inline
from scipy.signal import find_peaks
testdata = loadtxt("testdata.dat", usecols=(0,1))
testdata_x = testdata[100:200,0]
testdata_y = testdata[100:200,1]
plot(testdata_x, testdata_y)
show()
peaks = find_peaks(testdata_y)
peaks
However, I get the following output for peaks:
(array([ 7, 12, 36, 40, 65, 69, 93, 97]), {})
I cannot understand why I get an output as above and am struggling to find a solution.
I attempted also to pass the following:
peaks = find_peaks(testdata_y, testdata_x)
but this was to no avail.
How can I sort out the matter?
I have attached the data file here as a download link if necessary (hoested on filehosting.org)

Like the comments say, the values returned by find_peaks are the indices (or locations) of the peaks.
To find the values of these peaks, use the peak indices to get the values out of testdata_y. Then you can get the max.
%pylab inline
from scipy.signal import find_peaks
testdata = loadtxt("testdata.dat", usecols=(0,1))
testdata_x = testdata[100:200,0]
testdata_y = testdata[100:200,1]
plot(testdata_x, testdata_y)
show()
peaks = find_peaks(testdata_y)
peak_values = testdata_y[peaks[0]]
max_peak = max(peak_values)

Creating similar spectrogram in continues wavelet transform compared to discret wavelet transform

Using PyWavelets and Matplotbib.Specgram on a signal gives more detailed plots with pywt.dwt then pywt.cwt. How can I get a pywt.cwt specgram in a similar way?
With dwt:
import pywt
import pywt.data
import matplotlib.pyplot as plot
from scipy import signal
from scipy.io import wavfile
bA, bD = pywt.dwt(datamean, 'db2')
powerSpectrum, freqenciesFound, time, imageAxis = plot.specgram(bA, NFFT = 387, Fs=100)
plot.xlabel('Time')
plot.ylabel('Frequency')
plot.show()
with this spectrogram plot:
https://imgur.com/a/bYb8bBS
With cwt:
widths = np.arange(1,5)
coef, freqs = pywt.cwt(datamean, widths,'morl')
powerSpectrum, freqenciesFound, time, imageAxis = plot.specgram(coef, NFFT = 129, Fs=100)
plot.xlabel('Time')
plot.ylabel('Frequency')
plot.show()
with this spectrogram plot:
https://imgur.com/a/GIINzJp
and for better results:
sig = datamean
widths = np.arange(1, 31)
cwtmatr = signal.cwt(sig, signal.ricker, widths)
plt.imshow(cwtmatr, extent=[-1, 1, 1, 5], cmap='PRGn', aspect='auto',
vmax=abs(cwtmatr).max(), vmin=-abs(cwtmatr).max())
plt.show()
with this spectrogram plot:
https://imgur.com/a/TnXqgGR
How can I get for cwt (spectrogram plot 2 and 3) a similar spectogram plot and style like in the first one?
It seems like the 1st spectrogram plot compared to the 3rd has much more details.

This would be better as a comment, but since I lack the Karma to do that:
You don't want to make a spectrogram with wavelets, but a scalogram instead. What it looks like you're doing above is projecting your data in a scale subspace (that correlates to frequency), then taking those scales and finding the frequency content of them which is not what you probably want.
The detail and approximation coefficients are what you would want to use directly. Unfortunately, PyWavelets doesn't have a simple plotting function to do this for you, AFAIK. Matlab does, and their help page may be illuminating if I fail.
def scalogram(data):
wave='db4'
coeff=pywt.wavedec(data,wave)
levels=len(coeff)
lengths=[len(co) for co in coeff]
col=np.max(lengths)
im=np.ones([levels,col])
col=col.astype(float)
for level in range(levels):
#print [lengths[level],col]
y=coeff[level]
if lengths[1+level]<col:
x=col/(lengths[1+level]+1)*np.arange(1,len(y)+1)
xi=np.linspace(0,int(col),int(col))
yi=griddata(points=x,values=y,xi=xi,method='nearest')
else:
yi=y
im[level,:]=yi
im[im==0]=np.nan
tiles=sum(lengths)-lengths[0]
return im,tiles
Wxx,tiles=scalogram(data)
IM=plt.imshow(np.log10(abs(Wxx)),aspect='auto')
plt.show()
There are better ways of doing that, but it works. This produces a square matrix similar to spectrogram in "Wxx", and tiles is simply a counter of the number of time-frequency tilings to compare to the number used in a SFFT.
I've attached a picture of what these tilings look like

How to efficiently perform a top-hat (disk-like) smoothing on a healpix map?

I've got a high-resolution healpix map (nside = 4096) that I want to smooth in disks of a given radius, let's say 10 arcmin.
Being very new to healpy and having read the documentation I found that one - not so good - way to do this was to perform a "cone search", that is to find around each pixels the ones inside the disk, average them and give this new value to the pixel at the center. However this is very time-consuming.
import numpy as np
import healpy as hp
kappa = hp.read_map("zs_1.0334.fits") #Reading my file
NSIDE = 4096
t = 0.00290888 #10 arcmin
new_array = []
n = len(kappa)
for i in range(n):
a = hp.query_disc(NSIDE,hp.pix2vec(NSIDE,i),t)
new_array.append(np.mean(kappa[a]))
I think the healpy.sphtfunc.smoothing function could be of some help as it states that you can enter any custom beam window function but I don't understand how this works at all...
Thanks a lot for your help !

As suggested, I can easily make use of the healpy.sphtfunc.smoothing function by specifying a custom (circular) beam window.
To compute the beam window, which was my problem, healpy.sphtfunc.beam2bl is very useful and simple in the case of a top-hat.
The appropriated l_max would roughly be 2*Nside but it can be smaller depending on specific maps. One could for example compute the angular power-spectra (the Cls) and check if it dampens for smaller l than l_max which could help gain some more time.
Thanks a lot to everyone who helped in the comments section!

since I spent a certain amount of time trying to figure out how the function smoothing was working. There is a bit of code that allows you to do a top_hat smoothing.
Cheers,
import healpy as hp
import numpy as np
import matplotlib.pyplot as plt
def top_hat(b, radius):
return np.where(abs(b)<=radius, 1, 0)
nside = 128
npix = hp.nside2npix(nside)
#create a empy map
tst_map = np.zeros(npix)
#put a source in the middle of the map with value = 100
pix = hp.ang2pix(nside, np.pi/2, 0)
tst_map[pix] = 100
#Compute the window function in the harmonic spherical space which will smooth the map.
b = np.linspace(0,np.pi,10000)
bw = top_hat(b, np.radians(45)) #top_hat function of radius 45°
beam = hp.sphtfunc.beam2bl(bw, b, nside*3)
#Smooth map
tst_map_smoothed = hp.smoothing(tst_map, beam_window=beam)
hp.mollview(tst_map_smoothed)
plt.show()

Quantile-Quantile Plot using python statsmodels api

I am trying to see whether a normal distribution with specific parameters fits to a data set. However it seems qqplot does not work as it is expected to. The following small example shows this:
import numpy as np
import statsmodels.api as sm
import pylab
test = np.random.normal(20,5, 1000)
sm.qqplot(test, loc = 20, scale = 5 , line='45')
pylab.show()
As one can see I expect the points to be around the line with slope = 1 but it gives the following figure:
Can anyone explain me why this happens?

You can use line = '45' and it will work well if you have z-normalized data, meaning your distribution will have mean = 0 and sd = 1. In other cases you have several options, e.g. line = 's' or line = 'q' in case you want to see a fit against standardized line (the expected order statistics are scaled by the standard deviation of the given sample and have the mean added to them) or against line fit through the quartiles, which in my opinion is the one really meaning full and let's observe well the deviation of your data distribution from the normal one. Also, you can use line = 'r' for to see the fit to regression line. By default line is set to "None"
simply use code like this
import numpy as np
import statsmodels.api as sm
import pylab
test = np.random.normal(20, 5, 1000)
sm.qqplot(test, line='q')
pylab.show()

Please add "fit" as :
sm.qqplot(aaa, line = "45", fit = True)

I noticed that when I omitted the line='45' parameter from your code the following plot results.
We can see that what has happened is that, in the Q-Q plot that statsmodels makes the theoretical quantiles are not rescaled back to the dimensions of the original pseudosample, which is why the blue line is confined to the left edge of the your plot.
I don't know how to make statsmodels do what you want; however, there is another way — see https://stackoverflow.com/a/47189575/131187.

You can try setting the fit parameter to True

What's the correct usage of matplotlib.mlab.normpdf()?

I intend for part of a program I'm writing to automatically generate Gaussian distributions of various statistics over multiple raw text sources, however I'm having some issues generating the graphs as per the guide at:
python pylab plot normal distribution
The general gist of the plot code is as follows.
import numpy as np
import matplotlib.mlab as mlab
import matplotlib.pyplot as pyplot
meanAverage = 222.89219487179491 # typical value calculated beforehand
standardDeviation = 3.8857889432054091 # typical value calculated beforehand
x = np.linspace(-3,3,100)
pyplot.plot(x,mlab.normpdf(x,meanAverage,standardDeviation))
pyplot.show()
All it does is produce a rather flat looking and useless y = 0 line!
Can anyone see what the problem is here?
Cheers.

If you read documentation of matplotlib.mlab.normpdf, this function is deprycated and you should use scipy.stats.norm.pdf instead.
Deprecated since version 2.2: scipy.stats.norm.pdf
And because your distribution mean is about 222, you should use np.linspace(200, 220, 100).
So your code will look like:
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as pyplot
meanAverage = 222.89219487179491 # typical value calculated beforehand
standardDeviation = 3.8857889432054091 # typical value calculated beforehand
x = np.linspace(200, 220, 100)
pyplot.plot(x, norm.pdf(x, meanAverage, standardDeviation))
pyplot.show()

It looks like you made a few small but significant errors. You either are choosing your x vector wrong or you swapped your stddev and mean. Since your mean is at 222, you probably want your x vector in this area, maybe something like 150 to 300. This way you get all the good stuff, right now you are looking at -3 to 3 which is at the tail of the distribution. Hope that helps.

I see that, for the *args which are sending meanAverage, standardDeviation, the correct thing to be sent is:
mu : a numdims array of means of a
sigma : a numdims array of atandard deviation of a
Does this help?