I am trying to generate a white noise field with a specified power spectral density in V**2/Hz, however I am having issues when I try to validate my attempt.
In Python, here is a quick example of what I am doing - I am not sure what I am doing wrong!
from scipy.signal import spectrogram
import numpy as np
n = 10000
fs = 1024
# Desired power spectral density = 60dB re 1 V**2 / Hz
amp = 10.0**( 60.0 / 10.0 )
# Frequency resolution = 1 / duration
f_res = fs / n
# Normalize amplitude to a per Hz bin
amp *= f_res
# Construct uniformly random phases
phases = np.random.rand(n) * 2 * np.pi
spectrum = amp * (np.cos(phases) + 1j * np.sin(phases))
# Set the 0-Hz offset to 0
spectrum[0] = 0 + 0j
# Create timeseries using ifft:
ts = np.fft.ifft(spectrum)
# Use scipy spectrogram to validate approach
f, t, a = spectrogram(ts, fs, window='hann', scaling='density', mode='psd', return_onesided=False)
print(10.0*np.log10(np.mean(a)))
Running this, I get an answer of ~30 dB - not what I expected! I expected 60 dB. If I change the desired PSD to 70 dB, I get 50 dB, and if I change the desired to 90 dB, I get approximately 90 dB.
I feel like I am missing a scaling factor somewhere, or I am interpreting something incorrectly. Can anyone help me out here?
Related
The Situation
I am currently writing a program that will later on be used to analyze a signal that is somewhat of a asymmetric Gaussian. I am interested in how many frequencies I need to reproduce the signal somewhat exact and especially the amplitudes of those frequencies.
Before I input the real data I'm testing the program with a default (asymmetric) Gaussian, as can be seen in the code below.
My Problem
To ensure I that get the amplitudes right, I am rebuilding the original signal using the whole frequency spectrum, but there are some difficulties. I get to reproduce the signal somewhat well multiplying amp with 0.16, which I got by looking at the fraction rebuild/original. Of course, this is really unsatisfying and can't be the correct solution.
To be precise the difference is not dependant on the time length and seems to be a Gaussian too, following the form of the original, increasing in asymmetry according to the Skewnorm function itself. The amplitude of the difference function is correlated linear to 'height'.
My Question
I am writing this post because I am out of ideas for getting the amplitude right. Maybe anyone has had the same / a similar problem and can share their solution for this / give a hint.
Further information
Before focusing on a (asymmetric) Gaussian I analyzed periodic signals and rectangular pulses, which sadly were very unstable to variations in the time length of the input signal. In this context, I experimented with window functions, which seemed to speed up the process and increase the stability, the reason being that I had to integrate the peaks. Working with the Gaussian I got told to take each peak, received via the bare fft and ditch the integration approach, therefore my incertitude considering the amplitude described above. Maybe anyone got an opinion on the approach chosen by me and if necessary can deliver an improvement.
Code
from numpy.fft import fft, fftfreq
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import skewnorm
np.random.seed(1234)
def data():
height = 1
data = height * skewnorm.pdf(t, a=0, loc=t[int(N/2)])
# noise_power = 1E-6
# noise = np.random.normal(scale=np.sqrt(noise_power), size=t.shape)
# data += noise
return data
def fft_own(data):
freq = fftfreq(N, dt)
data_fft = fft(data) * np.pi
amp = 2/N * np.abs(data_fft) # * factor (depending on t1)
# amp = 2/T * np.abs(data_fft)**2
phase = np.angle(data_fft)
peaks, = np.where(amp >= 0) # use whole spectrum for rebuild
return freq, amp, phase, peaks
def rebuild(fft_own):
freq, amp, phase, peaks = fft_own
df = freq[1] - freq[0]
data_rebuild = 0
for i in peaks:
amplitude = amp[i] * df
# amplitude = amp[i] * 0.1
# amplitude = np.sqrt(amp[i] * df)
data_rebuild += amplitude * np.exp(0+1j * (2*np.pi * freq[i] * t
+ phase[i]))
f, ax = plt.subplots(1, 1)
# mask = (t >= 0) & (t <= t1-1)
ax.plot(t, data_init, label="initial signal")
ax.plot(t, np.real(data_rebuild), label="rebuild")
# ax.plot(t[mask], (data_init - np.real(data_rebuild))[mask], label="diff")
ax.set_xlim(0, t1-1)
ax.legend()
t0 = 0
t1 = 10 # diff(t0, t1) ∝ df
# T = t1- t0
N = 4096
t = np.linspace(t0, t1, int(N))
dt = (t1 - t0) / N
data_init = data()
fft_init = fft_own(data_init)
rebuild_init = rebuild(fft_init)
You should get a perfect reconstruction if you divide amp by N, and remove all your other factors.
Currently you do:
data_fft = fft(data) * np.pi # Multiply by pi
amp = 2/N * np.abs(data_fft) # Multiply by 2/N
amplitude = amp[i] * df # Multiply by df = 1/(dt*N) = 1/10
This means that you currently multiply by a total of pi * 2 / 10, or 0.628, that you shouldn't (only the 1/N factor in there is correct).
Correct code:
def fft_own(data):
freq = fftfreq(N, dt)
data_fft = fft(data)
amp = np.abs(data_fft) / N
phase = np.angle(data_fft)
peaks, = np.where(amp >= 0) # use whole spectrum for rebuild
return freq, amp, phase, peaks
def rebuild(fft_own):
freq, amp, phase, peaks = fft_own
data_rebuild = 0
for i in peaks:
data_rebuild += amp[i] * np.exp(0+1j * (2*np.pi * freq[i] * t
+ phase[i]))
Your program can be significantly simplified by using ifft. Simply set to 0 those frequencies in data_fft that you don't want to include in the reconstruction, and apply ifft to it:
data_fft = fft(data)
data_fft[np.abs(data_fft) < threshold] = 0
rebuild = ifft(data_fft).real
Note that the Fourier transform of a Gaussian is a Gaussian, so you won't be picking out individual peaks, you are picking a compact range of frequencies that will always include 0. This is an ideal low-pass filter.
How to apply a lowpass filter, with cutoff frequency varying linearly (or with a more general curve than linear) from e.g. 10000hz to 200hz along time, with numpy/scipy and possibly no other library?
Example:
at 00:00,000, lowpass cutoff = 10000hz
at 00:05,000, lowpass cutoff = 5000hz
at 00:09,000, lowpass cutoff = 1000hz
then cutoff stays at 1000hz during 10 seconds, then cutoff decreases down to 200hz
Here is how to do a simple 100hz lowpass:
from scipy.io import wavfile
import numpy as np
from scipy.signal import butter, lfilter
sr, x = wavfile.read('test.wav')
b, a = butter(2, 100.0 / sr, btype='low') # Butterworth
y = lfilter(b, a, x)
wavfile.write('out.wav', sr, np.asarray(y, dtype=np.int16))
but how to make the cutoff vary?
Note: I've already read Applying time-variant filter in Python but the answer is quite complex (and it applies to many kinds of filter in general).
One comparatively easy method is to keep the filter fixed and modulate signal time instead. For example, if signal time runs 10x faster a 10KHz lowpass will act like a 1KHz lowpass in standard time.
To do this we need to solve a simple ODE
dy 1
-- = ----
dt f(y)
Here t is modulated time y real time and f the desired cutoff at time y.
Prototype implementation:
from __future__ import division
import numpy as np
from scipy import integrate, interpolate
from scipy.signal import butter, lfilter, spectrogram
slack_l, slack = 0.1, 1
cutoff = 50
L = 25
from scipy.io import wavfile
sr, x = wavfile.read('capriccio.wav')
x = x[:(L + slack) * sr, 0]
x = x
# sr = 44100
# x = np.random.normal(size=((L + slack) * sr,))
b, a = butter(2, 2 * cutoff / sr, btype='low') # Butterworth
# cutoff function
def f(t):
return (10000 - 1000 * np.clip(t, 0, 9) - 1000 * np.clip(t-19, 0, 0.8)) \
/ cutoff
# and its reciprocal
def fr(_, t):
return cutoff / (10000 - 1000 * t.clip(0, 9) - 1000 * (t-19).clip(0, 0.8))
# modulate time
# calculate upper end of td first
tdmax = integrate.quad(f, 0, L + slack_l, points=[9, 19, 19.8])[0]
span = (0, tdmax)
t = np.arange(x.size) / sr
tdinfo = integrate.solve_ivp(fr, span, np.zeros((1,)),
t_eval=np.arange(0, span[-1], 1 / sr),
vectorized=True)
td = tdinfo.y.ravel()
# modulate signal
xd = interpolate.interp1d(t, x)(td)
# and linearly filter
yd = lfilter(b, a, xd)
# modulate signal back to linear time
y = interpolate.interp1d(td, yd)(t[:-sr*slack])
# check
import pylab
xa, ya, z = spectrogram(y, sr)
pylab.pcolor(ya, xa, z, vmax=2**8, cmap='nipy_spectral')
pylab.savefig('tst.png')
wavfile.write('capriccio_vandalized.wav', sr, y.astype(np.int16))
Sample output:
Spectrogram of first 25 seconds of BWV 826 Capriccio filtered with a time varying lowpass implemented via time bending.
you can use scipy.fftpack.fftfreq and scipy.fftpack.rfft to set thresholds
fft = scipy.fftpack.fft(sound)
freqs = scipy.fftpack.fftfreq(sound.size, time_step)
for the time_step I did twice the sampling rate of the sound
fft[(freqs < 200)] = 0
this would set all set all frequencies less than 200 hz to zero
for the time varying cut off, I'd split the sound and apply the filters then. assuming the sound has a sampling rate of 44100, the 5000hz filter would start at sample 220500 (five seconds in)
10ksound = sound[:220500]
10kfreq = scipy.fftpack.fftreq(10ksound.size, time_step)
10kfft = scipy.fftpack.fft(10ksound)
10kfft[(10kfreqs < 10000)] = 0
then for the next filter:
5ksound = sound[220500:396900]
5kfreq = scipy.fftpack.fftreq(10ksound.size, time_step)
5kfft = scipy.fftpack.fft(10ksound)
5kfft[(5kfreqs < 5000)] = 0
etc
edit: to make it "sliding" or a gradual filter instead of piece wise, you could make the "pieces" much smaller and apply increasingly bigger frequency thresholds to the corresponding piece(5000 -> 5001 -> 5002)
I'm trying to create a vibrato by oscillating between two 430Hz and 450Hz, storing the 16-bit sample in the list wav. However, the audible frequency seems to increase range of oscillation across the entire clip. Does anyone know why?
edit: rewrote code to be more clear/concise
# vibrato.py
maxamp = 2**15 - 1 # max signed short
wav = []
(t, dt) = (0, 1 / 44100)
while t < 6.0:
f = 440 + 10 * math.sin(2 * math.pi * 6 * t)
samp = maxamp * math.sin(2 * math.pi * f * t)
wav.append(samp)
t += dt
--
Update: because the response uses numpy, I'll update my code for plain python3
# vibrato.py
maxamp = 2**15 - 1 # max signed short
wav = []
(t, dt) = (0, 1 / 44100)
phase = 0
while t < 6.0:
f = 440 + 10 * math.sin(2 * math.pi * 6 * t)
phase += 2 * math.pi * f * t
samp = maxamp * math.sin(phase)
wav.append(samp)
t += dt
The issue has to do with an implied phase change that goes along with changing the frequency. In short, when you calculate the response relative to each point in a timeline, it's important to note that the phase of the oscillation will be different for each frequency at each time (except at the starting point where they're all the same). Therefore, moving between frequencies is like moving between different phases. For the case of moving between two distinct frequencies, this can be corrected for post hoc by adjusting the overall signal phases based on the frequency change. I've explained this in another answer so won't explain it again here, but here just show the initial plot that highlights the problem, and how to fix the issue. Here, the main thing added is the importance of a good diagnostic plot, and the right plot for this is a spectrogram.
Here's an example:
import numpy as np
dt = 1./44100
time = np.arange(0., 6., dt)
frequency = 440. - 10*np.sin(2*math.pi*time*1.) # a 1Hz oscillation
waveform = np.sin(2*math.pi*time*frequency)
Pxx, freqs, bins, im = plt.specgram(waveform, NFFT=4*1024, Fs=44100, noverlap=90, cmap=plt.cm.gist_heat)
plt.show()
Note that the span of the frequency oscillation is increasing (as you initially heard). Applying the correction linked to above gives:
dt = 1./defaults['framerate']
time = np.arange(0., 6., dt)
frequency = 440. - 10*np.sin(2*math.pi*time*1.) # a 1Hz oscillation
phase_correction = np.add.accumulate(time*np.concatenate((np.zeros(1), 2*np.pi*(frequency[:-1]-frequency[1:]))))
waveform = np.sin(2*math.pi*time*frequency + phase_correction)
Which is much closer to what was intended, I hope.
Another way to conceptualize this, which might make more sense in the context of looping through each time step (as the OP does), and as closer to the physical model, is to keep track of the phase at each step and determine the new amplitude considering both the amplitude and phase from the previous step, and combining these with the new frequency. I don't have the patience to let this run in pure Python, but in numpy the solution looks like this, and gives a similar result:
dt = 1./44100
time = np.arange(0., 6., dt)
f = 440. - 10*np.sin(2*math.pi*time*1.) # a 1Hz oscillation
delta_phase = 2 * math.pi * f * dt
phase = np.cumsum(delta_phase) # add up the phase differences along timeline (same as np.add.accumulate)
wav = np.sin(phase)
I am trying to apply a Hann window to a sinusoidal signal with the idea of applying an FFT to recover the frequency and the amplitude. This is a canonical case I have created to increase my understanding before I move onto my data (real time signal where I want to accurately determine the frequency content and amplitude). In the code below I need to multiple by an additional factor of 2.0 to recover the amplitudes. I understand multiplying by 2/N but now I am multiplying by 4/N. Has anyone come across this or can anyone provide an explanation to why this is? Here is my code:
import numpy as np
import matplotlib.pyplot as plt
# first create the time signal, which has two frequencies 13.2 hz and 43.9 hz
f_s = 100.0 # Hz sampling frequency
f = 1.0 # Hz
time = np.arange(0.0, 10.0, 1/f_s)
x = 5 * np.sin(13.2 * 2 * np.pi * f * time) + 3 * np.sin(43.9 * 2 * np.pi * f * time)
x = x + np.random.randn(len(time)) #inject some noise
# apply hann window and take the FFT
win = np.hanning(len(x))
FFT = np.fft.fft(win * x) * 2.0 # IT SEEMS I NEED AN ADDITIONAL FACTOR OF 2 TO RECOVER THE AMPLITUDES
n = len(FFT)
freq_hanned = np.fft.fftfreq(n, 1/f_s)
half_n = np.ceil(n/2.0)
fft_hanned_half = (2.0 / n) * FFT[:half_n]
freq_hanned_half = freq_hanned[:half_n]
# and plot
plt.plot(freq_hanned_half, np.abs(fft_hanned_half))
plt.xlabel("Frequency (Hz)")
plt.ylabel("Amplitude")
The mean value of the von Hann window is (approximately) 0.5, for N=1000 you have
>>> N=1000 ; print sum(np.hanning(N))/N
0.4995
>>>
Does this explain the necessity of multiplying by two to recover the discrete amplitudes?
I'm new to Python programming and I wanted to know if there was a way to create a high-pass filter for a periodic function like so:
import numpy as np
from scipy.signal import lfilter, firwin, butter
from pylab import figure, plot, show
sample_rate = .0167
nsamples = 480
F_1Hz = 1.38e-4
A_1Hz = 1.0
F_15Hz = .0011
A_15Hz = .5
t = np.arange(nsamples) / sample_rate
signal = A_1Hz * np.sin(2*np.pi*F_1Hz*t) + A_15Hz*np.sin(2*np.pi*F_15Hz*t)
signal[::120] = 2
figure(1)
plot(t,signal,'b')
show()
I want to keep the higher frequency ( .0011 Hz) as well as the spikes of 2 at the certain spots, however the amplitudes of the .0011 Hz needs to stay at .5 and the spikes need to stay at an amplitude of 2, so normalizing isn't an option. Moreover, if I made the function have the spikes of 2 at a non-periodic intervals(say a spike at only signal[prime numbers]) could I still filter it correctly, with the correct amplitudes?
One possibility is to use a custom high-pass filter. A simple way to make a high-pass filter is to start with a low-pass filter:
def lp_win_sinc(tw, fc, n):
m = int(np.ceil( 2./tw) * 2)
samps = np.arange(m+1)
shift = samps - m/2
shift[m/2] = 1
h = np.sin(2 * np.pi * fc * shift)/shift
h[m/2] = 2 * np.pi * fc
h = h * np.blackman(m+1)
h = h / h.sum()
s = np.zeros(n)
s[:len(h)] = h
return np.roll(s, -m/2)
Then construct a simple high-pass
def hp_win_sinc(tw, fc, n):
hp = -lp_win_sinc(tw, fc, n)
hp[0] = hp[0] + 1
return hp
(The ideas behind these are found in http://www.dspguide.com/pdfbook.htm, look at the chapter on windowed-sinc filters.)
Note: these are the impulse responses of the respective filters. To apply them to your data you can either convolve the impulse with your data, or you can fft your data and the impulse response and take the inverse fft of their product. In your case, e.g.
hp = hp_win_sinc(0.2, 0.001, len(signal))
f_hp = np.fft.rfft(hp)
f_d = np.fft.rfft(signal)
filt_sig = np.fft.irfft( f_hp * f_d)
plotting this quick result gives:
filtered data
Depending on your exact application, you might be able to simply adjust the gain to recover the 2.0 and 0.5 amplitudes. Hope this helps. Good luck!
The answer is quite likely no.
The reason behind this blunt answer is that your spikes (which have a value of 2) stand on top of the signal. If you filter anything away, your signal amplitude may change at the spikes.
If you could change this:
signal[::120] = 2
into
signal[::120] += 2
then such a filter can be constructed. What do you want to filter away? Anything below .0011 Hz?