We Keep Coding

How to go about data modelling?

I've spent the last 2 hours or so figuring out how to apply it to my two variables. I am supposed to demonstrate/explain how I would handle the relationship of the two following variables in data modelling:
Pressure24h DangerLevel24h
1000.2 45
1014.8 90
990.8 14
998.4 95
1002.1 46
1006 21
There is another 185,000 data to work with but that's just a very small sample of it. Pressure24h is measured in hectopascals and DangerLevel24h is measured in percentage. That's the only information I have to work with.
Is there any method that can be used to approach this?
I created a scatter plot to show the relationship but that was as far as I have gotten so far.
https://i.stack.imgur.com/Ty5Yn.png

Here's my code as discussed in the comments:
def lobf(*cords):
cords = cords[0]
print(cords)
x_mean, y_mean = 0, 0
print(cords)
for x, y in cords:
x_mean += x # get x sum
y_mean += y # get y sum
x_mean /= len(cords) # get x mean
y_mean /= len(cords) # get y mean
# Step 2 from https://www.varsitytutors.com/hotmath/hotmath_help/topics/line-of-best-fit
sigma_numerator, sigma_denominator = 0, 0
for xi, yi in cords:
sigma_numerator += (xi - x_mean)*(yi - y_mean) # get numerator
sigma_denominator += (xi - x_mean)**2 # get denominator
m = sigma_numerator/sigma_denominator # get slope
c = y_mean - m*x_mean # get y-intercept
return m ,c
data_values = [(2,2), (4,4)] # Sample data value you can put yours here
# Creating a for loop for every increment of 5 to avoid the blue blob you got.
# You can change the increment as per your choice
predicted_values = []
increment = 5
m ,c = lobf(data_values)
for i in range(data_values[-1][0]+increment, len(data_values)*100, increment): # You can consider dangerlevel24h as your x
"""
Starts with incrementing the last x value of your data
"""
predicted_values.append((i,i*m + c)) # appends x, y
print(predicted_values)
You can then plot every value from predicted_values. By iterating through every 5th or your desired iteration, you can avoid blue blobs to form. Also, this method will help you in predicting future values that aren't in your data. You can also try using Pearson's Theory of Correlation which is related to this method.

How to calculate average values of list elements in python?

I'd like to calculate average of each values in a list. To do so, I wrote a function which gets list as parameter and calculate the average and returns the list of average again.
Here is the signal:
random_data = [10 * random.uniform(0,1) for i in range(1000)]
random_peak = [100 * random.uniform(0,1) for i in range(50)] + [0] * 950
random.shuffle(peak)
for i in range(0, len(signal)):
signal = [peak[x] + random_data[x] for x in range(len(random_data))]
And now, I'd like to calculate m as following.
'''
m1 = 1/(number of signal) * x1
m2 = 1/(number of signal) * (x1+x2)
m3 = 1/(number of signal) * (x1+x2+x3)
...
'''
I wrote a following function to calculate m. How would I change the function to return list of m s?
def mean_values(s):
for i in range(len(s)):
m[i] = 1/len(s)*s[i]
return m[i]
mean_values(signal)
#mean_values(np.array(signal)

use m as a float instead of list it make more sense
to get mean
s = 1 / n * Σxi
you can use this to get new mean from a previous one
s' = s + (x1 - s) / n1
where s is the lastest mean, x1 the new value and n1 the new length
However in numpy their is a prebuilt function np.mean() which do that and manage python list too

Frequencies from a FFT shift based on size of data set?

I am working on finding the frequencies from a given dataset and I am struggling to understand how np.fft.fft() works. I thought I had a working script but ran into a weird issue that I cannot understand.
I have a dataset that is roughly sinusoidal and I wanted to understand what frequencies the signal is composed of. Once I took the FFT, I got this plot:
However, when I take the same dataset, slice it in half, and plot the same thing, I get this:
I do not understand why the frequency drops from 144kHz to 128kHz which technically should be the same dataset but with a smaller length.
I can confirm a few things:
Step size between data points 0.001
I have tried interpolation with little luck.
If I slice the second half of the dataset I get a different frequency as well.
If my dataset is indeed composed of both 128 and 144kHz, then why doesn't the 128 peak show up in the first plot?
What is even more confusing is that I am running a script with pure sine waves without issues:
T = 0.001
fs = 1 / T
def find_nearest_ind(data, value):
return (np.abs(data - value)).argmin()
x = np.arange(0, 30, T)
ff = 0.2
y = np.sin(2 * ff * np.pi * x)
x = x[:len(x) // 2]
y = y[:len(y) // 2]
n = len(y) # length of the signal
k = np.arange(n)
T = n / fs
frq = k / T * 1e6 / 1000 # two sides frequency range
frq = frq[:len(frq) // 2] # one side frequency range
Y = np.fft.fft(y) / n # dft and normalization
Y = Y[:n // 2]
frq = frq[:50]
Y = Y[:50]
fig, (ax1, ax2) = plt.subplots(2)
ax1.plot(x, y)
ax1.set_xlabel("Time (us)")
ax1.set_ylabel("Electric Field (V / mm)")
peak_ind = find_nearest_ind(abs(Y), np.max(abs(Y)))
ax2.plot(frq, abs(Y))
ax2.axvline(frq[peak_ind], color = 'black', linestyle = '--', label = F"Frequency = {round(frq[peak_ind], 3)}kHz")
plt.legend()
plt.xlabel('Freq(kHz)')
ax1.title.set_text('dV/dX vs. Time')
ax2.title.set_text('Frequencies')
fig.tight_layout()
plt.show()

Here is a breakdown of your code, with some suggestions for improvement, and extra explanations. Working through it carefully will show you what is going on. The results you are getting are completely expected. I will propose a common solution at the end.
First set up your units correctly. I assume that you are dealing with seconds, not microseconds. You can adjust later as long as you stay consistent.
Establish the period and frequency of the sampling. This means that the Nyquist frequency for the FFT will be 500Hz:
T = 0.001 # 1ms sampling period
fs = 1 / T # 1kHz sampling frequency
Make a time domain of 30e3 points. The half domain will contain 15000 points. That implies a frequency resolution of 500Hz / 15k = 0.03333Hz.
x = np.arange(0, 30, T) # time domain
n = x.size # number of points: 30000
Before doing anything else, we can define our time domain right here. I prefer a more intuitive approach than what you are using. That way you don't have to redefine T or introduce the auxiliary variable k. But as long as the results are the same, it does not really matter:
F = np.linspace(0, 1 - 1/n, n) / T # Notice F[1] = 0.03333, as predicted
Now define the signal. You picked ff = 0.2. Notice that 0.2Hz. 0.2 / 0.03333 = 6, so you would expect to see your peak in exactly bin index 6 (F[6] == 0.2). To better illustrate what is going on, let's take ff = 0.22. This will bleed the spectrum into neighboring bins.
ff = 0.22
y = np.sin(2 * np.pi * ff * x)
Now take the FFT:
Y = np.fft.fft(y) / n
maxbin = np.abs(Y).argmax() # 7
maxF = F[maxbin] # 0.23333333: This is the nearest bin
Since your frequency bins are 0.03Hz wide, the best resolution you can expect 0.015Hz. For your real data, which has much lower resolution, the error is much larger.
Now let's take a look at what happens when you halve the data size. Among other things, the frequency resolution becomes smaller. Now you have a maximum frequency of 500Hz spread over 7.5k samples, not 15k: the resolution drops to 0.066666Hz per bin:
n2 = n // 2 # 15000
F2 = np.linspace(0, 1 - 1 / n2, n2) / T # F[1] = 0.06666
Y2 = np.fft.fft(y[:n2]) / n2
Take a look what happens to the frequency estimate:
maxbin2 = np.abs(Y2).argmax() # 3
maxF2 = F2[maxbin2] # 0.2: This is the nearest bin
Hopefully, you can see how this applies to your original data. In the full FFT, you have a resolution of ~16.1 per bin with the full data, and ~32.2kHz with the half data. So your original result is within ~±8kHz of the right peak, while the second one is within ~±16kHz. The true frequency is therefore between 136kHz and 144kHz. Another way to look at it is to compare the bins that you showed me:
full: 128.7 144.8 160.9
half: 96.6 128.7 160.9
When you take out exactly half of the data, you drop every other frequency bin. If your peak was originally closest to 144.8kHz, and you drop that bin, it will end up in either 128.7 or 160.9.
Note: Based on the bin numbers you show, I suspect that your computation of frq is a little off. Notice the 1 - 1/n in my linspace expression. You need that to get the right frequency axis: the last bin is (1 - 1/n) / T, not 1 / T, no matter how you compute it.
So how to get around this problem? The simplest solution is to do a parabolic fit on the three points around your peak. That is usually a sufficiently good estimator of the true frequency in the data when you are looking for essentially perfect sinusoids.
def peakF(F, Y):
index = np.abs(Y).argmax()
# Compute offset on normalized domain [-1, 0, 1], not F[index-1:index+2]
y = np.abs(Y[index - 1:index + 2])
# This is the offset from zero, which is the scaled offset from F[index]
vertex = (y[0] - y[2]) / (0.5 * (y[0] + y[2]) - y[1])
# F[1] is the bin resolution
return F[index] + vertex * F[1]
In case you are wondering how I got the formula for the parabola: I solved the system with x = [-1, 0, 1] and y = Y[index - 1:index + 2]. The matrix equation is
[(-1)^2 -1 1] [a] Y[index - 1]
[ 0^2 0 1] # [b] = Y[index]
[ 1^2 1 1] [c] Y[index + 1]
Computing the offset using a normalized domain and scaling afterwards is almost always more numerically stable than using whatever huge numbers you have in F[index - 1:index + 2].
You can plug the results in the example into this function to see if it works:
>>> peakF(F, Y)
0.2261613409657391
>>> peakF(F2, Y2)
0.20401580936430794
As you can see, the parabolic fit gives an improvement, however slight. There is no replacement for just increasing frequency resolution through more samples though!

Scale/Transform/Normalise NumPy Array between Two Values

I have the following scenario:
value_range = [250.0, 350.0]
precision = 0.01
unique_values = len(np.arange(min(values_range),
max(values_range) + precision,
precision))
This means all values range between 250.0 and 350.0 with a precision of 0.01, giving a potential total of 10001 unique values that the data set can have.
# This is the data I'd like to scale
values_to_scale = np.arange(min(value_range),
max(value_range) + precision,
precision)
# These are the bins I want to assign to
unique_bins = np.arange(1, unique_values + 1)
You can see in the above example, each value in values_to_scale will map exactly to its corresponding item in the unique_bins array. I.e. a value of 250.0 (values_to_scale[0]) will equal 1.0 (unique_bins[0]) etc.
However, if my values_to_scale array looks like:
values_to_scale = np.array((250.66, 342.02))
How can I do the scaling/transformation to get the unique bin value? I.e. 250.66 should equal a value of 66 but how do I obtain this?
NOTE The value_range could equally be between -1 and 1, I'm just looking for a generic way to scale/normalise data between two values.

You're basically looking for a linear interpolation between min and max:
minv = min(value_range)
maxv = max(value_range)
unique_values = int(((maxv - minv) / precision) + 1)
((values_to_scale - minv) / (maxv + precision - minv) * unique_values).astype(int)
# array([ 65, 9202])

Splitting a value with multiple bins numpy histogram

I'm trying to program the process on this image:
On the image the 2 on the right-side is mapped to bin "80" since its corresponding value on the left-side is 80. The 4 on the right-side however has a corresponding value of 10 on the left-side, and because there is no bin for 10, the 4 needs to get split into two values.
To accomplish this I am using numpy's histogram with the "weight" parameter like this:
t1 = [80, 10]
t2 = [2, 4]
bins = np.arange(0, 200, 20)
h = np.histogram(t1,bins=bins,weights=t2)
The 2 gets mapped correctly, but the 4 gets mapped entirely to bin 0 (leftmost).
Output:
[4 0 0 0 2 0 0 0 0]
I think is due to the fact that the first bin is responsible for all directions in a range (0 to 20), instead of giving the magnitude when the direction doesn't equal to the exact same number as the bin.
So, I was wondering if anybody knows how I can rewrite this so the output will be:
[2 2 0 0 2 0 0 0 0]

Let's consider an easier task first:
Assume you would want to quantize the gradient direction (GD) as follows: floor(GD/20). You could use the following:
h = np.bincount(np.floor(GD.reshape((-1)) / 20).astype(np.int64), GM.reshape((-1)).astype(np.float64), minlength=13)
Where np.bincount simply accumulates the gradient magnitude (GM) based on the quantized gradient direction (GD). Notice that binlength controls the length of the histogram and it equals ceil(255/20).
However, you wanted soft assignment so you have to weight the GM contribution, you might want to try:
GD = GD.reshape((-1))
GM = GM.reshape((-1))
w = ((GD / 20) - np.floor(GD / 20)).astype(np.float64)
h1 = np.bincount(np.floor(GD / 20).astype(np.int64), GM.astype(np.float64) * (1.0-w), minlength=13)
h2 = np.bincount(np.ceil(GD / 20).astype(np.int64), GM.astype(np.float64) * w, minlength=13)
h = h1 + h2
p.s one might want to consider the np.bincount documentation https://docs.scipy.org/doc/numpy/reference/generated/numpy.bincount.html

Refer to Roy Jevnisek's answer, minlength should be 9 as there are 9 bins.
Also, since 180 degree is equivalent to 0 degree, the last element of h should be omitted and treated as the first element of h, as both the first and last elements of h represent the weighted count of 0 degree, ie:
h[0] = h[-1]
h = h[:-1]
Then the HOG can be plotted by:
GD = GD.reshape(-1)
GM = GM.reshape(-1)
w1 = (GD / 20) - np.floor(GD / 20)
w2 = np.ceil(GD / 20) - (GD / 20)
h1 = np.bincount(np.floor(GD / 20).astype('int32'), GM * w2, minlength=9)
h2 = np.bincount(np.ceil(GD / 20).astype('int32'), GM * w1, minlength=9)
h = h1 + h2
h[0] = h[-1]
h = h[:-1]
values = np.unique(np.floor(GD / 20).astype(np.int64))[:-1]
plt.title('Histogram of Oriented Gradients (HOG)')
plt.bar(values, h)
plt.show()