We Keep Coding

Estimating a Data Histogram with multiple gaussian distributions

Good Afternoon I am trying to make an estimate of a data histogram.
To do this I am estimated the number of gaussians to fit by using Kmeans.
Then I divide the data histogram using the min and max of the data histogram along with
the total number of clusters or centroids from Kmeans.
Then I take these segments of the original data and make them into separate lists
So I can operate on them separately calculating the mean and standard deviation for each.
When displayed with the original data the mean fits and the standard deviations look ok
The peak of these gaussians are however much larger then the region of the original data the came from.
I think I want to do some kind of normalization here to make the total probability from the multiple gaussians be the same as the original data.
Can anyone with better math skills suggest a plan of attack here?
It seems like the cumulative probability for each new gaussian will sum to one over the interval
from which it was extracted, and I need to use Zscore of the original data histogram to make some kind of normalization over just the interval of each new data set I constructed.
The complication is that the y axis is in frequency not in number of observation, maybe...
I appreciate any help with this ...
I have tried many things but the formulation of my approach is likely flawed.

How to add noise to a clean dataset based on a noisy one in Python?

Essentially, I have a noisy dataset (pandas dataframe) made of sequences of geolocation data with the latitude, the longitude, the timestamps and the mean of transport (which is the label) :
dataset looks like this
Because I was lacking data to train my model, I figured out that I would use a mapping API to create fake journeys between two data points with different means of transport. It worked! Therefore, I was able to create another clean dataset with the same columns.
The problem is that my first dataset is made of GPS data I've collected myself and is therefore very noisy whereas the second dataset is perfectly clean with evenly spaced out data points since it was automatically generated. Is there any way I can add noise to that dataset based on my original noisy dataset ? Should I only add noise to the timestamps and latitude/longitude ?
I thought about using the API to generate journeys that are already present in my noisy dataset and maybe try to mix the two datasets to create noise. I could also compare them to sort of "measure" the noise by comparing the noisy dataset with the clean one.
I couldn't find much, especially in the case of sequence data. I'm not necessarily looking for code, it could be a paper or anything else. Any idea ? Thank's !

I was dealing with very similar problem and I'd say you need to analyze the original dataset, determine the standard deviation from expected correct coordinates and apply random numbers with this deviation to the generated dataset. Standard deviation from "perfect" values can be different for each of the psarameters - time, lat, lon.
to calculate the deviation:
import numpy as np
st = abs(np.std(df["lat"])) # absolute value for later use
To apply random list:
import random
low = st*-1
high = st
list_of_errors = [random.uniform(low, high) for _ in range(len(df["lat"]))]
df["lat"] = df["lat"]+list_of_errors

what is the best algorithm to cluster this data

can some one help me find a good clustering algorithm that will cluster this into 3 clusters without defining the number of clusters.
i have tried many algorithms in its basic form.. nothing seems to work properly.
clustering = AgglomerativeClustering().fit(temp)
same way i tried the dbscan and kmeans too.. just used the guidelines from sklean. i couldn't get the expected results.
my original data set is a 1D list of numbers.. but the order of the numbers matters, so generated a 2D list as bellow.
temp = []
for i in range(len(avgs)):
temp.append([avgs[i], i+1])
clustering = AgglomerativeClustering().fit(temp)
in plotting piloting i used a similter range as the y axis
ax2.scatter(range(len(plots[i])), plots[i], c=np.random.rand(3,))
the order of the data matters, so this need to clustered into 3. and there might be some other data sets that the data is very good so that the result of that need to be just one cluster.
Link to the list if someone want to try
so i tried using the step detection and got the following image according to ur answer. but how can i find the values of the peaks.. if i get the max value i can get one of them.. but how to get the rest of it.. the second max is not an answer because the one right next to the max is the second max

Your data is not 2d coordinates. So don't choose an algorithm designed for that!
Instead your data appears to be sequential or time series.
What you want to use is a change point detection algorithm, capable of detecting a change in the mean value of a series.
A simple approach would be to compute the sum of the next 10 points minus the sum of the previous 10 points, then look for extreme values of this curve.

Is there a way to partially cluster data which contains clusters but does not wholly consist of clusters?

I have some 2D data (x,y) and I need to identify where there are many data points all close to each other in the x direction. There are 3 obvious clusters where all the x points are close together and the rest of the data does not fall into them. I was going to use a k-means clustering algorithm but that seems to be for clustering ALL of the data whereas I just want to label the 3 clusters data in the data that are obviously clusters and label the rest as normal data.
The data is in separate csv files which I process and then read into one big dataframe. So far while processing the data, I have filtered out files where processed data exceeds a certain length but this obviously means that sometimes part of the cluster is left out of the file or normal data is left out.

You could try something like DBSCAN which allows classification of points as "noise", and seems to be what you're after. There's a hierarchical version of this affiliated with the scikit project known as hdbscan
Google finds are various documents describing alternatives to k-means clustering.
The hdbscan docs also have a good description of comparing alternatives.

Simple machine learning problem (SVM, random forest

I am trying to solve a machine learning task but have encountered some problems. Any tips would be greatly appreciated. One of my questions is, how do you create a correlation matrix for 2 dataframes (data for 2 labels) of different sizes, to see if you can combine them into one.
Here is the whole text of the task
This dataset is composed of 1100 samples with 30 features each. The first column is the sample id. The second column in the dataset represents the label. There are 4 possible values for the labels. The remaining columns are numeric features.
Notice that the classes are unbalanced: some labels are more frequent than others. You need to decide whether to take this into account, and if so how.
Compare the performance of a Support-Vector Machine (implemented by sklearn.svm.LinearSVC) with that of a RandomForest (implemented by sklearn.ensemble.ExtraTreesClassifier). Try to optimize both algorithms' parameters and determine which one is best for this dataset. At the end of the analysis, you should have chosen an algorithm and its optimal set of parameters.
I have tried to make a correlation matrix for rows with the labels with lower cardinality but I am not convinced it is reliable
I tried to make two new dataframes from the rows that have labels 1 and 2. There are 100-150 entries for each of those 2 labels, compared to 400 for labels 0 and 3. I wanted to check if there is high correlation bewteen data labeled 1 and 2 to see if i could combine them but dont know if this is the right approach.I tried to make the dataframes the same size by appending zeros to the smaller one and then did a correlation matrix for both datasets together. is this a correct approach

your question and approach is not clear. can you modify the question with problem statement and few data sets that you have been given.
If you wanted to visualize your data set please plot them into 2,3 or 4 dimensions.
Here are many plotting tools like 3D scatter plot, pair plot, histogram and may more. use them to better understand your data sets.