We Keep Coding

How can I make clusters of time frame?

I have a Pandas Dataframe of Time.
0 2020-08-01 23:59:59
1 2020-08-01 23:59:49
2 2020-08-01 20:52:17
3 2020-08-01 19:02:34
4 2020-08-01 18:38:06
I want to add a column where I want to index by making a cluster. For eg. as follows:
0 2020-08-01 23:59:59 1
1 2020-08-01 23:59:49 1
2 2020-08-01 20:52:17 2
3 2020-08-01 19:02:34 3
4 2020-08-01 18:38:06 3
I have written this for this example as we can see 3 clusters can be made, which are the nearest/closest time stamps.
from sklearn.cluster import KMeans
mat = df['datetime'].values
kmeans = KMeans(n_clusters=3)
kmeans.fit(mat.iloc[:,1:])
y_kmeans = kmeans.predict(mat.iloc[:,1:])
df['cluster'] = y_kmeans
However, the above code also didn't work. Well, I have millions of data and obviously don't know how many clusters should I need to make. I read Elbow Method can be used but not exactly sure how it can be done. Can someone direct how it can be done?

kmeans assumes that you know the number of clusters.
If you want a method that determines the number of clusters algorithmically, you can e.g. use DBSCAN which forms a cluster whenever a group of data points is "close" to each other (closeness determined by the eps parameter). If you have a large number of samples and this is very costly, you can also try to explore any clusters in the data using a smaller (representative) subset of the data.

How to combine static features with time series in forecasting

I tried to find a similar question and its answers but was not successful in doing so. That's why I'm asking a question that might be asked before:
I'm working on a problem that outputs the cumulative water production of several water wells. The features I have are both time series (water rate and pump speed as functions of time) and static (depth of the wells, latitude and longitude of the well, thickness of the water bearing zone, etc.)
My input data can be shown as below for well#1.
dynamic data:
water rate pump speed total produced water
2000-01-01 10 4 1120
2000-01-02 20 8 1140
2000-01-03 10 4 1150
2000-01-04 10 3 1160
2000-01-05 10 4 1170
static data:
depth of the well_1 = 100
latitude and longitude of the well_1 = x1, y1
thickness of the water bearing zone of well_1 = 3
My question is how a RNN model (LSTM, GRU, ...) can be built that can take both dynamic and static features?

There are multiple options, and you need to experiment which one will be optimal for your case.
Option 1: You can treat your static features as fixed temporal data. So, you make a temporal dimension for each of your static features and let LSTM handle the rest.
For example your transformed data will look like this:
water rate pump speed total produced water depth_wall
2000-01-01 10 4 1120 100
2000-01-02 20 8 1140 100
2000-01-03 10 4 1150 100
2000-01-04 10 3 1160 100
2000-01-05 10 4 1170 100
Option 2: Designing multi-head networks.
TIME_SERIES_INPUT ------> LSTM -------\
*---> MERGE / Concatenate ---> [more layers]
STATIC_INPUTS --> [FC layer/ conv] ---/
Here is a paper explaining a combining strategy: https://arxiv.org/pdf/1712.08160.pdf
Here is another paper utilizing option 2: https://www.researchgate.net/publication/337159046_Classification_of_ECG_signals_by_dot_Residual_LSTM_Network_with_data_augmentation_for_anomaly_detection
Source code for paper 2: https://github.com/zabir-nabil/dot-res-lstm

LSTM_att proposed by Machine Learning Crop Yield Models Based on Meteorological Features and Comparison with a Process-Based Modelenter link description here seems to be a good option.
It applies static features to calculate the attention to aggregate the hidden states of time series and also provides a shortcut connection between each hidden state and final state (similar to ResNet). It outperforms baseline LSTM models.

Forecasting future occurrences with Random Forest

I'm currently exploring the use of Random Forests to predict future values of occurrences (my ARIMA model gave me really bad forecasting so I'm trying to evaluate other options). I'm fully aware that the bad results might be due to the fact that I don't have a lot of data and the quality isn't the greatest. My initial data consisted simply of the number of occurrences per date. I then added separate columns representing the day, month, year, day of the week (which was later one-hot encoded) and then I also added two columns with lagged values (one of them with the value observed in the day before and another with the value observed two days before). The final data is like this:
Count Year Month Day Count-1 Count-2 Friday Monday Saturday Sunday Thursday Tuesday Wednesday
196.0 2017.0 7.0 10.0 196.0 196.0 0 1 0 0 0 0 0
264.0 2017.0 7.0 11.0 196.0 196.0 0 0 0 0 0 1 0
274.0 2017.0 7.0 12.0 264.0 196.0 0 0 0 0 0 0 1
286.0 2017.0 7.0 13.0 274.0 264.0 0 0 0 0 1 0 0
502.0 2017.0 7.0 14.0 286.0 274.0 1 0 0 0 0 0 0
... ... ... ... ... ... ... ... ... ... ... ... ...
I then trained a random forest making the count the label (what I'm trying to predict) and all the rest the features. I also made 70/30 train/test split. Trained it on the train data and then used the test set to evaluate the model (code below):
rf = RandomForestRegressor(n_estimators = 1000, random_state = 42)
rf.fit(train_features, train_labels)
predictions = rf.predict(test_features)
The results I obtained were pretty good: MAE=1.71 and Accuracy of 89.84%.
First question: is there any possibility that I'm crazily overfitting the data? I just want to make sure I'm not making some big mistake that's giving me better results than I should get.
Second question: with the model trained, how do I use RF to predict future values? My goal was to give weekly forecasts for the number occurrences but I'm kind of stuck on how to do that.
If some who's a bit better and more experienced than me at this could help, I'd be very much appreciated! Thanks

Adressing your first question, random forest might tend to overfit, but that should be checked when comparing the MAE, MSE, RMSE of your test set. What do you mean with accuracy? Your R square? However, the way to work with models is to usually make them overfit at first, so you have a decent accuracy/mse/rmse and later perform regularization techniques to deal with this overfitting by setting a high min_child_weight or low max_depth, a high n_estimators is also good.
Secondly, to use your model to predict future values, you need to use the exact same model you trained, with the dataset you want to make your prediction on. Of course the features that were given in train must match the inputs that will be given when doing the forecasting. Furthermore, keep in mind that as time passes, this new information will be very valuable to improve your model by adding this new information to your train dataset.
forecasting = rf.predict(dataset_to_be_forecasted)

Predict the 2nd Class in Multi-class classification

Hello I have a multiclass classification model ready and trained on dataset
Label Feat1 Feat2 Feat3 Feat4
Class1 10 21 12 2
Class2 3 6 7 9
Class3 14 8 8 10
Class4 1 5 5 9
I currently can use the predict function in Sckit-Learn to apply the best model for predicting the single class . So I can get column Predicted_Label. How to approach the problem in order to get a list of prediction i.e 2nd or 3rd Best Prediction
Test_Data_Set
Feat1 Feat2 Feat3 Feat4 Predicted_Label Predicted_Label_2nd_Best_Prediction
1 3 10 7 Class1 [Class1,Class4]

Please refer to this question: Understanding predict_proba from MultiOutputClassifier
You need to use predict_proba() on your model to get probabilities for every class, for every row of your training dataset. In your case you will get an array of length 4 if you have 4 classes.
You can then get the second best prediction class for the second largest probability.
Multiclass MultiOutput Classification example on sklearn documentation
Note: Every array of length 4 from predict_proba() will add up to 1.

Does the test set need data cleaning in machine learning?

I am on an interesting machine learning project about the NYC taxi data (https://s3.amazonaws.com/nyc-tlc/trip+data/green_tripdata_2017-04.csv), the target is predicting the tip amount, the raw data looks like (2 data samples):
VendorID lpep_pickup_datetime lpep_dropoff_datetime store_and_fwd_flag \
0 2 2017-04-01 00:03:54 2017-04-01 00:20:51 N
1 2 2017-04-01 00:00:29 2017-04-01 00:02:44 N
RatecodeID PULocationID DOLocationID passenger_count trip_distance \
0 1 25 14 1 5.29
1 1 263 75 1 0.76
fare_amount extra mta_tax tip_amount tolls_amount ehail_fee \
0 18.5 0.5 0.5 1.00 0.0 NaN
1 4.5 0.5 0.5 1.45 0.0 NaN
improvement_surcharge total_amount payment_type trip_type
0 0.3 20.80 1 1.0
1 0.3 7.25 1 1.0
There are five different 'payment_type', indicated by numerical number 1,2,3,4,5
I find that only when the 'payment_type' is 1, the 'tip_amount' is meaningful, 'payment_type' 2,3,4,5 all have zero tip:
for i in range(1,6):
print(raw[raw["payment_type"] == i][['tip_amount', 'payment_type']].head(2))
gives:
tip_amount payment_type
0 1.00 1
1 1.45 1
tip_amount payment_type
5 0.0 2
8 0.0 2
tip_amount payment_type
100 0.0 3
513 0.0 3
tip_amount payment_type
59 0.0 4
102 0.0 4
tip_amount payment_type
46656 0.0 5
53090 0.0 5
First question: I want to build a regression model for 'tip_amount', if i use the 'payment_type' as a feature, can the model automatically handle this kind of behavior?
Second question: We know that the 'tip_amount' is actually not zero for 'payment_type' 2,3,4,5, just not being correctly recorded, if I drop these data samples and only keep the 'payment_type' == 1, then when using the model for unseen test dataset, it can not predict 'payment_type' 2,3,4,5 to zero tip, so I have to keep the 'payment_type' as an important feature right?
Third question: Let's say I keep all different 'payment_type' data samples and the model is able to predict zero tip amount for 'payment_type' 2,3,4,5 but is this what we really want? Because the underlying true tip should not be zero, it's just how the data looks like.

A common saying for machine learning goes garbage in, garbage out. Often, feature selection and data preprocessing is more important than your model architecture.
First question:
Yes
Second question:
Since payment_type of 2, 3, 4, 5 all result in 0, why not just keep it simple. Replace all payment types that are not 1 with 0. This will let your model easily correlate 1 to being paid and 0 to not being paid. It also reduces the amount of things your model will have to learn in the future.
Third question:
If the "underlying true tip" is not reflected in the data, then it is simply impossible for your model to learn it. Whether this inaccurate representation of the truth is what we want or not what we want is a decision for you to make. Ideally you would have data that shows the actual tip.
Preprocessing your data is very important and will help your model tremendously. Besides making some changes to your payment_type features, you should also look into normalizing your data, which will help your machine learning algorithm better generalize relations between your data.