Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I'm wondering if it is possible to include scikit-learn outlier detections like isolation forests in scikit-learn's pipelines?
So the problem here is that we want to fit such an object only on the training data and do nothing on the test data. Particularly, one might want to use cross-validation here.
How could a solution look like?
Build a class that inherits from TransformerMixin (and BaseEstimator for ParameterTuning).
Now define a fit_transform function that stores the state if the function has been called yet or not. If it hasn't been called yet, the function fits and predicts the outlier function on the data. If the function has been called before, the outlier detection already has been called on the training data, thus we assume that we now find the test data which we simply return.
Does such an approach have a chance to work or am I missing something here?
Your problem is basically the outlier detection problem.
Hopefully scikit-learn provides some functions to predict whether a sample in your train set is an outlier or not.
How does it work ? If you look at the documentation, it basically says:
One common way of performing outlier detection is to assume that the regular data come from a known distribution (e.g. data are Gaussian distributed). From this assumption, we generally try to define the “shape” of the data, and can define outlying observations as observations which stand far enough from the fit shape.
sklearn provides some functions that allow you to estimate the shape of your data. Take a look at : elliptic envelope and isolation forests.
As far as I am concerned, I prefer to use the IsolationForest algorithm that returns the anomaly score of each sample in your train set. Then you can take them off your training set.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
I'm trying to understand how Gridsearchcv's logic works. I looked at here, the official documentation, and the source code, but I couldn't figure out the following:
What is the general logic behind Gridsearchcv?
Clarifications:
If I use the default cv = 5, what are the % splits of the input data
into: train, validation, and test?
How often does Gridsearchcv perform such a split, and how does it decide which observation belong to train / validation / test?
Since cross validation is being done, where does any averaging come into play for the hyper parameter tuning? i.e. is the optimal hyper parameter value is one that optimizes some sort of average?
This question here shares my concern, but I don't know how up-to-date the information is and I am not sure I understand all the information there. For example, according to the OP, my understanding is that:
The test set is 25% of the input data set and is created once.
The union of the train set and validation set is correspondingly created once and this union is 75% of the original data.
Then, the procedure creates 5 (because cv = 5) further splits of this 75% into 60% train and 15% validation
The optimized hyper parameter value is one that optimizes the average of some metric over these 5 splits.
Is this understanding correct and still applicable now? And how does the procedure do the original 25%-75% split?
First your split your data into train and test. The testing set is left out for post training and optimization of the model. The gridsearchcv takes the 75% of your data and splits them into 5 slices. First it trains 4 slices and validates on 1, then takes 4 slices introducing the previously left out set for validation and tests on a new set etc... 5 times.
Then the performance of each run can be seen + the average of them to understand overall how your model behaves.
Since you are doing a gridsearch, the best_params will be saved at the end of your modeling to predict your test set.
So to summarize, the best parameters will be chosen and used for your model after the whole training, therefore, you can easily use them to predict(X_test)
Read more here.
Usually if you don't perform CV, the model will try to optimize its weights with preset parameters and the left out test set, will help to assess the model performance. However, for a real model training, it is highly important to re-split the training data into train and validation, where you use the validation to hypertune the parameters of the model (manually). However, over-hyptertuning the model to get the best performance on the validation set is cheating.
Theoretical K-Folds
More details
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 2 years ago.
Improve this question
I have a database that I have split into train and test datasets, fitted a XGBoost model on the train set, and made predictions using the fitted model on the test set. so far everything is good.
Now if I save the fitted model and want to use it on a completely new dataset to make predictions, what should my new database look like?
Does it have to contain the exact number of features?
Does a categorical feature have to have the same categories in both databases?
I assume, you are using one-hot encoding for lets say the color-feature?
So technically to avoid extra or new features in the test-data, you should form the feature-vector using train+test data.
Do one-hot encoding/featurization on the whole set of training+testing data. Now separate out training-dataset and testing-dataset.
Lets say [v1, v2, v3... vn] are the list of feature-names from train+test data.
Now form the training-data using this feature-name. As expected the feature-column corresponding to 5th color in the training-data would all be zero and THATS FINE
Use this same features-list for the test-data, now you shouldnt have any discrepancies in terms of new features coming up.
Hope that clarifies.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
I am working on a text classification problem. I have huge amount of data and when I am trying to fit data into the machine learning model it is causing a memory error. Is there any way through which I can fit data in parts to avoid memory error.
Additional information
I am using linearSVC model.
I have training data of 1.1 million rows.
I have vectorized text data using tfidf.
The shape of vectorized data (1121063, 4235687) which has to be
fitted into the model.
Or is there any other way out of this problem.
Unfortunately, I don't have any reproducible code for the same.
Thanks in advance.
The simple answer is not to use what I assume is the scikit-learn implementation of linearSVC and instead use some algorithm/implementation that allows training in batches. Most common of which are neural networks, but several other algorithms exists. In scikit-learn look for classifiers with the partial_fit method which will allow you to fit your classifier in batches. See e.g. this list
You could also try what's suggested from sklearn.svm import SVC (the second part, the first is using LinearSVC, which you did):
For large datasets consider using :class:'sklearn.svm.LinearSVC' or
:class:'sklearn.linear_model.SGDClassifier' instead, possibily after a :class:'sklearn.kernel_approximation.Nystroem' transformer.
If you check SGDClassifier() you can set the parameter "warm_start=True" so when you iterate trough your dataset it won't lose it's state.:
clf = SGDClassifier(warm_start=True)
for i in 'loop your data':
clf.fit(data[i])
Additionally you could reduce the dimension of your dataset by removing some words from your TFIDF model. Check the "max_df" and "min_df" parameters, they'll remove words with frequency higher than or lower than, can be a % or an unit.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 3 years ago.
Improve this question
Probably I'm missing something obvious--when I detrend my timeseries target data my model preforms way better. That's great. However, I'm trying to forecast an entire cycle and the trend ~is~ important. Is there a way to reconstitute the trend with these better scores or am I shooting in the foot by removing the trend in the first place?
mean absolute error with trend intact are on order of 0.001-0.003, with trend removed the scores are around 0.0001
Please provide more information.
What kind of model do you use?
Can you give an example of the time series e.g. pd.Series(data=[100,110,120,130,140])?
Have you checked for overfitting, meaning your model performs good on your current dataset but once new data comes in it performs really poor.
Does your time series really have a trend, or does it more or less move sideways (plot-wise speaking)?
Also you can combine different models, for example a linear model model might be a good choice for simulating the trend. Once you implemented the linear trend model you can add another model which tries to predict where the linear trend model is wrong. So esentially you could add a random forest algorithm which predicts the residuals of the linear model.
After you got both models you can simly sum up the prediction of both models. The linear one for the general trend and the random forest which tries to predict seasonality.
You can also look into models which recognize seasonality by nature, such as ARIMA models for example.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Could you please explain what the "fit" method in scikit-learn does? Why is it useful?
In a nutshell: fitting is equal to training. Then, after it is trained, the model can be used to make predictions, usually with a .predict() method call.
To elaborate: Fitting your model to (i.e. using the .fit() method on) the training data is essentially the training part of the modeling process. It finds the coefficients for the equation specified via the algorithm being used (take for example umutto's linear regression example, above).
Then, for a classifier, you can classify incoming data points (from a test set, or otherwise) using the predict method. Or, in the case of regression, your model will interpolate/extrapolate when predict is used on incoming data points.
It also should be noted that sometimes the "fit" nomenclature is used for non-machine-learning methods, such as scalers and other preprocessing steps. In this case, you are merely "applying" the specified function to your data, as in the case with a min-max scaler, TF-IDF, or other transformation.
Note: here are a couple of references...
fit method in python sklearn
http://scikit-learn.org/stable/tutorial/basic/tutorial.html