I use scikit-learn in Python to run RandomForestClassifier(). Because I want to visualize Random Forests to realize the correlation between different features, I use export_graphviz() to achieve this goal.
estimator1 = best_model1.estimators_[0]
from sklearn.tree import export_graphviz
export_graphviz(estimator1,
'tree_from_optimized_forest.dot',
rounded = True,
feature_names=X_train.columns,
class_names = ["No", "Yes"],
filled = True)
from subprocess import call
call(['dot', '-Tpng', 'tree_from_optimized_forest.dot', '-o', 'tree_from_optimized_forest.png', '-Gdpi=200'])
from IPython.display import Image
Image('tree_from_optimized_forest.png', "w")
However, unlike Decision Tree, Random Forests will produce many trees, which are depended on the number of n_estimators in RandomForestClassifier().
best_model1 = RandomForestClassifier(n_estimators= 100,
criterion='gini',
random_state= 42,
)
Besides, because DecisionTreeClassifier() uses all the samples to produce just one tree, we can explain directly the results on this single tree.
In opposite, Random Forests is trained to make several different trees, then voting inside these trees to decide the result. In addition, the content of these trees are different because Random Forests has the methods of Bootstrap, Bagging, Out-of-bag...and so on.
Therefore, I want to ask that if I only visualize one of trees from the result of RandomForestClassifier(), whether this tree has a certain reference value?
Can I directly explain the content of this tree as the analysis result of whole data? if not, whether DecisionTreeClassifier() is the only way to analyze the correlation between features through visualized image?
Thanks a lot!!
There have always been this relation in machine learning between the model's interpret-ability and complexity and your post is directly relating to this.
Some of the models that are quite simple but are used intensively for their interpret ability is the decision trees, but since they are not complex enough (suffer from a bias), they usually fail to learn very complex function, hence people came up with the random forest classifiers. Random forests reduce the bias of the vanilla decision tree and add more variance, but unfortunately in that process they took away the straightforward interpret-ability attribute with it.
Yet, there is still some tools that could help you gain some insight on the learnt function and the contribution of the features, one of those tools is treeinterpreter, you can learn more about it in this article.
Related
I have an extremely large dataset and would like to train several random forest models on partitions of the dataset, then average these models to come up with my final classifier. Since random forest is an ensemble method, this is an intuitively sound approach but I'm unsure whether it's possible to do using scikit-learn's random forest classifier. Any ideas?
I'd also be open to using a random forest classifier from another package as well, just not sure where to look.
Here is what I can think of:
Pandas + Scikit:
You can customize your own bootstrap algorithm where you randomly read a reasonably sized sample from the overall data set, and fit scikit trees on them (would be perfect if you randomize features at each node). Then pickle each tree and finally average them out to come up with your random forest.
Graphlab + SFrame Turi has its own big data library (SFrame, similar to Pandas) and machine learning library (graphlab, very similar to scikit). Very beautiful environment.
Blaze-Dask might have a little steeper learning curve for some people, but would be an efficient solution.
You can go with the memory-mapped numpy option also but it's going to be more cumbersome than the first three options, and I've never done it so I'll just leave this option here.
All in all, I would go with option 2.
I want to implement a AdaBoost model using scikit-learn (sklearn). My question is similar to another question but it is not totally the same. As far as I understand, the random_state variable described in the documentation is for randomly splitting the training and testing sets, according to the previous link. So if I understand correctly, my classification results should not be dependent on the seeds, is it correct? Should I be worried if my classification results turn out to be dependent on the random_state variable?
Your classification scores will depend on random_state. As #Ujjwal rightly said, it is used for splitting the data into training and test test. Not just that, a lot of algorithms in scikit-learn use the random_state to select the subset of features, subsets of samples, and determine the initial weights etc.
For eg.
Tree based estimators will use the random_state for random selections of features and samples (like DecisionTreeClassifier, RandomForestClassifier).
In clustering estimators like Kmeans, random_state is used to initialize centers of clusters.
SVMs use it for initial probability estimation
Some feature selection algorithms also use it for initial selection
And many more...
Its mentioned in the documentation that:
If your code relies on a random number generator, it should never use functions like numpy.random.random or numpy.random.normal. This approach can lead to repeatability issues in tests. Instead, a numpy.random.RandomState object should be used, which is built from a random_state argument passed to the class or function.
Do read the following questions and answers for better understanding:
Choosing random_state for sklearn algorithms
confused about random_state in decision tree of scikit learn
It does matter. When your training set differs then your trained state also changes. For a different subset of data you can end up with a classifier which is little different from the one trained with some other subset.
Hence, you should use a constant seed like 0 or another integer, so that your results are reproducible.
I have been playing around with sklearn a bit and following some simple examples online using the iris data.
I've now begun to play with some other datas. I'm not sure if this behaviour is correct and I'm misunderstanding but everytime I call fit(x,y) I get completely different tree data. So when I then run predictions I get varying differences (of around 10%), ie 60%, then 70%, then 65% etc...
I ran the code below twice to output 2 trees so I could read them in Word. I tried searching values from one doc in the other and a lot of them I couldn't find.
I kind of assumed fit(x, y) would always return the same tree - if this is the case then I assume my train data of floats is punking me.
clf_dt = tree.DecisionTreeClassifier()
clf_dt.fit(x_train, y_train)
with open("output2.dot", "w") as output_file:
tree.export_graphviz(clf_dt, out_file=output_file)
There is a random component to the algorithm, which you can read about in the user guide. The relevant part:
The problem of learning an optimal decision tree is known to be NP-complete under several aspects of optimality and even for simple concepts. Consequently, practical decision-tree learning algorithms are based on heuristic algorithms such as the greedy algorithm where locally optimal decisions are made at each node. Such algorithms cannot guarantee to return the globally optimal decision tree. This can be mitigated by training multiple trees in an ensemble learner, where the features and samples are randomly sampled with replacement.
If you want to achieve the same results each time, set the random_state parameter to an integer (by default it's None) and you should get the same result each time.
I am new to all these methods and am trying to get a simple answer to that or perhaps if someone could direct me to a high level explanation somewhere on the web. My googling only returned kaggle sample codes.
Are the extratree and randomforrest essentially the same? And xgboost uses boosting when it chooses the features for any particular tree i.e. sampling the features. But then how do the other two algorithms select the features?
Thanks!
Extra-trees(ET) aka. extremely randomized trees is quite similar to random forest (RF). Both methods are bagging methods aggregating some fully grow decision trees. RF will only try to split by e.g. a third of features, but evaluate any possible break point within these features and pick the best. However, ET will only evaluate a random few break points and pick the best of these. ET can bootstrap samples to each tree or use all samples. RF must use bootstrap to work well.
xgboost is an implementation of gradient boosting and can work with decision trees, typical smaller trees. Each tree is trained to correct the residuals of previous trained trees. Gradient boosting can be more difficult to train, but can achieve a lower model bias than RF. For noisy data bagging is likely to be most promising. For low noise and complex data structures boosting is likely to be most promising.
Edit 2:
There is now a lovely example in the sklearn documentation on this.
In order to see how many trees are necessary in my forest, I'd like to plot the OOB error as the number of trees used in the forest is increased. I'm in Python using a sklearn.ensemble.RandomForestClassifier but I can't find how to predict using a subset of trees in the forest. I could do this by making a new random forest on each iteration with increasing numbers of trees but this is too expensive.
It seems a similar task is possible with the Gradient Boosting object using the staged_decision_function method. See this example.
This is quite a simple procedure in R and can be achieved by simply calling plot(randomForestObject):
--Edit--
I see now the RandomForestClassifier object has an attribute estimators_ which returns all the DecisionTreeClassifier objects in a list. So to solve this I can iterate through that list, predicting the results from each tree and taking a 'cumulative average'. However, is there really no easier way to do this already implemented?
There is a discussion and code in this issue:
https://github.com/scikit-learn/scikit-learn/issues/4273
You can add trees one-by-one like this:
n_estimators = 100
forest = RandomForestClassifier(warm_start=True, oob_score=True)
for i in range(1, n_estimators + 1):
forest.set_params(n_estimators=i)
forest.fit(X, y)
print i, forest.oob_score_
The solution you propose also needs to get the oob indices for each tree, because you don't want to compute the score on all the training data.
I still feel this is a strange thing to do as the is really no natural ordering of the trees in the forest.
Can you explain what you use-case is? Do you want to find the minimum number of trees for a given accuracy to reduce prediction time? If you want fast prediction time, I'd suggest using GradientBoostingClassifier, which is usually much faster.