I need to one hot encode categorical variables on my pandas data frame.
My dataset is really big with over 2000 productIDs to be one hot encoded.
I tried pd.get_dummies and it always crashes.
I have also tried scikit-learn's OneHotEncoder which also crashes! (it works fine with a smaller subset of dataframe)
What other methods are there? What is the most efficient way to one hot encode categorical variables for very big data set?
My data frame:
Month User ProductID
1 A ProdA
3 A ProdB
11 A ProdC
12 A ProdD
Required output:
Month User ProdA ProdB ProdC ProdD
1 A 1 0 0 0
3 A 0 1 0 0
11 A 0 0 1 0
12 A 0 0 0 1
My dataset is really big with over 2000 productIDs and million of user rows.
This will result in a huge dataset. Presumably it's crashing because of memory.
Perhaps you should consider alternatives to full one-hot encoding.
One way is to create dummies of the top categories, and "other" for the rest.
tops = df.ProductID.value_counts().head(10)
will give you the top product ids. You can then use
df.ProductID[~df.ProductID.isin(tops)] = 'other'
and create dummies out of that.
If you have a response variable, you might alternatively use mean encoding.
For a feature with so many different possible values, one-hot encoding may not be the best option.
I suggest using Target Encoding (https://contrib.scikit-learn.org/categorical-encoding/). Unlike one-hot encoding, which will create k columns for k unique values of the feature, target encoding transforms the one feature into one column.
Related
I have twitter data that I want to cluster. It is text data and I learned that K means can not handle Non-Numerical data. I wanted to cluster data just on the basis of the tweets. The data looks like this.
I found this code that can converts the text into numerical data.
def handle_non_numerical_data(df):
columns = df.columns.values
for column in columns:
text_digit_vals = {}
def convert_to_int(val):
return text_digit_vals[val]
if df[column].dtype != np.int64 and df[column].dtype != np.float64:
column_contents = df[column].values.tolist()
unique_elements = set(column_contents)
x = 0
for unique in unique_elements:
if unique not in text_digit_vals:
text_digit_vals[unique] = x
x += 1
df[column] = list(map(convert_to_int, df[column]))
return df
df = handle_non_numerical_data(data)
print(df.head())
output
label tweet
0 9 24
1 5 11
2 17 45
3 14 138
4 18 112
I'm quite new to this and I don't think this is what I need to fit the data. What is a better way to handle Non-Numerical data (text) of this nature?
Edit: When running K means clustering Algorithm on raw text data I get this error.
ValueError: could not convert string to float
The most typical way of handling non-numerical data is to convert a single column into multiple binary columns. This is called "getting dummy variables" or a "one hot encoding" (among many other snobby terms).
There are other things you can do to translate the data to numbers, such as sentiment analysis (i.e. cetagorize each tweet into happy, sad, funny, angry, etc...), analyzing the tweets to determine if they are about a certain subject or not (i.e. Does this tweet talk about a virus?), the number of words in each tweet, the number of spaces per tweet, if it has good grammar or not, etc. As you can see, you are asking about a very broad subject.
When transforming data to binary columns, you get the number of unique values in your column and make that many new columns, each one of them filled with zeros and ones.
Let's focus on your first column:
import pandas as pd
df = pd.DataFrame({'account':['realdonaldtrump','naredramodi','pontifex','pmoindia','potus']})
account
0 realdonaldtrump
1 narendramodi
2 pontifex
3 pmoindia
4 potus
This is equivalent to:
pd.get_dummies(df, columns=['account'], prefix='account')
account_naredramodi account_pmoindia account_pontifex account_potus \
0 0 0 0 0
1 1 0 0 0
2 0 0 1 0
3 0 1 0 0
4 0 0 0 1
account_realdonaldtrump
0 1
1 0
2 0
3 0
4 0
This is one of many methods. You can check out this article about one hot encoding here.
NOTE: When you have many unique values, doing this will give you many columns and some algorithms will crash due to not having enough degrees of freedom (too many variables, not enough observations). Last, if you are running a regression, you will run into perfect multicollinearity if you do not drop one of the columns.
Going back to your example, if you want to turn all your columns into this kind of data, try:
pd.get_dummies(df)
However, I wouldn't do this for the tweet column because each tweet is its own unique value.
As k-means is a method of vector quantization, you should vectorize your textual data in one way or another.
See some examples of using k-means over text:
Word2Vec
tf-idf
I am cleaning a csv file on jupyter to do machine learning.
However, several columns have string values, like the column "description":
I know I need to use NLP to clean, but could not find how to do it on jupyter.
Could you advice me how to convert these values to numeric values?
Thank you
Numerical values are better for creating learning models than words or images.(Why? dimensionality reduction)
Common machine learning algorithms expect a numerical input.
The technique used to convert a word to a corresponding numerical value is called word embedding.
In word embedding, strings are converted to feature vectors(numbers).
Bag of words, word2vec, GloVe can be used for implementing this.
It is generally advisable to ignore those fields which wouldn't be significant for the model.So include description only if is absolutely essential.
The problem you are describing is that of converting categorical data, usually in the form of strings or numerical ID's to purely numerical data. I'm sure you are aware that using numerical ID's has a problem: it leads to the false interpretation that the data has some sort of order. Like apple < orange < lime, when this is not the case.
It is common to use one-hot encoding to produce numerical indicator variables. After encoding one column, you have N columns, where N is the amount of unique labels. The columns have a value of 1 when the corresponding categorical variable had that value and 0 otherwise. This is especially handy if there are few unique labels in one column. Both Pandas and sklearn have these sorts of functions available, albeit they are not as feature complete as one would hope.
The "description" column you have seems to be a bit trickier, because it actually includes language, not just categorical data. So that column would need to be parsed or handled in some other way. Although, the one-hot encoding scheme may very well be used for all the words in the description, producing a vector that has more 1's.
For example:
>>> import pandas as pd
>>> df = pd.DataFrame(['a', 'b', 'c', 'a', 'a', pd.np.nan])
>>> pd.get_dummies(df)
0_a 0_b 0_c
0 1 0 0
1 0 1 0
2 0 0 1
3 1 0 0
4 1 0 0
5 0 0 0
Additional processing would be needed to get the encoding word by word. This approach considers only the full values as variables.
I'm building a random forest in python using sklearn-learn, and I've applied "one hot" encoding to all of the categorical variables. Question: if I apply "one hot" to my DV,
do I apply all of its dummy columns as the DV, or should the DV be handled differently?
You need to apply one-hot encoding to all those columns where the values are not in numbers.You can handle DV with one-hot and other non-numerical columns with some other encoding as well. E.g: Suppose there is a column with city names, you need to change this into numerical form. This is called as DATA MOLDING. You can do this molding without one-hot as well.
E.g: there is DV column for diabetes with entry "yes" and "no". This is without one-hot encoding.
diabetes_map = {True : 1, False : 0}
df['diabetes'] = df['diabetes'].map(diabetes_map)
Depends on the type of problem you have. For binary or multi-class problems, you do not need to one hot encode dependent variable in scikit-learn. Doing one-hot encoding will change the shape of the output variable from single dimension to multi-dimensions. This is called as label-indicator matrix, where each column denotes the presence or absence of that label.
For example, doing one-hot encoding of the following:
['high', 'medium', 'low', 'high', 'low', 'high', 'medium']
will return this:
high medium low
1 0 0
0 1 0
0 0 1
1 0 0
0 0 1
1 0 0
0 1 0
Not all classifiers in scikit-learn are able to support this format, (even though they support multi-class classification) Even in those that do support this, this will trigger the multi-label classification (in which more than one label can be present at once) which is what you dont want in a multi-class problem.
i have an unbalanced data set which has two categorical values. one has around 500 values of a particular class and other is only one single datapoint with another class.Now i would like to split this data into test train with 80-20 ratio. but since this is unbalanced , i would like to have the second class to be present in both the test and train data.
I tried using the test-train-split from sklearn, but it does not give the second class data to be present in both of them. I even tried the stratified shuffle split, but that was also not giving data as i thought.
Is there any way we can split the data from a data frame forcing both the test and train datasets to have the single datapoint?. I am new to python so having difficulty figuring it out.
the data looks like:
a b c d label
1 0 0 1 1
1 1 1 0 1
..........
........
1 0 0 1 0.
the label has only 1 and 0 but the 0 is only one single observation but the rest of the 500 data points are having label as 1
With the information you gave, I suggest you to down/over sampling the data in order to give an equal weight to both your classes and then split your dataset as you like. Take a look at this library exposing different algorithms to deal with unbalanced data in python
Try to do oversampling as you have less amount of data points. Or else you can use neural network preferably MLP, That works fine with unbalanced data.
I have data of the form:
Feature 1 Feature 2 Feature 3 ---> Numerical Value
Problem is Feature 1 is like, String Values like Company Names, Feature 2 is also a string value like a Category and Feature 3 is just timestamp.
I want to train a model that given the features is able to predict the numerical value.
I know regression can be used for it.
But,
How do I convert the categorical features so that they can be used in regression?
How do I utilize the timestamp value for Prediction? Should I extract the month, the hour number (line from 0-23) and make them into more categorical values?
Thanks.
As we know machine learning algorithm are not capable to understand the text directly,so we need to convert these string values into one hot vector representation.
we use one hot encoder to perform “binarization” of the category and include it as a feature to train the model
So you can use pandas for this,
For example
import pandas as pd
df =pd.DataFrame({'A':["google","amazon","microsoft"]})
pd.get_dummies(df)
A_amazon A_google A_microsoft
0 1 0
1 0 0
0 0 1
After converting your variable into above format you can apply regression
Thanks