I have some data in the form of a panda dataframe that has the columns watercolor (string), place (string), temperature(float).
I want to use one hot encoding to turn the data into categories like
color: darkblue, lightblue, teal
1 0 0
0 1 0
For the strings it is no problem, but how do I set the intervals for the temperature(float)?
I tried writing:
output = pd.get_dummies(df.astype(str))
The problem is that all the unique float values are turned into a separate category like:
temperature: 37,6 37,7 37,9 38
0 1 0 0
1 0 0 0
That means that my program will overfit the data since all the temperatures are turned into separate categories. I would like to specify the interval for the third column(temperature). So I want to group all values from say 37,5-39 and from 39-41,5 and so on.
try use cut before creating the dummies columns
pd.cut(df['temperature'], [37.5, 39, 41,.....], labels=['37.5-39', '39-41',.....])
Related
I have a dataframe similar to the one below, in which activities assume binary values representing whether they require a doctor:
d = {'activity': ['Check-up', 'Assessment', 'Medication', 'Medication', 'Medication'], 'doctor_requirement': [1, 0,0,0,0]}
df = pd.DataFrame(data=d)
df
activity doctor_requirement
0 Check-up 1
1 Assessment 0
2 Medication 0
3 Medication 0
4 Medication 0
I would like to consider that a percentage of 'Medication' activities require a doctor. That is, to assign binary 1 to doctor_requirement for a percentage of 'Medication' visits. For instance, such that 50% of the activity 'Medication' requires a doctor (i.e. doctor_requirement = 1).
I would greatly appreciate your help, I've been looking online and can't seem to find how to apply such a condition.
Thanks in advance!
If you'd like a 50% chance then you can use:
df.loc[df['activity']=='Medication','doctor_requirement'] = np.random.choice([0,1],(df['activity']=='Medication').sum())
If you wish to contorl the probabilities for 0 and 1s, you can use np.random.choice's p parameter to specify odds.
df.loc[df['activity']=='Medication','doctor_requirement'] = np.random.choice([0,1],(df['activity']=='Medication').sum(),p=[0.99,0.01])
I have a Pandas data frame with different columns.
Some columns are just “yes” or “no” answers that I would need to replace with 1 and 0
Some columns are 1s and 2s, where 2 equals no - these 2 need to be replaced by 0
Other columns are Numerical categories, for example 1,2,3,4,5 where 1 = lion 2 = dog
Other columns are string categories, like: “A lot”, “A little” etc
The first 2 columns are the target variables
My problem issues:
If I just change all 2 to 0 in the data frame, it would end up changing the 2 in the target variable (which in this case act as a score rather than a “No”)
Another problem would be that columns with categories as numbers, will have their 2s changed to 0 as well
How can I clean this dataframe so that
2. all columns with either yes or 1 and those with either no or 2 -> become 1 and 0s
3. the two target variables -> stay as scores from 1-5
4. and all categorical variables remain unchanged until I do onehot encoding with them.
These are the steps I took:
To change all the “yes” or “no” to 0 and 1
df.replace(('Yes', 'No'), (1, 0), inplace=True)
Now in order to replace all the 2s that act as “No”s with 0s -
without it affecting neither the “2” that act as a score in first two target columns
nor the “2” that act as a category value in columns that have more than 2 unique values, I think I would need to combine the following two lines of code, is that correct? I am trying different ways to combine them but I keep getting errors
df.loc[:, df.nunique() <= 2] or df[df.columns.difference([‘target1 ‘,’target2 '])].replace(2, 0)
It would be better if you showed your code here and a sample of the database. I'm a bit confused. Here is what I gleaned:
First, here is a dummy dataset I created:
Here is the code that I think solves your two problems. If there is something missing, it's because I didn't quite get the explanation as I said.
import pandas as pd
import numpy as np
import os
filename = os.path.join(os.path.dirname(__file__),'data.csv')
sample = pd.read_csv(filename)
# This solves your first problem. Here we create a new column using numeric values instead of yes/no string values
#with a function
def create_answers_column(sample, colname):
def is_yes(a):
if a == 'yes':
return 1
else:
return 0
return sample[colname].apply(is_yes)
sample['Answers Numeric'] = create_answers_column(sample, 'Answers')
#This solves your second problem
#Using replace()
sample['Numbers'] = sample.Numbers.replace({2:0})
print(sample)
And here's the output:
Answers Numbers Animals Quantifiers Answers Numeric
0 yes 1 1 a lot 1
1 yes 0 2 little 1
2 no 0 3 many 0
3 yes 1 4 some 1
4 no 1 5 several 0
I have a dataset of stations
map_id longitude latitude zip_code
0 40830 -87.669147 41.857908 60608
1 40830 -87.669147 41.857908 60608
2 40120 -87.680622 41.829353 60609
3 40120 -87.680622 41.829353 60609
4 41120 -87.625826 41.831677 60616
As you can see, the first four rows are duplications and it is not an accident. They are the same stations, which are treated as separate stations of different lines.
I would like to eliminate such duplicates (it can be 2 or even 5 rows for some stations) and treat it as one station.
Moreover, I would like to create a new column "Hub", where aggregated rows will be treated a hub station. For example, as a boolean (0 for a regular station, 1 for a hub).
The desired output for the sample above with two cases of duplication -> transformed into 3 rows with 2 hubs.
map_id longitude latitude zip_code hub
0 40830 -87.669147 41.857908 60608 1
1 40120 -87.680622 41.829353 60609 1
1 41120 -87.625826 41.831677 60616 0
I appreciate any tips!
Looks to me like you want to drop duplicates and assign certain zipcodes as hub. If so, I would drop duplicates and use np.where to assign hubs. I included a non existent opcode to demonstrate how you can do this if more than one zipcode is designated as a hub
import numpy as np
df2=df.drop_duplicates(subset=['map_id','longitude','latitude','zip_code'], keep='first')
conditions=df2['zip_code'].isin(['60616','60619'])
df2['hub']=np.where(conditions,0,1)
df = df.groupby(['map_id','longitude','latitude','zip_code']).size().reset_index(name='hub')
df['hub'] = df['hub'].replace(1,0).apply(lambda x:min(x,1))
Output
map_id longitude latitude zip_code hub
0 40120 -87.680622 41.829353 60609 1
1 40830 -87.669147 41.857908 60608 1
2 41120 -87.625826 41.831677 60616 0
I need to one hot encode categorical variables on my pandas data frame.
My dataset is really big with over 2000 productIDs to be one hot encoded.
I tried pd.get_dummies and it always crashes.
I have also tried scikit-learn's OneHotEncoder which also crashes! (it works fine with a smaller subset of dataframe)
What other methods are there? What is the most efficient way to one hot encode categorical variables for very big data set?
My data frame:
Month User ProductID
1 A ProdA
3 A ProdB
11 A ProdC
12 A ProdD
Required output:
Month User ProdA ProdB ProdC ProdD
1 A 1 0 0 0
3 A 0 1 0 0
11 A 0 0 1 0
12 A 0 0 0 1
My dataset is really big with over 2000 productIDs and million of user rows.
This will result in a huge dataset. Presumably it's crashing because of memory.
Perhaps you should consider alternatives to full one-hot encoding.
One way is to create dummies of the top categories, and "other" for the rest.
tops = df.ProductID.value_counts().head(10)
will give you the top product ids. You can then use
df.ProductID[~df.ProductID.isin(tops)] = 'other'
and create dummies out of that.
If you have a response variable, you might alternatively use mean encoding.
For a feature with so many different possible values, one-hot encoding may not be the best option.
I suggest using Target Encoding (https://contrib.scikit-learn.org/categorical-encoding/). Unlike one-hot encoding, which will create k columns for k unique values of the feature, target encoding transforms the one feature into one column.
I have twitter data that I want to cluster. It is text data and I learned that K means can not handle Non-Numerical data. I wanted to cluster data just on the basis of the tweets. The data looks like this.
I found this code that can converts the text into numerical data.
def handle_non_numerical_data(df):
columns = df.columns.values
for column in columns:
text_digit_vals = {}
def convert_to_int(val):
return text_digit_vals[val]
if df[column].dtype != np.int64 and df[column].dtype != np.float64:
column_contents = df[column].values.tolist()
unique_elements = set(column_contents)
x = 0
for unique in unique_elements:
if unique not in text_digit_vals:
text_digit_vals[unique] = x
x += 1
df[column] = list(map(convert_to_int, df[column]))
return df
df = handle_non_numerical_data(data)
print(df.head())
output
label tweet
0 9 24
1 5 11
2 17 45
3 14 138
4 18 112
I'm quite new to this and I don't think this is what I need to fit the data. What is a better way to handle Non-Numerical data (text) of this nature?
Edit: When running K means clustering Algorithm on raw text data I get this error.
ValueError: could not convert string to float
The most typical way of handling non-numerical data is to convert a single column into multiple binary columns. This is called "getting dummy variables" or a "one hot encoding" (among many other snobby terms).
There are other things you can do to translate the data to numbers, such as sentiment analysis (i.e. cetagorize each tweet into happy, sad, funny, angry, etc...), analyzing the tweets to determine if they are about a certain subject or not (i.e. Does this tweet talk about a virus?), the number of words in each tweet, the number of spaces per tweet, if it has good grammar or not, etc. As you can see, you are asking about a very broad subject.
When transforming data to binary columns, you get the number of unique values in your column and make that many new columns, each one of them filled with zeros and ones.
Let's focus on your first column:
import pandas as pd
df = pd.DataFrame({'account':['realdonaldtrump','naredramodi','pontifex','pmoindia','potus']})
account
0 realdonaldtrump
1 narendramodi
2 pontifex
3 pmoindia
4 potus
This is equivalent to:
pd.get_dummies(df, columns=['account'], prefix='account')
account_naredramodi account_pmoindia account_pontifex account_potus \
0 0 0 0 0
1 1 0 0 0
2 0 0 1 0
3 0 1 0 0
4 0 0 0 1
account_realdonaldtrump
0 1
1 0
2 0
3 0
4 0
This is one of many methods. You can check out this article about one hot encoding here.
NOTE: When you have many unique values, doing this will give you many columns and some algorithms will crash due to not having enough degrees of freedom (too many variables, not enough observations). Last, if you are running a regression, you will run into perfect multicollinearity if you do not drop one of the columns.
Going back to your example, if you want to turn all your columns into this kind of data, try:
pd.get_dummies(df)
However, I wouldn't do this for the tweet column because each tweet is its own unique value.
As k-means is a method of vector quantization, you should vectorize your textual data in one way or another.
See some examples of using k-means over text:
Word2Vec
tf-idf