Pandas apply & map to every element of every column - python

How to apply a custom function to every element of every column if its the value is not null?
Lets say I have a data frame of 10 columns, out of which I want to apply a lower() function to every element of just 4 columns if pd.notnull(x), else just keep none as value.
I tried to use like this,
s.apply(lambda x: change_to_lowercase(x), axis = 1)
def change_to_lowercase(s):
s['A'] = s['A'].map(lambda x: x.lower() if pd.notnull(x) else x)
s['B'] = s['B'].map(lambda x: x.lower() if pd.notnull(x) else x)
s['C'] = s['C'].map(lambda x: x.lower() if pd.notnull(x) else x)
s['D'] = s['D'].map(lambda x: x.lower() if pd.notnull(x) else x)
return s
But since my columns are mixed datatype(which is NaN as float, rest as unicode). This is throwing me an error -
float has no attribute map.
How to get rid of this error?


I think you need DataFrame.applymap because working elementwise:
L = [[1.5, 'Test', np.nan, 2], ['Test', np.nan, 2,'TEST'], ['Test', np.nan,1.5, 2]]
df = pd.DataFrame(L, columns=list('abcd'))
print (df)
a b c d
0 1.5 Test NaN 2
1 Test NaN 2.0 TEST
2 Test NaN 1.5 2
cols = ['a','b']
#for python 2 change str to basestring
df[cols] = df[cols].applymap(lambda x: x.lower() if isinstance(x, str) else x)
print (df)
a b c d
0 1.5 test NaN 2
1 test NaN 2.0 TEST
2 test NaN 1.5 2


You are trying to map a Series and then in lambda you take the entire row.
You should also check for integers, floats etc that don't have a method .lower(). So the best is to check if it is a string, not just if it is not a notnull, in my opinion.
This works:
s = pd.DataFrame([{'A': 1.5, 'B':"Test", 'C': np.nan, 'D':2}])
s
A B C D
0 1.5 Test NaN 2
s1 = s.apply(lambda x: x[0].lower() if isinstance(x[0], basestring) else x[0]).copy()
s1
A 1.5
B test
C NaN
D 2
dtype: object
For python 3 to check if string isinstance(x[0], str)
To be able to select columns:
s1 = pd.DataFrame()
columns = ["A", "B"]
for column in columns:
s1[column] = s[column].apply(lambda x: x.lower() if isinstance(x, str) else x).copy()
s1
A B
0 1.5 test

Related

pandas combine columns without null keep string values

I want to combine columns without null and keep string values.
Example data:
a,b,c
123.jpg,213.jpg,987.jpg
,159.jpg,
There is my code:
cols = ['a','b','c']
df['combine_columns'] = df[cols].stack().groupby(level=0),agg(','.join)
print(df)
And the result:
a,b,c,combine_columns
123.jpg,213.jpg,987.jpg,"123.jpg,213.jpg,987.jpg"
,159.jpg,,159.jpg
But I want something like this:
a,b,c,combine_columns
123.jpg,213.jpg,987.jpg,""123.jpg","213.jpg","987.jpg""
,159.jpg,,"159.jpg"
How can I do this?

You can use apply with a list comprehension and pandas.notna as filter:
df['combine_columns'] = df.apply(lambda x: ','.join([e for e in x if pd.notna(e)]),
axis=1)
output:
a b c combine_columns
0 123.jpg 213.jpg 987.jpg 123.jpg,213.jpg,987.jpg
1 NaN 159.jpg NaN 159.jpg
Adding extra " in the string:
df['combine_columns'] = df.apply(lambda x: '"%s"' % ','.join([e for e in x if pd.notna(e)]),
axis=1)
output:
a b c combine_columns
0 123.jpg 213.jpg 987.jpg "123.jpg,213.jpg,987.jpg"
1 NaN 159.jpg NaN "159.jpg"

Fill NA values in dataframe by mapping a double indexed groupby object [duplicate]

This should be straightforward, but the closest thing I've found is this post:
pandas: Filling missing values within a group, and I still can't solve my problem....
Suppose I have the following dataframe
df = pd.DataFrame({'value': [1, np.nan, np.nan, 2, 3, 1, 3, np.nan, 3], 'name': ['A','A', 'B','B','B','B', 'C','C','C']})
name value
0 A 1
1 A NaN
2 B NaN
3 B 2
4 B 3
5 B 1
6 C 3
7 C NaN
8 C 3
and I'd like to fill in "NaN" with mean value in each "name" group, i.e.
name value
0 A 1
1 A 1
2 B 2
3 B 2
4 B 3
5 B 1
6 C 3
7 C 3
8 C 3
I'm not sure where to go after:
grouped = df.groupby('name').mean()
Thanks a bunch.

One way would be to use transform:
>>> df
name value
0 A 1
1 A NaN
2 B NaN
3 B 2
4 B 3
5 B 1
6 C 3
7 C NaN
8 C 3
>>> df["value"] = df.groupby("name").transform(lambda x: x.fillna(x.mean()))
>>> df
name value
0 A 1
1 A 1
2 B 2
3 B 2
4 B 3
5 B 1
6 C 3
7 C 3
8 C 3

fillna + groupby + transform + mean
This seems intuitive:
df['value'] = df['value'].fillna(df.groupby('name')['value'].transform('mean'))
The groupby + transform syntax maps the groupwise mean to the index of the original dataframe. This is roughly equivalent to #DSM's solution, but avoids the need to define an anonymous lambda function.

#DSM has IMO the right answer, but I'd like to share my generalization and optimization of the question: Multiple columns to group-by and having multiple value columns:
df = pd.DataFrame(
{
'category': ['X', 'X', 'X', 'X', 'X', 'X', 'Y', 'Y', 'Y'],
'name': ['A','A', 'B','B','B','B', 'C','C','C'],
'other_value': [10, np.nan, np.nan, 20, 30, 10, 30, np.nan, 30],
'value': [1, np.nan, np.nan, 2, 3, 1, 3, np.nan, 3],
}
)
... gives ...
category name other_value value
0 X A 10.0 1.0
1 X A NaN NaN
2 X B NaN NaN
3 X B 20.0 2.0
4 X B 30.0 3.0
5 X B 10.0 1.0
6 Y C 30.0 3.0
7 Y C NaN NaN
8 Y C 30.0 3.0
In this generalized case we would like to group by category and name, and impute only on value.
This can be solved as follows:
df['value'] = df.groupby(['category', 'name'])['value']\
.transform(lambda x: x.fillna(x.mean()))
Notice the column list in the group-by clause, and that we select the value column right after the group-by. This makes the transformation only be run on that particular column. You could add it to the end, but then you will run it for all columns only to throw out all but one measure column at the end. A standard SQL query planner might have been able to optimize this, but pandas (0.19.2) doesn't seem to do this.
Performance test by increasing the dataset by doing ...
big_df = None
for _ in range(10000):
if big_df is None:
big_df = df.copy()
else:
big_df = pd.concat([big_df, df])
df = big_df
... confirms that this increases the speed proportional to how many columns you don't have to impute:
import pandas as pd
from datetime import datetime
def generate_data():
...
t = datetime.now()
df = generate_data()
df['value'] = df.groupby(['category', 'name'])['value']\
.transform(lambda x: x.fillna(x.mean()))
print(datetime.now()-t)
# 0:00:00.016012
t = datetime.now()
df = generate_data()
df["value"] = df.groupby(['category', 'name'])\
.transform(lambda x: x.fillna(x.mean()))['value']
print(datetime.now()-t)
# 0:00:00.030022
On a final note you can generalize even further if you want to impute more than one column, but not all:
df[['value', 'other_value']] = df.groupby(['category', 'name'])['value', 'other_value']\
.transform(lambda x: x.fillna(x.mean()))

Shortcut:
Groupby + Apply + Lambda + Fillna + Mean
>>> df['value1']=df.groupby('name')['value'].apply(lambda x:x.fillna(x.mean()))
>>> df.isnull().sum().sum()
0
This solution still works if you want to group by multiple columns to replace missing values.
>>> df = pd.DataFrame({'value': [1, np.nan, np.nan, 2, 3, np.nan,np.nan, 4, 3],
'name': ['A','A', 'B','B','B','B', 'C','C','C'],'class':list('ppqqrrsss')})
>>> df['value']=df.groupby(['name','class'])['value'].apply(lambda x:x.fillna(x.mean()))
>>> df
value name class
0 1.0 A p
1 1.0 A p
2 2.0 B q
3 2.0 B q
4 3.0 B r
5 3.0 B r
6 3.5 C s
7 4.0 C s
8 3.0 C s

I'd do it this way
df.loc[df.value.isnull(), 'value'] = df.groupby('group').value.transform('mean')

The featured high ranked answer only works for a pandas Dataframe with only two columns. If you have a more columns case use instead:
df['Crude_Birth_rate'] = df.groupby("continent").Crude_Birth_rate.transform(
lambda x: x.fillna(x.mean()))

To summarize all above concerning the efficiency of the possible solution
I have a dataset with 97 906 rows and 48 columns.
I want to fill in 4 columns with the median of each group.
The column I want to group has 26 200 groups.
The first solution
start = time.time()
x = df_merged[continuous_variables].fillna(df_merged.groupby('domain_userid')[continuous_variables].transform('median'))
print(time.time() - start)
0.10429811477661133 seconds
The second solution
start = time.time()
for col in continuous_variables:
df_merged.loc[df_merged[col].isnull(), col] = df_merged.groupby('domain_userid')[col].transform('median')
print(time.time() - start)
0.5098445415496826 seconds
The next solution I only performed on a subset since it was running too long.
start = time.time()
for col in continuous_variables:
x = df_merged.head(10000).groupby('domain_userid')[col].transform(lambda x: x.fillna(x.median()))
print(time.time() - start)
11.685635566711426 seconds
The following solution follows the same logic as above.
start = time.time()
x = df_merged.head(10000).groupby('domain_userid')[continuous_variables].transform(lambda x: x.fillna(x.median()))
print(time.time() - start)
42.630549907684326 seconds
So it's quite important to choose the right method.
Bear in mind that I noticed once a column was not a numeric the times were going up exponentially (makes sense as I was computing the median).

def groupMeanValue(group):
group['value'] = group['value'].fillna(group['value'].mean())
return group
dft = df.groupby("name").transform(groupMeanValue)

I know that is an old question. But I am quite surprised by the unanimity of apply/lambda answers here.
Generally speaking, that is the second worst thing to do after iterating rows, from timing point of view.
What I would do here is
df.loc[df['value'].isna(), 'value'] = df.groupby('name')['value'].transform('mean')
Or using fillna
df['value'] = df['value'].fillna(df.groupby('name')['value'].transform('mean'))
I've checked with timeit (because, again, unanimity for apply/lambda based solution made me doubt my instinct). And that is indeed 2.5 faster than the most upvoted solutions.

To fill all the numeric null values with the mean grouped by "name"
num_cols = df.select_dtypes(exclude='object').columns
df[num_cols] = df.groupby("name").transform(lambda x: x.fillna(x.mean()))

df.fillna(df.groupby(['name'], as_index=False).mean(), inplace=True)

You can also use "dataframe or table_name".apply(lambda x: x.fillna(x.mean())).

Fillna in pandas with respect to similar line [duplicate]

This should be straightforward, but the closest thing I've found is this post:
pandas: Filling missing values within a group, and I still can't solve my problem....
Suppose I have the following dataframe
df = pd.DataFrame({'value': [1, np.nan, np.nan, 2, 3, 1, 3, np.nan, 3], 'name': ['A','A', 'B','B','B','B', 'C','C','C']})
name value
0 A 1
1 A NaN
2 B NaN
3 B 2
4 B 3
5 B 1
6 C 3
7 C NaN
8 C 3
and I'd like to fill in "NaN" with mean value in each "name" group, i.e.
name value
0 A 1
1 A 1
2 B 2
3 B 2
4 B 3
5 B 1
6 C 3
7 C 3
8 C 3
I'm not sure where to go after:
grouped = df.groupby('name').mean()
Thanks a bunch.

One way would be to use transform:
>>> df
name value
0 A 1
1 A NaN
2 B NaN
3 B 2
4 B 3
5 B 1
6 C 3
7 C NaN
8 C 3
>>> df["value"] = df.groupby("name").transform(lambda x: x.fillna(x.mean()))
>>> df
name value
0 A 1
1 A 1
2 B 2
3 B 2
4 B 3
5 B 1
6 C 3
7 C 3
8 C 3

fillna + groupby + transform + mean
This seems intuitive:
df['value'] = df['value'].fillna(df.groupby('name')['value'].transform('mean'))
The groupby + transform syntax maps the groupwise mean to the index of the original dataframe. This is roughly equivalent to #DSM's solution, but avoids the need to define an anonymous lambda function.

#DSM has IMO the right answer, but I'd like to share my generalization and optimization of the question: Multiple columns to group-by and having multiple value columns:
df = pd.DataFrame(
{
'category': ['X', 'X', 'X', 'X', 'X', 'X', 'Y', 'Y', 'Y'],
'name': ['A','A', 'B','B','B','B', 'C','C','C'],
'other_value': [10, np.nan, np.nan, 20, 30, 10, 30, np.nan, 30],
'value': [1, np.nan, np.nan, 2, 3, 1, 3, np.nan, 3],
}
)
... gives ...
category name other_value value
0 X A 10.0 1.0
1 X A NaN NaN
2 X B NaN NaN
3 X B 20.0 2.0
4 X B 30.0 3.0
5 X B 10.0 1.0
6 Y C 30.0 3.0
7 Y C NaN NaN
8 Y C 30.0 3.0
In this generalized case we would like to group by category and name, and impute only on value.
This can be solved as follows:
df['value'] = df.groupby(['category', 'name'])['value']\
.transform(lambda x: x.fillna(x.mean()))
Notice the column list in the group-by clause, and that we select the value column right after the group-by. This makes the transformation only be run on that particular column. You could add it to the end, but then you will run it for all columns only to throw out all but one measure column at the end. A standard SQL query planner might have been able to optimize this, but pandas (0.19.2) doesn't seem to do this.
Performance test by increasing the dataset by doing ...
big_df = None
for _ in range(10000):
if big_df is None:
big_df = df.copy()
else:
big_df = pd.concat([big_df, df])
df = big_df
... confirms that this increases the speed proportional to how many columns you don't have to impute:
import pandas as pd
from datetime import datetime
def generate_data():
...
t = datetime.now()
df = generate_data()
df['value'] = df.groupby(['category', 'name'])['value']\
.transform(lambda x: x.fillna(x.mean()))
print(datetime.now()-t)
# 0:00:00.016012
t = datetime.now()
df = generate_data()
df["value"] = df.groupby(['category', 'name'])\
.transform(lambda x: x.fillna(x.mean()))['value']
print(datetime.now()-t)
# 0:00:00.030022
On a final note you can generalize even further if you want to impute more than one column, but not all:
df[['value', 'other_value']] = df.groupby(['category', 'name'])['value', 'other_value']\
.transform(lambda x: x.fillna(x.mean()))

Shortcut:
Groupby + Apply + Lambda + Fillna + Mean
>>> df['value1']=df.groupby('name')['value'].apply(lambda x:x.fillna(x.mean()))
>>> df.isnull().sum().sum()
0
This solution still works if you want to group by multiple columns to replace missing values.
>>> df = pd.DataFrame({'value': [1, np.nan, np.nan, 2, 3, np.nan,np.nan, 4, 3],
'name': ['A','A', 'B','B','B','B', 'C','C','C'],'class':list('ppqqrrsss')})
>>> df['value']=df.groupby(['name','class'])['value'].apply(lambda x:x.fillna(x.mean()))
>>> df
value name class
0 1.0 A p
1 1.0 A p
2 2.0 B q
3 2.0 B q
4 3.0 B r
5 3.0 B r
6 3.5 C s
7 4.0 C s
8 3.0 C s

I'd do it this way
df.loc[df.value.isnull(), 'value'] = df.groupby('group').value.transform('mean')

The featured high ranked answer only works for a pandas Dataframe with only two columns. If you have a more columns case use instead:
df['Crude_Birth_rate'] = df.groupby("continent").Crude_Birth_rate.transform(
lambda x: x.fillna(x.mean()))

To summarize all above concerning the efficiency of the possible solution
I have a dataset with 97 906 rows and 48 columns.
I want to fill in 4 columns with the median of each group.
The column I want to group has 26 200 groups.
The first solution
start = time.time()
x = df_merged[continuous_variables].fillna(df_merged.groupby('domain_userid')[continuous_variables].transform('median'))
print(time.time() - start)
0.10429811477661133 seconds
The second solution
start = time.time()
for col in continuous_variables:
df_merged.loc[df_merged[col].isnull(), col] = df_merged.groupby('domain_userid')[col].transform('median')
print(time.time() - start)
0.5098445415496826 seconds
The next solution I only performed on a subset since it was running too long.
start = time.time()
for col in continuous_variables:
x = df_merged.head(10000).groupby('domain_userid')[col].transform(lambda x: x.fillna(x.median()))
print(time.time() - start)
11.685635566711426 seconds
The following solution follows the same logic as above.
start = time.time()
x = df_merged.head(10000).groupby('domain_userid')[continuous_variables].transform(lambda x: x.fillna(x.median()))
print(time.time() - start)
42.630549907684326 seconds
So it's quite important to choose the right method.
Bear in mind that I noticed once a column was not a numeric the times were going up exponentially (makes sense as I was computing the median).

def groupMeanValue(group):
group['value'] = group['value'].fillna(group['value'].mean())
return group
dft = df.groupby("name").transform(groupMeanValue)

I know that is an old question. But I am quite surprised by the unanimity of apply/lambda answers here.
Generally speaking, that is the second worst thing to do after iterating rows, from timing point of view.
What I would do here is
df.loc[df['value'].isna(), 'value'] = df.groupby('name')['value'].transform('mean')
Or using fillna
df['value'] = df['value'].fillna(df.groupby('name')['value'].transform('mean'))
I've checked with timeit (because, again, unanimity for apply/lambda based solution made me doubt my instinct). And that is indeed 2.5 faster than the most upvoted solutions.

To fill all the numeric null values with the mean grouped by "name"
num_cols = df.select_dtypes(exclude='object').columns
df[num_cols] = df.groupby("name").transform(lambda x: x.fillna(x.mean()))

df.fillna(df.groupby(['name'], as_index=False).mean(), inplace=True)

You can also use "dataframe or table_name".apply(lambda x: x.fillna(x.mean())).

how to remove 0's from a string without impacting other cells in pandas data frame?

I have a data frame which has "0's" and looks as below:
df = pd.DataFrame({
'WARNING':['4402,43527,0,7628,54337',4402,0,0,'0,1234,56437,76252',0,3602],
'FAILED':[0,0,'5555,6753,0','4572,0,8764,8753',9876,0,'0,4579,7514']
})
I want to remove the zeroes from the strings where there are multiple values such that the results df looks like this:
df = pd.DataFrame({
'WARNING':['4402,43527,7628,54337',4402,0,0,'1234,56437,76252',0,3602],
'FAILED':[0,0,'5555,6753','4572,8764,8753',9876,0,'4579,7514']
})
However the ones which have individual 0's in a cell should remain intact. How do I achieve this?

df = pd.DataFrame({
'WARNING':['0,0786,1230,01234,0',4402,0,0,'0,1234,56437,76252',0,3602],
'FAILED':[0,0,'5555,6753,0','4572,0,8764,8753',9876,0,'0,4579,7514']
})
df.apply(lambda x: x.str.strip('0,|,0')).replace(",0,", ",")
Output:
WARNING FAILED
0 786,1230,01234 NaN
1 NaN NaN
2 NaN 5555,6753
3 NaN 4572,0,8764,8753
4 1234,56437,76252 NaN
5 NaN NaN
6 NaN 4579,7514

I would solve it with a list comprehension.
In [1]: df.apply(lambda col: col.astype(str).apply(lambda x: ','.join([y for y in x.split(',') if y != '0']) if ',' in x else x), axis=0)
Out[1]: 
FAILED WARNING
0 0 4402,43527,7628,54337
1 0 4402
2 5555,6753 0
3 4572,8764,8753 0
4 9876 1234,56437,76252
5 0 0
6 4579,7514 3602
Breaking it down:
Iterate over all columns with df.apply(lambda col: ..., axis=0)
Convert each column's values to string with col.astype(str)
Apply a function to each "cell" of col with .apply(lambda x: ...)
The lambda function first checks if ',' exists in x, otherwise returns the original value of x
If ',' in x, it splits x by ',', which creates a list of y's
It keeps only the y != '0'
It joins everything at the end with a ','.join(...)

You can use regex with a negative look behind to replace 0, only if it not preceded by another digit.
import re
df.applymap(lambda x: re.sub(r'(?<![0-9])0,', '', str(x)))
WARNING FAILED
0 4402,43527,7628,54337 0
1 4402 0
2 0 5555,6753,0
3 0 4572,8764,8753
4 1234,56437,76252 9876
5 0 0
6 3602 4579,7514
For the test case W-B points out:
s = '0,0999,9990,999'
re.sub(r'(?<![0-9])0,', '', s)
#'0999,9990,999'

Replacing empty values with NaN in object/categorical variables

So I've searched SO for this and found a bunch of useful threads on how to replace empty values with NaN. However I can't get any of them to work on my DataFrame.
I've used:
df.replace('', np.NaN)
df3 = df.applymap(lambda x: np.nan if x == '' else x)
and even:
df.iloc[:,86:350] = df.iloc[:,86:350].apply(lambda x: x.str.strip()).replace('', np.nan)
and the code runs fine without error but when I look in my dataframe i still have b'' values instead of NaN. Any ideas on what I am missing?
I'm sorry for not giving the code to reproduce this as I don't know how to do that as I suspect it's specific to my dataframe which I imported from SPSS and these values were string variables in SPSS if that helps.

You were close with your second try:
df = df.applymap(lambda x: np.NaN if not x else x)
To show that both '' and b'' will evaluate to True in the conditional:
l = ['', b'']
for x in l:
if x:
print ('Not empty')
else:
print ('Empty')
>>> Empty
>>> Empty
Sample:
from pandas import DataFrame
from numpy import NaN
df = DataFrame([[1,2,''], ['',b'',3], [4, 5, b'']])
print (df)
# Output
0 1 2
0 1 2
1 b'' 3
2 4 5 b''
df2 = df.applymap(lambda x: NaN if not x else x)
print (df2)
# Output
0 1 2
0 1 2 NaN
1 NaN NaN 3
2 4 5 NaN

Categories