We Keep Coding

Filtering a column on pandas based on a string

I'm trying to filter a column on pandas based on a string, but the issue that I'm facing is that the rows are lists and not only strings.
A small example of the column
tags
['get_mail_mail']
['app', 'oflline_hub', 'smart_home']
['get_mail_mail', 'smart_home']
['web']
[]
[]
['get_mail_mail']
and I'm using this
df[df["tags"].str.contains("smart_home", case=False, na=False)]
but it's returning an empty dataframe.

You can explode, then compare and aggregate with groupby.any:
m = (df['tags'].explode()
.str.contains('smart_home', case=False, na=False)
.groupby(level=0).any()
)
out = df[m]
Or concatenate the string with a delimiter and use str.contains:
out = df[df['tags'].agg('|'.join).str.contains('smart_home')]
Or use a list comprehension:
out = df[[any(s=='smart_home' for s in l) for l in df['tags']]]
output:
tags
1 [app, oflline_hub, smart_home]
2 [get_mail_mail, smart_home]

You could try:
# define list of searching patterns
pattern = ["smart_home"]
df.loc[(df.apply(lambda x: any(m in str(v)
for v in x.values
for m in pattern),
axis=1))]
Output
tags
-- ------------------------------------
1 ['app', 'oflline_hub', 'smart_home']
2 ['get_mail_mail', 'smart_home']

Extracting the integers from a column of strings

I have 2 dataframes: longdf, and shortdf. Longdf is the ‘master’ list and I need to basically match values from shortdf to longdf, those that match, replace values in other columns. Both longdf and shortdf need extensive data cleaning.
The goal is to reach the df ‘goal.’ I was trying to use a for loop where I wanted to 1) extract all number in the df cell, and 2) strip the blank/cell spaces from the cell. First: How come this for loop doesn't work? Second: Is there a better way to do this?
import pandas as pd
a = pd.Series(['EY', 'BAIN', 'KPMG', 'EY'])
b = pd.Series([' 10wow this is terrible data8 ', '10/ USED TO BE ANOTHER NUMBER/ 2', ' OMG 106 OMG ', ' 10?7'])
y = pd.Series(['BAIN', 'KPMG', 'EY', 'EY' ])
z = pd.Series([108, 102, 106, 107 ])
goal = pd.DataFrame
shortdf = pd.DataFrame({'consultant': a, 'invoice_number':b})
longdf = shortdf.copy(deep=True)
goal = pd.DataFrame({'consultant': y, 'invoice_number':z})
shortinvoice = shortdf['invoice_number']
longinvoice = longdf['invoice_number']
frames = [shortinvoice, longinvoice]
new_list=[]
for eachitemer in frames:
eachitemer.str.extract('(\d+)').astype(float) #extracing all numbers in the df cell
eachitemer.str.strip() #strip the blank/whitespaces in between the numbers
new_list.append(eachitemer)
new_short_df = new_list[0]
new_long_df = new_list[1]

If I understand correctly, you want to take a series of strings that contain integers and remove all the characters that aren't integers. You don't need a for-loop for this. Instead, you can solve it with a simple regular expression.
b.replace('\D+', '', regex=True).astype(int)
Returns:
0 108
1 102
2 106
3 107
The regex replaces all characters that aren't numbers (denoted by \D) with an empty string, removing anything that's not a number. .astype(int) converts the series to the integer type. You can merge the result into your final dataframe as normal:
result = pd.DataFrame({
'consultant': a,
'invoice_number': b.replace('\D+', '', regex=True).astype(int)
})

apply lstrip on every element in dataframe of lists

In my dataframe each cell is a list with strings. The problem is that each string contains a whitespace infront of it
a={'names':[[' Peter',' Alex'],[' Josh',' Hans']]}
df=pd.DataFrame(a)
I want to remove the whitespaces.
For a single list i would use
y=[]
x = [' ab',' de',' cd']
for i in x:
d=i.strip()
y.append(d)
print (y)
['ab', 'de', 'cd']
so i tried to construct smth similar for a dataframe
stripped=[]
df=pd.DataFrame(a)
for index,row in df.iterrows():
d=df.names.apply(lambda x: x.lstrip())
stripped.append(d)
print(stripped)
which returns
'list' object has no attribute 'lstrip'
and if i call
for index,row in df.iterrows():
d=df.names.str.lstrip()
stripped.append(d)
print(stripped)
it returns Nan lists

this should work
df['names'] = df['names'].apply(lambda x: [i.strip() for i in x])
Output
names
0 [Peter, Alex]
1 [Josh, Hans]

Conversion of List of words (inside a dataframe) to a set of words

In my dataframe, I have a column with data as a list like [cell, protein, expression], I wanted to convert it as a set of words like cell, protein, expression, it should applies to entire column of the dataframe. Please suggest the possible way to do it.

try this
data['column_name'] = data['column_name'].apply(lambda x: ', '.join(x))

The issue is that df['Final_Text'] is not a list but rather a string. try using ast.literal_eval first:
import ast
from io import StringIO
# your sample df
s = """
,Final_Text
0,"['study', 'response', 'cell']"
1,"['cell', 'protein', 'effect']"
2,"['cell', 'patient', 'expression']"
3,"['patient', 'cell', 'study']"
4,"['study', 'cell', 'activity']"
"""
df = pd.read_csv(StringIO(s))
# convert you string of a list of to an actual list
df['Final_Text'] = df['Final_Text'].apply(ast.literal_eval)
# use a lambda expression with join to keep the text inside the list
df['Final_Text'] = df['Final_Text'].apply(lambda x: ', '.join(x))
Unnamed: 0 Final_Text
0 0 study, response, cell
1 1 cell, protein, effect
2 2 cell, patient, expression
3 3 patient, cell, study
4 4 study, cell, activity

Pythonic/efficient way to strip whitespace from every Pandas Data frame cell that has a stringlike object in it

I'm reading a CSV file into a DataFrame. I need to strip whitespace from all the stringlike cells, leaving the other cells unchanged in Python 2.7.
Here is what I'm doing:
def remove_whitespace( x ):
if isinstance( x, basestring ):
return x.strip()
else:
return x
my_data = my_data.applymap( remove_whitespace )
Is there a better or more idiomatic to Pandas way to do this?
Is there a more efficient way (perhaps by doing things column wise)?
I've tried searching for a definitive answer, but most questions on this topic seem to be how to strip whitespace from the column names themselves, or presume the cells are all strings.

Stumbled onto this question while looking for a quick and minimalistic snippet I could use. Had to assemble one myself from posts above. Maybe someone will find it useful:
data_frame_trimmed = data_frame.apply(lambda x: x.str.strip() if x.dtype == "object" else x)

You could use pandas' Series.str.strip() method to do this quickly for each string-like column:
>>> data = pd.DataFrame({'values': [' ABC ', ' DEF', ' GHI ']})
>>> data
values
0 ABC
1 DEF
2 GHI
>>> data['values'].str.strip()
0 ABC
1 DEF
2 GHI
Name: values, dtype: object

We want to:
Apply our function to each element in our dataframe - use applymap.
Use type(x)==str (versus x.dtype == 'object') because Pandas will label columns as object for columns of mixed datatypes (an object column may contain int and/or str).
Maintain the datatype of each element (we don't want to convert everything to a str and then strip whitespace).
Therefore, I've found the following to be the easiest:
df.applymap(lambda x: x.strip() if type(x)==str else x)

When you call pandas.read_csv, you can use a regular expression that matches zero or more spaces followed by a comma followed by zero or more spaces as the delimiter.
For example, here's "data.csv":
In [19]: !cat data.csv
1.5, aaa, bbb , ddd , 10 , XXX
2.5, eee, fff , ggg, 20 , YYY
(The first line ends with three spaces after XXX, while the second line ends at the last Y.)
The following uses pandas.read_csv() to read the files, with the regular expression ' *, *' as the delimiter. (Using a regular expression as the delimiter is only available in the "python" engine of read_csv().)
In [20]: import pandas as pd
In [21]: df = pd.read_csv('data.csv', header=None, delimiter=' *, *', engine='python')
In [22]: df
Out[22]:
0 1 2 3 4 5
0 1.5 aaa bbb ddd 10 XXX
1 2.5 eee fff ggg 20 YYY

The "data['values'].str.strip()" answer above did not work for me, but I found a simple work around. I am sure there is a better way to do this. The str.strip() function works on Series. Thus, I converted the dataframe column into a Series, stripped the whitespace, replaced the converted column back into the dataframe. Below is the example code.
import pandas as pd
data = pd.DataFrame({'values': [' ABC ', ' DEF', ' GHI ']})
print ('-----')
print (data)
data['values'].str.strip()
print ('-----')
print (data)
new = pd.Series([])
new = data['values'].str.strip()
data['values'] = new
print ('-----')
print (new)

Here is a column-wise solution with pandas apply:
import numpy as np
def strip_obj(col):
if col.dtypes == object:
return (col.astype(str)
.str.strip()
.replace({'nan': np.nan}))
return col
df = df.apply(strip_obj, axis=0)
This will convert values in object type columns to string. Should take caution with mixed-type columns. For example if your column is zip codes with 20001 and ' 21110 ' you will end up with '20001' and '21110'.

This worked for me - applies it to the whole dataframe:
def panda_strip(x):
r =[]
for y in x:
if isinstance(y, str):
y = y.strip()
r.append(y)
return pd.Series(r)
df = df.apply(lambda x: panda_strip(x))

I found the following code useful and something that would likely help others. This snippet will allow you to delete spaces in a column as well as in the entire DataFrame, depending on your use case.
import pandas as pd
def remove_whitespace(x):
try:
# remove spaces inside and outside of string
x = "".join(x.split())
except:
pass
return x
# Apply remove_whitespace to column only
df.orderId = df.orderId.apply(remove_whitespace)
print(df)
# Apply to remove_whitespace to entire Dataframe
df = df.applymap(remove_whitespace)
print(df)