We Keep Coding

Pandas CSV : Check for each row if a column is empty

I want to test for each row of a CSV file if some column are empty or not and change value of another column depending on that.
Here is what I have :
df = df.replace(r'^\s*$', np.NaN, regex=True)
df['Multi-line'] = pd.Series(dtype=object)
for i, row in df.iterrows():
if (row['Directory Number 1'] != np.NaN and row['Directory Number 2'] != np.NaN and row['Directory Number 3'] != np.NaN and row['Directory Number 4'] != np.NaN):
df.at[i,'Multi-line'] = 'Yes'
If 2 "Directory Number X" or more are not empty, I want the "Multi-line" column to be "Yes" and if 1 or 0 "Directory Number X" are not empty then "Multi-line" should be "No".
Here is only one if just to show you how it looks but in my test sample, all Multi-line are set to "Yes", it seems like the problem is inside the If condition with the row value and the np.nan but I don't know how to check if a row value is empty or not..
Thanks for you help !

I assume that you executed df = df.replace(r'^\s*$', np.NaN, regex=True)
before.
Then, to generate the new column, run:
df['Multi-line'] = df.apply(lambda row: 'Yes' if row.notna().sum() >= 2 else 'No', axis=1)
No need for explicit call to iterrows, as apply arranges just such
a loop, invoking the passed function for each row.
If your DataFrame has also other columns, especially when they can
have NaN values, then application of this lambda function should be
limited to just these 4 columns of interest.
In this case run:
cols = [ f'Directory Number {i}' for i in range(1, 5) ]
df['Multi-line'] = df[cols].apply(lambda row:
'Yes' if row.notna().sum() >= 2 else 'No', axis=1)
Note also that a check like if (row[s] != np.NaN): as proposed
in the other solution is a bad approach, since NaN by definition
is not equal to another NaN, so you can't just compare two NaNs.
To check it try:
s = np.nan
s2 = np.nan
s != s2 # True
s == s2 # False
Then save any "true" string in s, running s = 'xx' and repeat:
s != s2 # True
s == s2 # False
with just the same result.

You can use a counter instead
df = df.replace(r'^\s*$', np.NaN, regex=True)
df['Multi-line'] = pd.Series(dtype=object)
cnt=0;
str = ['Directory Number 1','Directory Number 2','Directory Number 3','Directory Number 4'];
for i, row in df.iterrows():
for s in str:
if (row[s] != np.NaN):
cnt+=1;
if (cnt>2):
df.at[i,'Multi-line'] = 'Yes'
else:
df.at[i,'Multi-line'] = 'No'
cnt=0;

Pandas append series from tuple into empty dataframe

I'm trying to split a dataframe with a certain logic.
Here's my attempt:
def split_df(df: pd.DataFrame):
train = pd.DataFrame(columns = df.columns)
valid = pd.DataFrame(columns = df.columns)
i = 0
for data in tqdm(df.iterrows()):
if i > 10:
break
if (len(valid[valid['category_id'] == data[1]['category_id']]) > 0):
tmp = pd.DataFrame(columns = df.columns, data = pd.Series(data[1]))
train.append(tmp,ignore_index=True)
i = i+1
else:
tmp = pd.DataFrame(columns = df.columns, data = pd.Series(data[1]))
train.append(tmp,ignore_index=True)
valid.append(tmp,ignore_index=True)
i = i+1
return (train, valid)
When I run this, I get a tuple of empty dataframes.
the i<10 is just for me to check outputs.
The splitting logic may be wrong, but it's not important for now.
I also try to avoid for loops, so if there's a better approach to this problem, I'll be glad to read about it.

Append does not modify the dataframe in place, so you need to reassign your variable to keep changes:
train = train.append(tmp,ignore_index=True)
valid = valid.append(tmp,ignore_index=True)

Search for a value anywhere in a pandas DataFrame

This seems like a simple question, but I couldn't find it asked before (this and this are close but the answers aren't great).
The question is: if I want to search for a value somewhere in my df (I don't know which column it's in) and return all rows with a match.
What's the most Pandaic way to do it? Is there anything better than:
for col in list(df):
try:
df[col] == var
return df[df[col] == var]
except TypeError:
continue
?

You can perform equality comparison on the entire DataFrame:
df[df.eq(var1).any(1)]

You should using isin , this is return the column , is want row check cold' answer :-)
df.isin(['bal1']).any()
A False
B True
C False
CLASS False
dtype: bool
Or
df[df.isin(['bal1'])].stack() # level 0 index is row index , level 1 index is columns which contain that value
0 B bal1
1 B bal1
dtype: object

You can try the code below:
import pandas as pd
x = pd.read_csv(r"filePath")
x.columns = x.columns.str.lower().str.replace(' ', '_')
y = x.columns.values
z = y.tolist()
print("Note: It take Case Sensitive Values.")
keyWord = input("Type a Keyword to Search: ")
try:
for k in range(len(z)-1):
l = x[x[z[k]].str.match(keyWord)]
print(l.head(10))
k = k+1
except:
print("")

This is a solution which will return the actual column you need.
df.columns[df.isin(['Yes']).any()]

Minimal solution:
import pandas as pd
import numpy as np
def locate_in_df(df, value):
a = df.to_numpy()
row = np.where(a == value)[0][0]
col = np.where(a == value)[1][0]
return row, col

How to print the column name on Pandas DataFrame row?

I have this DataFrame:
df = pd.DataFrame({'day':['1/1/2017','1/2/2017','1/3/2017','1/4/2017','1/5/2017','1/6/2017','1/7/2017'],
'event':['Rain','Sunny','Snow','Snow','Rain','Sunny','Sunny'],
'temperature': [32, 35, 28,24,32,31,''],'windspeed':[6,7,2,7,4,2,'']})
df
I am trying to find the headers for the missing values on index 6:
for x in df.loc[6]:
if x == '':
print(df.columns.values)
else: print(x)
I have tried searching and the closest I could get was what I have now. Ultimately I'm trying insert these values into the dataframe: temperature =
34, windspeed = 8.
But my first step was simply trying to build the loop/if statement that says if x=='' & [COLUMN_NAME] == 'temperature'... and that is where I got stuck. I'm new to python, just trying to learn Pandas. I need to only return the column I'm on, and not a list of all the columns.

There are better ways to do this, but this works.
for col, val in df.loc[6].iteritems():
if not val: # this is the same as saying "if val == '':"
print(col)
else:
print(val)

Modified from your code:
for i,x in enumerate(df.loc[6]):
if x == '':
print(df.columns[i])
else: print(x)

I would use list comprehension as follows:
listOfNulls = [ind for ind in df.loc[6].index if df.loc[6][ind] == '']
and when I print the listOfNulls, I get:
>>>> print(listOfNulls)
Out: ['temperature', 'windspeed']
The key here is it understand that df.loc[6] is a pandas Series which has indices. We are using the values of the Series to get the indices.

Efficiently update columns based on one of the columns split value

So here is my code updating many column values based on a condition of split values of the column 'location'. The code works fine, but as its iterating by row it's not efficient enough. Can anyone help me to make this code work faster please?
for index, row in df.iterrows():
print index
location_split =row['location'].split(':')
after_county=False
after_province=False
for l in location_split:
if l.strip().endswith('ED'):
df[index, 'electoral_district'] = l
elif l.strip().startswith('County'):
df[index, 'county'] = l
after_county = True
elif after_province ==True:
if l.strip()!='Ireland':
df[index, 'dublin_postal_district'] = l
elif after_county==True:
df[index, 'province'] = l.strip()
after_province = True

'map' was what I needed :)
def fill_county(column):
res = ''
location_split = column.split(':')
for l in location_split:
if l.strip().startswith('County'):
res= l.strip()
break
return res
df['county'] = map(fill_county, df['location'])