We Keep Coding

Discretize all the columns in a dataframe pyton

I have a dataframe where all the columns are continous variables, and I want to discretize them in binnings based on frequency (so the binnings have the same size).
In order to do this I just apply the pd.cut function and iterate through the columns, however I'm getting the following errors:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df_q[column] = pd.qcut(df_q[column], 3)
<ipython-input-46-87e2efb9d039>:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
df_q[column] = pd.qcut(df_q[column], 3)
<ipython-input-46-87e2efb9d039>:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
You can find a RepEx here:
import pandas as pd
from sklearn import datasets
import matplotlib.pyplot as plt
# Load data
data = datasets.load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
# Remove categorical variable and bin
df_q = df.loc[:, df.columns != "target"]
for column in df_q:
df_q[column] = pd.qcut(df_q[column], 3)

I do not get any error or warning message when running your code. Do try to make a copy of df before creating df_q.
df2 = df.copy()
df_q = df2.loc[:, df2.columns != "target"]

difference between df.loc[:, columns] and df.loc[:][columns]

I want to normalize some columns of a pandas data frame using MinMaxScaler in this way:
scaler = MinMaxScaler()
numericals = ["TX_TIME_SECONDS",'TX_Amount']
while I do in this way:
df.loc[:][numericals] = scaler.fit_transform(df.loc[:][numericals])
it's not done inplace and df is not changed;
whereas, when I do in this way:
df.loc[:, numericals] = scaler.fit_transform(df.loc[:][numericals])
the numerical columns of df are changed in place,
So, What's the difference between df.loc[:, ~] and df.loc[:][~]

df.loc[:][numericals] selects all rows and then selects columns "TX_TIME_SECONDS" and 'TX_Amount' of the returning object, and assigns some value to it. The problem is, the returning object might be a copy so this may not change the actual DataFrame.
The correct way of making this assignment is using df.loc[:, numericals], because with .loc you are guaranteed to modify the original DataFrame.

I suggest you read some documentation because this is pretty basic.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html
https://www.geeksforgeeks.org/python-pandas-dataframe-loc/

Getting rid of Error from Fillna in pandas

I tried filling the NA values of a column in a dataframe with:
df1 = data.copy()
df1.columns = data.columns.str.lower()
df2 = df1[['passangerid', 'trip_cost','class']]
df2['class'] = df2['class'].fillna(0)
df2
Although getting this error:
:5: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation:
df2['class'] = df2['class'].fillna(0, axis = 0)
Can someone please help?

First of all I'd advise you to follow the warning message and read up on the caveats in the provided link.
You're getting this warning (not an error) because your df2 is a slice of your df1, not a separate DataFrame.
To avoid getting this warning you can use .copy() method as:
df2 = df1[['passangerid', 'trip_cost','class']].copy()

Why do I get "a value is trying to be set on a copy of a slice" when creating a new column with apply?

I have something like this,
df1 = ...
df1['NEW_COLUMN'] = df1['SOME_COLUMN'].apply(lambda x: ...)
Although this works and I get the column 'NEW_COLUMN' added to the dataframe, I get this following annying warning. Why? And what is the solution?
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

If you simply want to avoid getting warned, you can set it in pandas options. If you understand why the warning is, and why is it happening then you can simply ignore it by adding this after importing pandas:
pd.options.mode.chained_assignment = None

Add copy() to avoid getting this warning
df = pd.DataFrame({"Value" : [0.12,0.22,0.32,0.11,0.54,0.55,0.98]})
df['Category'] = df.Value.apply(lambda x: 'Neg' if x < 0.5 else 'Pos').copy()

New column using apply function on other columns in dataframe

I have a dataframe where three of the columns are coordinates of data ('H_x', 'H_y' and 'H_z'). I want to calculate radius-vector of the data and add it as a new column in my dataframe. But I have some kind of problem with pandas apply function.
My code is:
def radvec(x, y, z):
rv=np.sqrt(x**2+y**2+z**2)
return rv
halo_field['rh_field']=halo_field.apply(lambda row: radvec(row['H_x'], row['H_y'], row['H_z']), axis=1)
The error I'm getting is:
group_sh.py:78: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-
docs/stable/indexing.html#indexing-view-versus-copy
halo_field['rh_field']=halo_field.apply(lambda row: radvec(row['H_x'], row['H_y'], row['H_z']), axis=1)
I get column that I want, but I'm still confused with this error message.
I'm aware there are similar questions here, but I couldn't find how to solve my problem. I'm fairly new to python. Can you help?
Edit: halo_field is a slice of another dataframe:
halo_field = halo_res[halo_res.N_subs==1]

The problem is you're working with a slice, which can be ambiguous:
halo_field = halo_res[halo_res.N_subs==1]
You have two options:
Work on a copy
You can explicitly copy your dataframe to avoid the warning and ensure your original dataframe is unaffected:
halo_field = halo_res[halo_res.N_subs==1].copy()
halo_field['rh_field'] = halo_field.apply(...)
Work on the original dataframe conditionally
Use pd.DataFrame.loc with a Boolean mask to update your original dataframe:
mask = halo_res['N_subs'] == 1
halo_res.loc[mask, 'rh_field'] = halo_res.loc[mask, 'rh_field'].apply(...)
Don't use apply
As a side note, in either scenario you can avoid apply for your function. For example:
halo_field['rh_field'] = (halo_field[['H_x', 'H_y', 'H_z']]**2).sum(1)**0.5