This question already has answers here:
Pandas still getting SettingWithCopyWarning even after using .loc
(3 answers)
Closed 6 years ago.
df_masked.loc[:, col] = df_masked.groupby([df_masked.index.month, df_masked.index.day])[col].\
transform(lambda y: y.fillna(y.median()))
Even after using a .loc, I get the foll. error, how do I fix it?
Anaconda\lib\site-packages\pandas\core\indexing.py:476: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
self.obj[item] = s
You could get this UserWarning if df_masked is a sub-DataFrame of some other DataFrame.
In particular, if data had been copied from the original DataFrame to df_masked then, Pandas emits the UserWarning to alert you that modifying df_masked will not affect the original DataFrame.
If you do not intend to modify the original DataFrame, then you are free to ignore the UserWarning.
There are ways to shut off the UserWarning on a per-statement basis. In particular, you could use df_masked.is_copy = False.
If you run into this UserWarning a lot, then instead of silencing the UserWarnings one-by-one, I think it is better to leave them be as you are developing your code. Be aware of what the UserWarning means, and if the modifying-the-child-does-not-affect-the-parent issue does not affect you, then ignore it. When your code is ready for production, or if you are experienced enough to not need the warnings, shut them off entirely with
pd.options.mode.chained_assignment = None
near the top of your code.
Here is a simple example which demonstrate the problem and (a) solution:
import pandas as pd
df = pd.DataFrame({'swallow':['African','European'], 'cheese':['gouda', 'cheddar']})
df_masked = df.iloc[1:]
df_masked.is_copy = False # comment-out this line to see the UserWarning
df_masked.loc[:, 'swallow'] = 'forest'
The reason why the UserWarning exists is to help alert new users to the fact that
chained-indexing such as
df.iloc[1:].loc[:, 'swallow'] = 'forest'
will not affect df when the result of the first indexer (e.g. df.iloc[1:])
returns a copy.
Related
This question already has answers here:
How to deal with SettingWithCopyWarning in Pandas
(20 answers)
Closed 3 years ago.
I tried renaming an index value with the code:
df_1.rename({'*****': "Favour Edwards"}, axis = 0, inplace = True)
and I got this message:
/home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/pandas/core/frame.py:4238: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame
See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
return super().rename(**kwargs)
Although my code still ran without any observable errors, i still checked out the link but unfortunately i'm still pretty new with coding and the vocabulary/ semantics of the documentation seemed sort of complex for me to really understand. Anyone who can break down the meaning in simpler terms?
Its a WARNING not ERROR (big distinction) that your operation may not have worked as expected and that you should check the results.
This question already has answers here:
How to deal with SettingWithCopyWarning in Pandas
(20 answers)
Closed 4 years ago.
I have a small dataframe, say this one :
Mass32 Mass44
12 0.576703 0.496159
13 0.576658 0.495832
14 0.576703 0.495398
15 0.576587 0.494786
16 0.576616 0.494473
...
I would like to have a rolling mean of column Mass32, so I do this:
x['Mass32s'] = pandas.rolling_mean(x.Mass32, 5).shift(-2)
It works as in I have a new column named Mass32s which contains what I expect it to contain but I also get the warning message:
A value is trying to be set on a copy of a slice from a DataFrame. Try
using .loc[row_indexer,col_indexer] = value instead
See the the caveats in the documentation:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
I'm wondering if there's a better way to do it, notably to avoid getting this warning message.
This warning comes because your dataframe x is a copy of a slice. This is not easy to know why, but it has something to do with how you have come to the current state of it.
You can either create a proper dataframe out of x by doing
x = x.copy()
This will remove the warning, but it is not the proper way
You should be using the DataFrame.loc method, as the warning suggests, like this:
x.loc[:,'Mass32s'] = pandas.rolling_mean(x.Mass32, 5).shift(-2)
This question already has an answer here:
df.loc causes a SettingWithCopyWarning warning message
(1 answer)
Closed 6 years ago.
In pandas data frame, I'm trying to map df['old_column'], apply user defined function f for each row and create a new column.
df['new_column'] = df['old_column'].map(lambda x: f(x))
This will give out "SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame." error.
I tried the following:
df.loc[:, 'new_column'] = df['old_column'].map(lambda x: f(x))
which doesn't help. What can I do?
A SettingWithCopy warning is raised for certain operations in pandas which may not have the expected result because they may be acting on copies rather than the original datasets. Unfortunately there is no easy way for pandas itself to tell whether or not a particular call will or won't do this, so this warning tends to be raised in many, many cases where (from my perspective as a user) nothing is actually amiss.
Both of your method calls are fine. If you want to get rid of the warning entirely, you can specify:
pd.options.mode.chained_assignment = None
See this StackOverflow Q&A for more information on this.
This question already has answers here:
How to deal with SettingWithCopyWarning in Pandas
(20 answers)
Closed 4 years ago.
I have a small dataframe, say this one :
Mass32 Mass44
12 0.576703 0.496159
13 0.576658 0.495832
14 0.576703 0.495398
15 0.576587 0.494786
16 0.576616 0.494473
...
I would like to have a rolling mean of column Mass32, so I do this:
x['Mass32s'] = pandas.rolling_mean(x.Mass32, 5).shift(-2)
It works as in I have a new column named Mass32s which contains what I expect it to contain but I also get the warning message:
A value is trying to be set on a copy of a slice from a DataFrame. Try
using .loc[row_indexer,col_indexer] = value instead
See the the caveats in the documentation:
http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
I'm wondering if there's a better way to do it, notably to avoid getting this warning message.
This warning comes because your dataframe x is a copy of a slice. This is not easy to know why, but it has something to do with how you have come to the current state of it.
You can either create a proper dataframe out of x by doing
x = x.copy()
This will remove the warning, but it is not the proper way
You should be using the DataFrame.loc method, as the warning suggests, like this:
x.loc[:,'Mass32s'] = pandas.rolling_mean(x.Mass32, 5).shift(-2)
This question already has answers here:
How to deal with SettingWithCopyWarning in Pandas
(20 answers)
Closed 3 years ago.
I have been reading this link on "Returning a view versus a copy". I do not really get how the chained assignment concept in Pandas works and how the usage of .ix(), .iloc(), or .loc() affects it.
I get the SettingWithCopyWarning warnings for the following lines of code, where data is a Panda dataframe and amount is a column (Series) name in that dataframe:
data['amount'] = data['amount'].astype(float)
data["amount"].fillna(data.groupby("num")["amount"].transform("mean"), inplace=True)
data["amount"].fillna(mean_avg, inplace=True)
Looking at this code, is it obvious that I am doing something suboptimal? If so, can you let me know the replacement code lines?
I am aware of the below warning and like to think that the warnings in my case are false positives:
The chained assignment warnings / exceptions are aiming to inform the
user of a possibly invalid assignment. There may be false positives;
situations where a chained assignment is inadvertantly reported.
EDIT : the code leading to the first copy warning error.
data['amount'] = data.apply(lambda row: function1(row,date,qty), axis=1)
data['amount'] = data['amount'].astype(float)
def function1(row,date,qty):
try:
if(row['currency'] == 'A'):
result = row[qty]
else:
rate = lookup[lookup['Date']==row[date]][row['currency'] ]
result = float(rate) * float(row[qty])
return result
except ValueError: # generic exception clause
print "The current row causes an exception:"
The point of the SettingWithCopy is to warn the user that you may be doing something that will not update the original data frame as one might expect.
Here, data is a dataframe, possibly of a single dtype (or not). You are then taking a reference to this data['amount'] which is a Series, and updating it. This probably works in your case because you are returning the same dtype of data as existed.
However it could create a copy which updates a copy of data['amount'] which you would not see; Then you would be wondering why it is not updating.
Pandas returns a copy of an object in almost all method calls. The inplace operations are a convience operation which work, but in general are not clear that data is being modified and could potentially work on copies.
Much more clear to do this:
data['amount'] = data["amount"].fillna(data.groupby("num")["amount"].transform("mean"))
data["amount"] = data['amount'].fillna(mean_avg)
One further plus to working on copies. You can chain operations, this is not possible with inplace ones.
e.g.
data['amount'] = data['amount'].fillna(mean_avg)*2
And just an FYI. inplace operations are neither faster nor more memory efficient. my2c they should be banned. But too late on that API.
You can of course turn this off:
pd.set_option('chained_assignment',None)
Pandas runs with the entire test suite with this set to raise (so we know if chaining is happening) on, FYI.