We Keep Coding

Pandas Dataframe: Function doesn't preserve my custom column order when returning df

I followed this code from user Lala la (https://stackoverflow.com/a/55803252/19896454)
to put 3 columns at the front and leave the rest with no changes. It works well inside the function but when returns the dataframe, it loses column order.
My desperate solution was to put the code on the main program...
Other functions in my code are able to return modified versions of the dataframe with no problem.
Any ideas what is happening?
Thanks!
def define_columns_order(df):
cols_to_order = ['LINE_ID','PARENT.CATEGORY', 'CATEGORY']
new_columns = cols_to_order + (df.columns.drop(cols_to_order).tolist())
df = df[new_columns]
return df

try using return(df.reindex(new_columns, axis=1)) and keep in mind DataFrame modifications are not in place, unless you specify inplace=True, therefore you need to explicitly assign the result returned by your function to your df variable

How to undo the changes made in original DataFrame?

I am working with a dataset. As a precautionary measure, I created a back-up copy using the following command.
Orig. Dataframe = df
df_copy = df.copy(deep = True)
Now, I dropped few columns from original dataframe (df) by mistake using inplace = True.
I tried to undo the operation, but no use.
So, the question is how to get my original dataframe (df) from copied dataframe (df_copy) ?

Yoy cannot restore it. Code like below dosen't work.
df = df_copy.copy(deep = True)
Every variables which reference original df keep reference after operation above.

How to filter multiple dataframes in a loop?

I have a lot of dataframes and I would like to apply the same filter to all of them without having to copy paste the filter condition every time.
This is my code so far:
df_list_2019 = [df_spain_2019,df_amsterdam_2019, df_venice_2019, df_sicily_2019]
for data in df_list_2019:
data = data[['host_since','host_response_time','host_response_rate',
'host_acceptance_rate','host_is_superhost','host_total_listings_count',
'host_has_profile_pic','host_identity_verified',
'neighbourhood','neighbourhood_cleansed','zipcode','latitude','longitude','property_type','room_type',
'accommodates','bathrooms','bedrooms','beds','amenities','price','weekly_price',
'monthly_price','cleaning_fee','guests_included','extra_people','minimum_nights','maximum_nights',
'minimum_nights_avg_ntm','has_availability','availability_30','availability_60','availability_90',
'availability_365','number_of_reviews','number_of_reviews_ltm','review_scores_rating',
'review_scores_checkin','review_scores_communication','review_scores_location', 'review_scores_value',
'instant_bookable','is_business_travel_ready','cancellation_policy','reviews_per_month'
]]
but it doesn't apply the filter to the data frame. How can I change the code to do that?
Thank you

The filter (column selection) is actually applied to every DataFrame, you just throw the result away by overriding what the name data points to.
You need to store the results somewhere, a list for example.
cols = ['host_since','host_response_time', ...]
filtered = [df[cols] for df in df_list_2019]

As soon as you write var = new_value, you do not change the original object but have the variable refering a new object.
If you want to change the dataframes from df_list_2019, you have to use an inplace=True method. Here, you could use drop:
keep = set(['host_since','host_response_time','host_response_rate',
'host_acceptance_rate','host_is_superhost','host_total_listings_count',
'host_has_profile_pic','host_identity_verified',
'neighbourhood','neighbourhood_cleansed','zipcode','latitude','longitude','property_type','room_type',
'accommodates','bathrooms','bedrooms','beds','amenities','price','weekly_price',
'monthly_price','cleaning_fee','guests_included','extra_people','minimum_nights','maximum_nights',
'minimum_nights_avg_ntm','has_availability','availability_30','availability_60','availability_90',
'availability_365','number_of_reviews','number_of_reviews_ltm','review_scores_rating',
'review_scores_checkin','review_scores_communication','review_scores_location', 'review_scores_value',
'instant_bookable','is_business_travel_ready','cancellation_policy','reviews_per_month'
])
for data in df_list_2019:
data.drop(columns=[col for col in data.columns if col not in keep], inplace=True)
But beware, pandas experts recommend to prefere the df = df. ... idiom to the df...(..., inplace=True) because it allows chaining the operations. So you should ask yourself if #timgeb's answer cannot be used. Anyway this one should work for your requirements.

Drop Columns in Pandas Dataframe: Inconsistency in Output

Problem: While dropping column labelled 'Happiness_Score' below, I'm getting it dropped in the parent Dataframe as well. This is not supposed to happen, would like clarification on this?
A = df_new
A.drop('Happiness_Score', axis = 1, inplace = True)
This is the output: As you can see the column gets dropped in df_new too; isn't inplace = True mean that it gets dropped only in the A Dataframe.
NOTE:
I'm able to workaround this by changing the code; now output is as expected.
B=df_new.drop('Happiness_Score', axis = 1)

Actually, when you do A = df_new
you are not creating a copy of the Dataframe, rather just a pointer. So to execute this correctly you should use A = df_new.copy()
When you are selecting a subset or indexing: A = df_new[condition] then it creates copy of a slice of a dataframe, so your workaround works too.

A = def_new creates a new reference to your original def_new, an not a new copy. You are binding A to the same thing def_new holding the reference to. And what happens when you do modification in a reference? It is reflected in the original object. I'll illustrate this with an example.
orgList = [1,2,3,4,5]
bkpList = orgList
print(bkpList is orgList) #OUTPUT: True
This is because both variables are pointing to same list. Modify any one, and change will be reflected in original list. Same thing can be observed in your dataframe case.
Solution: Keep a separate copy of your dataframe.

The variable A is a reference to df_new. Try creating A by doing a complete slice of df_new or df_new.copy().

Python Pandas SettingWithCopyWarning copies vs new objects

I'm working with a dataframe 'copy' created by sub-setting a previous one - see below:
import random
import pandas as pd
df = pd.DataFrame({'data':list(random.sample(range(10,100),25))})
df_filtered = df.query('data > 20 and data < 80')
df_filtered.rename(columns={'data':'observations'},inplace=True)
The problem is, when the rename method is called I receive a SettingWithCopy warning that, as I understand it, means I'm operating on a copy of the original (df in this case) object. The warning text is: "A value is trying to be set on a copy of a slice from a DataFrame"
I found this question that was answered using a different approach to subsetting. I prefer the Dataframe.query() method myself (syntax-wise). Is there a way I can create a new Dataframe object using the.query() method rather than the method suggested in the question I linked? I've tried a few options with iloc but haven't been successful thus-far.

You can always explicitly make a copy by calling .copy() on your filtered dataframe. Concretely, replace
df_filtered = df.query('data > 20 and data < 80')
with
df_filtered = df.query('data > 20 and data < 80').copy()
Does that get rid of the warning?

try this instead of using inplace=True:
In [12]: df_filtered = df.query('data > 20 and data < 80')
In [13]: df_filtered = df_filtered.rename(columns={'data':'observations'})
.rename() function returns a new object, so you can simply overwrite your DF with the returned new DF
if you use inplace the following is happening
from docs:
inplace : boolean, default False
Whether to return a new DataFrame. If True then value of copy is ignored.
Returns:
renamed : DataFrame (new object)
PS basically you should try to avoid using inplace=True and use df = df.function(...) technique instead