all,
I have (what should be) a very simple Pandas question. I'm trying to "refresh" the indices of my DataFrame, df.
I'm trying:
df3 = df2.reindex()
df3.head()
But this still gives me:
Dose
13539.0 1.0
13539.0 2.0
13539.0 5.0
13539.0 3.0
13539.0 4.0
I need to keep the "Dose" column in the same order, but make its indices 0 -> len(Dose).
In other words, my desired output is:
Dose
0 1.0
1 2.0
2 5.0
3 3.0
4 4.0
Clearly I've pre-processed something to mess this up, but I'd appreciate any insight as to how I can ammend it :-)
Thank you!!
df3.reset_index(drop=True, inplace=True)
When you reset the index, the old index is added as a column, and a new sequential index is used. Then you can use the drop parameter to avoid the old index being added as a column.
Refer to this Pandas documentation.
https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.reset_index.html
You might want to lookup the documentation for reset_index. In your case:
df3.reset_index(drop=True)
This solution does the job...
df2 = df2.reset_index()
df2 = df2['Dose']
Related
I have dataframe like as below
Re_MC,Fi_MC,Fin_id,Res_id,
1,2,3,4
,7,6,11
11,,31,32
,,35,38
df1 = pd.read_clipboard(sep=',')
I would like to fillna based on two steps
a) First, compare only Re_MC and Fi_MC. If a value is missing in either of these columns, copy it from the other column.
b) Despite doing step a, if there is still NA for either Re_MC or Fi_MC, copy values from Fin_id for Fi_MC and Res_id for Re_MC.
So, I tried the below two approaches
Approach 1 - This works but not efficient/elegant
df1['Re_MC'] = df1['Re_MC'].fillna(df1['Fi_MC'])
df1['Fi_MC'] = df1['Fi_MC'].fillna(df1['Re_MC'])
df1['Re_MC'] = df1['Re_MC'].fillna(df1['Res_id'])
df1['Fi_MC'] = df1['Fi_MC'].fillna(df1['Fin_id'])
Approach 2 - This doesn't work and provide incorrect output
df1['Re_MC'] = df1['Re_MC'].fillna(df1['Fi_MC']).fillna(df1['Res_id'])
df1['Fi_MC'] = df1['Fi_MC'].fillna(df1['Re_MC']).fillna(df1['Fin_id'])
Is there any other efficient way to fillna in a sequential manner? Meaning, we do step a first and then based on result of step a, we do step b
I expect my output to be like as shown below
updated code
df_new = (df_new
.fillna({'Re MC': df_new['Re Cust'],'Re MC': df_new['Re Cust_System']})
.fillna({'Fi MC' : df_new['Fi.Fi Customer'],'Final MC':df_new['Re.Fi Customer']})
.fillna({'Fi MC' : df_new['Re MC']})
.fillna({'Class Fi MC':df_new['Re MC']})
)
You can use dictionaries in fillna:
(df1
.fillna({'Re_MC': df1['Fi_MC'], 'Fi_MC': df1['Re_MC']})
.fillna({'Re_MC': df1['Res_id'], 'Fi_MC': df1['Fin_id']})
)
output:
Re_MC Fi_MC Fin_id Res_id
0 1.0 2.0 3 4
1 7.0 7.0 6 11
2 11.0 11.0 31 32
3 38.0 35.0 35 38
I have two data frames df1 and df2. Both have first column common SKUCode=SKU
df1:
df2:
I want to update df1 and set SKUStatus=0 if SKUCode matches SKU in df2.
I want to add new row to df1 if SKU from df2 has no match to SKUCode.
So after the operation df1 looks like following:
One way I could get this done is via df2.iterrows() and looping through values however I think there must be another neat way of doing this?
Thank you
import pandas as pdx
df1=pdx.DataFrame({'SKUCode':['A','B','C','D'],'ListPrice':[1798,2997,1798,999],'SalePrice':[1798,2997,1798,999],'SKUStatus':[1,1,1,0],'CostPrice':[500,773,525,300]})
df2=pdx.DataFrame({'SKUCode':['X','Y','B'],'Status':[0,0,0],'e_date':['31-05-2020','01-06-2020','01-06-2020']})
df1.merge(df2,left_on='SKUCode')
try this, using outer merge which gives both matching and non-matching records.
In [75]: df_m = df1.merge(df2, on="SKUCode", how='outer')
In [76]: mask = df_m['Status'].isnull()
In [77]: df_m.loc[~mask, 'SKUStatus'] = df_m.loc[~mask, 'Status']
In [78]: df_m[['SKUCode', "ListPrice", "SalePrice", "SKUStatus", "CostPrice"]].fillna(0.0)
output
SKUCode ListPrice SalePrice SKUStatus CostPrice
0 A 1798.0 1798.0 1.0 500.0
1 B 2997.0 2997.0 0.0 773.0
2 C 1798.0 1798.0 1.0 525.0
3 D 999.0 999.0 0.0 300.0
4 X 0.0 0.0 0.0 0.0
5 Y 0.0 0.0 0.0 0.0
I'm not sure exactly if I understood you correctly but I think you can use .loc. something along the lines of:
df1.loc[df2['SKUStatu'] != 0, 'SKUStatus'] = 1
You should have a look at pd.merge function [https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.merge.html].
First rename a column with the same name (e.g rename SKU to SKUCode). Then try:
df1.merge(df2, left_on='SKUCode')
If you provide input data (not screenshots), I can try with the appropriate parameters.
I have a dataframe, sega_df:
Month 2016-11-01 2016-12-01
Character
Sonic 12.0 3.0
Shadow 5.0 23.0
I would like to create multiple new columns, by applying a formula for each already existing column within my dataframe (to put it shortly, pretty much double the number of columns). That formula is (100 - [5*eachcell])*0.2.
For example, for November for Sonic, (100-[5*12.0])*0.2 = 8.0, and December for Sonic, (100-[5*3.0])*0.2 = 17.0 My ideal output is:
Month 2016-11-01 2016-12-01 Weighted_2016-11-01 Weighted_2016-12-01
Character
Sonic 12.0 3.0 8.0 17.0
Shadow 5.0 23.0 15.0 -3.0
I know how to create a for loop to create one column. This is for if only one month was in consideration:
for w in range(1,len(sega_df.index)):
sega_df['Weighted'] = (100 - 5*sega_df)*0.2
sega_df[sega_df < 0] = 0
I haven't gotten the skills or experience yet to create multiple columns. I've looked for other questions that may answer what exactly I am doing but haven't gotten anything to work yet. Thanks in advance.
One vectorised approach is to drown to numpy:
A = sega_df.values
A = (100 - 5*A) * 0.2
res = pd.DataFrame(A, index=sega_df.index, columns=('Weighted_'+sega_df.columns))
Then join the result to your original dataframe:
sega_df = sega_df.join(res)
I have an aviation dataset that I am trying to clean. There are some missing values for the NumEngines feature, but there are some instances where a missing value can be derived from an entry elsewhere in the dataframe (this is not always the case). Below is a mini example of my dataset to illustrate both cases. Note that first Cessna entry can be used to fill in the second, but this is not the case for Piper.
df = pd.DataFrame()
df["Make"] = ["Cessna","Piper","Cessna","Boeing"]
df["Model"] = ["Citation","PA32RT","Citation","737-300"]
df["NumEngines"] = [2,None,None,2]
How can I make it so that the resulting DataFrame would be
Make Model NumEngines
0 Cessna Citation 2.0
1 Piper PA32RT NaN
2 Cessna Citation 2.0
3 Boeing 737-300 2.0
I would bet transform('first') could make it again here:
df.groupby(['Make', 'Model']).transform('first')
Out[179]:
NumEngines
0 2.0
1 NaN
2 2.0
3 2.0
How do I multiply each element of a given column of my dataframe with a scalar?
(I have tried looking on SO, but cannot seem to find the right solution)
Doing something like:
df['quantity'] *= -1 # trying to multiply each row's quantity column with -1
gives me a warning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
Note: If possible, I do not want to be iterating over the dataframe and do something like this...as I think any standard math operation on an entire column should be possible w/o having to write a loop:
for idx, row in df.iterrows():
df.loc[idx, 'quantity'] *= -1
EDIT:
I am running 0.16.2 of Pandas
full trace:
SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead
See the the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
self.obj[item] = s
try using apply function.
df['quantity'] = df['quantity'].apply(lambda x: x*-1)
Note: for those using pandas 0.20.3 and above, and are looking for an answer, all these options will work:
df = pd.DataFrame(np.ones((5,6)),columns=['one','two','three',
'four','five','six'])
df.one *=5
df.two = df.two*5
df.three = df.three.multiply(5)
df['four'] = df['four']*5
df.loc[:, 'five'] *=5
df.iloc[:, 5] = df.iloc[:, 5]*5
which results in
one two three four five six
0 5.0 5.0 5.0 5.0 5.0 5.0
1 5.0 5.0 5.0 5.0 5.0 5.0
2 5.0 5.0 5.0 5.0 5.0 5.0
3 5.0 5.0 5.0 5.0 5.0 5.0
4 5.0 5.0 5.0 5.0 5.0 5.0
Here's the answer after a bit of research:
df.loc[:,'quantity'] *= -1 #seems to prevent SettingWithCopyWarning
More recent pandas versions have the pd.DataFrame.multiply function.
df['quantity'] = df['quantity'].multiply(-1)
The real problem of why you are getting the error is not that there is anything wrong with your code: you can use either iloc, loc, or apply, or *=, another of them could have worked.
The real problem that you have is due to how you created the df DataFrame. Most likely you created your df as a slice of another DataFrame without using .copy(). The correct way to create your df as a slice of another DataFrame is df = original_df.loc[some slicing].copy().
The problem is already stated in the error message you got " SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead"
You will get the same message in the most current version of pandas too.
Whenever you receive this kind of error message, you should always check how you created your DataFrame. Chances are you forgot the .copy()
Try df['quantity'] = df['quantity'] * -1.
A bit old, but I was still getting the same SettingWithCopyWarning. Here was my solution:
df.loc[:, 'quantity'] = df['quantity'] * -1
I got this warning using Pandas 0.22. You can avoid this by being very explicit using the assign method:
df = df.assign(quantity = df.quantity.mul(-1))
A little late to the game, but for future searchers, this also should work:
df.quantity = df.quantity * -1
You can use the index of the column you want to apply the multiplication for
df.loc[:,6] *= -1
This will multiply the column with index 6 with -1.
Also it's possible to use numerical indeces with .iloc.
df.iloc[:,0] *= -1
Update 2022-08-10
Python: 3.10.5 - pandas: 1.4.3
As Mentioned in Previous comments, one the applicable approaches is using lambda. But, Be Careful with data types when using lambda approach.
Suppose you have a pandas Data Frame like this:
# Create List of lists
products = [[1010, 'Nokia', '200', 1800], [2020, 'Apple', '150', 3000], [3030, 'Samsung', '180', 2000]]
# Create the pandas DataFrame
df = pd.DataFrame(products, columns=['ProductId', 'ProductName', 'Quantity', 'Price'])
# print DataFrame
print(df)
ProductId ProductName Quantity Price
0 1010 Nokia 200 1800
1 2020 Apple 150 3000
2 3030 Samsung 180 2000
So, if you want to triple the value of Quantity for all rows in Products and use the following Statement:
# This statement considers the values of Quantity as string and updates the DataFrame
df['Quantity'] = df['Quantity'].apply(lambda x:x*3)
# print DataFrame
print(df)
The Result will be:
ProductId ProductName Quantity Price
0 1010 Nokia 200200200 1800
1 2020 Apple 150150150 3000
2 3030 Samsung 180180180 2000
The above statement considers the values of Quantity as string.
So, in order to do the multiplication in the right way, the following statement with a convert could generate correct output:
# This statement considers the values of Quantity as integer and updates the DataFrame
df['Quantity'] = df['Quantity'].apply(lambda x:int(x)*3)
# print DataFrame
print(df)
Therefore the output will be like this:
ProductId ProductName Quantity Price
0 1010 Nokia 600 1800
1 2020 Apple 450 3000
2 3030 Samsung 540 2000
I Hope this could help :)