I have a pandas data frame that looks like this:
1: As you can see, I have the index "State" and "City"
And I want to filter by state using loc, for example using:
nuevo4.loc["Bulgaria"]
(The name of the Dataframe is "nuevo4"), but instead of getting the results, I want I get the error:
KeyError: 'Bulgaria'
I read the loc documentation online and I cannot see the fail here, I'm sorry if this is too obvious, the names are well spelled, and that...
You command should work see the below example. You might have some whitespace issues with your data:
import pandas as pd
import numpy as np
df = pd.DataFrame(np.arange(25).reshape(-1,5), index = pd.MultiIndex.from_tuples([('A',1), ('A',2),('B', 0), ('C', 1), ('C', 2)]))
print(df)
print(df.loc['A'])
Output:
0 1 2 3 4
A 1 0 1 2 3 4
2 5 6 7 8 9
B 0 10 11 12 13 14
C 1 15 16 17 18 19
2 20 21 22 23 24
Using loc:
0 1 2 3 4
1 0 1 2 3 4
2 5 6 7 8 9
Related
I have a pandas dataframe and I would like to add a column and set certain values based on certain conditions.
Initialization
See my following example:
import numpy as np
import pandas as pd
df=pd.DataFrame(np.reshape(np.arange(20),(5,4)).tolist(),columns=list('ABCD'))
df['E']=''
df.loc[0,'E']=1
df
Until here, everything is as wished. Output:
A B C D E
0 0 1 2 3 1
1 4 5 6 7
2 8 9 10 11
3 12 13 14 15
4 16 17 18 19
Problem
But now, I would like to set the value of the 'E' column in the first row for which the 'B' value is larger than 8 to 3 (the same as df.loc[2,'E']=1 but using that condition. I don't know which line it is a priori).
Solution attempts
I have tried different things:
df.loc[df['B']>8,'E'].iloc[0]=3
df
This doesn't change any value:
A B C D E
0 0 1 2 3 1
1 4 5 6 7
2 8 9 10 11
3 12 13 14 15
4 16 17 18 19
The same holds for
df.loc[df['B']>8]['E'].iloc[0]=3
Other things I've tried result in an error. df.loc[df['B']>8,'E']=3works well but I only want to assign the value in the first line meeting the condition, not in all lines.
So how do I achieve this?
Use .idxmax(). Since True is the max of True/False it will return the index of the first True value for a boolean index.
df.loc[(df['B']>8).idxmax(),'E']=3
I'm having problems with pd.rolling() method that returns several outputs even though the function returns a single value.
My objective is to:
Calculate the absolute percentage difference between two DataFrames with 3 columns in each df.
Sum all values
I can do this using pd.iterrows(). But working with larger datasets makes this method ineffective.
This is the test data im working with:
#import libraries
import pandas as pd
import numpy as np
#create two dataframes
values = {'column1': [7,2,3,1,3,2,5,3,2,4,6,8,1,3,7,3,7,2,6,3,8],
'column2': [1,5,2,4,1,5,5,3,1,5,3,5,8,1,6,4,2,3,9,1,4],
"column3" : [3,6,3,9,7,1,2,3,7,5,4,1,4,2,9,6,5,1,4,1,3]
}
df1 = pd.DataFrame(values)
df2 = pd.DataFrame([[2,3,4],[3,4,1],[3,6,1]])
print(df1)
print(df2)
column1 column2 column3
0 7 1 3
1 2 5 6
2 3 2 3
3 1 4 9
4 3 1 7
5 2 5 1
6 5 5 2
7 3 3 3
8 2 1 7
9 4 5 5
10 6 3 4
11 8 5 1
12 1 8 4
13 3 1 2
14 7 6 9
15 3 4 6
16 7 2 5
17 2 3 1
18 6 9 4
19 3 1 1
20 8 4 3
0 1 2
0 2 3 4
1 3 4 1
2 3 6 1
This method produces the output I want by using pd.iterrows()
RunningSum = []
for index, rows in df1.iterrows():
if index > 3:
Div = abs((((df2 / df1.iloc[index-3+1:index+1].reset_index(drop="True").values)-1)*100))
Average = Div.sum(axis=0)
SumOfAverages = np.sum(Average)
RunningSum.append(SumOfAverages)
#printing my desired output values
print(RunningSum)
[991.2698412698413,
636.2698412698412,
456.19047619047626,
616.6666666666667,
935.7142857142858,
627.3809523809524,
592.8571428571429,
350.8333333333333,
449.1666666666667,
1290.0,
658.531746031746,
646.031746031746,
597.4603174603175,
478.80952380952385,
383.0952380952381,
980.5555555555555,
612.5]
Finally, below is my attemt to use pd.rolling() so that I dont need to loop through each row.
def SumOfAverageFunction(vals):
Div = abs((((df2.values / vals.reset_index(drop="True").values)-1)*100))
Average = Div.sum()
SumOfAverages = np.sum(Average)
return SumOfAverages
RunningSums = df1.rolling(window=3,axis=0).apply(SumOfAverageFunction)
Here is my problem because printing RunningSums from above outputs several values and is not close to the results I'm getting using iterrows method. How do I solve this?
print(RunningSums)
column1 column2 column3
0 NaN NaN NaN
1 NaN NaN NaN
2 702.380952 780.000000 283.333333
3 533.333333 640.000000 533.333333
4 1200.000000 475.000000 403.174603
5 833.333333 1280.000000 625.396825
6 563.333333 760.000000 1385.714286
7 346.666667 386.666667 1016.666667
8 473.333333 573.333333 447.619048
9 533.333333 1213.333333 327.619048
10 375.000000 746.666667 415.714286
11 408.333333 453.333333 515.000000
12 604.166667 338.333333 1250.000000
13 1366.666667 577.500000 775.000000
14 847.619048 1400.000000 683.333333
15 314.285714 733.333333 455.555556
16 533.333333 441.666667 474.444444
17 347.619048 616.666667 546.666667
18 735.714286 466.666667 1290.000000
19 350.000000 488.888889 875.000000
20 525.000000 1361.111111 1266.666667
It's just the way rolling behaves, it's going to window around all of the columns and I don't know that there is a way around it. One solution is to apply rolling to a single column, and use the indexes from those windows to slice the dataframe inside your function. Still expensive, but probably not as bad as what you're doing.
Also the output of your first method looks wrong. You're actually starting your calculations a few rows too late.
import numpy as np
def SumOfAverageFunction(vals):
return (abs(np.divide(df2.values, df1.loc[vals.index].values)-1)*100).sum()
vals = df1.column1.rolling(3)
vals.apply(SumOfAverageFunction, raw=False)
i need a little bit of help with this unique problem. i have the following dataframe:
volume
index
1 5
1 10
1 10
2 6
2 8
2 5
3 15
3 5
i want to create a new dataframe that adds all the values in the respective indices (for index 1, add 5+10+10= 25 , etc) as shown below: how do i go about it?
volume
index
1 25
2 19
3 20
Try with sum
df = df.sum(level=0)
As a pandas newbie, I am still struggling with just making my dataframe look like the data presented in the question. I settled for this approach:
import pandas as pd
df = pd.DataFrame(
{ "volume" : [ 5,10,10,6,8,5,15,5 ]}, index=[1,1,1,2,2,2,3,3] )
print(df.groupby(level=0).sum())
To yield this dataframe:
volume
1 5
1 10
1 10
2 6
2 8
2 5
3 15
3 5
And this result:
volume
1 25
2 19
3 20
What I want to do is pretty simple, in other languages. I want to split a table, using a "for" loop to split a data frame every fifth row.
The idea is that I have dataframe that adds a new row, every so often, like answering a form with different questions and every answer is added to a specific column, like Google Forms with SpreadSheet.
What I have tried is the following:
import pandas as pd
dp=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]
df1=pd.DataFrame(data=dp)
for i in range(0, len(dp)):
if i%5==0:
df = df1.iloc[i,:]
print(df)
print(df)
Which I know isn't much but nevertheless it is a try. Now, what I can't do is create a new variable with the new dataframe every time the loop reaches the i mod 5 == 0 row.
I think you're trying to convert a flat list into rows and columns using a known number of fields.
I'd do something like this:
import numpy as np
import pandas as pd
numFields = 3 # this is five in your case
fieldNames = ['color', 'animal', 'amphibian'] # totally optional
# this is your 'dp'
inputData = ['brown', 'dog','false','green', 'toad','true']
flatDataArray = np.asarray(inputData)
reshapedData = flatDataArray.reshape(-1, numFields)
df = pd.DataFrame(reshapedData, columns=fieldNames) # you only need 'columns' if you want to name fields
print(df)
which gives:
color animal amphibian
0 brown dog false
1 green toad true
--UPDATE--
From your comment above, I see that you'd like an arbitrary number of dataframes- one for each five-row group. Why not create a list of dataframes (i.e. so you have dfs[0], dfs[1])?
# continuing with from where the previous code left off...
dfs = []
for group in reshapedData:
dfs.append(pd.DataFrame(group))
for df in dfs:
print(df)
which prints:
0
0 brown
1 dog
2 false
0
0 green
1 toad
2 true
numpy.split
lod = np.split(df1, np.arange(1, 16, 5))
print(*lod, sep='\n\n')
0
0 0
0
1 1
2 2
3 3
4 4
5 5
0
6 6
7 7
8 8
9 9
10 10
0
11 11
12 12
13 13
14 14
15 15
lod = np.split(df1, np.arange(0, 16, 5)[1:])
print(*lod, sep='\n\n')
0
0 0
1 1
2 2
3 3
4 4
0
5 5
6 6
7 7
8 8
9 9
0
10 10
11 11
12 12
13 13
14 14
0
15 15
I am working with ICD-9 codes for a data mining project using python and I am having trouble converting the specific codes into categories. For example, I am trying to change everything that's between 001 and 139 with 0, everything that's between 140 and 239 with 1, etc
This is what I have tried:
df = df.replace({'diag_1' : {'(1-139)' : 0, '(140-239)' : 1}})
You can use pd.cut to achieve this:
In [175]:
df = pd.DataFrame({'value':np.random.randint(0,20,10)})
df
Out[175]:
value
0 12
1 2
2 10
3 5
4 19
5 2
6 8
7 14
8 12
9 16
here we set bin intervals of (0-5) (5-15), (15-20):
In [183]:
df['new_value'] = pd.cut(df['value'], bins=[0,5,15,20], labels=[0,1,2])
df
Out[183]:
value new_value
0 12 1
1 2 0
2 10 1
3 5 0
4 19 2
5 2 0
6 8 1
7 14 1
8 12 1
9 16 2
I think in your case the following should work:
df['diag_1']= pd.cut(df['diag_1'], [1,140,240] , labels=[1,2,3])
you can set the bins and labels dynamically using np.arange or similar
There is nothing wrong with an if-statement.
newvalue = 1 if oldvalues <= 139 else 2
Apply this function as a lambda expression with map.