I want to make a data frame in pandas that look like this:
Id Name Gender Math Science English
1 Ram Male 98 92 80
2 Hari Male 30 40 23
3 Gita Female 60 65 77
4 Sita Female 50 45 55
5 Shyam Male 80 88 82
I wrote quote in python like this:
import pandas as pd
d = {'Id':[1,2,3,4,5], 'Name':['Ram','Hari','Gita','Sita','Shyam'],'Gender':['Male','Male','Female','Female','Male'],'Math':[98,30,60,50,80],'Science':[92,40,65,45,88],'English':[80,23,77,55,82]}
df = pd.DataFrame(data=d)
print (df)
It gave me output like this:
English Gender Id Math Name Science
0 80 Male 1 98 Ram 92
1 23 Male 2 30 Hari 40
2 77 Female 3 60 Gita 65
3 55 Female 4 50 Sita 45
4 82 Male 5 80 Shyam 88
How do I remove the first column with no attribute and also arrange attributes in such a way that is given in the question?
I want Id, Name, Gender, Math, Science, English. Thanks
If you don't want index, you can set it by unique column like Id.
import pandas as pd
d = {'Id':[1,2,3,4,5], 'Name':['Ram','Hari','Gita','Sita','Shyam'],'Gender':['Male','Male','Female','Female','Male'],'Math':[98,30,60,50,80],'Science':[92,40,65,45,88],'English':[80,23,77,55,82]}
df = pd.DataFrame(data=d)
df.set_index('Id', inplace=True)
print (df)
Output:
Name Gender Math Science English
Id
1 Ram Male 98 92 80
2 Hari Male 30 40 23
3 Gita Female 60 65 77
4 Sita Female 50 45 55
5 Shyam Male 80 88 82
Try to create directly the DataFrame instead of passing by "d"
df = pd.DataFrame({'Id': [1, 4, 7, 10], etc...})
Then use set_index to fix your Id as it :
df.set_index('Id')
Related
Student
Test
Grade
Jimmy
1
100
Jimmy
2
83
Jimmy
3
85
Drew
1
93
Drew
2
95
Drew
3
100
Smith
1
89
Smith
2
86
Smith
3
89
Billy
1
97
Billy
2
80
Billy
3
78
With the table given, I want to check if the student shares the same name, then I want to save their next test score so for Jimmy it would be:
Student
Test
Grade
Nexscore
Jimmy
1
100
83
Jimmy
2
83
85
Jimmy
3
85
Nan
If this only applies to Jimmy, then you can do the following: get True on indexes where Jimmy is.
df['Student'] == 'Jimmy'
And to get the Nexscore column, apply the shift operator.
All where the result from data frame df is written to df1:
import pandas as pd
df1 = df[df['Student'] == 'Jimmy'].copy()
df1['Nexscore'] = df1.Grade.shift(-1)
print(df1)
Output
Student Test Grade Nexscore
0 Jimmy 1 100 83.0
1 Jimmy 2 83 85.0
2 Jimmy 3 85 NaN
To check how many names are repeated, measure the length of the resulting dataframe:
print(len(df1))
if len(df1) > 1:
#your code
Two seperate similar DataFrames with different lengths
df2=
Index
First Name
Age
Gender
Weight
0
James
25
Male
155
1
John
27
Male
175
2
Patricia
23
Female
135
3
Mary
22
Female
125
4
Martin
30
Male
185
5
Margaret
29
Female
141
6
Kevin
22
Male
198
df1=
Index
First Name
Age
Gender
Weight
Height
0
James
25
Male
165
5'10
1
John
27
Male
175
5'9
2
Matthew
29
Male
183
6'0
3
Patricia
23
Female
135
5'3
4
Mary
22
Female
125
5'4
5
Rachel
29
Female
123
5'3
6
Jose
20
Male
175
5'11
7
Kevin
22
Male
192
6'2
df2 has some rows which are not in df1 and df1 has some values which are not in df2.
I am comparing df1 against df2. I have calculated the newentries with the following code
newentries = df2.loc[~df2['First Name'].isin(df1['First Name'])]
deletedentries = df1.loc[~df1['First Name'].isin(df2['First Name'])]
where newentries denote the rows/entries that are there in df2 but not in df1; deletedentries denote the rows/entries that are there in df1 but not in df2. The above code works perfectly fine.
I need to copy the height from df1 to df2 when the first names are equal.
df2.loc[df2['First Name'].isin(df1['First Name']),"Height"] = df1.loc[df1['First Name'].isin(df2['First Name']),"Height"]
The above code copies the values however indexing is causing an issue and the values are not copied to the corresponding rows, I tried to promote First Name as the Index but that doesn't solve the issue. Please help me with a solution
Also, I need to calculate the modified values, if the First Name is same, I need to check for modified values; for example in df1, the weight of James is 155 however in df2 the weight is 165, so I need to store the modified weight of James(165) and index(0) in a new dataframe without iteration; the iteration takes a long time because this is a sample of a big dataframe with a lot more rows and columns.
Desired output:
df2=
Index
First Name
Age
Gender
Weight
Height
0
James
25
Male
155
5'10
1
John
27
Male
175
5'9
2
Patricia
23
Female
135
5'3
3
Mary
22
Female
125
5'4
4
Martin
30
Male
185
5
Margaret
29
Female
141
6
Kevin
22
Male
198
6'2
Martin's and Margaret's heights are not there in df1, so their heights are not updated in df2
newentries=
Index
First Name
Age
Gender
Weight
Height
4
Martin
30
Male
185
5
Margaret
29
Female
141
deletedentries=
Index
First Name
Age
Gender
Weight
Height
2
Matthew
29
Male
183
6'0
5
Rachel
29
Male
123
5'3
6
Jose
20
Male
175
5'11
modval=
Index
First Name
Age
Gender
Weight
Height
0
James
165
7
Kevin
192
Building off of Rabinzel's answer:
output = df2.merge(df1, how='left', on='First Name', suffixes=[None, '_old'])
df3 = output[['First Name', 'Age', 'Gender', 'Weight', 'Height']]
cols = df1.columns[1:-1]
modval = pd.DataFrame()
for col in cols:
modval = pd.concat([modval, output[['First Name', col + '_old']][output[col] != output[col + '_old']].dropna()])
modval.rename(columns={col +'_old':col}, inplace=True)
newentries = df2[~df2['First Name'].isin(df1['First Name'])]
deletedentries = df1[~df1['First Name'].isin(df2['First Name'])]
print(df3, newentries, deletedentries, modval, sep='\n\n')
Output:
First Name Age Gender Weight Height
0 James 25 Male 155 5'10
1 John 27 Male 175 5'9
2 Patricia 23 Female 135 5'3
3 Mary 22 Female 125 5'4
4 Martin 30 Male 185 NaN
5 Margaret 29 Female 141 NaN
6 Kevin 22 Male 198 6'2
First Name Age Gender Weight
4 Martin 30 Male 185
5 Margaret 29 Female 141
First Name Age Gender Weight Height
2 Matthew 29 Male 183 6'0
5 Rachel 29 Male 123 5'3
6 Jose 20 Male 175 5'11
First Name Age Gender Weight
0 James NaN NaN 165.0
6 Kevin NaN NaN 192.0
for your desired output for df2 you can try this:
desired_df2 = df2.merge(df1[['First Name','Height']], on='First Name', how='left')
#if you want to change the "NaN" values just add ".fillna(fill_value=0)" for e.g 0 after the merge
print(desired_df2)
First Name Age Gender Weight Height
0 James 25 Male 155 5'10
1 John 27 Male 175 5'9
2 Patricia 23 Female 135 5'3
3 Mary 22 Female 125 5'4
4 Martin 30 Male 185 NaN
5 Margaret 29 Female 141 NaN
6 Kevin 22 Male 198 6'2
new and deleted entries is already right. for the moment I'm a bit stuck how to get the modval dataframe. I'll update my answer if I get a solution.
I have a DataFrame that looks like the below. "Name" represents a student name and values below each of the Test variables represent the test grade.
Name Test1 Test2 Test3
Ana 87 93 82
Cole 62 73 84
Sia 64 58 60
Max 93 95 99
Leah 93 90 85
Cam 76 80 83
The desired result is the DataFrame below. Where "MinTestGrade" represents that lowest grade each student earned between the 3. "TestNumber" is the Test they got the lowest grade on.
Name TestNumber MinTestGrade
Ana 3 82
Cole 1 62
Sia 2 58
Max 1 93
Leah 3 85
Cam 1 76
How can I do this using python?
You can pass idxmin and min to agg on axis to find the minimum grade and the column name, i.e. TestNumber, that it corresponds to for each student. Then join the outcome with "Name", rename the columns and finally strip the word "Test" from "TestNumber":
out = df[['Name']].join(df.filter(like='Test').agg(['idxmin', 'min'], axis=1)).rename(columns={'idxmin':'TestNumber', 'min':'MinTestGrade'})
out['TestNumber'] = out['TestNumber'].str.lstrip('Test').astype(int)
Output:
Name TestNumber MinTestGrade
0 Ana 3 82
1 Cole 1 62
2 Sia 2 58
3 Max 1 93
4 Leah 3 85
5 Cam 1 76
df.set_index("Name").agg(["idxmin", "min"], axis=1).reset_index()
# Name idxmin min
# 0 Ana Test3 82
# 1 Cole Test1 62
# 2 Sia Test2 58
# 3 Max Test1 93
# 4 Leah Test3 85
# 5 Cam Test1 76
input: (CSV file)
name subject internal_1_marks internal_2_marks final_marks
abc python 45 50 47
pqr java 45 46 46
pqr python 40 33 37
xyz java 45 43 49
xyz node 40 30 35
xyz ruby 50 45 47
Expected output: (CSV file)
name subject internal_1_marks internal_2_marks final_marks
abc python 45 50 47
pqr java 45 46 46
python 40 33 37
xyz java 45 43 49
node 40 30 35
ruby 50 45 47
I've tried this:
df = pd.read_csv("student_info.csv")
df.groupby(['name', 'subject']).sum().to_csv("output.csv")
but it's giving duplicate in first column as shown bellow.
name subject internal_1_marks internal_2_marks final_marks
abc python 45 50 47
pqr java 45 46 46
pqr python 40 33 37
xyz java 45 43 49
xyz node 40 30 35
xyz ruby 50 45 47
I need to remove duplicate in first column as shown in expected output.
Thanks.
Similar answer here
mask = df['name'].duplicated()
df.loc[mask.values,['name']] = ''
name subject internal_1_marks internal_2_marks final_marks
0 abc python 45 50 47
1 pqr java 45 46 46
2 python 40 33 37
3 xyz java 45 43 49
4 node 40 30 35
5 ruby 50 45 47
You can filter the dupes after the group by
df.groupby(['name', 'subject']).sum().reset_index().assign(name=lambda x: x['name'].where(~x['name'].duplicated(), '')).to_csv('filename.csv', index=False)
Also when reading the file you can pass index_col for the dupes
df = pd.read_csv('test.csv', index_col=[0])
I have a dictionary of DataFrame objects:
dictDF={0:df0,1:df1,2:df2}
Each DataFrame df0,df1,df2 represents a table in a specific date of time, where the first column identifies (like social security number) a person and the other columns are characteristics of this person such as
DataFrame df0
id Name Age Gender Job Income
10 Daniel 40 Male Scientist 100
5 Anna 39 Female Doctor 250
DataFrame df1
id Name Age Gender Job Income
67 Guto 35 Male Engineer 100
7 Anna 39 Female Doctor 300
9 Melissa 26 Female Student 36
DataFrame df2
id Name Age Gender Job Income
77 Patricia 30 Female Dentist 300
9 Melissa 27 Female Dentist 250
Note that the id (social security number) identifies exactly the person. For instance, the same "Melissa" arises in two different DataFrames. However, there are two different "Annas".
In these dataFrames the number of people and the people vary over time. Some people is represented in all dates and others are represented only in a specific date of time.
Is there a simple way to transform this dictionary of data frames in an (unbalanced) Panel object, where the ids arise in all dates and if the data a given id is not available it is replaced by NaN?
Off course, I can do that making a list of all ids and then checking in each date if a given id is represented. If it is represented, then I copy the data. Otherwise, I just write NaN.
I wonder if there an easy way using pandas tools.
I would recommend using a MultiIndex instead of a Panel.
First, add the period to each dataframe:
for n, df in dictDF.iteritems():
df['period'] = n
Then concatenate into a big dataframe:
big_df = pd.concat([df for df in dictDF.itervalues()], ignore_index=True)
Now set your index to period and id and you are guaranteed to have a unique index:
>>> big_df.set_index(['period', 'id'])
Name Age Gender Job Income
period id
0 10 Daniel 40 Male Scientist 100
5 Anna 39 Female Doctor 250
1 67 Guto 35 Male Engineer 100
7 Anna 39 Female Doctor 300
9 Melissa 26 Female Student 36
2 77 Patricia 30 Female Dentist 300
9 Melissa 27 Female Dentist 250
You can also reverse that order:
>>> big_df.set_index(['id', 'period']).sort_index()
Name Age Gender Job Income
id period
5 0 Anna 39 Female Doctor 250
7 1 Anna 39 Female Doctor 300
9 1 Melissa 26 Female Student 36
2 Melissa 27 Female Dentist 250
10 0 Daniel 40 Male Scientist 100
67 1 Guto 35 Male Engineer 100
77 2 Patricia 30 Female Dentist 300
You can even unstack the data quite easily:
big_df.set_index(['id', 'period'])[['Income']].unstack('period')
Income
period 0 1 2
id
5 250 NaN NaN
7 NaN 300 NaN
9 NaN 36 250
10 100 NaN NaN
67 NaN 100 NaN
77 NaN NaN 300