I want to update data frame X on values from dataframe from Y.
X = pd.DataFrame({'A': ['A0', 'A1', 'A2'],
'B': ['B0', 'B1', 'B2'],
'C': ['C0', 'C1', 'C2'],
'D': ['D0', 'D1', 'D2']})
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1 D1
2 A2 B2 C2 D2
Y = pd.DataFrame({'A': ['A0', 'A1'],
'B': ['B0', 'B1'],
'C': ['C0xx', 'C1xx'],
'D': ['D0xx', 'D1xx']})
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
And the result to be:
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
2 A2 B2 C2 D2
Of course my dataframe is match bigger.
1. Both DataFrames have the same index
This is the case you presented in the example given in your question.
You might want to use the update method:
>>> X.update(Y)
>>> X
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
2 A2 B2 C2 D2
It also works if lines are in a different order in X and Y:
>>> Y = pd.DataFrame({'A': ['A1', 'A0'],
'B': ['B1', 'B0'],
'C': ['C1xx', 'C0xx'],
'D': ['D1xx', 'D0xx']},
index=[1,0])
>>> Y
A B C D
1 A1 B1 C1xx D1xx
0 A0 B0 C0xx D0xx
>>> X.update(Y)
>>> X
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
2 A2 B2 C2 D2
2. Different indexes
If Y has a different index:
>>> Y = pd.DataFrame({'A': ['A0', 'A1'],
'B': ['B0', 'B1'],
'C': ['C0xx', 'C1xx'],
'D': ['D0xx', 'D1xx']},
index=[2,1])
>>> Y
A B C D
2 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
You can still use update if you can find another column usable as an index (identifying the lines so that they match the lines to be replaced). I take the example of the "A" column but a multiple index would work as well.
>>> X2, Y2 = X.set_index("A"), Y.set_index("A")
>>> X2.update(Y2)
>>> X2.reset_index(inplace=True)
>>> X2
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
2 A2 B2 C2 D2
I think you need combine_first with set_index if need add missing values by A, B columns in both df:
print (Y.set_index(['A','B']).combine_first(X.set_index(['A','B'])).reset_index())
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
2 A2 B2 C2 D2
Unfortunately update works bad:
Y = pd.DataFrame({'A': ['A0', 'A1'],
'B': ['B0', 'B1'],
'C': ['C0xx', 'C1xx'],
'D': ['D0xx', 'D1xx']}, index=[2,1])
print (X)
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1 D1
2 A2 B2 C2 D2
print (Y)
A B C D
2 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
X.update(Y)
print (X)
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1xx D1xx
2 A0 B0 C0xx D0xx
X.set_index(['A','B']).update(Y.set_index(['A','B']))
print (X)
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1 D1
2 A2 B2 C2 D2
print (Y.set_index(['A','B']).combine_first(X.set_index(['A','B'])).reset_index())
A B C D
0 A0 B0 C0xx D0xx
1 A1 B1 C1xx D1xx
2 A2 B2 C2 D2
Related
Lets say I have the dataframe below:
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1 D1
2 A2 B2 C2 D2
3 A3 B3 C3 D3
4 A4 B4 C4 D4
I am trying to write something that would essentially say; if column A contains A1, A2, or A4, then add a 'column E' populated by 'xx' in the rows where any of the three variables appear.
Then create a df2 which only contains the flagged rows and a df3 which has the flagged rows and column E subtracted. Resulting in df2:
A B C D E
0 A1 B1 C1 D1 xx
1 A2 B2 C2 D2 xx
2 A4 B4 C4 D4 xx
and df3:
A B C D
0 A0 B0 C0 D0
1 A3 B3 C3 D3
Python/pandas beginner here, so any and all help is much appreciated!
You can use boolean indexing:
mask = df["A"].isin(["A1", "A2", "A4"])
df_a = df[mask].copy()
df_a["E"] = "xx"
df_b = df[~mask] # add .copy()
print(df_a)
print(df_b)
Prints:
A B C D E
1 A1 B1 C1 D1 xx
2 A2 B2 C2 D2 xx
4 A4 B4 C4 D4 xx
A B C D
0 A0 B0 C0 D0
3 A3 B3 C3 D3
i have following 2 dataframes and i want to merge them.
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']},
index=[0, 1, 2, 3])
df2 = pd.DataFrame({'z': ['A4', 'A5', 'A6', 'A7'],
'e': ['B4', 'B5', 'B6', 'B7'],
'y': ['C4', 'C5', 'C6', 'C7'],
'f': ['D4', 'D5', 'D6', 'D7']},
index=[12,2, 43,24])
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1 D1
2 A2 B2 C2 D2
3 A3 B3 C3 D3
z y f
12 B4 C4 D4
32 B5 C5 D5
43 B6 C6 D6
24 B7 C7 D7
And i want like :
A B C D z y f
0 A0 B0 C0 D0 B4 C4 D4
1 A1 B1 C1 D1 B5 C5 D5
2 A2 B2 C2 D2 B6 C6 D6
3 A3 B3 C3 D3 B7 C7 D7
can anyone help me , tried below code but i didnt get the solution
pd.concat([df1, df2], axis=1)
Blockquote
You need reset_index first
df=pd.concat([df1.reset_index(drop=True),df2.reset_index(drop=True)],axis=1)
You could either reset the index and use pd.concat (like in YOBEN_S's answer), or stack the values with numpy.
>>> pd.DataFrame(np.hstack([df1, df2]), columns=[*df1.columns, *df2.columns])
A B C D z e y f
0 A0 B0 C0 D0 A4 B4 C4 D4
1 A1 B1 C1 D1 A5 B5 C5 D5
2 A2 B2 C2 D2 A6 B6 C6 D6
3 A3 B3 C3 D3 A7 B7 C7 D7
I'm trying to join two DataFrames by index that can contain columns in common and I only want to add one to the other if that specific value is NaN or doesn't exist. I'm using the pandas example, so I've got:
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']},
index=[0, 1, 2, 3])
as
A B C D
0 A0 B0 C0 D0
1 A1 B1 C1 D1
2 A2 B2 C2 D2
3 A3 B3 C3 D3
and
df4 = pd.DataFrame({'B': ['B2p', 'B3p', 'B6p', 'B7p'],
'D': ['D2p', 'D3p', 'D6p', 'D7p'],
'F': ['F2p', 'F3p', 'F6p', 'F7p']},
index=[2, 3, 6, 7])
as
B D F
2 B2p D2p F2p
3 B3p D3p F3p
6 B6p D6p F6p
7 B7p D7p F7p
and the searched result is:
A B C D F
0 A0 B0 C0 D0 Nan
1 A1 B1 C1 D1 Nan
2 A2 B2 C2 D2 F2p
3 A3 B3 C3 D3 F3p
6 Nan B6p Nan D6p F6p
7 Nan B7p Nan D7p F7p
This is a good use case for combine_first, where the row and column indices of the resulting dataframe will be the union of the two, i.e in the absence of an index in one of the dataframes, the value from the other is used (same behaviour as if it contained a NaN:
df1.combine_first(df4)
A B C D F
0 A0 B0 C0 D0 NaN
1 A1 B1 C1 D1 NaN
2 A2 B2 C2 D2 F2p
3 A3 B3 C3 D3 F3p
6 NaN B6p NaN D6p F6p
7 NaN B7p NaN D7p F7p
I would like to concatenate two df in both directions at the same time.
It means, if the index does not exist, it is created.
And if the column does not exist, it is created also.
import pandas as pd
df1 = pd.DataFrame({'A': ['A0', 'A1', 'A2', 'A3'],
'B': ['B0', 'B1', 'B2', 'B3'],
'C': ['C0', 'C1', 'C2', 'C3'],
'D': ['D0', 'D1', 'D2', 'D3']},
index=[0, 1, 2, 3])
df2 = pd.DataFrame({'A': ['A4'],
'D': ['D4']},
index=[4])
df3 = pd.DataFrame({'A': ['E4'],
'F': ['F4']},
index=[4])
result = pd.concat([df1, df2, df3])
It gives :
A B C D F
0 A0 B0 C0 D0 NaN
1 A1 B1 C1 D1 NaN
2 A2 B2 C2 D2 NaN
3 A3 B3 C3 D3 NaN
4 A4 NaN NaN D4 NaN
4 E4 NaN NaN NaN F4
Instead of :
A B C D F
0 A0 B0 C0 D0 NaN
1 A1 B1 C1 D1 NaN
2 A2 B2 C2 D2 NaN
3 A3 B3 C3 D3 NaN
4 E4 NaN NaN D4 F4
Lets say I have a pandas dataframe as follows:
A B C D
0 a0 b0 c0 d0
1 a1 b1 c1 d1
2 a2 b2 c2 d2
3 a3 b3 c3 d3
I would like to know how I can convert it to this.
A B
0 C c0 a0 b0
D d0 a0 b0
1 C c1 a1 b1
D d1 a1 b1
2 C c2 a2 b2
D d2 a2 b2
3 C c3 a3 b3
D d3 a3 b3
basically making a few columns as rows and creating a multi index.
Well, melt will pretty much get it in the form you want and then you can set the index as desired:
print df
0 a0 b0 c0 d0
1 a1 b1 c1 d1
2 a2 b2 c2 d2
3 a3 b3 c3 d3
Now use melt to stack (note, I reset the index and use that column as an id_var because it looks like you want the [0,1,2,3] index including in the stacking):
new = pd.melt(df.reset_index(),value_vars=['C','D'],id_vars=['index','A','B'])
print new
index A B variable value
0 0 a0 b0 C c0
1 1 a1 b1 C c1
2 2 a2 b2 C c2
3 3 a3 b3 C c3
4 0 a0 b0 D d0
5 1 a1 b1 D d1
6 2 a2 b2 D d2
7 3 a3 b3 D d3
Now just set the index (well sort it and then set the index to make it look like your desired output):
new = new.sort(['index']).set_index(['index','variable','value'])
print new
A B
index variable value
0 C c0 a0 b0
D d0 a0 b0
1 C c1 a1 b1
D d1 a1 b1
2 C c2 a2 b2
D d2 a2 b2
3 C c3 a3 b3
D d3 a3 b3
If you don't need the [0,1,2,3] as part of the stack, the melt command is a bit cleaner:
print pd.melt(df,value_vars=['C','D'],id_vars=['A','B'])
A B variable value
0 a0 b0 C c0
1 a1 b1 C c1
2 a2 b2 C c2
3 a3 b3 C c3
4 a0 b0 D d0
5 a1 b1 D d1
6 a2 b2 D d2
7 a3 b3 D d3