I have four pandas dataframes that can be generated with the below code:
#df 1
time1=pandas.Series([0,20,40,60,120])
pPAK2=pandas.Series([0,3,15,21,23])
cols=['time','pPAK2']
df=pandas.DataFrame([time1,pPAK2])
df=df.transpose()
df.columns=cols
df.to_csv('pPAK2.csv',sep='\t')
pak2_df=df
#df2
time2=pandas.Series([0,15,30,60,120])
cAbl=pandas.Series([0,15,34,10,0])
df=pandas.DataFrame([time2,cAbl])
df=df.transpose()
cols=['time','pcAbl']
df.columns=cols
df.to_csv('pcAbl.csv',sep='\t')
pcAbl_df=df
#df 3
time7=pandas.Series([0,60,120,240,480,960,1440])
pSmad3_n=pandas.Series([0,16,14,12,8,7.5,6])
scale_factor=40
pSmad3_n=pSmad3_n*scale_factor
#plt.plot(time7,pSmad3)
df=pandas.DataFrame([time7,pSmad3_n])
df=df.transpose()
cols=['time','pSmad3_n']
df.columns=cols
df.to_csv('pSmad3_n.csv',sep='\t')
smad3_df=df
#df4
time8=pandas.Series([0,240,480,1440])
PAI1_mRNA=pandas.Series([0,23,25,5])
scale_factor=5
PAI1_mRNA=PAI1_mRNA*scale_factor
df=pandas.DataFrame([time8,PAI1_mRNA])
df=df.transpose()
cols=['time','PAI1_mRNA']
df.columns=cols
df.to_csv('PAI1_mRNA.csv',sep='\t')
PAI1_df=df
#print dataframes
print PAI1_df
print pak2_df
print pcAbl_df
print smad3_df
I want to concatenate these dataframes by the time column with the pandas concat function but I can't get the output right. The output should look something like this, if were to just concatenate PAI1_df and pak2_df
time PAI1_mRNA pPAK2
0 0 0 0
1 20 'NaN' 3
2 40 'NaN' 15
3 60 'NaN' 21
4 120 'NaN' 23
5 240 115 'NaN'
6 480 125 'NaN'
7 1440 25 'NaN
I think it should be easy but there are a lot of features in the doc, does anybody know how to do this?
So you can concat it like this:
import pandas
df = pandas.concat([pak2_df.set_index('time'), pcAbl_df.set_index('time')], axis=1).reset_index()
print(df)
Prints:
time pPAK2 pcAbl
0 0 0 0
1 15 NaN 15
2 20 3 NaN
3 30 NaN 34
4 40 15 NaN
5 60 21 10
6 120 23 0
Related
I want to stack two columns on top of each other
So I have Left and Right values in one column each, and want to combine them into a single one. How do I do this in Python?
I'm working with Pandas Dataframes.
Basically from this
Left Right
0 20 25
1 15 18
2 10 35
3 0 5
To this:
New Name
0 20
1 15
2 10
3 0
4 25
5 18
6 35
7 5
It doesn't matter how they are combined as I will plot it anyway, and the new column name also doesn't matter because I can rename it.
You can create a list of the cols, and call squeeze to anonymise the data so it doesn't try to align on columns, and then call concat on this list, passing ignore_index=True creates a new index, otherwise you'll get the names as index values repeated:
cols = [df[col].squeeze() for col in df]
pd.concat(cols, ignore_index=True)
Many options, stack, melt, concat, ...
Here's one:
>>> df.melt(value_name='New Name').drop('variable', 1)
New Name
0 20
1 15
2 10
3 0
4 25
5 18
6 35
7 5
You can also use np.ravel:
import numpy as np
out = pd.DataFrame(np.ravel(df.values.T), columns=['New name'])
print(out)
# Output
New name
0 20
1 15
2 10
3 0
4 25
5 18
6 35
7 5
Update
If you have only 2 cols:
out = pd.concat([df['Left'], df['Right']], ignore_index=True).to_frame('New name')
print(out)
# Output
New name
0 20
1 15
2 10
3 0
4 25
5 18
6 35
7 5
Solution with unstack
df2 = df.unstack()
# recreate index
df2.index = np.arange(len(df2))
A solution with masking.
# Your data
import numpy as np
import pandas as pd
df = pd.DataFrame({"Left":[20,15,10,0], "Right":[25,18,35,5]})
# Masking columns to ravel
df2 = pd.DataFrame({"New Name":np.ravel(df[["Left","Right"]])})
df2
New Name
0 20
1 25
2 15
3 18
4 10
5 35
6 0
7 5
I ended up using this solution, seems to work fine
df1 = dfTest[['Left']].copy()
df2 = dfTest[['Right']].copy()
df2.columns=['Left']
df3 = pd.concat([df1, df2],ignore_index=True)
I'd like to merge this dataframe:
import pandas as pd
import numpy as np
df1 = pd.DataFrame([[1,10,100],[2,20,np.nan],[3,30,300]], columns=["A","B","C"])
df1
A B C
0 1 10 100
1 2 20 NaN
2 3 30 300
with this one:
df2 = pd.DataFrame([[1,422],[10,72],[2,278],[300,198]], columns=["ID","Value"])
df2
ID Value
0 1 422
1 10 72
2 2 278
3 300 198
to get an output:
df_output = pd.DataFrame([[1,10,100,422],[1,10,100,72],[2,20,200,278],[3,30,300,198]], columns=["A","B","C","Value"])
df_output
A B C Value
0 1 10 100 422
1 1 10 100 72
2 2 20 NaN 278
3 3 30 300 198
The idea is that for df2 the key column is "ID", while for df1 we have 3 possible key columns ["A","B","C"].
Please notice that the numbers in df2 are chosen to be like this for simplicity, and they can include random numbers in practice.
How do I perform such a merge? Thanks!
IIUC, you need a double merge/join.
First, melt df1 to get a single column, while keeping the index. Then merge to get the matches. Finally join to the original DataFrame.
s = (df1
.reset_index().melt(id_vars='index')
.merge(df2, left_on='value', right_on='ID')
.set_index('index')['Value']
)
# index
# 0 422
# 1 278
# 0 72
# 2 198
# Name: Value, dtype: int64
df_output = df1.join(s)
output:
A B C Value
0 1 10 100.0 422
0 1 10 100.0 72
1 2 20 NaN 278
2 3 30 300.0 198
Alternative with stack + map:
s = df1.stack().droplevel(1).map(df2.set_index('ID')['Value']).dropna()
df_output = df1.join(s.rename('Value'))
I'm writing a code to merge several dataframe together using pandas .
Here is my first table :
Index Values Intensity
1 11 98
2 12 855
3 13 500
4 24 140
and here is the second one:
Index Values Intensity
1 21 1000
2 11 2000
3 24 0.55
4 25 500
With these two df, I concanate and drop_duplicates the Values columns which give me the following df :
Index Values Intensity_df1 Intensity_df2
1 11 0 0
2 12 0 0
3 13 0 0
4 24 0 0
5 21 0 0
6 25 0 0
I would like to recover the intensity of each values in each Dataframes, for this purpose, I'm iterating through each line of each df which is very inefficient. Here is the following code I use:
m = 0
while m < len(num_df):
n = 0
while n < len(df3):
temp_intens_abs = df[m]['Intensity'][df3['Values'][n] == df[m]['Values']]
if temp_intens_abs.empty:
merged.at[n,"Intensity_df%s" %df[m]] = 0
else:
merged.at[n,"Intensity_df%s" %df[m]] = pandas.to_numeric(temp_intens_abs, errors='coerce')
n = n + 1
m = m + 1
The resulting df3 looks like this at the end:
Index Values Intensity_df1 Intensity_df2
1 11 98 2000
2 12 855 0
3 13 500 0
4 24 140 0.55
5 21 0 1000
6 25 0 500
My question is : Is there a way to directly recover "present" values in a df by comparing directly two columns using pandas? I've tried several solutions using numpy but without success.. Thanks in advance for your help.
You can try joining these dataframes: df3 = df1.merge(df2, on="Values")
I am quite new to pandas, and I have a numpy list looking like so:
something=[10,20,30,40,50]
When I convert it to a pandas dataframe hgowever, I have the entire list as one element:
dataset = pd.DataFrame({'something': something, \
'something2': something2}, \
columns=['something', 'something2'])
and I get:
something
0 [10,20,30,40,50]
What I would like is:
0 1 2 3 4
0 10 20 30 40 50
i.e list elements as individual columns.
You can do this using pd.Dataframe.from_records:
In [323]: df = pd.DataFrame.from_records([something])
In [324]: df
Out[324]:
0 1 2 3 4
0 10 20 30 40 50
For multiple lists, you can simply do this:
In [337]: something2 = [101,201,301,401,501]
In [338]: df = pd.DataFrame.from_records([something, something2])
In [339]: df
Out[339]:
0 1 2 3 4
0 10 20 30 40 50
1 101 201 301 401 501
EDIT: After OP's comment
If you want all lists to be creating multiple columns, you can try this:
In [349]: something
Out[349]: [10, 20, 30, 40, 50]
In [350]: something2
Out[350]: [101, 201, 301, 401, 501]
In [351]: something.extend(something2)
In [353]: df = pd.DataFrame.from_records([something])
In [354]: df
Out[354]:
0 1 2 3 4 5 6 7 8 9
0 10 20 30 40 50 101 201 301 401 501
pandas dataframe from dict could help:
something=[10,20,30,40,50]
something2 = [25,30,22,1,5]
data = {'something':something,'something2':something2}
pd.DataFrame.from_dict(data,orient='index')
0 1 2 3 4
something 10 20 30 40 50
something2 25 30 22 1 5
If you don't care for the indexes, and want them to be integers, reset_index should suffice:
pd.DataFrame.from_dict(data,orient='index').reset_index(drop=True)
If you are passing dictionary in Dataframe then by default, pandas treat the key as a column, you don't need to give columns name again, unless if you want different column names.
I tried following example:
import pandas as pd
something1=[10,20,30,40,50]
something2=[101,201,301,401,501]
pd.DataFrame([something1,something2])
Output
0 1 2 3 4
0 10 20 30 40 50
1 101 201 301 401 501
let me know if this works for you or not.
For example, how do I reorder each column sum and row sum in the following data with summed rows and columns?
import pandas as pd
data=[['fileA',47,15,3,5,7],['fileB',33,13,4,7,2],['fileC',25,17,9,3,5],
['fileD',25,7,1,4,2],['fileE',19,15,3,8,4], ['fileF',11,17,8,4,5]]
df = pd.DataFrame(data, columns=['filename','rows_cnt','cols_cnt','col_A','col_B','col_C'])
print(df)
filename rows_cnt cols_cnt col_A col_B col_C
0 fileA 47 15 3 5 7
1 fileB 33 13 4 7 2
2 fileC 25 17 9 3 5
3 fileD 25 7 1 4 2
4 fileE 19 15 3 8 4
5 fileF 11 17 8 4 5
df.loc[6]= df.sum(0)
filename rows_cnt cols_cnt col_A col_B col_C
0 fileA 47 15 3 5 7
1 fileB 33 13 4 7 2
2 fileC 25 17 9 3 5
3 fileD 25 7 1 4 2
4 fileE 19 15 3 8 4
5 fileF 11 17 8 4 5
6 fileA... 160 84 28 31 25
I made an image of the question.
How do I reorder the red frame in this image by the standard?
df.reindex([2,5,0,4,1,3,6], axis='index')
Is the only way to create the index manually like this?
data=[['fileA',47,15,3,5,7],['fileB',33,13,4,7,2],['fileC',25,17,9,3,5],
['fileD',25,7,1,4,2],['fileE',19,15,3,8,4], ['fileF',11,17,8,4,5]]
df = pd.DataFrame(data, columns=['filename','rows_cnt','cols_cnt','col_A','col_B','col_C'])
df = df.sort_values(by='cols_cnt', axis=0, ascending=False)
df.loc[6]= df.sum(0)
# to keep number original of index
df = df.reset_index(drop=False)
# need to remove this filename column, since need to sort by column (axis=1)
# unable sort with str and integer data type
df = df.set_index('filename', drop=True)
df = df.sort_values(by=df.index[-1], axis=1, ascending=False)
# set back the index of dataframe into original
df = df.reset_index(drop=False)
df = df.set_index('index', drop=True)
# try to set the fixed columns
fixed_cols = ['filename', 'rows_cnt','cols_cnt']
# try get the new order of columns by fixed the first three columns
# and then add with the remaining columns
new_cols = fixed_cols + (df.columns.drop(fixed_cols).tolist())
df[new_cols]