I have a dataframe with lots of data and 1 column that is structured like this:
index var_1
1 a=3:b=4:c=5:d=6:e=3
2 b=3:a=4:c=5:d=6:e=3
3 e=3:a=4:c=5:d=6
4 c=3:a=4:b=5:d=6:f=3
I am trying to structure the data in that column to look like this:
index a b c d e f
1 3 4 5 6 3 0
2 4 3 5 6 3 0
3 4 0 5 6 3 0
4 4 5 3 6 0 3
I have done the following thus far:
df1 = df['var1'].str.split(':', expand=True)
I can then loop through the cols of df1 and do another split on '=', but then I'll just have loads of disorganised label cols and value cols.
Use list comprehension with dictionaries for each value and pass to DataFrame constructor:
comp = [dict([y.split('=') for y in x.split(':')]) for x in df['var_1']]
df = pd.DataFrame(comp).fillna(0).astype(int)
print (df)
a b c d e f
0 3 4 5 6 3 0
1 4 3 5 6 3 0
2 4 0 5 6 3 0
3 4 5 3 6 0 3
Or use Series.str.split with expand=True for DataFrame, reshape by DataFrame.stack, again split, remove first level of MultiIndex and add new level by 0 column, last reshape by Series.unstack:
df = (df['var_1'].str.split(':', expand=True)
.stack()
.str.split('=', expand=True)
.reset_index(level=1, drop=True)
.set_index(0, append=True)[1]
.unstack(fill_value=0)
.rename_axis(None, axis=1))
print (df)
a b c d e f
1 3 4 5 6 3 0
2 4 3 5 6 3 0
3 4 0 5 6 3 0
4 4 5 3 6 0 3
Here's one approach using str.get_dummies:
out = df.var_1.str.get_dummies(sep=':')
out = out * out.columns.str[2:].astype(int).values
out.columns = pd.MultiIndex.from_arrays([out.columns.str[0], out.columns])
print(out.max(axis=1, level=0))
a b c d e f
index
1 3 4 5 6 3 0
2 4 3 5 6 3 0
3 4 0 5 6 3 0
4 4 5 3 6 0 3
You can apply "extractall" and "pivot".
After "extractall" you get:
0 1
index match
1 0 a 3
1 b 4
2 c 5
3 d 6
4 e 3
2 0 b 3
1 a 4
2 c 5
3 d 6
4 e 3
3 0 e 3
1 a 4
2 c 5
3 d 6
4 0 c 3
1 a 4
2 b 5
3 d 6
4 f 3
And in one step:
rslt= df.var_1.str.extractall(r"([a-z])=(\d+)") \
.reset_index(level="match",drop=True) \
.pivot(columns=0).fillna(0)
1
0 a b c d e f
index
1 3 4 5 6 3 0
2 4 3 5 6 3 0
3 4 0 5 6 3 0
4 4 5 3 6 0 3
#rslt.columns= rslt.columns.levels[1].values
Related
I have a dataframe as below:
df = pd.DataFrame({'ORDER':["A", "A", "A", "B", "B","B"], 'GROUP': ["A1C", "A1", "B1", "B1C", "M1", "M1C"]})
df['_A1_XYZ'] = 1
df['_A1C_XYZ'] = 2
df['_B1_XYZ'] = 3
df['_B1C_XYZ'] = 4
df['_M1_XYZ'] = 5
df
ORDER GROUP _A1_XYZ _A1C_XYZ _B1_XYZ _B1C_XYZ _M1_XYZ
0 A A1C 1 2 3 4 5
1 A A1 1 2 3 4 5
2 A B1 1 2 3 4 5
3 B B1C 1 2 3 4 5
4 B M1 1 2 3 4 5
5 B M1C 1 2 3 4 5
I want to create a column "NEW" based on column "GROUP" and all the columns that ends with XYZ as below:
Based on the value of GROUP for each row df["NEW"] = df["_XYZ"].
For example, for 1st row, GROUP = A1C, So "NEW" = 2 (_A1C_XYZ), Similarly for 2nd row "NEW" = 1 (_A1_XYZ)
My expected output
ORDER GROUP _A1_XYZ _A1C_XYZ _B1_XYZ _B1C_XYZ _M1_XYZ NEW
0 A A1C 1 2 3 4 5 2
1 A A1 1 2 3 4 5 1
2 A B1 1 2 3 4 5 3
3 B B1C 1 2 3 4 5 4
4 B M1 1 2 3 4 5 5
5 B M1C 1 2 3 4 5
Use pd.DataFrame.lookup:
df['NEW'] = df.lookup(df.index, '_'+df['GROUP']+'_XYZ')
df
Output:
ORDER GROUP _A1_XYZ _A1C_XYZ _B1_XYZ _B1C_XYZ _M1_XYZ _M1C_XYZ NEW
0 A A1C 1 2 3 4 5 6 2
1 A A1 1 2 3 4 5 6 1
2 A B1 1 2 3 4 5 6 3
3 B B1C 1 2 3 4 5 6 4
4 B M1 1 2 3 4 5 6 5
5 B M1C 1 2 3 4 5 6 6
Updated after question edited.
Or use stack and reindex,
(df['New'] = df.stack().reindex(zip(df.index, '_'+dfl['GROUP']+'_XYZ'))
.rename('NEW').reset_index(level=1, drop=True))
df
Output:
ORDER GROUP _A1_XYZ _A1C_XYZ _B1_XYZ _B1C_XYZ _M1_XYZ New
0 A A1C 1 2 3 4 5 2
1 A A1 1 2 3 4 5 1
2 A B1 1 2 3 4 5 3
3 B B1C 1 2 3 4 5 4
4 B M1 1 2 3 4 5 5
5 B M1C 1 2 3 4 5 NaN
#ScottBoston's answer is better if all of the values in the rows are also columns, but I thought I'd share mine! Essentially, I create a new dataframe with the relevant columns, drop the duplicates, change the column names, transpose the dataframe and merge the column back in...
a = df.iloc[:,2:].drop_duplicates()
a.columns = [col.split('_')[1] for col in df.columns if '_' in col]
a = a.T.rename({0:'NEW'}, axis=1)
df = pd.merge(df, a, how='left', left_on='GROUP', right_index=True)
df
output:
ORDER GROUP _A1_XYZ _A1C_XYZ _B1_XYZ _B1C_XYZ _M1_XYZ NEW
0 A A1C 1 2 3 4 5 2.0
1 A A1 1 2 3 4 5 1.0
2 A B1 1 2 3 4 5 3.0
3 B B1C 1 2 3 4 5 4.0
4 B M1 1 2 3 4 5 5.0
5 B M1C 1 2 3 4 5 NaN
I have a data-frame with n rows:
df = 1 2 3
4 5 6
4 2 3
3 1 9
6 7 0
9 2 5
I want to add a columns with the same value in groups of 3.
n (num rows) is for sure divided by 3.
So the new df will be:
df = 1 2 3 A
4 5 6 A
4 2 3 A
3 1 9 B
6 7 0 B
9 2 5 B
What is the best way to do so?
First remove last rows if not dividsable by 3 with DataFrame.iloc and then create 100% unique group by divide by 3 with integer division by 3:
print (df)
a b d
0 1 2 3
1 4 5 6
2 4 2 3
3 3 1 9
4 6 7 0
5 9 2 5
6 0 0 4 <- removed last row
N = 3
num = len(df) // N * N
df = df.iloc[:num]
df['groups'] = np.arange(len(df)) // N
print (df)
a b d groups
0 1 2 3 0
1 4 5 6 0
2 4 2 3 0
3 3 1 9 1
4 6 7 0 1
5 9 2 5 1
IIUC, groupby:
df['new_col'] = df.sum(1).groupby(np.arange(len(df))//3).transform('sum')
Output:
0 1 2 new_col
0 1 2 3 30
1 4 5 6 30
2 4 2 3 30
3 3 1 9 42
4 6 7 0 42
5 9 2 5 42
I have dataframe df with following data.
A B C D
1 1 3 1
1 2 9 8
1 3 3 9
2 1 2 9
2 2 1 4
2 3 9 5
2 4 6 4
3 1 4 1
3 2 0 4
4 1 2 6
5 1 2 4
5 2 8 3
grp = df.groupby('A')
Next I want to make all groups of dataframe df grouped on columns A to have same number of rows. Either Truncate extra rows or pad 0 rows. For above data, I want to make all groups to have 3 rows. I required the following results.
A B C D
1 1 3 1
1 2 9 8
1 3 3 9
2 1 2 9
2 2 1 4
2 3 9 5
3 1 4 1
3 2 0 4
3 0 0 0
4 1 2 6
4 0 0 0
4 0 0 0
5 1 2 4
5 2 8 3
5 0 0 0
Similarly, I may want to groupby on multiple columns, like
grp = df.groupby(['A','B'])
Use GroupBy.cumcount for counter column with DataFrame.reindex by MultiIndex.from_product:
df['g'] = df.groupby('A').cumcount()
mux = pd.MultiIndex.from_product([df['A'].unique(), range(3)], names=('A','g'))
df = (df.set_index(['A','g'])
.reindex(mux, fill_value=0)
.reset_index(level=1, drop=True)
.reset_index())
print (df)
A B C D
0 1 1 3 1
1 1 2 9 8
2 1 3 3 9
3 2 1 2 9
4 2 2 1 4
5 2 3 9 5
6 3 1 4 1
7 3 2 0 4
8 3 0 0 0
9 4 1 2 6
10 4 0 0 0
11 4 0 0 0
12 5 1 2 4
13 5 2 8 3
14 5 0 0 0
Another solution with DataFrame.merge with left join with helper DataFrame:
from itertools import product
df['g'] = df.groupby('A').cumcount()
df1 = pd.DataFrame(list(product(df['A'].unique(), range(3))), columns=['A','g'])
df = df1.merge(df, how='left').fillna(0).astype(int).drop('g', axis=1)
print (df)
A B C D
0 1 1 3 1
1 1 2 9 8
2 1 3 3 9
3 2 1 2 9
4 2 2 1 4
5 2 3 9 5
6 3 1 4 1
7 3 2 0 4
8 3 0 0 0
9 4 1 2 6
10 4 0 0 0
11 4 0 0 0
12 5 1 2 4
13 5 2 8 3
14 5 0 0 0
EDIT:
df['g'] = df.groupby(['A','B']).cumcount()
mux = pd.MultiIndex.from_product([df['A'].unique(),
df['B'].unique(),
range(3)], names=('A','B','g'))
df = (df.set_index(['A','B','g'])
.reindex(mux, fill_value=0)
.reset_index(level=2, drop=True)
.reset_index())
print (df.head(10))
A B C D
0 1 1 3 1
1 1 1 0 0
2 1 1 0 0
3 1 2 9 8
4 1 2 0 0
5 1 2 0 0
6 1 3 3 9
7 1 3 0 0
8 1 3 0 0
9 1 4 0 0
from itertools import product
df['g'] = df.groupby(['A','B']).cumcount()
df1 = pd.DataFrame(list(product(df['A'].unique(),
df['B'].unique(),
range(3))), columns=['A','B','g'])
df = df1.merge(df, how='left').fillna(0).astype(int).drop('g', axis=1)
In the example below, I want to update column C for the last 3 rows to the value 0.
Source Dataframe
A B C D E
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3
4 4 4 4 4
5 5 5 5 5
Target Dataframe
A B C D E
1 1 1 1 1
2 2 2 2 2
3 3 0 3 3
4 4 0 4 4
5 5 0 5 5
I tried something like
df.tail(3)['C']=0
but it does not work. Any idea?
You can settle for
df.loc[df.tail(3).index, 'C'] = 0
You can use:
df.iloc[-3:]['C'] = 0
Output:
A B C D E
0 1 1 1 1 1
1 2 2 2 2 2
2 3 3 0 3 3
3 4 4 0 4 4
4 5 5 0 5 5
Other way:
df[-3:]['C'] = 0
I have two df,
First df
A B C
1 1 3
1 1 2
1 2 5
2 2 7
2 3 7
Second df
B D
1 5
2 6
3 4
The column Bhas the same meaning in the both dfs. What is the most easy way add column D to the corresponding values in the first df? Output should be:
A B C D
1 1 3 5
1 1 2 5
1 2 5 6
2 2 7 6
2 3 7 4
Perform a 'left' merge in your case on column 'B':
In [206]:
df.merge(df1, how='left', on='B')
Out[206]:
A B C D
0 1 1 3 5
1 1 1 2 5
2 1 2 5 6
3 2 2 7 6
4 2 3 7 4
Another method would be to set 'B' on your second df as the index and then call map:
In [215]:
df1 = df1.set_index('B')
df['D'] = df['B'].map(df1['D'])
df
Out[215]:
A B C D
0 1 1 3 5
1 1 1 2 5
2 1 2 5 6
3 2 2 7 6
4 2 3 7 4