Given a dataframe:
>>> import pandas as pd
>>> lol = [['a', 1, 1], ['b', 1, 2], ['c', 1, 4], ['c', 2, 9], ['b', 2, 10], ['x', 2, 5], ['d', 2, 3], ['e', 3, 5], ['d', 2, 10], ['a', 3, 5]]
>>> df = pd.DataFrame(lol)
>>> df.rename(columns={0:'value', 1:'key', 2:'something'})
value key something
0 a 1 1
1 b 1 2
2 c 1 4
3 c 2 9
4 b 2 10
5 x 2 5
6 d 2 3
7 e 3 5
8 d 2 10
9 a 3 5
The goal is to keep the last N rows for the unique values of the key column.
If N=1, I could simply use the .drop_duplicates() function as such:
>>> df.drop_duplicates(subset='key', keep='last')
value key something
2 c 1 4
8 d 2 10
9 a 3 5
How do I keep the last 3 rows for each unique values of key?
I could try this for N=3:
>>> from itertools import chain
>>> unique_keys = {k:[] for k in df['key']}
>>> for idx, row in df.iterrows():
... k = row['key']
... unique_keys[k].append(list(row))
...
>>>
>>> df = pd.DataFrame(list(chain(*[v[-3:] for k,v in unique_keys.items()])))
>>> df.rename(columns={0:'value', 1:'key', 2:'something'})
value key something
0 a 1 1
1 b 1 2
2 c 1 4
3 x 2 5
4 d 2 3
5 d 2 10
6 e 3 5
7 a 3 5
But there must be a better way...
Is this what you want ?
df.groupby('key').tail(3)
Out[127]:
value key something
0 a 1 1
1 b 1 2
2 c 1 4
5 x 2 5
6 d 2 3
7 e 3 5
8 d 2 10
9 a 3 5
Does this help:
for k,v in df.groupby('key'):
print v[-2:]
value key something
1 b 1 2
2 c 1 4
value key something
6 d 2 3
8 d 2 10
value key something
7 e 3 5
9 a 3 5
Related
Given a dataframe like this:
df = pd.DataFrame([[11, 1, 1, 1, 1], [12, 2, 2, 2, 2], [12, 3, 3, 3, 3], [14, 4, 4, 4, 4]], columns=['a', 'b', 'c', 'd', 'e'])
a b c d e
0 11 1 1 1 1
1 12 2 2 2 2
2 12 3 3 3 3
3 14 4 4 4 4
I want to exchange values in column 'd' for rows having value == 12 in column a.
Final output should look like this:
a b c d e
0 11 1 1 1 1
1 12 2 2 3 2
2 12 3 3 2 3
3 14 4 4 4 4
How can I achieve this?
I have tried these:
df[df["a"] == 12]['d'] = df[df["a"] == 12]['d'].map({2: 3, 3: 2})
df.loc[df.a == 12]["d"].replace({2: 3, 3: 2}, inplace=True)
but these do not modify the original dataframe, though we can see changes in the series df[df["a"] == 12]['d'].map({2: 3, 3: 2}).
Can you show your code? If you use loc then, it changes the original one like below.
>>> df.loc[1, 'd'] = 3
>>> df
a b c d
0 1 1 1 1
1 2 2 2 3
2 3 3 3 3
3 4 4 4 4
Im trying to create top columns, which is the max of a couple of column rows. Pandas has a method nlargest but I cannot get it to work in rows. Pandas also has max and idxmax which does exactly what I want to do but only for the absolute max value.
df = pd.DataFrame(np.array([[1, 2, 3, 5, 1, 9], [4, 5, 6, 2, 5, 9], [7, 8, 9, 2, 5, 10]]), columns=['a', 'b', 'c', 'd', 'e', 'f'])
cols = df.columns[:-1].tolist()
df['max_1_val'] = df[cols].max(axis=1)
df['max_1_col'] = df[cols].idxmax(axis=1)
Output:
a b c d e f max_1_val max_1_col
0 1 2 3 5 1 9 5 d
1 4 5 6 2 5 9 6 c
2 7 8 9 2 5 10 9 c
But I am trying to get max_n_val and max_n_col so the expected output for top 3 would be:
a b c d e f max_1_val max_1_col max_2_val max_2_col max_3_val max_3_col
0 1 2 3 5 1 9 5 d 3 c 2 b
1 4 5 6 2 5 9 6 c 5 b 5 e
2 7 8 9 2 5 10 9 c 8 b 7 a
For improve performance is used numpy.argsort for positions, for correct order is used the last 3 items, reversed by indexing:
N = 3
a = df[cols].to_numpy().argsort()[:, :-N-1:-1]
print (a)
[[3 2 1]
[2 4 1]
[2 1 0]]
Then get columns names by indexing to c and for reordering values in d use this solution:
c = np.array(cols)[a]
d = df[cols].to_numpy()[np.arange(a.shape[0])[:, None], a]
Last create DataFrames, join by concat and reorder columns names by DataFrame.reindex:
df1 = pd.DataFrame(c).rename(columns=lambda x : f'max_{x+1}_col')
df2 = pd.DataFrame(d).rename(columns=lambda x : f'max_{x+1}_val')
c = df.columns.tolist() + [y for x in zip(df2.columns, df1.columns) for y in x]
df = pd.concat([df, df1, df2], axis=1).reindex(c, axis=1)
print (df)
a b c d e f max_1_val max_1_col max_2_val max_2_col max_3_val \
0 1 2 3 5 1 9 5 d 3 c 2
1 4 5 6 2 5 9 6 c 5 e 5
2 7 8 9 2 5 10 9 c 8 b 7
max_3_col
0 b
1 b
2 a
my dataframes:
df1 = pd.DataFrame(np.array([[1, 2, 3], [4, 2, 3], [7, 8, 8]]),columns=['a', 'b', 'c'])
df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 2, 3], [5, 8, 8]]),columns=['a', 'b', 'c'])
df1,df2:
a b c
0 1 2 3
1 4 2 3
2 7 8 8
a b c
0 1 2 3
1 4 2 3
2 5 8 8
I want to combine rows from columns a from both df's in all sequences but only where values in column b and c are equal.
Right now I have only solution for all in general with this code:
x = np.array(np.meshgrid(df1.a.values,
df2.a.values)).T.reshape(-1,2)
df = pd.DataFrame(x)
print(df)
0 1
0 1 1
1 1 4
2 1 5
3 4 1
4 4 4
5 4 5
6 7 1
7 7 4
8 7 5
expected output for df1.a and df2.a only for rows where df1.b==df2.b and df1.c==df2.c:
0 1
0 1 1
1 1 4
2 4 1
3 4 4
4 7 5
so basically i need to group by common rows in selected columns band c
You should try DataFrame.merge using inner merge:
df1.merge(df2, on=['b', 'c'])[['a_x', 'a_y']]
a_x a_y
0 1 1
1 1 4
2 4 1
3 4 4
4 7 5
I'm trying to create a new df column based on a condition to be validated in the all the rest of the columns per each row.
df = pd.DataFrame([[1, 5, 2, 8, 2], [2, 4, 4, 20, 5], [3, 3, 1, 20, 2], [4, 2, 2, 1, 0],
[5, 1, 4, -5, -4]],
columns=['a', 'b', 'c', 'd', 'e'],
index=[1, 2, 3, 4, 5])
I tried:
df['f'] = ""
df.loc[(df.any() >= 10), 'f'] = df['e'] + 10
However I get:
IndexingError: Unalignable boolean Series key provided
This is the desired output:
a b c d e f
1 1 5 2 8 2
2 2 4 4 20 5 15
3 3 3 1 20 2 12
4 4 2 2 1 0
5 5 1 4 -5 -4
Use
In [984]: df.loc[(df >= 10).any(1), 'f'] = df['e'] + 10
In [985]: df
Out[985]:
a b c d e f
1 1 5 2 8 2 NaN
2 2 4 4 20 5 15.0
3 3 3 1 20 2 12.0
4 4 2 2 1 0 NaN
5 5 1 4 -5 -4 NaN
Note that:
df.any()
a True
b True
c True
d True
e True
f True
dtype: bool
df.any() >= 10
a False
b False
c False
d False
e False
f False
dtype: bool
I assume you want to check if any value in a column is >= 10. That would be done with (df >= 10).any(axis=1).
You should be able to do this in one step, using np.where:
df['f'] = np.where((df >= 10).any(axis=1), df.e + 10, '')
df
a b c d e f
1 1 5 2 8 2
2 2 4 4 20 5 15
3 3 3 1 20 2 12
4 4 2 2 1 0
5 5 1 4 -5 -4
If you'd prefer NaNs instead of blanks, use:
df['f'] = np.where((df >= 10).any(axis=1), df.e + 10, np.nan)
df
a b c d e f
1 1 5 2 8 2 NaN
2 2 4 4 20 5 15.0
3 3 3 1 20 2 12.0
4 4 2 2 1 0 NaN
5 5 1 4 -5 -4 NaN
By using max
df['f'] = ""
df.loc[df.max(1)>=10,'f']=df.e+10
Out[330]:
a b c d e f
1 1 5 2 8 2
2 2 4 4 20 5 15
3 3 3 1 20 2 12
4 4 2 2 1 0
5 5 1 4 -5 -4
I have a pandas groupby object, from two keys.
gb = df.groupby(['A','B'])
How can I access a specific key say (2,4), how do I do it?
The group_by() method works well if there is only one key.
Any ideas?
I think you are looking for get_group:
In [1]: df = pd.DataFrame([[2, 4, 1], [2, 4, 2], [3, 4, 1]], columns=['A', 'B', 'C'])
In [2]: df
Out[2]:
A B C
0 2 4 1
1 2 4 2
2 3 4 1
In [3]: g = df.groupby(['A', 'B'])
In [4]: g.get_group((2,4))
Out[4]:
A B C
0 2 4 1
1 2 4 2
Use a tuple in get_group
In [49]: df = DataFrame(np.random.randint(10,size=15).reshape(5,3),columns=list('ABC'))
In [50]: df
Out[50]:
A B C
0 8 9 2
1 7 5 3
2 3 1 2
3 2 4 0
4 6 9 4
In [51]: df.groupby(['A','B']).sum()
Out[51]:
C
A B
2 4 0
3 1 2
6 9 4
7 5 3
8 9 2
In [52]: df.groupby(['A','B']).get_group((6,9))
Out[52]:
A B C
4 6 9 4