We Keep Coding

Add a column based on frequency for each group

I have a dataframe like this:
vehicle_id trip
0 0 0
1 0 0
2 0 0
3 0 1
4 0 1
5 1 0
6 1 0
7 1 1
8 1 1
9 1 1
10 1 1
11 1 1
12 1 2
13 2 0
14 2 1
15 2 2
I want to add a column that counts the frequency of each trip value for each 'vehicle id' group and drop the rows where the frequency is equal to 'one'. So after adding the column the frequency will be like this:
vehicle_id trip frequency
0 0 0 3
1 0 0 3
2 0 0 3
3 0 1 2
4 0 1 2
5 1 0 2
6 1 0 2
7 1 1 5
8 1 1 5
9 1 1 5
10 1 1 5
11 1 1 5
12 1 2 1
13 2 0 1
14 2 1 1
15 2 2 1
and the final result will be like this
vehicle_id trip frequency
0 0 0 3
1 0 0 3
2 0 0 3
3 0 1 2
4 0 1 2
5 1 0 2
6 1 0 2
7 1 1 5
8 1 1 5
9 1 1 5
10 1 1 5
11 1 1 5
what is the best solution for that? Also, what should I do if I intend to directly drop rows where the frequency is equal to 1 in each group (without adding the frequency column)?

Check the collab here :
https://colab.research.google.com/drive/1AuBTuW7vWj1FbJzhPuE-QoLncoF5W_7W?usp=sharing
You can use df.groupby() :
df["frequency"] = df.groupby(["vehicle_id","trip"]).transform("count")
But of course you need to create the frequency column before_hand :
df["frequency"] = 0
If I take your dataframe as example this gives :
import pandas as pd
dict = {"vehicle_id" : [0,0,0,0,0,1,1,1,1,1,1,1],
"trip" : [0,0,0,1,1,0,0,1,1,1,1,1]}
df = pd.DataFrame.from_dict(dict)
df["frequency"] = 0
df["frequency"] = df.groupby(["vehicle_id","trip"]).transform("count")
output :

Try:
df["frequency"] = (
df.assign(frequency=0).groupby(["vehicle_id", "trip"]).transform("count")
)
print(df[df.frequency > 1])
Prints:
vehicle_id trip frequency
0 0 0 3
1 0 0 3
2 0 0 3
3 0 1 2
4 0 1 2
5 1 0 2
6 1 0 2
7 1 1 5
8 1 1 5
9 1 1 5
10 1 1 5
11 1 1 5

Python: create new column conditionally on values from two other columns

I would like to combine two columns in a new column.
Lets suppose I have:
Index A B
0 1 0
1 1 0
2 1 0
3 1 0
4 1 0
5 1 2
6 1 2
7 1 2
8 1 2
9 1 2
10 1 2
Now I would like to create a column C with the entries from A from Index 0 to 4 and from column B from Index 5 to 10. It should look like this:
Index A B C
0 1 0 1
1 1 0 1
2 1 0 1
3 1 0 1
4 1 0 1
5 1 2 2
6 1 2 2
7 1 2 2
8 1 2 2
9 1 2 2
10 1 2 2
Is there a python code how I can get this? Thanks in advance!

If Index is an actual column you can use numpy.where and specify your condition
import numpy as np
df['C'] = np.where(df['Index'] <= 4, df['A'], df['B'])
Index A B C
0 0 1 0 1
1 1 1 0 1
2 2 1 0 1
3 3 1 0 1
4 4 1 0 1
5 5 1 2 2
6 6 1 2 2
7 7 1 2 2
8 8 1 2 2
9 9 1 2 2
10 10 1 2 2

if your index is your actual index
you can slice your indices with iloc and create your column with concat.
df['C'] = pd.concat([df['A'].iloc[:5], df['B'].iloc[5:]])
print(df)
A B C
0 1 0 1
1 1 0 1
2 1 0 1
3 1 0 1
4 1 0 1
5 1 2 2
6 1 2 2
7 1 2 2
8 1 2 2
9 1 2 2
10 1 2 2

group consecutive equal values and count up

The df:
a b
0 1
0 3
0 3
0 1
1 1
1 2
1 4
I would like to group by a and count up the equal consecutive rows in a group:
a b c
0 1 1
0 3 2
0 3 2
0 1 3
1 1 1
1 2 2
1 4 3
I tried:
df['c'] = df.b.groupby([df.a, df.b.diff().ne(0).cumsum()])
which gave me a type error:
Length of values does not match length of index

In your case , that is factor
s=df.b.diff().ne(0).cumsum().groupby(df.a).transform(lambda x : x.factorize()[0])+1
Out[276]:
0 1
1 2
2 2
3 3
4 1
5 2
6 3
Name: b, dtype: int32
df['c']=s
Or
df.b.groupby(df.a).apply(lambda x : x.diff().ne(0).cumsum())
Out[277]:
0 1
1 2
2 2
3 3
4 1
5 2
6 3
Name: b, dtype: int32

Another approach:
s = df.ne(df.shift()).any(1).astype(int)
df['c'] = s.groupby(df['a']).cumsum()
Output:
a b c
0 0 1 1
1 0 3 2
2 0 3 2
3 0 1 3
4 1 1 1
5 1 2 2
6 1 4 3

Pandas - updating sequence of values

I have this Sample DataFrame:
pd.DataFrame(data={1:[0,3,4,1], 2:[4,1,0,0], 3:[0,0,1,2], 4:[1,2,3,4] })
1 2 3 4
0 0 4 0 1
1 3 1 0 2
2 4 0 1 3
3 1 0 2 4
But i want to convert it to the format below:
pd.DataFrame(data={1:[1,1,1,1], 2:[0,2,0,2], 3:[0,3,3,0], 4:[4,0,4,4] })
1 2 3 4
0 1 0 0 4
1 1 2 3 0
2 1 0 3 4
3 1 2 0 4
Is there any way or a function to do this as i have more than 100,000 rows so for loops, dictionaries, lists won't work.

My entry:
data = df.reset_index().melt("index").query("value > 0")
out = data.pivot("index", "value", "value").fillna(0).astype(int)
giving
In [273]: out
Out[273]:
value 1 2 3 4
index
0 1 0 0 4
1 1 2 3 0
2 1 0 3 4
3 1 2 0 4
Unfortunately you'd have to clear the index and column names if you want to get rid of them, using either df.index.name = df.columns.name = None or df.rename_axis(None).rename_axis(None, 1) or something.

Using get_dummies:
s = pd.get_dummies(df, columns=df.columns, prefix_sep='', prefix='')
out = s.groupby(s.columns, axis=1).sum().drop('0', 1)
out.mask(out.ne(0)).fillna(dict(zip(out.columns, out.columns))).astype(int)
1 2 3 4
0 1 0 0 4
1 1 2 3 0
2 1 0 3 4
3 1 2 0 4

Using zip and np.isin
pd.DataFrame([ np.isin(y, x)*df.columns.values for x , y in zip([df.columns.values]*len(df),df.values)])
Out[900]:
0 1 2 3
0 0 2 0 4
1 1 2 0 4
2 1 0 3 4
3 1 0 3 4

Calculate changes of column in Pandas

I have a dataframe with data and I want calculate changes of values during time.
UserId DateTime Value
1 1 0
1 2 0
1 3 0
1 4 1
1 6 1
1 7 1
2 1 0
2 2 1
2 3 1
2 4 0
2 6 1
2 7 1
So after script execution I want to get a column with change identifier (for user and date). Only changes from 0 to 1 is interesting.
UserId DateTime Value IsChanged
1 1 0 0
1 2 0 0
1 3 0 0
1 4 1 1 <- Value was changed from 0 to 1
1 6 1 0
1 7 1 0
2 1 0 0
2 2 1 1 <- Value was changed from 0 to 1
2 3 1 0
2 4 0 0 <- Change from 1 to 0 not interesting
2 6 1 1 <- Value was changed from 0 to 1 for the user
2 7 1 0

What about this?
# df is your dataframe
df['IsChanged'] = (df['Value'].diff()==1).astype(int)
The only case you care about is Value being 0 before and 1 after, so you can simply calculate the change in value and check if it is equal to 1.
UserId DateTime Value IsChanged
0 1 1 0 0
1 1 2 0 0
2 1 3 0 0
3 1 4 1 1
4 1 6 1 0
5 1 7 1 0
6 2 1 0 0
7 2 2 1 1
8 2 3 1 0
9 2 4 0 0
10 2 6 1 1
11 2 7 1 0