We Keep Coding

How can I introduce a new column which does not count in every line?

I have following dataframe about auctions:
id.product_permutation
id.iteration
property.product
property.price
1
1
1
200
1
2
1
300
1
3
1
400
1
4
3
100
1
5
3
200
1
6
3
300
1
7
2
500
1
8
2
600
2
1
3
300
2
2
3
400
2
3
1
200
2
4
1
300
2
5
2
700
2
6
2
800
2
7
2
900
2
8
2
700
3
1
1
200
...
...
...
...
There are 3 different products in the auction and the column property.product tells which product is for sale at the moment. If the product number in property.product changes then the product is sold.
property.price tells what the price is at the moment.
If the number in id.product_permutation changes then the whole auction is over, all 3 items are sold and a new auction starts (with the same 3 items).
Now I would like to introduce a new column amount_of_sold_items which counts how many products were already sold (like in the following). I tried a lot, but unfortunately I do not get the desired result. Can anyone help me please to solve this issue?
id.product_permutation
id.iteration
property.product
property.price
amount_of_sold_items
1
1
1
200
0
1
2
1
300
0
1
3
1
400
0
1
4
3
100
1
1
5
3
200
1
1
6
3
300
1
1
7
2
500
2
1
8
2
600
2
1
NaN
NaN
NaN
3
2
1
3
300
0
2
2
3
400
0
2
3
1
200
1
2
4
1
300
1
2
5
2
700
2
2
6
2
800
2
2
7
2
900
2
2
8
2
700
2
2
NaN
NaN
NaN
3
3
1
1
200
0
...
...
...
...
...

df["n_items_sold"] = (df.groupby("id.product_permutation")["property.product"]
.transform(lambda x: x.diff().ne(0, fill_value=0).cumsum()))
For each id.product_permutation group, we assign a new series that looks at the turning points via difference not being equal to 0 (fill_value=0 is there to prevent counting the very first one as a turning point). Cumulative sum of these turning points keeps track of the items sold thus far.
This gives:
id.product_permutation id.iteration property.product property.price n_items_sold
0 1 1 1 200 0
1 1 2 1 300 0
2 1 3 1 400 0
3 1 4 3 100 1
4 1 5 3 200 1
5 1 6 3 300 1
6 1 7 2 500 2
7 1 8 2 600 2
8 2 1 3 300 0
9 2 2 3 400 0
10 2 3 1 200 1
11 2 4 1 300 1
12 2 5 2 700 2
13 2 6 2 800 2
14 2 7 2 900 2
15 2 8 2 700 2
16 3 1 1 200 0
To put [id_prod_perm, NaN, NaN, NaN, 3] rows at the end of each id.product_permuation, we can detect the changing points of id.product_permuation and insert columns to the transposed frame which, in effect, inserts rows to the original one when transposed:
# following is [8, 16] for the above example
changing_points = np.where(df["id.product_permutation"]
.diff().ne(0, fill_value=0))[0].tolist()
# insert to transpose and then come back
df = df.T
offset = 0 # helper for insertion location
for j, point in enumerate(changing_points, start=1):
# to the given point, insert a column with the same name
df.insert(loc=point+offset, column=point, value=[j, *[np.nan]*3, 3],
allow_duplicates=True)
# since an insertion enlarges the frame, old changing points
# need to increase, this is handled by the `offset`
offset += 1
# go back to original form, and also reset the index to 0..N-1
df = df.T.reset_index(drop=True)
to get
>>> df
id.product_permutation id.iteration property.product property.price n_items_sold
0 1.0 1.0 1.0 200.0 0.0
1 1.0 2.0 1.0 300.0 0.0
2 1.0 3.0 1.0 400.0 0.0
3 1.0 4.0 3.0 100.0 1.0
4 1.0 5.0 3.0 200.0 1.0
5 1.0 6.0 3.0 300.0 1.0
6 1.0 7.0 2.0 500.0 2.0
7 1.0 8.0 2.0 600.0 2.0
8 1.0 NaN NaN NaN 3.0
9 2.0 1.0 3.0 300.0 0.0
10 2.0 2.0 3.0 400.0 0.0
11 2.0 3.0 1.0 200.0 1.0
12 2.0 4.0 1.0 300.0 1.0
13 2.0 5.0 2.0 700.0 2.0
14 2.0 6.0 2.0 800.0 2.0
15 2.0 7.0 2.0 900.0 2.0
16 2.0 8.0 2.0 700.0 2.0
17 2.0 NaN NaN NaN 3.0
18 3.0 1.0 1.0 200.0 0.0

How to vectorize a non-overlapped dataframe to overlapped shiftting dataframe?

I would like to transform a regular dataframe to a multi-index dataframe with overlap and shift.
For example, the input dataframe is like this sample code:
import pandas as pd
import numpy as np
df = pd.DataFrame(data=np.arange(0, 12).reshape(-1, 2), columns=['d1', 'd2'], dtype=float)
df.index.name = 'idx'
print(df)
Output:
d1 d2
idx
0 0.0 1.0
1 2.0 3.0
2 4.0 5.0
3 6.0 7.0
4 8.0 9.0
5 10.0 11.0
What I want to output is: Make it overlap by batch and shift one row per time (Add a column batchid to label every shift), like this (batchsize=4):
d1 d2
idx batchid
0 0 0.0 1.0
1 0 2.0 3.0
2 0 4.0 5.0
3 0 6.0 7.0
1 1 2.0 3.0
2 1 4.0 5.0
3 1 6.0 7.0
4 1 8.0 9.0
2 2 4.0 5.0
3 2 6.0 7.0
4 2 8.0 9.0
5 2 10.0 11.0
My work so far:
I can make it work with iterations and concat them together. But it will take a lot of time.
batchsize = 4
ds, ids = [], []
idx = df.index.values
for bi in range(int(len(df) - batchsize + 1)):
ids.append(idx[bi:bi+batchsize])
for k, idx in enumerate(ids):
di = df.loc[pd.IndexSlice[idx], :].copy()
di['batchid'] = k
ds.append(di)
res = pd.concat(ds).fillna(0)
res.set_index('batchid', inplace=True, append=True)
Is there a way to vectorize and accelerate this process?
Thanks.

First we create a 'mask' that will tell us which elements go into which batch id
nrows = len(df)
batchsize = 4
mask_columns = {i:np.pad([1]*batchsize,(i,nrows-batchsize-i)) for i in range(nrows-batchsize+1)}
mask_df = pd.DataFrame(mask_columns)
df = df.join(mask_df)
this adds a few columns to df:
idx d1 d2 0 1 2
----- ---- ---- --- --- ---
0 0 1 1 0 0
1 2 3 1 1 0
2 4 5 1 1 1
3 6 7 1 1 1
4 8 9 0 1 1
5 10 11 0 0 1
This now looks like a df with 'dummies', and we need to 'reverse' the dummies:
df2 = df.set_index(['d1','d2'], drop=True)
df2[df2==1].stack().reset_index().drop(0,1).sort_values('level_2').rename(columns = {'level_2':'batchid'})
produces
d1 d2 batchid
-- ---- ---- ---------
0 0 1 0
1 2 3 0
3 4 5 0
6 6 7 0
2 2 3 1
4 4 5 1
7 6 7 1
9 8 9 1
5 4 5 2
8 6 7 2
10 8 9 2
11 10 11 2

You can accomplish this with list comprehension inside of a pd.concat with iloc using i as a variable that iterates through a range. This should be quicker:
batchsize = 4
df = (pd.concat([df.iloc[i:batchsize+i].assign(batchid=i)
for i in range(df.shape[0] - batchsize + 1)])
.set_index(['batchid'], append=True))
df
Out[1]:
d1 d2
idx batchid
0 0 0.0 1.0
1 0 2.0 3.0
2 0 4.0 5.0
3 0 6.0 7.0
1 1 2.0 3.0
2 1 4.0 5.0
3 1 6.0 7.0
4 1 8.0 9.0
2 2 4.0 5.0
3 2 6.0 7.0
4 2 8.0 9.0
5 2 10.0 11.0

How to group by data using one column perform some operation on another column and assign new groups pandas

I have a dataframe as below :
distance_along_path ID
0 0 1
1 2.2 1
2 4.5 1
3 7.0 1
4 0 2
5 0 3
6 3.0 2
7 5.0 3
8 0 4
9 2.0 4
10 5.0 4
11 0 5
12 3.0 5
11 7.0 4
12
I want be able to group these by id's first and the by distance_along_path values, every time a 0 is seen in distance along path for the id, new group is created and until the next 0 all these rows are under A group as indicated below
distance_along_path ID group
0 0 1 1
1 2.2 1 1
2 4.5 1 1
3 7.0 1 1
4 0 1 2
5 0 2 3
6 3.0 1 2
7 5.0 2 3
8 0 2 4
9 2.0 2 4
10 5.0 2 4
11 0 1 5
12 3.0 1 5
13 7.0 1 5
14 0 1 6
15 0 2 7
16 3.0 1 6
17 5.0 2 7
18 1.0 2 7
Thank you

try the following:
grp_id = df.groupby(['ID']).id.count().reset_index()
grp_distance = grp_id.groupby(['distance_along_path'].grp_id['distance_along_path']==0

Pandas: Drop duplicates based on row value

I have a dataframe and I want to drop duplicates based on different conditions....
A B
0 1 1.0
1 1 1.0
2 2 2.0
3 2 2.0
4 3 3.0
5 4 4.0
6 5 5.0
7 - 5.1
8 - 5.1
9 - 5.3
I want to drop all the duplicates from column A except rows with "-". After this, I want to drop duplicates from column A with "-" as a value based on their column B value. Given the input dataframe, this should return the following:-
A B
0 1 1.0
2 2 2.0
4 3 3.0
5 4 4.0
6 5 5.0
7 - 5.1
9 - 5.3
I have the following code but it's not very efficient for very large amounts of data, how can I improve this....
def generate(df):
str_col = df[df["A"] == "-"]
df.drop(df[df["A"] == "-"].index, inplace=True)
df = df.drop_duplicates(subset="A")
str_col = b.drop_duplicates(subset="B")
bigdata = df.append(str_col, ignore_index=True)
return bigdata.sort_values("B")

duplicated and eq:
df[~df.duplicated('A') # keep those not duplicates in A
| (df['A'].eq('-') # or those '-' in A
& ~df['B'].duplicated())] # which are not duplicates in B
Output:
A B
0 1 1.0
2 2 2.0
4 3 3.0
5 4 4.0
6 5 5.0
7 - 5.1
9 - 5.3

df.drop_duplicates(subset=['A', 'B'])
Given a full set of data:
A B C
0 1 1.0 0
1 1 1.0 1
2 2 2.0 2
3 2 2.0 3
4 3 3.0 4
5 4 4.0 5
6 5 5.0 6
7 - 5.1 7
8 - 5.1 8
9 - 5.3 9
Result:
A B C
0 1 1.0 0
2 2 2.0 2
4 3 3.0 4
5 4 4.0 5
6 5 5.0 6
7 - 5.1 7
9 - 5.3 9

groupby + head
df.groupby(['A','B']).head(1)
Out[7]:
A B
0 1 1.0
2 2 2.0
4 3 3.0
5 4 4.0
6 5 5.0
7 - 5.1
9 - 5.3

How to restrict the area for operation in python pandas dataframes?

I`d like to qualify my dropna option within the first 3 rows of the dataframe. The original dataframe is:
A C
0 0.0 0
1 NaN 1
2 2.0 2
3 3.0 3
4 NaN 4
5 5.0 5
6 6.0 6
And I would love to see:
A C
0 0.0 0
2 2.0 2
3 3.0 3
4 NaN 4
5 5.0 5
6 6.0 6
With only the row indexed 1 removed. Is it possible to make it within just one line of code?
Thanks!

You could use
In [594]: df[df.notnull().all(1) | (df.index > 3)]
Out[594]:
A C
0 0.0 0
2 2.0 2
3 3.0 3
4 NaN 4
5 5.0 5
6 6.0 6