I have a df such as example below and I am looking to identify users who message the same text within a given time period, such as <= 60 minutes for the example:
user = [1,2,3,4,5,6]
text = ['hello','hello','whats up','not now','not now','hello']
times = ['2010-09-14 16:51:00','2010-09-14 15:59:00',
'2010-09-14 15:14:00','2010-09-14 14:55:00','2010-09-14 15:47:00','2010-09-14 15:29:00']
df = pd.DataFrame({'userid':user,'message':text,'time':times})
My current method groups the text by a list of users who messaged each text:
group = df.groupby('message')['userid'].apply(list)
Then I return all the possible combinations of userid's from each list as an array of pair values and then retrieve the userid-text for each instance as a key for retrieving the time of each message for each pair from the original df
This method works but I have been trying to find a better way of grouping the users of each different text conditionally based on whether the time between each instance is less than a specified period of time, say 60 minutes for this example, taken as the difference between the two messages from the users. So "hello" for users 1 and 2 is less than 60 minutes apart so pass the condition and be added to the list for "hello".
The expected output for the example would therefore be:
userid
"hello" [1,2,6]
"not not" [4,5]
I haven't found any exact or similar solutions so any help is really appreciated. It may be that my approach to the problem is wrong!
Not sure it's the most elegant solution - but here's one using group-by and rolling. The advantage of this method is that it can work for large sets of data. It doesn't create the full cartesian product of all the users and times who sent the same message.
res = []
def collect_users(x):
if len(x) > 1:
s = set(x)
if res and res[-1].issubset(s):
res.pop()
res.append(set(x))
return 0
df.groupby("message").rolling("3600s").agg(collect_users)
The result comes as a list of sets:
[{1.0, 2.0, 6.0}, {4.0, 5.0}]
One option is to use a groupby to find the next matching message chronologically, merge it to the original dataframe, and then filter to things where the message gap is < 1 hour:
In [402]: df2 = df.merge(df.sort_values("time").groupby("message").shift(), left_index=True, right_index=True, suffixes=["_source", "_target"])
In [403]: df2.loc[df2['time_source'].sub(df2['time_target']).lt("1h"), ["message", "userid_source", "userid_target"]].astype('O')
Out[403]:
message userid_source userid_target
0 hello 1 2
1 hello 2 6
4 not now 5 4
Note that in your current data, 2 and 6 messaged hello 30 minutes apart and also appears here.
Related
I have a certain business requirement for which I am having trouble in implementing a faster solution (current solution takes 3 hrs per iteration)
Eg: Say I have a df
and there's a list :
l = [[a,b,c],[d,e,f]]
To do:
Compare all the list values across customer and check if they exist or not
If they exist then find the corresponding min and max date1
Currently the pseudo working code I have is :
for each customer:
group by customer and add column having code column into a list
for each list value:
check if particular list value exists (in case check if [a,b,c] exists in first loop)
if exists:
check for min date by group etc
This multiple for loop is taking too long to execute since I have 100k+ customers.
Any way to further improve this? I already eliminated one for loop reducing time from 10hrs to 3
l = [['a','b','c'],['d','e','f']]
Firstly flatten your list:
from pandas.core.common import flatten
l=list(flatten(l))
Then do boolean masking to check if the customer exists or not in your dataframe:
newdf=df[df['code'].isin(l)]
Finally do groupby():
#The below code groupby 'code':
newdf=newdf.groupby('code').agg(max_date1=('date1','max'),min_date1=('date1','min'))
#If You want to groupby customerid and code then use:
newdf=newdf.groupby(['customerid','code']).agg(max_date1=('date1','max'),min_date1=('date1','min'))
Now If you print newdf you will get your desired output
I slightly modified my approach.
Instead of looping through each customer (I have 100k+ customers)
I looped through each list :
checked if customers were present or not and then looped through filtered customers
This reduced the time by a couple of hours.
Thanks again for your help
I have a dataframe with three columns
The first column has 3 unique values I used the below code to create unique dataframes, However I am unable to iterate over that dataframe and not sure how to use that to iterate.
df = pd.read_excel("input.xlsx")
unique_groups = list(df.iloc[:,0].unique()) ### lets assume Unique values are 0,1,2
mtlist = []
for index, value in enumerate(unique_groups):
globals()['df%s' % index] = df[df.iloc[:,0] == value]
mtlist.append('df%s' % index)
print(mtlist)
O/P
['df0', 'df1', 'df2']
for example lets say I want to find out the length of the first unique dataframe
if I manually type the name of the DF I get the correct output
len(df0)
O/P
35
But I am trying to automate the code so technically I want to find the length and itterate over that dataframe normally as i would by typing the name.
What I'm looking for is
if I try the below code
len('df%s' % 0)
I want to get the actual length of the dataframe instead of the length of the string.
Could someone please guide me how to do this?
I have also tried to create a Dictionary using the below code but I cant figure out how to iterate over the dictionary when the DF columns are more than two, where key would be the unique group and the value containes the two columns in same line.
df = pd.read_excel("input.xlsx")
unique_groups = list(df["Assignment Group"].unique())
length_of_unique_groups = len(unique_groups)
mtlist = []
df_dict = {name: df.loc[df['Assignment Group'] == name] for name in unique_groups}
Can someone please provide a better solution?
UPDATE
SAMPLE DATA
Assignment_group Description Document
Group A Text to be updated on the ticket 1 doc1.pdf
Group B Text to be updated on the ticket 2 doc2.pdf
Group A Text to be updated on the ticket 3 doc3.pdf
Group B Text to be updated on the ticket 4 doc4.pdf
Group A Text to be updated on the ticket 5 doc5.pdf
Group B Text to be updated on the ticket 6 doc6.pdf
Group C Text to be updated on the ticket 7 doc7.pdf
Group C Text to be updated on the ticket 8 doc8.pdf
Lets assume there are 100 rows of data
I'm trying to automate ServiceNow ticket creation with the above data.
So my end goal is GROUP A tickets should go to one group, however for each description an unique task has to be created, but we can club 10 task once and submit as one request so if I divide the df's into different df based on the Assignment_group it would be easier to iterate over(thats the only idea which i could think of)
For example lets say we have REQUEST001
within that request it will have multiple sub tasks such as STASK001,STASK002 ... STASK010.
hope this helps
Your problem is easily solved by groupby: one of the most useful tools in pandas. :
length_of_unique_groups = df.groupby('Assignment Group').size()
You can do all kind of operations (sum, count, std, etc) on your remaining columns, like getting the mean value of price for each group if that was a column.
I think you want to try something like len(eval('df%s' % 0))
I have a dataframe (df_data) with 14 columns for info over 1 month. I pulled out one week's data (df1) and made a list of all the account numbers there (accounts1)
What I am trying to do is take that accounts1 list and have it go through each value in the list, checking if it is counted more than once in df_data and if so, to save that account number to a new list for repeats only.
Then I want to take that repeats list and pull the 14 columns out of the original df_data so I can have all the rows of all 14 columns for each occurrence of that account number.
I'm getting stuck with the list of repeated account numbers, I used the following code, which seems to have worked to create a list with results
cnt = collections.Counter(accounts1)
repeats.append([k for k, v in cnt.items() if v > 1])
print((repeats).count)
but the amount of elements in that list is right under 3,000. When I used the .unique and checked the difference it should be a little over 5,000. What am I doing wrong? And how can I then use those elements to pull the columns from the original dataframe?
Basically say I had
accounts1 = df1['accntnum'] = [0,1,2,5,8,2,5,0,0,7]
I would want it to cycle through and pull out each repeat from df_data and return a list of them like
repeats = [0, 2, 5, 7]
(There are numbers in the monthly df_data that are in weekly df1 but may not be repeated there yet)
Then I'd like to use that list to pull from df_data['accntnum'], thinking something like
df_repeats = df_data[df_data['accntnum'] isin repeats]]
Oh also, I'm really only interested in the first occurrence of a repeat. There is a date and time column that can help sort those out in the end though. Thank you in advance!
I am trying to make a list of the number of elements in each group in a pandas series. In my dataframe i have column called ID, and all values occur multiple times. I want to make a list containing the frequency of each element in the order by which they occur.
So an example of the column ID is [1,2,3,3,3,2,1,5,2,3,1,2,4,3]
this should produce [3,4,5,1,1] since the group-ID 1 occurs 3 times, the group-ID 2 occurs 4 times etc. I have made a code that does this perfectly:
group_list = df.ID.unique().tolist()
group_size = []
for i in group_list:
group_size.append(df.ID.value_counts()[i])
The problem is that it takes way to long to finish. I have 5 million rows, and i let it run for 50 minutes, and it still didn't finish! I tried running it for the first 30-50 rows and it works as intended.
To me it would be logical to simply use value_counts(sort=False) but it doesn't give me the group-ID frequencies in the order they occur in my series. I also tried implementing extend because i read it should be faster, but I get a "numpy.int64 object is not iterable".
Given a Series
ser = pd.Series([1,2,3,3,3,2,1,5,2,3,1,2,4,3])
You can do the following:
ser.value_counts().reindex(ser.unique()).tolist()
Out: [3, 4, 5, 1, 1]
Reindex will reorder the value_counts items based on the order they appear.
I have a file with a column containing IDs. Usually, an ID appears only once, but occasionally, they're associated with multiple records. I want to count how many times a given ID appeared, and then split into two separate dfs so I can run different operations on both. One df should be where IDs only appear once, and one should be where IDs appear multiple times.
I was able to successfully count the number of instances an ID appeared by grouping on ID and joining the counts back onto the original df, like so:
newdf = df.join(df.groupBy('ID').count(),on='ID')
This works nicely, as I get an output like so:
ID Thing count
287099 Foo 3
287099 Bar 3
287099 Foobar 3
321244 Barbar 1
333032 Barfoo 2
333032 Foofoo 2
But, now I want to split the df so that I have a df where count = 1, and count > 1. The below and variations thereof didn't work, however:
singular = df2.filter(df2.count == 1)
I get a 'TypeError: condition should be string or Column' error instead. When I tried displaying the type of the column, it says the count column is an instance. How can I get PySpark to treat the count column the way I need it to?
count is a method of dataframe,
>>> df2.count
<bound method DataFrame.count of DataFrame[id: bigint, count: bigint]>
Where as filter needs a column to operate on, change it as below,
singular = df2.filter(df2['count'] == 1)