I have a dataframe of 10,000 rows that I am trying to sum all possible combinations of those rows. According to my math, that's about 50 million combinations. I'll give a small example to simplify what my data looks like:
df = Ratio Count Score
1 6 11
2 7 12
3 8 13
4 9 14
5 10 15
And here's the desired result:
results = Min Ratio Max Ratio Total Count Total Score
1 2 13 23
1 3 21 36
1 4 30 50
1 5 40 65
2 3 15 25
2 4 24 39
2 5 34 54
3 4 17 27
3 5 27 42
4 5 19 29
This is the code that I came up with to complete the calculation:
for i in range(len(df)):
j = i + 1
while j <= len(df):
range_to_calc = df.iloc[i:j]
total_count = range_to_calc['Count'].sum()
total_score = range_to_calc['Score'].sum()
new_row = {'Min Ratio': range_to_calc.at[range_to_calc.first_valid_index(),'Ratio'],
'Max Ratio': range_to_calc.at[range_to_calc.last_valid_index(),'Ratio'],
'Total Count': total_count,
'Total Score': total_score}
results = results.append(new_row, ignore_index=True)
j = j + 1
This code works, but according to my estimates after running it for a few minutes, it would take 200 hours to complete. I understand that using numpy would be a lot faster, but I can't wrap my head around how to build multiple arrays to add together. (I think it would be easy if I was doing just 1+2, 2+3, 3+4, etc., but it's a lot harder because I need 1+2, 1+2+3, 1+2+3+4, etc.) Is there a more efficient way to complete this calculation so it can run in a reasonable amount of time? Thank you!
P.S.: If you're wondering what I want to do with a 50 million-row dataframe, I don't actually need that in my final results. I'm ultimately looking to divide the Total Score of each row in the results by its Total Count to get a Total Score Per Total Count value, and then display the 1,000 highest Total Scores Per Total Count, along with each associated Min Ratio, Max Ratio, Total Count, and Total Score.
After these improvements it takes ~2 minutes to run for 10k rows.
For the sum computation, you can pre-compute cumulative sum(cumsum) and save it. sum(i to j) is equal to sum(0 to j) - sum(0 to i-1).
Now sum(0 to j) is cumsum[j] and sum(0 to i - 1) is cumsum[i-1].
So sum(i to j) = cumsum[j] - cumsum[i - 1].
This gives significant improvement over computing sum each time for different combination.
Operation over numpy array is faster than the operation on pandas series, hence convert every colum to numpy array and then do the computation over it.
(From other answers): Instead of appending in list, initialise an empty numpy array of size ((n*(n+1)//2) -n , 4) and use it to save the results.
Use:
count_cumsum = np.cumsum(df.Count.values)
score_cumsum = np.cumsum(df.Score.values)
ratios = df.Ratio.values
n = len(df)
rowInCombination = (n * (n + 1) // 2) - n
arr = np.empty(shape = (rowInCombination, 4), dtype = int)
k = 0
for i in range(len(df)):
for j in range(i + 1, len(df)):
arr[k, :] = ([
count_cumsum[j] - count_cumsum[i-1] if i > 0 else count_cumsum[j],
score_cumsum[j] - score_cumsum[i-1] if i > 0 else score_cumsum[j],
ratios[i],
ratios[j]])
k = k + 1
out = pd.DataFrame(arr, columns = ['Total_Count', 'Total_Score',
'Min_Ratio', 'Max_Ratio'])
Input:
df = pd.DataFrame({'Ratio': [1, 2, 3, 4, 5],
'Count': [6, 7, 8, 9, 10],
'Score': [11, 12, 13, 14, 15]})
Output:
>>>out
Min_Ratio Max_Ratio Total_Count Total_Score
0 1 2 13 23
1 1 3 21 36
2 1 4 30 50
3 1 5 40 65
4 2 3 15 25
5 2 4 24 39
6 2 5 34 54
7 3 4 17 27
8 3 5 27 42
9 4 5 19 29
First of all, you can improve the algorithm. Then, you can speed up the computation using Numpy vectorization/broadcasting.
Here are the interesting point to improve the performance of the algorithm:
append of Pandas is slow because it recreate a new dataframe. You should never use it in a costly loop. Instead, you can append the lines to a Python list or even directly write the items in a pre-allocated Numpy vector.
computing partial sums take an O(n) time while you can pre-compute the cumulative sums and then just find the partial sum in constant time.
CPython loops are very slow, but the inner loop can be vectorized using Numpy thanks to broadcasting.
Here is the resulting code:
import numpy as np
import pandas as pd
def fastImpl(df):
n = len(df)
resRowCount = (n * (n+1)) // 2
k = 0
cumCounts = np.concatenate(([0], df['Count'].astype(int).cumsum()))
cumScores = np.concatenate(([0], df['Score'].astype(int).cumsum()))
ratios = df['Ratio'].astype(int)
minRatio = np.empty(resRowCount, dtype=int)
maxRatio = np.empty(resRowCount, dtype=int)
count = np.empty(resRowCount, dtype=int)
score = np.empty(resRowCount, dtype=int)
for i in range(n):
kStart, kEnd = k, k+(n-i)
jStart, jEnd = i+1, n+1
minRatio[kStart:kEnd] = ratios[i]
maxRatio[kStart:kEnd] = ratios[i:n]
count[kStart:kEnd] = cumCounts[jStart:jEnd] - cumCounts[i]
score[kStart:kEnd] = cumScores[jStart:jEnd] - cumScores[i]
k = kEnd
assert k == resRowCount
return pd.DataFrame({
'Min Ratio': minRatio,
'Max Ratio': maxRatio,
'Total Count': count,
'Total Score': score
})
Note that this code give the same results than the code in your question, but the original code does not give the expected results stated in the question. Note also that since inputs are integers, I forced Numpy to use integers for sake of performance (despite the algorithm should work with floats too).
This code is hundreds of thousand times faster than the original code on big dataframes and it succeeds to compute a dataframe of 10,000 rows in 0.7 second.
Others have explained why your algorithm was so slow so I will dive into that.
Let's take a different approach to your problem. In particular, look at how the Total Count and Total Score columns are calculated:
Calculate the cumulative sum for every row from 1 to n
Calculate the cumulative sum for every row from 2 to n
...
Calculate the cumulative sum for every row from n to n
Since cumulative sums are accumulative, we only need to calculate it once for row 1 to row n:
The cumsum of (2 to n) is the cumsum of (1 to n) - (row 1)
The cumsum of (3 to n) is the cumsum of (2 to n) - (row 2)
And so on...
In other words, the current cumsum is the previous cumsum minus its first row, then dropping the first row.
As you have theorized, pandas is a lot slower than numpy so we will convert everthing into numpy for speed:
arr = df[['Ratio', 'Count', 'Score']].to_numpy() # Convert to numpy array
tmp = np.cumsum(arr[:, 1:3], axis=0) # calculate cumsum for row 1 to n
tmp = np.insert(tmp, 0, arr[0, 0], axis=1) # create the Min Ratio column
tmp = np.insert(tmp, 1, arr[:, 0], axis=1) # create the Max Ratio column
results2 = [tmp]
for i in range(1, len(arr)):
tmp = results2[-1][1:] # current cumsum is the previous cumsum without the first row
diff = results2[-1][0] # the previous cumsum's first row
tmp -= diff # adjust the current cumsum
tmp[:, 0] = arr[i, 0] # new Min Ratio
tmp[:, 1] = arr[i:, 0] # new Max Ratio
results2.append(tmp)
# Assemble the result
results2 = np.concatenate(results2).reshape(-1,4)
results2 = pd.DataFrame(results2, columns=['Min Ratio', 'Max Ratio', 'Total Count', 'Total Score'])
During my test, this produces the results for a 10k row data frame in about 2 seconds.
Sorry to write late for this topic, but I'm just looking for a solution for a similar topic. The solution for this issue is simple because the combination is only in pairs. This is solved by uploading the dataframe to any DB and executing the following query whose duration is less than 10 seconds:
SEL f1.*,f2.*,f1.score+f2.score
FROM table_with_data_source f1, table_with_data_source f2
where f1.ratio<>f2.ratio;
The database will do it very fast even if there are 100,000 records or more.
However, none of the algorithms that I saw in the answers, actually perform a combinatorial of values. He only does it in pairs. The problem really gets complicated when it's a true combinatorial, for example:
Given: a, b, c, d and e as records:
a
b
c
d
e
The real combination would be:
a+b
a+c
a+d
a+e
a+b+c
a+b+d
a+b+e
a+c+d
a+c+e
a+d+e
a+b+c+d
a+b+c+e
a+c+d+e
a+b+c+d+e
b+c
b+d
b+e
b+c+d
b+c+e
b+d+e
c+d
c+e
c+d+e
d+e
This is a real combinatorial, which covers all possible combinations. For this case I have not been able to find a suitable solution since it really affects the performance of any HW. Anyone have any idea how to perform a real combinatorial using python? At the database level it affects the general performance of the database.
Related
I have the following dataframe
product_id
weight_in_g
1
50
2
120
3
130
4
200
5
42
6
90
I am trying to match products based on weights within a deviation of 50 using this loop
list1=[]
for row in df[['product_id', 'weight_in_g']].itertuples():
high = row[1] + 50
low = row[1] - 50
id = df['product_id'].loc[((df['weight_in_g'] >= low) & (df['weight_in_g'] <= high)) | (df['weight_in_g'] == 0)]
list1.append(id)
df['weight_matches'] = list1
del list1
Which gives me the output:
product_id
weight_in_g
weight_matches
1
50
1, 5, 6
2
120
2, 3, 6
3
130
3, 2, 6
4
200
4, 6
5
42
5, 6, 1
6
90
6, 1
I'm using this as a filter together with text embedding. So for that reason i'm including all values with "0" which is about 35% of the dataset (I'd rather keep the values instead of not matching 35% of my dataset)
This works with 10.000 and 20.000 rows, but if i'm going higher then that my notebook runs out of memory (13gb ram)
Is there any way i can make this more memory efficient?
You are using memory very inefficiently. The id variable is of type pd.Series, which stores 2 things: the indexes and the values, plus some small overhead. pandas defaults to int64 for integers so it takes 128 bits / 16 bytes to store a single product ID.
Unrelated to memory, you are also using loops, which are slow. You can have it run faster by using numpy broadcasting to take advantage of vectorization.
# If your product ID can fit into int32, convert it to conserve memory
product_ids = df["product_id"].to_numpy(dtype="int32")
# Convert columns to numpy arrays
weight = df["weight_in_g"].to_numpy()
weight_high = weight + 50
weight_low = weight - 50
# Raise weight by one dimension to prepare for broadcasting
weight = weight[:, None]
# A naive implementation of numpy broadcasting will require n*n bytes for the
# `mask` array. When n = 50_000, the memory demand is *at least* 2.5GB, which my
# weak PC can't afford. Chunking in a trade-off: bigger chunk size results in
# faster performance but demand more memory. Pick a size that best suits you.
chunk_size = 5000
weight_matches = []
for i in np.arange(0, len(weight), chunk_size):
chunk = weight[i : i + chunk_size]
mask = (chunk == 0) | ((weight_low <= chunk) & (chunk <= weight_high))
weight_matches += [product_ids[np.nonzero(m)] for m in mask]
df["weight_matches"] = weight_matches
The above code took 8.7s to run 50k rows. I was able to tweak it to run on 100k rows. However, keep in mind that every solution has a limit. If your requirement exceeds that limit, you need to find a new solution.
Here is one way to do it
# do a self join
# query where difference is +/- 50 in weights
# groupby and accumulate the matching products as list
# finally reset and rename the column
(df.merge(df , how='cross', suffixes=(None, '_y'))
.query('abs(weight_in_g - weight_in_g_y) <= 50')
.groupby(['product_id','weight_in_g'])['product_id_y'].agg(list)
).reset_index()
.rename(columns={'product_id_y':'weight_matches'})
product_id weight_in_g weight_matches
0 1 50 [1, 5, 6]
1 2 120 [2, 3, 6]
2 3 130 [2, 3, 6]
3 4 200 [4]
4 5 42 [1, 5, 6]
5 6 90 [1, 2, 3, 5, 6]
Below is a dataframe showing coordinate values from and to, each row having a corresponding value column.
I want to find the range of coordinates where the value column doesn't exceed 5. Below is the dataframe input.
import pandas as pd
From=[10,20,30,40,50,60,70]
to=[20,30,40,50,60,70,80]
value=[2,3,5,6,1,3,1]
df=pd.DataFrame({'from':From, 'to':to, 'value':value})
print(df)
hence I want to convert the following table:
to the following outcome:
Further explanation:
Coordinates from 10 to 30 are joined and the value column changed to 5
as its sum of values from 10 to 30 (not exceeding 5)
Coordinates 30 to 40 equals 5
Coordinate 40 to 50 equals 6 (more than 5, however, it's included as it cannot be divided further)
Remaining coordinates sum up to a value of 5
What code is required to achieve the above?
We can do a groupby on cumsum:
s = df['value'].ge(5)
(df.groupby([~s, s.cumsum()], as_index=False, sort=False)
.agg({'from':'min','to':'max', 'value':'sum'})
)
Output:
from to value
0 10 30 5
1 30 40 5
2 40 50 6
3 50 80 5
Update: It looks like you want to accumulate the values so the new groups do not exceed 5. There are several threads on SO saying that this can only be done with a for a loop. So we can do something like this:
thresh = 5
groups, partial, curr_grp = [], thresh, 0
for v in df['value']:
if partial + v > thresh:
curr_grp += 1
partial = v
else:
partial += v
groups.append(curr_grp)
df.groupby(groups).agg({'from':'min','to':'max', 'value':'sum'})
I want to find a more efficient way (in terms of peak memory usage and possibly time) to do the work of panda's groupby.ngroup so that I don't run into memory issues when working with large datasets (I provide reasons for why this column is useful to me below). Take this example with a small dataset. I can accomplish this task easily using groupby.ngroup.
import pandas as pd
import numpy as np
df = pd.DataFrame(np.array(
[[0, 1, 92],
[0, 0, 39],
[0, 0, 32],
[1, 0, 44],
[1, 1, 50],
[0, 1, 11],
[0, 0, 14]]), columns=['male', 'edu', 'wage'])
df['group_id'] = df.groupby(['male', 'edu']).ngroup()
df
male edu wage group_id
0 0 1 92 1
1 0 0 39 0
2 0 0 32 0
3 1 0 44 2
4 1 1 50 3
5 0 1 11 1
6 0 0 14 0
But when I start using larger datasets, the memory usage and computation time explodes and the memory usage in the groupby as a ratio over the memory usage of the dataframe increases almost three-fold for N=100,000,000 as compared to N=100,000. See below.
from memory_profiler import memory_usage
import time
N_values = [10**k for k in range(4, 9)]
stats = pd.DataFrame(index=N_values, dtype=float, columns=['time', 'basemem', 'groupby_mem'])
for N in N_values:
df = pd.DataFrame(
np.hstack([np.random.randint(0, 2, (N, 2)), np.random.normal(5, 1, (N, 1))]),
columns=['male', 'edu', 'wage']
)
def groupby_ngroup():
df.groupby(['male', 'edu']).ngroup()
def foo():
pass
basemem = max(memory_usage(proc=foo))
tic = time.time()
mem = max(memory_usage(proc=groupby_ngroup))
toc = time.time() - tic
stats.loc[N, 'basemem'] = basemem
stats.loc[N, 'groupby_mem'] = mem
stats.loc[N, 'time'] = toc
stats['mem_ratio'] = stats.eval('groupby_mem/basemem')
stats
time basemem groupby_mem mem_ratio
10000 0.037834 104.781250 105.359375 1.005517
100000 0.051785 108.187500 113.125000 1.045638
1000000 0.143642 128.156250 182.437500 1.423555
10000000 0.644650 334.148438 820.183594 2.454549
100000000 6.074531 2422.585938 7095.437500 2.928869
Why am I interested in this group identifier? Because I want to create columns that utilize pandas' groupby functions such as groupby.mean using the .map method as opposed to groupby.transform which takes a lot of memory and time. Furthermore, the .map approach can be used with dask dataframes as dask currently doesn't support .transform. With a column for "group_id" I can simply do means = df.groupby(['group_id'])['wage'].mean() and df['mean_wage'] = df['group_id'].map(means) to do the work of transform.
How about not using ngroup, and instead writing our own function to create group_id column?
Here is a code snippet that seems to give a slightly better performance:
from memory_profiler import memory_usage
import time
import pandas as pd
import numpy as np
N_values = [10**k for k in range(4, 9)]
stats = pd.DataFrame(index=N_values, dtype=float, columns=['time', 'basemem', 'groupby_mem'])
for N in N_values:
df = pd.DataFrame(
np.hstack([np.random.randint(0, 2, (N, 2)), np.random.normal(5, 1, (N, 1))]),
columns=['male', 'edu', 'wage']
)
def groupby_ngroup():
#df.groupby(['male', 'edu']).ngroup()
df['group_id'] = 2*df.male + df.edu
def foo():
pass
basemem = max(memory_usage(proc=foo))
tic = time.time()
mem = max(memory_usage(proc=groupby_ngroup))
toc = time.time() - tic
stats.loc[N, 'basemem'] = basemem
stats.loc[N, 'groupby_mem'] = mem
stats.loc[N, 'time'] = toc
stats['mem_ratio'] = stats.eval('groupby_mem/basemem')
stats
time basemem groupby_mem mem_ratio
10000 0.117921 2370.792969 79.761719 0.033643
100000 0.026921 84.265625 84.324219 1.000695
1000000 0.067960 130.101562 130.101562 1.000000
10000000 0.220024 308.378906 536.140625 1.738577
100000000 0.751135 2367.187500 3651.171875 1.542409
Essentially, we use the fact that the columns are numerical and treat them as binary numbers. The group_ids shall be the decimal equivalents.
Scaling it for three columns gives a similar result. For that, replace the dataframe initialization to the following:
df = pd.DataFrame(
np.hstack([np.random.randint(0, 2, (N, 3)), np.random.normal(5, 1, (N, 1))]),
columns=['male', 'edu','random1', 'wage']
)
and group_id function to:
def groupby_ngroup():
df['group_id'] = 4*df.male + 2*df.edu + df.random1
Following are the results of that test:
time basemem groupby_mem mem_ratio
10000 0.050006 78.906250 78.980469 1.000941
100000 0.033699 85.007812 86.339844 1.015670
1000000 0.066184 147.378906 147.378906 1.000000
10000000 0.322198 422.039062 691.179688 1.637715
100000000 1.233054 3167.921875 5183.183594 1.636146
Let us try using hash
list(map(hash,df.to_records().tolist()))
[4686582722376372986, 3632587615391525059, 2578593961740479157, -48845846747569345, 2044051356115000853, -583388452461625474, -1637380652526859201]
For a groupby where groupby variables are of unknown pattern, it seems that groupby.ngroup may be as good as it gets. But if your groupby variables are all categorical, e.g., take values 0,1,2,3...., then we can take inspiration from the solution given by #saurjog.
To generate the group ID, we can build a numerical expression that evaluates a special sum of the groupby variables. Consider the following functions
def gen_groupby_numexpr(cols, numcats):
txt = [cols[0]]
k = numcats[0]
for c,k_ in zip(cols[1:], numcats[1:]):
txt.append('{}*{}'.format(k, c))
k = k*k_
return ' + '.join(txt)
def ngroup_cat(df, by, numcats):
'''
by : list
the categorical (0,1,2,3...) groupby column names
numcats : list
the number of unique values for each column in "by"
'''
expr = gen_groupby_numexpr(by, numcats)
return df.eval(expr)
The function gen_groupby_numexpr generates the numerical expression and ngroup_cat generates the group id for the groupby variables in by with unique value counts numcats. Thus, consider the following dataset that matches our use case. It contains 3 categorical variables we will use to form the groupby, two of which take values in {0,1} and one takes values in {0,1,2}.
df2 = pd.DataFrame(np.hstack([np.random.randint(0, 2, (100, 2)),
np.random.randint(0, 3, (100, 1)),
np.random.randint(0, 20, (100, 1))]),
columns=['male', 'mar', 'edu', 'wage'])
If we generate the numerical expression we get:
'male + 2*mar + 4*edu'
Putting this altogether, we can generate the group id with
df2['group_id'] = ngroup_cat(df2, ['male', 'mar', 'edu'], [2, 2, 3])
from which we get 2*2*3=12 unique group IDs:
df2[['male', 'mar', 'edu', 'group_id']].drop_duplicates().sort_values(['group_id'])
male mar edu group_id
1 0 0 0 0
13 1 0 0 1
8 0 1 0 2
10 1 1 0 3
4 0 0 1 4
12 1 0 1 5
2 0 1 1 6
6 1 1 1 7
7 0 0 2 8
5 1 0 2 9
44 0 1 2 10
0 1 1 2 11
When I bench mark the solution above against groupby.ngroup it runs nearly 3 times faster on a dataset of N=10,000,000 and uses significantly less additional memory.
Now we can estimate these groupby means and then map them back to the whole dataframe to do the work of transform. I compute some benchmarks with mixed results on whether using transform or groupby then map is faster and less memory intensive. If you are computing means for groups of many variables then I think the latter is more efficient. Further, the latter can also be done in dask where transform is not yet supported.
I'm trying to avoid for loops applying a function on a per row basis of a pandas df. I have looked at many vectorization examples but have not come across anything that will work completely. Ultimately I am trying to add an additional df column with the summation of successful conditions with a specified value per each condition by row.
I have looked at np.apply_along_axis but that's just a hidden loop, np.where but I could not see this working for 25 conditions that I am checking
A B C ... R S T
0 0.279610 0.307119 0.553411 ... 0.897890 0.757151 0.735718
1 0.718537 0.974766 0.040607 ... 0.470836 0.103732 0.322093
2 0.222187 0.130348 0.894208 ... 0.480049 0.348090 0.844101
3 0.834743 0.473529 0.031600 ... 0.049258 0.594022 0.562006
4 0.087919 0.044066 0.936441 ... 0.259909 0.979909 0.403292
[5 rows x 20 columns]
def point_calc(row):
points = 0
if row[2] >= row[13]:
points += 1
if row[2] < 0:
points -= 3
if row[4] >= row[8]:
points += 2
if row[4] < row[12]:
points += 1
if row[16] == row[18]:
points += 4
return points
points_list = []
for indx, row in df.iterrows():
value = point_calc(row)
points_list.append(value)
df['points'] = points_list
This is obviously not efficient but I am not sure how I can vectorize my code since it requires the values per row for each column in the df to get a custom summation of the conditions.
Any help in pointing me in the right direction would be much appreciated.
Thank you.
UPDATE:
I was able to get a little more speed replacing the df.iterrows section with df.apply.
df['points'] = df.apply(lambda row: point_calc(row), axis=1)
UPDATE2:
I updated the function as follows and have substantially decreased the run time with a 10x speed increase from using df.apply and the initial function.
def point_calc(row):
a1 = np.where(row[:,2]) >= row[:,13], 1,0)
a2 = np.where(row[:,2] < 0, -3, 0)
a3 = np.where(row[:,4] >= row[:,8])
etc.
all_points = a1 + a2 + a3 + etc.
return all_points
df['points'] = point_calc(df.to_numpy())
What I am still working on is using np.vectorize on the function itself to see if that can be improved upon as well.
You can try it it the following way:
# this is a small version of your dataframe
df = pd.DataFrame(np.random.random((10,4)), columns=list('ABCD'))
It looks like that:
A B C D
0 0.724198 0.444924 0.554168 0.368286
1 0.512431 0.633557 0.571369 0.812635
2 0.680520 0.666035 0.946170 0.652588
3 0.467660 0.277428 0.964336 0.751566
4 0.762783 0.685524 0.294148 0.515455
5 0.588832 0.276401 0.336392 0.997571
6 0.652105 0.072181 0.426501 0.755760
7 0.238815 0.620558 0.309208 0.427332
8 0.740555 0.566231 0.114300 0.353880
9 0.664978 0.711948 0.929396 0.014719
You can create a Series which counts your points and is initialized with zeros:
points = pd.Series(0, index=df.index)
It looks like that:
0 0
1 0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
9 0
dtype: int64
Afterwards you can add and subtract values line by line if you want:
The condition within the brackets selects the rows, where the condition is true.
Therefore -= and += is only applied in those rows.
points.loc[df.A < df.C] += 1
points.loc[df.B < 0] -= 3
At the end you can extract the values of the series as numpy array if you want (optional):
point_list = points.values
Does this solve your problem?
Is there anyway to compare values within the same column of a pandas DataFrame?
The task at hand is something like this:
import pandas as pd
data = pd.DataFrame({"A": [0,-5,2,3,-3,-4,-4,-2,-1,5,6,7,3,-1]});
I need to find the maximum time (in indices) consecutive +/- values appear (Equivalently checking consecutive values because the sign can be encoded by True/False). The above data should yield 5 because there are 5 consecutive negative integers [-3,-4,-4,-2,-1]
If possible, I was hoping to avoid using a loop because the number of data points in the column may very well exceed millions in order.
I've tried using data.A.rolling() and it's variants, but can't seem to figure out any possible way to do this in a vectorized way.
Any suggestions?
Here's a NumPy approach that computes the max interval lengths for the positive and negative values -
def max_interval_lens(arr):
# Store mask of positive values
pos_mask = arr>=0
# Get indices of shifts
idx = np.r_[0,np.flatnonzero(pos_mask[1:] != pos_mask[:-1])+1, arr.size]
# Return max of intervals
lens = np.diff(idx)
s = int(pos_mask[0])
maxs = [0,0]
if len(lens)==1:
maxs[1-s] = lens[0]
else:
maxs = lens[1-s::2].max(), lens[s::2].max()
return maxs # Positive, negative max lens
Sample run -
In [227]: data
Out[227]:
A
0 0
1 -5
2 2
3 3
4 -3
5 -4
6 -4
7 -2
8 -1
9 5
10 6
11 7
12 3
13 -1
In [228]: max_interval_lens(data['A'].values)
Out[228]: (4, 5)