Vectorization to achieve performance - python

I want to avoid using for loop in the following code to achieve performance. Is vectorization suitable for this kind of problem?
a = np.array([[0,1,2,3,4],
[5,6,7,8,9],
[0,1,2,3,4],
[5,6,7,8,9],
[0,1,2,3,4]],dtype= np.float32)
temp_a = np.copy(a)
for i in range(1,a.shape[0]-1):
for j in range(1,a.shape[1]-1):
if a[i,j] > 3:
temp_a[i+1,j] += a[i,j] / 5.
temp_a[i-1,j] += a[i,j] / 5.
temp_a[i,j+1] += a[i,j] / 5.
temp_a[i,j-1] += a[i,j] / 5.
temp_a[i,j] -= a[i,j] * 4. / 5.
a = np.copy(temp_a)

You are basically doing convolution, with some special treatment for borders.
Try the following:
from scipy.signal import convolve2d
# define your filter
f = np.array([[0.0, 0.2, 0.0],
[0.2,-0.8, 0.2],
[0.0, 0.2, 0.0]])
# select parts of 'a' to be used for convolution
b = (a * (a > 3))[1:-1, 1:-1]
# convolve, padding with zeros ('same' mode)
c = convolve2d(b, f, mode='same')
# add the convolved result to 'a', excluding borders
a[1:-1, 1:-1] += c
# treat the special cases of the borders
a[0, 1:-1] += .2 * b[0, :]
a[-1, 1:-1] += .2 * b[-1, :]
a[1:-1, 0] += .2 * b[:, 0]
a[1:-1, -1] += .2 * b[:, -1]
It gives the following result, which is the same as you nested loops.
[[ 0. 2.2 3.4 4.6 4. ]
[ 6.2 2.6 4.2 3. 10.6]
[ 0. 3.4 4.8 6.2 4. ]
[ 6.2 2.6 4.2 3. 10.6]
[ 0. 2.2 3.4 4.6 4. ]]

My trail uses 3 filters, rot90, np.where, np.sum, and np.multiply. I am not sure which way to benchmark is more reasonable. If you do not take into account the time to create filters, it is roughly 4 times faster.
# Each filter basically does what `op` tries to achieve in a loop
filter1 = np.array([[0, 1 ,0, 0, 0],
[1, -4, 1, 0, 0],
[0, 1, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]) /5.
filter2 = np.array([[0, 0 ,1, 0, 0],
[0, 1, -4, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]) /5.
filter3 = np.array([[0, 0 ,0, 0, 0],
[0, 0, 1, 0, 0],
[0, 1, -4, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0]]) /5.
# only loop over the center of the matrix, a
center = np.array([[0, 0, 0, 0, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 0, 0, 0, 0]])
filter1 and filter2 can be rotated to represent 4 filters individually.
filter1_90_rot = np.rot90(filter1, k=1)
filter1_180_rot = np.rot90(filter1, k=2)
filter1_270_rot = np.rot90(filter1, k=3)
filter2_90_rot = np.rot90(filter2, k=1)
filter2_180_rot = np.rot90(filter2, k=2)
filter2_270_rot = np.rot90(filter2, k=3)
# Based on different index from `a` return different filter
filter_dict = {
(1,1): filter1,
(3,1): filter1_90_rot,
(3,3): filter1_180_rot,
(1,3): filter1_270_rot,
(1,2): filter2,
(2,1): filter2_90_rot,
(3,2): filter2_180_rot,
(2,3): filter2_270_rot,
(2,2): filter3
}
Main function
def get_new_a(a):
x, y = np.where(((a > 3) * center) > 0) # find pairs that match the condition
return a + np.sum(np.multiply(filter_dict[i, j], a[i,j])
for (i, j) in zip(x,y))
Note: There seem to be some numerical errors such that np.equal() would mostly return False between my result and OP's while np.close() would return true.
Timing results
def op():
temp_a = np.copy(a)
for i in range(1,a.shape[0]-1):
for j in range(1,a.shape[1]-1):
if a[i,j] > 3:
temp_a[i+1,j] += a[i,j] / 5.
temp_a[i-1,j] += a[i,j] / 5.
temp_a[i,j+1] += a[i,j] / 5.
temp_a[i,j-1] += a[i,j] / 5.
temp_a[i,j] -= a[i,j] * 4. / 5.
a2 = np.copy(temp_a)
%timeit op()
167 µs ± 2.72 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit get_new_a(a):
37.2 µs ± 2.68 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Note again, we ignore the time to create filter as I think it would be a one time thing. If you do want to include the time to create filters, it is roughly two times faster. You might think it is not fair becasue op's method contains two np.copy. The bottleneck of op's method, I think, is the for loop.
Reference:
numpy.multiply do a elementwise multiplication between two matrix.
np.rot90 does rotation for us. k is a parameter that you can decide how many times to rotate.
np.isclose can use this function to check whether two matrices are close within some error that you can define.

I came up with this solution:
a = np.array([[0,0,0,0,0],
[0,6,2,8,0],
[0,1,5,3,0],
[0,6,7,8,0],
[0,0,0,0,0]],dtype= np.float32)
up= np.zeros_like(a)
down= np.zeros_like(a)
right= np.zeros_like(a)
left = np.zeros_like(a)
def new_a(a,u,r,d,l):
c = np.copy(a)
c[c <= 3] = 0
up[:-2, 1:-1] += c[1:-1,1:-1] / 5.
down[2:, 1:-1] += c[1:-1,1:-1] / 5.
left[1:-1, :-2] += c[1:-1,1:-1]/ 5.
right[1:-1, 2:] += c[1:-1,1:-1] / 5.
a[1:-1,1:-1] -= c[1:-1,1:-1] * 4. / 5.
a += up + down + left + right
return a

Related

How to find the index where values in a list, increase value

I have a list that looks like:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
I need to append to a list, the index when the element changes from 0 to 1 (and not from 1 to 0).
I've tried to do the following, but it also registers when it changes from 1 to 0.
i = 0
while i != len(mot)-1:
if mot[i] != mot[i+1]:
mot_daily_index.append(i)
i += 1
Also, but not as important, is there a cleaner implementation?
Here is how you can do that with a list comprehension:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
mot_daily_index = [i for i,m in enumerate(mot) if i and m and not mot[i-1]]
print(mot_daily_index)
Output:
[7, 24]
Explanation:
list(enumerate([7,5,9,3])) will return [(0, 7), (1, 5), (2, 9), (3, 3)], so the i in i for i, m in enumerate, is the index of m during that iteration.
Use a list comprehension with a filter to get your indexes:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
idx = [i for i,v in enumerate(mot) if i and v > mot[i-1]]
print(idx)
Output:
[7, 24]
You could use
lst = [0, 0, 0, 1, 1, 1, 0, 1]
# 0 1 2 3 4 5 6 7
for index, (x, y) in enumerate(zip(lst, lst[1:])):
if x == 0 and y == 1:
print("Changed from 0 to 1 at", index)
Which yields
Changed from 0 to 1 at 2
Changed from 0 to 1 at 6
Here's a solution using itertools.groupby to group the list into 0's and 1's:
from itertools import groupby
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
mot_daily_index = []
l = 0
for s, g in groupby(mot):
if s == 1:
mot_daily_index.append(l)
l += len(list(g))
print(mot_daily_index)
Output:
[7, 24]
mot = [0,0,0,0,1,0,1,0,1,1,1,0,1,1,1,0,0,0,0]
mot_daily_index = [] # the required list
for i in range(len(a)-1):
if a[i]==0 and a[i+1]==1:
ind.append(i)
your code adds index whenever ith element is different from (i+1)th element
For a 3M element container, this answer is 67.2 times faster than the accepted answer.
This can be accomplished with numpy, by converting the list to a numpy.array.
The code for his answer, is a modification of the code from Find index where elements change value numpy.
That question wanted all transitions v[:-1] != v[1:], not just the small to large transitions, v[:-1] < v[1:], in this question.
Create a Boolean array, by comparing the array to itself, shifted by one place.
Use np.where to return the indices for True
This finds the index before the change, because the arrays are shifted for comparison, so use +1 to get the correct value.
import numpy as np
v = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0]
# convert to array
v = np.array(v)
# create a Boolean array
map_ = v[:-1] < v[1:]
# return the indices
idx = np.where(map_)[0] + 1
print(idx)
[out]:
array([ 7, 24], dtype=int64)
%timeit
# v is 3M elements
v = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0] * 100000
# accepted answer
%timeit [i for i,m in enumerate(v) if i and m and not v[i-1]]
[out]:
336 ms ± 14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# this answer
v = np.array(v)
%timeit np.where(v[:-1] < v[1:])[0] + 1
[out]:
5.03 ms ± 85.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
A oneliner using zip:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
[i+1 for i,m in enumerate(zip(mot[:-1],mot[1:])) if m[0]<m[1]]
# [7, 24]
Another list comprehension take:
mot = [0,1,1,1,1,0,0,0,1,0,0,1,1,1,0,1,1,1,0,0,0,0]
change_mot = [index+1 for index, value in enumerate(zip(mot[:-1], mot[1:], )) if value[1] - value[0] == 1]
Which yields
[1, 8, 11, 15]
This picks up the increase and records the index only if the increase = 1.

Conway's Game of Life: check if a cell is in the corner/border

I am trying to implement Game Of Life in Python. The new_state function should count the neighbours of a cell and decide, based on the rules (if statements) if it will die(turn 0) or stay alive(turn 1) in the next generation.
Is there a way to check if a value is in the corner/border of an array? The neighbouring cells of a cell will then be the immediate surrounding cells, there is no need to wrap the array. Right now, new_state throws out an index error. I am using numPy for this function.
import numpy
def new_state(array):
updated_array=[]
for x in range(len(array)):
for y in range(len(array[x])):
cell = array[x][y]
neighbours = (array[x-1][y-1], array[x][y-1], array[x+1][y-1], array[x+1][y], array[x+1][y+1], array[x][y+1],
array[x-1][y+1], array[x-1][y])
neighbours_count = sum(neighbours)
if cell == 1:
if neighbours_count == 0 or neighbours_count == 1:
updated_array.append(0)
elif neighbours_count == 2 or neighbours_count == 3:
updated_array.append(1)
elif neighbours_count > 3:
updated_array.append(0)
elif cell == 0:
if neighbours_count == 3:
updated_array.append(1)
return updated_array
To avoid the index error you can pad the array with zeroes:
def new_state(array):
array_padded = np.pad(array, 1)
updated_array=array.copy()
for x in range(1, array.shape[0]+1):
for y in range(1, array.shape[1]+1):
cell = array[x-1][y-1]
neighbours = (array_padded[x-1][y-1], array_padded[x][y-1],
array_padded[x+1][y-1], array_padded[x+1][y],
array_padded[x+1][y+1], array_padded[x][y+1],
array_padded[x-1][y+1], array_padded[x-1][y])
neighbours_count = sum(neighbours)
if cell == 1 and (neighbours_count < 2 or neighbours_count > 3):
updated_array[x-1, y-1] = 0
elif cell == 0 and neighbours_count == 3:
updated_array[x-1, y-1] = 1
return updated_array
Here's a much faster vectorized version:
from scipy.signal import correlate2d
def new_state_v(array):
kernel = np.ones((3,3))
kernel[1,1] = 0
neighbours = correlate2d(array, kernel, 'same')
updated_array=array.copy()
updated_array[(array == 1) & ((neighbours < 2) | (neighbours > 3))] = 0
updated_array[(array == 0) & (neighbours == 3)] = 1
return updated_array
Let's test
>>> array = np.random.randint(2, size=(5,5))
>>> array
array([[0, 0, 1, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 1],
[1, 1, 0, 0, 1],
[0, 1, 1, 1, 0]])
>>> new_state_v(array)
array([[0, 0, 1, 1, 0],
[0, 0, 1, 0, 0],
[0, 1, 0, 1, 0],
[1, 1, 0, 0, 1],
[1, 1, 1, 1, 0]])
Speed comparison:
array = np.random.randint(2, size=(1000,1000))
%timeit new_state(array)
7.76 s ± 94.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit new_state_v(array)
90.4 ms ± 703 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
I have not thoroughly tested this, but the way I'd go about it would be to allow the array index value to wrap - this way no additional logic besides the wrapping is required (which could be more bug prone when requiring modifications down the line).
Use:
import numpy as np
a = np.array([[1,2,3], [4,5,6], [7,8,9], [10,11,12]])
a[np.mod(x,a.shape[0]),np.mod(y,a.shape[1])]
where x and y are your original indexes.
Examples:
a[np.mod(4,a.shape[0]),np.mod(3,a.shape[1])]
>>> 1
# even wraps correctly with negative indexes!
a[np.mod(-1,a.shape[0]),np.mod(-1,a.shape[1])]
>>> 12

Non-Assert Way To Compare Two 2D Arrays for Accuracy

I am currently training a LSTM which classifies frames. What I am trying to do is compare two 2d numpy arrays to check for accuracy between my prediction and target. I have currently looked around for non-naive ways to solve this problem using NumPy / SciPy.
I am aware that there is np.testing.assert_array_equal(x, y) which uses Assertion to output the results. I am looking for a way to solve this issue using NumPy / SciPy so I can store the results rather than an Assert print out:
Arrays are not equal
(mismatch 14.285714285714292%)
x: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
y: array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0])
x = np.asarray([[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]])
y = np.asarray([[0, 0, 0], [0, 0, 0], [0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 0, 0], [0, 0, 0]])
try:
np.testing.assert_array_equal(x, y)
res = True
except AssertionError as err:
res = False
print (err)
I am looking for a way which I can store the mismatch of these two arrays without using a naive fashion (Two comparative loops):
accuracy = thisFunction(x,y)
I am sure there is something in NumPy which can solve this, I've had no luck with searching for built-in functions.
As hpaulj noted in the comment, you can use numpy.allclose() for checking the array equality, with acceptable difference of up to some tolerance value (see below or NumPy notes).
Here is a small illustration with two simple float arrays.
In [7]: arr1 = np.array([1.3, 1.4, 1.5, 3.4])
In [8]: arr2 = np.array([1.299999, 1.4, 1.4999999, 3.3999999999])
In [9]: np.allclose(arr1, arr2)
Out[9]: True
numpy.allclose will return True if the corresponding elements in the arrays are dissimilar (only up to the tolerance value). Else it would return False. NumPy default for relative & absolute tolerance values are rtol=1e-05, atol=1e-08 respectively.
Having said that, if you only want to compare int arrays, then you'd be better off with numpy.array_equal() which is approx. 8x faster than numpy.allclose.
In [17]: arr1 = np.random.randint(23045)
In [18]: arr2 = np.random.randint(23045)
In [19]: %timeit np.allclose(arr1, arr2)
22.9 µs ± 471 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [20]: %timeit np.array_equal(arr1, arr2)
3.99 µs ± 68.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
np.array_equal(x, y) is roughly equivalent to (x == y).all(). You can use this to compute the discrepancies:
def array_comp(x, y):
"""
Return the status of the comparison and the discrepancy.
For arrays of the same shape, the discrepancy is a ratio of mismatches to the total size.
For arrays of different shapes or sizes, the discrepancy is a message indicating the mismatch.
"""
if x.shape != y.shape:
return False, 'shape'
count = x.size - np.count_nonzero(x == y)
return count == 0, count / x.size

Replace 1's with 0's in a sequence

I have a huge list of 1's and 0's like this :
x = [1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1].
Full list here.
I want to create a new list y with the condition that , the 1's should be preserved only if the they occur in a sequence of >= than 10, else those 1's should be replaced by zeroes.
ex based on x above ^ , y should become:
y = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1].
So far I have the following :
Finding out where the changes are occurring and
Finding out what sequences are occurring with what frequency:
import numpy as np
import itertools
nx = np.array(x)
print np.argwhere(np.diff(nx)).squeeze()
answer = []
for key, iter in itertools.groupby(nx):
answer.append((key, len(list(iter))))
print answer
which gives me :
[0 3 8 14] # A
[(1, 1), (0, 3), (1, 5), (0, 6), (1, 10)] # B
#A which means the changes are happening after the 0th, 3rd and so on positions.
#B means there is one 1, followed by three 0's followed by five 1's followed by 6 zeroes followed by 10 1's.
How do I proceed to the final step of creating y where we will be replacing the 1's with 0's depending upon the sequence length?
PS: ##I'm humbled by these brilliant solutions from all the wonderful people.
Just check while you are iterating over the group-by. Something like:
>>> from itertools import groupby
>>> x = [1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1]
>>> result = []
>>> for k, g in groupby(x):
... if k:
... g = list(g)
... if len(g) < 10:
... g = len(g)*[0]
... result.extend(g)
...
>>> result
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
Note, this is faster than the corresponding pandas solution, for a dataset of this size at least:
In [11]: from itertools import groupby
In [12]: %%timeit
...: result = []
...: for k, g in groupby(x):
...: if k:
...: g = list(g)
...: if len(g) < 10:
...: g = len(g)*[0]
...: result.extend(g)
...:
181 µs ± 1.72 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [13]: %%timeit s = pd.Series(x)
...: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
...:
4.03 ms ± 176 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
And note, that's being generous with the pandas solution, not counting any time converting from list to pd.Series or converting back, including those:
In [14]: %%timeit
...: s = pd.Series(x)
...: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
...: s = s.tolist()
...:
4.92 ms ± 119 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Here is another numpy approach. Please take note of the benchmarks at the bottom of this post:
import numpy as np
import pandas as pd
from itertools import groupby
import re
from timeit import timeit
def f_pp(data):
switches = np.empty((data.size + 1,), bool)
switches[0] = data[0]
switches[-1] = data[-1]
switches[1:-1] = data[:-1]^data[1:]
switches = np.where(switches)[0].reshape(-1, 2)
switches = switches[switches[:, 1]-switches[:, 0] >= 10].ravel()
reps = np.empty((switches.size + 1,), int)
reps[1:-1] = np.diff(switches)
reps[0] = switches[0]
reps[-1] = data.size - switches[-1]
return np.repeat(np.arange(reps.size) & 1, reps)
def f_ja(data):
result = []
for k, g in groupby(data):
if k:
g = list(g)
if len(g) < 10:
g = len(g)*[0]
result.extend(g)
return result
def f_mu(s):
s = s.copy()
s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
return s
def vrange(starts, stops):
stops = np.asarray(stops)
l = stops - starts # Lengths of each range.
return np.repeat(stops - l.cumsum(), l) + np.arange(l.sum())
def f_ka(data):
x = data.copy()
d = np.where(np.diff(x) != 0)[0]
d2 = np.diff(np.concatenate(([0], d, [x.size])))
ind = np.where(d2 >= 10)[0] - 1
x[vrange(d[ind] + 1, d[ind + 1] + 2)] = 0
return x
def f_ol(data):
return list(re.sub(b'(?<!\x01)\x01{,9}(?!\x01)', lambda m: len(m.group()) * b'\x00', bytes(data)))
n = 10_000
data = np.repeat((np.arange(n) + np.random.randint(2))&1, np.random.randint(1, 20, (n,)))
datal = data.tolist()
datap = pd.Series(data)
kwds = dict(globals=globals(), number=100)
print(np.where(f_ja(datal) != f_pp(data))[0])
print(np.where(f_ol(datal) != f_pp(data))[0])
#print(np.where(f_ka(data) != f_pp(data))[0])
print(np.where(f_mu(datap).values != f_pp(data))[0])
print('itertools.groupby: {:6.3f} ms'.format(10 * timeit('f_ja(datal)', **kwds)))
print('re: {:6.3f} ms'.format(10 * timeit('f_ol(datal)', **kwds)))
#print('numpy Kasramvd: {:6.3f} ms'.format(10 * timeit('f_ka(data)', **kwds)))
print('pandas: {:6.3f} ms'.format(10 * timeit('f_mu(datap)', **kwds)))
print('numpy pp: {:6.3f} ms'.format(10 * timeit('f_pp(data)', **kwds)))
Sample output:
[] # Delta ja, pp
[] # Delta ol, pp
[ 749 750 751 ... 98786 98787 98788] # Delta mu, pp
itertools.groupby: 5.415 ms
re: 28.197 ms
pandas: 14.972 ms
numpy pp: 0.788 ms
Note only from scratch solutions considered. #Olivier's #juanpa.arrivillaga's and my approach yield same answer, #MaxU's doesn't. Couldn't get #Kazramvd's to finish reliably. (May be my fault - don't know pandas and didn't fully understand #Kazramvd's solution).
Note that is only one example, other conditions (like shorter lists, more switches, etc.) may change the ranking.
With list comprehension
From your encoded list B, you can use list comprehension to generate the new list.
b = [(1, 1), (0, 3), (1, 5), (0, 6), (1, 10)] # B
y = sum(([num and int(rep >= 10)] * rep for num, rep in b), [])
From the start with re
Alternatively, from the start this looks like something re could do as it can work with bytes.
import re
x = [1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
y = list(re.sub(b'(?<!\x01)\x01{,9}(?!\x01)', lambda m: len(m.group()) * b'\x00', bytes(x)))
Both solutions output:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
If you wanna use Numpy here is one Vectorized approach:
ind = np.where(np.diff(np.concatenate(([0], np.where(np.diff(x) != 0)[0], [x.size]))) >= 10)[0] - 1
x[vrange(d[ind] + 1, d[ind + 1] + 2)] = 0
If you want to use Python, here is an approach using itertools.chain, itertools.repeat and itertools.groupby within a list-comprehension:
chain.from_iterable(repeat(0, len(i)) if len(i) >= 10 else i for i in [list(g) for _, g in groupby(x)])
Demos:
# Python
In [28]: list(chain.from_iterable(repeat(0, len(i)) if len(i) >= 10 else i for i in [list(g) for _, g in groupby(x)]))
Out[28]: [1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
# Numpy
In [161]: x = np.array([1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1, 0, 0, 1, 1, 1, 1, 1, 1 ,1, 1, 1, 1, 0, 0])
In [162]: d = np.where(np.diff(x) != 0)[0]
In [163]: d2 = np.diff(np.concatenate(([0], d, [x.size])))
In [164]: ind = np.where(d2 >= 10)[0] - 1
In [165]: def vrange(starts, stops):
...: stops = np.asarray(stops)
...: l = stops - starts # Lengths of each range.
...: return np.repeat(stops - l.cumsum(), l) + np.arange(l.sum())
...:
In [166]: x[vrange(d[ind] + 1, d[ind + 1] + 2)] = 0
In [167]: x
Out[167]:
array([1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
For Vrange I used this answer https://codereview.stackexchange.com/questions/83018/vectorized-numpy-version-of-arange-with-multiple-start-stop but I think there might be more optimized approaches for that.
Try this:
y = []
for pair in b: ## b is the list which you called #B
add = 0
if pair[0] == 1 and pair[1] > 9:
add = 1
y.extend([add] * pair[1])
using Pandas:
import pandas as pd
In [130]: s = pd.Series(x)
In [131]: s
Out[131]:
0 1
1 0
2 0
3 0
4 1
..
20 1
21 1
22 1
23 1
24 1
Length: 25, dtype: int64
In [132]: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
In [133]: s.tolist()
Out[133]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
In [134]: s
Out[134]:
0 0
1 0
2 0
3 0
4 0
..
20 1
21 1
22 1
23 1
24 1
Length: 25, dtype: int64
for your "huge" list it takes approx. 7 ms on my old notebook:
In [141]: len(x)
Out[141]: 5124
In [142]: %%timeit
...: s = pd.Series(x)
...: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
...: res = s.tolist()
...:
6.56 ms ± 16.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Find location of the largest sum in 2d numpy array

So lets say I have a an array that looks similar to this :
array([[0, 0, 0, 0, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 0, 0, 0, 0]])
I would like to return the location of the center of the largest sum of values within a certain n*n square. So in this case it would be (2,2) if n = 3. If I let n = 4 it would be the same result.
Does numpy have a method for finding this location?
Approach #1 : We can use SciPy's 2D convolution to get summations in sliding windows of shape (n,n) and choose the index of the window with the biggest sum with argmax and translate to row, col indices with np.unravel_index, like so -
from scipy.signal import convolve2d as conv2
def largest_sum_pos_app1(a, n):
idx = conv2(a, np.ones((n,n),dtype=int),'same').argmax()
return np.unravel_index(idx, a.shape)
Sample run -
In [558]: a
Out[558]:
array([[0, 0, 0, 0, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 0, 0, 0, 0]])
In [559]: largest_sum_pos_app1(a, n=3)
Out[559]: (2, 2)
Approach #1S (Super charged) : We can boost it further by using uniform filter, like so -
from scipy.ndimage.filters import uniform_filter as unif2D
def largest_sum_pos_app1_mod1(a, n):
idx = unif2D(a.astype(float),size=n, mode='constant').argmax()
return np.unravel_index(idx, a.shape)
Approach #2 : Another based on scikit-image's sliding window creating tool view_as_windows, we would create sliding windows of shape (n,n) to give us a 4D array with the last two axes of shape (n,n) corresponding to the search window size. So, we would sum along those two axes and get the argmax index and translate it to the actual row, col positions.
Hence, the implementation would be -
from skimage.util.shape import view_as_windows
def largest_sum_pos_app2(a, n):
h = (n-1)//2 # half window size
idx = view_as_windows(a, (n,n)).sum((-2,-1)).argmax()
return tuple(np.array(np.unravel_index(idx, np.array(a.shape)-n+1))+h)
As also mentioned in the comments, a search square with an even n would be confusing given that it won't have its center at any element coordinate.
Runtime test
In [741]: np.random.seed(0)
In [742]: a = np.random.randint(0,1000,(1000,1000))
In [743]: largest_sum_pos_app1(a, n= 5)
Out[743]: (966, 403)
In [744]: largest_sum_pos_app1_mod1(a, n= 5)
Out[744]: (966, 403)
In [745]: largest_sum_pos_app2(a, n= 5)
Out[745]: (966, 403)
In [746]: %timeit largest_sum_pos_app1(a, n= 5)
...: %timeit largest_sum_pos_app1_mod1(a, n= 5)
...: %timeit largest_sum_pos_app2(a, n= 5)
...:
10 loops, best of 3: 57.6 ms per loop
100 loops, best of 3: 10.1 ms per loop
10 loops, best of 3: 47.7 ms per loop

Categories