I have a huge list of 1's and 0's like this :
x = [1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1].
Full list here.
I want to create a new list y with the condition that , the 1's should be preserved only if the they occur in a sequence of >= than 10, else those 1's should be replaced by zeroes.
ex based on x above ^ , y should become:
y = [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1].
So far I have the following :
Finding out where the changes are occurring and
Finding out what sequences are occurring with what frequency:
import numpy as np
import itertools
nx = np.array(x)
print np.argwhere(np.diff(nx)).squeeze()
answer = []
for key, iter in itertools.groupby(nx):
answer.append((key, len(list(iter))))
print answer
which gives me :
[0 3 8 14] # A
[(1, 1), (0, 3), (1, 5), (0, 6), (1, 10)] # B
#A which means the changes are happening after the 0th, 3rd and so on positions.
#B means there is one 1, followed by three 0's followed by five 1's followed by 6 zeroes followed by 10 1's.
How do I proceed to the final step of creating y where we will be replacing the 1's with 0's depending upon the sequence length?
PS: ##I'm humbled by these brilliant solutions from all the wonderful people.
Just check while you are iterating over the group-by. Something like:
>>> from itertools import groupby
>>> x = [1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1]
>>> result = []
>>> for k, g in groupby(x):
... if k:
... g = list(g)
... if len(g) < 10:
... g = len(g)*[0]
... result.extend(g)
...
>>> result
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
Note, this is faster than the corresponding pandas solution, for a dataset of this size at least:
In [11]: from itertools import groupby
In [12]: %%timeit
...: result = []
...: for k, g in groupby(x):
...: if k:
...: g = list(g)
...: if len(g) < 10:
...: g = len(g)*[0]
...: result.extend(g)
...:
181 µs ± 1.72 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [13]: %%timeit s = pd.Series(x)
...: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
...:
4.03 ms ± 176 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
And note, that's being generous with the pandas solution, not counting any time converting from list to pd.Series or converting back, including those:
In [14]: %%timeit
...: s = pd.Series(x)
...: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
...: s = s.tolist()
...:
4.92 ms ± 119 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Here is another numpy approach. Please take note of the benchmarks at the bottom of this post:
import numpy as np
import pandas as pd
from itertools import groupby
import re
from timeit import timeit
def f_pp(data):
switches = np.empty((data.size + 1,), bool)
switches[0] = data[0]
switches[-1] = data[-1]
switches[1:-1] = data[:-1]^data[1:]
switches = np.where(switches)[0].reshape(-1, 2)
switches = switches[switches[:, 1]-switches[:, 0] >= 10].ravel()
reps = np.empty((switches.size + 1,), int)
reps[1:-1] = np.diff(switches)
reps[0] = switches[0]
reps[-1] = data.size - switches[-1]
return np.repeat(np.arange(reps.size) & 1, reps)
def f_ja(data):
result = []
for k, g in groupby(data):
if k:
g = list(g)
if len(g) < 10:
g = len(g)*[0]
result.extend(g)
return result
def f_mu(s):
s = s.copy()
s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
return s
def vrange(starts, stops):
stops = np.asarray(stops)
l = stops - starts # Lengths of each range.
return np.repeat(stops - l.cumsum(), l) + np.arange(l.sum())
def f_ka(data):
x = data.copy()
d = np.where(np.diff(x) != 0)[0]
d2 = np.diff(np.concatenate(([0], d, [x.size])))
ind = np.where(d2 >= 10)[0] - 1
x[vrange(d[ind] + 1, d[ind + 1] + 2)] = 0
return x
def f_ol(data):
return list(re.sub(b'(?<!\x01)\x01{,9}(?!\x01)', lambda m: len(m.group()) * b'\x00', bytes(data)))
n = 10_000
data = np.repeat((np.arange(n) + np.random.randint(2))&1, np.random.randint(1, 20, (n,)))
datal = data.tolist()
datap = pd.Series(data)
kwds = dict(globals=globals(), number=100)
print(np.where(f_ja(datal) != f_pp(data))[0])
print(np.where(f_ol(datal) != f_pp(data))[0])
#print(np.where(f_ka(data) != f_pp(data))[0])
print(np.where(f_mu(datap).values != f_pp(data))[0])
print('itertools.groupby: {:6.3f} ms'.format(10 * timeit('f_ja(datal)', **kwds)))
print('re: {:6.3f} ms'.format(10 * timeit('f_ol(datal)', **kwds)))
#print('numpy Kasramvd: {:6.3f} ms'.format(10 * timeit('f_ka(data)', **kwds)))
print('pandas: {:6.3f} ms'.format(10 * timeit('f_mu(datap)', **kwds)))
print('numpy pp: {:6.3f} ms'.format(10 * timeit('f_pp(data)', **kwds)))
Sample output:
[] # Delta ja, pp
[] # Delta ol, pp
[ 749 750 751 ... 98786 98787 98788] # Delta mu, pp
itertools.groupby: 5.415 ms
re: 28.197 ms
pandas: 14.972 ms
numpy pp: 0.788 ms
Note only from scratch solutions considered. #Olivier's #juanpa.arrivillaga's and my approach yield same answer, #MaxU's doesn't. Couldn't get #Kazramvd's to finish reliably. (May be my fault - don't know pandas and didn't fully understand #Kazramvd's solution).
Note that is only one example, other conditions (like shorter lists, more switches, etc.) may change the ranking.
With list comprehension
From your encoded list B, you can use list comprehension to generate the new list.
b = [(1, 1), (0, 3), (1, 5), (0, 6), (1, 10)] # B
y = sum(([num and int(rep >= 10)] * rep for num, rep in b), [])
From the start with re
Alternatively, from the start this looks like something re could do as it can work with bytes.
import re
x = [1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
y = list(re.sub(b'(?<!\x01)\x01{,9}(?!\x01)', lambda m: len(m.group()) * b'\x00', bytes(x)))
Both solutions output:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
If you wanna use Numpy here is one Vectorized approach:
ind = np.where(np.diff(np.concatenate(([0], np.where(np.diff(x) != 0)[0], [x.size]))) >= 10)[0] - 1
x[vrange(d[ind] + 1, d[ind + 1] + 2)] = 0
If you want to use Python, here is an approach using itertools.chain, itertools.repeat and itertools.groupby within a list-comprehension:
chain.from_iterable(repeat(0, len(i)) if len(i) >= 10 else i for i in [list(g) for _, g in groupby(x)])
Demos:
# Python
In [28]: list(chain.from_iterable(repeat(0, len(i)) if len(i) >= 10 else i for i in [list(g) for _, g in groupby(x)]))
Out[28]: [1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
# Numpy
In [161]: x = np.array([1,0,0,0,1,1,1,1,1,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1, 0, 0, 1, 1, 1, 1, 1, 1 ,1, 1, 1, 1, 0, 0])
In [162]: d = np.where(np.diff(x) != 0)[0]
In [163]: d2 = np.diff(np.concatenate(([0], d, [x.size])))
In [164]: ind = np.where(d2 >= 10)[0] - 1
In [165]: def vrange(starts, stops):
...: stops = np.asarray(stops)
...: l = stops - starts # Lengths of each range.
...: return np.repeat(stops - l.cumsum(), l) + np.arange(l.sum())
...:
In [166]: x[vrange(d[ind] + 1, d[ind + 1] + 2)] = 0
In [167]: x
Out[167]:
array([1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
For Vrange I used this answer https://codereview.stackexchange.com/questions/83018/vectorized-numpy-version-of-arange-with-multiple-start-stop but I think there might be more optimized approaches for that.
Try this:
y = []
for pair in b: ## b is the list which you called #B
add = 0
if pair[0] == 1 and pair[1] > 9:
add = 1
y.extend([add] * pair[1])
using Pandas:
import pandas as pd
In [130]: s = pd.Series(x)
In [131]: s
Out[131]:
0 1
1 0
2 0
3 0
4 1
..
20 1
21 1
22 1
23 1
24 1
Length: 25, dtype: int64
In [132]: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
In [133]: s.tolist()
Out[133]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
In [134]: s
Out[134]:
0 0
1 0
2 0
3 0
4 0
..
20 1
21 1
22 1
23 1
24 1
Length: 25, dtype: int64
for your "huge" list it takes approx. 7 ms on my old notebook:
In [141]: len(x)
Out[141]: 5124
In [142]: %%timeit
...: s = pd.Series(x)
...: s[s.groupby(s.ne(1).cumsum()).transform('count').lt(10)] = 0
...: res = s.tolist()
...:
6.56 ms ± 16.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
Related
I have a list that looks like:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
I need to append to a list, the index when the element changes from 0 to 1 (and not from 1 to 0).
I've tried to do the following, but it also registers when it changes from 1 to 0.
i = 0
while i != len(mot)-1:
if mot[i] != mot[i+1]:
mot_daily_index.append(i)
i += 1
Also, but not as important, is there a cleaner implementation?
Here is how you can do that with a list comprehension:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
mot_daily_index = [i for i,m in enumerate(mot) if i and m and not mot[i-1]]
print(mot_daily_index)
Output:
[7, 24]
Explanation:
list(enumerate([7,5,9,3])) will return [(0, 7), (1, 5), (2, 9), (3, 3)], so the i in i for i, m in enumerate, is the index of m during that iteration.
Use a list comprehension with a filter to get your indexes:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
idx = [i for i,v in enumerate(mot) if i and v > mot[i-1]]
print(idx)
Output:
[7, 24]
You could use
lst = [0, 0, 0, 1, 1, 1, 0, 1]
# 0 1 2 3 4 5 6 7
for index, (x, y) in enumerate(zip(lst, lst[1:])):
if x == 0 and y == 1:
print("Changed from 0 to 1 at", index)
Which yields
Changed from 0 to 1 at 2
Changed from 0 to 1 at 6
Here's a solution using itertools.groupby to group the list into 0's and 1's:
from itertools import groupby
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
mot_daily_index = []
l = 0
for s, g in groupby(mot):
if s == 1:
mot_daily_index.append(l)
l += len(list(g))
print(mot_daily_index)
Output:
[7, 24]
mot = [0,0,0,0,1,0,1,0,1,1,1,0,1,1,1,0,0,0,0]
mot_daily_index = [] # the required list
for i in range(len(a)-1):
if a[i]==0 and a[i+1]==1:
ind.append(i)
your code adds index whenever ith element is different from (i+1)th element
For a 3M element container, this answer is 67.2 times faster than the accepted answer.
This can be accomplished with numpy, by converting the list to a numpy.array.
The code for his answer, is a modification of the code from Find index where elements change value numpy.
That question wanted all transitions v[:-1] != v[1:], not just the small to large transitions, v[:-1] < v[1:], in this question.
Create a Boolean array, by comparing the array to itself, shifted by one place.
Use np.where to return the indices for True
This finds the index before the change, because the arrays are shifted for comparison, so use +1 to get the correct value.
import numpy as np
v = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0]
# convert to array
v = np.array(v)
# create a Boolean array
map_ = v[:-1] < v[1:]
# return the indices
idx = np.where(map_)[0] + 1
print(idx)
[out]:
array([ 7, 24], dtype=int64)
%timeit
# v is 3M elements
v = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0] * 100000
# accepted answer
%timeit [i for i,m in enumerate(v) if i and m and not v[i-1]]
[out]:
336 ms ± 14 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
# this answer
v = np.array(v)
%timeit np.where(v[:-1] < v[1:])[0] + 1
[out]:
5.03 ms ± 85.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
A oneliner using zip:
mot = [0,0,0,0,0,0,0,1,1,1,1,1,1,1,1,1,1,1,1,1,0,0,0,0,1,1,1,0,0,0]
[i+1 for i,m in enumerate(zip(mot[:-1],mot[1:])) if m[0]<m[1]]
# [7, 24]
Another list comprehension take:
mot = [0,1,1,1,1,0,0,0,1,0,0,1,1,1,0,1,1,1,0,0,0,0]
change_mot = [index+1 for index, value in enumerate(zip(mot[:-1], mot[1:], )) if value[1] - value[0] == 1]
Which yields
[1, 8, 11, 15]
This picks up the increase and records the index only if the increase = 1.
I am trying to implement Game Of Life in Python. The new_state function should count the neighbours of a cell and decide, based on the rules (if statements) if it will die(turn 0) or stay alive(turn 1) in the next generation.
Is there a way to check if a value is in the corner/border of an array? The neighbouring cells of a cell will then be the immediate surrounding cells, there is no need to wrap the array. Right now, new_state throws out an index error. I am using numPy for this function.
import numpy
def new_state(array):
updated_array=[]
for x in range(len(array)):
for y in range(len(array[x])):
cell = array[x][y]
neighbours = (array[x-1][y-1], array[x][y-1], array[x+1][y-1], array[x+1][y], array[x+1][y+1], array[x][y+1],
array[x-1][y+1], array[x-1][y])
neighbours_count = sum(neighbours)
if cell == 1:
if neighbours_count == 0 or neighbours_count == 1:
updated_array.append(0)
elif neighbours_count == 2 or neighbours_count == 3:
updated_array.append(1)
elif neighbours_count > 3:
updated_array.append(0)
elif cell == 0:
if neighbours_count == 3:
updated_array.append(1)
return updated_array
To avoid the index error you can pad the array with zeroes:
def new_state(array):
array_padded = np.pad(array, 1)
updated_array=array.copy()
for x in range(1, array.shape[0]+1):
for y in range(1, array.shape[1]+1):
cell = array[x-1][y-1]
neighbours = (array_padded[x-1][y-1], array_padded[x][y-1],
array_padded[x+1][y-1], array_padded[x+1][y],
array_padded[x+1][y+1], array_padded[x][y+1],
array_padded[x-1][y+1], array_padded[x-1][y])
neighbours_count = sum(neighbours)
if cell == 1 and (neighbours_count < 2 or neighbours_count > 3):
updated_array[x-1, y-1] = 0
elif cell == 0 and neighbours_count == 3:
updated_array[x-1, y-1] = 1
return updated_array
Here's a much faster vectorized version:
from scipy.signal import correlate2d
def new_state_v(array):
kernel = np.ones((3,3))
kernel[1,1] = 0
neighbours = correlate2d(array, kernel, 'same')
updated_array=array.copy()
updated_array[(array == 1) & ((neighbours < 2) | (neighbours > 3))] = 0
updated_array[(array == 0) & (neighbours == 3)] = 1
return updated_array
Let's test
>>> array = np.random.randint(2, size=(5,5))
>>> array
array([[0, 0, 1, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 1],
[1, 1, 0, 0, 1],
[0, 1, 1, 1, 0]])
>>> new_state_v(array)
array([[0, 0, 1, 1, 0],
[0, 0, 1, 0, 0],
[0, 1, 0, 1, 0],
[1, 1, 0, 0, 1],
[1, 1, 1, 1, 0]])
Speed comparison:
array = np.random.randint(2, size=(1000,1000))
%timeit new_state(array)
7.76 s ± 94.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
%timeit new_state_v(array)
90.4 ms ± 703 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
I have not thoroughly tested this, but the way I'd go about it would be to allow the array index value to wrap - this way no additional logic besides the wrapping is required (which could be more bug prone when requiring modifications down the line).
Use:
import numpy as np
a = np.array([[1,2,3], [4,5,6], [7,8,9], [10,11,12]])
a[np.mod(x,a.shape[0]),np.mod(y,a.shape[1])]
where x and y are your original indexes.
Examples:
a[np.mod(4,a.shape[0]),np.mod(3,a.shape[1])]
>>> 1
# even wraps correctly with negative indexes!
a[np.mod(-1,a.shape[0]),np.mod(-1,a.shape[1])]
>>> 12
Set values for a window of size n of an array based on the current value of another array
Ignore values that the window overrides
Need to be able to change the window size (n) for different runs
This code works but it is very slow.
n = 3
def signal(arr):
signal = pd.Series(data=0, index=arr.index)
i = 0
while i < len(arr) - 1:
s = arr.iloc[i]
if s in [-1, 1]:
j = i + n
signal.iloc[i: j] = s
i = i + n
else:
i += 1
return signal
arr = [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, -1, -1, 0, 0, 0, 0]
signal = [0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, -1, -1, -1, 0, 0, 0]
Don't make arr a pandas series object but just a numpy array.
Try this:
import numpy as np
def signal(arr, n):
size = len(arr)
signal = np.zeros(size)
for i in range(size):
s = arr[i]
if s in [-1, 1]:
j = i + n
signal[i: j] = s
i = i + n
else:
i += 1
return signal
arr = [0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, -1, -1, 0, 0, 0, 0]
n = 3
signal(arr, n)
I benchmarked the two different solutions and this is way faster:
Original: 738 µs ± 21.9 µs per loop (mean ± std. dev. of 7 runs, 1000
loops each)
New: 9.56 µs ± 778 ns per loop (mean ± std. dev. of 7 runs, 100000
loops each)
I want to avoid using for loop in the following code to achieve performance. Is vectorization suitable for this kind of problem?
a = np.array([[0,1,2,3,4],
[5,6,7,8,9],
[0,1,2,3,4],
[5,6,7,8,9],
[0,1,2,3,4]],dtype= np.float32)
temp_a = np.copy(a)
for i in range(1,a.shape[0]-1):
for j in range(1,a.shape[1]-1):
if a[i,j] > 3:
temp_a[i+1,j] += a[i,j] / 5.
temp_a[i-1,j] += a[i,j] / 5.
temp_a[i,j+1] += a[i,j] / 5.
temp_a[i,j-1] += a[i,j] / 5.
temp_a[i,j] -= a[i,j] * 4. / 5.
a = np.copy(temp_a)
You are basically doing convolution, with some special treatment for borders.
Try the following:
from scipy.signal import convolve2d
# define your filter
f = np.array([[0.0, 0.2, 0.0],
[0.2,-0.8, 0.2],
[0.0, 0.2, 0.0]])
# select parts of 'a' to be used for convolution
b = (a * (a > 3))[1:-1, 1:-1]
# convolve, padding with zeros ('same' mode)
c = convolve2d(b, f, mode='same')
# add the convolved result to 'a', excluding borders
a[1:-1, 1:-1] += c
# treat the special cases of the borders
a[0, 1:-1] += .2 * b[0, :]
a[-1, 1:-1] += .2 * b[-1, :]
a[1:-1, 0] += .2 * b[:, 0]
a[1:-1, -1] += .2 * b[:, -1]
It gives the following result, which is the same as you nested loops.
[[ 0. 2.2 3.4 4.6 4. ]
[ 6.2 2.6 4.2 3. 10.6]
[ 0. 3.4 4.8 6.2 4. ]
[ 6.2 2.6 4.2 3. 10.6]
[ 0. 2.2 3.4 4.6 4. ]]
My trail uses 3 filters, rot90, np.where, np.sum, and np.multiply. I am not sure which way to benchmark is more reasonable. If you do not take into account the time to create filters, it is roughly 4 times faster.
# Each filter basically does what `op` tries to achieve in a loop
filter1 = np.array([[0, 1 ,0, 0, 0],
[1, -4, 1, 0, 0],
[0, 1, 0, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]) /5.
filter2 = np.array([[0, 0 ,1, 0, 0],
[0, 1, -4, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0],
[0, 0, 0, 0, 0]]) /5.
filter3 = np.array([[0, 0 ,0, 0, 0],
[0, 0, 1, 0, 0],
[0, 1, -4, 1, 0],
[0, 0, 1, 0, 0],
[0, 0, 0, 0, 0]]) /5.
# only loop over the center of the matrix, a
center = np.array([[0, 0, 0, 0, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 0, 0, 0, 0]])
filter1 and filter2 can be rotated to represent 4 filters individually.
filter1_90_rot = np.rot90(filter1, k=1)
filter1_180_rot = np.rot90(filter1, k=2)
filter1_270_rot = np.rot90(filter1, k=3)
filter2_90_rot = np.rot90(filter2, k=1)
filter2_180_rot = np.rot90(filter2, k=2)
filter2_270_rot = np.rot90(filter2, k=3)
# Based on different index from `a` return different filter
filter_dict = {
(1,1): filter1,
(3,1): filter1_90_rot,
(3,3): filter1_180_rot,
(1,3): filter1_270_rot,
(1,2): filter2,
(2,1): filter2_90_rot,
(3,2): filter2_180_rot,
(2,3): filter2_270_rot,
(2,2): filter3
}
Main function
def get_new_a(a):
x, y = np.where(((a > 3) * center) > 0) # find pairs that match the condition
return a + np.sum(np.multiply(filter_dict[i, j], a[i,j])
for (i, j) in zip(x,y))
Note: There seem to be some numerical errors such that np.equal() would mostly return False between my result and OP's while np.close() would return true.
Timing results
def op():
temp_a = np.copy(a)
for i in range(1,a.shape[0]-1):
for j in range(1,a.shape[1]-1):
if a[i,j] > 3:
temp_a[i+1,j] += a[i,j] / 5.
temp_a[i-1,j] += a[i,j] / 5.
temp_a[i,j+1] += a[i,j] / 5.
temp_a[i,j-1] += a[i,j] / 5.
temp_a[i,j] -= a[i,j] * 4. / 5.
a2 = np.copy(temp_a)
%timeit op()
167 µs ± 2.72 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
%timeit get_new_a(a):
37.2 µs ± 2.68 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Note again, we ignore the time to create filter as I think it would be a one time thing. If you do want to include the time to create filters, it is roughly two times faster. You might think it is not fair becasue op's method contains two np.copy. The bottleneck of op's method, I think, is the for loop.
Reference:
numpy.multiply do a elementwise multiplication between two matrix.
np.rot90 does rotation for us. k is a parameter that you can decide how many times to rotate.
np.isclose can use this function to check whether two matrices are close within some error that you can define.
I came up with this solution:
a = np.array([[0,0,0,0,0],
[0,6,2,8,0],
[0,1,5,3,0],
[0,6,7,8,0],
[0,0,0,0,0]],dtype= np.float32)
up= np.zeros_like(a)
down= np.zeros_like(a)
right= np.zeros_like(a)
left = np.zeros_like(a)
def new_a(a,u,r,d,l):
c = np.copy(a)
c[c <= 3] = 0
up[:-2, 1:-1] += c[1:-1,1:-1] / 5.
down[2:, 1:-1] += c[1:-1,1:-1] / 5.
left[1:-1, :-2] += c[1:-1,1:-1]/ 5.
right[1:-1, 2:] += c[1:-1,1:-1] / 5.
a[1:-1,1:-1] -= c[1:-1,1:-1] * 4. / 5.
a += up + down + left + right
return a
I have a sorted array of ints which might have repetitions. I would like to count consecutive equal values, restarting from zero when a value is different from the previous one. This is the expected result implemented with a simple python loop:
import numpy as np
def count_multiplicities(a):
r = np.zeros(a.shape, dtype=a.dtype)
for i in range(1, len(a)):
if a[i] == a[i-1]:
r[i] = r[i-1]+1
else:
r[i] = 0
return r
a = (np.random.rand(20)*5).astype(dtype=int)
a.sort()
print "given sorted array: ", a
print "multiplicity count: ", count_multiplicities(a)
Output:
given sorted array: [0 0 0 0 0 1 1 1 2 2 2 2 3 3 3 3 4 4 4 4]
multiplicity count: [0 1 2 3 4 0 1 2 0 1 2 3 0 1 2 3 0 1 2 3]
How can I get the same result in an efficient way using numpy? The array is very long, but the repetitions are just a few (say no more than ten).
In my special case I also know that values start from zero and that the difference between consecutive values is either 0 or 1 (no gaps in values).
Here's one cumsum based vectorized approach -
def count_multiplicities_cumsum_vectorized(a):
out = np.ones(a.size,dtype=int)
idx = np.flatnonzero(a[1:] != a[:-1])+1
out[idx[0]] = -idx[0] + 1
out[0] = 0
out[idx[1:]] = idx[:-1] - idx[1:] + 1
np.cumsum(out, out=out)
return out
Sample run -
In [58]: a
Out[58]: array([0, 0, 0, 0, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 4, 4, 4])
In [59]: count_multiplicities(a) # Original approach
Out[59]: array([0, 1, 2, 3, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 0, 1, 2])
In [60]: count_multiplicities_cumsum_vectorized(a)
Out[60]: array([0, 1, 2, 3, 0, 1, 2, 3, 4, 0, 1, 2, 3, 4, 0, 1, 2, 0, 1, 2])
Runtime test -
In [66]: a = (np.random.rand(200000)*1000).astype(dtype=int)
...: a.sort()
...:
In [67]: a
Out[67]: array([ 0, 0, 0, ..., 999, 999, 999])
In [68]: %timeit count_multiplicities(a)
10 loops, best of 3: 87.2 ms per loop
In [69]: %timeit count_multiplicities_cumsum_vectorized(a)
1000 loops, best of 3: 739 µs per loop
Related post.
I would use numba on such problems
import numba
nb_count_multiplicities = numba.njit("int32[:](int32[:])")(count_multiplicities)
X=nb_count_multiplicities(a)
Without rewriting your code at all it is about 50 percent faster than Divakar's vectorized solution.
Vectorizing is many times useful if it results in a shorter and maybe easier understandable code, but if you forcefully have to vectorize a code which could also be a problem for a quite expirienced programmer numba is the way to go.