Given a bool tensor in pytorch, I would like to have a "lockout period" of N values after each True value along each row. More specifically, in the example below, moving from left to right on any given row I would like to ensure that after each True the following N values are all False.
e.g.
N = 3
input = tensor([[0, 0, 0, 0, 1, 1, 0, 1, 0, 1],
[1, 1, 1, 0, 1, 0, 1, 1, 0, 1]])
# should output
tensor([[0, 0, 0, 0, 1, 0, 0, 0, 0, 1],
[1, 0, 0, 0, 1, 0, 0, 0, 0, 1]])
I can solve this with a double loop e.g.
for row in input:
for element in row:
# if sum of previous N entries > 0 set input[row,element] = 0
However I would like to solve this either (a) without looping at all or (b) with just a single loop (e.g. for column in input). Is there a way to achieve preferably (a), or otherwise (b)? I cannot assume the input tensor will be sparse or have any paritcular distribution.
Thanks Naga for the (Edit: since deleted) answer, but I need a pytorch solution and I am not sure that solution is O(n) as stated.
I found the following solution (looping over columns) seems to do the trick.
input = input.to(torch.bool)
for i, col in enumerate(input.t()):
input[:,i+1:i+1+N] = torch.mul(~col, input[:,i+1:i+1+N].t()).t()
Additionally based on a quick comparison on the sort of array size I'm interested in, it seems to be around 5x faster.
def a():
input = torch.randint(high=2, size = (200,100))
input = input.to(torch.bool)
N = 10
for i, col in enumerate(input.t()):
input[:,i+1:i+1+N] = torch.mul(~col, input[:,i+1:i+1+N].t()).t()
def b():
N = 10
a = np.random.randint(0, high=2, size = (200,100), dtype=int)
inds = np.where(a==1)
for r,c in np.nditer(inds):
if a[r,c]==1:
a[r,c+1:c+N]=0
%timeit a()
# 100 loops, best of 5: 2.47 ms per loop
%timeit b()
# 100 loops, best of 5: 12.8 ms per loop
Related
For a one dimensional numpy array of 1's and 0's, how can I effectively "mask" the array such that after the occurrence of a 1, the next n elements of the array are converted to zero. After the n elements have passed, the pattern repeats such that the first next occurrence of a 1 is preserved followed again by n zeros.
It is important that the first eligible occurrences of 1 are preserved, so a simple mask such as:
[true, false, false, true ...] won't work.
furthermore, the data set is massive so efficiency is important.
I've written crude python code to give me the desired results, but it is way too slow for what I need.
Here is an example:
data = [0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1]
n = 3
newData = []
tail = 0
for x in data:
if x == 1 and tail <= 0:
newData.append(1)
tail = n
else:
newData.append(0)
tail -= 1
print(newData)
newData: [0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1]
Is there possibly a vectorized numpy solution to this problem?
I'm processing tens of thousands of arrays, with more than a million elements in each array. So far using numpy functions has been the only way to manage this.
As far as I know, there is no option completely in numpy to do this. You could still use numpy to reduce the time for grabbing the indices, though.
data = [0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1]
n=3
def get_new_data(data,n):
new_data = np.zeros(len(data))
non_zero = np.argwhere(data).ravel()
idx = non_zero[0]
new_data[idx] =1
idx += n
for i in non_zero[1:]:
if i > idx:
new_data[i] = 1
idx+=n
return new_data
get_new_data(data, n)
A function like this should give you a better run time since you are not looping over the whole array.
If this is still not optimal to you, you can look at using numba, which works very well with numpy and is relatively easy to use.
You could do it like this:-
N = 3
data = [0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1]
newData = data.copy()
i = 0
M = [0 for _ in range(N)]
while i < len(newData) - N:
if newData[i] == 1:
newData[i + 1:i + 1 + N] = M
i += N
i += 1
print(newData)
I am currently training a LSTM which classifies frames. What I am trying to do is compare two 2d numpy arrays to check for accuracy between my prediction and target. I have currently looked around for non-naive ways to solve this problem using NumPy / SciPy.
I am aware that there is np.testing.assert_array_equal(x, y) which uses Assertion to output the results. I am looking for a way to solve this issue using NumPy / SciPy so I can store the results rather than an Assert print out:
Arrays are not equal
(mismatch 14.285714285714292%)
x: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
y: array([0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0])
x = np.asarray([[0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0], [0, 0, 0]])
y = np.asarray([[0, 0, 0], [0, 0, 0], [0, 0, 1], [0, 1, 0], [1, 0, 0], [0, 0, 0], [0, 0, 0]])
try:
np.testing.assert_array_equal(x, y)
res = True
except AssertionError as err:
res = False
print (err)
I am looking for a way which I can store the mismatch of these two arrays without using a naive fashion (Two comparative loops):
accuracy = thisFunction(x,y)
I am sure there is something in NumPy which can solve this, I've had no luck with searching for built-in functions.
As hpaulj noted in the comment, you can use numpy.allclose() for checking the array equality, with acceptable difference of up to some tolerance value (see below or NumPy notes).
Here is a small illustration with two simple float arrays.
In [7]: arr1 = np.array([1.3, 1.4, 1.5, 3.4])
In [8]: arr2 = np.array([1.299999, 1.4, 1.4999999, 3.3999999999])
In [9]: np.allclose(arr1, arr2)
Out[9]: True
numpy.allclose will return True if the corresponding elements in the arrays are dissimilar (only up to the tolerance value). Else it would return False. NumPy default for relative & absolute tolerance values are rtol=1e-05, atol=1e-08 respectively.
Having said that, if you only want to compare int arrays, then you'd be better off with numpy.array_equal() which is approx. 8x faster than numpy.allclose.
In [17]: arr1 = np.random.randint(23045)
In [18]: arr2 = np.random.randint(23045)
In [19]: %timeit np.allclose(arr1, arr2)
22.9 µs ± 471 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [20]: %timeit np.array_equal(arr1, arr2)
3.99 µs ± 68.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
np.array_equal(x, y) is roughly equivalent to (x == y).all(). You can use this to compute the discrepancies:
def array_comp(x, y):
"""
Return the status of the comparison and the discrepancy.
For arrays of the same shape, the discrepancy is a ratio of mismatches to the total size.
For arrays of different shapes or sizes, the discrepancy is a message indicating the mismatch.
"""
if x.shape != y.shape:
return False, 'shape'
count = x.size - np.count_nonzero(x == y)
return count == 0, count / x.size
Given a numpy array (let it be a bit array for simplicity), how can I construct a new array of the same shape where 1 stands exactly at the positions where in the original array there was a zero, preceded by at least N-1 consecutive zeros?
For example, what is the best way to implement function nzeros having two arguments, a numpy array and the minimal required number of consecutive zeros:
import numpy as np
a = np.array([0, 0, 0, 0, 1, 0, 0, 0, 1, 1])
b = nzeros(a, 3)
Function nzeros(a, 3) should return
array([0, 0, 1, 1, 0, 0, 0, 1, 0, 0])
Approach #1
We can use 1D convolution -
def nzeros(a, n):
# Define kernel for 1D convolution
k = np.ones(n,dtype=int)
# Get sliding summations for zero matches with that kernel
s = np.convolve(a==0,k)
# Look for summations that are equal to n value, which will occur for
# n consecutive 0s. Remember that we are using a "full" version of
# convolution, so there's one-off offsetting because of the way kernel
# slides across input data. Also, we need to create 1s at places where
# n consective 0s end, so we would need to slice out ending elements.
# Thus, we would end up with the following after int dtype conversion
return (s==n).astype(int)[:-n+1]
Sample run -
In [46]: a
Out[46]: array([0, 0, 0, 0, 1, 0, 0, 0, 1, 1])
In [47]: nzeros(a,3)
Out[47]: array([0, 0, 1, 1, 0, 0, 0, 1, 0, 0])
In [48]: nzeros(a,2)
Out[48]: array([0, 1, 1, 1, 0, 0, 1, 1, 0, 0])
Approach #2
Another way to solve and this could be considered as a variant of the 1D convolution approach, would be to use erosion, because if you look at the outputs, we can simply erode the mask of 0s from the starts until n-1 places. So, we can use scipy.ndimage.morphology's binary_erosion that also allow us to specify the portion of kernel center with its origin arg, hence we will avoid any slicing. The implementation would look something like this -
from scipy.ndimage.morphology import binary_erosion
out = binary_erosion(a==0,np.ones(n),origin=(n-1)//2).astype(int)
Using for loop:
def nzeros(a, n):
#Create a numpy array of zeros of length equal to n
b = np.zeros(n)
#Create a numpy array of zeros of same length as array a
c = np.zeros(len(a), dtype=int)
for i in range(0,len(a) - n):
if (b == a[i : i+n]).all(): #Check if array b is equal to slice in a
c[i+n-1] = 1
return c
Sample Output:
print(nzeros(a, 3))
[0 0 1 1 0 0 0 1 0 0]
So lets say I have a an array that looks similar to this :
array([[0, 0, 0, 0, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 0, 0, 0, 0]])
I would like to return the location of the center of the largest sum of values within a certain n*n square. So in this case it would be (2,2) if n = 3. If I let n = 4 it would be the same result.
Does numpy have a method for finding this location?
Approach #1 : We can use SciPy's 2D convolution to get summations in sliding windows of shape (n,n) and choose the index of the window with the biggest sum with argmax and translate to row, col indices with np.unravel_index, like so -
from scipy.signal import convolve2d as conv2
def largest_sum_pos_app1(a, n):
idx = conv2(a, np.ones((n,n),dtype=int),'same').argmax()
return np.unravel_index(idx, a.shape)
Sample run -
In [558]: a
Out[558]:
array([[0, 0, 0, 0, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 1, 1, 1, 0],
[0, 0, 0, 0, 0]])
In [559]: largest_sum_pos_app1(a, n=3)
Out[559]: (2, 2)
Approach #1S (Super charged) : We can boost it further by using uniform filter, like so -
from scipy.ndimage.filters import uniform_filter as unif2D
def largest_sum_pos_app1_mod1(a, n):
idx = unif2D(a.astype(float),size=n, mode='constant').argmax()
return np.unravel_index(idx, a.shape)
Approach #2 : Another based on scikit-image's sliding window creating tool view_as_windows, we would create sliding windows of shape (n,n) to give us a 4D array with the last two axes of shape (n,n) corresponding to the search window size. So, we would sum along those two axes and get the argmax index and translate it to the actual row, col positions.
Hence, the implementation would be -
from skimage.util.shape import view_as_windows
def largest_sum_pos_app2(a, n):
h = (n-1)//2 # half window size
idx = view_as_windows(a, (n,n)).sum((-2,-1)).argmax()
return tuple(np.array(np.unravel_index(idx, np.array(a.shape)-n+1))+h)
As also mentioned in the comments, a search square with an even n would be confusing given that it won't have its center at any element coordinate.
Runtime test
In [741]: np.random.seed(0)
In [742]: a = np.random.randint(0,1000,(1000,1000))
In [743]: largest_sum_pos_app1(a, n= 5)
Out[743]: (966, 403)
In [744]: largest_sum_pos_app1_mod1(a, n= 5)
Out[744]: (966, 403)
In [745]: largest_sum_pos_app2(a, n= 5)
Out[745]: (966, 403)
In [746]: %timeit largest_sum_pos_app1(a, n= 5)
...: %timeit largest_sum_pos_app1_mod1(a, n= 5)
...: %timeit largest_sum_pos_app2(a, n= 5)
...:
10 loops, best of 3: 57.6 ms per loop
100 loops, best of 3: 10.1 ms per loop
10 loops, best of 3: 47.7 ms per loop
I want to find the minimum-sized 2-dimensional ndarray within an ndarray that contains all values meeting a condition.
For example:
Let's say I have the array
x = np.array([[1, 1, 5, 3, 11, 1],
[1, 2, 15, 19, 21, 33],
[1, 8, 17, 22, 21, 31],
[3, 5, 6, 11, 23, 19]])
and call f(x, x % 2 == 0)
Then the return value of the program would be the array
[[2, 15, 19]
[8, 17, 22]
[5, 6, 11]]
Because it is the smallest rectangular array that includes all the even numbers (the condition).
I've found a way to find all the indices for which the condition is true by using np.argwhere and then slicing from the minimum to maximum indices from the original array, and I've done it using a for loop but I was wondering if there was a more efficient way to do it using numpy or scipy.
My current method:
def f(arr, cond_arr):
indices = np.argwhere(cond_arr)
min = np.amin(indices, axis = 0) #get first row, col meeting cond
max = np.amax(indices, axis = 0) #get last row, col meeting cond
return arr[min[0]:max[0] + 1, min[1] : max[1] + 1]
The function is pretty efficient already - but you can do better.
Instead of checking the condition for every row/column and then finding the minimum and maximum, we can collapse the condition into each axis (using reduction with the logical OR) and find the first/last indices:
def f2(arr, cond_arr):
c0 = np.where(np.logical_or.reduce(cond_arr, axis=0))[0]
c1 = np.where(np.logical_or.reduce(cond_arr, axis=1))[0]
return arr[c0[0]:c0[-1] + 1, c1[0]:c1[-1] + 1]
How it works:
With the example data cond_array looks like this:
>>> (x%2==0).astype(int)
array([[0, 0, 0, 0, 0, 0],
[0, 1, 0, 0, 0, 0],
[0, 1, 0, 1, 0, 0],
[0, 0, 1, 0, 0, 0]])
This are the column conditions:
>>> np.logical_or.reduce(cond_arr, axis=0).astype(int)
array([0, 1, 1, 1, 0, 0])
And this the row conditions:
>>> np.logical_or.reduce(cond_arr, axis=).astype(int)
array([0, 1, 1, 1])
Now we only need to find the first/last nonzero element for each of the two arrays.
Is it really faster?
%timeit f(x, x%2 == 0) # 10000 loops, best of 3: 24.6 µs per loop
%timeit f2(x, x%2 == 0) # 100000 loops, best of 3: 12.6 µs per loop
Well, a little bit... but it really shines with larger arrays:
x = np.random.randn(1000, 1000)
c = np.zeros((1000, 1000), dtype=bool)
c[400:600, 400:600] = True
%timeit f(x,c) # 100 loops, best of 3: 5.28 ms per loop
%timeit f2(x,c) # 1000 loops, best of 3: 225 µs per loop
Finally, this version has slightly more overhead but is generic over the number of dimensions:
def f3(arr, cond_arr):
s = []
for a in range(arr.ndim):
c = np.where(np.logical_or.reduce(cond_arr, axis=a))[0]
s.append(slice(c[0], c[-1] + 1))
return arr[s]