Related
I have N number of points, for example:
A = [2, 3]
B = [3, 4]
C = [3, 3]
.
.
.
And they're in an array like so:
arr = np.array([[2, 3], [3, 4], [3, 3]])
I need as output all pairwise distances in BFS (Breadth First Search) order to track which distance is which, like: A->B, A->C, B->C. For the above example data, the result would be [1.41, 1.0, 1.0].
EDIT: I have to accomplish it with numpy or core libraries.
If you can use it, SciPy has a function for this:
In [2]: from scipy.spatial.distance import pdist
In [3]: pdist(arr)
Out[3]: array([1.41421356, 1. , 1. ])
Here's a numpy-only solution (fair warning: it requires a lot of memory, unlike pdist)...
dists = np.triu(np.linalg.norm(arr - arr[:, None], axis=-1)).flatten()
dists = dists[dists != 0]
Demo:
In [4]: arr = np.array([[2, 3], [3, 4], [3, 3], [5, 2], [4, 5]])
In [5]: pdist(arr)
Out[5]:
array([1.41421356, 1. , 3.16227766, 2.82842712, 1. ,
2.82842712, 1.41421356, 2.23606798, 2.23606798, 3.16227766])
In [6]: dists = np.triu(np.linalg.norm(arr - arr[:, None], axis=-1)).flatten()
In [7]: dists = dists[dists != 0]
In [8]: dists
Out[8]:
array([1.41421356, 1. , 3.16227766, 2.82842712, 1. ,
2.82842712, 1.41421356, 2.23606798, 2.23606798, 3.16227766])
Timings (with the solution above wrapped in a function called triu):
In [9]: %timeit pdist(arr)
7.27 µs ± 738 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
In [10]: %timeit triu(arr)
25.5 µs ± 4.58 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
As an alternative method, but similar to ddejohn answer, we can use np.triu_indices which return just the upper triangular indices in the matrix, which may be more memory-efficient:
np.linalg.norm(arr - arr[:, None], axis=-1)[np.triu_indices(arr.shape[0], 1)]
This doesn't need additional modules like flattening and indexing. Its performance is similar to the aforementioned answer for large data (e.g. you can check it by arr = np.random.rand(10000, 2) on colab, which will be done near 4.6 s for both; It may beats the np.triu and flatten in larger data).
I have tested the memory usage one time by memory-profiler as follows, but it must be checked again if it be important in terms of memory usage (I'm not sure):
Update:
I have tried to limit the calculations just to the upper triangle, that speed the code up 2 to 3 times on the tested arrays. As array size grows, the performance difference between this loop and the previous methods by np.triu_indices or np.triu grows and be more obvious:
ind = np.arange(arr.shape[0] - 1)
sub_ind = ind + 1
result = np.zeros(sub_ind.sum())
j = 0
for i in range(ind.shape[0]):
result[j:j+ind[-1-i]+1] = np.linalg.norm(arr[ind[i]] - arr[sub_ind[i]:], axis=-1)
j += ind[-1-i]+1
Also, through this way, the memory consumption is reduced at least ~x4. So, this method made it possible to work on larger arrays and more quickly.
Benchmarks:
# arr = np.random.rand(100, 2)
100 loops, best of 5: 459 µs per loop (ddejohns --> np.triu & np.flatten)
100 loops, best of 5: 528 µs per loop (mine --> np.triu_indices)
100 loops, best of 5: 1.42 ms per loop (This method)
--------------------------------------
# arr = np.random.rand(1000, 2)
10 loops, best of 5: 49.9 ms per loop
10 loops, best of 5: 49.7 ms per loop
10 loops, best of 5: 30.4 ms per loop (~x1.7) The fastest
--------------------------------------
# arr = np.random.rand(10000, 2)
2 loops, best of 5: 4.56 s per loop
2 loops, best of 5: 4.6 s per loop
2 loops, best of 5: 1.85 s per loop (~x2.5) The fastest
Is there an efficient way to calculate sum of bits in each column over array in Python?
Example (Python 3.7 and Numpy 1.20.1):
Create numpy array with values 0 or 1
import numpy as np
array = np.array(
[
[1, 0, 1],
[1, 1, 1],
[0, 0, 1],
]
)
Compress size by np.packbits
pack_array = np.packbits(array, axis=1)
Expected result: sum of bits in each position (column) without np.unpackbits to get the same as array.sum(axis=0):
array([2, 1, 3])
I found just very slow solution:
dim = array.shape[1]
candidates = np.zeros((dim, dim)).astype(int)
np.fill_diagonal(candidates, 1)
pack_candidates = np.packbits(candidates, axis=1)
np.apply_along_axis(lambda c:np.sum((np.bitwise_and(pack_array, c) == c).all(axis=1)), 1, pack_candidates)
Using np.unpackbits can be problematic if the input array is big since the resulting array can be too big to fit in RAM, and even if it does fit in RAM, this would be far from being efficient since the huge array have to be written and read from the (slow) main memory. The same thing apply for CPU caches: smaller arrays can generally be computed faster. Moreover, np.unpackbits have a quite big overhead for small arrays.
AFAIK, this is not possible to do this operation very efficiently in Numpy while using a small amount of RAM (ie. using np.unpackbits, as pointed out by #mathfux). However, Numba can be used to speed up this computation, especially for small arrays. Here is the code:
#nb.njit('int32[::1](uint8[:,::1], int_)')
def bitSum(packed, m):
n = packed.shape[0]
assert packed.shape[1]*8-7 <= m <= packed.shape[1]*8
res = np.zeros(m, dtype=np.int32)
for i in range(n):
for j in range(m):
res[j] += bool(packed[i, j//8] & (128>>(j%8)))
return res
If you want a faster implementation, you can optimize the code by working on fixed-size tiles. However, this makes the code also more complex. Here is the resulting code:
#nb.njit('int32[::1](uint8[:,::1], int_)')
def bitSumOpt(packed, m):
n = packed.shape[0]
assert packed.shape[1]*8-7 <= m <= packed.shape[1]*8
res = np.zeros(m, dtype=np.int32)
for i in range(0, n, 4):
for j in range(0, m, 8):
if i+3 < n and j+7 < m:
# Highly-optimized 4x8 tile computation
k = j//8
b0, b1, b2, b3 = packed[i,k], packed[i+1,k], packed[i+2,k], packed[i+3,k]
for j2 in range(8):
shift = 7 - j2
mask = 1 << shift
res[j+j2] += ((b0 & mask) + (b1 & mask) + (b2 & mask) + (b3 & mask)) >> shift
else:
# Slow fallback computation
for i2 in range(i, min(i+4, n)):
for j2 in range(j, min(j+8, m)):
res[j2] += bool(packed[i2, j2//8] & (128>>(j2%8)))
return res
Here are performance results on my machine:
On the example array:
Initial code: 62.90 us (x1)
numpy_sumbits: 4.37 us (x14)
bitSumOpt: 0.84 us (x75)
bitSum: 0.77 us (x82)
On a random 2000x2000 array:
Initial code: 1203.8 ms (x1)
numpy_sumbits: 3.9 ms (x308)
bitSum: 2.7 ms (x446)
bitSumOpt: 1.5 ms (x802)
The memory footprint of the Numba implementations is much better too (at least 8 times smaller).
It seems there is no better option in numpy than numpy.unpackbits.
To be more clear, let's take another example:
array = np.array([[1, 0, 1, 0, 1, 1, 1, 0, 1],
[1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 1, 0, 0, 0, 0, 0, 0]])
pack_array = np.packbits(array, axis=1)
dim = array.shape[1]
Now, pack_array is calculated in this way:
[[1,0,1,0,1,1,1,0], [1,0,0,0,0,0,0,0]] -> [174, 128]
[[1,1,1,1,1,1,1,1], [1,0,0,0,0,0,0,0]] -> [255, 128]
[[0,0,1,0,0,0,0,0], [0,0,0,0,0,0,0,0]] -> [32, 0]
I've tested various algorithms and unpacking bits seems to be the fastest:
def numpy_sumbits(pack_array, dim):
out = np.unpackbits(pack_array, axis=1, count=dim)
arr = np.sum(out, axis=0)
return arr
def manual_sumbits(pack_array, dim):
arr = pack_array.copy()
out = np.empty((dim//8+1) * 8, dtype=int)
for i in range(8):
out[7 - i%8::8] = np.sum(arr % 2, axis=0)
arr = arr // 2
return out[:dim]
def numpy_sumshifts(pack_array, dim):
res = (pack_array.reshape(pack_array.size, -1) >> np.arange(8)) % 2
res = res.reshape(*pack_array.shape, 8)
return np.sum(res, axis=0)[:,::-1].ravel()[:dim]
print(numpy_unpackbits(pack_array, dim))
print(manual_unpackbits(pack_array, dim))
print(numpy_sumshifts(pack_array, dim))
>>>
[2 1 3 1 2 2 2 1 2]
[2 1 3 1 2 2 2 1 2]
[2 1 3 1 2 2 2 1 2]
%%timeit
numpy_sumbits(pack_array, dim)
>>> 3.49 ms ± 57.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
manual_sumbits(pack_array, dim)
>>> 10 ms ± 22.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
%%timeit
numpy_sumshifts(pack_array, dim)
>>> 20.1 ms ± 97.9 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
I'm looking for a numpy equivalent of my suboptimal Python code. The calculation I want to do can be summarized by:
The average of the peak of each section for each row.
Here the code with a sample array and list of indices. Sections can be of different sizes.
x = np.array([[1, 2, 3, 4],
[5, 6, 7, 8]])
indices = [2]
result = np.empty((1, x.shape[0]))
for row in x:
splited = np.array_split(row, indexes)
peak = [np.amax(a) for a in splited]
result[0, i] = np.average(peak)
Which gives: result = array([[3., 7.]])
What is the optimized numpy way to suppress both loop?
You could just take off the for loop and use axis instead:
result2 = np.mean([np.max(arr, 1) for arr in np.array_split(x_large, indices, 1)], axis=0)
Output:
array([3., 7.])
Benchmark:
x_large = np.array([[1, 2, 3, 4],
[5, 6, 7, 8]] * 1000)
%%timeit
result = []
for row in x_large:
splited = np.array_split(row, indices)
peak = [np.amax(a) for a in splited]
result.append(np.average(peak))
# 29.9 ms ± 177 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit np.mean([np.max(arr, 1) for arr in np.array_split(x_large, indices, 1)], axis=0)
# 37.4 µs ± 499 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
Validation:
np.array_equal(result, result2)
# True
I have multiple numpy arrays and I want to create new arrays doing something that is like an XOR ... but not quite.
My input is two arrays, array1 and array2.
My output is a modified (or new array, I don't really care) version of array1.
The modification is elementwise, by doing the following:
1.) If either array has 0 for the given index, then the index is left unchanged.
2.) If array1 and array2 are nonzero, then the modified array is assigned the value of array1's index subtracted by array2's index, down to a minimum of zero.
Examples:
array1: [0, 3, 8, 0]
array2: [1, 1, 1, 1]
output: [0, 2, 7, 0]
array1: [1, 1, 1, 1]
array2: [0, 3, 8, 0]
output: [1, 0, 0, 1]
array1: [10, 10, 10, 10]
array2: [8, 12, 8, 12]
output: [2, 0, 2, 0]
I would like to be able to do this with say, a single numpy.copyto statement, but I don't know how. Thank you.
edit:
it just hit me. could I do:
new_array = np.zeros(size_of_array1)
numpy.copyto(new_array, array1-array2, where=array1>array2)
Edit 2: Since I have received several answers very quickly I am going to time the different answers against each other to see how they do. Be back with results in a few minutes.
Okay, results are in:
array of random ints 0 to 5, size = 10,000, 10 loops
1.)using my np.copyto method
2.)using clip
3.)using maximum
0.000768184661865
0.000391960144043
0.000403165817261
Kasramvd also provided some useful timings below
You can use a simple subtraction and clipping the result with zero as the min:
(arr1 - arr2).clip(min=0)
Demo:
In [43]: arr1 = np.array([0,3,8,0]); arr2 = np.array([1,1,1,1])
In [44]: (arr1 - arr2).clip(min=0)
Out[44]: array([0, 2, 7, 0])
On large arrays it's also faster than maximum approach:
In [51]: arr1 = np.arange(10000); arr2 = np.arange(10000)
In [52]: %timeit np.maximum(0, arr1 - arr2)
22.3 µs ± 1.77 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [53]: %timeit (arr1 - arr2).clip(min=0)
20.9 µs ± 167 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [54]: arr1 = np.arange(100000); arr2 = np.arange(100000)
In [55]: %timeit np.maximum(0, arr1 - arr2)
671 µs ± 5.69 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
In [56]: %timeit (arr1 - arr2).clip(min=0)
648 µs ± 4.43 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
Note that if it's possible for arr2 to have negative values you should consider using an abs function on arr2 to get the expected result:
(arr1 - abs(arr2)).clip(min=0)
In [73]: np.maximum(0,np.array([0,3,8,0])-np.array([1,1,1,1]))
Out[73]: array([0, 2, 7, 0])
This doesn't explicitly address
If either array has 0 for the given index, then the index is left unchanged.
but the results match for all examples:
In [74]: np.maximum(0,np.array([1,1,1,1])-np.array([0,3,8,0]))
Out[74]: array([1, 0, 0, 1])
In [75]: np.maximum(0,np.array([10,10,10,10])-np.array([8,12,8,12]))
Out[75]: array([2, 0, 2, 0])
You can first simply subtract the arrays and then use boolean array indexing on the subtracted result to assign 0 where there are negative values as in:
# subtract
In [43]: subtracted = arr1 - arr2
# get a boolean mask by checking for < 0
# index into the array and assign 0
In [44]: subtracted[subtracted < 0] = 0
In [45]: subtracted
Out[45]: array([0, 2, 7, 0])
Applying the same for the other inputs specified by OP:
In [46]: arr1 = np.array([1, 1, 1, 1])
...: arr2 = np.array([0, 3, 8, 0])
In [47]: subtracted = arr1 - arr2
In [48]: subtracted[subtracted < 0] = 0
In [49]: subtracted
Out[49]: array([1, 0, 0, 1])
And for the third input arrays:
In [50]: arr1 = np.array([10, 10, 10, 10])
...: arr2 = np.array([8, 12, 8, 12])
In [51]: subtracted = arr1 - arr2
In [52]: subtracted[subtracted < 0] = 0
In [53]: subtracted
Out[53]: array([2, 0, 2, 0])
I want to create a list of points that would correspond to a grid. So if I want to create a grid of the region from (0, 0) to (1, 1), it would contain the points (0, 0), (0, 1), (1, 0) and (1, 0).
I know that that this can be done with the following code:
g = np.meshgrid([0,1],[0,1])
np.append(g[0].reshape(-1,1),g[1].reshape(-1,1),axis=1)
Yielding the result:
array([[0, 0],
[1, 0],
[0, 1],
[1, 1]])
My question is twofold:
Is there a better way of doing this?
Is there a way of generalizing this to higher dimensions?
I just noticed that the documentation in numpy provides an even faster way to do this:
X, Y = np.mgrid[xmin:xmax:100j, ymin:ymax:100j]
positions = np.vstack([X.ravel(), Y.ravel()])
This can easily be generalized to more dimensions using the linked meshgrid2 function and mapping 'ravel' to the resulting grid.
g = meshgrid2(x, y, z)
positions = np.vstack(map(np.ravel, g))
The result is about 35 times faster than the zip method for a 3D array with 1000 ticks on each axis.
Source: http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.gaussian_kde.html#scipy.stats.gaussian_kde
To compare the two methods consider the following sections of code:
Create the proverbial tick marks that will help to create the grid.
In [23]: import numpy as np
In [34]: from numpy import asarray
In [35]: x = np.random.rand(100,1)
In [36]: y = np.random.rand(100,1)
In [37]: z = np.random.rand(100,1)
Define the function that mgilson linked to for the meshgrid:
In [38]: def meshgrid2(*arrs):
....: arrs = tuple(reversed(arrs))
....: lens = map(len, arrs)
....: dim = len(arrs)
....: sz = 1
....: for s in lens:
....: sz *= s
....: ans = []
....: for i, arr in enumerate(arrs):
....: slc = [1]*dim
....: slc[i] = lens[i]
....: arr2 = asarray(arr).reshape(slc)
....: for j, sz in enumerate(lens):
....: if j != i:
....: arr2 = arr2.repeat(sz, axis=j)
....: ans.append(arr2)
....: return tuple(ans)
Create the grid and time the two functions.
In [39]: g = meshgrid2(x, y, z)
In [40]: %timeit pos = np.vstack(map(np.ravel, g)).T
100 loops, best of 3: 7.26 ms per loop
In [41]: %timeit zip(*(x.flat for x in g))
1 loops, best of 3: 264 ms per loop
Are your gridpoints always integral? If so, you could use numpy.ndindex
print list(np.ndindex(2,2))
Higher dimensions:
print list(np.ndindex(2,2,2))
Unfortunately, this does not meet the requirements of the OP since the integral assumption (starting with 0) is not met. I'll leave this answer only in case someone else is looking for the same thing where those assumptions are true.
Another way to do this relies on zip:
g = np.meshgrid([0,1],[0,1])
zip(*(x.flat for x in g))
This portion scales nicely to arbitrary dimensions. Unfortunately, np.meshgrid doesn't scale well to multiple dimensions, so that part will need to be worked out, or (assuming it works), you could use this SO answer to create your own ndmeshgrid function.
Yet another way to do it is:
np.indices((2,2)).T.reshape(-1,2)
Which can be generalized to higher dimensions, e.g.:
In [60]: np.indices((2,2,2)).T.reshape(-1,3)
Out[60]:
array([[0, 0, 0],
[1, 0, 0],
[0, 1, 0],
[1, 1, 0],
[0, 0, 1],
[1, 0, 1],
[0, 1, 1],
[1, 1, 1]])
To get the coordinates of a grid from 0 to 1, a reshape can do the work. Here are examples for 2D and 3D. Also works with floats.
grid_2D = np.mgrid[0:2:1, 0:2:1]
points_2D = grid_2D.reshape(2, -1).T
grid_3D = np.mgrid[0:2:1, 0:2:1, 0:2:1]
points_3D = grid_3D.reshape(3, -1).T
A simple example in 3D (can be extended to N-dimensions I guess, but beware of the final dimension and RAM usage):
import numpy as np
ndim = 3
xmin = 0.
ymin = 0.
zmin = 0.
length_x = 1000.
length_y = 1000.
length_z = 50.
step_x = 1.
step_y = 1.
step_z = 1.
x = np.arange(xmin, length_x, step_x)
y = np.arange(ymin, length_y, step_y)
z = np.arange(zmin, length_z, step_z)
%timeit xyz = np.array(np.meshgrid(x, y, z)).T.reshape(-1, ndim)
in: 2.76 s ± 185 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
which yields:
In [2]: xyx
Out[2]:
array([[ 0., 0., 0.],
[ 0., 1., 0.],
[ 0., 2., 0.],
...,
[999., 997., 49.],
[999., 998., 49.],
[999., 999., 49.]])
In [4]: xyz.shape
Out[4]: (50000000, 3)
Python 3.6.9
Numpy: 1.19.5
I am using the following to convert meshgrid to M X 2 array. Also changes the list of vectors to iterators can make it really fast.
import numpy as np
# Without iterators
x_vecs = [np.linspace(0,1,1000), np.linspace(0,1,1000)]
%timeit np.reshape(np.meshgrid(*x_vecs),(2,-1)).T
6.85 ms ± 93.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# With iterators
x_vecs = iter([np.linspace(0,1,1000), np.linspace(0,1,1000)])
%timeit np.reshape(np.meshgrid(*x_vecs),(2,-1)).T
5.78 µs ± 172 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
for N-D array using generator
vec_dim = 3
res = 100
# Without iterators
x_vecs = [np.linspace(0,1,res) for i in range(vec_dim)]
>>> %timeit np.reshape(np.meshgrid(*x_vecs),(vec_dim,-1)).T
11 ms ± 124 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# With iterators
x_vecs = (np.linspace(0,1,res) for i in range(vec_dim))
>>> %timeit np.reshape(np.meshgrid(*x_vecs),(vec_dim,-1)).T
5.54 µs ± 32.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)