I'm trying to vectorize some element calculations but having difficulty doing so without creating list comprehensions for local information to global information. I was told that I can accomplish what I want to do using logical arrays, but so far the examples I've found has not been helpful. While yes I can accomplish this with list comprehensions, speed is a main concern with my code.
I have a set of values that indicate indices in the "global" calculation that should not be adjusted.
For example, these "fixed" indices are
1 2 6
If my global calculation has ten elements, I would be able to set all the "free" values by creating a list of the set of the global indices and subtracting the fixed indices.
free = list(set(range(len(global)) - set(fixed))
[0, 3, 4, 5, 7, 8, 9]
in the global calculation, I would be able to adjust the "free" elements as shown in the following code snippet
global = np.ones(10)
global[free] = global[free] * 10
which should produce:
global = [10, 1, 1, 10, 10, 10, 1, 10, 10, 10]
my "local" calculation is a subset of the global one, where the local map indicates the corresponding indices in the global calculation.
local_map = [4, 2, 1, 8, 6]
local_values = [40, 40, 40, 40, 40]
but I need the values associated with the local map to retain their order for calculation purposes.
What would the equivalent of global[free] be on the local level?
the desired output would be something like this:
local_free = list(set(range(len(local)) - set(fixed))
local_values[local_free] *= 10
OUTPUT: local_values = [400, 40, 40, 400, 40]
I apologize if the question formatting is off, the code block formatting doesn't seem to be working in my browser, so please let me know if you need clarification.
For such comparison-related operations, NumPy has tools like np.setdiff1d and np.in1d among others. To solve our case, these two would be enough. I would assume that the inputs are NumPy arrays, as then we could use vectorized indexing methods supported by NumPy.
On the first case, we have -
In [97]: fixed = np.array([1,2,6])
...: global_arr = np.array([10, 1, 1, 10, 10, 10, 1, 10, 10, 10])
...:
To get the equivalent of list(set(range(len(global_arr)) - set(fixed)) in NumPy, we could make use of np.setdiff1d -
In [98]: np.setdiff1d(np.arange(len(global_arr)),fixed)
Out[98]: array([0, 3, 4, 5, 7, 8, 9])
Next up, we have -
In [99]: local_map = np.array([4, 2, 1, 8, 6])
...: local_values = np.array([42, 40, 48, 41, 43])
...:
We were trying to get -
local_free = list(set(range(len(local)) - set(fixed))
local_values[local_free] *= 10
Here, we can use np.in1d to get a mask to be an equivalent for local_free that could be used to index and assign into local_values with NumPy's boolean-indexing method -
In [100]: local_free = ~np.in1d(local_map,fixed)
...: local_values[local_free] *= 10
...:
In [101]: local_values
Out[101]: array([420, 40, 48, 410, 43])
Related
Suppose I have a numpy array such as:
a = np.arange(9)
>> array([0, 1, 2, 3, 4, 5, 6, 7, 8])
If I want to raise each element to succeeding powers of two, I can do it this way:
power_2 = np.power(a,2)
power_4 = np.power(a,4)
Then I can combine the arrays by:
np.c_[power_2,power_4]
>> array([[ 0, 0],
[ 1, 1],
[ 4, 16],
[ 9, 81],
[ 16, 256],
[ 25, 625],
[ 36, 1296],
[ 49, 2401],
[ 64, 4096]])
What's an efficient way to do this if I don't know the degree of the even monomial (highest multiple of 2) in advance?
One thing to observe is that x^(2^n) = (...(((x^2)^2)^2)...^2)
meaning that you can compute each column from the previous by taking the square.
If you know the number of columns in advance you can do something like:
import functools as ft
a = np.arange(5)
n = 4
out = np.empty((*a.shape,n),a.dtype)
out[:,0] = a
# Note: this works by side-effect!
# The optional second argument of np.square is "out", i.e. an
# array to write the result to (nonetheless the result is also
# returned directly)
ft.reduce(np.square,out.T)
out
# array([[ 0, 0, 0, 0],
# [ 1, 1, 1, 1],
# [ 2, 4, 16, 256],
# [ 3, 9, 81, 6561],
# [ 4, 16, 256, 65536]])
If the number of columns is not known in advance then the most efficient method is to make a list of columns, append as needed and only in the end use np.column_stack or np.c_ (if using np.c_ do not forget to cast the list to tuple first).
The straightforward approach is:
exponents = [2**n for n in a]
[a**e for e in exponents]
This works fine for relatively small numbers, but I see what looks like numerical overflow on the larger numbers. (Although I can compute those high powers just fine using scalars.)
The most elegant way I could think of is to not calculate the exponents beforehand. Since your exponents follow a very easy pattern, you can express everything using on list-comprehension.
result = [item**2*index for index,item in enumerate(a)]
If you are working with quite large datasets, this will cause some serious overhead. This statement will do all calculations immediately and save all calculated element in one large array. To mitigate this problem, you could you a generator expression, which will generate the data on the fly.
result = (item**2*index for index,item in enumerate(a))
See here for more details.
I need a fast way to keep a running maximum of a numpy array. For example, if my array was:
x = numpy.array([11,12,13,20,19,18,17,18,23,21])
I'd want:
numpy.array([11,12,13,20,20,20,20,20,23,23])
Obviously I could do this with a little loop:
def running_max(x):
result = [x[0]]
for val in x:
if val > result[-1]:
result.append(val)
else:
result.append(result[-1])
return result
But my arrays have hundreds of thousands of entries and I need to call this many times. It seems like there's got to be a numpy trick to remove the loop, but I can't seem to find anything that will work. The alternative will be to write this as a C extension, but it seems like I'd be reinventing the wheel.
numpy.maximum.accumulate works for me.
>>> import numpy
>>> numpy.maximum.accumulate(numpy.array([11,12,13,20,19,18,17,18,23,21]))
array([11, 12, 13, 20, 20, 20, 20, 20, 23, 23])
As suggested, there is scipy.maximum.accumulate:
In [9]: x
Out[9]: [1, 3, 2, 5, 4]
In [10]: scipy.maximum.accumulate(x)
Out[10]: array([1, 3, 3, 5, 5])
Numpy has а repeat function, that repeats each element of the array a given (per element) number of times.
I want to implement a function that does similar thing but repeats not individual elements, but variably sized blocks of consecutive elements. Essentially I want the following function:
import numpy as np
def repeat_blocks(a, sizes, repeats):
b = []
start = 0
for i, size in enumerate(sizes):
end = start + size
b.extend([a[start:end]] * repeats[i])
start = end
return np.concatenate(b)
For example, given
a = np.arange(20)
sizes = np.array([3, 5, 2, 6, 4])
repeats = np.array([2, 3, 2, 1, 3])
then
repeat_blocks(a, sizes, repeats)
returns
array([ 0, 1, 2,
0, 1, 2,
3, 4, 5, 6, 7,
3, 4, 5, 6, 7,
3, 4, 5, 6, 7,
8, 9,
8, 9,
10, 11, 12, 13, 14, 15,
16, 17, 18, 19,
16, 17, 18, 19,
16, 17, 18, 19 ])
I want to push these loops into numpy in the name of performance. Is this possible? If so, how?
Here's one vectorized approach using cumsum -
# Get repeats for each group using group lengths/sizes
r1 = np.repeat(np.arange(len(sizes)), repeats)
# Get total size of output array, as needed to initialize output indexing array
N = (sizes*repeats).sum() # or np.dot(sizes, repeats)
# Initialize indexing array with ones as we need to setup incremental indexing
# within each group when cumulatively summed at the final stage.
# Two steps here:
# 1. Within each group, we have multiple sequences, so setup the offsetting
# at each sequence lengths by the seq. lengths preceeeding those.
id_ar = np.ones(N, dtype=int)
id_ar[0] = 0
insert_index = sizes[r1[:-1]].cumsum()
insert_val = (1-sizes)[r1[:-1]]
# 2. For each group, make sure the indexing starts from the next group's
# first element. So, simply assign 1s there.
insert_val[r1[1:] != r1[:-1]] = 1
# Assign index-offseting values
id_ar[insert_index] = insert_val
# Finally index into input array for the group repeated o/p
out = a[id_ar.cumsum()]
This function is a great candidate to speed up using Numba:
#numba.njit
def repeat_blocks_jit(a, sizes, repeats):
out = np.empty((sizes * repeats).sum(), a.dtype)
start = 0
oi = 0
for i, size in enumerate(sizes):
end = start + size
for rep in range(repeats[i]):
oe = oi + size
out[oi:oe] = a[start:end]
oi = oe
start = end
return out
This is significantly faster than Divakar's pure NumPy solution, and a lot closer to your original code. I made no effort at all to optimize it. Note that np.dot() and np.repeat() can't be used here, but that doesn't matter when all the code gets compiled.
Plus, since it is njit meaning "nopython" mode, you can even use #numba.njit(nogil=True) and get multicore speedup if you have many of these calls to make.
i have a python numpy array with two rows. One row describes the start time of an event and the other one describes the end (here times as epoch integers). The code example means, that the event at index=0 starts at time=1 and ends at time=7.
start = [1, 8, 15, 30]
end = [7, 16, 20, 40]
timeranges = np.array([start,end])
I want to know, if the time ranges are intersected. That means i need a function/algorithm, that calculates the information, that the time range from 8 to 16 is intersected with the time range from 15 to 20.
My solution is, to use two intersected loops and check if any start time or end time is within an other timerange. But with ipython it lasts very long, because my timeranges are filled with nearly 10000 events.
Is there an elegant solution, to get the result in "short" time (e.g. below one minute)?
Store the data as a collection of (time,index_in_list,start_or_end). For example, if the input data is:
start = [1, 8, 15, 30]
end = [7, 16, 20, 40]
Transform the input data to a list of tuples as follows:
def extract_times(times,is_start):
return [(times[i],i,is_start) for i in range(len(times))]
Which yields:
extract_times(start,true) == [(1,0,true),(8,1,true),(15,2,true),(30,3,true)]
extract_times(end,false) == [(7,0,false),(16,1,false),(20,2,false),(40,3,false)]
Now, merge the two lists and store them.
Then, start traversing the lists from beginning to end, each time keeping track of the currently intersecting intervals, updating the state based on whether each new tuple is a beginning or and ending of an interval. This way you'll find all overlaps.
The complexity is O(n log(n)) for the sorting plus some overhead if there are lots of intersections.
Given that the input lists might not be sorted and to handle cases where we might see timeranges with multiple intersections, here's a brute-force comparison based method using broadcasting -
np.argwhere(np.triu(timeranges[1][:,None] > timeranges[0],1))
Sample runs
Original sample case :
In [81]: timeranges
Out[81]:
array([[ 1, 8, 15, 30],
[ 7, 16, 20, 40]])
In [82]: np.argwhere(np.triu(timeranges[1][:,None] > timeranges[0],1))
Out[82]: array([[1, 2]])
Multiple intersections case :
In [77]: timeranges
Out[77]:
array([[ 5, 7, 18, 12, 19],
[11, 17, 28, 19, 28]])
In [78]: np.argwhere(np.triu(timeranges[1][:,None] > timeranges[0],1))
Out[78]:
array([[0, 1],
[1, 3],
[2, 3],
[2, 4]])
If by within in "if any start time or end time is within an other timerange", you meant the boundaries are inclusive, change the comparison of > to >= in the solution code.
I am trying to generate a list to index coordinates (x, y and z), given a set of atom indices. My problem is quite simply how to elegantly go from this list:
atom_indices = [0, 4, 5, 8]
To this list:
coord_indices = [0, 1, 2, 12, 13, 14, 15, 16, 17, 24, 25, 26]
The easiest to read/understand way of doing this I've thought of so far is simply:
coord_indices = []
for atom in atom_indices:
coord_indices += [3 * atom,
3 * atom + 1,
3 * atom + 2]
But this doesn't seem very Pythonic. Is there a better way I haven't thought of without getting a list of lists or a list of tuples?
How about:
>>> atom_indices = [0, 4, 5, 8]
>>> coords = [3*a+k for a in atom_indices for k in range(3)]
>>> coords
[0, 1, 2, 12, 13, 14, 15, 16, 17, 24, 25, 26]
We can nest loops in list comprehensions in the same order we'd write the loops, i.e. this is basically
coords = []
for a in atom_indices:
for k in range(3):
coords.append(3*a+k)
Don't be afraid of for loops, though, if they're clearer in the situation. For reasons I've never fully understood, some people feel like they're being more clever when they write code horizontally instead of vertically, even though it makes it harder to debug.