Numpy has а repeat function, that repeats each element of the array a given (per element) number of times.
I want to implement a function that does similar thing but repeats not individual elements, but variably sized blocks of consecutive elements. Essentially I want the following function:
import numpy as np
def repeat_blocks(a, sizes, repeats):
b = []
start = 0
for i, size in enumerate(sizes):
end = start + size
b.extend([a[start:end]] * repeats[i])
start = end
return np.concatenate(b)
For example, given
a = np.arange(20)
sizes = np.array([3, 5, 2, 6, 4])
repeats = np.array([2, 3, 2, 1, 3])
then
repeat_blocks(a, sizes, repeats)
returns
array([ 0, 1, 2,
0, 1, 2,
3, 4, 5, 6, 7,
3, 4, 5, 6, 7,
3, 4, 5, 6, 7,
8, 9,
8, 9,
10, 11, 12, 13, 14, 15,
16, 17, 18, 19,
16, 17, 18, 19,
16, 17, 18, 19 ])
I want to push these loops into numpy in the name of performance. Is this possible? If so, how?
Here's one vectorized approach using cumsum -
# Get repeats for each group using group lengths/sizes
r1 = np.repeat(np.arange(len(sizes)), repeats)
# Get total size of output array, as needed to initialize output indexing array
N = (sizes*repeats).sum() # or np.dot(sizes, repeats)
# Initialize indexing array with ones as we need to setup incremental indexing
# within each group when cumulatively summed at the final stage.
# Two steps here:
# 1. Within each group, we have multiple sequences, so setup the offsetting
# at each sequence lengths by the seq. lengths preceeeding those.
id_ar = np.ones(N, dtype=int)
id_ar[0] = 0
insert_index = sizes[r1[:-1]].cumsum()
insert_val = (1-sizes)[r1[:-1]]
# 2. For each group, make sure the indexing starts from the next group's
# first element. So, simply assign 1s there.
insert_val[r1[1:] != r1[:-1]] = 1
# Assign index-offseting values
id_ar[insert_index] = insert_val
# Finally index into input array for the group repeated o/p
out = a[id_ar.cumsum()]
This function is a great candidate to speed up using Numba:
#numba.njit
def repeat_blocks_jit(a, sizes, repeats):
out = np.empty((sizes * repeats).sum(), a.dtype)
start = 0
oi = 0
for i, size in enumerate(sizes):
end = start + size
for rep in range(repeats[i]):
oe = oi + size
out[oi:oe] = a[start:end]
oi = oe
start = end
return out
This is significantly faster than Divakar's pure NumPy solution, and a lot closer to your original code. I made no effort at all to optimize it. Note that np.dot() and np.repeat() can't be used here, but that doesn't matter when all the code gets compiled.
Plus, since it is njit meaning "nopython" mode, you can even use #numba.njit(nogil=True) and get multicore speedup if you have many of these calls to make.
Related
How do I append values in an empty numpy array? I have an array of values that I want to perform a mathematical equation on and from those values I want to append it to a new empty array. What I have is:
qa = np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20])
values_a = np.array([])
for n in qa:
value = np.math.factorial(n + 19)/(np.math.factorial(n)*np.math.factorial(19))
values_array = np.append(values_a, value)
However, it isn't appending and not sure why?
You're missing the array name, you're using values_array instead of values_a. Here's a code that works using your logic:
qa = np.arange(21)
values_a = np.array([])
for n in qa:
value = np.math.factorial(n + 19)/(np.math.factorial(n)*np.math.factorial(19))
values_a = np.append(values_a, value)
Nevertheless, as some comments say, it is not recommended to use numpy.append inside a loop, so you can use scipy.special.factorial instead. It allows calculating the factorial of the whole array element-wise:
from scipy.special import factorial
factorial(qa + 19)/(factorial(qa) * factorial(19))
You are assigning the result of np.append to values_array, but next time through your loop you are overwriting that assignment. Instead, you want to assign the results of np.append into values_a so that you are building up values_a each successive iteration:
values_a = np.array([])
for n in qa:
value = np.math.factorial(n + 19)/(np.math.factorial(n)*np.math.factorial(19))
values_a = np.append(values_a, value)
Suppose I have a numpy array such as:
a = np.arange(9)
>> array([0, 1, 2, 3, 4, 5, 6, 7, 8])
If I want to raise each element to succeeding powers of two, I can do it this way:
power_2 = np.power(a,2)
power_4 = np.power(a,4)
Then I can combine the arrays by:
np.c_[power_2,power_4]
>> array([[ 0, 0],
[ 1, 1],
[ 4, 16],
[ 9, 81],
[ 16, 256],
[ 25, 625],
[ 36, 1296],
[ 49, 2401],
[ 64, 4096]])
What's an efficient way to do this if I don't know the degree of the even monomial (highest multiple of 2) in advance?
One thing to observe is that x^(2^n) = (...(((x^2)^2)^2)...^2)
meaning that you can compute each column from the previous by taking the square.
If you know the number of columns in advance you can do something like:
import functools as ft
a = np.arange(5)
n = 4
out = np.empty((*a.shape,n),a.dtype)
out[:,0] = a
# Note: this works by side-effect!
# The optional second argument of np.square is "out", i.e. an
# array to write the result to (nonetheless the result is also
# returned directly)
ft.reduce(np.square,out.T)
out
# array([[ 0, 0, 0, 0],
# [ 1, 1, 1, 1],
# [ 2, 4, 16, 256],
# [ 3, 9, 81, 6561],
# [ 4, 16, 256, 65536]])
If the number of columns is not known in advance then the most efficient method is to make a list of columns, append as needed and only in the end use np.column_stack or np.c_ (if using np.c_ do not forget to cast the list to tuple first).
The straightforward approach is:
exponents = [2**n for n in a]
[a**e for e in exponents]
This works fine for relatively small numbers, but I see what looks like numerical overflow on the larger numbers. (Although I can compute those high powers just fine using scalars.)
The most elegant way I could think of is to not calculate the exponents beforehand. Since your exponents follow a very easy pattern, you can express everything using on list-comprehension.
result = [item**2*index for index,item in enumerate(a)]
If you are working with quite large datasets, this will cause some serious overhead. This statement will do all calculations immediately and save all calculated element in one large array. To mitigate this problem, you could you a generator expression, which will generate the data on the fly.
result = (item**2*index for index,item in enumerate(a))
See here for more details.
I am trying to find elements from array(integer array) or list which are unique and those elements must not divisible by any other element from same array or list.
You can answer in any language like python, java, c, c++ etc.
I have tried this code in Python3 and it works perfectly but I am looking for better and optimum solution in terms of time complexity.
assuming array or list A is already sorted and having unique elements
A = [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
while i<len(A)-1:
while j<len(A):
if A[j]%A[i]==0:
A.pop(j)
else:
j+=1
i+=1
j=i+1
For the given array A=[2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] answer would be like ans=[2,3,5,7,11,13]
another example,A=[4,5,15,16,17,23,39] then ans would be like, ans=[4,5,17,23,39]
ans is having unique numbers
any element i from array only exists if (i%j)!=0, where i!=j
I think it's more natural to do it in reverse, by building a new list containing the answer instead of removing elements from the original list. If I'm thinking correctly, both approaches do the same number of mod operations, but you avoid the issue of removing an element from a list.
A = [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
ans = []
for x in A:
for y in ans:
if x % y == 0:
break
else: ans.append(x)
Edit: Promoting the completion else.
This algorithm will perform much faster:
A = [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]
if (A[-1]-A[0])/A[0] > len(A)*2:
result = list()
for v in A:
for f in result:
d,m = divmod(v,f)
if m == 0: v=0;break
if d<f: break
if v: result.append(v)
else:
retain = set(A)
minMult = 1
maxVal = A[-1]
for v in A:
if v not in retain : continue
minMult = v*2
if minMult > maxVal: break
if v*len(A)<maxVal:
retain.difference_update([m for m in retain if m >= minMult and m%v==0])
else:
retain.difference_update(range(minMult,maxVal,v))
if maxVal%v == 0:
maxVal = max(retain)
result = list(retain)
print(result) # [2, 3, 5, 7, 11, 13]
In the spirit of the sieve of Eratostenes, each number that is retained, removes its multiples from the remaining eligible numbers. Depending on the magnitude of the highest value, it is sometimes more efficient to exclude multiples than check for divisibility. The divisibility check takes several times longer for an equivalent number of factors to check.
At some point, when the data is widely spread out, assembling the result instead of removing multiples becomes faster (this last addition was inspired by Imperishable Night's post).
TEST RESULTS
A = [2,3,4,5,6,7,8,9,10,11,12,13,14,15,16] (100000 repetitions)
Original: 0.55 sec
New: 0.29 sec
A = list(range(2,5000))+[9697] (100 repetitions)
Original: 3.77 sec
New: 0.12 sec
A = list(range(1001,2000))+list(range(4000,6000))+[9697**2] (10 repetitions)
Original: 3.54 sec
New: 0.02 sec
I know that this is totally insane but i want to know what you think about this:
A = [4,5,15,16,17,23,39]
prova=[[x for x in A if x!=y and y%x==0] for y in A]
print([A[idx] for idx,x in enumerate(prova) if len(prova[idx])==0])
And i think it's still O(n^2)
If you care about speed more than algorithmic efficiency, numpy would be the package to use here in python:
import numpy as np
# Note: doesn't have to be sorted
a = [2, 2, 3, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 16, 29, 29]
a = np.unique(a)
result = a[np.all((a % a[:, None] + np.diag(a)), axis=0)]
# array([2, 3, 5, 7, 11, 13, 29])
This divides all elements by all other elements and stores the remainder in a matrix, checks which columns contain only non-0 values (other than the diagonal), and selects all elements corresponding to those columns.
This is O(n*M) where M is the max size of an integer in your list. The integers are all assumed to be none negative. This also assumes your input list is sorted (came to that assumption since all lists you provided are sorted).
a = [4, 7, 7, 8]
# a = [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
# a = [4, 5, 15, 16, 17, 23, 39]
M = max(a)
used = set()
final_list = []
for e in a:
if e in used:
continue
else:
used.add(e)
for i in range(e, M + 1):
if not (i % e):
used.add(i)
final_list.append(e)
print(final_list)
Maybe this can be optimized even further...
If the list is not sorted then for the above method to work, one must sort it. The time complexity will then be O(nlogn + Mn) which equals to O(nlogn) when n >> M.
I'm trying to vectorize some element calculations but having difficulty doing so without creating list comprehensions for local information to global information. I was told that I can accomplish what I want to do using logical arrays, but so far the examples I've found has not been helpful. While yes I can accomplish this with list comprehensions, speed is a main concern with my code.
I have a set of values that indicate indices in the "global" calculation that should not be adjusted.
For example, these "fixed" indices are
1 2 6
If my global calculation has ten elements, I would be able to set all the "free" values by creating a list of the set of the global indices and subtracting the fixed indices.
free = list(set(range(len(global)) - set(fixed))
[0, 3, 4, 5, 7, 8, 9]
in the global calculation, I would be able to adjust the "free" elements as shown in the following code snippet
global = np.ones(10)
global[free] = global[free] * 10
which should produce:
global = [10, 1, 1, 10, 10, 10, 1, 10, 10, 10]
my "local" calculation is a subset of the global one, where the local map indicates the corresponding indices in the global calculation.
local_map = [4, 2, 1, 8, 6]
local_values = [40, 40, 40, 40, 40]
but I need the values associated with the local map to retain their order for calculation purposes.
What would the equivalent of global[free] be on the local level?
the desired output would be something like this:
local_free = list(set(range(len(local)) - set(fixed))
local_values[local_free] *= 10
OUTPUT: local_values = [400, 40, 40, 400, 40]
I apologize if the question formatting is off, the code block formatting doesn't seem to be working in my browser, so please let me know if you need clarification.
For such comparison-related operations, NumPy has tools like np.setdiff1d and np.in1d among others. To solve our case, these two would be enough. I would assume that the inputs are NumPy arrays, as then we could use vectorized indexing methods supported by NumPy.
On the first case, we have -
In [97]: fixed = np.array([1,2,6])
...: global_arr = np.array([10, 1, 1, 10, 10, 10, 1, 10, 10, 10])
...:
To get the equivalent of list(set(range(len(global_arr)) - set(fixed)) in NumPy, we could make use of np.setdiff1d -
In [98]: np.setdiff1d(np.arange(len(global_arr)),fixed)
Out[98]: array([0, 3, 4, 5, 7, 8, 9])
Next up, we have -
In [99]: local_map = np.array([4, 2, 1, 8, 6])
...: local_values = np.array([42, 40, 48, 41, 43])
...:
We were trying to get -
local_free = list(set(range(len(local)) - set(fixed))
local_values[local_free] *= 10
Here, we can use np.in1d to get a mask to be an equivalent for local_free that could be used to index and assign into local_values with NumPy's boolean-indexing method -
In [100]: local_free = ~np.in1d(local_map,fixed)
...: local_values[local_free] *= 10
...:
In [101]: local_values
Out[101]: array([420, 40, 48, 410, 43])
if I have the following:
[0, 1, 2, 3, 4, 5, 6...] how can I reorder the list (actually make a new copy of the list) and then fill it as such:
[0, 1, 2, 3, 4, 5, 10, 9, 8, 7, 6, 11, 12, 13...]
I.e. Every five iterations, the list starts to decrement, or increment. The reason I want to do this is that I have a list of objects, and I want to fill a new list with the objects in a different order.
One technique I tried is:
copied_icons = [{key:'Object1'}, {key:'Object2'}, {key:'Object3'}...]
reversed_copied_icons = copied_icons[::-1]
left_to_right = []
for h in range(17):
left_to_right.append(copied_icons[h])
for j in range(18, 35):
left_to_right.append(reversed_copied_icons[j])
for k in range(36, 53):
left_to_right.append(copied_icons[k])
for l in range(54, 71):
left_to_right.append(reversed_copied_icons[l])
But for some reason this returns the list out of order and duplicates some of the objects. I am wondering if there is a simpler way to alternating incrementing and decrementing while filling my list.
There are two problems with your approach:
You are reversing the entire list, not just that slice. Let's say the list is [1,2,3,4], and we want to reverse the second half, i.e. get [1,2,4,3]; with your approach, you would take the third and fourth element from the reversed list, [4,3,2,1], and end up with [1,2,2,1]
The to-index in a range is exclusive, thus by using range(17) and then range(18,35) and so forth, you are missing out on the elements at index 17, 35, and 53
You can use a loop for the different parts to be reversed, and then replace that slice of the list with the same slice in reverse order.
lst = list(range(20))
for start in range(5, len(lst), 10):
lst[start:start+5] = lst[start+4:start-1:-1]
Or this way, as pointed out in comments, which also gets rid of those nasty off-by-one indices:
for start in range(5, len(lst), 10):
lst[start:start+5] = reversed(lst[start:start+5])
Afterwards, lst is [0, 1, 2, 3, 4, 9, 8, 7, 6, 5, 10, 11, 12, 13, 14, 19, 18, 17, 16, 15].
Or, in case the intervals to be reversed are irregular (as it seems to be in your question):
reverse = [(3, 7), (12,17)]
for start, end in reverse:
lst[start:end] = reversed(lst[start:end])
This appears to accomplish your objective:
def foo(lst, n=5):
""" lst=list to be re-ordered, n=item count before reversal """
new_list = list()
direction = 1
start = 0
end = start
while start < len(lst):
# process through the list in steps of 'n',
# except use min for possible stub at end.
end = start + min(n, len(lst) - start) # i.e. start+5
# If direction is 1, append list to new list. Otherwise, append reversed list.
new_list[start:end] = lst[start:end][::direction]
direction *= -1 # Switch directions
start = end # Jump to new starting position.
return new_list
lst = np.arange(20).tolist()
foo(lst,5)
[0, 1, 2, 3, 4, 9, 8, 7, 6, 5, 10, 11, 12, 13, 14, 19, 18, 17, 16, 15]
If the direction *= -1 line where to be removed, the code would simply copy the existing list (lst) in chunks of size 'n', the number of items you'd like before reversing the list.
Just above where the direction is to be changed, the [::direction] will be either [::1] in which case the list will be sorted in regular order or else [::-1] in which case the list will be reversed for the chunk of size n which is being processed. The third argument when slicing a list is the 'step size' argument, so a step of -1 returns a copy of the list in reverse order.
In case there is a stub, i.e. your stub is 2 if your list has 22 elements but your 'n' is in steps of 5, then you need to adjust your step size 'n' so you don't go past the end of your list. The min(n, len(lst) - start) will ensure you don't go past the end of the list. Alternatively, and probably clearer, you could use end = min(start + n, len(lst)).