I know that I can use something like string[3:4] to get a substring in Python, but what does the 3 mean in somesequence[::3]?
it means 'nothing for the first argument, nothing for the second, and jump by three'. It gets every third item of the sequence sliced.
Extended slices is what you want. New in Python 2.3
Python sequence slice addresses can be written as a[start:end:step] and any of start, stop or end can be dropped. a[::3] is every third element of the sequence.
seq[::n] is a sequence of each n-th item in the entire sequence.
Example:
>>> range(10)[::2]
[0, 2, 4, 6, 8]
The syntax is:
seq[start:end:step]
So you can do (in Python 2):
>>> range(100)[5:18:2]
[5, 7, 9, 11, 13, 15, 17]
Explanation
s[i:j:k] is, according to the documentation, "slice of s from i to j with step k". When i and j are absent, the whole sequence is assumed and thus s[::k] means "every k-th item".
Examples
First, let's initialize a list:
>>> s = range(20)
>>> s
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
Let's take every 3rd item from s:
>>> s[::3]
[0, 3, 6, 9, 12, 15, 18]
Let's take every 3rd item from s[2:]:
>>> s[2:]
[2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
>>> s[2::3]
[2, 5, 8, 11, 14, 17]
Let's take every 3rd item from s[5:12]:
>>> s[5:12]
[5, 6, 7, 8, 9, 10, 11]
>>> s[5:12:3]
[5, 8, 11]
Let's take every 3rd item from s[:10]:
>>> s[:10]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> s[:10:3]
[0, 3, 6, 9]
TL;DR
This visual example will show you how to a neatly select elements in a NumPy Matrix (2 dimensional array) in a pretty entertaining way (I promise). Step 2 below illustrate the usage of that "double colons" :: in question.
(Caution: this is a NumPy array specific example with the aim of illustrating the a use case of "double colons" :: for jumping of elements in multiple axes. This example does not cover native Python data structures like List).
One concrete example to rule them all...
Say we have a NumPy matrix that looks like this:
In [1]: import numpy as np
In [2]: X = np.arange(100).reshape(10,10)
In [3]: X
Out[3]:
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]])
Say for some reason, your boss wants you to select the following elements:
"But How???"... Read on! (We can do this in a 2-step approach)
Step 1 - Obtain subset
Specify the "start index" and "end index" in both row-wise and column-wise directions.
In code:
In [5]: X2 = X[2:9,3:8]
In [6]: X2
Out[6]:
array([[23, 24, 25, 26, 27],
[33, 34, 35, 36, 37],
[43, 44, 45, 46, 47],
[53, 54, 55, 56, 57],
[63, 64, 65, 66, 67],
[73, 74, 75, 76, 77],
[83, 84, 85, 86, 87]])
Notice now we've just obtained our subset, with the use of simple start and end indexing technique. Next up, how to do that "jumping"... (read on!)
Step 2 - Select elements (with the "jump step" argument)
We can now specify the "jump steps" in both row-wise and column-wise directions (to select elements in a "jumping" way) like this:
In code (note the double colons):
In [7]: X3 = X2[::3, ::2]
In [8]: X3
Out[8]:
array([[23, 25, 27],
[53, 55, 57],
[83, 85, 87]])
We have just selected all the elements as required! :)
Consolidate Step 1 (start and end) and Step 2 ("jumping")
Now we know the concept, we can easily combine step 1 and step 2 into one consolidated step - for compactness:
In [9]: X4 = X[2:9,3:8][::3,::2]
In [10]: X4
Out[10]:
array([[23, 25, 27],
[53, 55, 57],
[83, 85, 87]])
Done!
When slicing in Python the third parameter is the step. As others mentioned, see Extended Slices for a nice overview.
With this knowledge, [::3] just means that you have not specified any start or end indices for your slice. Since you have specified a step, 3, this will take every third entry of something starting at the first index. For example:
>>> '123123123'[::3]
'111'
remember that the foundations is what a[start:end:step] means. From there you can get a[1::2] get every odd index, a[::2] get every even, a[2::2] get every even starting at 2, a[2:4:2] get every even starting at 2 and ending at 4. Inspired by https://stackoverflow.com/a/3453102/1601580
You can also use this notation in your own custom classes to make it do whatever you want
class C(object):
def __getitem__(self, k):
return k
# Single argument is passed directly.
assert C()[0] == 0
# Multiple indices generate a tuple.
assert C()[0, 1] == (0, 1)
# Slice notation generates a slice object.
assert C()[1:2:3] == slice(1, 2, 3)
# If you omit any part of the slice notation, it becomes None.
assert C()[:] == slice(None, None, None)
assert C()[::] == slice(None, None, None)
assert C()[1::] == slice(1, None, None)
assert C()[:2:] == slice(None, 2, None)
assert C()[::3] == slice(None, None, 3)
# Tuple with a slice object:
assert C()[:, 1] == (slice(None, None, None), 1)
# Ellipsis class object.
assert C()[...] == Ellipsis
We can then open up slice objects as:
s = slice(1, 2, 3)
assert s.start == 1
assert s.stop == 2
assert s.step == 3
This is notably used in Numpy to slice multi-dimensional arrays in any direction.
Of course, any sane API should use ::3 with the usual "every 3" semantic.
The related Ellipsis is covered further at: What does the Ellipsis object do?
The third parameter is the step. So [::3] would return every 3rd element of the list/string.
Did I miss or nobody mentioned reversing with [::-1] here?
# Operating System List
systems = ['Windows', 'macOS', 'Linux']
print('Original List:', systems)
# Reversing a list
#Syntax: reversed_list = systems[start:stop:step]
reversed_list = systems[::-1]
# updated list
print('Updated List:', reversed_list)
source:
https://www.programiz.com/python-programming/methods/list/reverse
Python uses the :: to separate the End, the Start, and the Step value.
Related
If I have a large 2D numpy array and 2 arrays which correspond to the x and y indices I want to extract, It's easy enough:
h = np.arange(49).reshape(7,7)
# h = [[0, 1, 2, 3, 4, 5, 6],
# [7, 8, 9, 10, 11, 12, 13],
# [14, 15, 16, 17, 18, 19, 20],
# [21, 22, 23, 24, 25, 26, 27],
# [28, 29, 30, 31, 32, 33, 34],
# [35, 36, 37, 38, 39, 40, 41],
# [42, 43, 44, 45, 46, 47, 48]]
x_indices = np.array([1,3,4])
y_indices = np.array([2,3,5])
reduced_h = h[x_indices, y_indices]
#reduced_h = [ 9, 24, 33]
However, I would like to, for each x,y pair cut out a square (denoted by 'a' - the number of indices in each direction from the centre) surrounding this 'coordinate' and return an array of these little 2D arrays.
For example, for h, x,y_indices as above and a=1:
reduced_h = [[[1,2,3],[8,9,10],[15,16,17]], [[16,17,18],[23,24,25],[30,31,32]], [[25,26,27],[32,33,34],[39,40,41]]]
i.e one 3x3 array for each x-y index pair corresponding to the 3x3 square of elements centred on the x-y index. In general, this should return a numpy array which has shape (len(x_indices),2a+1, 2a+1)
By analogy to reduced_h[0] = h[x_indices[0]-1:x_indices[0]+1 , y_indices[0]-1:y_indices[0]+1] = h[1-1:1+1 , 2-1:2+1] = h[0:2, 1:3] my first try was the following:
h[x_indices-a : x_indices+a, y_indices-a : y_indices+a]
However, perhaps unsurprisingly, slicing between the arrays fails.
So the obvious next thing to try is to create this slice manually. np.arange seems to struggle with this but linspace works:
a=1
xrange = np.linspace(x_indices-a, x_indices+a, 2*a+1, dtype=int)
# xrange = [ [0, 2, 3], [1, 3, 4], [2, 4, 5] ]
yrange = np.linspace(y_indices-a, y_indices+a, 2*a+1, dtype=int)
Now can try h[xrange,yrange] but this unsurprisingly does this element-wise meaning I get only one (2a+1)x(2a+1) array (the same dimensions as xrange and yrange). It there a way to, for every index, take the right slices from these ranges (without loops)? Or is there a way to make the broadcast work initially without having to set up linspace explicitly? Thanks
You can index np.lib.stride_tricks.sliding_window_view using your x and y indices:
import numpy as np
h = np.arange(49).reshape(7,7)
x_indices = np.array([1,3,4])
y_indices = np.array([2,3,5])
a = 1
window = (2*a+1, 2*a+1)
out = np.lib.stride_tricks.sliding_window_view(h, window)[x_indices-a, y_indices-a]
out:
array([[[ 1, 2, 3],
[ 8, 9, 10],
[15, 16, 17]],
[[16, 17, 18],
[23, 24, 25],
[30, 31, 32]],
[[25, 26, 27],
[32, 33, 34],
[39, 40, 41]]])
Note that you may need to pad h first to handle windows around your coordinates that reach "outside" h.
I have a list of integers and I want to perform operations like addition, multiplication, floor division on every element of list slice (sub array) or at certain indexes (eg. range(start, end, jump) ) efficiently. The number being added or multiplied by each element of list slice is constant (say 'k').
For example:
nums = [23, 44, 65, 78, 87, 11, 33, 44, 3]
for i in range(2, 7, 2):
nums[i] //= 2 # here 2 is the constant 'k'
print(nums)
>>> [23, 44, 32, 78, 43, 11, 16, 44, 3]
I have to perform these operations several times on different slices/ranges and the constant 'k' varies for different slices/ranges. The obvious way to do this is to run a for loop and modify the value of elements, but that isn't fast enough. You can do this efficiently by using a numpy array because it supports bulk assignment/modification but I am looking for a way to do this in pure python.
One way to avoid the for loop is the following:
>>> nums = [23, 44, 65, 78, 87, 11, 33, 44, 3]
>>> nums[2:7:2] = [x//2 for x in nums[2:7:2]]
>>> nums
[23, 44, 32, 78, 43, 11, 16, 44, 3]
Is there a way to do multiple indexing in a numpy array as described below?
arr=np.array([55, 2, 3, 4, 5, 6, 7, 8, 9])
arr[np.arange(0,2):np.arange(5,7)]
output:
IndexError: too many indices for array
Desired output:
array([55,2,3,4,5],[2,3,4,5,6])
This problem might be similar to calculating a moving average over an array (but I want to do it without any function that is provided).
Here's an approach using strides -
start_index = np.arange(0,2)
L = 5 # Interval length
n = arr.strides[0]
strided = np.lib.stride_tricks.as_strided
out = strided(arr[start_index[0]:],shape=(len(start_index),L),strides=(n,n))
Sample run -
In [976]: arr
Out[976]: array([55, 52, 13, 64, 25, 76, 47, 18, 69, 88])
In [977]: start_index
Out[977]: array([2, 3, 4])
In [978]: L = 5
In [979]: out
Out[979]:
array([[13, 64, 25, 76, 47],
[64, 25, 76, 47, 18],
[25, 76, 47, 18, 69]])
I'm using a random.seed() to try and keep the random.sample() the same as I sample more values from a list and at some point the numbers change.....where I thought the one purpose of the seed() function was to keep the numbers the same.
Heres a test I did to prove it doesn't keep the same numbers.
import random
a=range(0,100)
random.seed(1)
a = random.sample(a,10)
print a
then change the sample much higher and the sequence will change(at least for me they always do):
a = random.sample(a,40)
print a
I'm sort of a newb so maybe this is an easy fix but I would appreciate any help on this.
Thanks!
If you were to draw independent samples from the generator, what would happen would be exactly what you're expecting:
In [1]: import random
In [2]: random.seed(1)
In [3]: [random.randint(0, 99) for _ in range(10)]
Out[3]: [13, 84, 76, 25, 49, 44, 65, 78, 9, 2]
In [4]: random.seed(1)
In [5]: [random.randint(0, 99) for _ in range(40)]
Out[5]: [13, 84, 76, 25, 49, 44, 65, 78, 9, 2, 83, 43 ...]
As you can see, the first ten numbers are indeed the same.
It is the fact that random.sample() is drawing samples without replacement that's getting in the way. To understand how these algorithms work, see Reservoir Sampling. In essence what happens is that later samples can push earlier samples out of the result set.
One alternative might be to shuffle a list of indices and then take either 10 or 40 first elements:
In [1]: import random
In [2]: a = range(0,100)
In [3]: random.shuffle(a)
In [4]: a[:10]
Out[4]: [48, 27, 28, 4, 67, 76, 98, 68, 35, 80]
In [5]: a[:40]
Out[5]: [48, 27, 28, 4, 67, 76, 98, 68, 35, 80, ...]
It seems that random.sample is deterministic only if the seed and sample size are kept constant. In other words, even if you reset the seed, generating a sample with a different length is not "the same" random operation, and may give a different initial subsequence than generating a smaller sample with the same seed. In other words, the same random numbers are being generated internally, but the way sample uses them to derive the random sequence is different depending on how large a sample you ask for.
You are assuming an implementation of random.sample something like this:
def samples(lst, k):
n = len(lst)
indices = []
while len(indices) < k:
index = random.randrange(n)
if index not in indices:
indices.append(index)
return [lst[i] for i in indices]
Which gives:
>>> random.seed(1)
>>> samples(list(range(20)), 5)
[4, 18, 2, 8, 3]
>>> random.seed(1)
>>> samples(list(range(20)), 10)
[4, 18, 2, 8, 3, 15, 14, 12, 6, 0]
However, that isn't how random.sample is actually implemented; seed does work how you think, it's sample that doesn't!
You simply need to re-seed it:
a = list(range(100))
random.seed(1) # seed first time
random.sample(a, 10)
>> [17, 72, 97, 8, 32, 15, 63, 57, 60, 83]
random.seed(1) # seed second time with same value
random.sample(a, 40)
>> [17, 72, 97, 8, 32, 15, 63, 57, 60, 83, 48, 26, 12, 62, 3, 49, 55, 77, 0, 92, 34, 29, 75, 13, 40, 85, 2, 74, 69, 1, 89, 27, 54, 98, 28, 56, 93, 35, 14, 22]
But in your case you're using a generator, not a list, so after sampling the first time a will shrink (from 100 to 90), and you will lose the elements that you had sampled, so it won't work. So just use a list and seed before every sampling.
I have a numpy array of numbers, for example,
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
I would like to find all the indexes of the elements within a specific range. For instance, if the range is (6, 10), the answer should be (3, 4, 5). Is there a built-in function to do this?
You can use np.where to get indices and np.logical_and to set two conditions:
import numpy as np
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
np.where(np.logical_and(a>=6, a<=10))
# returns (array([3, 4, 5]),)
As in #deinonychusaur's reply, but even more compact:
In [7]: np.where((a >= 6) & (a <=10))
Out[7]: (array([3, 4, 5]),)
Summary of the answers
For understanding what is the best answer we can do some timing using the different solution.
Unfortunately, the question was not well-posed so there are answers to different questions, here I try to point the answer to the same question. Given the array:
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
The answer should be the indexes of the elements between a certain range, we assume inclusive, in this case, 6 and 10.
answer = (3, 4, 5)
Corresponding to the values 6,9,10.
To test the best answer we can use this code.
import timeit
setup = """
import numpy as np
import numexpr as ne
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
# or test it with an array of the similar size
# a = np.random.rand(100)*23 # change the number to the an estimate of your array size.
# we define the left and right limit
ll = 6
rl = 10
def sorted_slice(a,l,r):
start = np.searchsorted(a, l, 'left')
end = np.searchsorted(a, r, 'right')
return np.arange(start,end)
"""
functions = ['sorted_slice(a,ll,rl)', # works only for sorted values
'np.where(np.logical_and(a>=ll, a<=rl))[0]',
'np.where((a >= ll) & (a <=rl))[0]',
'np.where((a>=ll)*(a<=rl))[0]',
'np.where(np.vectorize(lambda x: ll <= x <= rl)(a))[0]',
'np.argwhere((a>=ll) & (a<=rl)).T[0]', # we traspose for getting a single row
'np.where(ne.evaluate("(ll <= a) & (a <= rl)"))[0]',]
functions2 = [
'a[np.logical_and(a>=ll, a<=rl)]',
'a[(a>=ll) & (a<=rl)]',
'a[(a>=ll)*(a<=rl)]',
'a[np.vectorize(lambda x: ll <= x <= rl)(a)]',
'a[ne.evaluate("(ll <= a) & (a <= rl)")]',
]
rdict = {}
for i in functions:
rdict[i] = timeit.timeit(i,setup=setup,number=1000)
print("%s -> %s s" %(i,rdict[i]))
print("Sorted:")
for w in sorted(rdict, key=rdict.get):
print(w, rdict[w])
Results
The results are reported in the following plot for a small array (on the top the fastest solution) as noted by #EZLearner they may vary depending on the size of the array. sorted slice could be faster for larger arrays, but it requires your array to be sorted, for arrays with over 10 M of entries ne.evaluate could be an option. Is hence always better to perform this test with an array of the same size as yours:
If instead of the indexes you want to extract the values you can perform the tests using functions2 but the results are almost the same.
I thought I would add this because the a in the example you gave is sorted:
import numpy as np
a = [1, 3, 5, 6, 9, 10, 14, 15, 56]
start = np.searchsorted(a, 6, 'left')
end = np.searchsorted(a, 10, 'right')
rng = np.arange(start, end)
rng
# array([3, 4, 5])
a = np.array([1,2,3,4,5,6,7,8,9])
b = a[(a>2) & (a<8)]
Other way is with:
np.vectorize(lambda x: 6 <= x <= 10)(a)
which returns:
array([False, False, False, True, True, True, False, False, False])
It is sometimes useful for masking time series, vectors, etc.
This code snippet returns all the numbers in a numpy array between two values:
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56] )
a[(a>6)*(a<10)]
It works as following:
(a>6) returns a numpy array with True (1) and False (0), so does (a<10). By multiplying these two together you get an array with either a True, if both statements are True (because 1x1 = 1) or False (because 0x0 = 0 and 1x0 = 0).
The part a[...] returns all values of array a where the array between brackets returns a True statement.
Of course you can make this more complicated by saying for instance
...*(1-a<10)
which is similar to an "and Not" statement.
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
np.argwhere((a>=6) & (a<=10))
Wanted to add numexpr into the mix:
import numpy as np
import numexpr as ne
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
np.where(ne.evaluate("(6 <= a) & (a <= 10)"))[0]
# array([3, 4, 5], dtype=int64)
Would only make sense for larger arrays with millions... or if you hitting a memory limits.
This may not be the prettiest, but works for any dimension
a = np.array([[-1,2], [1,5], [6,7], [5,2], [3,4], [0, 0], [-1,-1]])
ranges = (0,4), (0,4)
def conditionRange(X : np.ndarray, ranges : list) -> np.ndarray:
idx = set()
for column, r in enumerate(ranges):
tmp = np.where(np.logical_and(X[:, column] >= r[0], X[:, column] <= r[1]))[0]
if idx:
idx = idx & set(tmp)
else:
idx = set(tmp)
idx = np.array(list(idx))
return X[idx, :]
b = conditionRange(a, ranges)
print(b)
s=[52, 33, 70, 39, 57, 59, 7, 2, 46, 69, 11, 74, 58, 60, 63, 43, 75, 92, 65, 19, 1, 79, 22, 38, 26, 3, 66, 88, 9, 15, 28, 44, 67, 87, 21, 49, 85, 32, 89, 77, 47, 93, 35, 12, 73, 76, 50, 45, 5, 29, 97, 94, 95, 56, 48, 71, 54, 55, 51, 23, 84, 80, 62, 30, 13, 34]
dic={}
for i in range(0,len(s),10):
dic[i,i+10]=list(filter(lambda x:((x>=i)&(x<i+10)),s))
print(dic)
for keys,values in dic.items():
print(keys)
print(values)
Output:
(0, 10)
[7, 2, 1, 3, 9, 5]
(20, 30)
[22, 26, 28, 21, 29, 23]
(30, 40)
[33, 39, 38, 32, 35, 30, 34]
(10, 20)
[11, 19, 15, 12, 13]
(40, 50)
[46, 43, 44, 49, 47, 45, 48]
(60, 70)
[69, 60, 63, 65, 66, 67, 62]
(50, 60)
[52, 57, 59, 58, 50, 56, 54, 55, 51]
You can use np.clip() to achieve the same:
a = [1, 3, 5, 6, 9, 10, 14, 15, 56]
np.clip(a,6,10)
However, it holds the values less than and greater than 6 and 10 respectively.