Related
I have got a 1-d array where I have a pattern in the enteries. I will give an example. In the array arr, I have
first 4 enteries with single digits, next 4 enteries with two digits and then the next 6 enteries with 3 digits.
(This single, double, triple digit thing is just to highlight the pattern. The actual array have float numbers of similar values). The example 1-d array looks like:
import numpy as np
arr = np.array([1, 2, 3, 4, 11, 12, 13, 14, 111, 123, 132, 145, 176, 129,
6, 5, 3, 2, 21, 82, 53, 34, 121, 133, 139, 165, 186, 119])
Now, one complete pattern has total 4+4+6 = 14 enteries. This pattern (or repeating unit) is repeated several hundred thousand times so the length of my array is a multiple of 14 (14 * 2 = 28 in the example arr above).
Question:
I want to extract all the one digit enteries (first 4 numbers of one repeating unit), all the two digit enteries
(next 4 numbers of one repeating unit), and all the three digit enteries (next 6 numbers of one repeating unit).
This way I want to have my big arr splitted into three 1-d arrays. So the desired output is
arr1 = array([1, 2, 3, 4, 6, 5, 3, 2])
arr2 = array([11, 12, 13, 14, 21, 82, 53, 34])
arr3 = array([111, 123, 132, 145, 176, 129, 121, 133, 139, 165, 186, 119])
My idea
One way could be to simply reshape it into 2d array since I know the number of repetitions (=28/14 = 2 in the example arr) and then use indexing to get all the first chunks of 4, 4 and 6
and then concatenate.
arr = arr.reshape(2, 14)
and then use slicing to get the chunks as
arr1 = np.concatenate(arr[:, 0:4])
arr2 = np.concatenate(arr[:, 4:8])
arr3 = np.concatenate(arr[:, 8:])
print (arr1, arr2, arr3)
# array([1, 2, 3, 4, 6, 5, 3, 2]),
# array([11, 12, 13, 14, 21, 82, 53, 34]),
# array([111, 123, 132, 145, 176, 129, 121, 133, 139, 165, 186, 119]))
But I am interested in knowing an alternative and efficient solution using some sort of masking and slicing without converting first to a 2-d array.
You can also build the mask:
# if you know where your indices are, otherwise use a formula
mask = np.zeros((3, 2, 14), dtype=bool)
mask[0,:, 0:4] = True
mask[1,:, 4:8] = True
mask[2,:, 8:] = True
arr1, arr2, arr3 = (arr[m.flatten()] for m in mask)
print (arr1, arr2, arr3)
Using a mask of the pattern as requested (and supposing that arr length is an exact multiple of the mask length):
mask1 = [True]*4 + [False]*10
mask2 = [False]*4 + [True]*4 + [False]*6
mask3 = [False]*8 + [True]*6
Then you directly get the desired arrays by doing:
n_masks = (len(arr) // len(mask1))
arr1 = arr[mask1 * n_masks]
arr2 = arr[mask2 * n_masks]
arr3 = arr[mask3 * n_masks]
You could access the indices directly
import numpy as np
arr = np.array([1, 2, 3, 4, 11, 12, 13, 14, 111, 123, 132, 145, 176, 129,
6, 5, 3, 2, 21, 82, 53, 34, 121, 133, 139, 165, 186, 119])
run_length = 14
repetitions = 2
indices1 = [run_length * i + j for i in range(repetitions) for j in range(4)]
arr1 = arr[indices1]
indices2 = [run_length * i + j for i in range(repetitions) for j in range(4, 8)]
arr2 = arr[indices2]
indices3 = [run_length * i + j for i in range(repetitions) for j in range(8, 14)]
arr3 = arr[indices3]
print(arr1)
print(arr2)
print(arr3)
Output
[1 2 3 4 6 5 3 2]
[11 12 13 14 21 82 53 34]
[111 123 132 145 176 129 121 133 139 165 186 119]
You could put everything in a function like this:
import numpy as np
arr = np.array([1, 2, 3, 4, 11, 12, 13, 14, 111, 123, 132, 145, 176, 129,
6, 5, 3, 2, 21, 82, 53, 34, 121, 133, 139, 165, 186, 119])
def extract(arr, run_length, repetitions, pattern_lengths):
chunks = [0] + np.cumsum(pattern_lengths).tolist()
for start, end in zip(chunks, chunks[1:]):
indices = [run_length * i + j for i in range(repetitions) for j in range(start, end)]
yield arr[indices]
arr1, arr2, arr3 = list(extract(arr, 14, 2, [4, 4, 6]))
print(arr1)
print(arr2)
print(arr3)
We could simply reshape into 2D (remember reshaping creates a view and has zero memory overhead and hence virtually free on runtime) with the number of cols same as the pattern lenth (14 in the sample case). Then, slice out the first 4 entries for first array output, next 4 for second and 8th col onwards for the last one.
Since, we need flattened output, we can do so with .ravel().
Hence -
In [44]: a2d = arr.reshape(-1,14) # 2d view into arr
...: arr1,arr2,arr3 = a2d[:,:4].ravel(),a2d[:,4:8].ravel(),a2d[:,8:].ravel()
In [45]: arr1
Out[45]: array([1, 2, 3, 4, 6, 5, 3, 2])
In [46]: arr2
Out[46]: array([11, 12, 13, 14, 21, 82, 53, 34])
In [47]: arr3
Out[47]: array([111, 123, 132, 145, 176, 129, 121, 133, 139, 165, 186, 119])
Now, say we are okay with 2D array outputs, then -
In [48]: arr1,arr2,arr3 = a2d[:,:4],a2d[:,4:8],a2d[:,8:]
In [49]: arr1
Out[49]:
array([[1, 2, 3, 4],
[6, 5, 3, 2]])
In [50]: arr2
Out[50]:
array([[11, 12, 13, 14],
[21, 82, 53, 34]])
In [51]: arr3
Out[51]:
array([[111, 123, 132, 145, 176, 129],
[121, 133, 139, 165, 186, 119]])
So, why take this? Because it's a view into the original input arr and hence as mentioned earlier has zero memory overhead and virtually free -
In [52]: np.shares_memory(arr,arr1)
Out[52]: True
and so on for other two arrays.
Suppose I have a numpy array img, with img.shape == (468,832,3). What does img[::2, ::2] do? It reduces the shape to (234,416,3) Can you please explain the logic?
Let's read documentation together (Source).
(Just read the bold part first)
The basic slice syntax is i:j:k where i is the starting index, j is the stopping index, and k is the step (k \neq 0). This selects the m elements (in the corresponding dimension) with index values i, i + k, ..., i + (m - 1) k where m = q + (r\neq0) and q and r are the quotient and remainder obtained by dividing j - i by k: j - i = q k + r, so that i + (m - 1) k < j.
...
Assume n is the number of elements in the dimension being sliced.
Then, if i is not given it defaults to 0 for k > 0 and n - 1 for k < 0
. If j is not given it defaults to n for k > 0 and -n-1 for k < 0 . If
k is not given it defaults to 1. Note that :: is the same as : and
means select all indices along this axis.
Now looking at your part.
[::2, ::2] will be translated to [0:468:2, 0:832:2] because you do not specify the first two or i and j in the documentation. (You only specify k here. Recall the i:j:k notation above.) You select elements on these axes at the step size 2 which means you select every other elements along the axes specified.
Because you did not specify for the 3rd dimension, all will be selected.
It slices every alternate row, and then every alternate column, from an array, returning an array of size (n // 2, n // 2, ...).
Here's an example of slicing with a 2D array -
>>> a = np.arange(16).reshape(4, 4)
>>> a
array([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]])
>>> a[::2, ::2]
array([[ 0, 2],
[ 8, 10]])
And, here's another example with a 3D array -
>>> a = np.arange(27).reshape(3, 3, 3)
>>> a
array([[[ 0, 1, 2],
[ 3, 4, 5],
[ 6, 7, 8]],
[[ 9, 10, 11],
[12, 13, 14],
[15, 16, 17]],
[[18, 19, 20],
[21, 22, 23],
[24, 25, 26]]])
>>> a[::2, ::2] # same as a[::2, ::2, :]
array([[[ 0, 1, 2],
[ 6, 7, 8]],
[[18, 19, 20],
[24, 25, 26]]])
Well, we have the RGB image as a 3D array of shape:
img.shape=(468,832,3)
Now, what does img[::2, ::2] do?
we're just downsampling the image (i.e. we're shrinking the image size by half by taking only every other pixel from the original image and we do this by using a step size of 2, which means to skip one pixel). This should be clear from the example below.
Let's take a simple grayscale image for easier understanding.
In [13]: arr
Out[13]:
array([[10, 11, 12, 13, 14, 15],
[20, 21, 22, 23, 24, 25],
[30, 31, 32, 33, 34, 35],
[40, 41, 42, 43, 44, 45],
[50, 51, 52, 53, 54, 55],
[60, 61, 62, 63, 64, 65]])
In [14]: arr.shape
Out[14]: (6, 6)
In [15]: arr[::2, ::2]
Out[15]:
array([[10, 12, 14],
[30, 32, 34],
[50, 52, 54]])
In [16]: arr[::2, ::2].shape
Out[16]: (3, 3)
Notice which pixels are in the sliced version. Also, observe how the array shape changes after slicing (i.e. it is reduced by half).
Now, this downsampling happens for all three channels in the image since there's no slicing happening in the third axis. Thus, you will get the shape reduced only for the first two axis in your example.
(468, 832, 3)
. . |
. . |
(234, 416, 3)
I am trying to solve the following problem. I have two matrices A and B and I want to create a new matrix C which consists of the rows of the matrices A and B depending on some condition which is encoded in the array v, i.e. if the i'th entry of v is a one then I want the i'th row of C to be the i'th row of B and if it is a zero then it should be the i'th row of A. I came up with the following solution
C = np.choose(v,A.T,B.T).T
but it is too slow. One obvious bad thing are the two transposes, but since np.choose does not take an axis argument I don't know how to get rid of them. Any ideas for a fast solution of this problem?
For Example let
A = np.arange(20).reshape([4,5])
and
B = 10 - A
Then one could imagine that one wants the matrix C to be the matrix of rows with smallest maximum norm. So we let
v = np.sum(A,axis=1)<np.sum(B,axis=1)
and then C is the matrix
C = np.choose(v,[A.T,B.T]).T
which is
array([[10, 9, 8, 7, 6],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]])
Seems like a good setup to use np.where to do the chosing operation based on the mask/binary input data -
C = np.where(v[:,None],B,A)
That v[:,None] part basically extends v to broadcastable shape as A and B allowing the broadcasting to let chosing work along the appropriate axis, axis=0 in this case for the two 2D arrays.
Sample run -
In [58]: A
Out[58]:
array([[82, 78, 57],
[14, 97, 32],
[72, 11, 49],
[98, 34, 41],
[89, 71, 52],
[34, 51, 55],
[26, 92, 59]])
In [59]: B
Out[59]:
array([[55, 67, 50],
[49, 64, 21],
[34, 18, 72],
[24, 61, 65],
[56, 59, 23],
[44, 77, 13],
[56, 55, 58]])
In [62]: v
Out[62]: array([1, 0, 0, 0, 0, 1, 1])
In [63]: np.where(v[:,None],B,A)
Out[63]:
array([[55, 67, 50],
[14, 97, 32],
[72, 11, 49],
[98, 34, 41],
[89, 71, 52],
[44, 77, 13],
[56, 55, 58]])
If v doesn't strictly consist of 0s and 1s only, use v[:,None]==1 as the first argument with np.where.
Another approach would be with boolean-indexing -
C = A.copy()
mask = v==1
C[mask] = B[mask]
Note : If v is already a boolean array, skip the comparison against 1 for the mask creation.
Runtime test -
In [77]: A = np.random.randint(11,99,(10000,3))
In [78]: B = np.random.randint(11,99,(10000,3))
In [79]: v = np.random.rand(A.shape[0])>0.5
In [82]: def choose_rows_copy(A, B, v):
...: C = A.copy()
...: C[v] = B[v]
...: return C
...:
In [83]: %timeit np.where(v[:,None],B,A)
10000 loops, best of 3: 107 µs per loop
In [84]: %timeit choose_rows_copy(A, B, v)
1000 loops, best of 3: 226 µs per loop
In python/numpy, how can I subset a multidimensional array where another one, of the same shape, is maximum along some axis (e.g. the first one)?
Suppose I have two 3*2*4 arrays, a and b. I want to obtain a 2*4 array containing the values of b at the locations where a has its maximal values along the first axis.
import numpy as np
np.random.seed(7)
a = np.random.rand(3*2*4).reshape((3,2,4))
b = np.random.rand(3*2*4).reshape((3,2,4))
print a
#[[[ 0.07630829 0.77991879 0.43840923 0.72346518]
# [ 0.97798951 0.53849587 0.50112046 0.07205113]]
#
# [[ 0.26843898 0.4998825 0.67923 0.80373904]
# [ 0.38094113 0.06593635 0.2881456 0.90959353]]
#
# [[ 0.21338535 0.45212396 0.93120602 0.02489923]
# [ 0.60054892 0.9501295 0.23030288 0.54848992]]]
print a.argmax(axis=0) #(I would like b at these locations along axis0)
#[[1 0 2 1]
# [0 2 0 1]]
I can do this really ugly manual subsetting:
index = zip(a.argmax(axis=0).flatten(),
[0]*a.shape[2]+[1]*a.shape[2], # a.shape[2] = 4 here
range(a.shape[2])+range(a.shape[2]))
# [(1, 0, 0), (0, 0, 1), (2, 0, 2), (1, 0, 3),
# (0, 1, 0), (2, 1, 1), (0, 1, 2), (1, 1, 3)]
Which would allow me to obtain my desired result:
b_where_a_is_max_along0 = np.array([b[i] for i in index]).reshape(2,4)
# For verification:
print a.max(axis=0) == np.array([a[i] for i in index]).reshape(2,4)
#[[ True True True True]
# [ True True True True]]
What is the smart, numpy way to achieve this? Thanks :)
Use advanced-indexing -
m,n = a.shape[1:]
b_out = b[a.argmax(0),np.arange(m)[:,None],np.arange(n)]
Sample run -
Setup input array a and get its argmax along first axis -
In [185]: a = np.random.randint(11,99,(3,2,4))
In [186]: idx = a.argmax(0)
In [187]: idx
Out[187]:
array([[0, 2, 1, 2],
[0, 1, 2, 0]])
In [188]: a
Out[188]:
array([[[49*, 58, 13, 69], # * are the max positions
[94*, 28, 55, 86*]],
[[34, 17, 57*, 50],
[48, 73*, 22, 80]],
[[19, 89*, 42, 71*],
[24, 12, 66*, 82]]])
Verify results with b -
In [193]: b
Out[193]:
array([[[18*, 72, 35, 51], # Mark * at the same positions in b
[74*, 57, 50, 84*]], # and verify
[[58, 92, 53*, 65],
[51, 95*, 43, 94]],
[[85, 23*, 13, 17*],
[17, 64, 35*, 91]]])
In [194]: b[a.argmax(0),np.arange(2)[:,None],np.arange(4)]
Out[194]:
array([[18, 23, 53, 17],
[74, 95, 35, 84]])
You could use ogrid
>>> x = np.random.random((2,3,4))
>>> x
array([[[ 0.87412737, 0.11069105, 0.86951092, 0.74895912],
[ 0.48237622, 0.67502597, 0.11935148, 0.44133397],
[ 0.65169681, 0.21843482, 0.52877862, 0.72662927]],
[[ 0.48979028, 0.97103611, 0.36459645, 0.80723839],
[ 0.90467511, 0.79118429, 0.31371856, 0.99443492],
[ 0.96329039, 0.59534491, 0.15071331, 0.52409446]]])
>>> y = np.argmax(x, axis=1)
>>> y
array([[0, 1, 0, 0],
[2, 0, 0, 1]])
>>> i, j = np.ogrid[:2,:4]
>>> x[i ,y, j]
array([[ 0.87412737, 0.67502597, 0.86951092, 0.74895912],
[ 0.96329039, 0.97103611, 0.36459645, 0.99443492]])
Suppose i have this numpy array
[[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]]
And i want to split it in 2 batches and then iterate:
[[1, 2, 3], Batch 1
[4, 5, 6]]
[[7, 8, 9], Batch 2
[10, 11, 12]]
What is the simplest way to do it?
EDIT: I'm deeply sorry i missed putting such info: Once i intend to carry on with the iteration, the original array would be destroyed due to splitting and iterating over batches. Once the batch iteration finished, i need to restart again from the first batch hence I should preserve that the original array wouldn't be destroyed. The whole idea is to be consistent with Stochastic Gradient Descent algorithms which require iterations over batches. In a typical example, I could have a 100000 iteration For loop for just 1000 batch that should be replayed again and again.
You can use numpy.split to split along the first axis n times, where n is the number of desired batches. Thus, the implementation would look like this -
np.split(arr,n,axis=0) # n is number of batches
Since, the default value for axis is 0 itself, so we can skip setting it. So, we would simply have -
np.split(arr,n)
Sample runs -
In [132]: arr # Input array of shape (10,3)
Out[132]:
array([[170, 52, 204],
[114, 235, 191],
[ 63, 145, 171],
[ 16, 97, 173],
[197, 36, 246],
[218, 75, 68],
[223, 198, 84],
[206, 211, 151],
[187, 132, 18],
[121, 212, 140]])
In [133]: np.split(arr,2) # Split into 2 batches
Out[133]:
[array([[170, 52, 204],
[114, 235, 191],
[ 63, 145, 171],
[ 16, 97, 173],
[197, 36, 246]]), array([[218, 75, 68],
[223, 198, 84],
[206, 211, 151],
[187, 132, 18],
[121, 212, 140]])]
In [134]: np.split(arr,5) # Split into 5 batches
Out[134]:
[array([[170, 52, 204],
[114, 235, 191]]), array([[ 63, 145, 171],
[ 16, 97, 173]]), array([[197, 36, 246],
[218, 75, 68]]), array([[223, 198, 84],
[206, 211, 151]]), array([[187, 132, 18],
[121, 212, 140]])]
consider array a
a = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]])
Option 1
use reshape and //
a.reshape(a.shape[0] // 2, -1, a.shape[1])
array([[[ 1, 2, 3],
[ 4, 5, 6]],
[[ 7, 8, 9],
[10, 11, 12]]])
Option 2
if you wanted groups of two rather than two groups
a.reshape(-1, 2, a.shape[1])
array([[[ 1, 2, 3],
[ 4, 5, 6]],
[[ 7, 8, 9],
[10, 11, 12]]])
Option 3
Use a generator
def get_every_n(a, n=2):
for i in range(a.shape[0] // n):
yield a[n*i:n*(i+1)]
for sa in get_every_n(a, n=2):
print sa
[[1 2 3]
[4 5 6]]
[[ 7 8 9]
[10 11 12]]
To avoid the error "array split does not result in an equal division",
np.array_split(arr, n, axis=0)
is better than np.split(arr, n, axis=0).
For example,
a = np.array([[170, 52, 204],
[114, 235, 191],
[ 63, 145, 171],
[ 16, 97, 173]])
then
print(np.array_split(a, 2))
[array([[170, 52, 204],
[114, 235, 191]]), array([[ 63, 145, 171],
[ 16, 97, 173]])]
print(np.array_split(a, 3))
[array([[170, 52, 204],
[114, 235, 191]]), array([[ 63, 145, 171]]), array([[ 16, 97, 173]])]
However, print(np.split(a, 3)) will raise an error since 4/3 is not an integer.
This is what I have used to iterate through. I use b.next() method to generate the indices, then pass the output to slice a numpy array, for example a[b.next()] where a is a numpy array.
class Batch():
def __init__(self, total, batch_size):
self.total = total
self.batch_size = batch_size
self.current = 0
def next(self):
max_index = self.current + self.batch_size
indices = [i if i < self.total else i - self.total
for i in range(self.current, max_index)]
self.current = max_index % self.total
return indices
b = Batch(10, 3)
print(b.next()) # [0, 1, 2]
print(b.next()) # [3, 4, 5]
print(b.next()) # [6, 7, 8]
print(b.next()) # [9, 0, 1]
print(b.next()) # [2, 3, 4]
print(b.next()) # [5, 6, 7]
Improving previous answer, to split based on batch size, you can use:
def split_by_batchsize(arr, batch_size):
return np.array_split(arr, (arr.shape[0]/batch_size)+1)
or with extra safety:
def split_by_batch_size(arr, batch_size):
nbatches = arr.shape[0]//batch_size
if nbatches != arr.shape[0]/batch_size:
nbatches += 1
return np.array_split(arr, nbatches)
example:
import numpy as np
ncols = 17
batch_size= 2
split_by_batchsize(np.random.random((ncols, 2)), batch_size)
# [array([[0.60482079, 0.81391257],
# [0.00175093, 0.25126441]]),
# array([[0.48591974, 0.77793401],
# [0.72128946, 0.3606879 ]]),
# array([[0.95649328, 0.24765806],
# [0.78844782, 0.56304567]]),
# array([[0.07310456, 0.76940976],
# [0.92163079, 0.90803845]]),
# array([[0.77838703, 0.98460593],
# [0.88397437, 0.39227769]]),
# array([[0.87599421, 0.7038426 ],
# [0.19780976, 0.12763436]]),
# array([[0.14263759, 0.9182901 ],
# [0.40523958, 0.0716843 ]]),
# array([[0.9802908 , 0.01067808],
# [0.53095143, 0.74797636]]),
# array([[0.7596607 , 0.97923229]])]
Sadly, simple iteration is faster than this fancy method. Thus, I did not suggest you to use this approach.
batch_size = 3
nrows = 1000
arr = np.random.random((nrows, 2))
%%timeit
for i in range((arr.shape[0] // batch_size) + 1):
idx = i*batch_size
foo = arr[idx:idx+batch_size,:]
# 345 µs ± 119 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%%timeit
for foo in split_by_batch_size(arr, batch_size):
pass
# 1.84 ms ± 377 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
The speed different seems came from np.array_split create list of array first.
do like this:
a = [[1, 2, 3],[4, 5, 6],
[7, 8, 9],[10, 11, 12]]
b = a[0:2]
c = a[2:4]