I have got a 1-d array where I have a pattern in the enteries. I will give an example. In the array arr, I have
first 4 enteries with single digits, next 4 enteries with two digits and then the next 6 enteries with 3 digits.
(This single, double, triple digit thing is just to highlight the pattern. The actual array have float numbers of similar values). The example 1-d array looks like:
import numpy as np
arr = np.array([1, 2, 3, 4, 11, 12, 13, 14, 111, 123, 132, 145, 176, 129,
6, 5, 3, 2, 21, 82, 53, 34, 121, 133, 139, 165, 186, 119])
Now, one complete pattern has total 4+4+6 = 14 enteries. This pattern (or repeating unit) is repeated several hundred thousand times so the length of my array is a multiple of 14 (14 * 2 = 28 in the example arr above).
Question:
I want to extract all the one digit enteries (first 4 numbers of one repeating unit), all the two digit enteries
(next 4 numbers of one repeating unit), and all the three digit enteries (next 6 numbers of one repeating unit).
This way I want to have my big arr splitted into three 1-d arrays. So the desired output is
arr1 = array([1, 2, 3, 4, 6, 5, 3, 2])
arr2 = array([11, 12, 13, 14, 21, 82, 53, 34])
arr3 = array([111, 123, 132, 145, 176, 129, 121, 133, 139, 165, 186, 119])
My idea
One way could be to simply reshape it into 2d array since I know the number of repetitions (=28/14 = 2 in the example arr) and then use indexing to get all the first chunks of 4, 4 and 6
and then concatenate.
arr = arr.reshape(2, 14)
and then use slicing to get the chunks as
arr1 = np.concatenate(arr[:, 0:4])
arr2 = np.concatenate(arr[:, 4:8])
arr3 = np.concatenate(arr[:, 8:])
print (arr1, arr2, arr3)
# array([1, 2, 3, 4, 6, 5, 3, 2]),
# array([11, 12, 13, 14, 21, 82, 53, 34]),
# array([111, 123, 132, 145, 176, 129, 121, 133, 139, 165, 186, 119]))
But I am interested in knowing an alternative and efficient solution using some sort of masking and slicing without converting first to a 2-d array.
You can also build the mask:
# if you know where your indices are, otherwise use a formula
mask = np.zeros((3, 2, 14), dtype=bool)
mask[0,:, 0:4] = True
mask[1,:, 4:8] = True
mask[2,:, 8:] = True
arr1, arr2, arr3 = (arr[m.flatten()] for m in mask)
print (arr1, arr2, arr3)
Using a mask of the pattern as requested (and supposing that arr length is an exact multiple of the mask length):
mask1 = [True]*4 + [False]*10
mask2 = [False]*4 + [True]*4 + [False]*6
mask3 = [False]*8 + [True]*6
Then you directly get the desired arrays by doing:
n_masks = (len(arr) // len(mask1))
arr1 = arr[mask1 * n_masks]
arr2 = arr[mask2 * n_masks]
arr3 = arr[mask3 * n_masks]
You could access the indices directly
import numpy as np
arr = np.array([1, 2, 3, 4, 11, 12, 13, 14, 111, 123, 132, 145, 176, 129,
6, 5, 3, 2, 21, 82, 53, 34, 121, 133, 139, 165, 186, 119])
run_length = 14
repetitions = 2
indices1 = [run_length * i + j for i in range(repetitions) for j in range(4)]
arr1 = arr[indices1]
indices2 = [run_length * i + j for i in range(repetitions) for j in range(4, 8)]
arr2 = arr[indices2]
indices3 = [run_length * i + j for i in range(repetitions) for j in range(8, 14)]
arr3 = arr[indices3]
print(arr1)
print(arr2)
print(arr3)
Output
[1 2 3 4 6 5 3 2]
[11 12 13 14 21 82 53 34]
[111 123 132 145 176 129 121 133 139 165 186 119]
You could put everything in a function like this:
import numpy as np
arr = np.array([1, 2, 3, 4, 11, 12, 13, 14, 111, 123, 132, 145, 176, 129,
6, 5, 3, 2, 21, 82, 53, 34, 121, 133, 139, 165, 186, 119])
def extract(arr, run_length, repetitions, pattern_lengths):
chunks = [0] + np.cumsum(pattern_lengths).tolist()
for start, end in zip(chunks, chunks[1:]):
indices = [run_length * i + j for i in range(repetitions) for j in range(start, end)]
yield arr[indices]
arr1, arr2, arr3 = list(extract(arr, 14, 2, [4, 4, 6]))
print(arr1)
print(arr2)
print(arr3)
We could simply reshape into 2D (remember reshaping creates a view and has zero memory overhead and hence virtually free on runtime) with the number of cols same as the pattern lenth (14 in the sample case). Then, slice out the first 4 entries for first array output, next 4 for second and 8th col onwards for the last one.
Since, we need flattened output, we can do so with .ravel().
Hence -
In [44]: a2d = arr.reshape(-1,14) # 2d view into arr
...: arr1,arr2,arr3 = a2d[:,:4].ravel(),a2d[:,4:8].ravel(),a2d[:,8:].ravel()
In [45]: arr1
Out[45]: array([1, 2, 3, 4, 6, 5, 3, 2])
In [46]: arr2
Out[46]: array([11, 12, 13, 14, 21, 82, 53, 34])
In [47]: arr3
Out[47]: array([111, 123, 132, 145, 176, 129, 121, 133, 139, 165, 186, 119])
Now, say we are okay with 2D array outputs, then -
In [48]: arr1,arr2,arr3 = a2d[:,:4],a2d[:,4:8],a2d[:,8:]
In [49]: arr1
Out[49]:
array([[1, 2, 3, 4],
[6, 5, 3, 2]])
In [50]: arr2
Out[50]:
array([[11, 12, 13, 14],
[21, 82, 53, 34]])
In [51]: arr3
Out[51]:
array([[111, 123, 132, 145, 176, 129],
[121, 133, 139, 165, 186, 119]])
So, why take this? Because it's a view into the original input arr and hence as mentioned earlier has zero memory overhead and virtually free -
In [52]: np.shares_memory(arr,arr1)
Out[52]: True
and so on for other two arrays.
Related
Input: There are two input arrays:
value_array = [56, 10, 65, 37, 29, 14, 97, 46]
index_array = [ 0, 0, 1, 0, 3, 0, 1, 1]
Output: I want to split value_array using index_array without using for-loop. So the output array will be:
split_array = [[56, 10, 37, 14], # index 0
[65, 97, 46], # index 1
[], # index 2
[29]] # index 3
Is there any way to do that using numpy without using any for-loop? I have looked at numpy.where but cannot figure it out how to do that.
For-loop: Here is the way to do that using for-loop. I want to avoid for-loop.
split_array = []
for i in range(max(index_array) + 1):
split_array.append([])
for i in range(len(value_array)):
split_array[index_array[i]].append(value_array[i])
Does this suffice?
Solution 1 (Note: for loop is not over the entire index array)
import numpy as np
value_array = np.array([56, 10, 65, 37, 29, 14, 97, 46])
index_array = np.array([ 0, 0, 1, 0, 3, 0, 1, 1])
max_idx = np.max(index_array)
split_array = []
for idx in range(max_idx + 1):
split_array.append([])
split_array[-1].extend(list(value_array[np.where(index_array == idx)]))
print(split_array)
[[56, 10, 37, 14], [65, 97, 46], [], [29]]
Solution 2
import numpy as np
value_array = np.array([56, 10, 65, 37, 29, 14, 97, 46])
index_array = np.array([ 0, 0, 1, 0, 3, 0, 1, 1])
value_array = value_array[index_array.argsort()]
split_idxs = np.squeeze(np.argwhere(np.diff(np.sort(index_array)) != 0) + 1)
print(np.array_split(value_array, split_idxs))
[array([56, 10, 37, 14]), array([65, 97, 46]), array([29])]
Indeed, you can use numpy by using arrays :
import numpy as np
value_array=np.array(value_array)
index_array=np.array(index_array)
split_array=[value_array[np.where(index_array==j)[0]] for j in set(index_array)]
You could do:
import numpy as np
value_array = np.array([56, 10, 65, 37, 29, 14, 97, 46])
index_array = np.array([ 0, 0, 1, 0, 3, 0, 1, 1])
# find the unique values in index array and the corresponding counts
unique, counts = np.unique(index_array, return_counts=True)
# create an array with 0 for the missing indices
zeros = np.zeros(index_array.max() + 1, dtype=np.int32)
zeros[unique] = counts # zeros = [4 3 0 1] 0 -> 4, 1 -> 3, 2 -> 0, 3 -> 1
# group by index array
so = value_array[np.argsort(index_array)] # so = [56 10 37 14 65 97 46 29]
# finally split using the counts
res = np.split(so, zeros.cumsum()[:-1])
print(res)
Output
[array([56, 10, 37, 14]), array([65, 97, 46]), array([], dtype=int64), array([29])]
The time complexity of this approach is O(N logN).
Additionally if you don't care about the missing indices, you could use the following:
_, counts = np.unique(index_array, return_counts=True)
res = np.split(value_array[np.argsort(index_array)], counts.cumsum()[:-1])
print(res)
Output
[array([56, 10, 37, 14]), array([65, 97, 46]), array([29])]
I have an array, of which this is a small sample. It repeats measurements 5 times, and I want to collate these blocks of five into a new array, where each block of five rows is now one row giving mean, median and standard deviation of the five initial rows
data =
[[1, 9, 66, 74, -0.274035]
[1, 9, 66, 74, -0.269245]
[1, 9, 66, 74, -0.271161]
[1, 9, 66, 74, -0.269245]
[1, 9, 66, 74, -0.266370]
[2, 10, 65, 73, 0.085277]
[2, 10, 65, 73, 0.086235]
[2, 10, 65, 73, 0.090068]
[2, 10, 65, 73, 0.087193]
[2, 10, 65, 73, 0.085277]
What I would like to do is keep the value of the value in the block for the first 4 column, then find the mean, median and standard deviation of the next column, working iteratively over blocks of five rows.
data2 =
[[1, 9, 66, 74, mean[0:5,4], median[0:5,4], std[0:5,4]]
[2, 10, 65, 73, mean[5:10,4], median[5:10,4], std[5:10,4]]]
or in numerical terms:
[[1, 9, 66, 74, -0.270011, -0.269245, 0.002528]
[2, 10, 65, 73, 0.08681, 0.086235, 0.001777]]
I've tried this, but just get are zeroes as an output:
index.shape
Out[119]: (10,)
repeat = 5
a = 0
b = repeat
length = int((len(index) - repeat) / repeat)
meanVre = np.zeros(length)
for _ in range(length):
np.append(meanVre, np.mean(data[a:b,5]))
a = a+5
b = b+5
(repeat is used as a variable rather than 5, as the amount of rows in the block is liable to change at a later date).
Any help you can give would be really appreciated.
def block_stats(data, blocksize = 5):
inputs = data[::blocksize, :4]
data_stat = data[:, 4].reshape(-1, blocksize)
means = np.mean(data_stat, axis = 1, keepdims = 1)
medians = np.median(data_stat, axis = 1, keepdims = 1)
stds = np.std(data_stat, axis = 1, keepdims = 1)
return np.vstack([inputs, means, medians, stds])
Is there a way to do multiple indexing in a numpy array as described below?
arr=np.array([55, 2, 3, 4, 5, 6, 7, 8, 9])
arr[np.arange(0,2):np.arange(5,7)]
output:
IndexError: too many indices for array
Desired output:
array([55,2,3,4,5],[2,3,4,5,6])
This problem might be similar to calculating a moving average over an array (but I want to do it without any function that is provided).
Here's an approach using strides -
start_index = np.arange(0,2)
L = 5 # Interval length
n = arr.strides[0]
strided = np.lib.stride_tricks.as_strided
out = strided(arr[start_index[0]:],shape=(len(start_index),L),strides=(n,n))
Sample run -
In [976]: arr
Out[976]: array([55, 52, 13, 64, 25, 76, 47, 18, 69, 88])
In [977]: start_index
Out[977]: array([2, 3, 4])
In [978]: L = 5
In [979]: out
Out[979]:
array([[13, 64, 25, 76, 47],
[64, 25, 76, 47, 18],
[25, 76, 47, 18, 69]])
When I was trying to solve a scientific problem with Python (Numpy), a 'shape mismatch' error came up: "shape mismatch: objects cannot be broadcast to a single shape". I managed to reproduce the same error in a simpler form, as shown below:
import numpy as np
nx = 3; ny = 5
ff = np.ones([nx,ny,7])
def test(x, y):
z = 0.0
for i in range(7):
z = z + ff[x,y,i]
return z
print test(np.arange(nx),np.arange(ny))
When I tried to call test(x,y) with x=1,y=np.arange(ny), everything works fine. So what's going on here? Why can't the both parameters be numpy arrays?
UPDATE
I have worked out the problem with some hints from #Saullo Castro. Here's some updated info for you guys who tried to help but feel unclear about my intention:
Basically I created a mesh grid with dimension nx*ny and another array ff that stores some value for each node. In the above code, ff has 7 values for each node and I was trying to sum up the 7 values to get a new nx*ny array.
However, the "shape mismatch" error is not due to the summing process as many of you might have guess now. I have misunderstood the rule of functions taking ndarray objects as input parameters. I tried to pass np.arange(nx), np.arange(ny) to test() is not gonna give me what I desired, even if nx==ny.
Back to my original intention, I solve the problem by creating another function and used np.fromfunction to created the array:
def tt(x, y):
return np.fromfunction(lambda a,b: test(a,b), (x, y))
which is not perfect but it works. (In this example there seems to be no need to create a new function, but in my actual code I modified it a bit so it can be used for slice of the grid)
Anyway, I do believe there's a much better way compared to my kind of dirty solution. So if you have any idea about that, please share with us :).
Let's look into an array similar to your ff array:
nx = 3; ny = 4
ff = np.arange(nx*ny*5).reshape(nx,ny,5)
#array([[[ 0, 1, 2, 3, 4],
# [ 5, 6, 7, 8, 9],
# [10, 11, 12, 13, 14],
# [15, 16, 17, 18, 19]],
#
# [[20, 21, 22, 23, 24],
# [25, 26, 27, 28, 29],
# [30, 31, 32, 33, 34],
# [35, 36, 37, 38, 39]],
#
# [[40, 41, 42, 43, 44],
# [45, 46, 47, 48, 49],
# [50, 51, 52, 53, 54],
# [55, 56, 57, 58, 59]]])
When you index using arrays of indices a, b, c like in ff[a, b, c], a, b, c must have the same shape, and numpy will build a new array based on the indices. For example:
ff[[0, 0, 1, 1, 2, 2], [0, 1, 0, 1, 2, 3], [0, 0, 0, 1, 1, 1]]
#array([ 0, 5, 20, 26, 51, 56])
This is called fancy indexing, which is like building an array with:
np.array([ff[0, 0, 0], ff[0, 1, 0], ff[1, 0, 0], ..., ff[2, 3, 1]])
In your case the f[x, y, i] will produce a shape mismatch error since a, b, c do not have the same shape.
Looks like you want to sum ff over the last dimension, with the 1st 2 dimensions covering their whole range. : is used to denote the whole range of a dimension:
def test():
z = 0.0
for i in range(7):
z = z + ff[:,:,i]
return z
print test()
But you can get the same result without looping, by using the sum method.
print ff.sum(axis=-1)
: is shorthand for 0:n
ff[0:nx, 0:ny, 0]==ff[:,:,0]
It is possible to index a block of ff with ranges, but you have to be much more careful about the shapes of the indexing arrays. For a beginner it is better to focus on getting slicing and broadcasting correct.
edit -
You can index an array like ff with arrays generated by meshgrid:
I,J = meshgrid(np.arange(nx),np.arange(ny),indexing='ij',sparse=False)
I.shape # (nx,ny)
ff[I,J,:]
also works with
I,J = meshgrid(np.arange(nx),np.arange(ny),indexing='ij',sparse=True)
I.shape # (nx,1)
J.shape # (1, ny)
ogrid and mgrid are alternatives to meshgrid.
Let's reproduce your problem in 2D case, so it is easier to see:
import numpy as np
a = np.arange(15).reshape(3,5)
x = np.arange(3)
y = np.arange(5)
Demo:
>>> a
array([[ 0, 1, 2, 3, 4],
[ 5, 6, 7, 8, 9],
[10, 11, 12, 13, 14]])
>>> a[x, y] # <- This is the error that you are getting
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: shape mismatch: objects cannot be broadcast to a single shape
# You are getting the error because x and y are different lengths,
# If x and y were the same lengths, the code would work:
>>> a[x, x]
array([ 0, 6, 12])
# mixing arrays and scalars is not a problem
>>> a[x, 2]
array([ 2, 7, 12])
It is not clear in your question what you are trying to do or what result are you expecting. It seems, though, that you are trying to calculate a total with your variable z.
Check if the sum method produces the result that you need:
import numpy as np
nx = 3; ny = 5
ff = ff = np.array(np.arange(nx*ny*7)).reshape(nx,ny,7)
print ff.sum() # 5460
print ff.sum(axis=0) # array([[105, 108, 111, 114, 117, 120, 123],
# [126, 129, 132, 135, 138, 141, 144],
# [147, 150, 153, 156, 159, 162, 165],
# [168, 171, 174, 177, 180, 183, 186],
# [189, 192, 195, 198, 201, 204, 207]]) shape(5,7)
print ff.sum(axis=1) # array([[ 70, 75, 80, 85, 90, 95, 100],
# [245, 250, 255, 260, 265, 270, 275],
# [420, 425, 430, 435, 440, 445, 450]]) shape (3,7)
print ff.sum(axis=2) # array([[ 21, 70, 119, 168, 217],
# [266, 315, 364, 413, 462],
# [511, 560, 609, 658, 707]]) shape (3,5)
I have a numpy array of numbers, for example,
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
I would like to find all the indexes of the elements within a specific range. For instance, if the range is (6, 10), the answer should be (3, 4, 5). Is there a built-in function to do this?
You can use np.where to get indices and np.logical_and to set two conditions:
import numpy as np
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
np.where(np.logical_and(a>=6, a<=10))
# returns (array([3, 4, 5]),)
As in #deinonychusaur's reply, but even more compact:
In [7]: np.where((a >= 6) & (a <=10))
Out[7]: (array([3, 4, 5]),)
Summary of the answers
For understanding what is the best answer we can do some timing using the different solution.
Unfortunately, the question was not well-posed so there are answers to different questions, here I try to point the answer to the same question. Given the array:
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
The answer should be the indexes of the elements between a certain range, we assume inclusive, in this case, 6 and 10.
answer = (3, 4, 5)
Corresponding to the values 6,9,10.
To test the best answer we can use this code.
import timeit
setup = """
import numpy as np
import numexpr as ne
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
# or test it with an array of the similar size
# a = np.random.rand(100)*23 # change the number to the an estimate of your array size.
# we define the left and right limit
ll = 6
rl = 10
def sorted_slice(a,l,r):
start = np.searchsorted(a, l, 'left')
end = np.searchsorted(a, r, 'right')
return np.arange(start,end)
"""
functions = ['sorted_slice(a,ll,rl)', # works only for sorted values
'np.where(np.logical_and(a>=ll, a<=rl))[0]',
'np.where((a >= ll) & (a <=rl))[0]',
'np.where((a>=ll)*(a<=rl))[0]',
'np.where(np.vectorize(lambda x: ll <= x <= rl)(a))[0]',
'np.argwhere((a>=ll) & (a<=rl)).T[0]', # we traspose for getting a single row
'np.where(ne.evaluate("(ll <= a) & (a <= rl)"))[0]',]
functions2 = [
'a[np.logical_and(a>=ll, a<=rl)]',
'a[(a>=ll) & (a<=rl)]',
'a[(a>=ll)*(a<=rl)]',
'a[np.vectorize(lambda x: ll <= x <= rl)(a)]',
'a[ne.evaluate("(ll <= a) & (a <= rl)")]',
]
rdict = {}
for i in functions:
rdict[i] = timeit.timeit(i,setup=setup,number=1000)
print("%s -> %s s" %(i,rdict[i]))
print("Sorted:")
for w in sorted(rdict, key=rdict.get):
print(w, rdict[w])
Results
The results are reported in the following plot for a small array (on the top the fastest solution) as noted by #EZLearner they may vary depending on the size of the array. sorted slice could be faster for larger arrays, but it requires your array to be sorted, for arrays with over 10 M of entries ne.evaluate could be an option. Is hence always better to perform this test with an array of the same size as yours:
If instead of the indexes you want to extract the values you can perform the tests using functions2 but the results are almost the same.
I thought I would add this because the a in the example you gave is sorted:
import numpy as np
a = [1, 3, 5, 6, 9, 10, 14, 15, 56]
start = np.searchsorted(a, 6, 'left')
end = np.searchsorted(a, 10, 'right')
rng = np.arange(start, end)
rng
# array([3, 4, 5])
a = np.array([1,2,3,4,5,6,7,8,9])
b = a[(a>2) & (a<8)]
Other way is with:
np.vectorize(lambda x: 6 <= x <= 10)(a)
which returns:
array([False, False, False, True, True, True, False, False, False])
It is sometimes useful for masking time series, vectors, etc.
This code snippet returns all the numbers in a numpy array between two values:
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56] )
a[(a>6)*(a<10)]
It works as following:
(a>6) returns a numpy array with True (1) and False (0), so does (a<10). By multiplying these two together you get an array with either a True, if both statements are True (because 1x1 = 1) or False (because 0x0 = 0 and 1x0 = 0).
The part a[...] returns all values of array a where the array between brackets returns a True statement.
Of course you can make this more complicated by saying for instance
...*(1-a<10)
which is similar to an "and Not" statement.
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
np.argwhere((a>=6) & (a<=10))
Wanted to add numexpr into the mix:
import numpy as np
import numexpr as ne
a = np.array([1, 3, 5, 6, 9, 10, 14, 15, 56])
np.where(ne.evaluate("(6 <= a) & (a <= 10)"))[0]
# array([3, 4, 5], dtype=int64)
Would only make sense for larger arrays with millions... or if you hitting a memory limits.
This may not be the prettiest, but works for any dimension
a = np.array([[-1,2], [1,5], [6,7], [5,2], [3,4], [0, 0], [-1,-1]])
ranges = (0,4), (0,4)
def conditionRange(X : np.ndarray, ranges : list) -> np.ndarray:
idx = set()
for column, r in enumerate(ranges):
tmp = np.where(np.logical_and(X[:, column] >= r[0], X[:, column] <= r[1]))[0]
if idx:
idx = idx & set(tmp)
else:
idx = set(tmp)
idx = np.array(list(idx))
return X[idx, :]
b = conditionRange(a, ranges)
print(b)
s=[52, 33, 70, 39, 57, 59, 7, 2, 46, 69, 11, 74, 58, 60, 63, 43, 75, 92, 65, 19, 1, 79, 22, 38, 26, 3, 66, 88, 9, 15, 28, 44, 67, 87, 21, 49, 85, 32, 89, 77, 47, 93, 35, 12, 73, 76, 50, 45, 5, 29, 97, 94, 95, 56, 48, 71, 54, 55, 51, 23, 84, 80, 62, 30, 13, 34]
dic={}
for i in range(0,len(s),10):
dic[i,i+10]=list(filter(lambda x:((x>=i)&(x<i+10)),s))
print(dic)
for keys,values in dic.items():
print(keys)
print(values)
Output:
(0, 10)
[7, 2, 1, 3, 9, 5]
(20, 30)
[22, 26, 28, 21, 29, 23]
(30, 40)
[33, 39, 38, 32, 35, 30, 34]
(10, 20)
[11, 19, 15, 12, 13]
(40, 50)
[46, 43, 44, 49, 47, 45, 48]
(60, 70)
[69, 60, 63, 65, 66, 67, 62]
(50, 60)
[52, 57, 59, 58, 50, 56, 54, 55, 51]
You can use np.clip() to achieve the same:
a = [1, 3, 5, 6, 9, 10, 14, 15, 56]
np.clip(a,6,10)
However, it holds the values less than and greater than 6 and 10 respectively.