Related
EDIT:
Originally posted with the question below,
but since a practical walkaround was found I changed the title to match the walkaround.
See the bottom of the details.
Question (OLD)
Similar to this question (which already solved), I want to create a boolean array from an array of indices, but I need a much faster solution.
The code in the above answer takes up to 80% of my code's execution time. So I want to speed it up somehow.
Details
In case you are wondering, here are the details of what I want to do.
I have a table that has about 50 columns and over 2 million rows.
All elements are uint8 in the range of 0 to 200.
import numpy as np
np.random.seed(0)
table = np.random.randint(0, 201, (10, 5), dtype=np.uint8) # rows=10, cols=5
The table looks like this:
array([[172, 10, 127, 140, 47],
[170, 196, 151, 117, 166],
[ 22, 183, 192, 33, 67],
[179, 78, 154, 82, 162],
[195, 118, 125, 139, 103],
[125, 9, 164, 116, 108],
[161, 159, 21, 81, 89],
[165, 102, 98, 36, 183],
[ 5, 112, 87, 58, 43],
[ 76, 70, 60, 75, 189]], dtype=uint8)
For each column, an accepted row indices is given by user as follows:
accepted_rows_for_column_0 = [1, 2, 5, 6]
accepted_rows_for_column_1 = [0, 1, 2, 4, 6, 8, 9]
accepted_rows_for_column_2 = [0, 1, 2, 3, 5, 6, 7]
accepted_rows_for_column_3 = [1, 2, 3, 4, 6, 8]
accepted_rows_for_column_4 = [2, 3, 6, 9]
# for convenient
accepted_rows = [accepted_rows_for_column_0, accepted_rows_for_column_1,
accepted_rows_for_column_2, accepted_rows_for_column_3,
accepted_rows_for_column_4]
# also, all unaccepted row indices are accessible
unaccepted_rows = ...
Here is some code to generate an actual size table for testing (if you want to try it out).
import numpy as np
import random
np.random.seed(0)
random.seed(0)
table = np.random.randint(0, 201, (2 * 10 ** 6, 50), dtype=np.uint8)
accepted_rows = [
np.array(sorted(random.sample(list(range(table.shape[0])), random.randint(table.shape[0] // 2, table.shape[0]))))
for _ in range(table.shape[1])
]
Now, I want to extract all rows where all columns are accepted.
In the example above (10x5 table), 2 and 6 are the target row indices.
expected_result = table[[2, 6]]
array([[ 22, 183, 192, 33, 67],
[161, 159, 21, 81, 89]], dtype=uint8)
The following is a solution using this answer.
def as_boolean_array(indices, size):
t = np.zeros(size, dtype=bool)
t[np.array(indices)] = True # This line is slow.
return t
indices = np.array([as_boolean_array(idx, len(table)) for idx in accepted_rows]).all(axis=0)
results = table[indices]
This is the fastest way I have found.
The execution time is about 350 msec even for a table with 2 million rows.
However, more than 300 msec of that time is spent on a single line that is doing fancy indexing.
As for the execution environment,
table is in memory, and there is about 2GB of free memory and 10GB of SSD disk.
Since the program itself runs in multiple processes,
parallelization using multiprocessing is ineffective.
Any suggestions?
EDIT:
As #myrtlecat and #Michael Szczesny mentioned in the comments section, if the indices are sorted, the intersection can be computed relatively fast.
By merging them first, I was able to greatly reduce the number of slow fancy indexing runs.
import sortednp as snp
indices = as_boolean_array(snp.kway_intersect(*accepted_rows), len(table))
results = table[indices]
Note: The execution time of kway_intersect seems to depend on the number of accepted indices. It it actually slower when most of the indices are accepted. In my case, I can easily get around this problem by using an unaccepted indices instead.
I am trying to collapse a fits data cube with Python. I know that special packages are doing it, but it is for a lecture purposes. I first extract a subcube in Z:
hdu.data = hdu.data[3365:3405, :, :]
subcube = hdu.data
The subcube has a dimension of Z=40, Y=50 and X=26. I want to collapse the cube in a all fashion way by a double loop in X and Y, in order to have a simple 2D image.
for i in range(1, xdim):
for j in range(1, ydim):
Sum[j,i] = subcube[:,j,i].sum()
I get an error message: IndexError: index 26 is out of bounds for axis 1 with size 26.
I know that python handle differently the cube dimensions as Z, Y, X and not X, Y, Z like IDL for example, but I can not figure out why I have the error.
Python indices start at 0. You need to do range(xdim) and range(ydim) in your for loops.
Python ranges starts with 0. Range for X is 0-25. For Y and Z the same.
Maybe simple double loop over subcube with new list creation can hel you?
z_flatten = [[sum(col) for col in row] for row in subcube]
The existing answers pointing out that Python is 0-indexed are correct, but no one pointed out yet that you don't even need to create an empty array with np.zeros or to use any for loops to do this.
Numpy already allows you to apply most operations along a specific axis of your array, as opposed to looping over the dimensions of your sub-cube and summing just one pixel at a time.
For example let's make a 3x4x4 data cube:
>>> cube = np.arange(3 * 4 * 4).reshape((3, 4, 4))
>>> cube
array([[[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[12, 13, 14, 15]],
[[16, 17, 18, 19],
[20, 21, 22, 23],
[24, 25, 26, 27],
[28, 29, 30, 31]],
[[32, 33, 34, 35],
[36, 37, 38, 39],
[40, 41, 42, 43],
[44, 45, 46, 47]]])
Say you want to sum all layers of a 3x3 slice of this cube:
>>> cube[:, :3, :3].sum(axis=0)
array([[48, 51, 54],
[60, 63, 66],
[72, 75, 78]])
In your case, the equivalent would be
subcube[:, :ydim, :xdim].sum(axis=0)
This is equivalent to what you're trying to do, but much more efficient.
As a general note, although you read your data cube out of a FITS file, since astropy.io.fits returns a Numpy array, any documentation or questions you can find about Numpy arrays apply--it generally isn't important at that point that it came from a FITS file. I point this out, just because it might help you in the future if you're struggling to perform operations on Numpy arrays.
So I have a 3D array with shape (28, 28, 60000), corresponding to 60000 28x28 images. I want to get random 24x24 chunks of each image by using the following function:
def crop(X):
x = random.randint(0,3)
y = random.randint(0,3)
return X[x:24+x, y:24+y,]
If I apply the function crop(X) to my matrix X, however, the same chunk from each sample is returned. How do I ensure each sample uses different randomly generated x and y values?
Here is my attempt at it.
Basically the idea is you will have to somehow split the matrix away from the last dimension (numpy doesn't let you apply over things which aren't a 1d array). You can do this using dsplit, and put it back together using dstack.
Then you would apply your crop function over each component. As a simplified example:
import random
a = np.array(range(300)).reshape(10,10,3)
def crop(X):
x = random.randint(0,3)
y = random.randint(0,3)
return X[x:3+x, y:3+y]
# we can loop over each component of the matrix by first splitting it
# off the last dimension:
b = [np.squeeze(x) for x in np.dsplit(a, a.shape[-1])]
# this will recreate the original matrix
c = np.dstack(b)
# so putting it together with the crop function
get_rand_matrix = [crop(np.squeeze(x)) for x in np.dsplit(a, a.shape[-1])]
desired_result = np.dstack(get_rand_matrix)
Here's a vectorized generic ( to handle non-squarish arrays as well) approach using NumPy broadcasting and linear indexing that generates the slices across all the images in one-go to produce a 3D array output, like so -
# Store shape
m,n,N = A.shape # A is the input array
# Set output block shape
out_blk_shape = (24,24)
x = np.random.randint(0,m-out_blk_shape[0]-1,(N))
y = np.random.randint(0,n-out_blk_shape[1]-1,(N))
# Get range arrays for the block across all images
R0 = np.arange(out_blk_shape[0])
R1 = np.arange(out_blk_shape[1])
# Get offset and thus all linear indices. Finally index into input array.
offset_idx = (y*n*N + x*N) + np.arange(N)
all_idx = R0[:,None]*n*N + R1*N + offset_idx[:,None,None]
out = A.ravel()[all_idx]
Sample run -
1) Inputs :
In [188]: A = np.random.randint(0,255,(6,7,2)) # Input array
In [189]: # Set output block shape
...: out_blk_shape = (3,2) # For demo reduced to a small shape
# Rest of the code stays the same.
In [190]: x # To select the start columns from the slice
Out[190]: array([1, 0])
In [191]: y # To select the start rows from the slice
Out[191]: array([1, 2])
In [192]: A[:,:,0]
Out[192]:
array([[ 75, 160, 110, 29, 77, 198, 78],
[237, 39, 219, 184, 73, 149, 144],
[138, 148, 243, 160, 165, 125, 17],
[155, 157, 110, 175, 91, 216, 61],
[101, 5, 209, 98, 212, 44, 63],
[213, 155, 96, 160, 193, 185, 157]])
In [193]: A[:,:,1]
Out[193]:
array([[201, 223, 7, 140, 98, 41, 167],
[139, 247, 134, 17, 74, 216, 0],
[ 44, 28, 26, 182, 45, 24, 34],
[178, 29, 233, 146, 157, 230, 173],
[111, 220, 234, 6, 246, 218, 149],
[200, 101, 23, 116, 166, 199, 233]])
2) Output :
In [194]: out
Out[194]:
array([[[ 39, 219],
[148, 243],
[157, 110]],
[[ 44, 28],
[178, 29],
[111, 220]]])
edit: it's an image so the suggested (How can I efficiently process a numpy array in blocks similar to Matlab's blkproc (blockproc) function) isn't really working for me
I have the following matlab code
fun = #(block_struct) ...
std2(block_struct.data) * ones(size(block_struct.data));
B=blockproc(im2double(Icorrected), [4 4], fun);
I want to remake my code, but this time in Python. I have installed Scikit and i'm trying to work around it like this
b = np.std(a, axis = 2)
The problem of course it's that i'm not applying the std for a number of blocks, just like above.
How can i do something like this? Start a loop and try to call the function for each X*X blocks? Then i wouldn't keep the size the it was.
Is there another more efficient way?
If there is no overlap in the windows you can reshape the data to suit your needs:
Find the mean of 3x3 windows of a 9x9 array.
import numpy as np
>>> a
array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8],
[ 9, 10, 11, 12, 13, 14, 15, 16, 17],
[18, 19, 20, 21, 22, 23, 24, 25, 26],
[27, 28, 29, 30, 31, 32, 33, 34, 35],
[36, 37, 38, 39, 40, 41, 42, 43, 44],
[45, 46, 47, 48, 49, 50, 51, 52, 53],
[54, 55, 56, 57, 58, 59, 60, 61, 62],
[63, 64, 65, 66, 67, 68, 69, 70, 71],
[72, 73, 74, 75, 76, 77, 78, 79, 80]])
Find the new shape
>>> window_size = (3,3)
>>> tuple(np.array(a.shape) / window_size) + window_size
(3, 3, 3, 3)
>>> b = a.reshape(3,3,3,3)
Find the mean along the first and third axes.
>>> b.mean(axis = (1,3))
array([[ 10., 13., 16.],
[ 37., 40., 43.],
[ 64., 67., 70.]])
>>>
2x2 windows of a 4x4 array:
>>> a = np.arange(16).reshape((4,4))
>>> window_size = (2,2)
>>> tuple(np.array(a.shape) / window_size) + window_size
(2, 2, 2, 2)
>>> b = a.reshape(2,2,2,2)
>>> b.mean(axis = (1,3))
array([[ 2.5, 4.5],
[ 10.5, 12.5]])
>>>
It won't work if the window size doesn't divide into the array size evenly. In that case you need some overlap in the windows or if you just want overlap numpy.lib.stride_tricks.as_strided is the way to go - a generic N-D function can be found at Efficient Overlapping Windows with Numpy
Another option for 2d arrays is sklearn.feature_extraction.image.extract_patches_2d and for ndarray's - sklearn.feature_extraction.image.extract_patches. Each manipulate the array's strides to produce the patches/windows.
I did the following
io.use_plugin('pil', 'imread')
a = io.imread('C:\Users\Dimitrios\Desktop\polimesa\\arizona.jpg')
B = np.zeros((len(a)/2 +1, len(a[0])/2 +1))
for i in xrange(0, len(a), 2):
for j in xrange(0, len(a[0]), 2):
x.append(a[i][j])
if i+1 < len(a):
x.append(a[i+1][j])
if j+1 < len(a[0]):
x.append(a[i][j+1])
if i+1 < len(a) and j+1 < len(a[0]):
x.append(a[i+1][j+1])
B[i/2][j/2] = np.std(x)
x[:] = []
and i think it's correct. Iterating over the image by 2 and taking each neighbour node, adding them to a list and calculating std.
edit* later edited for 4x4 blocks.
We can implement blockproc() in python the following way:
def blockproc(im, block_sz, func):
h, w = im.shape
m, n = block_sz
for x in range(0, h, m):
for y in range(0, w, n):
block = im[x:x+m, y:y+n]
block[:,:] = func(block)
return im
Now, let's apply it to implement contrast enhancement with local histogram equalization, with the low-contrast moon image (of size 512x512) as input and choosing 32x32 blocks:
from skimage import data, exposure
img = data.moon()
img = img / img.max()
m, n = 64, 64
img_eq = blockproc(img.copy(), (m, n), exposure.equalize_hist)
Display the input and output images:
Note that the function does in-place modification to the image, hence a copy of the input image is passed instead.
I have a multidimensional array called resultsten, with the following shape
print np.shape(resultsten)
(3, 3, 6, 10, 1, 9)
In some occasions, I use a part of this array in a program called cleanup, which then further tears this array apart into x, y, and z arrays:
x,y,z = cleanup(resultsten[0,:,:,:,:,:])
def cleanup(resultsmat):
x = resultsmat[:,:,:,:,2]
y = resultsmat[:,:,:,:,1]
z = resultsmat[:,:,:,:,4]
return x,y,z
However, it might also occur that I do not want to put the entire matrix of resultsten in my program cleanup, thus:
x,y,z = cleanup(resultsten[0,0,:,:,:,:])
This, of course gives an error, as the indices given to cleanup do not match the indices expected.
I was wondering if it is possible to have a variable amount of dimensions included in your slice.
I would like to know a command that takes all the entries for every dimension, up until the last dimension, where it only takes one index.
I've seen that is possible to do this for all dimensions except the first, e.g
resultsten[1,:,:,:,:,:]
gives the same result as:
resultsten[1,:]
I tried this:
resultsten[:,1]
but it does not give the required result, Python interprets it like this:
resultsten[:,1,:,:,:,:]
MWE:
def cleanup(resultsmat):
x = resultsmat[:,:,:,0,2]
y = resultsmat[:,:,:,0,1]
z = resultsmat[:,:,:,0,4]
return x,y,z
resultsten=np.arange(3*3*6*10*1*9).reshape(3,3,6,10,1,9)
x0,y0,z0 = cleanup(resultsten[0,:,:,:,:,:]) #works
x0,y0,z0 = cleanup(resultsten[0,0,:,:,:,:]) #does not work
I would use a list of slice objects:
import numpy as np
A = np.arange(2*3*4*5).reshape(2,3,4,5)
#[:] <-> [slice(None,None, None)]
sliceList = [slice(None, None, None)]*(len(A.shape)-1)
a,b,c,d,e = [A[sliceList+[i]] for i in range(A.shape[-1])]
Output:
>>> A[:,:,:,0]
array([[[ 0, 5, 10, 15],
[ 20, 25, 30, 35],
[ 40, 45, 50, 55]],
[[ 60, 65, 70, 75],
[ 80, 85, 90, 95],
[100, 105, 110, 115]]])
>>> a
array([[[ 0, 5, 10, 15],
[ 20, 25, 30, 35],
[ 40, 45, 50, 55]],
[[ 60, 65, 70, 75],
[ 80, 85, 90, 95],
[100, 105, 110, 115]]])