I'm doing some work whereby I have to load an manipulate CT images in a format called the Analyze 7.5 file format.
Part of this manipulation - which takes absolutely ages with large images - is loading the raw binary data to a numpy array and reshaping it to the correct dimensions. Here is an example:
headshape = (512,512,245) # The shape the image should be
headdata = np.fromfile("Analyze_CT_Head.img", dtype=np.int16) # loads the image as a flat array, 64225280 long. For testing, a large array of random numbers would do
head_shaped = np.zeros(shape=headshape) # Array to hold the reshaped data
# This set of loops is the problem
for ux in range(0, headshape[0]):
for uy in range(0, headshape[1]):
for uz in range(0, headshape[2]):
head_shaped[ux][uy][uz] = headdata[ux + headshape[0]*uy + (headshape[0]*headshape[1])*uz] # Note the weird indexing of the flat array - this is the pixel ordering I have to work with
I know numpy can do reshaping of arrays quickly, but I can't figure out the correct combination of transformations needed to replicate the effect of the nested loops.
Is there a way to replicate that strange indexing with some combination of numpy.reshape/numpy.ravel etc?
Take a look at the nibabel, a python library that implements readers/writers for the 'Analyze' format. It may have already solved this for you.
You could use reshape in combination with swapaxes
headshape = (2,3,4)
headdata = rand(2*3*4)
head_shaped_short = headdata.reshape(headshape[::-1]).swapaxes(0,2)
worked fine in my case.
numpy stores arrays flat in the memory. The strides attribute contains the necessary information how to map multidimensional indices to the flat indices in the memory.
Here is some further reading about numpy's memory layout.
This should work for you:
# get the number of bytes of the specified dtype
dtype = headdata.dtype
byte_count = dtype.itemsize
headdata = headdata.reshape(headshape)
x, y, z = headshape
headdata.strides = (byte_count, byte_count * x, byte_count * x * y)
# copy data to get back to standard memory layout
data = headdata.copy()
The code exploits setting the strides attribute to reflect your custom memory mapping and to create the (hopefully) correct multidimensional array. After that, it copies the whole array into data, in order to get back to a standard memory layout.
Related
I am in the process of converting some matlab code to python. I working with a 3d volume h x w x d represented as an numpy array, I am extracting smaller 3d patches from this volume using the function from SO here. So if I have 32x32x32 array and extract 16x16x16 patches I end up with a shape (2, 2, 2, 16, 16, 16) After processing each patch I would like to put it back into shape h x w x d basically reverse window_nd What would be the idiomatic numpy way without looping each dimension? Since I also need to work with 2d and 4d data I would like to avoid creating a function for each dimension.
Normally, writing back to as_strided views is not advised because it can cause race conditions, but since you only made blocks, this should work:
original_shaped_array = windowed_array.transpose(0,3,1,4,2,5).reshape(32,32,32)
Additionally, if you never copied the windowed array, and do calculations in-place, the data should be changed in the original array - a windowed view is simply a new view into the same data. Don't do this if there is any overlap
I am trying to assign provinces to an area for use in a game mod. I have two separate maps for area and provinces.
provinces file,
area file.
Currently I am reading in an image in Python and storing it in an array using PIL like this:
import PIL
land_prov_pic = Image.open(INPUT_FILES_DIR + land_prov_str)
land_prov_array = np.array(land_prov_pic)
image_size = land_prov_pic.size
for x in range(image_size[0]):
if x % 100 == 0:
print(x)
for y in range(image_size[1]):
land_prov_array[x][y] = land_prov_pic.getpixel((x,y))
Where you end up with land_prov_array[x][y] = (R,G,B)
However, this get's really slow, especially for large images. I tried reading it in using opencv like this:
import opencv
land_prov_array = cv2.imread(INPUT_FILES_DIR + land_prov_str)
land_prov_array = cv2.cvtColor(land_prov_array, cv2.COLOR_BGR2RGB) #Convert from BGR to RGB
But now land_prov_array[x][y] = [R G B] which is an ndarray and can't be inserted into a set. But it's way faster than the previous for loop. How do I convert [R G B] to (R,G,B) for every element in the array without for loops or, better yet, read it in that way?
EDIT: Added pictures, more description, and code blocks for readability.
It is best to convert the [R,G,B] array to tuple when you need it to be a tuple, rather than converting the whole image to this form. An array of tuples takes up a lot more memory, and will be a lot slower to process, than a numeric array.
The answer by isCzech shows how to create a NumPy view over a 3D array that presents the data as if it were a 2D array of tuples. This might not require the additional memory of an actual array of tuples, but it is still a lot slower to process.
Most importantly, most NumPy functions (such as np.mean) and operators (such as +) cannot be applied to such an array. Thus, one is obliged to iterate over the array in Python code (or with a #np.vectorize function), which is a lot less efficient than using NumPy functions and operators that work on the array as a whole.
For transformation from a 3D array (data3D) to a 2D array (data2D), I've used this approach:
import numpy as np
dt = np.dtype([('x', 'u1'), ('y', 'u1'), ('z', 'u1')])
data2D = data3D.view(dtype=dt).squeeze()
The .view modifies the data type and returns still a 3D array with the last dimension of size 1 which can be then removed by .squeeze. Alternatively you can use .squeeze(axis=-1) to only squeeze the last dimension (in case some of your other dimensions are of size 1 too).
Please note I've used uint8 ('u1') - your type may be different.
Trying to do this using a loop is very slow, indeed (compared to this approach at least).
Similar question here: Show a 2d numpy array where contents are tuples as an image
Can't seem to figure this one out. Very new to numpy.
I have a numpy array of shape (200,1,1000,1000) which corresponds to (number of images, channel, x_of_image, y_of_image). So I have 200 images with 1 channel that are 1000x1000 pixels each.
I want to take each of the 200 images (1,1000,1000), do a operation on the image portion (1000,1000), and append/concatenate it to a brand new array.
new_array = np.array([])
for image in original_array:
new_array = np.concatenate(new_array,original_array[0].operation())
New array would end up being the exact same shape as the original (200,1,1000,1000) just with different images because of the operation performed.
Bonus:
How would I just do the operation on some percentage of the array, say 50%?
This would output an array of (100,1,1000,1000)
Avoid calling np.concatenatein a loop. It allocates a new array and copies everything. This is slow and you may run into memory problems if the discarded copies pile up without being garbage collected.
How this should be done depends mostly on the operations you perform on the images. Most numpy operations are designed to work very well with multi-dimensional arrays.
Try to express the operation with numpy array functions. For example, normalizing the images to a range of 0..1 could be done like this:
new_array = original_array - original_array.min(axis=(-1, -2), keepdims=True)
new_array /= new_array.max(axis=(-1, -2), keepdims=True)
If the image operations are too complex to be broken down into numpy functions, allocate the new array first and modify it in place.
new_array = np.empty_like(original_array)
for i in range(new_array.shape[0]):
new_array[i] = complicated_operation(original_array[i])
Or copy the original array and work only on the copy:
new_array = original_array.copy()
for image in new_array:
image[:] = complicated_operation(image)
For some reason you do not want to pre-allocate, then store the images in a temporary list of arrays and concatenate them in the end:
new_images = []
for image in original_array:
new_images.append(image.operation())
new_array = np.stack(new_images)
If you really want to successively concatenate arrays, note that the arrays-to-be-concatenated are passed to the function as one sequence, like this:
new_array = np.array([])
for image in original_array:
new_array = np.concatenate([new_array, image.operation()])
Bonus: look up slicing. This is very basic numpy/Python and should definitely be in your toolbox.
original_array[::2, :, :, :] # take every second image
I am concatenating data to a numpy array like this:
xdata_test = np.concatenate((xdata_test,additional_X))
This is done a thousand times. The arrays have dtype float32, and their sizes are shown below:
xdata_test.shape : (x1,40,24,24) (x1 : [500~10500])
additional_X.shape : (x2,40,24,24) (x2 : [0 ~ 500])
The problem is that when x1 is larger than ~2000-3000, the concatenation takes a lot longer.
The graph below plots the concatenation time versus the size of the x2 dimension:
Is this a memory issue or a basic characteristic of numpy?
As far as I understand numpy, all the stack and concatenate functions are not extremely efficient. And for good reasons, because numpy tries to keep array memory contiguous for efficiency (see this link about contiguous arrays in numpy)
That means that every concatenate operation have to copy the whole data every time. When I need to concatenate a bunch of elements together I tend to do this :
l = []
for additional_X in ...:
l.append(addiional_X)
xdata_test = np.concatenate(l)
That way, the costly operation of moving the whole data is only done once.
NB : would be interested in the speed improvement that gives you.
If you have in advance the arrays you want to concatenate, I would suggest creating a new array with the total shape and fill it with the small arrays rather than concatenating, as every concatenation operation needs to copy the whole data to a new contiguous space of memory.
First, calculate the total size of the first axis:
max_x = 0
for arr in list_of_arrays:
max_x += arr.shape[0]
Second, create the end container:
final_data = np.empty((max_x,) + xdata_test.shape[1:], dtype=xdata_test.dtype)
which is equivalent to (max_x, 40, 24, 24) but dynamically typed.
Last, fill the numpy array:
curr_x = 0
for arr in list_of_arrays:
final_data[curr_x:curr_x+arr.shape[0]] = arr
curr_x += arr.shape[0]
The loop above, copies each of the arrays to a previously defined column/rows of the larger array.
By doing this, each of the N arrays will be copied to the exact final destination, rather than creating temporal arrays for each of the concatenation.
I'm trying to handle a memory problem in my application by using memory mapped arrays. However, as part of my computation I need to set values some values in my array to 0. Unfortunately, the array mask will require additional memory. Is there a way to do the following such that the mask is handled cleanly?
source_array = numpy.memmap(filename, dtype='float32', mode='w+', shape=shape)
#Load data into memory mapped numpy array
band.ReadAsArray(buf_obj = source_array)
#set values == 255 to 0
numpy.putmask(source_array, source_array >= 255.0, 0.0)
I believe the last line with source_array >= 255.0 must make a big array in memory, right? Aside from manually looping through each element, is there a memory efficient mechanism to set all my 255 values in source_array to 0?
Sorry realized that of course memmapping the mask isn't an optimal solution here. Numpy does not really have much to help loop through the array in chunks (which would be the cleanest way), though you can of course do that by hand. You might actually have some success with numexpr, which always does its calculation in chunks for speeding up numpy, but I did not try this.
I guess this wasn't quite what you wanted:
You can always use the out parameters to ufuncs and many other functions to ask numpy to store the result into that array directly (also to generally save memory). This means that if you create an empty memory map array you can do this:
# You could use tempfile.NamedTemporaryFile. But I will leave that to you:
mask = np.memmap(tempfile, shape=source_array.shape, dtype=bool, mode='w+')
np.greater_equal(source_array, 255.0, out=mask)
And then use the mask array in putmask. This should solve the problem.