Can't seem to figure this one out. Very new to numpy.
I have a numpy array of shape (200,1,1000,1000) which corresponds to (number of images, channel, x_of_image, y_of_image). So I have 200 images with 1 channel that are 1000x1000 pixels each.
I want to take each of the 200 images (1,1000,1000), do a operation on the image portion (1000,1000), and append/concatenate it to a brand new array.
new_array = np.array([])
for image in original_array:
new_array = np.concatenate(new_array,original_array[0].operation())
New array would end up being the exact same shape as the original (200,1,1000,1000) just with different images because of the operation performed.
Bonus:
How would I just do the operation on some percentage of the array, say 50%?
This would output an array of (100,1,1000,1000)
Avoid calling np.concatenatein a loop. It allocates a new array and copies everything. This is slow and you may run into memory problems if the discarded copies pile up without being garbage collected.
How this should be done depends mostly on the operations you perform on the images. Most numpy operations are designed to work very well with multi-dimensional arrays.
Try to express the operation with numpy array functions. For example, normalizing the images to a range of 0..1 could be done like this:
new_array = original_array - original_array.min(axis=(-1, -2), keepdims=True)
new_array /= new_array.max(axis=(-1, -2), keepdims=True)
If the image operations are too complex to be broken down into numpy functions, allocate the new array first and modify it in place.
new_array = np.empty_like(original_array)
for i in range(new_array.shape[0]):
new_array[i] = complicated_operation(original_array[i])
Or copy the original array and work only on the copy:
new_array = original_array.copy()
for image in new_array:
image[:] = complicated_operation(image)
For some reason you do not want to pre-allocate, then store the images in a temporary list of arrays and concatenate them in the end:
new_images = []
for image in original_array:
new_images.append(image.operation())
new_array = np.stack(new_images)
If you really want to successively concatenate arrays, note that the arrays-to-be-concatenated are passed to the function as one sequence, like this:
new_array = np.array([])
for image in original_array:
new_array = np.concatenate([new_array, image.operation()])
Bonus: look up slicing. This is very basic numpy/Python and should definitely be in your toolbox.
original_array[::2, :, :, :] # take every second image
Related
I have a stream of images coming in from a source which is, unfortunately a list of list of tuples of RGB values. I want to perform real-time processing on the image, but that code expects a Numpy array of shape (X,Y,3) where X and Y are the image height and width.
X = [[(R, G, B)...]]
img_arr = np.array([*X])
The above works fine but takes nearly a quarter of a second with my images which is obviously too slow. Interestingly, I also need to go back the other direction after the processing is done, and that code (which seems to work) is not so slow:
imgout = map(tuple, flipped_image)
Some relevant other questions:
why is converting a long 2D list to numpy array so slow?
Convert List of List of Tuples Into 2d Numpy Array
To answer the title of your question, numpy automatically lists and tuples to numpy arrays, so you can just use np.array(X), which will be about as fast as you can get:
img_arr = np.array(X)
A simple list comprehension will convert it back to the list-list-tuple form:
imgout = [[tuple(Z) for Z in Y] for Y in img_arr]
Code to generate a sample 10x10 X array:
X = [[tuple(Z) for Z in Y] for Y in np.random.randint(0,255,(10,10,3))]
I am trying to assign provinces to an area for use in a game mod. I have two separate maps for area and provinces.
provinces file,
area file.
Currently I am reading in an image in Python and storing it in an array using PIL like this:
import PIL
land_prov_pic = Image.open(INPUT_FILES_DIR + land_prov_str)
land_prov_array = np.array(land_prov_pic)
image_size = land_prov_pic.size
for x in range(image_size[0]):
if x % 100 == 0:
print(x)
for y in range(image_size[1]):
land_prov_array[x][y] = land_prov_pic.getpixel((x,y))
Where you end up with land_prov_array[x][y] = (R,G,B)
However, this get's really slow, especially for large images. I tried reading it in using opencv like this:
import opencv
land_prov_array = cv2.imread(INPUT_FILES_DIR + land_prov_str)
land_prov_array = cv2.cvtColor(land_prov_array, cv2.COLOR_BGR2RGB) #Convert from BGR to RGB
But now land_prov_array[x][y] = [R G B] which is an ndarray and can't be inserted into a set. But it's way faster than the previous for loop. How do I convert [R G B] to (R,G,B) for every element in the array without for loops or, better yet, read it in that way?
EDIT: Added pictures, more description, and code blocks for readability.
It is best to convert the [R,G,B] array to tuple when you need it to be a tuple, rather than converting the whole image to this form. An array of tuples takes up a lot more memory, and will be a lot slower to process, than a numeric array.
The answer by isCzech shows how to create a NumPy view over a 3D array that presents the data as if it were a 2D array of tuples. This might not require the additional memory of an actual array of tuples, but it is still a lot slower to process.
Most importantly, most NumPy functions (such as np.mean) and operators (such as +) cannot be applied to such an array. Thus, one is obliged to iterate over the array in Python code (or with a #np.vectorize function), which is a lot less efficient than using NumPy functions and operators that work on the array as a whole.
For transformation from a 3D array (data3D) to a 2D array (data2D), I've used this approach:
import numpy as np
dt = np.dtype([('x', 'u1'), ('y', 'u1'), ('z', 'u1')])
data2D = data3D.view(dtype=dt).squeeze()
The .view modifies the data type and returns still a 3D array with the last dimension of size 1 which can be then removed by .squeeze. Alternatively you can use .squeeze(axis=-1) to only squeeze the last dimension (in case some of your other dimensions are of size 1 too).
Please note I've used uint8 ('u1') - your type may be different.
Trying to do this using a loop is very slow, indeed (compared to this approach at least).
Similar question here: Show a 2d numpy array where contents are tuples as an image
I am trying to create an empty numpy array, and save all the images that I get from my device. The images come in as numpy array of shape (240,320,3). Creating an empty array to store these images seems like the correct thing to do. When I try to append however, I get this error:
ValueError: all the input arrays must have same number of dimensions
Code as follows:
import numpy as np
# will be appending many images of size (240,320,3)
images = np.empty((0,240,320,3),dtype='uint8')
# filler image to append
image = np.ones((240,320,3),dtype='uint8') * 255
images = np.append(images,image,axis=0)
I need to append many images to this array, so after 100 appends, the shape of the images array should be of shape (100,240,320,3) if done correctly.
Better than np.append is:
images = np.empty((100,240,320,3),dtype='uint8')
for i in range(100):
image = ....
images[i,...] = image
or
alist = []
for i in range(100):
image = ....
alist.append(image)
images = np.array(alist)
# or images = np.stack(alist, axis=0) for more control
np.append is just a cover for np.concatenate. So it makes a new array each time through the loop. By the time you add the 100th image, you have copied the first one 100 times!. The other disadvantage with np.append is that you have to adjust the dimensions of image, a frequent source of error. The other frequent error is getting that initial 'empty' array shape wrong.
Your images array has four dimensions, so you must append a four dimensional item to it. To do so, simply add a new axis to image like so:
images = np.append(images,image[np.newaxis, ...], axis=0)
In a sense, when passing an axis numpy.append is more akin to list.extend than list.append.
I have some 3D image data and want to build a stack of RGB images out of single channel stacks, i.e. I try to concatenate three arrays of shape (358, 1379, 1042) into one array of shape (358, 1379, 1042, 3). Inspired by skimage.color.gray2rgb I tried
np.concatenate((
stack1[..., np.newaxis],
stack2[..., np.newaxis],
stack3[..., np.newaxis]), axis=-1)
However, even though each of these stacks is only about 1GiB this fills my empty ~12GiB RAM immediately ... So I tried to pre-allocate an array of the final shape and then fill it with the stacks, like
rgb_stack = np.zeros(stack1.shape + (3,))
rgb_stack[:,:,:,0] = stack1
which also exhausted my RAM once I execute the second line. Finally I tried to explicitly copy the data from stack1 into rgb_stack by
rgb_stack = np.zeros(stack1.shape + (3,))
rgb_stack[:,:,:,0] = stack1.copy()
with the same result. What am I doing wrong?
To wrap up what can be learnt from the comments to the question; np.zeros creates an array of float64 which is almost 12GiB big. This by itself does not fill the RAM as Linux over commits and only sets the corresponding RAM aside once the array gets filled, which is in this case once it gets filled with the image data.
Thus creating zeros as another dtype solves the problem, e.g.
rgb_stack = np.zeros(stack1.shape + (3,), dtype=np.uint16)
rgb_stack[:,:,:,0] = stack1.copy()
works fine with uint16 stacks.
I'm doing some work whereby I have to load an manipulate CT images in a format called the Analyze 7.5 file format.
Part of this manipulation - which takes absolutely ages with large images - is loading the raw binary data to a numpy array and reshaping it to the correct dimensions. Here is an example:
headshape = (512,512,245) # The shape the image should be
headdata = np.fromfile("Analyze_CT_Head.img", dtype=np.int16) # loads the image as a flat array, 64225280 long. For testing, a large array of random numbers would do
head_shaped = np.zeros(shape=headshape) # Array to hold the reshaped data
# This set of loops is the problem
for ux in range(0, headshape[0]):
for uy in range(0, headshape[1]):
for uz in range(0, headshape[2]):
head_shaped[ux][uy][uz] = headdata[ux + headshape[0]*uy + (headshape[0]*headshape[1])*uz] # Note the weird indexing of the flat array - this is the pixel ordering I have to work with
I know numpy can do reshaping of arrays quickly, but I can't figure out the correct combination of transformations needed to replicate the effect of the nested loops.
Is there a way to replicate that strange indexing with some combination of numpy.reshape/numpy.ravel etc?
Take a look at the nibabel, a python library that implements readers/writers for the 'Analyze' format. It may have already solved this for you.
You could use reshape in combination with swapaxes
headshape = (2,3,4)
headdata = rand(2*3*4)
head_shaped_short = headdata.reshape(headshape[::-1]).swapaxes(0,2)
worked fine in my case.
numpy stores arrays flat in the memory. The strides attribute contains the necessary information how to map multidimensional indices to the flat indices in the memory.
Here is some further reading about numpy's memory layout.
This should work for you:
# get the number of bytes of the specified dtype
dtype = headdata.dtype
byte_count = dtype.itemsize
headdata = headdata.reshape(headshape)
x, y, z = headshape
headdata.strides = (byte_count, byte_count * x, byte_count * x * y)
# copy data to get back to standard memory layout
data = headdata.copy()
The code exploits setting the strides attribute to reflect your custom memory mapping and to create the (hopefully) correct multidimensional array. After that, it copies the whole array into data, in order to get back to a standard memory layout.