We Keep Coding

What is the correct way to reshape images after raveling?

At the moment I'm trying to np.ravel() my images so I can use np.append() freely, instead of using np.vstack() which many people here say it's not very fast given the loading/unloading things in memory and I worry it might slow my code down.
My idea was to just flatten the images, append them all and then use np.reshape(appended_images, [512,512,3,-1]) to create the tensor. The tensor is created all right, but upon checkup, the images aren't getting displayed, probably because one of these operations is not working the way I think it should be working.
Checking the final array im_stacked[:,:,:,0] with matplotlib returns a blank image, with a warning of values out of range. Upon inspection of only one channel of the image im_stacked[:,:,0,0] I'm faced with this:
This is just the image repeated over and over. Where is my mistake? Why is there some swapping occurring? Reshaping a single raveled image works fine.
Edit: Minimal code added
import numpy as np
import cv2 as cv
import matplotlib.pyplot as plt
#Image Loading, please use a 512x512x3 image
path = "./path/to/image.png"
im = cv.imread(path)
#Flattening the image
im_raveled = np.ravel(im)
#Starting with an empty array
im_stacked = np.array([])
#For the sake of simplicity this code is just repeated three times
im_stacked = np.append(im_stacked, im_raveled)
im_stacked = np.append(im_stacked, im_raveled)
im_stacked = np.append(im_stacked, im_raveled)
#Using a 515x512x3 image, reshaping the stacked array
im_reshaped = np.reshape(im_stacked, [512,512,3,-1])
#Plotting the images after reshaping
plt.figure()
plt.subplot(1,2,1)
#Plot only the first channel of the first image
plt.imshow(im_reshaped[:,:,0,0])
plt.subplot(1,2,2)
#Plot all channels of the first image
plt.imshow(im_reshaped[:,:,:,0])
plt.show()

Make a sample 3d array:
In [25]: image = np.random.randint(0,256,(512,512,3))
The best way:
In [26]: alist = []
In [27]: for i in range(5):
...: alist.append(image)
...:
It's easy to make an array from such list:
In [28]: np.array(alist).shape
Out[28]: (5, 512, 512, 3)
If you must join them on a new last dimension, use np.stack:
In [29]: np.stack(alist,-1).shape
Out[29]: (512, 512, 3, 5)
np.stack, np.vstack, and even np.append are all covers for np.concatenate. I hate np.append, since it leads too many naive users up the wrong path. It is not an list append clone.
If you must use repeated concatenates do something like:
In [30]: arr = np.zeros((0,512,512,3),image.dtype)
In [31]: arr = np.concatenate([arr,image], axis=0)
Traceback (most recent call last):
File "<ipython-input-31-1fc945fd1c90>", line 1, in <module>
arr = np.concatenate([arr,image], axis=0)
File "<__array_function__ internals>", line 5, in concatenate
ValueError: all the input arrays must have same number of dimensions, but the array at index 0 has 4 dimension(s) and the array at index 1 has 3 dimension(s)
oops, even with experience I have troubles getting that started.
In [32]: arr = np.concatenate([arr,image[None,...]], axis=0)
In [33]: arr.shape
Out[33]: (1, 512, 512, 3)
In [34]: arr = np.concatenate([arr,image[None,...]], axis=0)
In [35]: arr.shape
Out[35]: (2, 512, 512, 3)
Repeated concatenate is slow. concatenate takes a whole list of arrays, and should be used as such. Don't try to replicate list code in lists!
List append is easy because there's an obvious "empty" list, and you can efficiently add references to it. Arrays don't have an equivalent "empty" array. Dimensions matter, right from the start. I had to start with a (0,512,512,3) shape. If you don't know the needed dimensions, then don't take this approach.
As for your title question, this might work:
im_reshaped = np.reshape(im_stacked, [-1,512,512,3])
With the repeated np.append, you joined the ravelled arrays end to end, [(786432,),(786432,),(786432,),...]. Effectively the new dimension is a leading one, not a trailing one. It's a crude way of performing the list append and array build that I started with.

numpy Reshaping changes the images

I have an numpy array X which contains 2d images. numpy array dimensions are (1000,60,40) (1000=no.of img).
I want to feed this array to my model but requires dimensions to be
(1000,60,40,1) (appended 1 is for no. of channels).
so i reshape the array by
Y=X.reshape(1000,60,40,1)
as I was having wrong predictions I checked by re-reshaping the reshaped array to check if it was same as my orig img,
I did that by doing
Z=Y.reshape(1000,60,40)
And I saved them as PNG by doing
for i in range(1000):
misc.imsave('img_rereshaped'+str(i)+'.png',Z[i])
It gives some png files as output but they are not same as the respective original ones from the X numpy array
Am I reshaping in the wrong way or reshaping changes the input data and again reshaping the reshaped data would give different result than the original data?

To test whether the reshaping is causing a problem, it's better to test it without involving other potential errors coming from, say, misc.imsave() etc.
Running something like:
import numpy as np
a = np.random.rand(10,3)
b = np.reshape(a, [10, 3, 1])
c = np.reshape(b, [10, 3])
print(np.sum(c - a))
you'll see that going back and forth using reshape doesn't cause a problem.
Could be you're not using the PNG save correctly. Perhaps the function expects 3 channels for example. Try plotting it locally using matplotlib.

Speed up numpy array concatenate [duplicate]

Can't seem to figure this one out. Very new to numpy.
I have a numpy array of shape (200,1,1000,1000) which corresponds to (number of images, channel, x_of_image, y_of_image). So I have 200 images with 1 channel that are 1000x1000 pixels each.
I want to take each of the 200 images (1,1000,1000), do a operation on the image portion (1000,1000), and append/concatenate it to a brand new array.
new_array = np.array([])
for image in original_array:
new_array = np.concatenate(new_array,original_array[0].operation())
New array would end up being the exact same shape as the original (200,1,1000,1000) just with different images because of the operation performed.
Bonus:
How would I just do the operation on some percentage of the array, say 50%?
This would output an array of (100,1,1000,1000)

Avoid calling np.concatenatein a loop. It allocates a new array and copies everything. This is slow and you may run into memory problems if the discarded copies pile up without being garbage collected.
How this should be done depends mostly on the operations you perform on the images. Most numpy operations are designed to work very well with multi-dimensional arrays.
Try to express the operation with numpy array functions. For example, normalizing the images to a range of 0..1 could be done like this:
new_array = original_array - original_array.min(axis=(-1, -2), keepdims=True)
new_array /= new_array.max(axis=(-1, -2), keepdims=True)
If the image operations are too complex to be broken down into numpy functions, allocate the new array first and modify it in place.
new_array = np.empty_like(original_array)
for i in range(new_array.shape[0]):
new_array[i] = complicated_operation(original_array[i])
Or copy the original array and work only on the copy:
new_array = original_array.copy()
for image in new_array:
image[:] = complicated_operation(image)
For some reason you do not want to pre-allocate, then store the images in a temporary list of arrays and concatenate them in the end:
new_images = []
for image in original_array:
new_images.append(image.operation())
new_array = np.stack(new_images)
If you really want to successively concatenate arrays, note that the arrays-to-be-concatenated are passed to the function as one sequence, like this:
new_array = np.array([])
for image in original_array:
new_array = np.concatenate([new_array, image.operation()])
Bonus: look up slicing. This is very basic numpy/Python and should definitely be in your toolbox.
original_array[::2, :, :, :] # take every second image

Concatenate matrixes to tensor

I have two (or sometimes more) matrixes, which I want to combine to a tensor. The matrixes e.g. have the shape (100, 400) and when they are combined, they should have the dimensions (2, 100, 400).
How do I do that? I tried it the same way I created matrixes from vectors, but that didn't work:
tensor = numpy.concatenate(list_of_matrixes, axis=0)

Probably you want
tensor = np.array(list_of_matrices)

np.array([...]) just loves to combine the inputs into a new array along a new axis. In fact it takes some effort to prevent that.:)
To use concatenate you need to add an axis to your arrays. axis=0 means 'join on the current 1st axis', so it would produce a (200,400) array.
np.concatentate([arr1[None,...], arr2[None,...], axis=0)
would do the the trick, or more generally
np.concatenate([arr[None,...] for arr in list_arr], axis=0)
If you look at the code for dstack, hstack, vstack you'll see that they do this sort of dimension adjustment before passing the task to concatenate.
The np.array solution is easy, but the concatenate solution is a good learning opportunity.

Speed up a delta filter in python/numpy

I am writing a decompressor which (among other things) has to apply a delta filter to RGB images. That is, read images where only the first pixel is absolute (R1, G1, B1) and all the others are in the form (R[n]-R[n-1], G[n]-G[n-1], B[n]-B[n-1]), and convert them to standard RGB.
Right now I am using numpy as follows:
rgb = numpy.fromstring(data, 'uint8')
components = rgb.reshape(3, -1, order='F')
filtered = numpy.cumsum(components, dtype='uint8', axis=1)
frame = numpy.reshape(filtered, -1, order='F')
Where
line 1 creates a 1D array of the original image;
line 2 reshapes it in the form
[[R1, R2, ..., Rn], [G1, G2, ..., Gn], [B1, B2, ..., Bn]]
line 3 performs the actual defiltering
line 4 converts back again to a 1D array
The problem is that it is too slow for my needs. I profiled it and found out that a good amount of time is spent reshaping the array.
So I wonder: is there some way of avoiding reshaping or to speed it up?
Notes:
I'd prefer not to have to write a C extension for this.
I'm already using multithreading

For some reason I did not understand yet, the final reshape in your code copies the data. This can be avoided by using C order instead of Fortran order:
rgb = numpy.fromstring(data, 'uint8')
components = rgb.reshape(-1, 3)
filtered = numpy.cumsum(components, dtype='uint8', axis=0)
frame = filtered.reshape(-1)

First, when you read it in you can tell it a little more about the type, Try:
rgb = numpy.fromstring(data, '3uint8')
No reshape needed.
Next, for large operations, where you can get away with it (and cumsum qualifies), use the out= param to keep from moving the data...everything happens in place. Use:
rgb.cumsum(axis=0,out=rgb)
if you still want it flattened:
rgb = rgb.ravel()