I have 2000 images stored as single binary file "file.dat" and a head of 512 bytes to this file. Format of every image is 512*512*2 bytes (unsigned int 16). My task is to visualize all this images as video. How can I do this in python? My problem is starting from reading the sequence of images. I'm newbie in python.
Numpy is quite handy for reading in simple binary file formats.
From the sound of it, you have a large binary file of uin16's that you want to read into a 3D array and visualize. We don't have to load it all into memory, but for this example, we will.
Here's a basic idea of what the code would look like:
import numpy as np
import matplotlib.pyplot as plt
def main():
data = read_data('test.dat', 512, 512)
visualize(data)
def read_data(filename, width, height):
with open(filename, 'r') as infile:
# Skip the header
infile.seek(512)
data = np.fromfile(infile, dtype=np.uint16)
# Reshape the data into a 3D array. (-1 is a placeholder for however many
# images are in the file... E.g. 2000)
return data.reshape((width, height, -1))
def visualize(data):
# There are better ways to do this, but let's keep it simple
plt.ion()
fig, ax = plt.subplots()
im = ax.imshow(data[:,:,0], cmap=plt.cm.gray)
for i in xrange(data.shape[-1]):
image = data[:,:,i]
im.set(data=image, clim=[image.min(), image.max()])
fig.canvas.draw()
main()
Related
I'm working with matplotlib, specifically its imshow() operation. I have a multi-dimension array generated by random.rand function from NumPy.
data_array = np.random.rand(63, 4, 4, 3)
Now I want to generate images using the imshow() function from matplotlib using every 63 entries of this data array, and even this code below has generated the desired image I wanted.
plt.imshow(data_array[0]) #image constructed for the 1st element of the array
Now I wanted to save all the images produced using the imshow() from the array's entries and save those in a specific folder on my computer with a specific name. I tried with this code below.
def create_image(array):
return plt.imshow(array, interpolation='nearest', cmap='viridis')
count = 0
for i in data_array:
count += 63
image_machine = create_image(i)
image_machine.savefig('C:\Users\Asus\Save Images\close_'+str(count)+".png")
Using this code, I want to save each image produced using each entry of data_array using the imshow() function and save it to the specific folder 'C:\Users\Asus\Save Images' with a particular name encoding like 1st image will be saved as close_0.png
Please help me in this saving step, I'm stuck here.
You can do the following:
from pathlib import Path
import matplotlib.pyplot as plt
import numpy as np
output_folder = "path/to/folder/"
data_array = np.random.rand(63, 4, 4, 3)
for index, array in enumerate(data_array):
fig = plt.figure()
plt.imshow(array, interpolation="nearest", cmap="viridis")
fig.savefig(Path(output_folder, f"close_{index}.png"))
plt.close()
I have added the plt.close otherwise you will end up with a lot of images simultaneously open.
I'm working with large image datasets stored in a non-standard image format (.Tsm). Essentially it's a binary file with some headers at the start, very similar to FITS standard except stored in little-endian as opposed to FITS big-endian.
After reading the file header and formatting the metadata, I can read a single image using the following code
def __read_slice(self, file, img_num, dimensions):
"""Read a single image slice from .tsm file"""
pixel_range = self.metadata["pixel range"]
bytes_to_read = self.metadata["bytes to read"]
# position file pointer to correct byte
file.seek(self.HEADER_TOTAL_LEN + (bytes_to_read * img_num), 0)
all_bytes = file.read(bytes_to_read) # read image bytes
img = np.empty(len(pixels), dtype='uint16') # preallocate image vector
byte_idx = 0
for idx, pixel in enumerate(pixel_range):
img[idx] = (all_bytes[byte_idx + 1] << 8) + all_bytes[byte_idx]
byte_idx += 2
return np.reshape(img, (dimensions[1], dimensions[0])) # reshape array to correct dimensions
the trouble is the images can be very large (2048x2048) so even just loading in 20-30 frames for processing can take a significant amount of time. I'm new to python so i'm guessing the code here is pretty inefficient, especially the loop.
Is there a more efficient way to convert the byte data into 16bit integers?
You can try:
img= np.frombuffer(all_bytes, dtype='uint16')
Example:
>>> np.frombuffer(b'\x01\x02\x03\x04', dtype='uint16')
array([ 513, 1027], dtype=uint16)
I have a C++ program which writes pixel data for a 2D grid to a binary file. Each binary file may have numerous grid states back-to-back, and there may be multiple of these binary files.
e.g. I could have 10 binary files bin0, bin1, bin2... bin9 each holding data for 10 grid states for a total of 100 grid states to animate.
I'm looking for a fast way to create an animation from the grid states in these binary files.
My best attempt used python and PIL.Image to create a gif:
import glob
import numpy as np
from PIL import Image
from functools import partial
def create_images():
paths = glob.glob('./outfiles/dump*')
imgs = []
for path in sorted(paths, key=lambda x: int(x.split("p")[1])):
with open(path, 'rb') as ifile:
for dat in iter(partial(ifile.read, rows*cols), b''):
mem = memoryview(dat).cast('B', shape=[rows,cols])
arr = np.asarray(mem)
img = Image.fromarray(arr*255)
imgs.append(img)
return imgs
rows = 128
cols = 128
imgs = create_images()
imgs[0].save('./animation.gif', save_all=True, append_images=imgs[1:], loop=0)
but the final line where I actually write the gif can take a long time if I have potentially thousands of images, each with thousands of pixels. The rendering of the gif is also poor quality when the images are large.
Looking forward to suggestions for how to make this run fast using a different library from Pillow. Not bothered about sticking to Python if there is a better alternative using C/C++ (or other languages, but Python or C/C++ preferred), nor does the animation have to be a gif.
In my case I'm working with grid data which is either 0 or 1 (the context is Conway's Game of Life), so optimizations which take advantage of this would be welcome. Note, however, that currently each 1 or 0 occupies a whole byte in the binary file, therefore are not packed into bits.
EDIT
Just adding a helper python script to generate binary files as I described above for anyone who gives it a go.
import numpy as np
rows = 128
cols = 128
img_per_file = 10
num_files = 3
filesize = rows*cols*img_per_file
for i in range(num_files):
filename = 'bin'+str(i)
data = np.random.randint(2, size=filesize, dtype=np.uint8).tobytes()
f = open(filename, 'wb')
f.write(data)
f.close()
I am using the sliding window technic to an image and i am extracting the mean values of pixels of each one window. So the results are someting like this [[[[215.015625][123.55036272][111.66057478]]]].now the question is how could i save all these values for every one window into a txt file or at a CSV because i want to use them for further compare similarities? whatever i tried the error is same..that it is a 4D array and not an 1D or 2D. I ll appreciate any help really.! Thank you in advance
import cv2
import matplotlib.pyplot as plt
import numpy as np
# read the image and define the stepSize and window size
# (width,height)
image2 = cv2.imread("bird.jpg")# your image path
image = cv2.resize(image2, (224, 224))
tmp = image # for drawing a rectangle
stepSize = 10
(w_width, w_height) = (60, 60 ) # window size
for x in range(0, image.shape[1] - w_width, stepSize):
for y in range(0, image.shape[0] - w_height, stepSize):
window = image[x:x + w_width, y:y + w_height, :]
# classify content of the window with your classifier and
# determine if the window includes an object (cell) or not
# draw window on image
cv2.rectangle(tmp, (x, y), (x + w_width, y + w_height), (255, 0, 0), 2) # draw rectangle on image
plt.imshow(np.array(tmp).astype('uint8'))
# show all windows
plt.show()
mean_values=[]
mean_val, std_dev = cv2.meanStdDev(image)
mean_val = mean_val[:3]
mean_values.append([mean_val])
mean_values = np.asarray(mean_values)
print(mean_values)
Human Readable Option
Assuming that you want the data to be human readable, saving the data takes a little bit more work. My search showed me that there's this solution for saving 3D data to a text file. However, it's pretty simple to extend this example to 4D for your use case. This code is taken and adapted from that post, thank you Joe Kington and David Cheung.
import numpy as np
data = np.arange(2*3*4*5).reshape((2,3,4,5))
with open('test.csv', 'w') as outfile:
# We write this header for readable, the pound symbol
# will cause numpy to ignore it
outfile.write('# Array shape: {0}\n'.format(data.shape))
# Iterating through a ndimensional array produces slices along
# the last axis. This is equivalent to data[i,:,:] in this case.
# Because we are dealing with 4D data instead of 3D data,
# we need to add another for loop that's nested inside of the
# previous one.
for threeD_data_slice in data:
for twoD_data_slice in threeD_data_slice:
# The formatting string indicates that I'm writing out
# the values in left-justified columns 7 characters in width
# with 2 decimal places.
np.savetxt(outfile, twoD_data_slice, fmt='%-7.2f')
# Writing out a break to indicate different slices...
outfile.write('# New slice\n')
And then once the data has been saved all you need to do is load it and reshape it (np.load()) will default to reading in the data as a 2D array but np.reshape() will allow us to recover the structure. Again, this code is adapted from the previous post.
new_data = np.loadtxt('test.csv')
# Note that this returned a 2D array!
print(new_data.shape)
# However, going back to 3D is easy if we know the
# original shape of the array
new_data = new_data.reshape((2,3,4,5))
# Just to check that they're the same...
assert np.all(new_data == data)
Binary Option
Assuming that human readability is not necessary, I would recommend using the built-in *.npy format which is described here. This stores the data in a binary format.
You can save the array by doing np.save('NAME_OF_ARRAY.npy', ARRAY_TO_BE_SAVED) and then load it with SAVED_ARRAY = np.load('NAME_OF_ARRAY.npy').
You can also save several numpy array in a single zip file with the np.savez() function like so np.savez('MANY_ARRAYS.npz', ARRAY_ONE, ARRAY_TWO). And you load the zipped arrays in a similar fashion SEVERAL_ARRAYS = np.load('MANY_ARRAYS.npz').
I have to read the data from just one channel in a stereo wave file in Python.
For this I tried it with scipy.io:
import scipy.io.wavfile as wf
import numpy
def read(path):
data = wf.read(path)
for frame in data[1]:
data = numpy.append(data, frame[0])
return data
But this code is very slow, especially if I have to work with longer files.
So does anybody know a faster way to do this? I thought about the standard wave module by using wave.readframes(), but how are the frames stored there?
scipy.io.wavfile.read returns the tuple (rate, data). If the file is stereo, data is a numpy array with shape (nsamples, 2). To get a specific channel, use a slice of data. For example,
rate, data = wavfile.read(path)
# data0 is the data from channel 0.
data0 = data[:, 0]
The wave module returns the frames as a string of bytes, which can be converted to numbers with the struct module. For instance:
def oneChannel(fname, chanIdx):
""" list with specified channel's data from multichannel wave with 16-bit data """
f = wave.open(fname, 'rb')
chans = f.getnchannels()
samps = f.getnframes()
sampwidth = f.getsampwidth()
assert sampwidth == 2
s = f.readframes(samps) #read the all the samples from the file into a byte string
f.close()
unpstr = '<{0}h'.format(samps*chans) #little-endian 16-bit samples
x = list(struct.unpack(unpstr, s)) #convert the byte string into a list of ints
return x[chanIdx::chans] #return the desired channel
If your WAV file has some other sample size, you can use the (uglier) function in another answer I wrote here.
I've never used scipy's wavfile function so I can't compare speed, but the wave and struct approach I use here has always worked for me.
rate, audio = wavfile.read(path)
audio = np.mean(audio, axis=1)