We Keep Coding

OpenCV whole Images to bytes without Saving to Disk

Basically, I want to add a few bytes to my new PNG image file. An example case is like the following code:
img = open("example.png", "rb") # Open images and read as binnary
hex = img.read().hex() # Read images as Bytes to Hexadecimal.
add_hex = hex+"7feab1e74a4bdb755cca" # Add some bytes to it (as hex)
to_bytes_img = bytes.fromhex(add_hex) # Convert hex to bytes
with open("example2.png", "wb") as f: # Write images
f.write(to_bytes_img)
But, the problem is, I have a special case that requires me to perform the above operation using OpenCV (CV2). Where cv2.imread() only reads and stores Pixels as a numpy array (an array of pixels, not the whole file).
Then, I want to write that image into a new file cv2.imwrite(), which will rebuild the image and save the PNG on disk. My question is, how do I add some bytes to the PNG image file (in buffer/memory), before the cv2.imwrite() operation.
I could probably do it with with open() as above, but that would be very inefficient opening, writing, opening, writing to disk again.

Converting raw bytes to image

I have a file that contains a 240x320 image but its byte format I opened it in a hex editor and got what something like an array 16 columns 4800 raw.
Im completely new to this thats why im facing trouble I have tried using a python script but it gave an error on line 17, in data = columnvector[0][i]:
IndexError: list index out of range.
I have tried a java code but that was an error as well, I wanted to try some c# codes but none of the codes i found explains how i can feed my file to the code. This is the python code
import csv
import sys
import binascii
csv.field_size_limit(500 * 1024 * 1024)
columnvector = []
with open('T1.csv', 'r') as csvfile:
csvreader = csv.reader(csvfile,delimiter=' ', quotechar='|')
for row in csvreader:
columnvector.append(row)
headers =['42','4D','36','84','03','00','00','00','00','00','36','00','00','00','28','00','00','00',
'40','01','00','00','F0','00','00','00','01','00','18','00','00','00','00','00','00','84','03','00','C5','00',
'00','00','C5','00','00','00','00','00','00','00','00','00','00','00']
hexArray=[]
for i in range(0,76800):
data = columnvector[0][i]
hexArray.extend([data,data,data])
with open('T1.txt', 'wb') as f:
f.write(binascii.unhexlify(''.join(headers)))
f.write(binascii.unhexlify(''.join(hexArray)))
I want to convert the file to an image using any method I honestly don't care what method to use just as long as it gets the job done.
this is some the files
https://github.com/Mu-A/OV7670-files/tree/Help

You can make the binary data into images without writing any Python, just use ImageMagick in the Terminal. It is included in most Linux distros and is available for macOS and Windows.
If your image is 320x240 it should be:
320 * 240 bytes long if single channel (greyscale), or
320 * 240 * 3 if 3-channel RGB.
As your images are 76800, I am assuming they are greyscale.
So, in Terminal, to make that raw data into a JPEG, use:
magick -depth 8 -size 320x240 gray:T1 result.jpg
or, if you are using version 6 of ImageMagick, use:
convert -depth 8 -size 320x240 gray:T1 result.jpg
If you want a PNG with automatic contrast-stretch, use:
magick -depth 8 -size 320x240 gray:T1 -auto-level result.png
Unfortunately none of your images come out to anything sensible. Here is T1, for example:
The histograms do look somewhat sensible though:
I think you have something fundamentally wrong so I would try reverting to first principles to debug it. I would shine a torch into, or point the camera at a window and save a picture called bright.dat and then cover the lens with a black card and take another image called dark.dat. Then I would plot a histogram of the data and see if the bright one was at the rightmost end and the dark one was at the leftmost end. Make a histogram like this:
magick -depth 8 -size 320x240 Gray:bright.dat histogram:brightHist.png
and:
magick -depth 8 -size 320x240 Gray:dark.dat histogram:darkHist.png

for i in range(0,76800):
is a hardcoded value, and because columnvector[0][i] does not have that many values, you get that IndexError: list index out of range.
Consider why you need to set your range from 0-76800 or if the value can be dynamically sourced from len() of something.

Another simple way to make an image from a binary file is to convert it to a NetPBM image.
As your file is 320x240 and 8-bit binary greyscale, you just need to make a header with that information in it and append your binary file:
printf "P5\n320 240\n255\n" > image.pgm
cat T1 >> image.pgm
You can now open image.pgm with feh, Photoshop, GIMP or many other image viewers.

Tensorflow: How to encode and read bmp images?

I am trying to read .bmp images, do some augmentation on these, save them to a .tfrecords file and then open the .tfrecords files and use the images for image classification. I know that there is a tf.image.encode_jpeg() and a tf.image.encode_png() function, but there is no tf.image.encode_bmp() function. I know that .bmp images are uncompressed, so I've tried to simply base64-encode, np.tostring() and np.tobytes() the images, but I get the following error when trying to decode these formats:
tensorflow.python.framework.errors_impl.InvalidArgumentError: channels attribute 3 does not match bits per pixel from file <some long number>
My take is that tensorflow, in its encoding to jpeg or png, does something extra with the byte encoding of the images; saving information about array dimensionality, etc. However, I am quite clueless about this, so any help would be great!
Some code to show what it is I am trying to achieve:
with tf.gfile.FastGFile(filename, 'rb') as f:
image_data = f.read()
bmp_data = tf.placeholder(dtype=tf.string)
decode_bmp = tf.image.decode_bmp(self._decode_bmp_data, channels=3)
augmented_bmp = <do some augmentation on decode_bmp>
sess = tf.Session()
np_img = sess.run(augmented_bmp, feed_dict={bmp_data: image_data})
byte_img = np_img.tostring()
# Write byte_img to file using tf.train.Example
writer = tf.python_io.TFRecordWriter(<output_tfrecords_filename>)
example = tf.train.Example(features=tf.train.Features(feature={
'encoded_img': tf.train.Feature(bytes_list=tf.train.BytesList(value=[byte_img])}))
writer.write(example.SerializeToString())
# Read img from file
dataset = tf.data.TFRecordDataset(<img_file>)
dataset = dataset.map(parse_img_fn)
The parse_img_fn may be condensed to the following:
def parse_img_fn(serialized_example):
features = tf.parse_single_example(serialized_example, feature_map)
image = features['encoded_img']
image = tf.image.decode_bmp(image, channels=3) # This is where the decoding fails
features['encoded_img']
return features

in your comment, surely you mean encode instead of encrypt
The BMP file format is quite simplistic, consisting of a bunch of headers and pretty much raw pixel data. This is why BMP images are so big. I suppose this is also why TensorFlow developers did not bother to write a function to encode arrays (representing images) into this format. Few people still use it. It is recommended to use PNG instead, which performs lossless compression of the image. Or, if you can deal with lossy compression, use JPG.
TensorFlow doesn't do anything special for encoding images. It just returns the bytes that represent the image in that format, similar to what matplotlib does when you do save_fig (except MPL also writes the bytes to a file).
Suppose you produce a numpy array where the top rows are 0 and the bottom rows are 255. This is an array of numbers which, if you think it as a picture, would represent 2 horizontal bands, the top one black and the bottom one white.
If you want to see this picture in another program (GIMP) you need to encode this information in a standard format, such as PNG. Encoding means adding some headers and metadata and, optionally, compressing the data.
Now that it is a bit more clear what encoding is, I recommend you work with PNG images.
with tf.gfile.FastGFile('image.png', 'rb') as f:
# get the bytes representing the image
# this is a 1D array (string) which includes header and stuff
raw_png = f.read()
# decode the raw representation into an array
# so we have 2D array representing the image (3D if colour)
image = tf.image.decode_png(raw_png)
# augment the image using e.g.
augmented_img = tf.image.random_brightness(image)
# convert the array back into a compressed representation
# by encoding it into png
# we now end up with a string again
augmented_png = tf.image.encode_png(augmented_img, compression=9)
# Write augmented_png to file using tf.train.Example
writer = tf.python_io.TFRecordWriter(<output_tfrecords_filename>)
example = tf.train.Example(features=tf.train.Features(feature={
'encoded_img': tf.train.Feature(bytes_list=tf.train.BytesList(value=[augmented_png])}))
writer.write(example.SerializeToString())
# Read img from file
dataset = tf.data.TFRecordDataset(<img_file>)
dataset = dataset.map(parse_img_fn)
There are a few important pieces of advice:
don't use numpy.tostring. This returns a HUUGE representation because each pixel is represented as a float, and they are all concatenated. No compression, nothing. Try and check the file size :)
no need to pass back into python by using tf.Session. You can perform all the ops on TF side. This way you have an input graph which you can reuse as part of an input pipeline.

There is no encode_bmp in the tensorflow main package, but if you import tensorflow_io (also a Google officially supported package) you can find the encode_bmp method there.
For the documentation see:
https://www.tensorflow.org/io/api_docs/python/tfio/image/encode_bmp

How to convert all bytes files under the folder into images

import numpy
from PIL import Image
import binascii
def getMatrixfrom_bin(filename,width):
with open(filename, 'rb') as f:
content = f.read()
...
return fh
filename = "path\bin_filename(1)"
im = Image.fromarray(getMatrixfrom_bin(filename,512))
//getMatrixfrom_bin () is a function that generates a matrix from the binary bytes
im.save("path\bin_filename(1).png")
The code above can only generate a picture at a time, now I need to convert all the binary files under the path to images, how should I do?

If you are on a decent (i.e. Unix/Linux/macOS) platform, you can convert all your binary files to PNG images in parallel without writing any Python, if you use GNU Parallel and ImageMagick which are installed on most Linux distros and are available for macOS via homebrew.
So, the command to convert all files ending in .bin into PNG images, in parallel would be:
parallel 's=$(wc -c < {}); w=512; ((h=s/w)); convert -depth 8 -size ${w}x${h} gray:{} {.}.png' ::: *bin
That is a bit scary if you are not accustomed to it, so I'll break it down. Basically it is running "some stuff" in parallel for all files ending in .bin, so look again and it is:
parallel 'some stuff' ::: *.bin
What is the "some stuff"? Well, note that {} is short-hand for the file we are currently processing, so it is doing this:
s=$(wc -c < {}) # s=total bytes in current file, i.e. s=filesize
w=512 # w=image width
((h=s/w)) # h=s/w, i.e. h=height in pixels of current file
convert ...
The last line, the one starting convert is calling ImageMagick telling it your image depth is 8 bits, and the dimensions in pixels are WxH, it is then reading the current file into an image and saving it as a new image ending in PNG instead of the original extension. Easy!
Of course, if you knew the width was 500 pixels and the height was 400 pixels, life would be even easier:
parallel 'convert -depth 8 -size 500x400 gray:{} {.}.png' ::: *bin

Read .img medical image without header in python

I have a radiograph .img file without the header file. However, the researchers who have published the file have given this information about it
High resolution (2048 × 2048 matrix size, 0.175mm pixel size)
Wide density range (12-bit, 4096 gray scale)
Universal image format (no header, big-endian raw data)
I am trying to open the file using Python but unable to do so. Could someone suggest any method to read this image file?

I found some radiograph images, like yours, by downloading the JSRT database. I have tested the following code on the first image of this database: JPCLN001.IMG.
import matplotlib.pyplot as plt
import numpy as np
# Parameters.
input_filename = "JPCLN001.IMG"
shape = (2048, 2048) # matrix size
dtype = np.dtype('>u2') # big-endian unsigned integer (16bit)
output_filename = "JPCLN001.PNG"
# Reading.
fid = open(input_filename, 'rb')
data = np.fromfile(fid, dtype)
image = data.reshape(shape)
# Display.
plt.imshow(image, cmap = "gray")
plt.savefig(output_filename)
plt.show()
It produces an output file JPCLN001.PNG which looks like this:
I hope I have answered to your question.
Happy coding!

Just in case anybody else is looking at these images and wants to convert them in batches, or outside of Python and without needing any programming knowledge... you can convert them pretty readily at the command line in Terminal with ImageMagick (which is installed on most Linux distros anyway, and available for OS X and Windows) like this:
convert -size 2048x2048 -depth 16 -endian MSB -normalize gray:JPCLN130.IMG -compress lzw result.tif
which makes them into compressed 16-bit TIF files that can be viewed in any application. They also then take up half the space on disk without loss of quality since I specified LZW compression.
Likewise, if you want 16-bit PNG files, you can use:
convert -size 2048x2048 -depth 16 -endian MSB -normalize gray:JPCLN130.IMG result.png