We Keep Coding

How do I create image from binary data BSQ?

I've got a problem. I'm trying create image from binary data which I got from hyperspectral camera. The file which I have is in BSQ uint16 format. From the documentation I found out that images contained in the file (.dat) have a resolution of 1024x1024 and there are 24 images in total. The whole thing is to form a kind of "cube" which I want use in the future to creat multi-layered orthomosaic.
I would also like to add that I am completely new in python but I try to be up to date with everything I need. I hope that everything what I have written is clear and uderstandable.
At first I tried to use Numpy liblary to creating 3D array but ended up with an arrangement of random pixels.
from PIL import Image
import numpy as np
file=open('Sequence 1_000021.dat','rb')
myarray=np.fromfile(file,dtype=np.uint16)
print('Size of new array',":", len(myarray))
con_array=np.reshape(myarray,(24,1024,1024),'C')
naPIL=Image.fromarray(con_array[1,:,:])
naPIL.save('naPIL.tiff')
The result: enter image description here
Example of image which I want to achieve (thumbnail): enter image description here

As suspected it's just byte order, I get a sensible looking image when running the following code in a Jupyter notebook:
import numpy as np
from PIL import Image
# open as big-endian, convert to native order, then reshape as appropriate
raw = np.fromfile(
'./Sequence 1_000021.dat', dtype='>u2'
).astype('uint16').reshape((24, 1024, 1024))
# display inline
Image.fromarray(raw[1,:,:])

numpy.fromfile seems to be unable to read large files

I wanted to write some very simple python helper tool for my project which is reading binary data from an ECG record. I have found somewhere that numpy.fromfile is the most appropriate tool to approach it, so I wrote:
#!/usr/bin/env python3
import sys
import numpy as np
arrayOfNums = np.fromfile(sys.argv[1], 'short')
print("Converting " + sys.argv[1] + "...")
conversionOutput = open("output", "x")
conversionOutput.write(np.array2string(arrayOfNums, separator=' '))
conversionOutput.close()
print("Conversion done.")
I did that to write the data which is 2 byte records unseparated. The input file is somewhat large for a simple text file (over 7MB), however not large enough I think to cause numpy troubles.
The output I got in the file: [-32243 -32141 -32666 ... -32580 -32635 -32690]
Why the dots between? It seems to convert it okay, but omits almost everything it is supposed to save. Any help would be appreciated.

Numpy reads correctly your file. To avoid a long display, numpy uses the dots:
import numpy as np
a = np.random.random(10000)
Output:
>>> a
array([0.20902653, 0.80097215, 0.06909818, ..., 0.5963183 , 0.94024005,
0.31870234])
>>> a.shape
(10000,)
a contains 10000 values and not only the 6 displayed values.
Update
To display the full output:
import sys
np.set_printoptions(threshold=sys.maxsize)
print(a)

How do I encode a numpy array to a base64-encoded PNG without writing to disk?

I want to convert an image, represented as a numpy array, into a base64-encoded PNG string, without writing it as a file to disk first.
The goal is to send this over HTTP, and then display the image in a browser using Javascript. Basically trying to send a numpy image array to a browser without using disk.
I'm very comfortable on the python side, and quite unfamiliar on the JS side.
I don't want to use disk, and I don't want to use tensorflow (which has a related helper API).
I have tried researching, but everything that converts a numpy array to a PNG seems to want to save it to disk. I don't know how to avoid the disk I/O.

Many functions that create files will accept an instance of io.StringIO or io.BytesIO instead of a filename or file object.
Here's an example (in an ipython session) where I use numpngw.write_png (from numpngw, a package I wrote for creating PNG and animated PNG files from numpy arrays) to create a base64-encoded PNG file in a Python bytes object.
First, the imports:
In [39]: import numpy as np
In [40]: from numpngw import write_png
In [41]: import io
In [42]: import base64
Create an image to work with:
In [43]: img = np.random.randint(0, 256, size=(16, 16, 3)).astype(np.uint8)
Create a BytesIO object, and write the PNG "file" to it.
In [44]: f = io.BytesIO()
In [45]: write_png(f, img)
f.getvalue() gets the bytes in the PNG "file". Take a peek at the first 12 bytes:
In [46]: f.getvalue()[:12]
Out[46]: b'\x89PNG\r\n\x1a\n\x00\x00\x00\r'
Finally, use one of the many encodings available in the base64 library to encode the PNG data:
In [47]: b64 = base64.b64encode(f.getvalue())
In [48]: b64
Out[48]: b'iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAIAAACQkWg2AAAAInRFWHRDcmVhdGlvbiBUaW1lADIwMTktMTAtMThUMDI6NDY6MTRayfRM7AAAAFF0RVh0U29mdHdhcmUAbnVtcG5ndyAodmVyc2lvbiAwLjAuOS5kZXYwKSwgaHR0cHM6Ly9naXRodWIuY29tL1dhcnJlbldlY2tlc3Nlci9udW1wbmd3aGhwYwAAAxtJREFUeJwBEAPv/AChDcGkLZrMNsaw4b5CRQk90PGcAA05N+8BH3rlSGZVLOn/vleX3mclnyfC1TXEOJYAfLtv7F3kSclGhnJDU93XTTWA2RcAutMYkETtG2kUZMWvbSDAxm0uSm8y0Do6gNQpANrN6Xt3cIoybwKclwb9jEQoIrdE6acAd4IoaaSEuH2mUIZ0tQ5plhBU61nCj7jBqQBQFH2x7wd5HR3IA5jo/jOrlalRhR2xgYRboyOl+MhIxExUhNjqEpz66SGJQNbUrWoAYQIWoEgFXiewRwACPUd35eo0DxDWYcrONkmj1fBGe03/JirvaJLQ79F7uAG2vrCXACLBrKntrzxPLoFa7Wgzl1nNEdo7Fenn7Jd+d2TAAMFELJTkhXAhUpQe3NyIjY9y+QD0Lmn11S/UYf4BcTqE3lMbbQ2iQpS1kcKOtbC2zjx7KovsK1M+JRIClOUzATlVnp8ANJS/zfP2ZTzR4dKjV2XfF3bxFs9Kk9qb+ilBHlBUSr2VA6CHxxVtcuvlkeiDvSA6AFQjA+Hd5vsgjgAcEUPkJx2SgqazSjXcSFFyQHqhgif47p3WOVM2lv0rWc4kSRwE5QCXsYTzOWusqZPbHxZSIWDvPjZcESUQ6FKZ4Pa0XFfdD2rLxp/4PF6w33k75h6uHpMAJWGC0yVRXBRW726WM9BAoX7+KjhLaJ6FhJ5aeRYL092uW8DxFwgPTq/evQg6PgOZAOWyz+saW+gMZukrMeZyQLQcQ5QMzX0sFXBtT7rAs9sEKUIx/HLH6wuTCd+zkzbvbAD6cNATGxklmLG15gwnLIX5bSCFD3wTAhDZwaXTdvUDt6LGc7oma5QfuVKUPGXtm7MAIghIuKbNd6S1wsqG6Ga/cB+wMJYklc5AG4dGzFnK+RbpACrQtchNn9v6o2zXJxjaAIaIY6VW1Db6VNeP/tXKHrusaqy1lssoVdSNGiDMo17vmM4EsTkqSkNMFAMct78+WQCb6h4CwetL6rzaQvUx03qW4GBgPbIEMe2MIS9l7D53VKdtUjhpl28zOqI5bW3YFN4cXne4IEUrgwAAAABJRU5ErkJggg=='

Python error : index out of bounds

I was curious about image processing with python, so I found this great library imageio,
I tried to manipulate the pixels of a picture and save them in a new file,
but i had some problems with the loops
this is what the code looks like
enter image description here
and this the error that i Got !
IndexError: index 3507 is out of bounds for axis 0 with size 3507
the code :
# -*- coding: iso-8859-1 -*-
import imageio
import numpy as np
im = imageio.imread("JAFFRE009a.png")
taille=im.shape #taille is a tuple (Width,Height)
print taille # (4961,3507)
matrice_pixels=open("matrice.txt",'w')
for i in range(taille[1]):
line=""
for j in range(taille[0]):
line+=repr(im[i][j])
matrice_pixels.write(line+'\n')
matrice_pixels.close()

Because your image doesn't have squarred shape, reshape it before you go through your loop

EDIT
We can iterate through each row/column position and save to a file as below.It will take very long time depending upon file size.
Instead of writing your own function, you may want to take advantage of inbuilt binary save (which is more efficient) as
np.save('matrix.py', np_array)
You can load this file as np array and manipulate
Or as a text file using np.save [ will take longer ]
np.save('matrix.txt', np_array)
Working Code:
import imageio
import numpy as np
im = imageio.imread("9v9zU.png")
matrice_pixels=open("matric.txt","wb")
nx,ny = im.shape
for i in range(nx):
line=""
for j in range(ny):
line+=repr(im[i][j])
matrice_pixels.write(line+'\n')
matrice_pixels.close()
#Save as Binary data
np.save('matrix1.npy', im)
#Save as Human readable data
np.savetxt('matrix1.txt', im)
Alternately, you may want to look into off the shelf libraries that will do what you are intending to do.
For e.g. This SO link discusses how to remove section of the picture based upon its color using PIL library.
Also , in future, please DO NOT post a picture of your code. Copy/pase to SO window so that we can copy and modify. In this case I had write everything down line by line to test(thankfully code was not that long).

Trying to read numpy array into opencv - cv2.imdecode returns empty argument

Just getting back into coding after a few years out. Trying to write a piece of code that can go through a folder with .fits files, read out the image data, convert them to an array, read them into opencv, then perform edge detection on them. I can get it working fine with .png files as I don't have to play with arrays. I've done a lot of reading about, and that's enabled me to get to this point. Problem is, img is currently being returned empty, which causes everything else to freak. Any ideas please? Any help would be gratefully received!
#import packages
import cv2
from matplotlim import pyplot as plt
import os
from astropy.io import fits
from skimage import img_as_uint
import numpy as np
#create array with filenames
data = []
for root, dirs, files in os.walk(r'/Users/hannah/Desktop/firefountain-testset'):
for file in files:
if file.endswith('.fits'):
data.append(file)
#start my loop through the folder
for i in data:
fn = i
#read fits image data
hdulist = fits.open(fn)
img_data = hdulist[1].data
#put fits data into array with dtype set as original
imgraw=np.array(img_data, dtype = np.uint8)
#convert to uint16
img = img_as_uint(imgraw)
#crop to area of interest then add into array - possibly don't need second line
imgcrop = img[210:255,227:277]
imgcroparr = np.array(imgcrop)
#attempt to read into cv2 - this is where it loses the data
#all fine before this point I believe
imgfinal = cv2.imdecode(imgcroparr, 0,)
plt.imshow(imgfinal)
#imgfinal returns blank.
google drive with .fits files I'm using, plus .png to show what they should look like
Updated my code
for i in data:
fn = i
imgraw = fits.getdata(fn)
imgcrop = imgraw[210:255,227:277]
img = cv2.imdecode(imgcrop, 0,)
plt.imshow(img)
This opens the fits files no issue and imgcrop works as expected. cv2.imdecode(imgcrop, 0), however, throws up the following:
error: /Users/jhelmus/anaconda/conda-bld/work/opencv-2.4.8/modules/highgui/src/loadsave.cpp:307: error: (-215) buf.data && buf.isContinuous() in function imdecode_
Can't find anything on this error in this context online, getting more and more confused. It's probably something very silly and basic that I've forgotten to do, but I can't see it.