I am using Python 2.7
I am trying to use a large binary file that specifies the latitude/longitude of each pixel of an image.
Using this code: open('input', 'rb').read(50)
The file looks like this:
\x80\x0c\x00\x00\x00\x06\x00\x00.\x18\xca\xe4.\x18\xcc\xe4.\x18\xcf\xe4.\x18\xd1\xe4.\x18\xd3\xe4.\x18\xd5\xe4.\x18\xd7\xe4.\x18\xd9\xe4/\x18\xdb\xe4/\x18\xdd\xe4/\x18...
The read-me for the file gives the following information for decoding but I am unsure of how to apply them:
The files are in LSBF byte order. Files start with 2 4-byte integer values giving the pixel and line (x,y) size of the file. After the files have succeeding pairs of elements are 2-byte integer values of latitude and longitude multiplied 100 and truncated (e.g. 75.324 E is "-7532").
Thanks for any help.
Note the reason for doing this ultimately would be to draw/alter an image based on lat/long and not the pixel # in case anyone was wondering.
Generally when working with binary files you'll want to use the pack and unpack (https://docs.python.org/2/library/struct.html) functions. That will allow you to control the endianess of the data as well as the data types. In your specific case you'll probably want to read the header info first, something like
with open('input', 'rb') as fp:
# read the header bytes to get the number of elements
header_bytes = fp.read(8)
# convert the bytes to two unsigned integers.
x, y = unpack("II", header_bytes)
# Loop through the file getting the rest of the data
# read Y lines of X pixels
...
The above was not actually run or test, just trying to give you a general sense of an approach.
Related
s1 = wave.open('sound.wav', 'r')
frames_1 = []
for i in range(44100):
frames_1.append(struct.unpack('<h', s1.readframes(1)))
Basically, I am trying to unpack s1.readframes(1), which should return an integer value between ~-32000 and 32000. readframes() returns a byte object. It gives me the message "struct.error: unpack requires a buffer of 2 bytes".
I am not sure if I am even supposed to use struct.unpack() or if there is something else that I should do. I assume I am because to write the frames to the .wav file I have to first convert the frames to bytes using data = value.writeframesraw() and then use struct.pack('<h', data), however I am not sure if I am using unpack() correctly. If I try to change '<h' to something else, it just increases the number of bytes in the error message. Optimally I want to only use the wave library for this code, but if there are other libraries that I could import that can read and write .wav files easily then please recommend them.
I'm trying to write an algorithm that will save the filename and the 3 channel np.array stored in each filename to a csv (or similar filetype), and then be able to read in the csv and reproduce the color image image.
The format of my csv should look like this:
Filename RGB
0 foo.png np.array # the shape is 100*100*3
1 bar.png np.array
2 ... ...
As it stands, I'm iterating through each file saved in a directory and appending a list that later gets stored in a pandas.DataFrame:
df1= pandas.DataFrame()
df2= pandas.DataFrame()
directory= r'C:/my Directory'
fileList= os.listdir(directory)
filenameList= []
RGBList= []
for eachFile in fileList:
filenameList.append(eachFile)
RGBList.append(cv2.imread(directory + eachFile, 1).tostring())
df1["Filenames"]= filenameList
df2["RGB"]= RGBList
df1.to_csv('df1.csv')
df2.to_csv('df2.csv')
df1 functions as desired. I THINK df2 fuctions as intended. A print statement reveals the correct len of 30,000 for each row of the csv. However, when I read in the csv using pandas.read_csv('df2') and use a print statement to view the len of the first row, I get 110541. I intend to use np.fromstring() and np.reshape() to reshape the flattened np.array generated from np.tostring(), but I get the error:
ValueError: string size must be a multiple of element size
...because the number of elements is mismatched.
My question is:
Why is the len so much larger when I read in the csv?
Is there a more efficient way to write 3 channel color image pixel data to a csv that can easily be read back in?
If you write a single byte for each 8-bit pixel you will get a line with 1 byte per pixel. So, if your image is 80 pixels wide, you will get 80 bytes per line.
If you write a CSV, in human-readable ASCII, you will need more space. Imagine the first pixel is 186. So, you will write a 1, an 8, a 6 and a comma - i.e. 4 bytes now for the first pixel instead of a single byte in binary, and so on.
That means your file will be around 3-4x bigger, i.e. 110k instead of 30k, which is what you are seeing.
There is no "better way" to write a CSV - the problem is that is a fundamentally inefficient format designed for humans rather than computers. Why did you choose CSV? If it has to be legible for humans, you have no choice.
If it can be illegible to humans, but readily legible to computers, choose a different format such np.save() and np.load() - as you wisely have done already ;-)
I have some binary input files (extension ".bin") that describe a 2D field of ocean depth and which are all negative float numbers. I have been able to load them in matlab as follows:
f = fopen(filename,'r','b');
data = reshape(fread(f,'float32'),[128 64]);
This matlab code gives me double values between 0 and -5200. However, when I try to the same in Python, I strangely get values between 0 and 1e-37. The Python code is:
f = open(filename, 'rb')
data = np.fromfile(f, np.float32)
data.shape = (64,128)
The strange thing is that there is a mask value of 0 for land which shows up in the right places in the (64,128) array in both cases. It seems to just be the magnitude and sign of the numpy.float32 values that are off.
What am I doing wrong in the Python code?
numpy.fromfile isn't platform independant, especially the "byte-order" is mentioned in the documentation:
Do not rely on the combination of tofile and fromfile for data storage, as the binary files generated are are not platform independent. In particular, no byte-order or data-type information is saved.
You could try:
data = np.fromfile(f, '>f4') # big-endian float32
and:
data = np.fromfile(f, '<f4') # little-endian float32
and check which one (big endian or little endian) gives the correct values.
Base on your matlab fopen, the file is in big endian ('b'). But your python code does not take care of the endianness.
Background
The binary file contain successive raw output from a camera sensor which is in the form of a bayer pattern. i.e. the data is successive blocks containing information of the form shown below and where each block is a image in image stream
[(bayer width) * (bayer height) * sizeof(short)]
Objective
To read information from a specific block of data and store it as an array for processing. I was digging through the opencv documentation, and totally lost on how to proceed. I apologize for the novice question but any suggestions?
Assuming you can read the binary file (as a whole), I would try to use
Numpy to read it into a numpy.array. You can use numpy.fromstring and depending on the system the file was written on (little or big endian), use >i2 or <i2 as your data type (you can find the list of data types here).
Also note that > means big endian and < means little endian (more on that here)
You can set an offset and specify the length in order to read to read a certain block.
import numpy as np
with open('datafile.bin','r') as f:
dataBytes = f.read()
data = np.fromstring(dataBytes[blockStartIndex:blockEndIndex], dtype='>i2')
In case you cannot read the file as a whole, I would use mmap (requires a little knowledge of C) in order to break it down to multiple files and then use the method above.
OP here, with #lsxliron's suggestion I looked into using Numpy to achieve my goals and this is what I ended up doing
import numpy as np
# Data read length
length = (bayer width) * (bayer height)
# In terms of bytes: short = 2
step = 2 * length
# Open filename
img = open("filename","rb")
# Block we are interested in i
img.seek(i * step)
# Copy data as Numpy array
Bayer = np.fromfime(img,dtype=np.uint16,count=length)
Bayer now holds the bayer pattern values in the form of an numpy array success!!
I am reading in an image one byte at a time with with read(1), and appending it to a list. The image data is all hex data. When I print out the list with the print function it is in the format '\xd7'
['\xd7', '\xd7', '\xd7', '\xd7', '\xd7', '\xd7', '\xd7',...]
The problem is that now I need to perform some calculations on this hex data, however, it is in string format, and this '\xd' string format isn't supported by any of the int or hex conversion functions in python. They require a '0xd7' or just a 'd7'.
Thanks for the help
It's interpreting them as characters, so use ord to turn them into numbers. I.e. ord('\xd7') gives 215.
Also if you use Windows, or the program might have to run on Windows, make sure that you've got the file open in binary mode: open("imagefile.png","rb"). Makes no difference on other operating systems.
If you require 'd7' or '0xd7', rather than simply 0xd7 (viz, 215), hex() or '%x' are your friend.
>>> ord('\xd7')
215
>>> ord('\xd7') == 215 == 0xd7
True
>>> hex(ord('\xd7'))
'0xd7'
>>> '%x' % ord('\xd7')
'd7'
Also as observed in other answers, do make sure you open with the 'b' in the mode, otherwise it can get messed up, thinking it's UTF-8 or something like that, on certain sequences of bytes.
You could do something like this to get them into a numeric array:
import array
data = array.array('B') # array of unsigned bytes
with open("test.dat", 'rb') as input:
data = input.read(100)
data.fromstring(data)
print data
# array('B', [215, 215, 215, 215, 215, 215, 215])
read() can take a size value larger than 1: read(1024) will read 1K worth of bytes from the stream. That will be a lot faster than reading a byte at a time and appending it to the previous bytes.
What are you trying to do when printing the data? See the byte values, or display the image?
The data isn't in "string format", it's just bytes, but when you print them the print routine will escape non-printing values into something that will mean more to human eyes and brains. If you want to see the values without the escaping you can iterate over the bytes and convert them to their hexadecimal values, or decimal, or binary - whatever works for you and your application. The string formatting mini-language will be a good starting place.
If you are doing image processing, then you probably want to look at numpy.
There are a few packages that will help you read your image into memory too (PIL is mentioned above, another is my own mahotas or scikits.image).
If the data is in a file as raw data an you know the dimensions, you can do the following
import numpy as np
img = np.empty( (n_rows, n_cols), dtype=np.uint8) # create an empty image
img.data[:] = input_file.read()
to get your data into img.
An introductory website for image processing in python is http://pythonvision.org.