I work in a lab where we acquire electrophysiological recordings (across 4 recording channels) using custom Labview VIs, which save the acquired data as a .DAT (binary) file. The analysis of these files can then be continued in more Labview VIs, however I would like to analyse all my recordings in Python. First, I need walk through all of my files and convert them out of binary!
I have tried numpy.fromfile (filename), but the numbers I get out make no sense to me:
array([ 3.44316221e-282, 1.58456331e+029, 1.73060724e-077, ...,
4.15038967e+262, -1.56447362e-090, 1.80454329e+070])
To try and get further I looked up the .DAT header format to understand how to grab the bytes and translate them - how many bytes the data is saved in etc:
http://zone.ni.com/reference/en-XX/help/370859J-01/header/header/headerallghd_allgemein/
But I can't work out what to do. When I type "head filename" into terminal, below is what I see.
e.g. >> head 2014_04_10c1slice2_rest.DAT
DTL?
0???? ##????
empty array
PF?c ƀ????l?N?"1'.+?K13:13:27;0.00010000-08??t???DY
??N?t?x???D?
?uv?tgD?~I??
??N?t?x>?n??
????t?x>?n??
????t???D?
????t?x???D?
????t?x?~I??
????tgD>?n??
??N?tgD???D?
??N?t?x???D?
????t????DY
??N?t?x>?n??
??N?t????DY
?Kn$?t?????DY
??N?t??>?n??
??N?tgD>?n??
????t?x?~I??
????tgD>?n??
??N?tgD>?n??
??N?tgD???DY
????t?x???D?
????t???~I??
??N?tgD???DY
??N?tgD???D?
??N?t?~I??
??N?t?x???DY
??N?tF>?n??
??N?t?x??%Y
Any help or suggestions on what to do next would be really appreciated.
Thanks.
P.s. There is an old (broken) matlab file that seems to have been intended to convert these files. I think this could probably be helpful, but having spent a couple of days trying to understand it I am still stuck. http://www.mathworks.co.uk/matlabcentral/fileexchange/27195-load-labview-binary-data
Based on this link it looks like the following should do the trick:
binaryFile = open('Measurement_4.bin', mode='rb')
(data.offset,) = struct.unpack('>d', binaryFile.read(8))
Note that mode is set to 'rb' for binary.
With numpy you can directly do this as
data = numpy.fromfile('Measurement_4.bin', dtype='>d')
Please note that if you are just using Python as an intermediate and want to go back to LabVIEW with the data, you should instead use the function Read from Binary file.vi to read the binary file using native LabVIEW.
DAT is a pretty generic suffix, not necessarily something pointing to a specific format. If I'm understanding correctly, that help section is for DIAdem, which may be completely unrelated to how your data is saved from LV.
What you want is this help section, which tells you how LV flattens data to be stored on disk - http://zone.ni.com/reference/en-XX/help/371361J-01/lvconcepts/flattened_data/
You will need to look at the LV code to see exactly what kind of data you're saving and how the write file function is configured (byte order, size prepending, etc.) and then use that document to translate it to the actual representation.
Related
I'm trying to write a script to simplify my everyday life in the lab. I operate one ThermoFisher / FEI scanning electron microscope and I save all my pictures in the TIFF format.
The microscope software is adding an extensive custom TiffTag (code 34682) containing all the microscope / image parameters.
In my script, I would like to open an image, perform some manipulations and then save the data in a new file, including the original FEI metadata. To do so, I would like to use a python script using the tifffile module.
I can open the image file and perform the needed manipulations without problems. Retrieving the FEI metadata from the input file is also working fine.
I was thinking to use the imwrite function to save the output file and using the extratags optional argument to transfer to the output file the original FEI metadata.
This is an extract of the tifffile documentation about the extratags:
extratags : sequence of tuples
Additional tags as [(code, dtype, count, value, writeonce)].
code : int
The TIFF tag Id.
dtype : int or str
Data type of items in 'value'. One of TIFF.DATATYPES.
count : int
Number of data values. Not used for string or bytes values.
value : sequence
'Count' values compatible with 'dtype'.
Bytes must contain count values of dtype packed as binary data.
writeonce : bool
If True, the tag is written to the first page of a series only.
Here is a snippet of my code.
my_extratags = [(input_tags['FEI_HELIOS'].code,
input_tags['FEI_HELIOS'].dtype,
input_tags['FEI_HELIOS'].count,
input_tags['FEI_HELIOS'].value, True)]
tifffile.imwrite('output.tif', data, extratags = my_extratags)
This code is not working and complaining that the value of the extra tag should be ASCII 7-bit encoded. This looks already very strange to me because I haven't touched the metadata and I am just copying it to the output file.
If I convert the metadata tag value in a string as below:
my_extratags = [(input_tags['FEI_HELIOS'].code,
input_tags['FEI_HELIOS'].dtype,
input_tags['FEI_HELIOS'].count,
str(input_tags['FEI_HELIOS'].value), True)]
tifffile.imwrite('output.tif', data, extratags = my_extratags)
the code is working, the image is saved, the metadata corresponding to 'FEI_HELIOS' is created but it is empty!
Can you help me in finding what I am doing wrongly?
I don't need to use tifffile, but I would prefer to use python rather than ImageJ because I have already several other python scripts and I would like to integrate this new one with the others.
Thanks a lot in advance!
toto
ps. I'm a frequent user of stackoverflow, but this is actually my first question!
In principle the approach is correct. However, tifffile parses the raw values of certain tags, including FEI_HELIOS, to dictionaries or other Python types. To get the raw tag value for rewriting, it needs to be read from file again. In these cases, use the internal TiffTag._astuple function to get an extratag compatible tuple of the tag, e.g.:
import tifffile
with tifffile.TiffFile('FEI_SEM.tif') as tif:
assert tif.is_fei
page = tif.pages[0]
image = page.asarray()
... # process image
with tifffile.TiffWriter('copy1.tif') as out:
out.write(
image,
photometric=page.photometric,
compression=page.compression,
planarconfig=page.planarconfig,
rowsperstrip=page.rowsperstrip,
resolution=(
page.tags['XResolution'].value,
page.tags['YResolution'].value,
page.tags['ResolutionUnit'].value,
),
extratags=[page.tags['FEI_HELIOS']._astuple()],
)
This approach does not preserve Exif metadata, which tifffile cannot write.
Another approach, since FEI files seem to be written uncompressed, is to directly memory map the image data in the file to a numpy array and manipulate that array:
import shutil
import tifffile
shutil.copyfile('FEI_SEM.tif', 'copy2.tif')
image = tifffile.memmap('copy2.tif')
... # process image
image.flush()
Finally, consider tifftools for rewriting TIFF files where tifffile is currently failing, e.g. Exif metadata.
I have a pretty well functioning patch that uses two cameras, and employs cv.jit.track to track an object in "3D".
The console prints out sets of coordinates as the object moves through space (format [x,y,z]). My end goal is to translate this data into 3D modelling software to map a path that the object has travelled. One option for achieving this is a python script like this.
I need to determine how exactly to send the stream of data from the Max console outside of Max. Perhaps this is into a Python application, or a table, or literally a text file. Any suggestions are greatly appreciated!
excuse the code dump:
<pre><code>
----------begin_max5_patcher----------
3299.3oc6cz1aaia9yI+JDL5F1.7L3qhjGv.51ssaeaC31vvgtCExxpIpUVx
UVNuzC2+8IQRESEKISIJkljUihj7XIa879a7gr+xkWrXc1cQ6W38cduy6hK9
kKu3B4aU8FWnguXw1f6BSB1KusEoQ2ls9iKVptTQzcEx2N2KDV+lYGJRhJJt
eWj5KdwBueVeocAEgWGmd06yiBKTWEAvq.K8fLX0uPB4OQq.O7YROrMNs7KT
97A52Ldi7wVhJ+Ae+EGuS0yVdqvp27Wu7xperzYpCMNpi4yjTmPLVpiNcT21
n86CtJ5DxaeTgGv6MPu23FUhP9U+xm2OUhZgJIu.nRslJBPGKUBmcM08F1gs
PBXBZE17ECtziTQDK8vv9IH3oDDsCBBLoDDpGBR.jlTDDdjj.gMcjPWZdWEU
bylnaRh2WLNMONU4iTQrsn4su39jHSb3GxR1rvR0Rn+7a7UQ+wgQkleiiCHv
hUH.h.XB8qREWHDrRPfTAP+BJJ4NRePHnA24CYoE6i+h7AAgqnBJj6W+8H5i
KU8ISC1pXs+o73fjEsv+3SG+6v1nzCKLd5eHHLxLzvIrs3zRkpRt2xwvAY3U
bHlyv0uGujqRnmne0fCV4s3wpcR71ToKtHZqNwhE+sRZ3eEuMx6u+W799RtY
dPE1tr5G+0+wO58ehVGFr06eWDmDWbu2eNp330+wzfcO9y78A6CCVmD48Oyy
ze3E8auamXTbzOSd4MWDk+9nzpGjI+uoHFOg4XT90F4U6mvKNcW4ioWOFiVp
SjtIgH3DXofGBKFAVLYrwtCymtw6KdAIIiySiO2WEQQlRCUL3n7Hzz4N3iwE
qBiRRVmjE9o5u1a1GlmURdFZks4o35pOXXVRVthv.q3XeJyeYi+RfwbJpTav
fBOggozCaSkaeTx1rMMdtqyx2Dk23ACVZ7Cymz0QAO9dYDLrRIss+x7it9pF
eLg.gf7ks9Wler1xdkHk3XgRvqhe.L9L4Y2dh6jZqDwCNCKuqqihu5Z42J5A
ovoBqR7913MEWW8dDp0ge9gnznaBdvGUI9GuONKU9szhVHt9NuJOdSRbZjcR
jl5rjin72zgeNqCq8Z8JSG1+4jN7taiS2jcamQUJum2uMnHO9tyGUVkHOT63
AIofxfzCntETGIchlvPoqCRuZjMDfJqQgIighoCNFJFL+8zQluU4W99n9Swx
BAIVHITeUUZhNRn5nY13q0.imNwdGLlJc8Co6BB+jG1Rk89frIYKLU0REwfK
eG2QiiHSG+H7lUUrjh7RNxpM4AV4AvBhFIHU+R.Ft0Ac1sNLIZu2ltKqrLy8
dPu2lGrI9vdO1T3FrlQnbV.43gK98eRLGxuZMJ4v1fojngRpkM7NVgYyum+v
jr9bK1aup.LidUgYCW6lO+siJaWTpSQ1pIugGZiL+g1pTY+bwpqxCVWkbQUl
Edu8PZ7mOD4AmPcXHU8KcK2FTaWgyuRryscEURlByWDbo+Z3rzCVxy+dvhY3
S6kj.8rnEruD5.aq7uxLe9VGXqWerWgMfsUgtZ0p9Jz.V9h4lKtK3SkEjq92
3bynJJ1mxrYUVws3K.N+CDPvtcUsY9mGIEJMuYkEMJWCVHCPfkkPBDR2ACMT
JDL+kCrKORRvuyIBtr93SIXwHHXxS.AK8peFJhigpDDXDguOfHph9gQCOaH9
7uN5RJ5Cd+ljM+W4qkM9KmjqnVjqPDc3Vt.77mEjb.P7dC1EJ1WfFtoaKY8I
lvRcBy1VlAPwokxEkj38S8nJiTYzR0sNlbRqiOm1KaJ0dOrccTdugbkckr2z
1Ks.KqI43KjzOiT7PAC13jgFdZISYYLiMNJT.ZgM.Ty5CZEUkFR6YdfmfVU5
KlcuzEdq6ofVU6qU4mOb8EMiRYNLTF0fR6c9aoaPX3gs8kP1GRxBFCShHidW
5.YLZSCJyU1721jvqKUX5ew.zk5cNMJc+QnBcfQLQfPLAfhKcvxGrGHevrm1
9zQ6Pnj.obY96ifZohWPTqpXEGH1Irhrtx.XeXd7tBuhay9Nu679idah1Ub8
hyOALL5oZuPc.zgje.EO6Y2Vh7Qw2D48Eml4GJDbJESYCdlsoSYAYSPBQ0jG
k0B4M7DhnrmlDhNi9bVZTk97u06due28kp0e42u3b1oD0JLNJkXzSlR78iLe
OsiWfLkkwn19Drn6ZR7NWZMz3oPh34kgYsHiBGYsID+ut0lHG1x6G+vVpJmF
y7G4rVRdR1cLkz3cNSi3wNOoDx2lmzWTyhGjCTZ07WSyhG4ayS5+OoCidMpC
SetnCOEIOnSFZz4Nfeh5q4DO6rX8tWQEOcTyNKV7bd1YgBzwLEFw.FQeYL5r
Z4HlKdhFcVj3U0nyV6gVYGLhQmERe4M5rZhFoJXXDiNKD+0ZzYmBeePCucVr
DqsIzQu3FXVMQC4vQNvrP3y5AlEobDpVE1QLurPvy44kUGMSSciXIxe4Osr0
JvX9XmVV1raz94x7+xy7P8uXpVPk5gIve3s4X5DzEYWcUR2VimcErzdYUaGA
R8OseqYM77pMoR4yYQU0IO5f4QhpUueSRee1g7vZZqdSQ3cDc2DsuHNMnPWW
z6NtwYLtoswaTApzqXffJWRWtz30moFooFPUGkHxDOj2oD5XwxUU61szaZID
5.HDCnSQVfSHqOwFj0uOjkUlVSE5oVU8ZjkHSJ1MFIyYbqFaPMXjnYPpKmey
yhsUiNkmq7k5ujdeRxoRy4GU0Z3e9GkjzQN9np1aEmWZ2TkXprXY1ZwBOmhF
Aa55oIDWs6R8UlKUPSt0rM5f9v9rXPP4PVAUCdlIjqpRTavMhMdZTVylPtha
1n6UUDxY4aHjJco5wosAjZrrfHgZviHNi2UMG3r3MsW4MlBpvFBTuomgq3mb
RaUcMW8kis.SIr9vTAjafolPtha13OkzKWjCk5hPUQQZmARHW8JBbF2pNl6l
GbyFqFDvh3y50dR6nrIDV0zDBeE6waHkpKM0tPsJL9YTF7QyCCG5thZq7Qs+
ItZqc+HnoWswYhP6AEpp0oF0ZLImSUVcVfq85zBWlz9o765cRkv.wcCOIthn
Hg+Jxi20nSOhZSlVbaj89JWEpS0xlPpQrApNFOZ.nFbk1kIyP5XBmEJ.ngh9
SrX.BGfbv.ZpsAcV0tF4floeIgNEUQtYFBbVhqRGa5k31vEQnA30UGqtADog
GYR6djcylhZSYDTlEIcPwsLm6ceMWSxzF7Fws.uwpy3Oc74lPv5txympn0xN
tAOGdi6MATXKaBCrZ3yYTEMwYSCldVNrMsKRO+HlPtVnF2BtHF2u4GTZ+oOr
zD5rieDD1P7KgbEwQSESkPMypuIDVsln0LbAvckAa7Di41T2jZKUUWgGXFZB
qUHKAXOxZBM4oDaSVMP9fKBQzccdBe20ispQlhAi0pI.oyK4HVaS8d9Ct7C8
n810kbsWZ1vnGLNy5Ny8Iv8Lyp715u+ekwZ8OYmFxzqFWWWyUdsUUXzqiip+
Od3TbizCdSbGuoBqWVAX2wCk937UKnmtkQD26EL0FitdiTqRowu9bRQsrXMf
n0y.C+THhf2IuWdsIOiZt0BiyQxP0L.dDXpC9PsovUeaZ4HkC6N6+SulqJUV
Eg2u+7+j56T8RJX.4JtYy5VZQWJ35yfgG.lbguUK6o8XppT61wT2LoH13eCi
NuEkVZa.3Za.rI7LUXihnZZFMgbE2rpD4dWgPJQVpAfazzulPXlpigbUwTL2
S3gXUcmzyi1XlYuzjPOEsZmZeSev.id9f060BkOehAN+XiI77SDD6IBEp1wx
EPbqPNgMMKoWqKeJ7gFhTo6pyUuIjpyOJ+WXt6qiL1lp7obK7Wo2Sjzoxm.1
4DbaOIVT2IYMEtYwVnGHQaX2MB1uaztzUl.NCnsUpAVXooWzE0ljnIfYRBzY
HzqMjfMNKHbCewD9LzGEpygMpiT3CLia36d2yfTa7i0Oaj32rouT1wHb7IKB
GzFGDXgMbQ8Yirpe5MgzKIt1iqDxU71F8TnMRejOwXsOaBgQlK4EFMCIkaGg
fG.gX.M4C2AzVjEdNjUUydMWuIDSePcH0VjPecHDaVNODnAWGLGpH.ab0Q3m
OCYNB1xIXXWWxUST.wpBNsHydFm2Ed2xkbFuwVg2VTHU0+GS0Ede5kZf2p8C
Pvtc2DkWu4lkn7hsAeTs6k4KkfwoJP4dRXQdzMM2LzKBxCuNtHJr3PtZqRdm
9+2UWTse0ySODqUQKYVWpOaoezdP3gcYoZej7SQIIY2Ve7aWxf9Pvgjhlr0f
vvnzhla5dDExjahcNjK2M6.pfRnM210la.z2IOzqqNlw40LmkZYYd429wiAa
MlrsDMhq8NXJ6OR.xsfc0Al8fQelOgAjmT.TABRkTBD.E90aG+gifJgrbKrT
gg62oO3Bj6zkK+0K+e.gM30E
-----------end_max5_patcher-----------
</code></pre>
I am now learning to build a TTS project based on Tacotron-2.
Here, the original code in save_wav(wav, path, sr) function has a step to save a numpy array to .wav file by using
wav *= 32767 / max(0.01, np.max(np.abs(wav)))
scipy.io.wavfile.write(path, hparams.sample_rate, wav.astype(np.int16))
However, after obtained a numpy array using wav *= 32767 / max(0.01, np.max(np.abs(wav))), I want to convert it to a .mp3 file so that it will be easier to send it back as streaming response.
Right now, I can convert .wav bytes object to a .mp3 file, but the problem is that I don't know how to convert the numpy array to a .wav bytes object.
I searched about it and found that it seems like I need to set a header for the numpy array, but in almost all posts that I looked into indicated using modules like scipy.io.wave and audioop, which will first save the numpy array to a .wav file and then with open('filename.wav', 'rb').
(This is the link for scipy.io.wavfile.write module, where the filename param should be string or open file handle which, from my understanding, the generated .wav file will be saved on disk.)
Could anyone give any suggestion on how to achieve this?
Use io.BytesIO
There is a much simpler and more convenient solution using a little hack creating i/o interface of bytes. We can use it like file for write and read:
import io
from scipy.io.wavfile import write
bytes_wav = bytes()
byte_io = io.BytesIO(bytes_wav)
write(byte_io, <audio_sr>, <audio_numpy_array>)
result_bytes = byte_io.read()
Use your data sample rate and values array instead of <audio_sr> and <audio_numpy_array>.
You can operate with result_bytes as bytes of .wav file (as required).
P.S. Also check this simple gist of how to perform values array -> bytes -> values array for wav file.
I finally solved this problem by modifying and creating new modules based on scipy.io.wavfile.write and audio_segment.py of pydub.
Beside, when you want to do operation on wave/mp3 bytes without saving them as a .wav/.mp3 file (normally by using some handful APIs or python package module), you should manually add header for it. It will not be a too-tough task if you look into those excellent package source codes.
I have a bmp file. It is just a red square. I have to write a program with functions to make it have white stripes. Things I would need to do:
load the bmp file.
read and assess the bmp file.
code certain areas coordinates of the file to be colored white.
close the file
display the end product file as output
i am a novice, and am having trouble reading or displaying the original bmp file, let alone edit the content inside. it is not similar to opening a txt file and "readline()". also, when i copy paste the bmp file in the pydev projects src folder in eclipse, it does not show up on eclipse, so i don't know if how the computer would recognize that the file is there. i want to read up on it before posting here, but i don't seem to get much results googling, since i am not sure exactly what i should search for.
The easy way to do this is with a third-party image-processing library like PIL/Pillow. The code is simple enough that you could figure it out in a few minutes from the examples on the Image module docs…
But if you're not allowed to do that, let's look at how to do this manually.
First, BMP isn't a text file format, it's a binary format. That means you have to read it in binary mode. And you can't read it "line by line", because it doesn't have lines of text to read. Since a bytes object isn't mutable, you will probably want to copy it into a bytearray to work with. So:
with open('spam.bmp', 'rb') as f:
data = bytearray(f.read())
Next, you need to parse the BMP file format. I assume the main point of the exercise is figuring out how to do that yourself, so I'll give you a link to Wikipedia's article, which describes it better than the Microsoft docs, and you can go from there.
The struct module in the standard library will be very helpful for interpreting the headers; it's much easier to read a 32-bit little-endian number with struct.unpack_from('<L', data, offset) than with by reading data[offset], data[offset+1], etc. and re-combining them into a 32-bit number.
I'm guessing you can ignore all the options for BMP compression—otherwise, this would be way too hard an assignment. In fact, you can probably just assume that all of the headers will specify the most common variant and only code for that. But you might want to ask your teacher for feedback on that.
Now, once you've found the "pixel array" portion of the BMP, and you've figured out how to interpret it from the DIB header, you can just set pixels to white at whichever positions you want by setting the values at the appropriate indexes of the bytearray. For example, it may turn out to be as simple as:
pos = pixel_array_offset + row_size * y + pixel_size * x
data[pos:pos+3] = 255, 255, 255
Finally, once you've changed your red pixels to white, you can save it with:
with open('eggs.bmp', 'wb') as f:
f.write(data)
I'm reading an image file of dpx format, and want to extract the "Orientation" in the image section of the header, and also modify it. I have never tried to interpret binary data, so I'm a bit at a loss. I'm trying to use the struct module, but I really don't know how to do it properly. The file header specification is here:
http://www.fileformat.info/format/dpx/egff.htm
Thanks.
There seems to be a constant offset to the Orientation so if this is all you want to change then I wouldn't bother trying to parse it all, just work out the offset (which I think is just the size of the GENERICFILEHEADER plus one byte for the high byte of the orientation word) and read / manipulate it directly.
Using a bytearray would be my first choice. The offset varies by one depending on if it's in a big or little endian format, so something like this might work for you:
b = bytearray(your_byte_data)
big_endian = (b[0] == 0x52)
offset = 768 + big_endian
current_orientation = b[offset] # get current orientation
b[offset] = new_offset # set it to something new
open('out_file', 'wb').write(b)
You might want to consider using Imagemagic to do that. Open source and supports the dpx format.
The Python Imaging Library PIL has am attribute .info that might return the relevant data