so I think questions like this have been asked before but I'm having quite a bit of trouble getting this implemented.
I'm dealing with CSV files that contain floating points between -1 and 1. All of these floating points have to be converted to 16 bit 2s complement without the leading '0b'. From there, I will convert that number to a string representation of the 2s complement, and all of those from the CSV will be written will be written to a .dat file with no space in between. So for example, if I read in the CSV file and it has two entries [0.006534, -.1232], I will convert each entry to their respective 2s complement and write them one after another onto a .dat file.
The problem is I'm getting stuck in my code on how to convert the floating point to a 16 bit 2s complement. I've been looking at other posts like this and I've been told to use the .float() function but I've had no luck.
Can someone help me write a script that will take in a floating point number, and return the 16 bit 2s complement string of it? It has to be exactly 16 bits because I'm dealing with the MIT 16 standard.
I am using python 3.4 btw
To answer the question in the title: to convert a Python float to IEEE 754 half-precision binary floating-point format, you could use binary16:
>>> from binary16 import binary16
>>> binary16(0.006534)
b'\xb0\x1e'
>>> binary16(-.1232)
b'\xe2\xaf'
numpy produces similar results:
>>> import numpy as np
>>> np.array([0.006534, -.1232], np.float16).tostring()
b'\xb1\x1e\xe3\xaf'
>>> np.array([0.006534, -.1232], '>f2').tostring() # big-endian
b'\x1e\xb1\xaf\xe3'
My goal was to save the amplitudes as the ecg mit signal format 16
..snip..
the input is a .CSV file containing the f.p. values of the amplitude from a .WAV file (which is the recording of an ECG).
You could read the wav file directly and write the corresponding 16-bit two's complement amplitudes in little-endian byte order where any unused high-order bits are sign-extended from the most significant bit ('<h' struct format):
#!/usr/bin/env python3
import wave
with wave.open('ecg.wav') as wavfile, open('ecg.mit16', 'wb') as output_file:
assert wavfile.getnchannels() == 1 # mono
assert wavfile.getsampwidth() == 2 # 16bit
output_file.writelines(iter(lambda: wavfile.readframes(4096), b''))
There is a bug in Python 3 that .readframes() returns str instead of bytes sometimes. To workaround it, use if not data test that works on both empty str and bytes:
#!/usr/bin/env python3
import wave
with wave.open('ecg.wav') as wavfile, open('ecg.mit16', 'wb') as output_file:
assert wavfile.getnchannels() == 1 # mono
assert wavfile.getsampwidth() == 2 # 16bit
while True:
data = wavfile.readframes(4096)
if not data:
break
output_file.write(data)
Related
I'm reading some bytes from a binary file and I'm trying to convert them to decimals. They are big endian, so I try to unpack them as unpack('>f',bytes), but I'm getting the wrong result.
According to the specification I should be looking for
Fixed point numbers with 8 bits before the binary point and 24 bits
after the binary point. Three guard bits are reserved in the points to
eliminate most concerns over arithmetic overflow. Hence, the range for each component is 0xF0000000 to 0x0FFFFFFF representing a range of -16 to 16.
As an example I'm using 0x00d4f9c1, which should give me 0,831936, but I'm getting 1,95587[...]e-38.
The f designator is for floating point, which is entirely different from fixed point. You just need to convert that to an integer and divide by 2**24.
>>> x = 0x00d4f9c1
>>> x/(1<<24)
0.8319359421730042
>>>
I would like to parse and split a hexadecimal bytearray read from the vehicle CAN. The entire 64 bits were read as bytearray(b'\xa0\xc6\xa6\xc2\x06\xe3)B'), and I would like to split it based on bit positions. For example, I need the first 6 bits of 101000.
Based on some reading from here, I've completed the conversion from hex bytearray to binary string, and parse it successfully. My current approach is:
orig_data = bytearray(b'\xa0\xc6\xa6\xc2\x06\xe3)B')
def hex2bin(hex_string):
scale = 16
num_of_bits = 8
return bin(int(hex_string, scale))[2:].zfill(num_of_bits)
bin_str = hex2bin(bytes(orig_data).hex())
print(bin_str[:6])
Since I need to deal with a vast amount of data transferring at high speed, I was wondering if there is a faster way to do that than the current approach I adopted?
I am looking for a fast, preferably standard library mechanism to determine the bit-depth of wav file e.g. '16-bit' or '24-bit'.
I am using a subprocess call to Sox to get a plethora of audio metadata but a subprocess call is very slow and the only information I can only currently get reliably from Sox is the bit-depth.
The built in wave module does not have a function like "getbitdepth()" and is also not compatible with 24bit wav files - I could use a 'try except' to access the files metadata using the wave module (if it works, manually record that it is 16bit) then on except call sox instead (where sox will perform the analysis to accurately record its bitdepth). My concern is that that this approach feels like guess work. What if a an 8bit file is read? I would be manually assigning 16-bit when it is not.
SciPy.io.wavefile also is not compatible with 24bit audio so creates a similar issue.
This tutorial is really interesting and even includes some really low level (low level for Python at least) scripting examples to extract information from the wav files headers - unfortunately these scripts don't work for 16-bit audio.
Is there any way to simply (and without calling sox) determine what bit-depth the wav file I'm checking has?
The wave header parser script I'm using is as follows:
import struct
import os
def print_wave_header(f):
'''
Function takes an audio file path as a parameter and
returns a dictionary of metadata parsed from the header
'''
r = {} #the results of the header parse
r['path'] = f
fin = open(f,"rb") # Read wav file, "r flag" - read, "b flag" - binary
ChunkID=fin.read(4) # First four bytes are ChunkID which must be "RIFF" in ASCII
r["ChunkID"]=ChunkID
ChunkSizeString=fin.read(4) # Total Size of File in Bytes - 8 Bytes
ChunkSize=struct.unpack('I',ChunkSizeString) # 'I' Format is to to treat the 4 bytes as unsigned 32-bit inter
TotalSize=ChunkSize[0]+8 # The subscript is used because struct unpack returns everything as tuple
r["TotalSize"]=TotalSize
DataSize=TotalSize-44 # This is the number of bytes of data
r["DataSize"]=DataSize
Format=fin.read(4) # "WAVE" in ASCII
r["Format"]=Format
SubChunk1ID=fin.read(4) # "fmt " in ASCII
r["SubChunk1ID"]=SubChunk1ID
SubChunk1SizeString=fin.read(4) # Should be 16 (PCM, Pulse Code Modulation)
SubChunk1Size=struct.unpack("I",SubChunk1SizeString) # 'I' format to treat as unsigned 32-bit integer
r["SubChunk1Size"]=SubChunk1Size
AudioFormatString=fin.read(2) # Should be 1 (PCM)
AudioFormat=struct.unpack("H",AudioFormatString) ## 'H' format to treat as unsigned 16-bit integer
r["AudioFormat"]=AudioFormat[0]
NumChannelsString=fin.read(2) # Should be 1 for mono, 2 for stereo
NumChannels=struct.unpack("H",NumChannelsString) # 'H' unsigned 16-bit integer
r["NumChannels"]=NumChannels[0]
SampleRateString=fin.read(4) # Should be 44100 (CD sampling rate)
SampleRate=struct.unpack("I",SampleRateString)
r["SampleRate"]=SampleRate[0]
ByteRateString=fin.read(4) # 44100*NumChan*2 (88200 - Mono, 176400 - Stereo)
ByteRate=struct.unpack("I",ByteRateString) # 'I' unsigned 32 bit integer
r["ByteRate"]=ByteRate[0]
BlockAlignString=fin.read(2) # NumChan*2 (2 - Mono, 4 - Stereo)
BlockAlign=struct.unpack("H",BlockAlignString) # 'H' unsigned 16-bit integer
r["BlockAlign"]=BlockAlign[0]
BitsPerSampleString=fin.read(2) # 16 (CD has 16-bits per sample for each channel)
BitsPerSample=struct.unpack("H",BitsPerSampleString) # 'H' unsigned 16-bit integer
r["BitsPerSample"]=BitsPerSample[0]
SubChunk2ID=fin.read(4) # "data" in ASCII
r["SubChunk2ID"]=SubChunk2ID
SubChunk2SizeString=fin.read(4) # Number of Data Bytes, Same as DataSize
SubChunk2Size=struct.unpack("I",SubChunk2SizeString)
r["SubChunk2Size"]=SubChunk2Size[0]
S1String=fin.read(2) # Read first data, number between -32768 and 32767
S1=struct.unpack("h",S1String)
r["S1"]=S1[0]
S2String=fin.read(2) # Read second data, number between -32768 and 32767
S2=struct.unpack("h",S2String)
r["S2"]=S2[0]
S3String=fin.read(2) # Read second data, number between -32768 and 32767
S3=struct.unpack("h",S3String)
r["S3"]=S3[0]
S4String=fin.read(2) # Read second data, number between -32768 and 32767
S4=struct.unpack("h",S4String)
r["S4"]=S4[0]
S5String=fin.read(2) # Read second data, number between -32768 and 32767
S5=struct.unpack("h",S5String)
r["S5"]=S5[0]
fin.close()
return r
Esentially the same answer as from Matthias, but with copy-pastable code.
Requirements
pip install soundfile
Code
import soundfile as sf
ob = sf.SoundFile('example.wav')
print('Sample rate: {}'.format(ob.samplerate))
print('Channels: {}'.format(ob.channels))
print('Subtype: {}'.format(ob.subtype))
Explanation
Channels: Usually 2, meaning you have one left speaker and one right speaker.
Sample rate: Audio signals are analog, but we want to represent them digitally. Meaning we want to discretize them in value and in time. The sample rate gives how many times per second we get a value. The unit is Hz. The sample rate needs to be at least double of the highest frequency in the original sound, otherwise you get aliasing. Human hearing range goes from ~20Hz to ~20kHz, so you can cut off anything above 20kHZ. Meaning a sample rate of more than 40kHz does not make much sense.
Bit-depth: The higher the bit-depth, the more dynamic range can be captured. Dynamic range is the difference between the quietest and loudest volume of an instrument, part or piece of music. A typical value seems to be 16 bit or 24 bit. A bit-depth of 16 bit has a theoretical dynamic range of 96 dB, whereas 24 bit has a dynamic range of 144 dB (source).
Subtype: PCM_16 means 16 bit depth, where PCM stands for Pulse-Code Modulation.
Alternative
If you only look for a command line tool, then I can recommend MediaInfo:
$ mediainfo example.wav
General
Complete name : example.wav
Format : Wave
File size : 83.2 MiB
Duration : 8 min 14 s
Overall bit rate mode : Constant
Overall bit rate : 1 411 kb/s
Audio
Format : PCM
Format settings : Little / Signed
Codec ID : 1
Duration : 8 min 14 s
Bit rate mode : Constant
Bit rate : 1 411.2 kb/s
Channel(s) : 2 channels
Sampling rate : 44.1 kHz
Bit depth : 16 bits
Stream size : 83.2 MiB (100%)
I highly recommend the soundfile module (but mind you, I'm very biased because I wrote a large part of it).
There you can open your file as a soundfile.SoundFile object, which has a subtype attribute that holds the information you are looking for.
In your case that would probably be 'PCM_16' or 'PCM_24'.
Not clear when this update went out but the built in wave module appears to be compatible with 24 bit wav files. I'm using python 3.10.5
The wave_read sampwidth() method states that it returns bytes. I'm fairly sure just taking this value and multiplying by 8 will give us bit depth. For example:
with wave.open(path, 'rb') as wav:
bit_depth = wav.getsampwidth() * 8
getsampwidth() returns 2 for a 16 bit file and 3 for a 24 bit. No additional modules or subprocesses needed!
I'm reading a binary file with signal samples both in Octave and Python.
The thing is, I want to obtain the same values for both codes, which is not the case.
The binary file is basically a signal in complex format I,Q recorded as a 16bits Int.
So, based on the Octave code:
[data, cnt_data] = fread(fid, 2 * secondOfData * fs, 'int16');
and then:
data = data(1:2:end) + 1i * data(2:2:end);
It seems simple, just reading the binary data as 16 bits ints. And then creating the final array of complex numbers.
Threfore I assume that in Python I need to do as follows:
rel=int(f.read(2).encode("hex"),16)
img=int(f.read(2).encode("hex"),16)
in_clean.append(complex(rel,img))
Ok, the main problem I have is that both real and imaginary parts values are not the same.
For instance, in Octave, the first value is: -20390 - 10053i
While in Python (applying the code above), the value is: (23216+48088j)
As signs are different, the first thing I thought was that maybe the endianness of the computer that recorded the file and the one I'm using for reading the file are different. So I turned to unpack function, as it allows you to force the endian type.
I was not able to find an "int16" in the unpack documentation:
https://docs.python.org/2/library/struct.html
Therefore I went for the "i" option adding "x" (padding bytes) in order to meet the requirement of 32 bits from the table in the "struct" documentation.
So with:
struct.unpack("i","xx"+f.read(2))[0]
the result is (-1336248200-658802568j) Using
struct.unpack("<i","xx"+f.read(2))[0] provides the same result.
With:
struct.unpack(">i","xx"+f.read(2))[0]
The value is: (2021153456+2021178328j)
With:
struct.unpack(">i",f.read(2)+"xx")[0]
The value is: (1521514616-1143441288j)
With:
struct.unpack("<i",f.read(2)+"xx")[0]
The value is: (2021175386+2021185723j)
I also tried with numpy and "frombuffer":
np.frombuffer(f.read(1).encode("hex"),dtype=np.int16)
With provides: (24885+12386j)
So, any idea about what I'm doing wrong? I'd like to obtain the same value as in Octave.
What is the proper way of reading and interpreting the values in Python so I can obtain the same value as in Octave by applying fread with an'int16'?
I've been searching on the Internet for an answer for this but I was not able to find a method that provides the same value
Thanks a lot
Best regards
It looks like the binary data in your question is 5ab0bbd8. To unpack signed 16 bit integers with struct.unpack, you use the 'h' format character. From that (23216+48088j) output, it appears that the data is encoded as little-endian, so we need to use < as the first item in the format string.
from struct import unpack
data = b'\x5a\xb0\xbb\xd8'
# The wrong way
rel=int(data[:2].encode("hex"),16)
img=int(data[2:].encode("hex"),16)
c = complex(rel, img)
print c
# The right way
rel, img = unpack('<hh', data)
c = complex(rel, img)
print c
output
(23216+48088j)
(-20390-10053j)
Note that rel, img = unpack('<hh', data) will also work correctly on Python 3.
FWIW, in Python 3, you could also decode 2 bytes to a signed integer like this:
def int16_bytes_to_int(b):
n = int.from_bytes(b, 'little')
if n > 0x7fff:
n -= 0x10000
return n
The rough equivalent in Python 2 is:
def int16_bytes_to_int(b):
lo, hi = b
n = (ord(hi) << 8) + ord(lo)
if n > 0x7fff:
n -= 0x10000
return n
But having to do that subtraction to handle signed numbers is annoying, and using struct.unpack is bound to be much more efficient.
I'd like to get the exact sequence of bits from a file into a string using Python 3. There are several questions on this topic which come close, but don't quite answer it. So far, I have this:
>>> data = open('file.bin', 'rb').read()
>>> data
'\xa1\xa7\xda4\x86G\xa0!e\xab7M\xce\xd4\xf9\x0e\x99\xce\xe94Y3\x1d\xb7\xa3d\xf9\x92\xd9\xa8\xca\x05\x0f$\xb3\xcd*\xbfT\xbb\x8d\x801\xfanX\x1e\xb4^\xa7l\xe3=\xaf\x89\x86\xaf\x0e8\xeeL\xcd|*5\xf16\xe4\xf6a\xf5\xc4\xf5\xb0\xfc;\xf3\xb5\xb3/\x9a5\xee+\xc5^\xf5\xfe\xaf]\xf7.X\x81\xf3\x14\xe9\x9fK\xf6d\xefK\x8e\xff\x00\x9a>\xe7\xea\xc8\x1b\xc1\x8c\xff\x00D>\xb8\xff\x00\x9c9...'
>>> bin(data[:][0])
'0b11111111'
OK, I can get a base-2 number, but I don't understand why data[:][x], and I still have the leading 0b. It would also seem that I have to loop through the whole string and do some casting and parsing to get the correct output. Is there a simpler way to just get the sequence of 01's without looping, parsing, and concatenating strings?
Thanks in advance!
I would first precompute the string representation for all values 0..255
bytetable = [("00000000"+bin(x)[2:])[-8:] for x in range(256)]
or, if you prefer bits in LSB to MSB order
bytetable = [("00000000"+bin(x)[2:])[-1:-9:-1] for x in range(256)]
then the whole file in binary can be obtained with
binrep = "".join(bytetable[x] for x in open("file", "rb").read())
If you are OK using an external module, this uses bitstring:
>>> import bitstring
>>> bitstring.BitArray(filename='file.bin').bin
'110000101010000111000010101001111100...'
and that's it. It just makes the binary string representation of the whole file.
It is not quite clear what the sequence of bits is meant to be. I think it would be most natural to start at byte 0 with bit 0, but it actually depends on what you want.
So here is some code to access the sequence of bits starting with bit 0 in byte 0:
def bits_from_char(c):
i = ord(c)
for dummy in range(8):
yield i & 1
i >>= 1
def bits_from_data(data):
for c in data:
for bit in bits_from_char(c):
yield bit
for bit in bits_from_data(data):
# process bit
(Another note: you would not need data[:][0] in your code. Simply data[0] would do the trick, but without copying the whole string first.)
To convert raw binary data such as b'\xa1\xa7\xda4\x86' into a bitstring that represents the data as a number in binary system (base-2) in Python 3:
>>> data = open('file.bin', 'rb').read()
>>> bin(int.from_bytes(data, 'big'))[2:]
'1010000110100111110110100011010010000110...'
See Convert binary to ASCII and vice versa.