I am using this function found on GitHub to read some data from a HID-stream in Python. h.read(64)
def decode_bytes(byte_1, byte_2, byte_3, byte_4):
bytes_reversed_and_concatenated = byte_4 * (16 ** 6) + byte_3 * (16 ** 4) + byte_2 * (16 ** 2) + byte_1
bytes_hex = hex(bytes_reversed_and_concatenated)[2:]
bytes_decimal = str(round(struct.unpack('!f', bytes.fromhex(bytes_hex))[0], 1))
return bytes_decimal
The function converts four bytes (in hex-values as integers) from the stream to a Python float-value which is returned as a string. I've read that a C-struct float representation takes up four bytes, so I guess that explains that the function takes four bytes as an input. But apart from that, I'm pretty blank as to how and why the function works.
I have two questions:
First I would very much like to get a better understanding of how the function works. Why does it reverse the byte order and what is up with the 16 ** 6, 16 ** 4 and so on? I am having a hard time figuring out, what that does in Python.
Second I would like to reverse the function. Meaning I would like to be able to supply a float as an input and get out a list of four integer-hex-values, which I can write back via the HID-interface. But I have no idea, where to start.
I was hoping to get some pointers in the right direction. Any help is much appreciated.
So the comment from #user2357112 helped me figure everything out. The working and much simpler function now looks like this:
def decode_bytes(byte_1, byte_2, byte_3, byte_4):
return_value = struct.unpack('<f', bytes([byte_1, byte_2, byte_3, byte_4]))
return str(round(return_value[0], 1))
And if I want to wrap a float back up as a bytes array I do this:
struct.pack('<f', float(float_value))
Also I learned a bit about Endianness along the way. Thanks.
Related
Probably a stupid question but I stumbled across the int.to_bytes() function and I can't explain why I have to add "+7" if I calculate the bytes length and there is no hint in the documentation. I am sure I miss some bit/byte calculation magic but I need a little help from the community here.
Example what confused me: x.to_bytes((x.bit_length() + 7) // 8, byteorder='little') from https://docs.python.org/3/library/stdtypes.html#int.to_bytes
Thanks in advance
bit_length returns the number of bits necessary to represent an integer in binary, excluding the sign and leading zeros. So
x.bit_length() + 7) // 8
will just give you the number of bytes necessary to represent that integer x. You could also write something like
from math import ceil
ceil(x.bit_length / 8)
to get the same number.
The method to_bytes() requires this byte length as its first argument. To account for x == 0 you probably want to include that case to:
x.to_bytes(length=(min(x.bit_length(), 1) + 7) // 8, byteorder='little')
I am new in python and struggling to write a function that takes a Python bytearray as an argument and computes the proportion of ones in it.
So far I have tried below-mentioned code:
def cal_bit_proportion(test):
test = bytearray( 2 )
test[ 0 ] = 0b00000000
test[ 1 ] = 0b00000001
# bp = (need to calculate the proportion)
return bp # bp means bit proportion
It would be really great if someone helps me to solve it by using python 3.4+, or redirect me to appropriate source from where I can learn it.
You can do it like this:
sum(bin(x).count('1') for x in test)
That is, convert each byte in the bytearray to its binary string representation, and count the ones, then sum all of those counts.
Then you can divide by the total number of bits: 8 * len(test).
I'm reading a binary file with signal samples both in Octave and Python.
The thing is, I want to obtain the same values for both codes, which is not the case.
The binary file is basically a signal in complex format I,Q recorded as a 16bits Int.
So, based on the Octave code:
[data, cnt_data] = fread(fid, 2 * secondOfData * fs, 'int16');
and then:
data = data(1:2:end) + 1i * data(2:2:end);
It seems simple, just reading the binary data as 16 bits ints. And then creating the final array of complex numbers.
Threfore I assume that in Python I need to do as follows:
rel=int(f.read(2).encode("hex"),16)
img=int(f.read(2).encode("hex"),16)
in_clean.append(complex(rel,img))
Ok, the main problem I have is that both real and imaginary parts values are not the same.
For instance, in Octave, the first value is: -20390 - 10053i
While in Python (applying the code above), the value is: (23216+48088j)
As signs are different, the first thing I thought was that maybe the endianness of the computer that recorded the file and the one I'm using for reading the file are different. So I turned to unpack function, as it allows you to force the endian type.
I was not able to find an "int16" in the unpack documentation:
https://docs.python.org/2/library/struct.html
Therefore I went for the "i" option adding "x" (padding bytes) in order to meet the requirement of 32 bits from the table in the "struct" documentation.
So with:
struct.unpack("i","xx"+f.read(2))[0]
the result is (-1336248200-658802568j) Using
struct.unpack("<i","xx"+f.read(2))[0] provides the same result.
With:
struct.unpack(">i","xx"+f.read(2))[0]
The value is: (2021153456+2021178328j)
With:
struct.unpack(">i",f.read(2)+"xx")[0]
The value is: (1521514616-1143441288j)
With:
struct.unpack("<i",f.read(2)+"xx")[0]
The value is: (2021175386+2021185723j)
I also tried with numpy and "frombuffer":
np.frombuffer(f.read(1).encode("hex"),dtype=np.int16)
With provides: (24885+12386j)
So, any idea about what I'm doing wrong? I'd like to obtain the same value as in Octave.
What is the proper way of reading and interpreting the values in Python so I can obtain the same value as in Octave by applying fread with an'int16'?
I've been searching on the Internet for an answer for this but I was not able to find a method that provides the same value
Thanks a lot
Best regards
It looks like the binary data in your question is 5ab0bbd8. To unpack signed 16 bit integers with struct.unpack, you use the 'h' format character. From that (23216+48088j) output, it appears that the data is encoded as little-endian, so we need to use < as the first item in the format string.
from struct import unpack
data = b'\x5a\xb0\xbb\xd8'
# The wrong way
rel=int(data[:2].encode("hex"),16)
img=int(data[2:].encode("hex"),16)
c = complex(rel, img)
print c
# The right way
rel, img = unpack('<hh', data)
c = complex(rel, img)
print c
output
(23216+48088j)
(-20390-10053j)
Note that rel, img = unpack('<hh', data) will also work correctly on Python 3.
FWIW, in Python 3, you could also decode 2 bytes to a signed integer like this:
def int16_bytes_to_int(b):
n = int.from_bytes(b, 'little')
if n > 0x7fff:
n -= 0x10000
return n
The rough equivalent in Python 2 is:
def int16_bytes_to_int(b):
lo, hi = b
n = (ord(hi) << 8) + ord(lo)
if n > 0x7fff:
n -= 0x10000
return n
But having to do that subtraction to handle signed numbers is annoying, and using struct.unpack is bound to be much more efficient.
I have a file with big endian binaries. There are two numeric fields. The first has length 8 and the second length 12. How can I unpack the two numbers?
I am using the Python module struct (https://docs.python.org/2/library/struct.html) and it works for the first field
num1 = struct.unpack('>Q',payload[0:8])
but I don't know how I can unpack the second number. If I treat it as char(12), then I get something like '\x00\xe3AC\x00\x00\x00\x06\x00\x00\x00\x01'.
Thanks.
I think you should create a new string of bytes for the second number of length 16, fill the last 12 bytes with the string of bytes that hold your number and first 4 ones with zeros.
Then decode the bytestring with unpack with format >QQ, let's say to numHI, numLO variables. Then, you get final number with that: number = numHI * 2^64 + numLO*. AFAIR the integers in Python can be (almost) as large as you wish, so you will have no problems with overflows. That's only rough idea, please comment if you have problems with writing that in actual Python code, I'll then edit my answer to provide more help.
*^ is in this case the math power, so please use math.pow. Alternatively, you can use byte shift: number = numHI << 64 + numLO.
I'm using sndarray from pygame to play with basic sound synthesis. The problem is Whatever I do, I have an awful distortion on the generated sound.
In the code I'll provide at the end of the question, you'll see a bunch of code coming from here and there. Actually, the main stuff comes from a MIT's source I found on the interweb which is using Numeric to do mathematic stuff and handling arrays, and since I can't install it for now, I decided to use Numpy for this.
First, I thought the problem was coming from the Int format of my arrays, but if I cast the values to numpy.int16, I don't have sound anymore.
Plus, I can't find anything on google about that kind of behavior from pygame / sndarray.
Any idea ?
Thanks !
Code :
global_sample_rate = 44100
def sine_array_onecycle(hz, peak):
length = global_sample_rate / float(hz)
omega = numpy.pi * 2 / length
xvalues = numpy.arange(int(length)) * omega
return (peak * numpy.sin(xvalues))
def zipstereo(f):
return numpy.array(zip (f , f))
def make_sound(arr, n_samples = global_sample_rate):
return pygame.sndarray.make_sound( zipstereo( numpy.resize(numpy.array(arr), (n_samples,)) ) )
def sine(hz, peak):
snd = make_sound(sine_array_onecycle(hz, peak), global_sample_rate)
return snd
=> 'hope I didn't make any lame mistake, I'm pretty new in the world of python
Presuming you have some initialization code like
pygame.mixer.pre_init(44100, -16, 2) # 44.1kHz, 16-bit signed, stereo
sndarray expects you to be passing it 16-bit integer arrays, not float arrays.
Your "peak" value needs to make sense given the 16-bit integer representation. So, if your float array has values in the range -1.0 to +1.0, then you need to multiply by 2**15 to get it scaled appropriately.
To be clear, you may want a conversion like:
numpy.int16(float_array*(2**15))
My best guess of the situation is that you had a float array with a low peak value like 1.0, so when converting it to int16 most everything was getting converted to 0 or +/-1, which you wouldn't be able to hear. When passing the float array, you were probably just getting random bits (when interpreted as 16 bit integers) so then it sounded like harsh noise (I stumbled through this phase on my way to getting this working).