I'm fairly weak with structs but I have a feeling they're the best way to do this. I have a large string of binary data and need to pull 32 of those chars, starting at a specific index, and store them as an int. What is the best way to do this?
Since I need to start at an initial position I have been playing with struct.unpack_from(). Based on the format table here, I thought the 'i' formatting being 4 bytes is exactly what I needed but the code below executes and prints "(825307441,)" where I was expecting either the binary, decimal or hex form. Can anyone explain to me what 825307441 represents?
Also is there a method of extracting the data in a similar fashion but returning it in a list instead of a tuple? Thank you
st = "1111111111111111111111111111111"
test = struct.unpack_from('i',st,0)
print test
Just use int
>>> st = "1111111111111111111111111111111"
>>> int(st,2)
2147483647
>>> int(st[1:4],2)
7
You can slice the string any way you want to get the indices you desire. Passing 2 to int tells int that you are passing it a string in binary
Related
I use python 2.7 and windows.
I want to convert string'A123456' to bytes: b'\x0A\x12\x34\x56' and then concatenate it with other bytes (b'\xBB') to b'\xbb\x0A\x12\x34\x56'.
That is, I want to obtain b'\xbb\x0A\x12\x34\x56' from string'A123456' and b'\xBB'
This isn't too hard to do with binascii.unhexlify, the only problem you've got is that you want to zero pad your string when it's not an even number of nibbles (unhexlify won't accept a length 7 string).
So first off, it's probably best to make a quick utility function that does that, because doing it efficiently isn't super obvious, and you want a self-documenting name:
def zeropad_even(s):
# Adding one, then stripping low bit leaves even values unchanged, and rounds
# up odd values to next even value
return s.zfill(len(s) + 1 & ~1)
Now all you have to do is use that to fix up your string before unhexlifying it:
>>> from binascii import unhexlify
>>> unhexlify(zeropad_even('A123456'))
'\n\x124V'
>>> _ == b'\x0A\x12\x34\x56'
True
I included that last test just to show that you got the expected result; the repr of str tries to use printable ASCII or short escapes where available, so only the \x12 actually ends up in the repr; \x0A' becomes \n, \x34 is 4 and \x56 is V, but those are all equivalent ways to spell the same bytes.
I am hooking up an instrument to a laptop over TCP/IP. I have been using a python package to talk to it, and have it return numbers. There are two probes hooked up to this instrument, and I believe these are the bytes corresponding to the temperature readings of these two probes.
The instrument, by default, is set to Big Endian and these data should be of a 32-bit floating point variety - meaning that the variable (b) in the code chunk represents two numbers. b is representative of the output that I would get from the TCP functions.
>>> b = [16746, 42536, 16777, 65230]
>>>
My goal in this is to convert these into their float values, and automating the process. Currently, I am running b through the (hex) function to retrieve the hexadecimal equivalents of each byte:
>>> c =[hex(value) for value in b]
>>>
>>> c
>['0x416a', '0xa628', '0x4189', '0xfece']
>>>
... then I have manually created data_1 and data_2 below to match these hex values, then unpacked them using struct.unpack as I found in this other answer:
>>> data_1 = b'\x41\x6a\xa6\x28'
>>> import struct
>>> struct.unpack('>f', data_1)
>(14.665565490722656,)
>>> data_2 = b'\x41\x89\xfe\xce'
>>> struct.unpack('>f', data_2)
>(17.24941635131836,)
>>>
Some questions:
Am I fundamentally missing something? I am a biologist by trade, and usually a R programmer, so Python is relatively new to me.
I am primarily looking for a streamlined way to get from the TCP output (b) to the number outputs of struct.unpack. The eventual goal of this project is to constantly be polling the sensors for data, which will be graphed/displayed on screen as well as being saved to a .csv.
Thank you!
The function below produces same numbers you found:
import struct
def bigIntToFloat(bigIntlist):
pair = []
for bigInt in bigIntlist:
pair.append(bytes.fromhex(format(bigInt, '04x')))
if len(pair) == 2:
yield struct.unpack('>f', b''.join(pair))[0]
pair = []
The key parts are format(bigInt, '04x') which turns an integer into a hex value without the (in this case) unneeded '0x', while ensuring it's zero-padding to four characters, and bytes.fromhex, which turns the output of that into a bytes object suitable for struct.unpack.
As for whether you're missing something, that's hard for me to say, but I will say that the numbers you give look "reasonable" - that is, if you had the ordering wrong, I'd expect the numbers to be vastly different from each other, rather than slightly.
The simplest way is to use struct.pack to turn those numbers back into a byte string, then unpack as you were doing. pack and unpack can also work with multiple values at a time; the only snag is that pack expects individual arguments instead of a list, so you must put a * in front to expand the list.
>>> struct.unpack('>2f', struct.pack('>4H', *b))
(14.665565490722656, 17.24941635131836)
I'm currently trying to take integer arrays that actually represent other data types and convert them into the correct datatype.
So for example, if I had the integer array [1196773188, 542327116], I discover that this integer array represents a string from some other function, convert it, and realize it represents the string "DOUGLAS". The first number translates to the hexadecimal number 0x47554F44 and the second number represents the hexadecimal number 0x2053414C. Using a hex to string converter, these correspond to the strings 'GOUD' and 'SAL' respectively, spelling DOUGLAS in a little endian manner. The way the letters are backwards in individual elements of the array likely stem from the bytes being stored in a litte endian manner, although I might be mistaken on that.
These integer arrays could represent a number of datatypes, including strings, booleans, and floats.
I need to use Python 2.7, so I unfortunately can't use the bytes function.
Is there a simple way to convert an integer array to its corresponding datatype?
It seems that the struct module is the best way to go when converting between different types like this:
import struct
bufferstr = ""
dougarray = [1196773188, 542327116]
for num in dougarray:
bufferstr += struct.pack("i", num)
print bufferstr # Result is 'DOUGLAS'
From this point on we can easily convert 'DOUGLAS' to any datatype we want using struct.unpack():
print struct.unpack("f", bufferstr[0:4]) # Result is (54607.265625)
We can only unpack a certain number of bytes at a time however. Thank you all for the suggestions!
The function I'm writing gets a checksum (format: '*76') as a string (isolated from an NMEA string). This checksum in string format is called 'Obs' (Observed from string). It then computes the checksum from the rest of the string and gets that answer as hex (Terminal: 0x76), this will be called 'Com' (Computed from string). Now I need to convert one to the other to compare them agains each other.
I've tried stuff like:
HexObs = hex(Obs) #with Obs as '0x76' and '0*76'
Which gives me an error.
and
StrCom = str(Com)
Which gives: '118'
There were no previous questions in which I recognised my question.
Does anyone know how to convert one to the other? Tnx in advance.
I think you're problem is getting the original into an actual hex form
tobs = '76'
obs = hex(int('0x' + tobs, 16))
that will give you an actual hex value to compare
alternately you could use:
tobs = '76'
com = '0x76'
tcom = com[2:]
then compare tobs & tcom
To go from a string hex representation, use:
>>> int('0x76', 16)
118
The second argument is the base.
To go from an integer to a string hex representation, use:
>>> hex(118)
'0x76'
I have a file with big endian binaries. There are two numeric fields. The first has length 8 and the second length 12. How can I unpack the two numbers?
I am using the Python module struct (https://docs.python.org/2/library/struct.html) and it works for the first field
num1 = struct.unpack('>Q',payload[0:8])
but I don't know how I can unpack the second number. If I treat it as char(12), then I get something like '\x00\xe3AC\x00\x00\x00\x06\x00\x00\x00\x01'.
Thanks.
I think you should create a new string of bytes for the second number of length 16, fill the last 12 bytes with the string of bytes that hold your number and first 4 ones with zeros.
Then decode the bytestring with unpack with format >QQ, let's say to numHI, numLO variables. Then, you get final number with that: number = numHI * 2^64 + numLO*. AFAIR the integers in Python can be (almost) as large as you wish, so you will have no problems with overflows. That's only rough idea, please comment if you have problems with writing that in actual Python code, I'll then edit my answer to provide more help.
*^ is in this case the math power, so please use math.pow. Alternatively, you can use byte shift: number = numHI << 64 + numLO.