We Keep Coding

Convert bytes string literal to integer [duplicate]

This question already has answers here:
Convert bytes to int?
(7 answers)
Closed 8 months ago.
I receive a 32-bit number over the serial line, using num = ser.read(4). Checking the value of num in the shell returns something like a very unreadable b'\xcbu,\x0c'.
I can check against the ASCII table to find the values of "u" and ",", and determine that the hex value of the received number is actually equal to "cb 75 2c 0c", or in the format that Python outputs, it's b'\xcb\x75\x2c\x0c'. I can also type the value into a calculator and convert it to decimal (or run int(0xcb752c0c) in Python), which returns 3413453836.
How can I do this conversion from a binary string literal to an integer in Python?

I found two alternatives to solve this problem.
Using the int.from_bytes(bytes, byteorder, *, signed=False) method
Using the struct.unpack(format, buffer) from the builtin struct module
Using int.from_bytes
Starting from Python 3.2, you can use int.from_bytes.
Second argument, byteorder, specifies endianness of your bytestring. It can be either 'big' or 'little'. You can also use sys.byteorder to get your host machine's native byteorder.
from the docs:
The byteorder argument determines the byte order used to represent the integer. If byteorder is "big", the most significant byte is at the beginning of the byte array. If byteorder is "little", the most significant byte is at the end of the byte array. To request the native byte order of the host system, use sys.byteorder as the byte order value.
int.from_bytes(bytes, byteorder, *, signed=False)
Code applying in your case:
>>> import sys
>>> int.from_bytes(b'\x11', byteorder=sys.byteorder)
17
>>> bin(int.from_bytes(b'\x11', byteorder=sys.byteorder))
'0b10001'
Here is the official demonstrative code from the docs:
>>> int.from_bytes(b'\x00\x10', byteorder='big')
16
>>> int.from_bytes(b'\x00\x10', byteorder='little')
4096
>>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=True)
-1024
>>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=False)
64512
>>> int.from_bytes([255, 0, 0], byteorder='big')
16711680
Using the struct.unpack method
The function you need to achieve your goal is struct.unpack.
To understand where you can use it, we need to understand the parameters to give and their impact on the result.
struct.unpack(format, buffer)
Unpack from the buffer buffer (presumably packed by pack(format, ...)) according to the format string format. The result is a tuple even if it contains exactly one item. The buffer’s size in bytes must match the size required by the format, as reflected by calcsize().
The buffer is the bytes that we have to give and the format is how the bytestring should be read.
The information will be split into a string, format characters, that can be endianness, ctype, bytesize, ecc..
from the docs:
Format characters have the following meaning; the conversion between C and Python values should be obvious given their types. The ‘Standard size’ column refers to the size of the packed value in bytes when using standard size; that is, when the format string starts with one of '<', '>', '!' or '='. When using native size, the size of the packed value is platform-dependent.
This table represents the format characters currently avaiable in Python 3.10.6:
Format
C-Type
Standard Size
x
pad byte
c
char
1
b
signed char
1
B
uchar
1
?
bool
1
h
short
2
H
ushort
2
i
int
4
I
uint
4
l
long
4
L
ulong
4
q
long long
8
Q
unsigned long long
8
n
ssize_t
N
unsigned ssize_t
f
float
d
double
s
char[]
p
char[]
P
void*
and here is a table to use it to correct byte order:
Character
Byte order
Size
Alignment
#
native
native
Native
=
native
standard
None
<
little-endian
standard
None
>
big-endian
standard
None
!
network (= big-endian)
standard
None
Examples
Here is an example how you can use it:
>>> import struct
>>> format_chars = '<i' #4 bytes, endianness is 'little'
>>> struct.unpack(format_chars,b"f|cs")
(1935899750,)

check the builtin struct module.
https://docs.python.org/3/library/struct.html
in your case, it should probably be something like:
import struct
struct.unpack(">I", b'\xcb\x75\x2c\x0c')
but it depends on Endianness and signed/unsigned, so do read the entire doc.

Python convert Hex string to Signed Integer

I have a hex string, for instance: 0xb69958096aff3148
And I want to convert this to a signed integer like: -5289099489896877752
In Python, if I use the int() function on above hex number, it returns me a positive value as shown below:
>>> int(0xb69958096aff3148)
13157644583812673864L
However, if I use the "Hex" to "Dec" feature on Windows Calculator, I get the value as: -5289099489896877752
And I need the above signed representation.
For 32-bit numbers, as I understand, we could do the following:
struct.unpack('>i', s)
How can I do it for 64-bit integers?
Thanks.

If you want to convert it to 64-bit signed integer then you can still use struct and pack it as unsigned integer ('Q'), then unpack as signed ('q'):
>>> struct.unpack('<q', struct.pack('<Q', int('0xb69958096aff3148', 16)))
(-5289099489896877752,)

I would recommend the bitstring package available through conda or pip.
from bitstring import BitArray
b = BitArray('0xb69958096aff3148')
b.int
# returns
-5289099489896877752
Want the unsigned int?:
b.uint
# returns:
13157644583812673864

You could do a 64-bit version of this, for example:
def signed_int(h):
x = int(h, 16)
if x > 0x7FFFFFFFFFFFFFFF:
x -= 0x10000000000000000
return x
print(signed_int('0xb69958096aff3148'))
Output
-5289099489896877752

How can I convert two bytes of an integer back into an integer in Python?

I am currently using an Arduino that's outputting some integers (int) through Serial (using pySerial) to a Python script that I'm writing for the Arduino to communicate with X-Plane, a flight simulation program.
I managed to separate the original into two bytes so that I could send it over to the script, but I'm having a little trouble reconstructing the original integer.
I tried using basic bitwise operators (<<, >> etc.) as I would have done in a C++like program, but it does not seem to be working.
I suspect it has to do with data types. I may be using integers with bytes in the same operations, but I can't really tell which type each variable holds, since you don't really declare variables in Python, as far as I know (I'm very new to Python).
self.pot=self.myline[2]<<8
self.pot|=self.myline[3]

You can use the struct module to convert between integers and representation as bytes. In your case, to convert from a Python integer to two bytes and back, you'd use:
>>> import struct
>>> struct.pack('>H', 12345)
'09'
>>> struct.unpack('>H', '09')
(12345,)
The first argument to struct.pack and struct.unpack represent how you want you data to be formatted. Here, I ask for it to be in big-ending mode by using the > prefix (you can use < for little-endian, or = for native) and then I say there is a single unsigned short (16-bits integer) represented by the H.
Other possibilities are b for a signed byte, B for an unsigned byte, h for a signed short (16-bits), i for a signed 32-bits integer, I for an unsigned 32-bits integer. You can get the complete list by looking at the documentation of the struct module.

For example, using Big Endian encoding:
int.from_bytes(my_bytes, byteorder='big')

What you have seems basically like it should work, assuming the data stored in myline has the high byte first:
myline = [0, 1, 2, 3]
pot = myline[2]<<8 | myline[3]
print 'pot: {:d}, 0x{:04x}'.format(pot, pot) # outputs "pot: 515, 0x0203"
Otherwise, if it's low-byte first you'd need to do the opposite way:
myline = [0, 1, 2, 3]
pot = myline[3]<<8 | myline[2]
print 'pot: {:d}, 0x{:04x}'.format(pot, pot) # outputs "pot: 770, 0x0302"

This totally works:
long = 500
first = long & 0xff #244
second = long >> 8 #1
result = (second << 8) + first #500
If you are not sure of types in 'myline' please check Stack Overflow question How to determine the variable type in Python?.

To convert a byte or char to the number it represents, use ord(). Here's a simple round trip from an int to bytes and back:
>>> number = 3**9
>>> hibyte = chr(number / 256)
>>> lobyte = chr(number % 256)
>>> hibyte, lobyte
('L', '\xe3')
>>> print number == (ord(hibyte) << 8) + ord(lobyte)
True
If your myline variable is string or bytestring, you can use the formula in the last line above. If it somehow is a list of integers, then of course you don't need ord.

Convert int to single byte in a string?

I'm implementing PKCS#7 padding right now in Python and need to pad chunks of my file in order to amount to a number divisible by sixteen. I've been recommended to use the following method to append these bytes:
input_chunk += '\x00'*(-len(input_chunk)%16)
What I need to do is the following:
input_chunk_remainder = len(input_chunk) % 16
input_chunk += input_chunk_remainder * input_chunk_remainder
Obviously, the second line above is wrong; I need to convert the first input_chunk_remainder to a single byte string. How can I do this in Python?

In Python 3, you can create bytes of a given numeric value with the bytes() type; you can pass in a list of integers (between 0 and 255):
>>> bytes([5])
b'\x05'
bytes([5] * 5)
b'\x05\x05\x05\x05\x05'
An alternative method is to use an array.array() with the right number of integers:
>>> import array
>>> array.array('B', 5*[5]).tobytes()
b'\x05\x05\x05\x05\x05'
or use the struct.pack() function to pack your integers into bytes:
>>> import struct
>>> struct.pack('{}B'.format(5), *(5 * [5]))
b'\x05\x05\x05\x05\x05'
There may be more ways.. :-)
In Python 2 (ancient now), you can do the same by using the chr() function:
>>> chr(5)
'\x05'
>>> chr(5) * 5
'\x05\x05\x05\x05\x05'

In Python3, the bytes built-in accepts a sequence of integers. So for just one integer:
>>> bytes([5])
b'\x05'
Of course, thats bytes, not a string. But in Python3 world, OP would probably use bytes for the app he described, anyway.

How can I convert a hex ASCII string to a signed integer

Input = 'FFFF' # 4 ASCII F's
desired result ... -1 as an integer
code tried:
hexstring = 'FFFF'
result = (int(hexstring,16))
print result #65535
Result: 65535
Nothing that I have tried seems to recognized that a 'FFFF' is a representation of a negative number.

Python converts FFFF at 'face value', to decimal 65535
input = 'FFFF'
val = int(input,16) # is 65535
You want it interpreted as a 16-bit signed number.
The code below will take the lower 16 bits of any number, and 'sign-extend', i.e. interpret
as a 16-bit signed value and deliver the corresponding integer
val16 = ((val+0x8000)&0xFFFF) - 0x8000
This is easily generalized
def sxtn( x, bits ):
h= 1<<(bits-1)
m = (1<<bits)-1
return ((x+h) & m)-h

In a language like C, 'FFFF' can be interpreted as either a signed (-1) or unsigned (65535) value. You can use Python's struct module to force the interpretation that you're wanting.
Note that there may be endianness issues that the code below makes no attempt to deal with, and it doesn't handle data that's more than 16-bits long, so you'll need to adapt if either of those cases are in effect for you.
import struct
input = 'FFFF'
# first, convert to an integer. Python's going to treat it as an unsigned value.
unsignedVal = int(input, 16)
assert(65535 == unsignedVal)
# pack that value into a format that the struct module can work with, as an
# unsigned short integer
packed = struct.pack('H', unsignedVal)
assert('\xff\xff' == packed)
# ..then UNpack it as a signed short integer
signedVal = struct.unpack('h', packed)[0]
assert(-1 == signedVal)