We Keep Coding

Convert bytes string literal to integer [duplicate]

This question already has answers here:
Convert bytes to int?
(7 answers)
Closed 8 months ago.
I receive a 32-bit number over the serial line, using num = ser.read(4). Checking the value of num in the shell returns something like a very unreadable b'\xcbu,\x0c'.
I can check against the ASCII table to find the values of "u" and ",", and determine that the hex value of the received number is actually equal to "cb 75 2c 0c", or in the format that Python outputs, it's b'\xcb\x75\x2c\x0c'. I can also type the value into a calculator and convert it to decimal (or run int(0xcb752c0c) in Python), which returns 3413453836.
How can I do this conversion from a binary string literal to an integer in Python?

I found two alternatives to solve this problem.
Using the int.from_bytes(bytes, byteorder, *, signed=False) method
Using the struct.unpack(format, buffer) from the builtin struct module
Using int.from_bytes
Starting from Python 3.2, you can use int.from_bytes.
Second argument, byteorder, specifies endianness of your bytestring. It can be either 'big' or 'little'. You can also use sys.byteorder to get your host machine's native byteorder.
from the docs:
The byteorder argument determines the byte order used to represent the integer. If byteorder is "big", the most significant byte is at the beginning of the byte array. If byteorder is "little", the most significant byte is at the end of the byte array. To request the native byte order of the host system, use sys.byteorder as the byte order value.
int.from_bytes(bytes, byteorder, *, signed=False)
Code applying in your case:
>>> import sys
>>> int.from_bytes(b'\x11', byteorder=sys.byteorder)
17
>>> bin(int.from_bytes(b'\x11', byteorder=sys.byteorder))
'0b10001'
Here is the official demonstrative code from the docs:
>>> int.from_bytes(b'\x00\x10', byteorder='big')
16
>>> int.from_bytes(b'\x00\x10', byteorder='little')
4096
>>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=True)
-1024
>>> int.from_bytes(b'\xfc\x00', byteorder='big', signed=False)
64512
>>> int.from_bytes([255, 0, 0], byteorder='big')
16711680
Using the struct.unpack method
The function you need to achieve your goal is struct.unpack.
To understand where you can use it, we need to understand the parameters to give and their impact on the result.
struct.unpack(format, buffer)
Unpack from the buffer buffer (presumably packed by pack(format, ...)) according to the format string format. The result is a tuple even if it contains exactly one item. The buffer’s size in bytes must match the size required by the format, as reflected by calcsize().
The buffer is the bytes that we have to give and the format is how the bytestring should be read.
The information will be split into a string, format characters, that can be endianness, ctype, bytesize, ecc..
from the docs:
Format characters have the following meaning; the conversion between C and Python values should be obvious given their types. The ‘Standard size’ column refers to the size of the packed value in bytes when using standard size; that is, when the format string starts with one of '<', '>', '!' or '='. When using native size, the size of the packed value is platform-dependent.
This table represents the format characters currently avaiable in Python 3.10.6:
Format
C-Type
Standard Size
x
pad byte
c
char
1
b
signed char
1
B
uchar
1
?
bool
1
h
short
2
H
ushort
2
i
int
4
I
uint
4
l
long
4
L
ulong
4
q
long long
8
Q
unsigned long long
8
n
ssize_t
N
unsigned ssize_t
f
float
d
double
s
char[]
p
char[]
P
void*
and here is a table to use it to correct byte order:
Character
Byte order
Size
Alignment
#
native
native
Native
=
native
standard
None
<
little-endian
standard
None
>
big-endian
standard
None
!
network (= big-endian)
standard
None
Examples
Here is an example how you can use it:
>>> import struct
>>> format_chars = '<i' #4 bytes, endianness is 'little'
>>> struct.unpack(format_chars,b"f|cs")
(1935899750,)

check the builtin struct module.
https://docs.python.org/3/library/struct.html
in your case, it should probably be something like:
import struct
struct.unpack(">I", b'\xcb\x75\x2c\x0c')
but it depends on Endianness and signed/unsigned, so do read the entire doc.

python - pack exactly 24 bits into unsigned/signed int

I need to send exactly 24 bits over an ethernet connection, and the program on the other end expects an unsigned int in some cases and a signed int in others (C types). I want to use the struct class, but it doesn't have a type with 3 bytes built in (like uint24_t).
Similar questions to this have been asked, but the answer always involves sending 4 bytes and padding the data packet with zeros. I cannot do this, however, since I am not writing the program which is receiving the data, and it expects exactly 24 bits.
I am very new at this type of programming, so help is appreciated!

Using the struct module, create a string that contains exactly three 8-bit bytes.
import struct
# 24 bits: 01010101 10101010 11110000
byte1 = 0x55
byte2 = 0xaa
byte3 = 0xf0
data = struct.pack("BBB", byte1, byte2, byte3)
Depending on how you get the bits to send, you can define the string directly:
data = '\x55\xaa\xf0'

How can I convert a hex ASCII string to a signed integer

Input = 'FFFF' # 4 ASCII F's
desired result ... -1 as an integer
code tried:
hexstring = 'FFFF'
result = (int(hexstring,16))
print result #65535
Result: 65535
Nothing that I have tried seems to recognized that a 'FFFF' is a representation of a negative number.

Python converts FFFF at 'face value', to decimal 65535
input = 'FFFF'
val = int(input,16) # is 65535
You want it interpreted as a 16-bit signed number.
The code below will take the lower 16 bits of any number, and 'sign-extend', i.e. interpret
as a 16-bit signed value and deliver the corresponding integer
val16 = ((val+0x8000)&0xFFFF) - 0x8000
This is easily generalized
def sxtn( x, bits ):
h= 1<<(bits-1)
m = (1<<bits)-1
return ((x+h) & m)-h

In a language like C, 'FFFF' can be interpreted as either a signed (-1) or unsigned (65535) value. You can use Python's struct module to force the interpretation that you're wanting.
Note that there may be endianness issues that the code below makes no attempt to deal with, and it doesn't handle data that's more than 16-bits long, so you'll need to adapt if either of those cases are in effect for you.
import struct
input = 'FFFF'
# first, convert to an integer. Python's going to treat it as an unsigned value.
unsignedVal = int(input, 16)
assert(65535 == unsignedVal)
# pack that value into a format that the struct module can work with, as an
# unsigned short integer
packed = struct.pack('H', unsignedVal)
assert('\xff\xff' == packed)
# ..then UNpack it as a signed short integer
signedVal = struct.unpack('h', packed)[0]
assert(-1 == signedVal)

struct.error: unpack requires a string argument of length 4

Python says I need 4 bytes for a format code of "BH":
struct.error: unpack requires a string argument of length 4
Here is the code, I am putting in 3 bytes as I think is needed:
major, minor = struct.unpack("BH", self.fp.read(3))
"B" Unsigned char (1 byte) + "H" Unsigned short (2 bytes) = 3 bytes (!?)
struct.calcsize("BH") says 4 bytes.
EDIT: The file is ~800 MB and this is in the first few bytes of the file so I'm fairly certain there's data left to be read.

The struct module mimics C structures. It takes more CPU cycles for a processor to read a 16-bit word on an odd address or a 32-bit dword on an address not divisible by 4, so structures add "pad bytes" to make structure members fall on natural boundaries. Consider:
struct { 11
char a; 012345678901
short b; ------------
char c; axbbcxxxdddd
int d;
};
This structure will occupy 12 bytes of memory (x being pad bytes).
Python works similarly (see the struct documentation):
>>> import struct
>>> struct.pack('BHBL',1,2,3,4)
'\x01\x00\x02\x00\x03\x00\x00\x00\x04\x00\x00\x00'
>>> struct.calcsize('BHBL')
12
Compilers usually have a way of eliminating padding. In Python, any of =<>! will eliminate padding:
>>> struct.calcsize('=BHBL')
8
>>> struct.pack('=BHBL',1,2,3,4)
'\x01\x02\x00\x03\x04\x00\x00\x00'
Beware of letting struct handle padding. In C, these structures:
struct A { struct B {
short a; int a;
char b; char b;
}; };
are typically 4 and 8 bytes, respectively. The padding occurs at the end of the structure in case the structures are used in an array. This keeps the 'a' members aligned on correct boundaries for structures later in the array. Python's struct module does not pad at the end:
>>> struct.pack('LB',1,2)
'\x01\x00\x00\x00\x02'
>>> struct.pack('LBLB',1,2,3,4)
'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04'

By default, on many platforms the short will be aligned to an offset at a multiple of 2, so there will be a padding byte added after the char.
To disable this, use: struct.unpack("=BH", data). This will use standard alignment, which doesn't add padding:
>>> struct.calcsize('=BH')
3
The = character will use native byte ordering. You can also use < or > instead of = to force little-endian or big-endian byte ordering, respectively.

Typecasting in Python

I need to convert strings in Python to other types such as unsigned and signed 8, 16, 32, and 64 bit ints, doubles, floats, and strings.
How can I do this?

You can convert a string to a 32-bit signed integer with the int function:
string = "1234"
i = int(string) # i is a 32-bit integer
If the string does not represent an integer, you'll get a ValueError exception. Note, however, that if the string does represent an integer, but that integer does not fit into a 32-bit signed int, then you'll actually get an object of type long instead.
You can then convert it to other widths and signednesses with some simple math:
s8 = (i + 2**7) % 2**8 - 2**7 # convert to signed 8-bit
u8 = i % 2**8 # convert to unsigned 8-bit
s16 = (i + 2**15) % 2**16 - 2**15 # convert to signed 16-bit
u16 = i % 2**16 # convert to unsigned 16-bit
s32 = (i + 2**31) % 2**32 - 2**31 # convert to signed 32-bit
u32 = i % 2**32 # convert to unsigned 32-bit
s64 = (i + 2**63) % 2**64 - 2**63 # convert to signed 64-bit
u64 = i % 2**64 # convert to unsigned 64-bit
You can convert strings to floating point with the float function:
f = float("3.14159")
Python floats are what other languages refer to as double, i.e. they are 64-bits. There are no 32-bit floats in Python.

Python only has a single int type. To convert a string to an int, use int() like this:
>>> str = '123'
>>> num = int(str)
>>> num
123
Edit: Also to convert to float, use float() in the exact same way.

The following types -- for the most part -- don't exist in Python in the first place. In Python, strings are converted to ints, longs or floats, because that's all there is.
You're asking for conversions that aren't relevant to Python in the first place. Here's the list of types you asked for and their Python equivalent.
unsigned and signed int 8 bits, int
unsigned and signed int 16 bits, int
unsigned and signed int 32 bits, unsigned: long, signed int
unsigned and signed int 64 bits, long
double, float
float, float
string, this is what you had to begin with
I don't know what the following are, so I don't know a Python equivalent.
unsigned and signed 8 bit,
unsigned and signed 16 bit,
unsigned and signed 32 bit,
unsigned and signed 64 bit.
You already have all the conversions that matter: int(), long() and float().

I don't think this can necessarily be answered well without more information. As others have said, there are only int and long for integers in python -- the language doesn't adhere to the bit-width and signedness archetypes of lower-level programming languages.
If you're operating completely within python, then you're probably asking the wrong question. There's likely a better way to do what you need.
If you are interoperating with, for instance, C code, or over the network, then there are ways to do this, and it looks like the answer to your previous posting covered that avenue pretty handily.

I just now had a problem where I had a value passed as a 16 bit signed twos complement number from modbus.
I needed to convert this to a signed number.
I ended up writing this, which seems to work fine.
# convert a 32 bit (prob) integer as though it was
# a 16 bit 2's complement signed one
def conv_s16(i):
if (i & 0x8000):
s16 = -(((~i) & 0xFFFF) + 1)
else:
s16 = i
return s16