We Keep Coding

Difference between array of c_char inside and outside a structure in python for fortran library

I'm interfacing a fortran library with python using c_types. I initialize structures in python, pass them to fortran who populates them, et read them back in python. Everything works fine with array of numbers but now I'm stuck with interfacing string arrays.
I've tried example like this one and this was ok, but in this case the c_char array is not in a structure. So I've tried to modify the previous example putting the c_char array inside a structure. Here is the code I've used, with and without the structure:
Python code:
from ctypes import *
lib = CDLL("./libf.so")
if 1:
print(">>> Without structure")
func = getattr(lib, "fortran2py_")
nstring = pointer(c_long(2))
carr = (c_char * 255)()
func.argtypes = [POINTER(c_long), POINTER(c_char)]
print(type(carr))
print('before:',carr)
func(nstring, carr)
str1, str2 = ''.join([v.decode("utf-8") for v in carr]).rstrip("\x00").split("\x00")
print(str1, str2)
class Struct0(Structure):
_fields_ = [
("name", c_char * 255),
]
if 1:
print(">>> With structure")
func = getattr(lib, "fortran2pystr_")
nstring = pointer(c_long(2))
carr = Struct0()
func.argtypes = [POINTER(c_long), POINTER(Struct0)]
print(type(carr.name))
print('before:',carr.name)
func(nstring, byref(carr))
print('after:',carr.name)
Fortran code:
module c_interop
use iso_c_binding
implicit none
integer, parameter :: STRLEN = 64
type, bind(c) :: charStr
character(c_char) :: name(255)
end type charStr
contains
subroutine fortran2py(nstring, cstring_p) bind(C, name="fortran2py_")
integer(c_int), intent(in) :: nstring
character(c_char), dimension(*), intent(inout) :: cstring_p
integer :: i, j, ks, kf, n
character(len=STRLEN) :: mystr(2)
mystr(1) = "This is the first string."
mystr(2) = "Wow. Fortran + Python + Strings = Pain !"
ks = 1
do i = 1, nstring
n = len_trim(mystr(i))
kf = ks + (n - 1)
cstring_p(ks:kf) = transfer(mystr(i)(1:n), cstring_p(ks:kf))
cstring_p(kf + 1) = c_null_char
ks = ks + n + 1
enddo
end subroutine fortran2py
subroutine fortran2pystr(nstring, cstring_p) bind(C, name="fortran2pystr_")
integer(c_int), intent(in) :: nstring
type(charStr), intent(inout) :: cstring_p
integer :: i, j, ks, kf, n
character(len=STRLEN) :: mystr(2)
mystr(1) = "This is the first string."
mystr(2) = "Wow. Fortran + Python + Strings = Pain !"
ks = 1
do i = 1, nstring
n = len_trim(mystr(i))
kf = ks + (n - 1)
cstring_p%name(ks:kf) = transfer(mystr(i)(1:n), cstring_p%name(ks:kf))
cstring_p%name(kf + 1) = c_null_char
ks = ks + n + 1
enddo
end subroutine fortran2pystr
end module c_interop
I get no error, except that in the modified part, Fortran should fill the array of c_char carr.name looping on the elements of mystr, but the resulting string contain only the first element. When carr is not a structure but directly the c_char array, python can read all the content of mystr.
Output:
>>> Without structure
<class '__main__.c_char_Array_255'>
before: <__main__.c_char_Array_255 object at 0x151b3b092bf8>
This is the first string. Wow. Fortran + Python + Strings = Pain !
>>> With structure
<class 'bytes'>
before: b''
after: b'This is the first string.'
As you can see the type of carr and carr.name are also not the same. Do you have any idea of what is wrong with my modified code ? Thank you !

Listing [Python.Docs]: ctypes - A foreign function library for Python.
The cause it's a CTypes subtle behavior. c_char (and also c_wchar) arrays are silently converted to bytes (or str) when they are present as fields in a structure. This is being done via c_char_p (or c_wchar_p) which are NUL terminated, meaning that the "array" will be truncated if a NUL (0x00) char will be encountered, which is exactly your case. You can check that by looking at the field type. Don't know why this is (maybe to ease the usage), but there are cases when it does more harm than good. It can be reproduced with Python code only.
code00.py
#!/usr/bin/env python
import ctypes as cts
import sys
ARR_DIM = 10
CharArr = cts.c_char * ARR_DIM
class CharArrStruct(cts.Structure):
_fields_ = (
("data", CharArr),
)
def print_array(arr, text, size=ARR_DIM):
print(text)
for i in range(size):
print("{0:3d}".format(i), end=" - ")
try:
print(arr[i])
except IndexError:
print("IndexError!!!")
break
print()
def main(*argv):
arr = CharArr()
sarr = CharArrStruct()
print("Array (plain) type: {0:}".format(type(arr)))
print("Array (in structure) type: {0:}".format(type(sarr.data)))
string_separator = b"\x00"
print("\nString separator: {0:}".format(string_separator))
text = string_separator.join((b"abcd", b"efgh"))
arr[0:len(text)] = text
sarr.data = text
print_array(arr, "Plain array:")
print_array(sarr.data, "Structure with array:")
print("Strings (in structure): {0:}".format(sarr.data.split(string_separator)))
string_separator = b"\xFF"
print("\nString separator: {0:}".format(string_separator))
sarr.data = string_separator.join((b"abcd", b"efgh"))
print_array(sarr.data, "Structure with array:")
print("Strings (in structure): {0:}".format(sarr.data.split(string_separator)))
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.\n")
sys.exit(rc)
Output:
e:\Work\Dev\StackOverflow\q060093054>"e:\Work\Dev\VEnvs\py_pc064_03.07.06_test0\Scripts\python.exe" code00.py
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] 64bit on win32
Array (plain) type: <class '__main__.c_char_Array_10'>
Array (in structure) type: <class 'bytes'>
String separator: b'\x00'
Plain array:
0 - b'a'
1 - b'b'
2 - b'c'
3 - b'd'
4 - b'\x00'
5 - b'e'
6 - b'f'
7 - b'g'
8 - b'h'
9 - b'\x00'
Structure with array:
0 - 97
1 - 98
2 - 99
3 - 100
4 - IndexError!!!
Strings (in structure): [b'abcd']
String separator: b'\xff'
Structure with array:
0 - 97
1 - 98
2 - 99
3 - 100
4 - 255
5 - 101
6 - 102
7 - 103
8 - 104
9 - IndexError!!!
Strings (in structure): [b'abcd', b'efgh']
Done.
Notes:
As seen, data field type was changed
The simplest solution that came into my mind was to replace the string separator from NUL to another char that you are sure it won't appear in any of your strings. I chose 0xFF (255). I think it would also be possible with structures containing ctypes.POINTER(ctypes.c_char), but it would be a bit more complex (also, I didn't test it)
My Fortran knowledge is very close to 0, but something doesn't look right with fortran2pystr. I don't know how Fortran types are structured, but passing a char array wrapped in a struct pointer (indeed, they have the same address) from Python and handling it like a plain char array seems wrong. Changing the struct, would potentially be a recipe for disaster

How to convert bytearray in python 2.7 to decimal string?

I have the following byte array
>>> string_ba
bytearray(b'4\x00/\t\xb5')
which is easily converted to hex string with the next 2 lines:
hex_string = [chr(x).encode('hex') for x in string_ba]
hex_string = ''.join(hex_string)
that return
>>> hex_string.lower()
'34002f09b5'
which is expected. (This is an RFID card signature)
I convert this to decimal by doing the above and then converting from hex string to decimal string (padded with zeroes) with the next line. I have a limit of 10 characters in string, so I'm forced to remove the first 2 characters in the string to be able to convert it to, at most, 10 character decimal number.
dec_string = str(int(hex_string[2:], 16)).zfill(10)
>>> dec_string
'0003082677'
which is correct, as I tested this with an online converter (hex: 002f09b5, dec: 3082677)
The question is, if there's a way to skip converting from bytearray to hex_string, to obtain a decimal string. In other words to go straight from bytearray to dec_string
This will be running on Python 2.7.15.
>>> sys.version
'2.7.15rc1 (default, Apr 15 2018, 21:51:34) \n[GCC 7.3.0]'
I've tried removing the first element from bytearray and then converting it to string directly and joining. But this does not provide the desired result.
string_ba = string_ba[1:]
test_string = [str(x) for x in string_ba]
test_dec_string = ''.join(test_string).zfill(10)
>>> test_dec_string
'0000479181'
To repeat the question is there a way to go straight from bytearray to decimal string

You can use struct library to convert bytearray to decimal. https://codereview.stackexchange.com/questions/142915/converting-a-bytearray-into-an-integer maybe help you

A number (let's call it X) consisting of n digits (in a base, let's refer to it as B) is written as:
Dn-1Dn-2Dn-3...D2D1D0 (where each Di is a digit in base B)
and its value can be computed based on the formula:
VX = Σin=-01(Bi * Di) (notice that in this example the number is traversed from right to left - the traversing sense doesn't affect the final value).
As an example, number 2468 (B10) = 100 * 8 + 101 * 6 + 102 * 4 + 103 * 2 (= 8 + 60 + 400 + 2000).
An ASCII string is actually a number in base 256 (0x100), where each char (byte) is a digit.
Here's an alternative based on the above:
It only performs mathematical operations on integers (the conversion to string is done only at the end)
The traversing sense (from above) is helpful with the restriction (final (decimal) number must fit in a number of digits, and in case of overflow the most significant ones are to be ignored)
The algorithm is simple, starting from the right, compute the partial number value, until reaching the maximum allowed value, or exhausting the number digits (string chars)
code.py:
#!/usr/bin/env python
import sys
DEFAULT_MAX_DIGITS = 10
def convert(array, max_digits=DEFAULT_MAX_DIGITS):
max_val = 10 ** max_digits
number_val = 0
for idx, digit in enumerate(reversed(array)):
cur_val = 256 ** idx * digit
if number_val + cur_val > max_val:
break
number_val += cur_val
return str(number_val).zfill(max_digits)
def main():
b = bytearray("4\x00/\t\xb5")
print("b: {:}\n".format(repr(b)))
for max_digits in range(6, 15, 2):
print("Conversion of b (with max {:02d} digits): {:}{:s}".format(
max_digits, convert(b, max_digits=max_digits),
" (!!! Default case - required in the question)" if max_digits == DEFAULT_MAX_DIGITS else ""
))
if __name__ == "__main__":
print("Python {:s} on {:s}\n".format(sys.version, sys.platform))
main()
Outpput:
(py_064_02.07.15_test0) e:\Work\Dev\StackOverflow\q054091895>"e:\Work\Dev\VEnvs\py_064_02.07.15_test0\Scripts\python.exe" code.py
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
b: bytearray(b'4\x00/\t\xb5')
Conversion of b (with max 06 digits): 002485
Conversion of b (with max 08 digits): 03082677
Conversion of b (with max 10 digits): 0003082677 (!!! Default case - required in the question)
Conversion of b (with max 12 digits): 223341382069
Conversion of b (with max 14 digits): 00223341382069

CRC32 calculation in Python without using libraries

I have been trying to get my head around CRC32 calculations without much success, the values that I seem to get do not match what I should get.
I am aware that Python has libraries that are capable of generating these checksums (namely zlib and binascii) but I do not have the luxury of being able to use them as the CRC functionality do not exist on the micropython.
So far I have the following code:
import binascii
import zlib
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0x000000ffL] ^ (value >> 8)
return value
teststring = "test"
print "binascii calc: 0x%08x" % (binascii.crc32(teststring) & 0xffffffff)
print "zlib calc: 0x%08x" % (zlib.crc32(teststring) & 0xffffffff)
print "my calc: 0x%08x" % (crc32(teststring))
Then I get the following output:
binascii calc: 0xd87f7e0c
zlib calc: 0xd87f7e0c
my calc: 0x2780810c
The binascii and zlib calculations agree where as my one doesn't. I believe the calculated table of bytes is correct as I have compared it to examples available on the net. So the issue must be the routine where each byte is calculated, could anyone point me in the correct direction?
Thanks in advance!

I haven't looked closely at your code, so I can't pinpoint the exact source of the error, but you can easily tweak it to get the desired output:
import binascii
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0xff] ^ (value >> 8)
return -1 - value
# test
data = (
'',
'test',
'hello world',
'1234',
'A long string to test CRC32 functions',
)
for s in data:
print repr(s)
a = binascii.crc32(s)
print '%08x' % (a & 0xffffffffL)
b = crc32(s)
print '%08x' % (b & 0xffffffffL)
print
output
''
00000000
00000000
'test'
d87f7e0c
d87f7e0c
'hello world'
0d4a1185
0d4a1185
'1234'
9be3e0a3
9be3e0a3
'A long string to test CRC32 functions'
d2d10e28
d2d10e28
Here are a couple more tests that verify that the tweaked crc32 gives the same result as binascii.crc32.
from random import seed, randrange
print 'Single byte tests...',
for i in range(256):
s = chr(i)
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
seed(42)
print 'Multi-byte tests...'
for width in range(2, 20):
print 'Width', width
r = range(width)
for n in range(1000):
s = ''.join([chr(randrange(256)) for i in r])
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
output
Single byte tests... ok
Multi-byte tests...
Width 2
Width 3
Width 4
Width 5
Width 6
Width 7
Width 8
Width 9
Width 10
Width 11
Width 12
Width 13
Width 14
Width 15
Width 16
Width 17
Width 18
Width 19
ok
As discussed in the comments, the source of the error in the original code is that this CRC-32 algorithm inverts the initial crc buffer, and then inverts the final buffer contents. So value is initialised to 0xffffffff instead of zero, and we need to return value ^ 0xffffffff, which can also be written as ~value & 0xffffffff, i.e. invert value and then select the low-order 32 bits of the result.

If using binary data where the crc is chained over multiple buffers I used the following (using the OPs table):
def crc32(data, crc=0xffffffff):
for b in data:
crc = table[(b ^ crc) & 0xff] ^ (crc >> 8)
return crc
One can XOR the final result with -1 to agree with the online calculators.
crc = crc32(b'test')
print('0x{:08x}'.format(crc))
crc = crc32(b'te')
crc = crc32(b'st', crc)
print('0x{:08x}'.format(crc))
print('xor: 0x{:08x}'.format(crc ^ 0xffffffff))
output
0x278081f3
0x278081f3
xor: 0xd87f7e0c

Python - reading 10 bit integers from a binary file

I have a binary file containing a stream of 10-bit integers. I want to read it and store the values in a list.
It is working with the following code, which reads my_file and fills pixels with integer values:
file = open("my_file", "rb")
pixels = []
new10bitsByte = ""
try:
byte = file.read(1)
while byte:
bits = bin(ord(byte))[2:].rjust(8, '0')
for bit in reversed(bits):
new10bitsByte += bit
if len(new10bitsByte) == 10:
pixels.append(int(new10bitsByte[::-1], 2))
new10bitsByte = ""
byte = file.read(1)
finally:
file.close()
It doesn't seem very elegant to read the bytes into bits, and read it back into "10-bit" bytes. Is there a better way to do it?
With 8 or 16 bit integers I could just use file.read(size) and convert the result to an int directly. But here, as each value is stored in 1.25 bytes, I would need something like file.read(1.25)...

Here's a generator that does the bit operations without using text string conversions. Hopefully, it's a little more efficient. :)
To test it, I write all the numbers in range(1024) to a BytesIO stream, which behaves like a binary file.
from io import BytesIO
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if len(b) == 0:
break
n = int.from_bytes(b, 'big')
#Split n into 4 10 bit integers
t = []
for i in range(4):
t.append(n & 0x3ff)
n >>= 10
yield from reversed(t)
# Make some test data: all the integers in range(1024),
# and save it to a byte stream
buff = BytesIO()
maxi = 1024
n = 0
for i in range(maxi):
n = (n << 10) | i
#Convert the 40 bit integer to 5 bytes & write them
if i % 4 == 3:
buff.write(n.to_bytes(5, 'big'))
n = 0
# Rewind the stream so we can read from it
buff.seek(0)
# Read the data in 10 bit chunks
a = list(tenbitread(buff))
# Check it
print(a == list(range(maxi)))
output
True
Doing list(tenbitread(buff)) is the simplest way to turn the generator output into a list, but you can easily iterate over the values instead, eg
for v in tenbitread(buff):
or
for i, v in enumerate(tenbitread(buff)):
if you want indices as well as the data values.
Here's a little-endian version of the generator which gives the same results as your code.
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if not len(b):
break
n = int.from_bytes(b, 'little')
#Split n into 4 10 bit integers
for i in range(4):
yield n & 0x3ff
n >>= 10
We can improve this version slightly by "un-rolling" that for loop, which lets us get rid of the final masking and shifting operations.
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if not len(b):
break
n = int.from_bytes(b, 'little')
#Split n into 4 10 bit integers
yield n & 0x3ff
n >>= 10
yield n & 0x3ff
n >>= 10
yield n & 0x3ff
n >>= 10
yield n
This should give a little more speed...

As there is no direct way to read a file x-bit by x-bit in Python, we have to read it byte by byte. Following MisterMiyagi and PM 2Ring's suggestions I modified my code to read the file by 5 byte chunks (i.e. 40 bits) and then split the resulting string into 4 10-bit numbers, instead of looping over the bits individually. It turned out to be twice as fast as my previous code.
file = open("my_file", "rb")
pixels = []
exit_loop = False
try:
while not exit_loop:
# Read 5 consecutive bytes into fiveBytesString
fiveBytesString = ""
for i in range(5):
byte = file.read(1)
if not byte:
exit_loop = True
break
byteString = format(ord(byte), '08b')
fiveBytesString += byteString[::-1]
# Split fiveBytesString into 4 10-bit numbers, and add them to pixels
pixels.extend([int(fiveBytesString[i:i+10][::-1], 2) for i in range(0, 40, 10) if len(fiveBytesString[i:i+10]) > 0])
finally:
file.close()

Adding a Numpy based solution suitable for unpacking large 10-bit packed byte buffers like the ones you might receive from AVT and FLIR cameras.
This is a 10-bit version of #cyrilgaudefroy's answer to a similar question; there you can also find a Numba alternative capable of yielding an additional speed increase.
import numpy as np
def read_uint10(byte_buf):
data = np.frombuffer(byte_buf, dtype=np.uint8)
# 5 bytes contain 4 10-bit pixels (5x8 == 4x10)
b1, b2, b3, b4, b5 = np.reshape(data, (data.shape[0]//5, 5)).astype(np.uint16).T
o1 = (b1 << 2) + (b2 >> 6)
o2 = ((b2 % 64) << 4) + (b3 >> 4)
o3 = ((b3 % 16) << 6) + (b4 >> 2)
o4 = ((b4 % 4) << 8) + b5
unpacked = np.reshape(np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1), 4*o1.shape[0])
return unpacked
Reshape can be omitted if returning a buffer instead of a Numpy array:
unpacked = np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1).tobytes()
Or if image dimensions are known it can be reshaped directly, e.g.:
unpacked = np.reshape(np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1), (1024, 1024))
If the use of the modulus operator appears confusing, try playing around with:
np.unpackbits(np.array([255%64], dtype=np.uint8))
Edit: It turns out that the Allied Vision Mako-U cameras employ a different ordering than the one I originally suggested above:
o1 = ((b2 % 4) << 8) + b1
o2 = ((b3 % 16) << 6) + (b2 >> 2)
o3 = ((b4 % 64) << 4) + (b3 >> 4)
o4 = (b5 << 2) + (b4 >> 6)
So you might have to test different orders if images come out looking wonky initially for your specific setup.

Hex string to signed int in Python

How do I convert a hex string to a signed int in Python 3?
The best I can come up with is
h = '9DA92DAB'
b = bytes(h, 'utf-8')
ba = binascii.a2b_hex(b)
print(int.from_bytes(ba, byteorder='big', signed=True))
Is there a simpler way? Unsigned is so much easier: int(h, 16)
BTW, the origin of the question is itunes persistent id - music library xml version and iTunes hex version

In n-bit two's complement, bits have value:
bit 0 = 20
bit 1 = 21
bit n-2 = 2n-2
bit n-1 = -2n-1
But bit n-1 has value 2n-1 when unsigned, so the number is 2n too high. Subtract 2n if bit n-1 is set:
def twos_complement(hexstr, bits):
value = int(hexstr, 16)
if value & (1 << (bits - 1)):
value -= 1 << bits
return value
print(twos_complement('FFFE', 16))
print(twos_complement('7FFF', 16))
print(twos_complement('7F', 8))
print(twos_complement('FF', 8))
Output:
-2
32767
127
-1

import struct
For Python 3 (with comments' help):
h = '9DA92DAB'
struct.unpack('>i', bytes.fromhex(h))
For Python 2:
h = '9DA92DAB'
struct.unpack('>i', h.decode('hex'))
or if it is little endian:
h = '9DA92DAB'
struct.unpack('<i', h.decode('hex'))

Here's a general function you can use for hex of any size:
import math
# hex string to signed integer
def htosi(val):
uintval = int(val,16)
bits = 4 * (len(val) - 2)
if uintval >= math.pow(2,bits-1):
uintval = int(0 - (math.pow(2,bits) - uintval))
return uintval
And to use it:
h = str(hex(-5))
h2 = str(hex(-13589))
x = htosi(h)
x2 = htosi(h2)

This works for 16 bit signed ints, you can extend for 32 bit ints. It uses the basic definition of 2's complement signed numbers. Also note xor with 1 is the same as a binary negate.
# convert to unsigned
x = int('ffbf', 16) # example (-65)
# check sign bit
if (x & 0x8000) == 0x8000:
# if set, invert and add one to get the negative value, then add the negative sign
x = -( (x ^ 0xffff) + 1)

It's a very late answer, but here's a function to do the above. This will extend for whatever length you provide. Credit for portions of this to another SO answer (I lost the link, so please provide it if you find it).
def hex_to_signed(source):
"""Convert a string hex value to a signed hexidecimal value.
This assumes that source is the proper length, and the sign bit
is the first bit in the first byte of the correct length.
hex_to_signed("F") should return -1.
hex_to_signed("0F") should return 15.
"""
if not isinstance(source, str):
raise ValueError("string type required")
if 0 == len(source):
raise valueError("string is empty")
sign_bit_mask = 1 << (len(source)*4-1)
other_bits_mask = sign_bit_mask - 1
value = int(source, 16)
return -(value & sign_bit_mask) | (value & other_bits_mask)