We Keep Coding

Shared memory between C and python

I want to share memory between a program in C and another in python.
The c program uses the following structure to define the data.
struct Memory_LaserFrontal {
char Data[372]; // original data
float Med[181]; // Measurements in [m]
charD; // 'I': Invalid -- 'V': Valid
charS; // 'L': Clean -- 'S': Dirty
char LaserStatus[2];
};
From python I have managed to read the variable in memory using sysv_ipc but they have no structure and is seen as a data array. How can I restructure them?
python code:
from time import sleep
import sysv_ipc
# Create shared memory object
memory = sysv_ipc.SharedMemory(1234)
# Read value from shared memory
memory_value = memory.read()
print (memory_value)
print (len(memory_value))
while True:
memory_value = memory.read()
print (float(memory_value[800]))
sleep(0.1)
I have captured and printed the data in python, I have modified the sensor reading and the read data is also modified, confirming that the read data corresponds to the data in the sensor's shared memory. But without the proper structure y cant use the data.

You need to unpack your binary data structure into Python types. The Python modules struct and array can do this for you.
import struct
import array
NB: Some C compilers, but not the comomn ones, may pad your member variables to align each of them with the expected width for your CPU ( almost always 4 bytes ). This means that it may add padding bytes. You may have to experiment with the struct format parameter 'x' between the appropriate parts of your struct if this is the case. Python's struct module does not expect aligned or padded types by default, you need to inform it. See my note at the very end for a guess on what the padding might look like. Again, per #Max's comment, this is unlikely.
NB: I think the members charD and charS are really char D; and char S;
Assuming you want the floats as a Python list or equivalent we have to do some more work with the Python array module . Same for the char[] Data.
# Get the initial char array - you can turn it into a string if you need to later.
my_chars = array.array("b") # f for float, b for byteetc.
my_chars.from_bytes(memory_value[:372]) # happens that 372 C chars is 372 bytes.
Data = my_chars.tolist() # Could be bytes list
# advance to the member after Data
end_of_Data = struct.calcsize("372c")
# get the length in bytes that 181 floats take up
end_of_Med = struct.calcsize("181f") + end_of_Data
# now we know where the floats are
floats_as_bytes = memory_value[ end_of_Data : end_of_Med ]
# unpack the remaining parts
( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cccc", memory_value[end_of_Med:] )
Now use the array module to unpack to make a Python list
my_floats = array.array("f") # f for float, c for char etc.
my_floats.from_bytes(floats_as_bytes)
Now Data might be a list of Python bytes type that you need to convert to your preferred string encoding. Usually .decode('utf-8') is good enough.
Data_S = "".join(Data).decode('utf-8') # get a usable string in Data_S
Padding
struct Memory_LaserFrontal {
char Data[372]; // 372 is a multiple of 4, probably no padding
float Med[181]; // floats are 4 bytes, probably no padding
charD; // single char, expect 3 padding bytes after
charS; // single char, expect 3 padding bytes after
char LaserStatus[2]; // double char expect 2 padding bytes after.
};
So the last Python line above might be - where the 'x' indicates a padding byte that can be ignored.
( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cxxxcxxxccxx", memory_value[end_of_Med:] )

I always like to leave the full source code of the problem solved so others can use it if they have a similar problem.
thanks a lot all!
from time import sleep
import sysv_ipc
import struct
import array
# Create shared memory object
while True:
memory = sysv_ipc.SharedMemory(1234)
# Read value from shared memory
memory_value = memory.read()
#print (memory_value)
#print (len(memory_value))
# Get the initial char array - you can turn it into a string if you need to later.
my_chars = array.array("b") # f for float, c for char etc.
#my_chars.from_bytes(memory_value[:372]) # happens that 372 chars is 372 bytes.
Data = my_chars.tolist() # Could be bytes list
# advance to the member after Data
end_of_Data = struct.calcsize("372c")
# get the length in bytes that 181 floats take up
end_of_Med = struct.calcsize("181f") + end_of_Data
# now we know where the floats are
floats_as_bytes = memory_value[ end_of_Data : end_of_Med ]
# unpack the remaining parts
( D, S, LaserStatus_1, LaserStatus_2 ) = struct.unpack( "cccc", memory_value[end_of_Med:] )
print(len(floats_as_bytes)/4)
a=[]
for i in range(0,len(floats_as_bytes),4):
a.append(struct.unpack('<f', floats_as_bytes[i:i+4]))
print (a[0])
sleep(0.1)

Python Pillow Image.frombytes mode '1' bad result

Where am I wrong ? I want to create a basic white pict from bytes
from PIL import Image
if __name__ == "__main__":
data = [chr(1)] * 8192
data = "".join(data)
im = Image.frombytes('1', (128,64), data, 'raw')
im = im.convert("RGB")
im.save("image.png", "PNG")
But I get this:

Just use Image.new instead:
im = Image.new(mode='RGB', size=(128,64), color=(255,255,255))
If you really want to make it from bytes, it would be like this:
Image.frombytes(mode='RGB', size=(128,64), data=b'\xff'*128*64*3)
edit: Image.frombytes expects bytes, not a list of integers. To convert a list of integers to the right type, use this:
>>> bytes([0,1,2]) # Python 3
b'\x00\x01\x02'
>>> bytes(bytearray([0,1,2])) # Python 2
'\x00\x01\x02'
edit 2: mode='1' or the docs have bug (see comment thread). Assuming you have a list of zeros and ones, 1024 elements long, and you want to convert this to an 128x64 monochromatic image (one bit per pixel) then you'll have to pack the bytes manually:
bits = [int(not (y%13 and x%7)) for x in range(64) for y in range(128)]
# asymmetric grid
octets = [bits[i:i+8] for i in range(0, len(bits), 8)]
def bits2byte(bits8):
result = 0
for bit in bits8:
result <<= 1
result |= bit
return result
data = bytes(bytearray([bits2byte(octet) for octet in octets]))
im = Image.frombytes(mode='1', size=(128,64), data=data)
im.show()
Result:

In mode 1 each byte represents 8 pixels (there might be zero padding at end of each row if the width does not divide by 8). So to get a white image, you have to pass in only the byte b'\xff'
data = b'\xff' * 1024
im = Image.frombytes('1', (128,64), data)
Even if the Pillow docs say that there's one pixel per byte in this mode, that is not true for the frombytes and tobytes methods, at least.
Any other repeating input other than \xff (all white) or \x00 (all black) will give some sort of pinstripe pattern, like the one in your question.

python: struct pack size longer than expected -- why does this happen?

So I want to pack a list of tuples and then unpack it later.
from struct import *
from itertools import chain
a = [(1, 67), (213, 455), (9009, 8887)]
# converts 3x2 list to 6x1 list
b = list(chain(*a))
size=6
qq = pack('h'+'L'*size,size,*b)
# peak to get the list length
mysize = unpack('h',qq[:2])
mysize = mysize[0]
unpack('L',qq[2:6])
unpack('h'+'L'*mysize,qq)
unpack('L'*mysize, qq[2:]) # does not work
unpack('L'*mysize, qq[2:2+mysize*4]) # works
Using Python 2.7, the second to last line does not work. I tested len(qq), which is 28, when I was expecting 26.

According to the docs:
C types are represented in the machine’s native format and byte order, and properly aligned by skipping pad bytes if necessary (according to the rules used by the C compiler).
Since you are running on a 64 bit machine, smaller types such as h and L are padded to 8 bytes. You can use the formatting flags =, <, >, ! to remove the padding. For instance, adding "=" works
from struct import *
from itertools import chain
a = [(1, 67), (213, 455), (9009, 8887)]
# converts 3x2 list to 6x1 list
b = list(chain(*a))
size=6
qq = pack('=h'+'L'*size,size,*b)
# peak to get the list length
mysize = unpack('=h',qq[:2])
mysize = mysize[0]
unpack('=L',qq[2:6])
unpack('=h'+'L'*mysize,qq)
unpack('=' + 'L'*mysize, qq[2:]) # does not work
unpack('=' + 'L'*mysize, qq[2:2+mysize*4]) # works

reorder byte order in hex string (python)

I want to build a small formatter in python giving me back the numeric
values embedded in lines of hex strings.
It is a central part of my formatter and should be reasonable fast to
format more than 100 lines/sec (each line about ~100 chars).
The code below should give an example where I'm currently blocked.
'data_string_in_orig' shows the given input format. It has to be
byte swapped for each word. The swap from 'data_string_in_orig' to
'data_string_in_swapped' is needed. In the end I need the structure
access as shown. The expected result is within the comment.
Thanks in advance
Wolfgang R
#!/usr/bin/python
import binascii
import struct
## 'uint32 double'
data_string_in_orig = 'b62e000052e366667a66408d'
data_string_in_swapped = '2eb60000e3526666667a8d40'
print data_string_in_orig
packed_data = binascii.unhexlify(data_string_in_swapped)
s = struct.Struct('<Id')
unpacked_data = s.unpack_from(packed_data, 0)
print 'Unpacked Values:', unpacked_data
## Unpacked Values: (46638, 943.29999999943209)
exit(0)

array.arrays have a byteswap method:
import binascii
import struct
import array
x = binascii.unhexlify('b62e000052e366667a66408d')
y = array.array('h', x)
y.byteswap()
s = struct.Struct('<Id')
print(s.unpack_from(y))
# (46638, 943.2999999994321)
The h in array.array('h', x) was chosen because it tells array.array to regard the data in x as an array of 2-byte shorts. The important thing is that each item be regarded as being 2-bytes long. H, which signifies 2-byte unsigned short, works just as well.

This should do exactly what unutbu's version does, but might be slightly easier to follow for some...
from binascii import unhexlify
from struct import pack, unpack
orig = unhexlify('b62e000052e366667a66408d')
swapped = pack('<6h', *unpack('>6h', orig))
print unpack('<Id', swapped)
# (46638, 943.2999999994321)
Basically, unpack 6 shorts big-endian, repack as 6 shorts little-endian.
Again, same thing that unutbu's code does, and you should use his.
edit Just realized I get to use my favorite Python idiom for this... Don't do this either:
orig = 'b62e000052e366667a66408d'
swap =''.join(sum([(c,d,a,b) for a,b,c,d in zip(*[iter(orig)]*4)], ()))
# '2eb60000e3526666667a8d40'

The swap from 'data_string_in_orig' to 'data_string_in_swapped' may also be done with comprehensions without using any imports:
>>> d = 'b62e000052e366667a66408d'
>>> "".join([m[2:4]+m[0:2] for m in [d[i:i+4] for i in range(0,len(d),4)]])
'2eb60000e3526666667a8d40'
The comprehension works for swapping byte order in hex strings representing 16-bit words. Modifying it for a different word-length is trivial. We can make a general hex digit order swap function also:
def swap_order(d, wsz=4, gsz=2 ):
return "".join(["".join([m[i:i+gsz] for i in range(wsz-gsz,-gsz,-gsz)]) for m in [d[i:i+wsz] for i in range(0,len(d),wsz)]])
The input params are:
d : the input hex string
wsz: the word-size in nibbles (e.g for 16-bit words wsz=4, for 32-bit words wsz=8)
gsz: the number of nibbles which stay together (e.g for reordering bytes gsz=2, for reordering 16-bit words gsz = 4)

import binascii, tkinter, array
from tkinter import *
infile_read = filedialog.askopenfilename()
with open(infile, 'rb') as infile_:
infile_read = infile_.read()
x = (infile_read)
y = array.array('l', x)
y.byteswap()
swapped = (binascii.hexlify(y))
This is a 32 bit unsigned short swap i achieved with code very much the same as "unutbu's" answer just a little bit easier to understand. And technically binascii is not needed for the swap. Only array.byteswap is needed.

Converting an RGB color tuple to a hexidecimal string

I need to convert (0, 128, 64) to something like this "#008040". I'm not sure what to call the latter, making searching difficult.

Use the format operator %:
>>> '#%02x%02x%02x' % (0, 128, 64)
'#008040'
Note that it won't check bounds...
>>> '#%02x%02x%02x' % (0, -1, 9999)
'#00-1270f'

def clamp(x):
return max(0, min(x, 255))
"#{0:02x}{1:02x}{2:02x}".format(clamp(r), clamp(g), clamp(b))
This uses the preferred method of string formatting, as described in PEP 3101. It also uses min() and max to ensure that 0 <= {r,g,b} <= 255.
Update added the clamp function as suggested below.
Update From the title of the question and the context given, it should be obvious that this expects 3 ints in [0,255] and will always return a color when passed 3 such ints. However, from the comments, this may not be obvious to everyone, so let it be explicitly stated:
Provided three int values, this will return a valid hex triplet representing a color. If those values are between [0,255], then it will treat those as RGB values and return the color corresponding to those values.

I have created a full python program for it the following functions can convert rgb to hex and vice versa.
def rgb2hex(r,g,b):
return "#{:02x}{:02x}{:02x}".format(r,g,b)
def hex2rgb(hexcode):
return tuple(map(ord,hexcode[1:].decode('hex')))
You can see the full code and tutorial at the following link : RGB to Hex and Hex to RGB conversion using Python

This is an old question but for information, I developed a package with some utilities related to colors and colormaps and contains the rgb2hex function you were looking to convert triplet into hexa value (which can be found in many other packages, e.g. matplotlib). It's on pypi
pip install colormap
and then
>>> from colormap import rgb2hex
>>> rgb2hex(0, 128, 64)
'##008040'
Validity of the inputs is checked (values must be between 0 and 255).

I'm truly surprised no one suggested this approach:
For Python 2 and 3:
'#' + ''.join('{:02X}'.format(i) for i in colortuple)
Python 3.6+:
'#' + ''.join(f'{i:02X}' for i in colortuple)
As a function:
def hextriplet(colortuple):
return '#' + ''.join(f'{i:02X}' for i in colortuple)
color = (0, 128, 64)
print(hextriplet(color))
#008040

triplet = (0, 128, 64)
print '#'+''.join(map(chr, triplet)).encode('hex')
or
from struct import pack
print '#'+pack("BBB",*triplet).encode('hex')
python3 is slightly different
from base64 import b16encode
print(b'#'+b16encode(bytes(triplet)))

you can use lambda and f-strings(available in python 3.6+)
rgb2hex = lambda r,g,b: f"#{r:02x}{g:02x}{b:02x}"
hex2rgb = lambda hx: (int(hx[0:2],16),int(hx[2:4],16),int(hx[4:6],16))
usage
rgb2hex(r,g,b) #output = #hexcolor
hex2rgb("#hex") #output = (r,g,b) hexcolor must be in #hex format

In Python 3.6, you can use f-strings to make this cleaner:
rgb = (0,128, 64)
f'#{rgb[0]:02x}{rgb[1]:02x}{rgb[2]:02x}'
Of course you can put that into a function, and as a bonus, values get rounded and converted to int:
def rgb2hex(r,g,b):
return f'#{int(round(r)):02x}{int(round(g)):02x}{int(round(b)):02x}'
rgb2hex(*rgb)

Here is a more complete function for handling situations in which you may have RGB values in the range [0,1] or the range [0,255].
def RGBtoHex(vals, rgbtype=1):
"""Converts RGB values in a variety of formats to Hex values.
#param vals An RGB/RGBA tuple
#param rgbtype Valid valus are:
1 - Inputs are in the range 0 to 1
256 - Inputs are in the range 0 to 255
#return A hex string in the form '#RRGGBB' or '#RRGGBBAA'
"""
if len(vals)!=3 and len(vals)!=4:
raise Exception("RGB or RGBA inputs to RGBtoHex must have three or four elements!")
if rgbtype!=1 and rgbtype!=256:
raise Exception("rgbtype must be 1 or 256!")
#Convert from 0-1 RGB/RGBA to 0-255 RGB/RGBA
if rgbtype==1:
vals = [255*x for x in vals]
#Ensure values are rounded integers, convert to hex, and concatenate
return '#' + ''.join(['{:02X}'.format(int(round(x))) for x in vals])
print(RGBtoHex((0.1,0.3, 1)))
print(RGBtoHex((0.8,0.5, 0)))
print(RGBtoHex(( 3, 20,147), rgbtype=256))
print(RGBtoHex(( 3, 20,147,43), rgbtype=256))

Note that this only works with python3.6 and above.
def rgb2hex(color):
"""Converts a list or tuple of color to an RGB string
Args:
color (list|tuple): the list or tuple of integers (e.g. (127, 127, 127))
Returns:
str: the rgb string
"""
return f"#{''.join(f'{hex(c)[2:].upper():0>2}' for c in color)}"
The above is the equivalent of:
def rgb2hex(color):
string = '#'
for value in color:
hex_string = hex(value) # e.g. 0x7f
reduced_hex_string = hex_string[2:] # e.g. 7f
capitalized_hex_string = reduced_hex_string.upper() # e.g. 7F
string += capitalized_hex_string # e.g. #7F7F7F
return string

You can also use bit wise operators which is pretty efficient, even though I doubt you'd be worried about efficiency with something like this. It's also relatively clean. Note it doesn't clamp or check bounds. This has been supported since at least Python 2.7.17.
hex(r << 16 | g << 8 | b)
And to change it so it starts with a # you can do:
"#" + hex(243 << 16 | 103 << 8 | 67)[2:]

def RGB(red,green,blue): return '#%02x%02x%02x' % (red,green,blue)
background = RGB(0, 128, 64)
I know one-liners in Python aren't necessarily looked upon kindly. But there are times where I can't resist taking advantage of what the Python parser does allow. It's the same answer as Dietrich Epp's solution (the best), but wrapped up in a single line function. So, thank you Dietrich!
I'm using it now with tkinter :-)

There is a package called webcolors. https://github.com/ubernostrum/webcolors
It has a method webcolors.rgb_to_hex
>>> import webcolors
>>> webcolors.rgb_to_hex((12,232,23))
'#0ce817'

''.join('%02x'%i for i in input)
can be used for hex conversion from int number

If typing the formatting string three times seems a bit verbose...
The combination of bit shifts and an f-string will do the job nicely:
# Example setup.
>>> r, g, b = 0, 0, 195
# Create the hex string.
>>> f'#{r << 16 | g << 8 | b:06x}'
'#0000c3'
This also illustrates a method by which 'leading' zero bits are not dropped, if either the red or green channels are zero.

My course task required doing this without using for loops and other stuff, here is my bizarre solution lol.
color1 = int(input())
color2 = int(input())
color3 = int(input())
color1 = hex(color1).upper()
color2 = hex(color2).upper()
color3 = hex(color3).upper()
print('#'+ color1[2:].zfill(2)+color2[2:].zfill(2)+color3[2:].zfill(2))