We Keep Coding

convert mmap to ctypes void ** in python (swig)

I have used swig to generate a wrapper of a library in C.
I need to replicate a code in C in python, but I don't know how to go from mmap to void **
in C:
char *image_buf;
...
axidma_fd = open("/dev/axidma", O_RDWR|O_EXCL);
...
image_buf = mmap(NULL, image_byte_size, PROT_READ|PROT_WRITE,
MAP_SHARED, axidma_fd, 0);
...
trans.num_frame_buffers = 1;
trans.frame_buffers = (void **)&image_buf;
in python
axidma_fd = os.open("/dev/axidma", os.O_RDWR|os.O_EXCL)
...
image_buf_1 = mmap.mmap(vdma_fd,188, mmap.MAP_SHARED,mmap.ACCESS_WRITE| mmap.ACCESS_READ);
...
# Convert mmap to c_char
char_pointer = c_char.from_buffer(image_buf_1) ???? (but type is c_char not pointer)
# Double void
trans.frame_buffers = ??
How do I convert an mmap to a pointer?
How do I get the double pointer?
Would it be more efficient to do it in python directly or by instantiating the libc library?
for example like this
libc = ctypes.CDLL("libc.so.6")
image_mmap = ctypes.c_void_p(libc.mmap(0,188,mmap.ACCESS_WRITE| mmap.ACCESS_READ,mmap.MAP_SHARED,vdma_fd)
Thank you so much
EDITED 1
It explains me a little badly
if I do:
image_buf_1 = mmap.mmap(vdma_fd,188, mmap.MAP_SHARED,mmap.ACCESS_WRITE| mmap.ACCESS_READ)
image_map = c_void_p.from_buffer(image_buf_1)
image_map is None
>>> image_map
c_void_p(None)
On the other hand if I do
libc = ctypes.CDLL("libc.so.6")
image_mmap = ctypes.c_void_p(libc.mmap(0,188,mmap.ACCESS_WRITE| mmap.ACCESS_READ,mmap.MAP_SHARED,vdma_fd))
image_mmap is a valid pointer.
>>> image_mmap
c_void_p(18446744073709551615)
But when trying to assign it to
trans.frame_buffers = image_mmap
report:
trans.frame_buffers=image_mmap
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: in method 'axidma_video_transaction_frame_buffers_set', argument 2 of type 'void **'
I need convert mmap to void **
EDITED 2
Not working :(
I have defined:
class struct_axidma_video_transaction(Structure):
pass
struct_axidma_video_transaction.__slots__ = [
'channel_id',
'num_frame_buffers',
'frame_buffers',
'frame',
]
struct_axidma_video_transaction._fields_ = [
('channel_id', c_int),
('num_frame_buffers', c_int),
('frame_buffers', POINTER(POINTER(None))),
('frame', struct_axidma_video_frame),
]
I need to convert a mmap to a POINTER(POINTER to pass it to the frame_buffers parameter:
>>> from ctypes import c_int, c_void_p, byref, CDLL, POINTER,c_int64
>>> import mmap
>>> vdma_fd = os.open("/dev/axidma",os.O_RDWR|os.O_EXCL)
>>> type(vdma_fd)
<class 'int'>
>>> vdma_buf = mmap.mmap (vdma_fd,188,mmap.MAP_SHARED, mmap.ACCESS_READ | mmap.ACCESS_WRITE)
>>> type(vdma_buf)
<class 'mmap.mmap'>
>>> image_buf_p = c_void_p.from_buffer(vdma_buf)
>>> type(image_buf_p)
<class 'ctypes.c_void_p'>
>>> image_buf_pp = byref(image_buf_p)
>>> type(image_buf_pp)
<class 'CArgObject'>
>>> trans.frame_buffers = image_buf_pp
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected LP_c_void_p instance, got CArgObject
If I define:
>>> image_buf_pp = c_void_p.from_buffer(image_buf_p)
>>> type(image_buf_pp)
<class 'ctypes.c_void_p'>
>>> trans.frame_buffers = image_buf_pp
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: incompatible types, c_void_p instance instead of LP_c_void_p instance
>>>

Thanks in part to this answer: ctypes reference double pointer
try ctypes.byref(). Sample code:
import ctypes
import mmap
import os
FILENAME=#<your filename>
FILELENGTH=#<your file's length>
fd = os.open(FILENAME, os.O_RDWR|os.O_EXCL)
image_buf_1 = mmap.mmap(fd, FILELENGTH, mmap.MAP_SHARED|mmap.ACCESS_WRITE|mmap.ACCESS_READ)
image_map = ctypes.c_void_p.from_buffer(image_buf_1)
print(f"{image_map.value:x}")
print(f"{ctypes.byref(image_map)}") # this is the pointer you're looking for
os.system(f'cat /proc/$PPID/maps | grep {FILENAME}') # verify the pointer value
os.close(fd)
Sorry, I don't know exactly what you mean by "the double pointer"; mmap returns a void*, represented here by ctypes.byref(image_map).
As for what's more efficient, you can of course profile, but usually your investment comes from doing the I/O rather than setting up the I/O (I'm assuming that ctypes.DLL is implemented via dlopen() which is pretty lightweight for an already-loaded library). Unless you're opening a very large number of files, I'd guess you want to choose your method based on whether you want to use the data in Python (you may need the Python mmap object) or not.
--note: I briefly tried the libc method you cited, but mmap returned -1, Invalid argument. I didn't look into what I was doing wrong there.

Listing [Python.Docs]: mmap - Memory-mapped file support.
Check [SO]: C function called from Python via ctypes returns incorrect value (#CristiFati's answer) when working with CTypes.
Working with double (or multiple) pointers (that store array like data - like images are) is a bit tricky in CTypes (as seen in [SO]: C++ & Python: Pass and return a 2D double pointer array from python to c++ (#CristiFati's answer)).
Below is an example that uses arrays.
code00.py:
#!/usr/bin/env python
import ctypes as ct
import mmap
import os
import sys
#CharPtrPtr = ct.POINTER(ct.POINTER(ct.c_char))
def main(*argv):
h = 3
w = 4
fname = "file.txt"
with open(fname, mode="wb") as f:
f.write(b"\x00" * h * w)
fd = os.open(fname, os.O_RDWR | os.O_CREAT)
buf = mmap.mmap(fd, h * w, mmap.MAP_SHARED, mmap.ACCESS_WRITE | mmap.ACCESS_READ)
CharArrArr = ct.c_char * w * h
img = CharArrArr.from_buffer(buf)
for i in range(h):
for j in range(w):
img[i][j] = 0x41 + 2 * i * w + j
del img
buf.close()
os.close(fd)
with open(fname) as f:
print("File contents:", f.read())
if __name__ == "__main__":
print("Python {:s} {:03d}bit on {:s}\n".format(" ".join(elem.strip() for elem in sys.version.split("\n")),
64 if sys.maxsize > 0x100000000 else 32, sys.platform))
rc = main(*sys.argv[1:])
print("\nDone.")
sys.exit(rc)
Output:
(qaic-env) [cfati#cfati-5510-0:/mnt/e/Work/Dev/StackOverflow/q073614606]> python ./code00.py
Python 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] 064bit on linux
File contents: ABCDIJKLQRST
Done.
Looking at the edit, it seems that I didn't get it right. Why do you need the void** pointer? What are you doing with it (what is trans)? Please share the code, not just scattered snippets.
If i got things right you need to get mmap's return value (as void*), and pass its address (which is a void**) further. You can do it (using snippets from your code):
image_buf_1 = mmap.mmap(vdma_fd, 188, mmap.MAP_SHARED, mmap.ACCESS_WRITE | mmap.ACCESS_READ)
image_map = ctypes.c_void_p.from_buffer(image_buf_1)
trans.frame_buffers = ctypes.pointer(image_mmap)
The other problem (NULL pointer): You're passing fd 0 (which is StdIn) to libc.mmap (also check 2nd URL for your hidden bug), while for mmap.mmap you're using the "real" descriptor, So I assume for that descriptor, memory couldn't be properly mapped (you should check mmap results in both cases (image_buf_1)).

Python ctypes throws TypeError: one character bytes, bytearray or integer expected for functions with pointer references

I am new to both Python and ctypes the module. Trying to call C++ a function by loading a shared library. Here is the prototype of the function I want to call.
void foo_func(const char *binary, size_t binsz, size_t memsz, void *params, size_t paramssz, void *settings);
And here is the code I have written to call this function.
import ctypes
import pathlib
class virt_buff(ctypes.Structure):
_fields_ = [("x", ctypes.c_int), ("y", ctypes.c_int)]
if __name__ == "__main__":
libname = pathlib.Path().absolute() / "build/libfoo.so"
c_lib = ctypes.CDLL(libname)
func_param = virt_buff(7, 0)
with open("build/fib.bin", mode='rb') as file: # b is important -> binary
binary = file.read()
c_lib.foo_func(ctypes.c_char(binary), file.tell(), 0x9000 + (file.tell() & ~0xFFF), func_param, 4, NULL)
But when I run this code, it gives me the following output.
Traceback (most recent call last):
File "ct_test.py", line 32, in <module>
c_lib.foo_func(ctypes.c_char(binary), file.tell(), 0x9000 + (file.tell() & ~0xFFF), virt_param, 4, NULL)
TypeError: one character bytes, bytearray or integer expected
I tried a lot of things and nothing seems to work. Can anyone help me find out what the actual problem is?

The first error message is due to trying to pass a single character (c_char) and initializing it with a multiple character data blob. You want c_char_p for C char* instead. But there are other issues after that. Declaring .argtypes and .restype for your function correctly will help ctypes catch errors.
Make the following additional commented fixes:
import ctypes as ct
import pathlib
class virt_buff(ct.Structure):
_fields_ = [("x", ct.c_int), ("y", ct.c_int)]
if __name__ == "__main__":
libname = pathlib.Path().absolute() / "build/libfoo.so"
# Declare argument types and result type that match the C call for ctypes to error check
# void foo_func(const char *binary, size_t binsz, size_t memsz, void *params, size_t paramssz, void *settings);
c_lib = ct.CDLL(str(libname)) # on windows, CDLL(libname) failed to accept WindwowsPath.
c_lib.foo_func.argtypes = ct.c_char_p, ct.c_size_t, ct.c_size_t, ct.c_void_p, ct.c_size_t, ct.c_void_p
c_lib.foo_func.restype = None
func_param = virt_buff(7, 0)
with open("build/fib.bin", mode='rb') as file:
binary = file.read()
# Indent under with, so file.tell() is in scope,
# Pass address of structure using ct.byref() and size with ct.sizeof()
# Use Python None instead of C NULL.
c_lib.foo_func(binary, file.tell(), 0x9000 + (file.tell() & ~0xFFF), ct.byref(func_param), ct.sizeof(func_param), None)

Your parameter is almost certainly supposed to a POINTER to char, and not just a single char:
c_lib.foo_func(ctypes.c_char_p(binary), ...

ctypes dll declaration of variables

I'm trying to use following function with ctypes and I do have troubles how declare all the parameters and variables right.
The Documentation of the C code is following,
/* global variables */
int main ()
char sDeviceSerialNumber[32];
FEUSB_GetScanListPara( 0, "Device-ID", sDeviceSerialNumber ) ;
sDeviceSerialNumber is supposed to be a return value of the function which I need in Python for further use.
Python code:
def FEUSB_GetScanListPara(iIndex, cPara):
libfeusb.FEUSB_GetScanListPara.argtypes = [ctypes.c_int,
ctypes.c_wchar_p,
ctypes.POINTER(ctypes.c_char_p)
]
libfeusb.FEUSB_GetScanListPara.restype = ctypes.c_int
iIndex = ctypes.c_int(iIndex)
cValue_buffer = ctypes.create_string_buffer(32)
cValue = ctypes.c_char_p(ctypes.addressof(cValue_buffer))
value = libfeusb.FEUSB_GetScanListPara(iIndex,
cPara,
ctypes.byref(cValue)
)
if __name__ == "__main__":
i = 0
RFID.FEUSB_GetScanListPara(i, "Device-ID")
When I call the function with the code above, I get an error code, FEUSB_ERR_UNKNOWN_PARAMETER, therefore I assume that I do not declare the parameters correctly.
Any input is appreciated!
EDIT 1
def FEUSB_GetScanListPara(iIndex, cPara):
libfeusb.FEUSB_GetScanListPara.argtypes = [ctypes.c_int,
ctypes.c_char_p,
ctypes.c_char_p
]
libfeusb.FEUSB_GetScanListPara.restype = ctypes.c_int
cValue = ctypes.create_string_buffer(32)
value = libfeusb.FEUSB_GetScanListPara(iIndex, cPara,
ctypes.byref(cValue))
print("1.0", cPara, "back value", " = ", value)
print("1.1", cPara, " = ", cValue.value)
print("######")
if __name__ == "__main__":
data = RFID.FEUSB_GetScanListPara(i, b"Device-ID")
Python Console:
FEUSB_ClearScanList = 0
FEUSB_Scan = 0
FEUSB_GetScanListSize = 1
Traceback (most recent call last):
File "C:\xxxxx\3.1_ObidRFID_test\OBID_RFID_06.py", line 265, in <module>
data = RFID.FEUSB_GetScanListPara(i, b"Device-ID")
File "C:\xxxxx\3.1_ObidRFID_test\OBID_RFID_06.py", line 89, in FEUSB_GetScanListPara
value = libfeusb.FEUSB_GetScanListPara(iIndex, cPara, ctypes.byref(cValue))
ArgumentError: argument 3: <class 'TypeError'>: wrong type
EDIT 2
working code
def FEUSB_GetScanListPara(iIndex, cPara):
libfeusb.FEUSB_GetScanListPara.argtypes = [ctypes.c_int,
ctypes.c_char_p,
ctypes.c_char_p
]
libfeusb.FEUSB_GetScanListPara.restype = ctypes.c_int
cValue = ctypes.create_string_buffer(32)
return_value = libfeusb.FEUSB_GetScanListPara(0, b'Device-ID',
cValue)

Your declaration of .argtypes would match the C prototype of:
int FEUSB_GetScanListPara(int, wchar_t*, char**)
You haven't provided the exact C prototype, but from your example of:
FEUSB_GetScanListPara( 0, "Device-ID", sDeviceSerialNumber ) ;
and knowing wchar_t* is not a common interface parameter, you probably actually have simple char* declarations like:
int FEUSB_GetScanListPara(int, const char*, char*);
I'm assuming the 2nd parameter is an input parameter and 3rd parameter is an output parameter. Note that c_char_p corresponds to a byte string so use b'DeviceID' for cPara. Also if you have to allocate the buffer, the 3rd parameter is unlikely to be char**. If the API itself is not returning a pointer, but filling out an already allocated buffer, char* and hence ctypes.c_char_p is appropriate. You correctly use create_string_buffer() for an output parameter.
Note you don't need to wrap iIndex in a c_int. From .argtypes, ctypes knows the 1st parameter is a c_int and converts it for you. That's also the default if no .argtypes is provided, but better to be explicit and provide .argtypes.
This code should work. I don't have the DLL to verify:
import ctypes as ct
libfeusb = CDLL('./FESUB') # assuming in same directory
libfeusb.FEUSB_GetScanListPara.argtypes = ct.c_int, ct.c_char_p, ct.c_char_p
libfeusb.FEUSB_GetScanListPara.restype = ct.c_int
cValue = ct.create_string_buffer(32)
ret = libfeusb.FEUSB_GetScanListPara(0, b'Device-ID', cValue)
if ret == 0:
print(cValue.value)
else:
print('error:',ret)
If you still have issues, edit your question with a minimal, reproducible example. Make sure to provide the real C prototype.

Python and WinDLL calling doesn't return from the DLL call

My DLL has the following function:
extern "C" __declspec(dllexport) bool __cdecl PcapOpen(unsigned long inSize, void * inData, unsigned long *outSize, void * outData, Function_Modes eMode)
The function will write data out through outSize and outData (meaning these need to be passed to the function with the expectation that my function inside the DLL will modify them)
In addition, Function_Modes is:
typedef enum Function_Modes { Mode_Unknown = -1, Mode_Preview = 0, Mode_Execute = 1, Mode_Query } Function_Modes;
For reference in Perl I referenced the function like so:
my %Data;
$Data{Interface} = $interface;
my ($inData, $inSize) = buildData(\%Data);
$$outSize = pack("L", 65536);
$outData = "\x00"x65536;
my $pPcapOpen = Win32::API->new($dllName, "PcapOpen", "NPPPI", "I", "_cdecl");
my $return = $pPcapOpen->Call($inSize, $inData, $$outSize, $outData, $eMode);
I created a small Python code that should run the function and show the data it returns, but the call while "appears to access the DLL", does not return (the python code just after the call is never called)
pPcapDLL = ctypes.WinDLL("Pcap Interface.dll")
LP_c_char = ctypes.POINTER(ctypes.create_string_buffer(65536).__class__)
obuf = ctypes.create_string_buffer(65536)
obuflen = ctypes.c_ulong(65536)
optr = LP_c_char(obuf)
PcapOpen = pPcapDLL["PcapOpen"]
PcapOpen.restype = ctypes.c_bool
PcapOpen.argtypes = [ctypes.c_ulong, ctypes.c_void_p, ctypes.POINTER(LP_c_char), ctypes.POINTER(ctypes.c_size_t), ctypes.c_int]
outSize = ctypes.c_ulong(65536)
LP_c_char = ctypes.POINTER(ctypes.create_string_buffer(65536).__class__)
outData = ctypes.create_string_buffer(65536)
outData_ptr = LP_c_char(outData)
print("Calling PcapClose")
returnValue = PcapClose(inSize, inData, ctypes.byref(outSize), ctypes.byref(outData_ptr), eMode)
print("returnValue: {}".format(returnValue)) # <-- This doesn't get called
There appears to be very little documentation on how to handle DLL functions that return data - just that byref should be used, but its not 100% clear how to allocate the data for these function to use them - so that could be my issue
Thank you for assisting.

As I see it there are two changes you need to make:
The C function you are attempting to call uses the cdecl calling convention, so you must access the DLL using ctypes.cdll, not ctypes.windll. ctypes.windll is used for the stdcall calling convention. See also the Python ctypes documentation.
The fourth argument to your function is a pointer, but you appear to be passing it a pointer to a pointer. Try replacing ctypes.byref(outdata_ptr) with ctypes.byref(outdata).

Calling functions with arguments from CoreFoundation using ctypes

I am trying to access the CoreMidi functions from the CoreFoundation framework in MacOSX using ctypes in Python.
When I call a function that doesn't have any parameters, everything goes fine. For example, the following code:
from ctypes import *
core_midi = cdll.LoadLibrary("/System/Library/Frameworks/CoreMIDI.framework/Versions/A/CoreMIDI")
numOfDevices = core_midi.MIDIGetNumberOfDevices()
print numOfDevices
returns
3
which is the number of MIDI devices in my computer.
However, I am not able to execute functions which require parameters. Check this example (EDIT: as eryksun pointed out, I was using a char* as client_name, and the function prototype demanded a CFString. I corrected this in the code example below but I still get the same error):
core_midi = cdll.LoadLibrary("/System/Library/Frameworks/CoreMIDI.framework/Versions/A/CoreMIDI")
client_name = core_foundation.CFStringCreateWithCString(None, "MIDI Client", 0)
midi_client = c_uint()
result = core_midi.MIDIClientCreate(client_name, None, None, byref(midi_client))
print midi_client
print result
This code doesn't print anything at all, doesn't even raise an exception. The MIDIClientCreate function's prototype is:
extern OSStatus MIDIClientCreate(
CFStringRef name,
MIDINotifyProc notifyProc,
void *notifyRefCon,
MIDIClientRef *outClient );
MIDIClientRef is defined as a UInt32, and, as I understand, it receives a pointer to a MIDIClient structure that is created, that is why I use byref() to pass it as a parameter. If I just pass the variable without byref(), the function call returns a value -50, which probably indicates some bizarre error.
EDIT: I am not sure if I am creating the CFString correctly. I tried to test the result with the following code but it doesn't print anything on screen.
client_name = core_foundation.CFStringCreateWithCString(None, "MIDI Client", 0)
cstr = ctypes.create_string_buffer(20)
core_foundation.CFStringGetCString(client_name, cstr, 20, 0)
print cstr
Thanks!
EDIT: Answered by eryksun!
I didn't know this of course, but setting the pointers is not as obvious as my naive example was trying to do.
class _CFString(Structure):
pass
cf_path = ctypes.util.find_library("CoreFoundation")
cm_path = ctypes.util.find_library("CoreMIDI")
core_foundation = ctypes.cdll.LoadLibrary(cf_path)
core_midi = ctypes.cdll.LoadLibrary(cm_path)
CFStringRef = POINTER(_CFString)
midi_client = ctypes.c_uint()
core_foundation.CFStringCreateWithCString.restype = CFStringRef
core_foundation.CFStringCreateWithCString.argtypes = [c_void_p, c_char_p, c_uint32]
client_name = core_foundation.CFStringCreateWithCString(None, "MIDI Client", 0)
core_midi.MIDIClientCreate.argtypes = [CFStringRef, c_void_p, c_void_p, POINTER(c_uint32)]
result = core_midi.MIDIClientCreate(client_name, None, None, byref(midi_client))
print midi_client
print result
Actually, I though restype and argtypes didn't affect how functions were executed or how parameters were passed to them, but it seems that they do.
The above code results in:
c_uint(4657178L)
0
That is, my MIDI client is created somewhere and the function returns without error.
Thanks again eryksun!