We Keep Coding

Pass a pointer to an array value from Cython to C

I have a structure in C that looks like this:
typedef struct {
uint32_t id;
uint8_t *buf; // pointer to message data
} msg_t;
and some function that receives that a pointer to such structure and modifies it.
void recv_msg( msg_t *msg ) {
// Stuff happens to message here
return;
}
With cytpes I tried something like this:
from cytpes import CDLL, Structure, POINTER, c_ubyte, c_uint32
class Msg(Structure):
_fields_ = [("id", c_uint32), ("buf", POINTER(c_ubyte * 10))]
lib = CDLL("this_example.so")
get_msg = lib.recv_msg
get_msg.argtypes = [POINTER(Msg)]
get_msg.restype = None
sample_data_array = POINTER(c_ubyte * 10)()
data = sample_data_array(0, 0, 0, 0, 0, 0, 0, 0, 0, 0)
sample_msg = Msg(1, data)
get_msg(sample_msg)
print sample_msg.id, sample_msg.buf[0] # Should change data
I'm getting a TypeError('expected LP_c_ubyte_Array_10 instance, got list',)
I've also tried a similar approach using Cython:
from libc.stdint cimport uint8_t, uint32_t
cdef extern from "this_example.h":
ctypedef struct msg_t:
uint32_t id;
uint8_t *buf;
void get_msg (msg_t *)
def recv_msg():
# How I would do this I don't know
print msg.id, msg.buf[0]
I should also add, I don't want use numpy (but would reluctantly if I have to). Also, the length of the data array can vary, but I also have a length variable in the msg structure, so I can initialise it to the right length for sending, and just set it to default values and max length for receiving.
Any clues? Thanks!

I'm open to accepting other answers, but for now I thought I would post my solution in the hopes it might help someone else.
I solved it using cytpes and not Cython. I would really like to see a Cython solution though.
from ctypes import CDLL, Structure, POINTER, c_uint8, c_uint32, byref
class Msg(Structure):
_fields_ = [("id", c_uint32), ("buf", POINTER(c_uint8))]
lib = CDLL('this_example.so')
get_msg = lib.recv_msg
get_msg.argtypes = [POINTER(Msg)]
get_msg.restype = None
data = (c_uint8 * 8)()
sample_msg = Msg(1, data)
get_msg(byref(sample_msg))
print sample_msg.id, sample_msg.buf[0] # Should see changed data

How to access the value of a ctypes.LP_c_char pointer?

I have defined a struct :
class FILE_HANDLE(Structure):
_fields_ = [
("handle_bytes", c_uint),
("handle_type", c_int),
("f_handle", POINTER(c_char))
]
The struct is initialised :
buf = create_string_buffer(f_handle.handle_bytes)
fh = FILE_HANDLE(c_uint(8), c_int(0), buf)
I am passing it by reference to a function that populates it.
ret = libc.name_to_handle_at(dirfd, pathname, byref(fh), byref(mount_id), flags)
I can check with strace that the call works, but I have not been able to figure out how to access the value of fh.f_handle
fh.f_handle type is <ctypes.LP_c_char object at 0x7f1a7ca17560>
fh.f_handle.contents type is <ctypes.LP_c_char object at 0x7f1a7ca17560> but I get a SIGSEGV if I try to access its value.
How could I get 8 bytes from f_handle into a string or array ?

Everything actually looks right for what you've shown, but without seeing the explicit C definition of the structure and function you are calling it is difficult to see the problem.
Here's an example that works with what you have shown. I inferred what the C definitions should be from what you have declared in Python, but most likely your definition is different if you get a segfault.
C Code (Windows)
struct FILE_HANDLE
{
unsigned int handle_bytes;
int handle_type;
char* f_handle;
};
__declspec(dllexport) int name_to_handle_at(int dirfd, char* pathname, struct FILE_HANDLE* fh, int* mount_id, int flags)
{
unsigned int i;
printf("dirfd=%d pathname=%s fh->handle_bytes=%u fh->handle_type=%d flags=%d\n", dirfd, pathname, fh->handle_bytes, fh->handle_type, flags);
for(i = 0; i < fh->handle_bytes; ++i)
fh->f_handle[i] = 'A' + i;
*mount_id = 123;
return 1;
}
Python code (Works in Python 2 and 3):
from __future__ import print_function
from ctypes import *
class FILE_HANDLE(Structure):
_fields_ = [("handle_bytes", c_uint),
("handle_type", c_int),
("f_handle", POINTER(c_char))]
buf = create_string_buffer(8);
fh = FILE_HANDLE(8,0,buf)
libc = CDLL('test.dll')
mount_id = c_int(0)
ret = libc.name_to_handle_at(1,b'abc',byref(fh),byref(mount_id),7)
print('mount_id =',mount_id.value)
print('fh.f_handle =',fh.f_handle[:fh.handle_bytes])
Output
dirfd=1 pathname=abc fh->handle_bytes=8 fh->handle_type=0 flags=7
mount_id = 123
fh.f_handle = b'ABCDEFGH'
Note that since the structure is declared as a pointer to a single character, printing fh.f_handle.contents would only print b'A'. Using slicing, I've instructed Python to index the pointer up to the length allocated.
If this doesn't work for you, provide a Minimal, Complete, and Verifiable example (as I have) to reproduce your error exactly.

fh.f_handle is shown as LP_c_char because you defined the struct that way.
buf = create_string_buffer(8)
print type(buf)
fh = FILE_HANDLE(c_uint(8), c_int(0), buf)
print type(fh.f_handle)
Will output
<class 'ctypes.c_char_Array_8'>
<class 'ctypes.LP_c_char'>
You have defined your struct to accept a pointer to a c_char. So when you try to access fh.f_handle it will expect the value to be a memory address containing the address to the actual single c_char.
But by trying to input a c_char * 8 from the string buffer it will convert the first part of your buffer to a pointer.
Python tries to dereference your char[0] which means that it will look for a memory address with the value of the character you have defined in char[0]. That memory address is not valid, so your interpreter will signal a SIGSEGV.
Now to create a class which properly handles a variable length buffer is quite difficult. An easier option is to pass the buffer as an opaque handle, to access it afterwards you need to cast it back to a char array.
Example:
class FILE_HANDLE(Structure):
_fields_ = [
("handle_bytes", c_uint),
("handle_type", c_int),
("f_handle", c_void_p)
]
buf = create_string_buffer(8)
buf = cast(buf, c_void_p)
fh = FILE_HANDLE(c_uint(8), c_int(0), buf)
f_handle_value = (c_char * fh.handle_bytes).from_address(fh.f_handle)

Python equivalent of C struct (porting app form C to python)

I'm porting a simple bluetooth app, which sends "magic" packet on L2Cap protocol to bluetooth device..
I have a problem with converting struct object in C to python equivalent..
In c:
/* command types */
#define CFGBT_TYPE_SETREQ 0x00
#define CFGBT_TYPE_SETRES 0x01
#define CFGBT_TYPE_GETREQ 0x02
#define CFGBT_TYPE_GETRES 0x03
/* varid types */
#define CFG_VARIDT_UINT16 0x0000
#define CFG_VARIDT_UINT32 0x1000
#define CFG_VARIDT_STRING16 0x2000
typedef struct {
uint8_t type, status;
uint16_t varid;
uint8_t value[16];
} __attribute__((packed)) CFGBTFRAME;
static CFGBTFRAME c;
and then in app it's used like that:
/* set up */
c.type = CFGBT_TYPE_GETREQ;
c.varid = strtol(argv[3], NULL, 0);
c.status = 0;
memset(c.value, 0, 16);
/* send frame */
write(s, &c, sizeof(c));
Can you point me out how to construct same packet/stuct-like structure using python?
I know I will probably need to use ctypes and create "empty" class, but how to get all this together?

You can go about it with the struct module to pack values into a byte string, for example:
>>> import struct
>>> type, status, varid, value = 1, 0, 16, b'Hello'
>>> buffer = struct.pack('>BBH16s', type, status, varid, value)
>>> buffer
b'\x01\x00\x00\x10Hello\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'
Alternatively, you can use ctypes.Structure to define a class that will be representing your structure. It has the advantage of being easier to use with Python code, but you have to take into account alignment and padding issues and resolve them yourself (perhaps with struct).

If your goal is to group a set of key/value in an object, you may use dict or namedtuple.
a dict would be:
CFGBTFRAME = {
'type' : myType,
'status' : ...
}
access: CFGBTFRAME['type']
with namedtuple:
from collections import namedtuple
CFGBTFRAME = namedtuple('CFGBTFRAME', ['type', 'status', ...])
c = CFGBTFRAME()
c.type = myType
see http://docs.python.org/library/collections.html#collections.namedtuple for more information about namedtuple.

python using ctypes to work with dll - structure OUT argument

In the header file of the dll I have the following structure
typedef struct USMC_Devices_st{
DWORD NOD; // Number of the devices ready to work
char **Serial; // Array of 16 byte ASCII strings
char **Version; // Array of 4 byte ASCII strings
} USMC_Devices; // Structure representing connected devices
I would like to call a dll function:
DWORD USMC_Init( USMC_Devices &Str );
I tried with this:
class USMCDevices(Structure):
_fields_ = [("NOD", c_long),
("Serial", c_char_p),
("Version", c_char_p)]
usmc = cdll.USMCDLL #this is the dll file
init = usmc.USMC_Init
init.restype = c_int32; # return type
init.argtypes = [USMCDevices]; # argument
dev = USMCDevices()
init(dev)
I get an error here. I guess the problem is with "Serial" and "Version" which both are array corresponding to the NOD (number of devices).
Any ideas how to solve this problem?
I really appreciate your help!!!

Use POINTER(c_char_p) for the char ** pointers. Indexing Serial or Version creates a Python string for the given null-terminated string. Note that indexing in the array beyond NOD - 1 either produces garbage values or will crash the interpreter.
C:
#include <windows.h>
typedef struct USMC_Devices_st {
DWORD NOD; // Number of the devices ready to work
char **Serial; // Array of 16 byte ASCII strings
char **Version; // Array of 4 byte ASCII strings
} USMC_Devices;
char *Serial[] = {"000000000000001", "000000000000002"};
char *Version[] = {"001", "002"};
__declspec(dllexport) DWORD USMC_Init(USMC_Devices *devices) {
devices->NOD = 2;
devices->Serial = Serial;
devices->Version = Version;
return 0;
}
// build: cl usmcdll.c /LD
Python:
import ctypes
from ctypes import wintypes
class USMCDevices(ctypes.Structure):
_fields_ = [("NOD", wintypes.DWORD),
("Serial", ctypes.POINTER(ctypes.c_char_p)),
("Version", ctypes.POINTER(ctypes.c_char_p))]
usmc = ctypes.cdll.USMCDLL
init = usmc.USMC_Init
init.restype = wintypes.DWORD
init.argtypes = [ctypes.POINTER(USMCDevices)]
dev = USMCDevices()
init(ctypes.byref(dev))
devices = [dev.Serial[i] + b':' + dev.Version[i]
for i in range(dev.NOD)]
print('\n'.join(d.decode('ascii') for d in devices))
Output:
000000000000001:001
000000000000002:002

Python & Ctypes: Passing a struct to a function as a pointer to get back data

I've looked through other answers but can't seem to get this to work. I'm trying to call a function within a DLL for communicating with SMBus devices. This function takes a pointer to a struct, which has an array as one of it's fields. so...
In C:
typedef struct _SMB_REQUEST
{
unsigned char Address;
unsigned char Command;
unsigned char BlockLength;
unsigned char Data[SMB_MAX_DATA_SIZE];
} SMB_REQUEST;
I think I have to set values for the Address, Command and BlockLength while the DLL fills the Data array.
The function that requires this struct takes it as a pointer
SMBUS_API int SmBusReadByte( SMBUS_HANDLE handle, SMB_REQUEST *request );
So I've set up the struct in Python like so:
class SMB_REQUEST(ctypes.Structure):
_fields_ = [("Address", c_char),
("Command", c_char),
("BlockLength", c_char),
("Data", type(create_string_buffer(SMB_MAX_DATA_SIZE))]
*Note: I've also tried ctypes.c_char*SMB_MAX_DATA_SIZE for the data type*
To pass a pointer to a struct of this type to the function I have tried to initialise it first as follows:
data = create_string_buffer(SMB_MAX_DATA_SIZE)
smb_request = SMB_REQUEST('\x53', \x00', 1, data)
This responds with:
TypeError: expected string or Unicode object, c_char_Array_32 found
If I try leaving out the data array, like so:
smb_request = SMB_REQUEST('\x53', \x00', 1)
No, error.
However, then when I try to pass this to the function:
int_response = smbus_read_byte(smbus_handle, smb_request))
I get:
ArgumentError: argument 2: <type 'exceptions.TypeError'>: expected LP_SMB_REQUES
T instance instead of SMB_REQUEST
I've tried passing it as a pointer:
int_response = smbus_read_byte(smbus_handle, ctypes.POINTER(smb_request))
and I get:
----> 1
2
3
4
5
TypeError: must be a ctypes type
Here's how I've set up the art types:
smbus_read_byte.argtypes = (ctypes.c_void_p, ctypes.POINTER(SMB_REQUEST))
I've tried casting but still no go. Can anyone shed some light on this for me?
Update:
If I first initialise the struct like so:
smb_request = SMB_REQUEST('\xA6', '\x00', chr(1), 'a test string')
and then bass by reference:
int_response = smbus_receive_byte(smbus_handle, ctypes.byref(smb_request))
I get no error. However, the function returns -1 when it should return '0' for success and non-zero for a fail. Checking the value of smb_request.Data gives back 'a test string' so no change there.
Any suggestions as to what might be going on here would be greatly appreciated.
Thanks
UPDATE:
Since I've gotten a couple of enquiries about whether my handle is correct, here's how I'm using it. The header file for the DLL declares the following:
typedef void *SMBUS_HANDLE;
//
// This function call initializes the SMBus, opens the driver and
// allocates the resources associated with the SMBus.
// All SMBus API calls are valid
// after making this call except to re-open the SMBus.
//
SMBUS_API SMBUS_HANDLE OpenSmbus(void);
So here's how I'm doing this in python:
smbus_handle = c_void_p() # NOTE: I have also tried it without this line but same result
open_smbus = CDLL('smbus.dll').OpenSmbus
smbus_handle = open_smbus()
print 'SMBUS_API SMBUS_HANDLE OpenSmbus(void): ' + str(smbus_handle)
I call this before making the call to smbus_read_byte(). I have tried to set open_smbus.restype = c_void_p() but I get an error: TypeError: restype must be a type, a callable, or None

Here's a working example. It looks like you are passing the wrong type to the function.
Test DLL Code ("cl /W4 /LD x.c" on Windows)
#include <stdio.h>
#define SMBUS_API __declspec(dllexport)
#define SMB_MAX_DATA_SIZE 5
typedef void* SMBUS_HANDLE;
typedef struct _SMB_REQUEST
{
unsigned char Address;
unsigned char Command;
unsigned char BlockLength;
unsigned char Data[SMB_MAX_DATA_SIZE];
} SMB_REQUEST;
SMBUS_API int SmBusReadByte(SMBUS_HANDLE handle,SMB_REQUEST *request)
{
unsigned char i;
for(i = 0; i < request->BlockLength; i++)
request->Data[i] = i;
return request->BlockLength;
}
SMBUS_API SMBUS_HANDLE OpenSmbus(void)
{
return (void*)0x12345678;
}
Python code
from ctypes import *
SMB_MAX_DATA_SIZE = 5
ARRAY5 = c_ubyte * SMB_MAX_DATA_SIZE
class SMB_REQUEST(Structure):
_fields_ = [
("Address", c_ubyte),
("Command", c_ubyte),
("BlockLength", c_ubyte),
("Data", ARRAY5)]
smbus_read_byte = CDLL('x').SmBusReadByte
smbus_read_byte.argtypes = [c_void_p,POINTER(SMB_REQUEST)]
smbus_read_byte.restype = c_int
open_smbus = CDLL('x').OpenSmbus
open_smbus.argtypes = []
open_smbus.restype = c_void_p
handle = open_smbus()
print 'handle = %08Xh' % handle
smb_request = SMB_REQUEST(1,2,5)
print 'returned =',smbus_read_byte(handle,byref(smb_request))
print 'Address =',smb_request.Address
print 'Command =',smb_request.Command
print 'BlockLength =',smb_request.BlockLength
for i,b in enumerate(smb_request.Data):
print 'Data[%d] = %02Xh' % (i,b)
Output
handle = 12345678h
returned = 5
Address = 1
Command = 2
BlockLength = 5
Data[0] = 00h
Data[1] = 01h
Data[2] = 02h
Data[3] = 03h
Data[4] = 04h

You're almost there. You should use c_char * SMB_MAX_DATA_SIZE as the type for the definition of Data. This works for me on Mac OS X:
Shared library:
$ cat test.c
#include <stdio.h>
#define SMB_MAX_DATA_SIZE 16
typedef struct _SMB_REQUEST
{
unsigned char Address;
unsigned char Command;
unsigned char BlockLength;
unsigned char Data[SMB_MAX_DATA_SIZE];
} SMB_REQUEST;
int SmBusReadByte(void *handle, SMB_REQUEST *request)
{
printf("SmBusReadByte: handle=%p request=[%d %d %d %s]\n", handle,
request->Address, request->Command, request->BlockLength, request->Data);
return 13;
}
$ gcc test.c -fPIC -shared -o libtest.dylib
Python driver:
$ cat test.py
import ctypes
SMB_MAX_DATA_SIZE = 16
class SMB_REQUEST(ctypes.Structure):
_fields_ = [("Address", ctypes.c_ubyte),
("Command", ctypes.c_ubyte),
("BlockLength", ctypes.c_ubyte),
("Data", ctypes.c_char * SMB_MAX_DATA_SIZE)]
libtest = ctypes.cdll.LoadLibrary('libtest.dylib')
req = SMB_REQUEST(1, 2, 3, 'test')
result = libtest.SmBusReadByte(ctypes.c_voidp(0x12345678), ctypes.byref(req))
print 'result: %d' % result
$ python test.py
SmBusReadByte: handle=0x12345678 request=[1 2 3 test]
result: 13
UPDATE
You're having problems because you need to set the result type of open_smbus to void*. By default, ctypes assumes that functions return ints. You need to say this:
open_smbus.restype = ctypes.c_void_p
You were getting an error because you were using c_void_p() (note the extra parentheses). There's an important distinction between c_void_p and c_void_p(). The former is a type, and the latter is an instance of a type. c_void_p represents the C type void*, whereas c_void_p() represents an actual pointer instance (with a default value of 0).

Try changing
("Data", type(create_string_buffer(SMB_MAX_DATA_SIZE))
to
("Data", (c_char * SMB_MAX_DATA_SIZE)]