I am writing a Python wrapper for cpp APIs in that for one API I am trying to pass a NULL structure pointer as a parameter. Not sure how we can achieve that in Python.
Below is my sample implementation:
cpp_header.hpp
typedef enum {
E_FLAG_ON = 0,
E_FLAG_OFF
} option;
typedef struct {
float *a;
float b;
char *file_path;
option flag;
} inputs;
// API
int op_init(const inputs*);
This is what happening inside the init API:
Implementation.cpp
int op_init(const inputs* usr_ptr) {
internal_opration_read_set(&local_struct) { // local struct variable
// read one input file and update the structure values
}
if (usr_prt != NULL) {
internal_opration_update_set(usr_ptr, &local_struct) {
// update the values only if i send NOT NULL structure
}
}
}
From cpp test application I'm passing NULL structure to initialize.
test.cpp
int main() {
inputs *usr_cfg = NULL;
op_init(usr_cfg);
}
ctypes_imple.py
From ctypes import *
class inputs(Structure):
_fields_ = [('a', POINTER(c_float)),
('b', c_float),
('file_path', c_char_p),
('flag', option)]
# loading so file
so_lib = CDLL('some so')
# how we can initialize NULL structure pointer here?
so_lib.op_init() # how to send structure pointer as argument?
NOTE: this API reads inputs from a file and updates values to a structure in C. I am clueless how we can achieve the same in a Python wrapper? I mean updating values from so file runtime to a ctypes Python class.
Use None to pass a null pointer:
so_lib.op_init(None)
To send the actual structure instantiate one and send it. Best to define .argtypes and restype as well so ctypes doesn't have to guess and can perform better error checking:
so_lib.op_init.argtypes = POINTER(inputs),
so_lib.op_init.restype = c_int
arg = inputs() # all initialized to zero/null by default.
so_lib.op_init(arg)
Related
I have a hierarchical data structure of 3 structs in C++, which I renamed in my minimum working example (mwe) House, Room, and Objects. Room is a boost::optional type and has a std::vector<Object> member containing all Objects in this room. The Objects are just a container for some numbers.
I am aware that this is overly complex for such information, but it is necessary in the original code and cannot be changed easily. I tried to change it into a std::experimental::optional since we do not use c++17, but this broke some parts in our c++ code and I don't know whether it would actually solve the problem.
In C++, I have no issues at all and the boost::optional and all member variables work perfectly. But after binding everything I run into the weird problem that the std::vector objects is emptied as soon as I access any member variables of the Room. This can be either the length, width, area or the objects itself as shown in the Python example. If the objects are accessed for the first time, they are actually returned normally, but when trying the second access they are gone as if the had been moved. The same behavior applies if you perform x = myHouse.kitchen.objects.copy(). The list is in x and can even be accessed multiple times, but the information in kitchen is immediately lost. Weirdly, this also only applies to the objects and all other double members can be accessed indefinitely.
For the convenience of compiling everything for the mwe and debugging, I stuffed everything into a single cpp file, which is obviously not the case in the original code. For the mwe, the c++ code is compiled and included via cppimport, but also manual compilation does not change anything.
Here is the mwe:
mwe.cpp:
#include <boost/optional.hpp>
#include <iostream>
#include <map>
#include <string>
#include <vector>
#include <pybind11/complex.h>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>
/* implementing structs */
struct Object {
Object() = default;
Object(double price, double height) : price(price), height(height) {
};
double price = 0.0;
double height = 0.0;
};
struct Room {
Room() = default;
double length;
double width;
std::vector<Object> objects; // this is the buggy vector
double area() const {
return length * width;
};
};
struct House {
House() = default;
boost::optional<Room> bedroom;
boost::optional<Room> kitchen;
boost::optional<Room> livingroom;
std::map<std::string, std::vector<Object>> getObjects() {
std::map<std::string, std::vector<Object>> out;
if (bedroom) {
out.insert(std::make_pair("bedroom", bedroom->objects));
}
if (kitchen) {
out.insert(std::make_pair("kitchen", kitchen->objects));
}
if (livingroom) {
out.insert(std::make_pair("livingroom", livingroom->objects));
}
return out;
};
};
/* everything works fine in C++ -> get data this way to have complete object map */
House initSomethingInCpp() {
auto myHouse = House();
myHouse.bedroom = Room();
myHouse.kitchen = Room();
myHouse.bedroom->length = 10.0;
myHouse.bedroom->width = 2.0;
myHouse.kitchen->length = 5.0;
myHouse.kitchen->width = 3.0;
std::vector<Object> bedroomObjects;
std::vector<Object> kitchenObjects;
Object closet = Object(100.0, 2.5);
Object bed = Object(200.0, 1.0);
Object oven = Object(500.0, 1.5);
Object table = Object(50.0, 1.5);
bedroomObjects.push_back(closet);
bedroomObjects.push_back(bed);
kitchenObjects.push_back(oven);
kitchenObjects.push_back(table);
myHouse.bedroom->objects = bedroomObjects;
myHouse.kitchen->objects = kitchenObjects;
return myHouse;
};
namespace pybind11 {
/* taking care of boost type */
namespace detail {
/* boost::optional */
template<typename T>
struct type_caster<boost::optional<T>> : optional_caster<boost::optional<T>> {};
} // namespace detail
} // namespace pybind11
/* binding structs */
void init_house(pybind11::module& main) {
pybind11::class_<House> house(main, "House");
house.def(pybind11::init<>());
house.def_readwrite("bedroom", &House::bedroom);
house.def_readwrite("kitchen", &House::kitchen);
house.def_readwrite("livingroom", &House::livingroom);
house.def("get_objects", &House::getObjects);
};
void init_room(pybind11::module& main) {
pybind11::class_<Room> room(main, "Room");
room.def(pybind11::init<>());
room.def_readwrite("length", &Room::length);
room.def_readwrite("width", &Room::width);
room.def_readwrite("objects", &Room::objects);
room.def_property_readonly("area", &Room::area);
};
void init_objects(pybind11::module& main) {
pybind11::class_<Object> object(main, "Object");
object.def(pybind11::init<>());
object.def(pybind11::init<double, double>());
object.def_readonly("price", &Object::price);
object.def_readonly("heigth", &Object::height);
};
/* define module and bind init_in_cpp function */
PYBIND11_MODULE(mwe, m) {
init_house(m);
init_room(m);
init_objects(m);
m.def("init_something_in_cpp", &initSomethingInCpp);
};
execute.py:
import cppimport
#cppimport.set_quiet(False)
#cppimport.force_rebuild()
mod = cppimport.imp('mwe')
# get data
myHouse = mod.init_something_in_cpp()
print("\n")
print("all data is here")
objs = myHouse.get_objects()
print(objs)
print(myHouse.kitchen.area) # by accessing kitchen members, the objects list is emptied
print("\n")
print("kitchen objects are now missing")
objs = myHouse.get_objects()
print(objs)
print("but area still works:")
print(myHouse.kitchen.area) # everything but objects still works
print("\n")
print("also works directly with same variable")
print("bedroom objects are accessed:")
print(myHouse.bedroom.objects)
print("bedroom objects are accessed again:")
print(myHouse.bedroom.objects)
The execution gives the following output:
all data is here
{'bedroom': [mwe.Object object at 0x7fbc9c2a43f0, mwe.Object object at 0x7fbc9c2a4670], 'kitchen': [mwe.Object object at 0x7fbc9c2a4c30, mwe.Object object at 0x7fbc9c2a4df0]}
15.0
kitchen objects are now missing
{'bedroom': [mwe.Object object at 0x7fbc9c2a4e70, mwe.Object object at 0x7fbc9c2a4eb0], 'kitchen': []}
but area still works:
15.0
also works directly with same variable
bedroom objects are accessed:
[mwe.Object object at 0x7fbc9c2a4c30, mwe.Object object at 0x7fbc9c2a4df0]
bedroom objects are accessed again:
[]
Turns out this was actually a bug in pybind11 release 2.5 https://github.com/pybind/pybind11/issues/1919
It is fixed in the current master branch and future releases.
I have two different C functions and I would like to use them with ctypes in Python.
One function is establishing a connection and returns a pointer to an truct. The pointer shall be used as an argument in the second function to reuse the established connection.
C Code:
customStruct * connect()
{
customStruct *obj = connection_helper();
return obj;
}
void foo(customStruct * obj)
{
foo_helper(obj);
}
Python code:
from ctypes import *
lib = CDLL("./test.dll")
obj = lib.connect()
lib.foo(obj)
Unfortunately, I retrieve access violation errors when I call lib.foo(). I could recreate the customStruct struct in Python using a class with the _fields_ attribute, but since the struct consists of many other structs and since I don't want to access the struct members in Python itself, I'm thinking about an alternative how to create an identifier that can be reused.
I can change the definition of connect() as well as foo() as I'd like. I could also create another "identifier" struct if that would allow me to not have to recreate the struct in python.
Update:
It looks like I have to use the function byref() to achieve what I want.
https://docs.python.org/3/library/ctypes.html#ctypes.byref
The documentation states "The returned object can only be used as a foreign function call parameter" but I am not sure what to return in connect() then.
If you have an opaque structure (you do not know its members, or do not want to know its members), you should still create a class to represent that struct in python. You can then use this class to properly type your functions. This will help prevent bugs where you accidentally pass the wrong object as a "CustomStruct" pointer.
For example:
from ctypes import cdll, c_int, c_void_p
mylib = cdll.LoadLibrary('mylib')
class CustomStructP(c_void_p):
# subclassing c_void_p creates an opaque pointer type that is distinct
# from c_void_p, and can only be instantiated as a pointer
pass
create = mylib.create
create.argtypes = [c_int]
create.restype = CustomStructP
display = mylib.display
display.argtypes = [CustomStructP]
display.restype = None
delete = mylib.delete
delete.argtypes = [CustomStructP]
delete.restype = None
obj = create(10)
display(obj)
delete(obj)
display(CustomStructP()) # passing a null pointer
Now, if you tried something like: display(c_void_p()), you would get:
Traceback (most recent call last):
File "C:\Users\User\Documents\python\src\main.py", line 31, in <module>
display(c_void_p())
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
The C code I used was:
#include <stdio.h>
#include <stdlib.h>
struct customStruct {
int val;
};
struct customStruct *
create(int val) {
struct customStruct *obj = malloc(sizeof(struct customStruct));
obj->val = val;
return obj;
}
void
display(struct customStruct *obj) {
if (obj) {
printf("customStruct(%d) # %p\n", obj->val, obj);
}
else {
puts("customStruct is NULL");
}
}
void
delete(struct customStruct *obj) {
free(obj);
}
Like mentioned in comments already you need to set restype for the connect function and argtypes for the foo function on Python side.
In code it would look like this:
from ctypes import *
lib = cdll.LoadLibrary("some.dll")
lib.connect.restype = c_void_p
lib.foo.argtypes = c_void_p,
obj = lib.connect()
lib.foo(obj)
Test
A short test should verify that this gives the same pointer in your connection and foo function on the C side.
A slightly modified version of your code might look like this:
#include <stdlib.h>
#include <stdio.h>
typedef struct {
int x;
} customStruct;
static customStruct *connection_helper() {
return malloc(sizeof(customStruct));
}
customStruct *connect()
{
customStruct *obj = connection_helper();
printf("connect: %p\n", obj);
return obj;
}
void foo(customStruct * obj)
{
printf("foo: %p\n", obj);
//do something
}
If you run this you get something like:
connect: 0x7fa219e094a0
foo: 0x7fa219e094a0
I'm writing framework in Python 2.7 which would provide functionality to test C written APIs. I have the DLL and the C source code itself. The problem is that some of the C API functions require as an input structure of a callback functions and I can't find the way how to create structure in Python which fields are the C function pointers. I know how to create callback function in Python using ctypes, but I need to pack those callbacks in C structure to pass to C API. All these in code looks like this:
The C function pointers and the C structure:
#define DRIVERCALLAPI
typedef void*( DRIVERCALLAPI *fn_DRV_COMM_Open )(void*);
typedef void(DRIVERCALLAPI *fn_DRV_COMM_Close)(void*);
typedef int(DRIVERCALLAPI *fn_DRV_COMM_Transfer)(void *, unsigned char *, int);
typedef struct DriverCallbacks_t
{
fn_DRV_COMM_Open drvcomm_Open;
fn_DRV_COMM_Close drvcomm_Close;
fn_DRV_COMM_Transfer drvcomm_Transfer;
} DriverCallbacks;
typedef struct InitDataEntry_t
{
char iface[64];
void* handle;
} InitDataEntry;
Where handle points to an object of DriverCallbacks.
typedef struct InitDataContainer_t
{
uint32_t size;
uint32_t id;
InitDataEntry* data;
} InitDataContainer;
The pointer of a InitDataContainer should be passed to the API function.
void* dev_Create( void* args )
The API initializes the callback functions with appropriate functions to use them later. I need to somehow create Python structures of DriverCallbacks InitDataEntry and InitDataContainer. Any hint on how it can be achieved ? Thanks in advance !
After many experiments I finally found how to create Python structures which correspond to C structures with function pointers as fields. The idea is to use void pointers instead i.e. c_void_p from ctypes. For the provided example the python code would be:
from ctypes import *
class DriverCallbacks(Structure):
_fields_ = [
("drvcomm_Open", c_void_p),
("drvcomm_Close", c_void_p),
("drvcomm_Transfer", c_void_p)
]
class InitDataEntry(Structure):
_fields_ = [
("iface", 64 * c_byte),
("handle", c_void_p)
]
class InitDataContainer(Structure):
_fields_ = [
("size", c_uint),
("id", c_uint),
("data", POINTER(InitDataEntry))
]
The creation of the objects and the library function call could be like this (it's tested and works for me):
lib = cdll.LoadLibrary(LIBNAME)
driverFuncList = DriverCallbacks()
driverFuncList.drvcomm_Open = cast(lib.ftdi_open, c_void_p)
driverFuncList.drvcomm_Close = cast(lib.ftdi_close, c_void_p)
driverFuncList.drvcomm_Transfer = cast(lib.ftdi_transfer, c_void_p)
initData = InitDataEntry()
libc = cdll.msvcrt
libc.strcpy(byref(initData.iface), c_char_p("DriverCallbacks"))
initData.handle = cast(pointer(driverFuncList), c_void_p)
initDataCont = InitDataContainer()
initDataCont.size = c_uint(3)
initDataCont.id = c_uint(0)
initDataCont.data = pointer(initData)
ret = lib.dev_Create(byref(initDataCont))
The driverFuncList object can be filled also from within the C library if there is a such function which sets the callback function pointers.
I have a C++ myObject class that I expose via boost python using a wrapper structure:
struct myObjectWrapper{
static tuple compute(myObject& o,const Container& x0, const double& t0, Container& x){
double t;
int stat = o.evaluate(x0,t0,x,t);
return make_tuple(stat,t);
}
}
BOOST_PYTHON_MODULE(myModule)
{
// not shown here is code to expose Container class
class_<myObject>("MyObject")
.def("compute",&myObjectWrapper::compute)
;
}
Container is currently defined as:
typedef std::valarray<double> Container
and is exposed to python.
Now in python I can do.
x = Container()
(status,t) = obj.compute(Container([0.,0.,0.]),0.0,x)
print status, t, x[0]
This is not very pythonic. I would prefer to do:
(status,t,x) = obj.compute(Container([0.,0.,0.]),0.0)
print status, t, x[0]
I could write an additional wrapper in python, but I would prefer to avoid adding more wrappers.
The following code does't compile:
struct myObjectWrapper{
static tuple compute(myObject& o,const Container& x0, const double& t0){
double t;
Container x;
int stat = o.evaluate(x0,t0,x,t);
return make_tuple(stat,t,x);
}
}
Also I would prefer to steal the content of the local variable x and have python manage it rather than copy it:
return make_tuple(stat,t,std::move(x));
How do I achieve this?
In short, allocate the wrapper on the free store and use the manage_new_object result convert to transfer ownership to a Python object. This will cause Boost.Python to copy the pointer when constructing the Python object, rather than copying the pointee. For more details, see this answer.
Here is an auxiliary function that will transfer ownership to a Python object:
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
And one could use it as follows:
boost::python::tuple myObjectWrapper::compute(
myObject& o, const Container& x0, const double& t0)
{
auto x1 = std::make_unique<container>();
double t1 = 0;
int stat = self.evaluate(x0, t0, *x1, t1);
return boost::python::make_tuple(stat, t1, transfer_to_python(x1.release()));
}
Here is a complete example based on the original question that demonstrates using the transfer_to_python auxiliary function.
#include <boost/python.hpp>
#include <cassert>
#include <memory> // std::unique_ptr
// Mock legacy API.
struct container
{
container() {}
container(boost::python::object) {}
container(const container&)
{
// For this example, guarantee copy is not made.
assert(false);
}
};
struct my_object
{
int evaluate(container&, double&, container&, double&) { return 42; }
};
/// #brief Transfer ownership to a Python object. If the transfer fails,
/// then object will be destroyed and an exception is thrown.
template <typename T>
boost::python::object transfer_to_python(T* t)
{
// Transfer ownership to a smart pointer, allowing for proper cleanup
// incase Boost.Python throws.
std::unique_ptr<T> ptr(t);
// Use the manage_new_object generator to transfer ownership to Python.
namespace python = boost::python;
typename python::manage_new_object::apply<T*>::type converter;
// Transfer ownership to the Python handler and release ownership
// from C++.
python::handle<> handle(converter(*ptr));
ptr.release();
return python::object(handle);
}
// API wrapper.
boost::python::tuple my_object_compute(
my_object& self, container& x0, double t0)
{
auto x1 = std::make_unique<container>();
double t1 = 21;
int stat = self.evaluate(x0, t0, *x1, t1);
return boost::python::make_tuple(stat, t1, transfer_to_python(x1.release()));
}
BOOST_PYTHON_MODULE(example)
{
namespace python = boost::python;
python::class_<container>("Container")
.def(python::init<python::object>())
;
python::class_<my_object>("MyObject")
.def("compute", &my_object_compute)
;
}
Interactive usage:
>>> import example
>>> my_object = example.MyObject()
>>> status, t, x = my_object.compute(example.Container([1, 2, 3]), 4)
>>> assert(status == 42)
>>> assert(t == 21)
>>> assert(isinstance(x, example.Container))
What would be the C equivalent to this python code?
Thanks.
data = gather_me_some_data()
# where data = [ (metic, datapoints), ... ]
# and datapoints = [ (timestamp, value), ... ]
serialized_data = cPickle.dumps(data, protocol=-1)
length_prefix = struct.pack("!L", len(serialized_data))
message = length_prefix + serialized_data
C doesn't supports direct serialization mechanism because in C you can't get type information at run-time. You must yourself inject some type info at run-time and then construct required object by that type info. So define all your possible structs:
typedef struct {
int myInt;
float myFloat;
unsigned char myData[MY_DATA_SIZE];
} MyStruct_1;
typedef struct {
unsigned char myUnsignedChar;
double myDouble;
} MyStruct_2;
Then define enum which collects info about what structs in total you have:
typedef enum {
ST_MYSTRUCT_1,
ST_MYSTRUCT_2
} MyStructType;
Define helper function which lets to determine any struct size:
int GetStructSize(MyStructType structType) {
switch (structType) {
case ST_MYSTRUCT_1:
return sizeof(MyStruct_1);
case ST_MYSTRUCT_2:
return sizeof(MyStruct_2);
default:
// OOPS no such struct in our pocket
return 0;
}
}
Then define serialize function:
void BinarySerialize(
MyStructType structType,
void * structPointer,
unsigned char * serializedData) {
int structSize = GetStructSize(structType);
if (structSize != 0) {
// copy struct metadata to serialized bytes
memcpy(serializedData, &structType, sizeof(structType));
// copy struct itself
memcpy(serializedData+sizeof(structType), structPointer, structSize);
}
}
And de-serialization function:
void BinaryDeserialize(
MyStructType structTypeDestination,
void ** structPointer,
unsigned char * serializedData)
{
// get source struct type
MyStructType structTypeSource;
memcpy(&structTypeSource, serializedData, sizeof(structTypeSource));
// get source struct size
int structSize = GetStructSize(structTypeSource);
if (structTypeSource == structTypeDestination && structSize != 0) {
*structPointer = malloc(structSize);
memcpy(*structPointer, serializedData+sizeof(structTypeSource), structSize);
}
}
Serialization usage example:
MyStruct_2 structInput = {0x69, 0.1};
MyStruct_1 * structOutput_1 = NULL;
MyStruct_2 * structOutput_2 = NULL;
unsigned char testSerializedData[SERIALIZED_DATA_MAX_SIZE] = {0};
// serialize structInput
BinarySerialize(ST_MYSTRUCT_2, &structInput, testSerializedData);
// try to de-serialize to something
BinaryDeserialize(ST_MYSTRUCT_1, &structOutput_1, testSerializedData);
BinaryDeserialize(ST_MYSTRUCT_2, &structOutput_2, testSerializedData);
// determine which object was de-serialized
// (plus you will get code-completion support about object members from IDE)
if (structOutput_1 != NULL) {
// do something with structOutput_1
free(structOutput_1);
}
else if (structOutput_2 != NULL) {
// do something with structOutput_2
free(structOutput_2);
}
I think this is most simple serialization approach in C. But it has some problems:
struct must not have pointers, because you will never know how much memory one needs to allocate when serializing pointers and from where/how to serialize data into pointers.
this example has issues with system endianess - you need to be careful about how data is stored in memory - in big-endian or little-endian fashion and reverse bytes if needed [when casting char * to integal type such as enum] (...or refactor code to be more portable).
If you can use C++, there is the PicklingTools library