I'm trying to wrap the Patricia Tries (Perl's NET::Patricia) to be exposed in python. I am having difficulty with one of the classes.
So instances the patricia node (below) as viewed from python have a "data" property. Reading it goes fine, but writing to it breaks.
typedef struct _patricia_node_t {
u_int bit; /* flag if this node used */
prefix_t *prefix; /* who we are in patricia tree */
struct _patricia_node_t *l, *r; /* left and right children */
struct _patricia_node_t *parent;/* may be used */
void *data; /* pointer to data */
void *user1; /* pointer to usr data (ex. route flap info) */
} patricia_node_t;
Specifically:
>>> N = patricia.patricia_node_t()
>>> assert N.data == None
>>> N.data = 1
TypeError: in method 'patricia_node_t_data_set', argument 2 of type 'void *'
Now my C is weak. From what I read in the SWIG book, I think this means I need to pass it a pointer to data. According to the book :
Also, if you need to pass the raw pointer value to some external python library, you can do it by casting the pointer object to an integer... However, the inverse operation is not possible, i.e., you can't build a Swig pointer object from a raw integer value.
Questions:
am I understanding this correctly?
how do I get around this? Is %extends? typemap? Specifics would be very helpful.
Notes:
I can't change the C source, but I can extend it in additional .h files or the interface .i file.
From what I understand, that "data" field should be able to contain "anything" for some reasonable value of "anything" that I don't really know.
I haven't used SWIG in a while, but I am pretty sure that you want to use a typemap that will take a PyObject* and cast it to the required void* and vice versa. Be sure to keep track of reference counts, of course.
It looks like you should pass SWIG a pointer to an integer. For example, if this was all in C, your error would be like this:
void set(struct _patricia_node_t *tree, void *data) {
tree->data = data;
}
...
int value = 1;
set(tree, &value); // OK! HOORAY!
set(tree, value); // NOT OK! FIRE SCORPIONS!
And it seems to me you're doing the Python equivalent of set(tree, value). Now I'm not an expert with SWIG but perhaps you could pass a tuple instead of an integer? Does N.data = (1,) work? This was the answer suggested by an Allegro CL + SWIG example, but I dunno how well it applies to Python.
An alternative is use PyRadix, which uses the same underlying code.
Related
I'm making a C++ extension for Python, and I'm trying to do something like:
// this function assigns a C++ pointer to as attribute of a python object
void function1(PyObject* p){
// equivalent of p.attr = cpp_attr;
MyClass* cpp_attr = new MyClass();
PyObject* args = PyTuple_Pack(cpp_attr);
PyObject_SetAttrString(p, (char*)"attr", args);
}
I would like to retrieve this pointer and set it as attribute of another C++ object. I know how to get the PyObject* but after that I'm not sure what to do anymore
MySecondClass::MySecondClass(PyObject* p){
// get the attribute from p; equivalent of cpp_attr = p.attr
PyObject* cpp_attr = PyObject_getAttrString(p, (char*)"attr"));
// somehow get back the pointer to MyClass object created in function1
}
I looked at the documentation but I couldn't find anything that returns the original type. Is there anyway to do this?
Thanks
It's difficult to be absolutely certain, but I doubt that MyClass a Python object. This means that your attempt to store it as a Python object (e.g. using PyTuple_Pack) is completely wrong and will cause Python to malfunction in unexpected ways.
What will happen is that Python will attempt to interpret the pointer as a Python object, will try to use its normal reference counting mechanisms on that object (will change it in unpredictable ways), and ultimately try to deallocate that object (using Python mechanisms, not delete...) if some part of the object happens to equal 0.
There's a number of options, all basically centred around creating a wrapper object - a Python object defined in C++ that holds either a pointer or value of your C++ object.
Do it manually using the Python C API - This answer gives a very thorough example.
Look up the PyCapsule interface to create a quick wrapper around your object. You'd create your capsule with:
PyObject* cap = PyCapsule_New(cpp_attr, "MyClass",
[](PyObject* c) {
auto deleteme = reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass));
delete deleteme;
});
And you retrieve your C++ class from the capsule with:
reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass))
Use some tool like PyBind11, Cython, SWIG, etc to create the wrapper object for you.
Note also that PyObject_SetAttrString does not require the third argument to be a tuple (unless you specifically want to store a tuple...). You're likely getting it confused with PyObject_Call, where the args are passed as a tuple.
Assuming your call to PyTuple_Pack is correct, then you've created a PyTupleObject which has a structure:
typedef struct {
PyObject_VAR_HEAD
PyObject *ob_item[1];
} PyTupleObject;
The PyTupleObject inherits from the generic PyObject struct which has the following members:
struct _object *_ob_next;
struct _object *_ob_prev;
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
You can access the latter two with the macrosPy_REFCNT and Py_TYPE
The ob_item[1] member should be a pointer to the memory initially allocated. Based on how Macros are written in the documentation, you should be able to access it by
((PyTupleObject *)cpp_attr)->ob_item
And if you know the data type of the C++ pointer, then you should be able to cast it back. Maybe you can try
MyClass* cpp_att_again = reinterpret_cast<MyClass*>((PyTupleObject *)cpp_attr)->ob_item
Hopefully this points you in the right direction. You might be able to glean more insight from a similar question.
I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.
You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.
If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>
How can I pass a str value (containing 3000 {'0', '1'} bytes) obtained using python code as an argument to a python c extended function (extended using SWIG) which requires int * (fixed length int array) as an input argument? My code is such:
int *exposekey(int *bits) {
int a[1000];
for (int j=2000; j < 3000; j++) {
a[j - 2000] = bits[j];
}
return a;
}
What I've tried was to use ctypes (see below code):
import ctypes
ldpc = ctypes.cdll.LoadLibrary('./_ldpc.so')
arr = (ctypes.c_int * 3072)(<mentioned below>)
ldpc.exposekey(arr)
with 3072 {0, 1} entered in the position. Python returns syntax error : more than 255 arguments. This still doesn't help me to pass assigned str value instead of the initialized ctypes int array.
Other suggestion included using SWIG typemaps but how would that work for converting a str into int * ? Thanks in advance.
Regarding my comment, here are some more details about returning arrays from functions: [SO]: Returning an array using C. In short: ways handle this:
Make the returned variable static
Dynamically allocate it (using malloc (family) or new)
Turn it into an additional argument for the function
Getting that piece of C code to run within the Python interpreter is possible in 2 ways:
[Python 3.Docs]: Extending Python with C or C++ - which creates a C written Python module
A way of doing that is using swig which offers a simple interface for generating the module ([SWIG]: SWIG Basics) saving you the trouble of writing it yourself using [Python 3.Docs]: Python/C API Reference Manual
The other way around, leaving the code in a standard dll which can be accessed via [Python 3.Docs]: ctypes - A foreign function library for Python
Since they both are doing the same thing, mixing them together makes no sense. So, pick the one that best fits your needs.
1. ctypes
This is what you started with
It's one of the ways of doing things using ctypes
ctypes_demo.c:
#include <stdio.h>
#if defined(_WIN32)
# define CTYPES_DEMO_EXPORT_API __declspec(dllexport)
#else
# define CTYPES_DEMO_EXPORT_API
#endif
CTYPES_DEMO_EXPORT_API int exposekey(char *bitsIn, char *bitsOut) {
int ret = 0;
printf("Message from C code...\n");
for (int j = 0; j < 1000; j++)
{
bitsOut[j] = bitsIn[j + 2000];
ret++;
}
return ret;
}
Notes:
Based on comments, I changed the types in the function from int* to char*, because it's 4 times more compact (although it's still ~700% inefficient since 7 bits of each char are ignored versus only one of them being used; that can be fixed, but requires bitwise processing)
I took a and turned into the 2nd argument (bitsOut). I think this is best because it's caller responsibility to allocate and deallocate the array (the 3rd option from the beginning)
I also modified the index range (without changing functionality), because it makes more sense to work with low index values and add something to them in one place, instead of a high index values and subtract (the same) something in another place
The return value is the number of bits set (obviously, 1000 in this case) but it's just an example
printf it's just dummy, to show that the C code gets executed
When dealing with such arrays, it's recommended to pass their dimensions as well, to avoid out of bounds errors. Also, error handling is an important aspect
test_ctypes.py:
from ctypes import CDLL, c_char, c_char_p, c_int, create_string_buffer
bits_string = "010011000110101110101110101010010111011101101010101"
def main():
dll = CDLL("./ctypes_demo.dll")
exposekey = dll.exposekey
exposekey.argtypes = [c_char_p, c_char_p]
exposekey.restype = c_int
bits_in = create_string_buffer(b"\0" * 2000 + bits_string.encode())
bits_out = create_string_buffer(1000)
print("Before: [{}]".format(bits_out.raw[:len(bits_string)].decode()))
ret = exposekey(bits_in, bits_out)
print("After: [{}]".format(bits_out.raw[:len(bits_string)].decode()))
print("Return code: {}".format(ret))
if __name__ == "__main__":
main()
Notes:
1st, I want to mention that running your code didn't raise the error you got
Specifying function's argtypes and restype is mandatory, and also makes things easier (documented in the ctypes tutorial)
I am printing the bits_out array (only the first - and relevant - part, as the rest are 0) in order to prove that the C code did its job
I initialize bits_in array with 2000 dummy 0 at the beginning, as those values are not relevant here. Also, the input string (bits_string) is not 3000 characters long (for obvious reasons). If your bits_string is 3000 characters long you can simply initialize bits_in like: bits_in = create_string_buffer(bits_string.encode())
Do not forget to initialize bits_out to an array with a size large enough (in our example 1000) for its purpose, otherwise segfault might arise when trying to set its content past the size
For this (simple) function, the ctypes variant was easier (at least for me, since I don't use swig frequently), but for more complex functions / projects it will become an overkill and switching to swig would be the right thing to do
Output (running with Python3.5 on Win):
c:\Work\Dev\StackOverflow\q47276327>"c:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" test_ctypes.py
Before: [ ]
Message from C code...
After: [010011000110101110101110101010010111011101101010101]
Return code: 1000
2. swig
Almost everything from the ctypes section, applies here as well
swig_demo.c:
#include <malloc.h>
#include <stdio.h>
#include "swig_demo.h"
char *exposekey(char *bitsIn) {
char *bitsOut = (char*)malloc(sizeof(char) * 1000);
printf("Message from C code...\n");
for (int j = 0; j < 1000; j++) {
bitsOut[j] = bitsIn[j + 2000];
}
return bitsOut;
}
swig_demo.i:
%module swig_demo
%{
#include "swig_demo.h"
%}
%newobject exposekey;
%include "swig_demo.h"
swig_demo.h:
char *exposekey(char *bitsIn);
Notes:
Here I'm allocating the array and return it (the 2nd option from the beginning)
The .i file is a standard swig interface file
Defines the module, and its exports (via %include)
One thing that is worth mentioning is the %newobject directive that deallocates the pointer returned by exposekey to avoid memory leaks
The .h file just contains the function declaration, in order to be included by the .i file (it's not mandatory, but things are more elegant this way)
The rest is pretty much the same
test_swig.py:
from swig_demo import exposekey
bits_in = "010011000110101110101110101010010111011101101010101"
def main():
bits_out = exposekey("\0" * 2000 + bits_in)
print("C function returned: [{}]".format(bits_out))
if __name__ == "__main__":
main()
Notes:
Things make much more sense from Python programmer's PoV
Code is a lot shorter (that is because swig did some "magic" behind the scenes):
The wrapper .c wrapper file generated from the .i file has ~120K
The swig_demo.py generated module has ~3K
I used the same technique with 2000 0 at the beginning of the string
Output:
c:\Work\Dev\StackOverflow\q47276327>"c:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" test_swig.py
Message from C code...
C function returned: [010011000110101110101110101010010111011101101010101]
3. Plain Python C API
I added this part as a personal exercise
This is what swig does, but "manually"
capi_demo.c:
#include "Python.h"
#include "swig_demo.h"
#define MOD_NAME "capi_demo"
static PyObject *PyExposekey(PyObject *self, PyObject *args) {
PyObject *bitsInArg = NULL, *bitsOutArg = NULL;
char *bitsIn = NULL, *bitsOut = NULL;
if (!PyArg_ParseTuple(args, "O", &bitsInArg))
return NULL;
bitsIn = PyBytes_AS_STRING(PyUnicode_AsEncodedString(bitsInArg, "ascii", "strict"));
bitsOut = exposekey(bitsIn);
bitsOutArg = PyUnicode_FromString(bitsOut);
free(bitsOut);
return bitsOutArg;
}
static PyMethodDef moduleMethods[] = {
{"exposekey", (PyCFunction)PyExposekey, METH_VARARGS, NULL},
{NULL}
};
static struct PyModuleDef moduleDef = {
PyModuleDef_HEAD_INIT, MOD_NAME, NULL, -1, moduleMethods
};
PyMODINIT_FUNC PyInit_capi_demo(void) {
return PyModule_Create(&moduleDef);
}
Notes:
It requires swig_demo.h and swig_demo.c (not going to duplicate their contents here)
It only works with Python 3 (actually I got quite some headaches making it work, especially because I was used to PyString_AsString which is no longer present)
Error handling is poor
test_capi.py is similar to test_swig.py with one (obvious) difference: from swig_demo import exposekey should be replaced by from capi_demo import exposekey
The output is also the same to test_swig.py (again, not going to duplicate it here)
I have an API that I am trying to wrap using SWIG such that I can call the underlying C library from python.
I have got stuck with a particular API fn:
int update_tracks(track_t **phash_tracks,
const pdws_t *pdw_frame,
const rdws_t *rdw_frame,
lib_t *lib,
lib_meta_t *lib_meta,
const cfg_t *cfg);
Its the double pointer to track_t data structure that I can't handle.
All the single pointers work fine.
This is the only API fn that has a double pointer to track_t
All the others only have a single pointer, e.g.
void print_hash_tracks(const track_t *hash_tracks, const cfg_t *cfg,
enum TRKTYPE trktype);
I'm pretty sure I need to make a typemap in my SWIG interface file (interface.i) but I am finding the SWIG docs impenetrable.
What I think I need to do is create a typemap that whenever it sees the track_t** type, it takes a track_t* and converts it to its address, something like:
/* provide typemap to handle instances of track_t** parameters */
%typemap(in) track_t** (track_t *tracks) {
$1 = &tracks;
}
but I'm just getting segmentation faults when I run:
tracks = g3.track_t()
g3.update_tracks(tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
on the python side.
I feel like I've almost solved this but can't quite get the typemap specification right and at the same time struggling to understand the relevant documentation.
flexo - if you're out there - maybe you can shed some light on this, you seem to be the SO expert in this area..
UPDATE - m7ython (brilliant! another SWIG expert on SO)
Usage in C is pretty straigthforward
declare and initialise a track_t pointer to NULL:
track_t *hash_tracks = NULL;
then:
update_tracks(&hash_tracks, &pdw_frame, &rdw_frame,
&lib, &lib_meta, &cfg);
So the address of the pointer to track_t is passed as an arg to update_tracks(). The update_tracks() fn takes care of all the necessary mallocs for the data that gets put into hash_tracks, i.e. the hash table of track_t structs
All the other args are single pointers and I can create and populate them with no issues on the python side.
track_t is a struct containing a bunch of ints, floats, char* etc. e.g.
typedef struct
{
/* make struct hashable */
UT_hash_handle hh;
int id;
...
char name[MAX_BUF];
...
} track_t;
The reason that the track_t arg is a track_t** and not just a track_t* is because hash_tracks is a pointer to a hash table (using the UTHash library). hash_tracks points to the 1st track_t in the hash table. In the body of the update_tracks() fn track_t structs can be added/removed from the hash table, such that the pointer to the 1st track_t may change, i.e. hash_tracks may point to something else after the call to update_tracks(), hence the reason for passing a pointer to the pointer.
In other words, the track_t** arg, phash_tracks is being used both as an input and output type arg, hence the pointer to a pointer. All the other args are simply inputs, they don't change so they can be passed in as single pointers.
I attempted the 'helper fn' route with the following C fn:
track_t** make_phash_tracks(void)
{
track_t **phash_tracks;
phash_tracks = calloc(1, sizeof(track_t*));
return phash_tracks;
}
the use of calloc should ensure that *phash_tracks is NULL
this compiled and wrapped with no errors, but when I used it from the python side it segfaulted, e.g.
phash_tracks = g3.make_phash_tracks()
g3.update_tracks(phash_tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
checking the phash_tracks var just prior to calling update_tracks gave:
(Pdb) p phash_tracks
<Swig Object of type 'track_t **' at 0x7fb9e37c9030>
EDIT: Ok, I think I now understand what update_tracksdoes. It seems you can use the function in two ways. Either to update existing tracks, or to create tracks if you pass a pointer to a NULL pointer. I am not sure about the most elegant way to handle both cases in SWIG (or if this is even a problem), but here are some options.
1. phash_tracks is an output argument
First, you must pass *phash_tracks back to Python as a return value, and use the function in some form like
>>> int_res, tracks = g3.update_tracks(tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
or
>>> int_res, tracks = g3.update_tracks(pdw_frame, rdw_frame, lib, lib_meta, cfg)
This is accomplished by the following "argout" typemap:
%typemap(argout) track_t **phash_tracks {
%append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, SWIG_POINTER_OWN));
}
Maybe you don't want Python to take ownership of the track_t*, then replace SWIG_POINTER_OWN by 0.
2. Passing an empty phash_tracks
If you only want to use the update_tracks function to create tracks, you can do essentially what you are already doing. Use the following "in" typemap, and use the function as in the second example above (without the tracks parameter).
%typemap(in, numinputs=0) track_t **phash_tracks (track_t *tracks) {
tracks = NULL;
$1 = &tracks;
}
3. phash_tracks as an input (and output) argument
If you want to use update_tracks to update existing tracks, you should be able to use the "in" typemap I suggested before, and use the function from Python as in the first example (including the tracks parameter).
%typemap(in) track_t **phash_tracks (track_t *tracks) {
if ((SWIG_ConvertPtr($input, (void **) &tracks, $*1_descriptor, SWIG_POINTER_EXCEPTION | SWIG_POINTER_DISOWN)) == -1)
return NULL;
$1 = &tracks;
}
Note that it is important that Python disowns its tracks_t*.
4. Enabling both (2) and (3) above
You could basically use version (3) also to create tracks, if you could get swig to pass a wrapped NULL tracks_t*. I am not sure if SWIG allows this -- but maybe it does. Try using a helper function:
tracks_t* empty_tracks() { return NULL; }
Alternatively, you can modify the "in" typemap along the following lines, attempting to convert the provided argument to a track_t* and passing its address, or alternatively passing the address of a NULL track_t*.
%typemap(in) track_t **phash_tracks (track_t *tracks) {
// Alternatively, check if $input is a 0 integer `PyObject`...
if ((SWIG_ConvertPtr($input, (void **) &tracks, $*1_descriptor, SWIG_POINTER_DISOWN)) == -1)
tracks = NULL;
$1 = &tracks;
}
Then, from Python, just pass something else to create tracks:
>>> int_res, tracks = g3.update_tracks(0, pdw_frame, rdw_frame, lib, lib_meta, cfg)
I need to construct the following data type in Python for passing to a C function:
struct {
unsigned a,b,c;
char data[8];
};
However, I need to actually pass a pointer to the data field to the function, not a pointer to a struct, and I can't figure out how to do this.
Here is what I have so far:
from ctypes import *
class MyStruct(Structure):
_fields_ = [("a",c_uint), ("b",c_uint), ("c",c_uint), ("data",c_char*8)]
mystruct = MyStruct(0,1,8,"ABCDEFGH")
external_c_function(mystruct.data)
Now in C I have this function:
int external_c_function(char *data) {
int a = ((unsigned *)data)[-1];
int b = ((unsigned *)data)[-2];
int c = ((unsigned *)data)[-3];
...
}
The problem is, when the function gets called, "data" correctly points to "ABCDEFGH", but when I try to get the rest of the struct data preceding it, it is garbage. What am I doing wrong? Isn't mystruct held sequentially in memory like a real C struct? I suspect something funny is going on with the array: am I actually doing something silly like this?
struct {
unsigned a,b,c;
char *data; // -> char[8]
};
and if so, how do I do it correctly?
You pass a pointer to an element of a structure by reference, using the offset of the element:
external_c_function(byref(mystruct,MyStruct.data.offset))
It seems that when you reference mystruct.data, a copy of the data is made. I say this because the python command type(mystruct.data), returns str, rather than a C type.
I presume that you are not able to modify the external_c_function to accept the pointer at the start of the structure, as this would be the most obvious solution. Therefore you need to somehow do C style pointer arithmetic in python - i.e. get the address of mystruct (possibly using ctypes.pointer), then figure out a way to increment this pointer by the appropriate number of bytes.
I don't know how you can do such pointer arithmetic in python, or if it's even possible to do in any robust manner. However, you could always wrap external_c_function in another C function which does the necessary pointer arithmetic.
edit
Mark's answer solves the problem neatly. My comment about why the error occurs is still correct.