There is a libx.so which export 2 functions, and a struct,
typedef struct Tag {
int num;
char *name;
}Tag;
Tag *create(int n, char *n)
{
Tag *t = malloc(sizeof(Tag));
t->num = n;
t->name = n;
return t;
}
void use(Tag *t)
{
printf("%d, %s\n", t->num, t->name);
}
I want to call create in Python and then save the Tag *res returned by create, later I will call use and pass the Tag *res saved before to use, here is it (just to demonstrate):
>>>libx = ctypes.CDLL("./libx.so")
>>>res = libx.create(c_int(1), c_char_p("a"))
>>>libx.use(res)
The above code might be wrong, just to demonstrate what I want to do.
And my problem is that, how could I save the result returned by create? Because it returns a pointer to a user-defined struct, and I don't want to construct struct Tag's counterpart in Python, would c_void_p do the trick?
UPDATE
From #David's answer, I still don't quite understand one thing:
the pointer (c_char_p("a")) is only valid for the duration of the
call to create. As soon as create returns then that pointer is no
longer valid.
And I assign c_char_p("a") to t->name in create, when the call to create finishes, is t->name a dangling pointer? Because according to the quoted words, that pointer is no longer valid after create. Why c_char_p("a") is no longer valid?
The C code that you present is simply not going to work. You need to be much more precise about which party allocates and is responsible for the heap memory.
In your current example you pass c_char_p("a") to the C code. However, the pointer to that ctypes memory is only valid for the duration of the call to create. As soon as create returns then that pointer is no longer valid. But you took a copy of the pointer inside create. Thus the subsequent call to use is liable to fail.
You are going to need to take a copy of the contents of that string and store it in the struct. If you do that then you can use libx.create.restype = c_void_p safely.
But if you want the memory you allocated to be deallocated you will have to provide a destroy function to match the create function. With these changes the C code would look like this:
Tag *create(int n, char *s)
{
Tag *t = malloc(sizeof(Tag));
t->num = n;
t->name = strdup(s);
return t;
}
void destroy(Tag *t)
{
free(t->name);
free(t);
}
The Python code would look like this:
libx = ctypes.CDLL("./libx.so")
libx.create.restype = c_void_p
res = libx.create(c_int(1), c_char_p("a"))
libx.use(res)
libx.destroy(res)
Python does reference counting. You'll have to use Py_INCREF() and friends for objects that are returned from "external" libraries.
UPDATE: I don't know about .so loading by python, maybe the method proposed by #David Hefferman does this automagically.
UPDATE2: delete me!
Related
I have an API that I am trying to wrap using SWIG such that I can call the underlying C library from python.
I have got stuck with a particular API fn:
int update_tracks(track_t **phash_tracks,
const pdws_t *pdw_frame,
const rdws_t *rdw_frame,
lib_t *lib,
lib_meta_t *lib_meta,
const cfg_t *cfg);
Its the double pointer to track_t data structure that I can't handle.
All the single pointers work fine.
This is the only API fn that has a double pointer to track_t
All the others only have a single pointer, e.g.
void print_hash_tracks(const track_t *hash_tracks, const cfg_t *cfg,
enum TRKTYPE trktype);
I'm pretty sure I need to make a typemap in my SWIG interface file (interface.i) but I am finding the SWIG docs impenetrable.
What I think I need to do is create a typemap that whenever it sees the track_t** type, it takes a track_t* and converts it to its address, something like:
/* provide typemap to handle instances of track_t** parameters */
%typemap(in) track_t** (track_t *tracks) {
$1 = &tracks;
}
but I'm just getting segmentation faults when I run:
tracks = g3.track_t()
g3.update_tracks(tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
on the python side.
I feel like I've almost solved this but can't quite get the typemap specification right and at the same time struggling to understand the relevant documentation.
flexo - if you're out there - maybe you can shed some light on this, you seem to be the SO expert in this area..
UPDATE - m7ython (brilliant! another SWIG expert on SO)
Usage in C is pretty straigthforward
declare and initialise a track_t pointer to NULL:
track_t *hash_tracks = NULL;
then:
update_tracks(&hash_tracks, &pdw_frame, &rdw_frame,
&lib, &lib_meta, &cfg);
So the address of the pointer to track_t is passed as an arg to update_tracks(). The update_tracks() fn takes care of all the necessary mallocs for the data that gets put into hash_tracks, i.e. the hash table of track_t structs
All the other args are single pointers and I can create and populate them with no issues on the python side.
track_t is a struct containing a bunch of ints, floats, char* etc. e.g.
typedef struct
{
/* make struct hashable */
UT_hash_handle hh;
int id;
...
char name[MAX_BUF];
...
} track_t;
The reason that the track_t arg is a track_t** and not just a track_t* is because hash_tracks is a pointer to a hash table (using the UTHash library). hash_tracks points to the 1st track_t in the hash table. In the body of the update_tracks() fn track_t structs can be added/removed from the hash table, such that the pointer to the 1st track_t may change, i.e. hash_tracks may point to something else after the call to update_tracks(), hence the reason for passing a pointer to the pointer.
In other words, the track_t** arg, phash_tracks is being used both as an input and output type arg, hence the pointer to a pointer. All the other args are simply inputs, they don't change so they can be passed in as single pointers.
I attempted the 'helper fn' route with the following C fn:
track_t** make_phash_tracks(void)
{
track_t **phash_tracks;
phash_tracks = calloc(1, sizeof(track_t*));
return phash_tracks;
}
the use of calloc should ensure that *phash_tracks is NULL
this compiled and wrapped with no errors, but when I used it from the python side it segfaulted, e.g.
phash_tracks = g3.make_phash_tracks()
g3.update_tracks(phash_tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
checking the phash_tracks var just prior to calling update_tracks gave:
(Pdb) p phash_tracks
<Swig Object of type 'track_t **' at 0x7fb9e37c9030>
EDIT: Ok, I think I now understand what update_tracksdoes. It seems you can use the function in two ways. Either to update existing tracks, or to create tracks if you pass a pointer to a NULL pointer. I am not sure about the most elegant way to handle both cases in SWIG (or if this is even a problem), but here are some options.
1. phash_tracks is an output argument
First, you must pass *phash_tracks back to Python as a return value, and use the function in some form like
>>> int_res, tracks = g3.update_tracks(tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
or
>>> int_res, tracks = g3.update_tracks(pdw_frame, rdw_frame, lib, lib_meta, cfg)
This is accomplished by the following "argout" typemap:
%typemap(argout) track_t **phash_tracks {
%append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, SWIG_POINTER_OWN));
}
Maybe you don't want Python to take ownership of the track_t*, then replace SWIG_POINTER_OWN by 0.
2. Passing an empty phash_tracks
If you only want to use the update_tracks function to create tracks, you can do essentially what you are already doing. Use the following "in" typemap, and use the function as in the second example above (without the tracks parameter).
%typemap(in, numinputs=0) track_t **phash_tracks (track_t *tracks) {
tracks = NULL;
$1 = &tracks;
}
3. phash_tracks as an input (and output) argument
If you want to use update_tracks to update existing tracks, you should be able to use the "in" typemap I suggested before, and use the function from Python as in the first example (including the tracks parameter).
%typemap(in) track_t **phash_tracks (track_t *tracks) {
if ((SWIG_ConvertPtr($input, (void **) &tracks, $*1_descriptor, SWIG_POINTER_EXCEPTION | SWIG_POINTER_DISOWN)) == -1)
return NULL;
$1 = &tracks;
}
Note that it is important that Python disowns its tracks_t*.
4. Enabling both (2) and (3) above
You could basically use version (3) also to create tracks, if you could get swig to pass a wrapped NULL tracks_t*. I am not sure if SWIG allows this -- but maybe it does. Try using a helper function:
tracks_t* empty_tracks() { return NULL; }
Alternatively, you can modify the "in" typemap along the following lines, attempting to convert the provided argument to a track_t* and passing its address, or alternatively passing the address of a NULL track_t*.
%typemap(in) track_t **phash_tracks (track_t *tracks) {
// Alternatively, check if $input is a 0 integer `PyObject`...
if ((SWIG_ConvertPtr($input, (void **) &tracks, $*1_descriptor, SWIG_POINTER_DISOWN)) == -1)
tracks = NULL;
$1 = &tracks;
}
Then, from Python, just pass something else to create tracks:
>>> int_res, tracks = g3.update_tracks(0, pdw_frame, rdw_frame, lib, lib_meta, cfg)
This question is related to a previous question I asked. Namely this one if anyone is interested. Basically, what I want to do is to expose a C array to Python using a Py_buffer wrapped in a memoryview-object. I've gotten it to work using PyBuffer_FillInfo (work = I can manipulate the data in Python and write it to stdout in C), but if I try to roll my own buffer I get a segfault after the C function returns.
I need to create my own buffer because PyBuffer_FillInfo assumes that the format is char, making the itemsize field 1. I need to be able to provide items of size 1, 2, 4 and 8.
Some code, this is a working example:
Py_buffer *buf = (Py_buffer *) malloc(sizeof(*buf));
int r = PyBuffer_FillInfo(buf, NULL, malloc(sizeof(char) * 4), 4, 0, PyBUF_CONTIG);
PyObject *mv = PyMemoryView_FromBuffer(buf);
//Pack the memoryview object into an argument list and call the Python function
for (blah)
printf("%c\n", *buf->buf++); //this prints the values i set in the Python function
Looking at the implementation of PyBuffer_FillInfo, which is really simple, I rolled my own function to be able to provide custom itemsizes:
//buffer creation function
Py_buffer *getReadWriteBuffer(int nitems, int itemsize, char *fmt) {
Py_buffer *buf = (Py_buffer *) malloc(sizeof(*buf));
buf->obj = NULL
buf->buf = malloc(nitems * itemsize);
buf->len = nitems * itemsize;
buf->readonly = 0;
buf->itemsize = itemsize;
buf->format = fmt;
buf->ndim = 1;
buf->shape = NULL;
buf->strides = NULL;
buf->suboffsets = NULL;
buf->internal = NULL;
return buf;
}
How i use it:
Py_buffer *buf = getReadWriteBuffer(32, 2, "h");
PyObject *mv = PyMemoryView_FromBuffer(buf);
// pack the memoryview into an argument list and call the Python function as before
for (blah)
printf("%d\n", *buf->buf); //this prints all zeroes even though i modify the array in Python
return 0;
//the segfault happens somewhere after here
The result of using my own buffer object is a segfault after the C function returns. I really don't understand why this happens at all. Any help would be most appreciated.
EDIT
According to this question, which I failed to find before, itemsize > 1 might not even be supported at all. Which makes this question even more interesting. Maybe I could use PyBuffer_FillInfo with a large enough block of memory to hold what I want (32 C floats for example). In that case, the question is more about how to assign Python floats to the memoryview object in the Python function. Questions questions.
So, in lack of answers I decided to take another approach than the one I originally intended. Leaving this here in case someone else hits the same snag.
Basically, instead of creating a buffer (or bytearray, equiv.) in C and passing it to Python for the extension user to modify. I simply redesigned the code slightly, so that the user returns a bytearray (or any type that supports the buffer interface) from the Python callback function. This way I need not even worry about the size of the items since, in my case, all the C code does with the returned object is to extract its buffer and copy it to another buffer with a simple memcpy.
Code:
PYGILSTATE_ACQUIRE; //a macro i made
PyObject *result = PyEval_CallObject(python_callback, NULL);
if (!PyObject_CheckBuffer(result))
; //raise exception
Py_buffer *view = (Py_buffer *) malloc(sizeof(*view));
int error = PyObject_GetBuffer(result, view, PyBUF_SIMPLE);
if (error)
; //raise exception
memcpy(my_other_buffer, view->buf, view->len);
PyBuffer_Release(view);
Py_DECREF(result);
PYGILSTATE_RELEASE; //another macro
I hope this helps someone.
I need to construct the following data type in Python for passing to a C function:
struct {
unsigned a,b,c;
char data[8];
};
However, I need to actually pass a pointer to the data field to the function, not a pointer to a struct, and I can't figure out how to do this.
Here is what I have so far:
from ctypes import *
class MyStruct(Structure):
_fields_ = [("a",c_uint), ("b",c_uint), ("c",c_uint), ("data",c_char*8)]
mystruct = MyStruct(0,1,8,"ABCDEFGH")
external_c_function(mystruct.data)
Now in C I have this function:
int external_c_function(char *data) {
int a = ((unsigned *)data)[-1];
int b = ((unsigned *)data)[-2];
int c = ((unsigned *)data)[-3];
...
}
The problem is, when the function gets called, "data" correctly points to "ABCDEFGH", but when I try to get the rest of the struct data preceding it, it is garbage. What am I doing wrong? Isn't mystruct held sequentially in memory like a real C struct? I suspect something funny is going on with the array: am I actually doing something silly like this?
struct {
unsigned a,b,c;
char *data; // -> char[8]
};
and if so, how do I do it correctly?
You pass a pointer to an element of a structure by reference, using the offset of the element:
external_c_function(byref(mystruct,MyStruct.data.offset))
It seems that when you reference mystruct.data, a copy of the data is made. I say this because the python command type(mystruct.data), returns str, rather than a C type.
I presume that you are not able to modify the external_c_function to accept the pointer at the start of the structure, as this would be the most obvious solution. Therefore you need to somehow do C style pointer arithmetic in python - i.e. get the address of mystruct (possibly using ctypes.pointer), then figure out a way to increment this pointer by the appropriate number of bytes.
I don't know how you can do such pointer arithmetic in python, or if it's even possible to do in any robust manner. However, you could always wrap external_c_function in another C function which does the necessary pointer arithmetic.
edit
Mark's answer solves the problem neatly. My comment about why the error occurs is still correct.
I ran into a situation with pure python and C python module.
To summarize, how can I accept and manipulate python object in C module?
My python part will look like this.
#!/usr/bin/env python
import os, sys
from c_hello import *
class Hello:
busyHello = _sayhello_obj
class Man:
def __init__(self, name):
self.name = name
def getName(self):
return self.name
h = Hello()
h.busyHello( Man("John") )
in C, two things need to be resolved.
first, how can I receive object?
second, how can I call a method from the object?
static PyObject *
_sayhello_obj(PyObject *self, PyObject *args)
{
PyObject *obj;
// How can I fill obj?
char s[1024];
// How can I fill s, from obj.getName() ?
printf("Hello, %s\n", s);
return Py_None;
}
To extract an argument from an invocation of your method, you need to look at the functions documented in Parsing arguments and building values, such as PyArg_ParseTuple. (That's for if you're only taking positional args! There are others for positional-and-keyword args, etc.)
The object you get back from PyArg_ParseTuple doesn't have it's reference count increased. For simple C functions, you probably don't need to worry about this. If you're interacting with other Python/C functions, or if you're releasing the global interpreter lock (ie. allowing threading), you need to think very carefully about object ownership.
static PyObject *
_sayhello_obj(PyObject *self, PyObject *args)
{
PyObject *obj = NULL;
// How can I fill obj?
static char fmt_string = "O" // For "object"
int parse_result = PyArg_ParseTuple(args, fmt_string, &obj);
if(!parse_res)
{
// Don't worry about using PyErr_SetString, all the exception stuff should be
// done in PyArg_ParseTuple()
return NULL;
}
// Of course, at this point you need to do your own verification of whatever
// constraints might be on your argument.
For calling a method on an object, you need to use either PyObject_CallMethod or PyObject_CallMethodObjArgs, depending on how you construct the argument list and method name. And see my comment in the code about object ownership!
Quick digression just to make sure you're not setting yourself up for a fall later: If you really are just getting the string out to print it, you're better off just getting the object reference and passing it to PyObject_Print. Of course, maybe this is just for illustration, or you know better than I do what you want to do with the data ;)
char s[1024];
// How can I fill s, from obj.getName() ?
// Name of the method
static char method_name = "getName";
// No arguments? Score! We just need NULL here
char method_fmt_string = NULL;
PyObject *objname = PyObject_CallMethod(obj, obj_method, method_fmt_string);
// This is really important! What we have here now is a Python object with a newly
// incremented reference count! This means you own it, and are responsible for
// decrementing the ref count when you're done. See below.
// If there's a failure, we'll get NULL
if(objname == NULL)
{
// Again, this should just propagate the exception information
return NULL;
}
Now there are a number of functions in the String/Bytes Objects section of the Concrete Objects Layer docs; use whichever works best for you.
But do not forget this bit:
// Now that we're done with the object we obtained, decrement the reference count
Py_XDECREF(objname);
// You didn't mention whether you wanted to return a value from here, so let's just
// return the "None" singleton.
// Note: this macro includes the "return" statement!
Py_RETURN_NONE;
}
Note the use of Py_RETURN_NONE there, and note that it's not return Py_RETURN_NONE!
PS. The structure of this code is dictated to a great extent by personal style (eg. early returns, static char format strings inside the function, initialisation to NULL). Hopefully the important information is clear enough apart from stylistic conventions.
I'm trying to wrap the Patricia Tries (Perl's NET::Patricia) to be exposed in python. I am having difficulty with one of the classes.
So instances the patricia node (below) as viewed from python have a "data" property. Reading it goes fine, but writing to it breaks.
typedef struct _patricia_node_t {
u_int bit; /* flag if this node used */
prefix_t *prefix; /* who we are in patricia tree */
struct _patricia_node_t *l, *r; /* left and right children */
struct _patricia_node_t *parent;/* may be used */
void *data; /* pointer to data */
void *user1; /* pointer to usr data (ex. route flap info) */
} patricia_node_t;
Specifically:
>>> N = patricia.patricia_node_t()
>>> assert N.data == None
>>> N.data = 1
TypeError: in method 'patricia_node_t_data_set', argument 2 of type 'void *'
Now my C is weak. From what I read in the SWIG book, I think this means I need to pass it a pointer to data. According to the book :
Also, if you need to pass the raw pointer value to some external python library, you can do it by casting the pointer object to an integer... However, the inverse operation is not possible, i.e., you can't build a Swig pointer object from a raw integer value.
Questions:
am I understanding this correctly?
how do I get around this? Is %extends? typemap? Specifics would be very helpful.
Notes:
I can't change the C source, but I can extend it in additional .h files or the interface .i file.
From what I understand, that "data" field should be able to contain "anything" for some reasonable value of "anything" that I don't really know.
I haven't used SWIG in a while, but I am pretty sure that you want to use a typemap that will take a PyObject* and cast it to the required void* and vice versa. Be sure to keep track of reference counts, of course.
It looks like you should pass SWIG a pointer to an integer. For example, if this was all in C, your error would be like this:
void set(struct _patricia_node_t *tree, void *data) {
tree->data = data;
}
...
int value = 1;
set(tree, &value); // OK! HOORAY!
set(tree, value); // NOT OK! FIRE SCORPIONS!
And it seems to me you're doing the Python equivalent of set(tree, value). Now I'm not an expert with SWIG but perhaps you could pass a tuple instead of an integer? Does N.data = (1,) work? This was the answer suggested by an Allegro CL + SWIG example, but I dunno how well it applies to Python.
An alternative is use PyRadix, which uses the same underlying code.