I am trying to write a wrapper around libedit (a BSD alternative to readline with a slightly different API) and there is a way to add a named function to the C API.
For example in C:
static unsigned char show_help(EditLine *e, int ch) {
printf("Help");
}
el = el_init(argv[0], stdin, stdout, stderr);
el_set(el, EL_ADDFN, "help", "This is help", show_help);
el_set(el, EL_BIND, "\?", "help", NULL);
I call el_set to add a function and then bind that function later on.
I can't find a good way to allow me to wrap EL_ADDFN to bind Python methods dynamically. I could create a bunch of prenamed C function and wrap them all individually to python methods, but I would rather like emulate the C API as closely as possible.
Is there a way to call EL_ADDFN and determine which python method it is calling?
Try this: One single handler function (which I'll describe below). Wrap EL_ADDFN so that it records the mapping of name to python function, but always uses the one handler function. Wrap EL_BIND, so that it records the mapping of character to function name. Your handler function should look up the ch parameter in your character to name mapping and then lookup the name to function mapping and then call the function. (If ADDFN must be called before BIND, you could create a map of ch to function and populate that directly in the the BIND wrapper.)
In pseudo C:
const char *chmap[256]; // initialize to zero
struct hashtable *namemap; // up to you to find a
// hashtable implementation that
// will take const char * and map to
// PyObject * (function object);
static unsigned char python_func(EditLine *e, int ch) {
const char *name = chmap[ch];
// check for errors
PyObject *func = lookup(namemap, name);
// check for errors
PyObject *editline = convert(e); // or whatever you have
PyObject *result = PyObject_CallFunctionObjArgs(func, NULL);
// check result, convert to unsigned char, and return
}
So, ADDFN wrapper populates the hashtable, and the BIND operator populates the chmap.
Related
I am new to c++,
I created DLL which contains Class, and functions and all the return type of each function is PyObject (Python Object), So now i want to write the C++ application which loads DLL dynamically using LoadLibrary function.
Able to execute with adding the project to same solution and adding the reference to DLL.
I am able to load the DLL, but when i am calling the function it returns PyObject data type, how to store the return type of PyObject in C++?
You should take a look at Python's documentation on Concrete Objects Layer. Basically you have to convert PyObject into a C++ type using a function of the form Py*T*_As*T*(PyObject* obj) where T is the concrete type you want to retrieve.
The API assumes you know which function you should call. But, as stated in the doc, you can check the type before use:
...if you receive an object from a Python program and you are not sure that it has the right type, you must perform a type check first; for example, to check that an object is a dictionary, use PyDict_Check().
Here is an example to convert a PyObject into long:
PyObject* some_py_object = /* ... */;
long as_long(
PyLong_AsLong(some_py_object)
);
Py_DECREF(some_py_object);
Here is another, more complicated example converting a Python list into a std::vector:
PyObject* some_py_list = /* ... */;
// assuming the list contains long
std::vector<long> as_vector(PyList_Size(some_py_list));
for(size_t i = 0; i < as_vector.size(); ++i)
{
PyObject* item = PyList_GetItem(some_py_list, i);
as_vector[i] = PyLong_AsLong(item);
Py_DECREF(item);
}
Py_DECREF(some_py_list);
A last, more complicated example, to parse a Python dict into a std::map:
PyObject* some_py_dict = /* ... */;
// assuming the dict uses long as keys, and contains string as values
std::map<long, std::string> as_map;
// first get the keys
PyObject* keys = PyDict_Keys(some_py_dict);
size_t key_count = PyList_Size(keys);
// loop on the keys and get the values
for(size_t i = 0; i < key_count; ++i)
{
PyObject* key = PyList_GetItem(keys, i);
PyObject* item = PyDict_GetItem(some_py_dict, key);
// add to the map
as_map.emplace(PyLong_AsLong(key), PyString_AsString(item));
Py_DECREF(key);
Py_DECREF(item);
}
Py_DECREF(keys);
Py_DECREF(some_py_dict);
I have an API that I am trying to wrap using SWIG such that I can call the underlying C library from python.
I have got stuck with a particular API fn:
int update_tracks(track_t **phash_tracks,
const pdws_t *pdw_frame,
const rdws_t *rdw_frame,
lib_t *lib,
lib_meta_t *lib_meta,
const cfg_t *cfg);
Its the double pointer to track_t data structure that I can't handle.
All the single pointers work fine.
This is the only API fn that has a double pointer to track_t
All the others only have a single pointer, e.g.
void print_hash_tracks(const track_t *hash_tracks, const cfg_t *cfg,
enum TRKTYPE trktype);
I'm pretty sure I need to make a typemap in my SWIG interface file (interface.i) but I am finding the SWIG docs impenetrable.
What I think I need to do is create a typemap that whenever it sees the track_t** type, it takes a track_t* and converts it to its address, something like:
/* provide typemap to handle instances of track_t** parameters */
%typemap(in) track_t** (track_t *tracks) {
$1 = &tracks;
}
but I'm just getting segmentation faults when I run:
tracks = g3.track_t()
g3.update_tracks(tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
on the python side.
I feel like I've almost solved this but can't quite get the typemap specification right and at the same time struggling to understand the relevant documentation.
flexo - if you're out there - maybe you can shed some light on this, you seem to be the SO expert in this area..
UPDATE - m7ython (brilliant! another SWIG expert on SO)
Usage in C is pretty straigthforward
declare and initialise a track_t pointer to NULL:
track_t *hash_tracks = NULL;
then:
update_tracks(&hash_tracks, &pdw_frame, &rdw_frame,
&lib, &lib_meta, &cfg);
So the address of the pointer to track_t is passed as an arg to update_tracks(). The update_tracks() fn takes care of all the necessary mallocs for the data that gets put into hash_tracks, i.e. the hash table of track_t structs
All the other args are single pointers and I can create and populate them with no issues on the python side.
track_t is a struct containing a bunch of ints, floats, char* etc. e.g.
typedef struct
{
/* make struct hashable */
UT_hash_handle hh;
int id;
...
char name[MAX_BUF];
...
} track_t;
The reason that the track_t arg is a track_t** and not just a track_t* is because hash_tracks is a pointer to a hash table (using the UTHash library). hash_tracks points to the 1st track_t in the hash table. In the body of the update_tracks() fn track_t structs can be added/removed from the hash table, such that the pointer to the 1st track_t may change, i.e. hash_tracks may point to something else after the call to update_tracks(), hence the reason for passing a pointer to the pointer.
In other words, the track_t** arg, phash_tracks is being used both as an input and output type arg, hence the pointer to a pointer. All the other args are simply inputs, they don't change so they can be passed in as single pointers.
I attempted the 'helper fn' route with the following C fn:
track_t** make_phash_tracks(void)
{
track_t **phash_tracks;
phash_tracks = calloc(1, sizeof(track_t*));
return phash_tracks;
}
the use of calloc should ensure that *phash_tracks is NULL
this compiled and wrapped with no errors, but when I used it from the python side it segfaulted, e.g.
phash_tracks = g3.make_phash_tracks()
g3.update_tracks(phash_tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
checking the phash_tracks var just prior to calling update_tracks gave:
(Pdb) p phash_tracks
<Swig Object of type 'track_t **' at 0x7fb9e37c9030>
EDIT: Ok, I think I now understand what update_tracksdoes. It seems you can use the function in two ways. Either to update existing tracks, or to create tracks if you pass a pointer to a NULL pointer. I am not sure about the most elegant way to handle both cases in SWIG (or if this is even a problem), but here are some options.
1. phash_tracks is an output argument
First, you must pass *phash_tracks back to Python as a return value, and use the function in some form like
>>> int_res, tracks = g3.update_tracks(tracks, pdw_frame, rdw_frame, lib, lib_meta, cfg)
or
>>> int_res, tracks = g3.update_tracks(pdw_frame, rdw_frame, lib, lib_meta, cfg)
This is accomplished by the following "argout" typemap:
%typemap(argout) track_t **phash_tracks {
%append_output(SWIG_NewPointerObj(%as_voidptr(*$1), $*1_descriptor, SWIG_POINTER_OWN));
}
Maybe you don't want Python to take ownership of the track_t*, then replace SWIG_POINTER_OWN by 0.
2. Passing an empty phash_tracks
If you only want to use the update_tracks function to create tracks, you can do essentially what you are already doing. Use the following "in" typemap, and use the function as in the second example above (without the tracks parameter).
%typemap(in, numinputs=0) track_t **phash_tracks (track_t *tracks) {
tracks = NULL;
$1 = &tracks;
}
3. phash_tracks as an input (and output) argument
If you want to use update_tracks to update existing tracks, you should be able to use the "in" typemap I suggested before, and use the function from Python as in the first example (including the tracks parameter).
%typemap(in) track_t **phash_tracks (track_t *tracks) {
if ((SWIG_ConvertPtr($input, (void **) &tracks, $*1_descriptor, SWIG_POINTER_EXCEPTION | SWIG_POINTER_DISOWN)) == -1)
return NULL;
$1 = &tracks;
}
Note that it is important that Python disowns its tracks_t*.
4. Enabling both (2) and (3) above
You could basically use version (3) also to create tracks, if you could get swig to pass a wrapped NULL tracks_t*. I am not sure if SWIG allows this -- but maybe it does. Try using a helper function:
tracks_t* empty_tracks() { return NULL; }
Alternatively, you can modify the "in" typemap along the following lines, attempting to convert the provided argument to a track_t* and passing its address, or alternatively passing the address of a NULL track_t*.
%typemap(in) track_t **phash_tracks (track_t *tracks) {
// Alternatively, check if $input is a 0 integer `PyObject`...
if ((SWIG_ConvertPtr($input, (void **) &tracks, $*1_descriptor, SWIG_POINTER_DISOWN)) == -1)
tracks = NULL;
$1 = &tracks;
}
Then, from Python, just pass something else to create tracks:
>>> int_res, tracks = g3.update_tracks(0, pdw_frame, rdw_frame, lib, lib_meta, cfg)
I have the following setup:
GDAL library with Python bindings (SWIG)
Some glue code (Python)
A C library, interfaced with ctypes
I want to pass the underlying dataset pointer/handle of the SWIG Dataset object to my C library. How can I retrieve this pointer?
I do not want to interface the C library with SWIG.
It was actually quite easy, and I hope that my solution is portable. Given, that my C function definition looks somewhat like this:
int myfunc(GDALDatasetH ds);
Then my ctypes definition is like this:
_lib = C.LibraryLoader(C.CDLL).LoadLibrary(lib_path)
_myfunc = _lib.myfunc
_myfunc.argtypes = [C.c_void_p]
_myfunc.restype = C.POINTER(C.c_char)
And I can call the C function with:
ds = gdal.Open(path)
...
_myfunc(C.c_void_p(long(ds.this)))
My reservation with the ctypes approach for this problem is that the reference count of the ds object is not incremented automatically and will become a bad pointer if it were to go out of scope.
A better approach would be to define a C python extension module that would manage the data reference counter.
I'm using a static PyObject * to hold the object, obviously a real implementation would store it more intelligently.
static PyObject * ds;
PyObject* GiveDsToC(PyObject * self, PyObject * args)
{
PyObject * pThis=NULL;
unsigned long addr;
if(!PyArg_ParseTuple(args, "O", &ds))
return NULL;
/* Ensure the interpreter keeps ds around while we have it */
Py_INCREF(ds);
pThis = PyObject_GetAttrString(ds, "this"); // new reference
addr = PyLong_AsLong(pThis); // convert using __int__ method
Py_DECREF(pThis); // Release the object back
CallSomeCFunction(addr);
Py_RETURN_NONE;
}
void FinishedWithDS(void)
{
// Lock the GIL and decrement the reference counter
PyGILState_STATE state = PyGILState_Ensure();
Py_DECREF(ds);
PyGILState_Release(state);
}
I ran into a situation with pure python and C python module.
To summarize, how can I accept and manipulate python object in C module?
My python part will look like this.
#!/usr/bin/env python
import os, sys
from c_hello import *
class Hello:
busyHello = _sayhello_obj
class Man:
def __init__(self, name):
self.name = name
def getName(self):
return self.name
h = Hello()
h.busyHello( Man("John") )
in C, two things need to be resolved.
first, how can I receive object?
second, how can I call a method from the object?
static PyObject *
_sayhello_obj(PyObject *self, PyObject *args)
{
PyObject *obj;
// How can I fill obj?
char s[1024];
// How can I fill s, from obj.getName() ?
printf("Hello, %s\n", s);
return Py_None;
}
To extract an argument from an invocation of your method, you need to look at the functions documented in Parsing arguments and building values, such as PyArg_ParseTuple. (That's for if you're only taking positional args! There are others for positional-and-keyword args, etc.)
The object you get back from PyArg_ParseTuple doesn't have it's reference count increased. For simple C functions, you probably don't need to worry about this. If you're interacting with other Python/C functions, or if you're releasing the global interpreter lock (ie. allowing threading), you need to think very carefully about object ownership.
static PyObject *
_sayhello_obj(PyObject *self, PyObject *args)
{
PyObject *obj = NULL;
// How can I fill obj?
static char fmt_string = "O" // For "object"
int parse_result = PyArg_ParseTuple(args, fmt_string, &obj);
if(!parse_res)
{
// Don't worry about using PyErr_SetString, all the exception stuff should be
// done in PyArg_ParseTuple()
return NULL;
}
// Of course, at this point you need to do your own verification of whatever
// constraints might be on your argument.
For calling a method on an object, you need to use either PyObject_CallMethod or PyObject_CallMethodObjArgs, depending on how you construct the argument list and method name. And see my comment in the code about object ownership!
Quick digression just to make sure you're not setting yourself up for a fall later: If you really are just getting the string out to print it, you're better off just getting the object reference and passing it to PyObject_Print. Of course, maybe this is just for illustration, or you know better than I do what you want to do with the data ;)
char s[1024];
// How can I fill s, from obj.getName() ?
// Name of the method
static char method_name = "getName";
// No arguments? Score! We just need NULL here
char method_fmt_string = NULL;
PyObject *objname = PyObject_CallMethod(obj, obj_method, method_fmt_string);
// This is really important! What we have here now is a Python object with a newly
// incremented reference count! This means you own it, and are responsible for
// decrementing the ref count when you're done. See below.
// If there's a failure, we'll get NULL
if(objname == NULL)
{
// Again, this should just propagate the exception information
return NULL;
}
Now there are a number of functions in the String/Bytes Objects section of the Concrete Objects Layer docs; use whichever works best for you.
But do not forget this bit:
// Now that we're done with the object we obtained, decrement the reference count
Py_XDECREF(objname);
// You didn't mention whether you wanted to return a value from here, so let's just
// return the "None" singleton.
// Note: this macro includes the "return" statement!
Py_RETURN_NONE;
}
Note the use of Py_RETURN_NONE there, and note that it's not return Py_RETURN_NONE!
PS. The structure of this code is dictated to a great extent by personal style (eg. early returns, static char format strings inside the function, initialisation to NULL). Hopefully the important information is clear enough apart from stylistic conventions.
I've got a C python extension, and I would like to print out some diagnostics.
I'm receiving a string as a PyObject*.
What's the canonical way to obtain a string representation of this object, such that it usable as a const char *?
Use PyObject_Repr (to mimic Python's repr function) or PyObject_Str (to mimic str), and then call PyString_AsString to get char * (you can, and usually should, use it as const char*, for example:
PyObject* objectsRepresentation = PyObject_Repr(yourObject);
const char* s = PyString_AsString(objectsRepresentation);
This method is OK for any PyObject. If you are absolutely sure yourObject is a Python string and not something else, like for instance a number, you can skip the first line and just do:
const char* s = PyString_AsString(yourObject);
Here is the correct answer if you are using Python 3:
static void reprint(PyObject *obj) {
PyObject* repr = PyObject_Repr(obj);
PyObject* str = PyUnicode_AsEncodedString(repr, "utf-8", "~E~");
const char *bytes = PyBytes_AS_STRING(str);
printf("REPR: %s\n", bytes);
Py_XDECREF(repr);
Py_XDECREF(str);
}
If you need just print the object in Python 3 you can use one of these functions:
static void print_str(PyObject *o)
{
PyObject_Print(o, stdout, Py_PRINT_RAW);
}
static void print_repr(PyObject *o)
{
PyObject_Print(o, stdout, 0);
}
Try PyObject_Repr (to mimic Python's repr) or PyObject_Str (to mimic Python's str).
Docs:
Compute a string representation of
object o. Returns the string
representation on success, NULL on
failure. This is the equivalent of the
Python expression repr(o). Called by
the repr() built-in function.
For python >=3.3:
char* str = PyUnicode_1BYTE_DATA(py_object);
Yes, this is a non-const pointer, you can potentially modify the (immutable) string via it.
PyObject *module_name;
PyUnicode_AsUTF8(module_name)
For an arbitrary PyObject*, first call
PyObject_Repr() or PyObject_Str() to get a PyUnicode* object.
In Python 3.3 and up, call PyUnicode_AsUTF8AndSize. In addition to the Python string you want a const char * for, this function takes an optional address to store the length in.
Python strings are objects with explicit length fields that may contain null bytes, while a const char* by itself is typically a pointer to a null-terminated C string. Converting a Python string to a C string is a potentially lossy operation. For that reason, all the other Python C-API functions that could return a const char* from a string are deprecated.
If you do not care about losing a bunch of the string if it happens to contain an embedded null byte, you can pass NULL for the size argument. For example,
PyObject* foo = PyUnicode_FromStringAndSize("foo\0bar", 7);
printf("As const char*, ignoring length: %s\n",
PyUnicode_AsUTF8AndSize(foo, NULL));
prints
As const char*, ignoring length: foo
But you can also pass in the address of a size variable, to use with the const char*, to make sure that you’re getting the entire string.
PyObject* foo = PyUnicode_FromStringAndSize("foo\0bar", 7);
printf("Including size: ");
size_t size;
const char* data = PyUnicode_AsUTF8AndSize(foo, &size);
fwrite(data, sizeof(data[0]), size, stdout);
putchar('\n');
On my terminal, that outputs
$ ./main | cat -v
Including size: foo^#bar