I have the following setup:
GDAL library with Python bindings (SWIG)
Some glue code (Python)
A C library, interfaced with ctypes
I want to pass the underlying dataset pointer/handle of the SWIG Dataset object to my C library. How can I retrieve this pointer?
I do not want to interface the C library with SWIG.
It was actually quite easy, and I hope that my solution is portable. Given, that my C function definition looks somewhat like this:
int myfunc(GDALDatasetH ds);
Then my ctypes definition is like this:
_lib = C.LibraryLoader(C.CDLL).LoadLibrary(lib_path)
_myfunc = _lib.myfunc
_myfunc.argtypes = [C.c_void_p]
_myfunc.restype = C.POINTER(C.c_char)
And I can call the C function with:
ds = gdal.Open(path)
...
_myfunc(C.c_void_p(long(ds.this)))
My reservation with the ctypes approach for this problem is that the reference count of the ds object is not incremented automatically and will become a bad pointer if it were to go out of scope.
A better approach would be to define a C python extension module that would manage the data reference counter.
I'm using a static PyObject * to hold the object, obviously a real implementation would store it more intelligently.
static PyObject * ds;
PyObject* GiveDsToC(PyObject * self, PyObject * args)
{
PyObject * pThis=NULL;
unsigned long addr;
if(!PyArg_ParseTuple(args, "O", &ds))
return NULL;
/* Ensure the interpreter keeps ds around while we have it */
Py_INCREF(ds);
pThis = PyObject_GetAttrString(ds, "this"); // new reference
addr = PyLong_AsLong(pThis); // convert using __int__ method
Py_DECREF(pThis); // Release the object back
CallSomeCFunction(addr);
Py_RETURN_NONE;
}
void FinishedWithDS(void)
{
// Lock the GIL and decrement the reference counter
PyGILState_STATE state = PyGILState_Ensure();
Py_DECREF(ds);
PyGILState_Release(state);
}
Related
I try to write Python wrapper for C library using ctypes.
So far I have:
C.h
typedef struct
{
int erorrCode;
char * Key;
} A;
#ifdef __cplusplus
extern "C" {
#endif
EXPORT void __stdcall DestroyA(A &input);
#ifdef __cplusplus
}
#endif
C.cpp
EXPORT void __stdcall DestroyA(A &input)
{
delete []input.Key;
}
Python.py
import sys
import ctypes
class A(ctypes.Structure):
_fields_ = [
("erorrCode", ctypes.c_int),
("Key", ctypes.c_char_p)]
try:
libapi = ctypes.cdll.LoadLibrary('./lib.so')
except OSError:
print("Unable to load RAPI library")
sys.exit()
DestroyA = libapi.DestroyA
libapi.DestroyA.argtypes = [ctypes.POINTER(A)]
libapi.DestroyA.restype = None
a = A(1,b'random_string')
DestroyA(ctypes.byref(a)) #!!!here is segmentation fault
So, how can I fix the segmentation fault error?
Note: I cannot change the code on the C ++ side as long as there is a way to fix it on the Python side.
Listing [Python.Docs]: ctypes - A foreign function library for Python.
You have here Undefined Behavior (UB).
Python has builtin memory management for its objects, including CTypes ones. So, every time an object (PyObject which is basically anything - including a Python int), Python invokes one of the malloc functions family under the hood in order to allocate memory. Conversely, when the object is destroyed (manually or by GC), free is called.
What happened:
You created the object (behind the scenes, Python allocated some memory)
You called free on the object allocated by Python (which is wrong, not to mention that you also crossed the .dll boundary)
You need to call free only on pointers that you allocated. One such example: [SO]: python: ctypes, read POINTER(c_char) in python (#CristiFati's answer).
If you want to get rid of the object (and thus free the memory that it uses), let Python do it for you:
del a
Additional remarks:
You're using __stdcall functions with ctypes.CDLL. Again, that's UB (on 32bit). Use the "regular" calling convention (__cdecl)
You're passing a reference. That's C++ specific (although it's only a const ptr). To be C compatible, use:
EXPORT void destroyA(A *pInput);
How can I pass a str value (containing 3000 {'0', '1'} bytes) obtained using python code as an argument to a python c extended function (extended using SWIG) which requires int * (fixed length int array) as an input argument? My code is such:
int *exposekey(int *bits) {
int a[1000];
for (int j=2000; j < 3000; j++) {
a[j - 2000] = bits[j];
}
return a;
}
What I've tried was to use ctypes (see below code):
import ctypes
ldpc = ctypes.cdll.LoadLibrary('./_ldpc.so')
arr = (ctypes.c_int * 3072)(<mentioned below>)
ldpc.exposekey(arr)
with 3072 {0, 1} entered in the position. Python returns syntax error : more than 255 arguments. This still doesn't help me to pass assigned str value instead of the initialized ctypes int array.
Other suggestion included using SWIG typemaps but how would that work for converting a str into int * ? Thanks in advance.
Regarding my comment, here are some more details about returning arrays from functions: [SO]: Returning an array using C. In short: ways handle this:
Make the returned variable static
Dynamically allocate it (using malloc (family) or new)
Turn it into an additional argument for the function
Getting that piece of C code to run within the Python interpreter is possible in 2 ways:
[Python 3.Docs]: Extending Python with C or C++ - which creates a C written Python module
A way of doing that is using swig which offers a simple interface for generating the module ([SWIG]: SWIG Basics) saving you the trouble of writing it yourself using [Python 3.Docs]: Python/C API Reference Manual
The other way around, leaving the code in a standard dll which can be accessed via [Python 3.Docs]: ctypes - A foreign function library for Python
Since they both are doing the same thing, mixing them together makes no sense. So, pick the one that best fits your needs.
1. ctypes
This is what you started with
It's one of the ways of doing things using ctypes
ctypes_demo.c:
#include <stdio.h>
#if defined(_WIN32)
# define CTYPES_DEMO_EXPORT_API __declspec(dllexport)
#else
# define CTYPES_DEMO_EXPORT_API
#endif
CTYPES_DEMO_EXPORT_API int exposekey(char *bitsIn, char *bitsOut) {
int ret = 0;
printf("Message from C code...\n");
for (int j = 0; j < 1000; j++)
{
bitsOut[j] = bitsIn[j + 2000];
ret++;
}
return ret;
}
Notes:
Based on comments, I changed the types in the function from int* to char*, because it's 4 times more compact (although it's still ~700% inefficient since 7 bits of each char are ignored versus only one of them being used; that can be fixed, but requires bitwise processing)
I took a and turned into the 2nd argument (bitsOut). I think this is best because it's caller responsibility to allocate and deallocate the array (the 3rd option from the beginning)
I also modified the index range (without changing functionality), because it makes more sense to work with low index values and add something to them in one place, instead of a high index values and subtract (the same) something in another place
The return value is the number of bits set (obviously, 1000 in this case) but it's just an example
printf it's just dummy, to show that the C code gets executed
When dealing with such arrays, it's recommended to pass their dimensions as well, to avoid out of bounds errors. Also, error handling is an important aspect
test_ctypes.py:
from ctypes import CDLL, c_char, c_char_p, c_int, create_string_buffer
bits_string = "010011000110101110101110101010010111011101101010101"
def main():
dll = CDLL("./ctypes_demo.dll")
exposekey = dll.exposekey
exposekey.argtypes = [c_char_p, c_char_p]
exposekey.restype = c_int
bits_in = create_string_buffer(b"\0" * 2000 + bits_string.encode())
bits_out = create_string_buffer(1000)
print("Before: [{}]".format(bits_out.raw[:len(bits_string)].decode()))
ret = exposekey(bits_in, bits_out)
print("After: [{}]".format(bits_out.raw[:len(bits_string)].decode()))
print("Return code: {}".format(ret))
if __name__ == "__main__":
main()
Notes:
1st, I want to mention that running your code didn't raise the error you got
Specifying function's argtypes and restype is mandatory, and also makes things easier (documented in the ctypes tutorial)
I am printing the bits_out array (only the first - and relevant - part, as the rest are 0) in order to prove that the C code did its job
I initialize bits_in array with 2000 dummy 0 at the beginning, as those values are not relevant here. Also, the input string (bits_string) is not 3000 characters long (for obvious reasons). If your bits_string is 3000 characters long you can simply initialize bits_in like: bits_in = create_string_buffer(bits_string.encode())
Do not forget to initialize bits_out to an array with a size large enough (in our example 1000) for its purpose, otherwise segfault might arise when trying to set its content past the size
For this (simple) function, the ctypes variant was easier (at least for me, since I don't use swig frequently), but for more complex functions / projects it will become an overkill and switching to swig would be the right thing to do
Output (running with Python3.5 on Win):
c:\Work\Dev\StackOverflow\q47276327>"c:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" test_ctypes.py
Before: [ ]
Message from C code...
After: [010011000110101110101110101010010111011101101010101]
Return code: 1000
2. swig
Almost everything from the ctypes section, applies here as well
swig_demo.c:
#include <malloc.h>
#include <stdio.h>
#include "swig_demo.h"
char *exposekey(char *bitsIn) {
char *bitsOut = (char*)malloc(sizeof(char) * 1000);
printf("Message from C code...\n");
for (int j = 0; j < 1000; j++) {
bitsOut[j] = bitsIn[j + 2000];
}
return bitsOut;
}
swig_demo.i:
%module swig_demo
%{
#include "swig_demo.h"
%}
%newobject exposekey;
%include "swig_demo.h"
swig_demo.h:
char *exposekey(char *bitsIn);
Notes:
Here I'm allocating the array and return it (the 2nd option from the beginning)
The .i file is a standard swig interface file
Defines the module, and its exports (via %include)
One thing that is worth mentioning is the %newobject directive that deallocates the pointer returned by exposekey to avoid memory leaks
The .h file just contains the function declaration, in order to be included by the .i file (it's not mandatory, but things are more elegant this way)
The rest is pretty much the same
test_swig.py:
from swig_demo import exposekey
bits_in = "010011000110101110101110101010010111011101101010101"
def main():
bits_out = exposekey("\0" * 2000 + bits_in)
print("C function returned: [{}]".format(bits_out))
if __name__ == "__main__":
main()
Notes:
Things make much more sense from Python programmer's PoV
Code is a lot shorter (that is because swig did some "magic" behind the scenes):
The wrapper .c wrapper file generated from the .i file has ~120K
The swig_demo.py generated module has ~3K
I used the same technique with 2000 0 at the beginning of the string
Output:
c:\Work\Dev\StackOverflow\q47276327>"c:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe" test_swig.py
Message from C code...
C function returned: [010011000110101110101110101010010111011101101010101]
3. Plain Python C API
I added this part as a personal exercise
This is what swig does, but "manually"
capi_demo.c:
#include "Python.h"
#include "swig_demo.h"
#define MOD_NAME "capi_demo"
static PyObject *PyExposekey(PyObject *self, PyObject *args) {
PyObject *bitsInArg = NULL, *bitsOutArg = NULL;
char *bitsIn = NULL, *bitsOut = NULL;
if (!PyArg_ParseTuple(args, "O", &bitsInArg))
return NULL;
bitsIn = PyBytes_AS_STRING(PyUnicode_AsEncodedString(bitsInArg, "ascii", "strict"));
bitsOut = exposekey(bitsIn);
bitsOutArg = PyUnicode_FromString(bitsOut);
free(bitsOut);
return bitsOutArg;
}
static PyMethodDef moduleMethods[] = {
{"exposekey", (PyCFunction)PyExposekey, METH_VARARGS, NULL},
{NULL}
};
static struct PyModuleDef moduleDef = {
PyModuleDef_HEAD_INIT, MOD_NAME, NULL, -1, moduleMethods
};
PyMODINIT_FUNC PyInit_capi_demo(void) {
return PyModule_Create(&moduleDef);
}
Notes:
It requires swig_demo.h and swig_demo.c (not going to duplicate their contents here)
It only works with Python 3 (actually I got quite some headaches making it work, especially because I was used to PyString_AsString which is no longer present)
Error handling is poor
test_capi.py is similar to test_swig.py with one (obvious) difference: from swig_demo import exposekey should be replaced by from capi_demo import exposekey
The output is also the same to test_swig.py (again, not going to duplicate it here)
I have implemented the new python buffer interface in C++ outlined here:
https://docs.python.org/2/c-api/buffer.html
I have implemented my Py_buffer struct and filled it in:
template<typename T>
static int getbuffer(PyObject *obj, Py_buffer *view, int flags)
{
flags;
static const Py_ssize_t suboffsets[1] = { 0};
view->buf = (void*)(_Cast<T>(obj)->getbuffer());
view->obj = NULL;
view->len = _Cast<T>(obj)->getbuffersize();
view->itemsize = 1;
view->readonly = _Cast<T>(obj)->getreadonly();
view->ndim = 0;
view->format = NULL;
view->shape = NULL;
view->strides = NULL;
view->suboffsets = NULL;
view->internal = NULL;
return 0;
}
I am creating my Python buffer class in Python and handing it to C++. I am getting a pyObject along with my Py_Buffer. So now my question is, how am I supposed to write and resize this pyBuffer in C++? I can get access to the pointer directly and a size. But if its a newly created buffer how do I tell it how much space I need? There does not seem to be any sort of resize function for me to call.
I can use: int result = PyBuffer_FromContiguous(&m_view, const_cast<void*>(data), pySize, 'A');
to add data to my buffer. But my buffer must already have the correct size or it wont write. I do not think this is the correct way to be using it anyway.
Cython is not an option.
You shouldn't resize the Py_buffer directly, since it is just an interface to the data of a PyObject.
Instead, use PyByteArray_Resize() (or possibly _PyString_Resize()) on the underlying PyObject.
Explanation is defined below:
I have defined a new Python type named "Ex1".
typedef struct {
PyObject_HEAD
PyObject * int_id;
int * value;
} Ex1;
With this type in mind and all appropriate methods generated and validated in Python interpreted (it works pretty well). I want to be able to create a python object of the new Ex1 Type from C++ backend. A typical structure of what I need is:
int main
{
// Create Ex1 Object.
Ex1 Example;
// Call PythonC-API method to include Ex1 Object into the python interpreter.
// ¿Any function-method from Python API to perform this task?
}
Actually I managed to solve this problem using python docs:
https://docs.python.org/2/extending/newtypes.html
First of all it is necessary to define the appropiate methos as is described in the python docs (1) . Assuming the PyType created has two attributes (varp1 and varp2):
PyObject * create_Ex1(vartype1 var1, vartype2 var2)
{
PyObject * pInst = PyObject_CallObject((PyObject *)&Ex1Type, NULL);
((Ex1*)pInst)->varp1 = var1;
((Ex1*)pInst)->varp2 = var2;
return pInst;
}
and
static int Ex1_init(Ex1 *self, PyObject *args, PyObject *kwds)
{
// On init return -1 if error
self->varp1= NULL;
self->varp2= NULL;
return 0;
};
This is defined on the static PyTypeObject Ex1Type as is described in the python docs (1).
Then the object is created and initialized using this line:
PyObject* Ex1_obj = create_Ex1(var1, var2);
I have a struct in a dll that only contains function pointers (ie a vtable) that I would like to interact with in python (for test purposes). I am having a bit of trouble working out how to do this using ctypes.
What I have is:
struct ITest
{
virtual char const *__cdecl GetName() = 0;
virtual void __cdecl SetName(char const *name) = 0;
};
/* Factory function to create 'real' Test object */
extern "C" __declspec(dllexport) struct ITest * CALLCONV make_Test(char const * name);
A 'real' Test object will fill in the struct as appropriate. This gets compiled into a DLL (test.dll). I'd like, in python, to be able to call the factory method to get back a pointer to my Test struct and then call the function pointers contained in the struct, but I just can't seem to get my head around how it would work using ctypes. Does anyone have any pointers / examples of doing something similar or should I be using something like SWIG or Boost?
Thanks for any help.
Something like this should be a good starting point (I don't have your DLL compiled to test)
from ctypes import Structure, CFUNCTYPE, POINTER, c_char_p, windll
class ITest(Structure):
_fields_ = [
('GetName', CFUNCTYPE(c_char_p)),
('SetName', CFUNCTYPE(None, c_char_p)
]
test = windll.LoadLibrary('test.dll')
test.make_Test.restype = POINTER(ITest)
After this, you'll need to call make_Test() to get the struct, and try calling the functions. Perhaps with code like this:
itest = test.make_Test().contents
itest.SetName('asdf')
print itest.GetName()
Provide the dll or test and give me your results and I can help more if you still have problems.
The ctypes documentation says that you can create a ctypes.PYFUNCTYPE from an address.
If you get the address of the functions in your structure then you can wrap it as a Python function thanks to ctypes.PYFUNCTYPE and then call it as a regular ctype function.
I didn't test it myself but I think it maybe something to explore in your case
See http://docs.python.org/library/ctypes.html#ctypes.PYFUNCTYPE
I hope it helps