I am using the PyObject functionality to call c functions, and
return Py_BuildValue("theTypeToConvert", myCVariable);
to return things back to my python program, this all works fine.
However I have a custom C type
extern HANDLE pascal
how do I pass an instance of this back to python so I can give it to other c functions later, the closest I could think of was to use
Py_BuildValue("O&", etc)
but this apparently mangles the variable as I am not getting the correct results later on.
If I understand correctly that you want the object to be "opaque" from the Python perspective, i.e. just a pointer value that you can pass around in Python but not operate on the object it points to, then you might be after the Capsule object.
Official Python docs on capsules:
https://docs.python.org/2/c-api/capsule.html#capsules
See also:
Passing a C pointer around with the Python/C API
Related
I am new to programming, I defined a function in c++ with tuple method for two returned variables, after I compiled the files, In the python file, I try to access the two returned variables inside the dynamic library which have been compiled, but it is not working, there is segmentation error happened when I tried to run the python program. But I actually success with single return variable from c++, I think there is might be special trick for accessing the two returned variable with tuple method from python.
The following is the c++ code with two returned variables with tuple method
std::tuple<double, double> Cassie2d::Step(ControllerTorque* action)
{
dyn_model_.setState(mj_data_->qpos, mj_data_->qvel);
dyn_state_.UpdateDynamicState(&dyn_model_);
mju_copy(mj_data_->ctrl, action->torques, nU);
mj_step(mj_model_, mj_data_);
return std::make_tuple(mj_data_->qacc,mj_data_->time);
Render();
}
The following is the python method I was applied, due to both of the returned variables are double type.
lib.StepTorque.argtypes = [ctypes.c_void_p, ctypes.POINTER(ControllerTorque)]
lib.StepTorque.restype = ctypes.c_double
I guess the restype is not just equal to ctypes.c_double, because it worked for one returned variable and it might not work for two returned variables.
Really appreciate for the help!
With return std::make_tuple(mj_data_->qacc,mj_data_->time) your creating a structure which holds two variables, defined in parenthesis. If the function would be call from native C++ code you should be able to unpack it using std:tie, eq:
double a, b;
std::tie(a,b) = Step(arg);
However you need to call it directly from python, which has totally different syntax for returning multiple variables (C++ pair or tuple is kind of bypass missing functionality). I see two option in python.
Give your C++ function a two callbacks to python, and send variables separately this way:
void Cassie2d::Step(ControllerTorque* action, callback1, callback2)
{
dyn_model_.setState(mj_data_->qpos, mj_data_->qvel);
dyn_state_.UpdateDynamicState(&dyn_model_);
mju_copy(mj_data_->ctrl, action->torques, nU);
mj_step(mj_model_, mj_data_);
callback1(mj_data_->qacc);
callback2(mj_data_->time);
Render();
}
(Recommened) You can try to use native python option to serve multiple return. Left you C++ function in its initial form and call in python this way:
VarA, VarB = Step(action)
I assume you have a reference to Cassie2d::Step(ControllerTorque* action) method in your python code, however you provided very small snippet.
I am creating Python bindings for a C library.
In C the code to use the functions would look like this:
Ihandle *foo;
foo = MethFunc();
SetArribute(foo, 's');
I am trying to get this into Python. Where I have MethFunc() and SetAttribute() functions that could be used in my Python code:
import mymodule
foo = mymodule.MethFunc()
mymodule.SetAttribute(foo)
So far my C code to return the function looks like this:
static PyObject * _MethFunc(PyObject *self, PyObject *args) {
return Py_BuildValue("O", MethFunc());
}
But that fails by crashing (no errors)
I have also tried return MethFunc(); but that failed.
How can I return the function foo (or if what I am trying to achieve is completely wrong, how should I go about passing MethFunc() to SetAttribute())?
The problem here is that MethFunc() returns an IHandle *, but you're telling Python to treat it as a PyObject *. Presumably those are completely unrelated types.
A PyObject * (or any struct you or Python defines that starts with an appropriate HEAD macro) begins with pointers to a refcount and a type, and the first thing Python is going to do with any object you hand it is deal with those pointers. So, if you give it an object that instead starts with, say, two ints, Python is going to end up trying to access a type at 0x00020001 or similar, which is almost certain to segfault.
If you need to pass around a pointer to some C object, you have to wrap it up in a Python object. There are three ways to do this, from hackiest to most solid.
First, you can just cast the IHandle * to a size_t, then PyLong_FromSize_t it.
This is dead simple to implement. But it means these objects are going to look exactly like numbers from the Python side, because that's all they are.
Obviously you can't attach a method to this number; instead, your API has to be a free function that takes a number, then casts that number back to an IHandle* and calls a method.
It's more like, e.g., C's stdio, where you have to keep passing stdin or f as an argument to fread, instead of Python's io, where you call methods on sys.stdin or f.
But even worse, because there's no type checking, static or dynamic, to protect you from some Python code accidentally passing you the number 42. Which you'll then cast to an IHandle * and try to dereference, leading to a segfault…
And if you were hoping Python's garbage collector would help you know when the object is still referenced, you're out of luck. You need to make your users manually keep track of the number and call some CloseHandle function when they're done with it.
Really, this isn't that much better than accessing your code from ctypes, so hopefully that inspires you to keep reading.
A better solution is to cast the IHandle * to a void *, then PyCapsule_New it.
If you haven't read about capsules, you need to at least skim the main chapter. But the basic idea is that it wraps up a void* as a Python object.
So, it's almost as simple as passing around numbers, but solves most of the problems. Capsules are opaque values which your Python users can't accidentally do arithmetic on; they can't send you 42 in place of a capsule; you can attach a function that gets called when the last reference to a capsule goes away; you can even give it a nice name to show up in the repr.
But you still can't attach any behavior to capsules.
So, your API will still have to be a MethSetAttribute(mymodule, foo) instead of mymeth.SetAttribute(foo) if mymodule is a capsule, just as if it's an int. (Except now it's type-safe.)
Finally, you can build a new Python extension type for a struct that contains an IHandle *.
This is a lot more work. And if you haven't read the tutorial on Defining Extension Types, you need to go thoroughly read through that whole chapter.
But it means that you have an actual Python type, with everything that goes with it.
You can give it a SetAttribute method, and Python code can just call that method. You can give it whatever __str__ and __repr__ you want. You can give it a __doc__. Python code can do isinstance(mymodule, MyMeth). And so on.
If you're willing to use C++, or D, or Rust instead of C, there are some great libraries (PyCxx, boost::python, Pyd, rust-python, etc.) that can do most of the boilerplate for you. You just declare that you want a Python class and how you want its attributes and methods bound to your C attributes and methods and you get something you can use like a C++ class, except that it's actually a PyObject * under the covers. (And it'll even takes care of all the refcounting cruft for you via RAII, which will save you endless weekends debugging segfaults and memory leaks…)
Or you can use Cython, which lets you write C extension modules in a language that's basically Python, but extended to interface with C code. So your wrapper class is just a class, but with a special private cdef attribute that holds the IHandle *, and your SetAttribute(self, s) can just call the C SetAttribute function with that private attribute.
Or, as suggested by user, you can also use SWIG to generate the C bindings for you. For simple cases, it's pretty trivial—just feed it your C API, and it gives you back the code to build your Python .so. For less simple cases, I personally find it a lot more painful than something like PyCxx, but it definitely has a lower learning curve if you don't already know C++.
I am using Python C API 2.7.2 with my C++ console application. There is one doubt regarding Python C API Boolean Objects
I am using:
PyObject* myVariable = Py_True;
Do I need to deference myVariable with Py_DECREF(myVariable)?
The Python C API documentation says:-
The Python True object. This object has no methods. It needs to be
treated just like any other object with respect to reference counts.
I searched the questions but could not find a clear answer for it.
Thanks.
Although it isn't dynamically created, it must be reference counted because PyObject variables can hold ANY Python object. Otherwise there would need to be checks for Py_True and other special cases scattered throughout the Python runtime as well as any C/C++ code that uses the API. That would be messy and error prone.
It needs to be treated just like any other object with respect to reference counts.
This means that you must incref it when you take its reference
{
Py_INCREF(Py_True);
PyObject* myVariable = Py_True;
and you must decref it when you dispose of it.
Py_DECREF(myVariable);
}
I started on my first Python extension today and was only creating a very small wrapper around a C library as an exercise. As is typical with C libraries, you start of with an initialization function that yields a handler. You can pass that handler to functions and later you pass it to the cleanup function that frees memory.
When I started writing the wrapper I basically wanted to have a way to call each native C function from python. Quickly I hit the problem that I need to return an arbitrary pointer from C to Python only to give it from there to C again in another function. I doesn't matter how it looks as I don't use it in Python, I just store it and pass it around.
So how do you pass around a void pointer between Python and C?
Please note: I know it is not recommended to write such small wrappers using the extension system but rather ctypes and friends. This is just for practice right now.
PyLong_FromVoidPtr() and PyLong_AsVoidPtr() can be abused to inject malicious data into your program. I recommend against them.
Python has PyCapsule for exactly that job. Capsules provide a safe way to exchange void ptr between modules or Python space and C space. The capsules are type-safe, too. If you need some example, the socket / ssl modules and pyexpat / _elementtree modules use capsules to exchange CAPI structs.
http://docs.python.org/3/c-api/capsule.html
After some searching I found the functions PyLong_AsVoidPtr and PyLong_FromVoidPtr. This yields a nice way to convert between a void * and a PyObject:
# in init function
return PyLong_FromVoidPtr(handle);
# in function using handle
handle = PyLong_AsVoidPtr(python_handle);
The one problem now might be how to retrieve python_handle from the typical *args given to a function:
PyObject *python_handle;
PyArg_ParseTuple(args, "O", &python_handle);
Careful here: The argument given for the "O" object must be a pointer to a PyObject pointer: PyObject **. The "O" itself only denotes to pass this PyObject through without any handling and converting. And with this, you can pass around any pointers any way you like.
Note: I think this solution is not really pretty, because you now have to variables, one that is only needed for a short time.
Hallo,
I have some troubles understanding the python reference count.
What I want to do is return a tuple from c++ to python using the ctypes module.
C++:
PyObject* foo(...)
{
...
return Py_BuildValue("(s, s)", value1, value2);
}
Python:
pointer = c_foo(...) # c_foo loaded with ctypes
obj = cast(pointer, py_object).value
I'm was not sure about the ref count of obj, so I tried sys.getrefcount()
and got 3. I think it should be 2 (the getrefcount functions makes one ref itself).
Now I can't make Py_DECREF() before the return in C++ because the object gets deleted. Can I decrease the ref count in python?
edit
What happens to the ref count when the cast function is called? I'm not really sure from the documentation below. http://docs.python.org/library/ctypes.html#ctypes.cast
ctypes.cast(obj, type)
This function is similar to the cast operator in C. It returns a new instance of type which points to the same memory block as obj. type must be a pointer type, and obj must be an object that can be interpreted as a pointer.
On further research I found out that one can specify the return type of the function.
http://docs.python.org/library/ctypes.html#callback-functions
This makes the cast obsolete and the ref count is no longer a problem.
clib = ctypes.cdll.LoadLibrary('some.so')
c_foo = clib.c_foo
c_foo.restype = ctypes.py_object
As no additional answers were given I accept my new solution as the answer.
Your c++ code seems to be a classic wrapper using the official C-API and it's a bit weird since ctypes is usually used for using classic c types in python (like int, float, etc...).
I use personnally the C-API "alone" (without ctypes) but on my personnal experience, you don't have to worry about the reference counter in this case since you are returning a native python type with Py_BuildValue. When a function returns an object, the ownership of the returned object is given to the calling function.
You have to worry about Py_XINCREF/Py_XDECREF (better than Py_INCREF/Py_DECREF because it accepts NULL pointers) only when you want to change ownership of the object :
For example, you have created a wrapper of a map in python (let's call the typed object py_map). The element are of c++ class Foo and you have created an other python wrapper for them (let's call it py_Foo). If you create a function that wrap the [] operator, you are going to return a py_Foo object in python :
F = py_Map["key"]
but since the ownership is given to the calling function, you will call the destructor when you delete F and the map in c++ contains a pointer to a deallocated objet !
The solution is to write in c++ in the wrapper of [] :
...
PyObject* result; // My py_Foo object
Py_XINCREF(result); // transfer the ownership
return result;
}
You should take a look at the notion of borrowed and owned reference in python. This is essential to understand properly the reference counter.