We Keep Coding

Is it possible to assign an C++ pointer to an attribute of a Python object, and retrieve it as a C++ pointer in another C++ function?

I'm making a C++ extension for Python, and I'm trying to do something like:
// this function assigns a C++ pointer to as attribute of a python object
void function1(PyObject* p){
// equivalent of p.attr = cpp_attr;
MyClass* cpp_attr = new MyClass();
PyObject* args = PyTuple_Pack(cpp_attr);
PyObject_SetAttrString(p, (char*)"attr", args);
}
I would like to retrieve this pointer and set it as attribute of another C++ object. I know how to get the PyObject* but after that I'm not sure what to do anymore
MySecondClass::MySecondClass(PyObject* p){
// get the attribute from p; equivalent of cpp_attr = p.attr
PyObject* cpp_attr = PyObject_getAttrString(p, (char*)"attr"));
// somehow get back the pointer to MyClass object created in function1
}
I looked at the documentation but I couldn't find anything that returns the original type. Is there anyway to do this?
Thanks

It's difficult to be absolutely certain, but I doubt that MyClass a Python object. This means that your attempt to store it as a Python object (e.g. using PyTuple_Pack) is completely wrong and will cause Python to malfunction in unexpected ways.
What will happen is that Python will attempt to interpret the pointer as a Python object, will try to use its normal reference counting mechanisms on that object (will change it in unpredictable ways), and ultimately try to deallocate that object (using Python mechanisms, not delete...) if some part of the object happens to equal 0.
There's a number of options, all basically centred around creating a wrapper object - a Python object defined in C++ that holds either a pointer or value of your C++ object.
Do it manually using the Python C API - This answer gives a very thorough example.
Look up the PyCapsule interface to create a quick wrapper around your object. You'd create your capsule with:
PyObject* cap = PyCapsule_New(cpp_attr, "MyClass",
[](PyObject* c) {
auto deleteme = reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass));
delete deleteme;
});
And you retrieve your C++ class from the capsule with:
reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass))
Use some tool like PyBind11, Cython, SWIG, etc to create the wrapper object for you.
Note also that PyObject_SetAttrString does not require the third argument to be a tuple (unless you specifically want to store a tuple...). You're likely getting it confused with PyObject_Call, where the args are passed as a tuple.

Assuming your call to PyTuple_Pack is correct, then you've created a PyTupleObject which has a structure:
typedef struct {
PyObject_VAR_HEAD
PyObject *ob_item[1];
} PyTupleObject;
The PyTupleObject inherits from the generic PyObject struct which has the following members:
struct _object *_ob_next;
struct _object *_ob_prev;
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
You can access the latter two with the macrosPy_REFCNT and Py_TYPE
The ob_item[1] member should be a pointer to the memory initially allocated. Based on how Macros are written in the documentation, you should be able to access it by
((PyTupleObject *)cpp_attr)->ob_item
And if you know the data type of the C++ pointer, then you should be able to cast it back. Maybe you can try
MyClass* cpp_att_again = reinterpret_cast<MyClass*>((PyTupleObject *)cpp_attr)->ob_item
Hopefully this points you in the right direction. You might be able to glean more insight from a similar question.

Writing a pyo3 function equivalent to a Python function that returns its input object

I am looking to write a Rust backend for my library, and I need to implement the equivalent of the following function in pyo3:
def f(x):
return x
This should return the same object as the input, and the function getting the return value should hold a new reference to the input. If I were writing this in the C API I would write it as:
PyObject * f(PyObject * x) {
Py_XINCREF(x);
return x;
}
In PyO3, I find it quite confusing to navigate the differences between PyObject, PyObjectRef, &PyObject, Py<PyObject>, Py<&PyObject>.
The most naive version of this function is:
extern crate pyo3;
use pyo3::prelude::*;
#[pyfunction]
pub fn f(_py: Python, x: &PyObject) -> PyResult<&PyObject> {
Ok(x)
}
Among other things, the lifetimes of x and the return value are not the same, plus I see no opportunity for pyo3 to increase the reference count for x, and in fact the compiler seems to agree with me:
error[E0106]: missing lifetime specifier
--> src/lib.rs:4:49
|
4 | pub fn f(_py: Python, x: &PyObject) -> PyResult<&PyObject> {
| ^ expected lifetime parameter
|
= help: this function's return type contains a borrowed value, but the signature does not say whether it is borrowed from `_py` or `x`
There may be a way for me to manually increase the reference count using the _py parameter and use lifetime annotations to make the compiler happy, but my impression is that pyo3 intends to manage reference counts itself using object lifetimes.
What is the proper way to write this function? Should I be attempting to wrap it in a Py container?

A PyObject is a simple wrapper around a raw pointer:
pub struct PyObject(*mut ffi::PyObject);
It has multiple creation functions, each corresponding to different kinds of pointers that we might get from Python. Some of these, such as from_borrowed_ptr, call Py_INCREF on the passed-in pointer.
Thus, it seems like we can accept a PyObject, so long as it was created in the "right" manner.
If we expand this code:
#[pyfunction]
pub fn example(_py: Python, x: PyObject) -> PyObject {
x
}
We can see this section of code that calls our function:
let mut _iter = _output.iter();
::pyo3::ObjectProtocol::extract(_iter.next().unwrap().unwrap()).and_then(
|arg1| {
::pyo3::ReturnTypeIntoPyResult::return_type_into_py_result(example(
_py, arg1,
))
},
)
Our argument is created by a call to ObjectProtocol::extract, which in turn calls FromPyObject::extract. This is implemented for PyObject by calling from_borrowed_ptr.
Thus, using a bare PyObject as the argument type will correctly increment the reference count.
Likewise, when a PyObject is dropped in Rust, it will automatically decrease the reference count. When it is returned back to Python, ownership is transferred and it is up to the Python code to update the reference count appropriately.
All investigation done for commit ed273982 from the master branch, corresponding to v0.5.0-alpha.1.

According to the other answer, pyo3 takes care of building additional boilerplate around our functions in order to keep track of Python reference counting. In particular, the counter is already incremented when passing the object as an argument to the function. Nevertheless, the clone_ref method can be used to explicitly create a new reference to the same object, which will also increment its reference counter.
The output of the function must still be an actual Python object rather than a reference to it (which seems reasonable, as Python does not understand Rust references; pyo3 seems to ignore lifetime parameters in these functions).
#[pyfunction]
fn f(py: Python, x: PyObject) -> PyResult<PyObject> {
Ok(x.clone_ref(py))
}
From playing around with the function in Python land (AKA not a serious testbed), it at least seems to work as intended.
from dummypy import f
def get_object():
return f("OK")
a = [1, 2, 3]
if True:
b = f(a)
assert b is a
b[0] = 9001
print(a)
x = get_object()
print(x)

DLL conversion with ctypes on byref vector argument in python

My machine is Win7 with anaconda.
i am recently converting C++ dll functions to a python project.
There are numbers of difficulty I had overcame yet I have no idea how to deal with the following conversion:
typedef int (__stdcall *p_API_GetOrder)(vector<ApiOrder>& apiOrderList);
where,
class ApiOrder(Structure):
_fields_ = [
('Timestamp', c_long),
('Item', c_char * 16),
('Qty', c_long),
]
In python, I tried,
mydll.API_GetOrder(POINTER(ApiOrder()))
The error is:
TypeError: must be a ctypes type
i am not an export in C++ or programming. So not quite sure what byref is. It would be nice if someone can clear my concept.

POINTER(…) constructs a new pointer type, not a value of that type. So, when you do this:
mydll.API_GetOrder(POINTER(ApiOrder()))
… you’re passing a Python type object, not a ctypes wrapper around a C pointer object.
To get a pointer to a ctypes wrapper object, you want to call either pointer or byref. The former constructs a POINTER(…) instance, sets it to point to your object, and passes the wrapped pointer; the latter just directly passes a pointer to your object without constructing a pointer wrapper object, and usually that’s all you need. See Passing pointers in the docs for further details.
However, I don’t think this is going to do much good, for two reasons.
First, most functions that take a pointer to some struct and return an int are doing it so they can fill in that struct with useful values. Constructing a new empty struct and passing a pointer to it and not holding onto a reference to it means you have no way to look at whatever values got filled in.
Also, you probably want to check the return value.
In general, you need to do something like this:
order = ApiOrder()
ret = mydll.API_GetOrder(byref(order))
if ret:
do some error handling with either ret or errno
else:
so something with order
While we’re at it, you almost certainly want to set the argtypes and restype of the function, so ctypes knows how to convert things properly, and can give you an exception if you do something that makes no sense, instead of making it guess how to convert and pass things and segfault if it guesses wrong.
Also, for the case of functions that return a success-or-error int, it's usually better to assign a function to the restype, which looks up the error and raises an appropriate exception. (Use an errcheck if you need anything more flexible than just checking that an int return is nonzero or a pointer return is zero.)
But even this isn’t going to help here, because the function you’re trying to call doesn’t take a pointer to an ApiOrder in the first place, it takes a reference to a std::vector of them. So you need to call into the C++ stdlib to construct an object of that type, then you can byref that as the argument.
But usually, it’s easier to write some C++ code that provides a C API to the library, then use ctypes to call that C API, instead of trying to build and use C++ objects from Python.
Your C++ code would look something like this:
int call_getorder(p_API_GetOrder func, ApiOrder *apiOrderArray, size_t apiOrderCount) {
std::vector<ApiOrder> vec(apiOrderArray, apiOrderCount);
ret = func(vec);
if (ret) return ret;
std::copy(std::begin(vec), std::end(vec), apiOrderArray);
return 0;
}
Now, you can call this from Python by creating an array of 1 ApiOrder (or creating a POINTER to an ApiOrder and passing it directly, if you prefer):
orders = (ApiOrder*1)()
ret = mywrapperdll.call_order(mydll.API_GetOrder, byref(order), 1)
if ret:
do some error handling with either ret or errno
else:
do something with order[0]
Of course you're still going to want argtypes and restype.

Cython: std::function callbacks with custom parameter types

Please read this post before answering: Pass a closure from Cython to C++
In the accepted answers, it is neatly shown how a python function is converted into a std::function using Boost Python.
Following this example I'm able to wrap functions taking an std::function as an argument and call them using a python function as input. However, this only works when the std::function parameters are primitives like int, double, string etc.
Any guidance on how to make this work for custom types as well will be highly appreciated.

This won't be a complete answer - it assumes you can fill in the gaps from my previous answer that the question was based on. Unfortunately it is a little bit more complicated than that case.
Just to define the problem - assume you have a parameter of a custom C++ class, like:
class cpp_class {
// some non-trivial contents
};
and thus your C++ interface looks like this:
void call_some_std_func(std::function<void(cpp_class&)> callback) {
callback(5,std::string("hello"));
}
The first thing to do is to write a Cython wrapper for your C++ class (in principle you could make a Boost Python wrapper instead). Here you need to make a choice of about "ownership" of the C++ object. The first choice is to make a copy:
cdef extern from "cpp_file.hpp":
cppclass cpp_class:
pass # details
cdef class CyWrapper:
cdef cpp_class* ptr
def __dealloc__(self):
del self.ptr
# other details following standard wrapper pattern
cdef public make_CyWrapper(cpp_class& x):
obj = CyWrapper()
obj.ptr = new cpp_class(x)
return obj
I've created a wrapper class with a destructor that handles the memory and a publicly accessible constructor function that can be called from external code. This version is safe because your wrapper owns the object it holds and so there can be no writes to invalid memory. However, because it makes a copy, you can't make changes to the original C++ object.
A second option is to hold a pointer to an object you don't own. The code is basically identical except you remove the __dealloc__ and avoid making a copy in make_CyWrapper:
obj.ptr = &x // instead of new cpp_class(x)
This is unsafe - you need to ensure your C++ object outlives the Cython wrapper - but allows you to modify the object.
You could also imagine a few other options: you could take ownership of an existing object with your Cython wrapper (Such a scheme would have to pass by pointer rather than reference, or it could use move constructors); you could deconstruct your C++ class into a representation expressed in basic types and pass those to Python; you could use shared pointers to split the ownership; or you have a more elaborate way of marking your Cython wrapper as "invalid" once your held C++ instance is destructed.
What you do next depends on whether you're using Boost Python (for it's convenient, callable wrapping of Python objects) or if you're making your own version. (I showed both possibilities in the previous answer).
Assuming Boost Python, you need to do two things - tell it about the conversion and make sure that it imports the module that your wrapper is defined in (if you don't do this you get exciting segmentation faults)
struct convert_to_PyWrapper {
static PyObject* convert(const cpp_class& rhs) {
// the const_cast here is a bit dodgy, but was needed to make it work
return make_CyWrapper(const_cast<cpp_class&>(rhs));
}
};
inline void setup_boost_python() {
PyInit_your_module_name(); // named inityour_module_name in Python 2
boost::python::to_python_converter<
cpp_class,
convert_to_PyWrapper>();
}
You need to make sure that your Python/Cython code calls "setup_boost_python" before attempting to use the callback (if you put it at module level it's done on import, which is ideal).
If you're following my "manual" scheme (avoiding the dependency on Boost Python) then you need to modify the call_obj Cython function that does the C++ to Cython type conversion.
cdef public void call_obj(obj, cpp_class& c):
obj(make_CyWrapper(c))
You also need to ensure the wrapper Cython module is imported before use (otherwise you get segmentation faults). I did this in "py_object_wrapper.hpp" but providing it's done once somewhere you can place it where you like.
void operator()(cpp_class& a) {
PyInit_your_module_name();
if (held) {
call_obj(held,a);
}
}

Python reference count and ctypes

Hallo,
I have some troubles understanding the python reference count.
What I want to do is return a tuple from c++ to python using the ctypes module.
C++:
PyObject* foo(...)
{
...
return Py_BuildValue("(s, s)", value1, value2);
}
Python:
pointer = c_foo(...) # c_foo loaded with ctypes
obj = cast(pointer, py_object).value
I'm was not sure about the ref count of obj, so I tried sys.getrefcount()
and got 3. I think it should be 2 (the getrefcount functions makes one ref itself).
Now I can't make Py_DECREF() before the return in C++ because the object gets deleted. Can I decrease the ref count in python?
edit
What happens to the ref count when the cast function is called? I'm not really sure from the documentation below. http://docs.python.org/library/ctypes.html#ctypes.cast
ctypes.cast(obj, type)
This function is similar to the cast operator in C. It returns a new instance of type which points to the same memory block as obj. type must be a pointer type, and obj must be an object that can be interpreted as a pointer.

On further research I found out that one can specify the return type of the function.
http://docs.python.org/library/ctypes.html#callback-functions
This makes the cast obsolete and the ref count is no longer a problem.
clib = ctypes.cdll.LoadLibrary('some.so')
c_foo = clib.c_foo
c_foo.restype = ctypes.py_object
As no additional answers were given I accept my new solution as the answer.

Your c++ code seems to be a classic wrapper using the official C-API and it's a bit weird since ctypes is usually used for using classic c types in python (like int, float, etc...).
I use personnally the C-API "alone" (without ctypes) but on my personnal experience, you don't have to worry about the reference counter in this case since you are returning a native python type with Py_BuildValue. When a function returns an object, the ownership of the returned object is given to the calling function.
You have to worry about Py_XINCREF/Py_XDECREF (better than Py_INCREF/Py_DECREF because it accepts NULL pointers) only when you want to change ownership of the object :
For example, you have created a wrapper of a map in python (let's call the typed object py_map). The element are of c++ class Foo and you have created an other python wrapper for them (let's call it py_Foo). If you create a function that wrap the [] operator, you are going to return a py_Foo object in python :
F = py_Map["key"]
but since the ownership is given to the calling function, you will call the destructor when you delete F and the map in c++ contains a pointer to a deallocated objet !
The solution is to write in c++ in the wrapper of [] :
...
PyObject* result; // My py_Foo object
Py_XINCREF(result); // transfer the ownership
return result;
}
You should take a look at the notion of borrowed and owned reference in python. This is essential to understand properly the reference counter.