Hallo,
I have some troubles understanding the python reference count.
What I want to do is return a tuple from c++ to python using the ctypes module.
C++:
PyObject* foo(...)
{
...
return Py_BuildValue("(s, s)", value1, value2);
}
Python:
pointer = c_foo(...) # c_foo loaded with ctypes
obj = cast(pointer, py_object).value
I'm was not sure about the ref count of obj, so I tried sys.getrefcount()
and got 3. I think it should be 2 (the getrefcount functions makes one ref itself).
Now I can't make Py_DECREF() before the return in C++ because the object gets deleted. Can I decrease the ref count in python?
edit
What happens to the ref count when the cast function is called? I'm not really sure from the documentation below. http://docs.python.org/library/ctypes.html#ctypes.cast
ctypes.cast(obj, type)
This function is similar to the cast operator in C. It returns a new instance of type which points to the same memory block as obj. type must be a pointer type, and obj must be an object that can be interpreted as a pointer.
On further research I found out that one can specify the return type of the function.
http://docs.python.org/library/ctypes.html#callback-functions
This makes the cast obsolete and the ref count is no longer a problem.
clib = ctypes.cdll.LoadLibrary('some.so')
c_foo = clib.c_foo
c_foo.restype = ctypes.py_object
As no additional answers were given I accept my new solution as the answer.
Your c++ code seems to be a classic wrapper using the official C-API and it's a bit weird since ctypes is usually used for using classic c types in python (like int, float, etc...).
I use personnally the C-API "alone" (without ctypes) but on my personnal experience, you don't have to worry about the reference counter in this case since you are returning a native python type with Py_BuildValue. When a function returns an object, the ownership of the returned object is given to the calling function.
You have to worry about Py_XINCREF/Py_XDECREF (better than Py_INCREF/Py_DECREF because it accepts NULL pointers) only when you want to change ownership of the object :
For example, you have created a wrapper of a map in python (let's call the typed object py_map). The element are of c++ class Foo and you have created an other python wrapper for them (let's call it py_Foo). If you create a function that wrap the [] operator, you are going to return a py_Foo object in python :
F = py_Map["key"]
but since the ownership is given to the calling function, you will call the destructor when you delete F and the map in c++ contains a pointer to a deallocated objet !
The solution is to write in c++ in the wrapper of [] :
...
PyObject* result; // My py_Foo object
Py_XINCREF(result); // transfer the ownership
return result;
}
You should take a look at the notion of borrowed and owned reference in python. This is essential to understand properly the reference counter.
Related
I'm making a C++ extension for Python, and I'm trying to do something like:
// this function assigns a C++ pointer to as attribute of a python object
void function1(PyObject* p){
// equivalent of p.attr = cpp_attr;
MyClass* cpp_attr = new MyClass();
PyObject* args = PyTuple_Pack(cpp_attr);
PyObject_SetAttrString(p, (char*)"attr", args);
}
I would like to retrieve this pointer and set it as attribute of another C++ object. I know how to get the PyObject* but after that I'm not sure what to do anymore
MySecondClass::MySecondClass(PyObject* p){
// get the attribute from p; equivalent of cpp_attr = p.attr
PyObject* cpp_attr = PyObject_getAttrString(p, (char*)"attr"));
// somehow get back the pointer to MyClass object created in function1
}
I looked at the documentation but I couldn't find anything that returns the original type. Is there anyway to do this?
Thanks
It's difficult to be absolutely certain, but I doubt that MyClass a Python object. This means that your attempt to store it as a Python object (e.g. using PyTuple_Pack) is completely wrong and will cause Python to malfunction in unexpected ways.
What will happen is that Python will attempt to interpret the pointer as a Python object, will try to use its normal reference counting mechanisms on that object (will change it in unpredictable ways), and ultimately try to deallocate that object (using Python mechanisms, not delete...) if some part of the object happens to equal 0.
There's a number of options, all basically centred around creating a wrapper object - a Python object defined in C++ that holds either a pointer or value of your C++ object.
Do it manually using the Python C API - This answer gives a very thorough example.
Look up the PyCapsule interface to create a quick wrapper around your object. You'd create your capsule with:
PyObject* cap = PyCapsule_New(cpp_attr, "MyClass",
[](PyObject* c) {
auto deleteme = reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass));
delete deleteme;
});
And you retrieve your C++ class from the capsule with:
reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass))
Use some tool like PyBind11, Cython, SWIG, etc to create the wrapper object for you.
Note also that PyObject_SetAttrString does not require the third argument to be a tuple (unless you specifically want to store a tuple...). You're likely getting it confused with PyObject_Call, where the args are passed as a tuple.
Assuming your call to PyTuple_Pack is correct, then you've created a PyTupleObject which has a structure:
typedef struct {
PyObject_VAR_HEAD
PyObject *ob_item[1];
} PyTupleObject;
The PyTupleObject inherits from the generic PyObject struct which has the following members:
struct _object *_ob_next;
struct _object *_ob_prev;
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
You can access the latter two with the macrosPy_REFCNT and Py_TYPE
The ob_item[1] member should be a pointer to the memory initially allocated. Based on how Macros are written in the documentation, you should be able to access it by
((PyTupleObject *)cpp_attr)->ob_item
And if you know the data type of the C++ pointer, then you should be able to cast it back. Maybe you can try
MyClass* cpp_att_again = reinterpret_cast<MyClass*>((PyTupleObject *)cpp_attr)->ob_item
Hopefully this points you in the right direction. You might be able to glean more insight from a similar question.
I am looking to write a Rust backend for my library, and I need to implement the equivalent of the following function in pyo3:
def f(x):
return x
This should return the same object as the input, and the function getting the return value should hold a new reference to the input. If I were writing this in the C API I would write it as:
PyObject * f(PyObject * x) {
Py_XINCREF(x);
return x;
}
In PyO3, I find it quite confusing to navigate the differences between PyObject, PyObjectRef, &PyObject, Py<PyObject>, Py<&PyObject>.
The most naive version of this function is:
extern crate pyo3;
use pyo3::prelude::*;
#[pyfunction]
pub fn f(_py: Python, x: &PyObject) -> PyResult<&PyObject> {
Ok(x)
}
Among other things, the lifetimes of x and the return value are not the same, plus I see no opportunity for pyo3 to increase the reference count for x, and in fact the compiler seems to agree with me:
error[E0106]: missing lifetime specifier
--> src/lib.rs:4:49
|
4 | pub fn f(_py: Python, x: &PyObject) -> PyResult<&PyObject> {
| ^ expected lifetime parameter
|
= help: this function's return type contains a borrowed value, but the signature does not say whether it is borrowed from `_py` or `x`
There may be a way for me to manually increase the reference count using the _py parameter and use lifetime annotations to make the compiler happy, but my impression is that pyo3 intends to manage reference counts itself using object lifetimes.
What is the proper way to write this function? Should I be attempting to wrap it in a Py container?
A PyObject is a simple wrapper around a raw pointer:
pub struct PyObject(*mut ffi::PyObject);
It has multiple creation functions, each corresponding to different kinds of pointers that we might get from Python. Some of these, such as from_borrowed_ptr, call Py_INCREF on the passed-in pointer.
Thus, it seems like we can accept a PyObject, so long as it was created in the "right" manner.
If we expand this code:
#[pyfunction]
pub fn example(_py: Python, x: PyObject) -> PyObject {
x
}
We can see this section of code that calls our function:
let mut _iter = _output.iter();
::pyo3::ObjectProtocol::extract(_iter.next().unwrap().unwrap()).and_then(
|arg1| {
::pyo3::ReturnTypeIntoPyResult::return_type_into_py_result(example(
_py, arg1,
))
},
)
Our argument is created by a call to ObjectProtocol::extract, which in turn calls FromPyObject::extract. This is implemented for PyObject by calling from_borrowed_ptr.
Thus, using a bare PyObject as the argument type will correctly increment the reference count.
Likewise, when a PyObject is dropped in Rust, it will automatically decrease the reference count. When it is returned back to Python, ownership is transferred and it is up to the Python code to update the reference count appropriately.
All investigation done for commit ed273982 from the master branch, corresponding to v0.5.0-alpha.1.
According to the other answer, pyo3 takes care of building additional boilerplate around our functions in order to keep track of Python reference counting. In particular, the counter is already incremented when passing the object as an argument to the function. Nevertheless, the clone_ref method can be used to explicitly create a new reference to the same object, which will also increment its reference counter.
The output of the function must still be an actual Python object rather than a reference to it (which seems reasonable, as Python does not understand Rust references; pyo3 seems to ignore lifetime parameters in these functions).
#[pyfunction]
fn f(py: Python, x: PyObject) -> PyResult<PyObject> {
Ok(x.clone_ref(py))
}
From playing around with the function in Python land (AKA not a serious testbed), it at least seems to work as intended.
from dummypy import f
def get_object():
return f("OK")
a = [1, 2, 3]
if True:
b = f(a)
assert b is a
b[0] = 9001
print(a)
x = get_object()
print(x)
My machine is Win7 with anaconda.
i am recently converting C++ dll functions to a python project.
There are numbers of difficulty I had overcame yet I have no idea how to deal with the following conversion:
typedef int (__stdcall *p_API_GetOrder)(vector<ApiOrder>& apiOrderList);
where,
class ApiOrder(Structure):
_fields_ = [
('Timestamp', c_long),
('Item', c_char * 16),
('Qty', c_long),
]
In python, I tried,
mydll.API_GetOrder(POINTER(ApiOrder()))
The error is:
TypeError: must be a ctypes type
i am not an export in C++ or programming. So not quite sure what byref is. It would be nice if someone can clear my concept.
POINTER(…) constructs a new pointer type, not a value of that type. So, when you do this:
mydll.API_GetOrder(POINTER(ApiOrder()))
… you’re passing a Python type object, not a ctypes wrapper around a C pointer object.
To get a pointer to a ctypes wrapper object, you want to call either pointer or byref. The former constructs a POINTER(…) instance, sets it to point to your object, and passes the wrapped pointer; the latter just directly passes a pointer to your object without constructing a pointer wrapper object, and usually that’s all you need. See Passing pointers in the docs for further details.
However, I don’t think this is going to do much good, for two reasons.
First, most functions that take a pointer to some struct and return an int are doing it so they can fill in that struct with useful values. Constructing a new empty struct and passing a pointer to it and not holding onto a reference to it means you have no way to look at whatever values got filled in.
Also, you probably want to check the return value.
In general, you need to do something like this:
order = ApiOrder()
ret = mydll.API_GetOrder(byref(order))
if ret:
do some error handling with either ret or errno
else:
so something with order
While we’re at it, you almost certainly want to set the argtypes and restype of the function, so ctypes knows how to convert things properly, and can give you an exception if you do something that makes no sense, instead of making it guess how to convert and pass things and segfault if it guesses wrong.
Also, for the case of functions that return a success-or-error int, it's usually better to assign a function to the restype, which looks up the error and raises an appropriate exception. (Use an errcheck if you need anything more flexible than just checking that an int return is nonzero or a pointer return is zero.)
But even this isn’t going to help here, because the function you’re trying to call doesn’t take a pointer to an ApiOrder in the first place, it takes a reference to a std::vector of them. So you need to call into the C++ stdlib to construct an object of that type, then you can byref that as the argument.
But usually, it’s easier to write some C++ code that provides a C API to the library, then use ctypes to call that C API, instead of trying to build and use C++ objects from Python.
Your C++ code would look something like this:
int call_getorder(p_API_GetOrder func, ApiOrder *apiOrderArray, size_t apiOrderCount) {
std::vector<ApiOrder> vec(apiOrderArray, apiOrderCount);
ret = func(vec);
if (ret) return ret;
std::copy(std::begin(vec), std::end(vec), apiOrderArray);
return 0;
}
Now, you can call this from Python by creating an array of 1 ApiOrder (or creating a POINTER to an ApiOrder and passing it directly, if you prefer):
orders = (ApiOrder*1)()
ret = mywrapperdll.call_order(mydll.API_GetOrder, byref(order), 1)
if ret:
do some error handling with either ret or errno
else:
do something with order[0]
Of course you're still going to want argtypes and restype.
I am using Python C API 2.7.2 with my C++ console application. There is one doubt regarding Python C API Boolean Objects
I am using:
PyObject* myVariable = Py_True;
Do I need to deference myVariable with Py_DECREF(myVariable)?
The Python C API documentation says:-
The Python True object. This object has no methods. It needs to be
treated just like any other object with respect to reference counts.
I searched the questions but could not find a clear answer for it.
Thanks.
Although it isn't dynamically created, it must be reference counted because PyObject variables can hold ANY Python object. Otherwise there would need to be checks for Py_True and other special cases scattered throughout the Python runtime as well as any C/C++ code that uses the API. That would be messy and error prone.
It needs to be treated just like any other object with respect to reference counts.
This means that you must incref it when you take its reference
{
Py_INCREF(Py_True);
PyObject* myVariable = Py_True;
and you must decref it when you dispose of it.
Py_DECREF(myVariable);
}
I have a function that takes an int-pointer and exposed it via boost::python. How can I call this function from python?
in C++ with boost::python:
void foo(int* i);
...
def("foo", foo);
in python:
import foo_ext
i = 12
foo_ext.foo(i)
results in
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
Boost.Python.ArgumentError: Python argument types in
foo(int)
did not match C++ signature:
foo(int* i)
So how to pass a pointer?
Short answer is: You can't. Python does not have pointers
Long answer is: There are assorted workarounds depending on use-case.
I notice that you are using an int and an int* in your example. Int (along with float, str, and bool) is a special case because it is immutable in python.
Lets say that the object that you are passing in is not really an int.
Have a wrapper function that takes the argument as a reference, takes the address and passes it on to the actual function. This will work seamlessly in python.
Ok, so say it really was an int. Now you have a problem. You can not change the int you passed in. If you try the same solution, boost::python will complain about l-values at runtime. There are still several options.
Let's say that you do not need to see what the int looks like after the function exits and you know that the function will not squirrel away the pointer to dereference after the function returns:
Your wrapper should now take the int by value or by const reference. Everything else is the same.
Maybe you ONLY need to see the after state (the int is an OUT perimeter):
Your wrapper function will now take no arguments, and will pass the address of a local int to the actual function. It will return that value. If you function already has a return value it should now return a tuple.
Both the input and the output are important and you know that the function will not squirrel away the pointer to dereference after the function returns:
Combine the two above. The wrapper takes one int by value and returns a different int.
The function expects to squirrel away the pointer to dereference after the function returns:
There is no real good solution. You can create and expose an object in c++ that contains a c++ int. The wrapper will take that object by reference, extract the address of the contained int and pass it on to the actual function. Keeping the object alive in python (and safe from the garbage collector) until the library is done with it is now the python writer's problem, and if he goofs the data is corrupt or the interpretor crashes.
From python.org's boost.python HowTo
Perhaps you'd like the resulting
Python object to contain a raw pointer
to the argument? In that case, the
caveat is that if the lifetime of the
C++ object ends before that of the
Python object, that pointer will
dangle and using the Python object may
cause a crash.
Here's how to expose mutable C++
object during module initialisation:
scope().attr("a") = object(ptr(&class_instance));
In most cases you can avoid raw pointer passing to the function, but when it's really required you can make Python object for the C++ pointer to the original object using adapter in such way:
template<typename PtrT>
struct PtrAdapter {
auto& get(PtrT ptr) { return *ptr; }
};
then define mapping of the pointer type to Python object and allow implicit conversion:
class_<Cluster<LinksT>*, noncopyable>(typpedName<LinksT>("ClusterPtr", true, true)
, "Raw hierarchy cluster pointer\n")
.def("__call__", &PtrAdapter<Cluster<LinksT>*>::get,
return_internal_reference<>(),
"referenced cluster")
;
register_ptr_to_python<Cluster<LinksT>*>();
Note that original object type also should have mapping to the Python object (in this case Cluster<LinksT>).
Then for such C++ code:
Cluster<LinksT>* cl = clusters.head();
process(cl);
Id cid = cl->id();
You can use similar Python code:
cl = clusters.head()
process(cl)
cid = cl.id()