We Keep Coding

Is it possible to assign an C++ pointer to an attribute of a Python object, and retrieve it as a C++ pointer in another C++ function?

I'm making a C++ extension for Python, and I'm trying to do something like:
// this function assigns a C++ pointer to as attribute of a python object
void function1(PyObject* p){
// equivalent of p.attr = cpp_attr;
MyClass* cpp_attr = new MyClass();
PyObject* args = PyTuple_Pack(cpp_attr);
PyObject_SetAttrString(p, (char*)"attr", args);
}
I would like to retrieve this pointer and set it as attribute of another C++ object. I know how to get the PyObject* but after that I'm not sure what to do anymore
MySecondClass::MySecondClass(PyObject* p){
// get the attribute from p; equivalent of cpp_attr = p.attr
PyObject* cpp_attr = PyObject_getAttrString(p, (char*)"attr"));
// somehow get back the pointer to MyClass object created in function1
}
I looked at the documentation but I couldn't find anything that returns the original type. Is there anyway to do this?
Thanks

It's difficult to be absolutely certain, but I doubt that MyClass a Python object. This means that your attempt to store it as a Python object (e.g. using PyTuple_Pack) is completely wrong and will cause Python to malfunction in unexpected ways.
What will happen is that Python will attempt to interpret the pointer as a Python object, will try to use its normal reference counting mechanisms on that object (will change it in unpredictable ways), and ultimately try to deallocate that object (using Python mechanisms, not delete...) if some part of the object happens to equal 0.
There's a number of options, all basically centred around creating a wrapper object - a Python object defined in C++ that holds either a pointer or value of your C++ object.
Do it manually using the Python C API - This answer gives a very thorough example.
Look up the PyCapsule interface to create a quick wrapper around your object. You'd create your capsule with:
PyObject* cap = PyCapsule_New(cpp_attr, "MyClass",
[](PyObject* c) {
auto deleteme = reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass));
delete deleteme;
});
And you retrieve your C++ class from the capsule with:
reinterpret_cast<MyClass*>(PyCapsule_GetPointer(c, "MyClass))
Use some tool like PyBind11, Cython, SWIG, etc to create the wrapper object for you.
Note also that PyObject_SetAttrString does not require the third argument to be a tuple (unless you specifically want to store a tuple...). You're likely getting it confused with PyObject_Call, where the args are passed as a tuple.

Assuming your call to PyTuple_Pack is correct, then you've created a PyTupleObject which has a structure:
typedef struct {
PyObject_VAR_HEAD
PyObject *ob_item[1];
} PyTupleObject;
The PyTupleObject inherits from the generic PyObject struct which has the following members:
struct _object *_ob_next;
struct _object *_ob_prev;
Py_ssize_t ob_refcnt;
struct _typeobject *ob_type;
You can access the latter two with the macrosPy_REFCNT and Py_TYPE
The ob_item[1] member should be a pointer to the memory initially allocated. Based on how Macros are written in the documentation, you should be able to access it by
((PyTupleObject *)cpp_attr)->ob_item
And if you know the data type of the C++ pointer, then you should be able to cast it back. Maybe you can try
MyClass* cpp_att_again = reinterpret_cast<MyClass*>((PyTupleObject *)cpp_attr)->ob_item
Hopefully this points you in the right direction. You might be able to glean more insight from a similar question.

How to achieve polymorphism in Python C API?

I'm writing functools.partial object alternative, that accumulates arguments until their number become sufficient to make a call.
I use C API and I have tp_call implementation which when its called, returns modified version of self or PyObject*.
At first I followed Defining New Types guide and then realized, that I just can't return different types (PyObject * and MyObject*) from tp_call implementation.
Then I tried to not use struct with MyObject* definition and use PyObject_SetAttrString in tp_init instead, just like we do that in Python. But in that case I got AttributeError, because you can't set arbitrary attributes on object instances in Python.
What I need here is to make my tp_call implementation polymorphic, and make it able to return either MyObject which is subclass of PyObject, or PyObject type itself.
What is the sane way to do that?
UPDATE #0
That's the code:
static PyObject *Curry_call(Curry *self, PyObject *args,
PyObject *kwargs) {
PyObject * old_args = self->args;
self->args = PySequence_Concat(self->args, args);
Py_DECREF(old_args);
if (self->kwargs == NULL && kwargs != NULL) {
self->kwargs = kwargs;
Py_INCREF(self->kwargs);
} else if (self->kwargs != NULL && kwargs != NULL) {
PyDict_Merge(self->kwargs, kwargs, 1);
}
if ((PyObject_Size(self->args) +
(self->kwargs != NULL ? PyObject_Size(self->kwargs) : 0)) >=
self->num_args) {
return PyObject_Call(self->fn, self->args, self->kwargs);
} else {
return (PyObject *)self;
}
}
UPDATE #1
Why I initially abandoned this implementation - because I get segfault with it on subsequent calls of partial object. I thought that It happens because of casting Curry * to PyObject* issues. But now I have fixed the segfault by adding Py_INCREF(self); before return (PyObject *)self;. Very strange to me. Should I really INCREF self if I return it by C API ownership rules?

If you've defined your MyObject type correctly, you should be able to simply cast your MyObject * to a PyObject * and return that. The first member of a MyObject is a PyObject, and C lets you cast a pointer to a struct to a pointer to the struct's first member and vice versa. I believe the feature exists specifically to allow things like this.

I don't really know your whole code, but as long as MyObject is a PyObject (compatible, i.e. has the same "header" fields, make sure you have a length field), CPython is designed to just take your MyObject as a PyObject; simply cast the pointer to PyObject before returning it.
As you can see here, that is one of the things that is convenient when using C++: You can actually have subclasses with type safety, and you don't have to worry about someone just copying over half of your subclass' instance, for example.
EDIT: because it was asked "isn't this unsafe": yes. It is. But its only as unsafe as type handling in user code gets; CPython lets you do this, because it stores and checks the PyTypeObject *ob_type member of the PyObject struct contained. That's about as safe as for example C++'s runtime type checking is -- but it's implemented by python developers as opposed to GCC/clang/MSVC/icc/... developers.

Create a Python type from C that implements a __dict__?

How is a type created to have a __dict__ as per a "normal" class would have were it defined in Python?
Are there any examples of non-dynamic types with __dict__s?
Do types defined via Python's PyTypeObject pass through type_new?
There is a tp_dict member of PyTypeObject, but I can find no information on how it's used. There also seems to be something going on in typeobject.c's type_new but I can't decipher it clearly.
Here is some related information I've found:
__dict__ in class inherited from C extension module
How is __slots__ implemented in Python?

The following code will generate a class that implements a __dict__ in Python 2.x:
typedef struct {
PyObject_HEAD
PyObject* dict;
} BarObject;
static PyTypeObject BarObject_Type = {
PyObject_HEAD_INIT(NULL)
};
PyMODINIT_FUNC
initFoo(void)
{
PyObject *m;
m = Py_InitModule("Foo", NULL);
if (m == NULL)
return;
BarObject_Type.tp_new = PyType_GenericNew;
BarObject_Type.tp_name = "Foo.Bar";
BarObject_Type.tp_basicsize = sizeof(BarObject);
BarObject_Type.tp_getattro = PyObject_GenericGetAttr;
BarObject_Type.tp_setattro = PyObject_GenericSetAttr;
BarObject_Type.tp_flags = Py_TPFLAGS_DEFAULT;
BarObject_Type.tp_dictoffset = offsetof(BarObject,dict);
BarObject_Type.tp_doc = "Doc string for class Bar in module Foo.";
if (PyType_Ready(&BarObject_Type) < 0)
return;
Py_INCREF(&BarObject_Type);
PyModule_AddObject(m, "Bar", (PyObject*)&BarObject_Type);
}
The important bit is the tp_dictoffset member of the PyTypeObject struct (http://docs.python.org/c-api/typeobj.html):
If the instances of this type have a dictionary containing instance
variables, this field is non-zero and contains the offset in the
instances of the type of the instance variable dictionary; this offset
is used by PyObject_GenericGetAttr().
Do not confuse this field with tp_dict; that is the dictionary for
attributes of the type object itself.

To answer the last question first: No, type_new is only used for "heap types" that are dynamically defined at runtime (e.g. via a class statement). Statically defined types are initialised using PyType_Ready() instead.
To answer your first question: to create an extension type with a __dict__ descriptor, you need to dynamically allocate the type the same way the interpreter does for a class definition.
One way to get examples for that is to do as John suggests and generate some examples of your own with Cython.
For CPython 2.x you can look at the build_class method in the CPython source code (http://svn.python.org/view/python/trunk/Python/ceval.c?view=markup) to get an idea of the steps involved in a fully general solution.
If you're using Python 3 though, then this question may be of interest: What does Python's builtin __build_class__ do?
That is, as a CPython 3.x specific solution, the simplest thing to do is call builtins.__build_class__ with appropriate arguments via the C API.

I haven't done this, I'm embarrasingly bad on using the C-API, but according to the docs it should be enough with using PyObject_GenericGetAttr and PyObject_GenericSetAttr for the getattro and setattro methods. You also need to have a PyObject attribute called __dict__ of course.
Have you tried that?

How about writing some Python code defining a class and a method or two, compiling it with Cython and inspecting the generated C code?

Creating Instance of Python Extension Type in C

I am writing a simple Vector implementation as a Python extension module in C that looks mostly like this:
typedef struct {
PyObject_HEAD
double x;
double y;
} Vector;
static PyTypeObject Vector_Type = {
...
};
It is very simple to create instances of Vector while calling from Python, but I need to create a Vector instance in the same extension module. I looked in the documentation but I couldn't find a clear answer. What's the best way to do this?

Simplest is to call the type object you've created, e.g. with PyObject_CallFunction -- don't let the name fool you, it lets you call any callable, not just a function.
If you don't have a reference to your type object conveniently available as a static global to your C module, you can retrieve it in various ways, of course (e.g., from your module object with a PyObject_GetAttrString). But sticking that PyObject* into a static module-level C variable is probably simplest and most convenient.

Boost.Python - How to return by reference?

I'm using Boost.Python to create Python modules from C++ classes. And I ran into a problem with references.
Condider the following case where I have a class Foo with overloaded get methods that can either return by value or reference.
Specifying that the return by value should be used was easy once I typedefed a signature. But I think it should be possible return a reference as well by using a return_value_policy. However, using what seemed appropriate (doc); return_value_policy<reference_existing_object> did not seem to work.
Have I misunderstood what it does?
struct Foo {
Foo(float x) { _x = x; }
float& get() { return _x; }
float get() const { return _x; }
private:
float _x;
};
// Wrapper code
BOOST_PYTHON_MODULE(my_module)
{
using namespace boost::python;
typedef float (Foo::*get_by_value)() const;
typedef float& (Foo::*get_by_ref)();
class_<Foo>("Foo", init<float>())
.def("get", get_by_value(&Foo::get))
.def("get_ref", get_by_ref(&Foo::get),
return_value_policy<reference_existing_object>())//Doesn't work
;
}
Note: I know it could be dangerous to reference existing object without life-time managing.
Update:
It looks like it works for objects but not basic data types.
Take this revised example:
struct Foo {
Foo(float x) { _x = x; }
float& get() { return _x; }
float get() const { return _x; }
void set( float f ){ _x = f;}
Foo& self(){return *this;}
private:
float _x;
};
// Wrapper code
using namespace boost::python;
BOOST_PYTHON_MODULE(my_module)
{
typedef float (Foo::*get_by_value)() const;
class_<Foo>("Foo", init<float>())
.def("get", get_by_value(&Foo::get))
.def("get_self", &Foo::self,
return_value_policy<reference_existing_object>())
.def("set", &Foo::set);
;
}
Which in a test gave the expected result:
>>> foo1 = Foo(123)
>>> foo1.get()
123.0
>>> foo2 = foo1.get_self()
>>> foo2.set(1)
>>> foo1.get()
1.0
>>> id(foo1) == id(foo2)
False

In Python, there's the concept of immutable types. An immutable type can't have its value changed. Examples of built-in immutable types are int, float and str.
Having said that, you can't do what you want with boost::python, because Python itself does not allow you to change the value of the float returned by the function.
Your second example shows one solution, another would be to create thin-wrappers and exposing that:
void Foo_set_x(Foo& self, float value) {
self.get() = value;
}
class_<Foo>("Foo", init<float>())
...
.def("set", &Foo_set_x);
;
Which is a better solution than having to change the original C++ class.

I think you want return internal reference instead. I have used it before to do something similar.
Edit: Latest doc

I don't know much about Boost.Python, so I may misunderstand the question, in which case this is completely unhelpful. But here goes:
In Python you can't choose between returning by reference or by value, the distinction doesn't make sense in Python. I find it's easiest to think of it as everything being handled by reference.
You just have objects, and you have names for those objects. So
foo = "ryiuy"
Creates the string object "ryiuy" and then lets you refer to that string object with the name "foo". So in Python, when you get passed something, you get passed that object. There is no "values" as such, so you can't pass the value. But then again, it's also a valid viewpoint that there aren't references either, just objects and their names.
So the answer is, I guess, is that when you get a reference in C, you need to pass a reference to the object that reference references into Python. And when you get a value in C, you need to pass a reference to the object that you create from that value into Python.

Are you sure that the c++ object is being copied? You will get a new python object each time but which references the same c++ object. How are you determining that the object has been copied?