We Keep Coding

Revert cppyy automatic mapping of operator() to __getitem__ via C++ pythonization callback

As is also explained in this cppyy issue, an A& operator() on the C++ side is mapped to the python __getitem__.
On the issue it is suggested to add a special pythonization if this is not the wished for result.
An extra constraint in my case would be to add this to the C++ class itself to ensure that this pythonization is always applied.
I'm however having trouble figuring out how to properly do this via the Python C API. (1st time working with the that API so I'm a bit lost)
Minimal Reproducer somewhat contrived but shows the problem:
Note in the example below that struct A is code that I can't modify because that class is defined in library code. So the callback has to be in B.
import cppyy
cppyy.include("Python.h")
cppyy.cppdef(r"""
void myprint(PyObject* py){
PyObject* tmp = PyObject_Str(py);
Py_ssize_t size = 0;
const char* s = PyUnicode_AsUTF8AndSize(tmp, &size);
std::cout << std::string(s, size) << std::endl;
}
template <typename T>
struct A {
T& operator[](size_t idx) { return arr[idx]; }
const T& operator[](size_t idx) const { return arr[idx]; }
std::array<T, 10> arr{};
};
template <typename T>
struct B : public A<T> {
B& operator()() { return *this; };
static void __cppyy_pythonize__( PyObject* klass, const std::string& name){
std::cout << "Hello from pythonize" << std::endl;
PyObject* getitem = PyObject_GetAttrString(klass, "__getitem__");
myprint(getitem);
}
};
using tmp = B<double>;
""")
t = cppyy.gbl.B['double']
print(t.__getitem__.__doc__)
I can get the __getitem__ function from the PyObject* klass but, as explained in the docs, the callback happens at the very end after all the internal processing of the class.
Thus the __call__ function, which here is B& operator()(), has already been mapped to __getitem__.
Unfortunately, I can't for the life of me figure out how I would undo that mapping and get back that old __getitem__ function.
Is that operator[]() function even still accessible via the PyObject* klass ?
Any help/pointers would be much appreciated :)

First, to answer your question, to find the __getitem__ you want, get it from the base class of klass, not from klass directly. You can also do this in Python, rather than adding pythonizations in C++. In fact, doing this in Python is preferred as then you don't have to deal with the C-API.
However, since the actual bug report is not the one you referenced, but this one, and since the suggestion made there, which you followed here, makes this a classic XY-problem, let me also add that what you really want is to simply do PyObject_DelAttrString(klass, "__getitem__") in your code example.
Completely aside, the code that is giving you trouble here is from the Gaudi project, the core developers of which are the ones who asked for this automatic mapping in the first place. You may want to take this up with them.

C++/embedded Python: Can I retrieve Python file name and line number when a C++ function is called from Python

I run a Python script from a C++ program using PyRun_SimpleFile. I defined a custom module (using PyImport_AppendInittab, as done here), so it can be imported in my Python script and some C++ code gets executed by the Python script when functions from this module are used, this is done through a callback PyObject* MyFunction(PyObject *self, PyObject *args) being invoked by Python interpreter.
I want to know the current script file name and line number within the callback function being invoked.
I could not find any way to retrieve this. Is this possible?
Note: This question is definitely not a duplicate of How to get the caller's method name in the called method?. I'm trying to retrieve file name and line number from C++ code executing and later executed by a Python script.

You will need PyTraceBack_Here.
You can take a look at a traceback object's implementation here
Here is an example printig the traceback created by PyTraceBack_Here
#include <Python.h>
PyObject * mymodule_meth_test(PyObject * self) {
PyTraceBack_Here(PyEval_GetFrame());
PyObject * exc;
PyObject * val;
PyObject * tb;
PyErr_Fetch(&exc, &val, &tb);
PyTraceBack_Print(tb, PySys_GetObject("stderr"));
Py_RETURN_NONE;
}
PyMethodDef module_methods[] = {
{"test", (PyCFunction)mymodule_meth_test, METH_NOARGS, NULL},
{},
};
PyModuleDef module_def = {PyModuleDef_HEAD_INIT, "mymodule", NULL, -1, module_methods};
extern "C" PyObject * PyInit_mymodule() {
PyObject * module = PyModule_Create(&module_def);
return module;
}
From the tb object you should be able to extract the filename and line number.
It is an ordinary PyObject you can pass it to a python script or inspect it.
Here is how to extract the values without taking care of the refcounts:
int line = PyLong_AsLong(PyObject_GetAttrString(PyObject_GetAttrString(tb, "tb_frame"), "f_lineno"));
const char * filename = PyUnicode_AsUTF8(PyObject_GetAttrString(PyObject_GetAttrString(PyObject_GetAttrString(tb, "tb_frame"), "f_code"), "co_filename"));

Garbage collection and custom getter in SWIG

I'm not the author, but there's a public software package I use that seems to be leaking memory (Github issue). I'm trying to figure out how to patch it to make it work correctly.
To narrow the problem down, there's a struct, call it xxx_t. First %extend is used to make a member of the struct available in Python:
%extend xxx_t {
char *surface;
}
Then there's a custom getter. What exactly it does here isn't important except that it uses new to create a char*.
%{
char* xxx_t_surface_get(xxx *n) {
char *s = new char [n->length + 1];
memcpy (s, n->surface, n->length);
s[n->length] = '\0';
return s;
}
%}
Currently the code has this line to handle garbage collection:
%newobject surface;
This does not seem to work as expected. %newobject xxx_t::surface; also doesn't work. If I replace it with %newobject xxx_t_surface_get; that doesn't work because the getter function is escaped (inside %{ ... %}).
What is the right way to tell SWIG about the char* so it gets freed?

Before getting start it's worth pointing out one thing: Because you return char* it ends up using SWIG's normal string typemaps to produce a Python string.
Having said that let's understand what the code that currently gets generated looks like. We can start our investigation with the following SWIG interface definition to experiment with:
%module test
%inline %{
struct foobar {
};
%}
%extend foobar {
char *surface;
}
If we run something like this through SWIG we'll see a generated function which wraps your _surface_get code, something like this:
SWIGINTERN PyObject *_wrap_foobar_surface_get(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
foobar *arg1 = (foobar *) 0 ;
void *argp1 = 0 ;
int res1 = 0 ;
PyObject * obj0 = 0 ;
char *result = 0 ;
if (!PyArg_ParseTuple(args,(char *)"O:foobar_surface_get",&obj0)) SWIG_fail;
res1 = SWIG_ConvertPtr(obj0, &argp1,SWIGTYPE_p_foobar, 0 | 0 );
if (!SWIG_IsOK(res1)) {
SWIG_exception_fail(SWIG_ArgError(res1), "in method '" "foobar_surface_get" "', argument " "1"" of type '" "foobar *""'");
}
arg1 = reinterpret_cast< foobar * >(argp1);
result = (char *)foobar_surface_get(arg1);
resultobj = SWIG_FromCharPtr((const char *)result);
/* result is never used again from here onwards */
return resultobj;
fail:
return NULL;
}
The thing to note here however is that the result of calling your getter is lost when this wrapper returns. That is to say that it isn't even tied to the lifespan of the Python string object that gets returned.
So there are several ways we could fix this:
One option would be to ensure that the generated wrapper calls delete[] on the result of calling your getter, after the SWIG_FromCharPtr has happened. This is exactly what %newobject does in this instance. (See below).
Another alternative would be to keep the allocated buffer between calls, probably in some thread local storage and track the size to minimise allocations
Alternatively we could use some kind of RAII based object to own the temporary buffer and make sure it gets removed. (We could do something neat with operator void* if we wanted even).
If we change our interface to add %newobject like so:
%module test
%inline %{
struct foobar {
};
%}
%newobject surface;
%extend foobar {
char *surface;
}
Then we see that our generated code now looks like this:
// ....
result = (char *)foobar_surface_get(arg1);
resultobj = SWIG_FromCharPtr((const char *)result);
delete[] result;
We can see this in the real code from github too, so this isn't the bug that you're looking for.
Typically for C++ I'd lean towards the RAII option. And as it happens there's a neat way to do this from both a SWIG perspective and a C++ one: std::string. So we can fix your leak in a simple and clean way just by doing something like this:
%include <std_string.i> /* If you don't already have this... */
%extend xxx_t {
std::string surface;
}
%{
std::string xxx_t_surface_get(xxx *n) {
return std::string(n->surface, n->length);
}
%}
(You'll need to change the setter to match too though, unless you made it const so there is no setter)
The thing about this though is that it's still making two sets of allocations for the same output. Firstly the std::string object makes one allocation and then secondly an allocation occurs for the Python string object. That's all for something where the buffer already exists in C++ anyway. So whilst this change would be sufficient and correct to solve the leak you can also go further and write a version that does less duplicitous copying:
%extend xxx_t {
PyObject *surface;
}
%{
PyObject *xxx_t_surface_get(xxx *n) {
return SWIG_FromCharPtrAndSize(n->surface, n->length);
}
%}

How can I create python custom types from C++ using native Python/C API?

Explanation is defined below:
I have defined a new Python type named "Ex1".
typedef struct {
PyObject_HEAD
PyObject * int_id;
int * value;
} Ex1;
With this type in mind and all appropriate methods generated and validated in Python interpreted (it works pretty well). I want to be able to create a python object of the new Ex1 Type from C++ backend. A typical structure of what I need is:
int main
{
// Create Ex1 Object.
Ex1 Example;
// Call PythonC-API method to include Ex1 Object into the python interpreter.
// ¿Any function-method from Python API to perform this task?
}

Actually I managed to solve this problem using python docs:
https://docs.python.org/2/extending/newtypes.html
First of all it is necessary to define the appropiate methos as is described in the python docs (1) . Assuming the PyType created has two attributes (varp1 and varp2):
PyObject * create_Ex1(vartype1 var1, vartype2 var2)
{
PyObject * pInst = PyObject_CallObject((PyObject *)&Ex1Type, NULL);
((Ex1*)pInst)->varp1 = var1;
((Ex1*)pInst)->varp2 = var2;
return pInst;
}
and
static int Ex1_init(Ex1 *self, PyObject *args, PyObject *kwds)
{
// On init return -1 if error
self->varp1= NULL;
self->varp2= NULL;
return 0;
};
This is defined on the static PyTypeObject Ex1Type as is described in the python docs (1).
Then the object is created and initialized using this line:
PyObject* Ex1_obj = create_Ex1(var1, var2);

How to get a char* from a PyObject which points to a string

How can I get a char* from a PyObject which points to a string. For example, this is the python script,
Test.Connect("272.22.20.65", 1234)
and this is the C++ code,
static PyObject* Connect(PyObject *self, PyObject *args)
{
PyObject* pIP;
PyObject* pPort;
if (!PyArg_UnpackTuple(args, "Connect", 2, 2, &pIP, &pPort))
{
return NULL;
}
const char* zIP = GetAsString(pIP);
long iPort = PyLong_AsLong(pPort);
I want to get that IP address as a char* (GetAsString is a dummy function :D ). Please note that I'm using Python 3.1.
P.S.
I don't think this question got the correct answer ,
since there is no PyStringObject or PyString_AsString in Python 3. Isn't it ?

First you encode it, then you retrieve it. Don't forget to decref the temporary.

Here is my portable recipe for it, which makes use the default encoding, where that is applicable. It assumes you start with a PyObject*, named o. If you still have your input tuple from the function call, you can skip the first 2 lines.
PyObject* args = Py_BuildValue("(O)", o);
/* Py_DECREF(o); if o is not borrowed */
if (!args) return 0;
const char* s = 0;
if (!PyArg_ParseTuple(args, "s", &s)) {
Py_DECREF(args);
return 0;
}
/* s now points to a const char* - use it, delete args when done */
Py_DECREF(args);
PS: I have not tested it, but it should work with older versions of Python as well. There is nothing on it which is version specific.