We Keep Coding

SWIG and Boost::variant

I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.

You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.

If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>

How to find out what's happening in build-in __exit__?

An example is python's file.__exit__ (i.e. if it does anything in addition to close). Is this documented anywhere? I tried Googling but didn't find good results.

Python's built-in functions and types are written in C (in the reference implementation, CPython). You can read its source code, if you want. For the __exit__ method you're asking about, in Python 3, I think you are looking for the file Modules/_io/iobase.c:
static PyObject *
iobase_exit(PyObject *self, PyObject *args)
{
return PyObject_CallMethodObjArgs(self, _PyIO_str_close, NULL);
}
It looks like it doesn't do anything but call close.
The equivalent bit of code for Python 2 is in a differnt file, since it is still using its own IO classes (rather than the IO module, which is also available as a backport from Python 3). Look in Objects/fileobject.c.
static PyObject *
file_exit(PyObject *f, PyObject *args)
{
PyObject *ret = PyObject_CallMethod(f, "close", NULL);
if (!ret)
/* If error occurred, pass through */
return NULL;
Py_DECREF(ret);
/* We cannot return the result of close since a true
* value will be interpreted as "yes, swallow the
* exception if one was raised inside the with block". */
Py_RETURN_NONE;
}
I'm not exactly sure why this code needs a test for None where the Python 3 code doesn't, but you can still see that it doesn't do anything other than call close (and ignore its return value).

PyImport_Import fails - Returns NULL

First off, yes I have seen this and this, however they didn't resolve my problem/error.
So, I'm trying to call a Python function from C/C++, but when PyImport_Import() is called it returns NULL.
Code:
PyObject* fname = PyBytes_FromString("hello");
PyObject* module = PyImport_Import(fname);
Where hello is my hello.py file in the same directory as the executable. I have no idea where my error is, could some please point me to it?

Without knowing a whole bunch of information about how your system is set up and the contents of the Python files involved, it's difficult to diagnose your problem. Far better to let the Python runtime tell you what is wrong.
The documentation states
PyObject* PyImport_ImportModule(const char *name)
...
Return a new reference to the imported module, or NULL with an exception set on failure
(emphasis mine)
The exception handling documentation states that you can call PyErr_Print() to print the current exception to stderr.
So to put this into practice:
PyObject* fname = PyBytes_FromString("hello");
PyObject* module = PyImport_Import(fname);
if (module == nullptr)
{
PyErr_Print();
std::exit(1);
}
This will at least get you started on figuring out the error.
For future convenience, you probably want to create your own C++ exception class to wrap Python errors (and to avoid peppering your code with things like calls to exit).

I thought that PyBytes_FromString was the the 3.x alternative of PyString_From.
I was wrong. PyUnicode_FromString is the correct alternative.
(Thanks to #wakjah for given me the tip of using error handling)

Using SWIG to pass C++ object pointers to Python, than back to C++ again

I'm using SWIG to wrap 2 C++ objects, and I am embedding the Python interpreter in my application (i.e. calling PyInitialize() etc myself).
The first object is a wrapper for some application data.
The second is a "helper" object, also written in C++, which can perform certain operation based on what it finds in the data object.
The python script decides when/how/if to invoke the helper object.
So I pass a pointer to my C++ object to SWIG/Python thus:
swig_type_info *ty = SWIG_MangledTypeQuery("_p_MyDataObject");
if(ty == NULL)
{
Py_Finalize();
return false;
}
PyObject *data_obj = SWIG_NewPointerObj(PointerToMyDataObject, ty, 0);
if(data_obj == NULL)
{
Py_Finalize();
return false;
}
ty = SWIG_MangledTypeQuery("_p_MyHelperObject");
if(ty == NULL)
{
Py_Finalize();
return false;
}
PyObject *helper_obj = SWIG_NewPointerObj(PointerToMyHelperObject, ty, 0);
if(helper_obj == NULL)
{
Py_Finalize();
return false;
}
PyTuple_SetItem(pArgs, 0, data_obj);
PyTuple_SetItem(pArgs, 1, helper_obj);
PyObject *pValue = PyObject_CallObject(pFunc, pArgs);
if(pValue == NULL)
{
Py_Finalize();
return false;
}
In Python, we see something like:
def go(dataobj, helperobj):
## if conditions are right....
helperobj.helpme(dataobj)
Now, this largely works except for one thing. In my C++ code when I am preparing my arguments to pass on to the Python script, I observe the pointer value of PointerToMyDataObject.
When I set a breakpoint in the C++ implementation of helperobj.helpme(), I see that the memory address is different, though it seems to be a pointer to a valid instance of MyDataObject.
This is important to me, as "MyDataObject" is in fact a base class for a few possible derived classes. My helper object wants to perform an appropriate (determined by context) dynamic cast on the pointer it receives to point at the appropriate derived class. That's failing for what I think are obvious reasons now.
I've read some things about "shadow" objects in SWIG, which only adds to my confusion (apologies for my tiny brain :-P)
So, is SWIG making a copy of my object for some reason, and then passing a pointer to the copy? If it is, then I can understand why my assumptions about dynamic casts won't work.
I Tried to add this as a comment, but struggled with formatting, so..... more insight follows:
The problem has to do with pass-by-reference. Notice I have 2 implementations of the virtual method helpMe():
bool MyHelperObject::helpMe(MyDataObject mydata_obj)
{
return common_code(&mydata_obj);
}
bool MyHelperObject::helpMe(MyDataObject *mydata_obj)
{
return common_code(mydata_obj);
}
Although I provided python with a pointer, it is calling the pass-by-reference version. This explains why I'm getting different pointer values. But what can I do to force a call on the version that takes a pointer argument?

Based on what you've shown I think you want to make sure SWIG only gets to see the pointer version of helpMe. The non-pointer version will be creating a temporary copy and then passing that into the function and it sounds like that isn't what you want.
SWIG will have a hard time picking which version to use since it abstracts the pointer concept slightly to match Python better.
You can hide the non-pointer version from SWIG with %ignore before the declaration or %import that shows it to SWIG in your interface file:
%ignore MyHelperObject::helpMe(MyDataObject mydata_obj)
%import "some.h"

How to dynamically create a derived type in the Python C-API

Assume we have the type Noddy as defined in the tutorial on writing C extension modules for Python. Now we want to create a derived type, overwriting only the __new__() method of Noddy.
Currently I use the following approach (error checking stripped for readability):
PyTypeObject *BrownNoddyType =
(PyTypeObject *)PyType_Type.tp_alloc(&PyType_Type, 0);
BrownNoddyType->tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE;
BrownNoddyType->tp_name = "noddy.BrownNoddy";
BrownNoddyType->tp_doc = "BrownNoddy objects";
BrownNoddyType->tp_base = &NoddyType;
BrownNoddyType->tp_new = BrownNoddy_new;
PyType_Ready(BrownNoddyType);
This works, but I'm not sure if it is The Right Way To Do It. I would have expected that I have to set the Py_TPFLAGS_HEAPTYPE flag, too, because I dynamically allocate the type object on the heap, but doing so leads to a segfault in the interpreter.
I also thought about explicitly calling type() using PyObject_Call() or similar, but I discarded the idea. I would need to wrap the function BrownNoddy_new() in a Python function object and create a dictionary mapping __new__ to this function object, which seems silly.
What is the best way to go about this? Is my approach correct? Is there an interface function I missed?
Update
There are two threads on a related topic on the python-dev mailing list (1) (2). From these threads and a few experiments I deduce that I shouldn't set Py_TPFLAGS_HEAPTYPE unless the type is allocated by a call to type(). There are different recommendations in these threads whether it is better to allocate the type manually or to call type(). I'd be happy with the latter if only I knew what the recommended way to wrap the C function that is supposed to go in the tp_new slot is. For regular methods this step would be easy -- I could just use PyDescr_NewMethod() to get a suitable wrapper object. I don't know how to create such a wrapper object for my __new__() method, though -- maybe I need the undocumented function PyCFunction_New() to create such a wrapper object.

I encountered the same problem when I was modifying an extension to be compatible with Python 3, and found this page when I was trying to solve it.
I did eventually solve it by reading the source code for the Python interpreter, PEP 0384 and the documentation for the C-API.
Setting the Py_TPFLAGS_HEAPTYPE flag tells the interpreter to recast your PyTypeObject as PyHeapTypeObject, which contains additional members that must also be allocated. At some point the interpreter attempts to refer to these extra members and, if you leave them unallocated, it will cause a segfault.
Python 3.2 introduced the C structures PyType_Slot and PyType_Spec and the C function PyType_FromSpec that simplify the creation of dynamic types. In a nutshell, you use PyType_Slot and PyType_Spec to specify the tp_* members of the PyTypeObject and then call PyType_FromSpec to do the dirty work of allocating and initialising the memory.
From PEP 0384, we have:
typedef struct{
int slot; /* slot id, see below */
void *pfunc; /* function pointer */
} PyType_Slot;
typedef struct{
const char* name;
int basicsize;
int itemsize;
int flags;
PyType_Slot *slots; /* terminated by slot==0. */
} PyType_Spec;
PyObject* PyType_FromSpec(PyType_Spec*);
(The above isn't a literal copy from PEP 0384, which also includes const char *doc as a member of PyType_Spec. But that member doesn't appear in the source code.)
To use these in the original example, assume we have a C structure, BrownNoddy, that extends the C structure for the base class Noddy. Then we would have:
PyType_Slot slots[] = {
{ Py_tp_doc, "BrownNoddy objects" },
{ Py_tp_base, &NoddyType },
{ Py_tp_new, BrownNoddy_new },
{ 0 },
};
PyType_Spec spec = { "noddy.BrownNoddy", sizeof(BrownNoddy), 0,
Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, slots };
PyTypeObject *BrownNoddyType = (PyTypeObject *)PyType_FromSpec(&spec);
This should do everything in the original code, including calling PyType_Ready, plus what is necessary for creating a dynamic type, including setting Py_TPFLAGS_HEAPTYPE, and allocating and initialising the extra memory for a PyHeapTypeObject.
I hope that's helpful.

I apologize up front if this answer is terrible, but you can find an implementation of this idea in PythonQt, in particular I think the following files might be useful references:
PythonQtClassInfo.cpp
PythonQtClassInfo.h
PythonQtClassWrapper.cpp
PythonQtClassWrapper.h
This fragment from PythonQtClassWrapper_init jumps out at me as being somewhat interesting:
static int PythonQtClassWrapper_init(PythonQtClassWrapper* self, PyObject* args, PyObject* kwds)
{
// call the default type init
if (PyType_Type.tp_init((PyObject *)self, args, kwds) < 0) {
return -1;
}
// if we have no CPP class information, try our base class
if (!self->classInfo()) {
PyTypeObject* superType = ((PyTypeObject *)self)->tp_base;
if (!superType || (superType->ob_type != &PythonQtClassWrapper_Type)) {
PyErr_Format(PyExc_TypeError, "type %s is not derived from PythonQtClassWrapper", ((PyTypeObject*)self)->tp_name);
return -1;
}
// take the class info from the superType
self->_classInfo = ((PythonQtClassWrapper*)superType)->classInfo();
}
return 0;
}
It's worth noting that PythonQt does use a wrapper generator, so it's not exactly in line with what you're asking for, but personally I think trying to outsmart the vtable isn't the most optimal design. Basically, there are many different C++ wrapper generators for Python and people use them for a good reason - they're documented, there are examples floating around in search results and on stack overflow. If you hand roll a solution for this that nobody's seen before, it'll be that much harder for them to debug if they run into problems. Even if it's closed-source, the next guy who has to maintain it will be scratching his head and you'll have to explain it to every new person who comes along.
Once you get a code generator working, all you need to do is maintain the underlying C++ code, you don't have to update or modify your extension code by hand. (Which is probably not too far away from the tempting solution you went with)
The proposed solution is an example of breaking the type-safety that the newly introduced PyCapsule provides a bit more protection against (when used as directed).
So, while its possible it might not be the best long term choice to implement derived/subclasses this way, but rather wrap the code and let the vtable do what it does best and when the new guy has questions you can just point him at the documentation for whatever solution fits best.
This is just my opinion though. :D

One way to try and understand how to do this is to create a version of it using SWIG. See what it produces and see if it matches or is done a different way. From what I can tell the people who have been writing SWIG have an in depth understanding of extending Python. Can't hurt to see how they do things at any rate. It may help you understand this problem.