We Keep Coding

Pybind11 automatic type conversions memory management

I am using Pybind11 to make a Python C++ extension module, and have the following code:
#include <pybind11/pybind11.h>
#include <string.h>
namespace py = pybind11;
int getLength(char* arg) {
return strlen(arg);
}
PYBIND11_MODULE(example, m) {
m.def("getLength", &getLength, "Get length using strlen", py::arg("arg"));
}
From the type conversion documentations, when I call getLength from Python, Pybind11 converts Python's str type to C++ char*. I assume that the memory for the new char* argument is allocated on the heap. My question is: does Pybind11 deallocate this on return, or I need to add delete[] arg; at the end of my function?
I know that if I were to change my function, to accept py:str (Python's string type), then manually converted it to C++ char* I would be responsible for (de)allocation. Does this hold for built in conversions too?
Does Pybind11 handle memory management of the results of automatic conversions, or do I need to do that?
Though I couldn't find the answer in the docs, I link it anyway: https://pybind11.readthedocs.io/en/stable/advanced/cast/overview.html#list-of-all-builtin-conversions

In this case it will end up using the c-style string type caster, which in turn will instantiate a std::string here, and the memory will be cleaned up when this std::string's destructor is called. (unless it's an optimized small string which was allocated on the stack).
Although I am a little out of my depth here, it has to be a copy since python strings are (technically) immutable. But pybind11 also allows conversions to a std::string_view which does not make a copy.

How to unrevel a SWIG data type

I am writing C# scripts using Velocity API to automate the clinical trials' time-consuming repetitive procedures.
Velocity API is a library from Varian Medical Systems that was originally written in Python language. It was made available to C# developers through SWIG functions. Unluckily, they did not make a good job. At present, I am stuck with a data type that is not recognized by the C# compiler, namely:
pair <bool, vector>
"pair is a native data type for python and C++ but it is not a recognized data type by the C# compiler.
Through Visual Studio debugger, I browsed through Velocity.dll assembly. I could find the wrapped data type corresponding to the one showed above is:
SWIGTYPE_p_std__pairT_bool_std__vectorT_vsc__VectorR2d_t_t
If I use this latter data type in my C# script, then the C# does not complain. I can get an executable that is useless as I need to get access to the single items of the data pair returned by the method "getStructureHistognam". Unluckily the SWIGTYPE data type does not support any C# method to extract the single items. Nor does it support indexing, as a vector would.
My question for you is:
how can I unravel a variable of type SWIGTYPE_p_std__pairT_bool_std__vectorT_vsc__VectorR2d_t_t
and get access to each one of the two paired data?

SWIG generates opaque pointers for types it doesn't understand. They can be returned and passed to other functions, but not inspected. You would need to modify the SWIG .i source file to recognize the type by either writing appropriate typemaps or including appropriate template instantiations via #include <std_pair.i> and #include <std_vector.i>.
Here's a basic .i file that reveals a std::pair<bool,std::vector<int>> as an example:
%module test
%include <std_pair.i> // SWIG support for std::pair templates
%include <std_vector.i> // SWIG support for std::vector templates
%template(IntVector) std::vector<int>; // Support this template instantiation
%template(BoolIntVectorPair) std::pair<bool,std::vector<int>>; // and this one
%inline %{
#include <utility>
#include <vector>
std::pair<bool,std::vector<int>> get() {
return std::pair<bool,std::vector<int>>{true,{1,2,3}};
}
%}
My C# is rusty but building this file with Python works as shown below. You would use swig -csharp -c++ test.i to generate the wrapper, but I'm using swig -python -c++ test.i here:
>>> import test
>>> test.get()
(True, (1, 2, 3))

SWIG and Boost::variant

I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.

You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.

If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>

SWIG argument error when using "using std::vector" in python

This is very related to this question
Regardless of whether or not this is coding practice, I have come across code that looks like this
test.hh
#include <vector>
using std::vector;
class Test
{
public:
vector<double> data;
};
I am trying to swig this using swig3.0 using the following interface file
test.i
%module test_swig
%include "std_vector.i"
namespace std {
%template(VectorDouble) vector<double>;
};
%{
#include "test.hh"
%}
%naturalvar Test::data;
%include "test.hh"
And the following test code
test.py
t = test.Test()
jprint(t)
a = [1, 2, 3]
t.data = a # fails
doing so gives me the following error
in method 'Test_data_set', argument 2 of type 'vector< double >'
This can be fixed by either changing the using std::vector in test.hh to using namespace std or by removing using std::vector and changing vector<double> to std::vector<double>. This is not what I want.
The problem is that I was given this code as is. I am not allowed to make changes, but I am supposed to still make everything available in python via SWIG. What's going on here?
Thanks in advance.

To me, this looks like SWIG does not support the using std::vector; statement correctly. I think it's a SWIG bug. I can think of the following workarounds:
Add using namespace std; to the SWIG interface file (this will only affect the way wrappers are created; the using statement will not enter C++ code)
Add #define vector std::vector to the SWIG interface file (this will only work if vector is never used as std::vector)
Copy the declarations from the header file to the SWIG interface file, and change vector to std::vector. This will cause SWIG to generate correct wrappers, and again will not affect the C++ library code.

Using SWIG to wrap a C++ class that calls another objects member function

I am using swig to write a wrapper to a c++ class for use with python.
When I try to do from CSMPy import * (CSMPy is my module) I get this message:
ImportError: dlopen(/Users/MUL_mac2/anaconda/lib/python2.7/site-packages/_CSMPy.so, 2): Symbol not found: __ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m
Referenced from: /Users/MUL_mac2/anaconda/lib/python2.7/site-packages/_CSMPy.so
Expected in: dynamic lookup
A little bit of background:
I have one interface file that has an include to one header file containing my wrapper class:
This class has an object as a private member.
I then want to pass a number of objects of type std::deque<int> to a member function
of this object like so: this->object.Function(int_deque_a,int_deque_b) where object is a member of the class I am wrapping using swig.
When I comment the above line out everything works like a charm.
All the containers I am passing are valid datatypes to pass to this objects member function and contain the correct number of entries.
Everything compiles and this occurs only on import of the module.
What am I missing here?
I am using distutils to compile using python setup.py install
setup.py:
CSMPy_module = Extension('_CSMPy',
include_dirs = [Bunch of include directories here],
library_dirs = ['MyLibraryPath'],
libraries = ['MyLibrary'],
sources=['CSMPy_wrap.cxx', 'WrapperClass.cpp'],
)
setup (name = 'CSMPy',
version = '0.1',
author = "My name",
description = """Simple Test""",
ext_modules = [CSMPy_module],
py_modules = ["CSMPy"],
)
MyLibrary is a static library.
Edit 1:
I am providing you with a version of the code I can show to everyone
Setup.h
#include <iostream>
#include <vector>
#include <deque>
#include "VSet.h"
class Setup {
public:
Setup();
~Setup();
void InitializeSetup();
private:
std::deque<size_t> npes;
std::deque<size_t> epes;
std::deque<std::vector<size_t> > eni; //plist
std::deque<std::vector<csmp::int32> > enb; //pfverts
std::deque<std::vector<csmp::double64> > ncl; //pelmt
std::map<size_t, csmp::int32> bnf; //bflags
std::deque<csmp::int32> et;
csmp::VSet<2U> v;
};
Setup.cpp
#include "Setup.h"
Setup::Setup() {
std::cout<<"Setup initialized."<<std::endl;
}
Setup::~Setup() {
}
void Setup::InitializeSetup() {
for(size_t i = 0; i < this->eni.size(); i++) {
this->npes.push_back(this->eni[i].size());
}
for(size_t i = 0; i < this->enb.size(); i++) {
this->epes.push_back(this->enb[i].size());
}
this->v.Resize(this->et, npes, epes, this->ncl.size()); //This is the line that does not work
}
CSMPy.i
%module CSMPy
%{
#define SWIG_FILE_WITH_INIT
#include "stdlib.h"
#include <vector>
#include <deque>
#include <map>
#include "VSet.cpp"
#include "Setup.h"
#include "Number_Types.h"
%}
%include "Number_Types.h"
%include "std_map.i"
%include "std_vector.i"
%include "std_deque.i"
// Instantiate templates used by CSMPy
namespace std {
%template() pair<size_t, csmp::int32>;
%template() pair<size_t, csmp::double64>;
%template() pair<size_t, vector<size_t> >;
%template() pair<size_t, vector<csmp::int32> >;
%template() pair<size_t, vector<csmp::double64> >;
%template(Deque_SizeT) deque<size_t>;
%template(Deque_Int) deque<csmp::int32>;
%template(Vector_SizeT) vector<size_t>;
%template(Vector_Int32) vector<csmp::int32>;
%template(Vector_Double64) vector<csmp::double64>;
%template(Deque_Double64) deque<csmp::double64>;
%template(Deque_Vector_Int) deque<vector<csmp::int32> >;
%template(Deque_Vector_SizeT) deque<vector<size_t> >;
%template(Deque_Vector_Double64) deque<vector<csmp::double64> >;
%template(Map_SizeT_Int) map< size_t, csmp::int32>;
%template(Map_SizeT_Double64) map< size_t, csmp::double64>;
%template(Map_SizeT_Vector_SizeT) map< size_t, vector<size_t> >;
%template(Map_SizeT_Vector_Int) map< size_t, vector<csmp::int32> >;
%template(Map_SizeT_Vector_Double64) map< size_t, vector<csmp::double64> >;
}
%include "Setup.h"
Edit 2:
I did nm -gC myLib.so
I found this echo
__ZN4csmp4VSetILm2EE6ResizeERKNSt3__15dequeIiNS2_9allocatorIiEEEERKNS3_ImNS4_ImEEEESC_m
which on c++ tilt tells me:
csmp::VSet<2ul>::Resize(std::__1::deque<int, std::__1::allocator<int> > const&, std::__1::deque<unsigned long, std::__1::allocator<unsigned long> > const&, std::__1::deque<unsigned long, std::__1::allocator<unsigned long> > const&, unsigned long)
couple of notes on this, I have switched to using clang++ as my compiler and manually compiling. I have also put #include "VSet.cpp" in my .i file. (See edit in previous post)
I am now getting this error on import in python:
Symbol not found: __ZN4csmp5VData6InTextERSt14basic_ifstreamIcSt11char_traitsIcEE
Referenced from: ./_CSMPy.so
Expected in: flat namespace
I have also created a main that will instantiate the object and the call to Initialize() works.

It's not finding the symbol
__ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m
in the .so. Thanks to Dave for demangling this, we now know that it refers to
csmp::VSet<2ul>::Resize(
const std::deque<int>&,
const std::deque<unsigned long> &,
const std::deque<unsigned long> &)
So it is a little odd that you have two types of deques, based on what you posted.
Here are some things to try:
Verify that your _CSMP.so links to the STL library that comes with your compiler, you may have to specify an extra switch or field in your setup.py. Your code works when Resize is not there, you say, so that's not likely the problem.
turn on verbose output in your setup.py so that you can see the compilation and link command line parameters
make sure you %include std_deque.i in your SWIG .i file. You're not getting compile error so this is not likely the issue.
verify that you have instantiated your deque<int> with a %template(IntDeque) std::deque<int> in your .i, since Python knows nothing about C++ templates, and a template is not a class, but a recipe so compiler can create a class for you. If you really do use both int and unsigned long, you have to instantiate both. I'm only seeing int and size_t in your code. You can't assume that size_t is same as unsigned long.
Confirm that your DLL contains an intantiation of this Resize method for unsigned int. in your C++ code. I think you defined the size_t version via, or are the unsigned long unexpected?
About #5:
SWIG generates a header and a source file. In the header it puts functions that adhere to the Python C API and registers them in the Python interpreter, and in the body of those functions it figures out what C/C++ functions to call from your library. The fact that the above Resize is not found in DLL is an indication that SWIG thinks this overload of Resize is needed, so it is called from the function it generated, but your C++ lib did not instantiate it.
How is this possible? In your C++ lib, you have a class template with a Resize method. The trick with class templates is that the compiler will only generate code for the methods that are used in the DLL (so if your class defines 5 methods but your DLL only uses 1, it won't generate code for the other 4 methods), except if you explicitly instantiate the template in your library. You would do this by putting a statement
template class VSet<2ul>;
(whatever 2ul stands for) either in your C++ DLL, or the wrapper DLL via the %template directive in your .i file. This will instantiate all methods of VSet<2ul>, so Resize will be there too. IF the Resize thus generated has parameters deque<int> and deque<unsigned long>. Your code indicates that you are assuming that size_t is unsigned int. If size_t is typedefd to unsigned int, SWIG should be able to handle it, but maybe there is a bug. Better not assume. You could add a Resize overload for unsigned int. Or you could create an inline extension method in Setup taking two unsigneld long deques and calling the size_t version. Something like
%template DequeULong std::deque<unsigned long>
%extend Setup {
void Resize(const DequeInt& a, const DequeULong& b)
{
DequeSizet c;
... copy b into a DequeSizet
Resize(a, c);
}
}

The problem most likely isn't a compilation issue. It's much more likely that there's a mismatch between your header file and implementation files. The header promises an interface that you aren't implementing. You won't see the undefined reference in a standalone, C++-only application if you never call that member function in your C++ code.
The mismatch between header and implementation becomes a real problem when you tell SWIG to wrap that C++ header. The generated SWIG code contains a reference to that unimplemented function. The dynamic linking fails because that function is never defined.
So what function is it? Look at the error message:
Symbol not found: __ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m
This tells you exactly what's missing, but in a very convoluted (name mangled) way. Copy that symbol, open a terminal window, and issue the command echo <paste mangled name here> | c++filt:
echo __ZN4csmp4VSetILm2EE6ResizeERKSt5dequeIiSaIiEERKS2_ImSaImEESA_m | c++filt
The c++filt utility is a very useful capability on Macs and Linux boxes. In this case it gives you the unmangled name of the missing symbol. See my comment to Schollii's answer.

I'm starting a new answer because the fact that VSet<2U> is not wrapped makes most of my other answer not relevant to this problem (although everything in there is still correct). And the fact that Setup has a data member of type VSet<2U> is not relevant to SWIG since you aren't accessing Setup::v directly from Python.
Verify that Setup works without Python or SWIG: create a void main() where you instantiate a Setup and call its InitializeSetup() method, build and run. Most likely yo will get same runtime error due to symbol not found.
The Setup object code is looking for
csmp::VSet<2ul>::Resize(
const std::deque<int>&,
const std::deque<unsigned long> &,
const std::deque<unsigned long> &,
unsigned long)
So verify that your DLL has this symbol:
~> nm -gC yourLib.so
It probably doesn't. Are there other Resize overloads that were instantiated? This can give a clue.
There could be a variety of reasons why the compiler failed to instantiate the Resize for VSet<2U>. For example, a template class's method definitions must appear in the .h otherwise how it the compiler going to know what code to generate? If you tell the compiler to compile VSet.cpp it won't generate anything in the .o unless you explicitly instantiate the template for a specific type. Then the .o will contain object code for class of that specific templated class with that type. I like to have my method definitions separate class definition, but then I include the .cpp in the .h since any user of the .h would need to also nclude the .cpp so compiler can generate the right code. For you this would mean you would have at the bottom of VSet.h an #include "VSet.cpp" // templated code.