We Keep Coding

vector of enum not correctly handled by SWIG

Dears,
I use SWIG to generate Python bindings to a C++ API (and it works great!) but I have serious difficulty wrapping a function that takes a vector of enum as argument. I have built a minimal example to simplify the debugging, which I put as an attachment to this issue. It seems to me that the example should work, at least it works well for a vector of integer argument.
The need is pretty simple : we have a C++ method with the following signature:
void Run(const std::vector<double> & in, std::vector<int> & out, std::vector<testing::Status> & status)
where testing::Status is an enum
and we will like to obtain a Python method like:
out, status = Run(in)
Using the attached example, the swig executable does not raise any error, and the Python Run method can be ran, but the output value status cannot be used, an error is raised:
status: (<Swig Object of type 'testing::Status *' at 0x7fa441156450>, <Swig Object of type 'testing::Status *' at 0x7fa441156660>) swig/python detected a memory leak of type 'testing::Status *', no destructor found. swig/python detected a memory leak of type 'testing::Status *', no destructor found.
Here are the different files that can be used to reproduce the error:
mylib.h, the C++ to wrap in Python
#include <vector>
namespace testing
{
typedef enum
{
Ok = 0,
Error = 1,
} Status;
class Algo
{
public:
void Run(const std::vector<double> & in, std::vector<int> & out, std::vector<testing::Status> & status)
{
status.resize(in.size());
out.resize(in.size());
for (int i=0; i<in.size(); ++i) {
out[i] = i;
status[i] = Status::Ok;
}
}
};
}
mymodule.i, the SWIG interface file
%module mymodule
%{
#include "mylib.h"
%}
%include "std_vector.i"
%include "typemaps.i"
%define STD_TEMPLATE(TYPE...)
%template() TYPE;
%apply TYPE& OUTPUT {TYPE&}
%typemap(argout) const TYPE& {
// do nothing for const references
}
%typemap(out) (TYPE&) = (const TYPE&);
%enddef
STD_TEMPLATE (std::vector <int>);
STD_TEMPLATE (std::vector <double>);
STD_TEMPLATE (std::vector < testing::Status >);
%include "mylib.h"
build.sh, the build command line used to compile binaries
${swig_install}/bin/swig \
-I. \
-I${swig_install}/share/swig/${swig_version}/python \
-I${swig_install}/share/swig/${swig_version} \
-c++ -python \
-outdir . \
-o "mymodule.cxx" \
"mymodule.i"
g++ -L${python_install}/lib -lpython3 \
-I${python_install}/include/python \
-I. \
-std=c++11 -shared -fPIC \
mymodule.cxx -o _mymodule.so
run.py, the example in Python that raises the error
import mymodule as mm
algo = mm.Algo()
out, status = algo.Run([1.1, 2.2])
print("out:", out)
print("status:", status)

SWIG doesn't know what to do with the vector of enum outputs. One way is to handle the typemaps yourself:
mylib.i
%module mylib
%{
#include "mylib.h"
%}
%include "std_vector.i"
%include "typemaps.i"
%define STD_TEMPLATE(TYPE...)
%template() TYPE;
%apply TYPE& OUTPUT {TYPE&}
%typemap(argout) const TYPE& {
// do nothing for const references
}
%typemap(out) (TYPE&) = (const TYPE&);
%enddef
STD_TEMPLATE (std::vector <int>);
STD_TEMPLATE (std::vector <double>);
// Don't require an input parameter in Python.
// Create a temporary vector to hold the output result.
%typemap(in,numinputs=0) std::vector<testing::Status>& (std::vector<testing::Status> tmp) %{
$1 = &tmp;
%}
// Create a Python list object the same size as the vector
// and copy and convert the vector contents into it.
%typemap(argout) std::vector<testing::Status>& (PyObject* list) %{
list = PyList_New($1->size());
for(int x = 0; x < $1->size(); ++x)
PyList_SET_ITEM(list, x, PyLong_FromLong($1->at(x)));
$result = SWIG_Python_AppendOutput($result, list);
%}
%include "mylib.h"
Output (same mylib.h and run.py):
out: (0, 1)
status: [0, 0]

Python SWIG wrapper for C++ rvalue std::string &&

I'm trying to build a python wrapper for gnucash c++ parts. In QofBackend I encountered the method const std::string && get_message (). In python this message returns <Swig Object of type 'std::string *' at 0x7f4a20f5c9f0> instead of a string as in my setting as there is no swig typemap for std::string rvalue reference.
I didn't really find a simple explanation so I rebuilt an example setting and dug into c++ which I barely know. I managed to get the string into python but I'd like to know
if this typemap(out) approach is correct (also in respect of memory and error handling).
The conversion in set_s_workaround() is also just a workaround. I don't think that for gnucash the python code ever needs to set this value but for completeness sake it would be nice to also have a typemap(in) std::string&& and
get rid of get_s_workaround
get init_s_2 working.
/* example.hpp */
#include <string>
using namespace std;
struct struct1{
string s;
const std::string&& get_s();
void set_s(string&&);
void set_s_workaround(string);
void init_s();
void init_s_2();
void print_s();
};
string conv_rvalue_string(string);
/* example.cpp */
#include<iostream>
#include"example.hpp"
using namespace std;
void
struct1::set_s (std::string&& msg)
{
s = msg;
}
std::string
conv_rvalue_string (std::string msg)
{
return msg;
}
void
struct1::set_s_workaround(std::string msg)
{
set_s ( conv_rvalue_string(msg) );
}
void
struct1::init_s ()
{
set_s("Some content");
}
void
struct1::init_s_2()
{
std::string msg {"Couldn't find "};
/* set_s( msg ); */
}
void
struct1::print_s ()
{
cout<<get_s()<<endl;
}
const std::string&&
struct1::get_s ()
{
return std::move(s);
}
/* example.i */
%module example
%include "std_string.i"
%typemap(out) std::string&& {
std::string s = *$1;
$result = SWIG_From_std_string(s);
}
%{
#include <string>
#include "example.hpp"
%}
%include "example.hpp"
#!/bin/bash
swig3.0 -c++ -shadow -python example.i
g++ -fpic -c example.hpp example.cpp example_wrap.cxx -I/usr/include/python3.7
g++ -shared example_wrap.o example.o -o _example.so
# pyexample.py
import example
s1 = example.struct1()
s1.set_s_workaround('TEST')
s1.print_s()
print(s1.get_s())
Thanks for the help!

This std::string && typemap uses a temporary std::string that is passed to the method that takes a std::string rvalue reference.
The fragment dependecies in the %typemap ensure that the used fragments are ready in the wrapper file. We simply use the fragments that are already provided by "std_string.i".
/* example.i */
%module example
%include "std_string.i"
%typemap(out, fragment="SWIG_From_std_string") std::string&& {
$result = SWIG_From_std_string(*$1);
}
%typemap(in, fragment="SWIG_AsVal_std_string") std::string&& (std::string temp) {
int res = SWIG_AsVal_std_string($input, &temp);
$1 = &temp;
}
%{
#include <string>
#include "example.hpp"
%}
%include "example.hpp"

init_s2() would work like this:
void
struct1::init_s_2()
{
std::string msg {"Couldn't find "};
set_s( msg + "");
}
credits to S. Holtermann!
%typemap(out) std::string&& {
std::string s = *$1;
$result = SWIG_From_std_string(s);
}
can be shortened to
%typemap(out) std::string&& {
$result = SWIG_From_std_string(*$1);
}
without creating a local var (credits to S. Holtermann!)
A possible typemap(in) could be
%typemap(in) std::string && (std::string *temp){
int res = SWIG_AsPtr_std_string($input, &temp);
$1 = temp;
}
works
hasn't been tested extensively
does it cause memory leaks?

Swigging a class member function that is a template function

This question builds on the question: How to instantiate a template method of a template class with swig? .
However, compared to that question, the code I'm trying to wrap is a little different:
class MyClass {
public:
template <class T>
void f1(const string& firstArg, const T& value);
};
The MyClass is a regular C++ class, with one template function f1.
Attempt to wrap MyClass::f1:, i.e. the Swig .i file
%template(f1String) MyClass::f1<std::string>;
With the above, a Python client can do
o = MyClass
str1 = "A String"
o.f1String("", str1)
This interface require the Python client to learn about all different f1 function names, each one different depending on the type. Not so clean.
A cleaner interface can be obtained by overloading, extending in the interface file, e.g.
%extend MyClass {
void f1(const string& s, const string& s1){
$self->f1(s, s1);
}
void f1(const string& s, const int& anInt){
$self->f1(s, anInt);
}
}
This allow client code like this:
o = MyClass
str1 = "A String"
anInt = 34
o.f1("", str1)
o.f1("", anInt)
Question is, is there any way to obtain the interface above (by extending), without extending, using Swig?

Luckily the Python wrapper supports overloading, so you can simply instantiate the two methods with the same name and SWIG will do its magic to resolve the overloads at runtime. See 6.18 Templates in the chapter “SWIG and C++” of the documentation for more details.
test.i
%module example
%{
#include<iostream>
class MyClass {
public:
template <class T>
void f1(const std::string& firstArg, const T& value) {
std::cout << firstArg << ',' << value << '\n';
}
};
%}
%include <std_string.i>
class MyClass {
public:
template <class T>
void f1(const std::string& firstArg, const T& value);
};
%extend MyClass {
%template(f1) f1<std::string>;
%template(f1) f1<int>;
}
test.py
from example import *
o = MyClass()
str1 = "A String"
anInt = 34
o.f1("X", str1)
o.f1("Y", anInt)
Example workflow to compile and run:
$ swig -python -c++ test.i
$ g++ -Wall -Wextra -Wpedantic -I /usr/include/python2.7/ -fPIC -shared test_wrap.cxx -o _example.so -lpython2.7
$ python2.7 test.py
X,A String
Y,34

Swig downcasting from Base* to Derived*

I have the following c++ classes (simplified) which I am exposing to Python using SWIG:
struct Component
{
virtual void update();
}
struct DerivedComponent : public Component
{
void update() { cout << "DerivedComponent::update()" << endl; }
void speak() { cout << "DerivedComponent::speak()" << endl; }
}
class Entity
{
public:
Component* component(const std::string& class_name)
{
return m_components[class_name];
}
private:
std::unordered_map<std::string, Component*> m_components;
}
Now, in Python I can successfully call component("DerivedComponent").update() on an Entity instance. However, I cannot call component("DerivedComponent").speak() since the type returned by component("DerivedComponent") is reported as <class 'module.Component'>.
I obviously need to downcast the result of the component() function in order to call methods defined in DerivedComponent. I had hoped that Swig would perform automatic downcasting like I believe that Boost.Python does.
Short of defining a whole bunch of typecasting functions in c++ and exposing them to Python, is there any better solution for downcasting using either Swig or Python? What are my options?

You can do exactly what you want in Python, with a little work. It works as you hope because in Python downcasting is kind of meaningless as the return types of functions (or types in general) aren't strongly typed, so we can modify your Entity::component function to always return the most derived type no matter what it is.
To make that work with your C++/Python binding you need to write an 'out' typemap for Entity::component. I've written an example of how it might work. In this case we have to bodge it slightly because the only way to know what to downcast it to comes from the argument to the function. (If for example your base class had a method that returned this as a string/enum you could simplify this further and not depend on the input arguments).
%module test
%{
#include "test.hh"
%}
%include <std_string.i>
%typemap(out) Component * Entity::component {
const std::string lookup_typename = *arg2 + " *";
swig_type_info * const outtype = SWIG_TypeQuery(lookup_typename.c_str());
$result = SWIG_NewPointerObj(SWIG_as_voidptr($1), outtype, $owner);
}
%include "test.hh"
This uses the SWIG_TypeQuery function to ask the Python runtime to lookup the type based on arg2 (which for your example is the string).
I had to make some changes to your example header (named test.hh in my example) to fix a few issues before I could make this into a fully working demo, it ended up looking like:
#include <iostream>
#include <map>
#include <string>
struct Component
{
virtual void update() = 0;
virtual ~Component() {}
};
struct DerivedComponent : public Component
{
void update() { std::cout << "DerivedComponent::update()" << std::endl; }
void speak() { std::cout << "DerivedComponent::speak()" << std::endl; }
};
class Entity
{
public:
Entity() {
m_components["DerivedComponent"] = new DerivedComponent;
}
Component* component(const std::string& class_name)
{
return m_components[class_name];
}
private:
std::map<std::string, Component*> m_components;
};
I then built it with:
swig -py3 -c++ -python -Wall test.i
g++ -Wall -Wextra test_wrap.cxx -I/usr/include/python3.4/ -lpython3.4m -shared -o _test.so
With this in place I could then run the following Python:
from test import *
e=Entity()
print(e)
c=e.component("DerivedComponent")
print(c)
print(type(c))
c.update()
c.speak()
This works as you'd hope:
<test.Entity; proxy of <Swig Object of type 'Entity *' at 0xb7230458> >
Name is: DerivedComponent *, type is: 0xb77661d8
<test.DerivedComponent; proxy of <Swig Object of type 'DerivedComponent *' at 0xb72575d8> >
<class 'test.DerivedComponent'>
DerivedComponent::update()
DerivedComponent::speak()

I was looking to do something similar and came up with a similar but different solution based on this question.
If you know the possible types ahead of time and don't mind the extra overhead, you can have the 'out' typemap loop through and dynamic_cast to each to automatically return the object with its real type. SWIG already has this implemented for pointers with the %factory feature:
%factory(Component* /* or add method name. this is just the typemap filter */,
DerivedComponent1,
DerivedComponent2);
Looking at factory.swg and boost_shared_ptr.i I got this working for shared_ptr and dynamic_pointer_cast as well:
%define %_shared_factory_dispatch(Type)
if (!dcast) {
SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<Type> dobj
= SWIG_SHARED_PTR_QNAMESPACE::dynamic_pointer_cast<Type>($1);
if (dobj) {
dcast = 1;
SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<Type> *smartresult
= dobj ? new SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<Type>(dobj) : 0;
%set_output(SWIG_NewPointerObj(%as_voidptr(smartresult),
$descriptor(SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<Type> *),
SWIG_POINTER_OWN));
}
}%enddef
%define %shared_factory(BaseType,Types...)
%typemap(out) SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<BaseType> {
int dcast = 0;
%formacro(%_shared_factory_dispatch, Types)
if (!dcast) {
SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<BaseType> *smartresult
= $1 ? new SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<BaseType>($1) : 0;
%set_output(SWIG_NewPointerObj(%as_voidptr(smartresult),
$descriptor(SWIG_SHARED_PTR_QNAMESPACE::shared_ptr<BaseType> *),
SWIG_POINTER_OWN));
}
}%enddef
// Apply dynamic_pointer cast to all returned shared_ptrs of this type
%factory(Component /* must be a type for shared_ptr */,
DerivedComponent1,
DerivedComponent2);

swig, python and wchar_t problem

I am new to the Python C binding swig and have been trying to solve this problem for a while now. I have an external C library (Example.c) that I would like to call from Python. I read Swig tutorial and able to generate the wrapper in no time. The problem now is that when I invoke the API and I got this:
>>> import Example
>>> dir(Example)
['Example_CreateConnection', 'trimmed to fit the screen']
>>> Example.Example_CreateConnection("")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: in method 'Example_CreateConnection', argument 1 of type 'ExampleChar const *'
It seemed like it cannot find the type ExampleChar. The following is my swig file:
%module Example
%{
#include "ExampleSDK.h"
%}
%include "ExampleTypes.h"
%include "ExampleSDK.h"
ExampleTypes.h looks like this:
#ifndef ExampleTypes_H
#define ExampleTypes_H
typedef wchar_t ExampleChar;
#endif /* ExampleTypes_H */
ExampleSDK.h looks like this:
#ifndef ExampleSDK_H
#define ExampleSDK_H
#include "ExampleTypes.h"
void Example_CreateConnection(const ExampleChar *temp);
#endif /* ExampleSDK_H */
The following are the command lines being invoked to generate the wrapper:
swig -python -I. Example.i
gcc -c Example.c -I/Developer/SDKs/MacOSX10.6.sdk/usr/include/
gcc -c Example_wrap.c -I/usr/include/python2.6 -I.
gcc -bundle -flat_namespace -undefined suppress -o _Example.so Example_wrap.o Example.o -L/usr/lib/python2.6/config/ -lpython2.6
Here is how the Example.c looks like:
#include "runetype.h" // for Mac wchar_t definition
#include "ExampleSDK.h"
void Example_CreateConnection(const ExampleChar *temp)
{
//do nothing
}
I am not sure what is wrong with it. I hope someone will be able to point out the mistake(s) I have done over here. Thank you.
Regards,
Chuan Lim

Last time I used wchat_t with SWIG+Python I ended up needing to add something like:
%include "pywstrings.swg"
%include "pystrings.swg"
%include "std_string.i"
%include "typemaps.i"
%fragment("SWIG_AsVal_wchar_t", "header", fragment="<wchar.h>") {
SWIGINTERN int SWIG_AsVal_wchar_t(PyObject* p, wchar_t* c) {
return SWIG_OK;
}
}
%fragment("SWIG_From_wchar_t", "header", fragment="<wchar.h>") {
SWIGINTERNINLINE PyObject* SWIG_From_wchar_t(wchar_t c) {
return SWIG_Py_Void();
}
}
// Python -> C
%typemap(in) wchar_t const * {
$1 = PyString_to_wchar_t($input);
}
// C -> Python
%typemap(out) wchar_t * {
$result = wchar_t_to_PyObject($1);
}
in my Swig interface file.