We Keep Coding

Wrap a function that takes a struct of optional arguments using kwargs

In C it's not uncommon to see a function that takes a lot of inputs, many/most of which are optional group these up in a struct to make the interface cleaner for developers. (Even though you should be able to rely on a compiler accepting at least 127 arguments to a function nobody actually wants to write that many, especially as C has no overloading or default function argument support). As a hypothetical example we could consider the following struct/function pair (test.h) to illustrate the problem:
#include <stdbool.h>
typedef struct {
const char *name;
void *stuff;
int max_size;
char flags;
_Bool swizzle;
double frobination;
//...
} ComplexArgs;
void ComplexFun(const ComplexArgs *arg) {}
When it comes to wrapping this using SWIG we can get something working quickly using:
%module test
%{
#include "test.h"
%}
typedef bool _Bool;
%include "test.h"
That works and we can use it as follows:
import test
args=test.ComplexArgs()
args.flags=100;
args.swizzle=True
test.ComplexFun(args)
But that isn't exactly Pythonic. A Python developer would be more accustomed to seeing kwargs used to support this kind of calling:
import test
# Not legal in the interface currently:
test.ComplexFun(flags=100, swizzle=True)
How can we make that work? The SWIG -keyword command line option doesn't help either because there's only one actual argument to the function.

Normally in Python the way to modify function arguments and return values is to use a decorator. As a starting point I sketched out the following decorator, which solves the problem:
def StructArgs(ty):
def wrap(f):
def _wrapper(*args, **kwargs):
arg=(ty(),) if len(kwargs) else tuple()
for it in kwargs.iteritems():
setattr(arg[0], *it)
return f(*(args+arg))
return _wrapper
return wrap
It has some further neat properties when written like that:
It also doesn't break the syntax for calling the function directly with the single struct argument
It can support functions with mandatory positional arguments and a struct full of optional arguments as the last argument. (Although it can't use kwargs syntax for mandatory non-struct arguments currently)
The question then becomes one of simply applying that decorator to the right function inside the SWIG generated Python code. My plan was to wrap it up in the simplest possible macro I could, because the pattern is repeated across the library I'm wrapping lots. That turned out to be harder than I expected though. (And I'm apparently not the only one) I initially tried:
%feature("shadow") - I was pretty sure that would work, and indeed it does work for C++ member functions, but it doesn't work for free functions at global scope for some reason I didn't figure out.
%feature("autodoc") and %feature("docstring") - optimistically I'd hoped to be able to abuse them slightly, but no joy
%pythoncode right before the SWIG sees the function declaration on the C side. Generates the right code, but unfortunately SWIG immediately hides the function we decorated by adding ComplexFun = _test.ComplexFun. Couldn't find a way around it for quite a while.
Use %rename to hide the real function we call and then write a wrapper around the real function which was also decorated. That worked, but felt really inelegant, because it basically made writing the above decorator pointless instead of just writing it in the new wrapper.
Finally I found a neater trick to decorate the free function. By using %pythonprepend on the function I could insert something (anything, a comment, pass, empty string etc.) which as enough to suppress the extra code that was preventing #3 from working.
The final problem I encountered was that to make it all work as a single macro and get the position of the %pythoncode directive right (also still permit %includeing of the header file which contained the declaration) I had to call the macro before %include. That necessitated adding an additional %ignore to ignore the function if/when it's seen a second time in an actual header file. However the other problem it introduced is that we now wrap the function before the struct, so inside the Python module the type of the struct we need the decorator to populate isn't yet known when we call the decorator. That's easily enough fixed by passing a string to the decorator instead of a type and looking it up later, in the module globals().
So with that said the complete, working interface that wraps this becomes:
%module test
%pythoncode %{
def StructArgs(type_name):
def wrap(f):
def _wrapper(*args, **kwargs):
ty=globals()[type_name]
arg=(ty(),) if kwargs else tuple()
for it in kwargs.iteritems():
setattr(arg[0], *it)
return f(*(args+arg))
return _wrapper
return wrap
%}
%define %StructArgs(func, ret, type)
%pythoncode %{ #StructArgs(#type) %} // *very* position sensitive
%pythonprepend func %{ %} // Hack to workaround problem with #3
ret func(const type*);
%ignore func;
%enddef
%{
#include "test.h"
%}
typedef bool _Bool;
%StructArgs(ComplexFun, void, ComplexArgs)
%include "test.h"
This then was enough to work with the following Python code:
import test
args=test.ComplexArgs()
args.flags=100;
args.swizzle=True
test.ComplexFun(args)
test.ComplexFun(flags=100, swizzle=True)
Things you'd probably want to do before using this for real:
With this decorator and kwargs as currently written it's pretty hard to get any kind of TypeError back. Probably your C function has a way of indicating invalid combinations of inputs. Convert those into TypeError exceptions for Python users.
Adapt the macro to support mandatory positional arguments if needed.

Flexo's decoration is very impressive. I came across this problem for myself and hesitate to propose my solution except that it has one saving grace: simplicity. Also, my solution was for C++, but you might be able to modify it for C.
I declare my OptArgs struct like this:
struct OptArgs {
int oa_a {2},
double oa_b {22.0/7.0};
OptArgs& a(int n) { a = n; return *this; }
OptArgs& b(double n) { b = n; return *this; }
}
with the intention of calling the constructor from C++ with MyClass(required_arg, OptArgs().b(2.71)) for example.
Now I use the following in the .i file to move the SWIG-generated constructor out of the way and unpack keyword arguments:
%include "myclass.h"
%extend MyClass {
%pythoncode %{
SWIG__init__ = __init__
def __init__(self, *args, **kwargs):
if len(kwargs) != 0:
optargs = OptArgs()
for arg in kwargs:
set_method = getattr(optargs, arg, None)
# Deliberately let an error happen here if the argument is bogus
set_method(kwargs[arg])
args += (optargs,)
MyClass.SWIG__init__(self, *args)
%}
};
It's not perfect: it relies on the extension happening after the __init__ generated by SWIG has been declared, and is python-specific, but seems to work OK and is very, very simple.
I hope that's helpful.

SWIG: %ignore keeps giving "Syntax error in input(1)"

Forgive me if this question is silly, but I can't find a good example of %ignore being used around the web. I'm trying to generate a python wrapper for C++ code using the following command:
swig -python -c++ sample.i
I have an interface file like the following:
%module sample
%{
#include <file1.h>
#include <file2.h>
%}
%ignore vprint
%include <file1.h>
%include <file2.h>
%include <file3.h>
I'm trying to exclude the vprint function, defined in file1.h, because it takes in a va_list as a parameter, which SWIG doesn't support the wrapping of. When I include the %ignore statement, I get an error stating "Syntax error in input(1)". Without it, I get the normal error telling me I can't wrap the function I'm trying to ignore. Why might this error be occurring? Thanks in advance!

You certainly need a semicolon after the %ignore:
%ignore vprint;

SWIG argument error when using "using std::vector" in python

This is very related to this question
Regardless of whether or not this is coding practice, I have come across code that looks like this
test.hh
#include <vector>
using std::vector;
class Test
{
public:
vector<double> data;
};
I am trying to swig this using swig3.0 using the following interface file
test.i
%module test_swig
%include "std_vector.i"
namespace std {
%template(VectorDouble) vector<double>;
};
%{
#include "test.hh"
%}
%naturalvar Test::data;
%include "test.hh"
And the following test code
test.py
t = test.Test()
jprint(t)
a = [1, 2, 3]
t.data = a # fails
doing so gives me the following error
in method 'Test_data_set', argument 2 of type 'vector< double >'
This can be fixed by either changing the using std::vector in test.hh to using namespace std or by removing using std::vector and changing vector<double> to std::vector<double>. This is not what I want.
The problem is that I was given this code as is. I am not allowed to make changes, but I am supposed to still make everything available in python via SWIG. What's going on here?
Thanks in advance.

To me, this looks like SWIG does not support the using std::vector; statement correctly. I think it's a SWIG bug. I can think of the following workarounds:
Add using namespace std; to the SWIG interface file (this will only affect the way wrappers are created; the using statement will not enter C++ code)
Add #define vector std::vector to the SWIG interface file (this will only work if vector is never used as std::vector)
Copy the declarations from the header file to the SWIG interface file, and change vector to std::vector. This will cause SWIG to generate correct wrappers, and again will not affect the C++ library code.

Converting a C++ String Class to a Python String

I have a C++ class that is able to output strings in normal ASCII or wide format. I want to get the output in Python as a string. I am using SWIG (version 3.0.4) and have read the SWIG documentation. I believe I need to use the typemap construct to achieve my goal. I have written the following:
%typemap(out) my_namespace::MyString *
{
$result = PyString_AsString($1);
}
with no success. When I try to access a C++ string from Python, I get the following output:
<Swig Object of type 'MyString *' at 0x02B6FC68>
Obviously, I'm doing something wrong. Can anyone point me in the right direction? Thanks in advance.

I use pyboost for C++/python interfaces and is amazing and easy to do that. If you can, I recommend it. A simply std::string is automatically mapped to python string. In your case, may be any solution is to define a
__str __
method for the object or directly pass a char* (I see that in swig docs but I never followed this way).

It turns out the correct solution was:
%typemap(out) my_namespace::MyString &
{
$result = PyString_FromFormat($1->c_str());
}
Not very intuitive, but it works.

python pyparser fails at #pragma directive

The python C-parser pycparser fails at the following #pragma directive:
#pragma ghs section somestring="some_other_string"
generates error:
AssertionError: invalid #pragma directive
What's wrong with this #pragma?

Most likely nothing. The syntax, meaning and compiler behaviour of #pragma lines are implementation-defined. From n3797 S16.6:
A preprocessing directive of the form
# pragma pp-tokens(opt) new-line
causes the implementation to behave in an implementation-defined manner. The behavior might cause
translation to fail or cause the translator or the resulting program to behave in a non-conforming manner. Any pragma that is not recognized by the implementation is ignored.
The C standard has similar language.
If you want PyParser to do something other than throw an assertion error, you need to see what options are available to change its behaviour. Sorry, but that's beyond my scope.