I have a certain base of Python code (a Flask server). I need the server to perform a performance-critical operation, which I've decided to implement in C++. However, as the C++ part has other dependencies, trying out ctypes and boost.python yielded no results (not found symbols, other libraries even when setting up the environment, etc., basically, there were problems). I believe a suitable alternative would be for me to just compile the C++ part into an executable (a single function/procedure is required) and run it from python using commands or subprocess, communicating through stdin/out for example. The only thing I'm worried about is that this will slow down the procedure enough to matter and since I'm unable to create a shared object library, calling its function from python directly, I cannot benchmark the speedup.
When I compile the C++ code into an executable and run it with some sample data, the program takes ~5s to run. This does not account for opening the process from python, nor for passing data between the processes.
The question is: How big of a speedup can one expect by using ctypes/boost with a SO compared to creating a new process to run the procedure? If I regard the number to be big enough, it would motivate me to solve the encountered problems, basically, I'm asking if it's worth it.
If you're struggling with creating binding using Boost.Python, you can manually expose your API via c-functions and use them via FFI.
Here's a simple example, which briefly explains my idea. At first, you create a shared library, but add some extra functions here, which in the example I put into extern "C" section. It's necessary to use extern "C" since otherwise function names will be mangled and their actual names are likely to be different from those you've declared:
#include <cstdint>
#include <cstdio>
#ifdef __GNUC__
#define EXPORT __attribute__ ((visibility("default")))
#else // __GNUC__
#error "Unsupported compiler"
#endif // __GNUC__
class data_processor {
public:
data_processor() = default;
void process_data(const std::uint8_t *data, std::size_t size) {
std::printf("processing %zu bytes of data at %p\n", size, data);
}
};
extern "C" {
EXPORT void *create_processor() {
return new data_processor();
}
EXPORT void free_processor(void *data) {
delete static_cast<data_processor *>(data);
}
EXPORT void process_data(void *object, const std::uint8_t *data, const std::uint32_t size) {
static_cast<data_processor *>(object)->process_data(data, size);
}
}
Then you create function bindings in python. As you can see function declarations are almost the same as they are in the cpp file below. I used built-in types only (like void *, uint8_t and anything, but I believe FFI allows you to declare and use custom structs as well):
from cffi import FFI
mylib_api = FFI()
mylib_api.cdef("""
void *create_processor();
void free_processor(void *object);
void process_data(void *object, const uint8_t *data, uint32_t size);
""")
mylib = mylib_api.dlopen("mylib.so-file-location")
processor = mylib.create_processor()
try:
buffer = b"buffer"
mylib.process_data(processor, mylib_api.from_buffer("uint8_t[]", python_buffer=buffer), len(buffer))
finally:
mylib.free_processor(processor)
And that's basically it.
In my opinion inter-processing is going to be the last resort when nothing else works since:
you need to put a lot of efforts implementing details of your communication protocol, either if you use something popular, there could be a lot of issues, especially from c++-side;
inter-process communication is generally more expensive in terms of processor time.
I understand a basic C++ function wrapped for Python looks like this:
int square(int n)
{
return n*n;
}
static PyObject* square_wrapper(PyObject* self, PyObject* args)
{
int n = 0;
if(!PyArg_ParseTuple(args, "i", &n))
return Py_RETURN_NONE;
return Py_BuildValue("i", square(n));
}
Some of the C++ functions I'd like to wrap take in double arrays and modify them. This isn't a supported type by PyArg_ParseTuple. After doing some searching it is my understanding that using PyArg_ParseTuple to get a PyObject and then making that a PyArray_DOUBLE is an option. How would I go about doing this and how would I traverse such an array. I assume if I modify the array in the C++ extension that the Python version will be modified as well. Also, is NumPy the best way to make a double/float array to pass to my C++ extension?
I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.
You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.
If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>
I have experience working with both Boost Multiprecision and with Python's mpmath, separately.
When it gets to making both communicate (for example to create Python extensions in C++), my attempts have always involved some sort of wasteful float-to-string and string-to-float conversion.
My question is: is it possible to make both communicate in a more performant (and elegant) way? And by that I mean, is there a way to directly have C++ Boost Multiprecision load from and export to a Python mpmath.mpf object in the same vein as C's mpp does via pybind11?
I have been searching for this for quite a bit. The only other similar question I found was about just exporting from Boost Multiprecision to Python (in general) using pybind11, not to a mpmath object directly. And in that question, the OP ended up using the same approach I am trying to avoid (that is, converting from/to strings when communicating from/to C++ and Python).
This answers only partially to your question. Because the direct answer is: No, it is not possible in a clean way without a wasteful conversion to string, because mpmath is a purely python library without any parts of it written in C or C++, hence even if you try to skip "wasteful conversion" by seeking to use some sort of binary compatibility, your code will be very fragile: it will break every time when some python or mpmath internals are changed ever so slightly.
However I needed exactly the same thing. And so I settled down for an automated conversion registered via boost::python which checks and converts using strings. Actually inside python you also create mpmath.mpf objects from strings, so it's very much the same, except in the code below it is faster because it is written inside C++.
So here's what works for me:
#include <boost/python.hpp>
#include <iostream>
#include <limits>
#include <sstream>
#include <boost/math/constants/constants.hpp>
#include <boost/multiprecision/cpp_bin_float.hpp>
namespace py = ::boost::python;
using Prec80 = boost::multiprecision::number<boost::multiprecision::cpp_bin_float<80>>;
template<typename ArbitraryReal>
struct ArbitraryReal_to_python {
static PyObject* convert(const ArbitraryReal& val){
std::stringstream ss{};
ss << std::setprecision(std::numeric_limits<ArbitraryReal>::digits10+1) << val;
py::object mpmath = py::import("mpmath");
mpmath.attr("mp").attr("dps")=int(std::numeric_limits<ArbitraryReal>::digits10+1);
py::object result = mpmath.attr("mpf")(ss.str());
return boost::python::incref( result.ptr() );
}
};
template<typename ArbitraryReal>
struct ArbitraryReal_from_python {
ArbitraryReal_from_python(){
boost::python::converter::registry::push_back(&convertible,&construct,boost::python::type_id<ArbitraryReal>());
}
static void* convertible(PyObject* obj_ptr){
// Accept whatever python is able to convert into float
// This works with mpmath numbers. However if you want to accept strings as numbers this checking code can be a little longer to verify if string is a valid number.
double check = PyFloat_AsDouble(obj_ptr);
return (PyErr_Occurred()==nullptr) ? obj_ptr : nullptr;
}
static void construct(PyObject* obj_ptr, boost::python::converter::rvalue_from_python_stage1_data* data){
std::istringstream ss{ py::call_method<std::string>(obj_ptr, "__str__") };
void* storage=((boost::python::converter::rvalue_from_python_storage<ArbitraryReal>*)(data))->storage.bytes;
new (storage) ArbitraryReal;
ArbitraryReal* val=(ArbitraryReal*)storage;
ss >> *val;
data->convertible=storage;
}
};
struct Var
{
Prec80 value{"-71.23"};
Prec80 get() const { return value; };
void set(Prec80 val) { value = val; };
};
BOOST_PYTHON_MODULE(pysmall)
{
ArbitraryReal_from_python<Prec80>();
py::to_python_converter<Prec80,ArbitraryReal_to_python<Prec80>>();
py::class_<Var>("Var" )
.add_property("val", &Var::get, &Var::set);
}
Now you compile this code with this command:
g++ -O1 -g pysmall.cpp -o pysmall.so -std=gnu++17 -fPIC -shared -I/usr/include/python3.7m/ -lboost_python37 -lpython3.7m -Wl,-soname,"pysmall.so"
And here is an example python session:
In [1]: import pysmall
In [2]: a=pysmall.Var()
In [3]: a.val
Out[3]: mpf('-71.2299999999999999999999999999999999999999999999999999999999999999999999999999997072')
In [4]: a.val=123.12
In [5]: a.val
Out[5]: mpf('123.120000000000000000000000000000000000000000000000000000000000000000000000000000003')
The C++ code does not care whether mpmath is already imported in python. If it is, it obtains the exsiting library handle, if it is not then it imports it.
If you find any room for improvement in this snippet please let me know!
Here's a couple of useful references when I was writing this:
https://misspent.wordpress.com/2009/09/27/how-to-write-boost-python-converters/
https://github.com/bluescarni/mppp/blob/master/include/mp%2B%2B/extra/pybind11.hpp (but I didn't want to use pybind11, just boost::python)
EDIT: I have now finished implementing this in YADE , it works with EIGEN and CGAL libraries. The part concerning this question is in file ToFromPythonConverter.hpp
I am looking to wrap a c++ class which implements an algorithm I wrote using Opencv 2.3. I am aware that there is python wrappers for opencv as a whole but what I need is to wrap my own code which uses opencv. This seems logical because the lower level of my algorithm will be fast compiled c++ code and I will be free to call it from python and build a system around it.
My class is actually quite simple, it has 4 main methods:
void train( std::vector<cv::Mat> );
void save();
void load();
bool detect( cv::Mat );
This is essentially the majority of what I need to wrap. The problem is that I don't know how to best go about this. I have looked into ctypes, swig, boost python and pyplusplus. I have not been successful with any of the above to date.
I keep encountering issues with how wrap the opencv object cv::Mat. From python I will be using numpy arrays so I know I need conversion code from a numpy array to a cv::Mat and I have to register it.
I feel like someone else must have tried something like this at one point, if you can help me out it is much appreciated
Just to reiterate the goal: Wrap my c++ class which uses opencv into a python library so that I can use my algorithm from python.
I think I have the conversion somewhat figured out (with the help of the opencv source) but I still won't work from python.
Okay so I have been working with the code linked to in the above post (a conversion from numpy to cv::Mat) and I am still encountering problems. I am going to post my code and hopefully someone more knowledgeable than I can help me out here.
For example here is a simple class:
foo.h :
#include <opencv2/core/core.hpp>
class Foo {
public:
Foo();
~Foo();
cv::Mat image;
void bar( cv::Mat in );
};
foo.cpp :
#include "foo.h"
Foo::Foo(){}
Foo::~Foo(){}
void Foo::bar( cv::Mat in) {
image = in;
cv::Canny( image, image, 50, 100 );
cv::imwrite("image.png", image);
}
And here is where I have attempted to wrap this class using boost::python and bits of code from the above link:
wrap_foo.cpp
#include <boost/python.hpp>
#include <numpy/arrayobject.h>
#include <opencv2/core/core.hpp>
#include "foo.h"
using namespace cv;
namespace bp = boost::python;
//// Wrapper Functions
void bar(Foo& f, bp::object np);
//// Converter Functions
cv::Mat convertNumpy2Mat(bp::object np);
//// Wrapper Functions
void bar(Foo& f, bp::object np)
{
Mat img = convertNumpy2Mat(np);
f.bar(img);
return;
}
//// Boost Python Class
BOOST_PYTHON_MODULE(lib)
{
bp::class_<Foo>("Foo")
.def("bar", bar)
;
}
//// Converters
cv::Mat convertNumpy2Mat(bp::object np)
{
Mat m;
numpy_to_mat(np.ptr(),m);
return m;
}
The numpy_to_mat function is from the pano_cv library but from playing around with the official opencv source I know that this function is also there to some degree. The full file has the function below what I wrote above. This code compiles with bjam just fine but the when I import into python it crashes. The error is this: libFoo.so: undefined symbol: _ZN2cv3Mat10deallocateEv. I have tried a number of different things but I can't get this to work.
Help is most appreciated.
I found some code on google that converts numpy array to cv::mat in c++ using boost::python:
link