Related
I have a certain base of Python code (a Flask server). I need the server to perform a performance-critical operation, which I've decided to implement in C++. However, as the C++ part has other dependencies, trying out ctypes and boost.python yielded no results (not found symbols, other libraries even when setting up the environment, etc., basically, there were problems). I believe a suitable alternative would be for me to just compile the C++ part into an executable (a single function/procedure is required) and run it from python using commands or subprocess, communicating through stdin/out for example. The only thing I'm worried about is that this will slow down the procedure enough to matter and since I'm unable to create a shared object library, calling its function from python directly, I cannot benchmark the speedup.
When I compile the C++ code into an executable and run it with some sample data, the program takes ~5s to run. This does not account for opening the process from python, nor for passing data between the processes.
The question is: How big of a speedup can one expect by using ctypes/boost with a SO compared to creating a new process to run the procedure? If I regard the number to be big enough, it would motivate me to solve the encountered problems, basically, I'm asking if it's worth it.
If you're struggling with creating binding using Boost.Python, you can manually expose your API via c-functions and use them via FFI.
Here's a simple example, which briefly explains my idea. At first, you create a shared library, but add some extra functions here, which in the example I put into extern "C" section. It's necessary to use extern "C" since otherwise function names will be mangled and their actual names are likely to be different from those you've declared:
#include <cstdint>
#include <cstdio>
#ifdef __GNUC__
#define EXPORT __attribute__ ((visibility("default")))
#else // __GNUC__
#error "Unsupported compiler"
#endif // __GNUC__
class data_processor {
public:
data_processor() = default;
void process_data(const std::uint8_t *data, std::size_t size) {
std::printf("processing %zu bytes of data at %p\n", size, data);
}
};
extern "C" {
EXPORT void *create_processor() {
return new data_processor();
}
EXPORT void free_processor(void *data) {
delete static_cast<data_processor *>(data);
}
EXPORT void process_data(void *object, const std::uint8_t *data, const std::uint32_t size) {
static_cast<data_processor *>(object)->process_data(data, size);
}
}
Then you create function bindings in python. As you can see function declarations are almost the same as they are in the cpp file below. I used built-in types only (like void *, uint8_t and anything, but I believe FFI allows you to declare and use custom structs as well):
from cffi import FFI
mylib_api = FFI()
mylib_api.cdef("""
void *create_processor();
void free_processor(void *object);
void process_data(void *object, const uint8_t *data, uint32_t size);
""")
mylib = mylib_api.dlopen("mylib.so-file-location")
processor = mylib.create_processor()
try:
buffer = b"buffer"
mylib.process_data(processor, mylib_api.from_buffer("uint8_t[]", python_buffer=buffer), len(buffer))
finally:
mylib.free_processor(processor)
And that's basically it.
In my opinion inter-processing is going to be the last resort when nothing else works since:
you need to put a lot of efforts implementing details of your communication protocol, either if you use something popular, there could be a lot of issues, especially from c++-side;
inter-process communication is generally more expensive in terms of processor time.
I'm in the middle of trying to wrap a c++ project into a python api using SWIG and I'm running into an issue with code that has the following format.
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
typedef boost::variant<A,B,C> VariantType;
typedef std::vector<boost::variant<A,B,C>> VariantTypeList;
Classes A,B & C all come out in the python wrapper without a problem and seem to be usable. However when I try to add the following lines to the interface file
%template(VariantType) boost::variant<A,B,C>;
%template(VariantTypeList) std::vector<boost::variant<A,B,C>>;
I get an error that says
Boost\x64\include\boost\variant\variant.hpp(148): error : Syntax error in input(3).
So I go and look at the error and its a line that has a macro that is defined inside another header file specifically "boost/mpl/aux_/value_wknd.hpp" so I add that to the interface file with %include and now it appears that SWIG.exe crashes with an error helpfully stating
Access Violation
So long story short is there a way to wrap a boost::variant template type? Unfortunately this template definition is baked into the core of our library and I can't change it now. Also if it matters I'm on the MSVC 2013 compiler.
If it isn't possible to wrap the template type directly is it possible to work around this? I'm reading through the SWIG documentation to see if there is some typemap magic that can be applied but I'm fairly new to SWIG in general.
You can do this. I thought for quite a while about what the neatest Python interface to boost::variant actually is. My conclusion was that 99% of the time a Python user shouldn't even realise there's a variant type being use - unions and variants are basically just somewhat constrained duck-typing for C++.
So my goals were this:
wherever possible benefit from existing typemaps - we don't want to have to write our own std::string, int, typemaps from scratch.
anywhere a C++ function takes a boost::variant we should transparently accept any of the types the variant can hold for that function argument.
anywhere a C++ function returns a boost::variant we should transparently return it as the type the variant was holding when we got it back into Python.
allow Python users to explicitly create a variant object, e.g. an empty one, but don't expect that to ever actually happen. (Maybe that would be useful for reference output arguments, but I've not gone that far in this currently).
I didn't do this, but it would be fairly simple to add visitors from where this interface currently stands using the directors feature of SWIG.
It's pretty fiddly to do all that without adding some machinery into things. I wrapped everything up in a reusable file, this is the final working version of my boost_variant.i:
%{
#include <boost/variant.hpp>
static PyObject *this_module = NULL;
%}
%init %{
// We need to "borrow" a reference to this for our typemaps to be able to look up the right functions
this_module = m; // borrow should be fine since we can only get called when our module is loaded right?
// Wouldn't it be nice if $module worked *anywhere*
%}
#define FE_0(...)
#define FE_1(action,a1) action(0,a1)
#define FE_2(action,a1,a2) action(0,a1); action(1,a2)
#define FE_3(action,a1,a2,a3) action(0,a1); action(1,a2); action(2,a3)
#define FE_4(action,a1,a2,a3,a4) action(0,a1); action(1,a2); action(2,a3); action(3,a4)
#define FE_5(action,a1,a2,a3,a4,a5) action(0,a1); action(1,a2); action(2,a3); action(3,a4); action(4,a5)
#define GET_MACRO(_1,_2,_3,_4,_5,NAME,...) NAME
%define FOR_EACH(action,...)
GET_MACRO(__VA_ARGS__, FE_5, FE_4, FE_3, FE_2, FE_1, FE_0)(action,__VA_ARGS__)
%enddef
#define in_helper(num,type) const type & convert_type ## num () { return boost::get<type>(*$self); }
#define constructor_helper(num,type) variant(const type&)
%define %boost_variant(Name, ...)
%rename(Name) boost::variant<__VA_ARGS__>;
namespace boost {
struct variant<__VA_ARGS__> {
variant();
variant(const boost::variant<__VA_ARGS__>&);
FOR_EACH(constructor_helper, __VA_ARGS__);
int which();
bool empty();
%extend {
FOR_EACH(in_helper, __VA_ARGS__);
}
};
}
%typemap(out) boost::variant<__VA_ARGS__> {
// Make our function output into a PyObject
PyObject *tmp = SWIG_NewPointerObj(&$1, $&1_descriptor, 0); // Python does not own this object...
// Pass that temporary PyObject into the helper function and get another PyObject back in exchange
const std::string func_name = "convert_type" + std::to_string($1.which());
$result = PyObject_CallMethod(tmp, func_name.c_str(), "");
Py_DECREF(tmp);
}
%typemap(in) const boost::variant<__VA_ARGS__>& (PyObject *tmp=NULL) {
// I don't much like having to "guess" the name of the make_variant we want to use here like this...
// But it's hard to support both -builtin and regular modes and generically find the right code.
PyObject *helper_func = PyObject_GetAttrString(this_module, "new_" #Name );
assert(helper_func);
// TODO: is O right, or should it be N?
tmp = PyObject_CallFunction(helper_func, "O", $input);
Py_DECREF(helper_func);
if (!tmp) SWIG_fail; // An exception is already pending
// TODO: if we cared, we chould short-circuit things a lot for the case where our input really was a variant object
const int res = SWIG_ConvertPtr(tmp, (void**)&$1, $1_descriptor, 0);
if (!SWIG_IsOK(res)) {
SWIG_exception_fail(SWIG_ArgError(res), "Variant typemap failed, not sure if this can actually happen");
}
}
%typemap(freearg) const boost::variant<__VA_ARGS__>& %{
Py_DECREF(tmp$argnum);
%}
%enddef
This gives us a macro we can use in SWIG, %boost_variant. You can then use this in your interface file something like this:
%module test
%include "boost_variant.i"
%inline %{
struct A {};
struct B {};
%}
%include <std_string.i>
%boost_variant(TestVariant, A, B, std::string);
%inline %{
void idea(const boost::variant<A, B, std::string>&) {
}
boost::variant<A,B,std::string> make_me_a_thing() {
struct A a;
return a;
}
boost::variant<A,B,std::string> make_me_a_string() {
return "HELLO";
}
%}
Where the %boost_variant macro takes the first argument as a name for the type (much like %template would) and the remaining arguments as a list of all the types in the variant.
This is sufficient to allow us to run the following Python:
import test
a = test.A();
b = test.B();
test.idea(a)
test.idea(b)
print(test.make_me_a_thing())
print(test.make_me_a_string())
So how does that actually work?
We essentially duplicate SWIG's %template support here. (It's documented here as an option)
Most of the heavy lifting in my file is done using a FOR_EACH variadic macro. Largely that's the same as my previous answer on std::function, which was itself derived from several older Stack Overflow answers and adapted to work with SWIG's preprocessor.
Using the FOR_EACH macro we tell SWIG to wrap one constructor per type the variant can hold. This lets us explicitly construct variants from Python code, with two extra constructors added
By using constructors like this we can lean heavily on SWIG's overload resolution support. So given a Python object we can simply rely on SWIG to determine how to construct a variant from it. Which saves us a bunch of extra work, and uses the existing typemaps for each type within the variant.
The in typemap basically just delegates to the constructor, via a slightly convoluted route because it's surprisingly hard to find other functions in the same module programatically. Once that delegation has happened we use the normal conversion of a function argument to just pass the tempoary variant into the function as though it were what we were given.
We also synthesise a set of extra member functions, convert_typeN which internally just call boost::get<TYPE>(*this), where N and TYPE are the position of each type in the list of variant types.
Within the out typemap this then allows us to lookup a Python function, using which() to determine what the variant currently holds. We've then got largely SWIG generated code, using existing typemaps to make a given variant into a Python object of the underlying type. Again that saves us a lot of effort and makes everything plug and play.
If you're decided on SWIG (which wasn't clear to me from your post as you said to be fairly new to SWIG, so I'm under the assumption that this is a new project), then stop reading and ignore this answer.
But in case the bindings technology to use isn't fixed yet and you only need to bind Python, no other languages, an alternative is to use cppyy (http://cppyy.org, and full disclaimer: I'm main author). With that, the boost::variant type is directly available in Python and then you can make it look/behave more Pythonistic by writing Python code rather than SWIG .i code.
Example (note that cppyy has wheels for Windows on PyPI but built with MSVC2017, not MSVC2013, so I'll keep that caveat as to whether MSVC2013 is modern enough to build the code as I haven't tried):
import cppyy
cppyy.include("boost/variant/variant.hpp")
cppyy.include("boost/variant/get.hpp")
cpp = cppyy.gbl
std = cpp.std
boost = cpp.boost
cppyy.cppdef("""
class A
{
//constructors and such.
};
class B
{
//constructors and such.
};
class C
{
//constructors and such.
};
""")
VariantType = boost.variant['A, B, C']
VariantTypeList = std.vector[VariantType]
v = VariantTypeList()
v.push_back(VariantType(cpp.A()))
print(v.back().which())
v.push_back(VariantType(cpp.B()))
print(v.back().which())
v.push_back(VariantType(cpp.C()))
print(v.back().which())
print(boost.get['A'](v[0]))
try:
print(boost.get['B'](v[0]))
except Exception as e:
print(e) # b/c of type-index mismatch above
print(boost.get['B'](v[1])) # now corrected
print(boost.get['C'](v[2]))
which produces the expect output of:
$ python variant.py
0
1
2
<cppyy.gbl.A object at 0x5053704>
Could not instantiate get<B>:
B& boost::get(boost::variant<A,B,C>& operand) =>
Exception: boost::bad_get: failed value get using boost::get (C++ exception)
<cppyy.gbl.B object at 0x505370c>
<cppyy.gbl.C object at 0x5053714>
I am learning about C++, swig, and numpy for a lab I am working in. It has been specified that I must use swig (no scypy), I must use numpy arrays, and I may not "bring C code into the python world" such as by using %import "std_vector" in my interface file and just having the python user create one to send in. That being said, I am trying to get a 1d numpy array (if they need more dimensions I will just flatten it) to be passed into my C code by pointer exclusively - my boss doesn't want to have to take the time to copy everything because efficiency is very important. I believe we use c++ 14, python 2.7, and the latest version of swig, and I am using numpy.i as well.
I will provide the code below that I am currently using (just trying to get a minimum viable going here) but I am pretty new, and while it does work, I am not sure that is actually passing a pointer and not copying anything like I would like. Could someone please either confirm that it is, or show me how to make it do so? Thanks x10^99
//The C++ file I am wrapping:
#ifndef _np_array_to_array_h
#define _np_array_to_array_h
using namespace std;
double getMid(double* myArray, int size){
int half = size / 2;
return myArray[half];
}
#endif
///\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\//\/\
//The interface file:
%module np_array_to_array
%{
#define SWIG_FILE_WITH_INIT
#include "np_array_to_array.h"
#include <numpy/arrayobject.h>
%}
%include "numpy.i"
%init %{
import_array();
%}
%apply (double* IN_ARRAY1, int DIM1){(double* myArray, int size)};
%include "np_array_to_array.h"
//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
//How I compile on terminal:
swig -python -c++ np_array_to_array.i
g++ -fpic -c np_array_to_array_wrap.cxx -I/usr/include/python2.7
-I/home/sean/Desktop/SerangLab/Swig/numpy/numpy/core/include/ -I/home/sean/.local/lib/python2.7/site-packages/numpy/core/include/numpy/
g++ -shared np_array_to_array_wrap.o -o _np_array_to_array.so
This will compile and run, and create a successfully working python module that I import with (when in the same directory) "from np_array_to_array import *" and I can successfully run the getMid method, passing in a numpyArray and getting a double back out. As stated above, I am simply not sure if this is actually pass by pointer (not making any copies) or not, as I have not found anything that says one way or the other. Could someone please tell me, and if it is not, explain how one would do this? I believe it should be possible as I think numpy array uses c types and stores memory contiguously like c does.
You can investigate this fairly easily in the code that SWIG generates. There are two parts to this - a .py file and a .cxx file. In each of these there's some code generate for your getMid() function. The Python code just directly passes everything into the C++ code, which on my system ended up looking like this:
SWIGINTERN PyObject *_wrap_getMid(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
double *arg1 = (double *) 0 ;
int arg2 ;
PyArrayObject *array1 = NULL ;
int is_new_object1 = 0 ;
PyObject * obj0 = 0 ;
double result;
if (!PyArg_ParseTuple(args,(char *)"O:getMid",&obj0)) SWIG_fail;
{
npy_intp size[1] = {
-1
};
array1 = obj_to_array_contiguous_force_conversion(obj0, NPY_DOUBLE,
&is_new_object1);
if (!array1 || !require_dimensions(array1, 1) ||
!require_size(array1, size, 1)) SWIG_fail;
arg1 = (double*) array_data(array1);
arg2 = (int) array_size(array1,0);
}
result = (double)getMid(arg1,arg2);
resultobj = SWIG_From_double(static_cast< double >(result));
{
if (is_new_object1 && array1)
{
Py_DECREF(array1);
}
}
return resultobj;
fail:
...
It won't change much between SWIG and Python versions, although some of the SWIG options will change it a little bit.
The important point from the perspective of your question though seems to be the call to obj_to_array_contiguous_force_conversion. It has an argument that's used as an output parameter to indicate if a new object was allocated. If that ends up being set to true then after the call an object ends up getting released also.
From that alone it's pretty safe to conclude that the answer to your question is that it depends what input you pass to the function. If it already satisfies the constraints (i.e. it's contiguous) then you won't end up making a copy. Otherwise it will, since your C++ function requires a contiguous region.
It should also be a safe bet that if you use any of the numpy double types then you'll end up meeting this requirement and not making a copy, but for other data types that's less likely unless you've gone to a bit of effort.
I have a swigged C++ class container, MyContainer, holding objects of type MyObject, also a C++ class.
The following is the C++ header code (freemenot.h)
#ifndef freemenotH
#define freemenotH
#include <vector>
#include <string>
using std::string;
class MyObject
{
public:
MyObject(const string& lbl);
~MyObject();
string getLabel();
private:
string label;
};
class MyContainer
{
public:
MyContainer();
~MyContainer();
void addObject(MyObject* o);
MyObject* getObject(unsigned int t);
int getNrOfObjects();
private:
std::vector<MyObject*> mObjects;
};
#endif
and this is the source (freemenot.cpp)
#include "freemenot.h"
#include <iostream>
using namespace std;
/* MyObject source */
MyObject::MyObject(const string& lbl)
:
label(lbl)
{ cout<<"In object ctor"<<endl; }
MyObject::~MyObject() { cout<<"In object dtor"<<endl; }
string MyObject::getLabel() { return label; }
/* MyContainer source */
MyContainer::MyContainer() { cout<<"In container ctor"<<endl; }
MyContainer::~MyContainer()
{
cout<<"In container dtor"<<endl;
for(unsigned int i = 0; i < mObjects.size(); i++)
{
delete mObjects[i];
}
}
int MyContainer::getNrOfObjects() { return mObjects.size(); }
void MyContainer::addObject(MyObject* o) { mObjects.push_back(o); }
MyObject* MyContainer::getObject(unsigned int i) { return mObjects[i]; }
Observe that the objects are stored as RAW POINTERS in the vector. The class is such designed, and the container is thus responsible to free the objects in its destructor, as being done in the destructors for loop.
In C++ code, like below, an object o1 is added to the container c, which is returned to client code
MyContainer* getAContainerWithSomeObjects()
{
MyContainer* c = new MyContainer();
MyObject* o1 = new MyObject();
c.add(o1);
return c;
}
The returned container owns its objects, and are responsible to de-allocate these objects when done. In C++, access to the containers objects is fine after the function exits above.
Exposing the above classes to python, using Swig, will need an interface file. This interface file looks like this
%module freemenot
%{ #include "freemenot.h" %}
%include "std_string.i"
//Expose to Python
%include "freemenot.h"
And to generate a Python module, using CMake, the following CMake script was used.
cmake_minimum_required(VERSION 2.8)
project(freemenot)
find_package(SWIG REQUIRED)
include(UseSWIG)
find_package(PythonInterp)
find_package(PythonLibs)
get_filename_component(PYTHON_LIB_FOLDER ${PYTHON_LIBRARIES} DIRECTORY CACHE)
message("Python lib folder: " ${PYTHON_LIB_FOLDER})
message("Python include folder: " ${PYTHON_INCLUDE_DIRS})
message("Python libraries: " ${PYTHON_LIBRARIES})
set(PyModule "freemenot")
include_directories(
${PYTHON_INCLUDE_PATH}
${CMAKE_CURRENT_SOURCE_DIR}
)
link_directories( ${PYTHON_LIB_FOLDER})
set(CMAKE_MODULE_LINKER_FLAGS ${CMAKE_CURRENT_SOURCE_DIR}/${PyModule}.def)
set_source_files_properties(${PyModule}.i PROPERTIES CPLUSPLUS ON)
set_source_files_properties(${PyModule}.i PROPERTIES SWIG_FLAGS "-threads")
SWIG_ADD_LIBRARY(${PyModule}
MODULE LANGUAGE python
SOURCES ${PyModule}.i freemenot.cpp)
SWIG_LINK_LIBRARIES (${PyModule} ${PYTHON_LIB_FOLDER}/Python37_CG.lib )
# INSTALL PYTHON BINDINGS
# Get the python site packages directory by invoking python
execute_process(COMMAND python -c "import site; print(site.getsitepackages()[0])" OUTPUT_VARIABLE PYTHON_SITE_PACKAGES OUTPUT_STRIP_TRAILING_WHITESPACE)
message("PYTHON_SITE_PACKAGES = ${PYTHON_SITE_PACKAGES}")
install(
TARGETS _${PyModule}
DESTINATION ${PYTHON_SITE_PACKAGES})
install(
FILES ${CMAKE_CURRENT_BINARY_DIR}/${PyModule}.py
DESTINATION ${PYTHON_SITE_PACKAGES}
)
Generating the make files using CMake, and compiling using borlands bcc32 compiler, a Python module (freemenot) is generated and installed into a python3 valid sitepackages folder.
Then, in Python, the following script can be used to illuminate the problem
import freemenot as fmn
def getContainer():
c = fmn.MyContainer()
o1 = fmn.MyObject("This is a label")
o1.thisown = 0
c.addObject(o1)
return c
c = getContainer()
print (c.getNrOfObjects())
#if the thisown flag for objects in the getContainer function
#is equal to 1, the following call return an undefined object
#If the flag is equal to 0, the following call will return a valid object
a = c.getObject(0)
print (a.getLabel())
This Python code may look fine, but don't work as expected. Problem is that, when the function getContainer() returns, the memory for object o1 is freed, if the thisown flag is not set to zero. Accessing the object after this line, using the returned container will end up in disaster. Observe, there is not nothing wrong with this per se, as this is how pythons garbage collection works.
For the above use case being able to set the python objects thisown flag inside the addObject function, would render the C++ objects usable in Python.
Having the user to set this flag is no good solution.
One could also extend the python class with an "addObject" function, and modifying the thisown flag inside this function, and thereby hiding this memory trick from the user.
Question is, how to get Swig to do this, without extending the class?
I'm looking for using a typemap, or perhaps %pythoncode, but I seem not able to find a good working example.
The above code is to be used by, and passed to, a C++ program that is invoking the Python interpreter. The C++ program is responsible to manage the memory allocated in the python function, even after PyFinalize().
The above code can be downloaded from github https://github.com/TotteKarlsson/miniprojects
There are a number of different ways you could solve this problem, so I'll try and explain them each in turn, building on a few things along the way. Hopefully this is useful as a view into the options and innards of SWIG even if you only really need the first example.
Add Python code to modify thisown directly
The solution most like what you proposed relies on using SWIG's %pythonprepend directive to add some extra Python code. You can target it based on the C++ declaration of the overload you care about, e.g.:
%module freemenot
%{ #include "freemenot.h" %}
%include "std_string.i"
%pythonprepend MyContainer::addObject(MyObject*) %{
# mess with thisown
print('thisown was: %d' % args[0].thisown)
args[0].thisown = 0
%}
//Expose to Python
%include "freemenot.h"
Where the only notable quirk comes from the fact that the arguments are passed in using *args instead of named arguments, so we have to access it via position number.
There are several other places/methods to inject extra Python code (provided you're not using -builtin) in the SWIG Python documentation and monkey patching is always an option too.
Use Python's C API to tweak thisown
The next possible option here is to use a typemap calls the Python C API to perform the equivalent functionality. In this instance I've matched on the argument type and argument name, but that does mean the typemap here would get applied to all functions which receive a MyObject * named o. (The easiest solution here is to make the names describe the intended semantics in the headers if that would over-match currently which has the side benefit of making IDEs and documentation clearer).
%module freemenot
%{ #include "freemenot.h" %}
%include "std_string.i"
%typemap(in) MyObject *o {
PyObject_SetAttrString($input, "thisown", PyInt_FromLong(0)); // As above, but C API
$typemap(in,MyObject*); // use the default typemap
}
//Expose to Python
%include "freemenot.h"
The most noteworthy point about this example other than the typemap matching is the use of $typemap here to 'paste' another typemap, specifically the default one for MyObject* into our own typemap. It's worth having a look inside the generated wrapper file at a before/after example of what this ends up looking like.
Use SWIG runtime to get at SwigPyObject struct's own member directly
Since we're already writing C++ instead of going via the setattr in the Python code we can adapt this typemap to use more of SWIG's internals and skip a round-trip from C to Python and back to C again.
Internally within SWIG there's a struct that contains the details of each instance, including the ownership, type etc.
We could just cast from PyObject* to SwigPyObject* ourselves directly, but that would require writing error handling/type checking (is this PyObject even a SWIG one?) ourselves and become dependent on the details of the various differing ways SWIG can produce Python interfaces. Instead there's a single function we can call which just handles all that for us, so we can write our typemap like this now:
%module freemenot
%{ #include "freemenot.h" %}
%include "std_string.i"
%typemap(in) MyObject *o {
// TODO: handle NULL pointer still
SWIG_Python_GetSwigThis($input)->own = 0; // Safely cast $input from PyObject* to SwigPyObject*
$typemap(in,MyObject*); // use the default typemap
}
//Expose to Python
%include "freemenot.h"
This is just an evolution of the previous answer really, but implemented purely in the SWIG C runtime.
Copy construct a new instance before adding
There are other ways to approach this kind of ownership problem. Firstly in this specific instance your MyContainer assumes it can always call delete on every instance it stores (and hence owns in these semantics).
The motivating example for this would be if we were also wrapping a function like this:
MyObject *getInstanceOfThing() {
static MyObject a;
return &a;
}
Which introduces a problem with our prior solutions - we set thisown to 0, but here it would already have been 0 and so we still can't legally call delete on the pointer when the container is released.
There's a simple way to deal with this that doesn't require knowing about SWIG proxy internals - assuming MyObject is copy constructable then you can simply make a new instance and be sure that no matter where it came from it's going to be legal for the container to delete it. We can do that by adapting our typemap a little:
%module freemenot
%{ #include "freemenot.h" %}
%include "std_string.i"
%typemap(in) MyObject *o {
$typemap(in,MyObject*); // use the default typemap as before
$1 = new $*1_type(*$1); // but afterwards call copy-ctor
}
//Expose to Python
%include "freemenot.h"
The point to note here is the use of several more SWIG features that let us know the type of the typemap inputs - $*1_type is the type of the typemap argument dereferenced once. We could have just written MyObject here, as that's what it resolves to but this lets you handle things like templates if your container is really a template, or re-use the typemap in other similar containers with %apply.
The thing to watch for here now is leaks if you had a C++ function that you were deliberately allowing to return an instance without thisown being set on the assumption that the container would take ownership that wouldn't now hold.
Give the container a chance to manage ownership
Finally one of the other techniques I like using a lot isn't directly possible here as currently posed, but is worth mentioning for posterity. If you get the chance to store some additional data along side each instance in the container you can call Py_INCREF and retain a reference to the underlying PyObject* no matter where it came from. Provided you then get a callback at destruction time you can also call Py_DECREF and force the Python runtime to keep the object alive as long as the container.
You can also do that even when it's not possible to keep a 1-1 MyObject*/PyObject* pairing alive, by keeping a shadow container alive somewhere also. That can be hard to do unless you're willing to add another object into the container, subclass it or can be very certain that the initial Python instance of the container will always live long enough.
You're looking for %newobject. Here's a small example:
%module test
%newobject create;
%delobject destroy;
%inline %{
#include <iostream>
struct Test
{
Test() { std::cout << "create" << std::endl; }
~Test() { std::cout << "destroy" << std::endl; }
};
Test* create() { return new Test; }
void destroy(Test* t) { delete t; }
%}
Use:
>>> import test
>>> t1 = test.create() # create a test object
create
>>> t2 = test.Test() # don't really need a create function :)
create
>>> t3 = test.create() # and another.
create
>>> test.destroy(t2) # explicitly destroy one
destroy
>>>
>>>
>>>
>>> ^Z # exit Python and the other two get destroyed.
destroy
destroy
I just wanted thisown to be set to zero in the constructor. I did it in two ways
I simply added one line sed statement to my makefile to add 'self.thisown = 0' at the end of init() function of my class.
Using pythonappend. I figured out two caveats (a) %pythonappend statement has to be placed before c++ class definition, (b) c++ constructor overloads do not matter
%pythonappend MyApp::MyApp() %{
self.thisown = 0
%}
%include <MyApp.hpp>
How does module loading work in CPython under the hood? Especially, how does the dynamic loading of extensions written in C work? Where can I learn about this?
I find the source code itself rather overwhelming. I can see that trusty ol' dlopen() and friends is used on systems that support it but without any sense of the bigger picture it would take a long time to figure this out from the source code.
An enormous amount could be written on this topic but as far as I can tell, almost nothing has been — the abundance of webpages describing the Python language itself makes this difficult to search for. A great answer would provide a reasonably brief overview and references to resources where I can learn more.
I'm mostly concerned with how this works on Unix-like systems simply because that's what I know but I am interested in if the process is similar elsewhere.
To be more specific (but also risk assuming too much), how does CPython use the module methods table and initialization function to "make sense" of dynamically loaded C?
TLDR short version bolded.
References to the Python source code are based on version 2.7.6.
Python imports most extensions written in C through dynamic loading. Dynamic loading is an esoteric topic that isn't well documented but it's an absolute prerequisite. Before explaining how Python uses it, I must briefly explain what it is and why Python uses it.
Historically C extensions to Python were statically linked against the Python interpreter itself. This required Python users to recompile the interpreter every time they wanted to use a new module written in C. As you can imagine, and as Guido van Rossum describes, this became impractical as the community grew. Today, most Python users never compile the interpreter once. We simply "pip install module" and then "import module" even if that module contains compiled C code.
Linking is what allows us to make function calls across compiled units of code. Dynamic loading solves the problem of linking code when the decision for what to link is made at runtime. That is, it allows a running program to interface with the linker and tell the linker what it wants to link with. For the Python interpreter to import modules with C code, this is what's called for. Writing code that makes this decision at runtime is quite uncommon and most programmers would be surprised that it's possible. Simply put, a C function has an address, it expects you to put certain data in certain places, and it promises to have put certain data in certain places upon return. If you know the secret handshake, you can call it.
The challenge with dynamic loading is that it's incumbent upon the programmer to get the handshake right and there are no safety checks. At least, they're not provided for us. Normally, if we try to call a function name with an incorrect signature we get a compile or linker error. With dynamic loading we ask the linker for a function by name (a "symbol") at runtime. The linker can tell us if that name was found but it can’t tell us how to call that function. It just gives us an address - a void pointer. We can try to cast to a function pointer of some sort but it is solely up to the programmer to get the cast correct. If we get the function signature wrong in our cast, it’s too late for the compiler or linker to warn us. We’ll likely get a segfault after the program careens out of control and ends up accessing memory inappropriately. Programs using dynamic loading must rely on pre-arranged conventions and information gathered at runtime to make proper function calls. Here's a small example before we tackle the Python interpreter.
File 1: main.c
/* gcc-4.8 -o main main -ldl */
#include <dlfcn.h> /* key include, also in Python/dynload_shlib.c */
/* used for cast to pointer to function that takes no args and returns nothing */
typedef void (say_hi_type)(void);
int main(void) {
/* get a handle to the shared library dyload1.so */
void* handle1 = dlopen("./dyload1.so", RTLD_LAZY);
/* acquire function ptr through string with name, cast to function ptr */
say_hi_type* say_hi1_ptr = (say_hi_type*)dlsym(handle1, "say_hi1");
/* dereference pointer and call function */
(*say_hi1_ptr)();
return 0;
}
/* error checking normally follows both dlopen() and dlsym() */
File 2: dyload1.c
/* gcc-4.8 -o dyload1.so dyload1.c -shared -fpic */
/* compile as C, C++ does name mangling -- changes function names */
#include <stdio.h>
void say_hi1() {
puts("dy1: hi");
}
These files are compiled and linked separately but main.c knows to go looking for ./dyload1.so at runtime. The code in main assumes that dyload1.so will have a symbol "say_hi1". It gets a handle to dyload1.so's symbols with dlopen(), gets the address of a symbol using dlsym(), assumes it is a function that takes no arguments and returns nothing, and calls it. It has no way to know for sure what "say_hi1" is -- a prior agreement is all that keeps us from segfaulting.
What I've shown above is the dlopen() family of functions. Python is deployed on many platforms, not all of which provide dlopen() but most have similar dynamic loading mechanisms. Python achieves portable dynamic loading by wrapping the dynamic loading mechanisms of several operating systems in a common interface.
This comment in Python/importdl.c summarizes the strategy.
/* ./configure sets HAVE_DYNAMIC_LOADING if dynamic loading of modules is
supported on this platform. configure will then compile and link in one
of the dynload_*.c files, as appropriate. We will call a function in
those modules to get a function pointer to the module's init function.
*/
As referenced, in Python 2.7.6 we have these dynload*.c files:
Python/dynload_aix.c Python/dynload_beos.c Python/dynload_hpux.c
Python/dynload_os2.c Python/dynload_stub.c Python/dynload_atheos.c
Python/dynload_dl.c Python/dynload_next.c Python/dynload_shlib.c
Python/dynload_win.c
They each define a function with this signature:
dl_funcptr _PyImport_GetDynLoadFunc(const char *fqname, const char *shortname,
const char *pathname, FILE *fp)
These functions contain the different dynamic loading mechanisms for different operating systems. The mechanism for dynamic loading on Mac OS newer than 10.2 and most Unix(-like) systems is dlopen(), which is called in Python/dynload_shlib.c.
Skimming over dynload_win.c, the analagous function for Windows is LoadLibraryEx(). Its use looks very similar.
At the bottom of Python/dynload_shlib.c you can see the actual call to dlopen() and to dlsym().
handle = dlopen(pathname, dlopenflags);
/* error handling */
p = (dl_funcptr) dlsym(handle, funcname);
return p;
Right before this, Python composes the string with the function name it will look for. The module name is in the shortname variable.
PyOS_snprintf(funcname, sizeof(funcname),
LEAD_UNDERSCORE "init%.200s", shortname);
Python simply hopes there's a function called init{modulename} and asks the linker for it. Starting here, Python relies on a small set of conventions to make dynamic loading of C code possible and reliable.
Let's look at what C extensions must do to fulfill the contract that makes the above call to dlsym() work. For compiled C Python modules, the first convention that allows Python to access the compiled C code is the init{shared_library_filename}() function. For a module named spam compiled as shared library named “spam.so”, we might provide this initspam() function:
PyMODINIT_FUNC
initspam(void)
{
PyObject *m;
m = Py_InitModule("spam", SpamMethods);
if (m == NULL)
return;
}
If the name of the init function does not match the filename, the Python interpreter cannot know how to find it. For example, renaming spam.so to notspam.so and trying to import gives the following.
>>> import spam
ImportError: No module named spam
>>> import notspam
ImportError: dynamic module does not define init function (initnotspam)
If the naming convention is violated there's simply no telling if the shared library even contains an initialization function.
The second key convention is that once called, the init function is responsible for initializing itself by calling Py_InitModule. This call adds the module to a "dictionary"/hash table kept by the interpreter that maps module name to module data. It also registers the C functions in the method table. After calling Py_InitModule, modules may initialize themselves in other ways such as adding objects. (Ex: the SpamError object in the Python C API tutorial). (Py_InitModule is actually a macro that creates the real init call but with some info baked in like what version of Python our compiled C extension used.)
If the init function has the proper name but does not call Py_InitModule(), we get this:
SystemError: dynamic module not initialized properly
Our methods table happens to be called SpamMethods and looks like this.
static PyMethodDef SpamMethods[] = {
{"system", spam_system, METH_VARARGS,
"Execute a shell command."},
{NULL, NULL, 0, NULL}
};
The method table itself and the function signature contracts it entails is the third and final key convention necessary for Python to make sense of dynamically loaded C. The method table is an array of struct PyMethodDef with a final sentinel entry. A PyMethodDef is defined in Include/methodobject.h as follows.
struct PyMethodDef {
const char *ml_name; /* The name of the built-in function/method */
PyCFunction ml_meth; /* The C function that implements it */
int ml_flags; /* Combination of METH_xxx flags, which mostly
describe the args expected by the C func */
const char *ml_doc; /* The __doc__ attribute, or NULL */
};
The crucial part here is that the second member is a PyCFunction. We passed in the address of a function, so what is a PyCFunction? It's a typedef, also in Include/methodobject.h
typedef PyObject *(*PyCFunction)(PyObject *, PyObject *);
PyCFunction is a typedef for a pointer to a function that returns a pointer to a PyObject and that takes for arguments two pointers to PyObjects. As a lemma to convention three, C functions registered with the method table all have the same signature.
Python circumvents much of the difficulty in dynamic loading by using a limited set of C function signatures. One signature in particular is used for most C functions. Pointers to C functions that take additional arguments can be "snuck in" by casting to PyCFunction. (See the keywdarg_parrot example in the Python C API tutorial.) Even C functions that backup Python functions that take no arguments in Python will take two arguments in C (shown below). All functions are also expected to return something (which may just be the None object). Functions that take multiple positional arguments in Python have to unpack those arguments from a single object in C.
That's how the data for interfacing with dynamically loaded C functions is acquired and stored. Finally, here's an example of how that data is used.
The context here is that we're evaluating Python "opcodes", instruction by instruction, and we've hit a function call opcode. (see https://docs.python.org/2/library/dis.html. It's worth a skim.) We've determined that the Python function object is backed by a C function. In the code below we check if the function in Python takes no arguments (in Python) and if so, call it (with two arguments in C).
Python/ceval.c.
if (flags & (METH_NOARGS | METH_O)) {
PyCFunction meth = PyCFunction_GET_FUNCTION(func);
PyObject *self = PyCFunction_GET_SELF(func);
if (flags & METH_NOARGS && na == 0) {
C_TRACE(x, (*meth)(self,NULL));
}
It does of course take arguments in C - exactly two. Since everything is an object in Python, it gets a self argument. At the bottom you can see that meth is assigned a function pointer which is then dereferenced and called. The return value ends up in x.