This question is mostly academic.
It stems from some design questions I came across while working on wrappers for an existing C++ project.
When working with C/C++ extensions for Python, it is possible to dynamically load C-level functions that were wrapped in Python capsule objects in other extension modules (see the docs).
This means that, if you want to work with a C++ class wrapped in another module you have these two options (at least):
Link against the original C++ library and include the corresponding headers, and only use capsules to load functions that aren't in the original library (e.g. conversions between Python and C++ types).
Expose a C-like API by writing a factory functions that mimic the constructors and then wrapping the methods inside functions that take a pointer to an instance of the class as additional argument.
My question is this: Is there any way to share C++ classes between Python extension modules that allows the classes to be used in an idiomatic way (i.e. as they would be used in C++) while also loading them dynamically from another Python extension module?
The only possible way I can think of would be re-declaring the class with the needed method signatures under an anonymous namespace (to keep the names private to each module) and then defining the methods to wrap pointers to member functions that were loaded via capsules. Is that even possible?
Related
I would like to export a lot of abstract and normal classes from C++ to Python. It is essentially a whole package coded in C++ and would like it to be ran in Python. Instead of individually wrapping each classes, is there a way to easily wrap them and have them work in Python?
Using cppyy (http://cppyy.org) is an option, and about as automatic as it gets: just load the headers and the libraries. If there are truly a huge number of headers, then you can precompile them. Instantiation of templates and cross-inheritance is done dynamically, at run-time, so do not need to be prepared.
boost python and pybind11 allow you to "export" C++ classes as python object.
Both propose similar approaches based on templates. Pybind11 solution is said to be easier to use. If you use Microsoft toolchain, you can refer to this page to see how to design/build python bindings using pybind11:
https://learn.microsoft.com/en-us/visualstudio/python/working-with-c-cpp-python-in-visual-studio?view=vs-2019
Yes it is possible if you use for example pybind11 to wrap your C++ code. But you need to create a wrapper class around it.
There is an example of precisely this here
SWIG is another possibility. This one makes use of your existing C++ header file. See here and go down to section 6.6.3 for forcing abstract class constructor to be created using the %feature for customization
%feature("notabstract") YourClassHere
I got VS2008 project written in managed C++ that produces a utility DLL.
CLR Support: Common Language Runtime Support, Old Syntax (/clr:oldSyntax)
The resulting DLL exports a bunch of functions and complex types based on managed types like System::String, System::Byte and so on.
The DLL must be consumed (linked to) by a Python application. It seems that Python cannot use managed types/classes/structs.
Is it possible to convert the project so that it produces an "unmanaged" interface. That is, instead of working with System::String, I should somehow convert the types to std::string so that the Python app can work with?
The only "good news" is that the Python app that tries to consume the DLL needs to access only ONE exported function.
Additional question: if I only change the signature of that very function so that all the types of in/out parameters are unmanaged C++ would that work? The other exported functions will still work through managed types, but they won't be called at all.
My question is going to use examples from Python, but it seems like this could be a general question.
I've been using load-time dynamic linking, but for various reasons (it's recommended in the link below) I'd like to dynamically load the Python library:
HINSTANCE hModPython = LoadLibrary(_T("Python27.dll"));
I'm able to load Py_Initialize and other functions from the DLL, but it's a dirty process:
int (*pPy_Initialize)(void);
pPy_Initialize = (int (*)(void))GetProcAddress(hModPython, "Py_Initialize");
pPy_Initialize();
In this conversation it's said that:
Macros can make using these pointers transparent to any C code that calls routines in Python’s C API.
My question is really how to do what this writer suggests when I'm going to be importing a wide variety of functions, with various signatures. It would be nice to use the signatures that are already in Python.h (including that header somehow).
I would do like the system linker does: construct a symbol table containing all the function names. Then just initialise the pointers in that table. Function names can either be fixed string constants, or they might be read from the DLL itself (i.e. Win32 API to enumerate dll export functions?).
Significant drawback of that table approach, though, is impossibility to use it with existing code, which calls the functions by name (pPy_Initialize();) -- you'd have to use the pointer in the table, perhaps indexed via enum (pPy[Initialize]();).
Different signatures can be handled by using different tables (a table per signature). Signatures can also be stored along with the names in some symbolic form, and then you'd wrap it in some accessor magic which could parse and check it -- but that could quickly become too complex, like inventing yet another programming language.
IMHO, the only significant advantage of all that weird machinery over macros is that you might be able to load arbitrary DLLs with it. Other than that, I wouldn't go that route.
I know Cython's purpose is to create Python extensions modules, but can the compiled libraries made with Cython be loaded by non-python programs? If not, why?
I doubt that you can load them directly on a non-python program; looking at the C code generated by the simplest Cython script it's apparent that you need all the Python scaffolding to make it work. That said, you can do it indirectly for example from C++. In C++ I use boost.python to embed the Python interpreter and load some of my modules and scripts. This may seem convoluted, but allows you to quickly use whatever extensions you already have written in Python from C++, provided that you build the appropriate gluing code (see the boost.python wiki).
The disadvantage of this approach is that you are really loading a full Python interpreter just to be able to use some extension. This wasn't an issue for me since I already had the python extension and was embedding Python in my application to provide basic scripting ability, but I would not use this approach to write new libraries.
There are two mechanisms by which you can make cdef’ed Cython structures available externally:
Use a cdef public declaration – and/or also
Use the api keyword.
Here’s one of the examples of both mechanisms, from the aforelinked pages:
cdef public struct Vehicle:
int speed
float power
cdef api void activate(Vehicle *v):
if v.speed >= 88 and v.power >= 1.21:
print "Time travel achieved"
Each of these methods will instruct the Cython compiler to generate a header (“.h”) file that you can then integrate with your orthogonal C/C++ project.
A cdef public declaration yields a file named modulename.h; using the structures in this file will require you to link with the compiled Cython extension module.
An api declaration (which may be used simultaneously with cdef public, if you so desire) yields a modulename_api.h file; code consuming an api-based header will not need to link to the extension module – but it will need to call the cdef’d function import_modulename() before any of the API code can be used (a tactic the Cython-inclined users of NumPy will find familiar).
In my personal experience, it takes little effort to expose and subsequently employ encythoned structures as an external API in this fashion, provided that the struct layouts dovetail well with the consuming code, and that you’re willing to contend with manually managing the GIL in your C/C++ code in order to make things work.
I read through the following two threads on wrapping C library and C++ library, I am not sure I get it yet. The C++ library I am working with does use class and template, but not in any over-sophisticated way. What are issues or caveats of wrapping it with ctypes (besides the point that you can do so in pure python etc)?
PyCXX , Cython and boost::python are three other choices people mentioned, is there any consensus which one is more suitable for C++?
Thanks
Oliver
In defence of boost::python, given Alexander's answer on ctypes:
Boost python provides a very "c++" interface between c++ and python code - even doing things like allowed python subclasses of c++ classes to override virtual methods is relatively straightforward. Here's a potted list of good features:
Allow virtual methods of C++ classes to be overridden by python subclasses.
Bridge between std::vector<>, std::map<> instances and python lists and dictionaries (using vector_indexing_suite and map_indexing_suite)
Automatic sharing of reference counts in smart pointers (boost::shared_ptr, etc) with python reference counts (and you can extend this to any smart pointer).
Fine grained control of ownership when passing arguments and returning values from functions.
Basically, if you have a design where you want to expose a c++ interface in a way faithful to the language, then boost::python is probably the best way to do it.
The only downsides are increased compile time (boost::python makes extensive use of templates), and sometimes opaque error messages if you don't get things quite right.
For C++ a library to be accessible from Python it must use C export names, which basically means that a function named foo will be accessible from ctypes as foo.
This can be achieved only by enclosing the public interface with export C {}, which in turn disallows function overloading and templates therein (only the public interface of the library to be wrapped is relevant, the inner workings are not and may use any C++ features they like).
Reason for this is that C++ compilers use a mechanism called name mangling to generate unique names for overloaded or templated symbols. While ctypes would still find a function provided you knew its mangled name, the mangling scheme depends on the compiler/linker being used and is nothing you can rely on. In short: do not use ctypes to wrap libraries that use C++ features in their public interface.
Cython takes a different approach. It aids you at building a C extension module that does the interfacing with the original library. Therefore, linking to the C++ library is done by the regular C++ linkage mechanism, thus avoiding the aforementioned problem. The trouble with Cython is that C extension libraries need to to be recompiled for every platform, but anyway, this applies to the C++ library to be wrapped as well.
Personally, I'd say that in most cases the time to fire up Cython is a time that is well-spent and will eventually pay off in comparison to ctypes (with an exception for really simple Cish interfaces).
I don't have any experience with boost.python, so I can't comment on it (however, I don't have the impression that it is very popular either).