I have a poorly designed and big (> 300 public functions, >200 numeric constants defined with #define in the header file) that I have to wrap in Python. I have the dll and the h file. The library is updated yearly, till now in a backwards compatible way (i.e. just functions were added, a constant keep their numerical values, etc). But I have no guarantees as I do not control the library.
Using ctypes, I see two ways of wrapping this in Python:
Mapping every constant and function to python, 1 to 1
Redefining the API in Python and making calls to the library.
The first can be done in a (roughly) automatic way from the header file and therefore is easier to maintain and upgrade, the second requires a lot of python code but it will be easier to use.
I would appreciate some opinions based on your experience with this type of problem and some examples.
I recently used ctypesgen to create a ctypes wrapping for SDL, and complementary libraries (SDL_image, SDL_ttf, SDL_mixer).
For me, it worked fairly well. It generates Python 2.x, but I was able to get the desired 3.x code by using the "2to3" utility.
I think it's a good idea to use the ctypes wrapping as a foundation for a more "pythonic" api, and that's basically what I did (on a very simple level) with my pslab module.
So, if you're looking to do something similar, that would be one way.
Maintaining a Python library with a ctypes backend isn't an unmanageable approach. Obviously the initial investment is larger than using automated tools, but the API you are left with should be much better.
If you do take that route, aim to separate the Python API entirely from the C library though. Supporting multiple ctypes backends with one Python front end api isn't too bad - just query at runtime and dynamically load the correct ctypes wrapper module. I've done that to wrap different dll files and .so files for windows and linux but it would work for versions of a library as well.
Related
I have a Python code that needs to be able to execute a C++ code. I'm new to the idea of creating libraries but from what I have learned so far I need to know whether I need to use static or dynamic linking.
I have read up on the pros and cons of both but there is a lot of jargon thrown around that I do not understand yet and since I need to do this ASAP I was wondering if some light can be shed on this from somebody who can explain it simply to me.
So here's the situation. My C++ code generates some text files that have data. My Python code then uses those text files to plot the data. As a starter, I need to be able to run the C++ code directly from Python. Is DLL more suitable than SL? Or am I barking up the completely wrong tree?
Extra: is it possible to edit variables in my C++ code, compile it and execute it, all directly from Python?
It depends on your desired deployment. If you use dynamic linking will need to carefully manage the libraries (.so, .dll) on your path and ensure that the correct version is loaded. This can be helped if you include the version number in the filename, but then that has its own problems (security... displaying version numbers of your code is a bad idea).
Another benefit is that you can swap your library functionality without a re-compile as long as the interface does not change.
Statically linking is conceptually simpler and practically simpler. You only have to deploy one artefact (an .exe for example). I recommend you start with that until you need to move to the more complicated shared library setup.
Edit: I don't understand your "extra credit" question. What do you mean by "edit values"? If you mean can you modify variables that were declared in your C++ code, then yes you can as long as you use part of the public interface to do it.
BTW this advice is for the general decision. If you are linking from Python to C/C++ I think you need to use a shared library. Not sure as I haven't done it myself.
EDIT: To expand on "public interface". When you create a C++ library of whatever kind, you specify what functions are available to outside classes (look up how to to that). This is what I mean by public interface. Parts of your library are inaccessible but others (that you specify) are able to be called from client code (i.e. your python script). This allows you to modify the values that are stored in memory.
If you DO mean that you want to edit the actual C++ code from within your python I would suggest that you should re-design your application. You should be able to customise the run-time behaviour of your C++ library by providing the appropriate configuration.
If you give a solid example of what you mean by that we'll be able to give you better advice.
Yes it is possible!!
Try exploring subprocess module in python.
Following can be an example implementation of your scenario:
yourfile.cpp
#compilation
args = ['g++','-o','your_executable_name_with_path','yourfile.cpp_with_path']
your_compile = subprocess.Popen(args,stdin=subprocess.PIPE,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
output,compilation_error = your_compile.communicate()
your_compile.wait()
#successful compilation then there will be execuatble
if not compilation_error:
#execuation
args = ['your_executable_name_with_path'] #command to run a an execuatble
your_run = subprocess.Popen(args,stdin=subprocess.PIPE,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
your_code_output,runtime_error = your_run.communicate()
your_run.wait()
Further, you can tackle more cases and come up with an efficient design
I'm not quite sure how the idea of linking comes into what you are asking, but it sounds to me like you want to use something like SWIG, which allows you to create wrappers around C++ functions (and many other languages) which you can then call directly from your Python code.
Extra: is it possible to edit values in my C++ code, compile it and execute it directly from Python?
If I'm understanding this correctly, you want to use Python to change your C++ code, then compile and execute it? If this is the case, you may want to look into embedding the Python interpreter in your C++ program. This would mean doing things the other way around and having C++ run your Python script, instead of trying to do everything from Python.
I am new to object oriented programming and I am struggling to find a good tutorial on how to make a library in C++ that I can import into Python.
At the moment I am just trying to make a simple example that adds two numbers. I am confused about the process. Essentially I want to be able to do something like this in Python:
import MyCPPcode
MyCPPcode.Add(5,3) #function prints 5+3=8
I am not requesting a full example with code, just the steps that I need to take.
Do I need to make a .dll or a static library? I am using MS Visual Studio 2013.
Also, does the process tailor the C++ library code for Python in any way or will this library be available for other languages as well?
While I cannot guide you through the whole process, because I do not know python too well, Here is what I know:
It is absolutely possible. While not being something for someone who is new to object-oriented programing, it's called the python-C/C++ API. If you search for that in the python documentation there are several chapters about it.
While the example function you're showing might look like that from python, the process is a lot more redundant in c++ (behind the scenes). There are tools that combat that issue, for example Cython, but if you want to learn I'd suggest going pure python API.
As for availability with other languages... Well, the internal functions (i.e. adding two numbers) are of course general c++, so you can reuse them in other projects, but yes, the created library will be made to work with python, and not something else.
I have a platform for python scripting, and I would like to call matlab functions inside. I have found several threads tackling the issue, among them those two
How do I interact with MATLAB from Python?
Running m-files from Python
However, threads are either not recent, or not very detailed.
is mlabwrap reliable?
what would you advocate as a solution to call matlab functions / .m files in python script?
using win32com from python to call a matlab session --> is this a good idea ? could you point to more doc or examples on this topic?
looks like link to sourceForge is not up to date, last update 2010,
http://sourceforge.net/projects/mlabwrap/
Could you hint at some latest version ?
Thanks
I would still recommend mlabwrap as the solution for this.
I use mlabwrap on a regular (weekly?) basis, on both Linux and Windows, across several different versions of Python, and several different versions of Matlab. To answer your specific questions:
mlabwrap will reliably perform across platforms, Python, and Matlab versions. It does however have limitations, and it will reliably fail when pushed past those limitations. Usually, these can be worked around.
See my answer here for more information on calling Matlab functions vs. Matlab scripts through mlabwrap. This answer also describes how to workaround one of the primary limitations of mlabwrap, which is that not all Matlab objects can be directly converted into Python objects.
I don't know anything about calling Matlab using win32com.
I have used mlabwrap in what I'll term 'Python-primary' style, where the majority of the programming in Python, using Matlab as a library for specific mathematical functions that aren't available in scipy/numpy, and in a 'Matlab-primary' style, the majority of the programming is in Matlab, and the final results are importedinto Python for use in some external process.
For Python-primary, the thing to keep in mind is that not all Matlab functions will return Python-readable data. mlabwrap will return a MLabObjectProxy object from these functions. These commonly occur when you use Matlab functions to create objects that are passed into other Matlab functions to actually process the data. For example, you can use the digital signal processing toolbox to create a Welch spectrum object which you can then use to get the power spectrum of your data. Theoretically, you can pass these MLabObjectProxies into Matlab functions that require them. In my experience the more you pass these back and forth, the more likely you are to find a bug in mlabwrap. What you can do instead is write a simple Matlab wrapper function obtains the object, processes the data, and then returns appropriate output as an array.
You can also get around problems with the MLabObjectProxies by using the low-level commands in mlabwrap. For example, if I have a matlab_struct that is a struct array with field matlab_struct.label, and I only want the labels on the Python side, I can do the following:
# place matlab_struct into the Matlab workspace
mlab._set('matlab_struct', matlab_struct)
# convert the labels into a cell array
matlab_struct_labels = mlab.eval('{matlab_struct.labels}')
The main low-level commands available are mlab._set('variable_name', variable), mlab.eval('command string'), and mlab.get('variable_name').
If I'm doing a lot of heavy-duty processing in Matlab, say in a toolbox or plugin that isn't available elsewhere, I'll write what I call 'Matlab-primary' code, where I try to avoid passing data back and forth through mlabwrap, instead manipulating variables in the Matlab workspace by calling .m scripts, saving the resulting output to a data file, and importing that into my Python code.
Good luck!
mlabwrapper
is a great solution for python <-> MATLAB bridging. If something is not working for you just report concrete problems on SO :)
You have to note that mlabwrapper as project has been around for quite a while. http://mlabwrap.sourceforge.net/
I had problems with mlabwrap.cpp recently, for which I found the following github fork
Related Thereis is a copy of mlabwrap v1.1-pre
(http://mlabwrap.sourceforge.net/) patched as described here:
http://sourceforge.net/mailarchive/message.php?msg_id=27312822
with a patch fixing the error:
mlabraw.cpp:225: error: invalid conversion from ‘const mwSize*’ to
‘const int*’ Also note that in Ubuntu you need to sudo apt-get install
csh
For details see http://github.com/aweinstein/mlabwrap
mlab
After spending more time, I made a github mirror to update, bugfix and maintain the wrapper https://github.com/ewiger/mlab (patches and pull-requests are welcomed!)
It can be pip installed, i.e.
pip install mlab
I have excluded the cpp implementation for now. In the current way it works as the following:
For Linux/Mac library creates a pipe connection with MATLAB instance. The rest is serialization (partially pointed out by #brentlance), which is done using numpy.
For Windows library uses DCOM to communicate. (But I am still to fix version lookup using registry).
When to use mlab?
I would recommend to call very high level user-functions in MATLAB (mostly returning logical results or very standard built-in types as matrices) to minimize any communication with MATLAB. This approach is perfect for legacy code, but might require writing some wrapping interfaces to simplify function declarations.
Overall, the code is a bit cumbersome and is a patchwork of many. The core part (now this is matlabpipe and matlabcom) seems to do the job pretty well. Ultimately, I would not recommend mlab for full-scale productive application unless you are willing to spend time testing, bug-reporting, bug-fixing and feature-requesting all of your use-cases.
To answer your questions:
We used mlabwrap and it was reliable. The only difficulty was to compile it. We had a few issues with that. However, once you compile it, it runs well.
I would recommend matlab_wrapper, because it's much easier to install (pure Python, no compilation), supports various data structures (numeric, logical, struct, cell arrays) and runs on GNU/Linux, Windows, OSX.
Using win32com to communicate with MATLAB should work, but it's quite low-level and not portable.
Disclaimer: I'm the author of matlab_wrapper.
Since MATLAB R2014b, there is now a MATLAB API for Python. See the MATLAB documentation for more information. It however requires to start the MATLAB engine which can take some time.
I have two questions:
1)
I want to call a C library from a Python application. I don't want to wrap the whole API, I want only the functions and datatypes that are relevant to my purpose.
As I see it, I have two choices:
Use Cython to expose the relevant parts from the C library to Python.
Do the whole thing in Python, using ctypes to communicate with the external library.
I'm not sure whether 1) or 2) is the better choice.
We may have some changes in the function prototypes or need to support few more functions over time.So,I thought using cython or some other tool might be better.
Are there more advantages / disadvantages with either choice? Which approach do you recommend?
2).Also I am newbie to Python and cython. If possible can somebody give some example on how to call c library from python app? I browsed over net, but could not get any example working. I might be missing some thing.
Any help will be really appreciated.
There is a project I'm about to build in Smalltalk (Pharo). And there is a python library which I intend to use for the same. Now, there are 3 options:
Smalltalk wrapper for those python libraries
Porting the python library to Smalltalk
Write the library from scratch (in Smalltalk) for use in my project
The following are my queries:
What are 'basic' differences in porting/wrapping (No satisfactory explanation found anywhere yet)
How to know when to use which (of all the three)?
Any resources or pointers where I can get further help/some kick-start into my project.
Thanks!
Wrapper
Write functions in the native language whose sole purpose is to call the functions in the external library. The goal is to do as little as possible in the native language. For example, translating data types from the native language to the external library language, etc.
Wrappers make sense when the external library is:
written in a more efficient language than the native code (eg, a C++ library called from Python)
large/complex and would be time-consuming or error prone to translate
regularly updated; in a well-maintained library the interfaces (what your wrapper is concerned with) will change less often than the implementation of the features; so if you have wrappers around the functionality, updating to a new version of the library should be fairly straightforward
Porting
A port is simply a translation from one language to another. In general, the same logic is maintained as much as possible.
Porting makes sense when:
the native language is more efficient than the external library
the library is simple and one wants to save on the overhead involved with wrapping
one intends to make and maintain changes to the ported code in the native language
there are no plans to use the external library in its own language
one wants to learn one or both of the languages involved
Re-Write
Think of a Re-Write as a Port with a lot of refactoring. The goal is to take advantage of features of the native language to improve the library in some way (efficiency, readability, etc.)
Re-Writing makes sense in all of the same scenarios as porting. Deciding whether to do a simple port or a full re-write usually comes down to one question:
Is there a better way to implement the features of the external library in the native language?