choosing best approach to call c library from python app - python

I have two questions:
1)
I want to call a C library from a Python application. I don't want to wrap the whole API, I want only the functions and datatypes that are relevant to my purpose.
As I see it, I have two choices:
Use Cython to expose the relevant parts from the C library to Python.
Do the whole thing in Python, using ctypes to communicate with the external library.
I'm not sure whether 1) or 2) is the better choice.
We may have some changes in the function prototypes or need to support few more functions over time.So,I thought using cython or some other tool might be better.
Are there more advantages / disadvantages with either choice? Which approach do you recommend?
2).Also I am newbie to Python and cython. If possible can somebody give some example on how to call c library from python app? I browsed over net, but could not get any example working. I might be missing some thing.
Any help will be really appreciated.

Related

Advise needed for Static vs Dynamic linking

I have a Python code that needs to be able to execute a C++ code. I'm new to the idea of creating libraries but from what I have learned so far I need to know whether I need to use static or dynamic linking.
I have read up on the pros and cons of both but there is a lot of jargon thrown around that I do not understand yet and since I need to do this ASAP I was wondering if some light can be shed on this from somebody who can explain it simply to me.
So here's the situation. My C++ code generates some text files that have data. My Python code then uses those text files to plot the data. As a starter, I need to be able to run the C++ code directly from Python. Is DLL more suitable than SL? Or am I barking up the completely wrong tree?
Extra: is it possible to edit variables in my C++ code, compile it and execute it, all directly from Python?
It depends on your desired deployment. If you use dynamic linking will need to carefully manage the libraries (.so, .dll) on your path and ensure that the correct version is loaded. This can be helped if you include the version number in the filename, but then that has its own problems (security... displaying version numbers of your code is a bad idea).
Another benefit is that you can swap your library functionality without a re-compile as long as the interface does not change.
Statically linking is conceptually simpler and practically simpler. You only have to deploy one artefact (an .exe for example). I recommend you start with that until you need to move to the more complicated shared library setup.
Edit: I don't understand your "extra credit" question. What do you mean by "edit values"? If you mean can you modify variables that were declared in your C++ code, then yes you can as long as you use part of the public interface to do it.
BTW this advice is for the general decision. If you are linking from Python to C/C++ I think you need to use a shared library. Not sure as I haven't done it myself.
EDIT: To expand on "public interface". When you create a C++ library of whatever kind, you specify what functions are available to outside classes (look up how to to that). This is what I mean by public interface. Parts of your library are inaccessible but others (that you specify) are able to be called from client code (i.e. your python script). This allows you to modify the values that are stored in memory.
If you DO mean that you want to edit the actual C++ code from within your python I would suggest that you should re-design your application. You should be able to customise the run-time behaviour of your C++ library by providing the appropriate configuration.
If you give a solid example of what you mean by that we'll be able to give you better advice.
Yes it is possible!!
Try exploring subprocess module in python.
Following can be an example implementation of your scenario:
yourfile.cpp
#compilation
args = ['g++','-o','your_executable_name_with_path','yourfile.cpp_with_path']
your_compile = subprocess.Popen(args,stdin=subprocess.PIPE,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
output,compilation_error = your_compile.communicate()
your_compile.wait()
#successful compilation then there will be execuatble
if not compilation_error:
#execuation
args = ['your_executable_name_with_path'] #command to run a an execuatble
your_run = subprocess.Popen(args,stdin=subprocess.PIPE,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
your_code_output,runtime_error = your_run.communicate()
your_run.wait()
Further, you can tackle more cases and come up with an efficient design
I'm not quite sure how the idea of linking comes into what you are asking, but it sounds to me like you want to use something like SWIG, which allows you to create wrappers around C++ functions (and many other languages) which you can then call directly from your Python code.
Extra: is it possible to edit values in my C++ code, compile it and execute it directly from Python?
If I'm understanding this correctly, you want to use Python to change your C++ code, then compile and execute it? If this is the case, you may want to look into embedding the Python interpreter in your C++ program. This would mean doing things the other way around and having C++ run your Python script, instead of trying to do everything from Python.

How to make a C++ library for Python

I am new to object oriented programming and I am struggling to find a good tutorial on how to make a library in C++ that I can import into Python.
At the moment I am just trying to make a simple example that adds two numbers. I am confused about the process. Essentially I want to be able to do something like this in Python:
import MyCPPcode
MyCPPcode.Add(5,3) #function prints 5+3=8
I am not requesting a full example with code, just the steps that I need to take.
Do I need to make a .dll or a static library? I am using MS Visual Studio 2013.
Also, does the process tailor the C++ library code for Python in any way or will this library be available for other languages as well?
While I cannot guide you through the whole process, because I do not know python too well, Here is what I know:
It is absolutely possible. While not being something for someone who is new to object-oriented programing, it's called the python-C/C++ API. If you search for that in the python documentation there are several chapters about it.
While the example function you're showing might look like that from python, the process is a lot more redundant in c++ (behind the scenes). There are tools that combat that issue, for example Cython, but if you want to learn I'd suggest going pure python API.
As for availability with other languages... Well, the internal functions (i.e. adding two numbers) are of course general c++, so you can reuse them in other projects, but yes, the created library will be made to work with python, and not something else.

Maintainability of a python wrapping of a C library

I have a poorly designed and big (> 300 public functions, >200 numeric constants defined with #define in the header file) that I have to wrap in Python. I have the dll and the h file. The library is updated yearly, till now in a backwards compatible way (i.e. just functions were added, a constant keep their numerical values, etc). But I have no guarantees as I do not control the library.
Using ctypes, I see two ways of wrapping this in Python:
Mapping every constant and function to python, 1 to 1
Redefining the API in Python and making calls to the library.
The first can be done in a (roughly) automatic way from the header file and therefore is easier to maintain and upgrade, the second requires a lot of python code but it will be easier to use.
I would appreciate some opinions based on your experience with this type of problem and some examples.
I recently used ctypesgen to create a ctypes wrapping for SDL, and complementary libraries (SDL_image, SDL_ttf, SDL_mixer).
For me, it worked fairly well. It generates Python 2.x, but I was able to get the desired 3.x code by using the "2to3" utility.
I think it's a good idea to use the ctypes wrapping as a foundation for a more "pythonic" api, and that's basically what I did (on a very simple level) with my pslab module.
So, if you're looking to do something similar, that would be one way.
Maintaining a Python library with a ctypes backend isn't an unmanageable approach. Obviously the initial investment is larger than using automated tools, but the API you are left with should be much better.
If you do take that route, aim to separate the Python API entirely from the C library though. Supporting multiple ctypes backends with one Python front end api isn't too bad - just query at runtime and dynamically load the correct ctypes wrapper module. I've done that to wrap different dll files and .so files for windows and linux but it would work for versions of a library as well.

Best way to build an application based on R?

I'm looking for suggestions on how to go about building an application that uses R for analytics, table generation, and plotting. What I have in mind is an application that:
displays various data tables in different tabs, somewhat like in Excel, and the columns should be sortable by clicking.
takes user input parameters in some dialog windows.
displays plots dynamically (i.e. user-input-dependent) either in a tab or in a new pop-up window/frame
Note that I am not talking about a general-purpose fron-end/GUI for exploring data with R (like say Rattle), but a specific application.
Some questions I'd like to see addressed are:
Is an entirely R-based approach even possible ( on Windows ) ? The following passage from the Rattle article in R-Journal intrigues me:
It is interesting to note that the
first implementation of Rattle
actually used Python for implementing
the callbacks and R for the
statistics, using rpy. The release
of RGtk2 allowed the interface el-
ements of Rattle to be written
directly in R so that Rattle is a
fully R-based application
If it's better to use another language for the GUI part, which language is best suited for this? I'm looking for a language where it's relatively "painless" to build the GUI, and that also integrates very well with R. From this StackOverflow question How should I do rapid GUI development for R and Octave methods (possibly with Python)? I see that Python + PyQt4 + QtDesigner + RPy2 seems to be the best combo. Is that the consensus ?
Anyone have pointers to specific (open source) applications of the type I describe, as examples that I can learn from?
There are lots of ways to do this, including the python approach you mention. If you want to do it solely within R and if your aims are modest enough, the gWidgets package can be used. This exposes some of the features of either RGtk2, tcltk or qtbase (see the qtinterfaces project on r-forge) in a manner that is about as painless as can be. If you want more, look at using those packages directly. I'd recommend RGtk2 if you are going to share with others and if not, qtbase or tcltk.
Python + Qt4 + RPy = Much Win.
For example, see what Carson Farmer has done with Qgis and the ManageR plugin - its a full R interface to geographic data in the Qgis mapping package.
Depending on how much statistical functionality you need you might even get away without needing it at all, doing all the stats in Python, leveraging such goodies as the Numpy numeric package and the Qwt plotting canvas.
How about traditional LAMP + a R backend? Optionally s/MySQL/Postgres and optionally s/PHP/Perl Rapache looks pretty cool too: rapache.net
If you go for C++, take a look at rcpp and Rinside
Java can be combined with R using JRI
RServe gives you a TCP/IP protocol to interact with R. There's a Java client and a C++ client, so either of them can be used.
On a sidenote: Another thing you should be aware of, is that R contains quite some libraries written in Fortran and C, that can be called directly. Same goes for more advanced packages like VGAM, they also contain quite some C routines. Depending on what exactly you want to do, you might try to work with those ones, just to avoid the overhead of the R functions itself.
I've been looking for an overview of those myself, but AFAIK you'll have to do some effort to get everything. Some things you certainly should look at is The R language definition and R Internals.

Does one often use libraries outside the standard ones?

I am trying to learn Python and referencing the documentation for the standard Python library from the Python website, and I was wondering if this was really the only library and documentation I will need or is there more? I do not plan to program advanced 3d graphics or anything advanced at the moment.
Edit:
Thanks very much for the responses, they were very useful. My problem is where to start on a script I have been thinking of. I want to write a script that converts images into a web format but I am not completely sure where to begin. Thanks for any more help you can provide.
For the basics, yes, the standard Python library is probably all you'll need. But as you continue programming in Python, eventually you will need some other library for some task -- for instance, I recently needed to generate a tone at a specific, but differing, frequency for an application, and pyAudiere did the job just right.
A lot of the other libraries out there generate their documentation differently from the core Python style -- it's just visually different, the content is the same. Some only have docstrings, and you'll be best off reading them in a console, perhaps.
Regardless of how the other documentation is generated, get used to looking through the Python APIs to find the functions/classes/methods you need. When the time comes for you to use non-core libraries, you'll know what you want to do, but you'll have to find how to do it.
For the future, it wouldn't hurt to be familiar with C, either. There's a number of Python libraries that are actually just wrappers around C libraries, and the documentation for the Python libraries is just the same as the documentation for the C libraries. PyOpenGL comes to mind, but it's been a while since I've personally used it.
As others have said, it depends on what you're into. The package index at http://pypi.python.org/pypi/ has categories and summaries that are helpful in seeing what other libraries are available for different purposes. (Select "Browse packages" on the left to see the categories.)
One very common library, that should also fit your current needs, is the Python Image Library (PIL).
Note: the latest version is still in beta, and available only at Effbot site.
If you're just beginning, all you'll need to know is the stuff you can get from the Python website. Failing that a quick Google is the fastest way to get (most) Python answers these days.
As you develop your skills and become more advanced, you'll start looking for more exciting things to do, at which point you'll naturally start coming across other libraries (for example, pygame) that you can use for your more advanced projects.
It's very hard to answer this without knowing what you're planning on using Python for. I recommend Dive Into Python as a useful resource for learning Python.
In terms of popular third party frameworks, for web applications there's the Django framework and associated documentation, network stuff there's Twisted ... the list goes on. It really depends on what you're hoping to do!
Assuming that the standard library doesn't provide what we need and we don't have the time, or the knowledge, to implement the code we reuse 3rd party libraries.
This is a common attitude regardless of the programming language.
If there's a chance that someone else ever wanted to do what you want to do, there's a chance that someone created a library for it. A few minutes Googling something like "python image library" will find you what you need, or let you know that someone hasn't created a library for your purposes.

Categories