I'm working with an API which is distributed as a dll/so file that I need to dynamically link with my python program. To accomplish this, I want to use Cython.
I have been able to, in the past, link with the dll statically. This works well, except that the API comes in four different flavors, with theoretically infinitely more to come and the users should be able to compile them with whatever name they want (Kinda like a plugin system). Because of that, I can't just make an so/pxd file that statically links with just one library, or even links with a selection of them.
What I need is to be able to pass the so/dll name into the Cython code and have it "import" it. I know this can be done with ctypes via ctypes.cdll.LoadLibrary, but is this kind of thing possible in Cython? Am I going to have to use ctypes to do this?
I assume you are talking about writing C modules here. If so yes you can. I don't know what the equivalent on Windows is, but in Linux you can use dlopen and friends. There is a man page for it, and several web sites documenting it. Try this link "http://linux.die.net/man/3/dlopen" it provides you with a nice example near the bottom of the page. This is doing the same thing as what ctypes does, in fact I think this might even be what ctypes uses.
Related
I have a Python code that needs to be able to execute a C++ code. I'm new to the idea of creating libraries but from what I have learned so far I need to know whether I need to use static or dynamic linking.
I have read up on the pros and cons of both but there is a lot of jargon thrown around that I do not understand yet and since I need to do this ASAP I was wondering if some light can be shed on this from somebody who can explain it simply to me.
So here's the situation. My C++ code generates some text files that have data. My Python code then uses those text files to plot the data. As a starter, I need to be able to run the C++ code directly from Python. Is DLL more suitable than SL? Or am I barking up the completely wrong tree?
Extra: is it possible to edit variables in my C++ code, compile it and execute it, all directly from Python?
It depends on your desired deployment. If you use dynamic linking will need to carefully manage the libraries (.so, .dll) on your path and ensure that the correct version is loaded. This can be helped if you include the version number in the filename, but then that has its own problems (security... displaying version numbers of your code is a bad idea).
Another benefit is that you can swap your library functionality without a re-compile as long as the interface does not change.
Statically linking is conceptually simpler and practically simpler. You only have to deploy one artefact (an .exe for example). I recommend you start with that until you need to move to the more complicated shared library setup.
Edit: I don't understand your "extra credit" question. What do you mean by "edit values"? If you mean can you modify variables that were declared in your C++ code, then yes you can as long as you use part of the public interface to do it.
BTW this advice is for the general decision. If you are linking from Python to C/C++ I think you need to use a shared library. Not sure as I haven't done it myself.
EDIT: To expand on "public interface". When you create a C++ library of whatever kind, you specify what functions are available to outside classes (look up how to to that). This is what I mean by public interface. Parts of your library are inaccessible but others (that you specify) are able to be called from client code (i.e. your python script). This allows you to modify the values that are stored in memory.
If you DO mean that you want to edit the actual C++ code from within your python I would suggest that you should re-design your application. You should be able to customise the run-time behaviour of your C++ library by providing the appropriate configuration.
If you give a solid example of what you mean by that we'll be able to give you better advice.
Yes it is possible!!
Try exploring subprocess module in python.
Following can be an example implementation of your scenario:
yourfile.cpp
#compilation
args = ['g++','-o','your_executable_name_with_path','yourfile.cpp_with_path']
your_compile = subprocess.Popen(args,stdin=subprocess.PIPE,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
output,compilation_error = your_compile.communicate()
your_compile.wait()
#successful compilation then there will be execuatble
if not compilation_error:
#execuation
args = ['your_executable_name_with_path'] #command to run a an execuatble
your_run = subprocess.Popen(args,stdin=subprocess.PIPE,stderr=subprocess.PIPE,stdout=subprocess.PIPE)
your_code_output,runtime_error = your_run.communicate()
your_run.wait()
Further, you can tackle more cases and come up with an efficient design
I'm not quite sure how the idea of linking comes into what you are asking, but it sounds to me like you want to use something like SWIG, which allows you to create wrappers around C++ functions (and many other languages) which you can then call directly from your Python code.
Extra: is it possible to edit values in my C++ code, compile it and execute it directly from Python?
If I'm understanding this correctly, you want to use Python to change your C++ code, then compile and execute it? If this is the case, you may want to look into embedding the Python interpreter in your C++ program. This would mean doing things the other way around and having C++ run your Python script, instead of trying to do everything from Python.
I have a poorly designed and big (> 300 public functions, >200 numeric constants defined with #define in the header file) that I have to wrap in Python. I have the dll and the h file. The library is updated yearly, till now in a backwards compatible way (i.e. just functions were added, a constant keep their numerical values, etc). But I have no guarantees as I do not control the library.
Using ctypes, I see two ways of wrapping this in Python:
Mapping every constant and function to python, 1 to 1
Redefining the API in Python and making calls to the library.
The first can be done in a (roughly) automatic way from the header file and therefore is easier to maintain and upgrade, the second requires a lot of python code but it will be easier to use.
I would appreciate some opinions based on your experience with this type of problem and some examples.
I recently used ctypesgen to create a ctypes wrapping for SDL, and complementary libraries (SDL_image, SDL_ttf, SDL_mixer).
For me, it worked fairly well. It generates Python 2.x, but I was able to get the desired 3.x code by using the "2to3" utility.
I think it's a good idea to use the ctypes wrapping as a foundation for a more "pythonic" api, and that's basically what I did (on a very simple level) with my pslab module.
So, if you're looking to do something similar, that would be one way.
Maintaining a Python library with a ctypes backend isn't an unmanageable approach. Obviously the initial investment is larger than using automated tools, but the API you are left with should be much better.
If you do take that route, aim to separate the Python API entirely from the C library though. Supporting multiple ctypes backends with one Python front end api isn't too bad - just query at runtime and dynamically load the correct ctypes wrapper module. I've done that to wrap different dll files and .so files for windows and linux but it would work for versions of a library as well.
I am trying to develop a chrome extension in which i have coded all my
logic in 'python' . Now on browser action i want to pass parameters
and execute that .py file and return results to popup which will open
on browser action. To call a .py file from JavaScript i know i will
need to code a NPAPI plugin. But am confused between which approach i
should take. I have come across few options and am trying to choose
the easiest way to do it ..
Pyjamas Python Javascript Compiler : is a Python-to-Javascript
compiler which works as a language translator but on FAQ's of there
site last question suggest it will not run on Chrome. ( http://pyjs.org/)
FireBreath : FireBreath is a framework that allows easy creation of
powerful browser plugins. ( http://www.firebreath.org )
pyplugin - Python NPAPI plugin for XULRunner : It allows you to
build cross-platform graphical user interfaces using XUL and Python.
( http://pyplugin.com )
Plz Guide me to easiest way which will allow me to pass parameters and
execute that .py file and receive returned results from .py.
Thanx
Well, Pyjamas Python Javascript Compiler will not be complete -- not all python features are available in javascript, so it's impossible to convert all python to javascript. This may or may not do what you want, but I don't think it happens "on the fly", I think you have to write things on the desktop and run it through the "compiler" to get javascript out the other side.
FireBreath is the most awesomely amazing thing to ever hit the Internet -- I should know since I wrote it -- and it will absolutely allow you to do what you want, but you'll have to know how to tie into Python with C++ in order to do what you want. That said, you could probably use boost.python, which is included in the subset of boost that comes with FireBreath, but I've never used it so I don't know. You can pretty much do anything you want with an NPAPI plugin but you'll want to be real careful of security concerns.
A quick glance at pyplugin makes it look like pyplugin is basically what you'd be writing in FireBreath, but just a raw npapi plugin. If this will do what you want, it's probably the easiest way to go. It's designed to be used with XUL, which may be a problem since Chrome doesn't support XUL. You might also be able to modify it (since it's GPL) to do what you want. Of course, if you weren't planning to release your source, that could also be a problem.
The quickest way to solve your problem? Well, you'll have to decide; it'll take some more research, but I hope this is enough to at least get you started. Good luck!
scenario: a modular app that loads .py modules on the fly as it works. programmer (me) wishes to edit the code of a module and then re-load it into the program without halting execution.
can this be done?
i have tried running import a second time on an updated module.py, but the changes are not picked up
While reload does reload a module, as the other answer mentions, you need quite a few precautions to make it work smoothly -- and for some things you might believe would work easily, you're in for quite a shock in terms of amount of work actually needed.
If you ever use the form from module import afunction, then you've almost ensured reload won't work: you must exclusively import modules, never functions, classes, etc, from inside modules, if you want to have any hope of reload doing something useful all all (otherwise you'd have to somehow chases all the bits and pieces imported here and there from the module, and rebind each and every one of them -- eep;-). Note that I prefer following this rule anyway, whether I plan to do any reloading or not, but, with reload, it's crucial.
The difficult problem is: if you have, alive anywhere, instances of classes that existed in the previous version of the module, reload per se will do absolutely nothing to upgrade those instances. That problem is a truly hard one; one of the longest, hardest recipes in the Python Cookbook (2nd edition) is all about how to code your modules to support such "reload that actually upgrades existing instances". This only matters if you program in OOP style, of course, but... any Python program complex enough to need "reload this plugin" functionality is very likely to have lots of OOP in it, so it's hardly a minor issue.
The docs for reload are pretty complete and do mention this issue, but give no hint how to solve it. This recipe by Michael Hudson, from the Python Cookbook online, is better, but it's only the start of what we evolved into the printed (2nd edition) -- recipe 20.15, the online version of that is here (incomplete unless you sign up for a free time-limited preview of O'Reilly's commercial online books service).
reload() will do what you need. You just have to be careful about what code executes in your module upon re-compile; it is usually easiest if your module just contains definitions.
use the builtin command:
reload(module)
How do I unload (reload) a Python module?
You may get some use from the livecoding module.
Alex's answer and the others cover the general case.
However, since these modules you're working on are your own and you'll edit only them, it's possible to implement some kind of a local reloading mechanism rather than rely on the builtin reload. You can architect your module loader as something that loads up the files, compiles it and evals them into a specific namespace and tells the calling code that it's ready. You can take care of reloading by making sure that there's a decent init function which reinitialises the module properly. I assume you'll be editing only these.
It's a bit of work but might be interesting and worth your time.
I am trying to learn Python and referencing the documentation for the standard Python library from the Python website, and I was wondering if this was really the only library and documentation I will need or is there more? I do not plan to program advanced 3d graphics or anything advanced at the moment.
Edit:
Thanks very much for the responses, they were very useful. My problem is where to start on a script I have been thinking of. I want to write a script that converts images into a web format but I am not completely sure where to begin. Thanks for any more help you can provide.
For the basics, yes, the standard Python library is probably all you'll need. But as you continue programming in Python, eventually you will need some other library for some task -- for instance, I recently needed to generate a tone at a specific, but differing, frequency for an application, and pyAudiere did the job just right.
A lot of the other libraries out there generate their documentation differently from the core Python style -- it's just visually different, the content is the same. Some only have docstrings, and you'll be best off reading them in a console, perhaps.
Regardless of how the other documentation is generated, get used to looking through the Python APIs to find the functions/classes/methods you need. When the time comes for you to use non-core libraries, you'll know what you want to do, but you'll have to find how to do it.
For the future, it wouldn't hurt to be familiar with C, either. There's a number of Python libraries that are actually just wrappers around C libraries, and the documentation for the Python libraries is just the same as the documentation for the C libraries. PyOpenGL comes to mind, but it's been a while since I've personally used it.
As others have said, it depends on what you're into. The package index at http://pypi.python.org/pypi/ has categories and summaries that are helpful in seeing what other libraries are available for different purposes. (Select "Browse packages" on the left to see the categories.)
One very common library, that should also fit your current needs, is the Python Image Library (PIL).
Note: the latest version is still in beta, and available only at Effbot site.
If you're just beginning, all you'll need to know is the stuff you can get from the Python website. Failing that a quick Google is the fastest way to get (most) Python answers these days.
As you develop your skills and become more advanced, you'll start looking for more exciting things to do, at which point you'll naturally start coming across other libraries (for example, pygame) that you can use for your more advanced projects.
It's very hard to answer this without knowing what you're planning on using Python for. I recommend Dive Into Python as a useful resource for learning Python.
In terms of popular third party frameworks, for web applications there's the Django framework and associated documentation, network stuff there's Twisted ... the list goes on. It really depends on what you're hoping to do!
Assuming that the standard library doesn't provide what we need and we don't have the time, or the knowledge, to implement the code we reuse 3rd party libraries.
This is a common attitude regardless of the programming language.
If there's a chance that someone else ever wanted to do what you want to do, there's a chance that someone created a library for it. A few minutes Googling something like "python image library" will find you what you need, or let you know that someone hasn't created a library for your purposes.