I am developing a library for 3D and environmental audio in C++, which I wish to use from Python and a myriad of other languages. I wish it to work with Kivy on the iPhone down the line, so I want to use Cython instead of ctypes. To that end, I have already constructed and exposed a C-only api that uses only primitive types, i.e. no structs. I would ideally like to have only one implementation of the Python binding, and believe that on Mac/Linux this won't be a problem (I could be wrong, but that is off-topic for this question).
The problem is that it is using C++11 heavily, and consequently needs to compile with VC++ 2013. Python 2 is still typically compiled with VC++ 2008, and I believe Python 3 is only VC++ 2010.
I am familiar with the rules for dynamic runtime mixing: don't pass memory out and free it from the other side, never ever send handles back and forth, etc. I am following them. This means that I'm ctypes and cffi-ready. All functions are appropriately extern "c" and the like.
Can I use this with Cython safely, and if so how? My reading is giving me very mixed answers on whether or not it's safe to go use the export library produced by VC++ 2013 with VC++ 2008, and Cython seems to have no in-built functionality for dynamically linking (and that won't work on iOS anyway). The only thing I can think of that would work besides linking with the import library is to also bind the windows DLL manipulation functions and use those in Cython-but this defeats the one implementation goal pretty quickly.
I've already tried researching this myself, but there seems to be no good info; simply trying it will be inconclusive for weeks-linking is no guarantee of functionality, after all. This is the kind of thing that can very easily seem to work right without actually doing so; to that end, sources and links to further reading are much appreciated.
Related
What is the best/standard way to create Python interfaces for C++ libraries?
I know this question has been asked on here before but that was in 2008 and things may/likely have changed since then.
I've looked around and tested a few different methods but can't decide which is best. I've tried Swig, ctypes, and cppyy so far and think that cppyy is by far the easiest/fastest to implement. I've seen recommendations for Swig but it took a very long time to get Swig working and the results were not impressive. Is there a current standard? Why do people recommend Swig so much but I hear nothing of cppyy? Thank you.
SWIG has been around since February 1996 and supports a range of languages, not just Python. Although what is now cppyy started in February 2003, as RootPython, it has always been embedded with ROOT (http://root.cern.ch), and was not available standalone. A full, easy, installation with wheels on PyPI for all three major platforms only exists since March of this year and on conda-forge (for Linux & Mac) only since two months now. So, even though it has a long pedigree, within the wider Python world cppyy is really quite fresh, which is why I doubt many folks have heard of it yet, whereas SWIG is the (spiritual) ancestor of them all.
The reason for putting in the effort of making cppyy available, is that it offers quite a few features that other binders do not have, and would not be easy to add: a compliant C++17 parser (b/c of Clang/LLVM); automatic template instantiations, cross-inheritance and callbacks, all at run-time (b/c of Cling); and much better performance. It also does not create C extension modules, so you only need to recompile cppyy itself for different versions of Python, but none of the bound code.
Now, to your first question of what is the best. Well, it depends on the use case. For example, if you need more bindings than just Python, SWIG is your best bet. If you have lots of templates that you can not all instantiate at build time, need the performance and scaling, or have a C++ framework with lots of interfaces, then cppyy is hard to beat. If you have modern C++ and do not want any run-time dependency on external libraries, then PyBind11 is where it's at.
These days I can't recommend ctypes. The only real benefit is that it is a builtin module for most Pythons in the wild, but with the advent of PyPI and conda that has become moot. If you want a super-light C binder (not C++, but you can wrap those functions with C helpers), then go for CFFI.
As to your question of whether there is a standard: no, there is no one binder that is best for all use cases. There are even quite a few more than the ones you have mentioned, but many of them play in the same space (eg. SWIG vs. SIP, and PyBind11 vs. boost.python) and I wouldn't recommend them over the ones you already tried. I do want to point out AutoWIG, which is a generator utilizing Clang with PyBind11 or boost.python code as output; and cython, which is a Python-like code to write C extension modules and which has some (limited) C++ support. I've always felt that cython was neither here nor there, but lots of people like it, and it's used extensively in the scientific community and in math-heavy code, so that vouches for its quality.
Now, even though there is no "standard", all binders can convert proxies to PyCapsule objects and rebind them. So although it is a bit clunky at times, you can actually mix binders within one application.
One final point: CFFI and cppyy (through CFFI's backend) have near-native performance on PyPy. Unfortunately, cppyy isn't as up-to-date on PyPy as it is on CPython (e.g. cross-inheritance is still missing), but it's getting there. The other binders work through the Python C-API, which is fully functional on PyPy, but precludes the JIT from doing its work, with reduced performance as result.
Full disclaimer: I'm the author of cppyy, and these days I only use cppyy, CFFI, and PyBind11 for my binding needs.
I need to access data via USB from a beam profiler. I've tried using the USB module in python to access it, but unfortunately the company who makes this device "does not support development in Python". The project I am working on is to eventually create a GUI (via Python) to automate a motor and pull data from the device. So it has to be done in Python, or I'm going to have to discard the first half of the code and redo it in C++.
I think the reason the device can only interface with C/C++ is because of the header and library files that come with the driver download.
I've looked at Cython but am still very unsure how it can help me. I'm just trying to access the header files for the driver in python and somehow execute the C commands in python.
BTW I am using Anaconda (if that matters).
Thank-you for any clarification and help!
Check out boost.python
Here is an intro:
The Boost Python Library is a framework for interfacing Python and
C++. It allows you to quickly and seamlessly expose C++ classes
functions and objects to Python, and vice-versa, using no special
tools -- just your C++ compiler. It is designed to wrap C++ interfaces
non-intrusively, so that you should not have to change the C++ code at
all in order to wrap it, making Boost.Python ideal for exposing
3rd-party libraries to Python. The library's use of advanced
metaprogramming techniques simplifies its syntax for users, so that
wrapping code takes on the look of a kind of declarative interface
definition language (IDL).
It includes support for:
References and Pointers
Globally Registered Type Coercions
Automatic Cross-Module Type Conversions
Efficient Function Overloading
C++ to Python Exception Translation
Default Arguments
Keyword Arguments
Manipulating Python objects in C++
Exporting C++ Iterators as Python Iterators
Documentation Strings
and many more.
I'm trying to write a quick and dirty little program to poll a WASP-B ANT+ to wifi gateway on windows, using python (the language I'm most familiar with). The WASP-B device has an unpublished UDP protocol that is mixed up with NDA's and so forth, but they have a published .NET library for win32 programming. So ... I googled around a bit and found IronPython and python.NET. If I understand correctly, IronPython is a .NET application or something? Not really what I'm after, I just want to be able to use the "stuff" in the WASP-B's .NET library from within a python program, so I think python.NET is the better option for me.
As far as I can tell, python.NET isn't all that active though - does anyone here know if it's live, if it'll work with current (3.x) python on Windows and if I'm on the right track?
On sourceforge, python.NET hasn't been touched since Jan 2013 :
http://sourceforge.net/projects/pythonnet/
The WASP, for anyone interested :
http://www.npe-inc.com/products/products-wasp.html
If you're looking for a Python variant that plays very well with .NET, then I highly suggest Iron Python: http://ironpython.net/ it is an open-source variant of Python that was integrated with the .NET framework, allowing it to behave similarly to one of the .NET-friendly languages.
As far as being active, the last update was made on May 25, 2014, which is pretty recent. The updates are not frequent, but for the most part there is no need for them, beyond some improvements and fixes. I would suggest giving it a try and see if it works for you. It's hard to answer a question of something being the right thing for the job unless youn give it a try.
IronPython works wells within .NET environments (calling .NET assemblies). If you stay within .NET framework then IronPython is recommended. Note Python 3 port is in progress and the syntax is not supported yet (Jan 2015).
But CPython (main Python implementation) works well with Python.NET both for extending and embedding. The reason for using CPython with Python.NET is if you need libraries that call Python C-API (not supported by IronPython), like Numpy, Scipy, Pandas, Cython, Matplotlib, etc. Python 3 port is here:
http://www.lfd.uci.edu/~gohlke/pythonlibs/
https://github.com/renshawbay/pythonnet/
I'm building a rendering engine in Python for fun. I need to load 3D scenes. Any standard modern format like DAE, 3DS, or MAX would work: I can convert my files easily between standard formats.
OpenSceneGraph seems to be the most comprehensive and well-maintained solution. It would be ideal to be able to use it in Python without much hassle. Are there working Python bindings for OSG that are easy to install, work on Mac OS X (I'm on 10.8), and are compatible with the latest versions of OSG?
I searched around and came across osgswig (http://code.google.com/p/osgswig/) and PyOSG (http://sourceforge.net/projects/pyosg/), but they don't seem to be actively maintained. I don't see any recent activity related to these packages, and it seems that people had trouble running osgswig on OSX. Ideally, I'd like to find something that "just works", without major compilation hassles. I'd like to just install a package and be able to import a module that will let me load COLLADA or 3DS files.
I also came across pycollada (https://github.com/pycollada/pycollada). It seems active, but fairly early-stage. Ideally, I'd like a reasonably comprehensive package that supports specular maps, normal maps, and other reasonably advanced features. Animation would be nice as well.
In summary, I need to load 3D scenes in Python. Bindings for OSG would probably be ideal, because OSG is so comprehensive. But I need something that works on OSX. I would also prefer something that can be installed reasonably easily. Does something like this exist?
Thanks!
Take a look at Open Asset Import Library (short name: Assimp). It is a portable Open Source library to import various well-known 3D model formats in a uniform manner. http://www.assimp.org/
You should loot at panda3D (http://www.panda3d.org/), it's a game engine with extensive python bindings. It has the features you want : http://www.panda3d.org/manual/index.php/Features
I used it for a few years and it was a solid tool.
I made my own fork of a mirror of a clone of the osgswig project for a similar purpose. I have it working with OpenSceneGraph version 3.2.1 on Windows and Mac; and it's likely I will eventually polish it for linux too. I'm already delivering one product to customers based on my version of osgswig, and I'm considering making others. Find my fork here:
https://github.com/cmbruns/osgswig
If others show enough interest, I might be coaxed into creating binary installers for my version of the osgswig module, to make installation easier.
If you just want the easiest OpenSceneGraph bindings for OSG 3.2.1, you can stop reading this answer here. Read on for more of my thoughts for the future.
Though I am maintaining a fork of osgswig (as stated above), I sort of hate SWIG, and I would prefer to use bindings based on Boost.Python, rather than on SWIG. For large, complex C++ APIs, like OpenSceneGraph, Boost.Python can be much more elegant than SWIG, both for the API consumer, and for the binding maintainer (me, and me). I found one project using Boost.Python to wrap OSG, at https://code.google.com/p/osgboostpython/, but the developer is lovingly wrapping each part of the interface by hand, and has thus only completed a tiny fraction of the large OpenSceneGraph API.
Taking that Boost.Python based project as inspiration, I created yet another OpenSceneGraph Python binding project, at https://github.com/JaneliaSciComp/osgpyplusplus. Eventually, I want to use this osgpyplusplus project for all my python osg needs. And I would appreciate help in making it ready. Right now, osgpyplusplus suffers from the following weaknesses, compared to osgswig:
osgpyplusplus is not yet used in any working product
The build environment is tricky to set up, requiring both Boost.Python and Pyplusplus
I haven't paid much attention to osgpyplusplus recently, so it might rust away if I continue to ignore it.
Though osgpyplusplus probably wraps most of the OpenSceneGraph API, there are probably some important missing pieces that won't be identified until someone tries to develop a significant project with it.
It would be a lot of work for me to create a binary module installer for osgpyplusplus at this point, so please don't ask me to.
If I write a python script, anyone can simply point an editor to it and read it. But for programming written in C, one would have to use decompilers and hex tables and such. Why is that? I mean I simply can't open up the Safari web browser and look at its code.
Note: The author disavows a deep expertise in this subject. Some assertions may be incorrect.
Python actually is compiled into bytecode, which is what gets run by the python interpreter. Whenever you use a Python module, Python will generate a .pyc file with a name corresponding to the module. This is the equivalent of the .o file that's generated when you compile a C file.
So if you want something to disassemble, the .pyc file would be it :)
The process that Python goes through when compiling a module is pretty similar to what gcc or another C compiler does with C source code. The major difference is that it happens transparently as part of execution of the file. It's also optional: when running a non-module, i.e. an end-user script, Python will just interpret the code rather than compiling it first.
So really your question is "Why are python programs distributed as source rather than as compiled modules?" Or, put another way, "Why are C applications distributed as compiled binaries rather than as source code?"
It used to be very common for C applications to be distributed as source code. This was back before operating systems and their various subentities (i.e. linux distributions) became more established. Some distros, for example gentoo, still distribute apps as source code. Apps which are a bit more cutting edge or obscure are still distributed as source code for all platforms they target.
The reason for this is compatibility, and dependencies. The reason you can run the precompiled binary Safari on a Mac, or Firefox on Ubuntu Linux, is because it's been specifically built for that operating system, architecture (e.g. x86_64), and set of libraries.
Unfortunately, compilation of a large app is pretty slow, and needs to be redone at least partially every time the app is updated. Thus the motivation for binary distributions.
So why not create a binary distribution of Python? For one thing, as Aaron mentions, modules would need to be recompiled for each new version of the Python bytecode. But this would be similar to rebuilding a C app to link with a newer version of a dynamic library — Python modules are analogous in this sense to C libraries.
The real reason is that Python compilation is very much quicker than C compilation. This is in part, I think, because of the dynamic nature of the language, and also because it's not as thorough of a compilation. This has its tradeoffs: in particular, Python apps run much more slowly than do their C counterparts, because Python has to interpret the compiled bytecode into instructions for the processor, whereas the C app already contains such instructions.
That all being said, there is a program called py2exe that will take a Python module and distribution and build a precompiled windows executable, including in it the logic of the module and its dependencies, including Python itself. I guess the point of this is to avoid having to coerce people into installing Python on their Windows system just to run your app. Under linux, or I think even OS/X, Python is usually already installed, so precompilation is not really necessary. Linux systems also have super-dandy package managers that will transparently install dependencies such as Python if they are not already installed.
Python is a script language, runs in a virtual machine through an interpeter.
C is a compiled language, the code compiled to binary code which the computer can run without all that extra stuff Python needs.
This is sorta a big topic. You should look into your local friendly Computer Science curriculum, you'll find a lot of great stuff on this subject there.
The short answer is the Python is an "interpreted" language, which means that it requires a machine language program (the python interpreter) to run the python program, adding a layer of indirection. C or C++ are different. They are compiled directly to machine code, which runs directly on your processor.
There is a lot of additional voodoo to be learned here, however. Technically Python is compiled to a bytecode, and modern interpreters do more and more "Just in Time" compilation, so the boundaries between compiled and interpreted code are getting fuzzier all the time.
In several comments you asked: "Is it then possible to compile python to an executable binary file and then simply distribute that?"
From a theoretical viewpoint, there's no question the answer is yes -- a Python program could be compiled to, and distributed as, fully compiled machine code.
From a practical viewpoint, it's open to a lot more question. There are a few things like Unladen Swallow, Psyco, Shed Skin, and PyPy that you might want to know about though.
Unladen Swallow is primarily an attempt at making Python run faster, but part of the plan to do so involves using LLVM for its back-end. LLVM can (among other things) produce native machine code output. The last couple of releases of Unladen Swallow have used LLVM for native code generation, but 1) the most recent update on the web site is from late 2009, and 2) the release notes for that version say: "The Unladen Swallow team does not recommend wide adoption of the 2009Q3 release."
Psyco works as a plug-in for Python that basically does JIT compilation, so even though it can speed up execution (quite a lot in some cases), it doesn't produce a machine-code executable you can distribute. In short, while it's sort of similar to what you want, it's not intended to do exactly what you've asked for.
Shed Skin Python-to-C++ produces C++ as its output, and you then compile the C++ and (potentially) distribute the result of that. Shedskin is currently at version 0.5 -- i.e., nobody's claiming that it's a finished, released product. On the other hand, development is ongoing, and each release does seem to include pretty substantial improvements.
PyPy is a Python implementation written in Python. Their intent is to allow code production to be "plugged in" without affecting the rest of the implementation -- but while they currently support 4 different code generation models, I don't believe any of them results in producing native machine code that runs directly on the hardware.
Bottom line: work has been done and is being done with the intent of doing what you asked about, but at least to my knowledge there's not really anything I could reasonably recommend as a finished product that you can really depend on to do the job right now. The primary emphasis is really on execution speed, not producing standalone executables.
Yes, you can - it's called disassembling, and allows you to look at the code of Safari perfectly well. The thing is, C, among other languages, compiles to native code, i.e. code that your CPU can "understand" and execute.
More or less obviously, the level of abstraction present in the instruction set of your CPU is much smaller than that of a high level language like Python. The CPU instructions are not concerned with "downloading that URI", but more "check if that bit is set in a hardware register".
So, in conclusion, the level of complexity present in a native application is much higher when looking at the machine code, so many people simply can't make any sense of what is going on there, it's hard to get the big picture. With experience and time at your hands, it is possible though - people do it all the time, reversing applications and all.
you can't open up and read the code that actually runs for python either. Try
import dis
def foo():
for i in range(100):
print i
print dis.dis(foo)
That will show you the (human readable) bytcode of the foo program. equivalently, you can save the file and import it from the interactive python interpreter. This will create a .pyc file with the same basename as the script. open that with a hex editor and you are looking at the actually python bytecode.
The reason for the difference is that python changes up it's byte code between releases so that you would either need to distribute a different version of a binary only release for each version of python. This would be a pain.
With C, it's compiled to native code and so the byte code is much more stable making binary only releases possible.
because C code is complied to object (machine) code and python code is compiled into an intermediate byte code. I am not sure if you are even referring to the byte code of python - you must be referring to the source file itself which is directly executable (hiding the byte code from you!). C needs to be compiled and linked.
Python scripts are parsed and converted to binary only when they're run - i.e., they're text files and you can read them with an editor.
C code is compiled and linked to an executable binary file before they can be run. Normally, only this executable binary file is distributed - hence you need a decompiler. You can always view the source code, if you've access to it.
Not all C programs require decompilers. There's lots of C code distributed in source form. And some Python programs do require decompilers, if distributed as bytecode (.pyc files).
But, to the extent that your assumptions are valid, it's because C is a compiled language while Python is an interpreted language.
Python scripts are analogous to a man looking at a to-do list written in English (or language he understands). The man has to do all the work, every time that list of things has to be done.
If the man, instead of doing the steps on his own each time, creates and programs a robot which can carry out those steps again and again (and probably faster than him), that robot is analogous to the C program.
The man in the python case is called the "interpreter" and in the C case is called the "compiler", and the C robot is called the compiled program/executable.
When you look at the python program source, you see the to-do list. In case of the robot, you see the gears, motors and batteries, etc, which look very different from the to-do list. If you could get hold of the C "to-do" list, it looks somewhat like the python code, just in a different language.
G-WAN executes ANSI C scripts on the fly -making it just like Python scripts.
This can be server-side scripts (using G-WAN as a Web server) or any general-purpose C program and you can link any existing library.
Oh, and G-WAN C scripts are much faster than Python, PHP or Java...