We Keep Coding

Create a Python package from C++ library

I've been researching the topic of using C++ code in Python, but haven't found a generic clean flexible way to wrap C++ library in the Python package.
The question is whether it's possible to use existing complex C++ library to create a regular Python library, that can be called exactly like native Python libraries, such as NumPy or SciPy. If yes, any references would be much appreciated. If there are examples/tutorials available - it would be even more useful.
Thanks

There are many, many ways. Boost Python, http://www.boost.org/doc/libs/1_57_0/libs/python/doc/ , is very C++-specific and exploits C++ templates to the hilt (like all of Boost!-). Part of more general (less C++ specific) approaches include manual C coding of Python extensions, per https://docs.python.org/3/extending/extending.html ; SWIG, per http://www.swig.org/Doc1.3/SWIGPlus.html ; Cython, per http://docs.cython.org/src/userguide/wrapping_CPlusPlus.html ; ... and no doubt others I haven't come across yet.
The very existence of so many strong, actively maintained alternatives, hints that there's no "one size fits all" here! If you're a template wizard I bet you'll swear by Boost; if you're not, I guess you're more likely to swear at it -- and so on, and so forth.
Personally, I tend to end up using Cython (or even just ctypes!-) for experimenting, manual extension coding when I decide I want to do a lot of Python work using a certain C++ library (and performance is crucial) -- and SWIG at work, because that's the standard there. Haven't seriously used Boost in far too long -- a refresh on it goes on my not-so-tiny todo list for when my spare time gets more copious...:-).

Using Cython for game development?

How practical would it be to use Cython as the primary programming language for a game?
I am a experienced Python programmer and I absolutely love it, but I'm admittedly a novice when it comes to game programming specifically. I know that typically Python is considered too slow to do any serious game programming, which is why Cython is interesting to me. With Cython I can use a Python-like language with the speed of C.
I understand that I'll probably need to learn a bit of C/C++ anyway, but it seems like Cython would speed up development time quite a bit in comparison.
So, is it practical? And would I still be able to use C/C++ libraries like OpenGL, OpenAL, and Bullet Physics?

If you're working with a combination like that and your goal is to write a 3D game, you'd probably get better mileage out of a ready-made 3D engine with mature physics and audio bindings and a Python API like OGRE 3D or Panda3D. Even if you don't, this post about using Cython with Panda3D may be helpful.
I'm not sure about now, but back in 2007, the trade-off between the two was basically that:
Panda3D was better-documented and designed from the ground-up to be a C++ accelerated Python engine (apparently they made some API design decisions that don't occur to C++ engine projects) and, predictably, had a more mature Python API.
PyOgre was built on top of a much more advanced engine and had a larger and more vibrant community.
...however it's quite possible that may have changed, given that, passage of time aside, in 2007, Panda3D was still under a GPL-incompatible license and that drove off a lot of people. (Myself included)

I'm the developer for the Ignifuga Game Engine, it's 2D oriented and Python/Cython/SDL based. What I generally do is develop the code in Python, and then profile the engine to see if there are some obvious bottlenecks (the main loop, the rendering code are good candidates), and convert those modules to Cython. I then run all the code (Python and Cython based) through Cython, and compile it statically against SDL.
Another of Cython's big "pluses" is that binding to SDL, or any C based library, is almost trivial.
Regarding threads, the engine is currently single threaded with cooperative multitasking via Greenlets, though this comes from a design decision to mitigate potential multi threaded pitfalls that unexperienced developers might fall into, rather than a limitation on Cython's part.

at this date (12th of April 2011) unixmab83 is wrong.
Cython doesn't forbid the use of threads, you just needs to use the no_gil special statements.
Beside the bindins of c++ is now functional in cython.
We do use it for something which is close to gamedev. So while I cannot be final on this, cython is a valid candidate.

I've found that a lot of the time, especially for larger libraries, you wind up spending a tremendous amount of time just configuring the Cython project to build, knowing which structures to import, bridging the C code into Python in either direction etc. While Cython is a nice stopgap (and significantly more pleasant than pure C/C++ development), the amount of C++ you'd have to learn to effectively use it basically means you're going to have to bite the bullet and learn C++ anyway.
How about PyGame?

I know Cython and you do not have to know C/C++.
You will use static typing but very easy.
The hardest part is to get the compiling working, I think on Windows this is done over visual studio thing.
There is something like a standard library including math for example. The speed gain is not too big but this depends on your scope.
ctypes was much faster (pure C) but the connection to Python was very slow so that i decided to look for Cython which can still be dynamic.
For speed gain in a game Cython will be the right choice but i would name this performance also limited.

Threads!!! A good modern game must use threads. Cython practically forbids their use, holding to GIL (global interpreter lock) the entire time, making your code run in sequence.
If you are not writing a huge game, than Python/Cython is okay. But Cython is no good as a modern language without good thread support.

Coming from C, how should I learn Python? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I've got a good grasp on C, my first programming language. I know a reasonable number of tricks and techniques and have written quite a few programs, mostly for scientific stuff. Now I'd like to branch out and understand OOP, and Python seems like a good direction to go.
I've seen several questions on how to learn Python, but most of them were from folks who were looking to start programming for the first time. I don't need a tutorial that will tell me what a string is, but I do need one that can tell me how to make a string in Python. Any help on some good sources to look through? Bonus points if the source is free :)

I knew C before I knew Python. No offence intended, but I don't think that your C knowledge is that big a deal. Unless you read very, very slowly, just set out to learn Python. It won't take that long to skim through the material you're familiar with, and it's not as if a Python tutorial aimed at C programmers will make you a better Python programmer - it might teach you things in a different order, is all, and raise some specific things that you would do in C but that you should not do in Python.
Strings in Python actually are somewhat different from strings in C, and they're used differently. I strongly recommend learning them "from scratch", rather than thinking about them in terms of their differences from C strings. For one thing, in Python 2 it's best not to use Python's "string" class to represent strings: there's a separate unicode string class and for practical Python apps (pretty much anything involving user data), you need that. (Python 3 fixes this, making the str class a unicode string). You need to establish a good working practice for unicode/byte data and decode/encode.
A common mistake when learning a second programming language, is to think "I know how to program, I just need to translate what I do in C into Python". No, you don't. While it's true that an algorithm can be basically the same in different languages, the natural way to do a particular thing can be completely different in different languages. You will write better Python code if you learn to use Python idiomatically, than if you try to write Python like a C programmer. Many of the "tricks" you know that make sense in C will be either pointless or counter-productive in Python. Conversely many things that you should do happily in a typical Python program, like allocating and freeing a lot of memory, are things that in C you've probably learned to think twice about. Partly because the typical C program has different restrictions from the typical Python program, and partly because you just have to write more code and think harder to get that kind of thing right in C than you do in Python.
If you're learning the language because you urgently need to program a system/platform which has Python but doesn't have C, then writing Python programs that work like C programs is a reasonable interim measure. But that probably doesn't apply to you, and even if it did it's not the ultimate goal.
One thing you might be interested to look at because of your C experience, is the Python/C API. Python is great for many things, but it doesn't result in the fastest possible computational core of scientific apps [neither does C, probably, but let's not go into FORTRAN for now ;-)]. So if you're aiming to continue with scientific programming through your move in Python, and your programs are typically memory-bus- and CPU-bound doing immense amounts of number-crunching (billions of ops), then you might like to know how to escape into C if you ever need to. Consider it a last resort, though.
You do need to understand Python reasonably well before the Python/C API makes much sense, though.
Oh yes, and if you want to understand OOP in general, remember later on to take a look at something like Java, Objective-C, C++, or D. Python isn't just an OO language, it's a dynamic OO language. You might not realise it from comparing just C with Python, but dynamic vs static types is a completely independent issue from the OOP-ness of Python. Python objects are like hashtables that allow you to attach new fields willy-nilly, but objects in many other OO languages store data in ways which are much more like a C struct.

I learned everything I know about Python from the official documentation: http://docs.python.org/
And it's free.

dive into python is a good place to start
fire up an interpreter, IPython is even better than the plain Python interpreter
use dir() and help() to poke around
and don't forget to read through the official docs at least once

I'd recommend the book How to Think Like a Computer Scientist in Python. It really helped me get going in Python (now my favorite language) coming from Java, C and C++.

Diveintopython, official docs, "Learning python" by Mark Lutz(4th edition) is one of the best books.

As someone who has worked with Java for over 12 years, I found that picking up a problem and solving it in a new language is the best way to learn. I don't believe in reading - it wastes a huge amount of time, and you can easily end up reading for too long.
My advice is to find a problem and set off to solve it with Python. You will learn alot in the process.
Good luck

If you have a programming background, Python is pretty straightforward to pick up. The most onerous task is learning the libraries and idioms. The documentation at python.org is quite good and free. If you're doing number crunching you'll almost certainly want to become familiar with the numpy extension.

Is it reasonable to integrate python with c for performance?

I like to use python for almost everything and always had clear in my mind that if for some reason I was to find a bottleneck in my python code(due to python's limitations), I could always use a C script integrated to my code.
But, as I started to read a guide on how to integrate python. In the article the author says:
There are several reasons why one might wish to extend Python in C or C++, such as:
Calling functions in an existing library.
Adding a new builtin type to Python
Optimising inner loops in code
Exposing a C++ class library to Python
Embedding Python inside a C/C++ application
Nothing about performance. So I ask again, is it reasonable to integrate python with c for performance?

In my experience it is rarely necessary to optimize using C. I prefer to identify bottlenecks and improve algorithms in those areas completely in Python. Using hash tables, caching, and generally re-organizing your data structures to suit future needs has amazing potential for speeding up your program. As your program develops you'll get a better sense of what kind of material can be precalculated, so don't be afraid to go back and redo your storage and algorithms. Additionally, look for chances to kill "two birds with one stone", such as sorting objects as you render them instead of doing huge sorts.
When everything is worked to the best of your knowledge, I'd consider using an optimizer like Psyco. I've experienced literally 10x performance improvements just by using Psyco and adding one line to my program.
If all else fails, use C in the proper places and you'll get what you want.

* Optimising inner loops in code
Isn't that about performance ?

Performance is a broad topic so you should be more specific. If the bottleneck in your program involves a lot of networking then rewriting it in C/C++ probably won't make a difference since it's the network calls taking up time, not your code. You would be better off rewriting the slow section of your program to use fewer network calls thus reducing the time your program spends waiting on entwork IO. If your doing math intensive stuff such as solving differential equations and you know there are C librarys that can offer better performance then the way you are currently doing it in Python you may want to rewrite the section of your program to use those librarys to increase it's performance.

The C extensions API is notoriously hard to work with, but there are a number of other ways to integrate C code.
For some more usable alternatives see http://www.scipy.org/PerformancePython, in particular the section about using Weave for easy inlining of C code.
Also of interest is Cython, which provides a nice system for integrating with C code. Cython is used for optimization by some well-respected high-performance Python projects such as NumPy and Sage.
As mentioned above, Psyco is another attractive option for optimization, and one which requires nothing more than
import psyco
psyco.bind(myfunction)
Psyco will identify your inner loops and automatically substitute optimized versions of the routines.

C can definitely speed up processor bound tasks. Integrating is even easier now, with the ctypes library, or you could go for any of the other methods you mention.
I feel mercurial has done a good job with the integration if you want to look at their code as an example. The compute intensive tasks are in C, and everything else is python.

You will gain a large performance boost using C from Python (assuming your code is well written, etc) because Python is interpreted at run time, whereas C is compiled beforehand. This will speed up things quite a bit because with C, your code is simply running, whereas with Python, the Python interpreter must figure out what you are doing and interpret it into machine instructions.

I've been told for the calculating portion use C for the scripting use python. So yes you can integrate both. C is capable of faster calculations than that of python

Is Python good for big software projects (not web based)? [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Right now I'm developing mostly in C/C++, but I wrote some small utilities in Python to automatize some tasks and I really love it as language (especially the productivity).
Except for the performances (a problem that could be sometimes solved thanks to the ease of interfacing Python with C modules), do you think it is proper for production use in the development of stand-alone complex applications (think for example to a word processor or a graphic tool)?
What IDE would you suggest? The IDLE provided with Python is not enough even for small projects in my opinion.

We've used IronPython to build our flagship spreadsheet application (40kloc production code - and it's Python, which IMO means loc per feature is low) at Resolver Systems, so I'd definitely say it's ready for production use of complex apps.
There are two ways in which this might not be a useful answer to you :-)
We're using IronPython, not the more usual CPython. This gives us the huge advantage of being able to use .NET class libraries. I may be setting myself up for flaming here, but I would say that I've never really seen a CPython application that looked "professional" - so having access to the WinForms widget set was a huge win for us. IronPython also gives us the advantage of being able to easily drop into C# if we need a performance boost. (Though to be honest we have never needed to do that. All of our performance problems to date have been because we chose dumb algorithms rather than because the language was slow.) Using C# from IP is much easier than writing a C Extension for CPython.
We're an Extreme Programming shop, so we write tests before we write code. I would not write production code in a dynamic language without writing the tests first; the lack of a compile step needs to be covered by something, and as other people have pointed out, refactoring without it can be tough. (Greg Hewgill's answer suggests he's had the same problem. On the other hand, I don't think I would write - or especially refactor - production code in any language these days without writing the tests first - but YMMV.)
Re: the IDE - we've been pretty much fine with each person using their favourite text editor; if you prefer something a bit more heavyweight then WingIDE is pretty well-regarded.

You'll find mostly two answers to that – the religous one (Yes! Of course! It's the best language ever!) and the other religious one (you gotta be kidding me! Python? No... it's not mature enough). I will maybe skip the last religion (Python?! Use Ruby!). The truth, as always, is far from obvious.
Pros: it's easy, readable, batteries included, has lots of good libraries for pretty much everything. It's expressive and dynamic typing makes it more concise in many cases.
Cons: as a dynamic language, has way worse IDE support (proper syntax completion requires static typing, whether explicit in Java or inferred in SML), its object system is far from perfect (interfaces, anyone?) and it is easy to end up with messy code that has methods returning either int or boolean or object or some sort under unknown circumstances.
My take – I love Python for scripting, automation, tiny webapps and other simple well defined tasks. In my opinion it is by far the best dynamic language on the planet. That said, I would never use it any dynamically typed language to develop an application of substantial size.
Say – it would be fine to use it for Stack Overflow, which has three developers and I guess no more than 30k lines of code. For bigger things – first your development would be super fast, and then once team and codebase grow things are slowing down more than they would with Java or C#. You need to offset lack of compilation time checks by writing more unittests, refactorings get harder cause you never know what your refacoring broke until you run all tests or even the whole big app, etc.
Now – decide on how big your team is going to be and how big the app is supposed to be once it is done. If you have 5 or less people and the target size is roughly Stack Overflow, go ahead, write in Python. You will finish in no time and be happy with good codebase. But if you want to write second Google or Yahoo, you will be much better with C# or Java.
Side-note on C/C++ you have mentioned: if you are not writing performance critical software (say massive parallel raytracer that will run for three months rendering a film) or a very mission critical system (say Mars lander that will fly three years straight and has only one chance to land right or you lose $400mln) do not use it. For web apps, most desktop apps, most apps in general it is not a good choice. You will die debugging pointers and memory allocation in complex business logic.

In my opinion python is more than ready for developing complex applications. I see pythons strength more on the server side than writing graphical clients. But have a look at http://www.resolversystems.com/. They develop a whole spreadsheet in python using the .net ironpython port.
If you are familiar with eclipse have a look at pydev which provides auto-completion and debugging support for python with all the other eclipse goodies like svn support. The guy developing it has just been bought by aptana, so this will be solid choice for the future.
#Marcin
Cons: as a dynamic language, has way
worse IDE support (proper syntax
completion requires static typing,
whether explicit in Java or inferred
in SML),
You are right, that static analysis may not provide full syntax completion for dynamic languages, but I thing pydev gets the job done very well. Further more I have a different development style when programming python. I have always an ipython session open and with one F5 I do not only get the perfect completion from ipython, but object introspection and manipulation as well.
But if you want to write second Google
or Yahoo, you will be much better with
C# or Java.
Google just rewrote jaiku to work on top of App Engine, all in python. And as far as I know they use a lot of python inside google too.

I really like python, it's usually my language of choice these days for small (non-gui) stuff that I do on my own.
However, for some larger Python projects I've tackled, I'm finding that it's not quite the same as programming in say, C++. I was working on a language parser, and needed to represent an AST in Python. This is certainly within the scope of what Python can do, but I had a bit of trouble with some refactoring. I was changing the representation of my AST and changing methods and classes around a lot, and I found I missed the strong typing that would be available to me in a C++ solution. Python's duck typing was almost too flexible and I found myself adding a lot of assert code to try to check my types as the program ran. And then I couldn't really be sure that everything was properly typed unless I had 100% code coverage testing (which I didn't at the time).
Actually, that's another thing that I miss sometimes. It's possible to write syntactically correct code in Python that simply won't run. The compiler is incapable of telling you about it until it actually executes the code, so in infrequently-used code paths such as error handlers you can easily have unseen bugs lurking around. Even code that's as simple as printing an error message with a % format string can fail at runtime because of mismatched types.
I haven't used Python for any GUI stuff so I can't comment on that aspect.

Python is considered (among Python programmers :) to be a great language for rapid prototyping. There's not a lot of extraneous syntax getting in the way of your thought processes, so most of the work you do tends to go into the code. (There's far less idioms required to be involved in writing good Python code than in writing good C++.)
Given this, most Python (CPython) programmers ascribe to the "premature optimization is the root of all evil" philosophy. By writing high-level (and significantly slower) Python code, one can optimize the bottlenecks out using C/C++ bindings when your application is nearing completion. At this point it becomes more clear what your processor-intensive algorithms are through proper profiling. This way, you write most of the code in a very readable and maintainable manner while allowing for speedups down the road. You'll see several Python library modules written in C for this very reason.
Most graphics libraries in Python (i.e. wxPython) are just Python wrappers around C++ libraries anyway, so you're pretty much writing to a C++ backend.
To address your IDE question, SPE (Stani's Python Editor) is a good IDE that I've used and Eclipse with PyDev gets the job done as well. Both are OSS, so they're free to try!
[Edit] #Marcin: Have you had experience writing > 30k LOC in Python? It's also funny that you should mention Google's scalability concerns, since they're Python's biggest supporters! Also a small organization called NASA also uses Python frequently ;) see "One coder and 17,000 Lines of Code Later".

Nothing to add to the other answers, besides that if you choose python you must use something like pylint which nobody mentioned so far.

One way to judge what python is used for is to look at what products use python at the moment. This wikipedia page has a long list including various web frameworks, content management systems, version control systems, desktop apps and IDEs.
As it says here - "Some of the largest projects that use Python are the Zope application server, YouTube, and the original BitTorrent client. Large organizations that make use of Python include Google, Yahoo!, CERN and NASA. ITA uses Python for some of its components."
So in short, yes, it is "proper for production use in the development of stand-alone complex applications". So are many other languages, with various pros and cons. Which is the best language for your particular use case is too subjective to answer, so I won't try, but often the answer will be "the one your developers know best".

Refactoring is inevitable on larger codebases and the lack of static typing makes this much harder in python than in statically typed languages.

And as far as I know they use a lot of python inside google too.
Well i'd hope so, the maker of python still works at google if i'm not mistaken?
As for the use of Python, i think it's a great language for stand-alone apps. It's heavily used in a lot of Linux programs, and there are a few nice widget sets out there to aid in the development of GUI's.

Python is a delight to use. I use it routinely and also write a lot of code for work in C#. There are two drawbacks to writing UI code in Python. one is that there is not a single ui framework that is accepted by the majority of the community. when you write in c# the .NET runtime and class libraries are all meant to work together. With Python every UI library has at's own semantics which are often at odds with the pythonic mindset in which you are trying to write your program. I am not blaming the library writers. I've tried several libraries (wxwidgets, PythonWin[Wrapper around MFC], Tkinter), When doing so I often felt that I was writing code in a language other than Python (despite the fact that it was python) because the libraries aren't exactly pythonic they are a port from another language be it c, c++, tk.
So for me I will write UI code in .NET (for me C#) because of the IDE & the consistency of the libraries. But when I can I will write business logic in python because it is more clear and more fun.

I know I'm probably stating the obvious, but don't forget that the quality of the development team and their familiarity with the technology will have a major impact on your ability to deliver.
If you have a strong team, then it's probably not an issue if they're familiar. But if you have people who are more 9 to 5'rs who aren't familiar with the technology, they will need more support and you'd need to make a call if the productivity gains are worth whatever the cost of that support is.

I had only one python experience, my trash-cli project.
I know that probably some or all problems depends of my inexperience with python.
I found frustrating these things:
the difficult of finding a good IDE for free
the limited support to automatic refactoring
Moreover:
the need of introduce two level of grouping packages and modules confuses me.
it seems to me that there is not a widely adopted code naming convention
it seems to me that there are some standard library APIs docs that are incomplete
the fact that some standard libraries are not fully object oriented annoys me
Although some python coders tell me that they does not have these problems, or they say these are not problems.

Try Django or Pylons, write a simple app with both of them and then decide which one suits you best. There are others (like Turbogears or Werkzeug) but those are the most used.