We Keep Coding

PyPy and CPython: are big performance increases planned?

While I know projects promising large speed gains can result in let downs, I don't see much in the way of a roadmap for speeding up CPython and/or PyPy.
Is there something planned that promises a huge boost in speed for the core interpreter (e.g. --with-computed-gotos) in either of them? How about their standard libraries (e.g. Decimal in C, IO in C)?
I know HotPy(2) has an outline of a plan for speeding CPython up, but it sounds like an one-man project without much traction in core CPython.
PyPy has some information about where performance isn't great, but I can find no big goals for speedup in the docs.
So, are there known targets that could bring big performance improvement for Python implementations?

I'll answer the part about PyPy. I can't speak for CPython, but I think there are performance improvements that are being worked on (don't quote me on this though).
There is no project plan, since it's really not working that way. All the major parts (like "JIT" or "Garbage Collection") has been essentially done, however that completely does not mean everything is fast. There are definitely things that are slow and we generally improve on a case by case basis - submit a bug report if you think something is too slow. I have quite a few performance improvements on my plate that would definitely help twisted, but I have no idea about others.
Big things that are being worked on that might be worth mentioning:
Improved frames, that should help recursion and function calls that are not inlined (for example that contain loops)
Better string implementations for various kinds of usages, like concatenation, slicing etc.
Faster tracing
More compact tuples and objects, storing unwrapped results
Can I promise when how or how much it'll speed up things? Absolutely not, but on average we manage to have 10-30% speed improvements release-to-release, which is usually every 4 months or so, so I guess some stuff will get faster, but without you giving me a crystal ball or a time machine, I won't tell you for sure.
Cheers,
fijal

Your comments belie a lot of confusion...
PyPy and Python have currently very different performance capabilities.
Pypy is currently more than 5x faster than CPython on average.
HotPy has nothing to do with CPython. It's a one-man project and it's a whole new VM (not yet released, so I can't say anything about it's performance).
At the moment, there's a lot of activity in the PyPy project and they are improving it day by day.
There's a numpy port in a very advanced stage of development, they are improving ctypes, Cython compatibility, and soon there will be a complete Python3 implementation.
I believe PyPy is currently on pair with the V8 JavaScript engine and similar projects in terms of performance.
If speed and Python is what you want, pay attention to this project.

The answer is that PyPy is the plan to speed up CPython. PyPy aims to be an extremely conformant python interpreter which is highly optimized. The project has collected together all of the benchmarks they could find, and runs all of them for each build of pypy, to ensure against performance regressions. Check it out: http://speed.pypy.org/
I believe that by the time that the performance of cpython won't cut it anymore (for web dev work), pypy will be completely ready for prime-time. Raymond Hettinger (a core python dev) has called PyPy "python with the optimizations turned on".

Writing a DBMS in Python

I'm working on a basic DBMS as a pet project and planning to prototype in Python.
I figure there's a reason there are only a few Python databases, and my gut agrees that my favorite language will be too slow to act as an honest performing database, but I'm looking forward to using it to learn what I need quickly.
Would someone please contradict me? Is Python as ill-suited right now for this sort of thing as I think?
EDIT 4/16-
I've posted another getting-started-on-this-project type question if anyone is interested.
(Non-Relational) DBMS Design Resource

It's doubtful that anything you create as a pet project is going to turn out to be popular. Presumably you are mostly doing this as a learning experience and for fun.
Given these facts, there's no reason to stop yourself so early just because you think there might be performance problems. Just do it and have fun with it. The idea of a pure Python database will at least be academically interesting to others.
You can always do some performance profiling to find the bottlenecks and use the usual approaches in speeding things up (CPython, Cython, ctypes, etc.)
Don't be so quick to dismiss Python's huge benefits that you get in return for the performance hit. Namely rapid development.

If performance isn't a huge issue there's no reason Python can't do what you need, it certainly has all of the tools to do so. Designing a database certainly isn't a trivial undertaking, of course, but assuming you have the know-how and Python-fu to put in everything you need (of course, being helped by all of us here at SO ;) ) then the basic building blocks are all there.
For reference, there's at least one DBMS written in pure Python that I know of: KirbyBase

What should people new to Python know about its community and ecosystem? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I'm cobbling together some sort of an introduction to Python, but one that focuses on the community and the ecosystem around Python rather than just the language. With How to Think Like a Computer Scientist and other great tutorials, it's easy to get familiar with the language, but it took me a fair while before I knew what The Cheese Shop, or, err, PyPi is about, how pip and virtualenv work and why you should use them, where you should go for help, the interesting blogs that you should follow, how your code should look (PEP 8, writing pythonic code) and so on. The 'soft stuff'.
What confused you the most when you just started out with Python? Are there certain things that you would've wanted to know about, or resources you wish you would have stumbled upon earlier than you did? People to know about?
I found a few similar questions on StackOverflow (e.g. here) but nothing really close to what I'd like to hear from you guys. Hope this question doesn't feel too subjective to your tastes :-)
(And, if you'd like to help out, feel free to send a message.)

I think one of the most important thing a beginner need to know about Python ecosystem is that it's a general purpose language surrounded by specialized libs. Experienced pythonistas know them, but a newbie can't:
Don't stop to tkinter : go wx, gtk or qt.
Don't dev web code by hands : use TurboGears, Pylons, Web.py or Django.
Don't parse HTML / XML with hard tools : use lxml or beautifulsoup.
Don't make syscall to imagemagik : use PIL.
Don't make advanced maths manually : use NumPy and SciPy.
Don't access simple databases by hand : use ORM like SQLAlchemy.
Don't reinvent sysadmin wheels : use Fabric.
etc.
The main ones should be listed in a book with guidances to choose among them.

"What confused you the most when you just started out with Python?"
Rule 2 of learning Python: Any general-purpose module or framework you think you want has already been written. Several times.
The hard part is realizing that your idea is
Not unique.
Been already improved upon before you even starting thinking about it.
Already posted somewhere.
So, code less and search more. Search widely and flexibly until you find things that are similar to what you want to do.
Realize that you might have a name you think is descriptive. But other folks may call it something different. Join the community, adopt their naming. You may not like the phrase "ORM", but that's what it's called.
Realize that your idea, no matter how sound it seems, may be really poor. When you find a framework that seems to have "needless extra features", you may be missing something from your idea.
Realize that your idea, no matter how "intuitive" it seems, may be really poor. When you find a framework that seems "counter-intuitive", the problem could be yours. Learn theirs first, then compare and contrast after you've mastered theirs. Until you've mastered theirs, keep searching and learning.

A few points related to the ecosystem and indirectly the community:
I wished I had been reminded more about the Batteries included. I think people should be told to print-out the Table of Contents of the Standard Library and keep it under their pillow, for frequent reviews (that advice, I finally took, several years into it, from an online intro/beginner's video presentation!). The [relative] stability, extensive but relevant content of the Std Lib speaks to the thoughtful governance of the community leaders and its beloved BDFL.
I think newcomers can also be "warned" (the word is maybe too strong, too unfair) about the extreme variety of PiPy. This reflects the vibrant, smart and diverse (in terms of background, domain of application, interests...) collective of users and contributors. This however can be overwhelming and possibly risky as all packages in there are not "prime time ready" (But many are and "saved my life" many times over).
Even if you feel too new to Python, don't only use the libraries, do peek under the hood! This is true of many languages, but maybe particularly of Python, there's much to be learned from perusing various source code. The reasons this may be particular true for Python are intrinsic to the language itself (multi-paradigms, hi level of abstraction...) but also because of the relative uniformity of coding (and architecting) style and because of the general level of collaboration within the community.

"What confused you the most when you just started out with Python?"
Rule 1 of learning Python: Use the Source, Luke.
There are question on SO asking for "good" source from which to learn Python. The best answers amount to "read the libraries that came with Python."
One can say that the libraries that come with Python are quirky. In places. Which makes them all the better for learning from.
There's a community of like-minded people who are not clones.
Open source software is the highest-quality software you'll ever get to work with, but it's not created by paid developers who will rigidly enforce standards.

explaining what a PEP is, how it is written and who wrote them, where we can find them. PEPs give a lot of background informations about a specific feature of the language. they also are the tool which shows how fast evolving python is.
(i wish i had read some PEPs earlier, but i was not really aware of them, although they are frequently linked in the manual)

From PEP 20:
import this
(aka, the Zen of Python)

developing a python package that can be installed with easy_install etc... I consider it equivalent to developing a jar or dll etc....
on the same token, developing said package(s) with virtualenv or buildout
If I would have known those things sooner, I would have probably used python for more than just scripting way back when I first started using it.

Learn Go Or Improve My Python/Ruby Knowledge [closed]

Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 5 years ago.
Improve this question
I was reading about Go, and I can see that it's very good and can be a language used by many developers in some months, but I want to decide a simple thing: Learn Go or improve my Python or Ruby knowledge?
Years developing with Python: 1
Years developing with Ruby: 0.3

If you're just looking to have fun and expand your horizons, then I'd learn Go, since you already know some Python.
If you're looking to improve as a developer, I'd personally recommend working on an actual project (using Python, as it's the language you have the most experience with):
This will take your (Python and general) programming skills to a whole new level
If you choose an interesting project, for example a semi-popular open source project, you'll even have some concrete result to show for your extra work. This will help your resume, help you meet other people interested in programming, etc.
I personally believe that spending the time you would have spent learning a new language by coding actual things you can use, is usually a net win (unless of course, like I said, you're just doing it for some fun recreation or relaxation).

In reality, you should do both; if it's what you want. For me though, out of the two, I'd only look at Python. I have no real interest in languages that are so new.

It depends on what your goals and your needs are.
If you're looking to develop your skills for a job, then go with Python or Ruby. You're unlikely to see Go show up in the workplace for quite some time (if ever) unless you're working at Google. (Even then it's questionable.)
If you want to have fun, do what you want. I think the best decider is, pick a project you want to work on and then choose the language that is best suited for that project.
Other things to consider:
Each language is suited to certain tasks. Go is compiled into machine code, whereas Python and Ruby run in interpreters. Go lends itself to somewhat lower level work. It's also good for concurrent tasks. Higher level tasks might be more suited for Python or Ruby.
Go is an experimental language that's likely to experience changes. These changes may be backward incompatible. If you learn it now, in 6 months or a year you may have to re-learn some of it because it's changed. That said, it can be fun to be a part of something that's on the bleeding edge. And if it does happen to become "the next big thing," you're in on the ground level.

How long have you been working with Python?
If it were me, I'd do my best to maybe get a working knowledge of Go (basic syntax, some familiarity with unique language features), and continue with Python as I normally would.
Eventually you might come up with a small project that Go seems suited for (or you can come up with one now!) and really dive into the language that way.
There's no reason to limit yourself to just one. :)

It's up to you. You should probably do both if you can, because that way you will have more tools on your metaphorical programmer's belt.
There are a number of things that I think are worth considering whenever I'm in a similar dilemma.
Is a new language (not just unfamiliar, but actually new, like Go) likely to catch on? (If so, it will become practically NECESSARY to learn it, rather than just a good idea.)
You will need to spend some time to learn the unfamiliar language. Will this time investment result in some sort of positive return? The obvious one here is development time (i.e., can you eventually get more done and get back the time you spent to learn it), but if the language is superior in other ways (runs faster or with less memory, is best for your particular problem domain) those might factor in too.
Will learning the unfamiliar language allow you to solve a relevant/important/urgent problem that cannot be solved with what you already know?
Unfortunately, none of us can tell you how to weigh each of these concerns. You'll need to think about it really carefully and come to the answer on your own.

First of all, it's a very very personal question, and my first recommendation will be , if you think so, try Go for one month or so, learning the basics, and then deciding... Each one has the mind fitted more to some particular languages than another...
I also would recommend keep using both Python and Ruby, if you're interested, you have enough experience to use them comfortably (I think), so the next months you could experience a great boost in your experience using them...
For an strict utilitarian point of view, learning Go extensively could be a risk move, but worthy if begins to being using and you're one of the very few people with some experience when everyone are just beginning to learn the syntax...

These two languages accomplish different goals. Go is only 20% slower than C. If you want crazy speed and easy parallelism, then learn Go. However, readability was not a design goal. Also, it has no exceptions. It can be used for real-time products though ...

IMO you should improve your Python knowledge. Python is a widely adopted language, whereas
Go is still at a very, very early stage and there's no reason to believe that it will become successful.
From a purely utilitarian perspective, you will get a lot more value from learning Python.

It's not easy to answer this without knowing how good your Python is, or what you do, or where you'd use Go. If you're looking for employment, I suspect Python is the way to go. I'd be surprised to find anyone is using Go for major projects at the moment (outside Google).
Note also that Go is by no means finalised. See here for the Go roadmap, and note the potential changes in the future. So you may be trying to hit a moving target currently.

Python is a available for most operating systems, it's generally accepted as a scriptin language, and it has matured to production quality.
Go is a research language that's only available in beta quality on Linux and OS X. Nowhere else. It's interesting from an intellectual point of view in that you can learn and apply a few concepts (typed channels combined with easy multitasking) that are otherwise difficult to use.
As for Go, you might perhaps take a look at AT&T's "Plan 9" operating system first. It comes with a programming language called "Aleph" with also is C-based, has channels and multitasking. Looks to me like Go is a reimplementation of Aleph on more main-stream operating systems.

What are the drawbacks of Stackless Python? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I've been reading recently about Stackless Python and it seems to have many advantages compared with vanilla cPython. It has all those cool features like infinite recursion, microthreads, continuations, etc. and at the same time is faster than cPython (around 10%, if the Python wiki is to be believed) and compatible with it (at least versions 2.5, 2.6 and 3.0).
All these looks almost too good to be true. However, TANSTAAFL, I don't see much enthusiasm for Stackless among the Python community, and PEP 219 has never come into realization. Why is that? What are the drawbacks of Stackless? What skeletons are hidden in Stackless' closet?
(I know Stackless doesn't offer real concurrency, just an easier way of programming in the concurrent way. It doesn't really bother me.)

I don't know where that "Stackless is 10% faster" on the Wiki came from, but then again I've never tried to measure those performance numbers. I can't think of what Stackless does to make a difference that big.
Stackless is an amazing tool with several organizational/political problems.
The first comes from history. Christian Tismer started talking about what eventually became Stackless about 10 years ago. He had an idea of what he wanted, but had a hard time explaining what he was doing and why people should use it. This is partially because his background didn't have the CS training regarding ideas like coroutines and because his presentations and discussion are very implementation oriented, which is hard for anyone not already hip-deep in continuations to understand how to use it as a solution to their problems.
For that reason, the initial documentation was poor. There were some descriptions of how to use it, with the best from third-party contributors. At PyCon 2007 I gave a talk on "Using Stackless" which went over quite well, according to the PyCon survey numbers. Richard Tew has done a great job collecting these, updating stackless.com, and maintaining the distribution when new Python releases comes up. He's an employee of CCP Games, developers of EVE Online, which uses Stackless as an essential part of their gaming system.
CCP games is also the biggest real-world example people use when they talk about Stackless. The main tutorial for Stackless is Grant Olson's "Introduction to Concurrent Programming with Stackless Python", which is also game oriented. I think this gives people a skewed idea that Stackless is games-oriented, when it's more that games are more easily continuation oriented.
Another difficulty has been the source code. In its original form it required changes to many parts of Python, which made Guido van Rossum, the Python lead, wary. Part of the reason, I think, was support for call/cc that was later removed as being "too much like supporting a goto when there are better higher-level forms." I'm not certain about this history, so just read this paragraph as "Stackless used to require too many changes."
Later releases didn't require the changes, and Tismer continued to push for its inclusion in Python. While there was some consideration, the official stance (as far as I know) is that CPython is not only a Python implementation but it's meant as a reference implementation, and it won't include Stackless functionality because it can't be implemented by Jython or Iron Python.
There are absolutely no plans for "significant changes to the code base". That quote and reference hyperlink from Arafangion's (see the comment) are from roughly 2000/2001. The structural changes have long been done, and it's what I mentioned above. Stackless as it is now is stable and mature, with only minor tweaks to the code base over the last several years.
One final limitation with Stackless - there is no strong advocate for Stackless. Tismer is now deeply involved with PyPy, which is an implementation of Python for Python. He has implemented the Stackless functionality in PyPy and considers it much superior to Stackless itself, and feels that PyPy is the way of the future. Tew maintains Stackless but he isn't interested in advocacy. I considered being in that role, but couldn't see how I could make an income from it.
Though if you want training in Stackless, feel free to contact me! :)

it took quite long to find this discussion. At that
time I was not on PyPy but had a 2-years affair with psyco, until health stopped this all quite abruptly. I'm now active again and designing an
alternative approach - will present it on EuroPython 2012.
Most of Andrews statements are correct. Some
minor additions:
Stackless was significantly faster than CPython, 10 years ago, because I optimized the interpreter loop. At that time, Guido was not ready for that. A few years later, people did similar optimizations and even more and better ones, which makes Stackless a little bit slower, as expected.
On inclusion: well, in the beginning I was very pushy and convinced that Stackless is the way to go. Later, when it was almost possible to get included, I lost interest in that and preferred to let
it stay this way, partially out of frustration, partially to
keep control of Stackless.
The arguments like "other implementations cannot do it" felt always lame to me, as there are other examples where this argument could also be used. I thought I better forget about that and stay in good friendship with Guido, having my own distro.
Meanwhile things are changing again. I'm working on PyPy and Stackless as an extension Will talk about that sometimes later
Cheers -- Chris

If I recall correctly, Stackless was slated for inclusion into the official CPython, but the author of stackless told the CPython folks not to do so, because he planned to do some significant changes to the code base - presumeably he wanted the integration done later when the project was more mature.

I'm also interested in the answers here. I've played a bit with Stackless and it looks like it would be a good solid addition to standard Python.
PEP 219 does mention potential difficulties with calling Python code from C code, if Python wants to change to a different stack. There would need to be ways to detect and prevent this (to avoid trashing the C stack). I think this is tractable though, so I'm also wondering why Stackless must stand on its own.