We Keep Coding

Which python project to choose for a database-heavy application

I plan to develop a rather database heavy (~100 tables) web application in python. The focus is on providing a nice and task-optimized interface for people that edit or navigate through the data. Other focuses are:
Handle lots of data and complex queries.
Internationalization (translation, timezones, currencies)
Mailings (bulk emailing as well as notifications)
Easy integration into other websites (pull data from or push data to the application)
A role based authentication scheme. (ideally enforcing one role at a time)
It should be easy and fast (for python programmers) to create custom forms and workflows to work with the data.
I've read a lot about django, turbogears, pyramid, webcore, … but I'm still having a hard time to figure out where to start.
My current evaluation would suggest that turbogears is the way to go. Pyramid seems too much to learn about. Django seems to be too publishing focused. WebCore seems a bit to immature to base such a project on it.
Am I overlooking something? Are there other more suitable python frameworks? Is my information about some of them plain wrong? Which framework would you choose for this project, and why?

Imo the only part of django that might be "too" publishing orientated is the admin, but I have seen plenty of django applications doing stuff neatly.
Django has plenty of apps available covering what you want to do, but the only road block you might find is the part of: handle lots of data and complex queries. You will probably move out of django ORM land, but you might even move out of SQLAlchemy land too. Most of these projects use ORM's, so I would look into SQLAlchemy first, and evaluate how to use it for your needs.
Second, I would just go through the tutorials of the following projects, reading about them is good, but a small little tutorial/project (or mini prototype) is the only way to see if the project fits your programming style: pyramid, turbogears, and django. They have afaik the largest communities. The best tool will be the one you feel more confortable with. They all have good, excellent documentation, good supportive communities, and are mature enough for solid projects, and for very subtle differences, you probably can use any of them for your needs.

I have to agree with you: if you already have a database model, Django is not potentially the best way to go.
As of database binding, SQL Alchemy is definitely worth checking out, regardless of which framework you choose.
Some additions to your list:
Web.py - A little low-level, but a nice, mature framework (for SQLAlchemy usage, see cookbook on SQLAlchemy)
Tornado - Very good performance, supports Websockets which could be of a concern for some applications.
Edit: nowadays I would recommended Flask instead of Web.py. Tornado works really nicely with it as well.

The number of tables is not relevant for speeds etc. and not relevant for the choice of the framework. Recommendation: use SQLAlchemy as ORM between database and application. Go for Pyramid as web framework. Pyramid is easy, well-documented, test and very flexible in all aspects. Forms etc. can be easily created using "colander" + "deform" add-ons.

My gut says you want to use SQLAlchemy as the ORM. Turbogears does this out of the box, and probably is the largest player in the "not Django" space.
There was some work on pulling in SQLAlchemy for (or in addition to!) Django's ORM, but I don't know how current that work is (a quick google search found articles from 2008-2009 as the top hits)

Getting started with Pylons

I am just starting to use a web framework. I have decided I really like python and started looking at web frameworks. I don't really like django for a few reasons, but from what I have tried so far I found I really like pylons.
The problem I have is that I can't find that many articles/tutorials about pylons, especially 1.0 articles. Does anybody know any good getting started tutorials and articles about pylons?
Also, I am gonna need to implement users in my applications with a secure log in and have the users "own" a model. Any good advice/articles/tutorials about how I would do this?
When I was looking at some tutorial they mention virtual python environments. I don't really know what that is, why you would use them and how do you use them. Any help?
Finally, I can't find any good tutorials/articles about how to deploy pylons to a production environment. I own a VPS and am gonna deploy there. Any help with that?
Is there anything else I should know about pylons or python. I know the basics of python already.

The book suggested by meder (http://pylonsbook.com/en/1.1/) is a very good start. I upvoted his anwser because that's where I learned Pylons.
However, the book is written for Pylons 0.9.7 (the latest version before 0.10 and 1.0).
Pylons is the agglomeration of several high quality libraries. Learning Pylons is all about learning those libraries. Most of the book is about exploring those libraries. When you learn to develop web app in Pylons, what you really learn is to develop app in Python.
Right now, I think the book and the official website (http://pylonshq.com/docs/en/1.0/) are the two most valuable resources to learn Pylons.
Most of the changes that happenned between 0.9.7 and 1.0 are in the app start-up (which you probably won't really try to modify at the begining). Other than that, the libraries have been updated (sqlalchemy is now 0.6, etc.). Also, one change that may affect you: the url_to and redirect_to functions have been replaced by url and redirect. That's about it.

There is an entire book published free which covers Pylons 1.0:
http://pylonsbook.com/en/1.1/

You will definitely need to learn SQLAlchemy to master Pylons.
Official docs are pretty good start at it, http://www.sqlalchemy.org/docs/, and you may want to try Elixir extension, which provides a bit better declarative syntax.
You also should read docs on Routes module, http://routes.groovie.org/contents.html, especially on submappers and RESTful services, http://routes.groovie.org/restful.html
And you need to learn w/e templating system you choose. Mako, for example, have some non-obvious caveats, like much better performance of <%namespace/> vs <%include/>.

For authentication the homegrown decorator based approach works well also: http://wiki.pylonshq.com/display/pylonscookbook/Another+approach+for+authorization+in+pylons+%28decorator+based%2C+repoze.what+like%29

PHP Frameworks (CodeIgniter, Yii, CakePHP) vs. Django

I have to develop a site which has to accomodate around 2000 users a day and speed is a criterion for it. Moreover, the site is a user oriented one where the user will be able to log in and check his profile, register for specific events he/she wants to participate in. The site is to be hosted on a VPS server.Although I have pretty good experience with python and PHP but I have no idea how to use either of the framework. We have plenty of time to experiment and learn one of the above frameworks.Could you please specify which one would be preferred for such a scenario considering speed, features, and security of the site.
Thanks,
niting

This is a very subjective question but personally I'd recommend Django. Python is a very nice language to use and the Django framework is small, easy to use, well documented and also has a pretty active community.
This choice was made partly because of my dislike for PHP though, so take the recommendation with a pinch of salt.

Most of the frameworks out there nowadays are fast enough to serve whatever needs you will have. It really depends on in which environment you feel most comfortable. Though there are nuances here and there, MVC frameworks share a lot of the same principles, so whichever you choose to use is really a matter of which you most enjoy using.
So, if you like Python more, there's your answer. Use a Python framework, and Django is the best. If you like PHP more (which I personally don't), you've got some more decisions to make. But any of the PHP frameworks are fine. They really are. Just pick one that looks nice with comprehensive documentation and get to work.

I've worked with CakePHP and Django and I really recommend Django. I don't know too much about CodeIgniter, but I remember ruling it out when I was evaluating frameworks myself about a year ago. CakePHP seemed much more developed at the time.
First of all, the Django community is much bigger and has spent a lot of time focusing on reusable apps. This means that you get a lot of functionality for free. Pair this with the django admin, and you have a lot of things already done for you. I haven't kept up with the PHP frameworks much, but I'm pretty sure Django is also more developed.
This is more of a personal thing, but I just like Python over PHP. Compare the way models are done in CakePHP and Django: http://book.cakephp.org/view/67/Understanding-Models, http://docs.djangoproject.com/en/dev/topics/db/models/#topics-db-models. The python is clearly more readable.
Keep in mind that Django gives you an awesome ORM and builds your schema for you, i.e. you never have to touch the database if you don't want to. With the PHP frameworks, you have to do your own db design, which just slows me down at this point. You can always go in and add indexes for speed later.
This is probably the most biased, but if you are starting a new application - seriously - just stick with Django or Ruby on Rails. There is a reason everyone talks about them and they have the biggest communities and best developers behind them.
You can also check out Pinax for a lot of Django goodies.

Codeigniter it's fast and very documented also has a large community to and finaly friendly with the programmer.

CodeIgniter is a great PHP framework that is fast and has excellent documentation. Start reading through their user guide and it will give you a good idea how to work with the framework.

Extending Matchu:
Or, -If you like PHP more- its time to learn/growup about other things like Python. Its not hard to learn, and when you get started it gets very enjoyable.
Many people has done the PHPtoPython/Django port, like Mozilla, Netgeo, Nasa, TheOnion, etc.

If for the PHP part I would choose CodeIgniter - it doesn't get too much into your way. But it doesn't have any code/view/model generators out of the box, you need to type a bit.
But languages other than PHP appear to be more sexy.

I am using CodeIgniter 1.7.2 and for complex websites it's very good and powerfull, but it definitely is missing some kind of code generator which will allow for example to build an IT application in one click.
I had the impression (from watching a tutorial) that Django has it.

Django vs other Python web frameworks?

I've pretty much tried every Python web framework that exists, and it took me a long time to realize there wasn't a silver bullet framework, each had its own advantages and disadvantages. I started out with Snakelets and heartily enjoyed being able to control almost everything at a lower level without much fuss, but then I discovered TurboGears and I have been using it (1.x) ever since. Tools like Catwalk and the web console are invaluable to me.
But with TurboGears 2 coming out which brings WSGI support, and after reading up on the religious debates between the Django and WSGI camps, I'm really torn between "doing it the right way", e.g., learning WSGI, spending valuable time writing functionality that already exists in Django and other full-stack frameworks, as opposed to using Django or some high-level framework that does everything for me. The downsides with the latter that I can see are pretty obvious:
I'm not learning anything in the process
If I ever need to do anything lower level it's going to be a pain
The overhead required for just a basic site which uses authentication is insane. (IMO)
So, I guess my question is, which is the better choice, or is it just a matter of opinion, and should I suck it up and use Django if it achieves what I want with minimal fuss (I want authentication and a CRUD interface to my database)? I tried Werkzeug, Glashammer, and friends, but AuthKit and Repoze scared me off, as well as the number of steps involved to just setup basic authentication. I looked at Pylons, but the documentation seems lacking, and when referencing simple features like authentication or a CRUD interface, various wiki pages and documentation seemed to contradict each other, with different hacks for versions and such.
Thanks to S. Lott for pointing out that I wasn't clear enough. My question is: which of the following is worthwhile in the long run, but not painful in the short (e.g., some sort of middle ground, anyone?) - Learn WSGI, or stick with a "batteries-included" framework? If the latter, I would appreciate a suggestion as to whether I should give Django another try, stick with TurboGears 1.x, or venture into some other framework.
Also, I have tried CherryPy, but couldn't seem to find a good enough CRUD application that I could plop in and use right away.

the religious debates between the Django and WSGI camps
It would seem as though you're a tad bit confused about what WSGI is and what Django is. Saying that Django and WSGI are competing is a bit like saying that C and SQL are competing: you're comparing apples and oranges.
Django is a framework, WSGI is a protocol (which is supported by Django) for how the server interacts with the framework. Most importantly, learning to use WSGI directly is a bit like learning assembly. It's a great learning experience, but it's not really something you should do for production code (nor was it intended to be).
At any rate, my advice is to figure it out for yourself. Most frameworks have a "make a wiki/blog/poll in an hour" type exercise. Spend a little time with each one and figure out which one you like best. After all, how can you decide between different frameworks if you're not willing to try them out?

I'd say you're being a bit too pessimistic about "not learning anything" using Django or a similar full-stack framework, and underestimating the value of documentation and a large community. Even with Django there's still a considerable learning curve; and if it doesn't do everything you want, it's not like the framework code is impenetrable.
Some personal experience: I spent years, on and off, messing around with Twisted/Nevow, TurboGears and a few other Python web frameworks. I never finished anything because the framework code was perpetually unfinished and being rewritten underneath me, the documentation was often nonexistent or wrong and the only viable support was via IRC (where I often got great advice, but felt like I was imposing if I asked too many questions).
By comparison, in the past couple of years I've knocked off a few sites with Django. Unlike my previous experience, they're actually deployed and running. The Django development process may be slow and careful, but it results in much less bitrot and deprecation, and documentation that is actually helpful.
HTTP authentication support for Django finally went in a few weeks ago, if that's what you're referring to in #3.

I suggest taking another look at TG2. I think people have failed to notice some of the strides that have been made since the last version. Aside from the growing WSGI stack of utilities available there are quite a few TG2-specific items to consider. Here are a couple of highlights:
TurboGears Administration System - This CRUD interface to your database is fully customizable using a declarative config class. It is also integrated with Dojo to give you infinitely scrollable tables. Server side validation is also automated. The admin interface uses RESTful urls and HTTP verbs which means it would be easy to connect to programatically using industry standards.
CrudRestController/RestController - TurboGears provides a structured way to handle services in your controller. Providing you the ability to use standardized HTTP verbs simply by extending our RestController. Combine Sprox with CrudRestController, and you can put crud anywhere in your application with fully-customizable autogenerated forms.
TurboGears now supports mime-types as file extensions in the url, so you can have your controller render .json and .xml with the same interface it uses to render html (returning a dictionary from a controller)
If you click the links you will see that we have a new set of documentation built with sphinx which is more extensive than the docs of the past.
With the best web server, ORM, and template system(s) (pick your own) under the hood, it's easy to see why TG makes sense for people who want to get going quickly, and still have scalability as their site grows.
TurboGears is often seen as trying to hit a moving target, but we are consistent about releases, which means you won't have to worry about working out of the trunk to get the latest features you need. Coming to the future: more TurboGears extensions that will allow your application to grow functionality with the ease of paster commands.

Your question seems to be "is it worth learning WSGI and doing everything yourself," or using a "full stack framework that does everything for you."
I'd say that's a false dichotomy and there's an obvious third way. TurboGears 2 tries to provide a smooth path from a "do everything for you" style framework up to an understanding of WSGI middleware, and an ability to customize almost every aspect of the framework to suit your application's needs.
We may not be successful in every place at every level, but particularly if you've already got some TurboGears 1 experience I think the TG2 learning curve will be very, very easy at first and you'll have the ability to go deeper exactly when you need it.
To address your particular issues:
We provide an authorization system out of the box that matches the one you're used to from TG1.
We provide an out of the box "django admin" like interface called the tgext.admin, which works great with dojo to make a fancy spreadsheet like interface the default.
I'd also like to address a couple of the other options that are out there and talk a little bit about the benifits.
CherryPy. I think CherryPy is a great webserver and a nice minimalistic web-framework. It's not based on WSGI internally but has good WSGI support although it will not provide you with the "full stack" experience. But for custom setups that need to be both fast and aren't particularly suited to the defaults provided by Django or TurboGears, it's a great solution.
Django. I think Django is a very nice, tigtly integrated system for developing websites. If your application and style of working fits well within it's standard setup it can be fantastic. If however you need to tune your DB usage, replace the template language, use a different user authorization model or otherwise do things differently you may very likely find yourself fighting the framework.
Pylons Pylons like CherryPy is a great minimalistic web-framework. Unlike CherryPy it's WSGI enabled through the whole system and provides some sane defaults like SQLAlchemy and Mako that can help you scale well. The new official docs are of much better quality than the old wiki docs which are what you seem to have looked at.

Have you taken a look at CherryPy. It is minimalistic, yet efficient and simple. It is low level enough for not it to get in they way, but high enough to hide complexity. If I remember well, TurboGears was built on it.
With CherryPy, you have the choice of much everything. (Template framework, ORM if wanted, back-end, etc.)

Learn WSGI
WSGI is absurdly simple.. It's basically a function that looks like..
def application(environ, start_response) pass
The function is called when an HTTP request is received. environ contains various data (like the request URI etc etc), start_response is a callable function, used to set headers.
The returned value is the body of the website.
def application(environ, start_response):
start_response("200 OK", [])
return "..."
That's all there is to it, really.. It's not a framework, but more a protocol for web-frameworks to use..
For creating sites, using WSGI is not the "right way" - using existing frameworks is.. but, if you are writing a Python web-framework then using WSGI is absolutely the right way..
Which framework you use (CherryPy, Django, TurboGears etc) is basically personal preference.. Play around in each, see which you like the most, then use it.. There is a StackOverflow question (with a great answer) about this, "Recommendation for straight-forward python frameworks"

Have you checked out web2py? After recently evaluating many Python web frameworks recently I've decided to adopt this one. Also check out Google App Engine if you haven't already.

I'd say the correct answer depends on what you actually want and need, as what will be worthwhile in the long run depends on what you'll need in the long run. If your goal is to get applications deployed ASAP then the 'simpler' route, ie. Django, is surely the way to go. The value of a well-tested and well-documented system that exactly what you want can't be underestimated.
On the other hand if you have time to learn a variety of new things which may apply in other domains and want to have the widest scope for customisation then something like Turbogears is superior. Turbogears gives you maximum flexibility but you will have to spend a lot of time reading external docs for things like Repoze, SQLAlchemy, and Genshi to get anything useful done with it. The TG2 docs are deliberately less detailed than the TG1 docs in some cases because it's considered that the external docs are better than they used to be. Whether this sort of thing is an obstacle or an investment depends on your own requirements.

Django is definitely worth learning, and sounds like it will fit your purposes. The admin interface it comes with is easy to get up and running, and it does use authentication.
As for "anything lower level", if you mean sql, it is entirely possible to shove sql into you queries with the extra keyword. Stylistically, you always try to avoid that as much as possible.
As for "not learning anything"...the real question is whether your preference is to be primarily learning something lower-level or higher-level, which is hardly a question anyone here can answer for you.

Pylons seems a great tool for me:
a real web framework (CherryPy is just a web server),
small code base - reuse of other projects,
written entirely with WSGI in mind, based on Paste,
allows you to code the app right away and touch the low level bits if it's necessary,
I've used CherryPy and TurboGears and look at many other frameworks but none of them were so light and productive as Pylons is. Check the presentation at Google.

I'm a TurboGears fan, and this is exactly the reason why: a very nice trade-off between control and doing things right vs. easy.
You'll have to make up your own mind of course. Maybe you'd prefer to learn less, maybe more. Maybe the areas that I like knowledge/control (database for example), you couldn't care less about. And don't misunderstand. I'm not characterizing any frameworks as necessarily hard or wrong. It's just my subjective judgment.
Also I would recommend TurboGears 2 if at all possible. When it comes out, I think it will be much better than 1.0 in terms of what it has selected for defaults (genshi, pylons, SqlAlchemy)

I would suggest for TurboGears 2. They have done a fantastic job of integrating best of Python world.
WSGI: Assuming you are developing moderately complex projects/ business solutions in TG2 or some other framework say Grok. Even though these frameworks supports WSGI does that mean one who is using these frameworks have to learn WSGI? In most cases answer is No. I mean it's good have this knowledge no doubt.
WSGI knowledge is probably is more useful in cases like
you want to use some middleware or some other component which is not provided as part of the standard stack for eg. Authkit with TG or Grok without ZODB.
you are doing some integration.
CherryPy is good but think of handling your database commits/rollbacks at the end of transactions, exposing json, validations in such cases TG, Django like frameworks do it all for you.

Web2py is the secret sauce here. Don't miss checking it out.

Can anyone point out the pros and cons of TG2 over Django?

Django is my favorite python web framework. I've tried out others like pylons, web2py, nevow and others.
But I've never looked into TurboGears with much enthusiasm.
Now with TG2 out of beta I may give it a try. I'd like to know what are some of the pros and cons compared to Django.

TG2 has several advantages that I think are important:
Multi-database support
sharding/data partitioning support
longstanding support for aggregates, multi-column primary keys
a transaction system that handles multi-database transactions for you
an admin system that works with all of the above
out of the box support for reusable template snipits
an easy method for creating reusable template tag-libraries
more flexibility in using non-standard components
There are more, but I think it's also important to know that Django has some advantages over TG2:
Larger, community, more active IRC channel
more re-usable app-components
a bit more developed documentation
All of this means that it's a bit easier to get started in Django than TG2, but I personally think the added power and flexibility that you get is worth it. But your needs may always be different.

TG2 takes Pylons and changes some defaults - object dispatching instead of Routes, and Genshi instead of Mako. They believe there's only one way to do it, so apps can rely on the same API for any TurboGears website.
Similarities
TG2 and Django both distinguish between websites and components, so you'll eventually see reusable building blocks for TurboGears, too.
Differences
Django uses its own handlers for HTTP, routing, templating, and persistence. Django also has stellar documentation and an established community.
TurboGears defaults to best-of-breed libraries, which apparently are Paste, object dispatching, Genshi, and SqlAlchemy. This philosophy produces a better all-round toolset, but at the risk of instability - because it means throwing away backwards compatibility if and when better libraries appear.

Pros.
SQLAlchemy > django ORM
Multiple template languages out of the box (genshi,mako,jinja2)
more WSGI friendly
Object Dispatch > routes > regexp routing. You can get the first 2 with TG2
Almost all components are optional you can keep the core and use any ORM, template, auth library, etc.
Sprox > django forms
Cons.
- Admin is more basic (no inline objects yet!)
- less third party apps
- "app" system still in the making.
- given it's modularity you need to read documentation from different sources (SQLAlchemy, Genshi or Mako, repoze.who, Pylons, etc.)

I was struggling with the same question months ago and decided for Turbogears 2, and my reasoning was simple. "I'm new to python, I want to learn it not just for web-projects but as a substitute to php for scripting small helpers"
What I didn't like about Django, to me looks like a "close platform". ORM, Template system, sessions, etc they all are Django's
On the other hand, Turbogears 2 uses already known open platforms and just glued them, just like Appfuse does it for Java
With TurboGears 2 I learn SQLAlchemy that I can use later for small python scripts, or from the python shell to solve common tasks.
Main drawbacks are the lack of complete documentation and error messages.
Sometimes you have to search very deep to find simple solutions, the learning curve is steep, but it pays long term. The error messages where to me very confusing (coming from more than 10 years in Java development). I had lost many hours trying to find an "ascii encode error" when the real problem was a module not being imported.
That's my opinion, just remember I'm new to python and I could be wrong about many things stated here.

Besides what Nikhil gave in his answer, I think another minor difference is that Turbogears provdes some support for javascript widgets and integration with Mochikit.
Whereas Django steadfastly remains javascript framework neutral.
(At least this was true with older versions of Turbogears... this might have changed with TG2)
Edit: I just went over TG2 documentation and see that it did indeed change. Turbogears now uses ToscaWidgets which can use jQuery, ExtJS, Dojo, etc. underneath. This nicely makes it more framework neutral while still providing nice javascript widgets.
This strikes me as a pro for Turbogears if you don't have any javascript experience and a pro for Django if you are writing a lot of specialized javascript.

One of the most important questions is not just what technical features this platform provides or that platform provides, but the driving philosophy of the open source project and the nature of the community supporting it.
I've got no dog in this fight myself, but I found Mark Ramm's talk at DjangoCon 2008 to be very interesting on this point (Google will yield no end of subsequent discussion, no doubt).

Because Django uses its own ORM it limits you to learn that ORM for that specific web framework. I think using an web framework with a more popular ORM (like SqlAlchemy which TG uses) increases your employability chances. Just my 2 cents ..

Last I checked, django has a very poor data implementation. And that's a huge weakness in my book. Django's orm doesn't allow me to use the power of the underlying database. For example I can't use compound primary keys, which are important to good db design. It also doesn't support more than a single database, which is not a big deal until you really need it and find that you can't do it without resorting to doing it manually. Lastly if you have to make changes to your database structure in a team-friendly way, you have to try to choose between a set of 3rd party migration tools.
Turbogears seems to be more architecturally sound, doing its best to integrate individual tools that are awesome in their own right. And because TG is more of an integrator, you're able to switch out pieces to suit your preferences. Don't like SQL Alchemy? You can use SQLObject. Don't like Genshi templates? You can use Mako or even django's, although you're not exactly stuck with the default on django either.
Time for tg2's cons:
TG has a much smaller community, and community usually has its benefit.
Django has a much better name. I really like that name ;-)
Django seems simpler for the beginning web developer, with pretty cool admin tools.
TG has decent documentation, but you also need to go to Genshi's site to learn Genshi, SQL Alchemy's site to learn that, etc. Django has great docs.
My 2 cents.