Related
I'm a hobbyist programmer (only in TI-Basic before now), and after much, much, much debating with myself, I've decided to learn Python. I don't have a ton of free time to teach myself a hundred languages and all programming I do will be for personal use or for distributing to people who need them, so I decided that I needed one good, strong language to be good at. My questions:
Is python powerful enough to handle most things that a typical programmer might do in his off-time? I have in mind things like complex stat generators based on user input for tabletop games, making small games, automate install processes, and build interactive websites, but probably a hundred things along those lines
Does python handle networking tasks fairly well?
Can python source be obfuscated, or is it going to be open-source by nature? The reason I ask this is because if I make something cool and distribute it, I don't want some idiot script kiddie to edit his own name in and say he wrote it
And how popular is python, compared to other languages. Ideally, my language would be good and useful with help found online without extreme difficulty, but not so common that every idiot with computer knows python. I like the idea of knowing a slightly obscure language.
Thanks a ton for any help you can provide.
Is python powerful enough to handle
most things?
Yes. Period. Study EveOnline game for more information. Look at pygame framework. Free free to use Google to find more.
Does python handle networking tasks
fairly well?
Yes. Look at the number of Python web frameworks plus the Twisted framework. Feel free to use Google to search for Python networking.
Can python source be obfuscated?
Not usefully. This isn't C.
And how popular is python, compared to
other languages?
Look at the TIOBE index.
I think that Python is very powerful to do a lot of things, but just like Java and C++, it often depends on good third-party libraries. I come from a Java background but use Python for a lot of things, and it's been a fun ride. I've done things like statistics, and automation, not sure about the UI though that often depends on the toolkit more than the language.
Python networking works well. I don't know if I'd use it to build a fast algorithmic trading system or a VOIP application, but for most intents and purposes, especially at higher levels of abstraction, it's fine and easy to use. You would need external libraries for things like SSH or FTP.
Python is quite popular and has very good online support, active community, and major corporations (likeGoogle) that use it. I found the official online tutorial and reference to be excellent.
I have to say that I disagree with the "every idiot with a computer" line. There's a difference between knowing a language and using it right, and that's true about every language, even natural ones :) Python does have a lot of functional elements that are not as trivial to use for people coming from a procedural background, so there's always room for growth.
The one problem with Python compared to languages like C and Java is that it is not statically typed. This makes it much faster to write code, but also makes it *much easier) to make mistakes that can be quite nasty to debug. For instance, the same variable can contain a String reference at some point, and a reference to a list of strings at some other point.
Absolutely.
What type of networking? It has socket, http, xml, smtp/pop, telnet, and much more built in.
Python obfuscation won't be nearly as good as a compiled language. Usually that isn't a problem.
It's the 9th most popular tag on stackoverflow, so there's plenty of help available.
Is python powerful enough to handle most things that a typical programmer might do in his off-time? I have in mind things like complex stat generators based on user input for tabletop games, making small games, automate install processes, and build interactive websites, but probably a hundred things along those lines
Definitely. Python is a good tool for all of those except automating install processes, where it might be the right tool but more likely the right tool will likely be decided by what specifically you are automating.
Does python handle networking tasks fairly well?
Yes. You will want to look into Twisted.
Can python source be obfuscated, or is it going to be open-source by nature? The reason I ask this is because if I make something cool and distribute it, I don't want some idiot script kiddie to edit his own name in and say he wrote it
"Open source" refers to the licensing of your code, not the viewability of its source code. Hiding Python source code isn't especially possible, and the results of decompiling Python bytecode will result in much more readable code than the equivalent tools in languages like C. Don't worry about this! You can't prevent people from stealing your car or your computer if they are willing to break the law, and you can't do the same for your code in any language.
And how popular is python, compared to other languages. Ideally, my language would be good and useful with help found online without extreme difficulty, but not so common that every idiot with computer knows python. I like the idea of knowing a slightly obscure language.
This is an unanswerable question. Google will give you lots of conflicting results with different metrics, most of them useful. You're also being a bit silly ;)
As far as learning materials go, I recommend How to Think Like a Computer Scientist, which is a good text that does not presume any existing programming knowledge. It is available for free online, or you could buy a print copy if you prefer. (Don't bother learning 3.x yet. There is not enough library support to do much useful stuff like you want to do, and when there is picking it up will be a breeze; it's not very different than 2.5/6/7.)
Probably yes. Maybe the stat crunching thing will be kinda slow, and maybe a game depending on what kind of game, but generally the performance is good enough, and you save a lot of time on the actual programming. If you REALLY need performance, you can make a module in C, but usually there is a library written to do what you want..
I haven't used it, but there's a framework called Twisted that seems to be pretty good.
No. Bytecode can be decompiled easily, and it only works on a specific version of Python, so your code isn't as portable.
Python is pretty popular, and the Python Package Index has a big list of third party libraries. It's not as widespread as, say, Java, but a lot of people use it and you can probably get answers for what you want.
Points 1 and 2: HELL YEAH.
Point 4: kind of. Python is good at some network stuff. It's not Java or C++. Just use zlib (zip library) and pickle (serialization) for everything, and look at xmlrpclib if you need IPC.
Point 3: No. However, you can write C modules (for the performance critical, and hard-to-copy) parts of your code, and that would make it non-trivial to reverse-engineer.
Python is up to the task (and better) for 1, 2 and 4.
The best solution for 3 from what you describe would probably be to make your programs really open-source with GPL or BSD like licence. This way people will edit your super-cool sources (but often experienced programmers, not just script kiddies) and build on then but leave your name in for posterity.
I'm currently working on a component of a trading product that will allow a quant or strategy developer to write their own custom strategies. I obviously can't have them write these strategies in natively compiled languages (or even a language that compiles to a bytecode to run on a vm) since their dev/test cycles have to be on the order of minutes.
I've looked at lua, python, ruby so far and really enjoyed all of them so far, but still found them a little "low level" for my target users. Would I need to somehow write my own parser + interpreter to support a language with a minimum of support for looping, simple arithmatic, logical expression evaluation, or is there another recommendation any of you may have? Thanks in advance.
Mark-Jason Dominus, the author of Perl's Text::Template module, has some insights that might be relevant:
When people make a template module
like this one, they almost always
start by inventing a special syntax
for substitutions. For example, they
build it so that a string like %%VAR%%
is replaced with the value of $VAR.
Then they realize the need extra
formatting, so they put in some
special syntax for formatting. Then
they need a loop, so they invent a
loop syntax. Pretty soon they have a
new little template language.
This approach has two problems: First,
their little language is crippled. If
you need to do something the author
hasn't thought of, you lose. Second:
Who wants to learn another language?
If you write your own mini-language, you could end up in the same predicament -- maintaining a grammar and a parser for a tool that's crippled by design.
If a real programming language seems a bit too low-level, the solution may not be to abandon the language but instead to provide your end users with higher-level utility functions, so that they can operate with familiar concepts without getting bogged down in the weeds of the underlying language.
That allows beginning users to operate at a high level; however, you and any end users with a knack for it -- your super-users -- can still leverage the full power of Ruby or Python or whatever.
It sounds like you might need to create some sort of Domain Specific Language (DSL) for your users that could be built loosely on top of the target language. Ruby, Python and Lua all have their various quirks regarding syntax, and to a degree some of these can be massaged with clever function definitions.
An example of a fairly robust DSL is Cucumber which implements a an interesting strategy of converting user-specified verbiage to actual executable code through a series of regular expressions applied to the input data.
Another candidate might be JavaScript, or some kind of DSL to JavaScript bridge, as that would allow the strategy to run either client-side or server-side. That might help scale your application since client machines often have surplus computing power compared to a heavily loaded server.
Custom-made modules are going to be needed, no matter what you choose, that define your firm's high level constructs.
Here are some of the needs I envision -- you may have some of these covered already: a way to get current positions, current and historical quotes, previous performance data, etc... into the application. Define/backtest/send various kinds of orders (limit/market/stop, what exchange, triggers) or parameters of options, etc... You probably are going to need multiple sandboxes for testing as well as the real thing.
Quants want to be able to do matrix operations, stochastic calculus, PDEs.
If you wanted to do it in python, loading NumPy would be a start.
You could also start with a proprietary system designed to do mathematical financial research such as something built on top of Mathematica or Matlab.
I've been working on a Python Algorithmic Trading Library (actually for backtesting, not for real trading). You may want to take a look at it: http://gbeced.github.com/pyalgotrade/
Check out http://www.tadeveloper.com for a backtesting framework using MATLAB as a scripting language. MATLAB has the advantage that it is very powerful but you do not need to be a programmer to use it.
This might be a bit simplistic, but a lot of quant users are used to working with Excel & VBA macros. Would something like VBSCript be usable, as they may have some experience in this area.
Existing languages are "a little "low level" for my target users."
Yet, all you need is "a minimum of support for looping, simple arithmatic, logical expression evaluation"
I don't get the problem. You only want a few features. What's wrong with the list of languages you provided? They actually offer those features?
What's the disconnect? Feel free to update your question to expand on what the problem is.
I would use Common Lisp, which supports rapid development (you have a running image and can compile/recompile individual functions) and tailoring the language to your domain. You would provide functions and macros as building blocks to express strategies, and the whole language would be available to the user for combining these.
Is something along the lines of Processing the complexity level that you're shooting for? Processing is a good example of taking a full-blown language (Java) and reducing/simplifying the available syntax into only a subset applicable to the problem domain (problem domain = visualization in the case of Processing).
Here's a little side-by-side comparison from the Processing docs.
Java:
g.setColor(Color.black)
fillRect(0, 0, size.width, size.height);
Processing:
background(0);
As others have suggested, you may be able to simply write enough high-level functions such that most of the complexity is hidden from the user but you still retain the ability to do more low-level things when necessary. The Wiring language for Arduino follows this strategy of using a thin layer of high-level functions on top of C in order to make it more accessible to non-programmers and hobbyists.
Define the language first -- if possible, use the pseudo-language called EBN, it's very simple (see the Wikipedia entry).
Then once you have that, pick the language. Almost certainly you will want to use a DSL. Ruby and Lua are both really good at that, IMO.
Once you start working on it, you may find that you go back to your definition and tweak it. But that's the right order to do things, I think.
I have been in the same boat building and trading with my own software. Java is not great because you want something higher level like you say. I have had a lot of success using the eclipse project xtext. http://www.eclipse.org/Xtext It does all the plumbing of building parsers etc. for you and using eclipse you can quickly generate code with functional editors. I suggest looking into this as you consider other options as well. This combined with the eclipse modeling framework is very powerful for quickly building DSL's which sounds like you need. - Duncan
So this thread is definitely NOT a thread for why Python is better than Ruby or the inverse. Instead, this thread is for objective criticism on why you would pick one over the other to write a RESTful web API that's going to be used by many different clients, (mobile, web browsers, tablets etc).
Again, don't compare Ruby on Rails vs Django. This isn't a web app that's dependent on high level frameworks such as RoR or Django. I'd just like to hear why someone might choose one over the other to write a RESTful web API that they had to start tomorrow, completely from scratch and reasons they might go from one to another.
For me, syntax and language features are completely superfluous. The both offer an abundant amount of features and certainly both can achieve the same exact end goals. I think if someone flips a coin, it's a good enough reason to use one over the other. I'd just love to see what some of you web service experts who are very passionate about their work respond to why they would use one over the other in a very objective format.
I would say the important thing is that regardless of which you choose, make sure that your choice does not leak through your REST API. It should not matter to the client of your API which you chose.
I know Ruby, don't know python... you can see which way I'm leaning toward, right?
Choose the one you're most familiar with and most likely to get things done with the fastest.
Yeah, flip a coin. The truth is that you're going to find minimalist frameworks in either language. Heroku is a pretty strong reason to say Ruby but there may be other similar hosts for Python. But Heroku makes it stupid easy to deploy your api into the cloud whether it's Rails or some other Ruby project that uses Rack. WSGI doesn't give you this option.
As for as the actually implementation though, I'm guessing that you'll find that they're both completely competent languages and both a joy to program in.
I think they are fairly evenly matched in features. I prefer Python, but I have been using it for over a decade so I freely admit that what follows is totally biased.
IMHO Python is more mature - there are more libraries for it (although Ruby may be catching up), and the included libraries I think are better designed. The language evolution process is more mature too, with each proposed feature discussed in public via the PEPs before the decision is made to include them in a release. I get the impression that development of the Ruby language is much more ad-hoc.
Python is widely used in a lot of areas apart from web development - scientific computing, CGI rendering pipelines, distributed computing, Linux GUI tools etc. Ruby got very little attention before Rails came along, so I get the impression that most Ruby work is focused on web development. That may not be a problem if that is all you want to do with the language, but it does mean that Python has a more diverse user base and a more diverse set of libraries.
Python is faster too.
Ruby + Sinatra
Very easy to use with/as rack middleware - someone's already mentioned heroku
Either will do a great job and you'll gain in other ways from learning something new. Why not spend as couple of days with each? See how far you can get with a simple subset of the problem, then see how you feel. For bonus points report back here and answer your own question!
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
Right now I'm developing mostly in C/C++, but I wrote some small utilities in Python to automatize some tasks and I really love it as language (especially the productivity).
Except for the performances (a problem that could be sometimes solved thanks to the ease of interfacing Python with C modules), do you think it is proper for production use in the development of stand-alone complex applications (think for example to a word processor or a graphic tool)?
What IDE would you suggest? The IDLE provided with Python is not enough even for small projects in my opinion.
We've used IronPython to build our flagship spreadsheet application (40kloc production code - and it's Python, which IMO means loc per feature is low) at Resolver Systems, so I'd definitely say it's ready for production use of complex apps.
There are two ways in which this might not be a useful answer to you :-)
We're using IronPython, not the more usual CPython. This gives us the huge advantage of being able to use .NET class libraries. I may be setting myself up for flaming here, but I would say that I've never really seen a CPython application that looked "professional" - so having access to the WinForms widget set was a huge win for us. IronPython also gives us the advantage of being able to easily drop into C# if we need a performance boost. (Though to be honest we have never needed to do that. All of our performance problems to date have been because we chose dumb algorithms rather than because the language was slow.) Using C# from IP is much easier than writing a C Extension for CPython.
We're an Extreme Programming shop, so we write tests before we write code. I would not write production code in a dynamic language without writing the tests first; the lack of a compile step needs to be covered by something, and as other people have pointed out, refactoring without it can be tough. (Greg Hewgill's answer suggests he's had the same problem. On the other hand, I don't think I would write - or especially refactor - production code in any language these days without writing the tests first - but YMMV.)
Re: the IDE - we've been pretty much fine with each person using their favourite text editor; if you prefer something a bit more heavyweight then WingIDE is pretty well-regarded.
You'll find mostly two answers to that – the religous one (Yes! Of course! It's the best language ever!) and the other religious one (you gotta be kidding me! Python? No... it's not mature enough). I will maybe skip the last religion (Python?! Use Ruby!). The truth, as always, is far from obvious.
Pros: it's easy, readable, batteries included, has lots of good libraries for pretty much everything. It's expressive and dynamic typing makes it more concise in many cases.
Cons: as a dynamic language, has way worse IDE support (proper syntax completion requires static typing, whether explicit in Java or inferred in SML), its object system is far from perfect (interfaces, anyone?) and it is easy to end up with messy code that has methods returning either int or boolean or object or some sort under unknown circumstances.
My take – I love Python for scripting, automation, tiny webapps and other simple well defined tasks. In my opinion it is by far the best dynamic language on the planet. That said, I would never use it any dynamically typed language to develop an application of substantial size.
Say – it would be fine to use it for Stack Overflow, which has three developers and I guess no more than 30k lines of code. For bigger things – first your development would be super fast, and then once team and codebase grow things are slowing down more than they would with Java or C#. You need to offset lack of compilation time checks by writing more unittests, refactorings get harder cause you never know what your refacoring broke until you run all tests or even the whole big app, etc.
Now – decide on how big your team is going to be and how big the app is supposed to be once it is done. If you have 5 or less people and the target size is roughly Stack Overflow, go ahead, write in Python. You will finish in no time and be happy with good codebase. But if you want to write second Google or Yahoo, you will be much better with C# or Java.
Side-note on C/C++ you have mentioned: if you are not writing performance critical software (say massive parallel raytracer that will run for three months rendering a film) or a very mission critical system (say Mars lander that will fly three years straight and has only one chance to land right or you lose $400mln) do not use it. For web apps, most desktop apps, most apps in general it is not a good choice. You will die debugging pointers and memory allocation in complex business logic.
In my opinion python is more than ready for developing complex applications. I see pythons strength more on the server side than writing graphical clients. But have a look at http://www.resolversystems.com/. They develop a whole spreadsheet in python using the .net ironpython port.
If you are familiar with eclipse have a look at pydev which provides auto-completion and debugging support for python with all the other eclipse goodies like svn support. The guy developing it has just been bought by aptana, so this will be solid choice for the future.
#Marcin
Cons: as a dynamic language, has way
worse IDE support (proper syntax
completion requires static typing,
whether explicit in Java or inferred
in SML),
You are right, that static analysis may not provide full syntax completion for dynamic languages, but I thing pydev gets the job done very well. Further more I have a different development style when programming python. I have always an ipython session open and with one F5 I do not only get the perfect completion from ipython, but object introspection and manipulation as well.
But if you want to write second Google
or Yahoo, you will be much better with
C# or Java.
Google just rewrote jaiku to work on top of App Engine, all in python. And as far as I know they use a lot of python inside google too.
I really like python, it's usually my language of choice these days for small (non-gui) stuff that I do on my own.
However, for some larger Python projects I've tackled, I'm finding that it's not quite the same as programming in say, C++. I was working on a language parser, and needed to represent an AST in Python. This is certainly within the scope of what Python can do, but I had a bit of trouble with some refactoring. I was changing the representation of my AST and changing methods and classes around a lot, and I found I missed the strong typing that would be available to me in a C++ solution. Python's duck typing was almost too flexible and I found myself adding a lot of assert code to try to check my types as the program ran. And then I couldn't really be sure that everything was properly typed unless I had 100% code coverage testing (which I didn't at the time).
Actually, that's another thing that I miss sometimes. It's possible to write syntactically correct code in Python that simply won't run. The compiler is incapable of telling you about it until it actually executes the code, so in infrequently-used code paths such as error handlers you can easily have unseen bugs lurking around. Even code that's as simple as printing an error message with a % format string can fail at runtime because of mismatched types.
I haven't used Python for any GUI stuff so I can't comment on that aspect.
Python is considered (among Python programmers :) to be a great language for rapid prototyping. There's not a lot of extraneous syntax getting in the way of your thought processes, so most of the work you do tends to go into the code. (There's far less idioms required to be involved in writing good Python code than in writing good C++.)
Given this, most Python (CPython) programmers ascribe to the "premature optimization is the root of all evil" philosophy. By writing high-level (and significantly slower) Python code, one can optimize the bottlenecks out using C/C++ bindings when your application is nearing completion. At this point it becomes more clear what your processor-intensive algorithms are through proper profiling. This way, you write most of the code in a very readable and maintainable manner while allowing for speedups down the road. You'll see several Python library modules written in C for this very reason.
Most graphics libraries in Python (i.e. wxPython) are just Python wrappers around C++ libraries anyway, so you're pretty much writing to a C++ backend.
To address your IDE question, SPE (Stani's Python Editor) is a good IDE that I've used and Eclipse with PyDev gets the job done as well. Both are OSS, so they're free to try!
[Edit] #Marcin: Have you had experience writing > 30k LOC in Python? It's also funny that you should mention Google's scalability concerns, since they're Python's biggest supporters! Also a small organization called NASA also uses Python frequently ;) see "One coder and 17,000 Lines of Code Later".
Nothing to add to the other answers, besides that if you choose python you must use something like pylint which nobody mentioned so far.
One way to judge what python is used for is to look at what products use python at the moment. This wikipedia page has a long list including various web frameworks, content management systems, version control systems, desktop apps and IDEs.
As it says here - "Some of the largest projects that use Python are the Zope application server, YouTube, and the original BitTorrent client. Large organizations that make use of Python include Google, Yahoo!, CERN and NASA. ITA uses Python for some of its components."
So in short, yes, it is "proper for production use in the development of stand-alone complex applications". So are many other languages, with various pros and cons. Which is the best language for your particular use case is too subjective to answer, so I won't try, but often the answer will be "the one your developers know best".
Refactoring is inevitable on larger codebases and the lack of static typing makes this much harder in python than in statically typed languages.
And as far as I know they use a lot of python inside google too.
Well i'd hope so, the maker of python still works at google if i'm not mistaken?
As for the use of Python, i think it's a great language for stand-alone apps. It's heavily used in a lot of Linux programs, and there are a few nice widget sets out there to aid in the development of GUI's.
Python is a delight to use. I use it routinely and also write a lot of code for work in C#. There are two drawbacks to writing UI code in Python. one is that there is not a single ui framework that is accepted by the majority of the community. when you write in c# the .NET runtime and class libraries are all meant to work together. With Python every UI library has at's own semantics which are often at odds with the pythonic mindset in which you are trying to write your program. I am not blaming the library writers. I've tried several libraries (wxwidgets, PythonWin[Wrapper around MFC], Tkinter), When doing so I often felt that I was writing code in a language other than Python (despite the fact that it was python) because the libraries aren't exactly pythonic they are a port from another language be it c, c++, tk.
So for me I will write UI code in .NET (for me C#) because of the IDE & the consistency of the libraries. But when I can I will write business logic in python because it is more clear and more fun.
I know I'm probably stating the obvious, but don't forget that the quality of the development team and their familiarity with the technology will have a major impact on your ability to deliver.
If you have a strong team, then it's probably not an issue if they're familiar. But if you have people who are more 9 to 5'rs who aren't familiar with the technology, they will need more support and you'd need to make a call if the productivity gains are worth whatever the cost of that support is.
I had only one python experience, my trash-cli project.
I know that probably some or all problems depends of my inexperience with python.
I found frustrating these things:
the difficult of finding a good IDE for free
the limited support to automatic refactoring
Moreover:
the need of introduce two level of grouping packages and modules confuses me.
it seems to me that there is not a widely adopted code naming convention
it seems to me that there are some standard library APIs docs that are incomplete
the fact that some standard libraries are not fully object oriented annoys me
Although some python coders tell me that they does not have these problems, or they say these are not problems.
Try Django or Pylons, write a simple app with both of them and then decide which one suits you best. There are others (like Turbogears or Werkzeug) but those are the most used.
I'd like to do some server-side scripting using Python. But I'm kind of lost with the number of ways to do that.
It starts with the do-it-yourself CGI approach and it seems to end with some pretty robust frameworks that would basically do all the job themselves. And a huge lot of stuff in between, like web.py, Pyroxide and Django.
What are the pros and cons of the frameworks or approaches that you've worked on?
What trade-offs are there?
For what kind of projects they do well and for what they don't?
Edit: I haven't got much experience with web programing yet.
I would like to avoid the basic and tedious things like parsing the URL for parameters, etc.
On the other hand, while the video of blog created in 15 minutes with Ruby on Rails left me impressed, I realized that there were hundreds of things hidden from me - which is cool if you need to write a working webapp in no time, but not that great for really understanding the magic - and that's what I seek now.
CGI is great for low-traffic websites, but it has some performance problems for anything else. This is because every time a request comes in, the server starts the CGI application in its own process. This is bad for two reasons: 1) Starting and stopping a process can take time and 2) you can't cache anything in memory. You can go with FastCGI, but I would argue that you'd be better off just writing a straight WSGI app if you're going to go that route (the way WSGI works really isn't a whole heck of a lot different from CGI).
Other than that, your choices are for the most part how much you want the framework to do. You can go with an all singing, all dancing framework like Django or Pylons. Or you can go with a mix-and-match approach (use something like CherryPy for the HTTP stuff, SQLAlchemy for the database stuff, paste for deployment, etc). I should also point out that most frameworks will also let you switch different components out for others, so these two approaches aren't necessarily mutually exclusive.
Personally, I dislike frameworks that do too much magic for me and prefer the mix-and-match technique, but I've been told that I'm also completely insane. :)
How much web programming experience do you have? If you're a beginner, I say go with Django. If you're more experienced, I say to play around with the different approaches and techniques until you find the right one.
The simplest web program is a CGI script, which is basically just a program whose standard output is redirected to the web browser making the request. In this approach, every page has its own executable file, which must be loaded and parsed on every request. This makes it really simple to get something up and running, but scales badly both in terms of performance and organization. So when I need a very dynamic page very quickly that won't grow into a larger system, I use a CGI script.
One step up from this is embedding your Python code in your HTML code, such as with PSP. I don't think many people use this nowadays, since modern template systems have made this pretty obsolete. I worked with PSP for awhile and found that it had basically the same organizational limits as CGI scripts (every page has its own file) plus some whitespace-related annoyances from trying to mix whitespace-ignorant HTML with whitespace-sensitive Python.
The next step up is very simple web frameworks such as web.py, which I've also used. Like CGI scripts, it's very simple to get something up and running, and you don't need any complex configuration or automatically generated code. Your own code will be pretty simple to understand, so you can see what's happening. However, it's not as feature-rich as other web frameworks; last time I used it, there was no session tracking, so I had to roll my own. It also has "too much magic behavior" to quote Guido ("upvars(), bah").
Finally, you have feature-rich web frameworks such as Django. These will require a bit of work to get simple Hello World programs working, but every major one has a great, well-written tutorial (especially Django) to walk you through it. I highly recommend using one of these web frameworks for any real project because of the convenience and features and documentation, etc.
Ultimately you'll have to decide what you prefer. For example, frameworks all use template languages (special code/tags) to generate HTML files. Some of them such as Cheetah templates let you write arbitrary Python code so that you can do anything in a template. Others such as Django templates are more restrictive and force you to separate your presentation code from your program logic. It's all about what you personally prefer.
Another example is URL handling; some frameworks such as Django have you define the URLs in your application through regular expressions. Others such as CherryPy automatically map your functions to urls by your function names. Again, this is a personal preference.
I personally use a mix of web frameworks by using CherryPy for my web server stuff (form parameters, session handling, url mapping, etc) and Django for my object-relational mapping and templates. My recommendation is to start with a high level web framework, work your way through its tutorial, then start on a small personal project. I've done this with all of the technologies I've mentioned and it's been really beneficial. Eventually you'll get a feel for what you prefer and become a better web programmer (and a better programmer in general) in the process.
If you decide to go with a framework that is WSGI-based (for instance TurboGears), I would recommend you go through the excellent article Another Do-It-Yourself Framework by Ian Bicking.
In the article, he builds a simple web application framework from scratch.
Also, check out the video Creating a web framework with WSGI by Kevin Dangoor. Dangoor is the founder of the TurboGears project.
If you want to go big, choose Django and you are set. But if you want just to learn, roll your own framework using already mentioned WebOb - this can be really fun and I am sure you'll learn much more (plus you can use components you like: template system, url dispatcher, database layer, sessions, et caetera).
In last 2 years I built few large sites using Django and all I can say, Django will fill 80% of your needs in 20% of time. Remaining 20% of work will take 80% of the time, no matter which framework you'd use.
It's always worth doing something the hard way - once - as a learning exercise. Once you understand how it works, pick a framework that suits your application, and use that. You don't need to reinvent the wheel once you understand angular velocity. :-)
It's also worth making sure that you have a fairly robust understanding of the programming language behind the framework before you jump in -- trying to learn both Django and Python at the same time (or Ruby and Rails, or X and Y), can lead to even more confusion. Write some code in the language first, then add the framework.
We learn to develop, not by using tools, but by solving problems. Run into a few walls, climb over, and find some higher walls!
If you've never done any CGI programming before I think it would be worth doing one project - perhaps just a sample play site just for yourself - using the DIY approach. You'll learn a lot more about how all the various parts work than you would by using a framework. This will help in you design and debug and so on all your future web applications however you write them.
Personally I now use Django. The real benefit is very fast application deployment. The object relational mapping gets things moving fast and the template library is a joy to use. Also the admin interface gives you basic CRUD screens for all your objects so you don't need to write any of the "boring" stuff.
The downside of using an ORM based solution is that if you do want to handcraft some SQL, say for performance reasons, it much harder than it would have been otherwise, although still very possible.
If you are using Python you should not start with CGI, instead start with WSGI (and you can use wsgiref.handlers.CGIHandler to run your WSGI script as a CGI script. The result is something that is basically as low-level as CGI (which might be useful in an educational sense, but will also be somewhat annoying), but without having to write to an entirely outdated interface (and binding your application to a single process model).
If you want a less annoying, but similarly low-level interface, using WebOb would provide that. You would be implementing all the logic, and there will be few dark corners that you won't understand, but you won't have to spend time figuring out how to parse HTTP dates (they are weird!) or parse POST bodies. I write applications this way (without any other framework) and it is entirely workable. As a beginner, I'd advise this if you were interested in understanding what frameworks do, because it is inevitable you will be writing your own mini framework. OTOH, a real framework will probably teach you good practices of application design and structure. To be a really good web programmer, I believe you need to try both seriously; you should understand everything a framework does and not be afraid of its internals, but you should also spend time in a thoughtful environment someone else designed (i.e., an existing framework) and understand how that structure helps you.
OK, rails is actually pretty good, but there is just a little bit too much magic going on in there (from the Ruby world I would much prefer merb to rails). I personally use Pylons, and am pretty darn happy. I'd say (compared to django), that pylons allows you to interchange ints internal parts easier than django does. The downside is that you will have to write more stuff all by youself (like the basic CRUD).
Pros of using a framework:
get stuff done quickly (and I mean lighning fast once you know the framework)
everything is compying to standards (which is probably not that easy to achieve when rolling your own)
easier to get something working (lots of tutorials) without reading gazillion articles and docs
Cons:
you learn less
harder to replace parts (not that much of an issue in pylons, more so with django)
harder to tweak some low-level stuff (like the above mentioned SQLs)
From that you can probably devise what they are good for :-) Since you get all the code it is possible to tweak it to fit even the most bizzare situations (pylons supposedly work on the Google app engine now...).
For smaller projects, rolling your own is fairly easy. Especially as you can simply import a templating engine like Genshi and get alot happening quite quickly and easily. Sometimes it's just quicker to use a screwdriver than to go looking for the power drill.
Full blown frameworks provide alot more power, but do have to be installed and setup first before you can leverage that power. For larger projects, this is a negligible concern, but for smaller projects this might wind up taking most of your time - especially if the framework is unfamiliar.