When researching Google App Engine (GAE), it's clear that using Django is wildly popular for developing in Python on GAE. I've been scouring the web to find information on the costs and benefits of using Django, to find out why it's so popular. While I've been able to find a wide variety of sources on how to run Django on GAE and the various methods of doing so, I haven't found any comparative analysis on why Django is preferable to using the webapp framework provided by Google.
To be clear, it's immediately apparent why using Django on GAE is useful for developers with an existing skillset in Django (a majority of Python web developers, no doubt) or existing code in Django (where using GAE is more of a porting exercise). My team, however, is evaluating GAE for use on an all-new project and our existing experience is with TurboGears, not Django.
It's been quite difficult to determine why Django is beneficial to a development team when the BigTable libraries have replaced Django's ORM, sessions and authentication are necessarily changed, and Django's templating (if desirable) is available without using the entire Django stack.
Finally, it's clear that using Django does have the advantage of providing an "exit strategy" if we later wanted to move away from GAE and need a platform to target for the exodus.
I'd be extremely appreciative for help in pointing out why using Django is better than using webapp on GAE. I'm also completely inexperienced with Django, so elaboration on smaller features and/or conveniences that work on GAE are also valuable to me.
Django probably isn't the right choice for you, if you're sure that GAE is right for you. The strengths of the two technologies don't align very well - you completely lose a lot of Django's wonderful orm on GAE, and if you do use it, you write code that isn't really directly suitable to bigtable and the way GAE works.
The thing about GAE is that it gets the great scalability by forcing you to write code that scales easily from the ground up. You just can't do a number of things that scale poorly (of course, you can still write poorly scaling code, but you avoid some pitfalls). The tradeoff is that you really end up coding around the framework, if you use something like Django which is designed for a different environment.
If you see yourself ever leaving GAE for any reason, becoming invested in the infrastructure there is a problem for you. Coding for bigtable means that it will be harder to move to a different architecture (though the apache project is working to solve this for you with the HBase component of the Hadoop project). It would still be a lot of work to transition off of GAE.
What's the driving motivator behind using GAE, besides being a Google product, and a cool buzzword? Is there a reason that scaling using something like mediatemple's offering is unlikely to work well for you? Are you sure that the ways that GAE scales are right for your application? How does the cost compare to dedicated servers, if you're expecting to get to that performance realm? Can you solve your problem well using the tools GAE provides, as compared to a more traditional load-balanced server setup?
All this said, unless you absolutely positively need the borderline-ridiculous scaling that GAE offers, I'd personally suggest not letting that particular service structure your choice of framework. I like Django, so I'd say you should use it, but not on GAE.
Edit (June 2010):
As an update to this comment sometime later:
Google has announced sql-like capabilitys for GAE that aren't free, but will let you easily do things like run SQL-style commands to generate reports on your data.
Additionally, there are upcoming changes to the GAE query language which will allow complex queries in a far easier fashion. Look at the videos from Google I/O 2010.
Furthermore, there is work being done during the Summer of Code 2010 project which should bring no-sql support to django core, and by extension, make working with GAE significantly easier.
GAE is becoming more attractive as a hosting platform.
Edit (August 2011):
And Google just raised the cost to most users of the platform significantly by changing the pricing structure. The lockin problem has gotten better (if your application is big enough you can deploy the apache alternatives), but for most applications, running servers or VPS deployments is cheaper.
Very few people really have bigdata problems. "Oh my startup might scale someday" isn't a bigdata problem. Build stuff now and get it out the door using the standard tools.
We use django on our appengine instances mostly when we have to serve actual websites to the user. It has a great template engine, url routing and all the request/response/error handling built in. So even while we can't use the magic orm/admin stuff it has a lot going for it.
For api services we built something very simple on top of webob. It's far more lightweight because it doesn't need everything that django offers, and therefore a little quicker in some situations.
I've done lots of projects on GAE. Some in django, some in their normal framework.
For small things, I usually use their normal framework for simplicity and quickness. Like http://stdicon.com, http://yaml-online-parser.appspot.com/, or http://text-twist.appspot.com/.
For large things, I go with django to take advantage of all the nice middleware and plugins. Like http://metaward.com.
Basically my litmus test is Will this take me more than 2 weeks to write and be a REAL software project? If so, go with django for the addons.
It has the added benefit of, if your project is badly suited for BigTable then you quickly port off (like I did Is BigTable slow or am I dumb?)
I think all this answers are a bit obsolete.
Now you can use Google Cloud SQL
Django is a popular third-party Python web framework. When coupled
with Google Cloud SQL, all of its functionality can be fully supported
by applications running on App Engine. Support for using Google Cloud
SQL with Django is provided by a custom Django database backend which
wraps Django's MySQL backend.
https://cloud.google.com/python/django/appengine
one more fresh news is, that there is BETA support for PostgreSQL
I have experience using Django and not GAE. From my experiences with Django it was a very simplistic setup and the deployment process was incredibly easy in terms of web projects. Granted I had to learn Python to really get a good hold on things, but at the end of the day I would use it again on a project. This was almost 2 years ago before it reached 1.0 so I my knowledge is a bit outdated.
If you are worried about changing platforms, then this would be a better choice I suppose.
I cannot answer the question but you may want to look into web2py. It is similar to Django in many respects but its database abstraction layer works on GAE and supports most of the GAE functionality (not all but we try to catch up). In this way if GAE works for you great, if it does not, you can move your code to a different db (SQLite, MySQL, PostgreSQL, Oracle, MSSQL, FireBird, DB2, Informix, Ingres, and - soon - Sybase and MongoDB).
If you decide to run you app outside of GAE, you can still use Django. You won't really have that much luck with the GAE webapp
I am still very new to Google App engine development, but the interfaces Django provides do appear much nicer than the default. The benefits will depend on what you are using to run Django on the app engine. The Google App Engine Helper for Django allows you to use the full power of the Google App Engine with some Django functionality on the side.
Django non-rel attempts to provide as much of Django's power as possible, but running on the app-engine for possible extra scalability. In particular, it includes Django models (one of Django's core features), but this is a leaky abstraction due to the differences between relational databases and bigtable. There will most likely be tradeoffs in functionality and efficiency, as well as an increased number of bugs and quirks. Of course, this might be worth it in circumstances like those described in the question, but otherwise would strongly recommend using the helper at the start as then you have the option of moving towards either pure app-engine or Django non-rel later. Also, if you do switch to Django non-rel, your increased knowledge of how app engine works will be useful if the Django abstraction ever breaks - certainly much more useful than knowledge of the quirks/workarounds for Django non-rel if you swap the other way.
Related
I'm a data scientist, and I make apps using python. Where I make a lot of connections with databases, build and deploy my code on the cloud and I use models training and visualization.
My question is:
Is Django useful for making non-web apps?
if not are they any other frameworks? (similar to spring-boot for java)
Best regards
Django is a web framework which comes with many preinstalled apps. This will make your application bulkier than it is required.
For your usecase, you need not to use any frameworks. Just code it in python and you can use the available library to perform db operations.
If you are deploying your app on the cloud, and want expose some endpoints to trigger any of your internal functions, go with Flask framework. It is a light weight micro web framework
I would contrast the first answer and say absolutely you can use Django with no intentions of building a front end. The ORM is so powerful and easy to use along with the out-of-the-box signaling system among many other features. In my opinion a little extra "bulk" is worth the tools you have at your disposal with it. And of course you always have the option to add the front end later if you like.
I plan to develop a rather database heavy (~100 tables) web application in python. The focus is on providing a nice and task-optimized interface for people that edit or navigate through the data. Other focuses are:
Handle lots of data and complex queries.
Internationalization (translation, timezones, currencies)
Mailings (bulk emailing as well as notifications)
Easy integration into other websites (pull data from or push data to the application)
A role based authentication scheme. (ideally enforcing one role at a time)
It should be easy and fast (for python programmers) to create custom forms and workflows to work with the data.
I've read a lot about django, turbogears, pyramid, webcore, … but I'm still having a hard time to figure out where to start.
My current evaluation would suggest that turbogears is the way to go. Pyramid seems too much to learn about. Django seems to be too publishing focused. WebCore seems a bit to immature to base such a project on it.
Am I overlooking something? Are there other more suitable python frameworks? Is my information about some of them plain wrong? Which framework would you choose for this project, and why?
Imo the only part of django that might be "too" publishing orientated is the admin, but I have seen plenty of django applications doing stuff neatly.
Django has plenty of apps available covering what you want to do, but the only road block you might find is the part of: handle lots of data and complex queries. You will probably move out of django ORM land, but you might even move out of SQLAlchemy land too. Most of these projects use ORM's, so I would look into SQLAlchemy first, and evaluate how to use it for your needs.
Second, I would just go through the tutorials of the following projects, reading about them is good, but a small little tutorial/project (or mini prototype) is the only way to see if the project fits your programming style: pyramid, turbogears, and django. They have afaik the largest communities. The best tool will be the one you feel more confortable with. They all have good, excellent documentation, good supportive communities, and are mature enough for solid projects, and for very subtle differences, you probably can use any of them for your needs.
I have to agree with you: if you already have a database model, Django is not potentially the best way to go.
As of database binding, SQL Alchemy is definitely worth checking out, regardless of which framework you choose.
Some additions to your list:
Web.py - A little low-level, but a nice, mature framework (for SQLAlchemy usage, see cookbook on SQLAlchemy)
Tornado - Very good performance, supports Websockets which could be of a concern for some applications.
Edit: nowadays I would recommended Flask instead of Web.py. Tornado works really nicely with it as well.
The number of tables is not relevant for speeds etc. and not relevant for the choice of the framework. Recommendation: use SQLAlchemy as ORM between database and application. Go for Pyramid as web framework. Pyramid is easy, well-documented, test and very flexible in all aspects. Forms etc. can be easily created using "colander" + "deform" add-ons.
My gut says you want to use SQLAlchemy as the ORM. Turbogears does this out of the box, and probably is the largest player in the "not Django" space.
There was some work on pulling in SQLAlchemy for (or in addition to!) Django's ORM, but I don't know how current that work is (a quick google search found articles from 2008-2009 as the top hits)
I have no background in web applications, but have a fairly experience background in C++, and a quick learner.
I have spent some time learning Python and reading through SQLAlchemy. I kind of like the idea of coding in pure Python OO, and then use a nice SQLAlchemy mapper to persist everything. I like this decoupled approach (using pure Python classes along a mapper function to talk to DB) better than the ActiveRecord idea of Rails. I think eventually I would have more control over connecting the DB to the app. (I need to work with a DB that is updated by a background process. Something like a web crawler that fills the DB.)
At the same time, some stuff makes me think again about Rails. Like streamlined Email and Ajax handling in Rails.
Am I thinking the right way, that Rails is less flexible for Form Validation Manipulations, and working with DB? And is it harder in Pylons to handle Email (notifications), RSS, Ajax?
What would you suggest?
Thanks
Have a look at Django. It sounds like this exactely what you are looking for :-).
Have a look at these Python frameworks:
Django: Probably the single most popular Python framework, but for better (and worse) is very-much a full-stack solution.
Pylons: As reaction to the Django One Way of working, Pylons, for better (and worse), uses a much looser binding of the modules that make up your framework.
TurboGears: As an attempt at a happy medium between Django and Pylons, TurboGears is based on Pylons, but comes ready made with certain component choices, and the glue to hold it together.
Zope: Zope is more of an application server and framework than a "web framework". It just happens to be web based.
The first three are all inspired by Ruby on Rails, but each has it's own ideas for improvement. Zope predates Rails, and is it's own world.
I've used TurboGears to develop a few small apps. Kinda nice. At the time, their docs were kinda bad. I hope that's changed.
I've also directly used Python Paste a few times. Paste is the HTTP server base upon which Pylons, and in turn TurboGears, is based. Again very nice.
Also: When given the choice, I've always used SQLAlchemy as an ORM. It's truly an impressive piece of software that I've used for even non-web projects.
Hope this helps. Let us know what path you take. :-)
Rails is written in Ruby, not Python. If you have your heart set on Python, then go with Django. But please give Rails a fair shake; ActiveRecord is not the only ORM available either. I use DataMapper for some apps too. I may be biased, but I'm inclined to think the Rails ecosystem is bigger than that of Django too.
If you are looking for a project similar to rails, you should check out Masonite, A modern and developer-centric web framework in Python
You should also checkout web2py instead of Django. Just an alternative you might consider.
Here's two "A vs. B" articles and discussions regarding the two:
Django vs. web2py
Reddit.com community discussion
I wrote a few websites in Pylons over the years and I like it a lot. The great things about Pylons is that it consists mostly of third party libraries. That means that you're learning many useful libraries that can be used in you other projects, for example SQLAlchemy, WebOb, FormEncode, Beaker, Mako and so on ... Especially SQLAlchemy and Beaker are extremely useful in pretty much any context.
You could also have a look to Nagare, another full stack framework.
Some of Nagare based projects already in production can be found on the Nagare web site.
I used Web2Py for many small projects, including many goodies such as the "Workers" & Scheduler concepts, some event-driven updates in web page through the short tornado example in websocket_messaging.py. If you're looking for a small but powerful development framework that includes a small DB and display tables, it's just amazing. You even do not need to write a single HTML line. I do not see any competitor in this area. In my opinion it's far easier and faster than django, but django might provide much freedom in complex apps.
After much research, I've come up with a list of what I think might be the best way of putting together a Python based social network/cms, but have some questions about how some of these components fit together.
Before I ask about the particular components, here are some of the key features of the site to be built:
a modern almost desktop-like gui
future ability to host an advanced html5 sub-application (ex.http://www.lucidchart.com)
high scalability both for functionality and user load
user ability to password protect and permission manage content on per item/group basis
typical social network features
ability to build a scaled down mobile version in the future
Here's the list of tools I'm considering using:
Google App Engine
Python
Django
Pinax
Pyjamas
wxPython
And the questions:
Google App Engine -- this is an attempt to cut to the chase as many pieces of the puzzle seem to be in place.
Question: Am I limiting my options with this choice? Example: datastore not being relational? Should I wait
for SQL support under the Business version?
Python -- I considered 'drupal' at first, but in the end decided that being dependent on modules that may or
may not exist tomorrow + limitations of its templating system are a no-no. Learning its API, too, would be useless elsewhere
whereas Python seems like a swiss army knife of languages -- good for almost anything.
Question: v.2.5.2 is required by GAE, but python.org recommends 2.5.5. Which do I install?
Django -- v.0.96 is built into GAE. You seem to be able to upgrade it.
Questions: Any reason not to upgrade to the latest version? Ways to get around the lack of HTML5 support?
Pinax (http://pinaxproject.com) Rides on top of Django and appears to provide most of the social network functionality
anyone would want.
Question: Reasons NOT to use it? Alternatives?
Pyjamas and wxPython -- this is the part that gets a little confusing. The basic idea behind these is the ability
to build a GUI. I've considered Silverlight and Flash, before the GAE/Python route, but a few working versions of
HTML5 apps convinced me that enough of it ALREADY runs on the latest batch of browsers to chose the HTML5/Javascript
route instead.
Question: How do I extend/supplement Python/Django to build an app-like HTML5 interface? Are Pyjamas and wxPython
the way to go? Or should I change my thinking completely?
Answers to some/any of these questions would be of great help. Please excuse my ignorance if any of this doesn't make much sense.
My last venture into web programming was a decent sized LAMP website some 5-6 years ago. On the desktop side of things,
my programming experience boils down to very high level scripting languages that I keep on learning to accomplish very specific
tasks :)
As someone who has deployed a Django site to GAE, I can tell you that you are not going to reach the ideal solution. Django on GAE misses some of the best aspects of Django because the ORM doesn't work right. The best compromise may be to use Django-nonrel to add the features back in.
This introduces it's own problems though: because of the large number of files and memory used by a Django app you're code will be unloaded from memory quickly after the app becomes idle. That means that visitors will frequently hit an approximately 6 second delay on the first page view after the site's code has been unloaded from memory while GAE uncompresses the zipped modules. Once your site is busy this won't be a problem, but while your site is still young and unknown it will cause the appearance of performance problems. :-(
Second, I've also worked for a company that built a custom CMS and can tell you that the first 80% is pretty easy, especially with modern frameworks. However, the rest can be quite challenging. For example, user roles and custom content types are two challenging aspects. Therefore strongly consider standing on the backs of giants and finding a CMS or CMS framework that almost perfectly meets your needs and then extend it to do that extra bit you need.
So, that said, answering your points:
Yes, you're limiting your options but that may be OK. Most developers are more comfortable with the relational model than the nosql model. Therefore more open source software is built with it in mind. Also, GAE is a closed source platform which is also a deterrent to open source developers. App Engine Oil is a CMS framework that may suit you well and is optimized for App Engine. Also look at web2py which has support for GAE.
I've found myself to be extremely productive with Python. I used to write a lot of PHP now I find it ugly. That said, think about the total line count of code you'll have to write. If you can make Drupal work with high quality pre-made modules you may find yourself only needing 1/10th of the code. By the way, the trick with Drupal is to mainly use only high quality modules. Look at the history, make sure not to use development versions. Try to contact the authors on IRC. I'm not saying you should use Drupal but it is possible to have a reliable site with it (for example, whitehouse.gov)
You're in the classic GAE/Django problem. If you use 0.96 you get great performance but you miss a lot of the great 1.0+ features and you don't get the ORM and all of it's benefits. If you use a newer version of Django you get the performance/memory problems mentioned above.
I'm about to investigate pinax for my company. I've done a very cursor glance at it. I don't know if it has good support for non relational model backends. You'll probably need to look at django-nonrel. However know that you're going to be investing in relatively untried solutions here. A small percentage of Django users use Pinax and an even smaller percentage, if any, use it on a nonrelational backend. Therefore you're going to be in the highly experimental scenario you mentioned in point 2 above.
I can't offer personal experience on it. I've investigated pyjamas a few times. However I like writing HTML CSS and JS. I like to have control. I like progressive enhancement and knowing what users will see if they don't have the full capabilities. Also, I think any new app that doesn't explicitly address mobile clients is implicitly shooting themself in the foot. As many as 15% of Internet users only use the Internet via their smart phone. What kind of experience will they get with pyjamas?
You didn't mention this, but one thing I consider when choosing a platform is vendor lockin and portability. If you develop your solution for GAE and find that you're not able to do what you want, will you be able to port it to another solution elsewhere? How much work will it take? If you code heavily for GAE or make commitments to its architecture, you're stuck with it or with rewriting to move. Using Django or Web2py can help mitigate this.
That said, the big benefit of Python GAE is that you get to be very productive, see your results instantly, get hosting for free while your site is small and get excellent scalability. These are not small things. There is great value there.
Django is my favorite python web framework. I've tried out others like pylons, web2py, nevow and others.
But I've never looked into TurboGears with much enthusiasm.
Now with TG2 out of beta I may give it a try. I'd like to know what are some of the pros and cons compared to Django.
TG2 has several advantages that I think are important:
Multi-database support
sharding/data partitioning support
longstanding support for aggregates, multi-column primary keys
a transaction system that handles multi-database transactions for you
an admin system that works with all of the above
out of the box support for reusable template snipits
an easy method for creating reusable template tag-libraries
more flexibility in using non-standard components
There are more, but I think it's also important to know that Django has some advantages over TG2:
Larger, community, more active IRC channel
more re-usable app-components
a bit more developed documentation
All of this means that it's a bit easier to get started in Django than TG2, but I personally think the added power and flexibility that you get is worth it. But your needs may always be different.
TG2 takes Pylons and changes some defaults - object dispatching instead of Routes, and Genshi instead of Mako. They believe there's only one way to do it, so apps can rely on the same API for any TurboGears website.
Similarities
TG2 and Django both distinguish between websites and components, so you'll eventually see reusable building blocks for TurboGears, too.
Differences
Django uses its own handlers for HTTP, routing, templating, and persistence. Django also has stellar documentation and an established community.
TurboGears defaults to best-of-breed libraries, which apparently are Paste, object dispatching, Genshi, and SqlAlchemy. This philosophy produces a better all-round toolset, but at the risk of instability - because it means throwing away backwards compatibility if and when better libraries appear.
Pros.
SQLAlchemy > django ORM
Multiple template languages out of the box (genshi,mako,jinja2)
more WSGI friendly
Object Dispatch > routes > regexp routing. You can get the first 2 with TG2
Almost all components are optional you can keep the core and use any ORM, template, auth library, etc.
Sprox > django forms
Cons.
- Admin is more basic (no inline objects yet!)
- less third party apps
- "app" system still in the making.
- given it's modularity you need to read documentation from different sources (SQLAlchemy, Genshi or Mako, repoze.who, Pylons, etc.)
I was struggling with the same question months ago and decided for Turbogears 2, and my reasoning was simple. "I'm new to python, I want to learn it not just for web-projects but as a substitute to php for scripting small helpers"
What I didn't like about Django, to me looks like a "close platform". ORM, Template system, sessions, etc they all are Django's
On the other hand, Turbogears 2 uses already known open platforms and just glued them, just like Appfuse does it for Java
With TurboGears 2 I learn SQLAlchemy that I can use later for small python scripts, or from the python shell to solve common tasks.
Main drawbacks are the lack of complete documentation and error messages.
Sometimes you have to search very deep to find simple solutions, the learning curve is steep, but it pays long term. The error messages where to me very confusing (coming from more than 10 years in Java development). I had lost many hours trying to find an "ascii encode error" when the real problem was a module not being imported.
That's my opinion, just remember I'm new to python and I could be wrong about many things stated here.
Besides what Nikhil gave in his answer, I think another minor difference is that Turbogears provdes some support for javascript widgets and integration with Mochikit.
Whereas Django steadfastly remains javascript framework neutral.
(At least this was true with older versions of Turbogears... this might have changed with TG2)
Edit: I just went over TG2 documentation and see that it did indeed change. Turbogears now uses ToscaWidgets which can use jQuery, ExtJS, Dojo, etc. underneath. This nicely makes it more framework neutral while still providing nice javascript widgets.
This strikes me as a pro for Turbogears if you don't have any javascript experience and a pro for Django if you are writing a lot of specialized javascript.
One of the most important questions is not just what technical features this platform provides or that platform provides, but the driving philosophy of the open source project and the nature of the community supporting it.
I've got no dog in this fight myself, but I found Mark Ramm's talk at DjangoCon 2008 to be very interesting on this point (Google will yield no end of subsequent discussion, no doubt).
Because Django uses its own ORM it limits you to learn that ORM for that specific web framework. I think using an web framework with a more popular ORM (like SqlAlchemy which TG uses) increases your employability chances. Just my 2 cents ..
Last I checked, django has a very poor data implementation. And that's a huge weakness in my book. Django's orm doesn't allow me to use the power of the underlying database. For example I can't use compound primary keys, which are important to good db design. It also doesn't support more than a single database, which is not a big deal until you really need it and find that you can't do it without resorting to doing it manually. Lastly if you have to make changes to your database structure in a team-friendly way, you have to try to choose between a set of 3rd party migration tools.
Turbogears seems to be more architecturally sound, doing its best to integrate individual tools that are awesome in their own right. And because TG is more of an integrator, you're able to switch out pieces to suit your preferences. Don't like SQL Alchemy? You can use SQLObject. Don't like Genshi templates? You can use Mako or even django's, although you're not exactly stuck with the default on django either.
Time for tg2's cons:
TG has a much smaller community, and community usually has its benefit.
Django has a much better name. I really like that name ;-)
Django seems simpler for the beginning web developer, with pretty cool admin tools.
TG has decent documentation, but you also need to go to Genshi's site to learn Genshi, SQL Alchemy's site to learn that, etc. Django has great docs.
My 2 cents.