We're up to building a "Accounting Software (will call it as AS)" for mid and large sized companies. So AS is going to be comprehensive and have a lot of modules in relation. AS will run on cloud and has SOA approach.
What I'd like to ask is: Is using Python + Tornado good idea for development? What are the advantages and disadvantages? Especially when features like async (non-blocking), multithreading etc. are considered.
If you do not support this idea, which infrastructure is the best for our future AS you think?
Tornado is a good decision, if you need a lot of realtime events to be shown in your web application. For example chat (event: deliver new messages to all members of chat) or maybe some other actions (someone gives you a like and you know about it immediately). This is where async approach have all pros.
Databases
When you choose database, keep in mind, that you need an async driver for it.
For example to use MongoDB best choice is motor. To use Postgresql you probably need a momoko.
The cons of tornado are:
hard to start coding, if you are not familiar with async approach. For example, with django (most popular blocking python web framework) it is more easier to start, you have a lot of batteries included
smaller community, than django has
no ORM included
no admin part of site, you'll need to create it by yourself
Also here you can find some additional thoughts about this topic and an example of tornado application.
Related
I wanted to ask you about microservices in Python. As of writing this, i got pretty good with writing well structured flask-restful APIs and I wanted to go and learn about microservices in python.
Right now I have read up quite a lot of info regarding this and even searched online to find examples for this (1 example here) but I am not really sure exactly where to start as I don't want to invest too much time in a inefficient pattern.
So I wanted to know if anyone know any courses with examples for Python on building high load services. My only hints so far as asyncio and aiohttp for request handling and i'm not sure if using a message broker (such as zeromq or rabbitmq) would be a good idea as from what I read, it adds request lag.
Any advice would be great.
PS: The current pattern I'm stuck on is the API Gateway pattern and I would also want to know if it is a good direction as a start.
There are plenty of microservice frameworks in Python that can handle high load and get you a long way towards following best practices.
Try for example pymacaron (http://pymacaron.com/). Pymacaron is basically a flask app whose endpoints are auto-spawn from a swagger specification. To write a pymacaron microservice, you mostly have to:
(1) write a swagger specification for your api (which is always a good starting point, whatever language you are using). Your swagger file describes the get/post/etc calls of your api and which objects (json dicts) they get and return, but also which python method in your code that implement the endpoint.
(2) and implement your endpoints' methods.
Once you have done that, you get loads of things for free: you can package your code as a docker container, deploy it to amazon beanstalk, start asynchronous tasks from within your api calls, or get the api documentation with no extra work.
Here is an example of an helloworld api implemented with pymacaron: https://github.com/pymacaron/pymacaron-helloworld
I'm struggling with some architectural choices for a scalable internet-of-things application.
I've chosen to base my project on Twisted augmented with the Cyclone framework to provide many Tornado convenances (websockets, auth-decorators, secure-cookies, etc)
Using a Twisted core has worked beautifully for me. I have numerous IP protocol and hardware interfaces all of which turned out to have great library support inside of twisted (and adding new protocols and interfaces to my application are the most-likely angles I'll have project scope creep), all with Twisted needing very low CPU and providing for very high connection counts.
My problems are with second-order webapp functionality.
I pulled in Cyclone thinking that with it's auth goodies (OpenID, oauth, user-auth decorators and secure-cookies) it wouldn't take much to implement user/session/admin functionality in my webapp. After the 500+ lines of abstracting my database (via txmongo) and just building user logins it became clear I both:
Didn't understand how little Cyclone/Tornado bring in the user/session/admin space, and
Didn't understand the amount of code it takes to fill in the gaps if your trying to build a multi-user auth webapp
A friend pointed me at Flask, which initially I thought was completely redundant, until I found flask plugins. The combination of Flask-Login and Flask-Admin would completely cover my user, session and user-admin needs, negating me writing what I would guess to be about 2k lines of code. Unfortunately, the flask plugins are all rife with blocking code and calls to blocking libraries. I don't see them as compatible with my project even if WSGI containers are used given that the user/session functionality happens with every page load (additionally I don't see any short cuts that would allow me to port them to async world without work roughly equal to that of rewriting them)
My question is:
In the python async space (... hopefully in the Twisted space, given my protocol needs), are there any plugins or alternate frameworks that provide ready-to-go user/login/admin functionality similar to what is in Flask-Login and Flask-Admin?
P.S. I've looked at Klein as the obvious Twisted version of Flask, but it doesn't seem to have a plugin ecosystem, and I'm not finding any strong user/session/admin there.
P.P.S. By the time I wrote this question I had already written my own (crappy) user-login-session system. So what I'm really after is the "Admin" capability (automated CRUD functions on user-style records, including web UI rendering, all designed in a Twisted/async way). I asked about user/login in the question in case it turn out there is an already-integraded solution (such as flask-login and flask-admin) in which case I would happily drop my code and switch to that.
Do you really need everything async? Consider async WebSockets but sync page renders. If you must, add an async downstream proxy or load balancer which will virtually eliminate app server's IO overhead.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Which of these frameworks / libraries would be the best choise for building modern multiuser web application? I would love to have an asynchronous webserver which will allow me to scale easly.
What solution will give the best performance / scalability / most useful framework (in terms of easy of use and easy of developing)?
It would be great if it will provide good functionality (websockets, rpc, streaming, etc).
What are the pros and cons of each solution?
"Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design". If you are building something that is similar to a e-commerce site, then you should probably go with Django. It will get your work done quick. You dont have to worry about too many technology choices. It provides everything thing you need from template engine to ORM. It will be slightly opinionated about the way you structure your app, which is good If you ask me. And it has the strongest community of all the other libraries, which means easy help is available.
"Flask is a microframework for Python based on Werkzeug, Jinja 2 and good intentions". Beware - "microframework" may be misleading. This does not mean that Flask is a half-baked library. This mean the core of flask is very, very simple. Unlike Django, It will not make any Technology decisions for you. You are free to choose any template engine or ORM that pleases you. Even though it comes with Jinja template engine by default, you are always free to choose our own. As far as I know Flask comes in handy for writing APIs endpoints (RESTful services).
"Twisted is an event-driven networking engine written in python". This is a high-performance engine. The main reason for its speed is something called as deferred. Twisted is built on top of deferreds. For those of you who dont know about defereds, it is the mechanism through with asynchronous architecture is achieved. Twisted is very fast. But is not suitable for writing conventional webapps. If you want to do something low-level networking stuff, twisted is your friend.
"Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user". Tornado stands some where between Django and Flask. If you want to write something with Django or Flask, but if you need a better performance, you can opt for Tornado. it can handle C10k problem very well if it is architected right.
"Cyclone is a web server framework for Python that implements the Tornado API as a Twisted protocol". Now, what if you want something that is nearly as performant as Twisted but easy to write conventional webapps? Say hello to cyclone. I would prefer Cyclone over Tornado. It has an API that is very similar to Tornado. As a matter of fact, this is a fork of Tornado. But the problem is it has relativly small community. Alexandre Fiori is the only main commiter to the repo.
"Pyramid is a general, open source, Python web application development framework. Its primary goal is to make it easier for a Python developer to create web applications." I haven't really used Pyramid, but I went through the documentation. From what I understand, Pyramid is very similar to Flask and I think you can use Pyramid wherever Flask seems appropriate and vice-versa.
EDIT: Request to review any other frameworks are welcomed!
Source: http://dhilipsiva.com/2013/05/19/python-libraries-django-twisted-tornado-flask-cyclone-and-pyramid.html
This is obviously a somewhat biased answer, but that is not the same thing as a wrong answer; you should always use Twisted. I've answered similar questions before, but since your question is not quite the same, here are some reasons:
"Best Performance"
Twisted continuously monitors our performance at the speed.twistedmatrix.com website. We were also one of the first projects to be monitored by PyPy's similar site, thereby assuring the good performance of Twisted on the runtime that anyone concerned with high-performance applications in Python.
"Scalability"
To my knowledge, none of the listed frameworks have any built-in support for automatic scaling; they're all communication frameworks, so you have to do the work to communicate between your scaling nodes. However, Twisted has an advantage in its built-in support for local multi-processing. In fairness, there is a third-party add-on for Tornado that allows you to do the same thing. In recent releases, Twisted has added features that increase the number of ways you can share work between cores, and work is ongoing in that area. Twisted also has a couple of well-integrated, "native" RPC protocols which offer a construction-kit for whatever scaling idiom you want to pursue.
"Most Useful"
Lots of people seem to find Twisted very useful. So much so that many of them have extended it and made their extensions available to you.
"Functionality"
Out of the box, Twisted includes:
good support for test-driven development of all the following
TCP servers, clients, transport layer security
SSH client and server
IMAP4, ESMTP, POP3 clients and servers
DNS client and server
HTTP client and server
IRC, XMPP, OSCAR, MSN clients and servers
In this last department, at least, Twisted seems a clear winner for built-in functionality. And all this, in a package just over 2 megabytes!
I like #Glyph response.
Twisted is very comprehensive, rich python framework.
Twisted and Tornado have a very similar design.
And I like this design very much:
it's fast
easy to understand
easy to extend
doesn't require c-extensions
works on PyPy.
But I want to highlight Tornado, which I prefer and recently gain popularity.
Tornado, like Twisted, uses callback style programming, but it can be inlined using tornado.gen.engine (twisted.internet.inlineCallbacks in Twisted).
Codebase
The best comment is from http://cyclone.io site. cyclone tries to mix Twisted and Tornado because:
Twisted is one of the most mature libraries for non-blocking I/O available to the
public. Tornado is the open source version of FriendFeed’s web server,
one of the most popular and fast web servers for Python, with a very
decent API for building web applications.
The idea is to bridge Tornado's elegant and straightforward API to
Twisted's Event-Loop, enabling a vast number of supported protocols.
But in 2011 tornado.platform.twisted was out which brings similar functionality.
Performance
Tornado has much better performance. It also works seamlessly with PyPy, and get huge gain.
Scalability
The same like Twisted. Tornado has tornado.process and a lot of rpc services implemented on top of it.
Functionality
There are 71 Tornado based package, compared to 148 Twisted's and 48 Gevent's. But if you look carefully and compute median of packages upload time, you will see that Twisted ones are the oldest, then Gevent and Tornado the freshest.
Furthermore there is tornado.platform.twisted module which allows you to run code written for Twisted on Tornado.
Summary
With Tornado you can use a code from Twisted. There is no need to use cyclone which only twists your code (your code becomes more messy).
As for 2014, Tornado is considered as widely accepted and default async framework which works both on python2 and python3. Also the latest version 4.x brings a lot of functionality from https://docs.python.org/dev/library/asyncio.html.
I wrote an article, explaining why I consider that Tornado - the best Python web framework where I wrote much more about Tornado functionality.
(UPDATE: I'm sadly surprised about how few answers here recommend or even mention Gevent—I don't think it's in proportion to the popularity, performance and ease of use of this excellent library!)
Gevent and Twisted are not mutually exclusive, even though the contrary might seem obvious at first. There is a project called geventreactor which allows one to relatively smoothly leverage the best of both worlds, namely:
The efficient and cheap (cooperative green) thread model of Gevent, which is much easier to program in when it comes to concurrency—frankly, Twisted's inlineCallbacks is simply not up to the job in terms of performance when it comes to many coroutines, and neither in terms of ease/transparency of use: yield and Deferreds everywhere; often hard to build some abstractions; horrifyingly useless stack traces with both bare Deferreds as well as, and even more so with #inlineCallbacks.
All the built-in functionality of Twisted you can ever dream of, including but not limited to IReactorProcess.spawnProcess.
I'm personally currently using Gevent 1.0rc2 with Twisted 12.3 bridged by geventreactor. I have implemented my own as-of-yet unpublished additions and enhancements to geventreactor which I will publish soon, hopefully as part of geventreactor's original GitHub repository: https://github.com/jyio/geventreactor.
My current layout allows me to program in the nice programming model of Gevent, and leverage things such as a non-blocking socket, urllib2 and other modules. I can use regular Python code for doing regular things, as opposed to the learning curve and inconvenience of doing even simple, basic things the Twisted way. I can also easily use most 3rd party libraries that are normally either out of question with Twisted, or require the use of threads.
I can also completely avoid the awkward and often overly complex callback based programming by using greenlets (instead of Deferreds and callbacks, and/or #inlineCallbacks).
(This answer was written based on my personal experiences having used both Twisted and Gevent in real life projects, with significantly more experience using Twisted (but I don't claim to be a Twisted expert). The software I've had to write hasn't had to use too many of Twisted's features, so depending on the set of features you require of Twisted, the (relatively painless) extra complexity of mixing Gevent and Twisted might not be worth the trouble.)
I am currently working on a complex web interface and backend, that will need to address several issues.
Scalablility
multiple deployments of varying load demands
Very structured authorization groups
Different views for different user groups
admin panel
user/content management
Large managed database
current
long term stored data (histories)
Data Updates
Polling
Ex. Search queries, static pages/files, report generation per request
Pushing (likely websockets)
Ex. Real-time notifications
Varying protocols
Ex. HTTP, SSL, Websockets
I would like to use Python, because I have grown to really enjoy the language, and I am considering some combo of Django and Twisted.
I have some experience with Django, which I love for its MVT style of application programming, its authorization models, its admin panel, and its database API. However, it is not so strong in some of the data requirements that I need, in particular, the real-time aspects.
Now, I have not really used Twisted before, but I have seen many interesting things to it. In particular the async aspects, and the ability to run many protocols.
The problems in getting the two to work together are obvious in that Django is a blocking server and Twisted is designed to be non-blocking. I have seen some topics stating using the two together is possible and have had success with it. It also seems possible to run both and proxy them to accept different urls, but getting the authentication over the two may become tricky?
Having said all of that, I would like to ask if I am on the right track for implementing this system, as well as suggestions on how to use the two together, alternatives, or if I should just kick one out (at this point, I guess it'd have to be Django, because the real time stuff is necessary). I should mention that I have written some of the preliminary data models and views in Django already.
I am quite experienced on the client side of things (JS,CSS,HTML), but I am not so savvy in the server side of things. Any input would be helpful, thanks.
You can definitely use Twisted with Django. Several projects have used the two together to good effect. twistd web --wsgi provides a basic way to get it set up, and there's a great example with more bells and whistles, like static content by Alex Clemesha on github.
I need to create a project that has a web frontend to manage synchronous task execution (ala fabric), async tasks (AMQP), and long-polling/ajax for tabular viewing of results and queues/large, frequently changing datasets (think tail -f syslog). I have an existing Python codebase for a lot of the implementation-specific stuff.
After looking at a bunch of existing frameworks, the obvious answer appears to be Django+Celery. However, I do not want to "learn Django", nor do I need 95% of it's functionality. I just need simple auth, maybe sqlalchemy, easy ajax, amqp, xmlrpc would be helpful.
I would consider using Mongrel2, but I have a strong preference for RabbitMQ over 0MQ (for a few implementation-specific reasons).
I originally spent a great deal of time learning Twisted, and ended up getting a few hundred useful LOC out of it, but I found that I was twisting (lol) too much of my platform code to fit it's callback model. It actually 'fit the bill' very well (except with it's own amqp implementation), but it was so frustrating, and I went through so many iterations of code (one for each 'twisted ahah moment'), that it's 100% out.
Can somebody please help me wade through the mire? Tornado? Pylons? repoze? Pyramid? Flask? Bottle? CherryPy? Web2py? Paster/Webob? Anything else# http://wiki.python.org/moin/WebFrameworks?
Edit:
To be clear, integration with RabbitMQ (or another amqp provider) is of the utmost importance, and is really the crux of problem.
I don't have a full vision of python web frameworks but just want to share my point of view on 2 of them :
Bottle is light and works fine. If you want something easy to learn and easy to use that may be the right choice. I used it for quite simple front-end apps running locally and i liked it very much.
Tornado seems to me as a very good non-blocking server for real-time web app. Combined with tornadio it makes ajax-long-polling quite easy. However, it may be a little harder to learn than Bottle. I would recommend to have a look to the chat app in the example folder of tornadio.
I hope it helps
If you are going to use AMQP long term then I would steer clear of Celery because they use AMQP in a wierd way that suggests the developers did not understand the AMQP model.
bottle is a nice framework for knocking together RESTful apps (I use it to create mock servers for testing) and if you already have the code that does the real work, you may be surprised at how short a bottle app can be.
I'm currently building Python apps using RabbitMQ and using amqplib by way of kombu. I originally chose kombu in case I wanted to swap libraries and use pika or something else, but now I wish that I had just gone with amqplib and built a proper Pythonic AMQP model on top of that.
Do spend some time on the RabbitMQ site reading some of the blogs and slide presentations on AMQP before you get too deep into coding or you won't really understand the AMQP model and will make things harder for yourself.
Please don't use xmlrpc unless you have to talk to other apps. Bottle makes simple RESTful apps so simple, that XMLRPC is just uneccessary complexity.
A couple of suggestions.
CherryPy is a great low level framework. It doesn't provide a lot of functionality, but it provide a very easy system for mapping http requests to function calls.
web.py is another extremely lightweight and easy to use framework. It is more comprehensive than CherryPy, including templates and other features.
Plain wsgi is not a bad choice if your needs are extremely simple. It is a little more complicated to do simple stuff than CherryPy or Web.py. WSGI is the lowest common denominator, these days most web frameworks are built on top of it.