This question is related to an older question: MySQL tracking system. In short: I have to implement a tracking system that will have high loads using Python. For the database part I've settled on mongoDB (which sounds like the right tool for this job). The development language will be Python.
I was thinking of using several instances of a CherryPy application behind nginx. The reasoning behind this is that I don't want to handle all the wsgi part myself, but on the other hand I don't need a full blown web framework since the app will be simple and there's no need for ORM.
My questions are:
Should I use the CherryPy builtin server or should I use Apache with modwsgi (or another server altogether)?
Does this sound like a reasonable approach (nginx, mongoDB)? If not what would you recommend?
Thank you in advance.
Have you checked out Graphite? It sounds like exactly the kind of thing that you need (looking at your other question) and was designed for application and server monitoring by the Orbitz team. It's extremely robust and easy to use for this sort of thing.
Here's the project site: http://graphite.wikidot.com/
With some screenshots: http://graphite.wikidot.com/screen-shots
Sounds like MongoDB will be a good fit for this - fast updates with advanced operators, and M/R for batch offline processing. I think CherryPy behind Nginx should work well too. If you go the mod_wsgi route just watch out for this issue.
Related
I got 6 real-time videos which are inference output from heavy deep learning calculation from python, and I've tried to display them to PyQt that lots of threading issues with ugly GUI!!. So, I want to use a framework only to display well and fancy GUI! There seem lots of frameworks based on my google search, and I have no idea
which one is the best for my current project among Node js, Django, and Flask!
I need to display 6 real-time videos with 10~15 FPS.
Communicating well with python.
Easy to build GUI, I have some features like logging, displaying real-time graph(optional)
For video streaming, you will require to setup a socket URL for listening to incoming packet. Definitely, you can dedicate a server just for this to handle high amount of traffic.
Now, as such it dependents on your application, if you are just going to use it for streaming then Flask will do, it is lightweight. On the other hand, Django and NodeJS provides ready made function for socket programming and they are very useful.
I would say it doesn't really matter much as Django would be good because of its pre built features... Plus data transfer is very reliable in django along with your expertise in python, django is preferred. NodeJs would be easier to implement tho. Really prefers upon you, but both django and nodeJs works pretty well.
Sorry, no idea on flask
My opinion - flask faster development, node - better efficiency, some thoughts why are explained here: https://hinty.io/ivictbor/flask-vs-node/
I have developed a python program that parses a webpage and creates a new text document with the parsed data. I want to deliver this new information to the web. I have no idea where to start with something like this. Are there any free options where I can have a site automatically call this python code upon request and update the new data to its page? Or is the only feasible solution here to have my own website/server that uses my code? I'm honestly pretty overwhelmed with many of the options when I try to begin doing a web-search for a solution like this. I have done a decent amount of application programming before so i'm confident in my ability to learn new things, but web protocols are all new to me so its hard to find a starting point.
Ultimately I want this python code to run automatically, or per request of a user, and deliver to the data to them. It could even be through an email, although that is probably less practical.
I personally have good experience using Google Appengine (and its free for a limited amount of requests). The downside is that it does not allow C-extensions or Python3.
If you want to host your own server, tornado is a good option I think. Tornado supports both Python2 and Python3.
There are a great deal of options available.. from 'traditional' virtual server or website hosts like a2hosting or godaddy to 'Cloud Application Hosts' such as Amazon EC2, Heroku or OpenShift.
For your case, and without knowing more, I would suggest that an application hosting is more appropriate, and that you should take a look at Heroku and Openshift in particular.
Define carefully what you want to achieve (how the users access your application, what they see, how they interact with it... etc..) and then evaluate these options based on those requirements.
Most offer a free trial, or even free services, depending on what you need! Good luck
If you've never worked with web technologies before this will be a overwhelming task, since there's a lot of different technologies involved, and many possible ways to combine them.
You'll probably want to start by familiarizing yourself with the very basics of the HTTP protocol.
Then you should read a bit on CGI server-side programming (the article also has a quick overview on HTTP).
Python can run both on CGI and WSGI (if the server provider allows such access), so you may also want to read about WSGI.
Once you grasp all these concepts, you should check this question for actual python techniques.
Also, since you seem to be under the impression you must pay to have a website/app deployed, you should know there are companies that host python apps for free
I'm in the process of setting up a new web app and deciding whether to just do it with WSGI or go the full framework route with Django.
The app's foremost requirements:
1) The app has no UI what so ever and all of the data is exposed to clients via a REST api with JSON.
2) It will have data to persist so MongoDB & probably Amazon's SimpleDB will be used for the database side.
Is there a reason to use Django or can I get marginal speed improvement with WSGI only?
Previous server-side apps I've built were either with Java/Struts and Groovy/Grails on the JVM. My understanding is that Django is an MVC framework similar to Rails and Grails.
I've also played around with Google App Engine which uses WSGI as thin layer above your code for managing and routing requests.
I suggest you consider something between those two extremes. Flask is lightweight, very easy to use, and connects to your web server via wsgi. You can use regular python database connectors with it, and a few databases even have Flask-specific extension modules.
I have worked with Django for a couple of projects and I like it a lot, but since you are going to use mongoDB and a lot of JSON I suggest you use NodeJS as server side, with Express as framework, you can see a brief tutorial here:
http://howtonode.org/express-mongodb
One of the advantages of this is that you will use only javascript all along your project, I began working with this technology the last month in a Hackathon, and I can tell you that I'm very impressed of how fast and simple it is.
I've worked a bit with some django "apps" ,its really easy, but setting up the "apps" can be a bit of a long process. Django has a lot of nice features that you won't be using and I agree that you might be on one "extreme" here.
I need to create a project that has a web frontend to manage synchronous task execution (ala fabric), async tasks (AMQP), and long-polling/ajax for tabular viewing of results and queues/large, frequently changing datasets (think tail -f syslog). I have an existing Python codebase for a lot of the implementation-specific stuff.
After looking at a bunch of existing frameworks, the obvious answer appears to be Django+Celery. However, I do not want to "learn Django", nor do I need 95% of it's functionality. I just need simple auth, maybe sqlalchemy, easy ajax, amqp, xmlrpc would be helpful.
I would consider using Mongrel2, but I have a strong preference for RabbitMQ over 0MQ (for a few implementation-specific reasons).
I originally spent a great deal of time learning Twisted, and ended up getting a few hundred useful LOC out of it, but I found that I was twisting (lol) too much of my platform code to fit it's callback model. It actually 'fit the bill' very well (except with it's own amqp implementation), but it was so frustrating, and I went through so many iterations of code (one for each 'twisted ahah moment'), that it's 100% out.
Can somebody please help me wade through the mire? Tornado? Pylons? repoze? Pyramid? Flask? Bottle? CherryPy? Web2py? Paster/Webob? Anything else# http://wiki.python.org/moin/WebFrameworks?
Edit:
To be clear, integration with RabbitMQ (or another amqp provider) is of the utmost importance, and is really the crux of problem.
I don't have a full vision of python web frameworks but just want to share my point of view on 2 of them :
Bottle is light and works fine. If you want something easy to learn and easy to use that may be the right choice. I used it for quite simple front-end apps running locally and i liked it very much.
Tornado seems to me as a very good non-blocking server for real-time web app. Combined with tornadio it makes ajax-long-polling quite easy. However, it may be a little harder to learn than Bottle. I would recommend to have a look to the chat app in the example folder of tornadio.
I hope it helps
If you are going to use AMQP long term then I would steer clear of Celery because they use AMQP in a wierd way that suggests the developers did not understand the AMQP model.
bottle is a nice framework for knocking together RESTful apps (I use it to create mock servers for testing) and if you already have the code that does the real work, you may be surprised at how short a bottle app can be.
I'm currently building Python apps using RabbitMQ and using amqplib by way of kombu. I originally chose kombu in case I wanted to swap libraries and use pika or something else, but now I wish that I had just gone with amqplib and built a proper Pythonic AMQP model on top of that.
Do spend some time on the RabbitMQ site reading some of the blogs and slide presentations on AMQP before you get too deep into coding or you won't really understand the AMQP model and will make things harder for yourself.
Please don't use xmlrpc unless you have to talk to other apps. Bottle makes simple RESTful apps so simple, that XMLRPC is just uneccessary complexity.
A couple of suggestions.
CherryPy is a great low level framework. It doesn't provide a lot of functionality, but it provide a very easy system for mapping http requests to function calls.
web.py is another extremely lightweight and easy to use framework. It is more comprehensive than CherryPy, including templates and other features.
Plain wsgi is not a bad choice if your needs are extremely simple. It is a little more complicated to do simple stuff than CherryPy or Web.py. WSGI is the lowest common denominator, these days most web frameworks are built on top of it.
is there a difference between using FAPWS3 and MOD_WSGI when dealing with Django?
FAPWS3 seems alot faster when serving requests toward Python scripts. I would like to know if I'm missing out anything. :)
Any ideas?
The underlying web server is not the bottleneck, it is your application and database access. The differences between any underlying web server are going to very minimal or non existent in the context of an actual full application stack. You cannot base decisions on hello world type tests as they are pretty meaningless. Decisions should therefore be based on the quality and stability of the hosting solutions under load, as well as ease of configuration and support, including your own competence to manage a particular setup. If you have no idea how to configure and support a particular web server properly, eg., Apache, then why would you use it.
here is the best explanation what i ever seen in the web at the moment.
http://nichol.as/benchmark-of-python-web-servers
Quote from nichol.as
When you are just interested in quickly hosting your threaded
application you really can’t go wrong with Apache ModWSGI. Even though
Apache ModWSGI might put a little more strain on your memory
requirements there is a lot to go for in terms of functionality. For
example, protecting part of your website by using a LDAP server is as
easy as enabling a module. Standalone CherryPy also shows great
performance and functionality and is really a viable (fully Python)
alternative which can lower memory requirements.
When you are a little more adventurous you can look at uWSGI and
FAPWS3, they are relatively new compared to CherryPy and ModWSGI but
they show a significant performance increase and do have lower memory
requirements.