is there a difference between using FAPWS3 and MOD_WSGI when dealing with Django?
FAPWS3 seems alot faster when serving requests toward Python scripts. I would like to know if I'm missing out anything. :)
Any ideas?
The underlying web server is not the bottleneck, it is your application and database access. The differences between any underlying web server are going to very minimal or non existent in the context of an actual full application stack. You cannot base decisions on hello world type tests as they are pretty meaningless. Decisions should therefore be based on the quality and stability of the hosting solutions under load, as well as ease of configuration and support, including your own competence to manage a particular setup. If you have no idea how to configure and support a particular web server properly, eg., Apache, then why would you use it.
here is the best explanation what i ever seen in the web at the moment.
http://nichol.as/benchmark-of-python-web-servers
Quote from nichol.as
When you are just interested in quickly hosting your threaded
application you really can’t go wrong with Apache ModWSGI. Even though
Apache ModWSGI might put a little more strain on your memory
requirements there is a lot to go for in terms of functionality. For
example, protecting part of your website by using a LDAP server is as
easy as enabling a module. Standalone CherryPy also shows great
performance and functionality and is really a viable (fully Python)
alternative which can lower memory requirements.
When you are a little more adventurous you can look at uWSGI and
FAPWS3, they are relatively new compared to CherryPy and ModWSGI but
they show a significant performance increase and do have lower memory
requirements.
Related
is it possible to make the python as the web server and the front end is codeigniter?
For some reasons:
database security - like when you are saving data. the codeigniter will pass the data to python basehttpserver / or maybe flask (but i have not yet done the flask before)
SQL Injection.
it's like for example. front end codeigniter
form - send data.
back end python web service
receives data - will serve as an API. and the python will be the one in charge of saving data in MySQLdb.
In theory I don't see why this would be impossible.
You could easily write a web application using codeigniter and have the controllers just pass data along to a python based web service. If you're interested in a fully decoupled front-end/back-end, you could also use a queuing layer (such as RabbitMQ) in between the data entry facilities in your CI program, and the persistence web services in Python.
That said, I'm not clear why you would want to. CodeIgniter is PHP, and includes some very excellent data modelling components that integrate fully into the overall framework. Long story short, if you're using CodeIgniter, just have it connect to MySQL and do the data persistence for you.
Likewise, if you'd prefer to code your persistence in Python, why not just use Django? It's a fully realized Python web framework, and also features an excellent ORM and support for MySQL.
I don't really see how either technology gives you clear benefits, provided they are both used properly, for database security. Both have built-in methodologies for "cleansing" user provided data to prevent SQL injection (notes for Django and notes for CodeIgniter)
There are a great many other posts on StackOverflow dealing with preventing SQL injection with CodeIgniter and other frameworks. Just using python, or decoupling your front-end and back-end, will not provide you any additional security or protection guarantees. The only way to do that is to carefully architect your interactions with databases, using all the tools provided you by whatever framework you are using, or creating their equivalents if none are available (or switch to a better toolset).
Edit - expansion
Based on the comments above I figured it was actually worth writing a little more about the potential advantages and real challenges of a decoupled infrastructure.
In principle, it's easy to decouple a front-end from a more isolated backend. You could leverage in either Django or likely CodeIgniter (although I haven't personally seen it done in CI, just in Django) the existing model infrastructure, but deal with model objects in memory only on the frontend, and expand on the existing ORM functionality to use your backend services to actually store and retrieve data from a persistence layer (your database).
Practically, this can become quite a bit of work to do right. To gain the security advantages you desire, your decoupled backend needs to deal with the frontend as if in principle it is "hostile", or at the least, untrustworthy. So, be sure that you implement a method for the frontend to reliably authenticate itself to the backend. Ensure that all traffic is minimally using SSL between the frontend and the backend. Consider carefully your services architecture (the SOA layer in front of your backend logic) and make sure your APIs, where possible, are MECE (Mutually exclusive, collectively exhaustive).
I'm sure I'm missing some basic principles, but having recently participated in the design and build of a system along these lines I can assure you that the complexity can very quickly explode, so careful architectural discipline and adherence to both MECE and MVP (minimum viable product) is critical. A decoupled infrastructure can be an amazing end-product if it fits the need, and in use cases I've worked with it has been extremely effective. It isn't a one-size-fits-all, though, and hopefully some of the extra description here can help you make a more informed choice.
Hopefully this helps round out the topic answer. Basic principles: Design for what you need. Don't conflate complicated with secure. Simple can be secure, as can complex, but complexity breeds room for hard-to-plug security vulnerabilities, and simple gives the illusion of security by seeming easy. No approach guarantees a positive outcome, so don't try to cut corners; spend as much time as you can in research and design to minimize your time building, refactoring, and fixing.
I have developed a python program that parses a webpage and creates a new text document with the parsed data. I want to deliver this new information to the web. I have no idea where to start with something like this. Are there any free options where I can have a site automatically call this python code upon request and update the new data to its page? Or is the only feasible solution here to have my own website/server that uses my code? I'm honestly pretty overwhelmed with many of the options when I try to begin doing a web-search for a solution like this. I have done a decent amount of application programming before so i'm confident in my ability to learn new things, but web protocols are all new to me so its hard to find a starting point.
Ultimately I want this python code to run automatically, or per request of a user, and deliver to the data to them. It could even be through an email, although that is probably less practical.
I personally have good experience using Google Appengine (and its free for a limited amount of requests). The downside is that it does not allow C-extensions or Python3.
If you want to host your own server, tornado is a good option I think. Tornado supports both Python2 and Python3.
There are a great deal of options available.. from 'traditional' virtual server or website hosts like a2hosting or godaddy to 'Cloud Application Hosts' such as Amazon EC2, Heroku or OpenShift.
For your case, and without knowing more, I would suggest that an application hosting is more appropriate, and that you should take a look at Heroku and Openshift in particular.
Define carefully what you want to achieve (how the users access your application, what they see, how they interact with it... etc..) and then evaluate these options based on those requirements.
Most offer a free trial, or even free services, depending on what you need! Good luck
If you've never worked with web technologies before this will be a overwhelming task, since there's a lot of different technologies involved, and many possible ways to combine them.
You'll probably want to start by familiarizing yourself with the very basics of the HTTP protocol.
Then you should read a bit on CGI server-side programming (the article also has a quick overview on HTTP).
Python can run both on CGI and WSGI (if the server provider allows such access), so you may also want to read about WSGI.
Once you grasp all these concepts, you should check this question for actual python techniques.
Also, since you seem to be under the impression you must pay to have a website/app deployed, you should know there are companies that host python apps for free
I need to create a project that has a web frontend to manage synchronous task execution (ala fabric), async tasks (AMQP), and long-polling/ajax for tabular viewing of results and queues/large, frequently changing datasets (think tail -f syslog). I have an existing Python codebase for a lot of the implementation-specific stuff.
After looking at a bunch of existing frameworks, the obvious answer appears to be Django+Celery. However, I do not want to "learn Django", nor do I need 95% of it's functionality. I just need simple auth, maybe sqlalchemy, easy ajax, amqp, xmlrpc would be helpful.
I would consider using Mongrel2, but I have a strong preference for RabbitMQ over 0MQ (for a few implementation-specific reasons).
I originally spent a great deal of time learning Twisted, and ended up getting a few hundred useful LOC out of it, but I found that I was twisting (lol) too much of my platform code to fit it's callback model. It actually 'fit the bill' very well (except with it's own amqp implementation), but it was so frustrating, and I went through so many iterations of code (one for each 'twisted ahah moment'), that it's 100% out.
Can somebody please help me wade through the mire? Tornado? Pylons? repoze? Pyramid? Flask? Bottle? CherryPy? Web2py? Paster/Webob? Anything else# http://wiki.python.org/moin/WebFrameworks?
Edit:
To be clear, integration with RabbitMQ (or another amqp provider) is of the utmost importance, and is really the crux of problem.
I don't have a full vision of python web frameworks but just want to share my point of view on 2 of them :
Bottle is light and works fine. If you want something easy to learn and easy to use that may be the right choice. I used it for quite simple front-end apps running locally and i liked it very much.
Tornado seems to me as a very good non-blocking server for real-time web app. Combined with tornadio it makes ajax-long-polling quite easy. However, it may be a little harder to learn than Bottle. I would recommend to have a look to the chat app in the example folder of tornadio.
I hope it helps
If you are going to use AMQP long term then I would steer clear of Celery because they use AMQP in a wierd way that suggests the developers did not understand the AMQP model.
bottle is a nice framework for knocking together RESTful apps (I use it to create mock servers for testing) and if you already have the code that does the real work, you may be surprised at how short a bottle app can be.
I'm currently building Python apps using RabbitMQ and using amqplib by way of kombu. I originally chose kombu in case I wanted to swap libraries and use pika or something else, but now I wish that I had just gone with amqplib and built a proper Pythonic AMQP model on top of that.
Do spend some time on the RabbitMQ site reading some of the blogs and slide presentations on AMQP before you get too deep into coding or you won't really understand the AMQP model and will make things harder for yourself.
Please don't use xmlrpc unless you have to talk to other apps. Bottle makes simple RESTful apps so simple, that XMLRPC is just uneccessary complexity.
A couple of suggestions.
CherryPy is a great low level framework. It doesn't provide a lot of functionality, but it provide a very easy system for mapping http requests to function calls.
web.py is another extremely lightweight and easy to use framework. It is more comprehensive than CherryPy, including templates and other features.
Plain wsgi is not a bad choice if your needs are extremely simple. It is a little more complicated to do simple stuff than CherryPy or Web.py. WSGI is the lowest common denominator, these days most web frameworks are built on top of it.
I need to build a webservice that is very computationally intensive, and I'm trying to get my bearings on how best to proceed.
I expect users to connect to my service, at which point some computation is done for some amount of time, typically less than 60s. The user knows that they need to wait, so this is not really a problem. My question is, what's the best way to structure a service like this and leave me with the least amount of headache? Can I use Node.js, web.py, CherryPy, etc.? Do I need a load balancer sitting in front of these pieces if used? I don't expect huge numbers of users, perhaps hundreds or into the thousands. I'll need a number of machines to host this number of users, of course, but this is uncharted territory for me, and if someone can give me a few pointers or things to read, that would be great.
Thanks.
Can I use Node.js, web.py, CherryPy, etc.?
Yes. Pick one. Django is nice, also.
Do I need a load balancer sitting in front of these pieces if used?
Almost never.
I'll need a number of machines to host this number of users,
Doubtful.
Remember that each web transaction has several distinct (and almost unrelated) parts.
A front-end (Apache HTTPD or NGINX or similar) accepts the initial web request. It can handle serving static files (.CSS, .JS, Images, etc.) so your main web application is uncluttered by this.
A reasonably efficient middleware like mod_wsgi can manage dozens (or hundreds) of backend processes.
If you choose a clever backend processing component like celery, you should be able to distribute the "real work" to the minimal number of processors to get the job done.
The results are fed back into Apache HTTPD (or NGINX) via mod_wsgi to the user's browser.
Now the backend processes (managed by celery) are divorced from the essential web server. You achieve a great deal of parallelism with Apache HTTPD and mod_wsgi and celery allowing you to use every scrap of processor resource.
Further, you may be able to decompose your "computationally intensive" process into parallel processes -- a Unix Pipeline is remarkably efficient and makes use of all available resources. You have to decompose your problem into step1 | step2 | step3 and make celery manage those pipelines.
You may find that this kind of decomposition leads to serving a far larger workload than you might have originally imagined.
Many Python web frameworks will keep the user's session information in a single common database. This means that all of your backends can -- without any real work -- move the user's session from web server to web server, making "load balancing" seamless and automatic. Just have lots of HTTPD/NGINX front-ends that spawn Django (or web.py or whatever) which all share a common database. It works remarkably well.
I think you can build it however you like, as long as you can make it an asynchronous service so that the users don't have to wait.
Unless, of course, the users don't mind waiting in this context.
I'd recommend using nginx as it can handle rewrite/balancing/ssl etc with a minimum of fuss
If you want to make your web sevices asynchronous you can try Twisted. It is a framework oriented to asynchronous tasks and implements so many network protocols. It is so easy to offer this services via xml-rpc (just put xmlrpc_ as the prefix of your method). On the other hand it scales very well with hundreds and thousands of users.
Celery is also a good option to make the most computionally intensive tasks asynchronous. It integrates very well with Django.
This question is related to an older question: MySQL tracking system. In short: I have to implement a tracking system that will have high loads using Python. For the database part I've settled on mongoDB (which sounds like the right tool for this job). The development language will be Python.
I was thinking of using several instances of a CherryPy application behind nginx. The reasoning behind this is that I don't want to handle all the wsgi part myself, but on the other hand I don't need a full blown web framework since the app will be simple and there's no need for ORM.
My questions are:
Should I use the CherryPy builtin server or should I use Apache with modwsgi (or another server altogether)?
Does this sound like a reasonable approach (nginx, mongoDB)? If not what would you recommend?
Thank you in advance.
Have you checked out Graphite? It sounds like exactly the kind of thing that you need (looking at your other question) and was designed for application and server monitoring by the Orbitz team. It's extremely robust and easy to use for this sort of thing.
Here's the project site: http://graphite.wikidot.com/
With some screenshots: http://graphite.wikidot.com/screen-shots
Sounds like MongoDB will be a good fit for this - fast updates with advanced operators, and M/R for batch offline processing. I think CherryPy behind Nginx should work well too. If you go the mod_wsgi route just watch out for this issue.