We Keep Coding

AWS ECS Fargate + Python + Nginx

Hey,
I would like to start a small website that will be entirely handled in Python. I will be using the Flask framework for this. So far I had a lot of contact with AWS ECS and ELB service, but I admit, Python itself is still unknown to me. That's why I have a few questions:
1. I understand that from the point of view of a software engineer it is better to separate the backend and frontend - so it is best to create two separate Python projects based on Flask - one will be the API, the other the frontend, right? Generally, both should be separate services in the ECS service I guess.
2. In such configuration do they both have to use some kind of WSGI server, like gunicorn? Is this a good solution to run inside Fargate with multiple vCPU?
3. There are quite a few questions and myths around Nginx for this solution. Until now I assumed that if I use Application Load Balancer it should be enough (after all it also acts as reverse proxy). Is it necessary to use Nginx as a sidecar in ECS, are there any benefits of this? Assuming that using Nginx would be advisable, should it be only for the frontend or also for API?
Thank you really in advance for any supportive advice here - I know that I have asked for a lot of things.

In my consideration
1- if you want to have a microservice concept you can separate you application with front-end and back-end each of them has their freameworks. for front-end you can use Angular, React , Vuejs and so on. Python is backend technology and you can write strong restfull api to communicate with you front-end application
2- if you containerize your application with for example Docker and write Dockerfile for each service witch it is most common in microservice it is okay to run your container with any servers like nginx,apache or WSGI server(i did not work with this) then expose port (if it is needed) to be accessible
3- when you run your service in AWS Fargate it is possible to connect loadbalancer to your service and a service itself run tasks each task actually is one or more container with may be nginx server or something else , if you mean that it is normal to have nginx in your container.

Set keep-alive timeout for Flask server with parameter [duplicate]

Setting up Flask with uWSGI and Nginx can be difficult. I tried following this DigitalOcean tutorial and still had trouble. Even with buildout scripts it takes time, and I need to write instructions to follow next time.
If I don't expect a lot of traffic, or the app is private, does it make sense to run it without uWSGI? Flask can listen to a port. Can Nginx just forward requests?
Does it make sense to not use Nginx either, just running bare Flask app on a port?

When you "run Flask" you are actually running Werkzeug's development WSGI server, and passing your Flask app as the WSGI callable.
The development server is not intended for use in production. It is not designed to be particularly efficient, stable, or secure. It does not support all the possible features of a HTTP server.
Replace the Werkzeug dev server with a production-ready WSGI server such as Gunicorn or uWSGI when moving to production, no matter where the app will be available.
The answer is similar for "should I use a web server". WSGI servers happen to have HTTP servers but they will not be as good as a dedicated production HTTP server (Nginx, Apache, etc.).
Flask documents how to deploy in various ways. Many hosting providers also have documentation about deploying Python or Flask.

First create the app:
import flask
app = flask.Flask(__name__)
Then set up the routes, and then when you want to start the app:
import gevent.pywsgi
app_server = gevent.pywsgi.WSGIServer((host, port), app)
app_server.serve_forever()
Call this script to run the application rather than having to tell gunicorn or uWSGI to run it.
I wanted the utility of Flask to build a web application, but had trouble composing it with other elements. I eventually found that gevent.pywsgi.WSGIServer was what I needed. After the call to app_server.serve_forever(), call app_server.stop() when to exit the application.
In my deployment, my application is listening on localhost:port using Flask and gevent, and then I have Nginx reverse-proxying HTTPS requests to it.

You definitely need something like a production WSGI server such as Gunicorn, because the development server of Flask is meant for ease of development without much configuration for fine-tuning and optimization.
Eg. Gunicorn has a variety of configurations depending on the use case you are trying to solve. But the development flask server does not have these capabilities. In addition, these development servers show their limitations as soon as you try to scale and handle more requests.
With respect to needing a reverse proxy server such as Nginx is concerned it depends on your use case.
If you are deploying your application behind the latest load balancer in AWS such as an application load balancer(NOT classic load balancer), that itself will suffice for most use cases. No need to take effort into setting up NGINX if you have that option.
The purpose of a reverse proxy is to handle slow clients, meaning clients which take time to send the request. These reverse load balancers buffer the requests till the entire request is got from the clients and send them async to Gunicorn. This improves the performance of your application considerably.

Message communication between Django and Twisted

I've created remote objects monitoring application.
Application description: Twisted communicates with remote controllers and then saves measured data to database via "adbapi". Django is used as web interface.
Problem: Django uses models for database access. Twisted uses raw SQL queries, generates alerts and other before save operations. I want to move all logic from Twisted to Django. So that Twisted will be layer between remote controllers and Django. Django will use models and save measured data, perform various operations and so on.
Question: Best way to organize communication between Twisted and Django? Two-way communication bus between them. At the moment I've 3 options:
Perform http POST request via getPage from twisted.web.client and on other side handle it by Django.
Use RabbitMQ and pika module as transport on Twisted side. Use Django-standalone app as daemon on other side.
I believe that there should be a better solution than my offer.
P.S.:

Answer to my own question: for now I've used RabbitMQ as a transport. On twisted side used pika module via this adapter. On django side I created a django standalone script as a daemon. Daemon endlessly waiting for new messages from RabbitMQ via pika module.
Please, let me know if there is a better solution.

Working with Django Project on gunicorn and Tornado

I've a django project running on gunicorn server behind an nginx proxy.
I would like to add real time chat functionality to this project and for this purpose I'm considering to use Tornado websocket API.
But the problem is, Chat messages needs to be authenticated by Django User as request.is_authenticated(). Similarily I need to use two Django models for handling Chat in tornado. I'm uanble to figure out how to handle this situation keeping both gunicorn and tornado server different.
I've considered few options:
1. Creating a dedicated API on gunicorn server that will listen to special messages from tornado server. This can be done but extra overhead will be on gunicorn server
2. Session authentication can be done via gunicorn server as API and rest of database handling can be done on tornado itself.
Are there better ways to handle this? Without creating API and integrating them seamlessly?

It's possible to run your Django app on top of tornado instead of gunicorn (see https://github.com/bdarnell/django-tornado-demo). This isn't the best option for performance, but will let the django and tornado apps interact without building out a separate API.
Personally I recommend running django and tornado in the same process only as a stepping stone to transition to 100% tornado. If you intend to keep the django server then it is probably best to just build an authentication API that you can call from tornado.

Should I use orbited or gevent for integrating comet functionality into a django app

I have been working with Django for some time now and have written several apps on a setup that uses Apache 2 mod_wsgi and a PostgreSQL database on ubuntu.
I have aa app that uses xsendfile to serve files from Apache via a Django view, and also allow users to upload files via a form as well. All this working great, but I now want to ramp up the features (and the complexity I am sure) by allowing users to chat and to see when new files have been uploaded without refreshing their browser.
As I want this to be scale-able, I don't want to poll continually with AJAX as this is going to get very heavy with large numbers of users.
I have read more posts, sites and blogs then I can count on integrating comet functionality into a Django app but there are so many different opinions out there on how to do this that I am now completely confused.
Should I be using orbited, gevent, iosocket?
Where does Tornado fit into this debate?
I want the messages also be stored on the database, so do I need any special configuration
to prevent my application blocking when writing to the database?
Will running a chat server with Django have any impact on my ability to serve files from Apache?

I'd recommend using WebSockets for bidirectional realtime communication. Keep running Django as is and run a WebSocket server on another port. As far as your database blocking, yes, you'll need to keep that in mind as you write your WebSocket server and either use a non-blocking database driver, or address that in some way.
Client-side you'll want to use Socket.IO or web-socket-js to support flash fallback for older browsers which don't support flash.
For the server, I would lean towards gevent or tornado, personally. For gevent there is gevent-websocket and gevent-socketio, for tornado you get built-in WebSocket support and can use tornadio if you want to use Socket.IO. Eventlet and twisted both support WebSockets as well. There is also a pretty cool new project called autobahn which is built on twisted, and meinheld has WebSocket middleware you can use.
WebSockets are pretty exciting, and as such there are tons of great posts out there on the subject. I found these posts useful:
http://gehrcke.de/2011/06/the-best-and-simplest-tools-to-create-a-basic-websocket-application-with-flash-fallback-and-python-on-the-server-side/
http://codysoyland.com/2011/feb/6/evented-django-part-one-socketio-and-gevent/
http://toastdriven.com/blog/2011/jul/31/gevent-long-polling-you/
http://blog.jupo.org/post/8858247674/real-time-web-apps-with-django-and-websockets/

Instead of Apache + X-Sendfile you could use Nginx + X-Accel-Redirect. That way you can run a gevent/wsgi/django server behind Nginx with views that provide long-polling. No need for a separate websockets server.
I've used both Apache + X-Sendfile and Nginx + X-Accel-Redirect to serve (access-protected) content on Webfaction without any problems.