Is there anyway to make sure that an HttpResponse in django has successfully reached the end-user?
A normal TCP connection will end with FIN request! Could this be detected?
Currently, I save data in a temporary table and once the device receives the data, it sends a "confirmation message" to the server which in turn pulls data from temporary table and commits it.
Short answer is no. In production, there will be several layers of abstraction on top of django that prevent django from getting the level of detail that you want. In a small production set up, you will have
user --> nginx or apache --> gunicorn or wsgi running django
So at best, django will only be able to tell if nginx or apache received the response.
In larger scale production systems you will have
user --> load balancer --> nginx or apache --> gunicorn or wsgi running django
And in some even larger production scale systems you might see:
user --> cloudflare or incapsula (caching and firewall service) --> load balancer --> nginx or apache --> gunicorn or wsgi running django
So the short answer is, absent a special ACK or confirmation message, like the one that you already have in place, this can’t be done.
Related
I'm serving Wagtail pages in a very well tested web app. No errors show up in development.
When in Production (using Heroku and a hobby-tier Postgres db), those pages occasionally return an HTTP 404. If I refresh the browser a couple times, it goes back to serving the Page perfectly again.
Some details:
This happens with multiple pages;
I get emails with the 404 error when this happens, so Django's BrokenLinkEmailsMiddleware is kicking in;
Heroku web Dyno looks healthy based on its metrics;
Heroku Postgres has no logs for the Hobby tier, so I don't know what's going on on the db side;
Server is running behind a Cloudflare proxy server, but I can see the request hitting the origin and returning a 404.
I'm obviously not expecting a solution here since the above info is insufficent for that. But I'm looking for troubleshooting pointers, for example:
If the db has become under-dimensioned (too little RAM, too many simultaneous connections, etc.) - could that result in a Wagtail db query wrongly returning PageNotFound? Or would that return a 500 Server Error?
How can I test DB error-handling locally with Django?
How can I add a full Traceback for the Django 404 error, so I'm able to know exactly what line of code raised it?
What are other potential causes of intermittent HTTP 404 errors?
Any help here is welcome!
If I develop a REST service hosted in Apache and a Python plugin which services GET, PUT, DELETE, PATCH; and this service is consumed by an Angular client (or other REST interacting browser technology). Then how do I make it scale-able with RabbitMQ (AMQP)?
Potential Solution #1
Multiple Apache's still faces off against the browser's HTTP calls.
Each Apache instance uses an AMQP plugin and then posts message to a queue
Python microservices monitor a queue and pull a message, service it and return response
Response passed back to Apache plugin, in turn Apache generates the HTTP response
Does this mean the Python microservice no longer has any HTTP server code at all. This will change that component a lot. Perhaps best to decide upfront if you want to use this pattern as it seems it would be a task to rip out any HTTP server code.
Other potential solutions? I am genuinely puzzled as to how we're supposed to take a classic REST server component and upgrade it to be scale-able with RabbitMQ/AMQP with minimal disruption.
I would recommend switching wsgi to asgi(nginx can help here), Im not sure why you think rabbitmq is the solution to your problem, as nothing you described seems like that would be solved by using this method.
asgi is not supported by apache as far as I know, but it allows the server to go do work, and while its working it can continue to service new requests that come in. (gross over simplification)
If for whatever reason you really want to use job workers (rabbitmq, etc) then I would suggest returning to the user a "token" (really just the job_id) and then they can call with that token, and it will report back either the current job status or the result
I'm using the awesome Flask framework to create a website in which I use Flask-login for my user logins. This normally works fine, but sometimes I see strange issues with logins being mixed. We've got 3 flask dev-servers running on one machine (on different ports) and we're working in an office with about 10 people (with one shared ip). The problem is that sometimes one user is suddenly logged in as another user.
I can't really pinpoint when or under which circumstances this happens. But I also don't really know how I can debug it. Could the source of the evil be that we share an internet connection or is the problem that we run several flask dev-servers on one machine?
I don't know whether this also happens with people outside of our office (we're still in testing phase).
Can anybody give me some tips on how I can debug this?
Most likely you are using a web server which is caching (some of ) HTTP replies from Flask. These could include static media, generated media, PDFs, Office files.
A misconfigured front end web server may cache such a HTTP response containing media and the session cookie (Cookies header). After happily caching this response then the front end web server serves it to another user. The existing session cookie of this user gets overwritten with the session cookie from the cached HTTP response. Then, due to session switch, the user becomes the user whose HTTP response was cached.
Solutions
Fix your session middleware
Explicitly set no caching headers on the server side
Configure your front end web server not to cache responses with cookies
Further information in operationssecurity.org.
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
I am really new to sys admin stuff, and have only provisioned a VPS with nginx(serving the static files) and gunicorn as the web server.
I have lately been reading about different other stuff. I came to know about other tools:
nginx : high-performance HTTP server and reverse proxy, as well as an IMAP/POP3 proxy server
haproxy : high performance load balancer
varnish : caching HTTP reverse proxy
gunicorn : python WSGI http server
uwsgi : another python WSGI server
I have been reading about all the above 5 tools and I have confused myself as to which one is used for what purpose? Could someone please explain me in lay man terms what use is each of the tool put in, when used together and which specific concern do they address?
Let's say you plan to host a few websites on your new VPS. Let's look at the tools you might need for each site.
HTTP Servers
Website 'Alpha' just consists of a some pure HTML, CSS and Javascript. The content is static.
When someone visits website Alpha, their browser will issue an HTTP request. You have configured (via DNS and name server configuration) that request to be directed to the IP address of your VPS. Now you need your VPS to be able to accept that HTTP request, decide what to do with it, and issue a response that the visitor's browser can understand. You need an HTTP server, such as Apache httpd or NGINX, and let's say you do some research and eventually decide on NGINX.
Application Servers
Website 'Beta' is dynamic, written using the Django Web Framework.
WSGI is an protocol that describes the interface between a Python application (the django app) and an application server. So what you need now is an WSGI app server, which will be able to understand web requests, make appropriate 'calls' to the application's various objects, and return the results. You have many options here, including gunicorn and uWSGI. Let's say you do some research and eventually decide on uWSGI.
uWSGI can accept and handle HTTPS requests for static content as well, so if you wanted to you could have website Alpha served entirely by NGINX and website Beta served entirely by uWSGI. And that would be that.
Reverse Proxy Servers
But uWSGI has poor performance in dealing with static content, so you would rather use NGINX for static content like images, even on website Beta. But then something would have to distinguish between requests and send them to the right place. Is that possible?
It turns out NGINX is not just an HTTP server but also a reverse proxy server: it is capable of redirecting incoming requests to another place, like your uWSGI application server, or many other places, collecting the response(s) and sending them back to the original requester. Awesome! So you configure all incoming requests to go to NGINX, which will serve up static content or, when required, redirect it to the app server.
Load Balancing with multiple web servers
You are also hosting Website Gamma, which is a blog that is popular internationally and receives a ton of traffic.
For Gamma you decide to set up multiple web servers. All incoming requests are going to your original VPS with NGINX, and you configure NGINX to redirect the request to one of several other web servers based in round-robin fashion, and return the response to the original requester.
HAProxy is web server that specializes in balancing loads for high traffic sites. In this case, you were able to use NGINX to handle traffic for site Gamma. In other scenarios, one may choose to set up a high-availability cluster: e.g., send all requests to a server like HAProxy, which intelligently redirects traffic to a cluster of nginx servers similar to your original VPS.
Cache Server
Website Gamma exceeded the capacity of your VPS due to the sheer volume of traffic. Let's say you instead hosted website Delta, and the reason your web server is unable to handle Delta is due to a popular feature that is very content-heavy.
A cache server is able to understand what media content is being frequently requested and store this content differently, such that it can be more quickly served. This is achieved by reducing disk IO operations; the popular content can be stored in memory or virtual memory instead. You might decide to combine your existing NGINX stack with a technology like Varnish or Memchached to achieve this type of optimization and server website Gamma more effectively.
I will put a very concise (very informal) description for each one, in the order they would be hit when you make a request from your web browser:
HAProxy balances your traffic load, so if your webpage is receiving 5000 hits per second, you can't handle that with only one
webserver, so HAProxy will balance the hits among the webservers you
had behind.
Varnish is a cache server, it sits upfront your webservers and behind HAProxy, so if a resource is already cached by Varnish he will serve the request itself, instead
of passing the request to the webservers behind.
ngingx, gunicorn, uwsgi are web servers, that would be behind varnish and will get the requests that varnish will let pass
through. These web servers use optimized designs to handle high
loads (requests per second).
First gunicorn and uwsgi are both appservers. In other words they are responsible for running your python code in a stable and performant manner. Usually as a backend to a regular webserver.
The webserver would be nginx, it excels at serving static assets and passing the requests for dynamic content on to the appservers.
If the above doesn't give enough performance you add in varnish between nginx and the client, it should speed up repeated requests for the same thing.
haproxy is a load balancer, if you have several servers for the same content, this software will attempt to distribute requests among them optimally.
so basically:
your python code lives in the appserver (uwsgi or gunicorn)
your static webassets live in nginx
haproxy and varnish are software that allow you to better server very large amounts of requests
What is the simplest way to make Django /admin/ urls accessible to localhost only?
Options I have thought of:
Seperate the admin site out of the project (somehow) and run as a different virtual host (in Apache2)
Use a proxy in front of the hosting (Apache2) web server
Restrict the URL in Apache within WSGI somehow.
Is there a standard approach?
Thanks!
Id go for apache configuration:
<Location /admin>
Order Deny, Allow
Deny from all
Allow from 127.0.0.1
</Location>
HTH.
I'd go for the Apache configuration + run a proxy in front + restrict in WSGI :
I dislike Apache for communicating with web clients when dynamic content generation is involved. Because of it's execution model, a slow or disconnected client can tie up the Apache process. If you have a proxy in front ( i prefer nginx, but even a vanilla apache will do ), the proxy will worry about the clients and Apache can focus on a new dynamic content request.
Depending on your Apache configuration, a process can also slurp a lot of memory and hold onto it until it hits MaxRequests. If you have memory intensive code in /admin ( many people do ), you can end up with Apache processes that grab a lot more memory than they need. If you split your Apache config into /admin and /!admin , you can tweak your apache settings to have a larger number of /!admin servers which require a smaller potential footprint.
I'm paranoid server setups.
I want to ensure the proxy only sends /admin to a certain Apache port.
I want to ensure that Apache only receives /admin on certain apache port, and that it came from the proxy (with a secret header) or from localhost.
I want to ensure that the WSGI is only running the /admin stuff based on certain server/client conditions.