I'm using reverse proxy with Nginx.
When I POST a file to the Nginx, it seems that it will store the whole file in local and forward it to the backend server after received the whole file.
Is there a way to make Nginx receive & forward data synchronously？
It's already answered negatively from this SO link: nginx files upload streaming with proxy_pass
The answer of the above question was from one of the guys who is maintaining nginx code base. So you can forget it for now.
If it's really important to not transporting files twice, you may try nginx upload module if you have control over your upstream server. http://wiki.nginx.org/HttpUploadModule.
You mean streaming. Yeah, you probably want to play with proxy_buffering, proxy_store and/or proxy_temp_file_write_size:
http://wiki.nginx.org/HttpProxyModule#proxy_store
http://wiki.nginx.org/HttpProxyModule#proxy_buffering
http://wiki.nginx.org/HttpProxyModule#proxy_temp_file_write_size
Side note: since nginx is single-threaded, then you really want to use that feature (otherwise one upload may block entire server for quite a long time).
Related
I wrote a Flask web application for a system that our company uses. However, we have another web application, which is running on Node.JS. The "problem" is that my colleague writes everything on node, while I write everything in Python.
We want to implement both applications on one webpage - for example:
My application will run on example.com/assistant
His application will run on example.com/app1 and example.com/app2
How can we do this? Can we somehow implement the templates that I use with his templates and vice-versa?
Thank you in advance!
V
Serving different apps from the same domain
You can use haproxy for directing requests to specific service based on ACL rules.
You could use path_beg rule, to direct any request beginning with specific path to be directed to corresponding server. See example below.
/etc/haproxy/haproxy.cfg
# only relevant part of the config file
# assumes all apps are on one machine
frontend http-in
bind *:80
acl py_app1 path_beg /assistant
acl node_app1 path_beg /app1
acl node_app2 path_beg /app2
default_backend main_servers
backend py_app1
server flask_app 127.0.0.1:5000
backend node_app1
server nodejs1 127.0.0.1:4001
backend node_app2
server nodejs2 127.0.0.1:4002
backend main_servers
server other1 127.0.0.1:3000 # nginx, apache, or whatever
Sharing template code between apps
This would be harder, as you would need to both agree on some kind of format, which needs to be language and framework-agnostic, and probably logic-less.
Mustache claims to be "framework-agnostic way to render logic-free views". I used it sparringly a few years ago so this one is first that came to mind, however you should do more research on this, maybe there is some better fit.
Python implementation
JS implementation
The problem would be to actually keep the templates always in sync with apps, and not break functionality of the views. If a template changes then you would need to test all apps that use this template file. Also, you probably will block one another from updating your apps at different times, because if one of you change the template files, then you must come to a consensus, update all relevant apps, and deploy them at one time.
In my small web-site I feel need to make some data widely available, to avoid exchanging with database for every request made. E.g. this could be the list of current users show in the bottom of every page or the time of last update of ranking.
The stuff works in Python (Flask) running upon nginx + uwsgi (this docker image).
I wonder, do I have some small cache or shared memory for keeping such information "out of the box", or I need to take care of explicitly setting up some dedicated cache? Or perhaps some thing like this is provided by nginx?
alternatively I still can use database for it has its own cache I think, anyway
Sorry if question seems to be naive/silly - for I come from java world (where things a bit different as we serve all requests with one fat instance of java application) - and have some difficulty grasping what powers does wsgi/uwsgi provide. Thanks in advance!
Firstly, nginx has cache:
https://www.nginx.com/blog/nginx-caching-guide/
But for flask cacheing you also have options:
https://pythonhosted.org/Flask-Cache/
http://flask.pocoo.org/docs/1.0/patterns/caching/
Did you have a look at caching section from Flask docs?
It literally says:
Flask itself does not provide caching for you, but Werkzeug, one of the libraries it is based on, has some very basic cache support
You create a cache object once and keep it around, similar to how Flask objects are created. If you are using the development server you can create a SimpleCache object, that one is a simple cache that keeps the item stored in the memory of the Python interpreter:
from werkzeug.contrib.cache import SimpleCache
cache = SimpleCache()
-- UPDATE --
Or you could solve on the frontend side storing data in the web browser local storage.
If there's nothing in the local storage you call the DB, else you use the information from local storage rather than making db call.
Hope it helps.
My python application sits behind an Nginx instance. When I upload an image, which is one of the purpose of my app, I notice that nginx first saves the image in filesystem (used 'watch ls -l /tmp') and then hands it over to the app. Can I configure Nginx to work in-memory with image POST? My intent is to avoid touching the slow filesystem (the server runs on an embedded device).
Yes, set the proxy_max_temp_file_size to zero, or some other reasonably small value. Another option (which might be a better choice) is to set the proxy_temp_path to faster storage so that nginx can do a slightly better job of insulating the application from buggy or malicious hosts.
I have a URL route in my web.py application that I want to run to catch all URLs that hit the server, but only after any static assets are served.
For example, if theres is js/test.js in my static directory, the path http://a.com/js/tests.js should return the file contents. But I also have my url routing set up so that there is a regex that catches everything like this:
urls = ('/.*', 'CatchAllHandler')
So this should run only if no static asset was discovered. A request for http://a.com/js/test.js should return the static file test.js, but a request for http://a.com/js/nope.js should route through the CatchAllHandler.
I've looked into writing my own StaticMiddleware for this, but it will only help if the order of web.py operations is changed. Currently the middleware is executed after the URL routes have been processed. I need the middleware to run first, and let the url routing clean up the requests that were not served static assets.
The one idea I have is to use the notfound() function as my catch all handler, but that may not be best.
the url matching is python regex. You can test/play with python regex here
that said, this should work for you:
('/(?!static)(.*)', 'CatchAllHandler')
I haven't played with web.py's middleware, but my understanding.. WSGI middleware happens before web.py gets to seeing the request/response. I would think, provided your WSGI MiddleWare is properly configured, it would just work.
pouts That sucks. There is the hook stuff, which makes it really easy, I've don't that before, and it will see all the stuff before .. docs are here: http://webpy.org/cookbook/application_processors
but I guess in regards to your other comment, 'wanting it to work regardless of URL'. How would you know it's static content otherwise? I'm confused greatly. The EASIEST way, since for production you want some other web server running your web.py scripts, is to push all the static content into the web server. Then you can of course do whatever you want in the web server that needs doing. This is exactly what happens with mod_wsgi and apache for instance (you change /static to point to the directory IN the web server).
Perhaps if you shared an actual example of what you need done, I could help you more. Otherwise I've given you now 3 different ways to handle the problem (excluding using WSGI middleware). How many more do you need? :P
I'm maintaining an open-source document asset management application called NotreDAM, which is written in Django running on Apache an instance of TwistedWeb.
Whenever any user downloads a file, the application hangs for all users for the entire duration of the download. I've tracked down the download command to this point in the code, but I'm not enough versed with Python/Django to know why this may be happening.
response = HttpResponse(open(fullpath, 'rb').read(), mimetype=mimetype)
response["Last-Modified"] = http_date(statobj.st_mtime)
response["Content-Length"] = statobj.st_size
if encoding:
response["Content-Encoding"] = encoding
return response
Do you know how I could fix the application hanging while a file downloads?
The web server reads the whole file in the memory instead of streaming it. It is not well written code, but not a bug per se.
This blocks the Apache client (pre-forked) for the duration of whole file read. If IO is slow and the file is large it may take some time.
Usually you have several pre-forked Apache clients configured to satisfy this kind of requests, but on a badly configured web server you may exhibit this kind of problems and this is not a Django issue. Your web server is probably running only one pre-forked process, potentially in a debug mode.
notreDAM serves the asset files using the django.views.static.serve() command, which according to the Django docs "Using this method is inefficient and insecure. Do not use this in a production setting. Use this only for development." So there we go. I have to use another command.