Why serving static files are an issue in Python apps - python

I don't seem to understand the issue with serving static files in Python apps, for example Django. Now I am reading a documentation of WhiteNoise and first question is why it was developed in the first place? I mean what problem it solves? Why we can't save all the static files in /static folder and copy it to hosting.


I don't seem to understand the issue with serving static files in Python apps, for example Django.
As is specified in the Django documentation on managing static files:
This method is grossly inefficient and probably insecure, so it is unsuitable for production.
Indeed, Django's views to serve static files are likely not very efficient. A webserver will typically cache a certain amount of files, use compression to send these to the browser (if the browser accepts this), etc. Django's views to serve static/media files are not designed to be competitive with a webserver like apache/nginx.
Even if that is solved, it using a signle server to serve static content is not a good idea. Often Content Delivery Networks (CDNs) are used to find a server close to the client that can serve these files.
The Django documentation therefore contains articles to deploy static files on a dedicated server and how to configure this server.
The whitenoise package is simply a best effort attempt to serve these files with a Python view. Python is more designed to reduce programmer time, not processing time, so while whitenoise is efficient, very likely it can not compete with CDN.

Related

What's the point of Django's collectstatic?

In Django, the convention is to put all of your static files (i.e css, js) specific to your app into a folder called static. So the structure would look like this:
mysite/
manage.py
mysite/ --> (settings.py, etc)
myapp/ --> (models.py, views.py, etc)
static/
In mysite/settings.py I have:
STATIC_ROOT = 'staticfiles'
So when I run the command:
python manage.py collectstatic
It creates a folder called staticfiles at the root level (so same directory as myapp/)
What's the point of this? Isn't it just creating a copy of all my static files?

Collect static files from multiple apps into a single path
Well, a single Django project may use several apps, so while there you only have one myapp, it may actually be myapp1, myapp2, etc
By copying them from inside the individual apps into a single folder, you can point your frontend web server (e.g. nginx) to that single folder STATIC_ROOT and serve static files from a single location, rather than configure your web server to serve static files from multiple paths.
Persistent URLs with ManifestStaticFilesStorage
A note about the MD5 hash being appended to the filename for versioning: It's not part of the default behavior of collectstatic, as settings.STATICFILES_STORAGE defaults to StaticFilesStorage (which doesn't do that)
The MD5 hash will kick in e.g. if you set it to use ManifestStaticFilesStorage, which adds that behavior.
The purpose of this storage is to keep serving the old files in case
some pages still refer to those files, e.g. because they are cached by
you or a 3rd party proxy server. Additionally, it’s very helpful if
you want to apply far future Expires headers to the deployed files to
speed up the load time for subsequent page visits.

Django static files can be in many places. A file that is served as /static/img/icon.png could come from many places. By default:
FileSystemFinder will look for img/icon.png in each of STATICFILES_DIRS,
AppDirectoriesFinder will look for img/icon.png in the static subfolder in each of your INSTALLED_APPS. This allows libraries like Django Admin to add their own static files to your app.
Now: this only works if you run manage.py runserver with DEBUG=1. When you go live, the Django process will no longer serve the static assets. It would be inefficient to use Django for serving these, there are more specialised tools specifically for that.
Instead, you should do something like this:
find all of static files from every app
build a single directory that contains all of them
upload them somewhere (a static directory somewhere on your webserver or a third-party file storage)
configure your webserver (such as nginx) to serve /static/* directly from that directory and redirect any other requests to Django.
collectstatic is a ready-made script that prepares this directory for you, so that you can connect it directly to your deployment script.

In the production installation, you want to have persistent URLs. The URL doesn't change unless the file content changes.
This is to prevent having clients to have wrong version of CSS or JS file on their computer when opening a web page from Django. Django staticfiles detects file changes and updates URLs accordingly, so that if CSS or JS file changes the web browser downloads the new version.
This is usually achieved by adding MD5 hash to the filename during collectstatic run.
Edit: Also see related answer to multiple apps.

It's useful when there are multiple django apps within the site.
collectstatic will then collect static files from all the apps in one place - so that it could be served up in a production environment.

How to cleanly handle changes to external static files in a django project

Say I've got a django project in a git repo with static files hosted on Amazon S3. Pretty common setup. Now there's me and another person working on this project; one is working on the javascript part and I'm working on the python part. Is there a sensible way to take changes the other person makes to the static files hosted on Amazon S3 and bring them back into the django git project?
Django has a sensible one-way flow for static files: collectstatic grabs all of them and then if you use django-storages it pushes them out to Amazon S3 for you. But what about the reverse? How to pull the changed versions of the static files back into the /app/static/ folders in django to keep everything synchronized. I'm not aware of any django functionality. Multiple git repos? Something else?
BTW I am using Heroku, if that helps any.
Any suggestions for how to handle this nicely, or am I doing it all wrong?

js and css files are not read in a DJANGO project

I am reformulating a previous question because I think it was formulated as a Python problem but it is a Django one.
I am installing a project from github called publicmarkup:
The main page displays on the browser but no media (neither js files, nor css files) are read loaded. There is a module called mediasync that seems to be necessary to display correctly the css and the js files of the templates.
I think this a configuration problem. Here's the link settings.py file, Here it could be find the root path of the media folder.
And here's the doc for the mediasync modulet explains how to configure static_url
But I didn't understand anything since I am a newbie to djnago

In Django you have to set up your own workstation serve static files, in both dev and live modes.
You can read about it here.
This explains how to serve static files in your dev environment.
In live mode you serve your static files directly via your end web server, usually either Apache or Nginx.

A web application to serve static files

I am thinking to design my own web app to serve static files. I just don't want to use Amazon services..
So, can anyone tell me how to start the project? I am thinking to develop in Python - Django on Openshift (Redhat's).
This is how ideas are going through in my mind:
A dashboard helps me to add/ delete/ manage static files
To setup API kind of thing (end point: JSON objects) so that I can use this project to serve my web apps!
As openshift uses Apache!, I am thinking to dynamically edit htaccess and serve the files.. but not sure whether it would be possible or not
Or, I can use django's urls.py to serve the files but I don't think djano is actually made for.
Any ideas and suggestion?

How about this:
Use nginx to serve static files
Keep the files in some kind of predefined directory structure, build a django app as the dashbord with the filesystem as the backend. That is, moving, adding or deleting files from the dashboard changed them the filesystem and nginx doesn't have to be aware of this dashboard.
Do not use dynamic routing. Just layout and maintain the proper directory structure using the databoard.
Optionally, keep the directory structure and file metadata in some database server for faster searches and manipulation.
This should result in a very low overhead static file server.

Serving static files with dynamic URLs using nginx

I have some static files that I would like to serve, and I aware that Django and Python are not suitable for serving static files in a production environment.
My current set up is nginx with python/django behind it wither via nginx_fastcgi or uwsgi workers. Nginx serves up static files like the CSS and the Javascript, and all the dynamic pages are passed on to python to generate.
However there are urls like /user/input/battleships.in. Here battleships.in is a static file located on the filesystem at `$MEDIA_ROOT/users/$request.user.username. Is there a way to have django determine the correct url, and then proxy the request back to nginx to serve the static file?

Try to be not that trival =)
use X-accel module of Nginx. check
http://wiki.nginx.org/X-accel
How to make a private download area with django?
Django - Understanding X-Sendfile

Categories