I am using s3boto, django-pipeline and django-storages with this setting:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'utils.classutils.S3PipelineStorage'
MEDIA_ROOT = "/"
MEDIA_URL = '//s3.amazonaws.com/%s/' % AWS_STORAGE_BUCKET_NAME
STATIC_ROOT = '/static/'
STATIC_URL = '//s3.amazonaws.com/%s/static/' % AWS_STORAGE_BUCKET_NAME
STATICFILES_DIRS = (
os.path.join(SITE_ROOT, 'assets'),
)
and this custom storage for django-pipeline
from django.contrib.staticfiles.storage import CachedFilesMixin
from pipeline.storage import PipelineMixin
from storages.backends.s3boto import S3BotoStorage
class S3PipelineStorage(PipelineMixin, CachedFilesMixin, S3BotoStorage):
pass
but I keep on getting:
ValueError: The file 'project/head.png' could not be found with <utils.classutils.S3PipelineStorage object at 0x2163510>.
but this file does not exist anywhere! not in plugins, I tried finding it, I checked my static dirs, tried finding it in admin, and I don't even remember working with a file named like this! I tried findstatic and the file can't be found.
What could I be missing?
You certainly have a css file that point to a non-existing 'project/head.png' file, remove this reference, and it should work.
Short tip. I encountered this when I was setting up S3 for serving static files.
I wanted to serve static content from s3 and keep handling uploaded files to the media folder on the local machine. What I had then was one image that was referenced in a css-file on s3 that was pointing to an image in the media folder like this background: url(../media/some/image.png).
Obviously that file could not be found in s3 with the relative path that was set in the css and the upload crashed. It works locally but not when running staticfiles from s3.
Related
I am using Django 4.0.1 in my project, and right prior to deploying my site, I am faced with the issue of handling my static files. Due to the limit of my server, I have decided to instead serve these static files via CDN.
I have already configured my STATIC_URL option in settings.py:
STATIC_URL = 'assets/'
I am aware that in the Django documentation, they say that this url refers to the static files located in STATIC_ROOT. Of course, normally the latter is an absolute path on your server where the collectstatic command collects the static files and put them there, but I am wondering if I can configure this STATIC_ROOT to point a path which is not on my server.
To be precise, I want to know whether I can point STATIC_ROOT to my CDN storage. In that way I can still use STATIC_URL to refer to my static assets, while being able to serve them via CDN.
Yes, it's actually a recommended way to serve static files for years. Not sure why Django didn't add it to its core.
You can use django-storages, it supports almost every cloud provider. You can use it for media and/or static.
Yes definitely you can use it for any other server. For example while we use AWS S3 as a server to serve static files it should look like this :
USE_S3 = os.getenv('USE_S3') == 'TRUE'
if USE_S3:
# aws settings
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
AWS_DEFAULT_ACL = 'public-read'
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
# s3 static settings
AWS_LOCATION = 'static'
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
else:
STATIC_URL = '/staticfiles/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
I have configured an S3 bucket to store and serve static and media files for a Django website, currently just trying to get the static files needed for the admin pages and all that.
Here is all the static and AWS config info in my settings file:
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'config.storage_backends.MediaStorage'
#used to authenticate with S3
AWS_ACCESS_KEY_ID = 'AKIAWWJOJKZGFSJO2UPW' #not real one
AWS_SECRET_ACCESS_KEY = 'KNg1z5wXWiDRAIh4zLiHgbD2N3wtWZTK' #not real one
#for endpoints to send or retrieve files
AWS_STORAGE_BUCKET_NAME = 'my-static' #not real bucket name
AWS_DEFAULT_ACL = None
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400',}
AWS_LOCATION = 'static'
STATIC_ROOT = 'static'
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'config/static'),
]
Of course I replaced any sensitive variables with fake ones for the purpose of this post. I have gone through many tutorials and other posts and I seem to have my STATIC_URL configured correctly but whenever I runserver and go to the admin pages, none of the css is applied. I do not think it is properly retrieving the static files (they are all uploaded to the S3 bucket) from the bucket, but I am stuck on what to do.
I am sure you might have missed some configurations like public and static file hosting on your s3 bucket.
Please go through this article Using Amazon S3 to Store your Django Site's Static and Media Files
I have personally tried this and it worked for me. The article explains every step.
I have the following code in my settings.py in django.
DEFAULT_FILE_STORAGE = 'hhhh.utils.MediaRootS3BotoStorage'
STATICFILES_STORAGE = 'hhhh.utils.StaticRootS3BotoStorage'
S3DIRECT_REGION = 'us-west-2'
S3_URL = '//%s.s3.amazonaws.com/' % AWS_STORAGE_BUCKET_NAME
MEDIA_URL = '//%s.s3.amazonaws.com/media/' % AWS_STORAGE_BUCKET_NAME
MEDIA_ROOT = MEDIA_URL
STATIC_URL = S3_URL + 'static/'
STATIC_ROOT = STATIC_URL + 'static_root/'
Heroku is not serving the ststic files. any ideas. I have the allowed hosts set to my site and heroku.
If I am reading your variables correctly, your static URL is built as so:
S3_URL + static + static_root
So, if your s3 bucket is named hhhh, then the final URL is
//hhhh.s3.amazonaws.com/static/static_root
Do files exist in that location?
For more info, Heroku offers a sample settings.py file regarding Django static files here: serving static assets with Django: https://devcenter.heroku.com/articles/django-assets
This page speaks specifically to hosting s3 files on s3 with heroku:
http://www.jorgechang.com/blog/howto-deploy-a-fault-tolerant-django-app-on-aws-part-2-moving-static-media-files-to-s3/
The author's STATIC_ROOT variable is blank, because static locations of files are set- and then the code later refers to the files on an Amazons3 location- it seems his code collects static files from a specific place and puts them in S3, then references them from there. You seem to be attempting to refer directly to an amazon s3 URL on your static_root var, so this laws gives you an alternate way to go it.
I have setup an AWS S3 bucket in order to transfer my static files in a remote CDN using the application django-storages,
everything worked fine until I tried to compress my static files before uploading to S3 using django_compressor.
I have setup all the variables according django_compressor documentation for django-storages (https://django_compressor.readthedocs.org/en/latest/remote-storages/index.html)
I uploaded all the files in S3 using 'manage.py collectstatic' then:
When I do 'manage.py compress' I get this error:
CommandError: An error occured during rendering ../templates/base.html: 'https://my_bucket.s3.amazonaws.com/css/bootstrap.2.3.1.css' isn't accessible via COMPRESS_URL ('https://my_bucket.s3-external-3.amazonaws.com/') and can't be compressed
What's wrong with my setup?
Here is my settings.py configuration for django-storages and django_compressor:
COMPRESS_URL = 'https://mybucket_name.s3-external-3.amazonaws.com/'
STATIC_URL = COMPRESS_URL
DEFAULT_FILE_STORAGE = 'my_project.boto_custom.CachedS3BotoStorage'
AWS_ACCESS_KEY_ID = 'XXX'
AWS_SECRET_ACCESS_KEY = 'XXX'
AWS_STORAGE_BUCKET_NAME = 'mybucket_name'
COMPRESS_ROOT = STATIC_ROOT
COMPRESS_STORAGE = 'my_project.boto_custom.CachedS3BotoStorage'
STATICFILES_STORAGE = 'my_project.boto_custom.CachedS3BotoStorage'
COMPRESS_OFFLINE = True
Thanks for your help
I fixed it by adding one variable and it worked:
AWS_S3_CUSTOM_DOMAIN = 'my_bucket.s3-external-3.amazonaws.com'
If you have separate S3 buckets for static and media you can also put it in your subclass of S3BotoStorage like so:
class CachedS3BotoStorage(S3BotoStorage):
custom_domain = 'my_bucket.s3-external-3.amazonaws.com'
(or better yet set it to settings.AWS_S3_CUSTOM_STATIC_DOMAIN or something)
I'm moving over my django app to Heroku, and I was wondering what the proper way to handle static files is. Do I just push them via git to Heroku? Or should I be storing them on SW3 or something? Also, what should the STATIC_ROOT and such be?
Thanks!
You should store them externally on a service like S3 - while Heroku can serve static files, it's not designed to.
Here's a good primer on getting started with S3:
https://devcenter.heroku.com/articles/s3
Use django-storages http://django-storages.readthedocs.org/en/latest/index.html to collect static files to your S3 bucket and serve them accordingly.
These are the necessary settings you'll need to have for S3:
DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = 'access-id'
AWS_SECRET_ACCESS_KEY = 'secret-key'
AWS_STORAGE_BUCKET_NAME = 'bucket-name'
AWS_PRELOAD_METADATA = True # necessary to fix manage.py collectstatic command to only upload changed files instead of all files
MEDIA_ROOT and STATIC_ROOT are superceded by DEFAULT_FILE_STORAGE and STATICFILES_STORAGE respectively and hence not needed. You will, however, want to set MEDIA_URL and STATIC_URL to something like
STATIC_URL = 'https://bucket-name.s3.amazonaws.com/static/'
ADMIN_MEDIA_PREFIX = 'https://bucket-name.s3.amazonaws.com/static/admin/'
If you want to store your static and media files in different subfolders, this is a great solution: https://stackoverflow.com/a/10825691/674794
You'll want to set MEDIA_URL and STATIC_URL to the respective new folders, e.g.
MEDIA_URL = 'https://bucket-name.s3.amazonaws.com/media/'
STATIC_URL = 'https://bucket-name.s3.amazonaws.com/static/'
You'll also want to manually execute manage.py collectstatic and disable Heroku's automatic collectstatic as per https://devcenter.heroku.com/articles/django-assets#disabling_collectstatic, as Heroku's collectstatic will reupload every static file to S3 every time you push even if the files haven't been modified, adding a hefty transfer and request load to S3 and slowing down your pushes.
Then just continue using {{ STATIC_URL }} in your templates like usual and you should be set!
<link href='{{ STATIC_URL }}css/styles.css' type='text/css' rel='stylesheet'>
If you want to start off simple and choose not to immediately take that route though, you can do the quick hack in your urls configuration by following Cesar's mentioned post at Heroku - Handling static files in Django app, though there'll be a significant decrease in app performance.
While #Intenex's answer might still be the way to go if you have a lot of static content, for getting started, Heroku suggests using Whitenoise.
Here's Heroku's article aptly entitled "Django and Static Assets".
And the whitenoise documentation itself has a nice section on "Shouldn’t I be pushing my static files to S3 using something like Django-Storages?"