We Keep Coding

django admin static files are uploaded to s3 via `collectstatic` but not served when running locally

I'm following this tutorial for storing static and media files on s3. This is what my static files configuration in settings.py looks like:
USE_S3 = os.getenv('USE_S3') == 'TRUE'
## AWS Configuration
if USE_S3:
# aws settings
AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME')
AWS_DEFAULT_ACL = None # differs from the tutorial because the bucket is private
AWS_S3_CUSTOM_DOMAIN = f'{AWS_STORAGE_BUCKET_NAME}.s3.amazonaws.com'
AWS_S3_OBJECT_PARAMETERS = {'CacheControl': 'max-age=86400'}
# s3 static settings
AWS_LOCATION = 'static'
STATIC_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}/'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
else:
STATIC_URL = '/staticfiles/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_DIRS = (os.path.join(BASE_DIR, 'static'),)
MEDIA_URL = '/mediafiles/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'mediafiles'
After setting all the env variables I run python manage.py collectstatic and I can see that the files have been successfully uploaded to my s3 bucket. I.e. I see static/admin/ directory on s3 with fonts, css, etc.
However when I run the server locally the admin panel is missing all the css. I'm not sure why django cannot find the admin static files given that collectstatic worked. I found several other tutorials here, here and here but I can't find what I'm missing. For all of them it seems like it's supposed to "just work" after running collectstatic... What did I forget?
Note The admin panel is fine when USE_S3 == False

So the objects in the s3 bucket weren't public. I made the assets public and now it's working.

Django collectstatic not working on production with S3, but same settings work locally

I've been moving around some settings to make more defined local and production environments, and I must have messed something up.
Below are the majority of relevant settings. If I move the production.py settings (which just contains AWS-related settings at the moment) to base.py, I can update S3 from my local machine just fine. Similarly, if I keep those AWS settings in base.py and push to production, S3 updates appropriately. In addition, if I print something from production.py, it does print. However, if I make production.py my "local" settings on manage.py, or when I push to Heroku with the settings as seen below, S3 is not updating.
What about my settings is incorrect? (Well, I'm sure a few things, but specifically causing S3 not to update?)
Here's some relevant code:
__init__.py (in the directory with base, local, and production)
from cobev.settings.base import *
base.py
INSTALLED_APPS = [
...
'whitenoise.runserver_nostatic',
'django.contrib.staticfiles',
...
'storages',
]
...
STATIC_URL = '/static/'
STATICFILES_DIRS = [os.path.join(BASE_DIR, "global_static"),
os.path.join(BASE_DIR, "media", )
]
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
MEDIA_URL = '/media/'
local.py
# local_settings.py
from .base import *
...
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
production.py
from .base import *
# AWS Settings
AWS_ACCESS_KEY_ID = config('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = config('AWS_SECRET_ACCESS_KEY')
AWS_STORAGE_BUCKET_NAME = 'cobev'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_LOCATION = 'static'
AWS_DEFAULT_ACL = 'public-read'
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
DEFAULT_FILE_STORAGE = 'cobev.storage_backends.MediaStorage'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, AWS_LOCATION)
ADMIN_MEDIA_PREFIX = STATIC_URL + 'admin/'
# End AWS
wsgi.py
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cobev.settings.production")
application = get_wsgi_application()
from whitenoise.django import DjangoWhiteNoise
application = DjangoWhiteNoise(application)
manage.py
#!/usr/bin/env python
import os
import sys
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "cobev.settings.local")
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)

Ok, let me try, as discovered in comments from the question, you do S3 update using collectstatic, but this is a management command which is called using manage.py file where you set cobev.settings.local as settings which are not equal to cobev.settings.production which is used for wsgi.py file.
I think you should manage your settings file using, normal Django way, OS environment variable named DJANGO_SETTINGS_MODULE.
For sure you should be able to set it in any production environment you are running.

Then if that can help here is my config for AWS/Django/S3 in production :
Common Static config :
# STATIC FILE CONFIGURATION
# ------------------------------------------------------------------------------
# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-root
STATIC_ROOT = str(ROOT_DIR('staticfiles'))
# See: https://docs.djangoproject.com/en/dev/ref/settings/#static-url
STATIC_URL = '/static/'
# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#std:setting-STATICFILES_DIRS
STATICFILES_DIRS = [
str(APPS_DIR.path('static'))
]
# See: https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/#staticfiles-finders
STATICFILES_FINDERS = [
'django.contrib.staticfiles.finders.FileSystemFinder',
'django.contrib.staticfiles.finders.AppDirectoriesFinder'
]
Common Media config :
# MEDIA CONFIGURATION
# ------------------------------------------------------------------------------
# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-root
MEDIA_ROOT = str(APPS_DIR('media'))
# See: https://docs.djangoproject.com/en/dev/ref/settings/#media-url
MEDIA_URL = '/media/'
Production Static config :
# STATIC CONFIG PRODUCTION
# ------------------------------------------------------------------------------
# See: http://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html
AWS_STORAGE_BUCKET_NAME = 'mybucket-name-production'
AWS_ACCESS_KEY_ID = 'YOUR_KEY_ID'
AWS_SECRET_ACCESS_KEY = 'YOUR_SECRET_KEY'
AWS_S3_HOST = "s3.amazonaws.com"
AWS_S3_URL = 'https://{bucker_name}.s3.amazonaws.com/'.format(bucker_name=AWS_STORAGE_BUCKET_NAME)
AWS_LOCATION = 'static/'
AWS_S3_URL_PROTOCOL = 'https:'
AWS_S3_CUSTOM_DOMAIN = 'static.mydomain.com' # I use sub domaine to serve static
STATIC_URL = '{AWS_S3_URL_PROTOCOL}//{AWS_S3_CUSTOM_DOMAIN}/{AWS_LOCATION}'.format(
AWS_S3_URL_PROTOCOL=AWS_S3_URL_PROTOCOL,
AWS_S3_CUSTOM_DOMAIN=AWS_S3_CUSTOM_DOMAIN,
AWS_LOCATION=AWS_LOCATION)
AWS_QUERYSTRING_AUTH = False
AWS_IS_GZIPPED = True
AWS_EXPIREY = 60 * 60 * 24 * 14
# For s3boto
AWS_HEADERS = {
'Cache-Control': 'max-age=%d, s-maxage=%d, must-revalidate' % (AWS_EXPIREY, AWS_EXPIREY)
}
# For s3boto3
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=%d' % AWS_EXPIREY,
}
AWS_PRELOAD_METADATA = True
#AWS_S3_FILE_OVERWRITE = True
STATICFILES_STORAGE = 'config.storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'config.storages.DefaultStorage'
# MEDIA S3 CONFIG PRODUCTION
# --------------------------------------------------------------------------------
AWS_MEDIA_DIR = 'media'
MEDIA_URL = AWS_S3_URL + AWS_MEDIA_DIR + '/'
MEDIA_ROOT = MEDIA_URL
Here is my StaticStorage Class :
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = 'static'
file_overwrite = False
class DefaultStorage(S3Boto3Storage):
location = ''
file_overwrite = False
After That I add command in the folder .ebextensions in .config file for collectstatic :
# ./ebextensions/02_container_commands.config file :
container_commands:
0.3.0.push.static.to.s3:
command: "source /opt/python/run/venv/bin/activate && python manage.py collectstatic --ignore=.scss --noinput"
leader_only: true
ignoreErrors: true

It looks like you are using Whitenoise. Whitenoise allows django to serve its static files. It is different approach if you want to serve them from AWS.
So, you need to remove Whitenoise in order to utilize django-storages. Remove it from settings, middleware, wsgi.py, etc.
Also, you can remove everything from __init__.py in settings - settings file to be used is set by DJANGO_SETTINGS_MODULE env variable.
According to your STATICFILES_DIRS, media directory is included as static files. It is preferred for media to be served separately from static files (one of differences - static files are more likely to be cached and gziped) - i.e. also with AWS but from separate S3 bucket.
You may also add AWS CloudFront as CDN in front of your buckets later.

Cannot save django staticfiles to AWS S3

I ran into a strange problem while deploying staticfiles to S3.
I installed boto3 and django-storages.
Django: 2.1.2
Python: 3.6.7
When running locally, my staticfiles settings were following,
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
TEMPLATE_DIR = os.path.join(BASE_DIR,'templates')
STATIC_DIR = os.path.join(BASE_DIR,'static')
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
STATIC_URL = '/static/'
STATICFILES_DIRS = [
STATIC_DIR,
]
STATICFILES_STORAGE = 'whitenoise.storage.CompressedManifestStaticFilesStorage'
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media/')
When I want to save staticfiles to S3 bucket, I changed settings to
STATICFILES_DIRS = [
os.path.join(BASE_DIR, 'static'),
]
AWS_ACCESS_KEY_ID = 'key'
AWS_SECRET_ACCESS_KEY = 'key'
AWS_STORAGE_BUCKET_NAME = 'name'
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
AWS_S3_OBJECT_PARAMETERS = {
'CacheControl': 'max-age=86400',
}
AWS_STATIC_LOCATION = 'static'
AWS_MEDIA_LOCATION = 'media'
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_STATIC_LOCATION)
STATIC_ROOT = STATIC_URL
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, AWS_MEDIA_LOCATION)
MEDIA_ROOT = MEDIA_URL
STATICFILES_STORAGE = 'main.storage_backends.StaticRootS3BotoStorage'
DEFAULT_FILE_STORAGE = 'main.storage_backends.MediaStorage'
storage_backends.py
from storages.backends.s3boto3 import S3Boto3Storage
StaticRootS3BotoStorage = lambda: S3Boto3Storage(location='static')
class MediaStorage(S3Boto3Storage):
location = 'media'
file_overwrite = False
After running,
python manage.py collectstatic
I still get the following prompt,
You have requested to collect static files at the destination
location as specified in your settings:
/home/drogon/Crowdsocial_project/staticfiles
This will overwrite existing files!
Are you sure you want to do this?
It still saves to staticfiles directory locally, not on s3.
What am I doing wrong?
I followed these two tutorials,
https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html
https://www.codingforentrepreneurs.com/blog/s3-static-media-files-for-django/

Amazon S3 with Django can't load all the static files

I'm using Amazon S3 to be my static and media files storage. My Django project is running in Digital ocean ubuntu 16.04.
After running python manage.py collectstatic I found the CSS and js did not work in my website. And then I found that the CSS and js hadn't been upload in the S3. I think this is the reason why the cs and js did not work because they are not in there.
there is only 'static' folder in S3.
in this static folder there are not my project static files but the admin xadmin and another plug's static files
Above is the folder under static in S3.
When I check the url of the js it looks like this:
<link rel="stylesheet" href="https://myproject.s3.amazonaws.com/css/main.css?Signature=imJphDmnb4U%2BWOWHjE0Iagk2tow%3D&AWSAccessKeyId=AKIAI4LFEI2ASSMOYRTQ&Expires=1537337559">
<link rel="icon" href="https://myproject.s3.amazonaws.com/images/logo-blue.png?Signature=ACidpeC946mBazTtHx0McVIk6rM%3D&AWSAccessKeyId=AKIAI4LFEI2ASSMOYRTQ&Expires=1537337559">
But in my project the main.css is under static folder, the images are under media folder. This is so weird for me.
This is my project folder looks like:
And I found that, after running python manage.py collectstatic the system didn't copy the static and media at all
Here is the main part of my settings.py:
import os
import sys
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
sys.path.insert(0, os.path.join(BASE_DIR, 'apps'))
sys.path.insert(0, os.path.join(BASE_DIR, 'extra_apps'))
ROOT_URLCONF = 'myproject.urls'
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
AWS_ACCESS_KEY_ID = 'myproject'
AWS_SECRET_ACCESS_KEY = 'myproject'
AWS_STORAGE_BUCKET_NAME = 'myproject'
AWS_S3_FILE_OVERWRITE = False
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
# the sub-directories of media and static files
STATICFILES_LOCATION = 'static'
MEDIAFILES_LOCATION = 'media'
# a custom storage file, so we can easily put static and media in one bucket
STATICFILES_STORAGE = 'myproject.custom_storages.StaticStorage'
DEFAULT_FILE_STORAGE = 'myproject.custom_storages.MediaStorage'
# the regular Django file settings but with the custom S3 URLs
STATIC_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
MEDIA_URL = "https://%s/%s/" % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
I also created a file named custom_storages.py:
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage
class StaticStorage(S3Boto3Storage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3Boto3Storage):
location = settings.MEDIAFILES_LOCATION
I followed this tutorial https://blog.bitlabstudio.com/ultra-short-guide-to-django-and-amazon-s3-2c5aae805ce4
This issue is really very complicated to me. Any friend can help? This issue spends me 2 days to solve it.

How to tell Django collectstatic to use Amazon S3?

I have followed this howto to setup django + S3. Specifically:
import os
# AWS credentials
AWS_STORAGE_BUCKET_NAME = os.environ['AWS_STORAGE_BUCKET_NAME']
AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
# boto config
AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
AWS_HEADERS = { # see http://developer.yahoo.com/performance/rules.html#expires
'Expires': 'Thu, 31 Dec 2099 20:00:00 GMT',
'Cache-Control': 'max-age=94608000',
}
# For the static files
STATICFILES_LOCATION = 'static'
STATICFILES_STORAGE = 'myapp.custom_storages.StaticStorage'
STATIC_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, STATICFILES_LOCATION)
# For the media files
MEDIAFILES_LOCATION = 'media'
DEFAULT_FILE_STORAGE = 'myapp.custom_storages.MediaStorage'
MEDIA_URL = 'https://%s/%s/' % (AWS_S3_CUSTOM_DOMAIN, MEDIAFILES_LOCATION)
My custom storages are simple S3BotoStorages:
from django.conf import settings
from storages.backends.s3boto import S3BotoStorage
class StaticStorage(S3BotoStorage):
location = settings.STATICFILES_LOCATION
class MediaStorage(S3BotoStorage):
location = settings.MEDIAFILES_LOCATION
I would expect that collectstatic would honor this configuration (as explained in the howto), and use the myapp.custom_storages.StaticStorage to upload collected static files to S3. Instead, it just uses the local filesystem. Since I have:
STATIC_ROOT = os.path.join(BASE_DIR, 'mycollectstatic')
(just because having 'static' there seems too confusing to me), I can clearly see that:
» python manage.py collectstatic
You have requested to collect static files at the destination
location as specified in your settings:
/absolute-path/mycollectstatic
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel:
So it seems that collectstatic command is using STATIC_ROOT even when STATICFILES_STORAGE = 'myapp.custom_storages.StaticStorage'. Is this expected?
Should I configure STATIC_ROOT differently when using another STATICFILES_STORAGE? Where is this documented?