I´ve been doing some research, and I have found articles explaining how to use Django's(1.8) cache busting, but they don´t explain my situation.
I am using S3 (and it works) with the following setting in settings.py:
STATICFILES_STORAGE= 'pms.s3utils.StaticRootS3BotoStorage'
In order to use cache busting the docs say I have to set: STATICFILES_STORAGE='django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
I don´t know what setting to use in order to use both S3 and cache busting.
Thanks!
So I finally found a workaround.
In order to be able to upload my files to 2 different folders (static and uploads) in my S3 bucket I have this in my settings.py file:
STATICFILES_STORAGE = 'myapp.s3utils.StaticRootS3BotoStorage'
DEFAULT_FILE_STORAGE = 'myapp.s3utils.MediaRootS3BotoStorage'
And this in the myapp.s3utils.py file:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='uploads')
But I couldn´t use Django´s cache busting.
The solution was to change my myapp.s3utils.py file to:
from storages.backends.s3boto import S3BotoStorage
from django.contrib.staticfiles.storage import ManifestFilesMixin
class CustomS3Storage(ManifestFilesMixin, S3BotoStorage):
pass
StaticRootS3BotoStorage = lambda: CustomS3Storage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='uploads')
Basically it adds the ManiFestfilesMixin, which allows you to use cache busting.
As a side note, you can see that I am only using cache busting for the static files, but not for the uploads files. That is why the MediaRootS3BotoStorage calls the S3BotoStorage class instead of the CustomS3Storage.
I do it this way because the uploads files are not stored in my server (the static files are), they are stored directly in the S3 bucket, so when I ran the collectstatic they are not on my server, so I don´t have to add the hash to the names.
Related
I have deployed my django project on google cloud. One module of my app involves uploading files. When I am uploading my files on local server the files are successfully uploading but when I am trying to upload the files from the production server it is giving me the following error:
OSError at /uploadrecords
[Errno 30] Read-only file system: '/workspace/media/test.pdf'
Following is my code for uploading files and the settings:
#views.py
image = request.FILES['filerecord']
fs = FileSystemStorage()
filename = fs.save(image.name, image)
obj.upfile = fs.url(filename)
obj.save()
#setting.py
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
#MEDIA_ROOT = BASE_DIR / "media"
Kindly suggest me a solution
I assume you're using App Engine standard.
The file system of hosted apps is constrained:
https://cloud.google.com/appengine/docs/standard/python3/using-temp-files
You can use /tmp for temporary storage of uploads.
You can want to consider e.g. Google Cloud Storage for persistent storage of files.
The io module provides Python’s main facilities for dealing with various types of I/O.
Try to use it
import io
io.BytesIO(some_data)
I'd like Django to serve some media files (e.g. user-uploaded files) only for logged-in users. Since my site is quite low-traffic, I think I will keep things simple and do not use django-sendfile to tell Nginx when to serve a file. Instead I'll let Django/Gunicorn do the job. To me this seems a lot simpler and for a low traffic site this maybe more secure.
But what is the best way to organize the file storage location? Media files are all stored below MEDIA_ROOT and this directory is served by Nginx in production. If I upload my files to MEDIA_ROOT/protected/ I have to tell Nginx not to serve the files in the subdirectory protected.
But is this a good idea? It seems a litte risky to me to allow Nginx access /media/ in the first place and then protect the subdirectory /media/protected/. Wouldn't it be better not to use a subdirectory of MEDIA_ROOT to store protected files?
But if I try something like this quick-and-dirty in my model:
upload_to='../protected/documents/%Y/%m/'
Django complains:
SuspiciousFileOperation at /admin/core/document/add/
The joined path (/home/me/projects/project/protected/documents/2016/09/test.file) is located outside of the base path component (/home/me/projects/project/media)
So I thing it is not good practice to "leave" the MEDIA_ROOT.
What is the best solution to store and serve protected media files?
Serving media files ( that may be large files) from view directly is not good. You can use sendfile extension available in nginx server; a sample nginx configuration is like below.
location /projects/project/media/{
# this path is not public
internal;
# absolute path
alias /projects/project/media/;
}
change your view to
#login_required
def serve_protected_document(request, file):
document = get_object_or_404(ProtectedDocument, file="protected/documents/" + file)
# Split the elements of the path
path, file_name = os.path.split(file)
response = HttpResponse()
response["Content-Disposition"] = "attachment; filename=" + file_name
# nginx uses this path to serve the file
response["X-Accel-Redirect"] = document.name # path to file
return response
Link: More details on configuring sendfile extension on nginx is here
I now came up with the following solution:
I have this in my Django settings:
MEDIA_ROOT = "/projects/project/media/"
MEDIA_URL = "/media/
In my models I do either:
document = models.FileField(upload_to="public/documents")
or
document = models.FileField(upload_to="protected/documents")
This way, I now have the two subdirectories 'public' and 'protected' in my media files directory.
Nginx or Djangos development server only serves the files in the 'public' subdirectory.
For Djangos development server:
if os.environ["ENVIRONMENT_TYPE"] == 'development':
urlpatterns += static(settings.MEDIA_URL + "public/", document_root=settings.MEDIA_ROOT + "public/")
And for Nginx (used in production):
location /media/public/ {
alias /projects/project/media/public/;
}
When I want to serve a protected document, I do the following:
In urls.py:
url(r'^media/protected/documents/(?P<file>.*)$', core.views.serve_protected_document, name='serve_protected_document'),
And in views.py:
#login_required()
def serve_protected_document(request, file):
document = get_object_or_404(ProtectedDocument, file="protected/documents/" + file)
# Split the elements of the path
path, file_name = os.path.split(file)
response = FileResponse(document.file,)
response["Content-Disposition"] = "attachment; filename=" + file_name
return response
I would appreciate any comments! Are there better ways to implement this?
I am trying to set up an app where users can download their files stored in an S3 Bucket. I am able to set up my bucket, and get the correct file, but it won't download, giving me the this error: No such file or directory: 'media/user_1/imageName.jpg' Any idea why? This seems like a relatively easy problem, but I can't quite seem to get it. I can delete an image properly, so it is able to identify the correct image.
Here's my views.py
def download(request, project_id=None):
conn = S3Connection('AWS_BUCKET_KEY', 'AWS_SECRET_KEY')
b = Bucket(conn, 'BUCKET_NAME')
k = Key(b)
instance = get_object_or_404(Project, id=project_id)
k.key = 'media/'+str(instance.image)
k.get_contents_to_filename(str(k.key))
return redirect("/dashboard/")
The problem is that you are downloading to a local directory that doesn't exist (media/user1). You need to either:
Create the directory on the local machine first
Just use the filename rather than a full path
Use the full path, but replace slashes (/) with another character -- this will ensure uniqueness of filename without having to create directories
The last option could be achieved via:
k.get_contents_to_filename(str(k.key).replace('/', '_'))
See also: Boto3 to download all files from a S3 Bucket
Downloading files using boto3 is very simple, configure your AWS credentials at system level before using this code.
client = boto3.client('s3')
// if your bucket name is mybucket and the file path is test/abc.txt
// then the Bucket='mybucket' Prefix='test'
resp = client.list_objects_v2(Bucket="<your bucket name>", Prefix="<prefix of the s3 folder>")
for obj in resp['Contents']:
key = obj['Key']
//to read s3 file contents as String
response = client.get_object(Bucket="<your bucket name>",
Key=key)
print(response['Body'].read().decode('utf-8'))
//to download the file to local
client.download_file('<your bucket name>', key, key.replace('test',''))
replace is to locate the file in your local with s3 file name, if you don't replace it will try to save as 'test/abc.txt'.
import os
import boto3
import json
s3 = boto3.resource('s3', aws_access_key_id="AKIAxxxxxxxxxxxxJWB",
aws_secret_access_key="LV0+vsaxxxxxxxxxxxxxxxxxxxxxry0/LjxZkN")
my_bucket = s3.Bucket('s3testing')
# download file into current directory
for s3_object in my_bucket.objects.all():
# Need to split s3_object.key into path and file name, else it will give error file not found.
path, filename = os.path.split(s3_object.key)
my_bucket.download_file(s3_object.key, filename)
I implemented this library for generate barcodes images (http://kennethngedo.wordpress.com/2014/02/07/how-to-generate-barcode-in-django-using-reportlab/)
Everything works fine, the image is generated correctly, BUT... the image is created in a folder outside the project, and such I'm using Heroku for Production, I can't access to the image.
I'm using this Django structure (http://django-skel.readthedocs.org/en/latest/) specially adapted to work on Heroku with Amazon S3.
Do you know guys how can I upload the generated image on my Media folder on Amazon?
This is my Views.py where the image is created and saved:
from random import randint
from reportlab.lib.units import mm
from reportlab.graphics.barcode import *
from reportlab.graphics.shapes import Drawing, String
from django.shortcuts import render_to_response
class MyBarcodeDrawing(Drawing):
def __init__(self, text_value, *args, **kw):
barcode = createBarcodeDrawing('Code128', value=text_value, barHeight=10*mm, humanReadable=True)
Drawing.__init__(self,barcode.width,barcode.height,*args,**kw)
self.add(barcode, name='barcode')
def barcode_generator(barcode_value):
text = barcode_value
filename = "nightology_barcode_" + barcode_value
path_to_save = "media/barcodes/"
b = MyBarcodeDrawing(text)
b.save(formats=['gif','pdf'],outDir=path_to_save,fnRoot=filename)
barcodePicUrl = "barcode/"+ filename + ".gif"
return barcodePicUrl
I hope somebody could help me on this... I'll really appreciate.
Thanks!
I had a similar Problem but without the Amazon S3 part. For me it was really easy to create a new File in the media folder. I could simply use default_storage to get the path to the media folder:
from django.core.files.storage import default_storage
import os
# Get the path to the barcodes folder in the media folder
# check if the folder exists and create it if not
folderPath = default_storage.path('barcodes')
if not default_storage.exists('barcodes'):
os.mkdir(folderPath)
Since django-skel seems to be using a django-storage backend for Amazon S3 this should also work for your setup. If the Amazon S3 storage backend is not the default storage backend, you might have to use the S3Storage class directly.
I am getting a file posting from a file:
file = request.post['ufile']
I want to get the path. How can I get it?
You have to use the request.FILES dictionary.
Check out the official documentation about the UploadedFile object, you can use the UploadedFile.temporary_file_path attribute, but beware that only files uploaded to disk expose it (that is, normally, when using the TemporaryFileUploadHandler uploads handler).
upload = request.FILES['ufile']
path = upload.temporary_file_path
In the normal case, though, you would like to use the file handler directly:
upload = request.FILES['ufile']
content = upload.read() # For small files
# ... or ...
for chunk in upload.chunks():
do_somthing_with_chunk(chunk) # For bigger files
You should use request.FILES['ufile'].file.name
you will get like this /var/folders/v7/1dtcydw51_s1ydkmypx1fggh0000gn/T/tmpKGp4mX.upload
and use file.name, your upload file have to bigger than 2.5M.
if you want to change this , see File Upload Settings
We cannot get the file path from the post request, only the filename, because flask doesn't has the file system access. If you need to get the file and perform some operations on it then you can try creating a temp directory save the file there, you can also get the path.
import tempfile
import shutil
dirpath = tempfile.mkdtemp()
# perform some operations if needed
shutil.rmtree(dirpath) # remove the temp directory