I'd like Django to serve some media files (e.g. user-uploaded files) only for logged-in users. Since my site is quite low-traffic, I think I will keep things simple and do not use django-sendfile to tell Nginx when to serve a file. Instead I'll let Django/Gunicorn do the job. To me this seems a lot simpler and for a low traffic site this maybe more secure.
But what is the best way to organize the file storage location? Media files are all stored below MEDIA_ROOT and this directory is served by Nginx in production. If I upload my files to MEDIA_ROOT/protected/ I have to tell Nginx not to serve the files in the subdirectory protected.
But is this a good idea? It seems a litte risky to me to allow Nginx access /media/ in the first place and then protect the subdirectory /media/protected/. Wouldn't it be better not to use a subdirectory of MEDIA_ROOT to store protected files?
But if I try something like this quick-and-dirty in my model:
upload_to='../protected/documents/%Y/%m/'
Django complains:
SuspiciousFileOperation at /admin/core/document/add/
The joined path (/home/me/projects/project/protected/documents/2016/09/test.file) is located outside of the base path component (/home/me/projects/project/media)
So I thing it is not good practice to "leave" the MEDIA_ROOT.
What is the best solution to store and serve protected media files?
Serving media files ( that may be large files) from view directly is not good. You can use sendfile extension available in nginx server; a sample nginx configuration is like below.
location /projects/project/media/{
# this path is not public
internal;
# absolute path
alias /projects/project/media/;
}
change your view to
#login_required
def serve_protected_document(request, file):
document = get_object_or_404(ProtectedDocument, file="protected/documents/" + file)
# Split the elements of the path
path, file_name = os.path.split(file)
response = HttpResponse()
response["Content-Disposition"] = "attachment; filename=" + file_name
# nginx uses this path to serve the file
response["X-Accel-Redirect"] = document.name # path to file
return response
Link: More details on configuring sendfile extension on nginx is here
I now came up with the following solution:
I have this in my Django settings:
MEDIA_ROOT = "/projects/project/media/"
MEDIA_URL = "/media/
In my models I do either:
document = models.FileField(upload_to="public/documents")
or
document = models.FileField(upload_to="protected/documents")
This way, I now have the two subdirectories 'public' and 'protected' in my media files directory.
Nginx or Djangos development server only serves the files in the 'public' subdirectory.
For Djangos development server:
if os.environ["ENVIRONMENT_TYPE"] == 'development':
urlpatterns += static(settings.MEDIA_URL + "public/", document_root=settings.MEDIA_ROOT + "public/")
And for Nginx (used in production):
location /media/public/ {
alias /projects/project/media/public/;
}
When I want to serve a protected document, I do the following:
In urls.py:
url(r'^media/protected/documents/(?P<file>.*)$', core.views.serve_protected_document, name='serve_protected_document'),
And in views.py:
#login_required()
def serve_protected_document(request, file):
document = get_object_or_404(ProtectedDocument, file="protected/documents/" + file)
# Split the elements of the path
path, file_name = os.path.split(file)
response = FileResponse(document.file,)
response["Content-Disposition"] = "attachment; filename=" + file_name
return response
I would appreciate any comments! Are there better ways to implement this?
Related
I have deployed my django project on google cloud. One module of my app involves uploading files. When I am uploading my files on local server the files are successfully uploading but when I am trying to upload the files from the production server it is giving me the following error:
OSError at /uploadrecords
[Errno 30] Read-only file system: '/workspace/media/test.pdf'
Following is my code for uploading files and the settings:
#views.py
image = request.FILES['filerecord']
fs = FileSystemStorage()
filename = fs.save(image.name, image)
obj.upfile = fs.url(filename)
obj.save()
#setting.py
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
#MEDIA_ROOT = BASE_DIR / "media"
Kindly suggest me a solution
I assume you're using App Engine standard.
The file system of hosted apps is constrained:
https://cloud.google.com/appengine/docs/standard/python3/using-temp-files
You can use /tmp for temporary storage of uploads.
You can want to consider e.g. Google Cloud Storage for persistent storage of files.
The io module provides Python’s main facilities for dealing with various types of I/O.
Try to use it
import io
io.BytesIO(some_data)
Using flask. I have made an internal file browser/media player. This is for local network only so everyone who has access to the page has access to these files all ready.
Nevertheless I am dealing with 1000s of files in 1000s of locations. Is it possible to source a file in a html video player, or a img src that is local. The source files cant be moved, so cant go to the static folder etc...
like this
<video src="{{ clip }}" autoplay controls></video>
when clip is the file_path /projects/project_234/video/video_file.mov
I have all the variables needed just not able to get the file to play.
EDIT 01
It has come to my attention that mov files dont play in chrome only mp4's.
#app.route('/projects/<project>/<clip>', methods=['GET'])
def project_page_clip(project, clip):
file_path = request.args.get('file_path')
file_location = file_path
file_name = '90Sec_Approval.mp4'
if file_name:
return send_from_directory(file_location,file_name)
return render_template("project_selected_clip.html", file_path=file_path,
title=project, project=project, clip=clip)
So when clicked on the previous page this just opens the clip on a browser without rendering the project_selected_clip.html template
How can I get it to use the return send from directory as a src on the page instead?
After much deliberation, the best of many evils was to generate symlinks for the fles
This seems to be a big help
So after that... what works for me is to include this;
from Flask import send_file
top_dir = 'top/'
mid_dir = 'mid/'
more_dir = '/and/this/'
filename = 'some_photo.jpg'
#wip.route('/photo')
def photo():
return send_file(top_dir+mid_dir+more_dir+filename)
serves up the file!
Here is an answer to clarify how to use the send_file approach in an app setting with app.route and a url_for.
import os
from flask import Flask, url_for, send_file
app = Flask(__name__)
# set the path where you store your assets - it can be outside of the root dir
app.config['CUSTOM_STATIC_PATH'] = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../archive'))
# have a route listening for requests
#app.route('/this/can/be/anything/<path:filename>')
def send_media(filename):
path = os.path.join(app.config['CUSTOM_STATIC_PATH'], filename)
return send_file(path)
# in your template, you can then use the following
# <img src="{{ url_for('send_media', filename='<dir>/<filename>') }}">
I´ve been doing some research, and I have found articles explaining how to use Django's(1.8) cache busting, but they don´t explain my situation.
I am using S3 (and it works) with the following setting in settings.py:
STATICFILES_STORAGE= 'pms.s3utils.StaticRootS3BotoStorage'
In order to use cache busting the docs say I have to set: STATICFILES_STORAGE='django.contrib.staticfiles.storage.ManifestStaticFilesStorage'
I don´t know what setting to use in order to use both S3 and cache busting.
Thanks!
So I finally found a workaround.
In order to be able to upload my files to 2 different folders (static and uploads) in my S3 bucket I have this in my settings.py file:
STATICFILES_STORAGE = 'myapp.s3utils.StaticRootS3BotoStorage'
DEFAULT_FILE_STORAGE = 'myapp.s3utils.MediaRootS3BotoStorage'
And this in the myapp.s3utils.py file:
from storages.backends.s3boto import S3BotoStorage
StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='uploads')
But I couldn´t use Django´s cache busting.
The solution was to change my myapp.s3utils.py file to:
from storages.backends.s3boto import S3BotoStorage
from django.contrib.staticfiles.storage import ManifestFilesMixin
class CustomS3Storage(ManifestFilesMixin, S3BotoStorage):
pass
StaticRootS3BotoStorage = lambda: CustomS3Storage(location='static')
MediaRootS3BotoStorage = lambda: S3BotoStorage(location='uploads')
Basically it adds the ManiFestfilesMixin, which allows you to use cache busting.
As a side note, you can see that I am only using cache busting for the static files, but not for the uploads files. That is why the MediaRootS3BotoStorage calls the S3BotoStorage class instead of the CustomS3Storage.
I do it this way because the uploads files are not stored in my server (the static files are), they are stored directly in the S3 bucket, so when I ran the collectstatic they are not on my server, so I don´t have to add the hash to the names.
right now this is all I have:
import ftputil
a_host = ftputil.FTPHost("ftp_host", "username","pass") # login to ftp
for (dirname, subdirs, files) in a_host.walk("/"): # directory
for f in files:
fullpath = a_host.path.join(dirname, f)
if fullpath.endswith('html'):
#stucked
so I can log in to my ftp, and do a .walk in my files
the thing I am not able to manage is when the .walk finds a html file to also search in it for a string I want.
for example:
on my ftp - there is a index.html and a something.txt file
I want to find with .walk the index.html file, and then in index.html search for 'my string'
thanks
FTP is a protocol for file transfer only. It has not the ability by itself to execute remote commands which are needed to search the files on the remote server (there is a SITE command but it can usually not be used for such a purpose because it is not implemented or restricted to only a few commands).
This means your only option with FTP is to download the file and search it locally, i.e. transfer the file to the local system, open it there and look for the string.
I want to serve a HTML page from a subdirectory (not the 'static' one, via YAML, not the 'template' one, via Django or Jinja)
How can I reference the path + file name in the self.response.out.write(?) statement?
I am not sure I undersood, but I feel like you want to consider HTML files without template.
If so, you may want to try this solution...
You can define the path as follows:
path = os.path.join(os.path.dirname(__file__), 'your_directory/your_html.html')
Then, you can print the content of your HTML like this:
self.response.out.write(template.render(path, 0))