We Keep Coding

Storing files Django

I am building an website using django in which I have to store a lot of files.
I have a view, which takes file from the user and saves it.
def store(request):
if request.method == "POST":
sender = request.POST['sender']
file = request.POST['file']
I have a model in which i save these data:
class t_data(models.Model):
sender = models.TextField()
file = models.FileField(upload_to='files')
and I have this on my settings file:
MEDIA_ROOT = os.path.join(BASE_DIR,"media")
MEDIA_URL = '/media/'
Many files can have same name. Should I rename before saving them? How should I save them. There will be many users, So how can I distinguish files for each user?
my question is : **What is the best way to store files in a django application?**Please
Thank you

Try this solution. It creates a random string and adds it to the file name - How to Rename an Image or File in an Upload in Django

In Django how to convert an uploaded pdf file to an image file and save to the corresponding column in database?

I am creating an HTML template to show the cover of a pdf file(first page or user can choose one). I want Django to create the cover image automatically without extra upload.
The pdf file is uploaded using Django Modelform. Here is the structure of my code
models.py
class Pdffile(models.Model):
pdf = models.FileField(upload_to='pdfdirectory/')
filename = models.CharField(max_length=20)
pagenumforcover = models.IntegerField()
coverpage = models.FileField(upload_to='coverdirectory/')
form.py
class PdffileForm(ModelForm):
class Meta:
model = Pdffile
fields = (
'pdf',
'filename',
'pagenumforcover',
)
views.py
def upload(request):
if request.method == 'POST':
form = PdffileForm(request.POST, request.FILES)
if form.is_valid():
form.save()
return redirect('pdffilelist')
else:
form = PdffileForm()
return render(request, "uploadform.html", {'form': form})
def pdfcover(request, pk):
thispdf = get_object_or_404(Pdffile, pk=pk)
return render(request, 'pdfcover.html', {'thispdf': thispdf})
In the 'pdfcover.html', I want to use the Django template language so I can render different HTML for different uploaded pdf files. That's why I want to save the image file to the same column as the pdf file.
I am new to Python, new to Django, and obviously new to stack overflow. I have tried pdf2image and PyPDF2 and I believe they all could work however I just cannot find the right code. If you guys enlighten me I will be thankful.

In the pdf2image package there is a function called convert_from_path.
This is the description inside the package of what each of the parameters of the function does.
Parameters:
pdf_path -> Path to the PDF that you want to convert
dpi -> Image quality in DPI (default 200)
output_folder -> Write the resulting images to a folder (instead of directly in memory)
first_page -> First page to process
last_page -> Last page to process before stopping
fmt -> Output image format
jpegopt -> jpeg options `quality`, `progressive`, and `optimize` (only for jpeg format)
thread_count -> How many threads we are allowed to spawn for processing
userpw -> PDF's password
use_cropbox -> Use cropbox instead of mediabox
strict -> When a Syntax Error is thrown, it will be raised as an Exception
transparent -> Output with a transparent background instead of a white one.
single_file -> Uses the -singlefile option from pdftoppm/pdftocairo
output_file -> What is the output filename or generator
poppler_path -> Path to look for poppler binaries
grayscale -> Output grayscale image(s)
size -> Size of the resulting image(s), uses the Pillow (width, height) standard
paths_only -> Don't load image(s), return paths instead (requires output_folder)
use_pdftocairo -> Use pdftocairo instead of pdftoppm, may help performance
timeout -> Raise PDFPopplerTimeoutError after the given time
Because convert_from_path is designed to be able to turn every page in a pdf into an image the function returns an array of Image objects.
If you set the output_folder parameter each image will be saved to that location from the base directory. output_folder must be a full path in this case e.g. 'path/from/root/to/output_folder'. If you don't set it the images won't be saved when converted, only in memory.
By default if you do not set the output_file parameter the function will generate a random formatted filename such as 0a15a918-59ba-4f15-90f0-2ed5fbd0c36c-1.ext. Although if you do set a filename, because this filename is used for converting multiple pdf pages, if your output_file was 'file_name' then each file would be named starting from 'file_name0001-1.ext'.
Beware that if you set output_file and output_folder and try converting two different pdfs the second pdf will overwrite the image files of the first if they are in the same directory.
Here is some code modelled around yours in the question. This code assumes you have pdf2image installed.
I've added a built-in validator on the pdf FileField because else the code will crash if anything else but a pdf is uploaded.
validators=[FileExtensionValidator(allowed_extensions=['pdf'])]
I also created three constants for the upload directories and file format. If you need to change any of them then the rest of the code can remain the same.
COVER_PAGE_DIRECTORY = 'coverdirectory/'
PDF_DIRECTORY = 'pdfdirectory/'
COVER_PAGE_FORMAT = 'jpg'
Also I'm assuming you have the default settings setup for saving files.
settings.py
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
models.py
from django.core.validators import FileExtensionValidator
from django.db.models.signals import post_save
from pdf2image import convert_from_path
from django.conf import settings
import os
COVER_PAGE_DIRECTORY = 'coverdirectory/'
PDF_DIRECTORY = 'pdfdirectory/'
COVER_PAGE_FORMAT = 'jpg'
# this function is used to rename the pdf to the name specified by filename field
def set_pdf_file_name(instance, filename):
return os.path.join(PDF_DIRECTORY, '{}.pdf'.format(instance.filename))
# not used in this example
def set_cover_file_name(instance, filename):
return os.path.join(COVER_PAGE_DIRECTORY, '{}.{}'.format(instance.filename, COVER_PAGE_FORMAT))
class Pdffile(models.Model):
# validator checks file is pdf when form submitted
pdf = models.FileField(
upload_to=set_pdf_file_name,
validators=[FileExtensionValidator(allowed_extensions=['pdf'])]
)
filename = models.CharField(max_length=20)
pagenumforcover = models.IntegerField()
coverpage = models.FileField(upload_to=set_cover_file_name)
def convert_pdf_to_image(sender, instance, created, **kwargs):
if created:
# check if COVER_PAGE_DIRECTORY exists, create it if it doesn't
# have to do this because of setting coverpage attribute of instance programmatically
cover_page_dir = os.path.join(settings.MEDIA_ROOT, COVER_PAGE_DIRECTORY)
if not os.path.exists(cover_page_dir):
os.mkdir(cover_page_dir)
# convert page cover (in this case) to jpg and save
cover_page_image = convert_from_path(
pdf_path=instance.pdf.path,
dpi=200,
first_page=instance.pagenumforcover,
last_page=instance.pagenumforcover,
fmt=COVER_PAGE_FORMAT,
output_folder=cover_page_dir,
)[0]
# get name of pdf_file
pdf_filename, extension = os.path.splitext(os.path.basename(instance.pdf.name))
new_cover_page_path = '{}.{}'.format(os.path.join(cover_page_dir, pdf_filename), COVER_PAGE_FORMAT)
# rename the file that was saved to be the same as the pdf file
os.rename(cover_page_image.filename, new_cover_page_path)
# get the relative path to the cover page to store in model
new_cover_page_path_relative = '{}.{}'.format(os.path.join(COVER_PAGE_DIRECTORY, pdf_filename), COVER_PAGE_FORMAT)
instance.coverpage = new_cover_page_path_relative
# call save on the model instance to update database record
instance.save()
post_save.connect(convert_pdf_to_image, sender=Pdffile)
convert_pdf_to_image is a function that runs on the post_save signal of the Pdffile model. It gets run after your PdffileForm gets saved in your upload view so that we can create the cover image file from the saved pdf file.
cover_page_image = convert_from_path(
pdf_path=instance.pdf.path,
dpi=200,
first_page=instance.pagenumforcover,
last_page=instance.pagenumforcover,
fmt=COVER_PAGE_FORMAT,
output_folder=cover_page_dir,
)[0]
Changing dpi will change the quality of the image. In order to only convert one page the first_page and last_page parameters are the same. Because the result is an array we grab the first and only element in the list inside cover_page_image in this case.
Minor change to your upload view.
views.py
def upload(request):
form = PdffileForm()
if request.method == 'POST':
form = PdffileForm(request.POST, request.FILES)
# if form is not valid then form data will be sent back to view to show error message
if form.is_valid():
form.save()
return redirect('pdffilelist')
return render(request, "uploadform.html", {'form': form})
I don't know what your upload.html file looks like but I used the following which will work with the code provided.
upload.html
<h1>Upload PDF</h1>
<form method="POST" enctype="multipart/form-data">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Upload</button>
</form>
With an example pdf
Uploaded through the form
The resulting database record
The resulting file locations once uploaded
Final note:
Because FileFields have code to ensure that existing files don't get overwritten, The code
# get name of pdf_file
pdf_filename, extension = os.path.splitext(os.path.basename(instance.pdf.name))
new_cover_page_path = '{}.{}'.format(os.path.join(cover_page_dir, pdf_filename), COVER_PAGE_FORMAT)
# rename file to be the same as the pdf file
os.rename(cover_page_image.filename, new_cover_page_path)
# get the relative path to the cover page to store in model
new_cover_page_path_relative = '{}.{}'.format(os.path.join(COVER_PAGE_DIRECTORY, pdf_filename), COVER_PAGE_FORMAT)
instance.coverpage = new_cover_page_path_relative
ensures the pdf FileField filename is used to name the cover page because it is almost completely unique.

I used the explanation here, and everything works fine, except when from admin panel I oped the saved Pdffile object and try to change the pagenumforcover to another integer and then save it then it won't generate the new coverpage

How to auto-upload an image from django app to aws s3

When a user creates or registers for a new account on my website, an image is created(generated) and is supposed to be uploaded to the s3 bucket. The image is successfully created(verified by running the ls command on the server in the media directory) but it's not getting uploaded to s3. However, when I try uploading an image for a user account from the admin panel, changes are correctly reflected in s3 (i.e newly uploaded image from admin panel is shown in s3 bucket's directory, but this is not feasible as the users cannot be given admin panel access). I aim to auto-upload the generated image to the s3 bucket when a new account is created.
Here's some related code.
views.py
def signup(request):
if request.method == "POST":
base_form = UserForm(data=request.POST)
addnl_form = AddnlForm(data=request.POST)
if base_form.is_valid() and addnl_form.is_valid():
usrnm = base_form.cleaned_data['username']
if UserModel.objects.filter(user__username=usrnm).count()==0:
user = base_form.save()
user.set_password(user.password)
user.save()
#print(img)
addnl = addnl_form.save(commit=False )
addnl.user = user
img = qr.make_image() #create a qr code image, full code not included.
img.save('media/qrcodes/%s.png'%usrnm)
addnl.qr_gen = 'qrcodes/%s.png'%usrnm
addnl.save()
else:
messages.error(request,base_form.errors,addnl_form.errors)
else:
base_form = UserForm()
addnl_form = AddnlForm()
return render(request,'app/signup.html',{'base_form':base_form,'addnl_form':addnl_form} )
models.py
class UserModel(models.Model):
.
.
.
qr_gen = models.ImageField(upload_to='qrcodes',default=None,null=True,blank=True)
settings.py
DEFAULT_FILE_STORAGE = 'project.storage_backend.MediaStorage'
storage_backend.py
from storages.backends.s3boto3 import S3Boto3Storage
class MediaStorage(S3Boto3Storage):
location = 'media'
default_acl = 'public-read'
file_overwrite = False
UPDATE
Instead of auto-generating, an image and uploading it to s3, if I upload any image in the registration form, even in that case it's successfully uploading to s3, the only case where it fails is when I need to auto-upload without user intervention.
Please help me solve this problem. Thank you.

I would recommend taking a look at django-storages which automates all of this so you should only worry about the form and the view instead of anything else. In there you will find help on how to deal with images easily.

Instead of globally setting media storage settng, set it on the field
class UserModel(models.Model):
...
qr_gen = models.ImageField(upload_to='qrcodes',storage=MediaStorage())

Auto uploading an image directly to s3 requires one to directly communicate with S3's backend API. Plainly using django-storages or tweaking DEFAULT_FILE_STORAGE path isn't just enough as it only helps to point user-uploaded files to the specified s3 bucket/path. This problem can be tackled using the boto3 library's upload_file method.
Usage example:
import boto3
s3 = boto3.resource('s3')
s3.Bucket('mybucket').upload_file('/tmp/hello.txt', 'hello.txt')
Params:
Filename (str) -- The path to the file to upload.
Key (str) -- The name of the key to upload to.
ExtraArgs (dict) -- Extra arguments that may be passed to the client operation.
Callback (function) -- A method which takes a number of bytes transferred to be periodically called during the upload.

Wrong media location using Django management command

I'm downloading an image and saving it to a models image field, on a separate media app. Everything was working fine while the code was in a view but soon as I moved the code into a management command I can't get the image to save into the separate media location. Before it was correctly saving to /home/me/webapps/myapp_production_media/images now the files are being saved incorrectly to /home/me/webapps/myapp_production_django/src/media/images
command:
download_image('temp', model_instance.image, new_image)
def download_image(name, image, url):
input_file = StringIO(urllib2.urlopen(url).read())
output_file = StringIO()
img = Image.open(input_file)
if img.mode != "RGB":
img = img.convert("RGB")
img.save(output_file, "JPEG")
image.save(name+".jpg", ContentFile(output_file.getvalue()), save=False)
model:
class MyModel(models.Model):
image = models.ImageField(upload_to='images', null=True, blank=True)
I've added myapp_production_media/images path to the name+".jpg" but it doesn't change anything. I'm guessing this is happening because the image field in the model doesn't access the media_root in settings.py (as the code is a management command)?

Double-check that you are using the same settings file when running the management command. Production settings are usually not configured as default and in these cases need to be given on command line.

I suggest doing the following at the beginning of the command file.
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")
where app.setting is where the setting.py resides.

Flask and sqlalchemy: Get uploaded file using path stored on database

I'm using flask and sqlalchemy and I'm having issues regarding file uploading.
I'm uploading the file with a form and storing the URL to the DB, so I can display it inside on the page.
Here's the form's code:
boxart = request.files['boxart']
if boxart:
filename = secure_filename(boxart.filename)
boxart_path = os.path.join(app.config['UPLOAD_FOLDER']+"boxart/", filename)
boxart.save(boxart_path)
game_son.boxart = boxart_path
db.session.add(game_son)
db.session.commit()
So I go to my template and I can print the content of game_son.boxart and there is the full path to the file. If I click on the link, I can access, but the image will not display inside tag...
I tried getting it on the template by using:
<img src="{{ game_son.boxart_url() }}" />
Inside the model, I defined a boxart_url method, in case I need to parse the string before sending it
class Game_son(db.Model):
__searchable__ = ['version_name']
son_id = db.Column(db.Integer, primary_key=True)
version_name = db.Column(db.String(128), index=True, unique=False)
son_timestamp = db.Column(db.DateTime)
dad = db.Column(db.Integer, db.ForeignKey('game_dad.id'))
console = db.Column(db.Integer, db.ForeignKey('platform.id'))
boxart = db.Column(db.String(256))
thumbnail = db.Column(db.String(256))
genre = db.Column(db.String(32))
subgenre = db.Column(db.String(32))
def boxart_url(self):
return self.boxart
def __repr__(self):
return '<Game %r>: %r' % (self.version_name, self.dad)
The page's view just send the entire object (game) to the page.
The url that is arriving on the page is:
/home/removed_my_pc_user/app/media/boxart/image.jpg
Edit:
I just remembered that I'm using a virtual environment on the project.
Is it related to chmod?
Thanks in advance!!

Well, It's strange but I will answer my own question.
The solution is simple, I chose to store only the file names inside the database.
Then I created a route to a view that will return the file using the send_from_directory function from flask.ext.uploads
#app.route('/boxart/<filename>')
def uploaded_boxart(filename):
return send_from_directory(app.config['UPLOAD_FOLDER'],filename)
So, inside the template we can use the url_for() function to load the file, sending as a parameter the filename stored inside the database.
<img src="{{ url_for('uploaded_boxart', filename = game_son.boxart)}}" />
Also, I think that one of my mistakes was putting the media/ folder inside the app/ folder, and not inside the root directory. So I also reconfigured the UPLOAD_FOLDER in app.config.