I'm downloading an image and saving it to a models image field, on a separate media app. Everything was working fine while the code was in a view but soon as I moved the code into a management command I can't get the image to save into the separate media location. Before it was correctly saving to /home/me/webapps/myapp_production_media/images now the files are being saved incorrectly to /home/me/webapps/myapp_production_django/src/media/images
command:
download_image('temp', model_instance.image, new_image)
def download_image(name, image, url):
input_file = StringIO(urllib2.urlopen(url).read())
output_file = StringIO()
img = Image.open(input_file)
if img.mode != "RGB":
img = img.convert("RGB")
img.save(output_file, "JPEG")
image.save(name+".jpg", ContentFile(output_file.getvalue()), save=False)
model:
class MyModel(models.Model):
image = models.ImageField(upload_to='images', null=True, blank=True)
I've added myapp_production_media/images path to the name+".jpg" but it doesn't change anything. I'm guessing this is happening because the image field in the model doesn't access the media_root in settings.py (as the code is a management command)?
Double-check that you are using the same settings file when running the management command. Production settings are usually not configured as default and in these cases need to be given on command line.
I suggest doing the following at the beginning of the command file.
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "app.settings")
where app.setting is where the setting.py resides.
Related
This question already has answers here:
Save uploaded image to S3 with Django
(2 answers)
Closed 1 year ago.
My problem is that images stored in media folder are not transferring to S3 Bucket. I tested with other file from request and the file did transfer, so I assume settings.py must be OK.
From views.py ->
This works:
if request.method == 'POST':
imageFile = request.FILES['images']
upload = Upload(file=imageFile)
upload.save()
image_url = upload.file.url
print(image_url)
This does not work:
for i in os.listdir(folder):
f = os.path.join(conf_settings.MEDIA_ROOT,company, i)
upload = Upload(file=f)
upload.save()
No error but it just does not work.
This also does not work:
for i in os.listdir(folder):
with open(os.path.join(folder, i)) as f:
upload = Upload(file=f)
upload.save()
>The error I am getting is:
>
>Exception Value:
>'_io.TextIOWrapper' object has no attribute '_committed'
>
>at upload.save()
This is my storage_backend.py
from django.conf import settings
from storages.backends.s3boto3 import S3Boto3Storage
class MediaStorage(S3Boto3Storage):
location = 'media'
default_acl = 'public-read'
file_overwrite = True
This is my model.py
class Upload(models.Model):
uploaded_at = models.DateTimeField(auto_now_add=True)
file = models.FileField()
I am uploading a .ZIP file with images. Unzipping it and saving them to media folder, then I want to upload from media folder to S3 Bucket. This operation fails.
The file in request.FILES is the Zip file, which I am using to test that all settings.py for AWS should be correct because it does transfer correctly.
I believe my issue has to do with the way I am reading the file and passing it.
So after many hours....this actually worked. Although the transfer is a bit slow, im sure there must be a better way.
https://stackoverflow.com/a/53260957/11116189
I am creating an HTML template to show the cover of a pdf file(first page or user can choose one). I want Django to create the cover image automatically without extra upload.
The pdf file is uploaded using Django Modelform. Here is the structure of my code
models.py
class Pdffile(models.Model):
pdf = models.FileField(upload_to='pdfdirectory/')
filename = models.CharField(max_length=20)
pagenumforcover = models.IntegerField()
coverpage = models.FileField(upload_to='coverdirectory/')
form.py
class PdffileForm(ModelForm):
class Meta:
model = Pdffile
fields = (
'pdf',
'filename',
'pagenumforcover',
)
views.py
def upload(request):
if request.method == 'POST':
form = PdffileForm(request.POST, request.FILES)
if form.is_valid():
form.save()
return redirect('pdffilelist')
else:
form = PdffileForm()
return render(request, "uploadform.html", {'form': form})
def pdfcover(request, pk):
thispdf = get_object_or_404(Pdffile, pk=pk)
return render(request, 'pdfcover.html', {'thispdf': thispdf})
In the 'pdfcover.html', I want to use the Django template language so I can render different HTML for different uploaded pdf files. That's why I want to save the image file to the same column as the pdf file.
I am new to Python, new to Django, and obviously new to stack overflow. I have tried pdf2image and PyPDF2 and I believe they all could work however I just cannot find the right code. If you guys enlighten me I will be thankful.
In the pdf2image package there is a function called convert_from_path.
This is the description inside the package of what each of the parameters of the function does.
Parameters:
pdf_path -> Path to the PDF that you want to convert
dpi -> Image quality in DPI (default 200)
output_folder -> Write the resulting images to a folder (instead of directly in memory)
first_page -> First page to process
last_page -> Last page to process before stopping
fmt -> Output image format
jpegopt -> jpeg options `quality`, `progressive`, and `optimize` (only for jpeg format)
thread_count -> How many threads we are allowed to spawn for processing
userpw -> PDF's password
use_cropbox -> Use cropbox instead of mediabox
strict -> When a Syntax Error is thrown, it will be raised as an Exception
transparent -> Output with a transparent background instead of a white one.
single_file -> Uses the -singlefile option from pdftoppm/pdftocairo
output_file -> What is the output filename or generator
poppler_path -> Path to look for poppler binaries
grayscale -> Output grayscale image(s)
size -> Size of the resulting image(s), uses the Pillow (width, height) standard
paths_only -> Don't load image(s), return paths instead (requires output_folder)
use_pdftocairo -> Use pdftocairo instead of pdftoppm, may help performance
timeout -> Raise PDFPopplerTimeoutError after the given time
Because convert_from_path is designed to be able to turn every page in a pdf into an image the function returns an array of Image objects.
If you set the output_folder parameter each image will be saved to that location from the base directory. output_folder must be a full path in this case e.g. 'path/from/root/to/output_folder'. If you don't set it the images won't be saved when converted, only in memory.
By default if you do not set the output_file parameter the function will generate a random formatted filename such as 0a15a918-59ba-4f15-90f0-2ed5fbd0c36c-1.ext. Although if you do set a filename, because this filename is used for converting multiple pdf pages, if your output_file was 'file_name' then each file would be named starting from 'file_name0001-1.ext'.
Beware that if you set output_file and output_folder and try converting two different pdfs the second pdf will overwrite the image files of the first if they are in the same directory.
Here is some code modelled around yours in the question. This code assumes you have pdf2image installed.
I've added a built-in validator on the pdf FileField because else the code will crash if anything else but a pdf is uploaded.
validators=[FileExtensionValidator(allowed_extensions=['pdf'])]
I also created three constants for the upload directories and file format. If you need to change any of them then the rest of the code can remain the same.
COVER_PAGE_DIRECTORY = 'coverdirectory/'
PDF_DIRECTORY = 'pdfdirectory/'
COVER_PAGE_FORMAT = 'jpg'
Also I'm assuming you have the default settings setup for saving files.
settings.py
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
models.py
from django.core.validators import FileExtensionValidator
from django.db.models.signals import post_save
from pdf2image import convert_from_path
from django.conf import settings
import os
COVER_PAGE_DIRECTORY = 'coverdirectory/'
PDF_DIRECTORY = 'pdfdirectory/'
COVER_PAGE_FORMAT = 'jpg'
# this function is used to rename the pdf to the name specified by filename field
def set_pdf_file_name(instance, filename):
return os.path.join(PDF_DIRECTORY, '{}.pdf'.format(instance.filename))
# not used in this example
def set_cover_file_name(instance, filename):
return os.path.join(COVER_PAGE_DIRECTORY, '{}.{}'.format(instance.filename, COVER_PAGE_FORMAT))
class Pdffile(models.Model):
# validator checks file is pdf when form submitted
pdf = models.FileField(
upload_to=set_pdf_file_name,
validators=[FileExtensionValidator(allowed_extensions=['pdf'])]
)
filename = models.CharField(max_length=20)
pagenumforcover = models.IntegerField()
coverpage = models.FileField(upload_to=set_cover_file_name)
def convert_pdf_to_image(sender, instance, created, **kwargs):
if created:
# check if COVER_PAGE_DIRECTORY exists, create it if it doesn't
# have to do this because of setting coverpage attribute of instance programmatically
cover_page_dir = os.path.join(settings.MEDIA_ROOT, COVER_PAGE_DIRECTORY)
if not os.path.exists(cover_page_dir):
os.mkdir(cover_page_dir)
# convert page cover (in this case) to jpg and save
cover_page_image = convert_from_path(
pdf_path=instance.pdf.path,
dpi=200,
first_page=instance.pagenumforcover,
last_page=instance.pagenumforcover,
fmt=COVER_PAGE_FORMAT,
output_folder=cover_page_dir,
)[0]
# get name of pdf_file
pdf_filename, extension = os.path.splitext(os.path.basename(instance.pdf.name))
new_cover_page_path = '{}.{}'.format(os.path.join(cover_page_dir, pdf_filename), COVER_PAGE_FORMAT)
# rename the file that was saved to be the same as the pdf file
os.rename(cover_page_image.filename, new_cover_page_path)
# get the relative path to the cover page to store in model
new_cover_page_path_relative = '{}.{}'.format(os.path.join(COVER_PAGE_DIRECTORY, pdf_filename), COVER_PAGE_FORMAT)
instance.coverpage = new_cover_page_path_relative
# call save on the model instance to update database record
instance.save()
post_save.connect(convert_pdf_to_image, sender=Pdffile)
convert_pdf_to_image is a function that runs on the post_save signal of the Pdffile model. It gets run after your PdffileForm gets saved in your upload view so that we can create the cover image file from the saved pdf file.
cover_page_image = convert_from_path(
pdf_path=instance.pdf.path,
dpi=200,
first_page=instance.pagenumforcover,
last_page=instance.pagenumforcover,
fmt=COVER_PAGE_FORMAT,
output_folder=cover_page_dir,
)[0]
Changing dpi will change the quality of the image. In order to only convert one page the first_page and last_page parameters are the same. Because the result is an array we grab the first and only element in the list inside cover_page_image in this case.
Minor change to your upload view.
views.py
def upload(request):
form = PdffileForm()
if request.method == 'POST':
form = PdffileForm(request.POST, request.FILES)
# if form is not valid then form data will be sent back to view to show error message
if form.is_valid():
form.save()
return redirect('pdffilelist')
return render(request, "uploadform.html", {'form': form})
I don't know what your upload.html file looks like but I used the following which will work with the code provided.
upload.html
<h1>Upload PDF</h1>
<form method="POST" enctype="multipart/form-data">
{% csrf_token %}
{{ form.as_p }}
<button type="submit">Upload</button>
</form>
With an example pdf
Uploaded through the form
The resulting database record
The resulting file locations once uploaded
Final note:
Because FileFields have code to ensure that existing files don't get overwritten, The code
# get name of pdf_file
pdf_filename, extension = os.path.splitext(os.path.basename(instance.pdf.name))
new_cover_page_path = '{}.{}'.format(os.path.join(cover_page_dir, pdf_filename), COVER_PAGE_FORMAT)
# rename file to be the same as the pdf file
os.rename(cover_page_image.filename, new_cover_page_path)
# get the relative path to the cover page to store in model
new_cover_page_path_relative = '{}.{}'.format(os.path.join(COVER_PAGE_DIRECTORY, pdf_filename), COVER_PAGE_FORMAT)
instance.coverpage = new_cover_page_path_relative
ensures the pdf FileField filename is used to name the cover page because it is almost completely unique.
I used the explanation here, and everything works fine, except when from admin panel I oped the saved Pdffile object and try to change the pagenumforcover to another integer and then save it then it won't generate the new coverpage
When a user creates or registers for a new account on my website, an image is created(generated) and is supposed to be uploaded to the s3 bucket. The image is successfully created(verified by running the ls command on the server in the media directory) but it's not getting uploaded to s3. However, when I try uploading an image for a user account from the admin panel, changes are correctly reflected in s3 (i.e newly uploaded image from admin panel is shown in s3 bucket's directory, but this is not feasible as the users cannot be given admin panel access). I aim to auto-upload the generated image to the s3 bucket when a new account is created.
Here's some related code.
views.py
def signup(request):
if request.method == "POST":
base_form = UserForm(data=request.POST)
addnl_form = AddnlForm(data=request.POST)
if base_form.is_valid() and addnl_form.is_valid():
usrnm = base_form.cleaned_data['username']
if UserModel.objects.filter(user__username=usrnm).count()==0:
user = base_form.save()
user.set_password(user.password)
user.save()
#print(img)
addnl = addnl_form.save(commit=False )
addnl.user = user
img = qr.make_image() #create a qr code image, full code not included.
img.save('media/qrcodes/%s.png'%usrnm)
addnl.qr_gen = 'qrcodes/%s.png'%usrnm
addnl.save()
else:
messages.error(request,base_form.errors,addnl_form.errors)
else:
base_form = UserForm()
addnl_form = AddnlForm()
return render(request,'app/signup.html',{'base_form':base_form,'addnl_form':addnl_form} )
models.py
class UserModel(models.Model):
.
.
.
qr_gen = models.ImageField(upload_to='qrcodes',default=None,null=True,blank=True)
settings.py
DEFAULT_FILE_STORAGE = 'project.storage_backend.MediaStorage'
storage_backend.py
from storages.backends.s3boto3 import S3Boto3Storage
class MediaStorage(S3Boto3Storage):
location = 'media'
default_acl = 'public-read'
file_overwrite = False
UPDATE
Instead of auto-generating, an image and uploading it to s3, if I upload any image in the registration form, even in that case it's successfully uploading to s3, the only case where it fails is when I need to auto-upload without user intervention.
Please help me solve this problem. Thank you.
I would recommend taking a look at django-storages which automates all of this so you should only worry about the form and the view instead of anything else. In there you will find help on how to deal with images easily.
Instead of globally setting media storage settng, set it on the field
class UserModel(models.Model):
...
qr_gen = models.ImageField(upload_to='qrcodes',storage=MediaStorage())
Auto uploading an image directly to s3 requires one to directly communicate with S3's backend API. Plainly using django-storages or tweaking DEFAULT_FILE_STORAGE path isn't just enough as it only helps to point user-uploaded files to the specified s3 bucket/path. This problem can be tackled using the boto3 library's upload_file method.
Usage example:
import boto3
s3 = boto3.resource('s3')
s3.Bucket('mybucket').upload_file('/tmp/hello.txt', 'hello.txt')
Params:
Filename (str) -- The path to the file to upload.
Key (str) -- The name of the key to upload to.
ExtraArgs (dict) -- Extra arguments that may be passed to the client operation.
Callback (function) -- A method which takes a number of bytes transferred to be periodically called during the upload.
I'm trying to get files uploaded through the Django admin site to be placed on a network drive. Lets say the path to this drive is '\\FILESERVER\Django'.
My initial thought was to just set my media root to the same path I'd use to access the drive via File Explorer:
#settings.py
MEDIA_ROOT = r'\\FILESERVER\Django'
An Example Model:
#models.py
class Article(models.Model):
title = models.CharField(max_length=128)
pdf = models.FileField(upload_to='articles', blank=True, null=True)
def __str__(self):
return self.title
But when I upload a file, It just creates the folder on my local C: drive.
(e.g. C:\FILESERVER\Django\articles). Is there a way I can tell Django that this is supposed to be a path to a network drive?
Note: This is a Django 2.0.4 app running on a windows machine.
I believe what you're looking for is in pathlib. Have a look here:
https://docs.python.org/3/library/pathlib.html#methods-and-properties
Something like PureWindowsPath('//FILESERVER/Django').drive should do the trick for MEDIA_ROOT, IIRC.
If you use a network file server,you may need a customized django file storage class.
At the moment I'm on my way using the django-imagekit to show thumbnails in an image-heavy view of a gallery app. For this purpose I'm using the 'model-method', means I'm creating the thumbnails within the model.
Now with the comfort of the admin in mind (upload multiple picture at once) I also want to implement a multi-upload form in the admin-view. To ease things a little bit I tried to use an app I found on GitHub, django-admin-multiupload (I'm not able to link to it because of my low reputations but that's the exact name for it on GitHub).
When I only use django-imagekit, everything works fine and I get nice thumbnails, just like expected. When I only use django-admin-multiupload, everything works fine and I'm able to upload multiple images just like expected.
The problem starts when I'm using both apps at the same time. The multiupload works still fine but when I'm opening the view, and actually implementing the thumbnail (only using both and not implementing the thumbnail works fine), where the thumbnails should be shown I will get the following error:
OSError at /gallery/ - decoder jpeg not available
You can see the full error here: http://pastebin.com/gtVYEeG7
My confusion starts when starting only the single app and it works. So as far as my knowledge goes, all PIL issues could not be present.
To provide some more information: I'm using a virtualENV with the following list of packages:
pip
django
PIL
pilkit
django-imagekit
django-amdin-multiupload
To also provide some of my implementet code, here it is:
File: models.py
class Image(models.Model):
"""the model for the images"""
# the foreign key from the event
event = models.ForeignKey('Event', related_name='images',
blank=True, null=True)
# the image itself
# file = models.FileField('File', upload_to='gallery/images/')
file = models.ImageField('File', upload_to='gallery/images/')
image_thumbnail = ImageSpecField(source='file',
processors=[
ResizeToFill(300, 250)
],
format='JPEG',
options={'quality': 40})
# image title to represent it in the admin interface
image_name = models.CharField(max_length=35, default='img')
# publication date of the image
pub_date = models.DateTimeField('date published',
auto_now_add=True)
# for a better representation of the image
def __str__(self):
return self.image_name
File: admin.py
(this one is mostly as suggested in the example from the django-admin-multiupload repo, can be viewed here: https://github.com/gkuhn1/django-admin-multiupload/blob/master/example/gallery/admin.py)
from django.contrib import admin
from django.shortcuts import get_object_or_404
from gallery.models import Event, Image
from multiupload.admin import MultiUploadAdmin
# Register your models here.
# allows inline add of single images
class ImageInlineAdmin(admin.TabularInline):
model = Image
# used to define the process_uploaded_file function
# so it will not be duplicated in the Admin-Classes
class GalleryMultiuploadMixing(object):
def process_uploaded_file(self, uploaded, event, request):
image = event.images.create(file=uploaded)
return {
'url': image.file.url,
'thumbnail': image.file.url,
'id': image.id,
'name': image.image_name
}
# admin class for event model
class EventAdmin(GalleryMultiuploadMixing, MultiUploadAdmin):
inlines = [ImageInlineAdmin,]
multiupload_form = True
multiupload_list = False
def delete_file(self, pk, request):
obj = get_object_or_404(Image, pk=pk)
return obj.delete()
admin.site.register(Event, EventAdmin)
# admin class for image model
class ImageAdmin(GalleryMultiuploadMixing, MultiUploadAdmin):
multiupload_form = False
multiupload_list = True
admin.site.register(Image, ImageAdmin)
File: index.html
<td><img class="img-responsive" src="{{ image.image_thumbnail.url }}" /></td>
If there is any need for additional information please don't hesitate to ask.
Thank you in advance and I appreciate any help.
Edit: Today I tried another way and recognized that the error is only appearing to images that were uploaded with the django-admin-multiupload and not if only images are shown that were uploaded with the normal method. Maybe this could help to find a solution.
This error was mainly caused by a broken database that could be fixed by going back to an older version and reimplementing the new code. So there is no problem in django-admin-multiupload or django-imagekit.