I'm trying to save a user uploaded file directly to S3 without saving it locally. This project is using Django 1.9 and Boto3.
The relevant code is:
p=request.FILES['img'].read()
s3=boto3.resource('s3',
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
b = s3.Bucket(settings.AWS_STORAGE_BUCKET_NAME)
b.put_object(Key="media/test.jpg", Body=p)
This correctly uploads a file called 'test.jpg' to the media folder.
However, if I download 'test.jpg' from Amazon and try to open it in an image viewer, I get the message: "Error interpreting JPEG image file (Not a JPEG file: starts with 0xf0 0xef)". The jpg file is also only 26kb whereas the original was 116kb.
What is going wrong? I assume I am passing the wrong data as Body in the put_object method. But what should p be instead?
Update and Solutions
With JordonPhilips's help, I realised that because I had already opened the uploaded image earlier in the view with Pillow, the request.FILES['img'] socket had already been read.
The solution I went with was to remove the Pillow code, leaving the boto upload as the first access of request.FILES['img'].
However, I also figured out a solution if you want to do something to the image first (e.g. in Pillow):
from Pillow import Image
import cStringIO as StringIO
import boto3
and then in the view function:
im = Image.open(request.FILES['img'])
# whatever image analysis here
file2 = StringIO.StringIO()
im.save(file2,"jpeg",quality='keep')
s3 = boto3.resource( 's3', aws_access_key_id=settings. AWS_ACCESS_KEY_ID, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
b = s3.Bucket(settings.AWS_STORAGE_BUCKET_NAME)
b.put_object(Key="media/test.jpg", Body=file2.getvalue())
It looks like your problem was that you were trying to read the socket multiple times. You can only read the socket once, so you need to keep reference to important information.
Related
I'm a beginner in python and I'm trying to send someone my small python program together with a picture that'll display when the code is run.
I tried to first convert the image to a binary file thinking that I'd be able to paste it in the source code but I'm not sure if that's even possible as I failed to successfully do it.
You can base64-encode your JPEG/PNG image which will make it into a regular (non-binary string) like this:
base64 -w0 IMAGE.JPG
Then you want to get the result into a Python variable, so repeat the command but copy the output to your clipboard:
base64 -w0 IMAGE.JPG | xclip -selection clipboard # Linux
base64 -w0 IMAGE.JPG | pbcopy # macOS
Now start Python and make a variable called img and paste the clipboard into it:
img = 'PASTE'
It will look like this:
img = '/9j/4AAQSk...' # if your image was JPEG
img = 'iVBORw0KGg...' # if your image was PNG
Now do some imports:
from PIL import Image
import base64
import io
# Make PIL Image from base64 string
pilImage = Image.open(io.BytesIO(base64.b64decode(img)))
Now you can do what you like with your image:
# Print its description and size
print(pilImage)
<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=200x100>
# Save it to local disk
pilImage.save('result.jpg')
You can save a picture in byte format inside a variable in your program. You can then convert the bytes back into a file-like object using the BytesIO function of the io module and plot that object using the Image module from the Pillow library.
import io
import PIL.Image
with open("filename.png", "rb") as file:
img_binary = file.read()
img = PIL.Image.open(io.BytesIO(img_binary))
img.show()
To save the binary data inside your program without having to read from the source file you need to encode it with something like base64, use print() and then simply copy the output into a new variable and remove the file reading operation from your code.
That would look like this:
img_encoded = base64.encodebytes(img_binary)
print(img_binary)
img_encoded = " " # paste the output from the console into the variable
the output will be very long, especially if you are using a big image. I only used a very small png for testing.
This is how the program should look like at the end:
import io
import base64
import PIL.Image
# with open("filename.png", "rb") as file:
# img_binary = file.read()
# img_encoded = base64.encodebytes(img_binary)
img_encoded = b'iVBORw0KGgoAAAANSUhEUgAAADAAAAAwCAYAAABX[...]'
img = PIL.Image.open(io.BytesIO(base64.decodebytes(img_encoded)))
img.show()
You could perhaps have your Python program download the image from a site where you upload files such as Google Drive, Mega, or Imgur. That way, you can always access and view the image easily without the need of running the program or for example converting the binary back into the image in the method you mentioned.
Otherwise, you could always store the image as bytes in a variable and have your program read this variable. I'm assuming that you really wish to do it this way as it would be easier to distribute as there is only one file that needs to be downloaded and run.
Or you could take a look at pyinstaller which is made for python programs to be easily distributed across machines without the need to install Python by packaging it as an executable (.exe) file! That way you can include the image file together by embedding it into the program. There are plenty of tutorials for pyinstaller you could google up. Note: Include the '--onefile' in your parameters when running pyinstaller as this will package the executable into a single file that the person you're sending it to can easily open whoever it may be-- granted the executable file can run on the user's operating system. :)
I've been trying to send an image to firebase storage but when it gets to the storage, firebase can't render the image.
The image for now is pure base64.
versions:
Python 3.10.6
firebase==3.0.1
firebase-admin==6.0.1
Flask==2.0.3
dontpad.com for the base64 being used
Code:
def filePath(folderPath):
return f'{folderPath}/{date.today()}'
def fileUpload(file,folderPath):
fileName = filePath(folderPath)
from firebase_admin import storage
bucket = storage.bucket()
blob = bucket.blob(fileName)
blob.upload_from_string(file,'image/jpg' )
blob.make_public()
return blob.public_url
Additional info if needed will be provided when asked.
Expected:
Result:
What did I try?
Alternative data objects to replace base64 has been studied in the project but base64 is the only data I'm provided for the image so alternative ways have been discarded.
Most similar questions have used JavaScript, that's not my case, and they use different libraries with different methods and parameters so that hasn't helped my case.
Tried adding "data:image/jpeg;base64," to the start of the filename.
Tried replacing content type with "data_url" or "base64".
Tried uploading with and without the extension on the filename.
After much searching I finally found it. Here's the response.
Step 1
code for the user to provide the link and for PIL to resize it according to some conditions (not relevant to the question).
PIL opens the image file link like this:
img=Image.open(io.BytesIO(requests.get(url).content))
where url is the link.
Step2
then PIL must save it AND seek(0) on the io.BytesIO class:
blob = io.BytesIO()
img.save(blob, 'JPEG')
blob.seek(0)
response = twitter.upload_media(media=blob);
Step 3
Proceed according to the documendation:
twitter.update_status(status='Checkout this cool image!', media_ids=[response['media_id']])
Twython's documendation is outdated and StringIO() has changed package and also doesn't take bytes objects. I also can't get the logic behind making a bytes object to a string and then send it as a bytes object.
https://twython.readthedocs.io/en/latest/usage/advanced_usage.html
The media parameter also takes io.BytesIO class objects as shown above and in this simpler example below:
response = twitter.upload_media(media=io.BytesIO(requests.get(url).content));
I'm currently having some issues removing a file in python. I am creating a temporary file for pdf to image conversion. It is housed in a folder that holds a .ppm file and converts it to a .jpg file. It then deletes the temporary .ppm file. Here is the code:
import pdf2image
from PIL import Image
import os
images = pdf2image.convert_from_path('Path to pdf.pdf', output_folder='./folder name')
file = ''
for files in os.listdir('./folder name'):
if files.endswith(".ppm"):
file = files
path = os.path.join('folder name',file)
im = Image.open(path)
im.save("Path to image.jpg")
im.close()
os.remove(path)
The issue is at the end in the os.remove(path). I get the following error:
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'path to ppm file'
I would appreciate any help and thanks in advance.
Not really the answer to your question, but you can just output in the correct format at the start, and avoid the issue in the first place:
pdf2image.convert_from_path('Path to pdf.pdf', output_folder='./folder name', fmt='jpg')
To actually answer your question, I'm not sure why you're having the issue, because really the close() should prevent this problem. Perhaps check out this answer and try using a with statement? Or maybe the permissions release is just delayed, I'm curious what throwing that remove in a loop for as long as it throws an exception would do.
Edit: To set the name, you'll want to do something like:
images = pdf2image.convert_from_path('Path to pdf.pdf', output_folder='./folder name', fmt='jpg')
for image in images:
# Save the image
The pdf2image documentation looks like it recommends using a temporary folder, like in this example, and then you can just .save(...) the PIL image:
import tempfile
with tempfile.TemporaryDirectory() as path:
images_from_path = convert_from_path('/home/kankroc/example.pdf', output_folder=path)
# Do something here
Edit: I realized that the reason it was in use is probably because you need to close() all the images in images. You should read up on the pdf2image documentation and about the PIL images that it spits out for more details.
I've been looking for a way to download an image from a URL, preform some image manipulations (resize) actions on it, and then save it to a django ImageField. Using the two great posts (linked below), I have been able to download and save an image to an ImageField. However, I've been having some trouble manipulating the file once I have it.
Specifically, the model field save() method requires a File() object as the second parameter. So my data has to eventually be a File() object. The blog posts linked below show how to use urllib2 to save your an image URL into a File() object. This is great, however, I also want to manipulate the image using PIL as an Image() object. (or ImageFile object).
My preferred approach would be then to load the image URL directly into an Image() object, preform the resize, and convert it to a File() object and then save it in the model. However, my attempts to convert an Image() to a File() have failed. If at all possible, I want to limit the number of times I write to the disk, so I'd like to do this object transformation in Memory or using a NamedTemporaryFile(delete=True) object so I don't have to worry about extra files laying around. (Of course, I want the file to be written to disk once it is saved via the model).
import urllib2
from PIL import Image, ImageFile
from django.core.files import File
from django.core.files.temp import NamedTemporaryFile
inStream = urllib2.urlopen('http://www.google.com/intl/en_ALL/images/srpr/logo1w.png')
parser = ImageFile.Parser()
while True:
s = inStream.read(1024)
if not s:
break
parser.feed(s)
inImage = parser.close()
# convert to RGB to avoid error with png and tiffs
if inImage.mode != "RGB":
inImage = inImage.convert("RGB")
# resize could occur here
# START OF CODE THAT DOES NOT SEEM TO WORK
# I need to somehow convert an image .....
img_temp = NamedTemporaryFile(delete=True)
img_temp.write(inImage.tostring())
img_temp.flush()
file_object = File(img_temp)
# .... into a file that the Django object will accept.
# END OF CODE THAT DOES NOT SEEM TO WORK
my_model_instance.image.save(
'some_filename',
file_object, # this must be a File() object
save=True,
)
With this approach, the file appears corrupt whenever I view it as an image. Does anyone have any approach that takes a file file from a URL, allows one to manipulate it as an Image and then save it to a Django ImageField?
Any help is much appreciated.
Programmatically saving image to Django ImageField
Django: add image in an ImageField from image url
Update 08/11/2010: I did end up going with StringIO, however, I was stringIO was throwing an unusual Exception when I tried to save it in a Django ImageField. Specifically, the stack trace showed a name error:
"AttribueError exception "StringIO instance has no attribute 'name'"
After digging through the Django source, it looks like this error was caused when the model save tries to access the size attribute of the StringIO "File". (Though the error above indicates a problem with the name, the root cause of this error appears to be the lack of a size property on the StringIO image). As soon as I assigned a value to the size attribute of the image file, it worked fine.
In an attempt to kill 2 birds with 1 stone. Why not use a (c)StringIO object instead of a NamedTemporaryFile? You won't have to store it on disk anymore and I know for a fact that something like this works (I use similar code myself).
from cStringIO import StringIO
img_temp = StringIO()
inImage.save(img_temp, 'PNG')
img_temp.seek(0)
file_object = File(img_temp, filename)