We Keep Coding

Remove EXIF from Image Before Upload to S3 in Python

I want to remove exif from an image before uploading to s3. I found a similar question (here), but it saves as a new file (I don't want it). Then I found an another way (here), then I tried to implemented it, everything was ok when I tested it. But after I deployed to prod, some users reported they occasionally got a problem while uploading images with a size of 1 MB and above, so they must try it several times.
So, I just want to make sure is my code correct?, or maybe there is something I can improve.
from PIL import Image
# I got body from http Request
img = Image.open(body)
img_format = img.format
# Save it in-memory to remove EXIF
temp = io.BytesIO()
img.save(temp, format=img_format)
body = io.BytesIO(temp.getvalue())
# Upload to s3
s3_client.upload_fileobj(body, BUCKET_NAME, file_key)
*I'm still finding out if this issue is caused by other things.

You should be able to copy the pixel data and palette (if any) from an existing image to a new stripped image like this:
from PIL import Image
# Load existing image
existing = Image.open(...)
# Create new empty image, same size and mode
stripped = Image.new(existing.mode, existing.size)
# Copy pixels, but not metadata, across
stripped.putdata(existing.getdata())
# Copy palette across, if any
if 'P' in existing.mode: stripped.putpalette(existing.getpalette())
Note that this will strip ALL metadata from your image... EXIF, comments, IPTC, 8BIM, ICC colour profiles, dpi, copyright, whether it is progressive, whether it is animated.
Note also that it will write JPEG images with PIL's default quality of 75 when you save it, which may or may not be the same as your original image had - i.e. the size may change.
If the above stripping is excessive, you could just strip the EXIF like this:
from PIL import Image
im = Image.open(...)
# Strip just EXIF data
if 'exif' in im.info: del im.info['exif']
When saving, you could test if JPEG, and propagate the existing quality forward with:
im.save(..., quality='keep')
Note: If you want to verify what metadata is in any given image, before and after stripping, you can use exiftool or ImageMagick on macOS, Linux and Windows, as follows:
exiftool SOMEIMAGE.JPG
magick identify -verbose SOMEIMAGE.JPG

Only one image from 5 is downloaded and it knocks out an error

import requests
from PIL import Image
url_shoes_for_choice = [
"https://content.adidas.co.in/static/Product-CM7531/Unisex_OUTDOOR_SANDALS_CM7531_1.jpg",
"https://cdn.shopify.com/s/files/1/0080/1374/2161/products/product-image-897958210_640x.jpg?v=1571713841",
"https://cdn.chamaripashoes.com/media/catalog/product/cache/9/image/9df78eab33525d08d6e5fb8d27136e95/1/_/1_8_3.jpg",
"https://ae01.alicdn.com/kf/HTB1EyKjaI_vK1Rjy0Foq6xIxVXah.jpg_q50.jpg",
"https://www.converse.com/dw/image/v2/BCZC_PRD/on/demandware.static/-/Sites-cnv-master-catalog/default/dwb9eb8c43/images/a_107/167708C_A_107X1.jpg"
]
def img():
for url in url_shoes_for_choice:
image = requests.get(url, stream=True).raw
out = Image.open(image)
out.save('image/image.jpg', 'jpg')
if __name__=="__main__":
img()
Error:
OSError: cannot identify image file <_io.BytesIO object at 0x7fa185c52d58>

The problem is that one of the images is making issues with the byte data returned by the requests.get(url, stream=True).raw, I'm not sure but I guess the data of the 3rd image is invalid byte data so instead of getting the raw data we can just fetch the content and then by using BytesIO we can fix the byte data.
I fixed one more thing from your original code, I added numbering to your images so each can be saved with different name.
from io import BytesIO
def img():
for count, url in enumerate(url_shoes_for_choice):
image = requests.get(url, stream=True)
with BytesIO(image.content) as f:
with Image.open(f) as out:
# out.show() # See the images
out.save('image/image{}.jpg'.format(count))
(Though this works fine but I'm not sure what was the main issue. If anyone knows exactly what is the issue please comment and explain.)

I opened the first link in my browser and saved the image. It's actually a webp file.
$ file Unisex_OUTDOOR_SANDALS_CM7531_1.webp
Unisex_OUTDOOR_SANDALS_CM7531_1.webp: RIFF (little-endian) data, Web/P image, VP8 encoding, 500x500, Scaling: [none]x[none], YUV color, decoders should clamp
You explicitly tell the image library that it should expect a jpg. When you remove that parameter and let it figure it out on its own using out.save('image/image.jpg') the first image successfully downloads for me.
The first two images work this way if you make sure to save each under a different name:
def img():
i = 0
for url in url_shoes_for_choice:
i+=1
image = requests.get(url, stream=True).raw
out = Image.open(image)
out.save('image{}.jpg'.format(i))
the third is a valid jpeg file, as well as the fourth, but using the JFIF standard 1.01 which I hear the first time of. I'm pretty sure you'll have to figure out support for different such filetypes.
It is worth noting that if I download the images in chrome and open those with python, nothing fails. So chrome might be adding information to the file.
The documentation of PIL/pillow explains here that you need a new enough version for animated images, but that is not your problem.
Support for animated WebP files will only be enabled if the system
WebP library is v0.5.0 or later. You can check webp animation support
at runtime by calling features.check(“webp_anim”).

Uploading grayscale image Python

I am trying to upload an image that has been converted to grayscale, like this:
blob_path = os.path.join(os.path.split(__file__)[0], 'static/img/blob-masks/1.png')
blob = Image.open(blob_path).convert('L')
buffer = StringIO()
blob.save(buffer)
upload_image(buffer.getvalue(),"foo.png")
But it just seem to upload a black square.
If I got to the command line python and run:
col = Image.open("/static/img/blob-masks/5.png")
col.convert('L')
col.save("result_bw.png")
result_bw.png is perfect. What is going wrong?

Is there a reason you can't just upload the greyscale image after you convert it? Like:
image = Image.open('static/img/blob-masks/1.png')
image.convert('L')
image.save("temp/bw.png")
upload_image("temp/bw.png")
# maybe delete the temporary file when you're done
import os
os.remove("temp/bw.png")
I'm not sure how your upload_image() function works, but when I upload using django if I do any manipulations I write a temporary file and then re-import. If I don't manipulate the image at all I can just upload it directly.

Why won't jpeg save properly to AWS S3 from django

I'm trying to save a user uploaded file directly to S3 without saving it locally. This project is using Django 1.9 and Boto3.
The relevant code is:
p=request.FILES['img'].read()
s3=boto3.resource('s3',
aws_access_key_id=settings.AWS_ACCESS_KEY_ID,
aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
b = s3.Bucket(settings.AWS_STORAGE_BUCKET_NAME)
b.put_object(Key="media/test.jpg", Body=p)
This correctly uploads a file called 'test.jpg' to the media folder.
However, if I download 'test.jpg' from Amazon and try to open it in an image viewer, I get the message: "Error interpreting JPEG image file (Not a JPEG file: starts with 0xf0 0xef)". The jpg file is also only 26kb whereas the original was 116kb.
What is going wrong? I assume I am passing the wrong data as Body in the put_object method. But what should p be instead?
Update and Solutions
With JordonPhilips's help, I realised that because I had already opened the uploaded image earlier in the view with Pillow, the request.FILES['img'] socket had already been read.
The solution I went with was to remove the Pillow code, leaving the boto upload as the first access of request.FILES['img'].
However, I also figured out a solution if you want to do something to the image first (e.g. in Pillow):
from Pillow import Image
import cStringIO as StringIO
import boto3
and then in the view function:
im = Image.open(request.FILES['img'])
# whatever image analysis here
file2 = StringIO.StringIO()
im.save(file2,"jpeg",quality='keep')
s3 = boto3.resource( 's3', aws_access_key_id=settings. AWS_ACCESS_KEY_ID, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
b = s3.Bucket(settings.AWS_STORAGE_BUCKET_NAME)
b.put_object(Key="media/test.jpg", Body=file2.getvalue())

It looks like your problem was that you were trying to read the socket multiple times. You can only read the socket once, so you need to keep reference to important information.

Python/Django download Image from URL, modify, and save to ImageField

I've been looking for a way to download an image from a URL, preform some image manipulations (resize) actions on it, and then save it to a django ImageField. Using the two great posts (linked below), I have been able to download and save an image to an ImageField. However, I've been having some trouble manipulating the file once I have it.
Specifically, the model field save() method requires a File() object as the second parameter. So my data has to eventually be a File() object. The blog posts linked below show how to use urllib2 to save your an image URL into a File() object. This is great, however, I also want to manipulate the image using PIL as an Image() object. (or ImageFile object).
My preferred approach would be then to load the image URL directly into an Image() object, preform the resize, and convert it to a File() object and then save it in the model. However, my attempts to convert an Image() to a File() have failed. If at all possible, I want to limit the number of times I write to the disk, so I'd like to do this object transformation in Memory or using a NamedTemporaryFile(delete=True) object so I don't have to worry about extra files laying around. (Of course, I want the file to be written to disk once it is saved via the model).
import urllib2
from PIL import Image, ImageFile
from django.core.files import File
from django.core.files.temp import NamedTemporaryFile
inStream = urllib2.urlopen('http://www.google.com/intl/en_ALL/images/srpr/logo1w.png')
parser = ImageFile.Parser()
while True:
s = inStream.read(1024)
if not s:
break
parser.feed(s)
inImage = parser.close()
# convert to RGB to avoid error with png and tiffs
if inImage.mode != "RGB":
inImage = inImage.convert("RGB")
# resize could occur here
# START OF CODE THAT DOES NOT SEEM TO WORK
# I need to somehow convert an image .....
img_temp = NamedTemporaryFile(delete=True)
img_temp.write(inImage.tostring())
img_temp.flush()
file_object = File(img_temp)
# .... into a file that the Django object will accept.
# END OF CODE THAT DOES NOT SEEM TO WORK
my_model_instance.image.save(
'some_filename',
file_object, # this must be a File() object
save=True,
)
With this approach, the file appears corrupt whenever I view it as an image. Does anyone have any approach that takes a file file from a URL, allows one to manipulate it as an Image and then save it to a Django ImageField?
Any help is much appreciated.
Programmatically saving image to Django ImageField
Django: add image in an ImageField from image url
Update 08/11/2010: I did end up going with StringIO, however, I was stringIO was throwing an unusual Exception when I tried to save it in a Django ImageField. Specifically, the stack trace showed a name error:
"AttribueError exception "StringIO instance has no attribute 'name'"
After digging through the Django source, it looks like this error was caused when the model save tries to access the size attribute of the StringIO "File". (Though the error above indicates a problem with the name, the root cause of this error appears to be the lack of a size property on the StringIO image). As soon as I assigned a value to the size attribute of the image file, it worked fine.

In an attempt to kill 2 birds with 1 stone. Why not use a (c)StringIO object instead of a NamedTemporaryFile? You won't have to store it on disk anymore and I know for a fact that something like this works (I use similar code myself).
from cStringIO import StringIO
img_temp = StringIO()
inImage.save(img_temp, 'PNG')
img_temp.seek(0)
file_object = File(img_temp, filename)