We Keep Coding

How do I download a JPG file from Amazon S3 into memory using Python?

I'm struggling to download a JPG file from Amazon S3 using Python, I want to load this code onto Heroku so I need to the image to be loaded into memory rather than onto disk.
The code I'm using is:
import boto3
s3 = boto3.client(
"s3",
aws_access_key_id = access_key,
aws_secret_access_key = access_secret
)
s3.upload_fileobj(image_conv, bucket, Key = "image_3.jpg")
new_obj = s3.get_object(Bucket=bucket, Key="image_3.jpg")
image_dl = new_obj['Body'].read()
Image.open(image_dl)
I'm getting the error message:
File ..... line 2968, in open
fp = builtins.open(filename, "rb")
ValueError: embedded null byte
Calling image_dl returns a massive long list of what I assume are bytes, one small section looks like the following:
f\xbc\xdc\x8f\xfe\xb5q\xda}\xed\xcb\xdcD\xab\xe6o\x1c;\xb7\xa0\xf5\xf5\xae\xa6)\xbe\xee\xe6\xc3vn\xdfLVW:\x96\xa8\xa3}\xa4\xd8\xea\x8f*\x89\xd7\xcc\xe8\xf0\xca\xb9\x0b\xf4\x1f\xe7\x15\x93\x0f\x83ty$h\xa6\x83\xc8\x99z<K\xc3c\xd4w\xae\xa4\xc2\xfb\xcb\xee\xe0
The image before I uploaded to S3 returned the below and that's the format that I'm trying to return the image into. Is anyone able to help me on where I'm going wrong?
<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=1440x1440 at 0x7F2BB4005EB0>

Pillow's Image class needs either a filename to open, or a file-like object that it can call read on. Since you don't have a filename, you'll need to provide a stream. It's easiest to use BytesIO to turn the byte array into a strem:
import boto3
from PIL import Image
from io import BytesIO
bucket = "--example-bucket--"
s3 = boto3.client("s3")
with open("image.jpg", "rb") as image_conv:
s3.upload_fileobj(image_conv, bucket, Key="image_3.jpg")
new_obj = s3.get_object(Bucket=bucket, Key="image_3.jpg")
image_dl = new_obj['Body'].read()
image = Image.open(BytesIO(image_dl))
print(image.width, image.height)

Try first to load raw data into a BytesIO container:
from io import StringIO
from PIL import Image
file_stream = StringIO()
s3.download_fileobj(bucket, "image_3.jpg", file_stream)
img = Image.open(file_stream)
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.download_fileobj

Read Json files from Azure blob using python?

I need to read a JSON file from a blob container in Azure for doing some transformation on top of the JSON Files. I have seen few documentation and StackOverflow answers and developed a python code that will read the files from the blob.
I have tried the below script from one of the Stackoverflow answers to read JSON file but I get the below error
"TypeError: the JSON object must be str, bytes or byte array, not BytesIO"
I am new to python programming so not sure of the issue in the code. I tried with download_stream.content_as_text() but the file doesnt read the file without any error.
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
from io import BytesIO
import requests
from pandas import json_normalize
import json
filename = "sample.json"
container_name="test"
constr = ""
blob_service_client = BlobServiceClient.from_connection_string(constr)
container_client=blob_service_client.get_container_client(container_name)
blob_client = container_client.get_blob_client(filename)
streamdownloader=blob_client.download_blob()
stream = BytesIO()
streamdownloader.download_to_stream(stream)
# with open(stream) as j:
# contents = json.loads(j)
fileReader = json.loads(stream)
print(filereader)

You can use readallfunction. Please try this code:
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient
import json
filename = "sample.json"
container_name="test"
constr = ""
blob_service_client = BlobServiceClient.from_connection_string(constr)
container_client = blob_service_client.get_container_client(container_name)
blob_client = container_client.get_blob_client(filename)
streamdownloader = blob_client.download_blob()
fileReader = json.loads(streamdownloader.readall())
print(fileReader)
Result:

How to read image from Firebase using OpencCV?

Is there any idea for reading image from Firebase using OpenCV? Or do I have to download the pictures first and then do the cv.imread function from the local folder ?
Is there any way that I could just use cv.imread(link_of_picture_from_firebase)?

Here's how you can:
read a JPEG from disk,
convert to JSON,
upload to Firebase
Then you can:
retrieve the image back from Firebase
decode the JPEG data back into a Numpy array
save the retrieved image on disk
#!/usr/bin/env python3
import numpy as np
import cv2
from base64 import b64encode, b64decode
import pyrebase
config = {
"apiKey": "SECRET",
"authDomain": "SECRET",
"databaseURL": "SECRET",
"storageBucket": "SECRET",
"appId": "SECRET",
"serviceAccount": "FirebaseCredentials.json"
}
# Initialise and connect to Firebase
firebase = pyrebase.initialize_app(config)
db = firebase.database()
# Read JPEG image from disk...
# ... convert to UTF and JSON
# ... and upload to Firebase
with open("image2.jpg", 'rb') as f:
data = f.read()
str = b64encode(data).decode('UTF-8')
db.child("image").set({"data": str})
# Retrieve image from Firebase
retrieved = db.child("image").get().val()
retrData = retrieved["data"]
JPEG = b64decode(retrData)
image = cv2.imdecode(np.frombuffer(JPEG,dtype=np.uint8), cv2.IMREAD_COLOR)
cv2.imwrite('result.jpg',image)

Save Base64 String into Django ImageField

Im doing an application that uses Django in server-side.
Im trying to do that:
import uuid
from base64 import b64decode
from django.core.files.base import ContentFile
#staticmethod
def add_photo(user, person, image_base64):
photo = DatabasePersonPhoto()
photo.user = user
photo.person = person
image_data = b64decode(image_base64)
image_name = str(uuid.uuid4())+".jpg"
photo.image = ContentFile(image_data, image_name)
photo.save()
return photo
This is my Base64 String:
data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAUDBAQEAwUEBAQFBQUGBwwIBwcHBw8LCwkMEQ8SEhEPERETFhwXExQaFRERGCEYGh0dHx8fExciJCIeJBweHx7/2wBDAQUFBQcGBw4ICA4eFBEUHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh4eHh7/wAARCABkAGQDASIAAhEBAxEB/8QAHAAAAQQDAQAAAAAAAAAAAAAABQAEBgcCAwgB/8QANRAAAQQBAgQFAgUDBQEBAAAAAQIDBBEABSEGEjFBBxNRYXEUIhWBkaGxMvDxCCNCwdEWQ//EABsBAAIDAQEBAAAAAAAAAAAAAAMEAQIFBgAH/8QAJxEAAgEDAwQCAgMAAAAAAAAAAAECAxEhBBIxEzJBYQVxIlEUM5H/2gAMAwEAAhEDEQA/AOc3SPjbMAvlJJ6BNV75i4o73ZrNzEOVKpEaO64SRfKgmt851JJZOnvbJorursarHEHTpUoOKjRXXg2BzlCCeUHoTWF4/C+pvuNjy/LKiAAo2b9Ky8/DXTo/BHkNIF6hJALq10QAOtA9O2xwdSttWMsjclnkoXRuG9e1hfJp2lypJAIPI2SNtyL9fbCmjeH/ABfqU5+LH0OUXWFFDnOjkCVA0RZoXnWsLV48cEw47LHOoqUtKQAVHcmh1JN56dWjOFTi3yFXZUNyR63i0tTPwi6rejlBXhrxotCnvwCWEJAJBABIrsLvMNI4b12CpDczS5sYvLpvnZUOY7DYEb51ZM17SSwG0cwpQAKT19j85lO1DT5LDCkPoRIbPmIQAFFQH8dcFOvUlFxawXhXSdzknxfYcj6VpMV1Cm3EcxIKaIJJO4yvGEfc6TezZOd3cR8M8L8a6UlrWYDL6kcpUofa4k2LpQ3HTp0wHrX+n/w51eE4rTvqdKkOJoLac5kgbncG8b0OqjRpKm1kBqWqtTecZx+g98eNUUkXQy/tY/07QdM0OZNi8Qu6pKbBLDDTIQNunMbP7V16isoWVGdhSHGHk8q0EpPpYOOqrGp2l6OEbedr/iRXxixunYVeLJuE2lq6LwzpsZDYfYRIfABVzWQMIam43EjnlBaTQISlAAPpnsKS22klLhUALJI67b75hpkZzU9Rbc+nVIJVUZhAJU8obXVf0juT1xKSbauZ6m3lnuiakNIjJmTQlUuRYisUSo9geor2+byQ6A1KjL+sfdWqW+ac5lgggiwE7WB2u98cxeFmIj/1+qyfqpCgQq2+UhVikJ6Ght1sbbAYQYmxIkkLBVLmEABhkcxR2ANdB7nBTd3tj/paOFdkkix5ioRcLKqAHILqqG+/f5yNz5khxSmm3A04jZTZJBG938Uccp40mxdRah6nCeiNuggEmwR/g+uNOJIcfUpSJkRxTbvRRB2I3NE9+1ZW+12ZZJtXBumyFcxbcKklLgNlVi67d+5zfGmTI8x54uKIJoGwaA7fvg+fF/D3Ww4+pSngUkkgcvYAdqr/AKw3ougJfWFOzFKTy1d0D6+/tkyeD0UG9F1CWgh1laisH7z2I7gj2yVfiKhF81skIWilJBICT3Irt7Y20SHp+nspZSRzEkgCiPTfr74UdgN19RGVdgA7bE/xlEk3cs3ixCeHuJ1aRqkuG8oLaU5a2lncg9CN9vg7H2vK08buG9Ekz3NT0ttDLjxtXJ0UT6jsfXpk+8S+EJRKNW0fnTLjHnKS4AFp7g2K39T0oZVnGs16RGiatHBU2SUvoNnkWDSgfS6se94aMXFqUXgiE2m0VsvTZzKy2uI4SDsQD0xZO29TQtAU2sJHptiw3Wn+gvW9GlqQmI5JVOVcRs7DmouHske5vc9v0y4OA9NY4a4X/wDoZ0iO9qk1rljMqassIO9JINChudsoTX3T+LxoR5HGGCCQTQWrayoj5/LJZ4p8STGI+m6bGDbDKIw2aWpSEihZHNfb+xhKkW0rcyEItK/onL3EmlrX5aksyJK10FOqPLYNHajQ29L+cgPFPHWpwdcjwlOIhw7CipoUVpJ63XTbK1bn6iXQWpJbBoizVg9/85KOJ+GpyGnIc936txLIcjym2nEIUSCeSlpBsUR0rY0TWM0dHGms+QM60pPHgvDW9Si6zwc3PiOBTrADrZG5JB3sA9wenvmWnSI78RDyCgd7I36V1+D39MpLw64h1nT9MXpceEiWASCpZIDQPc/ttkn0rRNRK1LXqMlLjv3ENKIT0ugPyGK1tLZ8jFKtjglerraky245kNk821Hf2r+fyw+zADEUB2cphqiQQaIHqP775WTvDqkOplIlPocBFEqJIPt++NuLNV1NnTAzImqUyiguwQaJ/bbBfxnhLITqq3BO9NQqaZk2dqEt2EzZacDhFgdSAK9KB6nJnwFxUwrykQtRRqMaykoccAeT06gkWf0yhuIuOhE0iG3p7nMkoAKQNq6AfluOnpgjgYqTqI1afMkQWlrBbQy4W1K3FnYitsNHRNxbeAUtRmyOzIqoktDqXWVBLp+0PIo36HtVHKG8UdDj8H8T+S8lxWha5aFpQwEJYdA2KSAEg96G+2+TTw/41jPhMZ3VEzY6/wDitQW4PlQ616kD/wBL+LWix+KuANQ0/mSmbESHmFlVUobpFWBRGxv/AKGL04uE3CXAVyutyOXdShP6bPegyCAtlXKCLpQ6gj5BGLCuh6vo83Tm06/H82XG/wBgLC+UlA3F77kWRftixnbNYsD3L9nmkaV+LcQttNJUoFwKeUobJQDZv+B84R8WYivqWXEtJDe6PtGxNA1t8ZbTWmobBDTTTYJsgCt/yGAPFTRGVcDeaopS55hcSR1Cu29d6rMLTfIT1epUrWUfA3OiqcGvLKBYYUpSWVRStSCQlSFAEi+hv09ckmiMPzkJhvvONMMuKdcShRq6oC/YX+pyPSJCwwHEDldUQgkGt/7rCBnP6bo3lRT96kkuOHayetnv1O2dXG8lgy5NIKa7qrUB1uDp6UKWmgEJ+2/Unv8Ar65m0ddmB0L1Z5LraQtbDBKQgEbAkV27+xyJQkqcbDqnQFrWFcxFknsD02wwJWrRtRlT1BpZMcthSAAFpJu679cuoJYXIPe2GGl65AfD7OpLDi0+YEPOeYlQBrvv/jMTxA3qgXE1GM2mQByuNkmj6EE9vjGcaRqP4+7OZisPsLioSVuKKQ2K69wCSDt+2CteihtxEpqQhUkEK/2ySK/P12/TI6af2WU2voIRNJiRnwHg8/GUoFsI3G/RJ/Ot7yTSWuHNL4kjfjMD8biHT/OUwZTrCOdQtCUlsEiqoWKNm62IG8HayzMhrEkU+0CSKsq+fUfPTHurOxJLbLsSSmM40gtqQWwoFO5AIJHTtv0yu6Sdn4CxUWsEN0+S5A15t3SVPttrc2RzElG+wva69cv7wy4tnazFnpltEKjpEcEkAGgbJv2PTKZ0xlqE4++wlTzpBHOsBJSkjcJSCdyCRdnY9Bly8McPTNH4IimZFUxJnFTzgNhX3LsD22I7d8V1c4YdshqVNpW8MoDxCY+h4z1NloKQ2p8uJCRtSt/+8WF/FvS5w40fWmO4ULZbKTXYJA/kHFhqNe9NNsHOj+TwdL+QegyPeKSHHPD+Y2lClBoWeWttwQTtk1+nHc5G/EiOTwhqCWgSpTKgQO4rOC+NqqlXV+Ga1eO+P0csgIkPtIa5wlBtxR6E31G/p8Zv1kpJQ0ogi/6EkX+ZH8Y2inypASpfKSSNug6dP79McLjthZLVrN2ST0z6LTX4owJ3bNSQskpTSUA7EC7w8zEU1pLkl5S6UAAg3VkjoP1xop6NCbbU6lPm1uOt/A/T9MIFOs6npbjrbbSGUBJQhTnKSAboD8sve/BW1uTyah1MByO2/TKq5UqTQPToR1I369hgVxh1UZKlqTzIBQFBI2UDt77g1Z9cNx9TcZSmFqcZTY5qBVsQe+/TvjadGS26lTTylNrPMOU3XQV/G+Re2CUrgaC+4xODqPtcC9yBW99CPXCavMEtTtFxKwCADsSe19iaNY0jBS5rZU1zEk8yk0QR6/OE0EQZ6ylYS2SAVVsBV2PQgj9zlZ2aL020ywPCbg+Nq+rsStTKQwy4FJZKqJ7gncd+1nrnQXFMASY0VLSBYNbEiqrp+mVr4HRBOSJIfb8oVQqiRVE7VXp/nLr0xgT5gcSg+U0Nq6UO/vZzJrN3H4PyyEzeHIinyX/JC6Gy0A/pv0xZZiNMjvgrWne6xYjtQTqeyti2s/8A6ED2TX83jHVdPZnRnY8hCnG1jlIUSQSdvy3PbDBZJH9RF5itlNJT2Bs7dgCc5KLad0POxyj4lcKSOGteU4EqciFZDagOmwNH3o5H2H2m0KeWsFI7Hez6VnS3E+nMas9JYnND6Z8BCSRfKRsD7fPxlKcY+G+p6ROKopEhqyUAkkkE9CTtYzuPjtfvpqNbDRk6jTtS3QIM06qRKMpwDmPRPYDsBhVWoPoDaQooB322H99MYTIsqFJKZTCmlEWErFUPUZpWvmUOYgUbBBBr2zaUou1ngSs1yg3qcsyI5TIBVVAE9ReNdHlB1SoT5BcQbST3GN25ILSg8kmtwR3xm4Lc81olKkHmSa3+MlK6eTzk01gk8KOpiSsJcSlO5FC6F9P5xq24h3WHG3KUhZoFO2522/8AMHonvy0hooUVKG6QDucsPws8OZusTWZcwqabJBbQBa1G6v2A6/F4KpJRQWEdzuXb4IaQ+zw9HipSUKCQlxJ7jerF7D3AF9/XLpiNtREtw2EFSyBzDuB7nAfCujM6JAbiRmgt8p2O5UkHqSf7vJpo0D6ZBDyi46TZcI6+g9q9Myql5OyGXJRWTONE5GQCbPXFjopF0aBHtiyvSAdT2VErrjOeaZ6A7H+MWLOOo/2R+zXl2sFSwlbJaKE8i07gDA+lnnff053/AHWG9kc+5SPnFizqo90RReSA8ZcP6ZIRMiOslTbIK2zzfckn3ylVQGQ0FWsnnI3rpXxixZraTtYpX5GKSQ4pI2AG1ZJeAdPi6hMkoloLqUVygnp1xYsZfawUe5F4eH/CmhR46pSILZeLlcygCR16ZbvCMCLHZckMtJStN1Q9jixYpU4Q1Dlkw4PUXFoeX9y3VHmJyWkf1J7VixYpT8g63cecoUATZOLFiywI/9k=
The image file is generated, but I cant open it like an image.

I think this will be a best approach tried it and tested in django 1.10. based on this SO answer: https://stackoverflow.com/a/28036805/6143656
I made a function for decoded base64 file.
def decode_base64_file(data):
def get_file_extension(file_name, decoded_file):
import imghdr
extension = imghdr.what(file_name, decoded_file)
extension = "jpg" if extension == "jpeg" else extension
return extension
from django.core.files.base import ContentFile
import base64
import six
import uuid
# Check if this is a base64 string
if isinstance(data, six.string_types):
# Check if the base64 string is in the "data:" format
if 'data:' in data and ';base64,' in data:
# Break out the header from the base64 content
header, data = data.split(';base64,')
# Try to decode the file. Return validation error if it fails.
try:
decoded_file = base64.b64decode(data)
except TypeError:
TypeError('invalid_image')
# Generate file name:
file_name = str(uuid.uuid4())[:12] # 12 characters are more than enough.
# Get the file name extension:
file_extension = get_file_extension(file_name, decoded_file)
complete_file_name = "%s.%s" % (file_name, file_extension, )
return ContentFile(decoded_file, name=complete_file_name)
Then you can call the function
import decode_base64_file
p = Post(content='My Picture', image=decode_based64_file(your_base64_file))
p.save()

I found the solution.
I need to use only the parte without data:image/jpeg;base64,
In Python, we can do it with something like this:
image_base64 = image_base64.split('base64,', 1 )

fh = open("imageToSave.png", "wb")
fh.write(imgData.decode('base64'))
fh.close()
Edit (klaus-d): The code above gives an example, how to store an image file from BASE64 encoded data. It opens a file imageToSave.png in binary mode for writing, then decodes the base64 image data and write the result to the file. At the end it closes the file descriptor.

Decoding base64 from POST to use in PIL

I'm making a simple API in Flask that accepts an image encoded in base64, then decodes it for further processing using Pillow.
I've looked at some examples (1, 2, 3), and I think I get the gist of the process, but I keep getting an error where Pillow can't read the string I gave it.
Here's what I've got so far:
import cStringIO
from PIL import Image
import base64
data = request.form
image_string = cStringIO.StringIO(base64.b64decode(data['img']))
image = Image.open(image_string)
which gives the error:
IOError: cannot identify image file <cStringIO.StringIO object at 0x10f84c7a0>

You should try something like:
from PIL import Image
from io import BytesIO
import base64
data['img'] = '''R0lGODlhDwAPAKECAAAAzMzM/////wAAACwAAAAADwAPAAACIISPeQHsrZ5ModrLl
N48CXF8m2iQ3YmmKqVlRtW4MLwWACH+H09wdGltaXplZCBieSBVbGVhZCBTbWFydFNhdmVyIQAAOw=='''
im = Image.open(BytesIO(base64.b64decode(data['img'])))
Your data['img'] string should not include the HTML tags or the parameters data:image/jpeg;base64 that are in the example JSFiddle.
I've changed the image string for an example I took from Google, just for readability purposes.

There is a metadata prefix of data:image/jpeg;base64, being included in the img field. Normally this metadata is used in a CSS or HTML data URI when embedding image data into the document or stylesheet. It is there to provide the MIME type and encoding of the embedded data to the rendering browser.
You can strip off the prefix before the base64 decode and this should result in valid image data that PIL can load (see below), but you really need to question how the metadata is being submitted to your server as normally it should not.
import re
import cStringIO
from PIL import Image
image_data = re.sub('^data:image/.+;base64,', '', data['img']).decode('base64')
image = Image.open(cStringIO.StringIO(image_data))

Sorry for necromancy, but none of the answers worked completely for me. Here is code working on Python 3.6 and Flask 0.13.
Server:
from flask import Flask, jsonify, request
from io import BytesIO
from web import app
import base64
import re
import json
from PIL import Image
#app.route('/process_image', methods=['post'])
def process_image():
image_data = re.sub('^data:image/.+;base64,', '', request.form['data'])
im = Image.open(BytesIO(base64.b64decode(image_data)))
return json.dumps({'result': 'success'}), 200, {'ContentType': 'application/json'}
Client JS:
// file comes from file input
var reader = new FileReader();
reader.onloadend = function () {
var fileName = file.name;
$.post('/process_image', { data: reader.result, name: fileName });
};
reader.readAsDataURL(file);