We Keep Coding

Django broken response with file data

In my project I use AWS S3 Bucket for media files. Project contains media_app application, that provide get and post requests to files in remote storage (all requests go through server, so media_app is kinda proxy between frontend and AWS S3.
Today I faced with problem. When I try to get media file from AWS S3, Django correctly download it from remote bucket, save it in /tmp/ dir, get correct response metadata, but response itself is broken: it cause infinite response "in progress" state on frontend side (Tried with Postman and Swagger). "In progress" means "frontend application wait until response will be recieved", but on backend side there is no infinite loop. Django can provide next responses even if I use dev test server (Is is means, that no workers are blocked, cause django test server have only one worker).
Information about request and response (Django Silk):
There is a part of my view, that provide image download:
from django.http import Http404, StreamingHttpResponse
from wsgiref.util import FileWrapper
class ImageDetailAPIView(DetailDownloadAPIView):
queryset = Image.objects.filter(is_soft_deleted=False)\
.prefetch_related('thumbnails')
serializer_class = ImageSerializer
#swagger_auto_schema(responses=image_detail_responses)
def get(self, request, *args, **kwargs):
try:
instance = self.get_object()
except Http404:
raise FileNotFoundException
service = ImageDownloadService(instance)
chunk_size = 8192
file_path, status_code = service.download()
if status_code == 200:
response = StreamingHttpResponse(
FileWrapper(open(file_path, 'rb'), chunk_size),
content_type=instance.mime_type
)
response['Content-Length'] = instance.size
response['Content-Disposition'] = "attachment; filename=%s" % instance.file_name
return response
raise UnexpectedCaseException(is_for_file=True)
service architecture and code doesn't important, because I've already discover, that it works correctly: image correct identify in AWS and further download to /tmp/downloaded-file.webp. Replace StreamingHttpResponse by FileResponse doesn't work. I've tested it all with images with .png and .webp formats, and it doesn't work anyway.

I've solved the issue by using this post (but didn't find any cause of it):
add django.middleware.gzip.GZipMiddleware middleware to MIDDLEWARE project settings variable.
write a subclass for my existing FileWrapper (code below).
FixedFileWrapper:
class FixedFileWrapper(FileWrapper):
def __iter__(self):
self.filelike.seek(0)
return self
Maybe, someone can explain me the reason my issue?

Get Cloud Storage upload response

I am uploading a file to a Cloud Storage bucket using the Python SDK:
from google.cloud import storage
bucket = storage.Client().get_bucket('mybucket')
df = # pandas df to save
csv = df.to_csv(index=False)
output = 'test.csv'
blob = bucket.blob(output)
blob.upload_from_string(csv)
How can I get the response to know if the file was uploaded successfully? I need to log the response to notify the user about the operation.
I tried with:
response = blob.upload_from_string(csv)
but it always return a None object even when the operation has succeded.

You can try with tqdm library.
import os
from google.cloud import storage
from tqdm import tqdm
def upload_function(client, bucket_name, source, dest, content_type=None):
bucket = client.bucket(bucket_name)
blob = bucket.blob(dest)
with open(source, "rb") as in_file:
total_bytes = os.fstat(in_file.fileno()).st_size
with tqdm.wrapattr(in_file, "read", total=total_bytes, miniters=1, desc="upload to %s" % bucket_name) as file_obj:
blob.upload_from_file(file_obj,content_type=content_type,size=total_bytes,
)
return blob
if __name__ == "__main__":
upload_function(storage.Client(), "bucket", "C:\files\", "Cloud:\blob.txt", "text/plain")

Regarding how to get notifications about changes made into the buckets there is a few ways that you could also try:
Using Pub/Sub - This is the recommended way where Pub/Sub notifications send information about changes to objects in your buckets to Pub/Sub, where the information is added to a Pub/Sub topic of your choice in the form of messages. Here you will find an example using python, as in your case, and using other ways as gsutil, other supported languages or REST APIs.
Object change notification with Watchbucket: This will create a notification channel that sends notification events to the given application URL for the given bucket using a gsutil command.
Cloud Functions with Google Cloud Storage Triggers using event-driven functions to handle events from Google Cloud Storage configuring these notifications to trigger in response to various events inside a bucket—object creation, deletion, archiving and metadata updates. Here there is some documentation on how to implement it.
Another way is using Eventarc to build an event-driven architectures, it offers a standardized solution to manage the flow of state changes, called events, between decoupled microservices. Eventarc routes these events to Cloud Run while managing delivery, security, authorization, observability, and error-handling for you. Here there is a guide on how to implement it.
Here you’ll be able to find related post with the same issue and answers:
Using Storage-triggered Cloud Function.
With Object Change Notification and Cloud Pub/Sub Notifications for Cloud Storage.
Answer with a Cloud Pub/Sub topic example.

You can verify if the upload gets any error, then use the exception's response methods:
def upload(blob,content):
try:
blob.upload_from_string(content)
except Exception as e:
status_code = e.response.status_code
status_desc = e.response.json()['error']['message']
else:
status_code = 200
status_desc = 'success'
finally:
return status_code,status_desc
Refs:
https://googleapis.dev/python/google-api-core/latest/_modules/google/api_core/exceptions.html
https://docs.python.org/3/tutorial/errors.html

S3 presigned URL works 90 minutes after bucket creation

We generate presigned URLs in order for users to upload files directly into S3 buckets. Running integration tests we discovered a failing test where an HTTP PUT request on a presigned URL yielded a SignatureDoesNotMatch error response. Surprisingly, the same code worked fine using another bucket. We kept trying on the original bucket that caused the test to fail and were surprised when it suddenly started to work without any real code changes.
We noticed that it was roughly two hours after we had created the bucket when the test successfully ran through. Since we are located at UTC+0200 we suspected the issue to be somehow related to that time difference and/or some clock synching issue. We set out to confirm our suspicions that the same presigned URL would suddenly just work after enough time has passed. SPOILER: It does!
The following code creates a brand new bucket, generates a presigned URL suitable for file upload (ClientMethod='put_object'), and tries to HTTP PUT some data using the requests library. We re-try PUTting data every 60 seconds until it finally succeeds 5419 seconds (or 90 minutes) after the bucket was created.
Note: Even though the bucket is deleted afterwards, running the same script (using the same bucket name) now instantly succeeds. In case you want to re-confirm this behavior, make sure to use a different bucket name the second time around.
import logging
import time
import boto3
import requests
from botocore.client import Config
logger = logging.getLogger(__name__)
# region = "eu-central-1"
# region = "eu-west-1"
# region = "us-west-1"
region = "us-east-1"
s3_client = boto3.client('s3', region_name=region, config=Config(signature_version='s3v4'))
if __name__ == "__main__":
bucket_name = "some-globally-unique-bucket-name"
key_for_file = "test-file.txt"
# create bucket
if region == "us-east-1":
# https://github.com/boto/boto3/issues/125
s3_client.create_bucket(Bucket=bucket_name, ACL='private')
else:
s3_client.create_bucket(Bucket=bucket_name, ACL='private',
CreateBucketConfiguration={'LocationConstraint': region})
creation_time = time.time()
# generate presigned URL
file_data = b"Hello Test World"
expires_in = 4 * 3600
url = s3_client.generate_presigned_url(ClientMethod='put_object', ExpiresIn=expires_in,
Params={'Bucket': bucket_name, 'Key': key_for_file})
time_since_bucket_creation = time.time() - creation_time
time_interval = 60
max_time_passed = expires_in
success = False
try:
while time_since_bucket_creation < max_time_passed:
response = requests.put(url, data=file_data)
if response.status_code == 200:
success = True
break
if b"<Code>SignatureDoesNotMatch</Code>" in response.content:
reason = "SignatureDoesNotMatch"
else:
reason = str(response.content)
time_since_bucket_creation = time.time() - creation_time
print("="*50)
print(f"{time_since_bucket_creation:.2f} s after bucket creation")
print(f"unable to PUT data to url: {url}")
print(f"reason: {reason}")
print(response.content)
time.sleep(time_interval)
except KeyboardInterrupt:
print("Gracefully shutting down...")
if success:
print("YAY! File Upload was successful!")
time_since_bucket_creation = time.time() - creation_time
print(f"{time_since_bucket_creation:.2f} seconds after bucket creation")
s3_client.delete_object(Bucket=bucket_name, Key=key_for_file)
# delete bucket
s3_client.delete_bucket(Bucket=bucket_name)
We run integration tests with an AWS EKS cluster where we create a cluster along with some databases, S3 buckets, etc. and tear everything down after the tests have completed. Having to wait 90 minutes for the presigning of URLs to work is not feasible.
My Questions
Am I doing anything wrong?
Is this expected behavior?
Is there an acceptable workaround?
Can someone, please, confirm this behavior using the above code?
EDIT
I updated the code to create a bucket in the "us-east-1" region as suggested by "Michael - sqlbot" in the comments. The weird if statement is necessary as documented here. I am able to confirm Michael's suspicion that the behavior is NOT reproducible with "us-east-1".
In case it is of interest, the returned XML in the error case:
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<AWSAccessKeyId>REDACTED</AWSAccessKeyId>
<StringToSign>AWS4-HMAC-SHA256
20190609T170351Z
20190609/eu-central-1/s3/aws4_request
c143cb44fa45c56e52b04e61b777ae2206e0aaeed40dafc78e036878fa91dfd6</StringToSign>
<SignatureProvided>REDACTED</SignatureProvided>
<StringToSignBytes>REDACTED</StringToSignBytes>
<CanonicalRequest>PUT
/test-file.txt
X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTED%2F20190609%2Feu-central-1%2Fs3%2Faws4_request&X-Amz-Date=20190609T170351Z&X-Amz-Expires=14400&X-Amz-SignedHeaders=host
host:some-globally-unique-bucket-name.s3.eu-central-1.amazonaws.com
host
UNSIGNED-PAYLOAD</CanonicalRequest>
<CanonicalRequestBytes>REDACTED</CanonicalRequestBytes>
<RequestId>E6CBBC7D2E4D322E</RequestId>
<HostId>j1dM1MNaXaDhzMUXKhqdHd6+/Rl1C3GzdL9YDq0CuP8brQZQV6vbyE9Z63HBHiBWSo+hb6zHKVs=</HostId>
</Error>

Here's what you're bumping into:
A temporary redirect is a type of error response that signals to the requester that they should resend the request to a different endpoint. Due to the distributed nature of Amazon S3, requests can be temporarily routed to the wrong facility. This is most likely to occur immediately after buckets are created or deleted.
For example, if you create a new bucket and immediately make a request to the bucket, you might receive a temporary redirect, depending on the location constraint of the bucket. If you created the bucket in the US East (N. Virginia) AWS Region, you will not see the redirect because this is also the default Amazon S3 endpoint.
However, if the bucket is created in any other Region, any requests for the bucket go to the default endpoint while the bucket's DNS entry is propagated. The default endpoint redirects the request to the correct endpoint with an HTTP 302 response. Temporary redirects contain a URI to the correct facility, which you can use to immediately resend the request.
https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html
Note that the last part -- which you can use to immediately resend the request -- is not quite accurate. You can -- but if the request uses Signature Version 4, then following the redirect to the new hostname will result in a SignatureDoesNotMatch error because the hostname will be different. Back in the old days of Signature Version 2, the bucket name was included in the signature but the endpoint hostname itself was not, so the redirect to a different endpoint hostname would not invalidate the signature.
None of this would be a problem if boto were doing the right thing and using the correct regional endpoint to create the signed URL -- but for some reason, it uses the "global" (generic) endpoint -- which causes S3 to issue those redirects for the first few minutes of the bucket's lifetime, because DNS hasn't been updated, so the request misroutes to us-east-1 and gets redirected. That's why I suspected us-east-1 wouldn't exhibit the behavior.
This should be the default behavior, but it isn't; still, it seems like there should be a cleaner way to do this, automatically via configuration... and there may be... but I haven't found it in the documentation.
As a workaround, the client constructor accepts an endpoint_url argument, which seems to serve the purpose. As it turns out, s3.${region}.amazonaws.com is a valid endpoint for each S3 region, so they can be constructed from a region string.
s3_client = boto3.client('s3', region_name=region, endpoint_url=('https://s3.' + region + '.amazonaws.com'), config=...)
Long-time users of S3 may be suspicious of the claim that all regions support this, but it is accurate as of this writing. Originally, some regions formerly used a dash rather than a dot, e.g. s3-us-west-2.amazonaws.com and this is still valid in those older regions, but all regions now support the canonical form mentioned above.

How to Define Google Endpoints API File Download Message Endpoint

All the examples I can find on google endpoint api (e.g., tic-tac-toe sample) show strings, integers, enums, etc fields. None of the examples say anything about how to specify document (e.g., image or zip files) uploads or downloads using the API. Is this not possible?
If this is possible, can anyone share a code snippet on how to define google endpoint api on the server to allow downloads and uploads of files? For example, is there a way to set HTTPResponse headers to specify that an endpoint response will serve a zip file? How do we include the zip file in the response?
An example with python or php would be appreciated. If anyone from the endpoints-proto-datastore team is watching this discussion, please say whether or not file downloads are supported in endpoints at the moment. We hate to waste our time trying to figure this out if it is simply impossible. Thanks.
We are seeking a complete example for upload and download. We need to store the key for the uploaded file in our database during upload and retrieve it for download. The client app sends a token that the API needs to use to figure out what file to download. Hence, we would need to store the blob key generated during the upload process in our database. Our database would have the mapping between the token and the blob file's key.
class BlobDataFile(models.Model):
data_code = models.CharField(max_length=10) # Key used by client app to request file
blob_key = models.CharField()
By the way, our app is written in Django 1.7 with a mysql (modeled with models.Model) database. It is infuriating that all the examples for Google App Engine upload I can find is written for a standalone webapp Handlers (no urls.py/views.py solutions could be found anywhere). Hence, building a standalone uploader is as much of a challenge as writing the API code. If your solution has full urls.py/views.py example for uploading files and saving the blob_key in our BlobDataFile, it would be good enough for us.

f you use the blobstore use the get_serving_url function to read the images from url in the client, or use the messages.ByteField in the ResourceContainer and serialize the image with base64.b64decode
#the returned class
class Img(messages.Message):
message = messages.BytesField (1)
#The api class
#endpoints.api(name='helloImg', version='v1')
class HelloImgApi(remote.Service):
ID_RESOURCE = endpoints.ResourceContainer(
message_types.VoidMessage,
id=messages.StringField(1, variant=messages.Variant.STRING))
#endpoints.method(ID_RESOURCE, Img,
path='serveimage/{id}', http_method='GET', #ID is the blobstore key
name='greetings.getImage')
def image_get(self, request):
try:
blob_reader = blobstore.BlobReader(blob_key)
value = blob_reader.read()
return Img(message=value)
except:
raise endpoints.NotFoundException('image %s not found.' %
(request.id,))
APPLICATION = endpoints.api_server([HelloImgApi])
And this is the response (save it in the client with the proper format)
{
"message": "/9j/4AAQSkZJRgABAQAAAQABAAD//gA+Q1JFQVRPUjogZ2QtanBlZyB2MS4wICh1c2luZyBJSkcgSlBFRyB2NjIpLCBkZWZhdWx0IHF1YWxpdHkK/9sAQwAIBgYHBgUIBwcHCQkICgwUDQwLCwwZEhMPFB0aHx4dGhwcICQuJyAiLCMcHCg3KSwwMTQ0NB8nOT04MjwuMzQy/9sAQwEJCQkMCwwYDQ0YMiEcITIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIy/8AAEQgBZwKAAwEiAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/aAAwDAQACEQMRAD8A9/ooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACkpaKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAoopKACkZgilmIAHUntWTr3iTTfD1qZr2YBiPkjHLN9BXjniTxzqniSRreItb2ZOBDGeWH+0e/wBK5a+KhRWu/Y48TjadBa6vsei6z8TNF0udreDzL2VeD5ONgPpuP9M1zF18U9VuVZrWzt7OH/npKS5/DpmvP9kVqMyYkl/uDoPrUEs0kzbnbPoOwryZ46tPZ2PDqZlXm9HZHp/g74g3V54gFlqU2+Kf5Y3YBcN+HTNeqCvlqORopFkRiroQykdiK+k/D+oDVNBsrwHJkiUn645rvwFdzTjLdHpZZiZVIuEndo06KKK9E9YKKKKACikpaACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACkpaKAEoopaACkpaKACkpaKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigApKWoLu7gsrZ7i5lSKJBlnc4AFF7CbS1ZKSAMk1wPi74j22lb7PSytxeDhn6pH/ia5bxh8RrjVTJY6SzwWf3Wl6PJ/gK4iOAbfOnJVP1b6V5WJx9vdp/eeLjMzt7lL7ya4uLzWLt7q7naRzy0jngVE9wsSmO34z1c9TTZpzLhQNkY6KKhryG23dnhtuTuxOvWilPSu68DeApdbdNR1JGj04HKJ0M3+C+/etaVKVWXLE2oUZ1pcsEUvB/ge68STC4nDQ6ep5fHMnsv+Ne4afYW2mWMVnaRiOGJcKoqWCCK2hWGGNY40GFVRgAVLXv4fDxoxstz6fC4WGHjZbhRRRXQdQUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUlFAC0UUlAC0UUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFJRmsLxN4psfDViZbhg07D91CDyx/wqZSUVdkznGEeaT0Les65Y6FYvdXsoRR91e7H0Arw3xT4wvvE9zhyYrNT+7gU8fVvU1Q1zXr/wAQ37XN3IWOcJGPuoPQCqyqtoMsA03Ydl/+vXh4rGOp7sdj5vG5hKr7sNI/mIkSQKJJhlj91P8AGopZWlfcx+ntTWYuxZjkmkrhPNCg0V3PgPwO+tzLqN+hWwQ5RT/y1P8AhWlKlKrLlibUKEq0+WJJ4E8Btq8ialqcZWxU5SM8GU+/t/OvZo41iRURQqqMAAYAFEcSRRrHGoVFGAoHAFPr6GhQjRjZH1WGw0KEOWIUUUVudAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRSUALRRRQAlFLRQAUUUUAFJS0UAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUlFcn4y8Z23hq1MUZWW/kHyR5+77n2qJzjCPNIipUjTjzSehL4u8Y2nhizxxNfSD91CD+p9BXheo6lea1qD3V5K0s8h/AewHYUy8vLrVL6S5uZGmuJTksf5fSnZW2XauDKerf3fpXgYrFSrO3Q+YxmNlXlboHy2gwMGY9T2X/wCvVckkkk5JoPNFchwBSUtb/hLwvceJtTES5S0jOZpfQeg9zV04SnLliaUqcqklGO5f8D+DZfEd4Lm5Vk02Fvnb/nof7o/rXuUEEVtCkMKKkaDaqqMACo7Cxt9OsorS1jWOGJdqqKsV9FhsPGjGy3Pq8JhY4eFlv1FoooroOoKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigBKWiigApKWkoAWiiigAopKWgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigApM0ZrlvGXjC38M2O1dsl9KD5UWen+0faonNQjzSIqVI04uUthnjTxlb+G7QxRFZL+Qfu4/7v+0a8Muru51K9e4uJGlnlbJJ5JNF5eXOpXsl1dStLPK2WY08AWy4HMp6n+7XgYnEyrS8j5fGYyVeXkHFspVcGU/eb+77CoOtFFchwCUUtSWtrNe3UdtbxtJNIwVVHUmhJt2RSTbsi5omjXWvapFY2q/MxyzdkXuTX0BoWiWug6ZFZWq4VR8zd2Pcms7wd4Wg8NaWqEK15KA00nv6D2FdJ3r6DB4VUo3e7PqMBg1QjzS+JgKWiiu09AKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAEopaKAEopaKAEopaKAEopaSgBaKKKAEpaKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKSsjxH4htfDulveXLAt0jjB5dvQUpSUVdkykormlsVfFnim28NaaZXIe5cYhizyx9T7V4HqOoXWrX8t5dyGSeQ5JPb2HtU2s6zd67qUl7dyFnY8L2UegqCNBAokcfOfuqe3vXz+LxTqystj5jHYx15WXwoVVFuuT/AK0/+O1ETk5pSSxJPU0lcR51wpKWg0AJ1r2T4deDhpdqNVvo/wDTZl/dow/1Sn+prm/hz4Q/tK5XV76P/RYW/cow/wBYw7/QV7GABwK9nAYW372XyPfyzB2/fT+QUUUYr1T2xaKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAopKWgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAopKiuLiK1t3nncJFGpZmY4AFFxN2K+q6pa6Np8t7dyBIoxnnqT6D3r5+8S+I7rxLqjXU5KxDiGLPCL/jV/xt4ul8S6iViZl0+EkRJ/eP8AeNc5BEGy7/cH614eNxXtHyx2PnMwxvtXyR+FfiPhjCL5sgz/AHV9TSMxdixOSaV3Ltk8DsPSm15tzyb3EopaSgQVveE/Dc3iTV1gAK2yfNM/oPT6msmxsp9RvYbO2QvNK21QK+gfDHh+Dw7pEdpEAZD80sn95q7cFhvayu9kell+E9vPml8KNO1tYbK1itreMJFGoVVHYCpqMUV9ClY+oSsrIWiiigYUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAJS0UUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFJRQAhOASTwK8a+IvjM6ncNpFhJ/okTfvXU/6xh2+grofiR4y/s+3bR7CT/SpV/eup/1ant9TXjyqXYKOSa8rHYq37uPzPEzLGW/dQfqOiiMr46KOSfQVM7g4VRhF6ChsRp5aHj+I+ppleM2eA3cKKKKQgpDS123w88K/2zqI1C7TNlbNkAjiR/T6CtaNJ1ZqKNqFGVaahE634ceE/wCy7Iarex4vLhfkVhzGn+JrvhQAAMDpS19NSpRpwUYn2FGjGjBQj0CiiitDUKKKKACiiigAooooAKSlooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAEpaSigBaSiigBaKYzqv3mA+tNE8RbAlQn0zSuhXRLRSUUxi0UlLQAUUUUAFFJRQAVzfjPxRD4Z0hpAQ13KCsEfqfX6CtnU9St9J0+a9u3CQxLlj6+w96+d/EevXHiPWJb64JCn5Yo88IvYVx4vEeyjZbs4Mdi1QhZfEzPubia8uZLidzJNIxZmPUmpUXyU5++36Co4EAHmN/wEe9PJJ5718/KVz5acrsSiiioMxKKWgKWIABJPAAosNK5oaHo8+u6tDY24OXOXbsq9zX0HpWm2+k6dDZWyhY41AHv71zvgLwuNB0oT3Cf6dcANJkcoOy119fQ4LDeyhzPdn1OXYT2NPml8TFoooruPSCiiigAooooAKKKKAEpaKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAopKKACiiuJ8Y+P7bQVezsys+oYwR/DH9ff2rOpUjTjzSZnVqwpR5ps6PWde07QrYz31wqD+FOrN9BXmOt/FPULxjDpEIto+0jjc5/DoK4m6vL3Wbt7q9naRicl3PA9hTfMWIbYRg93PU14uIzCcnaGiPncVmlSbtT0Rcu7/U79vM1DUZj6B3P8hVdZkhkDpLcGQchw+0iqxJJyTk0lcDqSbu2eY6k27tnZaR8RdU00qkrNdQjtMcsPxr0bw9430zX3ECMYbnH+rk7/Q968Hp8UjwyrJG7I6nKspwQa6qGOq03q7o7cPmVak7N3R9OClrhfAnjP+2YRp984F9GPlY/8tB/jXc179KrGpHmifT0a0K0FOAtFJS1oaiUhIUEk4ApTXA/ErxZ/ZGnf2baSYvLlfmI6onr+NZ1KipxcmZVqsaUHORxvxG8WnWdROnWkn+hW7ckHiR/X6CuJij8x/8AZHJNRjLN6k1bACJtH4185WqucnJnyWIryqTc31FY54HAHApKKK5zlCkpaSgA+ld78NfC/wDaN9/a93Hm1t2xEpHDv6/Qfzrk9D0efXdXgsIAQZD87f3V7mvoTTdPg0vT4LK2QJFEoVRXpZfhueXtJbI9fK8J7SftJbL8y3RS0V7p9KFFFFABRRRQAUUUUAFFFFABRRRQAlLRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUlFV72CS5spoIpmhd0KrIo5U+tJiZwfj3x8NKV9L0qQNesMSyjkQj0H+1/KvI0Rp3aaZ2bJyzE5LGtjxD4W1LQb9xfK0kTNlbgch8+/rWUTkY6AdBXzuLrVJztLQ+Tx9erOpaenkK7lgABhR0AptFFcR54UUUUAFFFFAEttczWdzHc27lJY2DIw7GvffCniGLxFo0d0uFmX5Jk/ut/h3r59rpfBPiFtA11DIx+yXGI5h2Ho34V3YHEOlOz2Z6WW4t0anLLZnvNLTVYOoZTkEZBFBIAya+iPqzO17WbfQdIn1C5b5I1+Ve7N2A+pr5w1TVLjWNTnv7pt0srbj6AdgPYV1HxJ8V/wBuax9htpM2NoxAweJH7t/QVxkKGR/bvXiY2vzy5Vsj53McT7SXKtkWIEwN569qlo4xxRXlt3Z4zd2FFFFIQUdeBRXYfD7w3/bWsC6nTNpakMc9GbsK1pUnUmoo2oUZVqihE7v4e+GRouk/a7hP9MugGbI5RewrtKAAAAOlFfT06apxUUfZUaUaUFCPQWikpa0NAooooAKKKKACiiigAooooAKKSloAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooASloooAKSlooAr3dnBfW7QXMSyxMMFWGa8y8RfDF0L3GjtuXr5LdR9DXqtJWFbD06ytJHNiMJSrq00fM9zaz2czQ3EbRyKcFWGMVDXt/jTwjHrdm91axgX0YyB08wen19DXis0LQuykEYJBBGCD6H3rwMThZUJeR8vjMHPDSs9V3IqKWiuU4xKKKKACiiigD2v4da/wD2roQtZnzc2mEOTyy9j/SoPiZ4rGgaKbO3fF7dgquDyidz/SvNvCmv/wDCO65HeOW+zkFZlHda5bxT4lm8SeILi/lJCs22JP7qDoK9qlinOhbrsfRUca54bl+1sVlYu/qTWnDH5ceO561S02LcPNYfStGvLqy1seLXlryoSiiisTAKKKKAJrO0mvryG1t0LzTMEUD1NfQnh7RYdB0aCxiAyoy7f3m7muF+F3hwBX125Tk5jtgR27t/T869Pr3svw/JD2j3Z9NleF9nD2kt3+QUtFFekesFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABSUtFABRRRQAlLRRQAlFLRQAlFLRQAleX/Evwx5W7X7OLKHAvI1HbtIPcd69QqOeGO4geGVA8bqVZSOCDWValGrBxZjiKEa1NwkfM7LtIwcqeQfUUlbHiXQn8O69NprZNu+ZLVz/AHT/AA/hWMRivmKtN05OLPja1J0puEgooorMyCiiigA61zlzZFdV8sfdc7hXR0xokaVZCPmXoa1pVeRs3oVvZthEgijVB0Ap9FFZN3MW7u4lFFFABWn4f0eXXdagsIwdrnMjD+FB1NZley/DXw9/ZukHUJ0xc3YyMjlU7CuvB0PbVEuiO3AYb29VJ7Lc7O0tYrK0itoECRRKFUDsBU9FLX0iVtEfXJWVkFFFFMYUUUUAFFFFABRRRQAUUUlAC0UUUAFFFFABRRSUALRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFFFFABRRRQAUUUlAC0UUUAFFFFABRRSUALSUtJQBx/xE8Of274eeWFP9MtP3sRHU+orxFX86MSdG6MPQ19PMAykHoa8B8b6J/wAI94qlVVxZ3v7yP0BPUfnXlZjQuvaI8TNsNzL2qOfooIIJB60V4h86FFFFABRRSUDFpKKKACiijNOwG/4P0Jtf8QQwMD5EZ8yY/wCyO34178iLHGqIAFUYAHYVyPw90D+x9AWeVMXN1iR8jkL2FdhX0eCoeyp67s+sy7DexpK+7ClpM0V2HoC0UlFAC0UlFAC0UlFAC0UmaKAClpKKAFopKM0ALRSZozQAtFJmigBaKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigBKWiigAooooAKKKKACiiigAooooAKKKKACiiigBK474keH/AO2/DEskSZubTMseOpA6j8v5V2OaawDKQRkHqDUTipxcWRUgpxcX1PmGKTzoFc/eHyt9adWr4s0Y+HPF11ZgYtbj95Ce20nj8jkVknivmK1Nwm0z4zEUnTqOLFpKKM1iYhRSZpKYDs0maTNJmgY7NdF4J0M694khidc20H72Y9sDoPxP9a5vNe3/AA60P+yPDi3Eq4ubzEr56hf4R+XP412YKj7Sqr7I78vw/tqyvsjsRhQABgClzTM0Zr6I+sH5ozTM0ZoAfmjNMzRmgB9JTc0ZoAdmjNNzRmmA/NGaZmlzQA7NGabmjNADs0U3NGaAHZopM0ZoAWikzRmgB1FNpc0ALRSUUALRmkzRmgBc0uabS0AFLSUUALRSUUALRSUUALRSZpaACiiigApKWigAopKWgAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiikoAWikzRmgBaKbmjNAC0ZpM0maQDqSm5o3UAOzSZpu6k3UAcD8WdB/tHw6upQrm4sG3HHUxn735cH8DXkCSiaJJO54P1r6YuIo7m3kglUNHIpVlPcEV816jp76F4gvtIlziNz5ZPdeqn8q8rMKN/fR4ea0L2qIZmjNMzRurx7Hg2HZpM03dSbqdh2HZozTN1a+geHNQ8R3YhtI8RA/vJm+6g/x9quEHJ2ii4U5TfLFalzwfoEmv65FGUJtYmDzt2wO34174uEUKowAMAVi+HtBtPDumraW3zMeZJD1c1rb69/CYf2MLPdn1GBwvsKdnu9yXNGai30b66jtJc0ZqLdRuoAlzRmo91G6gZJmjNR7qXdTAkzRmo80uaAH5ozTM0uaAH5ozTM0uaAHZpc0zNLmgB2aM03NGaAHZpc03NGaAHZopKM0AOopuaWgBc0ZpM0uaAClpKKAFopM0tABS0lFAC0UmaWgAooooAKKKKAFooooAKKSloAKKKKACiiigAooooAKKKKACiiigAopKKAFpKKSgBc0UmaTNAC5ozTc0maQDs0hNNzSFqAHk0m6oy1IWoAk3Um6oi1NL0CJS1IXqEyUwyUATl6QyVWMvvUZmHrQBbMleSfGDSvLmstchXn/Uykfmp/mK9NM/vWH4rsl1rw3fWRALNGWT2Ycisq0OeDRhXh7Sm4nh/mBwHHRhmk3VQsZiY3hfh4z0/nVkvXz86fLKx8tUpcsmiXdSFqiDFmCgEknAA716B4U8FoDHf6uv8AtR25/m3+FXSoSqOyNaGFnWlaJT8J+CLnW3W6vd0FiDnOMNJ9P8a9fsbW1020S1s4VhhQYCr/ADNU1uFVQq4VQMADoKcLkHvXt0MPGitNz6HDYWFBabmn5lHmVni4HrThN71udRe30u+qYl96eJPegC1vpd1Vw9ODUAT7qN1RBqUGmMlzS5qPNKDQBJmlzUeaXNAEmaM0zNLmgB+aXNMzS0AOzS5plLQA7NLTaWgBaXNNzS0ALS5ptLQAtLSUUALmlpKKAFzS0lFAC0UlLQAUtJRQAtFJS0AFFFFAC0UlLQAtFFFABRRRQAUUUUAFFFFABRRRQAUUUUAFJRRQAUlFFABSZopKACkzRSGgAJpM0E000ABNNJpTTTSAQmmlqDTTQAhamF6GqJjQKwM9RNJQxqu5NAWHNNUD3GO9RSM1UZnftQFi292B3qpLqKL1cD8azLkzHOM1iXkFy+fvUCscZ4s0iHTNZOoWThrW4Y+YmeY2P9KxtzFtoGTXU6jo9xcIylGYHtXM3HhvVzN8ofb2rjq4RTlc4a2BjUlc6rw6unaawurh1luv4R1Cf/XrrU8Swt0avObLw3qPG8NXRWfh+4XG4muqFOMFaJ2U6UaceWKOsj11W6Grceqbqw7bR3XGTWrBp+3FWaGlHfFqspck1TitdtXI4cUAWUlJqdXNQJHU6rQBKrGpQaiUVKKAJAacDTBTxQA4GnA00U4UAOpRTRS0AOpaQUtAC0tJS0ALS0lLQAUtIKWgBaBRQKAFooooAWlpKBQAtFFFAC0UUUALRSUtABS0lLQAUUUUALRSUtABRRRQA6iiigBKWiigAooooAKKKKACiiigApM0UUAFJS0lABSGlpDQAUlFFACUlLSUANNIadSUANppp+KaRQAwimEVIRSEUAQsKjK1YK0wrQBVZKiaOrpSmGOgDPaEHtUL2wPatQx0wxe1IRkNZqe1RNYqf4RW0Yfak8n2pCMM2Cf3R+VNOnIf4B+VbvkD0pPIHpQBg/2Yn90U9dPA/hrc8gelHke1AGQtmB2qVbUDtWn5A9KcIfagDPW39qlWH2q6IvanCL2oAqLFipBHVkR04R0wIAlPC1KEpwSgZEFpwFSbKXbQAwClAp+2l20ANApcU7FLigY3FLilxS4oATFLilxRigBKXFLilxTASilxS4oASloxRQAUUtFABS0UUAFFFLQAUUUUAApaKKAClpKWgAooooAKWkpaACiiigB1FFFABRRRQAUUUUAFFFFABSUtFACUUUUAFJS0UAJRRRQAlJTqTFACUlOpMUANpMU6jFADMUmKfikxSAZikxT8UYoAj20m2pMUYoEQ7aQrU2KTbQBDspNlT7aTbQBBspPLqxto20AV/Lo8urG2jbQBX8ul8up9tG2gCDZS7Km20baAsRbKNlTbaNtAEWyl21JtpcUAR7aXbT8UYoAZtpcU/FGKAGYpcU7FGKAG4pcU7FGKAG4oxTsUYoATFGKdijFAxMUYpcUtADcUtLijFACYpaMUtMBKKWigBKWjFFABRS0UAFFFGKACjFLRQAUUUUAFFFLQAUUUUAFFFFADqKKKACiiigAooooAKKKKACiiigAooooATFFLRQAlFFFACUUtFACUlOpKQCYpMU7FFADcUmKdijFADcUmKfijFAhmKMU7FGKAG4pMU/FGKAGYoxT8UmKAG4pMU/FGKAGYoxT8UYoAZilxTsUYoAbijFOxS4oAZijFPxSYoAbijFPxRigBmKMU/FGKBjcUYp2KMUANxRinYoxQIbijFOxRigY3FGKdRigBMUYpaKAExRS0UAJRilooASlpaKAEopaKYCUYpaKAExS0UYoAKKKWgBMUUtFABRRRQAUUUUAFFLRQAmKWiigBaKKKAEooooAWiiigAooooAKKKKACiiigAooooAKKKKACiiigBKKKKADFFFFABRRRSATFGKKKADFFFFABijFFFABijFFFABiiiigAxRiiigAxRiiigAxRiiigAxRiiimAYoxRRQAUUUUAFFFFACUuKKKQCUtFFACYoxRRQAYoxRRQAUUUUALSUUUAFLRRQAUUUUAFFFFABRRRQAUUUUwCiiigAxRRRQAtFFFABRRRQAUUUUAf/9k="
}
in the client you can do this (in python for continuity)
import base64
myFile = open("mock.jpg", "wb")
img = base64.b64decode(value) #value is the returned string
myFile.write(img)
myFile.close()

Did you try converting the image to base64 string and send it as an argument of your request on the client side?
So you will be able to do that on the server side :
#strArg is the Base64 string sent from the client
img = base64.b64decode(strArg)
filename = 'someFileName.jpg'
with open(filename, 'wb') as f:
f.write(img)
#then you can save the file to your BlobStore

Cloud Storage API requests from GAE - 403 Access not configured

My GAE app is trying to manipulate files stored on Google Cloud Storage.
The files are stored in the default bucket for my app. I already managed to read/write files to that bucket using the GCS Python Client Library (https://developers.google.com/appengine/docs/python/googlecloudstorageclient/).
Unfortunately it does not support copy. Instead, I'm trying the JSON API with the API Client Library (https://google-api-client-libraries.appspot.com/documentation/storage/v1/python/latest/storage_v1.objects.html) and service account (https://developers.google.com/api-client-library/python/guide/google_app_engine#ServiceAccounts)
So far I'm getting an error 403 when requesting the cloud storage url.
Here's the code:
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/devstorage.read_write')
http = credentials.authorize(httplib2.Http(memcache))
service = discovery.build('storage', 'v1', http=http, developerKey='api_key_generated_from_the_dev_console')
bucket_name = app_identity.get_default_gcs_bucket_name()
# I'm planning to batch multiple requests, although there is just one in this example
batch = BatchHttpRequest()
# process_list_response outputs the exception if any
batch.add(service.objects().list(bucket=bucket_name), callback=process_list_response)
batch.execute(http=http)
Here's the log:
URL being requested:
https://www.googleapis.com/discovery/v1/apis/storage/v1/rest?userIp=x.x.x.x
Attempting refresh to obtain initial access_token
URL being requested:
https://www.googleapis.com/storage/v1/b/xxx.appspot.com/o?alt=json
HttpError 403 when requesting
https://www.googleapis.com/storage/v1/b/xxx-dev.appspot.com/o?alt=json
returned "Access Not Configured. Please use Google Developers Console
to activate the API for your project."
Here's what I've done in the dev console:
Google Cloud Storage and Google Cloud Storage JSON API are switched
to ON.
I created an API key which I use to build the service (is it necessary since I also use Oauth?)
Under Permissions, I added a member for my app with the email xxx#appspot.gserviceaccount.com
How can I make this work?

Posting this as an answer as it seems that my edit (we work together) was silently rejected, and a comment is too limited. This is not an answer but that is expanding the question.
Simpler example with a single http request. It seems that the JSON API is simply not working outside the API explorer. The XML/REST API works and returns a list of files in the bucket.
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/devstorage.read_write')
http = credentials.authorize(httplib2.Http(memcache))
bucket_name = app_identity.get_default_gcs_bucket_name()
# This works (200 with list of files in the content)
request_url = 'http://commondatastorage.googleapis.com/' + bucket_name
response, content = http.request(request_url, method="GET")
# This doesn't work (403, Access not configured)
request_url = 'https://www.googleapis.com/storage/v1/b/' + bucket_name + '/o?alt=json'
response, content = http.request(request_url, method="GET")
# This doesn't work (403, Access not configured), the key and project id header seem useless.
request_url = 'https://www.googleapis.com/storage/v1/b/' + bucket_name + '/o?alt=json&key=' + API_KEY
response, content = http.request(request_url, method="GET", headers={'x-goog-project-id': PROJECT_ID})
Also, looking at the code of AppAssertionCredentials, we can see:
kwargs: optional keyword args, including:
service_account_id: service account id of the application. If None or
unspecified, the default service account for the app is used.
self.service_account_id = kwargs.get('service_account_id', None)
Passing anything as service_account_id argument results in an exception:
Traceback (most recent call last):
File "/base/data/home/apps/.../1.37.../backup.py", line 61, in get
response, content = http.request(request_url, method="GET")
File "/base/data/home/apps/.../1.377.../oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/.../1.37.../oauth2client/client.py", line 491, in new_request
self._refresh(request_orig)
File "/base/data/home/apps/.../1.37.../oauth2client/appengine.py", line 197, in _refresh
raise AccessTokenRefreshError(str(e))
AccessTokenRefreshError
I have tested to pass the value returned by app_identity.get_service_account_name(), that doesn't work. (even though the documentation says it will use "the default service account for the app" if it is not set).
I have tested to pass the service account email found in the developer console that has the form: 3....-v0....#developer.gserviceaccount.com. Same token exception.
So, why are we getting a 403 Access not configured when the Cloudstorage JSON API is clearly enabled under our api/services?
And why is passing a service_account_id to AppAssertionCredentials failing with a AccessTokenRefreshError?
Edit:
The solution was ridiculous: turn OFF the Google Cloud Storage API, and turn it back ON.
I assume that the app was a "legacy" app, and doing so made the last bullet point 12 work here: https://developers.google.com/appengine/docs/python/googlecloudstorageclient/activate