I've been following the tutorials from google app engine for uploading an image
I've set up a simple HTML page for uploading a file using the "file" input type, and the name of the element is "file"
The form enctype is multipart/form-data and method is 'post'
Following the example, I would store the image data as a blob and store it with my object
myfile = MyFile()
myfile.filedata = db.Blob(self.request.get('file'))
myfile.filename = self.request.get('filename')
myfile.put()
But when I look at what's stored in filedata via datastore viewer, it wasn't binary image data, but instead just
236 bytes, SHA-1 = 63540e4ca6dba45e9ff3553787f7f23d330b7791
When the image I uploaded is definitely larger than 236 bytes.
Should the snippet from above retrieve the entire image and put it in a blob?
It seems like all that's being stored in the blob is the request header.
Related
I'm currently using the Azure Blob Storage SDK for Python. For my project I want to read/load the data from a specific blob without having to download it / store it on disk before accessing.
According to the documentation loading a specfic blob works for my with:
blob_client = BlobClient(blob_service_client.url,
container_name,
blob_name,
credential)
data_stream = blob_client.download_blob()
data = data_stream.readall()
The last readall() command returns me the byte information of the blob content (in my case a image).
With:
with open(loca_path, "wb") as local_file:
data_stream.readinto(my_blob)
it is possible to save the blob content on disk (classic downloading operation)
BUT:
Is it also possible to convert the byte data from data = data_stream.readall() directly into an image?
It already tried image_data = Image.frombytes(mode="RGB", data=data, size=(1080, 1920))
but it returns me an error not enough image data
Here is the sample code for reading the text without downloading the file.
from azure.storage.blob import BlockBlobService, PublicAccess
accountname="xxxx"
accountkey="xxxx"
blob_service_client = BlockBlobService(account_name=accountname,account_key=accountkey)
container_name="test2"
blob_name="a5.txt"
#get the length of the blob file, you can use it if you need a loop in your code to read a blob file.
blob_property = blob_service_client.get_blob_properties(container_name,blob_name)
print("the length of the blob is: " + str(blob_property.properties.content_length) + " bytes")
print("**********")
#get the first 10 bytes data
b1 = blob_service_client.get_blob_to_text(container_name,blob_name,start_range=0,end_range=10)
#you can use the method below to read stream
#blob_service_client.get_blob_to_stream(container_name,blob_name,start_range=0,end_range=10)
print(b1.content)
print("*******")
#get the next range of data
b2=blob_service_client.get_blob_to_text(container_name,blob_name,start_range=10,end_range=50)
print(b2.content)
print("********")
#get the next range of data
b3=blob_service_client.get_blob_to_text(container_name,blob_name,start_range=50,end_range=200)
print(b3.content)
For complete information you can check the document with Python libraries.
I need to upload an image string (as the one you get from requests.get(url).content) to google drive using the PyDrive package. I checked a similar question but the answer accepted there was to save it in a temporary file on a local drive and then upload that.
However, I cannot do that because of local storage and permission restrictions.
The accepted answer was previously to use SetContentString(image_string.decode('utf-8')) since
SetContentString requires a parameter of type str not bytes.
However the error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte came up, as in the comments on that answer.
Is there any way to do this without using a temporary file, using PIL/BytesIO/anything that can convert it to be uploaded correctly as a string or somehow using PIL manipulated as an image and uploaded using SetContentFile()?
A basic example of what I'm trying to do is:
img_content = requests.get('https://i.imgur.com/A5gIh7W.jpeg')
file = drive.CreateFile({...})
file.setContentString(img_content.decode('utf-8'))
file.Upload()
When I saw the document (Upload and update file content) of pydrive, it says as follows.
Managing file content is as easy as managing file metadata. You can set file content with either SetContentFile(filename) or SetContentString(content) and call Upload() just as you did to upload or update file metadata.
And, I searched about the method for directly uploading the binary data to Google Drive. But, I couldn't find it. From this situation, I thought that there might not be such method. So, in this answer, I would like to propose to upload the binary data using requests module. In this case, the access token is retrieved from the authorization script of pydrive. The sample script is as follows.
Sample script:
from pydrive.auth import GoogleAuth
import io
import json
import requests
url = 'https://i.imgur.com/A5gIh7W.jpeg' # Please set the direct link of the image file.
filename = 'sample file' # Please set the filename on Google Drive.
folder_id = 'root' # Please set the folder ID. The file is put to this folder.
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
metadata = {
"name": filename,
"parents": [folder_id]
}
files = {
'data': ('metadata', json.dumps(metadata), 'application/json'),
'file': io.BytesIO(requests.get(url).content)
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers={"Authorization": "Bearer " + gauth.credentials.access_token},
files=files
)
print(r.text)
Note:
In this script, it supposes that your URL is the direct link of the image file. Please be careful this.
In this case, uploadType=multipart is used. The official document says as follows. Ref
Use this upload type to quickly transfer a small file (5 MB or less) and metadata that describes the file, in a single request. To perform a multipart upload, refer to Perform a multipart upload.
When you want to upload the data of the large size, please use the resumable upload. Ref
References:
Upload and update file content of pydrive
Upload file data of Drive API
I am using the Dropbox API Explorer - get_thumbnail to get a thumbnail of images stored following the get_thumbnail documentation
The response returns a blob in binary. I am attempting to create a image file to store as an imagefield on my django app's database.
I followed this Quickstart to create the image.
# get_thumbnail request
url = "https://content.dropboxapi.com/2/files/get_thumbnail"
# product_path is a variable storing path from list_folder
new_product_path = "{\"path\":\"/%s\"}" %(product_path)
headers = {
"Authorization": "Bearer <api_token_ommitted>",
"Dropbox-API-Arg": new_product_path
}
r = requests.post(url, headers=headers)
thumbnail = Image.open(BytesIO(r.content))
# album is foreign key for Album model
product = Product.objects.create(album=album, name=converted_name, image=converted_url, thumbnail=thumbnail)
print('Product has been created!')
The results of print(r.content):
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x00\x00\x01\x00\x01\x00\x00\xff\xdb\x00C\x00\x06\x04\x05\x06\x05\x04\x06\x06\x05\x06\x07\x07\x06\x08\n\x10\n\n\t\t\n\x14\x0e\x0f\x0c\x10\x17\x14\x18\x18\x17\x14\x16\x16\x1a\x1d%\x1f\x1a\x1b#\x1c\x16\x16 , #&\')*)\x19\x1f-0-(0%()(\xff\xdb\x00C\x01\x07\x07\x07\n\x08\n\x13\n\n\x13(\x1a\x16\x1a((((((((((((((((((((((((((((((((((((((((((((((((((\xff\xc0\x00\x11\x08\x00$\x00#\x03\x01"\x00\x02\x11\x01\x03\x11\x01\xff\xc4\x00\x1f\x00\x00\x01\x05\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\xff\xc4\x00\xb5\x10\x00\x02\x01\x03\x03\x02\x04\x03\x05\x05\x04\x04\x00\x00\x01}\x01\x02\x03\x00\x04\x11\x05\x12!1A\x06\x13Qa\x07"q\x142\x81\x91\xa1\x08#B\xb1\xc1\x15R\xd1\xf0$3br\x82\t\n\x16\x17\x18\x19\x1a%&\'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\x83\x84\x85\x86\x87\x88\x89\x8a\x92\x93\x94\x95\x96\x97\x98\x99\x9a\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xff\xc4\x00\x1f\x01\x00\x03\x01\x01\x01\x01\x01\x01\x01\x01\x01\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\xff\xc4\x00\xb5\x11\x00\x02\x01\x02\x04\x04\x03\x04\x07\x05\x04\x04\x00\x01\x02w\x00\x01\x02\x03\x11\x04\x05!1\x06\x12AQ\x07aq\x13"2\x81\x08\x14B\x91\xa1\xb1\xc1\t#3R\xf0\x15br\xd1\n\x16$4\xe1%\xf1\x17\x18\x19\x1a&\'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x92\x93\x94\x95\x96\x97\x98\x99\x9a\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xff\xda\x00\x0c\x03\x01\x00\x02\x11\x03\x11\x00?\x00\xfa\x16\x8a\xc3_\x15h\xcd\x1c\x8c\xb7\xa8v\x9cm\xc1\x05\xbe\x80\xf5\xa7\xc3\xe2m\x1eB\xa0\xea6\xf1\x92\xa5\xb1+l \x0e\xb9\'\x8f\xd6\xbc\xa5R\r\xd93\xdcxj\xd1Wp\x7fs6h\xacx|K\xa2L3\x0e\xabg \xeb\xf2\xc9\x9a\xcd\xd6|w\xa3i\x9at\xd7m$\xb3\xac\\\x95\x8a2I\xa6\xe7\x14\xec\xd8G\rZ[A\xfd\xc7UEs\xd6\x9e0\xd1n,\x92\xe5\xaf\x16\x10\xc8\x1fd\xa0\x86\xc7\xb0\xefV-\xfcO\xa2\xcd\x07\x9a\xba\x9d\xa0PpCH\x14\x83\xee\x0f\xf3\xe9B\x9c_Pxj\xd1\xde\x0f\xeef\xcd\x15\x917\x894h#2j\x96x<\xe5d\r\xc7\xe1\x9ab\xf8\xa7B.\xe85{-\xeb\xc1\x1eg"\x9d\xd1>\xc2\xae\xfc\xaf\xeeg\xc2\xd3^\xddN\x00\x9a\xe2g\x00`\x06rq#\xbc\xb8\xe33\xc8p0>c\xd3\xd2\xa0\xa2\xbd\xbeT\xba\x18\xacMT\xef\xcc\xfe\xf3F\xcbX\xbb\xb4\x04C)Q\x90\xdcz\xd5\xa4\xf1>\xa6\xa1\xc7\xda\\\xab\x8c2\x93\xc1\xe75\x89EC\xa5\x07\xbaGB\xcc\xb1)[\x98\xd9\xb9\xf1\r\xf5\xc3\xc7#\xce\xfb\xd3\xa6\x0e1U\xe7\xd6/\'bd\x99\xc9 \x83\xcfPz\xd6u\x14\xd5(-\x90\xdeg\x89\x7fh\xb35\xed\xc4\xbc4\xcf\x8fL\xd4+,\x8ar\xae\xc0\xe7<\x1ae\x15VH\xe6\x9e&\xac\xdf4\xa4\xc2\x8a(\xa6b\x14QE\x00\x14QE\x00\x14QE\x00\x7f\xff\xd9'
I have tried to convert bytes data to image as follows, if your data in bytes format, I think following code will help you
bytes_data = b'\xff\xd8\xff\xe0\x00\x10JFIF\x00'
with open("path/image.png", "wb") as f:
f.write(bytes_data)
Add the new path of image to image field.
For more details refer following
How to generate temporary file in django and then destroy
Temporary file with specific file extension in Python 3
Programmatically saving image to Django ImageField
I am uploading a photo to blobstore using plupload. The BlobInfo object has some metadata
content_type: The content type of the blob.
creation: The creation date of the blob, or when it was uploaded.
filename: The file name that the user selected from their machine.
size: The size of the uncompressed blob.
md5_hash: The MD5 hash value of the uploaded blob.
My question is how can I get the other metadata of the photo either from plupload or serverside? Specifically, there is a metadata field of "description" that I need to retrieve.
I would use exif-py as follows:
blob_reader = blobstore.BlobReader(blob_key)
blob_reader_data = StringIO.StringIO(blob_reader.read())
tags = exifread.process_file(blob_reader_data)
the tags object it returns contain the metadata you're looking for.
I am using a Flask app to receive a mutipart/form-data request with an uploaded file (a video, in this example).
I don't want to save the file in the local directory because this app will be running on a server, and saving it will slow things down.
I am trying to use the file object created by the Flask request.files[''] method, but it doesn't seem to be working.
Here is that portion of the code:
#bp.route('/video_upload', methods=['POST'])
def VideoUploadHandler():
form = request.form
video_file = request.files['video_data']
if video_file:
s3 = boto3.client('s3')
s3.upload_file(video_file.read(), S3_BUCKET, 'video.mp4')
return json.dumps('DynamoDB failure')
This returns an error:
TypeError: must be encoded string without NULL bytes, not str
on the line:
s3.upload_file(video_file.read(), S3_BUCKET, 'video.mp4')
I did get this to work by first saving the file and then accessing that saved file, so it's not an issue with catching the request file. This works:
video_file.save(form['video_id']+".mp4")
s3.upload_file(form['video_id']+".mp4", S3_BUCKET, form['video_id']+".mp4")
What would be the best method to handle this file data in memory and pass it to the s3.upload_file() method? I am using the boto3 methods here, and I am only finding examples with the filename used in the first parameter, so I'm not sure how to process this correctly using the file in memory. Thanks!
First you need to be able to access the raw data sent to Flask. This is not as easy as it seems, since you're reading a form. To be able to read the raw stream you can use flask.request.stream, which behaves similarly to StringIO. The trick here is, you cannot call request.form or request.file because accessing those attributes will load the whole stream into memory or into a file.
You'll need some extra work to extract the right part of the stream (which unfortunately I cannot help you with because it depends on how your form is made, but I'll let you experiment with this).
Finally you can use the set_contents_from_file function from boto, since upload_file does not seem to deal with file-like objects (StringIO and such).
Example code:
from boto.s3.key import Key
#bp.route('/video_upload', methods=['POST'])
def VideoUploadHandler():
# form = request.form <- Don't do that
# video_file = request.files['video_data'] <- Don't do that either
video_file_and_metadata = request.stream # This is a file-like object which does not only contain your video file
# This is what you need to implement
video_title, video_stream = extract_title_stream(video_file_and_metadata)
# Then, upload to the bucket
s3 = boto3.client('s3')
bucket = s3.create_bucket(bucket_name, location=boto.s3.connection.Location.DEFAULT)
k = Key(bucket)
k.key = video_title
k.set_contents_from_filename(video_stream)