I am trying to upload some picture to Google App Engine using the Blobstore.
And some of the files contain non-ascii characters.
When I download these files, the filename for these downloaded file appeared to show the "key" in blobstore, rather than the original file name.
My site is http://wlhunaglearn.appspot.com/
I already add a save_as=blob_info.filename in my BlobstoreDownloadHandler , but it failed when the file name contains non-ascii characters.
Any suggestions?
Thanks in advance.
The following is my main.py file
# -*- encoding: utf-8 -*-
import os
import urllib
import webapp2
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
class MainHandler(webapp2.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" multiple name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
blob_info = upload_files[0]
self.redirect('/serve/%s' % blob_info.key())
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info, save_as=blob_info.filename)
app = webapp2.WSGIApplication([('/', MainHandler),
('/upload', UploadHandler),
('/serve/([^/]+)?', ServeHandler)],
debug=True)
Searching all the posts, I finally got a hint from an Answer to another Question
I figure out the correct way to display non-ascii file name is to
add a urllib.quote function to the last line of ServeHandler class
Therefore the ServeHandler class will be:
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info, save_as=urllib.quote(blob_info.filename.encode('utf-8')))
Related
I have the following code (taken from the sample code) to upload files directly to GCS but it does not preserve the file name of the original file. Instead it stored using a hash as the filename like
'L2FwcGhvc3RpbmdfcHJvZC9ibG9icy9BRW5CMlVxd3BLMk1ZVVg0SVVzSTlUQTMzRzEyVzQ3ZHQtQy1JSjh6ZU5aT0hWc3FzZ2k0NXpfNXZBUTZDRThSQTFNNi0xX0dEY1M4MEFwUFJYbGt4cUkxNkRpeGp6M0VUQS51QlBNM3BJNG9qXy0zVHZF'
The problem is that I need to show all the files uploaded to GCS to users so need to preserve the file identity to distinguish the files. They won't be able to identify their uploads from hashes
Here's the sample code... how do I ensure that it uploads with the file name provided? For instance, if I upload a file 'log.csv', it is uploaded as 'log.csv' rather than a hash which doesn't say if it's the file or not.
import os
import urllib
import webapp2
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
class MainHandler(webapp2.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload', gs_bucket_name='uploads')
self.response.out.write('<html><body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
blob_info = upload_files[0]
self.redirect('/serve/%s' % blob_info.key())
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)
app = webapp2.WSGIApplication([('/', MainHandler),
('/upload', UploadHandler),
('/serve/([^/]+)?', ServeHandler)],
debug=True)
The filename and other metadata are in the BlobInfo object in the datastore, not in the blobstore per se. See https://cloud.google.com/appengine/docs/python/blobstore/blobinfoclass for all details of the BlobInfo class (and https://cloud.google.com/appengine/docs/python/tools/webapp/blobstorehandlers for how the provided handlers deal with it on both upload and download).
So, instead of requiring the user to pass you the blob key (so you can use blobstore.BlobInfo.get(resource) directly on the request), you can allow the user to pass you the filename, use BlobInfo.gql to give you all the blobinfo objects for the given filename (no guarantee of uniqueness, of course -- unless you enforce it yourself in the app at upload time by rejecting duplicate filenames), then use the key() method of those blob info objects to get the required blob keys.
Access to the datastore (for blobs' metadata) is faster than access all the way to the blobstore, so e.g to present a list of uploaded files, BlobInfo.all() (returning a query to which you can add filters) will let you present filenames & other metadata (content type, creation datetime, size in bytes) to your users, for them to select which blob(s) they want to retrieve.
I have setup a simple python project on GAE, the user picks a file and it uploads into Google Cloud Storage. This all works fine, but I am getting an upload time of 10-11 seconds when I am uploading a 100kb file. How can I reduce this?
Extra info:
I have a 100Mbis upload speed and the result is the same with a 4kb image, the latency of the first post id about 6 seconds
import os
import urllib
import webapp2
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
class MainHandler(webapp2.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload', gs_bucket_name='bucket_name')
self.response.out.write('<html><body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
blob_info = upload_files[0]
self.redirect('/serve/%s' % blob_info.key())
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)
app = webapp2.WSGIApplication([('/', MainHandler),
('/upload', UploadHandler),
('/serve/([^/]+)?', ServeHandler)],
debug=True)
I just follow GAE's document (https://developers.google.com/appengine/docs/python/blobstore/#Python_Uploading_a_blob) to write upload handler to upload blobstore, when I select one file on computer and click Submit button on HTML page, it will show 'The url "/upload" does not match any handlers.'
Any comments is appreciated.
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-Type'] = 'text/html; charset=utf-8'
upload_url = blobstore.create_upload_url('/upload')
logging.info(upload_url)
self.response.out.write('<html><body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
logging.info('Upload handler')
upload_files = self.get_uploads('file') # 'file' is file upload field in the form
blob_info = upload_files[0]
logging.info(upload_files)
self.redirect('/serve/%s' % blob_info.key())
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
logging.info(resource)
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)
application = webapp2.WSGIApplication([
('/', MainPage),
('/upload', UploadHandler),
('/serve/([^/]+)?', ServeHandler),
], debug=True)
[Update1]
After I click submit button, I check dev-server Blobstore Viewer, I found the file has been uploaded there, however, my chrome browser still show 'The url "/upload" does not match any handlers.'. This is why?
I want to ask my question by self that maybe someone encounter similar issue as me.
After I change from
- url: /
script: AppWS.application
to
- url: (/.*)*
script: AppWS.application
everything is OK.
I'm using this link as an example to uploading images:
https://gist.github.com/jdstanhope/5079277
My HTML code:
<form action="/upload_image" method="post" id="form1" runat="server">
<div class="fileButtons">
<input type='file' id="imgInp" name="imgInput" accept="image/*"/><br><br>
<input type='button' id='remove' value='Remove' />
</div></form>
main.py:
class SetImage(webapp2.RequestHandler):
def post(self):
logging.debug("test")
id = str(self.request.get('id'))
image = self.request.get('imgInput')
app = webapp2.WSGIApplication([('/upload_image', SetImage),
('/', MainPage)], debug=True)
But when I add an image, nothing is being done, and the log console doesn't print:
logging.debug("test")
The recommended way of uploading images to GAE is by using blobstore.
Here is a quick breakdown of the doc to help you achieve this fast:
Imports:
import os
import urllib
import webapp2
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
Serve the form HTML. this form performs the POST to GAE with the selected files data:
class MainHandler(webapp2.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html><body>')
self.response.out.write('<form action="%s" method="POST"
enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File:
<input type="file" name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
Add a handler for receiving the POST data (binary content of the file). the last line in this function is to redirect the response to the location from where the file could be downloaded:
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
# 'file' is file upload field in the form
upload_files = self.get_uploads('file')
blob_info = upload_files[0]
self.redirect('/serve/%s' % blob_info.key())
Add handler to serve the image that you uploaded using the UploadHandler described above:
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)
And finally, define the routes for the app:
app = webapp2.WSGIApplication([('/', MainHandler),
('/upload', UploadHandler),
('/serve/([^/]+)?', ServeHandler)],
debug=True)
Regarding the issue of your log statement not firing: The default log level for dev_appserver.py is info, which you can override with the --dev_appserver_log_level flag.
Reference the new dev_appserver documentation.
To access uploaded files from a multipart/form-data POST request, the webapp2 documentation states that they're available in request.POST.
This is what I got from flex 4 file reference upload:
self.request =
Request: POST /UPLOAD
Accept: text/*
Cache-Control: no-cache
Connection: Keep-Alive
Content-Length: 51386
Content-Type: multipart/form-data; boundary=----------ei4cH2gL6ae0ei4ae0gL6GI3KM7ei4
Host: localhost:8080
User-Agent: Shockwave Flash
------------ei4cH2gL6ae0ei4ae0gL6GI3KM7ei4
Content-Disposition: form-data; name="Filename"
36823_117825034935819_100001249682611_118718_676534_n.jpg
------------ei4cH2gL6ae0ei4ae0gL6GI3KM7ei4
Content-Disposition: form-data; name="Filedata"; filename="36823_117825034935819_100001249682611_118718_676534_n.jpg"
Content-Type: application/octet-stream
���� [AND OTHER STRANGE CHARACTERS]
My class:
class Upload(webapp.RequestHandler):
def post(self):
content = self.request.get("Filedata")
return "done!"
Now what I'm missing in the Upload class in order to save that file to disk?
i have in the content var some strange characters (viewing in debug).
An App Engine application cannot:
write to the filesystem. Applications must use the App Engine
datastore for storing persistent data.
What you need to do is presenting a form with a file upload field to the user.
When the form is submitted, the file is uploaded and the Blobstore creates a blob from the file's contents and returns a blob key useful to retrieve and serve the blob later.
The maximum object size permitted is 2 gigabytes.
Here is a working snippet that you can try as is:
#!/usr/bin/env python
#
import os
import urllib
from google.appengine.ext import blobstore
from google.appengine.ext import webapp
from google.appengine.ext.webapp import blobstore_handlers
from google.appengine.ext.webapp import template
from google.appengine.ext.webapp.util import run_wsgi_app
class MainHandler(webapp.RequestHandler):
def get(self):
upload_url = blobstore.create_upload_url('/upload')
self.response.out.write('<html><body>')
self.response.out.write('<form action="%s" method="POST" enctype="multipart/form-data">' % upload_url)
self.response.out.write("""Upload File: <input type="file" name="file"><br> <input type="submit"
name="submit" value="Submit"> </form></body></html>""")
class UploadHandler(blobstore_handlers.BlobstoreUploadHandler):
def post(self):
upload_files = self.get_uploads('file')
blob_info = upload_files[0]
self.redirect('/serve/%s' % blob_info.key())
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)
def main():
application = webapp.WSGIApplication(
[('/', MainHandler),
('/upload', UploadHandler),
('/serve/([^/]+)?', ServeHandler),
], debug=True)
run_wsgi_app(application)
if __name__ == '__main__':
main()
EDIT1:
in your specific case you could use a BlobProperty (limited to 1MB) to store your request:
class Photo(db.Model):
imageblob = db.BlobProperty()
then adapt your webapp.RequestHandler to save your request:
class Upload(webapp.RequestHandler):
def post(self):
image = self.request.get("Filedata")
photo = Photo()
photo.imageblob = db.Blob(image)
photo.put()
EDIT2:
You don't need to change your app.yaml, just add a new handler and map it in your WSGI.
To retrieve the stored photo you should add another handler to serve your photos:
class DownloadImage(webapp.RequestHandler):
def get(self):
photo= db.get(self.request.get("photo_id"))
if photo:
self.response.headers['Content-Type'] = "image/jpeg"
self.response.out.write(photo.imageblob)
else:
self.response.out.write("Image not available")
then map your new DownloadImage class:
application = webapp.WSGIApplication([
...
('/i', DownloadImage),
...
], debug=True)
You would be able to get images with a url like:
yourapp/i?photo_id = photo_key
As requested, if for any odd reason you really want to serve your images using this kind of url www.mysite.com/i/photo_key.jpg , you may want to try with this:
class Download(webapp.RequestHandler):
def get(self, photo_id):
photo= db.get(db.Key(photo_id))
if photo:
self.response.headers['Content-Type'] = "image/jpeg"
self.response.out.write(photo.imageblob)
else:
self.response.out.write("Image not available")
with a slightly different mapping:
application = webapp.WSGIApplication([
...
('/i/(\d+)\.jpg', DownloadImage),
...
], debug=True)