I need to upload a file/document to Google Docs on a GAE application. This should be simple enough, but I'm having a lot of trouble with the API.
The context:
import gdata.docs.service
client = gdata.docs.service.DocsService()
client.ClientLogin('gmail', 'pass')
ms = gdata.MediaSource(#what goes in here?)
client.Upload(media_source=ms, title='title')
To upload I'm using client.Upload(), which takes a MediaSource (wrapper) object as a parameter. However, MediaSource() seems to only accept a filepath for a document: 'C:/Docs/ex.doc'.
Since I'm on GAE with no filesystem, I can only access the file through the Blobstore or a direct URL to the file. But how do I input that into MediaSource()?
There seems to be a way in Java to accomplish this by using MediaByteArraySource(), but nothing for Python.
If anyone is curious, here's how I solved this problem using the Document List API.
I didn't want to use the Drive SDK since it does complicate a lot of things. It's much simpler with the List API to just authenticate/login without the need for some OAuth trickery. This is using version 2.0.14 of the gdata Python library, which is not the current version (2.0.17), but it seems to have a simpler upload mechanism.
There's also slightly more (still sparse) documentation online for 2.0.14, though I had to piece this together from various sources and trial & error. The downside is that you cannot upload pdf's with this version. This code will not work with 2.0.17.
Here's the code:
import gdata.docs.service
import gdata.docs.data
from google.appengine.api import urlfetch
# get file from url
result = urlfetch.fetch('http://example.com/test.docx')
headers = result.headers
data = result.content
# authenticate client object
client = gdata.docs.service.DocsService()
client.ClientLogin('gmail', 'password')
# create MediaSource file wrapper
ms = gdata.MediaSource(file_handle=result.content,
content_type=headers['content-type'],
content_length=int(headers['content-length']))
# upload specific folder, return URL of doc
google_doc_name = 'title'
folder_uri = '/feeds/folders/private/full/folder:j7XO8SJj...'
entry = client.Upload(ms, google_doc_name, folder_or_uri=secret.G_FOLDER_URI)
edit_url = entry.GetAlternateLink().href
The Google Drive SDK docs include a complete sample application written in Python that runs on App Engine:
https://developers.google.com/drive/examples/python
You can use it as reference for your implementation and to see how to save a file from App Engine.
Related
I'm fairly new to django.
So heroku doesn't support image storage so I will have to use an other container for it. I've found a lot of tutorials for using Amazon S3 but I would like to use dropbox since it's free. Is this possible?
I've found this package https://django-storages.readthedocs.io/en/latest/ but I still don't understand how to use it. If anybody has used it please help me out. Thanks.
Sign up (if you haven’t already), go to the DropBox App Console, create a new application and generate the Access Token.
Use then the Python Dropbox SDK:
dbx = dropbox.Dropbox('access_token')
# create file
filename = '/local_files/file.json'
dbx.files_upload(f.read(), filename, mute=True)
# read file
filename = '/dropbox_root/file.json'
f, r = dbx.files_download(filename)
print(r.content)
You can see the Files on Heroku Medium post to see the details and few other options.
I am unable to retrieve the documents which are available in my collection inside the firestore database. Here is my code.
Every time I run this console dosen't print anything. I am following the documentation avaliable on this link https://firebase.google.com/docs/firestore/query-data/get-data, but it dosen't seems to work.
database_2 = firestore.client()
all_users_ref_2 = database_2.collection(u'user').stream()
for users in all_users_ref_2:
print(u'{} => {}'.format(users.id, users.to_dict()))
Do you have multiple projects? If so, double check that you open a client to the correct project. One quick way to confirm is to pass the project ID to the client:
db = firestore.Client('my-project-id')
Could be an authentication issue, you could download a service account key and use that in your project at the top.
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALs"] = "path/to/key.json"
or as mentioned
database_2 = firestore.Client("<project ID>")
make sure Client is a capital C
The App Engine documentation for the Blobstore gives a pretty thorough explanation of how to upload a file using the BlobstoreUploadHandler provided by the webapp framework.
However, I have a cgi.FieldStorage instance that I would like to store directly into the Blobstore. In other words, I don't need to upload the file since this is taken care of by other means; I just need to store it.
I've been looking through the blobstore module source to try to understand how the upload handler creates/generates blobstore keys and ultimately writes files to the blobstore itself, but I'm getting lost. It seems like the CreateUploadURLResponse in blobstore_service_pb is where the actual write would occur, but I'm not seeing the component that actually implements that functionality.
Update
There is also an implementation for storing files directly into the filesystem, which I think is what the upload handler does in the end. I am not entirely sure about this, so an explanation as to whether or not using the FileBlobStorage is the correct way to go would be appreciated.
After the deprecation of the files API you can no longer write directly to blobstore.
You should write to Google Cloud Storage instead. For that you can use the AE GCS client
Files written to Google Cloud Storage could be served by the Blobstore API by creating a blob key.
In app engine I can serve cloudstorage files like a pdf using the default bucket of my application:
http://storage.googleapis.com/<appid>.appspot.com/<file_name>
But how can I serve local cloudstorage files in the SDK, without making use of a blob_key?
I write to the default bucket like this:
gcs_file_name = '/%s/%s' % (app_identity.get_default_gcs_bucket_name(), file_name)
with gcs.open(gcs_file_name, 'w') as f:
f.write(data)
The name of the default bucket in the SDK = 'app_default_bucket'
In the SDK datastore I have a Kind: GsFileInfo showing: filename: /app_default_bucket/example.pdf
Update and workaround: You can get a serving url for NON image files like css, js and pdf.
gs_file = '/gs/%s/%s/%s' % (app_identity.get_default_gcs_bucket_name(), folder, filename)
serving_url = images.get_serving_url(blobstore.create_gs_key(gs_file))
UPDATE I found this feature to serve cloudstorage files using the SDK:
This feature has not been documented yet.
http://localhost:8080/_ah/gcs/app_default_bucket/filename
This meands we do not need the img serving url to serve NON images as shown below !!!
To create e serving url for cloudstorage files like images, css, js and pdf's in the default_bucket, I use this code for testing(SDK) and GAE production:
IMPORTANT: the images.get_serving_url() works also for NON images in the SDK!!
In the SDK you stll need the blobstore to read a blob and create a serving url for a cloudstorage object.
I also added the code to read, write and upload cloudstorage blobs in the SDK and GAE production.
The code can be found here.
This is the value that you see in the Development mode from app_identity_stub.py:
APP_DEFAULT_GCS_BUCKET_NAME = 'app_default_bucket'
The comments in this file explain it:
This service behaves the same as the production service, except using
constant values instead of app-specific values
You should get the correct URL in your production code.
EDIT:
This is from the support forum:
In development mode, the app engine tools simulate Google Cloud
Storage services locally. Objects in that simulated environment are
non-persistent so your app is failing because the desired object
doesn't exist in the local store. If you first create (and optionally
write to) the object you're trying to read, it should work fine in dev
mode (it did for me). Of course, objects in the production service are
persistent so there's no need for that extra step when running your
app in production mode (assuming the object already exists).
Hope that helps,
Marc Google Cloud Storage Team
This means you have to write a file first, then you can use it. If I understand correctly, you can use any bucket name for this purpose, including 'app_default_bucket'.
I was here earlier looking for answers and just wanted to share what I found, now that I have it working.
You can do this now, and it's only a little painful. Tricking the image or blobstore API isn't supported and doesn't seem to work any longer.
See:
https://cloud.google.com/storage/docs/access-control/signed-urls
https://cloud.google.com/storage/docs/access-control/create-signed-urls-gsutil
If you sign your URLs, you can give auto-expiring links to your content, for anonymous or paywalled consumption. You wouldn't want to serve your whole site this way, but for a PDF or whatnot, this is a valid and semi-secure option.
Missing from the documentation, you might need to drop the newline for the canonical extended headers. The storage endpoint will tell you what it expects when the signature is bad.
Also, your host should be: https://storage-download.googleapis.com/
If you're using App Engine, then the GoogleAccessId is: <projectname>#appspot.gserviceaccount.com
See: app_identity.get_service_account_name()
Example of how to generate the signature:
from google.appengine.api import app_identity
def signFile(path, verb='GET', md5='', contentType='',
expiration=''):
signatureRequest = '{}\n{}\n{}\n{}\n{}'.format(
verb, md5, contentType, expiration, path)
return app_identity.sign_blob(signatureRequest)
That returns a tuple of (privateKey, binarySignature).
Now you need to construct the URL. The signature should be base64 encoded, then urlencoded. See the following for how to finish constructing the URL. You should probable use the download host mentioned above.
Example URL from the docs:
https://storage.googleapis.
com/example-bucket/cat.jpeg?GoogleAccessId=example#example-project.iam.gservicea
ccount.com&Expires=1458238630&Signature=VVUgfqviDCov%2B%2BKnmVOkwBR2olSbId51kSib
uQeiH8ucGFyOfAVbH5J%2B5V0gDYIioO2dDGH9Fsj6YdwxWv65HE71VEOEsVPuS8CVb%2BVeeIzmEe8z
7X7o1d%2BcWbPEo4exILQbj3ROM3T2OrkNBU9sbHq0mLbDMhiiQZ3xCaiCQdsrMEdYVvAFggPuPq%2FE
QyQZmyJK3ty%2Bmr7kAFW16I9pD11jfBSD1XXjKTJzgd%2FMGSde4Va4J1RtHoX7r5i7YR7Mvf%2Fb17
zlAuGlzVUf%2FzmhLPqtfKinVrcqdlmamMcmLoW8eLG%2B1yYW%2F7tlS2hvqSfCW8eMUUjiHiSWgZLE
VIG4Lw%3D%3D
I hope this helps someone!
Oh yeah, you only need to do all the signature stuff if your bucket isn't publicly accessible (read-all).
I'm especially interested in solutions with source code available (Django independency is a plus, but I'm willing to hack my way through)
You can, of course, write your own handler. Other than that, your options currently are limited to:
gae-rest, which provides a RESTful interface to the datastore.
approcket, a tool for replicating between MySQL and App Engine.
The amusingly named GAEBAR - Google App Engine Backup and Restore.
Update: New version of Google AppEngine supports data import to and export from the online application natively. In their terms this is called upload_data and download_data respectively (names of subcommands of appcfg.py).
Please refer to Google documentation how to export and import data from/to GAE. This is probably the better way to do it today.
My old answer is below:
I use to_xml() method of the Model class to export the datastore.
class XmlExport(webapp.RequestHandler):
def get(self):
objects=MyModel.all().fetch(1000)
xml='<?xml version="1.0" encoding="UTF-8"?>\n<site>\n'
for o in objects:
xml = xml + o.to_xml()
xml = xml + '</site>'
self.response.headers['Content-Type']='text/xml; charset=utf-8'
self.response.out.write(xml)