I'm fairly new to django.
So heroku doesn't support image storage so I will have to use an other container for it. I've found a lot of tutorials for using Amazon S3 but I would like to use dropbox since it's free. Is this possible?
I've found this package https://django-storages.readthedocs.io/en/latest/ but I still don't understand how to use it. If anybody has used it please help me out. Thanks.
Sign up (if you haven’t already), go to the DropBox App Console, create a new application and generate the Access Token.
Use then the Python Dropbox SDK:
dbx = dropbox.Dropbox('access_token')
# create file
filename = '/local_files/file.json'
dbx.files_upload(f.read(), filename, mute=True)
# read file
filename = '/dropbox_root/file.json'
f, r = dbx.files_download(filename)
print(r.content)
You can see the Files on Heroku Medium post to see the details and few other options.
Related
experts, I want to built a automatic trading program in google app engine. How can I import my certification?
I have try to put it in root folder/ in storage/ and read it from storage and yield a new file in tmp folder, however, it did not work, either reply "read-only system" or "file can not found".
Can anybody show me how to import file (by path) in Google app engine?
THX
If you want to import files in Google App Engine using Python, you can put the file in a bucket in Cloud Storage and read the file from there. You may take the reference of this documentation which talks in detail on how you can specify the bucket and read the file from there-in.
You may also have a look at the code sample present in the Stackoverflow case.
I want to upload and retrieve file (that includes image and csv) for my django application.
My project is currently hosted on Google App Engine Flexible. From my understanding, I need to use Google Cloud Buckets for the process. But I could not find relevant material online for the process.
GoogleAppEngineCloudStorageClient API also provides the feature of writing a file and reading it:
cloudstorage.open(filename, mode='r', content_type=None, options=None, read_buffer_size=storage_api.ReadBuffer.DEFAULT_BUFFER_SIZE, retry_params=None)
Please point me to the relevant resources for the same. Since I am new to Django and Google Cloud, I would really appreciate if you could share code snippets with me.
Thanks in advance.
Check the Google storage section of Django Storages. It may help to read generally about the storage system used in Django before going to Google storage in detail.
As per the official documentation of Google App Engine Flexible, you can configure upload and retrieve files in different runtimes environments. I went through the documentation and I found that you need to create a Cloud Storage bucket in order to stock your files.Keep in mind that it's only for static content. I recommend to take a look at this documentation where you may find valuable information.
Please let me know how it goes.
In app engine I can serve cloudstorage files like a pdf using the default bucket of my application:
http://storage.googleapis.com/<appid>.appspot.com/<file_name>
But how can I serve local cloudstorage files in the SDK, without making use of a blob_key?
I write to the default bucket like this:
gcs_file_name = '/%s/%s' % (app_identity.get_default_gcs_bucket_name(), file_name)
with gcs.open(gcs_file_name, 'w') as f:
f.write(data)
The name of the default bucket in the SDK = 'app_default_bucket'
In the SDK datastore I have a Kind: GsFileInfo showing: filename: /app_default_bucket/example.pdf
Update and workaround: You can get a serving url for NON image files like css, js and pdf.
gs_file = '/gs/%s/%s/%s' % (app_identity.get_default_gcs_bucket_name(), folder, filename)
serving_url = images.get_serving_url(blobstore.create_gs_key(gs_file))
UPDATE I found this feature to serve cloudstorage files using the SDK:
This feature has not been documented yet.
http://localhost:8080/_ah/gcs/app_default_bucket/filename
This meands we do not need the img serving url to serve NON images as shown below !!!
To create e serving url for cloudstorage files like images, css, js and pdf's in the default_bucket, I use this code for testing(SDK) and GAE production:
IMPORTANT: the images.get_serving_url() works also for NON images in the SDK!!
In the SDK you stll need the blobstore to read a blob and create a serving url for a cloudstorage object.
I also added the code to read, write and upload cloudstorage blobs in the SDK and GAE production.
The code can be found here.
This is the value that you see in the Development mode from app_identity_stub.py:
APP_DEFAULT_GCS_BUCKET_NAME = 'app_default_bucket'
The comments in this file explain it:
This service behaves the same as the production service, except using
constant values instead of app-specific values
You should get the correct URL in your production code.
EDIT:
This is from the support forum:
In development mode, the app engine tools simulate Google Cloud
Storage services locally. Objects in that simulated environment are
non-persistent so your app is failing because the desired object
doesn't exist in the local store. If you first create (and optionally
write to) the object you're trying to read, it should work fine in dev
mode (it did for me). Of course, objects in the production service are
persistent so there's no need for that extra step when running your
app in production mode (assuming the object already exists).
Hope that helps,
Marc Google Cloud Storage Team
This means you have to write a file first, then you can use it. If I understand correctly, you can use any bucket name for this purpose, including 'app_default_bucket'.
I was here earlier looking for answers and just wanted to share what I found, now that I have it working.
You can do this now, and it's only a little painful. Tricking the image or blobstore API isn't supported and doesn't seem to work any longer.
See:
https://cloud.google.com/storage/docs/access-control/signed-urls
https://cloud.google.com/storage/docs/access-control/create-signed-urls-gsutil
If you sign your URLs, you can give auto-expiring links to your content, for anonymous or paywalled consumption. You wouldn't want to serve your whole site this way, but for a PDF or whatnot, this is a valid and semi-secure option.
Missing from the documentation, you might need to drop the newline for the canonical extended headers. The storage endpoint will tell you what it expects when the signature is bad.
Also, your host should be: https://storage-download.googleapis.com/
If you're using App Engine, then the GoogleAccessId is: <projectname>#appspot.gserviceaccount.com
See: app_identity.get_service_account_name()
Example of how to generate the signature:
from google.appengine.api import app_identity
def signFile(path, verb='GET', md5='', contentType='',
expiration=''):
signatureRequest = '{}\n{}\n{}\n{}\n{}'.format(
verb, md5, contentType, expiration, path)
return app_identity.sign_blob(signatureRequest)
That returns a tuple of (privateKey, binarySignature).
Now you need to construct the URL. The signature should be base64 encoded, then urlencoded. See the following for how to finish constructing the URL. You should probable use the download host mentioned above.
Example URL from the docs:
https://storage.googleapis.
com/example-bucket/cat.jpeg?GoogleAccessId=example#example-project.iam.gservicea
ccount.com&Expires=1458238630&Signature=VVUgfqviDCov%2B%2BKnmVOkwBR2olSbId51kSib
uQeiH8ucGFyOfAVbH5J%2B5V0gDYIioO2dDGH9Fsj6YdwxWv65HE71VEOEsVPuS8CVb%2BVeeIzmEe8z
7X7o1d%2BcWbPEo4exILQbj3ROM3T2OrkNBU9sbHq0mLbDMhiiQZ3xCaiCQdsrMEdYVvAFggPuPq%2FE
QyQZmyJK3ty%2Bmr7kAFW16I9pD11jfBSD1XXjKTJzgd%2FMGSde4Va4J1RtHoX7r5i7YR7Mvf%2Fb17
zlAuGlzVUf%2FzmhLPqtfKinVrcqdlmamMcmLoW8eLG%2B1yYW%2F7tlS2hvqSfCW8eMUUjiHiSWgZLE
VIG4Lw%3D%3D
I hope this helps someone!
Oh yeah, you only need to do all the signature stuff if your bucket isn't publicly accessible (read-all).
I know that I can't write files into the google app engine system, but I wonder if from the datastore I could programmatically build a txt file and serve it directly to download to the user of the application. I am not storing the file. I just want to serve it.
Any idea if this is possible?
Yes, it's possible.
You need to set the header to indicate that the file must be an attachment.
class MainHandler(webapp2.RequestHandler):
def test_download(self):
self.response.headers.add_header('content-disposition','attachment',filename='text.txt')
self.response.write("hello world")
You can see more information looking at the source for webapp2
Regarding "can't write files into the google app engine system", you can write to the blobstore instead. So if you need to generate a large file, you write it to the blobstore and serve it from there.
I need to upload a file/document to Google Docs on a GAE application. This should be simple enough, but I'm having a lot of trouble with the API.
The context:
import gdata.docs.service
client = gdata.docs.service.DocsService()
client.ClientLogin('gmail', 'pass')
ms = gdata.MediaSource(#what goes in here?)
client.Upload(media_source=ms, title='title')
To upload I'm using client.Upload(), which takes a MediaSource (wrapper) object as a parameter. However, MediaSource() seems to only accept a filepath for a document: 'C:/Docs/ex.doc'.
Since I'm on GAE with no filesystem, I can only access the file through the Blobstore or a direct URL to the file. But how do I input that into MediaSource()?
There seems to be a way in Java to accomplish this by using MediaByteArraySource(), but nothing for Python.
If anyone is curious, here's how I solved this problem using the Document List API.
I didn't want to use the Drive SDK since it does complicate a lot of things. It's much simpler with the List API to just authenticate/login without the need for some OAuth trickery. This is using version 2.0.14 of the gdata Python library, which is not the current version (2.0.17), but it seems to have a simpler upload mechanism.
There's also slightly more (still sparse) documentation online for 2.0.14, though I had to piece this together from various sources and trial & error. The downside is that you cannot upload pdf's with this version. This code will not work with 2.0.17.
Here's the code:
import gdata.docs.service
import gdata.docs.data
from google.appengine.api import urlfetch
# get file from url
result = urlfetch.fetch('http://example.com/test.docx')
headers = result.headers
data = result.content
# authenticate client object
client = gdata.docs.service.DocsService()
client.ClientLogin('gmail', 'password')
# create MediaSource file wrapper
ms = gdata.MediaSource(file_handle=result.content,
content_type=headers['content-type'],
content_length=int(headers['content-length']))
# upload specific folder, return URL of doc
google_doc_name = 'title'
folder_uri = '/feeds/folders/private/full/folder:j7XO8SJj...'
entry = client.Upload(ms, google_doc_name, folder_or_uri=secret.G_FOLDER_URI)
edit_url = entry.GetAlternateLink().href
The Google Drive SDK docs include a complete sample application written in Python that runs on App Engine:
https://developers.google.com/drive/examples/python
You can use it as reference for your implementation and to see how to save a file from App Engine.