I set up a required environment for Google Cloud Storage according to the manual.
I have installed "gsutil" and set up all paths.
My gsutil works perfectly, however, when I try to run the code below,
#!/usr/bin/python
import StringIO
import os
import shutil
import tempfile
import time
from oauth2_plugin import oauth2_plugin
import boto
# URI scheme for Google Cloud Storage.
GOOGLE_STORAGE = 'gs'
# URI scheme for accessing local files.
LOCAL_FILE = 'file'
uri=boto.storage_uri('sangin2', GOOGLE_STORAGE)
try:
uri.create_bucket()
print "done!"
except boto.exception.StorageCreateError, e:
print "failed"
It gives "403 Access denied" error.
Traceback (most recent call last):
File "/Volumes/WingIDE-101-4.0.0/WingIDE.app/Contents/MacOS/src/debug/tserver/_sandbox.py", line 23, in <module>
File "/Users/lsangin/gsutil/boto/boto/storage_uri.py", line 349, in create_bucket
return conn.create_bucket(self.bucket_name, headers, location, policy)
File "/Users/lsangin/gsutil/boto/boto/gs/connection.py", line 91, in create_bucket
response.status, response.reason, body)
boto.exception.GSResponseError: GSResponseError: 403 Forbidden
<?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message></Error>
Since I am new to this, it is kinda hard for me to figure out why.
Can someone help me?
Thank you.
The boto library should automatically find and use your $HOME/.boto file. One thing to check: make sure the project you're using is set as your default project for legacy access (at the API console, click on "Storage Access" and verify that it says "This is your default project for legacy access"). When I have that set incorrectly and I follow the create bucket example you referenced, I also get a 403 error, however, it doesn't make sense that this would work for you in gsutil but not with direct use of boto.
Try adding "debug=2" when you instantiate the storage_uri object, like this:
uri = boto.storage_uri(name, GOOGLE_STORAGE, debug=2)
That will generate some additional debugging information on stdout, which you can then compare with the debug output from an analogous, working gsutil example (via gsutil -D mb ).
Related
I'm trying to upload a video to Youtube using a python script.
So the code given here (upload_video.py) is supposed to work and I've followed the set up which includes enabling the Youtube API and getting OAuth secret keys and what not. You may notice that the code is in Python 2 so I used 2to3 to make it run with python3.7. The issue is that for some reason, I'm asked to login when I execute upload_video.py:
Now this should not be occuring as that's the whole point of having a client_secrets.json file, that you don't need to explicitly login. So once I exit this in-shell browser, Here's what I see:
Here's the first line:
/usr/lib/python3.7/site-packages/oauth2client/_helpers.py:255: UserWarning: Cannot access upload_video.py-oauth2.json: No such file or directory
warnings.warn(_MISSING_FILE_MESSAGE.format(filename))
Now I don't understand why upload_video.py-oauth2.json is needed as in the upload_video.py file, the oauth2 secret file is set as "client_secrets.json".
Anyways, I created the file upload_video.py-oauth2.json and copied the contents of client_secrets.json to it. I didn't get the weird login then but I got another error:
Traceback (most recent call last):
File "upload_video.py", line 177, in <module>
youtube = get_authenticated_service(args)
File "upload_video.py", line 80, in get_authenticated_service
credentials = storage.get()
File "/usr/lib/python3.7/site-packages/oauth2client/client.py", line 407, in get
return self.locked_get()
File "/usr/lib/python3.7/site-packages/oauth2client/file.py", line 54, in locked_get
credentials = client.Credentials.new_from_json(content)
File "/usr/lib/python3.7/site-packages/oauth2client/client.py", line 302, in new_from_json
module_name = data['_module']
KeyError: '_module'
So basically now I've hit a dead end. Any ideas about what to do now?
See the code of function get_authenticated_service in upload_video.py: you should not create the file upload_video.py-oauth2.json by yourself! This file is created upon the completion of the OAuth2 flow via the call to run_flow within get_authenticated_service.
Also you may read the doc OAuth 2.0 for Mobile & Desktop Apps for thorough info about the authorization flow on standalone computers.
I have already seen the examples on here of using python's os library to get a local file's time stamp in python by passing it the local path (i.e. /var/www/html/etc.../filename.txt), but when I try to pass getmtime a link, it cannot process it.
Here is what the code looks like:
import os
print(os.path.getmtime('https://www.sec.gov/Archives/edgar/data/1474439/000169655519000022/xslF345X03/wf-form4_156772823294389.xml'))
Here is the error I get:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python3.7/genericpath.py", line 55, in getmtime
return os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: 'https://www.sec.gov/Archives/edgar/data/1474439/000169655519000022/xslF345X03/wf-form4_156772823294389.xml'
I know that this link exists.
So it obviously doesn't like me passing it a link. Is there another function that you use to pass links, to get the last modification time of a remote file?
An URL is not necessarily a file. You can ask the remote server to tell you about the link, and the remote server may provide a Last-Modified header, or may not, at the remote server's discretion. It could also lie, if so instructed. In order to do this, you would need to make a HTTP request; the easiest way to do it from Python is the nice requests library.
import requests
import dateutil.parser
response = requests.head(url)
last_modified = response.headers.get('Last-Modified')
if last_modified:
last_modified = dateutil.parser.parse(last_modified)
There is this document from the Firebase about how to write and deploy the cloud function in Nodejs but can anyone help me out getting that very document in python. I am getting confused as I am a newbie in this field?
However, I tried to write my cloud function which looks like the below but constantly getting some errors that I am going to mention below:
import json
import firebase_admin
from firebase_admin import credentials
from firebase_admin import db
def go_firebase(request):
cred = credentials.Certificate('firebasesdk.json')
firebase_admin.initialize_app(cred, {
'databaseURL' : 'https://firebaseio.com/'
})
ref=db.reference('agents')
snapshot = ref.order_by_key().get()
for key, val in snapshot.items():
kw=val
dictfilt = lambda x, y: dict([ (i,x[i]) for i in x if i in set(y) ])
wanted_keys = ("address","name","phone","uid")
result = dictfilt(kw, wanted_keys)
data= json.dumps(result, sort_keys=True)
return data
And after deploying the function with http trigger, in the log, it is saying:
severity: "ERROR"
textPayload: "Traceback (most recent call last):
File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 313, in run_http_function
result = _function_handler.invoke_user_function(flask.request)
File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 215, in invoke_user_function
return call_user_function(request_or_event)
File "/env/lib/python3.7/site-packages/google/cloud/functions/worker.py", line 208, in call_user_function
return self._user_function(request_or_event)
File "/user_code/main.py", line 6, in go_firebase
cred = credentials.Certificate('firebasesdk.json')
File "/env/lib/python3.7/site-packages/firebase_admin/credentials.py", line 83, in __init__
with open(cert) as json_file:
FileNotFoundError: [Errno 2] No such file or directory: 'firebasesdk.json'
I have no idea why it is saying file not found because I have that json file in the same path where I am executing the function! I am using google cloud shell!
Can anyone be kind enough to tell me where I am going wrong?
The .json file that you are referring to is probably the dependencies file for Node.js Cloud Function.
How it works
Every Google Cloud Function has an extra file (additionally to the main code) that has all the libraries to be installed. For example if you are using requests library in your code then there is no way of running pip install requests before executing the main code. Therefore you add this library in the additional file and the Cloud Function will first read that file during deployment and will try to install all the libraries mentioned there.
For Node.js code the file with the libraries is a .json file. For the Python it is a requirements.txt file. For more information, you can refer to The Python Runtime documentation.
I am trying to use the Google Drive API to download publicly available files however whenever I try to proceed I get an import error.
For reference, I have successfully set up the OAuth2 such that I have a client id as well as a client secret , and a redirect url however when I try setting it up I get an error saying the object has no attribute urllen
>>> from apiclient.discovery import build
>>> from oauth2client.client import OAuth2WebServerFlow
>>> flow = OAuth2WebServerFlow(client_id='not_showing_client_id', client_secret='not_showing_secret_id', scope='https://www.googleapis.com/auth/drive', redirect_uri='https://www.example.com/oauth2callback')
>>> auth_uri = flow.step1_get_authorize_url()
>>> code = '4/E4h7XYQXXbVNMfOqA5QzF-7gGMagHSWm__KIH6GSSU4#'
>>> credentials = flow.step2_exchange(code)
And then I get the error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Python/2.7/site-packages/oauth2client/util.py", line
137, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Library/Python/2.7/site-packages/oauth2client/client.py", line
1980, in step2_exchange
body = urllib.parse.urlencode(post_data)
AttributeError: 'Module_six_moves_urllib_parse' object has no attribute
'urlencode'
Any help would be appreciated, also would someone mind enlightening me as to how I instantiate a drive_file because according to https://developers.google.com/drive/web/manage-downloads, I need to instantiate one and I am unsure of how to do so.
Edit: So I figured out why I was getting the error I got before. If anyone else is having the same problem then try running.
sudo pip install -I google-api-python-client==1.3.2
However I am still unclear about the drive instance so any help with that would be appreciated.
Edit 2: Okay so I figured out the answer to my whole question. The drive instance is just the metadata which results when we use the API to search for a file based on its id
So as I said in my edits try the sudo pip install and a file instance is just a dictionary of meta data.
I am trying to use the Remote API in a local Client, following instructions given in Google documentation (https://developers.google.com/appengine/docs/python/tools/remoteapi).
I first tried from the Remote API shell, and everything is working fine. I can connect to my remote datastore and fetch data.
But when I run the following script, which is very similar to Google's example :
#!/opt/local/bin/python2.7
import sys
SDK_PATH = "/usr/local/google_appengine/"
sys.path.append(SDK_PATH)
import dev_appserver
dev_appserver.fix_sys_path()
from google.appengine.ext.remote_api import remote_api_stub
import model.account
def auth_func():
return ('myaccount', 'mypasswd')
remote_api_stub.ConfigureRemoteApi('myappid', '/_ah/remote_api', auth_func)
# Fetch some data
entries = model.account.list()
for a in entries:
print a.name
then I get an error :
Traceback (most recent call last):
File "./remote_script_test.py", line 26, in <module>
entries = model.account.list()
[...]
File "/Developer/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/datastore/datastore_rpc.py", line 1333, in check_rpc_success
raise _ToDatastoreError(err)
google.appengine.api.datastore_errors.BadRequestError: Application Id (app) format is invalid: '_'
It says my application ID is a plain underscore '_', which is not the case since my app.yaml is correctly configured and when I do a
print app_identity.get_application_id()
from that script I get the correct application ID. I am under the impression that the GAE environment is not properly setup, but I could not figure out how to make it work. Does anyone have a full piece of code that works in this context ?
I am using Mac Os X Mountain Lion.
You may wish to start with remote_api_shell.py and remove stuff until it breaks, rather than add in things until it works.
For instance there's a line after the one you copied: remote_api_stub.MaybeInvokeAuthentication() is that necessary? There may be a number of necessary lines in remote_api_shell.py, maybe better to customize that rather than start from scratch.