I'm using the Google API services in some scripts and having some problems. This error is something weird, but here we go.
I have a script that is listing my Google Drive files.
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools
SCOPES = 'https://www.googleapis.com/auth/drive.readonly.metadata'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store)
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
files = DRIVE.files().list().execute().get('files', [])
for f in files:
print(f['name'], f['mimeType'],f['id'])
It works perfectly, I download the client_secret.json from Google API and save it in the same folder, then I start the script to check everythings is ok.
Then I start to edit my file to change the way how I execute it and don't read a file, instead, call the script and send the client_id and client_secret strings to the script, and the final version is this:
import sys
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools
# ID and SECRET arguments
client_id = sys.argv[1]
client_secret = sys.argv[2]
SCOPES = 'https://www.googleapis.com/auth/drive.readonly.metadata'
def listFiles(drive):
"""Receive the service and list the files"""
files = drive.files().list().execute().get('files', [])
for f in files:
print(f['name'], f['mimeType'],f['id'])
def main():
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.OAuth2WebServerFlow(client_id, client_secret, SCOPES)
creds = tools.run_flow(flow, store, tools.argparser.parse_args())
DRIVE = discovery.build('drive', 'v3', http=creds.authorize(Http()))
listFiles(DRIVE)
if __name__ == "__main__":
main()
The first time that I start this new version of the script it works, because the script in the old version has created the storage.json file.
Then I move my new version script to another folder or machine (where the storage.json file doesn't exists) to check if it works and then I get this:
$ python3 drive_list.py asdasdasdsa jijfkljflksdjflksdj
/usr/local/lib/python3.4/dist-packages/oauth2client/_helpers.py:255: UserWarning: Cannot access storage.json: No such file or directory
warnings.warn(_MISSING_FILE_MESSAGE.format(filename))
usage: drive_list.py [--auth_host_name AUTH_HOST_NAME]
[--noauth_local_webserver]
[--auth_host_port [AUTH_HOST_PORT [AUTH_HOST_PORT ...]]]
[--logging_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}]
drive_list.py: error: unrecognized arguments: asdasdasdsa jijfkljflksdjflksdj
The warning about the storage.json file is normal and appears in both script versions, is part of oauth2client.
This is the curios part, why the arguments are recognized when the storage.json file exists (ONLY created reading the client_secret.json)? if the first time that the script start it creates the file.
Is really weird this error, and I'm just trying to find what is happening exactly.
If someone can help me, I will be really thankful.
This is happening because you are importing the oauth2client.tools module.
To work correctly this module relies on the standard argparse module. In case you don't know, this standard module is used to write user-friendly command-line interfaces with easy management of command line arguments. This does not get along with your usage of sys.argv[1] and sys.argv[2] arguments.
To work around this, you can add new arguments to the command line like in the sample below. With this modification you would then run the tool like this
python3 drive_list.py -ci "your_client_id" -cs "your_client_secret"
Here is your code slightly modified to add the new command line arguments:
import argparse
from apiclient import discovery
from httplib2 import Http
from oauth2client import file, client, tools
# ID and SECRET arguments as new command line parameters
# Here is where you extend the oauth2client.tools startnd arguments
tools.argparser.add_argument('-ci', '--client-id', type=str, required=True, help='The client ID of your GCP project')
tools.argparser.add_argument('-cs', '--client-secret', type=str, required=True,
help='The client Secret of your GCP project')
SCOPES = 'https://www.googleapis.com/auth/drive.readonly.metadata'
def list_files(drive):
"""Receive the service and list the files"""
files = drive.files().list().execute().get('files', [])
for f in files:
print(f['name'], f['mimeType'], f['id'])
def main():
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
# You want to be sure to parse the args to take in consideration the new parameters
args = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
flow = client.OAuth2WebServerFlow(args.client_id, args.client_secret, SCOPES)
creds = tools.run_flow(flow, store, tools.argparser.parse_args())
drive_sdk = discovery.build('drive', 'v3', http=creds.authorize(Http()))
list_files(drive_sdk)
if __name__ == "__main__":
main()
Related
I'm hoping to use the Google Sheets API in a cloud function, which will run from my account's default service account, and I'm working in Python. However, I've only ever authenticated the Sheets library locally, using this bit of code:
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
def gen_creds(path_to_secret: str, rw_vs_ro: str):
"""
Generate the needed credentials to work with the Sheets v4 API based on your secret
json credentials file.
:param path_to_secret: The file path to your credentials json file
:param rw_vs_ro: A string, 'r_o' or 'r_w', representing whether creds should be readonly or readwrite
:return: The built service variable
"""
if rw_vs_ro == 'r_o':
scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly']
creds_nm = 'readonly_token.json'
else:
scopes = ['https://www.googleapis.com/auth/spreadsheets']
creds_nm = 'readwrite_token.json'
creds = None
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists(creds_nm):
creds = Credentials.from_authorized_user_file(creds_nm, scopes)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
path_to_secret, scopes)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(creds_nm, 'w') as token:
token.write(creds.to_json())
return build('sheets', 'v4', credentials=creds)
And I'm not entirely sure how to translate this to something that a cloud function would understand, as the cloud function won't be running as me, and lacks the same type of os path that I have access to locally. Would appreciate any insight into what the translation process looks like here--I was only able to find examples in JS, which wasn't perfect for what I was going for. Then, I would love to understand how to actually implement this code in a cloud function in GCP. Thanks!
When you deploy a cloud function, your main code will have access to all the files deployed within that function. This means all you need to do is include your readwrite_token.json/readonly_token.json files when deploying the package.
Once that's done, instead of simply passing the token files as strings, since the function’s directory can be different from the current working directory, you have to properly include the files as specified in this GCP Function Filesystem documentation
Also, you can't use InstalledAppFlow in the Cloud Function environment since this flow is meant for desktop os environments so better pray for the block to never be executed or replace with a different flow.
Actually, I found a simple answer to this question in the end--it's very easy to generate these credentials in GCP for Python! The exact replacement method for gen_creds is:
import google.auth
from googleapiclient.discovery import build
def gen_creds(rw_vs_ro: str):
"""
Generate the service credentials to be used to query a google sheet
:param rw_vs_ro: A string, 'r_o' or 'r_w', representing whether creds should be readonly or readwrite
:return: The built service variable
"""
if rw_vs_ro == 'r_o':
scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly']
if rw_vs_ro == 'r_w':
scopes = ['https://www.googleapis.com/auth/spreadsheets']
creds, project = google.auth.default(scopes=scopes)
service = build('sheets', 'v4', credentials=creds)
return service
Hope this is as helpful to others as it is to me!
I'm trying to run a google app script function remotely from a python flask app. This function creates google calendar events with inputs from a google sheet. I referred to this documentation from Google in order to set up the python script to run the appscript function. I followed every step required to deploy the app script project as an executable API and connected it to a google developer project and made OAuth 2.0 ID credentials as well.
From the API executable documentation, I got the following code and modified it to run as an object which can be called from the main server file.
from __future__ import print_function
from googleapiclient import errors
from googleapiclient.discovery import build
from httplib2 import Http
from oauth2client import file as oauth_file, client, tools
class CreateGCalEvent:
def main(self):
"""Runs the sample.
"""
SCRIPT_ID = 'my app script deployment ID was put here'
# Set up the Apps Script API
SCOPES = [
'https://www.googleapis.com/auth/script.scriptapp',
'https://www.googleapis.com/auth/drive.readonly',
'https://www.googleapis.com/auth/drive',
]
store = oauth_file.Storage('token.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('app_script_creds.json', SCOPES)
creds = tools.run_flow(flow, store)
service = build('script', 'v1', credentials=creds)
# Create an execution request object.
request = {"function": "getFoldersUnderRoot"}
try:
# Make the API request.
response = service.scripts().run(body=request,
scriptId=SCRIPT_ID).execute()
if 'error' in response:
# The API executed, but the script returned an error.
# Extract the first (and only) set of error details. The values of
# this object are the script's 'errorMessage' and 'errorType', and
# an list of stack trace elements.
error = response['error']['details'][0]
print("Script error message: {0}".format(error['errorMessage']))
if 'scriptStackTraceElements' in error:
# There may not be a stacktrace if the script didn't start
# executing.
print("Script error stacktrace:")
for trace in error['scriptStackTraceElements']:
print("\t{0}: {1}".format(trace['function'],
trace['lineNumber']))
else:
# The structure of the result depends upon what the Apps Script
# function returns. Here, the function returns an Apps Script Object
# with String keys and values, and so the result is treated as a
# Python dictionary (folderSet).
folderSet = response['response'].get('result', {})
if not folderSet:
print('No folders returned!')
else:
print('Folders under your root folder:')
for (folderId, folder) in folderSet.items():
print("\t{0} ({1})".format(folder, folderId))
except errors.HttpError as e:
# The API encountered a problem before the script started executing.
print(e.content)
Here is where the error comes. It can neither locate token.json nor the app_script_creds.json.
Now with a service account and any normal OAuth2.0 ID, when I create it, I will be given the option to download the credentials.json but here, this is all I seem to be getting, an App Script ID with no edit access or credentials to download as JSON. I created another OAuth ID in the same project as shown in the screenshot which has the edit access and json ready for download. When I used that json file inside the python script, It told me that it was expecting redirect uris, which I don't know for what it is or where to redirect to.
What do I need to do to get this working?
I adapted some code that I used for connecting to the App Scripts API. I hope it works for you too. The code is pretty much the same thing as this.
You can use from_client_secrets_file since you're already loading these credentials from the file. So, what the code does is look for a token file first. If the token file is not there, it logs in the user (prompting using the Google authorization screen) and stores the new token in the file as pickle.
Regarding the credentials in the Google console you need to pick the Desktop application when creating them because that is basically what a server is.
Note: with this, you can only have one user that will be doing all of these actions. This is because the server script will start a local server on the server machine to authenticate you, your client code will not see any of this.
import logging
import pickle
from pathlib import Path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
class GoogleApiService:
def __init__(self, , scopes):
"""
Args:
scopes: scopes required by the script. There needs to be at least
one scope specified.
"""
self.client_secrets= Path('credentials/credentials.json')
self.token_path = Path('credentials/token.pickle')
self.credentials = None
self.scopes = scopes
def get_service(self):
self.__authenticate()
return build('script', 'v1', credentials=self.credentials)
def __authenticate(self):
log.debug(f'Looking for existing token in {self.token_path}')
if self.token_path.exists():
with self.token_path.open('rb') as token:
self.credentials = pickle.load(token)
if self.__token_expired():
self.credentials.refresh(Request())
# If we can't find any token, we log in and save it
else:
self.__log_in()
self.__save_token()
def __log_in(self):
flow = InstalledAppFlow.from_client_secrets_file(
self.client_secrets,
self.scopes
)
self.credentials = flow.run_local_server(port=0)
def __save_token(self):
with self.token_path.open('wb') as token:
pickle.dump(self.credentials, token)
def __token_expired(self):
return self.credentials and self.credentials.expired and \
self.credentials.refresh_token
# Example for Google Apps Scripts
def main():
request = {'function': 'some_function', 'parameters': params}
gapi_service = GoogleApiService()
with gapi_service.get_service() as service:
response = service.scripts().run(
scriptId=self.script_id,
body=request
).execute()
if response.get('error'):
message = response['error']['details'][0]['errorMessage']
raise RuntimeError(message)
else:
return response['response']['result']
I'm hoping to use the Google Sheets API in a cloud function, which will run from my account's default service account, and I'm working in Python. However, I've only ever authenticated the Sheets library locally, using this bit of code:
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
def gen_creds(path_to_secret: str, rw_vs_ro: str):
"""
Generate the needed credentials to work with the Sheets v4 API based on your secret
json credentials file.
:param path_to_secret: The file path to your credentials json file
:param rw_vs_ro: A string, 'r_o' or 'r_w', representing whether creds should be readonly or readwrite
:return: The built service variable
"""
if rw_vs_ro == 'r_o':
scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly']
creds_nm = 'readonly_token.json'
else:
scopes = ['https://www.googleapis.com/auth/spreadsheets']
creds_nm = 'readwrite_token.json'
creds = None
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists(creds_nm):
creds = Credentials.from_authorized_user_file(creds_nm, scopes)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
path_to_secret, scopes)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open(creds_nm, 'w') as token:
token.write(creds.to_json())
return build('sheets', 'v4', credentials=creds)
And I'm not entirely sure how to translate this to something that a cloud function would understand, as the cloud function won't be running as me, and lacks the same type of os path that I have access to locally. Would appreciate any insight into what the translation process looks like here--I was only able to find examples in JS, which wasn't perfect for what I was going for. Then, I would love to understand how to actually implement this code in a cloud function in GCP. Thanks!
When you deploy a cloud function, your main code will have access to all the files deployed within that function. This means all you need to do is include your readwrite_token.json/readonly_token.json files when deploying the package.
Once that's done, instead of simply passing the token files as strings, since the function’s directory can be different from the current working directory, you have to properly include the files as specified in this GCP Function Filesystem documentation
Also, you can't use InstalledAppFlow in the Cloud Function environment since this flow is meant for desktop os environments so better pray for the block to never be executed or replace with a different flow.
Actually, I found a simple answer to this question in the end--it's very easy to generate these credentials in GCP for Python! The exact replacement method for gen_creds is:
import google.auth
from googleapiclient.discovery import build
def gen_creds(rw_vs_ro: str):
"""
Generate the service credentials to be used to query a google sheet
:param rw_vs_ro: A string, 'r_o' or 'r_w', representing whether creds should be readonly or readwrite
:return: The built service variable
"""
if rw_vs_ro == 'r_o':
scopes = ['https://www.googleapis.com/auth/spreadsheets.readonly']
if rw_vs_ro == 'r_w':
scopes = ['https://www.googleapis.com/auth/spreadsheets']
creds, project = google.auth.default(scopes=scopes)
service = build('sheets', 'v4', credentials=creds)
return service
Hope this is as helpful to others as it is to me!
I was writing a Python script to automate uploading some files to Google Drive. Since I'm still a newbie Python programmer and this is an exercise as much as anything else, I started following the Google Quickstart and decided to use their quickstart.py as a basis on which to base my own script. In the part where it talks about how to create credentials for your Python script, it refers to the "Create credentials" link, at https://developers.google.com/workspace/guides/create-credentials
I follow the link, get into one of my Google Cloud projects, and try to set up the OAuth consent screen, using an "Internal" project, as they tell you... but I can't. Google says:
“Because you’re not a Google Workspace user, you can only make your
app available to external (general audience) users. ”
So I try to create an "External" project, and then proceed to create a new client ID, using a Desktop application. Then I download the JSON credentials and put them in the same folder as my Python script, as "credentials.json". I then execute the Python script in order to authenticate it: the browser opens, I log into my Google account, give it my permissions... and then the browser hangs, because it's redirecting to a localhost URL and obviously my little Python script isn't listening in my computer at all.
I believe they must have changed this recently, because a year ago I started following the same Python tutorial and could create credentials without problems, but the Google Drive API docs haven't been updated yet. So... how do I create credentials for a Python script now?
EDIT: adding here the source code for my script. As I said, it's very similar to Google's "quickstart.py":
from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from googleapiclient.errors import HttpError
# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/drive.metadata', 'https://www.googleapis.com/auth/drive']
def main():
"""Shows basic usage of the Drive v3 API.
Prints the names and ids of the first 10 files the user has access to.
"""
creds = None
# The file token.pickle stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token_myappname.pickle'):
with open('token_myappname.pickle', 'rb') as token:
creds = pickle.load(token)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open('token_myappname.pickle', 'wb') as token:
pickle.dump(creds, token)
service = build('drive', 'v3', credentials=creds)
# Call the Drive v3 API
results = service.files().list(
pageSize=10, fields="nextPageToken, files(id, name)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
#print(items[0])
print('Files:')
for item in items:
#print (item)
print(u'{0} {1} {2}'.format(item['name'], item['owners'], item['parents']))
I propose you to use a service account to access to the Drive.
For that, you need to share the drive (or the folder) with the service account email. And then use this code
from googleapiclient.discovery import build
import google.auth
SCOPES = ['https://www.googleapis.com/auth/drive.metadata', 'https://www.googleapis.com/auth/drive']
def main():
credentials, project_id = google.auth.default(scopes=SCOPES)
service = build('drive', 'v3', credentials=credentials)
# Call the Drive v3 API
results = service.files().list(
q=f"'1YJ6gMgACOqVVbcgKviJKtVa5ITgsI1yP' in parents",
pageSize=10, fields="nextPageToken, files(id, name, owners, parents)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
#print(items[0])
print('Files:')
for item in items:
#print (item)
print(u'{0} {1} {2}'.format(item['name'], item['owners'], item['parents']))
If you run your code on Google Cloud, in a compute engine instance for example, you need to customize the VM with the service account that you authorized in your drive. (Don't use the compute engine default service account, else you will need extra configuration on your VM)
If you run your script outside GCP, you need to generate a service account key file and to store it on your local server. Then, create an environment variable GOOGLE_APPLICATION_CREDENTIALS that reference the full path of the stored key file.
Aside from the other solution posted by Guillaume Blaquiere, I also found another one on my own, which I wanted to post here in case it's helpful. All I had to do is to... erm, actually read the code I was copying and pasting, in particular this line:
creds = flow.run_local_server(port=0)
I checked Google's documentation outside of the Quickstart and found in the following: https://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html
It turns out, the example code was opening a local port in my computer to listen to the request, and it wasn't working probably due to the "port 0" part, or some other network problem.
So the workaround I found was to use a different auth method found in the docs:
creds = flow.run_console()
In this case, you paste manually in the command line the auth code given to you by Google. I just tried it, and have my credentials happily stored in my local pickle file.
I am using python to upload or download files on Google Drive. This is the most important part of my code:
from httplib2 import Http
from oauth2client import file, client, tools
from apiclient.http import MediaFileUpload
from apiclient import errors
import io
from apiclient.http import MediaIoBaseDownload
from apiclient import errors
import os
import csv
#####################################
try :
import argparse
flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
flags = None
SCOPES = 'https://www.googleapis.com/auth/drive.file'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
print("Create new data storage file ...")
flow = client.flow_from_clientsecrets('client_secrets.json', SCOPES)
flow.redirect_uri = client.OOB_CALLBACK_URN
authorize_url = flow.step1_get_authorize_url()
creds = tools.run_flow(flow, store, flags) \
if flags else tools.run(flow, store)
print ("Storage")
DRIVE = build('drive', 'v2', http=creds.authorize(Http()))
The python-script worked fine for 2 month. But since yesterday I get this error if I start my script:
400. That’s an error.
Error: admin_policy_enforced
Company security policies do not allow this app to connect to your Company account data. Please contact your local helpdesk for any support.
Request Details
client_id=XXXYYYZZZ.apps.googleusercontent.com
redirect_uri=urn:ietf:wg:oauth:2.0:oob
scope=https://www.googleapis.com/auth/drive.file
access_type=offline
response_type=code
My admin rights have not been changing till now. Do someone have an idea?