I'm creating a simple web app that saves some files in my Google Drive using the Auth2.0 authentication. Everything works well when I use the code below, but I struggle with some things here. The first one is to save the content in a different specified folder. The code below save the files in the root folder and I can't figure out why.The upload_folder_id evidently has the good value, that I omitted here.
from apiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
try:
import argparse
flags = argparse.ArgumentParser(parents=[tools.argparser]).parse_args()
except ImportError:
flags = None
SCOPES = 'https://www.googleapis.com/auth/drive.file'
store = file.Storage('storage.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', SCOPES)
creds = tools.run_flow(flow, store, flags) \
if flags else tools.run(flow, store)
DRIVE = build('drive', 'v3', http=creds.authorize(Http()))
upload_folder_id = <my_folder_id>
FILES = (('hello.txt', None)) # More files will be added after
for filename, mimeType in FILES:
metadata = {'name': filename}
if mimeType:
metadata['mimeType'] = mimeType
''' I tried all this possibilities too
metadata['parentId'] = upload_folder_id
metadata['folderId'] = upload_folder_id
'''
metadata['parents'] = [{'id': upload_folder_id}]
res = DRIVE.files().create(body=metadata, media_body=filename).execute()
if res:
print('Uploaded "%s" (%s)' % (filename, res['mimeType']))
The second is the authentication step. The first time that I connected the application, I was redirected to a google web page where I grant the API access. Is there a way to do this without going through the web browser to validate the access? (It's fine doing that for my local tests but I'll not be able to do this in the production server).
The code below save the files in the root folder and I can't figure out why.The upload_folder_id evidently has the good value, that I omitted here.
parents is supposed to be a string array, so there's no need for the id property to be there; just specify the parent folderId and I think you're good to go
Is there a way to do this without going through the web browser to validate the access?
The consent page is critical for the authentication process of the user, I don't think scraping can be a solution here. An alternative I can think of is for you to use Service Accounts, which is a server-to-server implementation so that the users don't need to login. The drawback here though is for the user doesn't own the data, the service account does.
Related
I enabled the google drive api, created a service account, added the service account email to the folder and successfully generated the .json credentials
But when I try to print the contents of the temp2 folder I get "no files found"
I'm attempting to do it this way
import logging
from google.oauth2.service_account import Credentials
from googleapiclient.discovery import build
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
creds = Credentials.from_service_account_file('bayfiles-779393795f34.json', scopes=['https://www.googleapis.com/auth/drive'])
service = build('drive', 'v3', credentials=creds)
query = "mimeType='application/vnd.google-apps.folder' or mimeType='application/vnd.google-apps.document' or mimeType='application/vnd.google-apps.spreadsheet' or mimeType='application/vnd.google-apps.presentation' and trashed = false and parents in '16x7o7CNCscM-lFqnKaXwUf1Bv-OTQK0W'"
results = service.files().list(q=query).execute()
items = results.get("files", [])
if not items:
logger.debug("No files found")
else:
# Print file names
for item in items:
logger.debug(f'The service account has access to the file "{item["name"]}" with ID "{item["id"]}"')
The strange thing is that the number of requests appears to me in the API page of my project
but through the code it is returned to me as if there were no files inside the temp2 folder, but it's wrong.
I tested locally, with my user account and your filter and your code works well. But as I said, strangely.
In fact, only the presentation not trashed in the mentioned folder are returned + all the other file mentioned in the OR clauses. Again, add parenthesis like that to narrow only to the current mentioned folder
query = "(mimeType='application/vnd.google-apps.folder' or mimeType='application/vnd.google-apps.document' or mimeType='application/vnd.google-apps.spreadsheet' or mimeType='application/vnd.google-apps.presentation') and trashed = false and parents in '16x7o7CNCscM-lFqnKaXwUf1Bv-OTQK0W'"
Are you sure about the Folder ID?
Let me show you how to avoid a service account key file (which is a bad practice).
You have to scope correctly your user credential. Use that CLI command for that
gcloud auth application-default login \
--scopes="https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/drive"
Keep in mind that is now your user account that will be used, and YOUR email must be granted on the Google Drive folder
If you want to use the service account identity instead of your own user email, you can use impersonation
gcloud auth application-default login \
--scopes="https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/drive" \
--impersonate-service-account="<service account email>"
You must have the role "project owner" or "service account token creator" to be able to impersonate a service account. But like that, you don't need a secret and sensitive file (JSON)
Update your code. You will see 2 changes
import logging
#from google.oauth2.service_account import Credentials -- No longer need it
import google.auth #Required for getting the default authentication
from googleapiclient.discovery import build
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
#creds = Credentials.from_service_account_file('bayfiles-779393795f34.json', scopes=['https://www.googleapis.com/auth/drive'])
# Prefer the default credential instead. Like that it works locally and in the cloud the same way
# Scopes are optional locally, but required in the cloud runtime environment
credentials, project_id = google.auth.default(scopes=["https://www.googleapis.com/auth/cloud-platform","https://www.googleapis.com/auth/drive"])
# With your user credential you also have to notify explicitly the quota project. It's optional for cloud runtime environment
service = build('drive', 'v3', credentials=credentials,client_options={"quota_project_id":project_id})
query = "mimeType='application/vnd.google-apps.folder' or mimeType='application/vnd.google-apps.document' or mimeType='application/vnd.google-apps.spreadsheet' or mimeType='application/vnd.google-apps.presentation' and trashed = false and parents in '16x7o7CNCscM-lFqnKaXwUf1Bv-OTQK0W'"
results = service.files().list(q=query).execute()
items = results.get("files", [])
if not items:
logger.debug("No files found")
else:
# Print file names
for item in items:
logger.debug(f'The service account has access to the file "{item["name"]}" with ID "{item["id"]}"')
I write a python script to upload file to google drive, but the script is redirecting to chrome for email user authentication.
is there any way to avoid redirecting to chrome for authentication.
I'm running on python 3.9.
here is my sample code:
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
drive = GoogleDrive(gauth)
upload_file_list = ['myfile.pdf']
for upload_file in upload_file_list:
gfile = drive.CreateFile({'parents': [{'id': '1B8ttlQMRUkjbrscevfa1DablIayzObh2'}]})
# Read file and set it as the content of this instance.
gfile.SetContentFile(upload_file)
gfile.Upload() # Upload the file.
The behaviour you are reporting is totally normal with OAuth 2.0 and the official Google APIs library.
What #Tanaike said is a good solution. You could use a service account to access Google Drive files without granting consent every time the token expires. With service accounts there are 2 options to achieve that:
Share the file/folder with the email address of the service account.
Use domain-wide delegation of authority to allow the service account to impersonate any user in your domain. Requires a domain using Google Workspace or Cloud Identity and Super Admin access to configure domain-wide delegation.
General information on how to make API calls with domain-wide delegation is available on this page https://developers.google.com/identity/protocols/oauth2/service-account#authorizingrequests.
Here is a working code sample:
from google.oauth2 import service_account
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
# Scopes required by this endpoint
# https://developers.google.com/drive/api/v3/reference/permissions/list
SCOPES = ["https://www.googleapis.com/auth/drive.readonly"]
# Variable that holds the file ID
DOCUMENT_ID = "i0321LSy8mmkx_Bw-XlDyzQ_b3Ny9m74u"
# Service account Credential file downloaded with domain-wide delegation of authority
# or with shared access to the file.
SERVICE_ACCOUNT_FILE = "serviceaccount.json";
# Creation of the credentials
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE,
scopes=SCOPES)
# [Impersonation] the service account will take action on behalf of the user,
# requires domain-wide delegation of authority.
delegated_credentials = credentials.with_subject('user#domain.com')
# The API call is attempted
try:
service = build('drive', 'v3', credentials=delegated_credentials)
# Retrieve the documents contents from the Docs service.
document = service.files().get(fileId=DOCUMENT_ID).execute()
print('The title of the document is: {}'.format(document.get('name')))
except HttpError as err:
print(err)
Keep in mind that to use user impersonation you will need to configure domain-wide delegation in the Admin console of the domain that has the files (this will also work for external files shared with users in the domain).
If you want to use this with regular consumer accounts you can't use user impersonation, instead you will share the file with the service account (read or write access) to later make API calls. Line 20 creates delegated credentials, this line needs to be removed if you will use this other approach.
I was writing a Python script to automate uploading some files to Google Drive. Since I'm still a newbie Python programmer and this is an exercise as much as anything else, I started following the Google Quickstart and decided to use their quickstart.py as a basis on which to base my own script. In the part where it talks about how to create credentials for your Python script, it refers to the "Create credentials" link, at https://developers.google.com/workspace/guides/create-credentials
I follow the link, get into one of my Google Cloud projects, and try to set up the OAuth consent screen, using an "Internal" project, as they tell you... but I can't. Google says:
“Because you’re not a Google Workspace user, you can only make your
app available to external (general audience) users. ”
So I try to create an "External" project, and then proceed to create a new client ID, using a Desktop application. Then I download the JSON credentials and put them in the same folder as my Python script, as "credentials.json". I then execute the Python script in order to authenticate it: the browser opens, I log into my Google account, give it my permissions... and then the browser hangs, because it's redirecting to a localhost URL and obviously my little Python script isn't listening in my computer at all.
I believe they must have changed this recently, because a year ago I started following the same Python tutorial and could create credentials without problems, but the Google Drive API docs haven't been updated yet. So... how do I create credentials for a Python script now?
EDIT: adding here the source code for my script. As I said, it's very similar to Google's "quickstart.py":
from __future__ import print_function
import pickle
import os.path
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request
from googleapiclient.errors import HttpError
# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/drive.metadata', 'https://www.googleapis.com/auth/drive']
def main():
"""Shows basic usage of the Drive v3 API.
Prints the names and ids of the first 10 files the user has access to.
"""
creds = None
# The file token.pickle stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token_myappname.pickle'):
with open('token_myappname.pickle', 'rb') as token:
creds = pickle.load(token)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open('token_myappname.pickle', 'wb') as token:
pickle.dump(creds, token)
service = build('drive', 'v3', credentials=creds)
# Call the Drive v3 API
results = service.files().list(
pageSize=10, fields="nextPageToken, files(id, name)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
#print(items[0])
print('Files:')
for item in items:
#print (item)
print(u'{0} {1} {2}'.format(item['name'], item['owners'], item['parents']))
I propose you to use a service account to access to the Drive.
For that, you need to share the drive (or the folder) with the service account email. And then use this code
from googleapiclient.discovery import build
import google.auth
SCOPES = ['https://www.googleapis.com/auth/drive.metadata', 'https://www.googleapis.com/auth/drive']
def main():
credentials, project_id = google.auth.default(scopes=SCOPES)
service = build('drive', 'v3', credentials=credentials)
# Call the Drive v3 API
results = service.files().list(
q=f"'1YJ6gMgACOqVVbcgKviJKtVa5ITgsI1yP' in parents",
pageSize=10, fields="nextPageToken, files(id, name, owners, parents)").execute()
items = results.get('files', [])
if not items:
print('No files found.')
else:
#print(items[0])
print('Files:')
for item in items:
#print (item)
print(u'{0} {1} {2}'.format(item['name'], item['owners'], item['parents']))
If you run your code on Google Cloud, in a compute engine instance for example, you need to customize the VM with the service account that you authorized in your drive. (Don't use the compute engine default service account, else you will need extra configuration on your VM)
If you run your script outside GCP, you need to generate a service account key file and to store it on your local server. Then, create an environment variable GOOGLE_APPLICATION_CREDENTIALS that reference the full path of the stored key file.
Aside from the other solution posted by Guillaume Blaquiere, I also found another one on my own, which I wanted to post here in case it's helpful. All I had to do is to... erm, actually read the code I was copying and pasting, in particular this line:
creds = flow.run_local_server(port=0)
I checked Google's documentation outside of the Quickstart and found in the following: https://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html
It turns out, the example code was opening a local port in my computer to listen to the request, and it wasn't working probably due to the "port 0" part, or some other network problem.
So the workaround I found was to use a different auth method found in the docs:
creds = flow.run_console()
In this case, you paste manually in the command line the auth code given to you by Google. I just tried it, and have my credentials happily stored in my local pickle file.
I'm trying use google drive api. I created a service account credentials and downloaded from console cloud. The problem is that I'm part of an organization in gsuit and when I try list my files, it's empty, but I have files in my drive.
from apiclient.discovery import build
from oauth2client.service_account import ServiceAccountCredentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(
"credentials.json", scopes=['https://www.googleapis.com/auth/drive'])
service = build('drive', 'v3', credentials=credentials)
print(service.files().list().execute())
What could be?
Actually you are not providing a lot of information but make sure on the api credentials you issued you selected the 'Other UI' option on the field 'Where will you be calling the API from' and you chose 'User data' instead of 'Application data', also the scope should be 'https://www.googleapis.com/auth/drive.readonly.metadata' for listing data.
'https://www.googleapis.com/auth/drive' is correct too but given that it is a gsuite account there can be limitations on generic scopes even for your own data.
Also you should do service = DRIVE.files().list().execute().get('files', [])
for f in files:
print(f['name'])
and enumerate that files array to get the files.
if that doesn't work have a look at the api docs and if you can't figure it out please post more details and try to do some debugging and post the results here.
Edit: Try using the restapi too with the appropriate credentials and see if the files are fetched successfully there. https://developers.google.com/drive/api/v2/reference/files/list
My Python (3.6.7) code uses oauth2client to access Google Photos APIs. It successfully authenticates, but when it tries to access the Google Photos albums, it seems to be using the username as the project_id.
from __future__ import print_function
from apiclient.discovery import build
from httplib2 import Http
from oauth2client import file, client, tools
# Setup the Photo v1 API
SCOPES = 'https://www.googleapis.com/auth/photoslibrary.readonly'
store = file.Storage('credentials.json')
creds = store.get()
if not creds or creds.invalid:
flow = client.flow_from_clientsecrets('scripts/client_id.json', SCOPES)
creds = tools.run_flow(flow, store)
service = build('photoslibrary', 'v1', http=creds.authorize(Http()))
# Call the Photo v1 API
results = service.albums().list(
pageSize=10, fields="nextPageToken,albums(id,title)").execute()
items = results.get('albums', [])
if not items:
print('No albums found.')
else:
print('Albums:')
for item in items:
print('{0} ({1})'.format(item['title'].encode('utf8'), item['id']))
When executing the above code, it prompts me the auth page. When I successfully authenticate, it shows me the following error:
HttpError 403 when requesting {URL} returned "Photos Library API has not been used in project 123456 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/photoslibrary.googleapis.com/overview?project=123456 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.">
Interestingly, the number in bold 123456 (obviously changed) is actually the first part of the client_id found in the client_id.json
But the project_id looks something like this: test1-235515
So what I got from this error is that the oauth2client client is passing the client_id instead of the project_id. So even though I have enabled the Photos API, it will never access it correctly.
Please help with this error. How can I manually change the project_id?
The project ID is different from the project number. You will be able to see both in your Google Cloud Console configuration. See this documentation for more on how to identify your projects [1].
A single Google Cloud project can have many different OAuth client IDs configured. See this documentation for information about creating OAuth client credentials [2]. You should be only have to make sure that the client you created belongs to the project for which you have enabled APIs. Going to the URL provided in the error message should take you to the right configuration page.
[1] https://cloud.google.com/resource-manager/docs/creating-managing-projects#identifying_projects
[2] https://support.google.com/cloud/answer/6158849?hl=en