We Keep Coding

How to connect Google Sheets from Google Cloud Datalab using Python SDK and service account?

I'm trying to fetch data from a shared Google Sheet using Python SDK from Google Cloud Datalab.
I did the following:
Created a new Notebook on Google Cloud Datalab
Obtained a service account which is used by the Notebook, as described on the Key concepts & components page: “To display the cloud project and service account names, click the user icon in the top-right corner of the Cloud Datalab notebook or notebook listing page in your browser”.
Shared the Google Sheet with the service account email.
Run any of the following code snippets
Using Google APIs Python SDK
import google.auth
from googleapiclient.discovery import build
credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/spreadsheets.readonly'])
service = build('sheets', 'v4', credentials=credentials)
sheet = service.spreadsheets()
result = sheet.values().get(spreadsheetId=GOOGLE_SHEET_ID,
range=SHEET_RANGE).execute()
values = result.get('values', [])
Got error:
HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/<my sheet id here>/values/<my sheet range here>?alt=json returned "Request had insufficient authentication scopes.">
Using gspread
%%bash
pip install gspread
import gspread
import google.auth
credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/spreadsheets.readonly'])
gc = gspread.authorize(credentials)
sheet = gc.open_by_key(GOOGLE_SHEET_ID)
worksheet = sheet.get_worksheet(MY_WORKSHEET)
Got error:
APIError: {'status': 'PERMISSION_DENIED', 'message': 'Request had insufficient authentication scopes.', 'details': [{'metadata': {'method': 'google.apps.sheets.v4.SpreadsheetsService.GetSpreadsheet', 'service': 'sheets.googleapis.com'}, 'reason': 'ACCESS_TOKEN_SCOPE_INSUFFICIENT', 'domain': 'googleapis.com', '#type': 'type.googleapis.com/google.rpc.ErrorInfo'}], 'code': 403}

GTasks API Service Account: Client is unauthorized to retrieve access tokens using this method

I have a service account with Domain wide delegation and its been working fine for Calendar APIs. I am able to impersonate as other users within the organization to CRUD events in Google Calendar without the Consent Screen. Now i'm trying to consume GTasks API but getting error.
For GTasks
credentials = service_account.Credentials.from_service_account_file('service-account.json', scopes=['https://www.googleapis.com/auth/tasks'])
delegated_credentials = credentials.with_subject('username#company.com')
service = build('tasks', 'v1', credentials=delegated_credentials)
tasklists = service.tasklists().list(maxResults=10).execute()
and got below error.
('unauthorized_client: Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested.', '{\n "error": "unauthorized_client",\n "error_description": "Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested."\n}')
For GCalendar, working fine
credentials = service_account.Credentials.from_service_account_file('service-account.json', scopes=['https://www.googleapis.com/auth/calendar'])
delegated_credentials = credentials.with_subject('username#company.com')
service = build('calendar', 'v3', credentials=delegated_credentials)
events = service.events().list(calendarId=SUBJECT,timeMin=now,maxResults=10,orderBy='startTime',singleEvents=True).execute()
I'm using this lib https://github.com/googleapis/google-api-python-client. Any hint what's wrong here?

Not all APIs support domain-wide delegation. If they do, you will see it mentioned in the Auth documentation.
The Calendar API supports domain-wide delegation, and the Tasks API does not.
https://developers.google.com/calendar/auth
https://developers.google.com/tasks/auth

Google Calendar and Tasks both support domain wide-delegation, I am using that. My suggestion is:
Check if your developer console has enabled Google Tasks API
Check if you've added Google Tasks into your API permission scope in your G Suite admin dashboard. May refer https://developers.google.com/identity/protocols/OAuth2ServiceAccount for more detail

Service account can't connect to a Google Sheet using domain wide delegation

I have a spreadsheet on our company Google Drive that I would like to be read by a service account. I've been stuck on this for some time now but I can't figure out what I'm missing.
For that I created a service account, enabled the domain wide delegation and added the needed scopes ('https://www.googleapis.com/auth/drive' and 'https://www.googleapis.com/auth/spreadsheets.readonly') in the G Suite Admin panel, matching the client id of my service account. I created a key for this service account and save it in a json file (file is key_sa.json).
Despite that, when running the following code, it fails:
SERVICE_ACCOUNT_JSON_FILE_PATH = 'key_sa.json'
SCOPES = ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/spreadsheets.readonly']
credentials = ServiceAccountCredentials.from_json_keyfile_name(
SERVICE_ACCOUNT_JSON_FILE_PATH, scopes=SCOPES)
delegated_credentials = credentials.create_delegated('user#company.com')
delegated_http = delegated_credentials.authorize(httplib2.Http())
delegated_credentials.refresh(delegated_http)
service = build('sheets', 'v4', http=delegated_http) # credentials=credentials)
The error message I got for the last line is:
oauth2client.client.HttpAccessTokenRefreshError: unauthorized_client: Client is unauthorized to retrieve access tokens using this method, or client not authorized for any of the scopes requested.
Any hint as to what I'm missing?

Try adding the following scopes as mentioned below:
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/spreadsheets.readonly',
'https://www.googleapis.com/auth/spreadsheets'
These scopes must be added to both in your code and the API permissions in Google Admin Security setting.
...spreadsheets.readonly
...spreadsheets
both the above permissions behave differently and so both need to be added.

Use Application Default Credentials on Google Compute Engine to access Sheets API

Does the ADC (Application Default Credentials) workflow only support Google Cloud APIs (for example, supports for Google Cloud Storage API, but not the Google Sheet API)?
I'm referring to google.auth's default method - not having to store any private keys with the code is a great win and the main benefit of making effective use of the ADC (Application Default Credentials) setup.
The following code works if I set the GOOGLE_APPLICATION_CREDENTIALS environmental variable to the private key file, say key.json. This is inline with the default method as per step 1 of the google.auth package: 1. If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set to the path of a valid service account JSON private key file, then it is loaded and returned.
import google.auth
from apiclient import discovery
credentials, project_id = google.auth.default(scopes=['https://www.googleapis.com/auth/spreadsheets'])
sheets = discovery.build('sheets', 'v4', credentials=credentials)
SPREADSHEETID = '....'
result = sheets.spreadsheets().values().get(spreadsheetId=SPREADSHEETID, range='Sheet1!A:B').execute()
print result.get('values', [])
Now, looking at step 4 of the method: 4. If the application is running in Compute Engine or the App Engine flexible environment then the credentials and project ID are obtained from the Metadata Service.
If i remove the GOOGLE_APPLICATION_CREDENTIALS environmental variable on a Google Compute instance, I get the following error:
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/..../values/Sheet1%21A%3AB?alt=json returned "Request had insufficient authentication scopes.">
This is not consistent with Google's wizard as per the Cloud Console:

Credentials are optional for the build constructor. Omit them and the Cloud Function service account will be used to authenticate against the Sheet.
from apiclient import discovery
sheets = discovery.build('sheets', 'v4')
SPREADSHEETID = '....'
result = sheets.spreadsheets().values().get(spreadsheetId=SPREADSHEETID, range='Sheet1!A:B').execute()
print result.get('values', [])

According to this documentation, the scope that you're using requires Oauth 2.0 authorization. Therefore, a user login and consent is required.

Receiving 403 error when accessing G Suite Admin SDK using google-api-python-client

i'm trying to write a simple script that will get the users list from my Google G Suite domain from Directory API of Admin SDK using google-api-python-client. I've read tons of documentation, tried hundreds of various requests, but always receive: googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/users?domain=example.com&alt=json returned "Not Authorized to access this resource/api"> error.
This is what i did:
In the Google Developer console https://console.developers.google.com:
created a new project
enabled 'Admin SDK' API.
created a Service account Key
saved the generated key into a 'service-key.json' file
In the G Suite Admin console:
API access is enabled
Admin SDK is enabled
'Client ID' of the service key ^^ is authorized to 'View users on your domain', scope: https://www.googleapis.com/auth/admin.directory.user.readonly in the API client access console section.
Created a simple test script:
#!/usr/bin/env python3
import json
from httplib2 import Http
from oauth2client.service_account import ServiceAccountCredentials
from apiclient.discovery import build
scopes = ['https://www.googleapis.com/auth/admin.directory.user.readonly']
credentials = ServiceAccountCredentials.from_json_keyfile_name(
'service-key.json', scopes)
account = credentials.authorize(Http())
service = build('admin', 'directory_v1', http=account)
response = service.users().list(domain='example.com').execute()
print(response)
Other:
tried also 'Enable G Suite Domain-wide Delegation' (used create_delegated() method on a ServiceAccountCredentials object)
i see in the Google Developer Console - Dashboard, that the script is issuing the proper requests - can see the 'directory.users.list' API methods are being issued, but fails with 403 error
Thanks in advance for any help!

#Mr.Rebot's suggestion worked for me. The basic service account wouldn't work but impersonating the admin (which you have enabled it to do), allows the API call to pass.