Following the instructions here:
https://developers.google.com/accounts/docs/OAuth2ServiceAccount?hl=en_US
I try running this python script on my google compute engine instance:
import httplib2
from googleapiclient.discovery import build
from oauth2client.gce import AppAssertionCredentials
credentials = AppAssertionCredentials("https://www.googleapis.com/auth/datastore")
http = credentials.authorize(httplib2.Http())
service = build('datastore', 'v1beta2', http=http)
x = service.datasets().lookup(body='', datasetId='surferjeff-easybates').execute(http=http)
But I still get this error:
Traceback (most recent call last):
File "C:/Users/surferjeff-easybates/Desktop/test.py", line 8, in <module>
x = service.datasets().lookup(body='', datasetId='surferjeff-easybates').execute(http=http)
File "C:\Python27\lib\site-packages\oauth2client\util.py", line 135, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Python27\lib\site-packages\googleapiclient\http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 401 when requesting https://www.googleapis.com/datastore/v1beta2/datasets/surferjeff-easybates/lookup?alt=json returned "Invalid Credentials">
And I have verified that my account has datastore enabled. What am I doing wrong?
The problem is related to a Google simple bug.
When you create a new instance using the cloud console and check the "Allow API access to all Google Cloud services in the same project", you are including this instance in all of the possible cloud scopes.
When you try to view which REST request generated this great functionality (by pressing the "Equivalent REST or command line" button) you see that the instance is generated using one global scope as follows:
"scopes": [
"https://www.googleapis.com/auth/cloud-platform"
]
If you attempt to create this instance using the Google API client (apiclient) with the same scopes you find out that it cannot get valid credentials to use the Datastore service.
The solution resides in the fact that the single scope is apparently not what Google assigns to your instance when you create it through the console.
If you open the instance details after creating it, you find out that it has the following scopes:
"scopes": [
"https://www.googleapis.com/auth/cloud-platform",
"https://www.googleapis.com/auth/datastore",
"https://www.googleapis.com/auth/userinfo.email"
]
Once you mention datastore's scope explicitly, things will work as expected.
Related
I am trying to make basic calls for gsuite, ala listing groups. I have poked around quite a bit and have seen several posts suggesting this is a scope issue. However, I have included as many scope paths as I can find that even looked remotely relevant. I still get an exception in the library because there is no "access_token" in the response. There IS an id_token, but nothing labelled "access_token", so the lib throws an exception.
Also, I am pretty sure the account has access to the "group settings API", and many other things.
Banging my head on the desk...
google-api-core==2.10.1
google-api-python-client==2.64.0
google-auth==2.12.0
google-auth-httplib2==0.1.0
google-auth-oauthlib==0.5.3
googleapis-common-protos==1.56.4
from __future__ import print_function
import os.path
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin',
'https://www.googleapis.com/auth/admin.directory.user',
"https://www.googleapis.com/auth/apps.groups.settings",
'https://www.googleapis.com/auth/admin',
'https://www.googleapis.com/auth/apps.groups.settings',
'https://www.googleapis.com/auth/indexing',
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/contacts',
'https://www.googleapis.com/auth/contacts.other.readonly'
'https://www.googleapis.com/auth/contacts.readonly',
'https://www.googleapis.com/auth/directory.readonly',
'https://www.googleapis.com/auth/user.addresses.read',
'https://www.googleapis.com/auth/user.birthday.read',
'https://www.googleapis.com/auth/user.emails.read',
'https://www.googleapis.com/auth/user.gender.read',
'https://www.googleapis.com/auth/user.organization.read',
'https://www.googleapis.com/auth/user.phonenumbers.read',
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/admin',
'https://www.googleapis.com/auth/service.management.readonly',
'https://www.googleapis.com/auth/monitoring',
'https://www.googleapis.com/auth/admin.reports.audit.readonly',
'https://www.googleapis.com/auth/admin.reports.usage.readonly',
'https://www.googleapis.com/auth/admin.datatransfer',
'https://www.googleapis.com/auth/admin.datatransfer.readonly',
'https://www.googleapis.com/auth/admin.directory.customer',
'https://www.googleapis.com/auth/admin.directory.customer.readonly',
'https://www.googleapis.com/auth/admin.directory.domain',
'https://www.googleapis.com/auth/admin.directory.domain.readonly',
'https://www.googleapis.com/auth/admin.directory.group',
'https://www.googleapis.com/auth/admin.directory.group.member',
'https://www.googleapis.com/auth/admin.directory.group.member.readonly',
'https://www.googleapis.com/auth/admin.directory.group.readonly',
'https://www.googleapis.com/auth/admin.directory.orgunit',
'https://www.googleapis.com/auth/admin.directory.orgunit.readonly',
'https://www.googleapis.com/auth/admin.directory.resource.calendar',
'https://www.googleapis.com/auth/admin.directory.resource.calendar.readonly',
'https://www.googleapis.com/auth/admin.directory.rolemanagement',
'https://www.googleapis.com/auth/admin.directory.rolemanagement.readonly',
'https://www.googleapis.com/auth/admin.directory.user',
'https://www.googleapis.com/auth/admin.directory.user.alias',
'https://www.googleapis.com/auth/admin.directory.user.alias.readonly',
'https://www.googleapis.com/auth/admin.directory.user.readonly',
'https://www.googleapis.com/auth/admin.directory.user.security',
'https://www.googleapis.com/auth/admin.directory.userschema',
'https://www.googleapis.com/auth/admin.directory.userschema.readonly',
'https://www.googleapis.com/auth/groups',
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/cloud-platform.read-only',
'https://www.googleapis.com/auth/apps.groups.migration']
SERVICE_ACCOUNT_FILE = 'privkey.json'
def main():
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = build('admin', 'directory_v1', credentials=credentials)
# this is where the library throws the exception
foobar = service.groups().list().execute()
if __name__ == '__main__':
main()
This is a partial stack trace:
Traceback (most recent call last):
File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 294, in jwt_grant
access_token = response_data["access_token"]
KeyError: 'access_token'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "../gsuite_auth_test6.py", line 93, in <module>
main()
File "../gsuite_auth_test6.py", line 90, in main
foobar = service.groups().list().execute()
File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/XXXXXXXXXX/.local/lib/python3.7/site-packages/googleapiclient/http.py", line 932, in execute
headers=self.headers,
File "/XXXXXXXXXXXXX/.local/lib/python3.7/site-packages/googleapiclient/http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google_auth_httplib2.py", line 209, in request
self.credentials.before_request(self._request, method, uri, request_headers)
File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/auth/credentials.py", line 133, in before_request
self.refresh(request)
File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/oauth2/service_account.py", line 411, in refresh
request, self._token_uri, assertion
File "/XXXXXXXXXXXXXX/.local/lib/python3.7/site-packages/google/oauth2/_client.py", line 299, in jwt_grant
six.raise_from(new_exc, caught_exc)
File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ('No access token in response.', {'id_token': '....'
Your research is right in that this can be caused by wrong scopes but there are more issues here that I will try to address.
I have included as many scope paths as I can find that even looked remotely relevant.
This is a problem. You cannot just throw all scopes to the wall and see what sticks. First you need to have enabled the relevant APIs on your GCP project. In this case you don't actually need the Groups Settings API, instead you need the Directory API, also known as Admin SDK API:
Admin SDK lets administrators of enterprise domains to view and manage resources like user, groups etc. It also provides audit and usage reports of domain.
Next, to know the scopes you can refer to Google's documentation. The groups.list method has the scopes that you need to use to list all Groups in the domain. In this case you can just use https://www.googleapis.com/auth/admin.directory.group.
Also, in the groups.list documentation I linked above you can see that you need to specify either the parameter customer for all groups within the Google Workspace account or domain for just one of its subdomains. You can also just use the my_customer alias to search everything within the account. The Python documentation here has its equivalent for the Python library. All of this means that your line to call groups.list should specify the customer like this:
foobar = service.groups().list(customer="my_customer").execute()
Finally, you seem to be using a service account's credentials. This means that you need to first enable domain-wide delegation for your service account so it can impersonate your user accounts, next, you have to impersonate an admin that already has the privileges to manage groups in the Admin Console. You can refer to the Google OAuth docs to check out how to do this, and here's also a Python sample. Pretty much you can just create a copy of the credentials while specifying the admin account, like this:
delegated_credentials=credentials.with_subject("admin#yourdomain.com")
Here's a sample based on your code that worked for me after taking all of the above into account:
from __future__ import print_function
import os.path
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/admin.directory.group']
SERVICE_ACCOUNT_FILE = 'privkey.json'
def main():
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
#delegate the admin credentials to the service account
delegated_credentials=credentials.with_subject("admin#yourdomain.com")
service = build('admin', 'directory_v1', credentials=delegated_credentials)
# make sure to add the customer parameter per the documentation requirements
foobar = service.groups().list(customer="my_customer").execute()
print(foobar)
if __name__ == '__main__':
main()
As you can see, aside from removing the unnecessary scopes I just changed two lines. First make sure that your GCP and service account configurations are in order and then don't forget to delegate access and specify the customer variable.
For full context I'm a newbie in scripting. I'm trying to get a full list of users from Google Admin using Google's API, but I'm struggling to even get passed the build() section of it. Whatever I tag onto service it just does not have that attribute, except for close()
My end goal is to pretty much get a full list of users, user aliases, and group names so I can check if they're already existing before creating/assigning one. Unfortunately, I'm stuck at this hurdle.
API I'm trying to use: https://developers.google.com/admin-sdk/directory
Code:
#!/usr/local/bin/env python3
import json
import os
import googleapiclient.discovery
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/admin.directory.user.readonly',
'https://www.googleapis.com/auth/admin.directory.user.alias.readonly',
'https://www.googleapis.com/auth/admin.directory.group.readonly']
SERVICE_ACCOUNT_FILE = os.get.environ('SERVICE_ACCOUNT_FILE')
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = googleapiclient.discovery.build('admin', 'directory_v1', credentials=credentials)
Here is the Error I keep running into if I were to just attempt to execute it with service.execute():
Traceback (most recent call last):
File "test.py", line 17, in <module>
service.execute()
AttributeError: 'Resource' object has no attribute 'execute'
If I were to run it without anything, just after the service = ... line, it does not spit out an error, so I'm assuming the service account file is correct.
The service variable, returned by build, corresponds to the class Resource.
This class can be used to interact with the API you specified when building the service. Depending on those parameters, the service will have specific API methods at its disposal, but execute() is not one of them.
Since you are building the service for Directory API, these will be the corresponding methods for interacting with this API:
Admin Directory API > Instance Methods
For example, if you wanted to call Users: list in order to list all users in your domain, you would do the following:
service = googleapiclient.discovery.build('admin', 'directory_v1', credentials=getCreds())
response = service.users().list(domain="YOUR_DOMAIN").execute()
The following lines from my Python app execute with no problems on my local machine.
import googleapiclient.discovery
project_id = 'some-project-id'
resource_manager = googleapiclient.discovery.build('cloudresourcemanager', 'v1')
iam_policy_request = resource_manager.projects().getIamPolicy(resource=project_id, body={})
iam_policy_response = iam_policy_request.execute(num_retries=3)
new_policy = dict()
new_policy['policy'] = iam_policy_response
del new_policy['policy']['version']
iam_policy_update_request = resourcemanager.projects().setIamPolicy(resource=project_id, body=new_policy)
update_result = iam_policy_update_request.execute(num_retries=3)
When I run the app in a GCE instance, and more precisely from within a Docker container inside the GCE instance, I get the exception:
URL being requested: POST https://cloudresourcemanager.googleapis.com/v1/projects/some-project-id:setIamPolicy?alt=json
Traceback (most recent call last):
File "/env/lib/python3.5/site-packages/google/api_core/grpc_helpers.py", line 54, in error_remapped_callable
return callable_(*args, **kwargs)
File "/env/lib/python3.5/site-packages/grpc/_channel.py", line 487, in __call__
return _end_unary_response_blocking(state, call, False, deadline)
File "/env/lib/python3.5/site-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.PERMISSION_DENIED, User not authorized to perform this action.)>
i.e. an authorization error. Oddly, when I open a Python terminal session inside the GCE instance and run the Python code line by line, I do not get the exception. It only throws the exception when the code is running as part of the app.
I am using a service account inside of the GCE instance, as opposed to my regular account on my local machine. But I don't think that is the problem since I am able to run the lines of code one by one inside of the instance while still relying on the service account roles.
I would like to be able to run the app without the exception within the Docker container inside of GCE. I feel like I'm missing something but can't figure out what the missing piece is.
Looking to your issue it seems an authentication issue, because your application is not properly authenticated :
1- First run this command it will let your application temporarily use your own user credentials:
gcloud beta auth application-default login
the output should be like this:
Credentials saved to file: $SOME_PATH/application_default_credentials.json
2-Then you have set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the path to the key file:
export GOOGLE_APPLICATION_CREDENTIALS=$SOME_PATH/application_default_credentials.json
Try to run you Application after that.
calling API and getting the error need to translate from English to France but getting this error so I have used this code but it is not working properly
code:
from __future__ import print_function
__author__ = 'jcgregorio#google.com (Joe Gregorio)'
from googleapiclient.discovery import build
def main():
service = build('translate', 'v2',
developerKey='AIzaSyDRRpR3GS1F1_jKNNM9HCNd2wJQyPG3oN0')
print(service.translations().list(
source='en',
target='fr',
q="flower"
).execute())
if __name__ == '__main__':
main()
Error
Traceback (most recent call last):
File "trail.py", line 19, in <module>
main()
File "trail.py", line 15, in main
q="flower"
File "build/bdist.linux-i686/egg/oauth2client/_helpers.py", line 133, in positional_wrapper
File "build/bdist.linux-i686/egg/googleapiclient/http.py", line 840, in execute
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/language/translate/v2?q=flower&source=en&alt=json&target=fr&key=AIzaSyDRRpR3GS1F1_jKNNM9HCNd2wJQyPG3oN0 returned "Bad Request">
Use the new Google Cloud client library and authorize your API using service account credentials JSON and calling export GOOGLE_APPLICATION_CREDENTIALS=your_service.json instead of using the API key. You can also use default application credentials by calling gcloud auth application-default login.
This is demonstrated in the Google Cloud Translate Python samples which should also help you get started more quickly.
Note the translate API requires billing, as described in the translate quickstart instructions, so be prepared to set that up for your project.
I'm porting our old user management scripts from the Google Provisioning API (which used the python gdata libraries) to the Google Directory API (the python admin-sdk libaries). So far most things have gone fine, however I've run into issues when attempting to do a discovery on what groups a user belongs to (which I need to remove membership from before a user deletion). Even stripping the code down to the barest essentials (replaced e-mails/credentials for public consumption):
#!/usr/bin/python
import httplib2
from apiclient import errors
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
SERVICE_ACCOUNT_EMAIL = 'XXXXXXXXXXXXXXXXXXXXXX#developer.gserviceaccount.com'
SERVICE_ACCOUNT_PKCS12_FILE_PATH = '/blah/blah/XXXXXXXX-privatekey.p12'
f = file(SERVICE_ACCOUNT_PKCS12_FILE_PATH, 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(SERVICE_ACCOUNT_EMAIL, key,
scope='https://www.googleapis.com/auth/admin.directory.user', sub='serviceaccount#our.tld')
service.users()
members = service.members().get(memberKey = 'serviceaccount#our.old', groupKey = 'googlegroup#our.tld').execute()
print members
This returns a 403 permissions error:
Traceback (most recent call last):
File "group_tests.py", line 39, in <module>
members = service.members().get(memberKey = 'serviceaccount#our.tld', groupKey = 'googlegroup#our.tld').execute()
File "/XXX/bin/gapps/lib/python2.6/site-packages/oauth2client/util.py", line 137, in positional_wrapper
return wrapped(*args, **kwargs)
File "/XXX/bin/gapps/lib/python2.6/site-packages/googleapiclient/http.py", line 729, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://www.googleapis.com/admin/directory/v1/groups/googlegroup%40our.tld/members/serviceaccount%40our.tld?alt=json returned "Insufficient Permission">
I don't recognize if the scope is wrong here, and if so what it should be? This service account is already set with permission for the following scopes (in Security>Advanced Security>API>Manage API client access):
https://www.googleapis.com/auth/admin.directory.user
https://www.googleapis.com/auth/admin.directory.group
Or should I be using groups instead of members? Like:
members = service.groups().get(memberKey = 'serviceaccount#our.old', groupKey = 'googlegroup#our.tld').execute()
Any pointers appreciated, I've been goggling around for any help on this for a week now to no avail.
Got this working:
First off, the scope in the credentials definition was incorrect.
admin.directory.user
changed to:
admin.directory.group
Also had the wrong initialization for the "build":
service.users()
changed to:
service.groups()
And the query statement itself was completely wrong, I went back to the reference doc and kept trying different changes until it took:
members = service.groups().list(domain = 'our.tld',userKey = 'user_whos_groups_i_want#our.tld',pageToken=None,maxResults=500).execute()
Hopefully this will be useful to someone else running into the same issue later. Please be aware that not all permission errors google will throw back are literally because of permissions, it may be your own code has conflicting scopes that you're trying to use.