In GAE standard environment, I'm struggling with registering the watch() call against Gmail API for the Pub/Sub push notification using google-api-python-client.
Here is the relevant excerpt from my code:
import googleapiclient.discovery
from oauth2client import service_account
SCOPES = ['https://www.googleapis.com/auth/gmail.modify']
SERVICE_ACCOUNT_FILE = '<My-project>-<short-ID>.json'
credentials = service_account.ServiceAccountCredentials.from_json_keyfile_name(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
gmail = googleapiclient.discovery.build('gmail', 'v1', credentials=credentials)
watchRequest = {
'labelIds' : ['INBOX'],
'topicName' : 'projects/<my-project>/topics/<my-topic>'
}
gmail.users().watch(userId='<email-I-need-to-watch>', body=watchRequest).execute()
After firing-off this part of the code I get:
Traceback (most recent call last):
File "/base/alloc/tmpfs/dynamic_runtimes/python27/54c5883f70296ec8_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 240, in Handle
handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
File "/base/alloc/tmpfs/dynamic_runtimes/python27/54c5883f70296ec8_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 299, in _LoadHandler
handler, path, err = LoadObject(self._handler)
File "/base/alloc/tmpfs/dynamic_runtimes/python27/54c5883f70296ec8_unzipped/python27_lib/versions/1/google/appengine/runtime/wsgi.py", line 85, in LoadObject
obj = __import__(path[0])
File "/base/data/home/apps/e~<my-project>-191008/20180124t154459.407164278206739455/main.py", line 68, in <module>
gmail.users().watch(userId='<email-I-need-to-watch>', body=watchRequest).execute()
File "/base/data/home/apps/e~<my-project>/20180124t154459.407164278206739455/lib/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/e~<my-project>/20180124t154459.407164278206739455/lib/googleapiclient/http.py", line 844, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/<email-I-need-to-watch>/watch?alt=json returned "Bad Request">
In regards to the authentication and authorization, here is what I have done so far:
I've created a Pub/Sub topic and this is the one I'm passing into the watch() request
I use G-Suite and the email inbox I intend to watch is part of my G-Suite business domain.
For this task I use a service account with enabled G-Suite Domain-wide Delegation. I've downloaded the .json service account file which I'm supplying in order to acquire the oauth2client.service_account.Credentials object (I see the access and refresh tokens being exchanged successfully in the logs). The json service file is placed in the same folder as my main.py script (root of my project).
In my G-Suite administration panel I've enabled the api access to the service account from 2. with the scope of https://www.googleapis.com/auth/gmail.modify. I'm using the gmail.modify access level as I intend to read, write and send both emails and drafts.
Is there something I'm missing out in my code or in the authentication and authorization steps?
Problem solved. I was missing the part of the code for impersonating a user from my domain in order to read his/her mailbox (as explained here).
The corrected code looks like this:
import googleapiclient.discovery
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/gmail.modify']
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES
)
credentials = credentials.with_subject('<email-I-need-to-watch>')
gmail = googleapiclient.discovery.build('gmail', 'v1', credentials=credentials)
watchRequest = {
'labelIds' : ['INBOX'],
'topicName' : 'projects/<my-project>/topics/<my-topic>'
}
gmail.users().watch(userId='me', body=watchRequest).execute()
Related
Since Microsoft has now dropped support for accessing an Exchange mailbox using basic authentication, I've had to upgrade some code to use oauth based access to the mailbox instead.
I've setup an Azure AD app following these docs:
https://learn.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth
Now I'd like to access emails from the mailbox using exchangelib and exchangelib.OAuth2Credentials
Here's the code I'm using:
from exchangelib import Account, OAuth2Credentials
AZURE_CLIENT_ID = "MY_CLIENT_ID"
AZURE_TENANT_ID = "MY_TENANT_ID"
AZURE_CLIENT_CREDENTIAL = "MY_CLIENT_CREDENTIAL"
MY_EMAIL = "me#example.com"
credentials = OAuth2Credentials(
client_id=AZURE_CLIENT_ID, client_secret=AZURE_CLIENT_CREDENTIAL, tenant_id=AZURE_TENANT_ID
)
account = Account(primary_smtp_address=MY_EMAIL, credentials=credentials, autodiscover=True)
for item in account.inbox.all().order_by('-datetime_received')[:100]:
print(item.subject, item.sender, item.datetime_received)
However I'm getting an error saying The SMTP address has no mailbox associated with it:
Traceback (most recent call last):
File "exchangelib_test.py", line 19, in <module>
account = Account(primary_smtp_address=MY_EMAIL, credentials=credentials, autodiscover=True)
File "/home/me/.local/lib/python3.6/site-packages/exchangelib/account.py", line 119, in __init__
email=primary_smtp_address, credentials=credentials, auth_type=auth_type, retry_policy=retry_policy
File "/home/me/.local/lib/python3.6/site-packages/exchangelib/autodiscover/discovery.py", line 114, in discover
ad_response = self._quick(protocol=ad_protocol)
File "/home/me/.local/lib/python3.6/site-packages/exchangelib/autodiscover/discovery.py", line 201, in _quick
return self._step_5(ad=ad)
File "/home/me/.local/lib/python3.6/site-packages/exchangelib/autodiscover/discovery.py", line 531, in _step_5
ad.raise_errors()
File "/home/me/.local/lib/python3.6/site-packages/exchangelib/autodiscover/properties.py", line 313, in raise_errors
raise ErrorNonExistentMailbox('The SMTP address has no mailbox associated with it')
exchangelib.errors.ErrorNonExistentMailbox: The SMTP address has no mailbox associated with it
Does anyone have an idea what might be going wrong here? The SMTP address definitely does exist and is an Exchange mailbox which I can log in to and view via Outlook.
If your using the client credentials flow then you will also need to use Impersonation access_type=IMPERSONATION eg
account = Account(primary_smtp_address=MY_EMAIL, credentials=credentials, autodiscover=True, access_type=IMPERSONATION)
generally with Office365 you don't want to use Autodiscover as the endpoint is always outlook.office365.com there is a sample of bypassing that in the samples page https://ecederstrand.github.io/exchangelib/ eg
config = Configuration(server='outlook.office365.com',
credentials=credentials)
account = Account(primary_smtp_address='john#example.com', config=config,
autodiscover=False, access_type=IMPERSONATION)
I am following the steps in https://developers.google.com/gmail/api/quickstart/python to create their example application that can print a Gmail user's labels. I get JSONDecodeError: Extra data instead when I execute quickstart.py. See The Error below.
The ultimate goal is to automate emailing invoices. However, I'm not familiar with the underlying concepts and without a working example I'm especially unclear on how to proceed.
Original Question
Any suggestions on either a) how to fix the issue or b) better resources with which to learn the concepts involved?
Resolution
When I deleted an extraneous ND found at the end of the credentials.json file, the JSONDecoderError problem disappeared. I don't know how it got there; presumably, the error was mine.
Follow-up comments
#DaImTo pointed out in their answer that the tutorial's example code is for installed applications and that "for google workspace automation you would want to use a service account and configure domain wide delegation to say a dummy user on your workspace account". See their answer for a slightly more in-depth explanation and suggested code for service account authorisation.
After correcting the credentials.json file, a new issue presented itself. I will not go into that issue here since, as per #DaImTo's answer, the tutorial is not the correct template for creating google workspace automation.
Future Updates
I will update how I progress with #DaImTo's suggestions; either here in the question body or as a comment or link.
The Error
Traceback (most recent call last):
File "C:\Users\Miranda and Warren\Documents\Invoice\Invoicing\quickstart.py", line 56, in <module>
main()
File "C:\Users\Miranda and Warren\Documents\Invoice\Invoicing\quickstart.py", line 30, in main
flow = InstalledAppFlow.from_client_secrets_file(
File "C:\Users\Miranda and Warren\Documents\Invoice\Invoicing\NDIS\lib\site-packages\google_auth_oauthlib\flow.py", line 205, in from_client_secrets_file
client_config = json.load(json_file)
File "C:\Python310\lib\json\__init__.py", line 293, in load
return loads(fp.read(),
File "C:\Python310\lib\json\__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "C:\Python310\lib\json\decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 406 (char 405)
The Tutorial Code
The quickstart.py code I copied and pasted from their start up guide is reproduced below:
from __future__ import print_function
import os.path
from google.auth.transport.requests import Request
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
from googleapiclient.errors import HttpError
# If modifying these scopes, delete the file token.json.
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
def main():
"""Shows basic usage of the Gmail API.
Lists the user's Gmail labels.
"""
creds = None
# The file token.json stores the user's access and refresh tokens, and is
# created automatically when the authorization flow completes for the first
# time.
if os.path.exists('token.json'):
creds = Credentials.from_authorized_user_file('token.json', SCOPES)
# If there are no (valid) credentials available, let the user log in.
if not creds or not creds.valid:
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request())
else:
flow = InstalledAppFlow.from_client_secrets_file(
'credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save the credentials for the next run
with open('token.json', 'w') as token:
token.write(creds.to_json())
try:
# Call the Gmail API
service = build('gmail', 'v1', credentials=creds)
results = service.users().labels().list(userId='me').execute()
labels = results.get('labels', [])
if not labels:
print('No labels found.')
return
print('Labels:')
for label in labels:
print(label['name'])
except HttpError as error:
# TODO(developer) - Handle errors from gmail API.
print(f'An error occurred: {error}')
if __name__ == '__main__':
main()
I followed the steps for "Authorize credentials for a desktop application". The following redacted JSON is what I had on my machine. Note the 'ND' at the end of the file.
(Invalid) contents of credentials.json
{
"installed": {
"client_id": "[46 characters here].apps.googleusercontent.com",
"project_id": "ndis-automation",
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_secret": "[6 characters]-[7 characters]-[20 characters]",
"redirect_uris": ["http://localhost"]
}
}ND
Extra data: line 1 column 406
Is probably caused by an invalid credentials.json file it would be interesting to see what it looks like.
What your question should have been 😁
The ultimate goal is to automate emailing invoices.
If you want to use gmail automated with google workspace, then that tutoiral is not going to help you.
The authorization method in that tutorial is for installed applications. For google workspace automation you would want to use a service account and configure domain wide delegation to say a dummy user on your workspace account.
The code for service account authorization is different, and you need to set the subject to the user on your domain which you have configured delegation for.
from google.oauth2 import service_account
from googleapiclient.discovery import build
credentials = service_account.Credentials.from_service_account_file(
credentials_file,
scopes=['https://www.googleapis.com/auth/gmail.send'])
impersonate = 'no-reply#daimto.com'
credentials = credentials.with_subject(impersonate)
service = build('gmail', 'v1', credentials=credentials)
I am trying to access Google Sheet (read write mode) from Python (runs in GKE). I have tried both outh2client as well as google-auth approach but it gives the same error every time:
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/1kvHv1OBCzr9GnFxRu9RTJC7jjQjc9M4rAiDnhyak2Sg/values/vm_metrics%21A10?alt=json returned "Request had insufficient authentication scopes.". Details: "[{'#type': 'type.googleapis.com/google.rpc.ErrorInfo', 'reason': 'ACCESS_TOKEN_SCOPE_INSUFFICIENT', 'domain': 'googleapis.com', 'metadata': {'method': 'google.apps.sheets.v4.SpreadsheetsService.GetValues', 'service': 'sheets.googleapis.com'}}]">
This is my code using outh2client:
from googleapiclient.discovery import build
from oauth2client import client
creds=client.GoogleCredentials.get_application_default().create_scoped(
['https://www.googleapis.com/auth/spreadsheets'])
service = build('sheets', 'v4', credentials=creds)
sheet = service.spreadsheets()
sheet.values().get(spreadsheetId='whatev', range='Sheet1!A:C').execute()
This is my code using google-auth:
import google.auth
SCOPES = ['https://www.googleapis.com/auth/spreadsheets']
creds, project=google.auth.default(scopes=SCOPES)
service = build('sheets', 'v4', credentials=creds)
sheet = service.spreadsheets()
sheet.values().get(spreadsheetId='XXXXXXXXXX', range='Sheet1!A:C').execute()
AFAICT, this is the same question as Google Sheet API access with Application Default credentials. The solution (from that SO post) is
from googleapiclient.discovery import build
from oauth2client import client
creds = client.GoogleCredentials.get_application_default().create_scoped(
['https://www.googleapis.com/auth/spreadsheets.readonly']
)
response = service.spreadsheets().values().get(
spreadsheetId='XXXXXXXXXX',
range='Sheet1!A:C'
).execute()
rows = response['values']
However, this will not work with a human user's application default credentials. You cannot run gcloud auth application-default login and use your user credentials. You'll get this error:
Traceback (most recent call last):
File "<ipython-input-2-2a9ba7e9e38f>", line 8, in <module>
sheet.values().get(spreadsheetId='1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms', range='Class Data!A2:E''....').execute()
File "/Users/dking/miniconda3/lib/python3.7/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Users/dking/miniconda3/lib/python3.7/site-packages/googleapiclient/http.py", line 855, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/1BxiMVs0XRA5nFMdKvBdBZjgmUUqptlbs74OgvE2upms/values/Class%20Data%21A2%3AE....?alt=json returned "Request had insufficient authentication scopes.">
AFAICT, there is no way to use a human user's application default credentials, which is tremendously annoying.
I have a gmail account in my g-suite organization for which I want to automatically read its gmail messages.
Since this needs to be run periodically with an automated process, the regular OAuth flow is not useful, since someone needs to open a browser and give permission to the automated access.
So far I've created a service account with domain-wide authority, as documented here.
The code I am using is:
import httplib2
from apiclient import discovery
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly', ]
CLIENT_SECRET_FILE = '/path/to/client_secrets.json'
def get_credentials():
from oauth2client.service_account import ServiceAccountCredentials
credentials = ServiceAccountCredentials.from_json_keyfile_name(CLIENT_SECRET_FILE, SCOPES)
delegated_credentials = credentials.create_delegated('myproject#myproject-123456.iam.gserviceaccount.com')
return delegated_credentials
def main():
credentials = get_credentials()
http = credentials.authorize(httplib2.Http())
service = discovery.build('gmail', 'v1', http=http)
results = service.users().messages().list(userId='user_to_impersonate#mycompany.com').execute()
if __name__ == '__main__':
main()
But I get a 400 Bad Request error:
Traceback (most recent call last):
File "gmail.py", line 77, in <module>
main()
File "gmail.py", line 65, in main
results = service.users().messages().list(userId='user_to_impersonate#mycompany.com').execute()
File "/usr/local/lib/python3.6/site-packages/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/googleapiclient/http.py", line 842, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 400 when requesting https://www.googleapis.com/gmail/v1/users/user_to_impersonate%40mycompany.com/messages?alt=json returned "Bad Request">
Is it possible at all to access a specific gmail account without having to grant permission from the browser? Do I need to perform any special step first? Or, is there any way get more information for debugging?
Based on the docs, you need to use delegated_credentials = credentials.create_delegated('user_to_impersonate#mycompany.com') instead of delegated_credentials = credentials.create_delegated('myproject#myproject-123456.iam.gserviceaccount.com').
I am trying to get push notifications regarding all the new inbox messages I receive in my email. I have configured the pubsub client as mentioned in the pubsub doc. Following is the code:
import httplib2
from apiclient import discovery
from oauth2client import client as oauth2client
from lib.common.models.main.users import User
from oauth2client.client import SignedJwtAssertionCredentials,\
AccessTokenRefreshError
PUBSUB_SCOPES = ['https://www.googleapis.com/auth/pubsub']
def create_pubsub_client(http=None):
credentials = oauth2client.GoogleCredentials.get_application_default()
if credentials.create_scoped_required():
credentials = credentials.create_scoped(PUBSUB_SCOPES)
if not http:
http = httplib2.Http()
credentials.authorize(http)
return discovery.build('pubsub', 'v1', http=http)
pubsub = create_pubsub_client()
topic = 'projects/<project-name>/topics/<Topic-name>'
policy = {
'policy': {
'bindings': [{
'role': 'roles/pubsub.publisher',
'members': ['serviceAccount:<my-service-account>'],
}],
}
}
resp = pubsub.projects().topics().setIamPolicy(resource=topic, body=policy).execute()
request = {
'labelIds': ['INBOX'],
'topicName': 'projects/<project-name>/topics/<Topic-name>'
}
f = file('link-to.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(
'service-account-email',
key,
scope='https://mail.google.com/',
)
user = User.query.filter(User.id == 143).first()
accesskey = user.get_access_key(True)
credentials.access_token = accesskey['key']
service = discovery.build('gmail','v1',credentials=credentials)
service.users().watch(userId='me', body=request).execute()
When I run the above Program, I encounter the following error:
Traceback (most recent call last):
File "/home/kmittal/workspace/engine/workers/newPubSubClient.py", line 82, in
service.users().watch(userId='me', body=request).execute()
File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 135, in positional_wrapper
return wrapped(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: https://www.googleapis.com/gmail/v1/users/me/watch?alt=json returned "Invalid topicName does not match projects/SOME_ANOTHER_PROJECT/topics/*">
As per this other answer, it looks like you can only create a watch for topics that are part of your own project, not another project. The documentation seems to back this up as it implies you must specify myproject (your own project ID):
The topic name can be any name you choose under your project (i.e.
matching projects/myproject/topics/*, where myproject is the Project
ID listed for your project in the Google Developers Console).