I made a personal use app that just queries my gmail for a certain type of email. The app is successfully running on a pi night and day making requests to the Gmail service every thirty seconds.
However every week or so I get this error:
messages = gmail.users().messages().list(userId='me').execute() #
q='is:unread'
File "C:\Users\rexma\Anaconda3\envs\stan_env\lib\site-
packages\oauth2client\_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\rexma\Anaconda3\envs\stan_env\lib\site-
packages\googleapiclient\http.py", line 839, in execute
method=str(self.method), body=self.body, headers=self.headers)
File "C:\Users\rexma\Anaconda3\envs\stan_env\lib\site-
packages\googleapiclient\http.py", line 166, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "C:\Users\rexma\Anaconda3\envs\stan_env\lib\site-
packages\oauth2client\transport.py", line 186, in new_request
credentials._refresh(orig_request_method)
File "C:\Users\rexma\Anaconda3\envs\stan_env\lib\site-
packages\oauth2client\client.py", line 761, in _refresh
self._do_refresh_request(http)
File "C:\Users\rexma\Anaconda3\envs\stan_env\lib\site-
packages\oauth2client\client.py", line 819, in _do_refresh_request
raise HttpAccessTokenRefreshError(error_msg, status=resp.status)
oauth2client.client.HttpAccessTokenRefreshError: invalid_grant: Token has
been expired or revoked.
Which forces me to go back in, delete my old credentials and rerun the authorization. I've tried to do the other fixes online like setting
flow.authorization_url(access_type = 'offline', approval_prompt='force')
But the 'OAuth2WebServerFlow' object has no attribute 'authorization_url'.
Here is my full authorization code:
def authenticate(CLIENT_SECRET_FILE, APPLICATION_NAME, SCOPE, credential_name):
home_dir = os.path.expanduser('~')
credential_dir = os.path.join(home_dir, '.credentials')
if not os.path.exists(credential_dir):
os.makedirs(credential_dir)
credential_path = os.path.join(credential_dir, credential_name)
store = Storage(credential_path)
credentials = store.get()
if not credentials or credentials.invalid:
flow = flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPE)
flow.user_agent = APPLICATION_NAME
# flow.authorization_url(access_type='offline', approval_prompt='force')
credentials = run_flow(flow, store)
return credentials
I'm definitely still a scrub with OAuth2, even having read this. I just want to know how I can have my app use the refresh token in the json credentials to keep running without expiring.
Any help is much appreciated.
Related
I am using service account credentials and gspread to read from a Google Sheets worksheet.
from oauth2client.service_account import ServiceAccountCredentials
import gspread
gspread version: 5.6.2
oauth2client version: 4.1.3
python version: 3.8.2
This is the code I am using to read the data
def __init__(self):
SCOPES = ['https://www.googleapis.com/auth/spreadsheets',
'https://www.googleapis.com/auth/drive']
creds = ServiceAccountCredentials.from_json_keyfile_name('credentials.json', SCOPES)
self.file = gspread.authorize(creds)
self.workbook = self.file.open("Example Data")
print(self.workbook)
self.populateAllData()
def populateAllData(self):
self.worksheet = self.workbook.get_worksheet_by_id(0)
self.config_worksheet = self.workbook.get_worksheet_by_id(xxxxx)
self.config_worksheet_local = self.config_worksheet.get_values()
self.worksheet_local = self.worksheet.get_values()
At the first call of get_values, I get the following error.
File "main.py", line 31, in populateAllData
self.config_worksheet_local = self.config_worksheet.get_values()
File "/Users/xxxxxx/Library/Python/3.8/lib/python/site-packages/gspread/utils.py", line 674, in wrapper
return f(*args, **kwargs)
File "/Users/xxxxxx/Library/Python/3.8/lib/python/site-packages/gspread/worksheet.py", line 403, in get_values
return fill_gaps(self.get(range_name, **kwargs))
File "/Users/xxxxxx/Library/Python/3.8/lib/python/site-packages/gspread/utils.py", line 674, in wrapper
return f(*args, **kwargs)
File "/Users/xxxxxx/Library/Python/3.8/lib/python/site-packages/gspread/worksheet.py", line 713, in get
response = self.spreadsheet.values_get(range_name, params=params)
File "/Users/xxxxxxx/Library/Python/3.8/lib/python/site-packages/gspread/spreadsheet.py", line 182, in values_get
r = self.client.request("get", url, params=params)
File "/Users/xxxxxx/Library/Python/3.8/lib/python/site-packages/gspread/client.py", line 92, in request
raise APIError(response)
gspread.exceptions.APIError: {'code': 500, 'message': 'Internal error encountered.', 'status': 'INTERNAL'}
This piece of code with the credentials provided was working properly till a few days back, but now it consistently fails with this error. I saw people facing the same issue in the past because of making too many requests or reading from a sheet with large amount of data. But that is not the problem in my case.
Has anyone faced the same problem? Or can guide me to where I can read more logs or get more details about the error, and a possible path to debug this?
I'm trying to access blogger API using the official Google python library and a service account with a JSON key file made at the Google Developer console.
I'm using Linux Debian.
google-api-python-client=2.65.0
The thing is that I'm getting this traceback:
Traceback (most recent call last):
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/discovery.py", line 287, in build
content = _retrieve_discovery_doc(
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/discovery.py", line 422, in _retrieve_discovery_doc
resp, content = req.execute(num_retries=num_retries)
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
return wrapped(*args, **kwargs)
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/http.py", line 923, in execute
resp, content = _retry_request(
File "/path/to/myproject/lib/python3.9/site-packages/googleapiclient/http.py", line 191, in _retry_request
resp, content = http.request(uri, method, *args, **kwargs)
File "/path/to/myproject/lib/python3.9/site-packages/httplib2/__init__.py", line 1322, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/path/to/myproject/lib/python3.9/site-packages/httplib2/__init__.py", line 1072, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/path/to/myproject/lib/python3.9/site-packages/httplib2/__init__.py", line 996, in _conn_request
conn.request(method, request_uri, body, headers)
File "/usr/lib/python3.9/http/client.py", line 1279, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1325, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1274, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/lib/python3.9/http/client.py", line 1034, in _send_output
self.send(msg)
File "/usr/lib/python3.9/http/client.py", line 995, in send
self.sock.sendall(data)
OSError: [Errno 9] Bad file descriptor
Which point to some sort of error at the OS when opening some file, but I can't imagine which is the file that is causing trouble. The credentials seem like are doing all right when recovered from the json file and at the moment of the exception is just sending this:
b'GET /discovery/v1/apis/blogger/v3/rest HTTP/1.1\r\nHost: www.googleapis.com\r\ncontent-length: 0\r\nuser-agent: Python-httplib2/0.10.3 (gzip)\r\naccept-encoding: gzip, deflate\r\n\r\n'
through the internet. This also seems to do fine from my browser.
The code is here below:
import json
import google.auth
from google.oauth2 import service_account
import googleapiclient.discovery
from oauth2client import client
from oauth2client.service_account import ServiceAccountCredentials
SCOPES = ['https://www.googleapis.com/auth/blogger']
def get_key_dict():
SERVICE_ACCOUNT_FILE = '/path/to/myproject/apis/blogger/private_secrets.json'
with open(SERVICE_ACCOUNT_FILE, 'r') as keyfile:
return json.load(keyfile)
key_dict = get_key_dict()
credentials = ServiceAccountCredentials.from_json_keyfile_dict(key_dict, scopes=SCOPES)
# Here it breaks!
service = googleapiclient.discovery.build('blogger', 'v3', credentials=credentials, static_discovery=False,)
try:
users = service.users()
# Retrieve this user's profile information
thisuser = users.get(userId="self").execute()
print("This user's display name is: %s" % thisuser["displayName"])
blogs = service.blogs()
# Retrieve the list of Blogs this user has write privileges on
thisusersblogs = blogs.listByUser(userId="self").execute()
for blog in thisusersblogs["items"]:
print("The blog named '%s' is at: %s" % (blog["name"], blog["url"]))
posts = service.posts()
# List the posts for each blog this user has
for blog in thisusersblogs["items"]:
print("The posts for %s:" % blog["name"])
request = posts.list(blogId=blog["id"])
while request != None:
posts_doc = request.execute()
if "items" in posts_doc and not (posts_doc["items"] is None):
for post in posts_doc["items"]:
print(" %s (%s)" % (post["title"], post["url"]))
request = posts.list_next(request, posts_doc)
except client.AccessTokenRefreshError:
print(
"The credentials have been revoked or expired, please re-run"
"the application to re-authorize"
)
if you check the documentation Authorizing requests and identifying your application you will find that it states that you can use Oauth2 for authorizing a user to the blogger api and api keys. There is no mention of using service account authorization with this api.
To my knowledge as i tried about five years ago this api does not support service account authentication. You will need to use Oauth2.
I am building a python client-side application that uses Firestore. I have successfully used Google Identity Platform to sign up and sign in to the Firebase project, and created a working Firestore client using google.cloud.firestore.Client which is authenticated as a user:
import json
import requests
import google.oauth2.credentials
from google.cloud import firestore
request_url = f"https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key={self.__api_key}"
headers = {"Content-Type": "application/json; charset=UTF-8"}
data = json.dumps({"email": self.__email, "password": self.__password, "returnSecureToken": True})
response = requests.post(request_url, headers=headers, data=data)
try:
response.raise_for_status()
except (HTTPError, Exception):
content = response.json()
error = f"error: {content['error']['message']}"
raise AuthError(error)
json_response = response.json()
self.__token = json_response["idToken"]
self.__refresh_token = json_response["refreshToken"]
credentials = google.oauth2.credentials.Credentials(self.__token,
self.__refresh_token,
client_id="",
client_secret="",
token_uri=f"https://securetoken.googleapis.com/v1/token?key={self.__api_key}"
)
self.__db = firestore.Client(self.__project_id, credentials)
I have the problem, however, that when the token has expired, I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Missing or invalid authentication."
debug_error_string = "{"created":"#1613043524.699081937","description":"Error received from peer ipv4:172.217.16.74:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Missing or invalid authentication.","grpc_status":16}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/my_app/src/controllers/im_alive.py", line 20, in run
self.__device_api.set_last_updated(utils.device_id())
File "/home/my_app/src/api/firestore/firestore_device_api.py", line 21, in set_last_updated
"lastUpdatedTime": self.__firestore.SERVER_TIMESTAMP
File "/home/my_app/src/api/firestore/firestore.py", line 100, in update
ref.update(data)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/document.py", line 382, in update
write_results = batch.commit()
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/batch.py", line 147, in commit
metadata=self._client._rpc_metadata,
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/gapic/firestore_client.py", line 1121, in commit
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/dist-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/dist-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Missing or invalid authentication.
I have tried omitting the token and only specifying the refresh token, and then calling credentials.refresh(), but the expires_in in the response from the https://securetoken.googleapis.com/v1/token endpoint is a string instead of a number (docs here), which makes _parse_expiry(response_data) in google.oauth2._client.py:257 raise an exception.
Is there any way to use the firestore.Client from either google.cloud or firebase_admin and have it automatically handle refreshing tokens, or do I need to switch to the manually calling the Firestore RPC API and refreshing tokens at the correct time?
Note: There are no users interacting with the python app, so the solution must not require user interaction.
Can't you just pass the string cast as integer _parse_expiry(int(float(response_data))) ?
If it is not working you could try to make a call and refresh token after getting and error 401, see my answer for the general idea on how to handle tokens.
As mentioned by #Marco, it is recommended that you use a service account if it's going to be used in an environment without user. When you use service account, you can just set GOOGLE_APPLICATION_CREDENTIALS environment variable to location of service account json file and just instantiate the firestore Client without any credentials (The credentials will be picked up automatically):
import firestore
client = firestore.Client()
and run it as (assuming Linux):
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
$ python file.py
Still, if you really want to use user credentials for the script, you can install the Google Cloud SDK, then:
$ gcloud auth application-default login
This will open browser and for you to select account and login. After logging in, it creates a "virtual" service account file corresponding to your user account (that will also be loaded automatically by clients). Here too, you don't need to pass any parameters to your client.
See also: Difference between “gcloud auth application-default login” and “gcloud auth login”
I'm trying to use the Google Slides API on Google App Engine, and despite using the Google code samples (specifically for OAuth2 & the Slides API on App Engine), I'm running into problems.
Here is my App Engine code, with unnecessary cruft removed (everything's in main.app). What I'm doing is trying to posting a string from an HTML form and then build a blank presentation. I've already used the Slides API with a simple script that I prototyped; I'm now trying to make this self-serve via an App Engine app, but it's the change in authentication that's tripping me up.
from googleapiclient import discovery
from oauth2client import client
from oauth2client.contrib import appengine
from google.appengine.api import memcache
CLIENT_SECRETS = os.path.join(os.path.dirname(__file__), 'client_secrets.json')
MISSING_CLIENT_SECRETS_MESSAGE = """[omitted]""" % CLIENT_SECRETS
http = httplib2.Http()
service = discovery.build('slides', 'v1', http=http)
decorator = appengine.OAuth2DecoratorFromClientSecrets(
CLIENT_SECRETS,
scope='https://www.googleapis.com/auth/presentations https://www.googleapis.com/auth/drive',
message=MISSING_CLIENT_SECRETS_MESSAGE)
class SlideBuilder(webapp2.RequestHandler):
#decorator.oauth_required
def post(self):
programslug = self.request.get('programid')
presoname = str(programslug) + ' Mentors'
presentationbody = {
'title': presoname
}
presentation = service.presentations().create(body=presentationbody).execute()
I want to point out that I downloaded the most recent client_secrets.json directly from the API console, so that should match up correctly for CLIENT_SECRETS.
The error I'm getting (on dev server; but it's also on the live app) is this:
Traceback (most recent call last):
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/Users/jedc/pm-tools/oauth2client/contrib/appengine.py", line 644, in check_oauth
resp = method(request_handler, *args, **kwargs)
File "/Users/jedc/pm-tools/main.py", line 113, in post
presentation = service.presentations().create(body=presentationbody).execute()
File "/Users/jedc/pm-tools/oauth2client/_helpers.py", line 133, in positional_wrapper
return wrapped(*args, **kwargs)
File "/Users/jedc/pm-tools/googleapiclient/http.py", line 840, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 401 when requesting https://slides.googleapis.com/v1/presentations?alt=json returned "Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.">
It feels like there's something subtle but dumb that I'm doing here. I'd appreciate any help or pointers to figure out what that is!
This error happening because your http is not authorized with the credentials.
To authorize the http with the credentials you should use the decorator.
decorator = appengine.OAuth2DecoratorFromClientSecrets(
CLIENT_SECRETS,
scope='https://www.googleapis.com/auth/presentations https://www.googleapis.com/auth/drive',
message=MISSING_CLIENT_SECRETS_MESSAGE)
http = decorator.http()
service = discovery.build('slides', 'v1', http=http)
This will fix your problem.
For further reference read this app engine decorators documentation from Google
I am using "bigquery_service = build('bigquery', 'v2', http=http)" and it is executing correctly in my laptop, but when I am executing this query on the server I am getting the following error.
File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "QueryAPI.py", line 117, in localMain
exportDataToGCS(canonicalDate);
File "QueryAPI.py", line 177, in exportDataToGCS
bigquery_service = build('bigquery', 'v2', http=http)
File "/homeBigQuery/src/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/homeBigQuery/src/googleapiclient/discovery.py", line 198, in build
resp, content = http.request(requested_url)
File "/homeBigQuery/src/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/homeBigQuery/src/oauth2client/client.py", line 538, in new_request
redirections, connection_type)
File "/homeBigQuery/src/httplib2/__init__.py", line 1570, in request
(response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
File "/homeBigQuery/src/httplib2/__init__.py", line 1317, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, headers)
File "/homeBigQuery/src/httplib2/__init__.py", line 1286, in _conn_request
response = conn.getresponse()
File "/usr/lib64/python2.6/httplib.py", line 980, in getresponse
raise ResponseNotReady()
You could try this 3rd party library for BigQuery. It's a thin wrapper around the Google BigQuery API.
To login to BigQuery:
import bigquery as bq
with open('my_key.pem', 'r') as key:
key_content = key.read()
bq_client = bq.get_client(project_id='foo',
service_account='bar',
private_key=key_content)
To submit a load job:
schema = [{'name':'foo', 'type':'STRING'}]
job = bq_client.import_data_from_uris(source_uris=['gs://foo/bar'],
dataset='foo',
table='bar',
schema=schema,
source_format=bq.client.JOB_FORMAT_CSV,
write_disposition=bq.client.JOB_WRITE_TRUNCATE,
field_delimiter=',',
skip_leading_rows=1)
try:
job_resource = bq_client.wait_for_job(job, timeout=60)
except bq.errors.BigQueryTimeoutException:
logging.critical('BigQuery loading timeout.')
Have you tried removing the "http=http" parameter ?
here is an example of the use of the BQ API:
import httplib2
from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
# REPLACE WITH YOUR Project ID
PROJECT_NUMBER = 'XXXXXXXXXXX'
# REPLACE WITH THE SERVICE ACCOUNT EMAIL FROM GOOGLE DEV CONSOLE
SERVICE_ACCOUNT_EMAIL = 'XXXXX#developer.gserviceaccount.com'
# OBTAIN THE KEY FROM THE GOOGLE APIs CONSOLE
# More instructions here: http://goo.gl/w0YA0
f = file('key.p12', 'rb')
key = f.read()
f.close()
credentials = SignedJwtAssertionCredentials(
SERVICE_ACCOUNT_EMAIL,
key,
scope='https://www.googleapis.com/auth/bigquery')
http = httplib2.Http()
http = credentials.authorize(http)
service = build('bigquery', 'v2')
datasets = service.datasets()
response = datasets.list(projectId=PROJECT_NUMBER).execute(http)
print 'Dataset list:'
for dataset in response['datasets']:
print '%s' % dataset['datasetReference']['datasetId']
Solved this by specifying the proxy server host and port for the server that I was running this on. While initiating an authorized http service object with the help of credentials, I modified the code as follows:
BEFORE:
http = httplib2.Http()
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)
FIX:
http = httplib2.Http(proxy_info = httplib2.ProxyInfo(httplib2.socks.PROXY_TYPE_HTTP, 'your-proxy-host-name', your-proxy-port))
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)