We Keep Coding

Firestore client in python (as user) using firebase_admin or google.cloud.firestore

I am building a python client-side application that uses Firestore. I have successfully used Google Identity Platform to sign up and sign in to the Firebase project, and created a working Firestore client using google.cloud.firestore.Client which is authenticated as a user:
import json
import requests
import google.oauth2.credentials
from google.cloud import firestore
request_url = f"https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key={self.__api_key}"
headers = {"Content-Type": "application/json; charset=UTF-8"}
data = json.dumps({"email": self.__email, "password": self.__password, "returnSecureToken": True})
response = requests.post(request_url, headers=headers, data=data)
try:
response.raise_for_status()
except (HTTPError, Exception):
content = response.json()
error = f"error: {content['error']['message']}"
raise AuthError(error)
json_response = response.json()
self.__token = json_response["idToken"]
self.__refresh_token = json_response["refreshToken"]
credentials = google.oauth2.credentials.Credentials(self.__token,
self.__refresh_token,
client_id="",
client_secret="",
token_uri=f"https://securetoken.googleapis.com/v1/token?key={self.__api_key}"
)
self.__db = firestore.Client(self.__project_id, credentials)
I have the problem, however, that when the token has expired, I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Missing or invalid authentication."
debug_error_string = "{"created":"#1613043524.699081937","description":"Error received from peer ipv4:172.217.16.74:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Missing or invalid authentication.","grpc_status":16}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/my_app/src/controllers/im_alive.py", line 20, in run
self.__device_api.set_last_updated(utils.device_id())
File "/home/my_app/src/api/firestore/firestore_device_api.py", line 21, in set_last_updated
"lastUpdatedTime": self.__firestore.SERVER_TIMESTAMP
File "/home/my_app/src/api/firestore/firestore.py", line 100, in update
ref.update(data)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/document.py", line 382, in update
write_results = batch.commit()
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/batch.py", line 147, in commit
metadata=self._client._rpc_metadata,
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/gapic/firestore_client.py", line 1121, in commit
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/dist-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/dist-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Missing or invalid authentication.
I have tried omitting the token and only specifying the refresh token, and then calling credentials.refresh(), but the expires_in in the response from the https://securetoken.googleapis.com/v1/token endpoint is a string instead of a number (docs here), which makes _parse_expiry(response_data) in google.oauth2._client.py:257 raise an exception.
Is there any way to use the firestore.Client from either google.cloud or firebase_admin and have it automatically handle refreshing tokens, or do I need to switch to the manually calling the Firestore RPC API and refreshing tokens at the correct time?
Note: There are no users interacting with the python app, so the solution must not require user interaction.

Can't you just pass the string cast as integer _parse_expiry(int(float(response_data))) ?
If it is not working you could try to make a call and refresh token after getting and error 401, see my answer for the general idea on how to handle tokens.

As mentioned by #Marco, it is recommended that you use a service account if it's going to be used in an environment without user. When you use service account, you can just set GOOGLE_APPLICATION_CREDENTIALS environment variable to location of service account json file and just instantiate the firestore Client without any credentials (The credentials will be picked up automatically):
import firestore
client = firestore.Client()
and run it as (assuming Linux):
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
$ python file.py
Still, if you really want to use user credentials for the script, you can install the Google Cloud SDK, then:
$ gcloud auth application-default login
This will open browser and for you to select account and login. After logging in, it creates a "virtual" service account file corresponding to your user account (that will also be loaded automatically by clients). Here too, you don't need to pass any parameters to your client.
See also: Difference between “gcloud auth application-default login” and “gcloud auth login”

Google Calendar API HttpError 404 when using service account

I am trying to get a list of events on my calendar from a service account so that I do not have to authenticate myself to display my calendar. When I run the following code I get a 404 error:
SCOPES = ['https://www.googleapis.com/auth/calendar.readonly']
SERVICE_ACCOUNT_FILE = os.path.join(os.path.dirname(
os.path.realpath(__file__)), 'static/calendarSync/service-account.json')
calId = "blakewright1021#gmail.com"
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
google_account = googleapiclient.discovery.build(
'calendar', 'v3', credentials=credentials)
cal_list = google_account.calendarList() # pylint: disable=no-member
#page_token = None
#calendar_list = cal_list.list(pageToken=page_token).execute()
# for calendar_list_entry in calendar_list['items']:
# print("Here: ")
# print(calendar_list_entry['summary'])
calendar = cal_list.get(
calendarId=calId)
output = calendar.execute()['summary']
When I use the code that is commented out to try and print a list of calendars, it does not print anything so the list must be empty.
Here is the traceback:
Internal Server Error: /
Traceback (most recent call last):
File "C:\Python38\lib\site-packages\django\core\handlers\exception.py", line 34, in inner
response = get_response(request)
File "C:\Python38\lib\site-packages\django\core\handlers\base.py", line 115, in _get_response
response = self.process_exception_by_middleware(e, request)
File "C:\Python38\lib\site-packages\django\core\handlers\base.py", line 113, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "C:\Users\blake\djangoCalendar\locallibrary\calendarSync\views.py", line 40, in index
output = calendar.execute()['summary']
File "C:\Python38\lib\site-packages\googleapiclient\_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Python38\lib\site-packages\googleapiclient\http.py", line 907, in execute
raise HttpError(resp, content, uri=self.uri)
googleapiclient.errors.HttpError: <HttpError 404 when requesting https://www.googleapis.com/calendar/v3/users/me/calendarList/blakewright1021%40gmail.com?alt=json returned "Not Found">
I have shared my calendar with the service account. I am also able to access the calendar when I authenticate without a service account. What am I doing wrong?

CalendarList is the list on the bottom left of the google calendar web application.
Unless you have inserted the calendar into the service accounts calendarlist using clanedarlist.insert, its not going to show up in that list.
Once you have shared a calendar with the service account just do a calendar.get on the calendar id you shared and you will be able to access it. You can then do a calendarlist.insert if you really want it in the calendar list.

How to properly build a service object for the Google Sheets API v4

I am trying to achieve a simple goal: run python (2.7) script to read/update google sheet using Sheets API v4. I followed these steps:
Created google cloud platform account
Created service account and downloaded .json file with credentials
Installed google client library
Shared the spreadsheet with service account
Wrote a very primitive script that should read the data from a spreadsheet
I have also read a lot of official documentation on how to connect python script to google sheets. Main articles and guides:
https://developers.google.com/identity/protocols/oauth2/service-account#python_1
https://developers.google.com/sheets/api/guides/values#python
https://developers.google.com/sheets/api/quickstart/python
This is the script I am trying to run:
from googleapiclient.discovery import build
from google.oauth2 import service_account
SCOPES = ['https://www.googleapis.com/auth/spreadsheets']
SERVICE_ACCOUNT_FILE = 'service_account.json'
spreadsheet_id = '1KDCwwoIavfSINhDP9OKmieQux6LxhVC_ESsDQEma_7Y'
range_name = 'A1:A2'
creds = service_account.Credentials.from_service_account_file(SERVICE_ACCOUNT_FILE, scopes=SCOPES)
service = build('sheets', 'v4', credentials=creds)
result = service.spreadsheets().values().get(spreadsheetId=spreadsheet_id, range=range_name).execute()
rows = result.get('values', [])
print('{0} rows retrieved.'.format(len(rows)))
When I run this script api call is not being made. Script always fails when build function tries to create service object that is needed to make a request.
And this is the error log I am getting:
Traceback (most recent call last):
File "google_sheets_update.py", line 15, in <module>
service = build('sheets', 'v4', credentials=creds)
File "/env/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/env/lib/python2.7/site-packages/googleapiclient/discovery.py", line 286, in build
adc_key_path=adc_key_path,
File "/env/lib/python2.7/site-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
return wrapped(*args, **kwargs)
File "/env/lib/python2.7/site-packages/googleapiclient/discovery.py", line 455, in build_from_document
if isinstance(client_options, six.moves.collections_abc.Mapping):
AttributeError: '_MovedItems' object has no attribute 'collections_abc'
I know that attribute error is explicit but I wasn't able to find any resource that will explain why this error occurs here and how to fix it. If someone could point me to what am I doing wrong and how to solve it I will appreciate it.

Not able to run google sheet api from composer

I'm running a google sheet extract python api from compute engine and its works fine
and I'm running the same from the composer, but its not working
In composer, I'm login to the vm with the same user and running ssh command and I'm using the same service account in google sheet & vm
The following is the error message I'm getting
File "/home/utetwork_multiplier.py", line 15, in sheet_reade
result = service.spreadsheets().values().get(spreadsheetId=spreadsheet_id, range=range_name).execute(
File "/hsite-packages/googleapiclient/_helpers.py", line 130, in positional_wrappe
return wrapped(*args, **kwargs
File "/on3.5/site-packages/googleapiclient/http.py", line 849, in execut
raise HttpError(resp, content, uri=self.uri
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/14r0cQ1RCXhLyd7i0VCrmddXOFiugFnfioRb6cYI_BWQ/values/Master%21A%3AJ?alt=json returned "Request had insufficient authentication scopes."
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/contrib/operators/ssh_operator.py", line 164, in execut
.format(self.command, error_msg)
airflow.exceptions.AirflowException: error running cmd: set -e;cd src/digital_platform && ../../venvs/bdp/bin/python -m.marketing.adnetwork_multiplier, error: Traceback (most recent call last)
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_mai
"__main__", mod_spec
File "/usr/lib/python3.5/runpy.py", line 85, in _run_cod
exec(code, run_globals
File "/home/tt/src/digital_platform//marketing/adnetwork_multiplier.py", line 74, in <module
main(
File "/home/tt/src/digital_platform//marketing/adnetwork_multiplier.py", line 28, in mai
data = sheet_reader(range_name
File "/home/tt/src/digital_platform//marketing/adnetwork_multiplier.py", line 15, in sheet_reade
result = service.spreadsheets().values().get(spreadsheetId=spreadsheet_id, range=range_name).execute(
File "/home//venvs/bdp/lib/python3.5/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrappe
return wrapped(*args, **kwargs
File "/home//venvs/bdp/lib/python3.5/site-packages/googleapiclient/http.py", line 849, in execut
raise HttpError(resp, content, uri=self.uri
googleapiclient.errors.HttpError: <HttpError 403 when requesting https://sheets.googleapis.com/v4/spreadsheets/14r0cQ1RCXhLydcYI_BWQ/values/Master%21A%3AJ?alt=json returned "Request had insufficient authentication scopes."
During handling of the above exception, another exception occurred
Scopes
scope = [
'https://www.googleapis.com/auth/bigquery',
'https://www.googleapis.com/auth/spreadsheets.readonly',
'https://www.googleapis.com/auth/spreadsheets'
]
home = os.path.expanduser('~')
csf = os.path.join(home, '.client_secret.json')
token_filename = os.path.join(home, '.google_auth.dat')
flow = flow_from_clientsecrets(csf, scope=scope, message="%s is missing" % csf)
storage = Storage(token_filename)
credentials = storage.get()
if credentials is None or credentials.invalid:
flags = tools.argparser.parse_args(args=[])
flags.noauth_local_webserver = True
credentials = tools.run_flow(flow, storage, flags=flags)

From the error message you apear to be running Method: spreadsheets.values.get
That method requires one of the following scopes to access
https://www.googleapis.com/auth/drive
https://www.googleapis.com/auth/drive.readonly
https://www.googleapis.com/auth/drive.file
https://www.googleapis.com/auth/spreadsheets
https://www.googleapis.com/auth/spreadsheets.readonly
The error message states
403 when requesting https://sheets.googleapis.com/v4/spreadsheets/14r0cQ1RCXhLydcYI_BWQ/values/Master%21A%3AJ?alt=json returned "Request had insufficient authentication scopes.
Which means that the user you have authenticated with was not authenticated with one of the required scopes above. I suggest that you log out the user delete the credentials for that user and run your code again making sure that when it pops up the consent screen it is asking for one of the required scopes. I suspect you are running your script with outdated grants from the user.

Google API Build Service Object returns KeyError: 'rootUrl'

I have a flask application where I can run a script (with the help of Flask-script) that makes use of google api discovery using the code below:
app_script.py
import argparse
import csv
import httplib2
from apiclient import discovery
from oauth2client import client
from oauth2client.file import Storage
from oauth2client import tools
def get_auth_credentials():
flow = client.flow_from_clientsecrets(
'/path/to/client_screts.json', # file downloaded from Google Developers Console
scope='https://www.googleapis.com/auth/webmasters.readonly',
redirect_uri='urn:ietf:wg:oauth:2.0:oob')
storage = Storage('/path/to/storage_file.dat')
credentials = storage.get()
if credentials is None or credentials.invalid:
parser = argparse.ArgumentParser(parents=[tools.argparser])
flags = parser.parse_args(['--noauth_local_webserver'])
credentials = tools.run_flow(flow=flow, storage=storage, flags=flags)
return credentials
def main():
credentials = get_auth_credentials()
http_auth = credentials.authorize(httplib2.Http())
# build the service object
service = discovery.build('webmasters', 'v3', http_auth)
Now the problem is every time I shutdown my computer upon booting and running the script again, I get the following error when trying to build the service object:
terminal:
$ python app.py runscript
No handlers could be found for logger "oauth2client.util"
Traceback (most recent call last):
File "app.py", line 5, in <module>
testapp.manager.run()
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/flask_script/__init__.py", line 412, in run
result = self.handle(sys.argv[0], sys.argv[1:])
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/flask_script/__init__.py", line 383, in handle
res = handle(*args, **config)
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/flask_script/commands.py", line 216, in __call__
return self.run(*args, **kwargs)
File "/home/user/development/testproject/testapp/__init__.py", line 16, in runscript
metrics_collector.main()
File "/home/user/development/testproject/testapp/metrics_collector.py", line 177, in main
service = discovery.build('webmasters', 'v3', http_auth)
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/googleapiclient/discovery.py", line 206, in build
credentials=credentials)
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/oauth2client/util.py", line 140, in positional_wrapper
return wrapped(*args, **kwargs)
File "/home/user/.virtualenvs/testproject/local/lib/python2.7/site-packages/googleapiclient/discovery.py", line 306, in build_from_document
base = urljoin(service['rootUrl'], service['servicePath'])
KeyError: 'rootUrl'
intalled:
google-api-python-client==1.4.2
httplib2==0.9.2
Flask==0.10.1
Flask-Script==2.0.5
The script runs sometimes*, but thats the problem I don't know why it runs sometimes and others doesn't
*What I tried to make it work was to, delete all the cookies, download the client_secrets.json from the Google Developers Console again, remove the storage_file.dat, remove all .pyc files from the project
Can anyone help me see what's going on?

From a little bit of research here, it seems that the No handlers could be found for logger "oauth2client.util" error can actually be masking a different error. You need to use the logging module and configure your system to output.
Solution
Just add the following to configure logging:
import logging
logging.basicConfig()
Other helpful/related posts
Python - No handlers could be found for logger "OpenGL.error"
SOLVED: Error trying to access "google drive" with python (google quickstart.py source code)

Thank you so much for the tip Avantol13, you were right there was an error being masked.
The problem was that the following line:
service = discovery.build('webmasters', 'v3', http_auth)
should have actually be:
service = discovery.build('webmasters', 'v3', http=http_auth)
All working now. Thanks