We Keep Coding

Firestore client in python (as user) using firebase_admin or google.cloud.firestore

I am building a python client-side application that uses Firestore. I have successfully used Google Identity Platform to sign up and sign in to the Firebase project, and created a working Firestore client using google.cloud.firestore.Client which is authenticated as a user:
import json
import requests
import google.oauth2.credentials
from google.cloud import firestore
request_url = f"https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key={self.__api_key}"
headers = {"Content-Type": "application/json; charset=UTF-8"}
data = json.dumps({"email": self.__email, "password": self.__password, "returnSecureToken": True})
response = requests.post(request_url, headers=headers, data=data)
try:
response.raise_for_status()
except (HTTPError, Exception):
content = response.json()
error = f"error: {content['error']['message']}"
raise AuthError(error)
json_response = response.json()
self.__token = json_response["idToken"]
self.__refresh_token = json_response["refreshToken"]
credentials = google.oauth2.credentials.Credentials(self.__token,
self.__refresh_token,
client_id="",
client_secret="",
token_uri=f"https://securetoken.googleapis.com/v1/token?key={self.__api_key}"
)
self.__db = firestore.Client(self.__project_id, credentials)
I have the problem, however, that when the token has expired, I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Missing or invalid authentication."
debug_error_string = "{"created":"#1613043524.699081937","description":"Error received from peer ipv4:172.217.16.74:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Missing or invalid authentication.","grpc_status":16}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/my_app/src/controllers/im_alive.py", line 20, in run
self.__device_api.set_last_updated(utils.device_id())
File "/home/my_app/src/api/firestore/firestore_device_api.py", line 21, in set_last_updated
"lastUpdatedTime": self.__firestore.SERVER_TIMESTAMP
File "/home/my_app/src/api/firestore/firestore.py", line 100, in update
ref.update(data)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/document.py", line 382, in update
write_results = batch.commit()
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/batch.py", line 147, in commit
metadata=self._client._rpc_metadata,
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/gapic/firestore_client.py", line 1121, in commit
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/dist-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/dist-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Missing or invalid authentication.
I have tried omitting the token and only specifying the refresh token, and then calling credentials.refresh(), but the expires_in in the response from the https://securetoken.googleapis.com/v1/token endpoint is a string instead of a number (docs here), which makes _parse_expiry(response_data) in google.oauth2._client.py:257 raise an exception.
Is there any way to use the firestore.Client from either google.cloud or firebase_admin and have it automatically handle refreshing tokens, or do I need to switch to the manually calling the Firestore RPC API and refreshing tokens at the correct time?
Note: There are no users interacting with the python app, so the solution must not require user interaction.

Can't you just pass the string cast as integer _parse_expiry(int(float(response_data))) ?
If it is not working you could try to make a call and refresh token after getting and error 401, see my answer for the general idea on how to handle tokens.

As mentioned by #Marco, it is recommended that you use a service account if it's going to be used in an environment without user. When you use service account, you can just set GOOGLE_APPLICATION_CREDENTIALS environment variable to location of service account json file and just instantiate the firestore Client without any credentials (The credentials will be picked up automatically):
import firestore
client = firestore.Client()
and run it as (assuming Linux):
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
$ python file.py
Still, if you really want to use user credentials for the script, you can install the Google Cloud SDK, then:
$ gcloud auth application-default login
This will open browser and for you to select account and login. After logging in, it creates a "virtual" service account file corresponding to your user account (that will also be loaded automatically by clients). Here too, you don't need to pass any parameters to your client.
See also: Difference between “gcloud auth application-default login” and “gcloud auth login”

Google App Engine Python - YouTube API v3 - Insufficient Permission

I successfully duplicated the Tasks API from this video, but I am unable to successfully translate this format to using the YouTube API.
Here is my .py file:
import httplib2
import os
import sys
import jinja2
import webapp2
import logging
import pprint
from oauth2client import tools
from oauth2client.client import flow_from_clientsecrets
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.client import AccessTokenRefreshError
from oauth2client.tools import argparser, run_flow
from oauth2client.appengine import OAuth2Decorator
from apiclient.discovery import build
from google.appengine.ext.webapp import util
from google.appengine.ext.webapp import template
decorator = OAuth2Decorator(
client_id = '*my client ID*',
client_secret = '*my client secret*',
scope='https://www.googleapis.com/auth/youtube')
service = build("youtube", "v3")
class MainHandler(webapp2.RequestHandler):
#decorator.oauth_required
def get (self):
self.response.headers['Content-Type'] = 'text/plain'
channels_list = service.channels().list(
mine=True,
part="id"
).execute(http = decorator.http())
self.response.out.write (pprint.pformat(channels_list))
app = webapp2.WSGIApplication (
[
('/', MainHandler),
(decorator.callback_path, decorator.callback_handler()),
],
debug=True)
Here is my Traceback:
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:\Users\...\testapp\oauth2client\appengine.py", line 733, in check_oauth
resp = method(request_handler, *args, **kwargs)
File "C:\Users\...\testapp\testapp.py", line 35, in get
).execute(http = decorator.http())
File "C:\Users\...\testapp\oauth2client\util.py", line 129, in positional_wrapper
return wrapped(*args, **kwargs)
File "C:\Users\...\testapp\apiclient\http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 403 when requesting https://www.googleapis.com/youtube/v3/channels?part=id&mine=true&alt=json returned "Insufficient Permission">
INFO 2015-06-13 12:27:54,515 module.py:788] default: "GET / HTTP/1.1" 500 2278
I have checked and double-checked that I have both the YouTube Data API and YouTube Analytics API enabled this client ID. I have even disabled them and re-enabled to check, but I am still getting this error.
I am new to GAE and its methods, so maybe I am not understanding an error in the Traceback.
One note is that I was getting "ImportError: No module named..." for apiclient, httplib2, oauth2client, and uritemplate, so I moved those folders directly into my app file (and didn't get that error again). Not sure if moving those directly into the folder is causing errors.

Well I hope I didn't waste anyone's time, but if anyone else has this issue, I found this question, and although I didn't have a memcache issue, I revoked permission for my app from the user account I was using, refreshed the app and gave permission again and now it seems to be working.
I guess this was a Google-side issue. Thanks to anyone who spent the time to read my question.

Google App Engine + Google Drive cache error

I don't know why my project show the next cookie error. Could someone help me?
PATH
->test
->lib
->public
->templates
- app.yaml
- main.py
- client_secrets.json
- session-secret (python -c "import os;print os.urandom(64)" > session.secret)
When I use my App Engine Launcher (release: "1.7.5") and check out my localhost web page
I chose my Google account to add permissions in accounts.google.com/AccountChooser?service....... (redirect) and then accept conditions of the scopes
The log console shows the next error:
Traceback (most recent call last):
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files\Google\google_appengine\lib\webapp2-2.5.2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:\Program Files\Google\google_appengine\pruebasDocs\main.py", line 320, in get
creds = self.GetCodeCredentials()
File "C:\Program Files\Google\google_appengine\pruebasDocs\main.py", line 194, in GetCodeCredentials
session.set_secure_cookie(name='userid', value=userid)
File "lib\sessions.py", line 160, in set_secure_cookie
self.set_cookie(name, value, expires_days=expires_days, **kwargs)
File "lib\sessions.py", line 141, in set_cookie
self.response.headers._headers.append(('Set-Cookie', str(vals.OutputString(None))))
**AttributeError: ResponseHeaders instance has no attribute '_headers'**
lo.....t:8080/?code=4/00VfZ4DJ8d0P99v1kwn0yjBofcbq.gn6ceL8RBx0XYKs_1NgQtmXj_6WohwI
MAIN.PY
def GetCodeCredentials(self):
# Other frameworks use different API to get a query parameter.
code = self.request.get('code')
if not code:
# returns None to indicate that no code was passed from Google Drive.
return None
# Auth flow is a controller that is loaded with the client information,
# including client_id, client_secret, redirect_uri etc
oauth_flow = self.CreateOAuthFlow()
# Perform the exchange of the code. If there is a failure with exchanging
# the code, return None.
try:
creds = oauth_flow.step2_exchange(code)
except FlowExchangeError:
return None
# Create an API service that can use the userinfo API. Authorize it with our
# credentials that we gained from the code exchange.
users_service = CreateService('oauth2', 'v2', creds)
# Make a call against the userinfo service to retrieve the user's information.
# In this case we are interested in the user's "id" field.
userid = users_service.userinfo().get().execute().get('id')
# Store the user id in the user's cookie-based session.
session = sessions.LilCookies(self, SESSION_SECRET)
session.set_secure_cookie(name='userid', value=userid)
SESSIONS.PY
# output all their cookies to the headers at once before a response flush.
for vals in new_cookie.values():
self.response.headers._headers.append(('Set-Cookie', vals.OutputString(None)))

Error in listing the google drive files

I am trying to list all my google drive files in an application using google app engine. I am using the oauth protocol to access the drive files. On clicking the link to list the files, i get an error, which is listed below.
Please help. Thanks in advance.
Here is my program
#!/usr/bin/env python
#
# Copyright 2013 Google Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
"""Starting template for Google App Engine applications.
Use this project as a starting point if you are just beginning to build a Google
App Engine project. Remember to download the OAuth 2.0 client secrets which can
be obtained from the Developer Console <https://code.google.com/apis/console/>
and save them as 'client_secrets.json' in the project directory.
"""
import httplib2
import logging
import os
from google.appengine.api import users
from apiclient import discovery
from oauth2client import appengine
from oauth2client import client
from google.appengine.api import memcache
import webapp2
import jinja2
JINJA_ENVIRONMENT = jinja2.Environment(
loader=jinja2.FileSystemLoader(os.path.dirname(__file__)),
autoescape=True,
extensions=['jinja2.ext.autoescape'])
# CLIENT_SECRETS, name of a file containing the OAuth 2.0 information for this
# application, including client_id and client_secret, which are found
# on the API Access tab on the Google APIs
# Console <http://code.google.com/apis/console>
CLIENT_SECRETS = os.path.join(os.path.dirname(__file__), 'client_secrets.json')
# Helpful message to display in the browser if the CLIENT_SECRETS file
# is missing.
MISSING_CLIENT_SECRETS_MESSAGE = """
<h1>Warning: Please configure OAuth 2.0</h1>
<p>
To make this sample run you will need to populate the client_secrets.json file
found at:
</p>
<p>
<code>%s</code>.
</p>
<p>with information found on the APIs Console.
</p>
""" % CLIENT_SECRETS
http = httplib2.Http(memcache)
service = discovery.build('drive', 'v2', http=http)
decorator = appengine.oauth2decorator_from_clientsecrets(
CLIENT_SECRETS,
scope=[
'https://www.googleapis.com/auth/userinfo.email',
'https://www.googleapis.com/auth/userinfo.profile',
'https://www.googleapis.com/auth/drive',
'https://www.googleapis.com/auth/drive.appdata',
'https://www.googleapis.com/auth/drive.apps.readonly',
'https://www.googleapis.com/auth/drive.file',
'https://www.googleapis.com/auth/drive.metadata.readonly',
'https://www.googleapis.com/auth/drive.readonly',
'https://www.googleapis.com/auth/drive.scripts',
],
message=MISSING_CLIENT_SECRETS_MESSAGE)
class MainHandler(webapp2.RequestHandler):
#decorator.oauth_aware
def get(self):
user= users.get_current_user()
if user:
variables = {
'user': users.get_current_user(),
'url': decorator.authorize_url(),
'has_credentials': decorator.has_credentials()
}
template = JINJA_ENVIRONMENT.get_template('main.html')
self.response.write(template.render(variables))
else:
self.response.write(users.create_login_url("/"))
class listFile(webapp2.RequestHandler):
#decorator.oauth_aware
def get(self):
if not decorator.has_credentials():
self.reponse.write("<a href='")
self.reponse.write(decorator.authorize_url())
self.response.write(">Authorize</a>")
else:
a=service.files().list().execute()
self.response.write(a)
app = webapp2.WSGIApplication(
[
('/', MainHandler),
('/list', listFile),
(decorator.callback_path, decorator.callback_handler()),
],
debug=True)
The error i get is
Internal Server Error
The server has either erred or is incapable of performing the requested operation.
Traceback (most recent call last):
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
rv = self.handle_exception(request, response, e)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
rv = self.router.dispatch(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
return handler.dispatch()
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "/base/data/home/runtimes/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "/base/data/home/apps/s~test-ak2/1.371630947154576708/oauth2client/appengine.py", line 777, in setup_oauth
resp = method(request_handler, *args, **kwargs)
File "/base/data/home/apps/s~test-ak2/1.371630947154576708/main.py", line 104, in get
a=service.files().list().execute()
File "/base/data/home/apps/s~test-ak2/1.371630947154576708/oauth2client/util.py", line 132, in positional_wrapper
return wrapped(*args, **kwargs)
File "/base/data/home/apps/s~test-ak2/1.371630947154576708/apiclient/http.py", line 723, in execute
raise HttpError(resp, content, uri=self.uri)
HttpError: <HttpError 401 when requesting https://www.googleapis.com/drive/v2/files?alt=json returned "Login Required">

When you are requesting the file list:
service.files().list().execute()
the client library needs to know how to authenticate the HTTP request. The way it does this is is uses a special HTTP object that knows how to add the right credentials to the request.
This can be set either at initialization of the service, or it can be specified when you are calling execute().
In your case you are specifying it when you initialize the service object:
http = httplib2.Http(memcache)
service = discovery.build('drive', 'v2', http=http)
That's fine - but when you use the service object, it doesn't know about the decorator, so what you need to do is get an HTTP object from the decorator:
service.files().list().execute(http=decorator.http())
and pass that to the execute() method. This will ensure the credentials associated with the logged in user are added to the request to the Drive API.

python twitter 404

I'm trying to use twitter-python to connect to twitter's API, and I keep getting this pesky 404 error!
Code is below:
import sys
import twitter
def main():
api = twitter.Api()
statuses = api.GetPublicTimeline()
print [s.user.name for s in statuses]
It should come back with the names of people who've posted status updates on the public timeline, but I get this traceback instead.
Traceback (most recent call last):
File "new.py", line 14, in <module>
main()
File "new.py", line 10, in main
statuses = api.GetPublicTimeline()
File "/usr/lib/pymodules/python2.7/twitter.py", line 1319, in GetPublicTimeline
json = self._FetchUrl(url, parameters=parameters)
File "/usr/lib/pymodules/python2.7/twitter.py", line 2030, in _FetchUrl
url_data = opener.open(url, encoded_post_data).read()
File "/usr/lib/python2.7/urllib2.py", line 400, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 513, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 438, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 372, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 521, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
If I try to authenticate, all of my requests come back as 401.
Thanks for your help!

on https://code.google.com/p/python-twitter/ they say:
The python-twitter library now only supports oAuth authentication as the Twitter devs have indicated that oAuth is the only method that will be supported moving forward.
api = twitter.Api(consumer_key='consumer_key',
consumer_secret='consumer_secret', access_token_key='access_token', access_token_secret='access_token_secret')
Is that relevant to your problem ?

Twitter is one of the better APIs when it comes to making the oAuth process painless for developers. If you log in to Twitter and go to the developer dashboard at https://dev.twitter.com/apps, you can register a new application (or configure your existing apps).
Viewing the application details will give you an "oAuth Tool" tab, where you'll find all the relevant oAuth values for that app: Consumer Key, Consumer Secret, Access Token, and Access Token Secret.
If you're creating an application that will always authenticate with Twitter as the same user, you can simply hard-code these values into your code or store them in a config file somewhere. If your application needs to authenticate with Twitter on behalf of multiple users (eg, so that it can post statuses on behalf of a user), then you'll need to go through the oAuth dance which is described at https://dev.twitter.com/docs/auth/3-legged-authorization.