We Keep Coding

Firestore client in python (as user) using firebase_admin or google.cloud.firestore

I am building a python client-side application that uses Firestore. I have successfully used Google Identity Platform to sign up and sign in to the Firebase project, and created a working Firestore client using google.cloud.firestore.Client which is authenticated as a user:
import json
import requests
import google.oauth2.credentials
from google.cloud import firestore
request_url = f"https://identitytoolkit.googleapis.com/v1/accounts:signInWithPassword?key={self.__api_key}"
headers = {"Content-Type": "application/json; charset=UTF-8"}
data = json.dumps({"email": self.__email, "password": self.__password, "returnSecureToken": True})
response = requests.post(request_url, headers=headers, data=data)
try:
response.raise_for_status()
except (HTTPError, Exception):
content = response.json()
error = f"error: {content['error']['message']}"
raise AuthError(error)
json_response = response.json()
self.__token = json_response["idToken"]
self.__refresh_token = json_response["refreshToken"]
credentials = google.oauth2.credentials.Credentials(self.__token,
self.__refresh_token,
client_id="",
client_secret="",
token_uri=f"https://securetoken.googleapis.com/v1/token?key={self.__api_key}"
)
self.__db = firestore.Client(self.__project_id, credentials)
I have the problem, however, that when the token has expired, I get the following error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
return callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.7/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAUTHENTICATED
details = "Missing or invalid authentication."
debug_error_string = "{"created":"#1613043524.699081937","description":"Error received from peer ipv4:172.217.16.74:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Missing or invalid authentication.","grpc_status":16}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/home/my_app/src/controllers/im_alive.py", line 20, in run
self.__device_api.set_last_updated(utils.device_id())
File "/home/my_app/src/api/firestore/firestore_device_api.py", line 21, in set_last_updated
"lastUpdatedTime": self.__firestore.SERVER_TIMESTAMP
File "/home/my_app/src/api/firestore/firestore.py", line 100, in update
ref.update(data)
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/document.py", line 382, in update
write_results = batch.commit()
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/batch.py", line 147, in commit
metadata=self._client._rpc_metadata,
File "/usr/local/lib/python3.7/dist-packages/google/cloud/firestore_v1/gapic/firestore_client.py", line 1121, in commit
request, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/dist-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/dist-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/dist-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.Unauthenticated: 401 Missing or invalid authentication.
I have tried omitting the token and only specifying the refresh token, and then calling credentials.refresh(), but the expires_in in the response from the https://securetoken.googleapis.com/v1/token endpoint is a string instead of a number (docs here), which makes _parse_expiry(response_data) in google.oauth2._client.py:257 raise an exception.
Is there any way to use the firestore.Client from either google.cloud or firebase_admin and have it automatically handle refreshing tokens, or do I need to switch to the manually calling the Firestore RPC API and refreshing tokens at the correct time?
Note: There are no users interacting with the python app, so the solution must not require user interaction.

Can't you just pass the string cast as integer _parse_expiry(int(float(response_data))) ?
If it is not working you could try to make a call and refresh token after getting and error 401, see my answer for the general idea on how to handle tokens.

As mentioned by #Marco, it is recommended that you use a service account if it's going to be used in an environment without user. When you use service account, you can just set GOOGLE_APPLICATION_CREDENTIALS environment variable to location of service account json file and just instantiate the firestore Client without any credentials (The credentials will be picked up automatically):
import firestore
client = firestore.Client()
and run it as (assuming Linux):
$ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json
$ python file.py
Still, if you really want to use user credentials for the script, you can install the Google Cloud SDK, then:
$ gcloud auth application-default login
This will open browser and for you to select account and login. After logging in, it creates a "virtual" service account file corresponding to your user account (that will also be loaded automatically by clients). Here too, you don't need to pass any parameters to your client.
See also: Difference between “gcloud auth application-default login” and “gcloud auth login”

Python Script Returning an error when trying to Obtain TFS work Items

I have written the following Pythin script to try and gather TFS (Team Foundation Server) work item data. Below is my script:-
from tfs import TFSAPI
user="andrew.xxxx"
password="xxxxxxxx"
tfsAddress = "http://man-tfsmig-1:8080/"
print(tfsAddress)
client = TFSAPI(tfsAddress, project="DefaultCollection/xxxxxxxx", user=user, password=password)
work_item = client.get_workitem(28274)
When I run the code I get the following error, but I do not know what the problem is:-
Traceback (most recent call last):
File ".\TFS_Release_Notes_app.py", line 13, in <module>
work_item = client.get_workitem(28274)
File "C:\Python\lib\site-packages\tfs\connection.py", line 70, in get_workitem
return self.get_workitems(id_, fields)[0]
File "C:\Python\lib\site-packages\tfs\connection.py", line 80, in get_workitems
work_items_batch_info = self.__get_workitems(work_items_batch, fields=fields, expand=expand)
File "C:\Python\lib\site-packages\tfs\connection.py", line 65, in __get_workitems
object_class=Workitem)
File "C:\Python\lib\site-packages\tfs\connection.py", line 46, in get_tfs_object
raw = self.rest_client.send_get(uri=uri, payload=payload, project=project)
File "C:\Python\lib\site-packages\tfs\connection.py", line 323, in send_get
return self.__send_request('GET', uri, None, payload=payload, project=project, json=json)
File "C:\Python\lib\site-packages\tfs\connection.py", line 360, in __send_request
response.raise_for_status()
File "C:\Python\lib\site-packages\requests\models.py", line 939, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://xxxx-xxxxx-1:8080/DefaultCollection/_apis/wit/workitems?ids=28274&$expand=all&api-version=1.0

It's telling you the error: 404, not found.
That means the URI you're providing for the API is incorrect.
It's generating this URI: http://xxxx-xxxxx-1:8080/DefaultCollection/_apis/wit/workitems. Validate if that is correct.
Usually, when TFS is running on port 8080 over HTTP, there's a /tfs/ virtual directory.

I needed to add the following line to authenticate:-
# Use NTLM authorization
from requests_ntlm import HttpNtlmAuth
client = TFSAPI("https://tfs.tfs.ru/tfs/", user=user, password=password, auth_type=HttpNtlmAuth)

Python requests gives SSL unknown protocol

I am trying to send a request to an API I have set up on an AWS machine.
The code I use is as follows:
import requests
import json
report_dict = {
"client_name": "Wayne Enterprises",
"client_id": 123,
"report_type": "api_testing",
"timestamp_generated": "2015-07-29T11:00:00Z",
"report_data": {"revenue": 9000.00}
}
report_json = json.dumps(report_dict)
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=False)
Doing this, I get:
Traceback (most recent call last):
File "art2_java_test.py", line 124, in <module>
main()
File "art2_java_test.py", line 9, in main
test_post_good_data()
File "art2_java_test.py", line 29, in test_post_good_data
resp = requests.post("https://my-url.com:8080/my-api/reports", data=report_json,verify=Fal
se)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 109, in post
return request('post', url, data=data, json=json, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\api.py",
line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\sessions
.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests-2.7.0-py2.7.egg\requests\adapters
.py", line 428, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:5
90)
But when I send the request as http instead of https, it (usually) works fine. I've found some evidence that this can have to do with proxy servers, but I am not using one. Are there any other potential reasons for this error? This is a website only available on my company's local network, if that's relevant.

.... https://my-url.com:8080/my-api/reports
...But when I send the request as http instead of https, it (usually) works fine.
My guess is that you are trying the same port 8080 for http and https. But, servers usually listen on a single port either for http or https and not both. This means that if your client is trying to start the TLS handshake needed for https against this port it will get a plain error message back. The client then tries to interpret this error message as TLS and returns some weird error messages, because the response is not TLS at all.

check Python requests with charles proxy for HTTPS

I want to debug some python requests using charles proxy.
I need to include the certificate for charles on the call, but is not working
import requests
endpoint_url = 'https://www.httpsnow.org/'
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
print "empexo"
print r
I have added the https address on Charles,
I get on Charles:
SSLHandshake: Remote host closed connection during handshake
and on python the log with error
empexo
Traceback (most recent call last):
File "/Users/iosdev/DopPy/GetCelebs.py", line 15, in <module>
r = requests.get(endpoint_url, verify=True, cert='/Users/iosdev/DopPy/charles.crt')
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 65, in get
return request('get', url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/api.py", line 49, in request
response = session.request(method=method, url=url, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 461, in request
resp = self.send(prep, **send_kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/Users/iosdev/VenvPY26/lib/python2.6/site-packages/requests/adapters.py", line 431, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 336265225] _ssl.c:341: error:140B0009:SSL routines:SSL_CTX_use_PrivateKey_file:PEM lib
Process finished with exit code 1

I found this thread while I was troubleshooting a similar issue. In the scenario I ran into the cert argument was being used to define the path to a ".crt" file when the verify argument should have been used instead.
The correct usage ended up looking like:
requests.get(endpoint_url, verify='/path/to/file.crt')
See Requests' documentation for more details: https://2.python-requests.org/en/v1.1.0/user/advanced/#ssl-cert-verification
As an aside, I find employing Request's ability to specify the path to a ".crt" via the REQUESTS_CA_BUNDLE environmental variable more effective when using Charles Proxy for local debugging.
Running something like the following in shell saves having to specify the path to Charles' ".crt" for every Requests call:
REQUESTS_CA_BUNDLE=/path/to/file.crt
export REQUESTS_CA_BUNDLE

python-requests authenticated proxy httplib.BadStatusLine

Authenticated proxy through python-requests returns the following error:
>>> import requests
>>> proxies = {'https': 'http://username:password#proxy.company.com:8080',}
>>> requests.get('https://api.github.com/',proxies=proxies,verify=False)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/site-packages/requests/api.py", line 55, in get
return request('get', url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 335, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 327, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='proxy.company.com', port=8080): Max retries exceeded with url: https://api.github.com/ (Caused by <class 'httplib.BadStatusLine'>: )
Authenticated proxy works fine in curl:
$ curl --proxy-user username:password --proxy http://proxy.company.com:8080 -k https://api.github.com/
{
"current_user_url": "https://api.github.com/user",
"authorizations_url": "https://api.github.com/authorizations",
"emails_url": "https://api.github.com/user/emails",
"emojis_url": "https://api.github.com/emojis",
"events_url": "https://api.github.com/events",
"feeds_url": "https://api.github.com/feeds",
"following_url": "https://api.github.com/user/following{/target}",
"gists_url": "https://api.github.com/gists{/gist_id}",
"hub_url": "https://api.github.com/hub",
"issue_search_url": "https://api.github.com/legacy/issues/search/{owner}/{repo}/{state}/{keyword}",
"issues_url": "https://api.github.com/issues",
"keys_url": "https://api.github.com/user/keys",
"notifications_url": "https://api.github.com/notifications",
"organization_repositories_url": "https://api.github.com/orgs/{org}/repos/{?type,page,per_page,sort}",
"organization_url": "https://api.github.com/orgs/{org}",
"public_gists_url": "https://api.github.com/gists/public",
"rate_limit_url": "https://api.github.com/rate_limit",
"repository_url": "https://api.github.com/repos/{owner}/{repo}",
"repository_search_url": "https://api.github.com/legacy/repos/search/{keyword}{?language,start_page}",
"current_user_repositories_url": "https://api.github.com/user/repos{?type,page,per_page,sort}",
"starred_url": "https://api.github.com/user/starred{/owner}{/repo}",
"starred_gists_url": "https://api.github.com/gists/starred",
"team_url": "https://api.github.com/teams",
"user_url": "https://api.github.com/users/{user}",
"user_organizations_url": "https://api.github.com/user/orgs",
"user_repositories_url": "https://api.github.com/users/{user}/repos{?type,page,per_page,sort}",
"user_search_url": "https://api.github.com/legacy/user/search/{keyword}"
}
How do I troubleshoot?

Requests has support for HTTP proxies just since 2.0: https://github.com/kennethreitz/requests/pull/1515
BTW, it does respect proxy environment variables, so you can just set:
export http_proxy="http://username:password#proxy.company.com:8080"
export https_proxy=$http_proxy
More details about the state of proxy support in requests 1.x can be found in this blog post: https://lukasa.co.uk/2013/07/Python_Requests_And_Proxies/

Your proxy is indeed HTTP not HTTPS. To use proxy with HTTPS it needs to use connect protocol, not HTTP, with presumably CONNECT basic auth. Now I do not have a HTTPS proxy with auth available, and do not know how to install such system, but what happens if you have just "https://username:password#proxy.company.com:8080" there instead?
On the other hand, you might want to make sure that requests/urllib3 are up to date. See this bug report.