We Keep Coding

Decode Firebase JWT in Python using PyJWT

I have written the following code :
def check_token(token):
response = requests.get("https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com")
key_list = response.json()
decoded_token = jwt.decode(token, key=key_list, algorithms=["RS256"])
print(f"Decoded token : {decoded_token}")
I am trying to decode the token provided by firebase client side to verify it server-side.
The above code is throwing the following exception :
TypeError: Expecting a PEM-formatted key.
I have tried to not pass a list to the jwt.decode method, only the key content and i have a bigger error that the library could not deserialize the Key.
I was following this answer but i am getting this error.
Is it a requests conversion problem ? What am i doing wrong ?

The 2nd parameter key in decode() seems to take a string value instead of list. The Google API request returns a dict/map containing multiple keys. The flow goes like:
Fetch public keys from the Google API endpoint
Then read headers without validation to get the kid claim then use it to get appropriate key from that dict
That is a X.509 Certificate and not the public key as in this answer so you need to get public key from that.
The following function worked for me:
import jwt
import requests
from cryptography.hazmat.backends import default_backend
from cryptography import x509
def check_token(token):
n_decoded = jwt.get_unverified_header(token)
kid_claim = n_decoded["kid"]
response = requests.get("https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com")
x509_key = response.json()[kid_claim]
key = x509.load_pem_x509_certificate(x509_key.encode('utf-8'), backend=default_backend())
public_key = key.public_key()
decoded_token = jwt.decode(token, public_key, ["RS256"], options=None, audience="<FIREBASE_PROJECT_ID>")
print(f"Decoded token : {decoded_token}")
check_token("FIREBASE_ID_TOKEN")

Coinlist exchange API authentication - Python

For the past month I've been trying to understand coinlist API (https://coinlist.co) since there is no API wrapper available it's been a hard task. I could figure it out that their API docs are really similar to coinbase exchange, and tried to extrapolate but with no success.
import json, hmac, hashlib, time, requests
from requests.auth import AuthBase
# Before implementation, set environmental variables with the names API_KEY and API_SECRET
API_KEY = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
API_SECRET = 'xxxx/xxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxx=='
# Create custom authentication for Coinlist API
class CoinlistWalletAuth(AuthBase):
def __init__(self, api_key, secret_key):
self.api_key = api_key
self.secret_key = secret_key
def __call__(self, request):
timestamp = str(int(time.time()))
message = timestamp + request.method + request.path_url + (request.body or '')
signature = hmac.new(self.secret_key, message, hashlib.sha256).hexdigest()
request.headers.update({
'CL-ACCESS-SIGN': signature,
'CL-ACCESS-TIMESTAMP': timestamp,
'CL-ACCESS-KEY': self.api_key,
})
return request
auth = CoinlistWalletAuth(API_KEY, API_SECRET)
#Test1 - Fetching account balance
response = requests.get('https://trade-api.coinlist.co/v1/accounts', auth=auth)
I am getting this TypeError: key: expected bytes or bytearray, but got 'str' when calling for the response.
Docs say - You must base64-encode the signature (the output of the sha256 HMAC). Why is it failing?

Two things:
TypeError: key: expected bytes or bytearray, but got 'str'
hmac.new(key, msg=None, digestmod='')
Return a new hmac object. key is a bytes or bytearray object giving the secret key. (...)
import base64
...
secret_key_as_bytes = base64.b64decode(self.secret_key, altchars=None)
The CL-ACCESS-SIG header is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string timestamp + HTTP method + path + body (where '+' represents string concatenation) and base64-encode the output.
digest = hmac.new(secret_key_as_bytes, message, hashlib.sha256).digest()
signature = str(base64.b64encode(digest))

wish it's not too late
hmac.new()
function need the key argument to be bytes,so:
byte_key = bytes(self.secret_key, 'UTF-8') # key.encode() would also work in this case
but it's no done yet, if you go with this, you would get
TypeError: Unicode-objects must be encoded before hashing
to fix this , your message should be encoded.
message = message.encode()
h = hmac.new(byte_key, message, hashlib.sha256).hexdigest()
now it's done.
Howerver I got 400. sad.
I found someone have done this:
https://gist.github.com/badmofo/d109fbc447fea64d99e5ca58bdf53d7b

Generate YubiOTP verification HMAC-SHA-1 signatures in Python

I'm a little bit confused by what I need to do here for Python, but from the Yubikey API documentation for verifying Yubikeys that have YubiOTP the HMAC signature needs to be generated a specific way - from their documentation:
Generating signatures
The protocol uses HMAC-SHA-1 signatures. The HMAC key to use is the
client API key.
Generate the signature over the parameters in the message. Each
message contains a set of key/value pairs, and the signature is always
over the entire set (excluding the signature itself), and sorted in
alphabetical order of the keys. More precisely, to generate a message
signature do:
Alphabetically sort the set of key/value pairs by key order.
Construct a single line with each ordered key/value pair
concatenated using &, and each key and value contatenated with =. Do
not add any linebreaks. Do not add whitespace. For example:
a=2&b=1&c=3.
Apply the HMAC-SHA-1 algorithm on the line as an octet string using
the API key as key (remember to base64decode the API key obtained from
Yubico).
Base 64 encode the resulting value according to RFC 4648, for
example, t2ZMtKeValdA+H0jVpj3LIichn4=.
Append the value under key h to the message.
Now my understanding of their API from their documentation states the following valid request parameters:
id - the Client ID from Yubico API
otp - the YubiOTP value from the YubiOTP component of a yubikey.
h - the HMAC-SHA1 signature for the request
timestamp - empty does nothing, 1 includes the timestamp in the reply from the server
nonce - A 16 to 40 character long string with random unique data.
sl - a value of 0 to 100 indicating percentage of syncing required by client, or strings "fast" or "Secure" to use server values; if nonexistent server decides
timeout - # of seconds to wait for sync responses; let server decide if absent.
I have a total of two functions I'm trying to use to try and handle all these things and generate the URL. Namely, we the HMAC support function and the verify_url_generate which generates the URL (and API_KEY is statically coded - my API Secret Key from Yubico):
def generate_signature(message, key=base64.b64decode(API_KEY)):
message = bytes(message, 'UTF-8')
digester = hmac.new(key, message, hashlib.sha1)
digest = digester.digest()
signature = base64.urlsafe_b64encode(digest)
return str(signature, 'UTF-8')
def verify_url_generate(otp):
nonce = "".join(secrets.choice(ascii_lowercase) for _ in range(40))
data = OrderedDict(
{
"id": None,
"nonce": None,
"otp": None,
"sl": 50,
"timeout": 10,
"timestamp": 1
}
)
data['otp'] = otp
data['id'] = CLIENT_ID
data['nonce'] = nonce
args = ""
for key, value in data.items():
args += f"{key}={value}&"
sig = generate_signature(args[:-1])
url = YUBICO_API_URL + args + "&h=" + sig
print(url)
return
Any URL generated by this triggers a notice about "BAD_SIGNATURE" from the remote site - any URL generated minus the HMAC sig (h=) parameter works. So we know the issue isn't the URL, it's the HMAC signature.
Does anyone know what I'm doing wrong with my HMAC generation approach, by passing the HMAC sig generator the concatenated args from the ordered dict in key=value separated by & for each parameter format?

Can you try using standard_b64encode and then using urllib.parse.quote(url) in your final URL?
I ask because this page says that "As such, all parameters must be properly URL encoded. In particular, some base64 characters (such as "+") in the value fields needs to be escaped." which means it is expecting +(or %2B) in the args and does a unquote and then normal decode.

Using a Zapier Custom Request Webhook with JSON Web Tokens

I need to access an API that uses JSON Web Tokens as their authentication method. Is there a good way to use a python code step to create this token then add that token as a header to a custom request webhook step?

My experience authenticating with APIs has been using the simple API key method. As such I first read your question and didn't fully understand. I decided to do some research and hopefully learn something along the way, and I certainly did. I share my findings and answer below:
For starters I began reading into JSON Web Tokens(JWT) which lead me to the JWT website, which was an excellent resource. It very clearly spells out the components that make up a JWT and how they need to be formatted, I would highly recommend having a look.
From the JWT website I found that a JWT is made up of three components:
A base64 URL safe encoded header.
A base64 URL safe encoded payload.
A base64 URL safe encoded signature.
All three of the above combined form the correctly formatted JWT. Fortunately the JWT website has a list of libraries made for Python. Unfortunately none of these third-party libraries are available in the vanilla Python offered by the Zapier code module. To get this done required reading some source code and leveraging what libraries we do have available. So after a few hours and lots of trial and error I was able to come up with the following solution for generating a correctly formatted JWT:
import hashlib
import hmac
import requests
from base64 import urlsafe_b64encode
def base64url_encode(payload):
if not isinstance(payload, bytes):
payload = payload.encode('utf-8')
encode = urlsafe_b64encode(payload)
return encode.decode('utf-8').rstrip('=')
def generate_JWT(header, payload, secret):
encoded_header = base64url_encode(header)
encoded_payload = base64url_encode(payload)
signature = hmac.new(secret,
encoded_header + "." + encoded_payload,
hashlib.sha256)
encoded_signature = base64url_encode(signature.digest())
return encoded_header + "." + encoded_payload + "." + encoded_signature
def get_request(url, jwt):
headers = {
"Authorization" : "Bearer " + jwt
}
result = requests.get(url, headers=headers)
return result
secret = "yoursecrettoken"
header = '{"alg":"HS256","typ":"JWT"}'
payload = '{"sub":"1234567890","name":"John Doe","iat":1516239022}'
jwt = generate_JWT(header, payload, secret)
response = get_request("https://SomeApiEndpoint.com/api/", jwt)
You can test the output of this against the JWT's debugger here.
Note: For the encoding to work properly for the header and payload objects you have to convert them to a string object. I tried doing this by calling the JSON.dumps() function and passing the dictionary objects, but when I encoded the return values they did not match what was shown on the JWT debugger. The only solution I could find was by wrapping the dictionary objects in quotations and ensuring there were no spaces within it.
And so with the JWT in hand you can use it in your Zapier Webhooks custom get request step, or you could save the zap and send the request in the same code module using Python's request library as I have in my code example.
Thanks for the learning opportunity, and I hope this helps.

I had a similar issue trying to generate and use a JWT in a custom integration. Unfortunately, this code above did not work for my situation. I'm currently using the below javascript and it seems to be functioning perfectly.
const toBase64 = obj => {
const str = JSON.stringify (obj);
return Buffer.from(str).toString ('base64');
};
const replaceSpecialChars = b64string => {
// this will match the special characters and replace them with url-safe substitutes
return b64string.replace (/[=+/]/g, charToBeReplaced => {
switch (charToBeReplaced) {
case '=':
return '';
case '+':
return '-';
case '/':
return '_';
}
});
};
const crypto = require('crypto');
const signatureFunction = crypto.createSign('RSA-SHA256');
const headerObj = {
alg: 'RS256',
typ: 'JWT',
};
const payloadObj = {
iat: Math.round(Date.now() / 1000), // lists the current Epoch time
exp: Math.round(Date.now() / 1000) + 3600, // adds one hour
sub: '1234567890'
name: 'John Doe'
};
const base64Header = toBase64(headerObj);
const base64Payload = toBase64(payloadObj);
const base64UrlHeader = replaceSpecialChars(base64Header);
const base64UrlPayload = replaceSpecialChars(base64Payload);
signatureFunction.write(base64UrlHeader + '.' + base64UrlPayload);
signatureFunction.end();
// The private key without line breaks
const privateKey = `-----BEGIN PRIVATE KEY-----
MIIJQwIBADANBgkqhkiG9w0BAQEFAASCCS0wggkpAgEAAoICAQC5Q+0Je6sZ6BuX
cTsN7pEzAaj4819UE7gM+Tf7U5AKHSKk3hN5UILtp5EuEO7h7H+lyknn/5txltA4
-----END PRIVATE KEY-----`;
const signatureBase64 = signatureFunction.sign(privateKey, 'base64');
const signatureBase64Url = replaceSpecialChars(signatureBase64);
console.log("Your JWT is: " + base64UrlHeader + "." + base64UrlPayload + "." + signatureBase64Url);
I have this code in a Zapier code-step prior to calling the custom integration and pass in the generated token object to authenticate the call.
Hopefully, this helps someone!

Should I use Base64 of HMAC digest or just HMAC hex digest?

Legend
I expose an API which requires client to sign requests by sending two headers:
Authorization: MyCompany access_key:<signature>
Unix-TimeStamp: <unix utc timestamp in seconds>
To create a signature part, the client should use a secret key issued by my API service.
In Python (Py3k) it could look like:
import base64
import hmac
from hashlib import sha256
from datetime import datetime
UTF8 = 'utf-8'
AUTH_HEADER_PREFIX = 'MyCompany'
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature_base64 = base64.b64encode(new_hmac.digest())
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=str(signature_base64, UTF8).strip()
)
if __name__ == '__main__':
message = str(datetime.utcnow().timestamp())
signature = create_signature('my access key', 'my secret key', message)
print(
'Request headers are',
'Authorization: {}'.format(signature),
'Unix-Timestamp: {}'.format(message),
sep='\n'
)
# For message='1457369891.672671',
# access_key='my access key'
# and secret_key='my secret key' will ouput:
#
# Request headers are
# Authorization: MyCompany my access key:CUfIjOFtB43eSire0f5GJ2Q6N4dX3Mw0KMGVaf6plUI=
# Unix-Timestamp: 1457369891.672671
I wondered if I could avoid dealing with encoding digest of bytes to Base64 and just use HMAC.hexdigest() to retrieve a string.
So that my function will change to:
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature = new_hmac.hexdigest()
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=signature
)
But then I found that Amazon uses similar approach as in my first code snippet:
Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;
Signature = Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) );
Seeing that Amazon doesn't use hex digest I stopped myself to move forward with it because maybe they know something I don't.
Update
I've measured a performance and found hex digest to be faster:
import base64
import hmac
import string
from hashlib import sha256
UTF8 = 'utf-8'
MESSAGE = '1457369891.672671'
SECRET_KEY = 'my secret key'
NEW_HMAC = create_hmac()
def create_hmac():
new_hmac = hmac.new(bytes(SECRET_KEY, UTF8), digestmod=sha256)
new_hmac.update(bytes(MESSAGE, UTF8))
return new_hmac
def base64_digest():
return base64.b64encode(NEW_HMAC.digest())
def hex_digest():
return NEW_HMAC.hexdigest()
if __name__ == '__main__':
from timeit import timeit
print(timeit('base64_digest()', number=1000000,
setup='from __main__ import base64_digest'))
print(timeit('hex_digest()', number=1000000,
setup='from __main__ import hex_digest'))
Results with:
3.136568891000934
2.3460130329913227
Question #1
Does someone know why do they stick to Base64 of bytes digest and don't use just hex digest? Is there some solid reason to keep using this approach over hex digest?
Question #2
According to RFC2716 the format of Authorization header value when using Basic Authentication
is:
Authorization: Base64(username:password)
So basically you wrap with Base64 two values (user's id and password) seprated by colon.
As you can see in my code snippet and in Amazon's documentation nor me, nor Amazon do that for own custom value of the Authorization header.
Would it be a better style to wrap the whole pair as Base64(access_key:signature) to stick closer to this RFC or it doesn't matter at all?

Amazon does use the hex digest in Signature Version 4.
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
Your example is from Signature Version 2, the older algorithm, which does use Base-64 encoding for the signature (and which also is not supported in the newest AWS regions).
So, your concern that AWS knows something you don't is misplaced, since their newer algorithm uses it.
In the Authorization: header, it really doesn't make a difference other than a few extra octets.
Where Base-64 gets messy is when the signature is passed in the query string, because + and (depending on who you ask) / and = require special handling -- they need to be url-escaped ("percent-encoded") as %2B, %2F, and %3D respectively... or you have to make accommodations for the possible variations on the server... or you have to require the use of a non-standard Base-64 alphabet, where + / = becomes - ~ _ the way CloudFront does it. (This particular non-standard alphabet is only one of multiple non-standard options, all "solving" the same problem of magic characters in URLs with Base-64).
Go with hex-encoding.
You will almost inevitably find would-be consumers of your API that find Base-64 to be "difficult."