For the past month I've been trying to understand coinlist API (https://coinlist.co) since there is no API wrapper available it's been a hard task. I could figure it out that their API docs are really similar to coinbase exchange, and tried to extrapolate but with no success.
import json, hmac, hashlib, time, requests
from requests.auth import AuthBase
# Before implementation, set environmental variables with the names API_KEY and API_SECRET
API_KEY = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
API_SECRET = 'xxxx/xxxxxxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxx=='
# Create custom authentication for Coinlist API
class CoinlistWalletAuth(AuthBase):
def __init__(self, api_key, secret_key):
self.api_key = api_key
self.secret_key = secret_key
def __call__(self, request):
timestamp = str(int(time.time()))
message = timestamp + request.method + request.path_url + (request.body or '')
signature = hmac.new(self.secret_key, message, hashlib.sha256).hexdigest()
request.headers.update({
'CL-ACCESS-SIGN': signature,
'CL-ACCESS-TIMESTAMP': timestamp,
'CL-ACCESS-KEY': self.api_key,
})
return request
auth = CoinlistWalletAuth(API_KEY, API_SECRET)
#Test1 - Fetching account balance
response = requests.get('https://trade-api.coinlist.co/v1/accounts', auth=auth)
I am getting this TypeError: key: expected bytes or bytearray, but got 'str' when calling for the response.
Docs say - You must base64-encode the signature (the output of the sha256 HMAC). Why is it failing?
Two things:
TypeError: key: expected bytes or bytearray, but got 'str'
hmac.new(key, msg=None, digestmod='')
Return a new hmac object. key is a bytes or bytearray object giving the secret key. (...)
import base64
...
secret_key_as_bytes = base64.b64decode(self.secret_key, altchars=None)
The CL-ACCESS-SIG header is generated by creating a sha256 HMAC using the base64-decoded secret key on the prehash string timestamp + HTTP method + path + body (where '+' represents string concatenation) and base64-encode the output.
digest = hmac.new(secret_key_as_bytes, message, hashlib.sha256).digest()
signature = str(base64.b64encode(digest))
wish it's not too late
hmac.new()
function need the key argument to be bytes,so:
byte_key = bytes(self.secret_key, 'UTF-8') # key.encode() would also work in this case
but it's no done yet, if you go with this, you would get
TypeError: Unicode-objects must be encoded before hashing
to fix this , your message should be encoded.
message = message.encode()
h = hmac.new(byte_key, message, hashlib.sha256).hexdigest()
now it's done.
Howerver I got 400. sad.
I found someone have done this:
https://gist.github.com/badmofo/d109fbc447fea64d99e5ca58bdf53d7b
Related
I am using Auth0 in my DRF project to generate tokens and authenticate. Everything works fine if I normally encode and decode the tokens. But I have written a method requires_scope which determines if the required scope (which the decorator on API tells) for any API. The method is below:
def requires_scope(required_scope):
"""Determines if the required scope is present in the Access Token
Args:
required_scope (str): The scope required to access the resource
"""
def require_scope(f):
#wraps(f)
def decorated(*args, **kwargs):
token = get_token_auth_header(args[0])
decoded = jwt.decode(token, verify=False, algorithms=settings.AUTH0_ALGORITHMS)
if decoded.get("scope"):
token_scopes = decoded["scope"].split()
for token_scope in token_scopes:
if token_scope == required_scope:
return f(*args, **kwargs)
response = JsonResponse({'message': 'You don\'t have access to this resource'})
response.status_code = 403
return response
return decorated
return require_scope
now when I use the decorator on API for any specific scope, it does not decode the JWT and shows the following ValueError error:
('Could not deserialize key data. The data may be in an incorrect format, it may be encrypted with an unsupported algorithm, or it may be an unsupported key type (e.g. EC curves with explicit parameters).', [_OpenSSLErrorWithText(code=75497580, lib=9, reason=108, reason_text=b'error:0480006C:PEM routines::no start line')])
This is my decoding method:
def jwt_decode_token(token):
header = jwt.get_unverified_header(token)
jwks = requests.get('https://{}/.well-known/jwks.json'.format(settings.SOCIAL_AUTH_AUTH0_DOMAIN)).json()
public_key = None
for jwk in jwks['keys']:
if jwk['kid'] == header['kid']:
public_key = jwt.algorithms.RSAAlgorithm.from_jwk(json.dumps(jwk))
# public_key = "-----BEGIN PUBLIC KEY-----\n" + jwk['x5c'][0] + "\n-----END PUBLIC KEY-----\n"
# print(public_key)
if public_key is None:
raise Exception('Public key not found.')
issuer = 'https://{}/'.format(settings.SOCIAL_AUTH_AUTH0_DOMAIN)
return jwt.decode(
token,
public_key,
audience=settings.AUTH0_TOKEN_API_AUDIENCE,
issuer=issuer,
algorithms=['RS256']
)
and this is the payload I am passing to API Call (it contains the scope):
payload = f"{{\"client_id\":\"{settings.SOCIAL_AUTH_AUTH0_KEY}\",\
\"client_secret\":\"{settings.SOCIAL_AUTH_AUTH0_SECRET}\",\
\"audience\":\"{settings.AUTH0_TOKEN_API_AUDIENCE}\",\
\"grant_type\":\"password\",\
\"username\":\"{email}\",\
\"password\":\"{password}\",\
\"scope\":\"read:messages\"}}"
What's wrong I am doing? Can anyone help?
The reason why it complains about the key is because it is still trying to do the verification. In order to correctly disable this with pyjwt is:
jwt.decode(token, algorithms=["RS256"], options={"verify_signature": False})
See the pyjwt documentation (interestingly, I cannot skip the algorithms block): https://pyjwt.readthedocs.io/en/latest/usage.html#reading-the-claimset-without-validation
I have written the following code :
def check_token(token):
response = requests.get("https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com")
key_list = response.json()
decoded_token = jwt.decode(token, key=key_list, algorithms=["RS256"])
print(f"Decoded token : {decoded_token}")
I am trying to decode the token provided by firebase client side to verify it server-side.
The above code is throwing the following exception :
TypeError: Expecting a PEM-formatted key.
I have tried to not pass a list to the jwt.decode method, only the key content and i have a bigger error that the library could not deserialize the Key.
I was following this answer but i am getting this error.
Is it a requests conversion problem ? What am i doing wrong ?
The 2nd parameter key in decode() seems to take a string value instead of list. The Google API request returns a dict/map containing multiple keys. The flow goes like:
Fetch public keys from the Google API endpoint
Then read headers without validation to get the kid claim then use it to get appropriate key from that dict
That is a X.509 Certificate and not the public key as in this answer so you need to get public key from that.
The following function worked for me:
import jwt
import requests
from cryptography.hazmat.backends import default_backend
from cryptography import x509
def check_token(token):
n_decoded = jwt.get_unverified_header(token)
kid_claim = n_decoded["kid"]
response = requests.get("https://www.googleapis.com/robot/v1/metadata/x509/securetoken#system.gserviceaccount.com")
x509_key = response.json()[kid_claim]
key = x509.load_pem_x509_certificate(x509_key.encode('utf-8'), backend=default_backend())
public_key = key.public_key()
decoded_token = jwt.decode(token, public_key, ["RS256"], options=None, audience="<FIREBASE_PROJECT_ID>")
print(f"Decoded token : {decoded_token}")
check_token("FIREBASE_ID_TOKEN")
don't do much hashing or encrypted server connections but trying to make the best of it,
I'm trying to play around with the coinbase API as their documentation is pretty direct https://docs.pro.coinbase.com/?python#signing-a-message
I've reached a point where I'm stuck getting the error 'bytes' object has no attribute 'encode' when trying pass my message variable into a new hmac, all my parameters for my key, secretkey, passphrase are all strings, I have tried encoding them as ascii and as base64 and as utf-8 and I get the same error
I'm assuming my encryption ordering is somehow the problem but I haven't been able to find anything useful on other stack overflow that matches my issue so far, hoping someone can at least point out what I'm missing
def get_accounts(self):
print("getting accounts")
secret = bytes(self.secret, 'UTF-8')
#unicode objects must be encoded before hashing?? what is that??
timestamp = str(time.time())
message = timestamp + 'GET' + self.baseurl + 'accounts' + ''
message = bytes(message, 'UTF-8')
hmac_key = base64.b64decode(secret)
signature = hmac.new(hmac_key, message, hashlib.sha256)
print("signature ",signature)
#getting stuck here
signature_b64 = signature.digest().encode('base64').rstrip('\n')
headers = {
'CB-ACCESS-SIGN': signature_b64,
'CB-ACCESS-TIMESTAMP': timestamp,
'CB-ACCESS-KEY': self.api_key,
'CB-ACCESS-PASSPHRASE': self.passphrase,
'Content-Type': 'application/json'
}
r = requests.get(self.baseurl + 'accounts', headers=headers)
print (r.json())
okay I got it to produce a normal base64 encoded string, the coinbase api says the signature isn't valid but that's technically a separate issue so I'll leave that alone for now
signature_b64 = base64.b64encode(bytes(signature, 'UTF-8'))
signature_b64 = signature_b64.decode("utf-8")
print('final',signature_b64)
I'm trying to create a signature for an API call - for which the documentation provides these instructions:
timestamp = str(int(time.time()))
message = timestamp + request.method + request.path_url + (request.body or '')
signature = hmac.new(self.secret_key, message, hashlib.sha256).hexdigest()
However, I always get this error:
Exception has occurred: TypeError key: expected bytes or bytearray, but got 'str'
File "/Users/dylanbrandonuom/BouncePay_Code/src/coinbase/Coinbase_API.py", line 26, in __call__
signature = hmac.new(self.secret_key, message, hashlib.sha256).hexdigest()
File "/Users/dylanbrandonuom/BouncePay_Code/src/coinbase/Coinbase_API.py", line 40, in <module>
r = requests.get(api_url + 'user', auth=auth)
I've tried changing
signature = hmac.new(self.secret_key, message, hashlib.sha256).hexdigest()
to
signature = hmac.new(b'self.secret_key', message, hashlib.sha256).hexdigest()
but had no success.
Here is the second part of the error:
api_url = 'https://api.coinbase.com/v2/'
auth = CoinbaseWalletAuth(API_KEY, API_SECRET)
r = requests.get(api_url + 'user', auth=auth)
Is anyone able to let me know why this keeps occurring?
I'm thinking it might be the message variable with request.method and request.path_url, but I'm not sure.
The error message you're seeing tells you that you're passing a (unicode) string as the key argument to hmac.new(), but it expects bytes (or a bytearray).
This means that self.secret_key is a string, rather than a bytes object. There's no indication in your question where in your code self.secret_key is being assigned, but on the assumption that it's a constant somewhere, it might look like this:
SECRET = 'some secret key'
If so, changing that line to something like
SECRET = b'some secret key'
… ought to work. If you're assigning self.secret_key in some other way, it's impossible to know how to fix the problem without seeing that code.
Legend
I expose an API which requires client to sign requests by sending two headers:
Authorization: MyCompany access_key:<signature>
Unix-TimeStamp: <unix utc timestamp in seconds>
To create a signature part, the client should use a secret key issued by my API service.
In Python (Py3k) it could look like:
import base64
import hmac
from hashlib import sha256
from datetime import datetime
UTF8 = 'utf-8'
AUTH_HEADER_PREFIX = 'MyCompany'
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature_base64 = base64.b64encode(new_hmac.digest())
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=str(signature_base64, UTF8).strip()
)
if __name__ == '__main__':
message = str(datetime.utcnow().timestamp())
signature = create_signature('my access key', 'my secret key', message)
print(
'Request headers are',
'Authorization: {}'.format(signature),
'Unix-Timestamp: {}'.format(message),
sep='\n'
)
# For message='1457369891.672671',
# access_key='my access key'
# and secret_key='my secret key' will ouput:
#
# Request headers are
# Authorization: MyCompany my access key:CUfIjOFtB43eSire0f5GJ2Q6N4dX3Mw0KMGVaf6plUI=
# Unix-Timestamp: 1457369891.672671
I wondered if I could avoid dealing with encoding digest of bytes to Base64 and just use HMAC.hexdigest() to retrieve a string.
So that my function will change to:
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature = new_hmac.hexdigest()
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=signature
)
But then I found that Amazon uses similar approach as in my first code snippet:
Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;
Signature = Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) );
Seeing that Amazon doesn't use hex digest I stopped myself to move forward with it because maybe they know something I don't.
Update
I've measured a performance and found hex digest to be faster:
import base64
import hmac
import string
from hashlib import sha256
UTF8 = 'utf-8'
MESSAGE = '1457369891.672671'
SECRET_KEY = 'my secret key'
NEW_HMAC = create_hmac()
def create_hmac():
new_hmac = hmac.new(bytes(SECRET_KEY, UTF8), digestmod=sha256)
new_hmac.update(bytes(MESSAGE, UTF8))
return new_hmac
def base64_digest():
return base64.b64encode(NEW_HMAC.digest())
def hex_digest():
return NEW_HMAC.hexdigest()
if __name__ == '__main__':
from timeit import timeit
print(timeit('base64_digest()', number=1000000,
setup='from __main__ import base64_digest'))
print(timeit('hex_digest()', number=1000000,
setup='from __main__ import hex_digest'))
Results with:
3.136568891000934
2.3460130329913227
Question #1
Does someone know why do they stick to Base64 of bytes digest and don't use just hex digest? Is there some solid reason to keep using this approach over hex digest?
Question #2
According to RFC2716 the format of Authorization header value when using Basic Authentication
is:
Authorization: Base64(username:password)
So basically you wrap with Base64 two values (user's id and password) seprated by colon.
As you can see in my code snippet and in Amazon's documentation nor me, nor Amazon do that for own custom value of the Authorization header.
Would it be a better style to wrap the whole pair as Base64(access_key:signature) to stick closer to this RFC or it doesn't matter at all?
Amazon does use the hex digest in Signature Version 4.
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
Your example is from Signature Version 2, the older algorithm, which does use Base-64 encoding for the signature (and which also is not supported in the newest AWS regions).
So, your concern that AWS knows something you don't is misplaced, since their newer algorithm uses it.
In the Authorization: header, it really doesn't make a difference other than a few extra octets.
Where Base-64 gets messy is when the signature is passed in the query string, because + and (depending on who you ask) / and = require special handling -- they need to be url-escaped ("percent-encoded") as %2B, %2F, and %3D respectively... or you have to make accommodations for the possible variations on the server... or you have to require the use of a non-standard Base-64 alphabet, where + / = becomes - ~ _ the way CloudFront does it. (This particular non-standard alphabet is only one of multiple non-standard options, all "solving" the same problem of magic characters in URLs with Base-64).
Go with hex-encoding.
You will almost inevitably find would-be consumers of your API that find Base-64 to be "difficult."