I'm a little bit confused by what I need to do here for Python, but from the Yubikey API documentation for verifying Yubikeys that have YubiOTP the HMAC signature needs to be generated a specific way - from their documentation:
Generating signatures
The protocol uses HMAC-SHA-1 signatures. The HMAC key to use is the
client API key.
Generate the signature over the parameters in the message. Each
message contains a set of key/value pairs, and the signature is always
over the entire set (excluding the signature itself), and sorted in
alphabetical order of the keys. More precisely, to generate a message
signature do:
Alphabetically sort the set of key/value pairs by key order.
Construct a single line with each ordered key/value pair
concatenated using &, and each key and value contatenated with =. Do
not add any linebreaks. Do not add whitespace. For example:
a=2&b=1&c=3.
Apply the HMAC-SHA-1 algorithm on the line as an octet string using
the API key as key (remember to base64decode the API key obtained from
Yubico).
Base 64 encode the resulting value according to RFC 4648, for
example, t2ZMtKeValdA+H0jVpj3LIichn4=.
Append the value under key h to the message.
Now my understanding of their API from their documentation states the following valid request parameters:
id - the Client ID from Yubico API
otp - the YubiOTP value from the YubiOTP component of a yubikey.
h - the HMAC-SHA1 signature for the request
timestamp - empty does nothing, 1 includes the timestamp in the reply from the server
nonce - A 16 to 40 character long string with random unique data.
sl - a value of 0 to 100 indicating percentage of syncing required by client, or strings "fast" or "Secure" to use server values; if nonexistent server decides
timeout - # of seconds to wait for sync responses; let server decide if absent.
I have a total of two functions I'm trying to use to try and handle all these things and generate the URL. Namely, we the HMAC support function and the verify_url_generate which generates the URL (and API_KEY is statically coded - my API Secret Key from Yubico):
def generate_signature(message, key=base64.b64decode(API_KEY)):
message = bytes(message, 'UTF-8')
digester = hmac.new(key, message, hashlib.sha1)
digest = digester.digest()
signature = base64.urlsafe_b64encode(digest)
return str(signature, 'UTF-8')
def verify_url_generate(otp):
nonce = "".join(secrets.choice(ascii_lowercase) for _ in range(40))
data = OrderedDict(
{
"id": None,
"nonce": None,
"otp": None,
"sl": 50,
"timeout": 10,
"timestamp": 1
}
)
data['otp'] = otp
data['id'] = CLIENT_ID
data['nonce'] = nonce
args = ""
for key, value in data.items():
args += f"{key}={value}&"
sig = generate_signature(args[:-1])
url = YUBICO_API_URL + args + "&h=" + sig
print(url)
return
Any URL generated by this triggers a notice about "BAD_SIGNATURE" from the remote site - any URL generated minus the HMAC sig (h=) parameter works. So we know the issue isn't the URL, it's the HMAC signature.
Does anyone know what I'm doing wrong with my HMAC generation approach, by passing the HMAC sig generator the concatenated args from the ordered dict in key=value separated by & for each parameter format?
Can you try using standard_b64encode and then using urllib.parse.quote(url) in your final URL?
I ask because this page says that "As such, all parameters must be properly URL encoded. In particular, some base64 characters (such as "+") in the value fields needs to be escaped." which means it is expecting +(or %2B) in the args and does a unquote and then normal decode.
Related
I have a big yaml file that I want to store as a secret in my kubernetes cluster. The following command succeeds:
k create secret generic values --from-file=my-values.yaml
But in my code, I want to use the k8s python client.
So I want to do something like this:
def make_k8s_client(kubeconig):
....
def create_secret(name, data, client_api):
secret = client.V1Secret(
api_version="v1",
kind="Secret",
metadata=client.V1ObjectMeta(name=name),
data=data,
)
client_api.create_namespaced_secret(namespace="default",
body=secret)
k8s_api = make_k8s_client("path-to-kubeconfig")
with open("path-to/my-values.yaml") as f:
values = yaml.load(f)
If I pass the yaml like this:
create_secret("mysecret", values, k8s_api)
I get this error:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Secret in version \"v1\" cannot be handled as a Secret: v1.Secret.Data: decode base64: illegal base64 data at input byte 0, error found in #10 byte of ...|pe\": \"abc\", \"def|..., bigger context ...|{\"apiVersion\": \"v1\", \"data\": {\"k8sType\": \"abc\", \"secret\": \"mysecret\", \"type\": \"mytype","reason":"BadRequest","code":400}
If I pass the secret like this:
create_secret("mysecret", base64.urlsafe_b64encode(json.dumps(values).encode()).decode(), k8s_api)
I get this error:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Secret in version \"v1\" cannot be handled as a Secret: v1.Secret.Data: ReadMapCB: expect { or n, but found \", error found in #10 byte of ...| \"data\": \"eyJrOHNUeX|..., bigger context ...|{\"apiVersion\": \"v1\", \"data\": \"eyJrOHNUeXBlIjogImF3cyIsICJnYXJkZW5lclNlY3JldCI6IC|...","reason":"BadRequest","code":400}
How do I have to encode the json file in order to be able to pass it to the python k8s client?
Data contains the secret data. Each key must consist of alphanumeric characters, '-', '_' or '.'. The serialized form of the secret data is a base64 encoded string, representing the arbitrary (possibly non-string) data value here (1).
V1Secret(
api_version="v1",
kind="Secret",
metadata=client.V1ObjectMeta(name=name),
data={
'my-values.yaml': base64.b64encode(json.dumps(values).encode()).decode("utf-8")
},
How to create and use a Secret
Haven't worked in Python much and I'm obviously not sending the proper signature being asked for. How do I hash it and pass it in properly?
SIGNED endpoints require an additional parameter, signature, to be sent in the query string or request body.
Endpoints use HMAC SHA256 signatures. The HMAC SHA256 signature is a keyed HMAC SHA256 operation. Use your secretKey as the key and totalParams as the value for the HMAC operation.
The signature is not case sensitive.
totalParams is defined as the query string concatenated with the request body.
Full Documentation:
https://github.com/binance-exchange/binance-official-api-docs/blob/master/rest-api.md
import requests, json, time, hashlib
apikey = "myactualapikey"
secret = "myrealsecret"
test = requests.get("https://api.binance.com/api/v1/ping")
servertime = requests.get("https://api.binance.com/api/v1/time")
servertimeobject = json.loads(servertime.text)
servertimeint = servertimeobject['serverTime']
hashedsig = hashlib.sha256(secret)
userdata = requests.get("https://api.binance.com/api/v3/account",
params = {
"signature" : hashedsig,
"timestamp" : servertimeint,
},
headers = {
"X-MBX-APIKEY" : apikey,
}
)
print(userdata)
I am getting
{"code":-1100,"msg":"Illegal characters found in parameter 'signature'; legal range is '^[A-Fa-f0-9]{64}$'."}
This:
hashedsig = hashlib.sha256(secret)
Gives you a hash object, not a string. You need to get the string in hex form:
hashedsig = hashlib.sha256(secret).hexdigest()
You could have figured this out by comparing the documentation you linked (which shows they require hex strings) with your original hashedsig and the functions it provides.
Secondly, as a commenter pointed out, you need to apply HMAC, not just SHA256:
params = urlencode({
"signature" : hashedsig,
"timestamp" : servertimeint,
})
hashedsig = hmac.new(secret.encode(), params.encode(), hashlib.sha256).hexdigest()
You can find similar code here: http://python-binance.readthedocs.io/en/latest/_modules/binance/client.html
I am writing a basic TACACS client module. I can get the packets on the wire via struct.pack ok, as I've checked against wireshark. But I am not sure how to handle the byte-wise XOR of the md5 and not even sure if I am creating the hash properly. The RFCs for TACACS say take specific items from the TACACS header and 'secret key' and encrypt it five times, concatenating the previous hash during each run. See below
def encrypt(packet, tac_key):
key = packet[4:8] + bytes(tac_key, 'utf-8') + packet[:1] + packet[2:3]
h1 = md5()
h1.update(key)
hash = h1.digest()
previous = h1.digest()
for i in range(2, 6):
new = md5()
key = key + previous
new.update(key)
hash += new.digest()
previous = new.digest()
return hash
The packet has already been run through struct.pack
body = struct.pack(fmt, TAC_AUTHEN_LOGIN, TAC_PLUS_PRIV_LVL_USER, TAC_PLUS_AUTHEN_TYPE_PAP, TAC_PLUS_AUTHEN_SVC_LOGIN,
user_len, port_len, rem_addr_len, data_len, bytearray(user, 'utf-8'),
bytearray(port, 'utf-8'), bytearray(data, 'utf-8)'))
Is this the correct way to do this or should I not use struct.pack on the body of the packet and just encrypt it?
I was not generating a new pseudo_pad with server's response header. Therefore, the response remained obfuscated.
Legend
I expose an API which requires client to sign requests by sending two headers:
Authorization: MyCompany access_key:<signature>
Unix-TimeStamp: <unix utc timestamp in seconds>
To create a signature part, the client should use a secret key issued by my API service.
In Python (Py3k) it could look like:
import base64
import hmac
from hashlib import sha256
from datetime import datetime
UTF8 = 'utf-8'
AUTH_HEADER_PREFIX = 'MyCompany'
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature_base64 = base64.b64encode(new_hmac.digest())
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=str(signature_base64, UTF8).strip()
)
if __name__ == '__main__':
message = str(datetime.utcnow().timestamp())
signature = create_signature('my access key', 'my secret key', message)
print(
'Request headers are',
'Authorization: {}'.format(signature),
'Unix-Timestamp: {}'.format(message),
sep='\n'
)
# For message='1457369891.672671',
# access_key='my access key'
# and secret_key='my secret key' will ouput:
#
# Request headers are
# Authorization: MyCompany my access key:CUfIjOFtB43eSire0f5GJ2Q6N4dX3Mw0KMGVaf6plUI=
# Unix-Timestamp: 1457369891.672671
I wondered if I could avoid dealing with encoding digest of bytes to Base64 and just use HMAC.hexdigest() to retrieve a string.
So that my function will change to:
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature = new_hmac.hexdigest()
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=signature
)
But then I found that Amazon uses similar approach as in my first code snippet:
Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;
Signature = Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) );
Seeing that Amazon doesn't use hex digest I stopped myself to move forward with it because maybe they know something I don't.
Update
I've measured a performance and found hex digest to be faster:
import base64
import hmac
import string
from hashlib import sha256
UTF8 = 'utf-8'
MESSAGE = '1457369891.672671'
SECRET_KEY = 'my secret key'
NEW_HMAC = create_hmac()
def create_hmac():
new_hmac = hmac.new(bytes(SECRET_KEY, UTF8), digestmod=sha256)
new_hmac.update(bytes(MESSAGE, UTF8))
return new_hmac
def base64_digest():
return base64.b64encode(NEW_HMAC.digest())
def hex_digest():
return NEW_HMAC.hexdigest()
if __name__ == '__main__':
from timeit import timeit
print(timeit('base64_digest()', number=1000000,
setup='from __main__ import base64_digest'))
print(timeit('hex_digest()', number=1000000,
setup='from __main__ import hex_digest'))
Results with:
3.136568891000934
2.3460130329913227
Question #1
Does someone know why do they stick to Base64 of bytes digest and don't use just hex digest? Is there some solid reason to keep using this approach over hex digest?
Question #2
According to RFC2716 the format of Authorization header value when using Basic Authentication
is:
Authorization: Base64(username:password)
So basically you wrap with Base64 two values (user's id and password) seprated by colon.
As you can see in my code snippet and in Amazon's documentation nor me, nor Amazon do that for own custom value of the Authorization header.
Would it be a better style to wrap the whole pair as Base64(access_key:signature) to stick closer to this RFC or it doesn't matter at all?
Amazon does use the hex digest in Signature Version 4.
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
Your example is from Signature Version 2, the older algorithm, which does use Base-64 encoding for the signature (and which also is not supported in the newest AWS regions).
So, your concern that AWS knows something you don't is misplaced, since their newer algorithm uses it.
In the Authorization: header, it really doesn't make a difference other than a few extra octets.
Where Base-64 gets messy is when the signature is passed in the query string, because + and (depending on who you ask) / and = require special handling -- they need to be url-escaped ("percent-encoded") as %2B, %2F, and %3D respectively... or you have to make accommodations for the possible variations on the server... or you have to require the use of a non-standard Base-64 alphabet, where + / = becomes - ~ _ the way CloudFront does it. (This particular non-standard alphabet is only one of multiple non-standard options, all "solving" the same problem of magic characters in URLs with Base-64).
Go with hex-encoding.
You will almost inevitably find would-be consumers of your API that find Base-64 to be "difficult."
I am trying to validate an incoming webhook and so far the resulting hash is not matching the test hash generated by the api.
The docs list the following example for Ruby however I am using Python/Django so any help to 'convert' this function would be appreciated!
Ruby Function
# request_signature - the signature sent in Webhook-Signature
# request_body - the JSON body of the webhook request
# secret - the secret for the webhook endpoint
require "openssl"
digest = OpenSSL::Digest.new("sha256")
calculated_signature = OpenSSL::HMAC.hexdigest(digest, secret, request_body)
if calculated_signature == request_signature
# Signature ok!
else
# Invalid signature. Ignore the webhook and return 498 Token Invalid
end
This is roughly what I have put together myself so far using https://docs.python.org/3/library/hashlib.html.
Python Attempt
import hashlib
secret = "xxxxxxxxxxxxxxxxxx"
json_data = {json data}
h = hashlib.new('sha256')
h.update(secret)
h.update(str(json_data))
calculated_signature = h.hexdigest()
if calculated_signature == webhook_signature:
do_something()
else:
return 498
When I run the above the hashes never match obviously due to my incorrect Python implementation.
Any help/pointers would be greatly appreciated!
I believe it should be something like this:
import hmac
import hashlib
digester = hmac.new(secret, request_body, hashlib.sha256)
calculated_signature = digester.hexdigest()
A few notes:
Use the actual request body. Don't rely on str(json_data) equalling the request body. This will almost certainly fail as python will print out inner strings using repr which will likely leave a bunch of spurious u"..." that aren't actually in the response. json.dumps won't necessarily do better because there could be whitespace differences that are isignificant to JSON, but are very significant to the hmac signature.
hmac is your friend :-)