I am trying to validate an incoming webhook and so far the resulting hash is not matching the test hash generated by the api.
The docs list the following example for Ruby however I am using Python/Django so any help to 'convert' this function would be appreciated!
Ruby Function
# request_signature - the signature sent in Webhook-Signature
# request_body - the JSON body of the webhook request
# secret - the secret for the webhook endpoint
require "openssl"
digest = OpenSSL::Digest.new("sha256")
calculated_signature = OpenSSL::HMAC.hexdigest(digest, secret, request_body)
if calculated_signature == request_signature
# Signature ok!
else
# Invalid signature. Ignore the webhook and return 498 Token Invalid
end
This is roughly what I have put together myself so far using https://docs.python.org/3/library/hashlib.html.
Python Attempt
import hashlib
secret = "xxxxxxxxxxxxxxxxxx"
json_data = {json data}
h = hashlib.new('sha256')
h.update(secret)
h.update(str(json_data))
calculated_signature = h.hexdigest()
if calculated_signature == webhook_signature:
do_something()
else:
return 498
When I run the above the hashes never match obviously due to my incorrect Python implementation.
Any help/pointers would be greatly appreciated!
I believe it should be something like this:
import hmac
import hashlib
digester = hmac.new(secret, request_body, hashlib.sha256)
calculated_signature = digester.hexdigest()
A few notes:
Use the actual request body. Don't rely on str(json_data) equalling the request body. This will almost certainly fail as python will print out inner strings using repr which will likely leave a bunch of spurious u"..." that aren't actually in the response. json.dumps won't necessarily do better because there could be whitespace differences that are isignificant to JSON, but are very significant to the hmac signature.
hmac is your friend :-)
Related
I'm a little bit confused by what I need to do here for Python, but from the Yubikey API documentation for verifying Yubikeys that have YubiOTP the HMAC signature needs to be generated a specific way - from their documentation:
Generating signatures
The protocol uses HMAC-SHA-1 signatures. The HMAC key to use is the
client API key.
Generate the signature over the parameters in the message. Each
message contains a set of key/value pairs, and the signature is always
over the entire set (excluding the signature itself), and sorted in
alphabetical order of the keys. More precisely, to generate a message
signature do:
Alphabetically sort the set of key/value pairs by key order.
Construct a single line with each ordered key/value pair
concatenated using &, and each key and value contatenated with =. Do
not add any linebreaks. Do not add whitespace. For example:
a=2&b=1&c=3.
Apply the HMAC-SHA-1 algorithm on the line as an octet string using
the API key as key (remember to base64decode the API key obtained from
Yubico).
Base 64 encode the resulting value according to RFC 4648, for
example, t2ZMtKeValdA+H0jVpj3LIichn4=.
Append the value under key h to the message.
Now my understanding of their API from their documentation states the following valid request parameters:
id - the Client ID from Yubico API
otp - the YubiOTP value from the YubiOTP component of a yubikey.
h - the HMAC-SHA1 signature for the request
timestamp - empty does nothing, 1 includes the timestamp in the reply from the server
nonce - A 16 to 40 character long string with random unique data.
sl - a value of 0 to 100 indicating percentage of syncing required by client, or strings "fast" or "Secure" to use server values; if nonexistent server decides
timeout - # of seconds to wait for sync responses; let server decide if absent.
I have a total of two functions I'm trying to use to try and handle all these things and generate the URL. Namely, we the HMAC support function and the verify_url_generate which generates the URL (and API_KEY is statically coded - my API Secret Key from Yubico):
def generate_signature(message, key=base64.b64decode(API_KEY)):
message = bytes(message, 'UTF-8')
digester = hmac.new(key, message, hashlib.sha1)
digest = digester.digest()
signature = base64.urlsafe_b64encode(digest)
return str(signature, 'UTF-8')
def verify_url_generate(otp):
nonce = "".join(secrets.choice(ascii_lowercase) for _ in range(40))
data = OrderedDict(
{
"id": None,
"nonce": None,
"otp": None,
"sl": 50,
"timeout": 10,
"timestamp": 1
}
)
data['otp'] = otp
data['id'] = CLIENT_ID
data['nonce'] = nonce
args = ""
for key, value in data.items():
args += f"{key}={value}&"
sig = generate_signature(args[:-1])
url = YUBICO_API_URL + args + "&h=" + sig
print(url)
return
Any URL generated by this triggers a notice about "BAD_SIGNATURE" from the remote site - any URL generated minus the HMAC sig (h=) parameter works. So we know the issue isn't the URL, it's the HMAC signature.
Does anyone know what I'm doing wrong with my HMAC generation approach, by passing the HMAC sig generator the concatenated args from the ordered dict in key=value separated by & for each parameter format?
Can you try using standard_b64encode and then using urllib.parse.quote(url) in your final URL?
I ask because this page says that "As such, all parameters must be properly URL encoded. In particular, some base64 characters (such as "+") in the value fields needs to be escaped." which means it is expecting +(or %2B) in the args and does a unquote and then normal decode.
I have a Rails app that provides a JSON API that is consumed by a python script. Security is important and I've been using HMAC to do it. the rails app and the python script both know the secret key and the signature that they encrypt with it is the URL and the body of the request.
My problem is that the signature of the request doesn't change each time. If it was intercepted then an attacker could send the exact same request with the same digest and I think it would authenticate, een though the attacker doesn't know the secret key.
So I think I need to have something like a timestamp of the request included in the signature - the problem is I don't know how to get at that in python and ruby.
This is my python code:
import hmac
import hashlib
import requests
fetch_path = url_base + '/phone_messages/pending'
fetch_body = '{}'
fetch_signature = fetch_path + ':' + fetch_body
fetch_hmac = hmac.new(api_key.encode('utf-8'), fetch_signature.encode('utf-8'), haslib.sha1).hexdigest()
and this is my ruby code:
signature = "#{request.url}:#{request.body.to_json.to_s}"
hmac_digest = OpenSSL::HMAC.hexdigest('sha1', secret_key, signature)
Question: I need to have something like a timestamp of the request included in the signature
For example:
import hmac, hashlib, datetime
api_key = 'this is a key'
fetch_path = 'http://phone_messages/pending'
fetch_body = '{}'
fetch_data = fetch_path + ':' + fetch_body
for n in range(3):
fetch_signature = fetch_data + str(datetime.datetime.now().timestamp() )
fetch_hmac = hmac.new(api_key.encode('utf-8'), fetch_signature.encode('utf-8'), hashlib.sha1).hexdigest()
print("{}:{} {}".format(n, fetch_signature, fetch_hmac))
Output:
0:http://phone_messages/pending:{}1538660666.768066 cfa49feaeaf0cdc5ec8bcf1057446c425863e83a
1:http://phone_messages/pending:{}1538660666.768358 27d0a5a9f33345babf0c824f45837d3b8863741e
2:http://phone_messages/pending:{}1538660666.768458 67298ad0e9eb8bb629fce4454f092b74ba8d6c66
I recommended, to discus Security at security.stackexchange.com.
As a starting point, read: what-is-a-auth-key-in-the-security-of-the-computers
I resolved this by putting the timestamp (seconds since epoch) in the body of the post request, or parameter of the get request. I simply used the timestamp as the signature for encoding, which means the HMAC hash is different for every request that comes in a different second.
Then to prevent an attacker just using a previously seen timestamp I verified on the server that the timestamp is not more than 5 seconds before the current.
An attacker with a really fast turn around of intercepting a communication and sending an attack could still get through, but I couldn't drop the timeout below 5 seconds because it's already getting some requests timing out.
Since the whole thing is done under SSL I think it should be secure enough.
I'm trying to work with Duo's python client (https://github.com/duosecurity/duo_client_python) and I believe I'm simply missing something with my novice python eyes. I somehow need to authenticate my requests to the API using an HMAC signature -- which I've not worked with before but seems trivial to generate (they even provide a python function). Doc here https://duo.com/docs/adminapi#authentication
I'm at a loss as to HOW to craft this signature for authentication, prior to passing my request to the API. From https://duo.com/docs/adminapi#authentication:
The API uses HTTP Basic Authentication to authenticate requests. Use your >Duo application’s integration key as the HTTP Username.
Generate the HTTP Password as an HMAC signature of the request. This will >be different for each request and must be re-generated each time.
It then goes into which parameters are added and necessary for the HMAC signature to be generated properly. I understand this part. My issue is how and when to pass the HMAC signature that I generate. As the Duo doc doesn't specifically go into that, I'm thinking that this is something I should already know [were I not such a python novice].
For example, simple auth calls work fine (as they need no authentication):
root#box:~# python -m duo_client.client --ikey XXXXXXXX --skey XXXXXXXX --host XXXXXXXX.duosecurity.com --method GET --path /auth/v2/check 200 OK
{
"response":
{ "time": 1496437236 }
,
"stat": "OK"
}
However, calls using admin require authentication, and thus fail with:
root#box:~# python -m duo_client.client --ikey XXXXXXXX --skey XXXXXXXX --host XXXXXXXX.duosecurity.com --method GET --path /admin/v1/users signature=XXXXXXXX
401 Unauthorized
{
"code": 40103,
"message": "Invalid signature in request credentials",
"stat": "FAIL"
}
Thanks in advance for any insight!
==EDIT==
So I thought I'd post the function that Duo provide for creating the signature, which appears similar to what is happening in the StackOverflow question I found that I thought might help (Python, HTTPS GET with basic authentication). From https://duo.com/docs/adminapi#authentication:
def sign(method, host, path, params, skey, ikey):
"""
Return HTTP Basic Authentication ("Authorization" and "Date") headers.
method, host, path: strings from request
params: dict of request parameters
skey: secret key
ikey: integration key
"""
# create canonical string
now = email.Utils.formatdate()
canon = [now, method.upper(), host.lower(), path]
args = []
for key in sorted(params.keys()):
val = params[key]
if isinstance(val, unicode):
val = val.encode("utf-8")
args.append(
'%s=%s' % (urllib.quote(key, '~'), urllib.quote(val, '~')))
canon.append('&'.join(args))
canon = '\n'.join(canon)
# sign canonical string
sig = hmac.new(skey, canon, hashlib.sha1)
auth = '%s:%s' % (ikey, sig.hexdigest())
# return headers
return {'Date': now, 'Authorization': 'Basic %s' % base64.b64encode(auth)}
I've used the above function in a simple script to print out (just so I can visualize) what should be getting passed within the script I'll create to server our needs -- I added the function to a script and used print by adding the following:
# Printing Signature Headers ### TESTING ###
print sign('GET', 'XXXhostXXX', '/admin/v1/users', 'XXXparamXXX', 'XXXskeyXXX', 'XXXXikeyXXXX')
However, I'm getting this error:
root#box:~# ./duo.py
Traceback (most recent call last):
File "./duo.py", line 39, in
print sign('GET', 'XXXhostXXX', '/admin/v1/users', 'XXXparamXXX', 'XXXskeyXXX', 'XXXikeyXXX')
File "./duo.py", line 18, in sign
for key in sorted(params.keys()):
AttributeError: 'str' object has no attribute 'keys'
Am I just missing something here? I've been looking for what could cause that error but I thought I'd ask here as well.
This was an "issue" on Duo's side...wrong skey was being used and my user did not have the proper access.
Legend
I expose an API which requires client to sign requests by sending two headers:
Authorization: MyCompany access_key:<signature>
Unix-TimeStamp: <unix utc timestamp in seconds>
To create a signature part, the client should use a secret key issued by my API service.
In Python (Py3k) it could look like:
import base64
import hmac
from hashlib import sha256
from datetime import datetime
UTF8 = 'utf-8'
AUTH_HEADER_PREFIX = 'MyCompany'
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature_base64 = base64.b64encode(new_hmac.digest())
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=str(signature_base64, UTF8).strip()
)
if __name__ == '__main__':
message = str(datetime.utcnow().timestamp())
signature = create_signature('my access key', 'my secret key', message)
print(
'Request headers are',
'Authorization: {}'.format(signature),
'Unix-Timestamp: {}'.format(message),
sep='\n'
)
# For message='1457369891.672671',
# access_key='my access key'
# and secret_key='my secret key' will ouput:
#
# Request headers are
# Authorization: MyCompany my access key:CUfIjOFtB43eSire0f5GJ2Q6N4dX3Mw0KMGVaf6plUI=
# Unix-Timestamp: 1457369891.672671
I wondered if I could avoid dealing with encoding digest of bytes to Base64 and just use HMAC.hexdigest() to retrieve a string.
So that my function will change to:
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature = new_hmac.hexdigest()
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=signature
)
But then I found that Amazon uses similar approach as in my first code snippet:
Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;
Signature = Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) );
Seeing that Amazon doesn't use hex digest I stopped myself to move forward with it because maybe they know something I don't.
Update
I've measured a performance and found hex digest to be faster:
import base64
import hmac
import string
from hashlib import sha256
UTF8 = 'utf-8'
MESSAGE = '1457369891.672671'
SECRET_KEY = 'my secret key'
NEW_HMAC = create_hmac()
def create_hmac():
new_hmac = hmac.new(bytes(SECRET_KEY, UTF8), digestmod=sha256)
new_hmac.update(bytes(MESSAGE, UTF8))
return new_hmac
def base64_digest():
return base64.b64encode(NEW_HMAC.digest())
def hex_digest():
return NEW_HMAC.hexdigest()
if __name__ == '__main__':
from timeit import timeit
print(timeit('base64_digest()', number=1000000,
setup='from __main__ import base64_digest'))
print(timeit('hex_digest()', number=1000000,
setup='from __main__ import hex_digest'))
Results with:
3.136568891000934
2.3460130329913227
Question #1
Does someone know why do they stick to Base64 of bytes digest and don't use just hex digest? Is there some solid reason to keep using this approach over hex digest?
Question #2
According to RFC2716 the format of Authorization header value when using Basic Authentication
is:
Authorization: Base64(username:password)
So basically you wrap with Base64 two values (user's id and password) seprated by colon.
As you can see in my code snippet and in Amazon's documentation nor me, nor Amazon do that for own custom value of the Authorization header.
Would it be a better style to wrap the whole pair as Base64(access_key:signature) to stick closer to this RFC or it doesn't matter at all?
Amazon does use the hex digest in Signature Version 4.
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
Your example is from Signature Version 2, the older algorithm, which does use Base-64 encoding for the signature (and which also is not supported in the newest AWS regions).
So, your concern that AWS knows something you don't is misplaced, since their newer algorithm uses it.
In the Authorization: header, it really doesn't make a difference other than a few extra octets.
Where Base-64 gets messy is when the signature is passed in the query string, because + and (depending on who you ask) / and = require special handling -- they need to be url-escaped ("percent-encoded") as %2B, %2F, and %3D respectively... or you have to make accommodations for the possible variations on the server... or you have to require the use of a non-standard Base-64 alphabet, where + / = becomes - ~ _ the way CloudFront does it. (This particular non-standard alphabet is only one of multiple non-standard options, all "solving" the same problem of magic characters in URLs with Base-64).
Go with hex-encoding.
You will almost inevitably find would-be consumers of your API that find Base-64 to be "difficult."
I have a CherryPy web application that requires authentication. I have working HTTP Basic Authentication with a configuration that looks like this:
app_config = {
'/' : {
'tools.sessions.on': True,
'tools.sessions.name': 'zknsrv',
'tools.auth_basic.on': True,
'tools.auth_basic.realm': 'zknsrv',
'tools.auth_basic.checkpassword': checkpassword,
}
}
HTTP auth works great at this point. For example, this will give me the successful login message that I defined inside AuthTest:
curl http://realuser:realpass#localhost/AuthTest/
Since sessions are on, I can save cookies and examine the one that CherryPy sets:
curl --cookie-jar cookie.jar http://realuser:realpass#localhost/AuthTest/
The cookie.jar file will end up looking like this:
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
localhost FALSE / FALSE 1348640978 zknsrv 821aaad0ba34fd51f77b2452c7ae3c182237deb3
However, I'll get an HTTP 401 Not Authorized failure if I provide this session ID without the username and password, like this:
curl --cookie 'zknsrv=821aaad0ba34fd51f77b2452c7ae3c182237deb3' http://localhost/AuthTest
What am I missing?
Thanks very much for any help.
So, the short answer is you can do this, but you have to write your own CherryPy tool (a before_handler), and you must not enable Basic Authentication in the CherryPy config (that is, you shouldn't do anything like tools.auth.on or tools.auth.basic... etc) - you have to handle HTTP Basic Authentication yourself. The reason for this is that the built-in Basic Authentication stuff is apparently pretty primitive. If you protect something by enabling Basic Authentication like I did above, it will do that authentication check before it checks the session, and your cookies will do nothing.
My solution, in prose
Fortunately, even though CherryPy doesn't have a way to do both built-in, you can still use its built-in session code. You still have to write your own code for handling the Basic Authentication part, but in total this is not so bad, and using the session code is a big win because writing a custom session manager is a good way to introduce security bugs into your webapp.
I ended up being able to take a lot of things from a page on the CherryPy wiki called Simple authentication and access restrictions helpers. That code uses CP sessions, but rather than Basic Auth it uses a special page with a login form that submits ?username=USERNAME&password=PASSWORD. What I did is basically nothing more than changing the provided check_auth function from using the special login page to using the HTTP auth headers.
In general, you need a function you can add as a CherryPy tool - specifically a before_handler. (In the original code, this function was called check_auth(), but I renamed it to protect().) This function first tries to see if the cookies contain a (valid) session ID, and if that fails, it tries to see if there is HTTP auth information in the headers.
You then need a way to require authentication for a given page; I do this with require(), plus some conditions, which are just callables that return True. In my case, those conditions are zkn_admin(), and user_is() functions; if you have more complex needs, you might want to also look at member_of(), any_of(), and all_of() from the original code.
If you do it like that, you already have a way to log in - you just submit a valid session cookie or HTTPBA credentials to any URL you protect with the #require() decorator. All you need now is a way to log out.
(The original code instead has an AuthController class which contains login() and logout(), and you can use the whole AuthController object in your HTTP document tree by just putting auth = AuthController() inside your CherryPy root class, and get to it with a URL of e.g. http://example.com/auth/login and http://example.com/auth/logout. My code doesn't use an authcontroller object, just a few functions.)
Some notes about my code
Caveat: Because I wrote my own parser for HTTP auth headers, it only parses what I told it about, which means just HTTP Basic Auth - not, for example, Digest Auth or anything else. For my application that's fine; for yours, it may not be.
It assumes a few functions defined elsewhere in my code: user_verify() and user_is_admin()
I also use a debugprint() function which only prints output when a DEBUG variable is set, and I've left these calls in for clarity.
You can call it cherrypy.tools.WHATEVER (see the last line); I called it zkauth based on the name of my app. Take care NOT to call it auth, or the name of any other built-in tool, though .
You then have to enable cherrypy.tools.WHATEVER in your CherryPy configuration.
As you can see by all the TODO: messages, this code is still in a state of flux and not 100% tested against edge cases - sorry about that! It will still give you enough of an idea to go on, though, I hope.
My solution, in code
import base64
import re
import cherrypy
SESSION_KEY = '_zkn_username'
def protect(*args, **kwargs):
debugprint("Inside protect()...")
authenticated = False
conditions = cherrypy.request.config.get('auth.require', None)
debugprint("conditions: {}".format(conditions))
if conditions is not None:
# A condition is just a callable that returns true or false
try:
# TODO: I'm not sure if this is actually checking for a valid session?
# or if just any data here would work?
this_session = cherrypy.session[SESSION_KEY]
# check if there is an active session
# sessions are turned on so we just have to know if there is
# something inside of cherrypy.session[SESSION_KEY]:
cherrypy.session.regenerate()
# I can't actually tell if I need to do this myself or what
email = cherrypy.request.login = cherrypy.session[SESSION_KEY]
authenticated = True
debugprint("Authenticated with session: {}, for user: {}".format(
this_session, email))
except KeyError:
# If the session isn't set, it either wasn't present or wasn't valid.
# Now check if the request includes HTTPBA?
# FFR The auth header looks like: "AUTHORIZATION: Basic <base64shit>"
# TODO: cherrypy has got to handle this for me, right?
authheader = cherrypy.request.headers.get('AUTHORIZATION')
debugprint("Authheader: {}".format(authheader))
if authheader:
#b64data = re.sub("Basic ", "", cherrypy.request.headers.get('AUTHORIZATION'))
# TODO: what happens if you get an auth header that doesn't use basic auth?
b64data = re.sub("Basic ", "", authheader)
decodeddata = base64.b64decode(b64data.encode("ASCII"))
# TODO: test how this handles ':' characters in username/passphrase.
email,passphrase = decodeddata.decode().split(":", 1)
if user_verify(email, passphrase):
cherrypy.session.regenerate()
# This line of code is discussed in doc/sessions-and-auth.markdown
cherrypy.session[SESSION_KEY] = cherrypy.request.login = email
authenticated = True
else:
debugprint ("Attempted to log in with HTTBA username {} but failed.".format(
email))
else:
debugprint ("Auth header was not present.")
except:
debugprint ("Client has no valid session and did not provide HTTPBA credentials.")
debugprint ("TODO: ensure that if I have a failure inside the 'except KeyError'"
+ " section above, it doesn't get to this section... I'd want to"
+ " show a different error message if that happened.")
if authenticated:
for condition in conditions:
if not condition():
debugprint ("Authentication succeeded but authorization failed.")
raise cherrypy.HTTPError("403 Forbidden")
else:
raise cherrypy.HTTPError("401 Unauthorized")
cherrypy.tools.zkauth = cherrypy.Tool('before_handler', protect)
def require(*conditions):
"""A decorator that appends conditions to the auth.require config
variable."""
def decorate(f):
if not hasattr(f, '_cp_config'):
f._cp_config = dict()
if 'auth.require' not in f._cp_config:
f._cp_config['auth.require'] = []
f._cp_config['auth.require'].extend(conditions)
return f
return decorate
#### CONDITIONS
#
# Conditions are callables that return True
# if the user fulfills the conditions they define, False otherwise
#
# They can access the current user as cherrypy.request.login
# TODO: test this function with cookies, I want to make sure that cherrypy.request.login is
# set properly so that this function can use it.
def zkn_admin():
return lambda: user_is_admin(cherrypy.request.login)
def user_is(reqd_email):
return lambda: reqd_email == cherrypy.request.login
#### END CONDITIONS
def logout():
email = cherrypy.session.get(SESSION_KEY, None)
cherrypy.session[SESSION_KEY] = cherrypy.request.login = None
return "Logout successful"
Now all you have to do is enable both builtin sessions and your own cherrypy.tools.WHATEVER in your CherryPy configuration. Again, take care not to enable cherrypy.tools.auth. My configuration ended up looking like this:
config_root = {
'/' : {
'tools.zkauth.on': True,
'tools.sessions.on': True,
'tools.sessions.name': 'zknsrv',
}
}