I have a Rails app that provides a JSON API that is consumed by a python script. Security is important and I've been using HMAC to do it. the rails app and the python script both know the secret key and the signature that they encrypt with it is the URL and the body of the request.
My problem is that the signature of the request doesn't change each time. If it was intercepted then an attacker could send the exact same request with the same digest and I think it would authenticate, een though the attacker doesn't know the secret key.
So I think I need to have something like a timestamp of the request included in the signature - the problem is I don't know how to get at that in python and ruby.
This is my python code:
import hmac
import hashlib
import requests
fetch_path = url_base + '/phone_messages/pending'
fetch_body = '{}'
fetch_signature = fetch_path + ':' + fetch_body
fetch_hmac = hmac.new(api_key.encode('utf-8'), fetch_signature.encode('utf-8'), haslib.sha1).hexdigest()
and this is my ruby code:
signature = "#{request.url}:#{request.body.to_json.to_s}"
hmac_digest = OpenSSL::HMAC.hexdigest('sha1', secret_key, signature)
Question: I need to have something like a timestamp of the request included in the signature
For example:
import hmac, hashlib, datetime
api_key = 'this is a key'
fetch_path = 'http://phone_messages/pending'
fetch_body = '{}'
fetch_data = fetch_path + ':' + fetch_body
for n in range(3):
fetch_signature = fetch_data + str(datetime.datetime.now().timestamp() )
fetch_hmac = hmac.new(api_key.encode('utf-8'), fetch_signature.encode('utf-8'), hashlib.sha1).hexdigest()
print("{}:{} {}".format(n, fetch_signature, fetch_hmac))
Output:
0:http://phone_messages/pending:{}1538660666.768066 cfa49feaeaf0cdc5ec8bcf1057446c425863e83a
1:http://phone_messages/pending:{}1538660666.768358 27d0a5a9f33345babf0c824f45837d3b8863741e
2:http://phone_messages/pending:{}1538660666.768458 67298ad0e9eb8bb629fce4454f092b74ba8d6c66
I recommended, to discus Security at security.stackexchange.com.
As a starting point, read: what-is-a-auth-key-in-the-security-of-the-computers
I resolved this by putting the timestamp (seconds since epoch) in the body of the post request, or parameter of the get request. I simply used the timestamp as the signature for encoding, which means the HMAC hash is different for every request that comes in a different second.
Then to prevent an attacker just using a previously seen timestamp I verified on the server that the timestamp is not more than 5 seconds before the current.
An attacker with a really fast turn around of intercepting a communication and sending an attack could still get through, but I couldn't drop the timeout below 5 seconds because it's already getting some requests timing out.
Since the whole thing is done under SSL I think it should be secure enough.
Related
I'm trying to get make an API for the first time and I've made my app but it says I have to do a local authentication with instructions here:
Link to TDAmeritrade authentication
But it says I have to go on https://auth.tdameritrade.com/auth?response_type=code&redirect_uri={URLENCODED REDIRECT URI}&client_id={URLENCODED Consumer Key}%40AMER.OAUTHAP where I plug in the "url encoded redirect uri" and "urlencoded consumer key" and I dont know how to get the URI. Let's say if I'm using local host 1111 do i just plug in "localhost:1111"? because that didnt work
Perhaps that doesn't even matter? because I was writing the following:
import requests
from config import consumer_key
#daily prices generator
endpoint = "https://api.tdameritrade.com/v1/marketdata/{}/pricehistory".format("AAPL")
#parameters
import time
timeStamp=time.time()
timeStamp=int(timeStamp)
parameters = {'api_key':consumer_key,
'periodType':'day',
'frequencyType':"minute",
'frequency':'5',
'period':'1',
'endDate':str(timeStamp+86400),
'startDate':str(timeStamp),
'extendedHourData':'true'}
#caller
stuff = requests.get(url = endpoint, params = parameters)
#reformater
lister = stuff.json()
lister
which returned "{'error': 'The API key in request query param is either null or blank or invalid.'}"
TDA has some rules
timeStamp needs to be in milliseconds
Can only get past 31 days in minute format
There is also some format constraints.
frequenceType=minute --> then use periodType=day
frequencyType=daily --> then use periodType=month
I am using python and cookielib to talk to an HTTP server that has its date incorrectly set. I have no control over this server, so fixing its time is not a possibility. Unfortunately, the server's incorrect time messes up cookielib because the cookies appear to be expired.
Interestingly, if I go to the same website with any web browser, the browser accepts the cookie and it gets saved. I assume that modern webbrowsers come across misconfigured web servers all the time and see that their Date header is set incorrectly, and adjust cookie expiration dates accordingly.
Has anyone come across this problem before? Is there any way of handling it within Python?
I hacked together a solution that includes live-monkey patching of the urllib library. Definitely not ideal, but if others find a better way, please let me know:
cook_proc = urllib2.HTTPCookieProcessor(cookielib.LWPCookieJar())
cookie_processing_lock = threading.Lock()
def _process_cookies(request, response):
'''Process cookies, but do so in a way that can handle servers with bad
clocks set.'''
# We do some real monkey hacking here, so put it in a lock.
with cookie_processing_lock:
# Get the server date.
date_header = cookielib.http2time(
response.info().getheader('Date') or '')
# Save the old cookie parsing function.
orig_parse = cookielib.parse_ns_headers
# If the server date is off by more than an hour, we'll adjust it.
if date_header:
off_by = time.time() - date_header
if abs(off_by) > 3600:
logging.warning("Server off %.1f hrs."%(abs(off_by)/3600))
# Create our monkey patched
def hacked_parse(ns_headers):
try:
results = orig_parse(ns_headers)
for r in results:
for r_i, (key, val) in enumerate(r):
if key == 'expires':
r[r_i] = key, val + off_by
logging.info("Fixing bad cookie "
"expiration time for: %s"%r[0][0])
logging.info("COOKIE RESULTS: %s", results)
return results
except Exception as e:
logging.error("Problem parse cookie: %s"%e)
raise
cookielib.parse_ns_headers = hacked_parse
response = cook_proc.http_response(request, response)
# Make sure we set the cookie processor back.
cookielib.parse_ns_headers = orig_parse
Legend
I expose an API which requires client to sign requests by sending two headers:
Authorization: MyCompany access_key:<signature>
Unix-TimeStamp: <unix utc timestamp in seconds>
To create a signature part, the client should use a secret key issued by my API service.
In Python (Py3k) it could look like:
import base64
import hmac
from hashlib import sha256
from datetime import datetime
UTF8 = 'utf-8'
AUTH_HEADER_PREFIX = 'MyCompany'
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature_base64 = base64.b64encode(new_hmac.digest())
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=str(signature_base64, UTF8).strip()
)
if __name__ == '__main__':
message = str(datetime.utcnow().timestamp())
signature = create_signature('my access key', 'my secret key', message)
print(
'Request headers are',
'Authorization: {}'.format(signature),
'Unix-Timestamp: {}'.format(message),
sep='\n'
)
# For message='1457369891.672671',
# access_key='my access key'
# and secret_key='my secret key' will ouput:
#
# Request headers are
# Authorization: MyCompany my access key:CUfIjOFtB43eSire0f5GJ2Q6N4dX3Mw0KMGVaf6plUI=
# Unix-Timestamp: 1457369891.672671
I wondered if I could avoid dealing with encoding digest of bytes to Base64 and just use HMAC.hexdigest() to retrieve a string.
So that my function will change to:
def create_signature(access_key, secret_key, message):
new_hmac = hmac.new(bytes(secret_key, UTF8), digestmod=sha256)
new_hmac.update(bytes(message, UTF8))
signature = new_hmac.hexdigest()
return '{prefix} {access_key}:{signature}'.format(
prefix=AUTH_HEADER_PREFIX,
access_key=access_key,
signature=signature
)
But then I found that Amazon uses similar approach as in my first code snippet:
Authorization = "AWS" + " " + AWSAccessKeyId + ":" + Signature;
Signature = Base64( HMAC-SHA1( YourSecretAccessKeyID, UTF-8-Encoding-Of( StringToSign ) ) );
Seeing that Amazon doesn't use hex digest I stopped myself to move forward with it because maybe they know something I don't.
Update
I've measured a performance and found hex digest to be faster:
import base64
import hmac
import string
from hashlib import sha256
UTF8 = 'utf-8'
MESSAGE = '1457369891.672671'
SECRET_KEY = 'my secret key'
NEW_HMAC = create_hmac()
def create_hmac():
new_hmac = hmac.new(bytes(SECRET_KEY, UTF8), digestmod=sha256)
new_hmac.update(bytes(MESSAGE, UTF8))
return new_hmac
def base64_digest():
return base64.b64encode(NEW_HMAC.digest())
def hex_digest():
return NEW_HMAC.hexdigest()
if __name__ == '__main__':
from timeit import timeit
print(timeit('base64_digest()', number=1000000,
setup='from __main__ import base64_digest'))
print(timeit('hex_digest()', number=1000000,
setup='from __main__ import hex_digest'))
Results with:
3.136568891000934
2.3460130329913227
Question #1
Does someone know why do they stick to Base64 of bytes digest and don't use just hex digest? Is there some solid reason to keep using this approach over hex digest?
Question #2
According to RFC2716 the format of Authorization header value when using Basic Authentication
is:
Authorization: Base64(username:password)
So basically you wrap with Base64 two values (user's id and password) seprated by colon.
As you can see in my code snippet and in Amazon's documentation nor me, nor Amazon do that for own custom value of the Authorization header.
Would it be a better style to wrap the whole pair as Base64(access_key:signature) to stick closer to this RFC or it doesn't matter at all?
Amazon does use the hex digest in Signature Version 4.
Authorization: AWS4-HMAC-SHA256 Credential=AKIDEXAMPLE/20150830/us-east-1/iam/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=5d672d79c15b13162d9279b0855cfba6789a8edb4c82c400e06b5924a6f2b5d7
http://docs.aws.amazon.com/general/latest/gr/sigv4-add-signature-to-request.html
Your example is from Signature Version 2, the older algorithm, which does use Base-64 encoding for the signature (and which also is not supported in the newest AWS regions).
So, your concern that AWS knows something you don't is misplaced, since their newer algorithm uses it.
In the Authorization: header, it really doesn't make a difference other than a few extra octets.
Where Base-64 gets messy is when the signature is passed in the query string, because + and (depending on who you ask) / and = require special handling -- they need to be url-escaped ("percent-encoded") as %2B, %2F, and %3D respectively... or you have to make accommodations for the possible variations on the server... or you have to require the use of a non-standard Base-64 alphabet, where + / = becomes - ~ _ the way CloudFront does it. (This particular non-standard alphabet is only one of multiple non-standard options, all "solving" the same problem of magic characters in URLs with Base-64).
Go with hex-encoding.
You will almost inevitably find would-be consumers of your API that find Base-64 to be "difficult."
I am trying to validate an incoming webhook and so far the resulting hash is not matching the test hash generated by the api.
The docs list the following example for Ruby however I am using Python/Django so any help to 'convert' this function would be appreciated!
Ruby Function
# request_signature - the signature sent in Webhook-Signature
# request_body - the JSON body of the webhook request
# secret - the secret for the webhook endpoint
require "openssl"
digest = OpenSSL::Digest.new("sha256")
calculated_signature = OpenSSL::HMAC.hexdigest(digest, secret, request_body)
if calculated_signature == request_signature
# Signature ok!
else
# Invalid signature. Ignore the webhook and return 498 Token Invalid
end
This is roughly what I have put together myself so far using https://docs.python.org/3/library/hashlib.html.
Python Attempt
import hashlib
secret = "xxxxxxxxxxxxxxxxxx"
json_data = {json data}
h = hashlib.new('sha256')
h.update(secret)
h.update(str(json_data))
calculated_signature = h.hexdigest()
if calculated_signature == webhook_signature:
do_something()
else:
return 498
When I run the above the hashes never match obviously due to my incorrect Python implementation.
Any help/pointers would be greatly appreciated!
I believe it should be something like this:
import hmac
import hashlib
digester = hmac.new(secret, request_body, hashlib.sha256)
calculated_signature = digester.hexdigest()
A few notes:
Use the actual request body. Don't rely on str(json_data) equalling the request body. This will almost certainly fail as python will print out inner strings using repr which will likely leave a bunch of spurious u"..." that aren't actually in the response. json.dumps won't necessarily do better because there could be whitespace differences that are isignificant to JSON, but are very significant to the hmac signature.
hmac is your friend :-)
Short version: How do I make signed URLs "on-demand" to mimic Nginx's X-Accel-Redirect behavior (i.e. protecting downloads) with Amazon CloudFront/S3 using Python.
I've got a Django server up and running with an Nginx front-end. I've been getting hammered with requests to it and recently had to install it as a Tornado WSGI application to prevent it from crashing in FastCGI mode.
Now I'm having an issue with my server getting bogged down (i.e. most of its bandwidth is being used up) due to too many requests for media being made to it, I've been looking into CDNs and I believe Amazon CloudFront/S3 would be the proper solution for me.
I've been using Nginx's X-Accel-Redirect header to protect the files from unauthorized downloading, but I don't have that ability with CloudFront/S3--however they do offer signed URLs. I'm no Python expert by far and definitely don't know how to create a Signed URL properly, so I was hoping someone would have a link for how to make these URLs "on-demand" or would be willing to explain how to here, it would be greatly appreciated.
Also, is this the proper solution, even? I'm not too familiar with CDNs, is there a CDN that would be better suited for this?
Amazon CloudFront Signed URLs work differently than Amazon S3 signed URLs. CloudFront uses RSA signatures based on a separate CloudFront keypair which you have to set up in your Amazon Account Credentials page. Here's some code to actually generate a time-limited URL in Python using the M2Crypto library:
Create a keypair for CloudFront
I think the only way to do this is through Amazon's web site. Go into your AWS "Account" page and click on the "Security Credentials" link. Click on the "Key Pairs" tab then click "Create a New Key Pair". This will generate a new key pair for you and automatically download a private key file (pk-xxxxxxxxx.pem). Keep the key file safe and private. Also note down the "Key Pair ID" from amazon as we will need it in the next step.
Generate some URLs in Python
As of boto version 2.0 there does not seem to be any support for generating signed CloudFront URLs. Python does not include RSA encryption routines in the standard library so we will have to use an additional library. I've used M2Crypto in this example.
For a non-streaming distribution, you must use the full cloudfront URL as the resource, however for streaming we only use the object name of the video file. See the code below for a full example of generating a URL which only lasts for 5 minutes.
This code is based loosely on the PHP example code provided by Amazon in the CloudFront documentation.
from M2Crypto import EVP
import base64
import time
def aws_url_base64_encode(msg):
msg_base64 = base64.b64encode(msg)
msg_base64 = msg_base64.replace('+', '-')
msg_base64 = msg_base64.replace('=', '_')
msg_base64 = msg_base64.replace('/', '~')
return msg_base64
def sign_string(message, priv_key_string):
key = EVP.load_key_string(priv_key_string)
key.reset_context(md='sha1')
key.sign_init()
key.sign_update(message)
signature = key.sign_final()
return signature
def create_url(url, encoded_signature, key_pair_id, expires):
signed_url = "%(url)s?Expires=%(expires)s&Signature=%(encoded_signature)s&Key-Pair-Id=%(key_pair_id)s" % {
'url':url,
'expires':expires,
'encoded_signature':encoded_signature,
'key_pair_id':key_pair_id,
}
return signed_url
def get_canned_policy_url(url, priv_key_string, key_pair_id, expires):
#we manually construct this policy string to ensure formatting matches signature
canned_policy = '{"Statement":[{"Resource":"%(url)s","Condition":{"DateLessThan":{"AWS:EpochTime":%(expires)s}}}]}' % {'url':url, 'expires':expires}
#sign the non-encoded policy
signature = sign_string(canned_policy, priv_key_string)
#now base64 encode the signature (URL safe as well)
encoded_signature = aws_url_base64_encode(signature)
#combine these into a full url
signed_url = create_url(url, encoded_signature, key_pair_id, expires);
return signed_url
def encode_query_param(resource):
enc = resource
enc = enc.replace('?', '%3F')
enc = enc.replace('=', '%3D')
enc = enc.replace('&', '%26')
return enc
#Set parameters for URL
key_pair_id = "APKAIAZVIO4BQ" #from the AWS accounts CloudFront tab
priv_key_file = "cloudfront-pk.pem" #your private keypair file
# Use the FULL URL for non-streaming:
resource = "http://34254534.cloudfront.net/video.mp4"
#resource = 'video.mp4' #your resource (just object name for streaming videos)
expires = int(time.time()) + 300 #5 min
#Create the signed URL
priv_key_string = open(priv_key_file).read()
signed_url = get_canned_policy_url(resource, priv_key_string, key_pair_id, expires)
print(signed_url)
#Flash player doesn't like query params so encode them if you're using a streaming distribution
#enc_url = encode_query_param(signed_url)
#print(enc_url)
Make sure that you set up your distribution with a TrustedSigners parameter set to the account holding your keypair (or "Self" if it's your own account)
See Getting started with secure AWS CloudFront streaming with Python for a fully worked example on setting this up for streaming with Python
This feature is now already supported in Botocore, which is the underlying library of Boto3, the latest official AWS SDK for Python. (The following sample requires the installation of the rsa package, but you can use other RSA package too, just define your own "normalized RSA signer".)
The usage looks like this:
from botocore.signers import CloudFrontSigner
# First you create a cloudfront signer based on a normalized RSA signer::
import rsa
def rsa_signer(message):
private_key = open('private_key.pem', 'r').read()
return rsa.sign(
message,
rsa.PrivateKey.load_pkcs1(private_key.encode('utf8')),
'SHA-1') # CloudFront requires SHA-1 hash
cf_signer = CloudFrontSigner(key_id, rsa_signer)
# To sign with a canned policy::
signed_url = cf_signer.generate_presigned_url(
url, date_less_than=datetime(2015, 12, 1))
# To sign with a custom policy::
signed_url = cf_signer.generate_presigned_url(url, policy=my_policy)
Disclaimer: I am the author of that PR.
As many have commented already, the initially accepted answer doesn't apply to Amazon CloudFront in fact, insofar Serving Private Content through CloudFront requires the use of dedicated CloudFront Signed URLs - accordingly secretmike's answer has been correct, but it is meanwhile outdated after he himself took the time and Added support for generating signed URLs for CloudFront (thanks much for this!).
boto now supports a dedicated create_signed_url method and the former binary dependency M2Crypto has recently been replaced with a pure-Python RSA implementation as well, see Don't use M2Crypto for cloudfront URL signing.
As increasingly common, one can find one or more good usage examples within the related unit tests (see test_signed_urls.py), for example test_canned_policy(self) - see setUp(self) for the referenced variables self.pk_idand self.pk_str (obviously you'll need your own keys):
def test_canned_policy(self):
"""
Generate signed url from the Example Canned Policy in Amazon's
documentation.
"""
url = "http://d604721fxaaqy9.cloudfront.net/horizon.jpg?large=yes&license=yes"
expire_time = 1258237200
expected_url = "http://example.com/" # replaced for brevity
signed_url = self.dist.create_signed_url(
url, self.pk_id, expire_time, private_key_string=self.pk_str)
# self.assertEqual(expected_url, signed_url)
This is what I use for create a policy so that I can give access to multiple files with the same "signature":
import json
import rsa
import time
from base64 import b64encode
url = "http://your_domain/*"
expires = int(time.time() + 3600)
pem = """-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----"""
key_pair_id = 'ABX....'
policy = {}
policy['Statement'] = [{}]
policy['Statement'][0]['Resource'] = url
policy['Statement'][0]['Condition'] = {}
policy['Statement'][0]['Condition']['DateLessThan'] = {}
policy['Statement'][0]['Condition']['DateLessThan']['AWS:EpochTime'] = expires
policy = json.dumps(policy)
private_key = rsa.PrivateKey.load_pkcs1(pem)
signature = b64encode(rsa.sign(str(policy), private_key, 'SHA-1'))
print '?Policy=%s&Signature=%s&Key-Pair-Id=%s' % (b64encode(policy),
signature,
key_pair_id)
I can use it for all files under http://your_domain/* for example:
http://your_domain/image2.png?Policy...
http://your_domain/image2.png?Policy...
http://your_domain/file1.json?Policy...
secretmike's answer works, but it is better to use rsa instead of M2Crypto.
I used boto which uses rsa.
import boto
from boto.cloudfront import CloudFrontConnection
from boto.cloudfront.distribution import Distribution
expire_time = int(time.time() +3000)
conn = CloudFrontConnection('ACCESS_KEY_ID', 'SECRET_ACCESS_KEY')
##enter the id or domain name to select a distribution
distribution = Distribution(connection=conn, config=None, domain_name='', id='', last_modified_time=None, status='')
signed_url = distribution.create_signed_url(url='YOUR_URL', keypair_id='YOUR_KEYPAIR_ID_example-APKAIAZVIO4BQ',expire_time=expire_time,private_key_file="YOUR_PRIVATE_KEY_FILE_LOCATION")
Use the boto documentation
I find simple solutions do not need change s3.generate_url ways,
just select your Cloudfront config: Yes, Update bucket policy.
After that change from :
https://xxxx.s3.amazonaws.com/hello.png&Signature=sss&Expires=1585008320&AWSAccessKeyId=kkk
to
https://yyy.cloudfront.net/hello.png&Signature=sss&Expires=1585008320&AWSAccessKeyId=kkk
with yyy.cloudfront.net is your CloudFront domain
refer to: https://aws.amazon.com/blogs/developer/accessing-private-content-in-amazon-cloudfront/