I'm having an issue and not quite sure how to explain it but I will try my best.
So I'm attempting to authenticate with an API which requires grabbing a private key that is provided by the website in hex representation (e.g. an example token is "665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569") and decode to ascii to create a keyed hash so I may pass it in an http header which is part of the authentication process.
When it comes to python2 I can simply
import hashlib
import hmac
import requests
headers = {
"custom header": hmac.new("665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569".decode("hex"),
msg="whatever",
digestmod=hashlib.sha256).hexdigest()
}
requests.get("my url", headers=headers)
However, I cannot get this working in python3 despite several hours of googling, various SO posts and looking at the official docs for hmac.
This seems to stem from the differences between how python2 and 3 handle strings.
In python2 running "665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569".decode("hex") returns this string of characters "f\ ��Q~S�?ۛM -0�V��GE�i" which is passed to hmac.new()
Somethings I have tried in Python3 after searching around:
bytes.fromhex('665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569').decode('utf-8')
bytes.fromhex('665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569').decode('ascii')
import binascii
binascii.unhexlify(b"665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569")
But these all error or output different returns that hmac.new() won't accept. I'm assuming there's a simple fix that I'm just ignorant on since I'm not very knowledgeable about the nuances of how p2 and p3 handle strings.
One of your attempts is correct:
In [1]: import binascii
...: binascii.unhexlify(b"665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569")
...:
Out[1]: b'f\\ \xb3\xc4Q~\x02S\x11\x16\x0b\x7f\xec?\xdb\x9bM\t\x1f\x14-0\x8cV\x8d\x0e\xecGE\xf5i'
If you get a wrong result from hmac afterwards, you can post a question about that specific scenario, with some examples comparing python2/3.
You may be running into a problem with the message itself, which needs to explicitly use bytes, not a string. These two give the same values:
Python 3:
In [10]: hmac.new(binascii.unhexlify(b"665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569"),
...: msg="whatever".encode('utf-8'),
...: digestmod=hashlib.sha256).hexdigest()
Out[10]: '79ca98357629c22a094c67a02638076573ec41d2c5ce8996435656f8488552d0'
Python 2:
>>> hmac.new("665c20b3c4517e025311160b7fec3fdb9b4d091f142d308c568d0eec4745f569".decode("hex"),
... msg="whatever",
... digestmod=hashlib.sha256).hexdigest()
'79ca98357629c22a094c67a02638076573ec41d2c5ce8996435656f8488552d0'
Related
The following python code produces a valid JWT token, using pyjwt:
>>> import jwt
>>> payload = {'nested': [{'name': 'me', 'id': '1'}]}
>>> token = jwt.encode(payload, 'secret')
>>> token.decode()
ey[...]ko0Zq_k
pyjwt also supports calls from the command line interface. But the docs only show examples with = separated key value pairs and not with nested payloads.
My best guess was this:
$ pyjwt --key=secret encode nested=[{name=me, id=1}]
ey[...]0FRW9gyU # not the same token as above :(
Which didn't work. Is it simply not supported?
As mentioned, your command line token when decoded returns this json object:
{'nested': '[{name=me,', 'id': '1}]'}
A quick dive into the __main__.py of jwt package gives this little snippet:
... snipped
def encode_payload(args):
# Try to encode
if args.key is None:
raise ValueError('Key is required when encoding. See --help for usage.')
# Build payload object to encode
payload = {}
for arg in args.payload:
k, v = arg.split('=', 1)
... some additional handling on v for time, int, float and True/False/None
... snipped
As you can see the key and value of the payload is determined directly based on the split('=', 1), so it anything passed the first = in your command line following a key will always be determined as a single value (with some conversion afterwards).
So in short, nested dicts in CLI is not supported.
However, the semi-good news is, there are certain ways you can work around these:
Run an impromptu statement off Python's CLI directly like so:
> python -c "import jwt; print(jwt.encode({'nested':[{'name':'me', 'id':'1'}]}, 'secret').decode('utf-8'))"
# eyJ...Zq_k
Not exactly ideal, but it gives you what you need.
Save the same script into a .py capable of taking args and execute it on Python's CLI:
import sys, jwt
my_json = sys.argv[0]
token = jwt.encode(eval(my_json), 'secret')
print(token.decode('utf-8'))
# run in CLI
> python my_encode.py "{'nested':[{'name':'me', 'id':'1'}]}"
# eyJ...Zq_k
Note the use of eval() here is not ideal because of security concerns. This is just my lazy way of implementing it because I don't want to write a parser for the args. If you absolutely must use CLI for your implementation and it's exposed, I would highly recommend you invest the effort into cleansing and parsing the argvs more carefully.
The most contrived way: you can try to modify the Lib\site-packages\jwt\__main__.py function (at your own peril) to suit your need until official support is added. I'd caution you should be rather comfortable with writing your own parse though before considering messing with the main code. I took a few stab at it before I realize the limitations you will be running into:
a. The main encode() method doesn't consider a list as a valid JSON object (but it should). So right off the bat you must have a dict like string to manipulate.
b. The code always forces numbers to be cast as int or float if possible. You'll need to escape it somehow or entirely change the way it handle numbers.
My attempt went something like this:
def func(result, payload):
for arg in payload:
k, v = arg.split('=', 1)
if v.startswith('{') and v.endswith('}'):
result[k] = func({}, v[1:-1])
else:
... the rest of the existing code
However I quickly ran into the limitation of the original arguments are already space delimited and assume it's a k, v pair, I would need to further handle another delimiter like , as well as capability to handle lists, and it could get messier. It's definitely doable, and the effect is immediate i.e. the CLI runs directly off of this __main__.py, but it's more work than I'd like to invest at the moment so I leave it with your capable hands.
The effort to overcome these issues to achieve what you need might be more than necessary, depend on your skill and comfort level. So pick your battle... if CLI is not absolutely necessary, I'd suggest just use the .py methods instead.
I am trying to create a token for an API call in R. I have example code and output in Python, but I am unable to replicate in R. I know little to nothing about encoding, decoding, etc. Hoping someone can shed some light on what I can do to make these outputs match. Here is a toy example.
R Code:
library(RCurl)
library(digest)
api_key = "abcdefghijklmnopqrstuvwxyz123456789=="
decoded_api_key = base64Decode(api_key)
hmac_binary = hmac(decoded_api_key, "MySpecialString", "sha512")
hmac_encoded = base64Encode(digest(hmac_binary))
print(as.character(hmac_encoded))
# ZmZjZDBlMjkyNzg3NDNmYWM1ZDcyNjVkNmY4ZmM1OGQ=
Python:
import hmac
import hashlib
import base64
api_key = "abcdefghijklmnopqrstuvwxyz123456789=="
decoded_api_key = base64.b64decode(api_key)
hmac_binary = hmac.new(decoded_api_key, "MySpecialString", hashlib.sha512)
hmac_encoded = base64.b64encode(hmac_binary.digest())
print(hmac_encoded)
# MduxNfXVkwcOtCpBWJEl96S43boYVYTtHb4waR21ARCMo6iokKuxbwEJMTkuytbrCOxvBqKCYiaZiV/AyHTEcw==
The answers I obtain are given at the end of the code blocks. Clearly they don't match. I'd like someone to help me change my R code to match the Python output.
Thanks in advance.
The digest() function in R doesn't do the same thing as the .digest() method in python. It doesn't extract the value, it computes a new digest for whatever you pass in. Also the hmac function will by default return a string with the bytes in it, but you want to base64 encode the actual bytes so you need to make sure to request the raw values. Finally, a base64 string should have a multiple of 4 characters in the string. The extra padding seems to return a different value. So this should give the same value as the python code
api_key = "abcdefghijklmnopqrstuvwxyz123456789="
decoded_api_key = base64Decode(api_key)
hmac_binary = hmac(decoded_api_key, "MySpecialString", "sha512", raw=TRUE)
hmac_encoded = base64Encode(hmac_binary)
print(as.character(hmac_encoded))
# [1] "MduxNfXVkwcOtCpBWJEl96S43boYVYTtHb4waR21ARCMo6iokKuxbwEJMTkuytbrCOxvBqKCYiaZiV/AyHTEcw=="
I'm using Python 3 for this.
Basically, I'm making a API call using urllib and getting the error:
"ValueError: Single '}' encountered in format string"
I have looked at a variety of other solutions but they don't seem to work.
Basically what I am doing is:
import urllib.request
import urllib.parse
def query_person(first, last):
person_request = urllib.request.urlopen('http://api.querysite.com/content/search/index:AUTHOR?query=authlast%28%27{last}}%27%29%20AND%20authfirst%28%27{first}}%27%29&'.format(first=first,last=last))
return(person_request)
print(query_person("John", "Doe"))
The actual API will not be reproducible since it requires an API key (ommited obviously) as well as the need to be on a verified network.
I think the issue has to do with "{last}}%27%29%20AND%20authfirst%28%27{first}}" having an extra bracket. For example, if I wanted to just query it in my url bar without python or .format(), it would look like:
http://api.querysite.com/content/search/index:AUTHOR?query=authlast%28%27Doe}%27%29%20AND%20authfirst%28%27John}%27%29&
or more specifically: Doe}%27%29%20AND%20authfirst%28%27John}%27%29&
If I use the latter method in python, I have no issues, but it does not allow me to input names to query of course.
You need to double up on your single brace if you want it to remain in the string:
For example:
'{first}}}'.format(first='John') == 'John}'
In your case:
person_request = urllib.request.urlopen('http://api.querysite.com/content/search/index:AUTHOR?query=authlast%28%27{last}}}%27%29%20AND%20authfirst%28%27{first}}}%27%29&'.format(first=first,last=last))
I'm following the directions on the API documentation precisely, and after some frustration I finally put together something directly from their examples on http://docs.amazonwebservices.com/AWSECommerceService/2011-08-01/DG/rest-signature.html
I've tried this python script on a few machines and have gotten the same result on all of them.
import hmac
from base64 import b64encode
from hashlib import sha256
secret_key = '1234567890'
to_sign = """GET
webservices.amazon.com
/onca/xml
AWSAccessKeyId=AKIAI44QH8DHBEXAMPLE&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=ItemAttributes%2COffers%2CImages%2CReviews&Service=AWSECommerceService&Timestamp=2009-01-01T12%3A00%3A00Z&Version=2009-01-06"""
print b64encode(hmac.new(secret_key, to_sign, sha256).digest())
The instructions say that the signature using this request, and this key, is Nace+U3Az4OhN7tISqgs1vdLBHBEijWcBeCqL5xN9xg= but I get O6UTkH+m4zAQUvB+WXUZJeA8bZcKAdkc4crKgHtbc6s=
(Before anyone says anything: The example page displays the requests wrapped at 65 characters; I've already tried it. This doesn't provide a solution, and is not stated in the instructions for signature creation.)
EDIT: I found the answer, see below.
Well, look at that... The docs were wrong.
I stumbled on an old (nearly) duplicate of this question: Calculating a SHA hash with a string + secret key in python
It looks like the AWSAccessKeyId value changed from 00000000000000000000 to AKIAI44QH8DHBEXAMPLE in the example requests page.
Updating this in the script prints the expected key, Nace+U3Az4OhN7tISqgs1vdLBHBEijWcBeCqL5xN9xg=
import hmac
from base64 import b64encode
from hashlib import sha256
secret_key = '1234567890'
to_sign = """GET
webservices.amazon.com
/onca/xml
AWSAccessKeyId=00000000000000000000&ItemId=0679722769&Operation=ItemLookup&ResponseGroup=ItemAttributes%2COffers%2CImages%2CReviews&Service=AWSECommerceService&Timestamp=2009-01-01T12%3A00%3A00Z&Version=2009-01-06"""
print b64encode(hmac.new(secret_key, to_sign, sha256).digest())
You might check out the Bottlenose library, https://github.com/dlo/bottlenose, I have found that it makes dealing with AWS Product API much more friendly.
(This question is related - but not the same - to this one)
This is the POST data that I get from a github hook:
payload=%7B%22pusher%22%3A%7B%22name%22%3A%22none%22%7D%2C%22repository%22%3A%7B%22name%22%3A%22test%22%2C%22size%22%3A84%2C%22has_wiki%22%3Atrue%2C%22created_at%22%3A%222012%2F01%2F12%2001%3A04%3A25%20-0800%22%2C%22watchers%22%3A1%2C%22private%22%3Afalse%2C%22fork%22%3Afalse%2C%22url%22%3A%22https%3A%2F%2Fgithub.com%2Fgonvaled%2Ftest%22%2C%22pushed_at%22%3A%222012%2F01%2F12%2001%3A05%3A26%20-0800%22%2C%22has_downloads%22%3Atrue%2C%22open_issues%22%3A0%2C%22has_issues%22%3Atrue%2C%22homepage%22%3A%22%22%2C%22description%22%3A%22%22%2C%22forks%22%3A1%2C%22owner%22%3A%7B%22name%22%3A%22gonvaled%22%2C%22email%22%3A%22gonvaled%40gonvaled.com%22%7D%7D%2C%22forced%22%3Afalse%2C%22after%22%3A%2214209371dcbdd95cc3ef5c4a07d80edd42f1295c%22%2C%22deleted%22%3Afalse%2C%22ref%22%3A%22refs%2Fheads%2Fmaster%22%2C%22commits%22%3A%5B%5D%2C%22before%22%3A%2214209371dcbdd95cc3ef5c4a07d80edd42f1295c%22%2C%22compare%22%3A%22https%3A%2F%2Fgithub.com%2Fgonvaled%2Ftest%2Fcompare%2F1420937...1420937%22%2C%22created%22%3Afalse%7D
Which I can decode using this:
urllib.unquote(data)
Getting this:
payload={"pusher":{"name":"none"},"repository":{"name":"test","size":84,"has_wiki":true,"created_at":"2012/01/12 01:04:25 -0800","watchers":1,"private":false,"fork":false,"url":"https://github.com/gonvaled/test","pushed_at":"2012/01/12 01:05:26 -0800","has_downloads":true,"open_issues":0,"has_issues":true,"homepage":"","description":"","forks":1,"owner":{"name":"gonvaled","email":"gonvaled#gonvaled.com"}},"forced":false,"after":"14209371dcbdd95cc3ef5c4a07d80edd42f1295c","deleted":false,"ref":"refs/heads/master","commits":[],"before":"14209371dcbdd95cc3ef5c4a07d80edd42f1295c","compare":"https://github.com/gonvaled/test/compare/1420937...1420937","created":false}
I can see the JSON there, after the payload= bit. The question I have is: what format is the full data? How can I get just the payload bit, using standard python libraries; I would prefer to avoid splitting the string myself, since I do not know the special cases.
The github help page gives this suggested implementation for a Sinatra server:
post '/' do
push = JSON.parse(params[:payload])
"I got some JSON: #{push.inspect}"
end
How can this params array be handled in python, with standard libraries? What is the most pythonic implementation of that Ruby code? My end goal is to have the full POST data accessible as a python dictionary.
Try this:
import json
import urlparse
data = urlparse.parse_qs(r)
print json.loads(r['payload'][0])
where r is the string you received as response.
See http://docs.python.org/library/urlparse.html#urlparse.parse_qs
import urlparse
import json
s = "payload=%7B%22pusher%22%3A%7B%22name%22%3A%22none%22%7D%2C%22repository%22%3A%7B%22name%22%3A%22test%22%2C%22size%22%3A84%2C%22has_wiki%22%3Atrue%2C%22created_at%22%3A%222012%2F01%2F12%2001%3A04%3A25%20-0800%22%2C%22watchers%22%3A1%2C%22private%22%3Afalse%2C%22fork%22%3Afalse%2C%22url%22%3A%22https%3A%2F%2Fgithub.com%2Fgonvaled%2Ftest%22%2C%22pushed_at%22%3A%222012%2F01%2F12%2001%3A05%3A26%20-0800%22%2C%22has_downloads%22%3Atrue%2C%22open_issues%22%3A0%2C%22has_issues%22%3Atrue%2C%22homepage%22%3A%22%22%2C%22description%22%3A%22%22%2C%22forks%22%3A1%2C%22owner%22%3A%7B%22name%22%3A%22gonvaled%22%2C%22email%22%3A%22gonvaled%40gonvaled.com%22%7D%7D%2C%22forced%22%3Afalse%2C%22after%22%3A%2214209371dcbdd95cc3ef5c4a07d80edd42f1295c%22%2C%22deleted%22%3Afalse%2C%22ref%22%3A%22refs%2Fheads%2Fmaster%22%2C%22commits%22%3A%5B%5D%2C%22before%22%3A%2214209371dcbdd95cc3ef5c4a07d80edd42f1295c%22%2C%22compare%22%3A%22https%3A%2F%2Fgithub.com%2Fgonvaled%2Ftest%2Fcompare%2F1420937...1420937%22%2C%22created%22%3Afalse%7D"
L = urlparse.parse_qsl(s)
for k, v in L:
print k
print json.loads(v)
gives
payload
{u'forced': False, u'compare': u'https://github.com/gonvaled/...1420937', ...
u'before': u'14209371dcbdd95cc3ef5c4a07d80edd42f1295c'}