We Keep Coding

AWS S3 X-Amz-Expires with authorization headers is ignored

I use the following code in Python to generate authorization headers for getting an object in an S3 bucket (where AWS_SECRET_ACCESS_KEY, AWS_ACCESS_KEY_ID, AWS_DEFAULT_REGION, and DEFAULT_BUCKET are all set via environment variables in another piece of code)
def get_s3_headers(object_name):
'''Get authorization headers required to access a certain object in S3.'''
headers = _sig_v4_headers(pre_auth_headers={},
service='s3',
host=f'{DEFAULT_BUCKET}.s3.amazonaws.com',
method='GET',
path='/' + object_name,
query={ 'X-Amz-Expires' : '120' },
payload=b'')
return headers
def _sig_v4_headers(pre_auth_headers, service, host, method, path, query, payload):
# Define use of signature v4
algorithm = 'AWS4-HMAC-SHA256'
now = datetime.datetime.utcnow()
amzdate = now.strftime('%Y%m%dT%H%M%SZ')
datestamp = now.strftime('%Y%m%d')
payload_hash = hashlib.sha256(payload).hexdigest()
credential_scope = f'{datestamp}/{AWS_DEFAULT_REGION}/{service}/aws4_request'
pre_auth_headers_lower = {
header_key.lower(): ' '.join(header_value.split())
for header_key, header_value in pre_auth_headers.items()
}
required_headers = {
'host': host,
'x-amz-content-sha256': payload_hash,
'x-amz-date': amzdate,
}
headers = {**pre_auth_headers_lower, **required_headers}
header_keys = sorted(headers.keys())
signed_headers = ';'.join(header_keys)
def signature():
def canonical_request():
canonical_uri = urllib.parse.quote(path, safe='/~')
quoted_query = sorted(
(urllib.parse.quote(key, safe='~'), urllib.parse.quote(value, safe='~'))
for key, value in query.items()
)
canonical_querystring = '&'.join(f'{key}={value}' for key, value in quoted_query)
canonical_headers = ''.join(f'{key}:{headers[key]}\n' for key in header_keys)
print(canonical_querystring)
return f'{method}\n{canonical_uri}\n{canonical_querystring}\n' + \
f'{canonical_headers}\n{signed_headers}\n{payload_hash}'
def sign(key, msg):
return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()
string_to_sign = f'{algorithm}\n{amzdate}\n{credential_scope}\n' + \
hashlib.sha256(canonical_request().encode('utf-8')).hexdigest()
date_key = sign(('AWS4' + AWS_SECRET_ACCESS_KEY).encode('utf-8'), datestamp)
region_key = sign(date_key, AWS_DEFAULT_REGION)
service_key = sign(region_key, service)
request_key = sign(service_key, 'aws4_request')
return sign(request_key, string_to_sign).hex()
return {
'uri' : f'https://{host}{path}',
'headers' : {
**pre_auth_headers,
'x-amz-date': amzdate,
'x-amz-content-sha256': payload_hash,
'Authorization': f'{algorithm} Credential={AWS_ACCESS_KEY_ID}/{credential_scope}, '
f'SignedHeaders={signed_headers}, Signature=' + signature(),
}
}
Calling get_s3_headers with a valid object key returns something similar to the following:
"headers": {
"Authorization": "AWS4-HMAC-SHA256 Credential=AKIAYS3VM3EBIFL7FKE5/20220324/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=<string of characters>",
"x-amz-content-sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"x-amz-date": "20220324T193132Z"
},
"uri": "https://hotspot-storage.s3.amazonaws.com/posts/61e0dd8056716196cf357434"
Calling GET on https://hotspot-storage.s3.amazonaws.com/posts/61e0dd8056716196cf357434?X-Amz-Expires=120 returns the correct image with no expiration errors. Changing X-Amz-Expires to any other value returns a "Signature not valid" exception as expected. However, I can still use the link and the headers after 120 seconds, as if X-Amz-Expires doesn't actually do anything. I have zero clue why this is not working, so any help would be much appreciated.
EDIT
Changing X-Amz-Expires to 0 in both the query string and Python code still allows access to the link. The expiration time always defaults to 15 minutes.

Can't signing a Message using sha512 in python at iconomi

I am trying to send an authentificated message over an API at iconomi.com.
I am used to sign message when dealing with other exchange API, but can't be authentificated with this specific one.
I read the official documentation for authentification:
You generate the ICN-SIGN header by creating a sha512 HMAC using the
base64-decoded secret key on the prehash string timestamp + method +
requestPath + body (where + represents string concatenation) and
base64-encode the output, where:
the timestamp value is the same as the ICN-TIMESTAMP header. the body
is the request body string or omitted if there is no request body
(typically for GET requests). method must always be in upper case
Example: base64_encode(HMAC_SHA512(secret_key, timestamp + upper_case(method) + requestPath + body))
I found also a java client example on the official github, please see bellow signature generation in java :
private String generateServerDigest(String method, String uri, long timestamp, String body) {
//return timestamp + request.getMethodValue() + uri + body;
String checkDigestString = timestamp + method + uri + body;// "GET+/v1/daa-list+123123123"; //timestamp in epoch milliseconds
// hash server composited digest with algorithm and apikeys secret
SecretKeySpec signingKey = new SecretKeySpec(apiSecret.getBytes(), "HmacSHA512");
Mac mac;
try {
mac = Mac.getInstance(signingKey.getAlgorithm());
mac.init(signingKey);
} catch (NoSuchAlgorithmException | InvalidKeyException e) {
log.warn("Could not ={}", signingKey.getAlgorithm());
return null;
}
return Base64.getEncoder().encodeToString(mac.doFinal(checkDigestString.getBytes()));
}
Please note that checkDigestString code timestamp + method + uri + body and comment GET+/v1/daa-list+123123123 are already different on the official doc.
And this is my python implementation attempt :
def sign(timestamp,method,requestPath,body):
global api_secret
base64_decoded_secret_key = base64.b64decode(api_secret)
content_to_hash = (str(timestamp) + method.upper() + requestPath + body).encode('utf-8')
sign_digest = hmac.new(base64_decoded_secret_key, content_to_hash , hashlib.sha512).digest()
return base64.b64encode(sign_digest).decode('utf-8')
When I try this signature method with requestPath = "/v1/user/balance" (which required to be authentificated), it fail without error...
Does any one used with both java and python may help me to convert this signature method to python ?

This code will work for GET:
import time,requests
import hashlib,hmac,base64
api_key = "my api key"
api_secret = "my api secret"
defaut_encoding = "utf8"
uri = "https://api.iconomi.com"
requestPath = "/v1/user/balance"
api_url_target = uri+requestPath # https://api.iconomi.com/v1/user/balance
method="GET"
body=""
icn_timestamp = int(1000.*time.time())
message = (str(icn_timestamp) + method.upper() + requestPath + body).encode(defaut_encoding)
signature_digest = hmac.new(api_secret.encode(defaut_encoding), message, hashlib.sha512).digest() #here digest is byte
b64_signature_digest= base64.b64encode(signature_digest).decode(defaut_encoding)
headers_sign= {
"ICN-API-KEY":api_key,
"ICN-SIGN":b64_signature_digest,
"ICN-TIMESTAMP":str(icn_timestamp)
}
s=requests.session()
res = s.get(api_url_target,headers=headers_sign,timeout=3, verify=True).content
print (res)
Update for #Karl comment, this code will work for POST:
import time,requests
import hashlib,hmac,base64,json
api_key = "my api key"
api_secret = "my api secret"
ticker = "my ticker strategy"
defaut_encoding = "utf8"
uri = "https://api.iconomi.com"
requestPath = "/v1/strategies/"+ticker+"/structure"
api_url_target = uri+requestPath # https://api.iconomi.com/v1/strategies/{my ticker strategy}/structure
method="POST"
body="{'ticker': ticker, 'values': [{'rebalancedWeight': 1., 'targetWeight':1., 'assetTicker': 'XMR', 'assetName': 'Monero', 'assetCategory': 'Privacy'}]}"
icn_timestamp = int(1000.*time.time())
message = (str(icn_timestamp) + method.upper() + requestPath + body).encode(defaut_encoding)
signature_digest = hmac.new(api_secret.encode(defaut_encoding), message, hashlib.sha512).digest() #here digest is byte
b64_signature_digest= base64.b64encode(signature_digest).decode(defaut_encoding)
headers_sign= {
"ICN-API-KEY":api_key,
"ICN-SIGN":b64_signature_digest,
"ICN-TIMESTAMP":str(icn_timestamp)
}
s=requests.session()
res = s.post(api_url_target,headers=headers_sign,json = json.loads(body), timeout=3, verify=True).content
print (res)

REST request to App Store Connect API response with '401' | 'NOT_AUTHORIZED'

we launched an iOS-App and I want to grab some Information (e.g. Installations, Updates, Reviews) from the App Store Connect API.
I create an JSON Web Token as described in the official Apple documentation: Link
Afterwards I make a request with the token in the header. Now I get an '401' | 'NOT_AUTHORIZED' each time as an answer, see the following picture:
REST Response
In the following snippets you can see my python code (I tried to solve it in Python and R, but the result is always the same).
First, I create an JWT:
from datetime import datetime, timedelta
from jose import jwt, jws
import ecdsa
KEY_ID = "XXXXXXXXXX"
ISSUER_ID = "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"
PRIVATE_KEY = open('AuthKey_XXXXXXXXXX.p8', 'r').read()
TIMESTAMP = int( (datetime.now() - timedelta(minutes = 45)).timestamp() * 1000)
claim = {"iss" : ISSUER_ID,
"exp" : TIMESTAMP,
"aud" : "appstoreconnect-v1"}
header = {
"alg": "ES256",
"kid": KEY_ID,
"typ": "JWT"
}
# Create the JWT
encoded = jwt.encode(claim, PRIVATE_KEY, algorithm='ES256', headers=header)
Now when I print encoded, I get to following JWT (looks valid for me):
'eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NiIsImtpZCI6IlhYWFhYWFhYWFgifQ.eyJpc3MiOiJYWFhYWFhYWC1YWFhYLVhYWFgtWFhYWC1YWFhYWFhYWFhYWFgiLCJleHAiOjE1NDUzOTc1MTQ1ODAsImF1ZCI6ImFwcHN0b3JlY29ubmVjdC12MSJ9.eTl6iaAW-Gp67FNmITrWCpLTtJzVdLYXIl5_KKgqaNgzwyGo7npBOBo9_u5PtLNnssQFEwJWbPND-6Ww5ACgEg'
Even if I decode the first two parts of the JWT via Base64 I get the right Header (it also contains the right algorithm for encoding: 'alg': 'ES256') and Claim:
from jose.utils import base64url_decode
print(base64url_decode(b'eyJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NiIsImtpZCI6IlhYWFhYWFhYWFgifQ'))
print(base64url_decode(b'eyJpc3MiOiJYWFhYWFhYWC1YWFhYLVhYWFgtWFhYWC1YWFhYWFhYWFhYWFgiLCJleHAiOjE1NDUzOTc1MTQ1ODAsImF1ZCI6ImFwcHN0b3JlY29ubmVjdC12MSJ9'))
See the following picture: Output Base64 Decoding
So now, that I'm think that the JWT-Object is ready I send the request to the API:
import requests
JWT = 'Bearer ' + encoded
URL = 'https://api.appstoreconnect.apple.com/v1/apps'
HEAD = {'Authorization': JWT}
print(HEAD)
R = requests.get(URL, headers=HEAD)
R.json()
And now we can see my problem, see the picture: Header | REST Response
Please note that I have hidden the KEY_ID, ISSUER_ID and PRIVATE_KEY for the example.

Your token contains an expiry time
"exp": 1545397514580,
which equals September 12th, 50941.
When I delete the last three digits
"exp": 1545397514,
I get December 21st, 2018
which makes much more sense.
Change that line
TIMESTAMP = int( (datetime.now() - timedelta(minutes = 45)).timestamp() * 1000)
to
TIMESTAMP = int( (datetime.now() - timedelta(minutes = 45)).timestamp())
exp is a timestamp that is defined as seconds since 01.01.1970 00:00
See also here

First of all, please don't open files without context managers. This string:
PRIVATE_KEY = open('AuthKey_XXXXXXXXXX.p8', 'r').read()
should be:
with open('AuthKey_XXXXXXXXXX.p8', 'r') as f:
PRIVATE_KEY = f.read()
It save you from many problems with unclosed files in future.
Then, check what token you've read from file. Is it correct?
The next problem I see is timestamp. "The token's expiration time, in Unix epoch time;" You provide it in milliseconds, I guess.

Here is working solution for me. Without delay it return 401 error
KEY_ID = "xxxxx"
ISSUER_ID = "xxxxx"
EXPIRATION_TIME = int(round(time.time() + (20.0 * 60.0))) # 20 minutes timestamp
PATH_TO_KEY = '../credentials/AuthKey_xxxxx.p8'
with open(PATH_TO_KEY, 'r') as f:
PRIVATE_KEY = f.read()
header = {
"alg": "ES256",
"kid": KEY_ID,
"typ": "JWT"
}
payload = {
"iss": ISSUER_ID,
"exp": EXPIRATION_TIME,
"aud": "appstoreconnect-v1"
}
# Create the JWT
token = jwt.encode(header, payload, PRIVATE_KEY)
JWT = 'Bearer ' + token.decode()
HEAD = {'Authorization': JWT}
# Without delay I got 401 error
time.sleep(5)
URL = 'https://api.appstoreconnect.apple.com/v1/apps';
r = requests.get(URL, params={'limit': 200}, headers=HEAD)

Aws lambda function failing - Python

This is a very weird problem that I have got stuck into, will really appreciate if someone could provide some direction. I am trying to access value for request_url from web_token.py module.
when I only try to run web_token.py separately over pycharm and print request_url it works fine and generates the url. I zip both these files and upload it to lambda function but when testing it I get an error "Unable to import module 'retrieve_accounts': No module named boto.sts". I even tried putting the code of web_token.py inside retrieve_accounts.py but getting the same error. I am sure I am missing something very basic it looks boto.sts is not being recognized while running the python script. Can somebody please give some guidance. Thank You!
retrieve_accounts.py
import boto3
import web_token
def get_account(event, context):
client = boto3.client('dynamodb')
NameID = "testname#org.com"
ManagerEmail = "test1#eorg.com"
response = client.scan(
TableName='Sandbox-Users',
ScanFilter={
'NameID': {
'AttributeValueList': [
{
'S': NameID,
},
],
'ComparisonOperator': 'EQ'
}
}
)
if response["Count"] > 0:
client = boto3.client('dynamodb')
response = client.get_item(
Key={
'NameID': {
'S': NameID,
},
'ManagerEmail': {
'S': ManagerEmail,
},
},
TableName='Sandbox-Users',
)
return web_token.request_url ----------->here
else:
response = client.put_item(
Item={
'NameID': {
'S': NameID,
},
'ManagerEmail': {
'S': ManagerEmail,
}
},
TableName='Sandbox-Users'
)
return "Create Account"
web_token.py
import httplib
import urllib, json
from boto.sts import STSConnection -------->Error here
sts_connection = STSConnection()
assumed_role_object = sts_connection.assume_role(
role_arn="arn:aws:iam::454084028794:role/AMPSandboxRole",
role_session_name="AssumeRoleSession"
)
# Step 3: Format resulting temporary credentials into JSON
json_string_with_temp_credentials = '{'
json_string_with_temp_credentials += '"sessionId":"' +
assumed_role_object.credentials.access_key + '",'
json_string_with_temp_credentials += '"sessionKey":"' +
assumed_role_object.credentials.secret_key + '",'
json_string_with_temp_credentials += '"sessionToken":"' +
assumed_role_object.credentials.session_token + '"'
json_string_with_temp_credentials += '}'
# Step 4. Make request to AWS federation endpoint to get sign-in token.
Construct the parameter string with the sign-in action request, a 12-hour session duration, and the JSON
document with temporary credentials as parameters.
request_parameters = "?Action=getSigninToken"
request_parameters += "&SessionDuration=43200"
request_parameters += "&Session=" +
urllib.quote_plus(json_string_with_temp_credentials)
request_url = "/federation" + request_parameters
conn = httplib.HTTPSConnection("signin.aws.amazon.com")
conn.request("GET", request_url)
r = conn.getresponse()
# Returns a JSON document with a single element named SigninToken.
signin_token = json.loads(r.read())
request_parameters = "?Action=login"
request_parameters += "&Issuer=sandbox.com"
request_parameters += "&Destination=" +
urllib.quote_plus("https://console.aws.amazon.com/")
request_parameters += "&SigninToken=" + signin_token["SigninToken"]
request_url = "https://signin.aws.amazon.com/federation" +
request_parameters

AWS Lambda Python environments include boto3 (and botocore). They don't include the older boto (a precursor to boto3), hence the import failure.
You could potentially include boto in your upload but it's not advisable to mix boto and boto3 if you can avoid it. Use one or the other, preferably boto3.

Unable to get OAuth "Request Token" while working with the Tumblr API using Python

I've been using libraries to handle OAuth so far, but lately I've been digging deeper trying to understand the underlying OAuth process. Currently, I'm trying to connect to Tumblr API v2 using OAuth 1.0a with this simple code:
import urllib, urllib2, time, random, hmac, base64, hashlib
def makenonce():
random_number = ''.join( str( random.randint( 0, 9 ) ) for _ in range( 40 ) )
m = hashlib.md5( str( time.time() ) + str( random_number ) )
return m.hexdigest()
def encodeparams(s):
return urllib.quote( str( s ), safe='~' )
# Actual key and secret from a test app created using a dummy Tumblr account
consumer_key = '97oAujQhSaQNv4XDXzCjdZlOxwNyhobmDwmueJBCHWsFFsW7Ly'
consumer_secret = '5q1dpF659SOgSUb0Eo52aAyoud8N8QOuJu6enCG92aDR6WoMlf'
#oauth URLs
request_tokenURL = 'http://www.tumblr.com/oauth/request_token'
#oauth params
oauth_parameters = {
'oauth_consumer_key' : consumer_key,
'oauth_nonce' : makenonce(),
'oauth_timestamp' : str(int(time.time())),
'oauth_signature_method' : "HMAC-SHA1",
'oauth_version' : "1.0"
}
normalized_parameters = encodeparams( '&'.join( ['%s=%s' % ( encodeparams( str( k ) ), encodeparams( str( oauth_parameters[k] ) ) ) for k in sorted( oauth_parameters )] ) )
# Since I'm focusing only on getting the request token for now, I set this to POST.
normalized_http_method = 'POST'
normalized_http_url = encodeparams( request_tokenURL )
signature_base_string = '&'.join( [normalized_http_method, normalized_http_url, normalized_parameters] )
oauth_key = consumer_secret + '&'
hashed = hmac.new( oauth_key, signature_base_string, hashlib.sha1 )
oauth_parameters['oauth_signature'] = base64.b64encode( hashed.digest() )
oauth_header = 'Authorization: OAuth realm="http://www.tumblr.com",' + 'oauth_nonce="' + oauth_parameters['oauth_nonce'] + '",' + 'oauth_timestamp="' + oauth_parameters['oauth_timestamp'] + '",' + 'oauth_consumer_key="' + oauth_parameters['oauth_consumer_key'] + '",' + 'oauth_signature_method="HMAC-SHA1",oauth_version="1.0",oauth_signature="' + oauth_parameters['oauth_signature'] +'"'
# sample oauth_header generated by the code above:
# Authorization: OAuth realm="http://www.tumblr.com",oauth_nonce="c200a0e06f30b84b851ac3e99a71054b",oauth_timestamp="1315231855",oauth_consumer_key="97oAujQhSaQNv4XDXzCjdZlOxwNyhobmDwmueJBCHWsFFsW7Ly",oauth_signature_method="HMAC-SHA1",oauth_version="1.0",oauth_signature="kVAlmwolCX0WJIvTF9MB2UV5rnU="
req = urllib2.Request( request_tokenURL )
req.add_header( 'Authorization', oauth_header )
# If all goes well, Tumblr should send me the oauth request token.
print urllib2.urlopen( req ).read()
Instead of the OAuth Request token, Tumblr returns HTTP Error 401: Unauthorized.
Things I've tried without any success:
Changed oauth_version from "1.0" to "1.0a", and changed it back again.
A guide on OAuth mandated adding the '&' at the end of consumer_secret to get the oauth_key. I tried removing the '&' later to see if that made any difference.
Checked if the OAuth parameters were sorted, and they were.
Did not add the string "Authorization: " to oauth_header, then added it back later. Neither made any difference.
Where have I gone wrong?

Solved it after making just 2 simple changes in the above code:
normalized_http_method = 'GET' #not POST
oauth_header = 'OAuth realm="http://www...' # The word "Authorization" is unnecessary. I had taken this out earlier as listed in "Things I've tried without any success", but the error listed in (1) threw me off track. With (1) solved, I could see how "Authorization" was indeed unnecessary.
The OAuth Request token Tumblr sent me when I finally got it right:
oauth_token=mbRUgyDkPePfkEztiLELMqUl1kyNXEcaTCCwpb7SoXDF9mhiTF&oauth_token_secret=5pXllXGKA8orAaUat1G7ckIfMfYup8juMBAgEELUkeMZoC3pv6&oauth_callback_confirmed=true
↑
This is a one-time only token and I've listed it here just for the sake of completeness.