We Keep Coding

How to authenticate Azure REST APIs to Service Bus with SAS

I'm trying to send a message to the Service Bus queue using the REST API provided in this document:
https://learn.microsoft.com/en-us/rest/api/servicebus/send-message-to-queue
Do note that I cannot use the Azure Libraries for this task as there are non available that I know of for Service Now and I'm setting up the test trigger in Python to simulate the REST API calls that would be made from Service Now.
I had a similar question with regards to storage queues and I've tried reusing the same solution however the Service Bus would respond with "Missing Authorization Header"
This is my code that uses the Authorization in the header:
import requests
api = f"https://{service_namespace}.servicebus.windows.net/{queue}/messages?"
msg = """<QueueMessage>
<MessageText>Testing 1234</MessageText>
</QueueMessage>
"""
header = {
"Authorization": f"SharedAccessSignature sr=https://{service_namespace}.servicebus.windows.net/{queue}&sig={sig}&se={se}&skn={skn}",
"Content-Type": "application/atom+xml;type=entry;charset=utf-8"
}
resp = requests.post(api, data=msg, headers=header)
print(resp)
print(resp.text)
print(resp.headers)
Here, sig is the primary key I got from the Service Bus's Queue under Shared Access Policy
se is the epoch time 2 years from now (w/o mili seconds)
skn is the name of the policy
The final response I get is
<Response [401]>
{'Content-Length': '0', 'Server': 'Microsoft-HTTPAPI/2.0', 'Strict-Transport-Security': 'max-age=31536000', 'Date': 'Thu, 24 Feb 2022 09:27:17 GMT'}
If I post without the Auth in the header and use the solution in the highlighted question above,
This is the API structure BTW:
f"https://{service_namespace}.servicebus.windows.net/{queue}/messages?sig={sig}&se={se}&skn={skn}"
I get this error:
<Error><Code>401</Code><Detail>MissingToken: The authorization header was not found. To know more visit https://aka.ms/sbResourceMgrExceptions. . TrackingId:<redacted>, SystemTracker:<redacted>.servicebus.windows.net:<redacted>/messages, Timestamp:2022-02-24T09:31:09</Detail></Error>
{'Transfer-Encoding': 'chunked', 'Content-Type': 'application/xml; charset=utf-8', 'Server': 'Microsoft-HTTPAPI/2.0', 'Strict-Transport-Security': 'max-age=31536000', 'Date': 'Thu, 24 Feb 2022 09:31:09 GMT'}
I am not sure how to proceed with this, any tips and suggestion would be much appreciated.

The reason you are getting this error is because you are computing the shared access signature incorrectly. You can learn more about it here.
To generate SAS token using python, please see the code below which is taken from here:
import time
import urllib
import hmac
import hashlib
import base64
def get_auth_token(sb_name, eh_name, sas_name, sas_value):
"""
Returns an authorization token dictionary
for making calls to Event Hubs REST API.
"""
uri = urllib.parse.quote_plus("https://{}.servicebus.windows.net/{}" \
.format(sb_name, eh_name))
sas = sas_value.encode('utf-8')
expiry = str(int(time.time() + 10000))
string_to_sign = (uri + '\n' + expiry).encode('utf-8')
signed_hmac_sha256 = hmac.HMAC(sas, string_to_sign, hashlib.sha256)
signature = urllib.parse.quote(base64.b64encode(signed_hmac_sha256.digest()))
return {"sb_name": sb_name,
"eh_name": eh_name,
"token":'SharedAccessSignature sr={}&sig={}&se={}&skn={}' \
.format(uri, signature, expiry, sas_name)
}

Calling AWS APIGateway with a Cognito user pool

I have a REST API configured on my AWS account using AWS APIGateway. The REST API uses a Cognito Authorizer for authentication.
I am using a Python Script to interact with the API. In this Python script, I managed to authenticate a user. For the sake of informatio, here is how I do it handling the case where the user needs to update its password (first authentication) :
region = 'eu-west-1'
cognito = boto3.client('cognito-idp', region)
clientId = 'abcdefghijk27261819'
username = "myUsername"
password = "myPassword"
response = cognito.initiate_auth(AuthFlow='USER_PASSWORD_AUTH', AuthParameters={"USERNAME": username,
"PASSWORD": password}, ClientId = clientId)
try :
response['ChallengeName']
response = cognito.respond_to_auth_challenge(ClientId=clientId, ChallengeName=response['ChallengeName'], Session=response['Session'],
ChallengeResponses={"USERNAME": username,
"NEW_PASSWORD": "myNewPassword"})
except KeyError:
pass
# store credentials
accessToken = response['AuthenticationResult']['AccessToken']
idToken = response['AuthenticationResult']['IdToken']
I am able to verify that the identity token idToken is the one expected by the API through the APIGateway console :
However, even though the token seems to be ok, I can't call the API.
I tried two calls :
HOST = "https://<APIID>.execute-api.eu-west-1.amazonaws.com"
STAGE = "dev"
BASE_URL = f"{HOST}/{STAGE}"
r = requests.post(f"{BASE_URL}/S3Operations/CreateSimulationBucket", headers={"Authorization": idToken}) # the call I want
r = requests.post(f"{BASE_URL}", headers={"Authorization": idToken}) # the test call
and both calls return an error :
# first call
{'Content-Type': 'application/json', 'Content-Length': '23', 'Connection': 'keep-alive', 'Date': 'Tue, 22 Jun 2021 07:36:47 GMT', 'x-amzn-RequestId': 'd87ee0c8-d004-4199-8974-1727830c9f25', 'x-amzn-ErrorType': 'ForbiddenException', 'x-amz-apigw-id': 'BUPqaFQcDoEF3Xg=', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 5e828cc6ff056cb59ec35c3467ec45f5.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'AMS1-C1', 'X-Amz-Cf-Id': '6sbLUxCEmUWQYTWD6BHAommN6eTVieLmL-6XOJh30fNW-IhdcztJIQ=='}
# second call
{'Content-Type': 'application/json', 'Content-Length': '1379', 'Connection': 'keep-alive', 'Date': 'Tue, 22 Jun 2021 07:45:19 GMT', 'x-amzn-RequestId': 'b69a442e-3b3c-4447-bec8-8eff344ac7cb', 'x-amzn-ErrorType': 'IncompleteSignatureException', 'x-amz-apigw-id': 'BUQ6bFLqjoEFjhA=', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 043fc2faaa02eeb59193e3fa300adb6b.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'AMS1-C1', 'X-Amz-Cf-Id': '8btFflDUE2I7LxiKbvnmreOE-oKrIwbudgORbWaRrASNUmIPdYtwYQ=='}
My questions are :
Why the first call return a ForbiddenException ? I believe my paths are correctly configured :
Why the second call returns a IncompleteSignatureException ?
PROGRESS
For question 1. Actually, I could find information on AWS documentation page here and was able to understand I simply called an unauthorized method.
For question 2. I was able to diagnose the issue more precisely through the use of API Gateway responses. As shown below, I crafted custom messages for all possible issues
And I was able to find out that the issue was a non-provided API Key. Indeed, my API has a UsagePlan and therefore requires an API Key.
I have modified my script to get this API Key as shown below
session = boto3.Session(profile_name='my-profile')
region = 'eu-west-1' # or 'us-west-1' or whatever
cognito = session.client('cognito-idp', region)
apigw = session.client('apigateway', region)
# Cognito information
cognito_userpool_client_id = 'abcdefghijk27261819'
cognito_username = 'my_username'
cognito_password = 'my_password|'
# API information
apigw_id = 'my_api_id'
def get_api_key(apigw_client, apigw_id):
return apigw_client.get_api_key(apiKey=apigw_id, includeValue=True)['value']
response = cognito.initiate_auth(AuthFlow='USER_PASSWORD_AUTH', AuthParameters={"USERNAME": cognito_username,
"PASSWORD": cognito_password}, ClientId = cognito_userpool_client_id)
# store credentials
accessToken = response['AuthenticationResult']['AccessToken']
idToken = response['AuthenticationResult']['IdToken']
# Call API
HOST = "https://<APIID>.execute-api.eu-west-1.amazonaws.com"
STAGE = "dev"
BASE_URL = f"{HOST}/{STAGE}"
r = requests.post(f"{BASE_URL}/S3Operations/CreateSimulationBucket", headers={"Authorization": idToken, "x-api-key": get_api_key(apigw, apigw_id)})
despite these modifications I still receive the same error :
{'Content-Type': 'application/json', 'Content-Length': '23', 'Connection': 'keep-alive', 'Date': 'Tue, 22 Jun 2021 15:24:27 GMT', 'x-amzn-RequestId': '29743683-dbaf-4d60-bbae-8f2581e1c45a', 'x-amzn-ErrorType': 'ForbiddenException', 'x-amz-apigw-id': 'BVUKxF9FjoEFgbA=', 'issue': 'Invalid API Key', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 bdba42cf1410fb617eeb4ffd3e0b9cb7.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'AMS1-C1', 'X-Amz-Cf-Id': 'fMbbQSLlDd7OHErlPCoVNJRZM_P9mq5DYxgsfkJVy4sKHw9lijFYeA=='}

have you deployed the api in api gateway? Have you tried constructing the api call in postman before you do it in python?

Ok. After some analysis, I was able to understand the issue.
For question 1. I could find information on AWS documentation page here and was able to understand I simply called an unauthorized method i.e. calling the root of the API was not authorized because the root had no registered method/ressource.
For question 2. It took me some time to get to the heart of the problem. Indeed, by crafting custom gateway responses, I knew the problem was coming from an Invalid API Key. But this was in fact confusing because the real problem came from an invalid Authorizer !
Prior to solving the issue, here was my SAM template I was using for generating the resources :
API:
Type: AWS::Serverless::Api
Properties:
StageName: !Sub ${Env}
# Authentication on the API will be performed via a Key
Auth:
Authorizers:
CognitoAuthorizer:
UserPoolArn: !GetAtt CognitoUserPool.Arn
ApiKeyRequired: true
# this creates a key, a usage plan and associate them
UsagePlan:
CreateUsagePlan: PER_API
UsagePlanName: !Sub ${Project}-${Env}-UsagePlan
So ... I define a Cognito Authorizer ... But I am not actually using it. I was able to identify that in the AWS API Gateway console. Going into the resource page I arrive to the path I want to call and I click on the Method box :
And I arrive at the page shown below. Prior to the correction the Authorization field was None and that was causing my issue
Indeed ... I was making an API call with an Authorization header while there was no Authorization set on the method itself. Now I find it confusing that AWS does not send a more detailed error message and only send a Invalid API key. It took me a while to understand what was causing the issue because of that because I assumed there was an Authorizer on the method.
In order to solve the issue, I had to modify my SAM template as shown below and this modification allowed to set the Cognito Authorizer by default for all methods of the API and allowed me to make the call successfully :
API:
Type: AWS::Serverless::Api
Properties:
StageName: !Sub ${Env}
# Authentication on the API will be performed via a Key
Auth:
Authorizers:
CognitoAuthorizer:
UserPoolArn: !GetAtt CognitoUserPool.Arn
DefaultAuthorizer : CognitoAuthorizer
ApiKeyRequired: true
# this creates a key, a usage plan and associate them
UsagePlan:
CreateUsagePlan: PER_API
UsagePlanName: !Sub ${Project}-${Env}-UsagePlan
Notice the addition of the record DefaultAuthorizer : CognitoAuthorizer
Hope this will help others !

Trigger DAG from google cloud function

I have been trying to execute a DAG in Cloud Composer (composer-1.8.1-airflow-1.10.3) when a file arrives at a GCS bucket based on the cloud composer documentation with Cloud Function in Python
I haven't been able to make it work, every time it triggers, the function returns this message to the log
Exception: Bad response from application: 400 / {'Date': 'Mon, 30 Dec 2019 18:07:02 GMT', 'Content-Type': 'text/html', 'Content-Length': '192', 'Server': 'gunicorn/19.9.0', 'Via': '1.1 google', 'Alt-Svc': 'quic=":443"; ma=2592000; v="46,43",h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000'} / '<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>400 Bad Request</title>\n<h1>Bad Request</h1>\n<p>The browser (or proxy) sent a request that this server could not understand.</p>\n'
The request URL s¡generate and the open id token seems correct
The error is raised in the last part of the code
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
google_open_id_connect_token)}, **kwargs)
if resp.status_code == 403:
raise Exception('Service account {} does not have permission to '
'access the IAP-protected application.'.format(
signer_email))
elif resp.status_code != 200:
raise Exception(
'Bad response from application: {!r} / {!r} / {!r}'.format(
resp.status_code, resp.headers, resp.text))
Thanks

The error code 400 Bad Request reflected in the initial question more often corresponds to the client side problem, indeed the input parameters for DAG triggering should be inspected as a first, the function that invokes IAP call in the code example:
def make_iap_request(url, client_id, method='GET', **kwargs)
Ensure, that you have applied dependencies required to run against IAP protected URL via requirements.txt file.
Do not forget to verify the essential arguments that you pass to this function:
url: The Identity-Aware Proxy-protected URL to fetch.
client_id: The client ID used by Identity-Aware Proxy.
So far I haven't found even better hand on experience about triggering the DAG then explained by #Jacob Ferriero in his great article.

How to use OAuth Applciation token generated by Docusign Python client [ApiClient.request_jwt_application_token()]?

I am writing a back end rest application using python that creates envelopes from templates and returns the front end application the url to the created envelope to be displayed.
Using the Python client, I am able to create a valid OAuth token using the following lines:
api_client = ApiClient("https://demo.docusign.net/restapi")
resp = api_client.request_jwt_application_token(
client_id=client_id, # the integrator key
oauth_host_name=auth_server, # 'account-d.docusign.com'
private_key_bytes=private_key_file.read(), # private key temp file containing key in bytes
expires_in=expiration # 3600
)
The call is successful, and returns the following:
resp
{
'access_token': 'eyJ0eXAi...81z9D5w',
'data': None,
'expires_in': '28800',
'refresh_token': None,
'scope': None,
'token_type': 'Application'
}
And the proper Authorization headers are set in the client:
api_client.default_headers
{
'X-DocuSign-SDK': 'Python',
'User-Agent': 'Swagger-Codegen/1.0.0/python',
'Authorization': 'Application eyJ0eXAi...81z9D5w'
}
However, whenever I try to make any call, I get the following response:
envelope_api = EnvelopeApi(api_client)
envelope_api.get_envelope(account_id, envelope_id)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
...
docusign_esign.client.api_exception.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-cache', 'Content-Length': '120', 'Content-Type': 'application/json; charset=utf-8', 'X-DocuSign-TraceToken': '14ac...388', 'Date': 'Fri, 20 Dec 2019 17:47:43 GMT', 'Vary': 'Accept-Encoding'})
HTTP response body: b'{"errorCode":"USER_AUTHENTICATION_FAILED","message":"One or both of Username and Password are invalid. invalid_request"}'
Is there an additional step to utilize this Application token? The Python JWT example is severely out of date, and the Python client repository does not mention any required changes to use the new call
Commit introducing new application token functionality

You're generating an application token which is only useful in specific circumstances like with use with Datafeeds. In general, you want to be generating a user token, which will include a UserId as one of the parameters.
To do so, you need the api_client.configure_jwt_authorization_flow method. An example of how to use that is available on GitHub: https://github.com/docusign/eg-01-python-jwt/blob/master/example_base.py

Get status code for request sent using pysimplesoap

I'm using Pysimplesoap for sending a data to the web-service. I want to know the status of the request code. I'm able to print a traceback using trace=True. Over there it does prints the status code and other response variables but how do I get the to store all the traceback into a variable and then check the status code for the same?
Here's my code :_
client = SoapClient(
location = url,
action = 'http://tempuri.org/IService_1_0/',
namespace = "http://tempuri.org/",
soap_ns='soap', ns = False,trace = True
)
data = {'AWB_Number' : '406438762211111', 'Weight':'0.023' ,'Length':'16.4','Height':'4.5','Width':'9.9'}
response= client.UpdateShipment(
ShipmentNumber = data['AWB_Number'],
Weight = Decimal(data['Weight']),
Length = Decimal(data['Length']),
Height = Decimal(data['Height']),
Width = Decimal(data['Width']) ,
InboundLane = "2",
SequenceNumber = "1",
)
I do get a traceback :-
Content-length: 526
Content-type: text/xml; charset="UTF-8"
<?xml version="1.0" encoding="UTF-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soap:Header/>
<soap:Body>
<UpdateShipment xmlns="http://tempuri.org/">
<SequenceNumber>1</SequenceNumber><Weight>0.023</Weight><Height>4.5</Height><Width>9.9</Width><Length>16.4</Length><ShipmentNumber>406438762211111</ShipmentNumber><InboundLane>2</InboundLane></UpdateShipment>
</soap:Body>
</soap:Envelope>
status: 200
content-length: 293
x-powered-by: ASP.NET
server: Microsoft-IIS/7.5
date: Sat, 23 Aug 2014 07:27:38 GMT
content-type: text/xml; charset=utf-8
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><UpdateShipmentResponse xmlns="http://tempuri.org/"><UpdateShipmentResult/></UpdateShipmentResponse></s:Body></s:Envelope>
===============================================================================
There's a status code mention there as 200, but I dont get to store this traceback into a variable to get to know the status code of it. A human need to intervene to have a look at the status code. How do my program gets to know the status code?

The SoapClient instance retains a .response attribute, containing information about the response. What that information is depends on the transport picked.
If you have just PySimpleSoap installed, the urllib2 library is picked and the status code is not part of the client.response attribute; the information is not retained from the actual response from urllib2, only the HTTP headers are preserved.
The pycurl transport gives you even less info; client.response is always an empty dictionary then.
Only if you also installed httplib2 will you get anything useful out of this; client.response is then set to a dictionary that includes a status code:
>>> import pysimplesoap
>>> client = pysimplesoap.client.SoapClient(wsdl='http://www.webservicex.net/stockquote.asmx?WSDL')
>>> response = client.GetQuote('GOOG')
>>> client.response
{'status': '200', 'content-length': '991', 'x-aspnet-version': '4.0.30319', 'vary': 'Accept-Encoding', 'server': 'Microsoft-IIS/7.0', '-content-encoding': 'gzip', 'cache-control': 'private, max-age=0', 'date': 'Sat, 23 Aug 2014 08:05:19 GMT', 'x-powered-by': 'ASP.NET', 'content-type': 'text/xml; charset=utf-8'}
>>> client.response['status']
'200'
Note that the value is a string, not an integer.
The httplib2 transport is picked by default if available.
As for the trace option; that sets up a logging module log, calls logging.basicConfig() with a log level and gets on with it. You could add a custom handler to the pysimplesoap.client.log object, but I wouldn't bother, really. If you see the status logged to the trace, then in all likelihood you are already using httplib2 and can access the status code more directly. For urllib2, for example, no status is logged either. That's because it is the client.response.items() values that are being logged here.

From the pysimplesoap source code, at line 261 of client, the send method is written which is used to send the http request from client's end
Taking cue from it,
def send(self, method, xml):
...
response, content = self.http.request(
location, 'POST', body=xml, headers=headers)
self.response = response
self.content = content
...
you should be able to access client.response to get the raw response for the request. From there, try doing a client.response.status, or do a dir(client.response) to find out any supportive method to get status code.
EDIT
As mentioned here in transport module, you can specifiy a library (like urllib2 or httplib2). to make it the default one to be picked up.