We Keep Coding

SQS to AWS Lambda Function with AWS Chalice and BOTO3

I am using AWS SQS to store information coming in from an external server and then sending it to a Lambda function to process it and dequeue the information.
The information that I am sending in is in the form of a JSON and is being used as a python dictionary.
def lambda_handler(event, context):
for record in event['Records']:
messageHandler(record)
return {
'statusCode': 200,
'body': json.dumps('Batch Processed')
}
Assuming that the code for the messageHandler is working and properly implemented, how do I catch the messages from the queue in their batches. This is all being deployed by AWS Chalice without the use of CLI.
I am well out of my depth right now and have no idea why this is not working when I deploy it but is working when I trigger a normal Lambda Function in the AWS Console through the SQS Send/Recieve Message feature. As far as I know the triggers are set up correctly and they should have no issue.
If you have any questions please let me know.

The event that you are processing will look something like this:
{
"Records": [
{
"messageId": "11d6ee51-4cc7-4302-9e22-7cd8afdaadf5",
"receiptHandle": "AQEBBX8nesZEXmkhsmZeyIE8iQAMig7qw...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1573251510774",
"SequenceNumber": "18849496460467696128",
"MessageGroupId": "1",
"SenderId": "AIDAIO23YVJENQZJOL4VO",
"MessageDeduplicationId": "1",
"ApproximateFirstReceiveTimestamp": "1573251510774"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:fifo.fifo",
"awsRegion": "us-east-2"
}
]
}
where the "body" is your json encoded message. You'll want your message handler function to do something like this:
def message_handler(event):
message = json.loads(event["body"])
# do stuff...
The return value from the lambda is pretty pointless if it is being used as the event target from sqs.

How to retrieve a URL query string parameter from inside an AWS Lambda Python function?

How do you access URL querystring parameters from inside an AWS Lambda function served though an API Gateway?
I have both the API gateway + Lambda function setup so I can call it from a public URL. My Python function is simply:
def lambda_handler(event, context):
print('event:', event)
print('context:', context)
I've configured the API's GET "Method Request" handler to pass through the "abc" querystring parameter.
I've also configured the API's GET "Integration Request" handler to map "abc" from "method.request.querystring.abc".
However, when I access my URL, e.g. https://myapp.execute-api.us-east-1.amazonaws.com/prod/myfunc?abc=123, the only thing logged is:
event: {}
context: <bootstrap.LambdaContext object at 0x7fc7a6cb0850>
What am I doing wrong? Why isn't "abc" being passed through in the event dictionary?

Check Use Lambda Proxy integration in the Integration Request to have it pass all request details in the event.

I have a similar problem and I know how frustrating it is. Use this mapping template:
{
"method": "$context.httpMethod",
"body" : $input.json('$'),
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
},
"queryStringParameters": {
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
},
"pathParameters": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
}
}
Then you should find your event looks like this:
{
"method":"GET",
"body":{
},
"headers":{
},
"queryParams":{
"id":"459463732",
"command":"join_session"
},
"pathParams":{
}
}
The context is used for other information, such as IP addresses and timeout settings.

Send a request with a body to AWS Lambda

I have uploaded an AWS Lambda function where the lambda_handler looks as follows:
import json
def lambda_handler(event, context):
print(event)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!'),
'event': event
}
Problem 1: returning event
When I test it using the Lambda Management Console I can create a test event with parameters which also return the exact same format and all works fine:
{
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
However, when I use Postman, then I get something totally else, which returns to me:
{
"message": "Internal server error"
}
I suspect its because the event looks something more like:
{'resource': '/hello', 'path': '/hello', 'httpMethod': 'GET', 'headers': {'Accept': '*/*', ... etc
Problem 2: adding json parameters in body creates an error
When I try in Postman to add in the body > raw > JSON(application/JSON) the keys above, then I get the error:
ERROR: The request could not be satisfied
Questions
I have two questions:
How do I pass parameters in the body and be able to capture it in AWS lambda using the event or context?
How do I return the event properly?

Assuming you have set up your Lambda as proxy integration in AWS API Gateway. If you want to attach query string params and no body then your method type should be GET.
The event you would receive in your Lambda for request /GET your-path/?myKey1=value1&myKey2=value2 should something like:
{
"resource": "",
"path": "/your-path",
"httpMethod": "GET",
"headers": {
},
"queryStringParameters": {
"myKey1": "value1",
"myKey2": "value2"
},
"pathParameters": {
},
"body": "{}"
}
You can access query string params in queryStringParameters property.
If you send request using Postman and attach body then your Lambda integration type should be POST/PUT. Data that you add in Postman request body would be available in event["body"].
Last thing if you test Lambda directly in the console then event will be received as you put in the body. You will need to format your event according to the integration method type. If it is POST/PUT then:
{
"body": {
"someValue": {..}
}
}
If it is GET then:
{
"queryStringParameters": {
"myKey1": "value1",
"myKey2": "value2"
}
}

Figrued it out, after help from #Althar Khan.
Apparently the API Gateway of AWS Lambda only accepts certain properties:
...
return {
"statusCode": 200,
"headers": {
"my_header": "my_value"
},
"body": JSON.stringify(responseBody),
"isBase64Encoded": false
};
In this response, there are four fields: statusCode, headers, body, and isBase64Encoded.
In this example, the function's response is in the format that the API Gateway expects. For additional information, see Output Format of a Lambda Function for Proxy Integration.

serverless invoke local with mocked SNS event

I'm having some trouble invoking locally using serverless.
Using Python3.6 runtime
The function in serverless.yml is:
functions:
myFunction:
events:
- sns: arn:aws:sns:us-east-1:123456789:myTopic
My command is
sls invoke local -f myFunction -s dev -r us-east-1 -p events/myMockedSnsMessage.json
Serverless doesn't like the syntax on the myMockedSnsMessage.json
When I log the event in my lambda function I get something like:
{
'Records': [
{
'Sns': {
'Message': '{"version":"0","id":"a965ce94-fcb2-ad15-319d-04adab1072d0","detail-type":"AWS API Call via CloudTrail",...}'
}
}
]
}
That is, the SNS message is a string with valid JSON inside
How should I store a mock event for an SNS message and still have valid JSON so serverless doesn't yell at me for bad JSON syntax?

Perhaps use double-quotes, and stringify the json. e.g.
{
"Records": [
{
"Sns": {
"Message": "{\"version\":\"0\",\"id\":\"a965ce94-fcb2-ad15-319d-04adab1072d0\",\"detail-type\":\"AWS API Call via CloudTrail\",...}"
}
}
]
}
If you need more examples, the AWS documentation has sample events that you can modify, and when you use AWS console to test your lambda, you can select many different event templates from a dropdown.

Azure Function - Python - ServiceBus Output Binding - Setting Custom Properties

I have an Azure Function written in Python that has an Service Bus (Topic) output binding. The function is triggered by another queue, we process some files from a blobl storage and then put another message in a queue.
My function.json file looks like that:
{
"bindings": [
{
"type": "serviceBus",
"connection": "Omnibus_Input_Send_Servicebus",
"name": "outputMessage",
"queueName": "validation-output-queue",
"accessRights": "send",
"direction": "out"
}
],
"disabled": false
}
In my function, I can send a message to another queue like that:
with open(os.environ['outputMessage'], 'w') as output_message:
output_message.write('This is my output test message !')
It is working fine. Now I'd like to send a message to a topic. I've created a subscription with an SQLFilter and I need to set some custom properties to the BrokeredMessage.
From the azure sdk for python, I've found that I can add custom properties like that (I've installed the azure module using pip):
from azure.servicebus import Message
sent_msg = Message(b'This is the third message',
broker_properties={'Label': 'M3'},
custom_properties={'Priority': 'Medium',
'Customer': 'ABC'}
)
My new function.json file looks like that:
{
"bindings": [
{
"type": "serviceBus",
"connection": "Omnibus_Input_Send_Servicebus",
"name": "outputMessage",
"topicName": "validation-output-topic",
"accessRights": "send",
"direction": "out"
}
],
"disabled": false
}
And I've modify my function like that:
from azure.servicebus import Message
sent_msg = Message(b'This is the third message',
broker_properties={'Label': 'M3'},
custom_properties={'Priority': 'Medium',
'Customer': 'ABC'}
)
with open(os.environ['outputMessage'], 'w') as output_message:
output_message.write(sent_msg)
When I run the function, I get this exception:
TypeError: expected a string or other character buffer object
I tried to use the buffer and the memoryview function but still get another exception:
TypeError: cannot make memory view because object does not have the buffer interface
I am wondering if the actual binding supports BrokeredMessage and how to deal with it ?

The ServiceBus output binding for Python (and other script languages) only supports a simple string mapping, where the string you specify becomes the content of the BrokeredMessage created behind the scenes. To set any extended properties or do anything more sophisticated, you'll have to drop down to using the Azure Python SDK yourself in your function.

In the same situation, where I need to add user properties in the output service bus queue/topic, I used azure.servicebus.ServiceBusClient directly.
sb.Message class has a user_properties setter:
def main(
httpreq: func.HttpRequest,
context: func.Context ):
sbClient : sb.ServiceBusClient = sb.ServiceBusClient.from_connection_string( os.getenv("AzureWebJobsServiceBus") )
topicClient : sb.TopicClient = sbClient.get_topic('scoring-testtopic')
message = sb.Message( httpreq.get_body().decode( 'UTF-8' ))
message.user_properties = {
'#AzureWebJobsParentId' : context.invocation_id,
'Prom' : '31000001'
}
topicClient.send( message )