I'm having some trouble invoking locally using serverless.
Using Python3.6 runtime
The function in serverless.yml is:
functions:
myFunction:
events:
- sns: arn:aws:sns:us-east-1:123456789:myTopic
My command is
sls invoke local -f myFunction -s dev -r us-east-1 -p events/myMockedSnsMessage.json
Serverless doesn't like the syntax on the myMockedSnsMessage.json
When I log the event in my lambda function I get something like:
{
'Records': [
{
'Sns': {
'Message': '{"version":"0","id":"a965ce94-fcb2-ad15-319d-04adab1072d0","detail-type":"AWS API Call via CloudTrail",...}'
}
}
]
}
That is, the SNS message is a string with valid JSON inside
How should I store a mock event for an SNS message and still have valid JSON so serverless doesn't yell at me for bad JSON syntax?
Perhaps use double-quotes, and stringify the json. e.g.
{
"Records": [
{
"Sns": {
"Message": "{\"version\":\"0\",\"id\":\"a965ce94-fcb2-ad15-319d-04adab1072d0\",\"detail-type\":\"AWS API Call via CloudTrail\",...}"
}
}
]
}
If you need more examples, the AWS documentation has sample events that you can modify, and when you use AWS console to test your lambda, you can select many different event templates from a dropdown.
Related
I am not able to find any code sample or relevant documentation on python library for google cloud
Want to restart managed instance groups all vms via cloud function.
To list instances I am using something like this
import googleapiclient.discovery
def list_instances(compute, project, zone):
result = compute.instances().list(project=project, zone=zone).execute()
return result['items'] if 'items' in result else None
in requirement file I have
google-api-python-client==2.31.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
From command line this is possible via SDK ->
https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/rolling-action/restart
gcloud compute instance-groups managed rolling-action restart NAME [--max-unavailable=MAX_UNAVAILABLE] [--region=REGION | --zone=ZONE] [GCLOUD_WIDE_FLAG …]
But in python I am not able to write any code.
This is an incomplete answer since the python docs are pretty unreadable to me.
Looking at the gcloud cli code (which I couldn't find an official repo for so I looked here),
the restart command is triggered by something called a "minimal action".
minimal_action = (client.messages.InstanceGroupManagerUpdatePolicy.
MinimalActionValueValuesEnum.RESTART)
In the Python docs, there's references to these fields in the applyUpdatesToInstances method.
So I think the relevant code is something similar to:
compute.instanceGroupManagers().applyUpdatesToInstances(
project=project,
zone=zone,
instanceGroupManager='NAME',
body={"allInstances": True, "minimalAction": "RESTART"},
)
There may or may not be a proper Python object for the body, the docs aren't clear.
And the result seems to be an Operation object of some kind, but I don't know if there's execute() method or not.
This is confusing, because gcloud compute instance-groups managed rolling-action is syntactic sugar that does two things:
It turns on Proactive updater, by setting appropriate UpdatePolicy on the InstanceGroupManager resource
And it changes version name on the same resource to trigger an update.
It is covered in the docs in https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups#performing_a_rolling_replace_or_restart
Compare the gcloud and API tabs to get the idea.
Unfortunately I am illiterate in Python, so I am not able to translate it into Python code :(.
Using the documentation that #Grzenio provided, use patch() method to restart the instance group. See patch documentation to check its parameters.
This could be written in python using the code below. I provided the required parameters project,zone,instanceGroupManager and body. The value of body is from the example in the documentation.
import googleapiclient.discovery
import json
project = 'your-project-id'
zone = 'us-central1-a' # the zone of your instance group
instanceGroupManager = 'instance-group-1' # instance group name
body = {
"updatePolicy": {
"minimalAction": "RESTART",
"type": "PROACTIVE"
},
"versions": [{
"instanceTemplate": "global/instanceTemplates/instance-template-1",
"name": "v2"
}]
}
compute = googleapiclient.discovery.build('compute', 'v1')
rolling_restart = compute.instanceGroupManagers().patch(
project=project,
zone=zone,
instanceGroupManager=instanceGroupManager,
body=body
)
restart_operation = rolling_restart.execute() # execute the request
print(json.dumps(restart_operation,indent=2))
This will return an operation object and the instance group should restart in the rolling fashion:
{
"id": "3206367254887659944",
"name": "operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"zone": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a",
"operationType": "patch",
"targetLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/instanceGroupManagers/instance-group-1",
"targetId": "810482163278776898",
"status": "RUNNING",
"user": "serviceaccountused#your-project-id.iam.gserviceaccount.com",
"progress": 0,
"insertTime": "2021-12-01T20:10:47.654-08:00",
"startTime": "2021-12-01T20:10:47.670-08:00",
"selfLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/operations/operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"kind": "compute#operation"
}
I am using AWS SQS to store information coming in from an external server and then sending it to a Lambda function to process it and dequeue the information.
The information that I am sending in is in the form of a JSON and is being used as a python dictionary.
def lambda_handler(event, context):
for record in event['Records']:
messageHandler(record)
return {
'statusCode': 200,
'body': json.dumps('Batch Processed')
}
Assuming that the code for the messageHandler is working and properly implemented, how do I catch the messages from the queue in their batches. This is all being deployed by AWS Chalice without the use of CLI.
I am well out of my depth right now and have no idea why this is not working when I deploy it but is working when I trigger a normal Lambda Function in the AWS Console through the SQS Send/Recieve Message feature. As far as I know the triggers are set up correctly and they should have no issue.
If you have any questions please let me know.
The event that you are processing will look something like this:
{
"Records": [
{
"messageId": "11d6ee51-4cc7-4302-9e22-7cd8afdaadf5",
"receiptHandle": "AQEBBX8nesZEXmkhsmZeyIE8iQAMig7qw...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1573251510774",
"SequenceNumber": "18849496460467696128",
"MessageGroupId": "1",
"SenderId": "AIDAIO23YVJENQZJOL4VO",
"MessageDeduplicationId": "1",
"ApproximateFirstReceiveTimestamp": "1573251510774"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:fifo.fifo",
"awsRegion": "us-east-2"
}
]
}
where the "body" is your json encoded message. You'll want your message handler function to do something like this:
def message_handler(event):
message = json.loads(event["body"])
# do stuff...
The return value from the lambda is pretty pointless if it is being used as the event target from sqs.
How do you access URL querystring parameters from inside an AWS Lambda function served though an API Gateway?
I have both the API gateway + Lambda function setup so I can call it from a public URL. My Python function is simply:
def lambda_handler(event, context):
print('event:', event)
print('context:', context)
I've configured the API's GET "Method Request" handler to pass through the "abc" querystring parameter.
I've also configured the API's GET "Integration Request" handler to map "abc" from "method.request.querystring.abc".
However, when I access my URL, e.g. https://myapp.execute-api.us-east-1.amazonaws.com/prod/myfunc?abc=123, the only thing logged is:
event: {}
context: <bootstrap.LambdaContext object at 0x7fc7a6cb0850>
What am I doing wrong? Why isn't "abc" being passed through in the event dictionary?
Check Use Lambda Proxy integration in the Integration Request to have it pass all request details in the event.
I have a similar problem and I know how frustrating it is. Use this mapping template:
{
"method": "$context.httpMethod",
"body" : $input.json('$'),
"headers": {
#foreach($param in $input.params().header.keySet())
"$param": "$util.escapeJavaScript($input.params().header.get($param))" #if($foreach.hasNext),#end
#end
},
"queryStringParameters": {
#foreach($param in $input.params().querystring.keySet())
"$param": "$util.escapeJavaScript($input.params().querystring.get($param))" #if($foreach.hasNext),#end
#end
},
"pathParameters": {
#foreach($param in $input.params().path.keySet())
"$param": "$util.escapeJavaScript($input.params().path.get($param))" #if($foreach.hasNext),#end
#end
}
}
Then you should find your event looks like this:
{
"method":"GET",
"body":{
},
"headers":{
},
"queryParams":{
"id":"459463732",
"command":"join_session"
},
"pathParams":{
}
}
The context is used for other information, such as IP addresses and timeout settings.
I currently have a lambda function that updates a DynamoDB table with a value passed as a parameter. I am able to run the following within the Lambda console with a test parameter set to "TEST":
import boto3
import json
def lambda_handler(event, context):
# TODO implement
update_ami(event)
def update_ami(ami_id):
#DO STUFF
I am attempting to call this from an SSM Automation built from the following JSON document:
{
"description":"Test Execute Lambda Function.",
"schemaVersion":"0.3",
"assumeRole":"MYARN",
"parameters":{},
"mainSteps":[
{
"name": "invokeMyLambdaFunction",
"action": "aws:invokeLambdaFunction",
"maxAttempts": 3,
"timeoutSeconds": 120,
"onFailure": "Abort",
"inputs": {
"FunctionName": "MyLambdaFunction",
"Payload": "TESTER"
}
}
]
}
Executing this automation results in the following error:
Automation Step Execution fails when it is invoking the lambda function. Get Exception from Invoke API of lambda Service. Exception Message from Invoke API: [Could not parse request body into json: Unrecognized token 'TESTER': was expecting ('true', 'false' or 'null')
I have also tried passing the Payload input as a JSON object instead of a string, and adjusted my lambda method accordingly:
JSON Automation:
...
"inputs": {
"FunctionName": "MyLambdaFunction",
"Payload": {
"ami_id": "AMI-TESTER"
}
}
...
Lambda Python:
def lambda_handler(event, context):
# TODO implement
update_ami(event['ami-id'])
This results in the following error coming from the Automation Document editor within the SSM console:
Input {ami_id=TESTER} is of type class java.util.LinkedHashMap, but expected type is String.
So in a nutshell... How do I pass a single string from an Automation document to a Lambda Function?
The error which is shown below looks to be issue with payload not passed as string:
Input {ami_id=TESTER} is of type class java.util.LinkedHashMap, but expected type is String
Please try to use escape character before double quotes.
"inputs": {
"FunctionName": "MyLambdaFunction",
"Payload": "{
\"ami_id\": \"AMI-TESTER\"
}"
}
AWS has provided proper syntax, if you see this Url
{
"name":"updateSsmParam",
"action":"aws:invokeLambdaFunction",
"timeoutSeconds":1200,
"maxAttempts":1,
"onFailure":"Abort",
"inputs":{
"FunctionName":"Automation-UpdateSsmParam",
"Payload":"{\"parameterName\":\"latestAmi\", \"parameterValue\":\"{{createImage.ImageId}}\"}"
}
}
I have an Azure Function written in Python that has an Service Bus (Topic) output binding. The function is triggered by another queue, we process some files from a blobl storage and then put another message in a queue.
My function.json file looks like that:
{
"bindings": [
{
"type": "serviceBus",
"connection": "Omnibus_Input_Send_Servicebus",
"name": "outputMessage",
"queueName": "validation-output-queue",
"accessRights": "send",
"direction": "out"
}
],
"disabled": false
}
In my function, I can send a message to another queue like that:
with open(os.environ['outputMessage'], 'w') as output_message:
output_message.write('This is my output test message !')
It is working fine. Now I'd like to send a message to a topic. I've created a subscription with an SQLFilter and I need to set some custom properties to the BrokeredMessage.
From the azure sdk for python, I've found that I can add custom properties like that (I've installed the azure module using pip):
from azure.servicebus import Message
sent_msg = Message(b'This is the third message',
broker_properties={'Label': 'M3'},
custom_properties={'Priority': 'Medium',
'Customer': 'ABC'}
)
My new function.json file looks like that:
{
"bindings": [
{
"type": "serviceBus",
"connection": "Omnibus_Input_Send_Servicebus",
"name": "outputMessage",
"topicName": "validation-output-topic",
"accessRights": "send",
"direction": "out"
}
],
"disabled": false
}
And I've modify my function like that:
from azure.servicebus import Message
sent_msg = Message(b'This is the third message',
broker_properties={'Label': 'M3'},
custom_properties={'Priority': 'Medium',
'Customer': 'ABC'}
)
with open(os.environ['outputMessage'], 'w') as output_message:
output_message.write(sent_msg)
When I run the function, I get this exception:
TypeError: expected a string or other character buffer object
I tried to use the buffer and the memoryview function but still get another exception:
TypeError: cannot make memory view because object does not have the buffer interface
I am wondering if the actual binding supports BrokeredMessage and how to deal with it ?
The ServiceBus output binding for Python (and other script languages) only supports a simple string mapping, where the string you specify becomes the content of the BrokeredMessage created behind the scenes. To set any extended properties or do anything more sophisticated, you'll have to drop down to using the Azure Python SDK yourself in your function.
In the same situation, where I need to add user properties in the output service bus queue/topic, I used azure.servicebus.ServiceBusClient directly.
sb.Message class has a user_properties setter:
def main(
httpreq: func.HttpRequest,
context: func.Context ):
sbClient : sb.ServiceBusClient = sb.ServiceBusClient.from_connection_string( os.getenv("AzureWebJobsServiceBus") )
topicClient : sb.TopicClient = sbClient.get_topic('scoring-testtopic')
message = sb.Message( httpreq.get_body().decode( 'UTF-8' ))
message.user_properties = {
'#AzureWebJobsParentId' : context.invocation_id,
'Prom' : '31000001'
}
topicClient.send( message )