I'm new to AWS and struggling with step function.
My workflow is like this:
client ('search_word')-> api gateway -> lambda function (invoke step function) -> step function (generate search output) -> client
Here's my invoke lambda function.
import json
import boto3
import uuid
client = boto3.client('stepfunctions')
def lambda_handler(event, context):
transactionId = str(uuid.uuid1())
print(transactionId)
input = {'TransactionId':transactionId,'text':'search_word'}
response = client.start_execution(
stateMachineArn='arn:aws:states:ap-northeast-2:xxxxxxxxxx:stateMachine:MyStateMachine',
name=transactionId,
input=json.dumps(input)
)
print(response)
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
I want to get the execution result from step function and pass it to client. But I have no idea how to do it.
The workflow doesn't have to be what I suggested as long as I can give the execution result of step function to the client.
Here's my step function.
{
"Comment": "A simple AWS Step Functions state machine.",
"StartAt": "Tokenize",
"States": {
"Tokenize": {
"Type": "Task",
"Resource": "arn:aws:lambda:ap-northeast-2:xxxxxxxx:function:search_ko",
"Next": "Search"
},
"Search": {
"Type": "Task",
"Resource": "arn:aws:lambda:ap-northeast-2:xxxxxxx:function:BM-25-Get-Index",
"End": true
}
}
}
Please help.
Thanks in advance!
You are trying to combine api gateway which is synchronous, with step functions which are asynchronous. If you use step functions your flow should be asynchronous. You will not get the response with a single API call. You will need to create another API which will keep "polling" the step function to see if it has successfully executed and what the response is.
Here is the flow you can try out.
API call that initializes the step function. You can try directly connecting the API gateway post call with your step function and see if it returns the execution arn in response. If it does you won't need a lambda. If it doesn't, then stick to your current lambda and make it return execution arn.
Your app then gets the execution arn and uses it to call another API periodically. You can either call API periodically or have the lambda in backend keep polling for response if you think the time it takes to complete step function execution will be less than 30 seconds (API gateway timeout limit).
Your second API/Lambda then finds the result of execution using DescribeExecution, and returns the response.
Edit:
If you believe your step functions will execute consistently under 30 seconds, you can try using a single lambda to start the step function and then keep polling it for completion using DescribeExecution.
Related
hello everyone I almost finished a bot with lex v2 that uses lambda, I did the integration using the kommunicate client to do before, but now I need the chatbot to start with an intent automatically, this is my first problem.
Also I would like to know if on lambda in the answers there is a way to send a response that makes an elicit slot, and then after 5 seconds send another to move to confirmation intent on lex v2.. I tried with time and asyncio, but it seems that the code does not go on, I can only get the first answer with the slot elicit, I wish that after about 5 seconds I refer to confirmation intent on lex v2, this is my code:
if carFound is not None:
# if resp['total'] < 30:
# print('Conferma intento')
# carDialog = {
# "type": "ConfirmIntent"
# }
# else:
# print('Filtri args')
# carDialog ={
# "type": "ElicitSlot",
# "slotToElicit": "Args"
# }
response = {
"sessionState": {
"dialogAction": {
"type": "ElicitSlot",
"slotToElicit": "Args"
},
"intent": {
'name': intent,
'slots': slots,
"state": "Fulfilled",
}
},
"messages": carFound,
}
practically after every call to my api that sends me carFound as a payload with all the machines, I should verify that when resp['total'] and less than 30 I refer in addition to the answers also to confirmation intent after some time.
as i said yet i tried with sleep() function of python, and i still have only the response that has the elicitslot,maybe sleep and asyncio are not well for lexv2...
I verified through the test of lambda, with my inputs that the condition of <30 is true, so the problem is on the response i think.
for the welcome i don't know how i can do, i want that my bot start with the intent Welcome for example, without write nothing on my kommunicate client that is on a website.
I am not able to find any code sample or relevant documentation on python library for google cloud
Want to restart managed instance groups all vms via cloud function.
To list instances I am using something like this
import googleapiclient.discovery
def list_instances(compute, project, zone):
result = compute.instances().list(project=project, zone=zone).execute()
return result['items'] if 'items' in result else None
in requirement file I have
google-api-python-client==2.31.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
From command line this is possible via SDK ->
https://cloud.google.com/sdk/gcloud/reference/compute/instance-groups/managed/rolling-action/restart
gcloud compute instance-groups managed rolling-action restart NAME [--max-unavailable=MAX_UNAVAILABLE] [--region=REGION | --zone=ZONE] [GCLOUD_WIDE_FLAG …]
But in python I am not able to write any code.
This is an incomplete answer since the python docs are pretty unreadable to me.
Looking at the gcloud cli code (which I couldn't find an official repo for so I looked here),
the restart command is triggered by something called a "minimal action".
minimal_action = (client.messages.InstanceGroupManagerUpdatePolicy.
MinimalActionValueValuesEnum.RESTART)
In the Python docs, there's references to these fields in the applyUpdatesToInstances method.
So I think the relevant code is something similar to:
compute.instanceGroupManagers().applyUpdatesToInstances(
project=project,
zone=zone,
instanceGroupManager='NAME',
body={"allInstances": True, "minimalAction": "RESTART"},
)
There may or may not be a proper Python object for the body, the docs aren't clear.
And the result seems to be an Operation object of some kind, but I don't know if there's execute() method or not.
This is confusing, because gcloud compute instance-groups managed rolling-action is syntactic sugar that does two things:
It turns on Proactive updater, by setting appropriate UpdatePolicy on the InstanceGroupManager resource
And it changes version name on the same resource to trigger an update.
It is covered in the docs in https://cloud.google.com/compute/docs/instance-groups/rolling-out-updates-to-managed-instance-groups#performing_a_rolling_replace_or_restart
Compare the gcloud and API tabs to get the idea.
Unfortunately I am illiterate in Python, so I am not able to translate it into Python code :(.
Using the documentation that #Grzenio provided, use patch() method to restart the instance group. See patch documentation to check its parameters.
This could be written in python using the code below. I provided the required parameters project,zone,instanceGroupManager and body. The value of body is from the example in the documentation.
import googleapiclient.discovery
import json
project = 'your-project-id'
zone = 'us-central1-a' # the zone of your instance group
instanceGroupManager = 'instance-group-1' # instance group name
body = {
"updatePolicy": {
"minimalAction": "RESTART",
"type": "PROACTIVE"
},
"versions": [{
"instanceTemplate": "global/instanceTemplates/instance-template-1",
"name": "v2"
}]
}
compute = googleapiclient.discovery.build('compute', 'v1')
rolling_restart = compute.instanceGroupManagers().patch(
project=project,
zone=zone,
instanceGroupManager=instanceGroupManager,
body=body
)
restart_operation = rolling_restart.execute() # execute the request
print(json.dumps(restart_operation,indent=2))
This will return an operation object and the instance group should restart in the rolling fashion:
{
"id": "3206367254887659944",
"name": "operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"zone": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a",
"operationType": "patch",
"targetLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/instanceGroupManagers/instance-group-1",
"targetId": "810482163278776898",
"status": "RUNNING",
"user": "serviceaccountused#your-project-id.iam.gserviceaccount.com",
"progress": 0,
"insertTime": "2021-12-01T20:10:47.654-08:00",
"startTime": "2021-12-01T20:10:47.670-08:00",
"selfLink": "https://www.googleapis.com/compute/v1/projects/your-project-id/zones/us-central1-a/operations/operation-1638418246759-5d221f9977443-33811aed-eed3ee88",
"kind": "compute#operation"
}
I am using AWS SQS to store information coming in from an external server and then sending it to a Lambda function to process it and dequeue the information.
The information that I am sending in is in the form of a JSON and is being used as a python dictionary.
def lambda_handler(event, context):
for record in event['Records']:
messageHandler(record)
return {
'statusCode': 200,
'body': json.dumps('Batch Processed')
}
Assuming that the code for the messageHandler is working and properly implemented, how do I catch the messages from the queue in their batches. This is all being deployed by AWS Chalice without the use of CLI.
I am well out of my depth right now and have no idea why this is not working when I deploy it but is working when I trigger a normal Lambda Function in the AWS Console through the SQS Send/Recieve Message feature. As far as I know the triggers are set up correctly and they should have no issue.
If you have any questions please let me know.
The event that you are processing will look something like this:
{
"Records": [
{
"messageId": "11d6ee51-4cc7-4302-9e22-7cd8afdaadf5",
"receiptHandle": "AQEBBX8nesZEXmkhsmZeyIE8iQAMig7qw...",
"body": "Test message.",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1573251510774",
"SequenceNumber": "18849496460467696128",
"MessageGroupId": "1",
"SenderId": "AIDAIO23YVJENQZJOL4VO",
"MessageDeduplicationId": "1",
"ApproximateFirstReceiveTimestamp": "1573251510774"
},
"messageAttributes": {},
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:fifo.fifo",
"awsRegion": "us-east-2"
}
]
}
where the "body" is your json encoded message. You'll want your message handler function to do something like this:
def message_handler(event):
message = json.loads(event["body"])
# do stuff...
The return value from the lambda is pretty pointless if it is being used as the event target from sqs.
I have a AWS lambda script that is triggered by a post request. The script takes about 10 minutes to run. When the POST request/trigger is received, I want to return a guid/identifier before/at the same time as running the script. That way, a user can use that guid to check the progress of the script via a db config I have not yet made.
My problem is that when my lambda_handler gives a return object, the script stops running. I want to return the guid, and keep the script running.
You can't do that with only lambda function.
If you want to achieve your goal, you must use Lambda with Step Functions to build a complete workflow:
https://docs.aws.amazon.com/step-functions/latest/dg/welcome.html
You have to break your lambda into 2 parts. The first one return the guid/identifier. The other continue processing the logic after the first lambda returns the result.
Step Functions configuration example:
{
"StartAt": "LambdaFunction1",
// The first lambda that return the guid/identifier
"States": {
"LambdaFunction1": {
"Type": "Task",
"Resource": "arn:aws:lambda1:here",
"Next": "LambdaFunction2"
},
"LambdaFunction2": {
"Type": "Task",
"Resource": "arn:aws:lambda2:here",
"End": true
}
}
}
The AWS Lambda handler has a signature of
def lambda_handler(event, context):
However, I cannot find any documentation as to the event's structure when the trigger is an S3 Bucket receiving a put
I thought that it might be defined in the s3 console, but couldn't find that there.
Anyone have any leads?
The event from S3 to Lambda function will be in json format as shown below,
{
"Records":[
{
"eventVersion":"2.0",
"eventSource":"aws:s3",
"awsRegion":"us-east-1",
"eventTime":The time, in ISO-8601 format, for example, 1970-01-01T00:00:00.000Z, when S3 finished processing the request,
"eventName":"event-type",
"userIdentity":{
"principalId":"Amazon-customer-ID-of-the-user-who-caused-the-event"
},
"requestParameters":{
"sourceIPAddress":"ip-address-where-request-came-from"
},
"responseElements":{
"x-amz-request-id":"Amazon S3 generated request ID",
"x-amz-id-2":"Amazon S3 host that processed the request"
},
"s3":{
"s3SchemaVersion":"1.0",
"configurationId":"ID found in the bucket notification configuration",
"bucket":{
"name":"bucket-name",
"ownerIdentity":{
"principalId":"Amazon-customer-ID-of-the-bucket-owner"
},
"arn":"bucket-ARN"
},
"object":{
"key":"object-key",
"size":object-size,
"eTag":"object eTag",
"versionId":"object version if bucket is versioning-enabled, otherwise null",
"sequencer": "a string representation of a hexadecimal value used to determine event sequence,
only used with PUTs and DELETEs"
}
}
},
{
// Additional events
}
]
}
here is the link for aws documentation which can guide you. http://docs.aws.amazon.com/lambda/latest/dg/with-s3-example.html
I think your easiest route is just to experiment quickly:
Create a bucket using the console
Create a lambda that is triggered by puts to the bucket using the console
Ensure you choose the default execution role, so you create cloudwatch logs
The lambda function just needs to "print(event)" when called, which is then logged
Save an object to the bucket
You'll then see the event structure in the log - its pretty self explanatory.
Please refer this URL to get Event Message Structure: http://docs.aws.amazon.com/AmazonS3/latest/dev/notification-content-structure.html