I use aws lambda to perform custom actions as Auto Scaling terminates instances. It looks like this
def scaledown_handler(event, context):
# customs actions
client = boto3.client('autoscaling')
response = client.complete_lifecycle_action(LifecycleHookName=event['detail']['LifecycleHookName'],
LifecycleActionToken=event['detail']['LifecycleActionToken'],
AutoScalingGroupName=event['detail']['AutoScalingGroupName'],
LifecycleActionResult='CONTINUE',
InstanceId=event['detail']['EC2InstanceId'])
The problem is that the function just hangs on client.complete_lifecycle_action() and finishes by timeout without any response and my ec2 instances are always "Waiting for Terminate Lifecycle Action".
aws autoscaling complete-lifecycle-action in aws CLI works fine, but i need to be done this from AWS lambda. How can I find out why does complete_lifecycle_action() hang without a response?
If you don't have a NAT gateway in your VPC then the Lambda function won't have access to anything outside the VPC. The AWS API exists outside your VPC, so the Lambda function is getting a network timeout trying to access it.
You have to add a NAT Gateway to your VPC in order for Lambda functions (and other things in your VPC that don't have a public IP) to access anything outside the VPC.
You need to use put_lifecycle_hook() API.
http://www.callumpember.com/auto-scaling-lifecycle-hooks/
On this link, you can get the complete python script for executing the custom actions before terminating the instance.
Related
I just wrote a python script that connect to the AWS ElastiCache. And it just check the connection.
from redis import Redis
try:
redis = Redis(host='xxx.cache.amazonaws.com',
port=6379,
db=0)
if not redis.ping():
raise("REDIS can't be initialized")
return True
except Exception as e:
print(str(e))
return False
When the following code is ran under my EC2 it has no problem and the response is reeally fast. Then I wrapped this using AWS Lambda and invoke the function locally. The returned result is still normal. But when the AWS Lambda is deployed on cloud the function got stuck on the redis.ping() until the function went timeout after 30 seconds.
I am not sure why the behavior is totally different.
Thanks in advance.
In a nutshell, this is what I did
Follow AWS Knowledge Center Videos: How do I give internet access to my Lambda function in a VPC? and created a Lambda function that would have internet access in the VPC
Follow Modifying a subnet group and make sure that the Redis's subnet is same as the Subnet of Lambda function
I am using event bridge to trigger a lambda function at 8 everyday to perform some ETL operations.
At times i receive requests to trigger the lambda manually ondemand. How can i achieve that using the same lambda function.
There are many ways to run the Lambda on demand like running it in the AWS Console, connecting it to an API Gateway and triggering it from the API Gateway etc.
But the easiest way is to use Lambda URLs
It will give you an URL that you can envoke that will run the Lambda.
When trying to use requests in my function, it only timeout without additional error. I was trying to use rapidapi for amazon. I already have the host and key, but when I test my function it always timeout. My zip file and my function were on the same directory and I knew that my code was correct.
I just figured out that my VPC configuration in Lambda was can only access within the resources of the VPC. I just removed the VPC and it now runs. But when your lambda function will connect to your database, you need to add and configure your VPC.
I have a Lambda function that has a Python handler that submits a job to AWS Batch via boto3 client:
client = boto3.client('batch', 'us-east-1')
def handle_load(event, context):
hasher = hashlib.sha1()
hasher.update(str(time.time()).encode())
job_name = f"job-{hasher.hexdigest()[:10]}"
job_queue = os.environ.get("job_queue")
job_definition = os.environ.get("job_definition")
logger.info(f"Submitting job named '{job_name}' to queue '{job_queue}' "
f"with definition '{job_definition}'")
response = client.submit_job(
jobName=job_name,
jobQueue=job_queue,
jobDefinition=job_definition,
)
logger.info(f"Submission successful, job ID: {response['jobId']}")
I can see this Lambda function submit the Batch job in CloudWatch logs but it always times out before the response comes back. I never see these jobs show up in the queue, so I'm not sure where things go after they are submitted, it seems that the Lambda is always timing out before the response comes back, I have little else to go on.
I have successfully added a job to the queue via AWS CLI, using the same queue and definition ARNs that are used in the Lambda's Python code. This job can be seen in the queue under the runnable tab (presumably the job will be started at some point in the near future).
The job submission with AWS CLI comes back instantly, so there must be something amiss on the Lambda configuration preventing the job submission. Perhaps I'm not using the correct role for the Lambda that submits the job, or have some other permissions that are amiss causing the timeout? The Lambda has permission for the batch:SubmitJob action allowed on all resources.
If an AWS Lambda function is not connected to a VPC, then by default it is connected to the Internet. This means it can call AWS API functions, which resides on the Internet.
If your Lambda function is configured to use a VPC, it will not have Internet access by default. This is good for connecting to other resources in a VPC, but if you wish to communicate with an AWS service, you'll either:
A NAT Gateway in a public subnet, with the Lambda function connected to a private subnet that has a Route Table rule that points to the NAT Gateway, or
A VPC endpoint that connects to the desired service. Unfortunately, AWS Batch does not have a VPC Endpoint.
So, if your Lambda function does not need to connect to other resources in the VPC, you can disconnect it and it should work. Otherwise, use a NAT Gateway.
Based on the comments. Lambda in a VPC does not have access to internet. You need to setup internet gateway in public subnet and NAT gateway in private subnet with your lambda to be able to access AWS Batch endpoints. Alternatively have to use VPC interface endpoint for AWS Batch. From docs:
Connect your function to private subnets to access private resources. If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address.
Also you need to add permissions to your lambda's execution role to be able to create network interface in VPC:
ec2:CreateNetworkInterface
ec2:DescribeNetworkInterfaces
ec2:DeleteNetworkInterface
I have two simple lambda functions. Lambda 1 is invoking lambda 2 (both do a simple print for text).
If both lambdas are outside of a VPC then the invocation succeeds, however as soon as I set them both in to access a VPC (I need to test within a VPC as the full process will be wtihin a VPC) the invocation times out.
Do I have to give my lambda access to the internet to be able invoke a second lambda within the same VPC?
If your lambda functions are inside a VPC you need to configure your both lambda functions into private subnet not public subnet. That is the AWS recommended way.
If you are invoking the second Lambda from the first using Amazon API Gateway, then your Lambda will need to have access to the internet. Follow this guide to configure a NAT Gateway (last step).
regarding the VPC: in order to connect to your VPC and access resources there, the Lambdas must reside in the same region as your VPC and also be configured access to your VPC.
Please follow the steps provided in this AWS Guide: Configuring a Lambda Function to Access Resources in an Amazon VPC. This guide advises to use AWS CLI commands to do this and does not show how to configure it through the console.
You will need to be familiar with Amazon networking particulars (VPCs, Security Groups and Subnets), IAM security for the VPC and have a CLI environment setup. You are going to grant the Lambda Function access to this VPC using IDs and IAM execution roles via the CLI.