I have a small AWS Lambda function that looks like this:
It grabs the credentials to connect to Snowflake from SSM Parameter Store, and then calls snowflake.connector.connect. It's meant to obviously go grab data from my Snowflake data warehouse. However, the code hangs and never finishes the snowflake.connector.connect call.
I believe my subnets and networking are set up properly:
Just to test and develop, I set my security group to allow all inbound and outbound traffic on all ports.
I have my Lambda running in a private subnet, and a route table that directs 0.0.0.0/0 to the NAT Gateway instance. In my code, I print(requests.get('http://216.58.192.142')) just to prove that I do indeed have internet connectivity.
I have many large dependencies that don't fit in the 200MB deploy package for Lambdas, so I have my dependencies mounted in an EFS file system at /mnt/efs path, and I add /mnt/efs/python to my PYTHONPATH in the code before I start to import those dependencies.
import boto3
import snowflake.connector
print("Getting snowflake creds")
session = boto3.session.Session()
ssm = session.client("ssm")
obj = ssm.get_parameter(Name="snowflake", WithDecryption=True)
sf_creds = json.loads(obj.get("Parameter").get("Value"))
def get_data(event, context):
print(requests.get('http://216.58.192.142'))
print("Executing")
print("got parameter, connecting")
con = snowflake.connector.connect(
user=sf_creds["USER"],
password=sf_creds["PASSWORD"],
account=sf_creds["ACCOUNT"],
ocsp_response_cache_filename="/tmp/ocsp_response_cache"
)
print("connected")
When I run this same exact code locally on my MacBook, I am able to connect fairly quickly, within a second or two. My snowflake-python-connector version is 2.3.2.
However, no matter how long I try, the connect method hands when it executes in an AWS Lambda function. I'm really not sure what is going on.
I've verified that
the AWS Lambda function is connected to the internet (it receives a [200 OK] from the requests.get call).
security groups are as permissive as possible (allow all traffic on all ports both inbound and outbound)
I have not touched the NACL
Really, I'm at a loss as to why this is happening, especially given that the code works fine on my local machine - could someone try to point me in the right direction?
Lambda can respond with 200, but there can be exception in logs. Check out the CloudWatch logs of this lambda.
Related
When trying to use requests in my function, it only timeout without additional error. I was trying to use rapidapi for amazon. I already have the host and key, but when I test my function it always timeout. My zip file and my function were on the same directory and I knew that my code was correct.
I just figured out that my VPC configuration in Lambda was can only access within the resources of the VPC. I just removed the VPC and it now runs. But when your lambda function will connect to your database, you need to add and configure your VPC.
I have a Azure Function, which makes calculations and stores and reads data from my own Cosmos DB and one external database via REST API.
From Azure Portal, I can see the "outboundIpAddresses" and "possibleOutboundIpAddresses" (subscriptions > {your subscription} > providers > Microsoft.Web > sites). Totally 12 IP addresses. When I run the function locally (VS Code), everything goes smoothly. However, when I deploy that function, I get the following error:
Result: Failure Exception: CosmosHttpResponseError: (Forbidden) Request originated from client IP <IP-address> through public internet. This is blocked by your Cosmos DB account firewall settings
This itself is self-explanatory, but the problem is that the IP-address mentioned in the error message does not belong neither "outboundIpAddresses" or "possibleOutboundIpAddresses". And almost every time the function gets triggered, the client IP in the error message changes.
Do you have any ideas why this happens and how to solve the issue?
Is your function app in Consumption plan? If yes, when a function app that runs on the Consumption plan is scaled, a new range of outbound IP addresses may be assigned. When running on the Consumption plan, you may need to whitelist the entire data center.
On further note, if you are into app service plan, you have the option of assigning dedicated IP address .
I have a Lambda function that has a Python handler that submits a job to AWS Batch via boto3 client:
client = boto3.client('batch', 'us-east-1')
def handle_load(event, context):
hasher = hashlib.sha1()
hasher.update(str(time.time()).encode())
job_name = f"job-{hasher.hexdigest()[:10]}"
job_queue = os.environ.get("job_queue")
job_definition = os.environ.get("job_definition")
logger.info(f"Submitting job named '{job_name}' to queue '{job_queue}' "
f"with definition '{job_definition}'")
response = client.submit_job(
jobName=job_name,
jobQueue=job_queue,
jobDefinition=job_definition,
)
logger.info(f"Submission successful, job ID: {response['jobId']}")
I can see this Lambda function submit the Batch job in CloudWatch logs but it always times out before the response comes back. I never see these jobs show up in the queue, so I'm not sure where things go after they are submitted, it seems that the Lambda is always timing out before the response comes back, I have little else to go on.
I have successfully added a job to the queue via AWS CLI, using the same queue and definition ARNs that are used in the Lambda's Python code. This job can be seen in the queue under the runnable tab (presumably the job will be started at some point in the near future).
The job submission with AWS CLI comes back instantly, so there must be something amiss on the Lambda configuration preventing the job submission. Perhaps I'm not using the correct role for the Lambda that submits the job, or have some other permissions that are amiss causing the timeout? The Lambda has permission for the batch:SubmitJob action allowed on all resources.
If an AWS Lambda function is not connected to a VPC, then by default it is connected to the Internet. This means it can call AWS API functions, which resides on the Internet.
If your Lambda function is configured to use a VPC, it will not have Internet access by default. This is good for connecting to other resources in a VPC, but if you wish to communicate with an AWS service, you'll either:
A NAT Gateway in a public subnet, with the Lambda function connected to a private subnet that has a Route Table rule that points to the NAT Gateway, or
A VPC endpoint that connects to the desired service. Unfortunately, AWS Batch does not have a VPC Endpoint.
So, if your Lambda function does not need to connect to other resources in the VPC, you can disconnect it and it should work. Otherwise, use a NAT Gateway.
Based on the comments. Lambda in a VPC does not have access to internet. You need to setup internet gateway in public subnet and NAT gateway in private subnet with your lambda to be able to access AWS Batch endpoints. Alternatively have to use VPC interface endpoint for AWS Batch. From docs:
Connect your function to private subnets to access private resources. If your function needs internet access, use NAT. Connecting a function to a public subnet does not give it internet access or a public IP address.
Also you need to add permissions to your lambda's execution role to be able to create network interface in VPC:
ec2:CreateNetworkInterface
ec2:DescribeNetworkInterfaces
ec2:DeleteNetworkInterface
import pymysql
from botocore.vendored import requests
def lambda_handler(event,context):
conn=pymysql.connect (host ="rootrestdatabase.cd6kbmibgfod.ap-south-1.rds.amazonaws.com", user="****" , passwd="*****",db="restawsdatabase")
i want to connect rds in lambda . but i gor error like
"errorMessage": "(2003, \"Can't connect to MySQL server on 'rootrestdatabase.cd6kbmibgfod.ap-south-1.rds.amazonaws.com' (timed out)\")",
"errorType": "OperationalError",
"stackTrace": [
[
i have tried it in local machine it works fine but when i deploy code in lamba it didnt worked. also both lambda function and rds are in same region. what shoud i have to do?
To communicate with RDS instances, lambda functions have to be in the same VPC - a network timeout error is a great indicator of that. However, if your RDS instance is publicly accessible, make sure the security groups, which you attached to the function, allow traffic that you need.
When you are connecting Lambda to My SQL DB make sure to follow the below steps:
The Execution Role which is assigned to Lambda must have Permission to EC2 Full access.
Go to VPC Tab in Lambda and make sure you have VPC there. Edit and add Required VPC Connection with proper Subnets and Security Groups.
Once this is done wait for 2 minutes and start testing your Lambda. It should connect to your required DB in VPC
I use aws lambda to perform custom actions as Auto Scaling terminates instances. It looks like this
def scaledown_handler(event, context):
# customs actions
client = boto3.client('autoscaling')
response = client.complete_lifecycle_action(LifecycleHookName=event['detail']['LifecycleHookName'],
LifecycleActionToken=event['detail']['LifecycleActionToken'],
AutoScalingGroupName=event['detail']['AutoScalingGroupName'],
LifecycleActionResult='CONTINUE',
InstanceId=event['detail']['EC2InstanceId'])
The problem is that the function just hangs on client.complete_lifecycle_action() and finishes by timeout without any response and my ec2 instances are always "Waiting for Terminate Lifecycle Action".
aws autoscaling complete-lifecycle-action in aws CLI works fine, but i need to be done this from AWS lambda. How can I find out why does complete_lifecycle_action() hang without a response?
If you don't have a NAT gateway in your VPC then the Lambda function won't have access to anything outside the VPC. The AWS API exists outside your VPC, so the Lambda function is getting a network timeout trying to access it.
You have to add a NAT Gateway to your VPC in order for Lambda functions (and other things in your VPC that don't have a public IP) to access anything outside the VPC.
You need to use put_lifecycle_hook() API.
http://www.callumpember.com/auto-scaling-lifecycle-hooks/
On this link, you can get the complete python script for executing the custom actions before terminating the instance.