I want to package and deploy a simple project on AWS Lambda, using Zappa, but without the Zappa requirements overhead.
Given this simple scenario:
lambda_handler.py
def handle(event, context):
print('Hello World')
I have a deploy.sh script that does that:
#!/usr/bin/env bash
source venv/bin/activate
zappa package -o lambda.zip
aws lambda update-function-code --function-name lambda-example --zip-file fileb://./lambda.zip
This works, BUT the final lambda.zip is way bigger then it needs to be:
I know that for this specific case the Zappa is not needed, but in the real project I'm using some libraries that requires https://github.com/Miserlou/lambda-packages, and using Zappa is the simplest way to install them.
How do I generate the python lambda package without this overhead?
First, you can use slim_handler that allows to upload larger files than 50M. Second, as #bddb already mentioned you can eclude some files such as .pyc, zip etc with the exclude property. Please find more details here:
https://github.com/Miserlou/Zappa#package
Here is an example how your zappa_settings.json could look like:
{
"dev": {
...
"slim_handler": false, // Useful if project >50M. Set true to just upload a small handler to Lambda and load actual project from S3 at runtime. Default false.
"exclude": ["*.gz", "*.rar"], // A list of regex patterns to exclude from the archive. To exclude boto3 and botocore (available in an older version on Lambda), add "boto3*" and "botocore*".
}
}
Related
I would like to use a serverless lambda that will execute commands from a tool called WSO2 API CTL as I would on linux cli. I am not sure of how to mimic the downloading and calling of the commands as if I were on a linux machine using either Nodejs or Python via the lambda?
I am okay with creating and setting up the lambda and even getting it in the right VPC so that the commands will reach an application on an EC2 instance but I am stuck at how to actually execute the linux commands using either Nodejs or Python and which one would be better, if any.
After adding the following I get an error trying to download:
os.system("curl -O https://apim.docs.wso2.com/en/latest/assets/attachments/learn/api-controller/apictl-3.2.1-linux-x64.tar.gz")
Warning: Failed to create the file apictl-3.2.1-linux-x64.tar.gz: Read-only
It looks like there is no specific reason to download apictl during the initialisation of your Lambda. Therefore, I would propose to bundle it with your deployment package.
The advantage of this approach are:
Quicker initialisation
Less code in your Lambda
You could extend your CI/CD pipeline to download the application during build and then add it to your ZIP archive that you deploy.
I'm working on an ML project that utilizes AWS Lambda for building models and generating predictions. Lambdas are written in python and use several ML libraries like pandas, sklearn, numpy, and scikit-learn.
These lambdas use shared code that packaged by Lambda's layer.
I use AWS CDK for project deployment. CDK code is written in TypeScript, don't ask why I mix Python and Typescript, it's not relevant in this situation.
The size of the package (lambda code + layer) exceeds the maximum allowed size of 250MB because of ML libraries.
After AWS announcement of containerized lambdas support, I decided to try it out to overcome the 250MB limit. However, I didn't find any good example that fit my situation, so I'm trying to build it myself.
The CDK code looks like this:
...
// Create a lambda layer from code
// Code is located in lambda-code/ml directory and it looks
// like any Python package with main ML and DB connection functions
const mlLayer = new PythonLayerVersion(this, 'mlLayer', {
entry: './lambda-code/ml/',
})
...
// Lambda function is specified like
const classifyTransactionLambda = new DockerImageFunction(this, 'classifyTransactionLambda', {
code: DockerImageCode.fromImageAsset('./lambda-code/classify'),
memorySize: 512,
layers: [mlLayer],
tracing: Tracing.ACTIVE,
environment: {
BUCKET_NAME: mlModelsBucket.bucketName,
ENV: env
}
});
...
The structure of the code looks like this:
Dockerfile in classify lambda:
# Use the python lambda image from AWS ECR
FROM public.ecr.aws/lambda/python:3.7
COPY requirements.txt ./
RUN pip3 install -r requirements.txt
COPY index.py ./
CMD ["index.classify_transaction_handler"]
When I run cdk deploy I'm getting the following error:
This lambda function uses a runtime that is incompatible with this layer (FROM_IMAGE is not in [python3.7])
Does anyone run into problem like this? Is this error mean that mlLayer version is not compatible with lambda classifyTransactionLambda?
Any help would be very appreciated!
At this point
Functions defined as container images do not support layers. When you build a container image, you can package your preferred runtimes and dependencies as a part of the image.
https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html
So I modified my build to copy all layer/library code to each lambda function before building an image.
I have built a deployment package with pandas, numpy, etc for my sample code to run. The size is some 46 MB. Doubt is, do I have to zip my code update every time and again update the entire deployment package to AWS S3 for a simple code update too?
Is there any other way, by which, I can avoid the 45 MB upload cost of S3 everytime and just upload the few KBs of code?
I would recommend creating a layer in AWS lambda.
First you need to create an instance of Amazon Linux (using the AMI specified in https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html - at this time (26th of March 2019) it is amzn-ami-hvm-2017.03.1.20170812-x86_64-gp2 ) or a docker container with the same environment as the lambda execution environment.
I personally do it with docker.
For example, to create a layer for Python 3.6, I would run
sudo docker run --rm -it -v "$PWD":/var/task lambci/lambda:build-python3.6 bash
Then I would create a folder python/lib/python3.6/site-packages in /var/task in the docker container (so it will be accessible later on in the directory on the host machine where I started docker)
do pip3 install <your packages here> -t python/lib/python3.6/site-packages
zip up the folder python and upload it as a layer and use it in my AWS lambda function.
NB! The paths in the zip file should look like "python/lib/python3.6/site-packages/{your package names}"
Now the heavy dependencies are in separate layer and you don't have re-upload them every time you update the function, you only need to update the code
Split the application into two parts. The first part would be the lambda function that only includes your application code. The next part a lambda layer. The lambda layer will include onky the the dependencies and be uploaded once.
A lambda layer can be uploaded and attached to the lambda function. When your function is invoked, AWS will combine the lambda function with the lambda layer then execute the entire package.
When updating your code, you will only need to update the lambda function. Since the package is much smaller you can edit it using the web editor, or you can zip it and upload it directly to lambda using the cli tools.
Exmample: aws lambda update-function-code --function-name Yourfunctionname --zip-file fileb://Lambda_package.zip
Here are video instructions and examples on creating a lambda layer for dependencies.It demonstrates using the pymsql library, but you can install any of your libraries there.
https://geektopia.tech/post.php?blogpost=Create_Lambda_Layer_Python
I got an AWS python Lambda function which contains few python files and also several dependencies.
The app is build using Chalice so by that the function will be mapped like any REST function.
Before the deployment in prod env, I want to test it locally, so I need to pack all this project (python files and dependencies), I tried to look over the web for the desired solution but I couldn't find it.
I managed to figrue how to deploy one python file, but a whole project did not succeed.
Take a look to the Atlassian's Localstack: https://github.com/atlassian/localstack
It's a full copy of the AWS cloud stack, locally.
I use Travis : I hooked it to my master branch in git, so that when I push on this branch, Travis tests my lambda, with a script that uses pytest, after having installed all its dependencies with pip install. If all the tests passed, it then deploy the lambda in AWS in my prod-env.
I'm new to AWS Lambda and pretty new to Python.
I wanted to write a python lambda that uses the AWS API.
boto is the most popular python module to do this so I wanted to include it.
Looking at examples online I put import boto3 at the top of my Lambda and it just worked- I was able to use boto in my Lambda.
How does AWS know about boto? It's a community module. Are there a list of supported modules for Lambdas? Does AWS cache its own copy of community modules?
AWS Lambda's Python environment comes pre-installed with boto3. Any other libraries you want need to be part of the zip you upload. You can install them locally with pip install whatever -t mysrcfolder.
The documentation seems to suggest boto3 is provided by default on AWS Lambda:
AWS Lambda includes the AWS SDK for Python (Boto 3), so you don't need to include it in your deployment package. However, if you want to use a version of Boto3 other than the one included by default, you can include it in your deployment package.
As far as I know, you will need to manually install any other dependencies in your deployment package, as shown in the linked documentation, using:
pip install foobar -t <project path>
AWS Lambda includes the AWS SDK for Python (Boto 3), so you don't need to include it in your deployment package.
This link will give you a little more in-depth info on Lambda environment
https://aws.amazon.com/blogs/compute/container-reuse-in-lambda/
And this too
https://alestic.com/2014/12/aws-lambda-persistence/