I'm currently trying to use the Python requests module in an AWS Lambda function. Here are the steps that I've taken so far:
I created a new directory, and installed the requests module in it, using the command pip3 install requests -t .
I then wrote a simple Python script, test_requests.py, within the directory, which looks like this:
import requests
def my_handler(event, context):
r = requests.get("http://google.com")
return r
I zipped the entire directory, including the requests module, using zip test_requests.zip *
I then uploaded the function to AWS with the following command: aws lambda create-function --function-name test_requests --zip-file fileb://test_requests.zip --handler test_requests.my_handler --runtime python3.6 --region us-east-1 --role xxxMY_ROLE_ARNxxx
Finally, I invoked the function with this command: aws lambda invoke --function-name test-requests --payload {} --region us-east-1 lambda_response.txt
When I made this command, I got an unhandled exception back from Lambda. The output file, lambda_response.txt contained this: {"errorMessage": "module 'requests' has no attribute 'get'", "errorType": "AttributeError", "stackTrace": [["/var/task/test_requests.py", 3, "my_handler", "r = requests.get('http://google.com')"]]}
I've seen several questions regarding AWS lambda, and being unable to import modules properly. Those questions all seemed focused around lambda being unable to find the module. In this case, it seems that lambda has found requests, but is unable to access all of its attributes.
I figured out what I was doing wrong. zip test.zip * only zips the top level of the directory structure. I needed the -r flag in order to capture everything.
Related
So I have this similar problem with this person.
How to create password encrypted zip file in python through AWS lambda
We have the exact same problem but i already did everything from the answers in that thread but to no avail.
I have a lambda script that runs on python3.9 I need to compress the files in my s3 as a zip file that is password protected and i need to put it in another s3.
This is how it goes
import pyminizip
def zip_to_client():
# reportTitles = os.listdir(tempDir)
dateGenerated = datetime.now(tz=atz).strftime("%Y-%m-%d")
pyminizip.compress("Daily_Booking_Report.csv", subfolder + str(dateGenerated) +'/'+str(id)+'/'
, "/tmp/test.zip", "awesomepassword", 9)
s3 = boto3.resource('s3')
s3.meta.client.upload_file(Filename = '/tmp/test.zip', Bucket = bucket, Key = subfolder + 'test.zip', ExtraArgs={'Tagging':'archive=90days'})
print("SUCCESS: Transferred report into S3")
i'm not sure if it works but i can't debug it because lambda shows me the error:
Response
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'pyminizip'",
"errorType": "Runtime.ImportModuleError",
"requestId": "0000111000",
"stackTrace": []
}
I made sure that i put import pyminizip as well as pip installing it in the directory.
pip install pyminizip -t .
so far this is what the lambda directory looks like
https://ibb.co/ZGmLBbv
i've tried everything from putting it in a lambda layer to pip installing different versions from python 3.7 to 3.9
This is a common case when you create a lambda layer and get import error. And this occurs when you don't have created python files in a defined directory like python/python38/site-packages...
or
second reason might be a dependency is missing. In that use use docker and follow steps from here : https://www.geeksforgeeks.org/how-to-install-python-packages-for-aws-lambda-layers/.
When I check the cloud watch logs of my Lambda function, I see theses errors:
[ERROR] Runtime.ImportModuleError: Unable to import module 'trigger_bitbucket_pipeline_from_s3': No module named 'requests'
File structure:
/bin
--trigger_bitbucket_pipeline_from_s3.zip
/src
--trigger_bitbucket_pipeline_from_s3.py
--/requests (lib folder)
lambda.tf
Lambda.tf:
data "archive_file" "lambda_zip" {
type = "zip"
source_file = "${path.module}/src/trigger_bitbucket_pipeline_from_s3.py"
output_file_mode = "0666"
output_path = "${path.module}/bin/trigger_bitbucket_pipeline_from_s3.zip"
}
resource "aws_lambda_function" "processing_lambda" {
filename = data.archive_file.lambda_zip.output_path
function_name = "triggering_pipleline_lambda"
handler = "trigger_bitbucket_pipeline_from_s3.lambda_handler"
source_code_hash = data.archive_file.lambda_zip.output_base64sha256
role = aws_iam_role.processing_lambda_role.arn
runtime = "python3.9"
}
My lambda function in src/trigger_bitbucket_pipeline_from_s3.py is pretty straightforward for now:
import logging
import requests
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info(f'## EVENT: {event}')
return {
'statusCode': 200,
}
What am I doing wrong? I have already double checked file names.
That is because there is no module named 'requests' in lambda, remembrer lambda is serverless so you need to configure all your dependencies before you run it.
One way to solve this is to install that dependency locally in your project:
pip install requests -t ./
Then create again the .zip file (with the dependency in it) and upload to your lambda function.
And other way to solve it is to use a custom layer in AWS lambda that contains the relevant 'requests' site-packages you require. Example:
https://dev.to/razcodes/how-to-create-a-lambda-layer-in-aws-106m
You typically receive this error when your Lambda environment can't find the specified library in the Python code. This is because Lambda isn't prepackaged with all Python libraries.
To resolve this error, create a deployment package or Lambda layer that includes the libraries that you want to use in your Python code for Lambda.
Make sure that you put the library that you import for Python inside the /python folder.
In your local environment install all library files into the python folder by running the following:
pip install librarywhatyouneed -t python/
There are dependencies to create all python libraries prepackaged and zip the python with all dependencies and put into the layer associated with add Layer lambda created on AWS.
I am trying to run AWS CLI commands on a Lambda function. I referred to How to use AWS CLI within a Lambda function (aws s3 sync from Lambda) :: Ilya Bezdelev and generated a zip file with the awscli package. When I try to run the lambda function I get the following error:
START RequestId: d251660b-4998-4061-8886-67c1ddbbc98c Version: $LATEST
[INFO] 2020-06-22T19:15:45.232Z d251660b-4998-4061-8886-67c1ddbbc98c
Running shell command: /opt/aws --version
Traceback (most recent call last):
File "/opt/aws", line 19, in <module>
import awscli.clidriver
ModuleNotFoundError: No module named 'awscli'
What could be the issue here?
Everything in the 'site-packages' folder needs to be directly in the zip, and subsequently the /opt/ folder for the lambda, NOT nested inside a 'site-packages' folder which is what the tutorial results in unfortunately when you use his commands verbatim.
AS #tvmaynard said, you first need to add all the packages inside the same path as aws script of the AWS-CLI, by using this command:
cp -r ../${VIRTUAL_ENV_DIR}/lib/python${PYTHON_VERSION}/site-packages/. .
But, Even after that you will face a problem that there is some libraries that AWS-CLI is dependent on and must be installed in the Runtime Python as PyYAML, to install it you need to have access to Python Runtime inside the lambda, which is Not allowed.
Even if, you try to solve this by telling the interpreter where to search for the PyYAML library and installed it inside /tmp/, as follow:
import sys
import subprocess
stderr = subprocess.PIPE
stdout = subprocess.PIPE
cmd_1 = "pip install PyYAML -t /tmp/ --no-cache-dir"
subprocess.Popen(
args=cmd_1, start_new_session=True, shell=True, text=True, cwd=None,
stdout=stdout, stderr=stderr
)
sys.path.insert(1, '/tmp/')
import yaml
You will be able to use the library by importing it only inside the lambda function context as if you added a layer to the lambda, But not from the underlying command line which is linked to the python interpreter inside the lambda Runtime and have a default path to search for the libraries it needs.
You will also, Pay More Money to install this and the may needed other libraries, every time you trigger your lambda, Which if on Production ENV, will add more money to your Bill, that doesn't generate value.
I know the concept of using a deployment package is relatively straightforward, but I've been banging my head on this issue for the last few hours. I am following the documentation from AWS on packaging up Lambda dependencies. I want to write a simple Lambda function to update an entry in a PostgreSQL table upon some event.
I first make a new directory to work in:
mkdir lambdas-deployment && cd lambdas-deployment
Then I make a new virtual environment and install my packages:
virtualenv v-env
source v-env/bin/activate
pip3 install sqlalchemy boto3 psycopg2
My trigger-yaml-parse.py function (it doesn't actually use the sqlalchemy library yet, but I'm just trying to import it successfully):
import logging
import json
import boto3
import sqlalchemy
def lambda_handler(event, context):
records = event['Records']
s3_records = filter(lambda record: record['eventSource'] == 'aws:s3', records)
object_created_records = filter(lambda record: record['eventName'].startswith('ObjectCreated'), s3_records)
for record in object_created_records:
key = record['s3']['object']['key']
print(key)
I've been following the instructions in the AWS documentation.
zip -r trigger-yaml-parse.zip $VIRTUAL_ENV/lib/python3.6/site-packages/
I then add in my function code:
zip -g trigger-yaml-parse.zip trigger-yaml-parse.py
I get an output of updating: trigger-yaml-parse.py (deflated 48%).
Then I upload my new zipped deployment to my S3 build bucket:
aws s3 cp trigger-yaml-parse.zip s3://lambda-build-bucket
I choose upload from S3 in the AWS Lambda console:
However, my Lambda function fails upon execution with the error:
START RequestId: 396c6c3c-3f5b-4df9-b7f1-057842a87eb3 Version: $LATEST
Unable to import module 'trigger-yaml-parse': No module named 'sqlalchemy'
What am I doing wrong? I've followed the documentation from AWS literally step for step.
I think your problem might be in this line:
zip -r trigger-yaml-parse.zip $VIRTUAL_ENV/lib/python3.6/site-packages/
When you create the zip file the compressed files will have the complete path you had in your disk. The python runtime in lambda will not be able to find the libraries.
Instead you should do something like this
cd $VIRTUAL_ENV/lib/python3.6/site-packages/
zip -r /full/path/to/trigger-yaml-parse.zip .
Run unzip -t against both files and you will see the difference.
from AWS documentation:
"Zip packages uploaded with incorrect permissions may cause execution
failure. AWS Lambda requires global read permissions on code files and
any dependent libraries that comprise your deployment package"
So you can use zip info to check permissions:
zipinfo trigger-yaml-parse.zip
-r-------- means only the file owner has permissions.
I need to do a rest-call within a python script, that runs once per day.
I can't pack the "requests" package into my python-package using the AWS Lambdas. I get the error: "Unable to import module 'lambda_function': No module named lambda_function"
I broke it down to the hello_world predefined script. I can pack it into a zip and upload it. Everything works. As soon as I put "import requests" into the file, I get this error.
Here is what I already did:
The permissions of the zip and the project folder (including subfolders) are set to `chmod 777`. So permissions shouldn't be a problem.
The script itself is within the root folder. When you open the zip file, you directly see it.
I installed the requests package into the root-folder of the project using `sudo pip install requests -t PATH_TO_ROOT_FOLDER`
The naming of everything looks like this:
zip-file: lambda_function.zip
py-file: lambda_function.py
handler method: lambda_handler(event, context)
handler-definition in the "webconfig: lambda_function.lambda_handler
The file I want to run in the end looks like this:
import requests
import json
def lambda_handler(event, context):
url = 'xxx.elasticbeanstalk.com/users/login'
headers = {"content-type": "application/json", "Authorization": "Basic Zxxxxxxxxx3NjxxZxxxxzcw==" }
response = requests.put(url, headers=headers, verify=False)
return 'hello lambda_handler'
I'm glad for ANY kind of help. I already used multiple hours on this issue.
EDIT: On Oct-21-2019 Botocore removed the vendored version of requests: https://github.com/boto/botocore/pull/1829.
EDIT 2: (March 10, 2020): The deprecation date for the Lambda service to bundle the requests module in the AWS SDK is now January 30, 2021. https://aws.amazon.com/blogs/compute/upcoming-changes-to-the-python-sdk-in-aws-lambda/
EDIT 3: (Nov 22, 2022): AWS cancelled the deprecation so you can continue to use requests as described below. AWS Blog
To use requests module, you can simply import requests from botocore.vendored. For example:
from botocore.vendored import requests
def lambda_handler(event, context):
response = requests.get("https://httpbin.org/get", timeout=10)
print(response.json())
you can see this gist to know more modules that can be imported directly in AWS lambda.
If you're working with Python on AWS Lambda, and need to use requests, you better use urllib3, it is currently supported on AWS Lambda and you can import it directly, check the example on urllib3 site.
import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://httpbin.org/robots.txt')
r.data
# b'User-agent: *\nDisallow: /deny\n'
r.status
# 200
I finally solved the problem: The structure in my zip file was broken. It is important that the python script and the packed dependencies (as folders) are in the root of the zip file. This solved my problem.
It's a bit depressing if you find such easy errors after hours of try and failure.
I believe you have lambda_function.py on the Lambda console. You need to first create the Lambda function deployment package, and then use the console to upload the package.
You create a directory, for example project-dir on your system (locally)
create lambda_function.py in project-dir, copy the content of lambda_function.py from lambda console and paste it in project-dir/lambda_function.py
pip install requests -t /path/to/project-dir
Zip the content of the project-dir directory, which is your deployment package (Zip the directory content, not the directory)
Go to the Lambda console, select upload zip file in code entry type and upload your deployment package. Import requests should work without any error.
With this command download the folder package
pip install requests -t .
Run this command on your local machine, then zip your working directory, then upload to aws.
Most of the comments somehow correct, but not enough informative for AWS beginners. Here is my long resume what needs to be done for accessing requests functionality:
1. Creates root folder for AWS Lambda function
% mkdir lambda-function
2. Go inside crated root folder
% cd lambda-function
3. Create entry point Python file for AWS Lambda.
% vi lambda_function.py
4. Paste a code into lambda_function.py
import requests
def lambda_handler(event, context):
response = requests.get("https://www.test.com/")
print(response.text)
return response.text
5. Install requests library. Note:package folder created
% pip install --target ./package requests
6. Go inside package
% cd package
7. Zip package
zip -r ../deployment-package.zip .
8. Go into parent folder
% cd ..
9. Zip deployment packge and lambda function file
% zip -g deployment-package.zip lambda_function.py
In the AWS Lambda functions tap "Upload from" and pick ".zip file". Navigate to your zip package zip file: deployment-package.zip.
After upload all files will be inside AWS Lambda function.
python 3.8 windows 10
lambda is looking for a specific folder structure and we are going to recreate in this manner in the steps below (https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-create):
make a folder on your desktop called "python," open a cmd terminal: cd desktop
pip install --target python requests
right click your python folder and zip it and rename the zip to 'requests.zip' - now if you look inside the zip you should see the python folder.
aws console > lambda > layers > create layer => name layer/upload requests.zip
aws console > functions > create function => in the "designer" box select layers and then "add layers." Choose custom layers and select
your layer.
Go back to the function screen by clicking on the lambda symbol in the designer box. Now you can see "function code" again. Click lambda_function.py
Now you can import requests like this:
import json
import requests
def lambda_handler(event, context):
# TODO implement
response = requests.get('your_URL')
return {
'statusCode': 200,
'body': json.dumps(response.json())
}
Copy whatever you have in the lambda_function fron AWS lambda console and paste it in a new python script and save it as lambda_function.py.
Make a new folder (I name it as package) and save requests module in it by running the following code in terminal: pip install -t package requests
Move lambda_function.py into the folder (package).
Go to the folder and select all content and zip them.
Go back to the AWS Lambda console. select the function and under the Function code section, click on 'Action' (on the right side) and select Upload a .zip file.
Upload the folder. lambda_function should be uploaded automatically.
Run and Enjoy.
Add a layer to your lambda function
by specifying this arn (ap-south-1)
arn:aws:lambda:ap-south-1:770693421928:layer:Klayers-p38-requests-html:10