I have done pip install --target ./package retry and then packaged my code and ./package folder in to a zip and uploaded it to AWS lambda. However, from package.retry import retry results in :
{ "errorMessage": "Unable to import module
'lambda_function_update_error_table': No module named 'packageretry'",
"errorType": "Runtime.ImportModuleError" }
But, if the retry folder which is installed through pip is in root directory, from retry import retry works fine.
How can I solve this issue? Thanks in advance!!
Try downloading the .whl file and then after manually extracting it again zip it with your lambda code. now once you deployment package is ready upload it to the Lambda function and then try again.
you can get the .whl file form https://pypi.org/search/?q=retry
Related
So I have this similar problem with this person.
How to create password encrypted zip file in python through AWS lambda
We have the exact same problem but i already did everything from the answers in that thread but to no avail.
I have a lambda script that runs on python3.9 I need to compress the files in my s3 as a zip file that is password protected and i need to put it in another s3.
This is how it goes
import pyminizip
def zip_to_client():
# reportTitles = os.listdir(tempDir)
dateGenerated = datetime.now(tz=atz).strftime("%Y-%m-%d")
pyminizip.compress("Daily_Booking_Report.csv", subfolder + str(dateGenerated) +'/'+str(id)+'/'
, "/tmp/test.zip", "awesomepassword", 9)
s3 = boto3.resource('s3')
s3.meta.client.upload_file(Filename = '/tmp/test.zip', Bucket = bucket, Key = subfolder + 'test.zip', ExtraArgs={'Tagging':'archive=90days'})
print("SUCCESS: Transferred report into S3")
i'm not sure if it works but i can't debug it because lambda shows me the error:
Response
{
"errorMessage": "Unable to import module 'lambda_function': No module named 'pyminizip'",
"errorType": "Runtime.ImportModuleError",
"requestId": "0000111000",
"stackTrace": []
}
I made sure that i put import pyminizip as well as pip installing it in the directory.
pip install pyminizip -t .
so far this is what the lambda directory looks like
https://ibb.co/ZGmLBbv
i've tried everything from putting it in a lambda layer to pip installing different versions from python 3.7 to 3.9
This is a common case when you create a lambda layer and get import error. And this occurs when you don't have created python files in a defined directory like python/python38/site-packages...
or
second reason might be a dependency is missing. In that use use docker and follow steps from here : https://www.geeksforgeeks.org/how-to-install-python-packages-for-aws-lambda-layers/.
When I check the cloud watch logs of my Lambda function, I see theses errors:
[ERROR] Runtime.ImportModuleError: Unable to import module 'trigger_bitbucket_pipeline_from_s3': No module named 'requests'
File structure:
/bin
--trigger_bitbucket_pipeline_from_s3.zip
/src
--trigger_bitbucket_pipeline_from_s3.py
--/requests (lib folder)
lambda.tf
Lambda.tf:
data "archive_file" "lambda_zip" {
type = "zip"
source_file = "${path.module}/src/trigger_bitbucket_pipeline_from_s3.py"
output_file_mode = "0666"
output_path = "${path.module}/bin/trigger_bitbucket_pipeline_from_s3.zip"
}
resource "aws_lambda_function" "processing_lambda" {
filename = data.archive_file.lambda_zip.output_path
function_name = "triggering_pipleline_lambda"
handler = "trigger_bitbucket_pipeline_from_s3.lambda_handler"
source_code_hash = data.archive_file.lambda_zip.output_base64sha256
role = aws_iam_role.processing_lambda_role.arn
runtime = "python3.9"
}
My lambda function in src/trigger_bitbucket_pipeline_from_s3.py is pretty straightforward for now:
import logging
import requests
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info(f'## EVENT: {event}')
return {
'statusCode': 200,
}
What am I doing wrong? I have already double checked file names.
That is because there is no module named 'requests' in lambda, remembrer lambda is serverless so you need to configure all your dependencies before you run it.
One way to solve this is to install that dependency locally in your project:
pip install requests -t ./
Then create again the .zip file (with the dependency in it) and upload to your lambda function.
And other way to solve it is to use a custom layer in AWS lambda that contains the relevant 'requests' site-packages you require. Example:
https://dev.to/razcodes/how-to-create-a-lambda-layer-in-aws-106m
You typically receive this error when your Lambda environment can't find the specified library in the Python code. This is because Lambda isn't prepackaged with all Python libraries.
To resolve this error, create a deployment package or Lambda layer that includes the libraries that you want to use in your Python code for Lambda.
Make sure that you put the library that you import for Python inside the /python folder.
In your local environment install all library files into the python folder by running the following:
pip install librarywhatyouneed -t python/
There are dependencies to create all python libraries prepackaged and zip the python with all dependencies and put into the layer associated with add Layer lambda created on AWS.
I have a python3.6 script that uses sqlalchemy, pandas and numpy. To get this working on AWS Lambda, I took the following steps.
Created a new, clean directory
Create a new virtualenv
Create a holding directory (mkdir dist)
Install packages pip install sqlalchemy numpy pandas
Navigate to packages cd env/lib/python3.6/site-packages
Zip packages to holding directory zip -r path/dist/Transfer.zip .
Navigate to root
Zip python file zip -g dist/Transfer.zip my_python.py
Upload to S3
Direct Lambda > Configuration > Code entry type > Upload a file from S3 > path to my file
Set Handler to my_python.lambda_handler
Save and test
I always get the error
{
"errorMessage": "Unable to import module 'my_python'"
}
With the logs as
Unable to import module 'heap_consolidation_lambda': Missing required dependencies ['numpy']
Why can it not see numpy? Fwiw, numpy is the third import, so apparently it has no issues with sqlalchemy and pandas.
I am trying to upload a python lambda function with zipped dependencies but for some reason I am constantly getting
"errorMessage": "Unable to import module 'CreateThumbnail'"
whenever I test it.
Here are the steps I took which were almost identical to these docs.
Created and activate a virtualenv with virtualenv ~/lambda_env and source ~/lambda_env/bin/activate
Install Pillow and boto3 with pip install Pillow and pip install boto3
Zip dependencies with cd $VIRTUAL_ENV/lib/python2.7/site-packages and zip -r9 ~/CreateThumbnail.zip *
Add the actual python lambda function to the zip file with zip -g ~/CreateThumbnail.zip CreateThumbnail.py where CreateThumbnail.py is
from __future__ import print_function
import boto3
import os
import sys
import uuid
from PIL import Image
import PIL.Image
s3_client = boto3.client('s3')
def resize_image(image_path, resized_path):
with Image.open(image_path) as image:
image.thumbnail(tuple(x / 2 for x in image.size))
image.save(resized_path)
def handler(event, context):
for record in event['Records']:
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
upload_path = '/tmp/resized-{}'.format(key)
s3_client.download_file(bucket, key, download_path)
resize_image(download_path, upload_path)
s3_client.upload_file(upload_path, '{}resized'.format(bucket), key)
Then in the console I set the handler to be CreateThumbnail.handler
Then I upload CreateThumbnail.zip via the aws console and click 'save & test' I get
"errorMessage": "Unable to import module 'CreateThumbnail'"
I am very confused by this because feel like I am following the docs. Can anyone tell me what I am doing wrong here?
Perhaps check out the lambda-uploader project... It handles the packaging of dependencies and is config based.
https://github.com/rackerlabs/lambda-uploader/
Also these links may be helpful:
http://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html
https://markn.ca/2015/10/python-extension-modules-in-aws-lambda/
http://www.perrygeo.com/running-python-with-compiled-code-on-aws-lambda.html
The problem lies in the packaging hierarchy. After you install the dependencies, zip the lambda function as follows (in the example below, lambda_function is the name of my function)
Try this:
pip install requests -t .
zip -r9 lambda_function.zip .
zip -g lambda_function.zip lambda_function.py
Do not let your browser automatically unzip the lambda "project" file after downloading. This seems to corrupt the file when it is re-zipped and used.
The tutorial you pointed out uses python 3.8
And you seem to be using python 2.7
That may be the reason.
I am doing a similar tutorial, but they give us the zip ready to upload but warning to select python 3.7 and not 3.8 or it will fail to run correctly.
I need to do a rest-call within a python script, that runs once per day.
I can't pack the "requests" package into my python-package using the AWS Lambdas. I get the error: "Unable to import module 'lambda_function': No module named lambda_function"
I broke it down to the hello_world predefined script. I can pack it into a zip and upload it. Everything works. As soon as I put "import requests" into the file, I get this error.
Here is what I already did:
The permissions of the zip and the project folder (including subfolders) are set to `chmod 777`. So permissions shouldn't be a problem.
The script itself is within the root folder. When you open the zip file, you directly see it.
I installed the requests package into the root-folder of the project using `sudo pip install requests -t PATH_TO_ROOT_FOLDER`
The naming of everything looks like this:
zip-file: lambda_function.zip
py-file: lambda_function.py
handler method: lambda_handler(event, context)
handler-definition in the "webconfig: lambda_function.lambda_handler
The file I want to run in the end looks like this:
import requests
import json
def lambda_handler(event, context):
url = 'xxx.elasticbeanstalk.com/users/login'
headers = {"content-type": "application/json", "Authorization": "Basic Zxxxxxxxxx3NjxxZxxxxzcw==" }
response = requests.put(url, headers=headers, verify=False)
return 'hello lambda_handler'
I'm glad for ANY kind of help. I already used multiple hours on this issue.
EDIT: On Oct-21-2019 Botocore removed the vendored version of requests: https://github.com/boto/botocore/pull/1829.
EDIT 2: (March 10, 2020): The deprecation date for the Lambda service to bundle the requests module in the AWS SDK is now January 30, 2021. https://aws.amazon.com/blogs/compute/upcoming-changes-to-the-python-sdk-in-aws-lambda/
EDIT 3: (Nov 22, 2022): AWS cancelled the deprecation so you can continue to use requests as described below. AWS Blog
To use requests module, you can simply import requests from botocore.vendored. For example:
from botocore.vendored import requests
def lambda_handler(event, context):
response = requests.get("https://httpbin.org/get", timeout=10)
print(response.json())
you can see this gist to know more modules that can be imported directly in AWS lambda.
If you're working with Python on AWS Lambda, and need to use requests, you better use urllib3, it is currently supported on AWS Lambda and you can import it directly, check the example on urllib3 site.
import urllib3
http = urllib3.PoolManager()
r = http.request('GET', 'http://httpbin.org/robots.txt')
r.data
# b'User-agent: *\nDisallow: /deny\n'
r.status
# 200
I finally solved the problem: The structure in my zip file was broken. It is important that the python script and the packed dependencies (as folders) are in the root of the zip file. This solved my problem.
It's a bit depressing if you find such easy errors after hours of try and failure.
I believe you have lambda_function.py on the Lambda console. You need to first create the Lambda function deployment package, and then use the console to upload the package.
You create a directory, for example project-dir on your system (locally)
create lambda_function.py in project-dir, copy the content of lambda_function.py from lambda console and paste it in project-dir/lambda_function.py
pip install requests -t /path/to/project-dir
Zip the content of the project-dir directory, which is your deployment package (Zip the directory content, not the directory)
Go to the Lambda console, select upload zip file in code entry type and upload your deployment package. Import requests should work without any error.
With this command download the folder package
pip install requests -t .
Run this command on your local machine, then zip your working directory, then upload to aws.
Most of the comments somehow correct, but not enough informative for AWS beginners. Here is my long resume what needs to be done for accessing requests functionality:
1. Creates root folder for AWS Lambda function
% mkdir lambda-function
2. Go inside crated root folder
% cd lambda-function
3. Create entry point Python file for AWS Lambda.
% vi lambda_function.py
4. Paste a code into lambda_function.py
import requests
def lambda_handler(event, context):
response = requests.get("https://www.test.com/")
print(response.text)
return response.text
5. Install requests library. Note:package folder created
% pip install --target ./package requests
6. Go inside package
% cd package
7. Zip package
zip -r ../deployment-package.zip .
8. Go into parent folder
% cd ..
9. Zip deployment packge and lambda function file
% zip -g deployment-package.zip lambda_function.py
In the AWS Lambda functions tap "Upload from" and pick ".zip file". Navigate to your zip package zip file: deployment-package.zip.
After upload all files will be inside AWS Lambda function.
python 3.8 windows 10
lambda is looking for a specific folder structure and we are going to recreate in this manner in the steps below (https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-create):
make a folder on your desktop called "python," open a cmd terminal: cd desktop
pip install --target python requests
right click your python folder and zip it and rename the zip to 'requests.zip' - now if you look inside the zip you should see the python folder.
aws console > lambda > layers > create layer => name layer/upload requests.zip
aws console > functions > create function => in the "designer" box select layers and then "add layers." Choose custom layers and select
your layer.
Go back to the function screen by clicking on the lambda symbol in the designer box. Now you can see "function code" again. Click lambda_function.py
Now you can import requests like this:
import json
import requests
def lambda_handler(event, context):
# TODO implement
response = requests.get('your_URL')
return {
'statusCode': 200,
'body': json.dumps(response.json())
}
Copy whatever you have in the lambda_function fron AWS lambda console and paste it in a new python script and save it as lambda_function.py.
Make a new folder (I name it as package) and save requests module in it by running the following code in terminal: pip install -t package requests
Move lambda_function.py into the folder (package).
Go to the folder and select all content and zip them.
Go back to the AWS Lambda console. select the function and under the Function code section, click on 'Action' (on the right side) and select Upload a .zip file.
Upload the folder. lambda_function should be uploaded automatically.
Run and Enjoy.
Add a layer to your lambda function
by specifying this arn (ap-south-1)
arn:aws:lambda:ap-south-1:770693421928:layer:Klayers-p38-requests-html:10