We Keep Coding

Python Lambda missing dependencies when set up through Amplify

I've been trying to configure an Amplify project with a Python based Lambda backend API.
I have followed the tutorials by creating an API through the AWS CLI and installing all the dependencies through pipenv.
When I cd into the function's directory, my Pipfile looks like this:
name = "pypi"
url = "https://pypi.python.org/simple"
verify_ssl = true
[dev-packages]
[packages]
src = {editable = true, path = "./src"}
flask = "*"
flask-cors = "*"
aws-wsgi = "*"
boto3 = "*"
[requires]
python_version = "3.8"
And when I run amplify push everything works and the Lambda Function gets created successfully.
Also, when I run the deploy pipeline from the Amplify Console, I see in the build logs that my virtual env is created and my dependencies are downloaded.
Something else that was done based on github issues (otherwise build would definitely fail) was adding the following to amplify.yml:
backend:
phases:
build:
commands:
- ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
- ln -fs /usr/local/bin/python3.8 /usr/bin/python3
- pip3 install --user pipenv
- amplifyPush --simple
Unfortunately, from the Lambda's logs (both dev and prod), I see that it fails importing every dependency that was installed through Pipenv. I added the following in index.py:
import os
os.system('pip list')
And saw that NONE of my dependencies were listed so I was wondering if the Lambda was running through the virtual env that was created, or was just using the default Python.
How can I make sure that my Lambda is running the virtualenv as defined in the Pipfile?

Lambda functions do not run in a virtualenv. Amplify uses pipenv to create a virtualenv and download the dependencies. Then Amplify packages those dependencies, along with the lambda code, into a zip file which it uploads to AWS Lambda.
Your problem is either that the dependencies are not packaged with your function or that they are packaged with a bad directory structure. You can download the function code to see exactly how the packaging went.

Python serverless: ModuleNotFoundError

I'm trying to use serverless framework with a python project.
I created a hello world example that I run in offline mode. It works well but when I try to import a python package I get ModuleNotFoundError.
Here is my serverless.yaml file:
service: my-test
frameworkVersion: "3"
provider:
name: aws
runtime: python3.8
functions:
hello:
handler: lambdas.hello.hello
events:
- http:
path: /hello
method: get
plugins:
- serverless-python-requirements
- serverless-offline
In lambdas.hello.py:
import json
import pandas
def hello(event, context):
body = {
"message": 'hello world',
}
response = {"statusCode": 200, "body": json.dumps(body)}
return response
In my Pipfile:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
pandas = "*"
[requires]
python_version = "3.8"
To run it, I use the command $ sls offline start
Then When I query on postman http://localhost:3000/dev/hello I get the error ModuleNotFoundError.
If I remove the line import pandasin hello.py file, it works.
I don't understand why I get this error as serverless-python-requirements is supposed to check the pipfile and pandas is in my pipfile.
How can I use pandas (or any other python package) in my lambdas with serverless framework in offline mode ?

The serverless-python-requirements plugin is used to bundle your dependencies and package them for deployment. This only comes to effect when you run sls deploy.
From the plugin page -
The plugin will now bundle your python dependencies specified in your requirements.txt or Pipfile when you run sls deploy
Read more about python packaging here - https://www.serverless.com/blog/serverless-python-packaging
Since you are running your service locally, this plugin will not be used.
Your dependencies need to be installed locally.
perform the below steps to make it work -
Create a virtual environment in you serverless directory.
install the plugin serverless plugin install -n serverless-offline
install pandas using pip
run sls offline start

Your lambda function don't have the panda module installed
You need to use the serverless-python-requirements plugin : https://www.serverless.com/plugins/serverless-python-requirements. To use it you need docker on your machine and to create a requirement.txt file in your service with the packages you need in your lambda

problem with command heroku run -a <name of app> pipenv run upgrade

I have made a code with python flask and I am following the next steps to deploy it:
Deploying to Heroku (takes 7 minutes)
Install heroku (if you don't have it yet)
$ npm i heroku -g
Login to heroku on the command line (if you have not already)
$ heroku login -i
Create an application (if you don't have it already)
$ heroku create <your_application_name>
Enviroment Variables (takes 2 minutes)
Now navigate to your heroku dashboard and look for your application settings, we have to manually add our environment variables into heroku:
You cannot create a .env file on Heroku, instead you need to manually create all the variables under your project settings.
Open your .env file and copy and paste each variable (FLASK_APP, DB_CONNECTION_STRING, etc.) to Heroku.
Deploying your database to Heroku (takes 3 minutes)
You local MySQL Database now has to be uploaded to a cloud, there are plenty of services that provide MySQL database hosting but we recommend JawDB because it has a Free Tier, its simple and 100% integrated with Heroku.
Go to your heroku project dashboard and look to add a new heroku add-on.
Look for JawDB MySQL and add it to your project (it may ask for a Credit Card but you will not be charged as long as your remain within 5mb database size, enough for your demo.
Once JawDB is added to your project look for the Connection String inside your JawDB dashboard, something like:
mysql://tqqa0ui0cga32nxd:eqi8nchjbpwth82v#c584md9egjnm02sk.5btxwkvyhwsf.us-east-1.rds.amazonaws.com:3306/45fds423rbtbr
Copy the connection string and create a new environment variable on your project settings.
Run migrations on heroku: After your database is connected, you have to create the tables and structure, you can do that by running the pipenv run upgrade command on the production server like this:
$ heroku run -a=<your_app_name> pipenv run upgrade
:warning: Note: Notice that you have to replace <your app name> with your application name, you also have to be logged into heroku in your terminal (you can do that by typing heroku login -i)
Push to the Heroku codebase
Commit and push to heroku, make sure you have added and committed your changes and push to heroku
$ git push heroku main hh
That is it!
Now the problem is when I run this command:
heroku run -a=<your_app_name> pipenv run upgrade
And the respons is:
bash: pipenv: command not found
This is my Pipfile.txt:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
flask = "*"
sqlalchemy = "*"
flask-sqlalchemy = "*"
flask-migrate = "*"
flask-swagger = "*"
psycopg2-binary = "*"
python-dotenv = "*"
mysql-connector-python = "*"
flask-cors = "*"
gunicorn = "*"
mysqlclient = "*"
flask-admin = "*"
cloudinary = "*"
flask-login = "*"
pipenv = "*"
[requires]
python_version = "3.8"
[scripts]
start="flask run -p 3000 -h 0.0.0.0"
init="flask db init"
migrate="flask db migrate"
upgrade="flask db upgrade"
deploy="echo 'Please follow this 3 steps to deploy: https://github.com/4GeeksAcademy/flask-rest-hello/blob/master/README.md#deploy-your-website-to-heroku' "
These are the commands I run before deploying:
pipenv install;
mysql -u root -e "CREATE DATABASE example";
pipenv run init;
pipenv run migrate;
pipenv run upgrade;
If I don't run the upgrade in heroku this is what Release Log in heroku:
sqlalchemy.exc.DatabaseError: (mysql.connector.errors.DatabaseError) 2003 (HY000): Can't connect to MySQL server on 'localhost' (111)
(Background on this error at: http://sqlalche.me/e/14/4xp6)

It seems you are missing the pipenv tool or missing in PATH. You may install it using:
$ pip install pipenv
If pipenv already is installed check the PATH variable and if you are able to locate pipenv using $ which pipenv. Have a look at the docs regarding PATH:
https://pipenv-fork.readthedocs.io/en/latest/advanced.html

Install dependency’s extras if package’s specific extras is requested

My project has Celery as a dependency. It’s a hard dependencies, ie. my project can’t live without it. However, it can use Redis as its backend, which my app doesn’t need specifically.
I want my package to be set up so if a user installs dependencies with poetry install -E redis, it would install the redis block of Celery (as if it were specified in pyproject.toml as celery = { version="^4.4.0", extras=["redis"] }).
However, if a user uses a plain poetry install (without -E redis), i don’t want Celery’s Redis dependencies (as if it were only specified as celery = "^4.4.0") to be installed.
Is there a way to put this into Poetry config? Or should I track the optional requirements of celery[redis] and manually add them to my pyproject.toml file?
I already checked the Poetry documentation on this matter, but it doesn’t offer a way to specify the same dependency (celery in my case) with different options.

This should work by defining redis as an optional extra, e.g.:
[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = ""
authors = ["finswimmer <finswimmer#example.org>"]
[tool.poetry.dependencies]
python = "^3.6"
celery = "^4.4.7"
redis = { version = "^3.5.3", optional = true }
[tool.poetry.dev-dependencies]
[tool.poetry.extras]
redis = ["redis"]
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"

How do I configure pip.conf in AWS Elastic Beanstalk?

I need to deploy a Python application to AWS Elastic Beanstalk, however this module requires dependencies from our private PyPi index. How can I configure pip (like what you do with ~/.pip/pip.conf) so that AWS can connect to our private index while deploying the application?
My last resort is to modify the dependency in requirements.txt to -i URL dependency before deployment, but there must be a clean way to achieve this goal.

In .ebextensions/files.config add something like this:
files:
"/opt/python/run/venv/pip.conf":
mode: "000755"
owner: root
user: root
content: |
[global]
find-links = <URL>
trusted-host = <HOST>
index-url = <URL>
Or whatever other configurations you'd like to set in your pip.conf. This will place the pip.conf file in the virtual environment of your application, which will be activated before pip -r requirements.txt is executed. Hopefully this helps!