I've been trying to configure an Amplify project with a Python based Lambda backend API.
I have followed the tutorials by creating an API through the AWS CLI and installing all the dependencies through pipenv.
When I cd into the function's directory, my Pipfile looks like this:
name = "pypi"
url = "https://pypi.python.org/simple"
verify_ssl = true
[dev-packages]
[packages]
src = {editable = true, path = "./src"}
flask = "*"
flask-cors = "*"
aws-wsgi = "*"
boto3 = "*"
[requires]
python_version = "3.8"
And when I run amplify push everything works and the Lambda Function gets created successfully.
Also, when I run the deploy pipeline from the Amplify Console, I see in the build logs that my virtual env is created and my dependencies are downloaded.
Something else that was done based on github issues (otherwise build would definitely fail) was adding the following to amplify.yml:
backend:
phases:
build:
commands:
- ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
- ln -fs /usr/local/bin/python3.8 /usr/bin/python3
- pip3 install --user pipenv
- amplifyPush --simple
Unfortunately, from the Lambda's logs (both dev and prod), I see that it fails importing every dependency that was installed through Pipenv. I added the following in index.py:
import os
os.system('pip list')
And saw that NONE of my dependencies were listed so I was wondering if the Lambda was running through the virtual env that was created, or was just using the default Python.
How can I make sure that my Lambda is running the virtualenv as defined in the Pipfile?
Lambda functions do not run in a virtualenv. Amplify uses pipenv to create a virtualenv and download the dependencies. Then Amplify packages those dependencies, along with the lambda code, into a zip file which it uploads to AWS Lambda.
Your problem is either that the dependencies are not packaged with your function or that they are packaged with a bad directory structure. You can download the function code to see exactly how the packaging went.
Related
I have a Python app that takes the value of a certificate in a Dockerfile and updates it. However, I'm having difficulty knowing how to get the app to work within Gitlab.
When I push the app with the Dockerfile to be updated I want the app to run in the Gitlab pipeline and update the Dockerfile. I'm a little stuck on how to do this. I'm thinking that I would need to pull the repo, run the app and then push back up.
Would like some advice on if this is the right approach and if so how I would go about doing so?
This is just an example of the Dockerfile to be updated (I know this image wouldn't actually work, but the app would only update the ca-certificate present in the DF:
#syntax=docker/dockerfile:1
#init the base image
FROM alpine:3.15
#define present working directory
#WORKDIR /library
#run pip to install the dependencies of the flask app
RUN apk add -u \
ca-certificates=20211220 \
git=3.10
#copy all files in our current directory into the image
COPY . /library
EXPOSE 5000
#define command to start the container, need to make app visible externally by specifying host 0.0.0.0
CMD [ "python3", "-m", "flask", "run", "--host=0.0.0.0"]
gitlab-ci.yml:
stages:
- build
- test
- update_certificate
variables:
PYTHON_IMG: "python:3.10"
pytest_installation:
image: $PYTHON_IMG
stage: build
script:
- pip install pytest
- pytest --version
python_requirements_installation:
image: $PYTHON_IMG
stage: build
script:
- pip install -r requirements.txt
unit_test:
image: $PYTHON_IMG
stage: test
script:
- pytest ./tests/test_automated_cert_checker.py
cert_updater:
image: $PYTHON_IMG
stage: update_certificate
script:
- pip install -r requirements.txt
- python3 automated_cert_updater.py
I'm aware there's a lot of repetition with installing the requirements multiple times and that this is an area for improvement. I doesn't feel like it's necessary for the app to be built into an image because it's only used for updating the DF.
requirements.txt installs pytest and BeautifulSoup4
Additional context: The pipeline that builds the Dockerimage already exists and builds successfully. I am looking for a way to run this app once a day which will check if the ca-certificate is still up to date. If it isn't then the app is run, the ca-certificate in the Dockerfile is updated and then the updated Dockerfile is re built automatically.
My thoughts are that I may need to set the gitlab-ci.yml up pull the repo, run the app (that updates the ca-certificate) and then re push it, so that a new image is built based upon the update to the certificate.
The Dockerfile shown here is just a basic example showing that the actual DF in the repo looks like.
What you probably want to do is identify the appropriate version before you build the Dockerfile. Then, pass a --build-arg with the ca-certificates version. That way, if the arg changes, then the cached layer becomes invalid and will install the new version. But if the version is the same, the cached layer would be used.
FROM alpine:3.15
ARG CA_CERT_VERSION
RUN apk add -u \
ca-certificates=$CA_CERT_VERSION \
git=3.10
# ...
Then when you build your image, you should figure out the appropriate ca-certificates version and pass it as a build-arg.
Something like:
version="$(python3 ./get-cacertversion.py)" # you implement this
docker build --build-arg CA_CERT_VERSION=$version -t myimage .
Be sure to add appropriate bits to leverage docker caching in GitLab.
I'm trying to use serverless framework with a python project.
I created a hello world example that I run in offline mode. It works well but when I try to import a python package I get ModuleNotFoundError.
Here is my serverless.yaml file:
service: my-test
frameworkVersion: "3"
provider:
name: aws
runtime: python3.8
functions:
hello:
handler: lambdas.hello.hello
events:
- http:
path: /hello
method: get
plugins:
- serverless-python-requirements
- serverless-offline
In lambdas.hello.py:
import json
import pandas
def hello(event, context):
body = {
"message": 'hello world',
}
response = {"statusCode": 200, "body": json.dumps(body)}
return response
In my Pipfile:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
pandas = "*"
[requires]
python_version = "3.8"
To run it, I use the command $ sls offline start
Then When I query on postman http://localhost:3000/dev/hello I get the error ModuleNotFoundError.
If I remove the line import pandasin hello.py file, it works.
I don't understand why I get this error as serverless-python-requirements is supposed to check the pipfile and pandas is in my pipfile.
How can I use pandas (or any other python package) in my lambdas with serverless framework in offline mode ?
The serverless-python-requirements plugin is used to bundle your dependencies and package them for deployment. This only comes to effect when you run sls deploy.
From the plugin page -
The plugin will now bundle your python dependencies specified in your requirements.txt or Pipfile when you run sls deploy
Read more about python packaging here - https://www.serverless.com/blog/serverless-python-packaging
Since you are running your service locally, this plugin will not be used.
Your dependencies need to be installed locally.
perform the below steps to make it work -
Create a virtual environment in you serverless directory.
install the plugin serverless plugin install -n serverless-offline
install pandas using pip
run sls offline start
Your lambda function don't have the panda module installed
You need to use the serverless-python-requirements plugin : https://www.serverless.com/plugins/serverless-python-requirements. To use it you need docker on your machine and to create a requirement.txt file in your service with the packages you need in your lambda
I have made a code with python flask and I am following the next steps to deploy it:
Deploying to Heroku (takes 7 minutes)
Install heroku (if you don't have it yet)
$ npm i heroku -g
Login to heroku on the command line (if you have not already)
$ heroku login -i
Create an application (if you don't have it already)
$ heroku create <your_application_name>
Enviroment Variables (takes 2 minutes)
Now navigate to your heroku dashboard and look for your application settings, we have to manually add our environment variables into heroku:
You cannot create a .env file on Heroku, instead you need to manually create all the variables under your project settings.
Open your .env file and copy and paste each variable (FLASK_APP, DB_CONNECTION_STRING, etc.) to Heroku.
Deploying your database to Heroku (takes 3 minutes)
You local MySQL Database now has to be uploaded to a cloud, there are plenty of services that provide MySQL database hosting but we recommend JawDB because it has a Free Tier, its simple and 100% integrated with Heroku.
Go to your heroku project dashboard and look to add a new heroku add-on.
Look for JawDB MySQL and add it to your project (it may ask for a Credit Card but you will not be charged as long as your remain within 5mb database size, enough for your demo.
Once JawDB is added to your project look for the Connection String inside your JawDB dashboard, something like:
mysql://tqqa0ui0cga32nxd:eqi8nchjbpwth82v#c584md9egjnm02sk.5btxwkvyhwsf.us-east-1.rds.amazonaws.com:3306/45fds423rbtbr
Copy the connection string and create a new environment variable on your project settings.
Run migrations on heroku: After your database is connected, you have to create the tables and structure, you can do that by running the pipenv run upgrade command on the production server like this:
$ heroku run -a=<your_app_name> pipenv run upgrade
:warning: Note: Notice that you have to replace <your app name> with your application name, you also have to be logged into heroku in your terminal (you can do that by typing heroku login -i)
Push to the Heroku codebase
Commit and push to heroku, make sure you have added and committed your changes and push to heroku
$ git push heroku main hh
That is it!
Now the problem is when I run this command:
heroku run -a=<your_app_name> pipenv run upgrade
And the respons is:
bash: pipenv: command not found
This is my Pipfile.txt:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
flask = "*"
sqlalchemy = "*"
flask-sqlalchemy = "*"
flask-migrate = "*"
flask-swagger = "*"
psycopg2-binary = "*"
python-dotenv = "*"
mysql-connector-python = "*"
flask-cors = "*"
gunicorn = "*"
mysqlclient = "*"
flask-admin = "*"
cloudinary = "*"
flask-login = "*"
pipenv = "*"
[requires]
python_version = "3.8"
[scripts]
start="flask run -p 3000 -h 0.0.0.0"
init="flask db init"
migrate="flask db migrate"
upgrade="flask db upgrade"
deploy="echo 'Please follow this 3 steps to deploy: https://github.com/4GeeksAcademy/flask-rest-hello/blob/master/README.md#deploy-your-website-to-heroku' "
These are the commands I run before deploying:
pipenv install;
mysql -u root -e "CREATE DATABASE example";
pipenv run init;
pipenv run migrate;
pipenv run upgrade;
If I don't run the upgrade in heroku this is what Release Log in heroku:
sqlalchemy.exc.DatabaseError: (mysql.connector.errors.DatabaseError) 2003 (HY000): Can't connect to MySQL server on 'localhost' (111)
(Background on this error at: http://sqlalche.me/e/14/4xp6)
It seems you are missing the pipenv tool or missing in PATH. You may install it using:
$ pip install pipenv
If pipenv already is installed check the PATH variable and if you are able to locate pipenv using $ which pipenv. Have a look at the docs regarding PATH:
https://pipenv-fork.readthedocs.io/en/latest/advanced.html
My project has Celery as a dependency. It’s a hard dependencies, ie. my project can’t live without it. However, it can use Redis as its backend, which my app doesn’t need specifically.
I want my package to be set up so if a user installs dependencies with poetry install -E redis, it would install the redis block of Celery (as if it were specified in pyproject.toml as celery = { version="^4.4.0", extras=["redis"] }).
However, if a user uses a plain poetry install (without -E redis), i don’t want Celery’s Redis dependencies (as if it were only specified as celery = "^4.4.0") to be installed.
Is there a way to put this into Poetry config? Or should I track the optional requirements of celery[redis] and manually add them to my pyproject.toml file?
I already checked the Poetry documentation on this matter, but it doesn’t offer a way to specify the same dependency (celery in my case) with different options.
This should work by defining redis as an optional extra, e.g.:
[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = ""
authors = ["finswimmer <finswimmer#example.org>"]
[tool.poetry.dependencies]
python = "^3.6"
celery = "^4.4.7"
redis = { version = "^3.5.3", optional = true }
[tool.poetry.dev-dependencies]
[tool.poetry.extras]
redis = ["redis"]
[build-system]
requires = ["poetry>=1.0"]
build-backend = "poetry.masonry.api"
I need to deploy a Python application to AWS Elastic Beanstalk, however this module requires dependencies from our private PyPi index. How can I configure pip (like what you do with ~/.pip/pip.conf) so that AWS can connect to our private index while deploying the application?
My last resort is to modify the dependency in requirements.txt to -i URL dependency before deployment, but there must be a clean way to achieve this goal.
In .ebextensions/files.config add something like this:
files:
"/opt/python/run/venv/pip.conf":
mode: "000755"
owner: root
user: root
content: |
[global]
find-links = <URL>
trusted-host = <HOST>
index-url = <URL>
Or whatever other configurations you'd like to set in your pip.conf. This will place the pip.conf file in the virtual environment of your application, which will be activated before pip -r requirements.txt is executed. Hopefully this helps!