I have a Python Serverless project that uses a private Git (on Github) repo.
Requirements.txt file looks like this:
itsdangerous==0.24
boto3>=1.7
git+ssh://git#github.com/company/repo.git#egg=my_alias
Configurations of the project mainly looks like this
plugins:
- serverless-python-requirements
- serverless-wsgi
custom:
wsgi:
app: app.app
packRequirements: false
pythonRequirements:
dockerizePip: true
dockerSsh: true
When I deploy using this command:
sls deploy --aws-profile my_id --stage dev --region eu-west-1
I get this error:
Command "git clone -q ssh://git#github.com/company/repo.git /tmp/pip-install-a0_8bh5a/my_alias" failed with error code 128 in None
What am I doing wrong? I'm suspecting either the way I configured my SSH key for Github access or the configurations of the serverless package.
So the only way I managed to sort this issue was
Configure the SSH WITH NO PASSPHRASE. Following steps here.
In serverless.yml, I added the following:
custom:
wsgi:
app: app.app
packRequirements: false
pythonRequirements:
dockerizePip: true
dockerSsh: true
dockerSshSymlink: ~/.ssh
Notice I added dockerSshSymlink to report the location of the ssh files on my local machine; ~/.ssh.
In requirements.txt, I added my private dependency like this:
git+ssh://git#github.com/my_comp/my_repo.git#egg=MyRepo
All works.
Although not recommeneded. Have you tried using sudo sls deploy --aws-profile my_id --stage dev --region eu-west-1
This error can be also created by using the wrong password or ssh key.
Related
I'm trying to use serverless framework with a python project.
I created a hello world example that I run in offline mode. It works well but when I try to import a python package I get ModuleNotFoundError.
Here is my serverless.yaml file:
service: my-test
frameworkVersion: "3"
provider:
name: aws
runtime: python3.8
functions:
hello:
handler: lambdas.hello.hello
events:
- http:
path: /hello
method: get
plugins:
- serverless-python-requirements
- serverless-offline
In lambdas.hello.py:
import json
import pandas
def hello(event, context):
body = {
"message": 'hello world',
}
response = {"statusCode": 200, "body": json.dumps(body)}
return response
In my Pipfile:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
pandas = "*"
[requires]
python_version = "3.8"
To run it, I use the command $ sls offline start
Then When I query on postman http://localhost:3000/dev/hello I get the error ModuleNotFoundError.
If I remove the line import pandasin hello.py file, it works.
I don't understand why I get this error as serverless-python-requirements is supposed to check the pipfile and pandas is in my pipfile.
How can I use pandas (or any other python package) in my lambdas with serverless framework in offline mode ?
The serverless-python-requirements plugin is used to bundle your dependencies and package them for deployment. This only comes to effect when you run sls deploy.
From the plugin page -
The plugin will now bundle your python dependencies specified in your requirements.txt or Pipfile when you run sls deploy
Read more about python packaging here - https://www.serverless.com/blog/serverless-python-packaging
Since you are running your service locally, this plugin will not be used.
Your dependencies need to be installed locally.
perform the below steps to make it work -
Create a virtual environment in you serverless directory.
install the plugin serverless plugin install -n serverless-offline
install pandas using pip
run sls offline start
Your lambda function don't have the panda module installed
You need to use the serverless-python-requirements plugin : https://www.serverless.com/plugins/serverless-python-requirements. To use it you need docker on your machine and to create a requirement.txt file in your service with the packages you need in your lambda
Error:
Running "serverless" from node_modules
Deploying serverless-flask to stage dev (us-east-1)
✖ Stack serverless-flask-dev failed to deploy (0s)
Environment: darwin, node 16.0.0, framework 3.1.1 (local) 3.1.1v (global), plugin 6.0.0, SDK 4.3.1
Credentials: Local, "default" profile
Docs: docs.serverless.com
Support: forum.serverless.com
Bugs: github.com/serverless/serverless/issues
Error:
Error: spawn docker ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
at onErrorNT (node:internal/child_process:480:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
I'm following these instructions (https://www.serverless.com/blog/flask-python-rest-api-serverless-lambda-dynamodb/) and can't seem to figure this out since the base app is in python and not javascript... most people who have solved this solved it using javascript.
To solve this issue you need to update your serverless.yml file with these changes in the custom block
custom:
pythonRequirements:
pythonBin: python3
dockerizePip: "false"
I also face the same issue my issue was with dockerizePip it was set to
dockerizePip: non-linux
either remove this entry from serverless.yml file or just set it to false
To be able to deploy your project with serverless-python-requirements you need to have docker on your machine (if you are on windows consider using docker desktop or a linux vm)
Why do I need Docker ?
When you do a sls deploy, serverless-python-requirements launch a docker container to install all the dependencies you've put in your requirements.txt file that will be used during the deployement process
You are getting this error because your container is not launch correctly
I'm very new to DevOps, so this may be a very silly question. I'm trying to deploy a python Web scraping script onto an azure webapp using GitHub actions. This script is meant to be run for a long period of time as it is analyzing websites word by word for hours. It then logs the results to .log files.
I know a bit of how GitHub actions work, I know that I can trigger jobs when I push to the repo for instance. However, I'm a bit confused as to how one runs the app or a script on an azure resource (like a VM or webapp) for example. Does this process involve SSH-ing into the resource and then automatically run the cli command "python main.py" or "docker-compose up", or is there something more sophisticated involved?
For better context, this is my script inside of my workflows folder:
on:
[push]
env:
AZURE_WEBAPP_NAME: emotional-news-service # set this to your application's name
WORKING_DIRECTORY: '.' # set this to the path to your path of working directory inside GitHub repository, defaults to the repository root
PYTHON_VERSION: '3.9'
STARTUP_COMMAND: 'docker-compose up --build -d' # set this to the startup command required to start the gunicorn server. default it is empty
name: Build and deploy Python app
jobs:
build-and-deploy:
runs-on: ubuntu-latest
environment: dev
steps:
# checkout the repo
- uses: actions/checkout#master
# setup python
- name: Setup Python
uses: actions/setup-python#v1
with:
python-version: ${{ env.PYTHON_VERSION }}
# setup docker compose
- uses: KengoTODA/actions-setup-docker-compose#main
with:
version: '1.26.2'
# install dependencies
- name: python install
working-directory: ${{ env.WORKING_DIRECTORY }}
run: |
sudo apt install python${{ env.PYTHON_VERSION }}-venv
python -m venv --copies antenv
source antenv/bin/activate
pip install setuptools
pip install -r requirements.txt
python -m spacy download en_core_web_md
# Azure login
- uses: azure/login#v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- uses: azure/appservice-settings#v1
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
mask-inputs: false
general-settings-json: '{"linuxFxVersion": "PYTHON|${{ env.PYTHON_VERSION }}"}' #'General configuration settings as Key Value pairs'
# deploy web app
- uses: azure/webapps-deploy#v2
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
package: ${{ env.WORKING_DIRECTORY }}
startup-command: ${{ env.STARTUP_COMMAND }}
# Azure logout
- name: logout
run: |
az logout
most of the script above was taken from: https://github.com/Azure/actions-workflow-samples/blob/master/AppService/python-webapp-on-azure.yml.
is env.STARTUP_COMMAND the "SSH and then run the command" part that I was thinking of, or is it something else entirely?
I also have another question: is there a better way to view logs from that python script running from within the azure resource? The only way I can think of is to ssh into it and then type in "cat 'whatever.log'".
Thanks in advance!
Instead of using STARTUP_COMMAND: 'docker-compose up --build -d' you can use the startup file name.
startUpCommand: 'gunicorn --bind=0.0.0.0 --workers=4 startup:app'
or
StartupCommand: 'startup.txt'
The StartupCommand parameter defines the app in the startup.py file. By default, Azure App Service looks for the Flask app object in a file named app.py or application.py. If your code doesn't follow this pattern, you need to customize the startup command. Django apps may not need customization at all. For more information, see How to configure Python on Azure App Service - Customize startup command.
Also, because the python-vscode-flask-tutorial repository contains the same startup command in a file named startup.txt, you could specify that file in the StartupCommand parameter rather than the command, by using StartupCommand: 'startup.txt'.
Refer: here for more info
I am trying to deploy the python serverless application on AWS.
I follow pretty straightforward tutorial and I do following steps:
Install serverless
npm install -g serverless
Generate template project sls create --template aws-python3 --name sls-demo --path sls-demo
My handler.py file looks as follows:
import json
def hello(event, context):
body = {
"message": "Go Serverless v1.0! Your function executed successfully!",
"input": event
}
response = {
"statusCode": 200,
"body": json.dumps(body)
}
return response
And my serverless.yml configuration file looks as follows:
service: sls-demo
frameworkVersion: '2'
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
region: eu-central-1
functions:
hello:
handler: handler.hello
I have already installed aws cli on my machine, configured it with aws credentials and when I run the deployment command sls deploy it finishes successfully.
I test the lambda function with following command sls invoke --function hello and the result returns successfully:
{
"statusCode": 200,
"body": "{\"message\": \"Go Serverless v1.0! Your function executed successfully!\", \"input\": {}}"
}
Now I want to introduce some extra dependencies in my lambda function, dockerize it and deploy using the serverless-python-requirements plugin.
For this I do following steps:
Create virtual environment python -m venv ./venv
Activate virtual environment source venv/bin/activate
Install numpy dependency pip install numpy
Freeze python dependencies pip freeze > requirements.txt
Install serverless-python-requirements plugin sls plugin install -n serverless-python-requirements
My updated handler.py file looks as follows:
import json
import numpy
def hello(event, context):
array = numpy.arange(15).reshape(3, 5)
body = {
"message": "Go Serverless v1.0! Your function executed successfully!",
"input": event,
"array": array
}
response = {
"statusCode": 200,
"body": json.dumps(body)
}
return response
And my updated serverless.yml configuration file looks as follows:
service: sls-demo
frameworkVersion: '2'
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
region: eu-central-1
functions:
hello:
handler: handler.hello
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: non-linux
After these changes, when I run the deployment command sls deploy it fails with the following error:
Serverless: Recoverable error occurred (Inaccessible host: `sls-demo-dev-serverlessdeploymentbucket-xyz.s3.eu-central-1.amazonaws.com'. This service may not be available in the `eu-central-1' region.), sleeping for ~5 seconds. Try 1 of 4
I enabled debug logs for the serverless by exporting this flag export SLS_DEBUG=* and the exception stack trace looks as follows:
Serverless: Recoverable error occurred (UnknownEndpoint: Inaccessible host: `sls-demo-dev-serverlessdeploymentbucket-xyz.s3.eu-central-1.amazonaws.com'. This service may not be available in the `eu-central-1' region.
at Request.ENOTFOUND_ERROR (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:507:46)
at Request.callListeners (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at ClientRequest.error (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:339:22)
at ClientRequest.<anonymous> (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/http/node.js:96:19)
at ClientRequest.emit (events.js:315:20)
at ClientRequest.EventEmitter.emit (domain.js:467:12)
at TLSSocket.socketErrorListener (_http_client.js:469:9)
at TLSSocket.emit (events.js:315:20)
at TLSSocket.EventEmitter.emit (domain.js:467:12)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
----------------------------------------------------------------------------------------------------), sleeping for ~7 seconds. Try 1 of 4
This error happens when serverless is trying to upload the build artifact to the s3 bucket:
Serverless: Uploading service sls-demo.zip file to S3 (33.14 MB)...
I suspect that this error happens due to the large size of the file, but I don't know how to resolve this problem, or if I miss something in the configuration.
I tried to upload the large file in the mentioned s3 bucket using the aws cli and it works without any problem:
aws s3 cp large_file.zip s3://sls-demo-dev-serverlessdeploymentbucket-xyz/large_file.zip
I don't know how to troubleshoot this problem, and I was unable to find any answer regarding this error in the internet. Any help appreciated.
I am trying to set up an application running in a python 3 App Engine Flexible environment. I have an app.yaml file:
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT application:app
runtime_config:
python_version: 3
I have a requirements.txt listing some packages my app needs:
Flask==0.12
gunicorn==19.7.1
...
I also have a common functions package that is located in a GCP source respository (git). I don't want to host it publicly on PyPi. Is it possible to still include it as a requirement? Something like:
git+https://source.developers.google.com/p/app/r/common
Using the above ask for a username and password when I try it on my local machine, even though I have a helper set up:
git config credential.helper gcloud.sh
You can add -i http://yourhost.com --trusted-host yourhost.com flags to requirements.txt file.