Hi everyone and sorry for the silly question but its my day 2 with Yaml.
Problem statement:
I have a python code which runs for 12 mins, (so I cant use Cloud Function to automate it), hence using cloud build as a hack.
Steps done so far:
I have my code in google cloud repository and I used cloud build to build an image and created a google cloud build trigger. Now I want to run the main.py python code each time I trigger the build ( which I will do using a Cloud Scheduler as described here
Folder structure (as shown in the image below)
cloudbuild.yaml which I managed to write so far
steps:
- name: 'gcr.io/$PROJECT_ID/p2p-cloudbuild'
entrypoint: '/bin/bash'
args: ['-c','virtualenv /workspace/venv' ]
# Create a Python virtualenv stored in /workspace/venv that will persist across container runs.
- name: 'gcr.io/$PROJECT_ID/p2p-cloudbuild'
entrypoint: 'venv/bin/pip'
args: ['install', '-V', '-r', 'requirements.txt']
# Installs any dependencies listed in the project's requirements.txt.
Question : How do I add the step to call/execute 'my_function' inside main.py file ?
appreciate your help.
steps:
- name: 'gcr.io/$PROJECT_ID/p2p-cloudbuild'
entrypoint: '/bin/bash'
args: ['-c','virtualenv /workspace/venv' ]
# Create a Python virtualenv stored in /workspace/venv that will persist across container runs.
- name: 'gcr.io/$PROJECT_ID/p2p-cloudbuild'
entrypoint: 'venv/bin/pip'
args: ['install', '-V', '-r', 'requirements.txt']
# Installs any dependencies listed in the project's requirements.txt.
Let's say I have a file main.py:
def foo():
return "bar"
This actually can be simplified into:
- name: 'gcr.io/$PROJECT_ID/p2p-cloudbuild'
entrypoint: '/bin/bash'
args:
- '-c'
- |
virtualenv /workspace/venv
source /workspace/venv/bin/activate
pip install -V -r requirements.txt
python -c 'from main import foo; print (foo())'
Related
I am trying to test a lambda function locally, the function is created from the public docker image from aws, however I want to install my own python library from my github, according to the documentation AWS sam Build I have to add a variable to be taken in the Dockerfile like this:
Dockerfile
FROM public.ecr.aws/lambda/python:3.8
COPY lambda_preprocessor.py requirements.txt ./
RUN yum install -y git
RUN python3.8 -m pip install -r requirements.txt -t .
ARG GITHUB_TOKEN
RUN python3.8 -m pip install git+https://${GITHUB_TOKEN}#github.com/repository/library.git -t .
And to pass the GITHUB_TOKEN I can create a .json file containing the variables for the docker environment.
.json file named env.json
{
"LambdaPreprocessor": {
"GITHUB_TOKEN": "TOKEN_VALUE"
}
}
And simply pass the file address in the sam build: sam build --use-container --container-env-var-file env.json Or directly the value without the .json with the command: sam build --use-container --container-env-var GLOBAL_ENV_VAR=TOKEN_VALUE
My problem is that I don't get the GITHUB_TOKEN variable either with the .json file or by putting it directly in the command with --container-env-var GITHUB_TOKEN=TOKEN_VALUE
Using sam build --use-container --container-env-var GLOBAL_ENV_VAR=TOKEN_VALUE --debug shows that it doesn't take it when creating the lambda image.
The only way that has worked for me is to put the token directly in the Dockerfile not as an build argument.
Promt output
Building image for LambdaPreprocessor function
Setting DockerBuildArgs: {} for LambdaPreprocessor function
Does anyone know why this is happening, am I doing something wrong?
If you need to see the template.yaml this is the lambda definition.
template.yaml
LambdaPreprocessor:
Type: AWS::Serverless::Function
Properties:
PackageType: Image
Architectures:
- x86_64
Timeout: 180
Metadata:
Dockerfile: Dockerfile
DockerContext: ./lambda_preprocessor
DockerTag: python3.8-v1
I'm doing it with vscode and wsl 2 with ubuntu 20.04 lts on windows 10
I am having this issue too. What I have learned is that in the Metadata field there is DockerBuildArgs: that you can also add. Example:
Metadata:
DockerBuildArgs:
MY_VAR: <some variable>
When I add this it does make it to the DockerBuildArgs dict.
I'm running into an issue where it seems I can only run a python command in either the dockerfile or Kubernetes. Right now I have two python scripts, the first script setting up keys and tokens so the second script can run properly.
My dockerfile looks something like this:
FROM python:3.8.0-alpine
WORKDIR /code
COPY script1.py .
COPY script2.py .
# Install python libraries
RUN pip install --upgrade pip
RUN apk add build-base
RUN apk add linux-headers
RUN pip install -r requirements.txt
CMD [ "python", "-u", "script2.py"]
and my Kubernetes yaml file is:
apiVersion: apps/v1
kind: Deployment
metadata:
name: script2
labels:
app: script2-app
spec:
selector:
matchLabels:
app: script2-app
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: script2-app
spec:
containers:
- name: script2-app
image: script2:v1.0.1
ports:
- containerPort: 5000
env:
- name: KEYS_AND_TOKENS
valueFrom:
secretKeyRef:
name: my_secret
key: KEYS_AND_TOKENS
command:
- "python"
- "script1.py"
The issue starts with the 'command' portion in the yaml file. Without it, Kubernetes will run the container as usual. (the container can still run without the keys and tokens. It will just log that some functions failed to run then move on.) However, when I include the 'command' portion, script1 will run and successfully set up the keys. But once script1 finishes, nothing else happens. The deployment continues to run but script2 never starts.
The reason I am doing it this way is because script2 may need to restart on occasion due to internet connection failures causing it to crash. Sence all script1 is doing is setting up keys and tokens, it only needs to run once, then things will be set up for as long as the pod lives. I don't want to verify the keys and tokens every time script2 restarts. This is why the two scripts are separate and why I'm only running script1 in startup.
Any help would be much appreciated!
What is happening?
The command supplied through the yaml file overrides the CMD in the dockerfile (you can refer the kubernetes documentation here). So, when you supply command for executing the script1 in the yaml file, it overrides the command (CMD) in dockerfile for the execution of script2 because of which you must be getting errors as per your code logic.
How to resolve?
Step 1: Create a bash file as follows: (naming it "run.sh")
#!/bin/bash
exec python3 /path/to/script1.py
exec python3 /path/to/script2.py
Step 2: Update the dockerfile as:
FROM python:3.8.0-alpine
WORKDIR /code
COPY script1.py .
COPY script2.py .
COPY run.sh . # copying the bash file
# Install python libraries
RUN pip install --upgrade pip
RUN apk add build-base
RUN apk add linux-headers
RUN pip install -r requirements.txt
RUN chmod a+x run.sh # making it executable
CMD ["./run.sh"] # executing the scripts through bash file
Step 3: Remove the command from the kubernetes deployment yaml
But if you want to keep the command in the yaml file then you should replace it with "./run.sh"
Note:
Ideally you should not run two different scripts like this. If you need to set up tokens/keys you should do that in a sub-module which you can call in your main script. You can handle network connectivity issues, etc. through a combination of exception handling and a retry mechanism.
I'm very new to DevOps, so this may be a very silly question. I'm trying to deploy a python Web scraping script onto an azure webapp using GitHub actions. This script is meant to be run for a long period of time as it is analyzing websites word by word for hours. It then logs the results to .log files.
I know a bit of how GitHub actions work, I know that I can trigger jobs when I push to the repo for instance. However, I'm a bit confused as to how one runs the app or a script on an azure resource (like a VM or webapp) for example. Does this process involve SSH-ing into the resource and then automatically run the cli command "python main.py" or "docker-compose up", or is there something more sophisticated involved?
For better context, this is my script inside of my workflows folder:
on:
[push]
env:
AZURE_WEBAPP_NAME: emotional-news-service # set this to your application's name
WORKING_DIRECTORY: '.' # set this to the path to your path of working directory inside GitHub repository, defaults to the repository root
PYTHON_VERSION: '3.9'
STARTUP_COMMAND: 'docker-compose up --build -d' # set this to the startup command required to start the gunicorn server. default it is empty
name: Build and deploy Python app
jobs:
build-and-deploy:
runs-on: ubuntu-latest
environment: dev
steps:
# checkout the repo
- uses: actions/checkout#master
# setup python
- name: Setup Python
uses: actions/setup-python#v1
with:
python-version: ${{ env.PYTHON_VERSION }}
# setup docker compose
- uses: KengoTODA/actions-setup-docker-compose#main
with:
version: '1.26.2'
# install dependencies
- name: python install
working-directory: ${{ env.WORKING_DIRECTORY }}
run: |
sudo apt install python${{ env.PYTHON_VERSION }}-venv
python -m venv --copies antenv
source antenv/bin/activate
pip install setuptools
pip install -r requirements.txt
python -m spacy download en_core_web_md
# Azure login
- uses: azure/login#v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}
- uses: azure/appservice-settings#v1
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
mask-inputs: false
general-settings-json: '{"linuxFxVersion": "PYTHON|${{ env.PYTHON_VERSION }}"}' #'General configuration settings as Key Value pairs'
# deploy web app
- uses: azure/webapps-deploy#v2
with:
app-name: ${{ env.AZURE_WEBAPP_NAME }}
package: ${{ env.WORKING_DIRECTORY }}
startup-command: ${{ env.STARTUP_COMMAND }}
# Azure logout
- name: logout
run: |
az logout
most of the script above was taken from: https://github.com/Azure/actions-workflow-samples/blob/master/AppService/python-webapp-on-azure.yml.
is env.STARTUP_COMMAND the "SSH and then run the command" part that I was thinking of, or is it something else entirely?
I also have another question: is there a better way to view logs from that python script running from within the azure resource? The only way I can think of is to ssh into it and then type in "cat 'whatever.log'".
Thanks in advance!
Instead of using STARTUP_COMMAND: 'docker-compose up --build -d' you can use the startup file name.
startUpCommand: 'gunicorn --bind=0.0.0.0 --workers=4 startup:app'
or
StartupCommand: 'startup.txt'
The StartupCommand parameter defines the app in the startup.py file. By default, Azure App Service looks for the Flask app object in a file named app.py or application.py. If your code doesn't follow this pattern, you need to customize the startup command. Django apps may not need customization at all. For more information, see How to configure Python on Azure App Service - Customize startup command.
Also, because the python-vscode-flask-tutorial repository contains the same startup command in a file named startup.txt, you could specify that file in the StartupCommand parameter rather than the command, by using StartupCommand: 'startup.txt'.
Refer: here for more info
I've been trying to understand how to go about deploying my Python Function App to Azure using Bitbucket pipelines.
I've read some answers on the web, and it seems pretty simple once I have my python app zipped.
It can easily be done using this answer: Azure Function and BitBucket build pipelines
script:
- pipe: microsoft/azure-functions-deploy:1.0.2
variables:
AZURE_APP_ID: $AZURE_APP_ID
AZURE_PASSWORD: $AZURE_PASSWORD
AZURE_TENANT_ID: $AZURE_TENANT_ID
FUNCTION_APP_NAME: '<string>'
ZIP_FILE: '<string>'
However, I can't, for the life of me, find the format Azure Functions is expecting the zip file to be in.
Where do the requirements go? Even better - what pipeline spec comes before this one that creates the sought after ZIP_FILE?
Thanks!
I tried this solution and work also for me, but there is a deprecation problem. The pipe microsoft/azure-functions-deploy is using a deprecated image for azure cli: microsoft/azure-cli, you can read this here.
So you can use the atlassian version of this pipe but for python doesn't work for me because in the command:
az functionapp deployment source config-zip...
there isn't specified --build-remote.
So my solution is not use pipe but write your step with azure-cli commands:
- step:
name: Deploy on Azure
image: mcr.microsoft.com/azure-cli:latest
script:
- az login --service-principal --username ${AZURE_APP_ID} --password ${AZURE_PASSOWRD} --tenant ${AZURE_TENANT_ID}
- az functionapp deployment source config-zip -g ${RESOURCE_GROUP_NAME} -n 'functioAppName' --src 'function.zip' --build-remote
This is a step that work in my case, another solution can be write a step that use this
Ended up finding the answer scattered in different places:
image: python:3.8
pipelines:
branches:
master:
- step:
name: Build function zip
caches:
- pip
script:
- apt-get update
- apt-get install -y zip
- pip install --target .python_packages/lib/site-packages -r requirements.txt
- zip -r function.zip .
artifacts:
- function.zip
- step:
name: Deploy zip to Azure
deployment: Production
script:
- pipe: microsoft/azure-functions-deploy:1.0.0
variables:
AZURE_APP_ID: $AZURE_APP_ID
AZURE_PASSWORD: $AZURE_PASSWORD
AZURE_TENANT_ID: $AZURE_TENANT_ID
ZIP_FILE: 'function.zip'
FUNCTION_APP_NAME: $FUNCTION_NAME
I have been reading this tutorial:
https://prakhar.me/docker-curriculum/
along with other tutorials, and Docker docks and I am still not completely clear on how to do this task.
The problem
My local machine is running Mac OS X, and I would like to set up a development environment for a python project. In this project I need to run call an api from a docker repo bamos/openface. The project also has some dependencies such as yaml, etc. If I just mount my local to openface, ie:
docker run -v path/to/project:/root/project -p 9000:9000 -p 8000:8000 -t -i bamos/openface /bin/bash
Then I need to install yaml and other dependencies, and every time I exit the container the installations would be lost. Additionally, it is also much slower for some reason. So the right way to do this is using Docker compose, but I am not sure how to proceed from here.
UPDATE
In response to the comments, I will now update the problem:
Right now my Dockerfile looks like this:
FROM continuumio/anaconda
ADD . /face-off
WORKDIR /face-off
RUN pip install -r requirements.txt
EXPOSE 5000
CMD [ "python", "app.py" ]
It is important that I build from anaconda since a lot of my code will use numpy and scipy. Now I also need bamos/openface, so I tried adding that to my docker-compose.yml file:
version: '2'
services:
web:
build: .
command: python app.py
ports:
- "5000:5000"
volumes:
- .:/face-off
openface:
build: bamos/openface
However, I am getting error:
build path path/to/face-off/bamos/openface either does not exist, is not accessible, or is not a valid URL
So I need to pass bamos/openface the right way so I can build a container with it. Right now bamos/openface is listed when I do docker images.