I'm trying to wrap a scraping project in a Docker container to run it on a droplet. The spider scraps a website and then writes the data to a postgres database. The postgres database is already running and managed by Digitalocean.
When I run the command locally to test, everything is fine:
docker compose up
I can visualize the spider writing on the database.
Then, I use github action to build and push my docker image on a registry each time I push the code with the script:
name: CI
# 1
# Controls when the workflow will run.
on:
# Triggers the workflow on push events but only for the master branch
push:
branches: [ master ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
inputs:
version:
description: 'Image version'
required: true
#2
env:
REGISTRY: "registry.digitalocean.com/*****-registery"
IMAGE_NAME: "******-scraper"
POSTGRES_USERNAME: ${{ secrets.POSTGRES_USERNAME }}
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
POSTGRES_HOSTNAME: ${{ secrets.POSTGRES_HOSTNAME }}
POSTGRES_PORT: ${{ secrets.POSTGRES_PORT }}
POSTGRES_DATABASE: ${{ secrets.POSTGRES_DATABASE }}
SPLASH_URL: ${{ secrets.SPLASH_URL }}
#3
jobs:
build-compose:
name: Build docker-compose
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v2
- name: Insall doctl
uses: digitalocean/action-doctl#v2
with:
token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
- name: Login to DO Container Registry with short-lived creds
run: doctl registry login --expiry-seconds 1200
- name: Remove all old images
run: if [ ! -z "$(doctl registry repository list | grep "****-scraper")" ]; then doctl registry repository delete-manifest ****-scraper $(doctl registry repository list-tags ****-scraper | grep -o "sha.*") --force; else echo "No repository"; fi
- name: Build compose
run: docker compose -f docker-compose.yaml up -d
- name: Push to Digital Ocean registery
run: docker compose push
deploy:
name: Deploy from registery to droplet
runs-on: ubuntu-latest
needs: build-compose
Then I ssh root#ipv4 manually to my droplet in order to install docker, docker compose and run the image from the registry with:
# Login to registry
docker login -u DO_TOKEN -p DO_TOKEN registry.digitalocean.com
# Stop running container
docker stop ****-scraper
# Remove old container
docker rm ****-scraper
# Run a new container from a new image
docker run -d --restart always --name ****-scraper registry.digitalocean.com/****-registery/****-scraper
As soon as the python script starts on the droplet I have the error:
psycopg2.OperationalError: could not connect to server: No such file
or directory Is the server running locally and accepting connections
on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
It seems like I'm doing something wrong and I can't find how to fix this so far.
I would appreciate some help explanations.
Thanks,
My Dockerfile:
# As Scrapy runs on Python, I run the official Python 3 Docker image.
FROM python:3.9.7-slim
# Set the working directory to /usr/src/app.
WORKDIR /usr/src/app
# Install libpq-dev for psycopg2 python package
RUN apt-get update \
&& apt-get -y install libpq-dev gcc
# Copy the file from the local host to the filesystem of the container at the working directory.
COPY requirements.txt ./
# Install Scrapy specified in requirements.txt.
RUN pip3 install --no-cache-dir -r requirements.txt
# Copy the project source code from the local host to the filesystem of the container at the working directory.
COPY . .
# For Slash
EXPOSE 8050
# Run the crawler when the container launches.
CMD [ "python3", "./****/launch_spiders.py" ]
My docker-compose.yaml
version: "3"
services:
splash:
image: scrapinghub/splash
restart: always
command: --maxrss 2048 --max-timeout 3600 --disable-lua-sandbox --verbosity 1
ports:
- "8050:8050"
launch_spiders:
restart: always
build: .
volumes:
- .:/usr/src/app
image: registry.digitalocean.com/****-registery/****-scraper
depends_on:
- splash
Try installing binary packages of psycopg2-binary instead of psycopg2. Then you don't need gcc and libpq-dev. Probably you have mixed versions of postgreSQL.
Problem solved!
The .env file with all my credentials was in the .dockerignore. It was then impossible to locate this .env when building the image.
Related
I'm trying to run Django from a Docker container on Heroku, but to make that work, I need to run python manage.py collectstatic during my build phase. To achieve that, I wrote the following Dockerfile:
# Set up image
FROM python:3.10
WORKDIR /usr/src/app
ENV PYTHONDONTWRITEBYTECODE=1
ENV PYTHONUNBUFFERED=1
# Install poetry and identify Python dependencies
RUN pip install poetry
COPY pyproject.toml /usr/src/app/
# Install Python dependencies
RUN set -x \
&& apt update -y \
&& apt install -y \
libpq-dev \
gcc \
&& poetry config virtualenvs.create false \
&& poetry install --no-ansi
# Copy source into image
COPY . /usr/src/app/
# Collect static files
RUN python -m manage collectstatic -v 3 --no-input
And here's the docker-compose.yml file I used to run the image:
services:
db:
image: postgres
env_file:
- .env.docker.db
volumes:
- db:/var/lib/postgresql/data
networks:
- backend
ports:
- "5433:5432"
web:
build: .
restart: always
env_file:
- .env.docker.web
ports:
- "8001:$PORT"
volumes:
- .:/usr/src/app
depends_on:
- db
networks:
- backend
command: gunicorn --bind 0.0.0.0:$PORT myapp.wsgi
volumes:
db:
networks:
backend:
driver: bridge
The Dockerfile builds just fine, and I can even see that collectstatic is running and collecting the appropriate files during the build. However, when the build is finished, the only evidence that collectstatic ran is an empty directory called staticfiles. If I run collectstatic again inside of my container, collectstatic works just fine, but since Heroku doesn't persist files created after the build stage, they disappear when my app restarts.
I found a few SO answers discussing how to get collectstatic to run inside a Dockerfile, but that's not my problem; my problem is that it does run, but the collected files don't show up in the container. Anyone have a clue what's going on?
UPDATE: This answer did the trick. My docker-compose.yml was overriding the changes made by collectstatic with this line:
volumes:
- .:/usr/src/app
If, like me, you want to keep the bind mount for ease of local development (so that you don't need to re-build each time), you can edit the command for the web service as follows:
command: bash -c "python -m manage collectstatic && gunicorn --bind 0.0.0.0:$PORT myapp.wsgi"
Note that the image would have run just fine as-is had I pushed it to Heroku (since Heroku doesn't use the docker-compose.yml file), so this was just a problem affecting containers I created on my local machine.
You are overriding the content of /usr/src/app in your container when you added the
volumes:
- .:/usr/src/app
to your docker compose file.
Remove it since you already copied everything during the build.
I am running FastApi via docker by creating a sevice called ingestion-data in docker-compose. My Dockerfile :
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7
# Environment variable for directory containing our app
ENV APP /var/www/app
ENV PYTHONUNBUFFERED 1
# Define working directory
RUN mkdir -p $APP
WORKDIR $APP
COPY . $APP
# Install missing dependencies
RUN pip install -r requirements.txt
AND my docker-compose.yml file
version: '3.8'
services:
ingestion-service:
build:
context: ./app
dockerfile: Dockerfile
ports:
- "80:80"
volumes:
- .:/app
restart: always
I am not sure why this is not picking up any change automatically when I make any change in any endpoint of my application. I have to rebuild my images and container every time.
Quick answer: Yes :)
In the Dockerfile, you copying your app into /var/www/app.
The instructions form the Dockerfile are executed when you build your image (docker build -t <imgName>:<tag>)
If you change the code later on, how could the image be aware of that?
However, you can mount a volume(a directory) from your host machine, into the container when you execute the docker run / docker-compose up command, right under /var/www/app. You'll then be able to change the code in your local directory and the changes will automatically be seen in the container as well.
Perhaps you want to mount the current working directory(the one containing your app) at /var/www/app?
volumes:
- .:/var/www/app
I am trying to incorporate a python container and a dynamodb container into one stack file to experiment with Docker swarm. I have done tutorials on docker swarm seeing web apps running across multiple nodes before but never built anything independently. I am able to run docker-compose up with no issues, but struggling with swarm.
My docker-compose.yml looks like
version: '3.3'
services:
dynamodb:
image: "amazon/dynamodb-local"
ports:
- "8000:8000"
track-count:
image: "my-app"
links:
- "dynamodb:localhost"
Running docker stack deploy -c docker-compose.yml trial_stack brings up no errors however printing 'hello world' as the first line of python code is not displayed in the terminal. I get the following as CMD line output
Ignoring unsupported options: links
Creating network trial_stack_default
Creating service trial_stack_dynamodb
Creating service trial_stack_track-count
My question is:
1) Why is the deploy service ignoring the links? I have noticed this is repeated in the docs https://docs.docker.com/engine/reference/commandline/stack_deploy/ but unsure if this will cause my stack to fail.
2) Assuming the links issue is fixed, where will any command line output be shown, to confirm the system is running? Currently I only have one node, my local machine, which is the manager.
For reference, my python image is being built by the following Dockerfile:
FROM python:3.8-slim-buster
RUN mkdir /app
WORKDIR /app
RUN pip install --upgrade pip
COPY ./requirements.txt ./
RUN pip install -r ./requirements.txt
COPY / /
COPY /resources/secrets.py /resources/secrets.py
CMD [ "python", "/main.py" ]
You can update docker-compose.yaml to enable tty for the services for which you want to see the stdout on console.
Updated docker-compose.yaml should look like this:
version: '3.3'
services:
dynamodb:
image: "amazon/dynamodb-local"
ports:
- "8000:8000"
track-count:
image: "my-app"
tty: true
links:
- "dynamodb:localhost"
and then when once you have the task deployed, to check service logs you can run:
# get the service name
docker stack services <STACK_NAME>
# display the service logs, edited based on user's suggestion
docker service logs --follow --raw <SERVICE_NAME>
Installing the package in alpine get stuck
it stuck at
(6/12) Installing ncurses-terminfo (6.1_p20190105-r0) OR
(10/12) Installing python2 (2.7.16-r1)
Sometimes it works properly.
Command: sudo docker-compose build
Tried proxy but didn't worked
# Docker Upstart and SysVinit configuration file
#
# THIS FILE DOES NOT APPLY TO SYSTEMD
#
# Please see the documentation for "systemd drop-ins":
# https://docs.docker.com/engine/admin/systemd/
#
# Customize location of Docker binary (especially for development testing).
#DOCKERD="/usr/local/bin/dockerd"
# Use DOCKER_OPTS to modify the daemon startup options.
DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4"
# If you need Docker to use an HTTP proxy, it can also be specified here.
export http_proxy="http://127.0.0.1:3128/"
# This is also a handy place to tweak where Docker's temporary files go.
#export DOCKER_TMPDIR="/mnt/bigdrive/docker-tmp"
Also tried by increating the MTU
docker-compose.yml
version: '3.7'
services:
admin-api:
container_name: admin-api
build:
context: .
dockerfile: Dockerfile
environment:
- HOME=/home
- NODE_ENV=dev
- DB_1=mongodb://mongo:27017/DB_1
- DB_2=mongodb://mongo:27017/DB_2
volumes:
- '.:/app'
- '/app/node_modules'
- '$HOME/.aws:/home/.aws'
ports:
- '4004:4004'
networks:
- backend
links:
- mongo
mongo:
container_name: mongo
image: mongo:4.2.0-bionic
ports:
- "27018:27017"
networks:
- backend
networks:
backend:
driver: bridge
driver_opts:
com.docker.network.driver.mtu: 1500
Dockerfile
# base image
FROM node:8.16.1-alpine
# set working directory
WORKDIR /app
# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH
# install and cache app dependencies
COPY package.json /app/package.json
RUN apk add --update-cache py-pip && \
pip install awscli && \
apk --purge -v del py-pip && \
rm -rf /var/cache/apk/*
RUN npm install --silent
RUN npm install -g nodemon
# start app
CMD nodemon
EXPOSE 4004
My work is dependent on AWS and it requires AWS credentials, I installed the AWS using pip and mounted the /home/.aws (local) with /home/.aws container but when I am creating or building the container it gets stuck and doesn't show any error. While building the container, I also checked the network monitor, it shows receiving packets 0 bytes/s
Tried --verbose but it didn't get any useful information
Django server is running well in localhost. however, When I try to run server on the docker container, it doesn't find the manage.py file when using docker-compose file and even I run the container manually and run the server, it doesn't appear in browser. how can I solve this problem?
So I wrote all the code testing on my local server and using the dockerfile, I built the image of my project.
and I tried to run server on the docker container, suddenly this doesn't run.
what's worse, if I use docker-compose to run the server, it doesn't find the manage.py file though I already checked with 'docker run -it $image_name sh'
here is the code of my project
I am new to docker and new to programming.
hope you can give me a help. thanks!
file structure
current directory
└─example
└─db.sqlite3
└─docker-compose.yml
└─Dockerfile
└─manage.py
└─Pipfile
└─Pipfile.lock
Docker file
# Base image - Python version
FROM python:3.6-alpine
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Set work directory
WORKDIR /code
# Copy Pipfile
COPY Pipfile /code
COPY Pipfile.lock /code
# Install dependencies
RUN pip install pipenv
RUN pipenv install --system
# Copy files
COPY . /code/
docker-compose.yml
# docker-compose.yml
version: '3.3'
services:
web:
build: .
command: python /code/manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
ports:
- 8000:8000
expected result : running server in web browser like in chrome
actual result :
when using docker-compose :
ERROR like this in the prompt : web_1 | python: can't open file '/code/manage.py': [Errno 2] No such file or directory
when running the container manually with 'docker run -it $image_name sh' and 'python manage.py runserver' on the shell :
server is running but, doesn't connect to web browser. (doesn't show up in browser like chrome'
Yo have done same thing in many ways. You have copy source files using a COPY command and then you have mounted a host volume in your docker-compose.yml file. In first place you don't need a volume because volume mounts are to persisting data generated by and used by Docker containers.
Following simplified Dockerfile and docker-compose file would fix the problem.
# Base image - Python version
FROM python:3.6-alpine
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Copy files
COPY . /code/
# Set work directory
WORKDIR /code
# Install dependencies
RUN pip install pipenv
RUN pipenv install --system
docker-compose.yml -:
# docker-compose.yml
version: '3.3'
services:
web:
build: .
command: python ./manage.py runserver 0.0.0.0:8000
ports:
- 8000:8000