I'm trying to use this tutorial to upload a docker container to AWS ECR for Lambda. My problem is that my python script uses psycopg2, and I couldn't figure out how to install psycopg2 inside the Docker image. I know that I need postgres-devel for the libq library and gcc for compiling, but it still doesn't work.
My requirements.txt:
pandas==1.3.0
requests==2.25.1
psycopg2==2.9.1
pgcopy==1.5.0
Dockerfile:
FROM public.ecr.aws/lambda/python:3.8
WORKDIR /app
COPY my_script.py .
COPY some_file.csv .
COPY requirements.txt .
RUN yum install -y postgresql-devel gcc*
RUN pip install -r requirements.txt
CMD ["/app/my_script.handler"]
After building, running the image, and testing the lambda function locally, I get this error message:
psycopg2.OperationalError: SCRAM authentication requires libpq version 10 or above
So I think the container has the wrong version of postgres(-devel). But I'm not sure how to install the proper version? Any tips for deploying a psycopg2 script to docker for lambda usage?
This might be a little old and too late to answer but figure I post what worked for me.
FROM public.ecr.aws/lambda/python:3.8
COPY . ${LAMBDA_TASK_ROOT}
RUN yum install -y gcc python27 python27-devel postgresql-devel
RUN pip3 install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"
CMD [ "app.handler" ]
Related
I have a Dockerfile that needs to install the latest package code from a private git repo, however because the dockerfile/url/commit doesn't change (I just follow the latest in master), Docker will cache this request and won't pull the latest code.
I can disable build caching entirely which fixes the issue - but this results in a slow build.
How can I just force docker not to use the cache for the one command?
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7
COPY ./requirements.txt /app
RUN pip install --upgrade pip
RUN pip install -r requirements.txt
# This needs to be separate to trigger to invalidate the build cache
RUN pip install -e git+https://TOKEN#github.com/user/private-package.git#egg=private_package
COPY ./main.py /app
COPY ./app /app/app
Add
ARG foo=bar
Before RUN pip install -e ... in your docker file.
Then in your script with docker build .... add as a parameter
--build-arg foo="$(date -s)"
I have basic python docker container file like this:
FROM python:3.8
RUN pip install --upgrade pip
EXPOSE 8000
ENV PYTHONDONTWRITEBYTECODE=1
# Turns off buffering for easier container logging
ENV PYTHONUNBUFFERED=1
# Install pip requirements
COPY requirements.txt .
RUN python -m pip install -r requirements.txt
WORKDIR /app
COPY . /app
RUN useradd appuser && chown -R appuser /app
USER appuser
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "app:app"]
I want to run my flask application in a docker container by using this definition file. Locally I can start a new virtual env, install everything via pip install -r requirements.txt on python 3.8 and it does not fail.
When building the docker image it fails to install all packages from the requirements.txt. For example this package fails:
ERROR: Could not find a version that satisfies the requirement cvxopt==1.2.5.post1
ERROR: No matching distribution found for cvxopt==1.2.5.post1
When I comment out the package in the requirements.txt everything seems to work. The package itself claims to be compatible with python >2.7. Same behavior for the package pywin32==228 here.
Looing at the wheel files in the package, cvxopt.1.2.5.post1 only contains a build for Windows. For Linux (such as the docker container), you should use cvxopt.1.2.5.
You should replace the version with 1.2.5 (pip install cvxopt==1.2.5)
The latest version cvxopt 1.2.5.post1 is not compatible with all architectures: https://pypi.org/project/cvxopt/1.2.5.post1/#files
The previous one is compatible with a lot more hardware and should be able to run on your Docker image: https://pypi.org/project/cvxopt/1.2.5/#files
I have a folder in an ubuntu VM called "MovieFlix" that contains a dockerfile a python flask app and a "templates" folder with html templates inside. I have managed to build a docker image with the same dockerfile successfully twice but I had to delete it in order to edit my python file . The third time I try to build my docker image the image is not build and I get
Package python3 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
dh-python
E: Package 'python3' has no installation candidate
The command '/bin/sh -c apt-get install -y python3 python3-pip' returned a non-zero code: 100
My DockerFile :
FROM ubuntu:16.04
MAINTAINER bill <bill#gmailcom>
RUN apt-get update
RUN apt-get install -y python3 python3-pip
RUN apt-get install -y bcrypt
RUN pip3 install flask pymongo flask_bcrypt
RUN pip3 install Flask-PyMongo py-bcrypt
RUN mkdir /app
RUN mkdir -p /app/templates
COPY webservice.py /app/webservice.py
ADD templates /app/templates
EXPOSE 5000
WORKDIR /app
ENTRYPOINT ["python3" , "-u" , "webservice.py" ]
I tried installing python3-pip but it is already installed in my ubuntu VM
I would appreciate your help . Thank you in advance .
Run below commands in order :
1. sudo apt-get update
2. sudo apt-get install dh-python
SOLVED : I deleted all inactive docker containers and build my image again with the same code
I've created flask app and try to dockerize it. It uses machine learning libraries, I had some problems with download it so my Dockerfile is a little bit messy, but Image was succesfully created.
from alpine:latest
RUN apk add --no-cache python3-dev \
&& pip3 install --upgrade pip
WORKDIR /app
COPY . /app
FROM python:3.5
RUN pip3 install gensim
RUN pip3 freeze > requirements.txt
RUN pip3 --no-cache-dir install -r requirements.txt
EXPOSE 5000
ENV PATH=/venv/bin:$PATH
ENV FLASK_APP /sentiment-service/__init__.py
CMD ["python","-m","flask", "run", "--host", "0.0.0.0", "--port", "5000"]
and when i try:
docker run my_app:latest
I get
/usr/local/bin/python: No module named flask
Of course I have Flask==1.1.1 in my requirements.txt file.
Thanks for any help!
The problem is here:
RUN pip3 freeze > requirements.txt
The > operator in bash overwrites the content of the file. If you want to append to your requirements.txt, consider using >> operator:
RUN pip3 freeze >> requirements.txt
Thank you All. Finally I rebuilded my app, simplified requirements, exclude alpine and use python 3.7 in my Dockerfile.
I could run app locally, but Docker probably could not find some file from path, or get some other error from app, that is why it stopped just after starting.
I'm trying to build a docker image with docker-compose in my ARM64 rasperry pi but it seems to be imposible.
This is my dockerfile:
FROM python:3.6-slim
RUN apt-get update && apt-get -y install python3-dev
RUN apt-get -y install python3-numpy
RUN apt-get -y install python3-pandas
ENTRYPOINT ["python3", "app.py"]
It seems to be OK, but when app.py is run, it gives an error: "Module numpy not found", and the same for pandas module.
If I try to install numpy and pandas using pip:
RUN pip install numpy pandas
It gives me an error or, more usually, the raspberry just gets frozen and I have to unplug it to recover.
I have tried with different versions of python for the source image and also using several ubuntu images and installing python.
Any idea of how can I install numpy and pandas in docker for my raspberry pi (ARM64)?
Thanks
The problems seems to be with the python version. I'm using a python3.6 docker image but, both python3-numpy and python3-pandas packages require python3.5, so when those packages are installed a new version of python is also installed. This is why when I'm trying to import those modules the python interpreter can't found them, because they are installed for another python version.
Finaly I solved it using a generic docker debian image and installing python3.5 myself instead of using a python docker image.
FROM debian:stretch-slim
RUN apt-get update && apt-get -y dist-upgrade
RUN apt-get -y install build-essential libssl-dev libffi-dev python3.5 libblas3 libc6 liblapack3 gcc python3-dev python3-pip cython3
RUN apt-get -y install python3-numpy python3-sklearn
RUN apt-get -y install python3-pandas
COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt
(Disclaimer: The Raspberry Pi 3 B+ is probably too slow to install big dependecies like numpy)
This Dockerfile worked for me on the Raspberry Pi 3 B+ with Software-Version: Linux raspberrypi 5.10.63-v7+ (Consider updating it)
FROM python:3.9-buster
WORKDIR /
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
I am not sure, but I think it helped also to clean docker i.e. remove all images and containers with the following commands:
Warning: This commands deletes all images and containers!
$ docker container prune
$ docker image prune -a
Or reset Docker completely (deletes also volumes and networks):
$ docker system prune --volumes
I recommend to create requirements.txt file.
Inside you can declare packets to install.
The `Dockerfile':
FROM python
COPY app.py /workdir/
COPY requirements.txt /workdir/
WORKDIR /workdir
RUN pip install --trusted-host pypi.python.org -r requirements.txt
CMD python app.py
edit
I create Dockerfile which import pandas lib and then checking if it work:
cat Dockerfile
FROM python
COPY app.py /workdir/
WORKDIR /workdir
RUN python -m pip install pandas
CMD python app.py