I created a multi-stage Docker image to reduce the size of my python application. The content of the Dockerfile is:
FROM python:3.10-slim AS image-compiling
# updates pip and prepare for installing wheels
RUN pip install --upgrade pip && \
pip install wheel && \
pip cache purge
# install needed modules from wheels
RUN pip install --user \
Flask==2.2.2 Flask-RESTful==0.3.9 Flask-Cors==3.0.10 keras==2.9.0 && \
pip cache purge
# brand new image
FROM python:3.10-slim AS image-deploy
# get python 'user environment'
COPY --from=image-compiling /root/.local /root/.local
# get app content
COPY app ./app
# define working directory and the standard command (run it detached)
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["wsgi.py"]
Then I pushed the image my_image to the Google Cloud Storage of my project my_proj with:
$ docker push us.gcr.io/my_proj/my_image
When I try to start a Google Cloud Run service with this image, it fails with the Python error:
Error 2022-09-14 11:41:13.744 EDTTraceback (most recent call last): File "/app/wsgi.py", line 10, in <module> from flask import render_template ModuleNotFoundError: No module named 'flask'
Warning 2022-09-14 11:41:13.805 EDTContainer called exit(1).
But flask is installed in the image, and a container created locally from this image does run it normally.
Why the container in Google Cloud Run is not able to find the module flask?
Isn't it created completely?
Reason of the problem:
The problem in this case is not with the fact that the image was created using the multi-stage approach. The issue is that the Python modules were installed using the option --user.
This option was included so that the needed Python modules were stored in the folder /root/.local, which could be copied directly to the new deploy image.
However, for some reason, Python applications in containers in Google Cloud Run are not able to find the modules if they are in the folder /root/.local. Probably a user other than root is used there. Thus, using the --user argument of pip does not work, which also inhibits the multi-stage approach in this type of scenario.
Solution:
Do a "mono-stage" image with global Python modules, i.e., a Dockerfile like:
FROM python:3.10-slim
# updates pip and prepare for installing wheels
RUN pip install --upgrade pip && \
pip install wheel && \
pip cache purge
# install needed modules from wheels
RUN pip install \
Flask==2.2.2 Flask-RESTful==0.3.9 Flask-Cors==3.0.10 keras==2.9.0 && \
pip cache purge
# get app content
COPY app ./app
# define working directory and the standard command (run it detached)
WORKDIR /app
ENTRYPOINT ["python"]
CMD ["wsgi.py"]
I would like to use my computer GPU to run the program. For that custom container has been created using: https://blog.softwaremill.com/setting-up-tensorflow-with-gpu-acceleration-the-quick-way-add80cd5c988 (please see Rolling your own custom container section). I tried to use ENTRYPOINT["mp"] for creating a user-defined executable command 'mp'. However, there is the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "mp": executable file not found in $PATH: unknown.
Using mp executable command, I would like to run 'mp train'
Link for my docker file: https://github.com/sandeepsinghsengar/docker-file/blob/main/docker
I have copied the docker file's contents here also.
FROM tensorflow/tensorflow:latest-gpu
# ^or just latest-gpu-jupyter if you need Jupyter
WORKDIR /tmp #my main docker's working directory is /tmp
# Set desired Python version
ENV python_version 3.7
# Install desired Python version (the current TF image is be based on Ubuntu at the moment)
RUN apt install -y python${python_version}
# Set default version for root user - modified version of this solution:
https://jcutrer.com/linux/upgrade-python37-ubuntu1810
RUN update-alternatives --install /usr/local/bin/python python
/usr/bin/python${python_version} 1
RUN /usr/local/bin/python -m pip install --upgrade pip
# Update pip: https://packaging.python.org/tutorials/installing-
packages/#ensure-pip-setuptools-and-wheel-are-up-to-date
RUN python -m pip install --upgrade pip setuptools wheel
#RUN apt-get install python-six
# By copying over requirements first, we make sure that Docker will "cache"
# our installed requirements in a dedicated FS layer rather than reinstall
# them on every build
COPY requirements.txt requirements.txt
# Install the requirements
RUN python -m pip install -r requirements.txt
ENTRYPOINT ["mp"]
Normal python commands are running under docker. Any suggestion will be highly appreciable.
Let me know for further information.
I'm building a docker image that suppose to run a python script that depends on some packages (numpy) I install during the build.
During build everything appears to be installed correctly but when I run the container it behaves like those packages were never installed. What seems to be the problem with my code ?
My docker file looks like this :
FROM myimage as intermediate
WORKDIR ./app
COPY ./mathServer ./mathServer/
RUN apt-get update
RUN apt-get install sudo
RUN sudo apt-get install python3-pip -y
RUN sudo pip3 install numpy
RUN sudo pip3 install httpserver
RUN pip3 list
WORKDIR ./app
COPY --from=intermediate ./app/* ./
CMD ["sh","-c","python3 mathServer/mathServer.py"]
I would expect docker run myimage to run mathServer.py successfully but instead it complains about numpy package.
"importError: No module named 'numpy''"
Also if I replace command "python3 mathServer/mathServer.py" with command "pip3 list" pip3 command does not exist. Somehow the packages installed during build are not available when I'm actually running the container.
Please check your docker build log. Numpy requests c compiler and fortran compiler to build and install. It is likely the installation was not successful.
Consider try pre-build dockers such as https://hub.docker.com/r/continuumio/miniconda/, and add numpy via RUN <PATH_TO>/conda install numpy -y
Or https://hub.docker.com/r/continuumio/anaconda3 that already have numpyinstalled.
Background
I have been struggling for the past few days to deploy a Lambda that uses Pillow, and I am deploying using Python 3.6. It may be noteworthy also that I am developing this on a Windows 10 environment.
First Attempts
I began by having pip install my packages strictly in my workspace by doing the following:
pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\pillow"
I have other packages, and tried installing the packages in the same manor, one of them specifically was praw and I did so by:
pip3 install praw -t "D:\Work and Projects\...\...\<projectdir>\praw"
After zipping the contents of my project together, I uploaded my package up to Lambda and upon my first test I received the error:
Unable to import module 'motw_lambda': cannot import name '_imaging'
I then removed the Pillow package in an attempt to see where this issue was stemming from (Pillow or praw or one of the other packages). With Pillow removed, the execution succeeded. I then removed the pillow package in my package and tried:
pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\PIL"
and
pip3 install pillow -t "D:\Work and Projects\...\...\<projectdir>\Pillow"
But got the same error with the package '_imaging'.
Further Attempts
I then followed the directions per this resource and this. I also tried using virualenv and even lambda-uploader.
Strange enough, I get the same error! I am all out of options here, and feel either I am doing something dumb, or that this is not possible on Lambda-Python3.6 currently (Although I can't image someone else hasn't used pillow in a py3.6-lambda yet...)
Any info, help, or generic resources would be appreciated!
Basically, you have to compile the libraries (eg, PIL) either using Docker or, even better, an EC2 instance.
Launch an Docker container like this: docker run --rm -it -v "%cd%:/code" lambci/lambda:build-python3.6 sh
Inside there, navigate to the /code dir (cd /code), create a virtualenv (virtualenv env), activate it (source env/bin/activate) and finally, install your library (pip install pillow).
Once you have installed your library, you can exit the container. The secret here is to move your package library to the root folder (where your main .py file is). For example, move the folder env/lib/python3.6/site-packages/PIL to the root.
Then, zip your PIL folder together with your .py file and you're set!
Full Example:
The following example compiles and compresses PIL and other common Python libraries to run in AWS Lambda.
Dockerfile:
FROM lambci/lambda:build-python3.6
WORKDIR /code
CMD ["sh", "entrypoint.sh"]
entrypoint.sh:
#!/bin/sh
set -ex
cd /code/
if [ ! -d "PIL" ]; then
# Create virtual env, activate it and install PIL
virtualenv env && source env/bin/activate && pip install pillow requests
# Copy necessary files to the root folder
rm -f build-output.zip
#mkdir PIL
cp -f -r env/lib/python3.6/site-packages/PIL .
cp -f -r env/lib/python3.6/site-packages/requests .
# below are the dependencies for the requests pkg
cp -f -r env/lib/python3.6/site-packages/urllib3 .
cp -f -r env/lib/python3.6/site-packages/certifi .
cp -f -r env/lib/python3.6/site-packages/chardet .
cp -f -r env/lib/python3.6/site-packages/idna .
# Remove temp files
rm -r env
fi
# ZIP it
zip -9 build-output *.py
zip -9 -r build-output PIL
zip -9 -r build-output requests
zip -9 -r build-output urllib3
zip -9 -r build-output certifi
zip -9 -r build-output chardet
zip -9 -r build-output idna
To build (Windows):
docker build -t build_lambda .
docker run --rm -v "%cd%:/code" build_lambda
For anyone else also new to aws python and running into this issue, you can use the layers feature, and there are existing layers here you can link to and this worked for me.
https://github.com/keithrozario/Klayers
Pillow specifically on us-east-1:
arn:aws:lambda:us-east-1:770693421928:layer:Klayers-python38-Pillow:2
You can use a precompiled version of the PIL available here:
https://github.com/Miserlou/lambda-packages
Just extract PIL folder to the deployment package and it should work.
Finally found a pretty easy solution. The trick is to install Pillow in a directory locally, zip it up, and then create a Lambda Layer. Now, the Python version you run locally must match the Python version of the Lambda. In my case locally I have Python 3.10, and the Lambda is 3.9, so I installed 3.9 locally just to use for this.
On Mac I used homebrew to install pyenv to get 3.9, but however you want to install is fine.
!Only needed if you do not have the correct version of Python locally!
brew install pyenv
pyenv install 3.9 (or whatever version)
(if you do this, then you can run 'pyenv local 3.9.16' or whatever version you installed, then 'Python3 --versions' should show that version.
For example, pyenv installed in MY_USER_DIR/.pyenv/versions/3.9.16/bin/python3
OR you can set your env to with pyenv to use the specific version.
Then cd into an empty dir, and run the following:
YOUR_LOCAL_PATH_TO_CORRECT_VERSION/python3 -m pip install \
--platform manylinux2014_aarch64 \
--target=./python/lib/python3.9/site-packages \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
Pillow
In the above, manylinux2014_aarch64 is for the ARM runtime. So your Lambda will have to be set to that, or find the value for the X86 version. And if you are not using 3.9, then change that to your version.
Your are now good to go to create the zip for your layer. Just run the following, and then upload the zip to a layer, and make sure you select the correct Python version and architecture for your Lambda.
in the same dir you ran the above command:
zip -r ../pillowLayer.zip .
I am creating a docker containing python and php. I am writing a python script to connect to a MQTT broker residing in another docker.
In my dockerfile I ensure that I install the paho client by using the following commands:
RUN apt-get install -y python3-dev
RUN apt-get install -y libffi-dev
RUN apt-get install -y libssl-dev
ADD https://bootstrap.pypa.io/get-pip.py /tmp/get-pip.py
RUN cat /tmp/get-pip.py | python3
RUN pip install paho-mqtt
RUN pip install python-etcd
However when I run the python script I get the following error:
ImportError: No module named paho.mqtt.client
The docker installation does not show any error with regards to paho-mqtt installation.
It will be great if someone can guide on this.
I think I have found the problem,
You have installed Python3 but for some reason the interpreter defaults to version 2.7 in Linux.
Try using pip3 install paho-mqtt python-etcd instead.
Or if it does not work, you can literally copy and paste the paho folder from your Python2.7 site-packages folder to your Python3 site-packages folder. I have just verified paho-mqtt 1.2 for Python2 is exactly the same as paho-mqtt 1.2 for Python3 using a Meld diff tool. Please note, when you directly copy and paste pip list will not display the package you copied.
site-packages are usually inside your system lib folder. It depends upon how Python is installed. In my case everything is inside $HOME/.pyenv folder.
Remember Python2 has it's own site-packages folder and Python3 has it's own site-packages folder where Python searches for the packages. Sometimes if you are using a Debian based Linux distro please make sure to check inside the dist-packages folder as well to see if you can find the package you are looking for.
You can try install Paho lib:
git clone https://github.com/eclipse/paho.mqtt.python
Once you have the code, it can be installed from your repository as well:
cd paho.mqtt.python
python setup.py install