I'm building a docker image that suppose to run a python script that depends on some packages (numpy) I install during the build.
During build everything appears to be installed correctly but when I run the container it behaves like those packages were never installed. What seems to be the problem with my code ?
My docker file looks like this :
FROM myimage as intermediate
WORKDIR ./app
COPY ./mathServer ./mathServer/
RUN apt-get update
RUN apt-get install sudo
RUN sudo apt-get install python3-pip -y
RUN sudo pip3 install numpy
RUN sudo pip3 install httpserver
RUN pip3 list
WORKDIR ./app
COPY --from=intermediate ./app/* ./
CMD ["sh","-c","python3 mathServer/mathServer.py"]
I would expect docker run myimage to run mathServer.py successfully but instead it complains about numpy package.
"importError: No module named 'numpy''"
Also if I replace command "python3 mathServer/mathServer.py" with command "pip3 list" pip3 command does not exist. Somehow the packages installed during build are not available when I'm actually running the container.
Please check your docker build log. Numpy requests c compiler and fortran compiler to build and install. It is likely the installation was not successful.
Consider try pre-build dockers such as https://hub.docker.com/r/continuumio/miniconda/, and add numpy via RUN <PATH_TO>/conda install numpy -y
Or https://hub.docker.com/r/continuumio/anaconda3 that already have numpyinstalled.
Related
I am new to several concepts/tools here but have been working my way through tutorials to get this issue figured out. (I've spent a majority of my career writing embedded software or windows apps using C/C++/C#)
So, I'm getting started working in an AWS environment that another member of my team spun up a few months before I joined. I've noticed several lambdas have code just copy/pasted in them and so wanted to build a python package and import it as a layer then use it in all of our lambdas where the code is copied.
Through some reading I understand I can't just build my python package on mac os and found that I should build the package inside docker.
So, this is what I have tried so far:
Docker File
FROM amazonlinux:2
RUN yum update -y && yum install -y initscripts
RUN yum install python3 -y
RUN python3 -m venv myenv
RUN source myenv/bin/activate
RUN yum install python3-pip -y
RUN yum install zip -y
RUN pip3 install wheel
COPY . .
The last line in the docker file just dumps all of my python code/shell scripts into the docker container.
Now, from terminal:
docker build -t myimage .
docker run -it myimage /bin/bash
Once I have the container prompt:
./build-package.sh
Where build-package.sh looks like this:
pip3 install setuptools
pip3 install wheel
python3 setup.py sdist
python3 setup.py bdist_wheel
cd dist
pip3 install geocalc-1.0.0-py3-none-any.whl
cd ..
Now I am verifying my package is installed:
pip3 list
From which I get this output:
Package Version
geocalc 1.0.0
pip 20.2.2
setuptools 49.1.3
wheel 0.38.4
So I know my geocalc pacakge is installed. I find out where it is located with pip3 show geocalc. It is located under /usr/local/lib/python3.7/site-packages
Also ran a few tests inside the docker container to use the package and was able to confirm it was working as expected.
Next:
cd /usr/local/lib/python3.7/site-packages
zip -r9 ${OLDPWD}/function.zip .
In a separate terminal window:
docker ps
I get my container's ID from the ps command output and then run
docker cp bb87bf25976f:/function.zip .
So now I have my funciton.zip file from my docker container. Time to deploy to AWS:
aws lambda publish-layer-version \
--layer-name layerNameHere \
--description "Geospatial Calculations" \
--zip-file fileb://function.zip \
--compatible-runtimes python3.7
This command outputs the arn of the newly created layer. I copy this and go over to my lambda and add the layer using the arn.
At the top of my lambda:
from geocalc import geocalculator
Now when I run the lambda I get this error:
"errorMessage": "Unable to import module 'lambda_function': No module named 'geocalc'"
A couple of other details regarding python code:
geocalc.py
import math
class geocalculator:
.
.
.
setup.py
from setuptools import setup
setup(
name='geocalc',
version='1.0.0',
description='Package with a few commonly used geospatial calculations',
...
packages=['geocalc'],
)
If you've read this far, thanks for taking the time. Any suggestions about why my layer is not working properly?
I would like to use my computer GPU to run the program. For that custom container has been created using: https://blog.softwaremill.com/setting-up-tensorflow-with-gpu-acceleration-the-quick-way-add80cd5c988 (please see Rolling your own custom container section). I tried to use ENTRYPOINT["mp"] for creating a user-defined executable command 'mp'. However, there is the following error:
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: exec: "mp": executable file not found in $PATH: unknown.
Using mp executable command, I would like to run 'mp train'
Link for my docker file: https://github.com/sandeepsinghsengar/docker-file/blob/main/docker
I have copied the docker file's contents here also.
FROM tensorflow/tensorflow:latest-gpu
# ^or just latest-gpu-jupyter if you need Jupyter
WORKDIR /tmp #my main docker's working directory is /tmp
# Set desired Python version
ENV python_version 3.7
# Install desired Python version (the current TF image is be based on Ubuntu at the moment)
RUN apt install -y python${python_version}
# Set default version for root user - modified version of this solution:
https://jcutrer.com/linux/upgrade-python37-ubuntu1810
RUN update-alternatives --install /usr/local/bin/python python
/usr/bin/python${python_version} 1
RUN /usr/local/bin/python -m pip install --upgrade pip
# Update pip: https://packaging.python.org/tutorials/installing-
packages/#ensure-pip-setuptools-and-wheel-are-up-to-date
RUN python -m pip install --upgrade pip setuptools wheel
#RUN apt-get install python-six
# By copying over requirements first, we make sure that Docker will "cache"
# our installed requirements in a dedicated FS layer rather than reinstall
# them on every build
COPY requirements.txt requirements.txt
# Install the requirements
RUN python -m pip install -r requirements.txt
ENTRYPOINT ["mp"]
Normal python commands are running under docker. Any suggestion will be highly appreciable.
Let me know for further information.
I have a folder in an ubuntu VM called "MovieFlix" that contains a dockerfile a python flask app and a "templates" folder with html templates inside. I have managed to build a docker image with the same dockerfile successfully twice but I had to delete it in order to edit my python file . The third time I try to build my docker image the image is not build and I get
Package python3 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
dh-python
E: Package 'python3' has no installation candidate
The command '/bin/sh -c apt-get install -y python3 python3-pip' returned a non-zero code: 100
My DockerFile :
FROM ubuntu:16.04
MAINTAINER bill <bill#gmailcom>
RUN apt-get update
RUN apt-get install -y python3 python3-pip
RUN apt-get install -y bcrypt
RUN pip3 install flask pymongo flask_bcrypt
RUN pip3 install Flask-PyMongo py-bcrypt
RUN mkdir /app
RUN mkdir -p /app/templates
COPY webservice.py /app/webservice.py
ADD templates /app/templates
EXPOSE 5000
WORKDIR /app
ENTRYPOINT ["python3" , "-u" , "webservice.py" ]
I tried installing python3-pip but it is already installed in my ubuntu VM
I would appreciate your help . Thank you in advance .
Run below commands in order :
1. sudo apt-get update
2. sudo apt-get install dh-python
SOLVED : I deleted all inactive docker containers and build my image again with the same code
I'm trying to build a docker image with docker-compose in my ARM64 rasperry pi but it seems to be imposible.
This is my dockerfile:
FROM python:3.6-slim
RUN apt-get update && apt-get -y install python3-dev
RUN apt-get -y install python3-numpy
RUN apt-get -y install python3-pandas
ENTRYPOINT ["python3", "app.py"]
It seems to be OK, but when app.py is run, it gives an error: "Module numpy not found", and the same for pandas module.
If I try to install numpy and pandas using pip:
RUN pip install numpy pandas
It gives me an error or, more usually, the raspberry just gets frozen and I have to unplug it to recover.
I have tried with different versions of python for the source image and also using several ubuntu images and installing python.
Any idea of how can I install numpy and pandas in docker for my raspberry pi (ARM64)?
Thanks
The problems seems to be with the python version. I'm using a python3.6 docker image but, both python3-numpy and python3-pandas packages require python3.5, so when those packages are installed a new version of python is also installed. This is why when I'm trying to import those modules the python interpreter can't found them, because they are installed for another python version.
Finaly I solved it using a generic docker debian image and installing python3.5 myself instead of using a python docker image.
FROM debian:stretch-slim
RUN apt-get update && apt-get -y dist-upgrade
RUN apt-get -y install build-essential libssl-dev libffi-dev python3.5 libblas3 libc6 liblapack3 gcc python3-dev python3-pip cython3
RUN apt-get -y install python3-numpy python3-sklearn
RUN apt-get -y install python3-pandas
COPY requirements.txt /tmp/
RUN pip3 install -r /tmp/requirements.txt
(Disclaimer: The Raspberry Pi 3 B+ is probably too slow to install big dependecies like numpy)
This Dockerfile worked for me on the Raspberry Pi 3 B+ with Software-Version: Linux raspberrypi 5.10.63-v7+ (Consider updating it)
FROM python:3.9-buster
WORKDIR /
COPY requirements.txt requirements.txt
RUN pip3 install -r requirements.txt
I am not sure, but I think it helped also to clean docker i.e. remove all images and containers with the following commands:
Warning: This commands deletes all images and containers!
$ docker container prune
$ docker image prune -a
Or reset Docker completely (deletes also volumes and networks):
$ docker system prune --volumes
I recommend to create requirements.txt file.
Inside you can declare packets to install.
The `Dockerfile':
FROM python
COPY app.py /workdir/
COPY requirements.txt /workdir/
WORKDIR /workdir
RUN pip install --trusted-host pypi.python.org -r requirements.txt
CMD python app.py
edit
I create Dockerfile which import pandas lib and then checking if it work:
cat Dockerfile
FROM python
COPY app.py /workdir/
WORKDIR /workdir
RUN python -m pip install pandas
CMD python app.py
I'm using the Linux DSVM image: microsoft-dsvm linux-data-science-vm-ubuntu linuxdsvmubuntu
My python code fails at first line import pandas as pd with a python error, module not found.
When i remote SSH into the node and run a pip install pandas it tells me it's already installed. Same goes for numpy etc.
I've tried to setup a start task with /bin/bash -c "pip install pandas" etc. but it fails with command pip not found.
Again when running from the SSH shell pip is on the PATH and there is no problem running it.
Can anyone point me in the right direction?
The simple tutorials from microsoft works fine as they don't rely on any external packages. So I'm able to upload my python file and datasets etc. from blob storage onto the machine. And python runs ok. It's just like all the data science specific packages and pip is missing when the task is running, but its there when i SSH into the node.
Bonus question, is jupyter suppose to be running on port 8000?
First you have to install pip in your compute nodes.
bin/bash -c "sudo apt-get -y update && export DEBIAN_FRONTEND=noninteractive && sudo apt-get install -y python3-pip && sudo pip3 install pandas;"
Provide this command as a startup task to the azure batch pool which will install pip and pandas in your virtual machines.
Same way put all the libraries that you want to install in a requirements.txt and give sudo pip3 install -r requirements.txt after installing pip.