I need to change the Dockerfile below, coming from pytorch_geometric (a popular PyTorch package), from CUDA9.0 to CUDA10.0.
FROM ubuntu:16.04
RUN apt-get update && apt-get install -y --no-install-recommends apt-utils ca-certificates apt-transport-https gnupg-curl && \
rm -rf /var/lib/apt/lists/* && \
NVIDIA_GPGKEY_SUM=d1be581509378368edeec8c1eb2958702feedf3bc3d17011adbf24efacce4ab5 && \
NVIDIA_GPGKEY_FPR=ae09fe4bbd223a84b2ccfce3f60f4b3d7fa2af80 && \
apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub && \
apt-key adv --export --no-emit-version -a $NVIDIA_GPGKEY_FPR | tail -n +5 > cudasign.pub && \
echo "$NVIDIA_GPGKEY_SUM cudasign.pub" | sha256sum -c --strict - && rm cudasign.pub && \
echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/cuda.list && \
echo "deb https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64 /" > /etc/apt/sources.list.d/nvidia-ml.list
ENV CUDA_VERSION 9.0.176
ENV NCCL_VERSION 2.4.2
ENV CUDA_PKG_VERSION 9-0=$CUDA_VERSION-1
ENV CUDNN_VERSION 7.4.2.24
RUN apt-get update && apt-get install -y --no-install-recommends \
cuda-cudart-$CUDA_PKG_VERSION && \
ln -s cuda-9.0 /usr/local/cuda && \
rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \
cuda-libraries-$CUDA_PKG_VERSION \
libnccl2=$NCCL_VERSION-1+cuda9.0 && \
apt-mark hold libnccl2 && \
rm -rf /var/lib/apt/lists/*
RUN apt-get update && apt-get install -y --allow-unauthenticated --no-install-recommends \
cuda-libraries-dev-$CUDA_PKG_VERSION \
cuda-nvml-dev-$CUDA_PKG_VERSION \
cuda-minimal-build-$CUDA_PKG_VERSION \
cuda-command-line-tools-$CUDA_PKG_VERSION \
cuda-core-9-0=9.0.176.3-1 \
cuda-cublas-dev-9-0=9.0.176.4-1 \
libnccl-dev=$NCCL_VERSION-1+cuda9.0 && \
rm -rf /var/lib/apt/lists/*
ENV LIBRARY_PATH /usr/local/cuda/lib64/stubs
# NVIDIA docker 1.0.
LABEL com.nvidia.volumes.needed="nvidia_driver"
LABEL com.nvidia.cuda.version="${CUDA_VERSION}"
RUN echo "/usr/local/nvidia/lib" >> /etc/ld.so.conf.d/nvidia.conf && \
echo "/usr/local/nvidia/lib64" >> /etc/ld.so.conf.d/nvidia.conf
ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:${PATH}
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64
# NVIDIA container runtime.
ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
ENV NVIDIA_REQUIRE_CUDA "cuda>=9.0"
# PyTorch (Geometric) installation
RUN rm /etc/apt/sources.list.d/cuda.list && \
rm /etc/apt/sources.list.d/nvidia-ml.list
RUN apt-get update && apt-get install -y \
curl \
ca-certificates \
vim \
sudo \
git \
bzip2 \
libx11-6 \
&& rm -rf /var/lib/apt/lists/*
# Create a working directory.
RUN mkdir /app
WORKDIR /app
# Create a non-root user and switch to it.
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
&& chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user
# All users can use /home/user as their home directory.
ENV HOME=/home/user
RUN chmod 777 /home/user
# Install Miniconda.
RUN curl -so ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-4.5.12-Linux-x86_64.sh \
&& chmod +x ~/miniconda.sh \
&& ~/miniconda.sh -b -p ~/miniconda \
&& rm ~/miniconda.sh
ENV PATH=/home/user/miniconda/bin:$PATH
ENV CONDA_AUTO_UPDATE_CONDA=false
# Create a Python 3.6 environment.
RUN /home/user/miniconda/bin/conda install conda-build \
&& /home/user/miniconda/bin/conda create -y --name py36 python=3.6.5 \
&& /home/user/miniconda/bin/conda clean -ya
ENV CONDA_DEFAULT_ENV=py36
ENV CONDA_PREFIX=/home/user/miniconda/envs/$CONDA_DEFAULT_ENV
ENV PATH=$CONDA_PREFIX/bin:$PATH
# CUDA 9.0-specific steps.
RUN conda install -y -c pytorch \
cuda90=1.0 \
magma-cuda90=2.4.0 \
"pytorch=1.1.0=py3.6_cuda9.0.176_cudnn7.5.1_0" \
torchvision=0.2.1 \
&& conda clean -ya
# Install HDF5 Python bindings.
RUN conda install -y h5py=2.8.0 \
&& conda clean -ya
RUN pip install h5py-cache==1.0
# Install TorchNet, a high-level framework for PyTorch.
RUN pip install torchnet==0.0.4
# Install Requests, a Python library for making HTTP requests.
RUN conda install -y requests=2.19.1 \
&& conda clean -ya
# Install Graphviz.
RUN conda install -y graphviz=2.38.0 \
&& conda clean -ya
RUN pip install graphviz==0.8.4
# Install OpenCV3 Python bindings.
RUN sudo apt-get update && sudo apt-get install -y --no-install-recommends \
libgtk2.0-0 \
libcanberra-gtk-module \
&& sudo rm -rf /var/lib/apt/lists/*
RUN conda install -y -c menpo opencv3=3.1.0 \
&& conda clean -ya
# Install PyTorch Geometric.
RUN CPATH=/usr/local/cuda/include:$CPATH \
&& LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH \
&& DYLD_LIBRARY_PATH=/usr/local/cuda/lib:$DYLD_LIBRARY_PATH
RUN pip install --verbose --no-cache-dir torch-scatter \
&& pip install --verbose --no-cache-dir torch-sparse \
&& pip install --verbose --no-cache-dir torch-cluster \
&& pip install --verbose --no-cache-dir torch-spline-conv \
&& pip install torch-geometric
# Set the default command to python3.
CMD ["python3"]
I've tried starting it with FROM pytorch/pytorch:1.1.0-cuda10.0-cudnn7.5-runtime and commenting everything up to # PyTorch (Geometric) installation and the section on # CUDA 9.0-specific steps. for
RUN conda install -c pytorch pytorch
RUN conda install -c fragcolor cuda10.0 && conda clean -ya
and commenting out
# Install Graphviz.
RUN conda install -y graphviz=2.38.0 \
&& conda clean -ya
RUN pip install graphviz==0.8.4
which didn't seem to work even with CUDA9.0
This makes the docker work and load, pytorch to be able to be imported and cuda to work as well. However, when I try to import torch_geometric I get ModuleNotFoundError: No module named 'torch_scatter.scatter_cuda'
Since the package is well-maintained (4.5k stars, mentioned in the pytorch website), it seems to me it's likely my fault and something general about how to adapt from CUDA9.0 to CUDA10.0.
I'd appreciate any advice on what I could be doing wrong or a way of changing it without removing so many lines from the original Dockerfile, which is probably what's causing the issue.
Have you ever try with nvidia/cuda Docker base image?
try
FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu16.04
RUN apt-get update && apt-get install -y --no-install-recommends \
apt-utils \
python3.6 \
python-dev \
python-pip \
python-setuptools \
&& \
rm -rf /var/lib/apt/lists/* && \
apt-get update
RUN pip install --upgrade pip==9.0.3 && \
pip --no-cache-dir install --upgrade torch==1.1.0 && \
pip --no-cache-dir install --upgrade torchvision==0.3.0
The versions of the packages I wrote are stable to use with Dockerfile.
I checked it and it worked well without any collision.
Related
I tried to alias python3 to python3.8 in the Dockerfile. But It doesn't work for me. I am using ubuntu:18.04.
Step 25/41 : RUN apt-get update && apt-get install -y python3.8
---> Using cache
---> 9fa81ca14a53
Step 26/41 : RUN alias python3="python3.8" && python3 --version
---> Running in d7232d3c8b8f
Python 3.6.9
As you can see the python3 is still 3.6.9. How can I fix this issue?
Thanks.
EDIT
Just attach my Dockerfile:
##################################################################################################################
# Build
#################################################################################################################
#FROM openjdk:8
FROM ubuntu:18.04
############## Linux and perl packages ###############
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer && \
apt-get update -y && \
apt-get install curl groff python-gdbm -y;
# Fix certificate issues, found as of
# https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/983302
RUN apt-get update && \
apt-get install -y ca-certificates-java && \
apt-get clean && \
update-ca-certificates -f && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer;
# Setup JAVA_HOME, this is useful for docker commandline
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME
# install git
RUN apt-get update && \
apt-get install -y mysql-server && \
apt-get install -y uuid-runtime git jq python python-dev python-pip python-virtualenv libdbd-mysql-perl && \
rm -rf /var/lib/apt/lists/* && \
apt-get install perl && \
perl -MCPAN -e 'CPAN::Shell->install("Inline")' && \
perl -MCPAN -e 'CPAN::Shell->install("DBI")' && \
perl -MCPAN -e 'CPAN::Shell->install("List::MoreUtils")' && \
perl -MCPAN -e 'CPAN::Shell->install("Inline::Python")' && \
perl -MCPAN -e 'CPAN::Shell->install("LWP::Simple")' && \
perl -MCPAN -e 'CPAN::Shell->install("JSON")' && \
perl -MCPAN -e 'CPAN::Shell->install("LWP::Protocol::https")';
RUN apt-get update && \
apt-get install --yes cpanminus
RUN cpanm \
CPAN::Meta \
YAML \
DBI \
Digest::SHA \
Module::Build \
Test::Most \
Test::Weaken \
Test::Memory::Cycle \
Clone
# Install perl modules for network and SSL (and their dependencies)
RUN apt-get install --yes \
openssl \
libssl-dev \
liblwp-protocol-https-perl
RUN cpanm \
LWP \
LWP::Protocol::https
# New module for v1.2 annotation
RUN perl -MCPAN -e 'CPAN::Shell->install("Text::NSP::Measures::2D::Fisher::twotailed")'
#############################################
############## python packages ###############
# python packages
RUN pip install pymysql==0.10.1 awscli boto3 pandas docopt fastnumbers tqdm pygr
############## python3 packages ###############
# python3 packages
RUN apt-get update && \
apt-get install -y python3-pip && \
python3 -m pip install numpy && \
python3 -m pip install pandas && \
python3 -m pip install sqlalchemy && \
python3 -m pip install boto3 && \
python3 -m pip install pymysql && \
python3 -m pip install pymongo;
RUN python3 -m pip install pyfaidx
#############################################
#############################################
############# expose tcp ports
EXPOSE 3306/tcp
EXPOSE 80/tcp
EXPOSE 8080
############# RUN entrypoint.sh
# commented out for testing
ENTRYPOINT ["./entrypoint.sh"]
© 2022 GitHub, Inc.
Terms
When I install the package pyfaidx with default python3.6, it raises an error. I found that python3.8 can install it. Thus, I want to switch to python3.8 to install all py3 packages.
Bash alias that you define in your RUN statement will be available only in the current shell session. When the current RUN statement finishes executing, you exit the session, effectively forgetting any aliases you set up there.
See also: How can I set Bash aliases for docker containers in Dockerfile?
Another option is to use update-alternatives, e.g.,
# update-alternatives --install `which python3` python3 `which python3.8` 20
update-alternatives: using /usr/bin/python3.8 to provide /usr/bin/python3 (python3) in auto mode
# python3 --version
Python 3.8.0
This may interfere with other container packages that do require 3.6 which was default on Ubuntu 18.04 back in the day. Furthermore, pip's authors do not recommend using pip to install system-wide packages like that. In fact, newer pip versions will emit a warning when attempting to use pip globally along the lines of your Dockerfile.
Therefore a better course of action is using a virtualenv:
# apt install -y python3-venv python3.8-venv
...
# python3.8 -m venv /usr/local/venv
# /usr/local/venv/bin/pip install -U pip setuptools wheel
# /usr/local/venv/bin/pip install -U pyfaidx
... (etc)
You can also "enter" your virtualenv by activating it:
root#a1d0210118a8:/# source /usr/local/venv/bin/activate
(venv) root#a1d0210118a8:/# python -V
Python 3.8.0
See also: Use different Python version with virtualenv.
We found 1 vulnerability in base docker image "pyjwt version 2.3.0 has 1 vulnerability" Fixed in version pyjwt 2.4.0
Below is the Dockerfile
FROM ubuntu:22.04
# hadolint ignore=DL3015
# hadolint ignore=DL3008
RUN apt-get clean
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get -y update
RUN apt-get -y upgrade apt \
&& apt-get install -y unoconv ghostscript software-properties-common \
&& add-apt-repository ppa:ondrej/php -y \
&& apt -y install php7.4 \
&& apt-get install -y curl jq php7.4-bcmath php7.4-xml zip unzip php7.4-zip \
&& apt-get install -y php7.4-fpm php7.4-amqp composer nginx openssl php7.4-curl ca-certificates \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" \
&& unzip awscliv2.zip \
&& ./aws/install \
&& rm awscliv2.zip
# Setup services
COPY ./src/scripts/nginx.conf /etc/nginx/nginx.conf
COPY ./src/scripts/run.sh /opt/run.sh
RUN chmod -R a+rw /etc/nginx
RUN chmod -R a+rw /etc/php/7.4/fpm
RUN chmod +x /opt/run.sh
EXPOSE 8080 8443
CMD [ "/opt/run.sh" ]
I have tried many things like update installing python3 and updating pyjwt package with pip install pyjwt==2.4.0. But it didn't work. It seems like one of the above package from Dockerfile is using pyjwt(2.3.0) and I don't know how do i update it.
You can try uninstall python3-jwt package with apt and install new version with pip
RUN apt purge --autoremove python3-jwt -y
RUN pip3 install PyJWT==2.4.0
I tried to install pybgpstream without sudo rights on a machine.
Python installation is system wide. I had to install everything under ~/.local/lib...
After installing libbgstream binary I wanted to install pybgpstream with
pip install --global-option build_ext --global-option '--include-dir=/home/USER/.local/include' --global-option '--library-dir=/home/USER/.local/lib' pybgpstream
After successfull installation I tried running some script with pybgpstream. Doesn't work:
ImportError: /home/USER/.local/lib/python2.7/site-packages/_pybgpstream.so: undefined symbol: _Py_FalseStruct
I added /home/USER/.local/lib/ to LD_PATH...
I don't know what's wrong...
There's a docker solution to work with pybgpstream. Here is the docker file
FROM python:3.8
RUN apt update \
&& apt install -y build-essential curl zlib1g-dev libbz2-dev libcurl4-openssl-dev librdkafka-dev python3-gi-cairo nano \
&& mkdir ~/src && cd ~/src/ && \
curl -LO https://research.wand.net.nz/software/wandio/wandio-4.2.3.tar.gz && \
tar zxf wandio-4.2.3.tar.gz && \
cd wandio-4.2.3/ && ./configure && make install && ldconfig && \
cd ~/src/ && \
curl -LO https://github.com/CAIDA/libbgpstream/releases/download/v2.1.0/libbgpstream-2.1.0.tar.gz && \
tar zxf libbgpstream-2.1.0.tar.gz && \
cd libbgpstream-2.1.0/ && ./configure && make check install && ldconfig && \
pip install pybgpstream && \
pip install ipython && \
pip install statsmodels && \
pip install matplotlib && \
pip install ipykernel && \
pip install pylint && \
pip install autopep8
CMD ["/bin/bash"]
You can also use this container every time and run bgp streams
FROM webdevops/base:ubuntu-16.04
RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install- recommends \
apache2 \
openssh-client \
python3 \
python3-dev \
python3-venv \
python3-psycopg2 \
python3-pip \
pyflakes3 \
pylint3 \
pep8 \
pep257 \
postgresql-client \
libapache2-mod-wsgi-py3 \
&& apt-get clean \
&& rm -fr /var/lib/apt/lists/*
RUN mkdir /var/www/html/hotels-project
RUN cd /var/www/html/hotels-project/ \
&& python3 -m venv hotels-venv \
&& /bin/bash -c "source hotels-venv/bin/activate"
RUN pip install 'django<2.0'
RUN pip install requests
RUN pip install psycopg2
show message:
ERROR: Service 'apache-python' failed to build: The command '/bin/sh
-c pip install 'django<2.0'' returned a non-zero code: 127
You have two issues in your docker file.
Using pip instead of pip3
Activating virtualenv in one step and running commands in another step
In Dockerfile for every RUN step, you get a fresh terminal. So any source command you executed in previous RUN statement is no more active.
So your code should be something like this
RUN cd /var/www/html/hotels-project/ \
&& python3 -m venv hotels-venv \
&& /bin/bash -c "source hotels-venv/bin/activate" \
&& pip3 install -r requirements.txt
And requirements.txt should have below content
django<2.0
requests==X.XX
psycopg2==y.yy
That's how you should do it
How do I install R version 3.4.0 in my docker image. I've installed python using:
RUN yum -y install https://centos6.iuscommunity.org/ius-release.rpm \
&& yum -y install python36u \
&& yum -y install python36u-devel \
&& yum -y install python36u-pip \
&& yum -y install python36u-tkinter.x86_64
Similarly I need to install R:
I've specified following things in file so far for R:
ENV R_BASE_VERSION 3.4.0
RUN Rscript -e 'install.packages("devtools",dependencies=TRUE)' \
&&Rscript -e 'install.packages("methods",dependencies=TRUE)' \
&&Rscript -e 'install.packages("jsonlite",dependencies=TRUE)' \
Please suggest .I'm new to docker
I think I need to do something like below:
ENV R_BASE_VERSION 3.4.1
## Now install R and littler, and create a link for littler in /usr/local/bin
## Also set a default CRAN repo, and make sure littler knows about it too
RUN apt-get update \
&& apt-get install -t unstable -y --no-install-recommends \
littler \
r-cran-littler \
r-base=${R_BASE_VERSION}* \
r-base-dev=${R_BASE_VERSION}* \
r-recommended=${R_BASE_VERSION}* \
&& echo 'options(repos = c(CRAN = "https://cran.rstudio.com/"), download.file.method = "libcurl")' >> /etc/R/Rprofile.site \
&& echo 'source("/etc/R/Rprofile.site")' >> /etc/littler.r \
&& ln -s /usr/share/doc/littler/examples/install.r /usr/local/bin/install.r \
&& ln -s /usr/share/doc/littler/examples/install2.r /usr/local/bin/install2.r \
&& ln -s /usr/share/doc/littler/examples/installGithub.r /usr/local/bin/installGithub.r \
&& ln -s /usr/share/doc/littler/examples/testInstalled.r /usr/local/bin/testInstalled.r \
&& install.r docopt \
&& rm -rf /tmp/downloaded_packages/ /tmp/*.rds \
&& rm -rf /var/lib/apt/lists/*
But I do not know what is this litter and all. I just need R to be installed and then i will install required packages as I have specified above.
Edits : First line in my docker file installs node4.
Here are two DockerFile to install Python, R and NodeJS
The first one installs Python 3.4.2, R 3.1.1 and nodejs 4.8.4:
From node:4
RUN apt-get update && apt-get remove -y python && apt-get install -y python3 r-base
RUN cp /usr/bin/python3 /usr/bin/python
This second one installs Python 3.5.3, R 3.4.1 and nodejs 4.8.4:
From r-base:3.4.1
RUN apt-get update && apt-get install -y python3 nodejs
RUN cp /usr/bin/python3 /usr/bin/python
Choose the one that best fits your needs.
If your public base image (the base image of your own image) is really node:4, then it is not yum based but apt-get based to manage packages.
Thus you shoud install R the following way:
RUN apt-get update && apt-get install -y r-base
You should use some R images, like
https://hub.docker.com/_/r-base/
or some oher image in this list
https://hub.docker.com/search/?isAutomated=0&isOfficial=0&page=1&pullCount=0&q=R&starCount=0