I'm trying to update an existing Dockerfile to switch from python3.5 to python3.8, previously it was creating a symlink for python3.5 and pip3 like this:
RUN ln -s /usr/bin/pip3 /usr/bin/pip
RUN ln -s /usr/bin/python3 /usr/bin/python
I've updated the Dockerfile to install python3.8 from deadsnakes:ppa
apt-get install python3-pip python3.8-dev python3.8-distutils python3.8-venv
if I remove python3-pip, it complains about gcc
C compiler or Python headers are not installed on this system. Try to run: sudo apt-get install gcc python3-dev
with these installations in place I'm trying to update existing symlink creation something like this:
RUN ln -s /usr/bin/pip3 /usr/local/lib/python3.8/dist-packages/pip
RUN ln -s /usr/bin/pip /usr/local/lib/python3.8/dist-packages/pip
RUN ln -s /usr/bin/python3.8 /usr/bin/python3
it fails, saying
ln: failed to create symbolic link '/usr/bin/python3': File exists
which I assume fails because python3 points to python3.6.
if I try: RUN ln -s /usr/bin/python3.8 /usr/bin/python it doesn't complain about symlink and image gets build successfully, but fails while installing requirements later (we use Makefile targets to install dependencies inside the container using pip and pip-sync):
ERROR: Cannot uninstall 'python-apt'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
which I assume because python-apt gets installed as part of the default python3.6 installation and python3.8 pip can't uninstall it.
PS: my Dockerfile image is based on Ubunut 18.04 which comes with python3.6 as default.
How can I properly switch Dockerfile / image from python3.5 to python3.8? so I can later use pip directly and it points to python3.8's pip
Replacing the system python in this way is usually not a good idea (as it can break operating-system-level programs which depend on those executables) -- I go over that a little bit in this video I made "why not global pip / virtualenv?"
A better way is to create a prefix and put that on the PATH earlier (this allows system executables to continue to work, but bare python / python3 / etc. will use your other executable)
in the case of deadsnakes which it seems like you're using, something like this should work:
FROM ubuntu:bionic
RUN : \
&& apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
software-properties-common \
&& add-apt-repository -y ppa:deadsnakes \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
python3.8-venv \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& :
RUN python3.8 -m venv /venv
ENV PATH=/venv/bin:$PATH
the ENV line is the key here, that puts the virtualenv on the beginning of the path
$ docker build -t test .
...
$ docker run --rm -ti test bash -c 'which python && python --version && which pip && pip --version'
/venv/bin/python
Python 3.8.5
/venv/bin/pip
pip 20.1.1 from /venv/lib/python3.8/site-packages/pip (python 3.8)
disclaimer: I'm the maintainer of deadsnakes
Why not just build a new image from ubuntu:18.04 with the desired config you need?
Like this:
FROM ubuntu:18.04
RUN apt update && apt install software-properties-common -y
RUN add-apt-repository ppa:deadsnakes/ppa && install python3.8 -y
RUN ln -s /usr/bin/pip3 /usr/bin/pip && \
ln -s /usr/bin/python3.8 /usr/bin/python
You can install and enable your python version.
# Python 3.8 and pip3
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa -y
RUN apt-get install -y python3.8
RUN ln -s /usr/bin/python3.8 /usr/bin/python
RUN apt-get install -y python3-pip
Sometimes, modifying the OS (like getting new Ubuntu clean os) is not favorable, because the current OS is too complicated. For example, my base OS is FROM ufoym/deepo:all-cu101.
So, to modify the existing python (3.6) to python 3.8, I added these 2 lines:
RUN apt-get update -qq && apt-get install -y -qq python3.8
RUN rm /usr/bin/python && rm /usr/bin/python3 && ln -s /usr/bin/python3.8 /usr/bin/python && ln -s /usr/bin/python3.8 /usr/bin/python3 \
&& rm /usr/local/bin/python && rm /usr/local/bin/python3 && ln -s /usr/bin/python3.8 /usr/local/bin/python && ln -s /usr/bin/python3.8 /usr/local/bin/python3 \
&& apt-get install -y python3-pip python-dev python3.8-dev && python3 -m pip install pip --upgrade
The first step is to install the python3.8;
The second step is to modify the softlink of python and python3 to point to python3.8
After that, install python3-pip, and update it to make sure the pip is using the current python 3.8 environment.
Related
I tried to alias python3 to python3.8 in the Dockerfile. But It doesn't work for me. I am using ubuntu:18.04.
Step 25/41 : RUN apt-get update && apt-get install -y python3.8
---> Using cache
---> 9fa81ca14a53
Step 26/41 : RUN alias python3="python3.8" && python3 --version
---> Running in d7232d3c8b8f
Python 3.6.9
As you can see the python3 is still 3.6.9. How can I fix this issue?
Thanks.
EDIT
Just attach my Dockerfile:
##################################################################################################################
# Build
#################################################################################################################
#FROM openjdk:8
FROM ubuntu:18.04
############## Linux and perl packages ###############
RUN apt-get update && \
apt-get install -y openjdk-8-jdk && \
apt-get install -y ant && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer && \
apt-get update -y && \
apt-get install curl groff python-gdbm -y;
# Fix certificate issues, found as of
# https://bugs.launchpad.net/ubuntu/+source/ca-certificates-java/+bug/983302
RUN apt-get update && \
apt-get install -y ca-certificates-java && \
apt-get clean && \
update-ca-certificates -f && \
rm -rf /var/lib/apt/lists/* && \
rm -rf /var/cache/oracle-jdk8-installer;
# Setup JAVA_HOME, this is useful for docker commandline
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64/
RUN export JAVA_HOME
# install git
RUN apt-get update && \
apt-get install -y mysql-server && \
apt-get install -y uuid-runtime git jq python python-dev python-pip python-virtualenv libdbd-mysql-perl && \
rm -rf /var/lib/apt/lists/* && \
apt-get install perl && \
perl -MCPAN -e 'CPAN::Shell->install("Inline")' && \
perl -MCPAN -e 'CPAN::Shell->install("DBI")' && \
perl -MCPAN -e 'CPAN::Shell->install("List::MoreUtils")' && \
perl -MCPAN -e 'CPAN::Shell->install("Inline::Python")' && \
perl -MCPAN -e 'CPAN::Shell->install("LWP::Simple")' && \
perl -MCPAN -e 'CPAN::Shell->install("JSON")' && \
perl -MCPAN -e 'CPAN::Shell->install("LWP::Protocol::https")';
RUN apt-get update && \
apt-get install --yes cpanminus
RUN cpanm \
CPAN::Meta \
YAML \
DBI \
Digest::SHA \
Module::Build \
Test::Most \
Test::Weaken \
Test::Memory::Cycle \
Clone
# Install perl modules for network and SSL (and their dependencies)
RUN apt-get install --yes \
openssl \
libssl-dev \
liblwp-protocol-https-perl
RUN cpanm \
LWP \
LWP::Protocol::https
# New module for v1.2 annotation
RUN perl -MCPAN -e 'CPAN::Shell->install("Text::NSP::Measures::2D::Fisher::twotailed")'
#############################################
############## python packages ###############
# python packages
RUN pip install pymysql==0.10.1 awscli boto3 pandas docopt fastnumbers tqdm pygr
############## python3 packages ###############
# python3 packages
RUN apt-get update && \
apt-get install -y python3-pip && \
python3 -m pip install numpy && \
python3 -m pip install pandas && \
python3 -m pip install sqlalchemy && \
python3 -m pip install boto3 && \
python3 -m pip install pymysql && \
python3 -m pip install pymongo;
RUN python3 -m pip install pyfaidx
#############################################
#############################################
############# expose tcp ports
EXPOSE 3306/tcp
EXPOSE 80/tcp
EXPOSE 8080
############# RUN entrypoint.sh
# commented out for testing
ENTRYPOINT ["./entrypoint.sh"]
© 2022 GitHub, Inc.
Terms
When I install the package pyfaidx with default python3.6, it raises an error. I found that python3.8 can install it. Thus, I want to switch to python3.8 to install all py3 packages.
Bash alias that you define in your RUN statement will be available only in the current shell session. When the current RUN statement finishes executing, you exit the session, effectively forgetting any aliases you set up there.
See also: How can I set Bash aliases for docker containers in Dockerfile?
Another option is to use update-alternatives, e.g.,
# update-alternatives --install `which python3` python3 `which python3.8` 20
update-alternatives: using /usr/bin/python3.8 to provide /usr/bin/python3 (python3) in auto mode
# python3 --version
Python 3.8.0
This may interfere with other container packages that do require 3.6 which was default on Ubuntu 18.04 back in the day. Furthermore, pip's authors do not recommend using pip to install system-wide packages like that. In fact, newer pip versions will emit a warning when attempting to use pip globally along the lines of your Dockerfile.
Therefore a better course of action is using a virtualenv:
# apt install -y python3-venv python3.8-venv
...
# python3.8 -m venv /usr/local/venv
# /usr/local/venv/bin/pip install -U pip setuptools wheel
# /usr/local/venv/bin/pip install -U pyfaidx
... (etc)
You can also "enter" your virtualenv by activating it:
root#a1d0210118a8:/# source /usr/local/venv/bin/activate
(venv) root#a1d0210118a8:/# python -V
Python 3.8.0
See also: Use different Python version with virtualenv.
I am trying to install python and pip & Ansible using Dockerfile but I get this error
/bin/sh: 1: python: not found
The command '/bin/sh -c curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py && python get-pip.py && python -m pip install --upgrade "pip < 21.0" && pip install ansible --upgrade' returned a non-zero code: 127
ERROR: Service 'jenkins' failed to build : Build failed
Here is my Dockerfile:
FROM jenkins/jenkins
USER root
RUN curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py && \
python get-pip.py && \
python -m pip install --upgrade "pip < 21.0" && \
pip install ansible --upgrade
USER jenkins
Note: I used the same instructions on another Dockerfile and it went without errors. Here is the Dockerfile from CentOS image:
FROM centos:7
RUN yum update -y && \
yum -y install openssh-server && \
yum install -y passwd
RUN useradd remote_user && \
echo "password" | passwd remote_user --stdin && \
mkdir /home/remote_user/.ssh && \
chmod 700 /home/remote_user/.ssh
COPY remote-key.pub /home/remote_user/.ssh/authorized_keys
RUN chown remote_user:remote_user -R /home/remote_user && \
chmod 600 /home/remote_user/.ssh/authorized_keys
RUN /usr/sbin/sshd-keygen
RUN yum -y install mysql
RUN curl -O https://bootstrap.pypa.io/pip/2.7/get-pip.py && \
python get-pip.py && \
python -m pip install --upgrade "pip < 21.0" && \
pip install awscli --upgrade
CMD /usr/sbin/sshd -D
Since I'm not entirely sure my comments were fully understandable, here is how I would install ansible in your current base image jenkins/jenkins.
Notes:
I fixed the tag to lts since building from latest is a bit on the edge. You can change that to whatever tag suits your needs.
That base image is itself based on Ubuntu and not CentOS as reported in your title (hence using apt and not yum/dnf)
I used two RUN directives (one for installing python, the other for ansible) but you can merge them in a single instruction if you want to further limit the number of layers.
FROM jenkins/jenkins:lts
USER root
RUN apt-get update && \
apt-get install -y python3-pip && \
rm -rf /var/lib/apt/lists/*
RUN pip install --upgrade pip && \
pip install ansible && \
pip cache purge
USER jenkins
I deleted RUN instructions and replaced it with :
RUN apt-get update
RUN apt-get install -y ansible
Worked like a charm.
I have a couple of Python modules that I use inside my Docker container and they require a higher version of Python that what's being used. I install Python and install the modules using:
RUN apt-get update || : && apt-get install python3 -y
RUN apt-get install -y python3-pip
COPY requirements.txt /project
RUN pip3 install -r requirements.txt
Expecting I would be using the latest version of Python in my Docker container but when I go into it's shell and run python3 --version is comes as 3.4.2 which is incredibly old for my program. How do I make the default Python to be the latest I installed above without messing over the System-level python?
The image runtime I'm using for the Docker container is: node:9-slim
I don't think you can find a prebuilt python3.9 package on a debian 8 distribution as your environment is pretty old.
The only solution is you build the python3.9 out from source code in your base container. A full workable Dockerfile as next:
FROM node:9-slim
RUN apt update; \
apt install -y build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev; \
wget https://www.python.org/ftp/python/3.9.7/Python-3.9.7.tgz; \
tar -zxvf Python-3.9.7.tgz; \
cd Python-3.9.7; \
./configure --prefix=/usr/local/python3; \
make && make install; \
ln -sf /usr/local/python3/bin/python3.9 /usr/bin/python3; \
ln -sf /usr/local/python3/bin/pip3.9 /usr/bin/pip3
Verify it:
$ docker build -t myimage:1 .
$ docker run --rm -it myimage:1 python3 --version
Python 3.9.7
$ docker run --rm -it myimage:1 pip3 --version
pip 21.2.3 from /usr/local/python3/lib/python3.9/site-packages/pip (python 3.9)
I am building a ubuntu docker image that is going to run my python application, and I have some libraries that require python <= 3.6 to work otherwise it will throw errors.
My problem is that when I install pip, it will always automatically use python 3.8, and I'm not sure how to let pip use an older version of python, this is the installation in my Dockerfile
RUN apt-get update && \
apt-get upgrade -y && \
apt-get install -y software-properties-common && \
add-apt-repository ppa:deadsnakes/ppa && \
apt-add-repository universe && \
apt-get update && \
apt-get install -y \
libmysqlclient-dev \
netcat \
python3 \
python-dev \
build-essential \
python3-setuptools \
python3-pip \
supervisor && \
pip install -U pip setuptools && \
rm -rf /var/lib/apt/lists/*
I tried to change python3-pip by just python-pip but when I run it it gives me the following error
E: Unable to locate package python-pip
I've tried a lot of solutions but always the same problem
Outside of Docker, if python3.6 is the python you need, you can do:
python3.6 -m pip install
In Docker right now obviously python3 is pointing to Python 3.8 so you must first install python3.6 and find out how to call it (python3.6 or python3). You might need to compile it from source and probably create some symbolic link. This can get very ugly to do inside a Docker, but you can try to write a shell script with all commands and to run the shell script inside a Docker. Or if you are lucky you may find a ready Python3.6 Docker package that works for you and apt-get install it instead of python3 the same way as you do now.
I installed Python 3.6 on Ubuntu 16.04 on Docker using the ppa:jonathonf/python-3.6 repository. Now I'd like to install xapian so I can use it with Python. I have not found any ready-made packages, so I am trying to build it from sources. I set PYTHON3 and PYTHON3_LIB parameters to point to Python 3.6. During the build process I get the following error:
ImportError: libxapian.so.30: cannot open shared object file: No such file or directory
I tried xapian versions 1.3.7 and 1.4.5 without luck.
How can I install xapian?
Here's a Dockerfile to reproduce my error:
FROM ubuntu:16.04
RUN apt-get update \
&& apt-get install -y software-properties-common python-software-properties
RUN add-apt-repository ppa:jonathonf/python-3.6
RUN apt-get update \
&& apt-get install -y python3-pip docker.io python3.6 python3.6-dev software-properties-common \
python-software-properties build-essential wget unzip cmake python3-sphinx \
&& cd /usr/local/bin \
&& ln -s /usr/bin/python3.6 python
RUN python -m pip install --upgrade pip
# install xapian 1.4.5
RUN apt-get update && apt-get install -y curl uuid-dev zlib1g-dev
WORKDIR /root
RUN curl --silent --show-error --fail --next -O https://oligarchy.co.uk/xapian/1.4.5/xapian-core-1.4.5.tar.xz
RUN curl --silent --show-error --fail --next -O https://oligarchy.co.uk/xapian/1.4.5/xapian-bindings-1.4.5.tar.xz
RUN tar xvf xapian-core-1.4.5.tar.xz
RUN tar xvf xapian-bindings-1.4.5.tar.xz
WORKDIR /root/xapian-core-1.4.5
RUN ./configure && make && make install
WORKDIR /root/xapian-bindings-1.4.5
RUN ./configure PYTHON3=/usr/bin/python3.6 PYTHON3_LIB=/usr/lib/python3.6 --with-python3 && make && make install
RUN python -c "import xapian"
The problem is that the Xapian library (libxapian.so.30) is being installed into /usr/local/lib by default, but Ubuntu doesn't know that it's been put there yet. You can tell it by adding:
RUN ldconfig
after installing the core (so before you change WORKDIR to build the bindings).
There's some helpful information about ldconfig and library search paths on Ubuntu in the answers to this Unix Stackexchange question.