We Keep Coding

Docker Invalid Reference Format Error while building in bamboo

In my dockerfile, I was previously using FROM python:3.9-alpineon top of which librdkafka 1.9.2 is built and this was successful. But today, with the same docker file, the build failed by throwing the below error:
#error "confluent-kafka-python requires librdkafka v2.0.2 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html".
When I searched in the internet, alpine:edge seems to have the newest version of librdkafka package. So I changed the dockerfile to FROM python:3.9-alpine:edge. But on building, this threw my an error:
Step 1/41 : FROM python:3.9-alpine:edge build 25-Jan-2023 10:25:20 invalid reference format build 25-Jan-2023 10:25:20 [?1h=[41m[37;1mAn error occurred when executing task '
I am new to docker and I used https://www.docker.com/blog/how-to-use-the-alpine-docker-official-image/ for the format. Please do help me regarding this.
This is my dockerfile currently:
FROM python:3.9-alpine:edge
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
# COPY kong.ini kong.ini
# COPY iot.ini iot.ini
# COPY project.ini project.ini
# COPY eom.ini eom.ini
# COPY notify.ini notify.ini
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]```

Error while building confluent-kafka in docker for alphine image

I am trying to build a python application which require confluent-kafka package. But while building in bamboo, I got the below error
fatal error: librdkafka/rdkafka.h: No such file or directory
build 13-Dec-2022 11:46:59 23 | #include <librdkafka/rdkafka.h>
build 13-Dec-2022 11:46:59 | ^~~~~~~~~~~~~~~~~~~~~~
My dockerfile is as such:
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
# libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
I am unable to gauge why the error is coming since librdkafka is already installed.My requirement is to use alpine image. Can anyone please help me regarding this?

Docker build process and persistence of files to final image for use in container

I'm working on containerizing a somewhat complex application, and I'm running into some issues that are probably down to a lack of understanding of how Docker works and I've done a large amount of googling and reading but still haven't seem to gotten a solution.
I'm currently using docker-compose to launch the containers, and for building using docker-compose up --build at build time.
I've got a .dockerignore file going to try and limit my build context as best I can or build-times can take a very long time.
my docker-compose.yml looks something like
services:
base_image:
container_name: base_image_generator
build:
context: .
My dockerfile looks something like
FROM ubuntu:20.04 as dds_install
WORKDIR /app
COPY ./rti/rti_connext_dds-6.1.0-pro-host-x64Linux.run \
./rti/rti_connext_dds-6.1.0-pro-target-x64Linux4gcc7.3.0.rtipkg ./
RUN chmod +x rti_connext_dds-6.1.0-pro-host-x64Linux.run \
&& ./rti_connext_dds-6.1.0-pro-host-x64Linux.run --mode unattended
RUN /opt/rti_connext_dds-6.1.0/bin/rtipkginstall \
-u rti_connext_dds-6.1.0-pro-target-x64Linux4gcc7.3.0.rtipkg
FROM ubuntu:20.04
ENV TZ="America/Los Angeles"
ENV NODE_OPTIONS=--max_old_space_size=16384
ARG DEBIAN_FRONTEND=noninteractive
# Copy the data from the dds build
COPY --from=dds_install \
/opt/rti_connext_dds-6.1.0 /opt/rti_connext_dds-6.1.0
# Copy in the license file
COPY ./rti/rti_license.dat /opt/rti_connext_dds-6.0.1
# Add the NDDSHOME to our path
ENV NDDSHOME /opt/rti_connext_dds-6.1.0
ENV PATH $PATH:$NDDSHOME/bin
RUN apt-get update && apt-get install -y software-properties-common \
&& add-apt-repository ppa:deadsnakes/ppa \
&& apt-get update \
&& apt-get install -y \
python3.6 \
python3.6-venv \
python3.6-dev
RUN apt-get update \
&& apt-get -y upgrade \
&& apt-get install -y \
build-essential \
cmake \
cmake-gui \
curl \
ffmpeg \
git \
gnupg2 \
gnome-keyring \
libboost-chrono-dev \
libboost-filesystem-dev \
libboost-program-options-dev \
libboost-system-dev \
libboost-thread-dev \
libboost-timer-dev \
libcurl4-openssl-dev \
libssl-dev \
pass \
python3-pkg-resources \
python3-pip \
python3-venv \
python-is-python3 \
tzdata \
vim \
wget \
nano
RUN python3.6 -m venv /root/venv
ENV PATH="/root/venv/bin:$PATH"
# Get node 14, this is the LTS version of node and install it
RUN curl -fsSL https://deb.nodesource.com/setup_14.x | bash -
RUN apt-get update && apt-get -y install nodejs
RUN npm install -g #angular/cli
RUN pip install --upgrade pip && pip install \
cython \
jinja2 \
pip-login \
pycurl \
setuptools \
vcstool \
wheel \
xmltodict \
gdal==2.2.3
I then tag the image that is generated, and move onto another docker-compose.yml that uses the generated image, and use another Dockerfile to copy source code into the Docker build context, and do a build process, and login to a private pip repo and pull down some python packages using pip.
The issue I'm running into is that my python venv isn't persisting. If I run the pip list command prior to the docker build finishing by issuing a RUN pip list the list that is printed is as expected. When I issue a docker-compose up, to bring a container up my app doesn't function properly though because it's missing all of the python packages that I installed, although running back using docker exec -it <container_name> bash shows that my virtualenv is being used (verified with a which pip)

Docker image taking long to build

Here are the contents of the Dockerfile, i changed from an alpine image to the slim-buster image. Im really struggling to see why its taking so long i think its got to do with all the aps im updating and installing in apt-get update. I might be reinstalling packages i don't need perhaps or doing something i don't need to, is there a way i can speed this up?
# pull official base image
FROM python:3.8-slim-buster
# set work directory
WORKDIR /opt/workspace
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
# Need this crap for wkhtml old install because any version after this doesn't work with charts and javascript
RUN echo "deb http://security.debian.org/debian-security jessie/updates main" >> /etc/apt/sources.list
# Pillow and Psycopg Dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
wget \
libpq-dev \
libpng-dev \
libjpeg-dev \
python-dev \
postgresql-client \
python3-pip \
python3-setuptools \
python3-wheel \
python3-cffi \
libssl1.0.0 \
libpng12-0 \
xfonts-base \
xfonts-75dpi \
libcairo2 \
libpango-1.0-0 \
libpangocairo-1.0-0 \
libgdk-pixbuf2.0-0 \
libffi-dev \
shared-mime-info\
gcc \
musl-dev \
python3-dev \
tk-dev \
uuid-dev \
&& rm -rf /var/lib/apt/lists/*
# fetch wait for it script
RUN wget -q -O /usr/bin/wait-for-it https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh && \
chmod +x /usr/bin/wait-for-it
RUN pip install psycopg2
# bunch of wkhtmltopdf shit only works with charts on this image.
RUN wget http://archive.ubuntu.com/ubuntu/pool/main/libj/libjpeg-turbo/libjpeg-turbo8_2.0.3-0ubuntu1_amd64.deb
RUN dpkg -i libjpeg-turbo8_2.0.3-0ubuntu1_amd64.deb
RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
RUN dpkg -i wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
# Install Dependencies
COPY requirements.txt /opt/workspace/requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# copy entrypoint.sh
COPY ./entrypoint.sh /opt/workspace/entrypoint.sh
# copy project
COPY . /opt/workspace
# run entrypoint.sh
ENTRYPOINT ["/opt/workspace/entrypoint.sh"]
EDIT:
After Receiving an answer I have updated my dockerfile to the following. Bare in mind the only packages i need to install using apt-get are the packages that wkhtmltopdf requires to run! it is running so much faster now and is working a lot better.
FROM python:3.8-slim-buster
# set work directory
WORKDIR /opt/workspace
# set environment variables
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1
ENV DEBIAN_FRONTEND=noninteractive
# Need this crap for wkhtml old install
RUN echo "deb http://security.debian.org/debian-security jessie/updates main" >> /etc/apt/sources.list
RUN apt-get update && apt-get install -y --no-install-recommends \
wget \
fontconfig \
xfonts-base \
xfonts-75dpi \
libssl1.0.0 \
libpq-dev \
libpng-dev \
libjpeg-dev \
libffi-dev \
libpng12-0 \
libxext6 \
libx11-6 \
libxrender1 \
gcc \
&& rm -rf /var/lib/apt/lists/*
# fetch wait for it script
RUN wget -q -O /usr/bin/wait-for-it https://raw.githubusercontent.com/vishnubob/wait-for-it/master/wait-for-it.sh && \
chmod +x /usr/bin/wait-for-it
# bunch of wkhtmltopdf shit only works with charts on this image.
RUN wget http://archive.ubuntu.com/ubuntu/pool/main/libj/libjpeg-turbo/libjpeg-turbo8_2.0.3-0ubuntu1_amd64.deb
RUN dpkg -i libjpeg-turbo8_2.0.3-0ubuntu1_amd64.deb
RUN wget https://github.com/wkhtmltopdf/wkhtmltopdf/releases/download/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
RUN dpkg -i wkhtmltox-0.12.2.1_linux-trusty-amd64.deb
# Install Dependencies
COPY requirements.txt /opt/workspace/requirements.txt
RUN pip install --no-cache-dir -r requirements.txt
# copy entrypoint.sh
COPY ./entrypoint.sh /opt/workspace/entrypoint.sh
# copy project
COPY . /opt/workspace
# run entrypoint.sh
ENTRYPOINT ["/opt/workspace/entrypoint.sh"]

There's a bunch of issues here. First, there are packages you don't need:
You're installing python twice. You're installing python, the Debian Python package, but the Docker python image has its own version of Python (in /usr/local), with dev headers already there. You don't need this, and it can lead to confusion because you end up with two versions of Python (https://pythonspeed.com/articles/importerror-docker/).
musl-dev is unnecessary. Debian uses glibc, not musl. I suspect this is holdover from Alpine.
You are installing a compiler, a whole bunch of C headers in general, all those *-dev packages. It's quite possible you don't need to at all! On Alpine, you need to compile everything because Alpine can't use normal binary wheels. (https://pythonspeed.com/articles/alpine-docker-python/) Since you're on Debian, quite possibly all your depedencies have binary wheels. You would still need a compiler for your own code, but if it's pure Python quite possibly not.
So my suggestion: just drop the whole apt-get line. Pretty good chance it'll Just Work without it.

can not build qtermwidget from source

I'm trying to build qtermwidget from source, but it gives me error.
I've successfully built lxqt-build-tools and then installed pyqt5 from both pip3 and apt:
sudo -H pip3 install -U pyqt5 pyqtwebengine
sudo apt install python3-sip-dev python3-pyqt5
and then I ran this:
mkdir -p /tmp/EAF && cd /tmp/EAF
git clone https://github.com/lxqt/qtermwidget
cd qtermwidget
mkdir build && cd build
cmake .. -DQTERMWIDGET_BUILD_PYTHON_BINDING=ON -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=/usr
and it works fine. but when I run make command, it gives me this error:
[ 87%] Built target qtermwidget5
Byte-compiling /tmp/EAF/qtermwidget/build/pyqt//__init__.py to /tmp/EAF/qtermwidget/build/pyqt//__pycache__/__init__.cpython-36.pyc
[ 87%] Built target __tmp_EAF_qtermwidget_build_pyqt____pycache_____init__.cpython-36.pyc
[ 89%] Generating sip/sipQTermWidgetpart0.cpp, sip/sipQTermWidgetpart1.cpp, sip/sipQTermWidgetpart2.cpp, sip/sipQTermWidgetpart3.cpp, sip/sipQTermWidgetpart4.cpp, sip/sipQTermWidgetpart5.cpp, sip/sipQTermWidgetpart6.cpp, sip/sipQTermWidgetpart7.cpp
sip: Unable to find file "QtGui/QtGuimod.sip"
pyqt/CMakeFiles/python_module_QTermWidget.dir/build.make:62: recipe for target 'pyqt/sip/sipQTermWidgetpart0.cpp' failed
make[2]: *** [pyqt/sip/sipQTermWidgetpart0.cpp] Error 1
make[2]: *** Deleting file 'pyqt/sip/sipQTermWidgetpart0.cpp'
CMakeFiles/Makefile2:179: recipe for target 'pyqt/CMakeFiles/python_module_QTermWidget.dir/all' failed
make[1]: *** [pyqt/CMakeFiles/python_module_QTermWidget.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2
I'm using gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 and cmake version 3.16.0 and GNU Make 4.1 in elementary OS 5.1 Hera. I once tried building sip and pyqt5 from source, didn't change anything for me.

The PyQt5 distributed by Ubuntu does not share the necessary .sip to compile QTermWidget so it is necessary to compile sip and pyqt5 manually. It seems you tried and it didn't work since it seems you used the wrong flags. Considering the above, I analyzed how sip, pyqt5 and qtermwidget are compiled in Arch Linux and managed to implement a Dockerfile that allowed me to compile QTermWidget.
So considering the above the procedure is:
sudo apt-get update && apt-get install \
-y --no-install-recommends \
build-essential \
git \
ca-certificates \
wget \
cmake \
pkg-config \
python3-dev \
libglib2.0-dev \
qt5-default \
qttools5-dev
mkdir -p /tmp/EAF
cd /tmp/EAF && \
git clone https://github.com/lxqt/lxqt-build-tools.git \
&& cd lxqt-build-tools \
&& mkdir build && cd build \
&& cmake .. \
&& make && sudo make install
cd /tmp/EAF && \
wget https://www.riverbankcomputing.com/static/Downloads/sip/4.19.19/sip-4.19.19.tar.gz && \
tar xvzf sip-4.19.19.tar.gz && \
cd sip-4.19.19 && \
python3 configure.py --sip-module PyQt5.sip && \
make && \
sudo make install
cd /tmp/EAF && \
wget https://www.riverbankcomputing.com/static/Downloads/PyQt5/5.13.2/PyQt5-5.13.2.tar.gz && \
tar xvzf PyQt5-5.13.2.tar.gz && \
cd PyQt5-5.13.2 && \
python3 configure.py --confirm-license && \
make && \
sudo make install
cd /tmp/EAF && \
git clone https://github.com/lxqt/qtermwidget \
&& cd qtermwidget \
&& mkdir build && cd build \
&& cmake .. -DQTERMWIDGET_BUILD_PYTHON_BINDING=ON -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_INSTALL_LIBDIR=/lib \
&& make && sudo make install