We Keep Coding

Docker Invalid Reference Format Error while building in bamboo

In my dockerfile, I was previously using FROM python:3.9-alpineon top of which librdkafka 1.9.2 is built and this was successful. But today, with the same docker file, the build failed by throwing the below error:
#error "confluent-kafka-python requires librdkafka v2.0.2 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html".
When I searched in the internet, alpine:edge seems to have the newest version of librdkafka package. So I changed the dockerfile to FROM python:3.9-alpine:edge. But on building, this threw my an error:
Step 1/41 : FROM python:3.9-alpine:edge build 25-Jan-2023 10:25:20 invalid reference format build 25-Jan-2023 10:25:20 [?1h=[41m[37;1mAn error occurred when executing task '
I am new to docker and I used https://www.docker.com/blog/how-to-use-the-alpine-docker-official-image/ for the format. Please do help me regarding this.
This is my dockerfile currently:
FROM python:3.9-alpine:edge
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
# COPY kong.ini kong.ini
# COPY iot.ini iot.ini
# COPY project.ini project.ini
# COPY eom.ini eom.ini
# COPY notify.ini notify.ini
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]```

AWS lambda stuck importing library

I am trying to deploy my ECR image to aws lambda. The image works fine locally, but on aws, it gets stuck importing this library https://github.com/jianfch/stable-ts.
import json
import boto3
import requests
import numpy
print("All imports ok 1 ...")
from stable_whisper import load_model
print("All imports ok 2 ...")
The first statement is printed but it gets stuck on importing and the second statement never got printed until it timed out.
Docker File:
# Build FFmpeg
FROM public.ecr.aws/lambda/python:3.8 as lambda-base
COPY requirements.txt ./
COPY myfunction.py ./
RUN pip3 install -r requirements.txt
WORKDIR /ffmpeg_sources
RUN yum install autoconf automake bzip2 bzip2-devel cmake libxcb libxcb-devel \
freetype-devel gcc gcc-c++ git libtool make pkgconfig zlib-devel -y -q
# Compile NASM assembler
RUN curl -OL https://www.nasm.us/pub/nasm/releasebuilds/2.15.05/nasm-2.15.05.tar.bz2
RUN tar xjvf nasm-2.15.05.tar.bz2
RUN cd nasm-2.15.05 && sh autogen.sh && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile Yasm assembler
RUN curl -OL https://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz
RUN tar xzvf yasm-1.3.0.tar.gz
RUN cd yasm-1.3.0 && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile FFmpeg
RUN curl -OL https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2
RUN tar xjvf ffmpeg-snapshot.tar.bz2
RUN cd ffmpeg && \
export PATH="/ffmpeg_sources/bin:$PATH" && \
export PKG_CONFIG_PATH="/ffmpeg_sources/ffmpeg_build/lib/pkgconfig" && \
./configure \
--prefix="/ffmpeg_sources/ffmpeg_build" \
--pkg-config-flags="--static" \
--extra-cflags="-I/ffmpeg_sources/ffmpeg_build/include" \
--extra-ldflags="-L/ffmpeg_sources/ffmpeg_build/lib" \
--extra-libs=-lpthread \
--extra-libs=-lm \
--enable-libxcb \
--bindir="/ffmpeg_sources/bin" && \
make && \
make install
# Final image with code and dependencies
FROM lambda-base
COPY myfunction.py /var/task/
CMD ["myfunction.lambda_handler"]
inside the requirements.txt, I tried both stable-ts and git+https://github.com/jianfch/stable-ts.git
I appreciate any help.

stable_whisper has a lot of dependencies and some of them contain compiled code (ffmpeg).
Python packages that contain compiled code aren't always compatible with Lambda runtimes by default. I don’t know how to build ffmpeg, but I can point you to a useful AWS sample, which utilizes python packages based on this dependency. Maybe it will contribute to solving your problem or maybe others will be able to help you further.
Sample Dockerfile:
# Install dependencies
…
# Build FFmpeg
FROM public.ecr.aws/lambda/python:3.8 as ffmpeg
WORKDIR /ffmpeg_sources
RUN yum install autoconf automake bzip2 bzip2-devel cmake libxcb libxcb-devel \
freetype-devel gcc gcc-c++ git libtool make pkgconfig zlib-devel -y -q
# Compile NASM assembler
RUN curl -OL https://www.nasm.us/pub/nasm/releasebuilds/2.15.05/nasm-2.15.05.tar.bz2
RUN tar xjvf nasm-2.15.05.tar.bz2
RUN cd nasm-2.15.05 && sh autogen.sh && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile Yasm assembler
RUN curl -OL https://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz
RUN tar xzvf yasm-1.3.0.tar.gz
RUN cd yasm-1.3.0 && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile FFmpeg
RUN curl -OL https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2
RUN tar xjvf ffmpeg-snapshot.tar.bz2
RUN cd ffmpeg && \
export PATH="/ffmpeg_sources/bin:$PATH" && \
export PKG_CONFIG_PATH="/ffmpeg_sources/ffmpeg_build/lib/pkgconfig" && \
./configure \
--prefix="/ffmpeg_sources/ffmpeg_build" \
--pkg-config-flags="--static" \
--extra-cflags="-I/ffmpeg_sources/ffmpeg_build/include" \
--extra-ldflags="-L/ffmpeg_sources/ffmpeg_build/lib" \
--extra-libs=-lpthread \
--extra-libs=-lm \
--enable-libxcb \
--bindir="/ffmpeg_sources/bin" && \
make && \
make install
# Final image with code and dependencies
FROM lambda-base
# Copy FFMpeg binary
COPY --from=ffmpeg /ffmpeg_sources/bin/ffmpeg /usr/bin/

Error while building confluent-kafka in docker for alphine image

I am trying to build a python application which require confluent-kafka package. But while building in bamboo, I got the below error
fatal error: librdkafka/rdkafka.h: No such file or directory
build 13-Dec-2022 11:46:59 23 | #include <librdkafka/rdkafka.h>
build 13-Dec-2022 11:46:59 | ^~~~~~~~~~~~~~~~~~~~~~
My dockerfile is as such:
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
# libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
I am unable to gauge why the error is coming since librdkafka is already installed.My requirement is to use alpine image. Can anyone please help me regarding this?

How to install Python2.7.5 in Ubuntu docker image?

I have specific requirement to install Python 2.7.5 in Ubuntu, I could install 2.7.18 without any issues
Below is my dockerfile
ARG UBUNTU_VERSION=18.04
FROM ubuntu:$UBUNTU_VERSION
RUN apt-get update -y \
&& apt-get install -y python2.7.x \
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["python"]
however if I set it to python2.7.5
ARG UBUNTU_VERSION=18.04
FROM ubuntu:$UBUNTU_VERSION
RUN apt-get update -y \
&& apt-get install -y python2.7.5 \
&& rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["python"]
it is throwing the following error
E: Couldn't find any package by regex 'python2.7.5'
I want to install Python 2.7.5 along with relevant PIP, what should I do?

This version is no longer available in canonical mirrors.
It has been released in 2013.
As a result, having both python and pip working together since then is challenging.
Python 2.7.5 + PIP on centos7
It may be the simplest way if ubuntu is not a requirement.
ARG CENTOS_VERSION=7
FROM centos:$CENTOS_VERSION
# Python 2.7.5 is installed with centos7 image
# Add repository for PIP
RUN yum install -y epel-release
# Install pip
RUN yum install -y python-pip
RUN python --version
ENTRYPOINT [ "python" ]
Python 2.7.5 on ubuntu
I've been able to install it from source
It has not been a success to install pip :
https://bootstrap.pypa.io/pip/2.7/get-pip.py
ARG UBUNTU_VERSION=18.04
FROM ubuntu:$UBUNTU_VERSION
ARG PYTHON_VERSION=2.7.5
# Install dependencies
# PIP - openssl version > 1.1 may be an issue (try older ubuntu images)
RUN apt-get update \
&& apt-get install -y wget gcc make openssl libffi-dev libgdbm-dev libsqlite3-dev libssl-dev zlib1g-dev \
&& apt-get clean
WORKDIR /tmp/
# Build Python from source
RUN wget https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz \
&& tar --extract -f Python-$PYTHON_VERSION.tgz \
&& cd ./Python-$PYTHON_VERSION/ \
&& ./configure --enable-optimizations --prefix=/usr/local \
&& make && make install \
&& cd ../ \
&& rm -r ./Python-$PYTHON_VERSION*
RUN python --version
ENTRYPOINT [ "python" ]
Python 2.7.6 + pip on ubuntu
Ubuntu 14.04 still has mirrors working (how long ???).
Python packages are really close to your expectations.
You may try to run your scripts with that one.
ARG UBUNTU_VERSION=14.04
FROM ubuntu:$UBUNTU_VERSION
RUN apt-get update \
&& apt-get install -y python python-pip \
&& apt-get clean
RUN python --version
ENTRYPOINT [ "python" ]
Python 2.7.5 + pip, not working but could on ubuntu
Here is what I've tried with no success.
ARG UBUNTU_VERSION=16.04
FROM ubuntu:$UBUNTU_VERSION
# Install dependencies
RUN apt-get update \
&& apt-get install -y wget gcc make openssl libffi-dev libgdbm-dev libsqlite3-dev libssl-dev zlib1g-dev \
&& apt-get clean
WORKDIR /tmp/
# Build python from source
RUN wget https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz \
&& tar --extract -f Python-$PYTHON_VERSION.tgz \
&& cd ./Python-$PYTHON_VERSION/ \
&& ./configure --enable-optimizations --prefix=/usr/local \
&& make && make install \
&& cd ../ \
&& rm -r ./Python-$PYTHON_VERSION*
# Build pip from source
RUN wget https://bootstrap.pypa.io/pip/2.7/get-pip.py \
&& python get-pip.py
RUN python --version
ENTRYPOINT [ "python" ]
Python 2.7.9 with pip - as requested in comment
You can use this dockerfile, building python includes pip.
ARG UBUNTU_VERSION=16.04
FROM ubuntu:$UBUNTU_VERSION
ARG PYTHON_VERSION=2.7.9
# Install dependencies
RUN apt-get update \
&& apt-get install -y wget gcc make openssl libffi-dev libgdbm-dev libsqlite3-dev libssl-dev zlib1g-dev \
&& apt-get clean
WORKDIR /tmp/
# Build Python from source
RUN wget https://www.python.org/ftp/python/$PYTHON_VERSION/Python-$PYTHON_VERSION.tgz \
&& tar --extract -f Python-$PYTHON_VERSION.tgz \
&& cd ./Python-$PYTHON_VERSION/ \
&& ./configure --with-ensurepip=install --enable-optimizations --prefix=/usr/local \
&& make && make install \
&& cd ../ \
&& rm -r ./Python-$PYTHON_VERSION*
RUN python --version \
&& pip --version
ENTRYPOINT [ "python" ]

The simplest possible solution:
sudo apt-get install libssl-dev openssl
wget https://www.python.org/ftp/python/2.7.5/Python-2.7.5.tgz
tar xzvf Python-2.7.5.tgz
cd Python-2.7.5
./configure
make
sudo make install
After installation completed set installed python as default one.

Python lib: undefined symbol: _Py_FalseStruct

I tried to install pybgpstream without sudo rights on a machine.
Python installation is system wide. I had to install everything under ~/.local/lib...
After installing libbgstream binary I wanted to install pybgpstream with
pip install --global-option build_ext --global-option '--include-dir=/home/USER/.local/include' --global-option '--library-dir=/home/USER/.local/lib' pybgpstream
After successfull installation I tried running some script with pybgpstream. Doesn't work:
ImportError: /home/USER/.local/lib/python2.7/site-packages/_pybgpstream.so: undefined symbol: _Py_FalseStruct
I added /home/USER/.local/lib/ to LD_PATH...
I don't know what's wrong...

There's a docker solution to work with pybgpstream. Here is the docker file
FROM python:3.8
 
RUN apt update \
&& apt install -y build-essential curl zlib1g-dev libbz2-dev libcurl4-openssl-dev librdkafka-dev python3-gi-cairo nano \
&& mkdir ~/src && cd ~/src/ && \
curl -LO https://research.wand.net.nz/software/wandio/wandio-4.2.3.tar.gz && \
tar zxf wandio-4.2.3.tar.gz && \
cd wandio-4.2.3/ && ./configure && make install && ldconfig && \
cd ~/src/ && \
curl -LO https://github.com/CAIDA/libbgpstream/releases/download/v2.1.0/libbgpstream-2.1.0.tar.gz && \
tar zxf libbgpstream-2.1.0.tar.gz && \
cd libbgpstream-2.1.0/ && ./configure && make check install && ldconfig && \
pip install pybgpstream && \
pip install ipython && \
pip install statsmodels && \
pip install matplotlib && \
pip install ipykernel && \
pip install pylint && \
pip install autopep8
 
CMD ["/bin/bash"]
You can also use this container every time and run bgp streams