my jenkins CI has started to fail.
I'm building a module that depends on another module of mine
here is the error message. it's not failing on localhost. The build is done using dockers
Collecting cryptography>=2.2.1 (from pyOpenSSL==18.0.0->api_common_module==4.0.3->mobile_module)
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /usr/bin/python2 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-3qDpun --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /tmp/python-packages -- setuptools>=18.5 wheel "cffi>=1.7,!=1.11.3; python_implementation != 'PyPy'":
Looking in links: /tmp/python-packages
Collecting setuptools>=18.5
Could not find a version that satisfies the requirement setuptools>=18.5 (from versions: )
No matching distribution found for setuptools>=18.5
Here is the docker file
FROM alpine:3.6
LABEL project="mobile_module"
LABEL description="Mobile module"
# Install needed packages. Notes:
# * dumb-init: a proper init system for containers, to reap zombie children
# * musl: standard C library
# * linux-headers: commonly needed, and an unusual package name from Alpine.
# * build-base: used so we include the basic development packages (gcc)
# * bash: so we can access /bin/bash
# * git: to ease up clones of repos
# * ca-certificates: for SSL verification during Pip and easy_install
# * python: the binaries themselves
# * python-dev: are used for gevent e.g.
# * py-setuptools: required only in major version 2, installs easy_install so we can install Pip.
ENV PACKAGES="\
dumb-init \
musl \
linux-headers \
build-base \
bash \
git \
ca-certificates \
python2 \
python2-dev \
py-setuptools \
openssh \
sshpass \
openssl \
openssl-dev \
ansible \
libffi-dev \
py2-pip \
supervisor \
nginx \
uwsgi-python \
"
# Add the packages
RUN apk add --update $PACKAGES \
&& rm -rf /var/cache/apk/* \
&& echo
# make some useful symlinks that are expected to exist
RUN if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python2.7 /usr/bin/python; fi \
&& if [[ ! -e /usr/bin/python-config ]]; then ln -sf /usr/bin/python2.7-config /usr/bin/python-config; fi \
&& if [[ ! -e /usr/bin/easy_install ]]; then ln -sf /usr/bin/easy_install-2.7 /usr/bin/easy_install; fi \
&& echo
# Install and upgrade Pip
RUN easy_install pip \
&& pip install --upgrade pip \
&& if [[ ! -e /usr/bin/pip ]]; then ln -sf /usr/bin/pip2.7 /usr/bin/pip; fi \
&& echo
RUN pip install cffi
RUN apk add --no-cache gcc musl-dev
# **company_name** user and group
#RUN getent group www-data
#RUN getent group company_name
RUN adduser -D company_name && adduser company_name company_name
RUN adduser -D www-data -G company_name
# RUN groups www-data
# RUN groups company_name
ENV APP_DIR /var/www/webservice
# Setup uwsgi
RUN mkdir ${APP_DIR} \
&& mkdir -p ${APP_DIR}/.python-eggs \
&& chown -R www-data:www-data ${APP_DIR} \
&& chmod 777 /run/ -R \
&& chmod 777 /root/ -R
WORKDIR ${APP_DIR}
COPY uwsgi/webservice.ini /etc/uwsgi/apps-available/webservice.ini
RUN mkdir /etc/uwsgi/apps-enabled/
RUN ln -f -s /etc/uwsgi/apps-available/webservice.ini /etc/uwsgi/apps-enabled/webservice.ini
RUN pip install uwsgitop
# Setup nginx
RUN chown -R www-data:www-data /var/lib/nginx
# RUN cat /etc/nginx/nginx.conf
COPY nginx/nginx.conf /etc/nginx/nginx.conf
COPY nginx/webservice /etc/nginx/conf.d/webservice.conf
RUN rm -f /etc/nginx/conf.d/default.conf
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
RUN ln -sf /dev/stderr /var/log/nginx/error.log
RUN mkdir -p /run/nginx
# Install application packages
COPY python-packages /tmp/python-packages
RUN pip install -f /tmp/python-packages --no-index mobile_module
RUN mkdir -p /data/mobile_module/newrelic
COPY newrelic/newrelic.ini /data/mobile_module/newrelic/newrelic.ini
any ideas why is that?
The root cause was upgrading PyOpenSSL from 17.3.0 to 18.0. reverted that and was solved.
Related
In my dockerfile, I was previously using FROM python:3.9-alpineon top of which librdkafka 1.9.2 is built and this was successful. But today, with the same docker file, the build failed by throwing the below error:
#error "confluent-kafka-python requires librdkafka v2.0.2 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html".
When I searched in the internet, alpine:edge seems to have the newest version of librdkafka package. So I changed the dockerfile to FROM python:3.9-alpine:edge. But on building, this threw my an error:
Step 1/41 : FROM python:3.9-alpine:edge build 25-Jan-2023 10:25:20 invalid reference format build 25-Jan-2023 10:25:20 [?1h=[41m[37;1mAn error occurred when executing task '
I am new to docker and I used https://www.docker.com/blog/how-to-use-the-alpine-docker-official-image/ for the format. Please do help me regarding this.
This is my dockerfile currently:
FROM python:3.9-alpine:edge
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
# COPY kong.ini kong.ini
# COPY iot.ini iot.ini
# COPY project.ini project.ini
# COPY eom.ini eom.ini
# COPY notify.ini notify.ini
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]```
I am trying to build a python application which require confluent-kafka package. But while building in bamboo, I got the below error
fatal error: librdkafka/rdkafka.h: No such file or directory
build 13-Dec-2022 11:46:59 23 | #include <librdkafka/rdkafka.h>
build 13-Dec-2022 11:46:59 | ^~~~~~~~~~~~~~~~~~~~~~
My dockerfile is as such:
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
# libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
I am unable to gauge why the error is coming since librdkafka is already installed.My requirement is to use alpine image. Can anyone please help me regarding this?
I have a Dockerfile that starts with
FROM python:3.7-slim-buster
and I want to install node.js and npm in it. How can I install them in this image?
This should work:
FROM python:3.7-slim-buster
# setup dependencies
RUN apt-get update
RUN apt-get install xz-utils
RUN apt-get -y install curl
# Download latest nodejs binary
RUN curl https://nodejs.org/dist/v14.15.4/node-v14.15.4-linux-x64.tar.xz -O
# Extract & install
RUN tar -xf node-v14.15.4-linux-x64.tar.xz
RUN ln -s /node-v14.15.4-linux-x64/bin/node /usr/local/bin/node
RUN ln -s /node-v14.15.4-linux-x64/bin/npm /usr/local/bin/npm
RUN ln -s /node-v14.15.4-linux-x64/bin/npx /usr/local/bin/npx
To run node start it with docker run -it <containerName> /bin/bash
Then node, npm and npx are available
npm globally packages need to add links if you want to use it as command.
FROM python:3.7-slim-buster
ENV NODE_VERSION 14.15.4
#if build in `china`, debian mirrors, npm registry change to china source
ARG AREA=london
RUN set -ex \
&& if [ 'china' = "$AREA" ] ; then \
sed -i "s#http://deb.debian.org#https://mirrors.aliyun.com#g" /etc/apt/sources.list; \
fi \
&& apt-get update \
&& apt-get install -y git xz-utils curl \
# install node
&& curl "https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.xz" -O \
&& tar -xf "node-v$NODE_VERSION-linux-x64.tar.xz" \
&& ln -s "/node-v$NODE_VERSION-linux-x64/bin/node" /usr/local/bin/node \
&& ln -s "/node-v$NODE_VERSION-linux-x64/bin/npm" /usr/local/bin/npm \
&& ln -s "/node-v$NODE_VERSION-linux-x64/bin/npx" /usr/local/bin/npx \
# npm install bump, openapi-generator
&& if [ 'china' = "$AREA" ] ; then \
npm config set registry https://registry.npm.taobao.org/; \
fi \
&& npm install -g bump-cli#2.1.0 \
&& ln -s "/node-v$NODE_VERSION-linux-x64/bin/bump" /usr/local/bin/bump \
# clear
&& npm cache clean --force \
&& rm -rf /var/lib/apt/lists/* \
&& rm -f "/node-v$NODE_VERSION-linux-x64.tar.xz" \
&& apt-get clean \
&& apt-get autoremove
Just use the official installation for Debian from here :
RUN curl -fsSL https://deb.nodesource.com/setup_19.x | bash - &&\
apt-get install -y nodejs
I got this error when i run command
sudo docker-compose up
Docker File:
FROM alpine
ARG AWS_RDS_USER
ARG AWS_RDS_PASSWORD
ARG AWS_RDS_HOST
ARG AWS_RDS_DATABASE
ARG LOCALE_SERVICE_URL
ARG CRYPTO_KEY
ENV APP_DIR=/app
ENV APP_ENV=production
ENV DATABASE_CONNECTION_STRING=mysql://${AWS_RDS_USER}:${AWS_RDS_PASSWORD}#${AWS_RDS_HOST}/${AWS_RDS_DATABASE}
ENV LOCALE_SERVICE_URL=$LOCALE_SERVICE_URL
ENV CRYPTO_KEY=$CRYPTO_KEY
COPY build/requirements.txt build/app.ini ${APP_DIR}/
COPY build/nginx.conf /etc/nginx/nginx.conf
COPY api ${APP_DIR}/api
RUN apk add --no-cache curl python pkgconfig python-dev openssl-dev libffi-dev musl-dev make gcc
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python
RUN apk update && \
apk add --virtual .build-deps autoconf gcc make g++ python-dev && \
apk add nginx uwsgi uwsgi-python py2-pip py-mysqldb && \
chown -R nginx:nginx ${APP_DIR} && \
chmod 777 /run/ -R && \
chmod 777 /root/ -R && \
pip2 install --upgrade pip && \
pip2 install -r ${APP_DIR}/requirements.txt && \
apk del .build-deps && \
rm -fR tmp/* && \
pw_migrate migrate --database=$DATABASE_CONNECTION_STRING --directory=$APP_DIR/api/migrations -v
EXPOSE 80
CMD nginx && uwsgi --ini ${APP_DIR}/app.ini
For solution, I tried to install below packages
1) gcc package.
2) libffi packages.
3) pip openssl packages.
But still error is not resolved. Any help should be appreciated
Try the solution suggested here
This is because you need a working compiler, the easiest way around
this is too install the build-base package like so:
apk add --no-cache --virtual .pynacl_deps build-base python3-dev
libffi-dev This will install various tools that are required to
compile pynacl and pip install pynacl will now succeed.
Note it is optional to use the --virtual flag but it makes it easy to
trim the image because you can run apk del .pynacl_deps later in your
dockerfile as they are not needed any more and would reduce the
overall size of the image.
FROM webdevops/base:ubuntu-16.04
RUN apt-get update && apt-get -y upgrade && apt-get install -y --no-install- recommends \
apache2 \
openssh-client \
python3 \
python3-dev \
python3-venv \
python3-psycopg2 \
python3-pip \
pyflakes3 \
pylint3 \
pep8 \
pep257 \
postgresql-client \
libapache2-mod-wsgi-py3 \
&& apt-get clean \
&& rm -fr /var/lib/apt/lists/*
RUN mkdir /var/www/html/hotels-project
RUN cd /var/www/html/hotels-project/ \
&& python3 -m venv hotels-venv \
&& /bin/bash -c "source hotels-venv/bin/activate"
RUN pip install 'django<2.0'
RUN pip install requests
RUN pip install psycopg2
show message:
ERROR: Service 'apache-python' failed to build: The command '/bin/sh
-c pip install 'django<2.0'' returned a non-zero code: 127
You have two issues in your docker file.
Using pip instead of pip3
Activating virtualenv in one step and running commands in another step
In Dockerfile for every RUN step, you get a fresh terminal. So any source command you executed in previous RUN statement is no more active.
So your code should be something like this
RUN cd /var/www/html/hotels-project/ \
&& python3 -m venv hotels-venv \
&& /bin/bash -c "source hotels-venv/bin/activate" \
&& pip3 install -r requirements.txt
And requirements.txt should have below content
django<2.0
requests==X.XX
psycopg2==y.yy
That's how you should do it