We Keep Coding

AWS lambda stuck importing library

I am trying to deploy my ECR image to aws lambda. The image works fine locally, but on aws, it gets stuck importing this library https://github.com/jianfch/stable-ts.
import json
import boto3
import requests
import numpy
print("All imports ok 1 ...")
from stable_whisper import load_model
print("All imports ok 2 ...")
The first statement is printed but it gets stuck on importing and the second statement never got printed until it timed out.
Docker File:
# Build FFmpeg
FROM public.ecr.aws/lambda/python:3.8 as lambda-base
COPY requirements.txt ./
COPY myfunction.py ./
RUN pip3 install -r requirements.txt
WORKDIR /ffmpeg_sources
RUN yum install autoconf automake bzip2 bzip2-devel cmake libxcb libxcb-devel \
freetype-devel gcc gcc-c++ git libtool make pkgconfig zlib-devel -y -q
# Compile NASM assembler
RUN curl -OL https://www.nasm.us/pub/nasm/releasebuilds/2.15.05/nasm-2.15.05.tar.bz2
RUN tar xjvf nasm-2.15.05.tar.bz2
RUN cd nasm-2.15.05 && sh autogen.sh && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile Yasm assembler
RUN curl -OL https://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz
RUN tar xzvf yasm-1.3.0.tar.gz
RUN cd yasm-1.3.0 && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile FFmpeg
RUN curl -OL https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2
RUN tar xjvf ffmpeg-snapshot.tar.bz2
RUN cd ffmpeg && \
export PATH="/ffmpeg_sources/bin:$PATH" && \
export PKG_CONFIG_PATH="/ffmpeg_sources/ffmpeg_build/lib/pkgconfig" && \
./configure \
--prefix="/ffmpeg_sources/ffmpeg_build" \
--pkg-config-flags="--static" \
--extra-cflags="-I/ffmpeg_sources/ffmpeg_build/include" \
--extra-ldflags="-L/ffmpeg_sources/ffmpeg_build/lib" \
--extra-libs=-lpthread \
--extra-libs=-lm \
--enable-libxcb \
--bindir="/ffmpeg_sources/bin" && \
make && \
make install
# Final image with code and dependencies
FROM lambda-base
COPY myfunction.py /var/task/
CMD ["myfunction.lambda_handler"]
inside the requirements.txt, I tried both stable-ts and git+https://github.com/jianfch/stable-ts.git
I appreciate any help.

stable_whisper has a lot of dependencies and some of them contain compiled code (ffmpeg).
Python packages that contain compiled code aren't always compatible with Lambda runtimes by default. I don’t know how to build ffmpeg, but I can point you to a useful AWS sample, which utilizes python packages based on this dependency. Maybe it will contribute to solving your problem or maybe others will be able to help you further.
Sample Dockerfile:
# Install dependencies
…
# Build FFmpeg
FROM public.ecr.aws/lambda/python:3.8 as ffmpeg
WORKDIR /ffmpeg_sources
RUN yum install autoconf automake bzip2 bzip2-devel cmake libxcb libxcb-devel \
freetype-devel gcc gcc-c++ git libtool make pkgconfig zlib-devel -y -q
# Compile NASM assembler
RUN curl -OL https://www.nasm.us/pub/nasm/releasebuilds/2.15.05/nasm-2.15.05.tar.bz2
RUN tar xjvf nasm-2.15.05.tar.bz2
RUN cd nasm-2.15.05 && sh autogen.sh && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile Yasm assembler
RUN curl -OL https://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz
RUN tar xzvf yasm-1.3.0.tar.gz
RUN cd yasm-1.3.0 && \
./configure --prefix="/ffmpeg_sources/ffmpeg_build" \
--bindir="/ffmpeg_sources/bin" && \
make && make install
# Compile FFmpeg
RUN curl -OL https://ffmpeg.org/releases/ffmpeg-snapshot.tar.bz2
RUN tar xjvf ffmpeg-snapshot.tar.bz2
RUN cd ffmpeg && \
export PATH="/ffmpeg_sources/bin:$PATH" && \
export PKG_CONFIG_PATH="/ffmpeg_sources/ffmpeg_build/lib/pkgconfig" && \
./configure \
--prefix="/ffmpeg_sources/ffmpeg_build" \
--pkg-config-flags="--static" \
--extra-cflags="-I/ffmpeg_sources/ffmpeg_build/include" \
--extra-ldflags="-L/ffmpeg_sources/ffmpeg_build/lib" \
--extra-libs=-lpthread \
--extra-libs=-lm \
--enable-libxcb \
--bindir="/ffmpeg_sources/bin" && \
make && \
make install
# Final image with code and dependencies
FROM lambda-base
# Copy FFMpeg binary
COPY --from=ffmpeg /ffmpeg_sources/bin/ffmpeg /usr/bin/

Error while building confluent-kafka in docker for alphine image

I am trying to build a python application which require confluent-kafka package. But while building in bamboo, I got the below error
fatal error: librdkafka/rdkafka.h: No such file or directory
build 13-Dec-2022 11:46:59 23 | #include <librdkafka/rdkafka.h>
build 13-Dec-2022 11:46:59 | ^~~~~~~~~~~~~~~~~~~~~~
My dockerfile is as such:
FROM python:3.9-alpine
RUN adduser -D pythonwebapi
WORKDIR /home/pythonwebapi
COPY requirements.txt requirements.txt
COPY logger_config.py logger_config.py
RUN echo 'http://dl-3.alpinelinux.org/alpine/v3.9/main' >> /etc/apk/repositories
RUN apk update \
&& apk upgrade \
&& apk add --no-cache build-base \
autoconf \
bash \
bison \
boost-dev \
cmake \
flex \
# libressl-dev \
zlib-dev
RUN apk add make gcc g++
RUN apk add libffi-dev
RUN apk update && apk --no-cache add librdkafka-dev
RUN apk add postgresql-dev gcc python3-dev musl-dev
RUN pip install --upgrade pip && pip install -r requirements.txt && pip install gunicorn
RUN apk del gcc g++ make
RUN pip install --no-cache-dir six pytest numpy cython
RUN pip install --no-cache-dir pandas
RUN pip install --no-cache-dir confluent-kafka
ARG ARROW_VERSION=3.0.0
ARG ARROW_SHA1=c1fed962cddfab1966a0e03461376ebb28cf17d3
ARG ARROW_BUILD_TYPE=release
ENV ARROW_HOME=/usr/local \
PARQUET_HOME=/usr/local
#Download and build apache-arrow
RUN mkdir -p /arrow \
&& wget -q https://github.com/apache/arrow/archive/apache-arrow-${ARROW_VERSION}.tar.gz -O /tmp/apache-arrow.tar.gz \
&& echo "${ARROW_SHA1} *apache-arrow.tar.gz" | sha1sum /tmp/apache-arrow.tar.gz \
&& tar -xvf /tmp/apache-arrow.tar.gz -C /arrow --strip-components 1 \
&& mkdir -p /arrow/cpp/build \
&& cd /arrow/cpp/build \
&& cmake -DCMAKE_BUILD_TYPE=$ARROW_BUILD_TYPE \
-DOPENSSL_ROOT_DIR=/usr/local/ssl \
-DCMAKE_INSTALL_LIBDIR=lib \
-DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
-DARROW_WITH_BZ2=ON \
-DARROW_WITH_ZLIB=ON \
-DARROW_WITH_ZSTD=ON \
-DARROW_WITH_LZ4=ON \
-DARROW_WITH_SNAPPY=ON \
-DARROW_PARQUET=ON \
-DARROW_PYTHON=ON \
-DARROW_PLASMA=ON \
-DARROW_BUILD_TESTS=OFF \
.. \
&& make -j$(nproc) \
&& make install \
&& cd /arrow/python \
&& python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet \
&& python setup.py install \
&& rm -rf /arrow /tmp/apache-arrow.tar.gz
COPY app app
COPY init_app.py ./
ENV FLASK_APP init_app.py
RUN chown -R pythonwebapi:pythonwebapi ./
RUN chown -R 777 ./
USER pythonwebapi
EXPOSE 8000 7000
ENTRYPOINT ["gunicorn","--timeout", "7000","init_app:app","-k","uvicorn.workers.UvicornWorker","-b","0.0.0.0"]
I am unable to gauge why the error is coming since librdkafka is already installed.My requirement is to use alpine image. Can anyone please help me regarding this?

Build and install aerospike 4.0.0 in python 3.8 alpine 3.16

I am trying to install aerospike for python 3.8.14 on alpine 1.16.2.
Following offical documanetion I've already built aerospike-client-c 4.6.18 (release), successfully installed aerospike using pip.
Problem is that I got following error:
uv_async_send: symbol not found
ImportError: Error relocating /usr/local/lib/python3.8/site-packages/aerospike.cpython-38-x86_64-linux-gnu.so: uv_async_send: symbol not found
I've tried use ./install_libuv in aearospike-c-client but it didn't help.
I am not familair with all these building stuff, so maybe I am missing something.
Any ideas I can work out?
Command that failed:
python3 -c 'import aerospike'
My current Dockerfile:
ARG PYTHON_VERSION=3.8
FROM python:${PYTHON_VERSION}-alpine as aerospike-c-builder
ENV AS_C_VERSION=4.6.18
WORKDIR /src
RUN apk update
RUN apk add --no-cache \
build-base \
bash linux-headers \
libuv libuv-dev \
openssl openssl-dev \
lua5.1 lua5.1-dev zlib-dev
RUN wget https://artifacts.aerospike.com/aerospike-client-c/${AS_C_VERSION}/aerospike-client-c-src-${AS_C_VERSION}.zip \
&& unzip aerospike-client-c-src-${AS_C_VERSION}.zip \
&& mv aerospike-client-c-src-${AS_C_VERSION} aerospike-client-c \
&& cd aerospike-client-c \
&& make EVENT_LIB=libuv
# Stage 2: Build Aerospike Python client
FROM python:${PYTHON_VERSION}-alpine as aerospike-python-builder
ENV AS_PYTHON_VERSION=4.0.0
WORKDIR /src
COPY --from=aerospike-c-builder /src/aerospike-client-c/target/Linux-x86_64/ aerospike-client-c/
ENV PREFIX=/src/aerospike-client-c/
RUN apk update
RUN apk add --no-cache \
build-base \
bash \
linux-headers \
zlib-dev \
openssl-dev \
git
ENV DOWNLOAD_C_CLIENT 0
RUN pip --no-cache-dir install aerospike==${AS_PYTHON_VERSION}

Django/Python Docker Libreoffice Subprocess

I am trying to use libreoffice in my Django app, to convert a docx file to pdf using python subprocess.
I have included libreoffice in my dockerfile:
Dockerfile:
FROM python:3.8-alpine
ENV PYTHONUNBUFFERED 1
COPY ./requirements.txt /requirements.txt
COPY ./behavioursolutiondjango /behavioursolutiondjango
COPY ./scripts /scripts
WORKDIR /behavioursolutiondjango
EXPOSE 8000
RUN python -m venv /py && \
/py/bin/pip install --upgrade pip && \
apk add --update python3-dev \
xmlsec xmlsec-dev \
gcc \
libc-dev \
libreoffice \
libffi-dev && \
apk add --update --no-cache postgresql-client && \
apk add --update --no-cache --virtual .tmp-deps \
build-base postgresql-dev musl-dev linux-headers && \
/py/bin/pip install -r /requirements.txt && \
apk del .tmp-deps && \
adduser --disabled-password --no-create-home app && \
mkdir -p /vol/web/static && \
mkdir -p /vol/web/media && \
chown -R app:app /vol && \
chmod -R 755 /vol && \
chmod -R +x /scripts
ENV PATH="/scripts:/py/bin:$PATH"
USER app
CMD ["run.sh"]
And have the following running to do the conversion:
subprocess.call(["soffice", "--headless", "--convert-to", "pdf", new_cert.cert.path])
But I am running into the following error:
LibreOffice 7.2 - Fatal Error: The application cannot be started.
User installation could not be completed.
I have spent hours on this and cannot figure out what im missing.
I would be more than happy to use something other than Libreoffice, but cannot find something that will work, other than libreoffice.

Python docker build up error

I got this error when i run command
sudo docker-compose up
Docker File:
FROM alpine
ARG AWS_RDS_USER
ARG AWS_RDS_PASSWORD
ARG AWS_RDS_HOST
ARG AWS_RDS_DATABASE
ARG LOCALE_SERVICE_URL
ARG CRYPTO_KEY
ENV APP_DIR=/app
ENV APP_ENV=production
ENV DATABASE_CONNECTION_STRING=mysql://${AWS_RDS_USER}:${AWS_RDS_PASSWORD}#${AWS_RDS_HOST}/${AWS_RDS_DATABASE}
ENV LOCALE_SERVICE_URL=$LOCALE_SERVICE_URL
ENV CRYPTO_KEY=$CRYPTO_KEY
COPY build/requirements.txt build/app.ini ${APP_DIR}/
COPY build/nginx.conf /etc/nginx/nginx.conf
COPY api ${APP_DIR}/api
RUN apk add --no-cache curl python pkgconfig python-dev openssl-dev libffi-dev musl-dev make gcc
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python
RUN apk update && \
apk add --virtual .build-deps autoconf gcc make g++ python-dev && \
apk add nginx uwsgi uwsgi-python py2-pip py-mysqldb && \
chown -R nginx:nginx ${APP_DIR} && \
chmod 777 /run/ -R && \
chmod 777 /root/ -R && \
pip2 install --upgrade pip && \
pip2 install -r ${APP_DIR}/requirements.txt && \
apk del .build-deps && \
rm -fR tmp/* && \
pw_migrate migrate --database=$DATABASE_CONNECTION_STRING --directory=$APP_DIR/api/migrations -v
EXPOSE 80
CMD nginx && uwsgi --ini ${APP_DIR}/app.ini
For solution, I tried to install below packages
1) gcc package.
2) libffi packages.
3) pip openssl packages.
But still error is not resolved. Any help should be appreciated

Try the solution suggested here
This is because you need a working compiler, the easiest way around
this is too install the build-base package like so:
apk add --no-cache --virtual .pynacl_deps build-base python3-dev
libffi-dev This will install various tools that are required to
compile pynacl and pip install pynacl will now succeed.
Note it is optional to use the --virtual flag but it makes it easy to
trim the image because you can run apk del .pynacl_deps later in your
dockerfile as they are not needed any more and would reduce the
overall size of the image.