We Keep Coding

Can't load BERTopic model

since Google Colab upgraded from Python 3.7 to 3.8, I can't seem to load my saved BERTopic model which trained on Google Colab prior to the upgrade.
Also, the "use fallback runtime version" feature is not available in the Command Palette anymore [It was available only up until Mid-December].
Is there, any possible way to load the old BERTopic model in Google Colab? I can't figure out the exact source of conflict.
I ran these codes to try in Python 3.7 version. I'm having trouble importing BERTopic.
!sudo apt-get update -y
!sudo apt-get install python3.7
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1
!sudo update-alternatives --config python3
!apt-get install python3-pip
!python -m pip install --upgrade pip --user
!pip install bertopic
ERROR: Could not build wheels for hdbscan, which is required to install pyproject.toml-based projects.

you can run install build essential before you install hdbscan
apt-get update && apt-get install -y build-essential

How do I install a specific python version without pyenv within nvidia docker so that it interacts well with poetry?

I am trying to build a docker image that contains cuda, cudnn and python, each with specific versions that are templatable as a base for downstream users.
(In this example I have replace all the irrelevant templating with hard-coded versions, this is just FYI as a motivation).
Please note that the following questions are not duplicates:
How to install python in a docker image? does not involve poetry
Integrating Python Poetry with Docker Does not concern itself with installing dependencies
How do I integrate pyenv, poetry, and docker? This works for me already, I am looking for a different solution
I have achieved what I want using pyenv to install the specific python version within docker inside the nvidia image.
However, this solution is not optimal since the resulting image is about 1.5GB larger than what I think should be possible. Sidenote: I know that there are other ways to reduce the image size further that I have not done in this example. This is not the question here.
I have prepared a dummy pyproject.toml and poetry.lock to demonstrate the issue that I am currently facing:
pyproject.toml
[tool.poetry]
name = "example_project"
version = "1.0.0"
description = ""
authors = ["RunOrVeith"]
[tool.poetry.dependencies]
python = ">=3.8,<3.11"
scipy = "^1.9.3"
[build-system]
requires = ["poetry-core>=1.1.0"]
build-backend = "poetry.core.masonry.api"
Working Dockerfile.pyenv
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ARG PYTHON_VERSION=3.8
ENV DEBIAN_FRONTEND=noninteractive
# Set-up necessary Env vars for PyEnv
ENV PYENV_ROOT /root/.pyenv
ENV PATH $PYENV_ROOT/shims:$PYENV_ROOT/bin:$PATH
ENV PATH="/root/.local/bin/:$PATH"
# Install essentials for pyenv https://github.com/pyenv/pyenv/wiki
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget ca-certificates \
curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev mecab-ipadic-utf8 git \
&& rm -rf /var/lib/apt/lists/*
# Install pyenv
RUN set -ex \
&& curl https://pyenv.run | bash \
&& pyenv update \
&& pyenv install $PYTHON_VERSION \
&& pyenv global $PYTHON_VERSION \
&& pyenv rehash \
&& pip install --upgrade pip
# Install poetry
RUN curl -sSL https://install.python-poetry.org | python - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example # The template that I want to provide ends here, this is just for demoing the issue
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
The version that doesn't work Dockerfile.plain
FROM nvidia/cuda:11.0.3-cudnn8-runtime-ubuntu20.04 as base
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHON_VERSION=3.8
ENV PATH="/root/.local/bin/:$PATH"
RUN apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys A4B469963BF863CC \
&& apt update \
&& apt install -y git curl \
&& apt install -y --no-install-recommends make build-essential
# Don't be confused, distutils-3.9 also installs python 3.8 https://github.com/deadsnakes/issues/issues/150
RUN apt install -y --no-install-recommends python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-distutils python${PYTHON_VERSION}-venv \
&& update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 10 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 10 \
&& apt-get install -y --no-install-recommends python3-pip python3-setuptools \
&& update-alternatives --install /usr/local/bin/pip pip /usr/bin/pip 10 \
&& update-alternatives --install /usr/local/bin/pip3 pip3 /usr/bin/pip 10 \
&& apt-get clean
WORKDIR /virtualenvs
RUN curl -sSL https://install.python-poetry.org | python${PYTHON_VERSION} - \
&& poetry --version && poetry config virtualenvs.create false
FROM base as example
WORKDIR /app
COPY pyproject.toml .
COPY poetry.lock .
RUN poetry install --no-interaction --no-ansi
You can build this using
DOCKER_BUILDKIT=1 docker build -t github:example-plain --target example -f Dockerfile.plain .
and then run using
docker run -it github:example-plain bash
Here is the issue:
All the following commands are run from within the docker image.
According to poetry, everything is installed:
root#5e1ffb1f971c:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
However using regular pip, there is nothing, and imports also fail.
If I use poetry to import something, it also does not work.
root#5e1ffb1f971c:/app# pip --version
pip 20.0.2 from /usr/lib/python3/dist-packages/pip (python 3.8)
root#5e1ffb1f971c:/app# pip freeze
root#5e1ffb1f971c:/app# python -c "import scipy"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
root#5e1ffb1f971c:/app# poetry run python -c "import scipy"
Skipping virtualenv creation, as specified in config file.
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'scipy'
What is also interesting is that if I upgrade pip with poetry it tells me it can't uninstall pip, I am assuming this is due to this ubuntu patch that tries to prevent me from breaking the system (even though I just install pip).
Afterwards, the poetry pip executable also points somewhere else.
root#5e1ffb1f971c:/app# poetry run pip install --upgrade pip
Skipping virtualenv creation, as specified in config file.
Collecting pip
Using cached pip-22.3.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 20.0.2
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Can't uninstall 'pip'. No files were found to uninstall.
Successfully installed pip-22.3.1
root#5e1ffb1f971c:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /usr/local/lib/python3.8/dist-packages/pip (python 3.8)
So how do I set this up so that I get a fresh python install of whichever version I configure, and it works with poetry? It is also required that the python and python3 aliases point to whatever poetry is using.
Reference with working version:
If I do the same commands with the working version using pyenv, it looks like this:
root#c0a9af7f05b4:/app# pip freeze
numpy==1.23.4
scipy==1.9.3
root#c0a9af7f05b4:/app# poetry show
Skipping virtualenv creation, as specified in config file.
numpy 1.23.4 NumPy is the fundamental package for array computing with Python.
scipy 1.9.3 Fundamental algorithms for scientific computing in Python
root#c0a9af7f05b4:/app# poetry run pip --version
Skipping virtualenv creation, as specified in config file.
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)
root#c0a9af7f05b4:/app# pip --version
pip 22.3.1 from /root/.pyenv/versions/3.8.15/lib/python3.8/site-packages/pip (python 3.8)

Unable to install tensorflow v1 Python package in Docker image

I am trying to install tensorflow<2.0,>=1.15 pip package during the Docker build. I am not able to build it, and I am getting this error in my terminal during the pip installation:
> [12/12] RUN pip3 install --no-cache-dir -r requirements.txt:
#16 0.488 ERROR: Could not find a version that satisfies the requirement tensorflow<2.0,>=1.15 (from versions: none)
#16 0.489 ERROR: No matching distribution found for tensorflow<2.0,>=1.15
To replicate the error:
Dockerfile:
FROM python:3.7-slim-buster
RUN apt-get update
RUN apt-get install -y unzip
RUN apt-get install -y build-essential
RUN apt-get install -y python-all-dev
RUN apt-get install -y libexiv2-dev
RUN apt-get install -y libboost-python-dev
RUN apt-get install -y wget
COPY . /usr/src/app
WORKDIR /usr/src/app
ENV PYTHONUNBUFFERED True
RUN pip3 install --upgrade pip
RUN pip3 install --no-cache-dir -r requirements.txt
requirements.txt:
tensorflow>=1.15,<2.0
I have tried to build FROM (first line in the Dockerfile) other Python versions, either 3.7 or lower, never newer. Still the same result.
I use Docker Desktop for Mac M1 version 4.3.2, Engine version 20.10.11.
When I run it on Fedora Linux, I can build it successfully.
I suspect that this can be Docker-related. There might be a difference between Docker Desktop and Docker for Linux. But I might be doing something wrong.
Have some of you folks ever encountered the same error? How did you solve this? Thanks for any tips.

Tensorflow 1.x does not support the Mac M1 chip. It is recommended to install Tensorflow >=2.5 on Mac M1
Take a look at these release notes from Mac Tensorflow:
https://github.com/apple/tensorflow_macos/releases

Docker build fails if pip is installed and updated to 10.0.1 in a single RUN section

How would you explain docker build failure with Dockerfile1, and it's success with Dockerfile2 (see below).
1)
// Dockerfile1
FROM ubuntu:16.04
RUN apt-get -y update && \
apt-get -y install python-pip python-dev build-essential && \
pip install --upgrade pip && \
pip install --upgrade virtualenv
docker build . fails with the following err
Collecting pip
Downloading
https://files.pythonhosted.org/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl (1.3MB)
Installing collected packages: pip
Found existing installation: pip 8.1.1
Not uninstalling pip at /usr/lib/python2.7/dist-packages, outside
environment /usr
Successfully installed pip-10.0.1
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
from pip import main
ImportError: cannot import name main
The command '/bin/sh -c apt-get -y update && apt-get -y install
python-pip python-dev build-essential && pip install --upgrade pip && pip install --upgrade virtualenv && virtualenv /venv' returned a non-zero code: 1
However, it succeeds if we split it into two RUN.
2)
// Dockerfile2
FROM ubuntu:16.04
RUN apt-get -y update && \
apt-get -y install python-pip python-dev build-essential && \
pip install --upgrade pip
RUN pip install --upgrade virtualenv
The installation failure for pip is related to this reported issue. So my questions:
Why does docker build fail in the first case? If we just run those command in bash, there wont be any error.
Why does docker build succeed in the second case? How is it related to layering concept in docker?
Why specifying pip version in Dockerfile1 (i.e. pip install --upgrade pip=0.9.3) solves the problem too?
Update (May 6, 2018):
I've figured out the issue. What happens here is as below:
apt-get -y install python-pip installs an old version of pip whose shim script import pip's main directly.
pip install --upgrade pip installs pip 10.0.1 and moves main into an internal directory _internal. It adds its shim script to PATH.
Calling pip fails as it still calls the old shim script as it's path is cached. Running hash -d pip in between fixes the issue.
So apparently, splitting install and update into two RUN sections has similar effect as hash -d pip. Workarounds (also suggested by Andriy Maletsky) are 1) pin pip update to 9.0.3, or 2) install (latest) pip from source in the first place, or 3) use hash -r in between, or 4) use another RUN command for later use of pip.

The problem is that pip executable (/usr/bin/pip) breaks while updating pip from version 9 to version 10.
Possible solutions:
1. Do not update and use pip v9
2. Do not use apt-get to install pip. Download it manually.
Why does docker build fail in the first case? If we just run those command in bash, there wont be any error.
No, there will be an error. I ran those commands inside docker run --rm -it ubuntu:16.04 bash and got it.
Why does docker build succeed in the second case? How is it related to layering concept in docker?
I believe you made a mistake somewhere in second RUN and it's silencing an error (in that place which you didn't provide). For example, this will work (because ; used instead of && and execution doesn't break after bad command):
RUN pip install --upgrade virtualenv && \
virtualenv /venv; source /venv/bin/activate
Why specifying pip version in Dockerfile1 (i.e. pip install --upgrade pip=0.9.3) solves the problem too?
Because this pip bug appeared in version 10.
P.S. You should not update or manually change files you added to your system via apt-get (you are doing this via pip install --upgrade pip).

OpenCV 3.1.0 with Python 3.5

After following several different tutorials, guides and steps recommended in other SO answers I didn't manage to install OpenCV for use with Python 3.5 in my Ubuntu 16.04 system.
As long as OpenCV 3.1.0 officially supports Python 3.x, how do I install it appropriately?

I managed installing Python 3.5 and OpenCV library appropriately in my system after gathering steps and troubleshooting solutions over different tutorials and guides.
The installation is performed under a virtualenv, so there is no need to clean up previous install attempts footprints from your system.
Following the steps presented here you'll install:
openCV 3.1.0
opencv_contrib 3.1.0
numpy
scipy
scikit
matplotlib
cython
venv
At the end, it may take up to 20Gb of space if you haven't already installed any of those packages previously.
You will need gcc-4.9+ for compiling OpenCV, I tested it with gcc-5.4
Install OpenCV dependencies
sudo apt-get build-dep -y opencv
Create and setup a virtualenv
sudo apt-get install python3-venv
python3.5 -m venv python35-opencv31
source ~/python35-opencv31/bin/activate
pip install matplotlib
pip install numpy
pip install scipy
pip install scikit-learn
pip install cython
pip install -U scikit-image
Compile OpenCV 3.1.0 and openvc_contrib 3.1.0
Dependencies
sudo apt-get install build-essential cmake libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev
Getting repositories
mkdir ~/git
cd ~/git
git clone https://github.com/opencv/opencv.git
cd ./opencv
git checkout 3.1.0
cd ~/git
git clone https://github.com/Itseez/opencv_contrib.git
cd ./opencv_contrib
git checkout 3.1.0
Making sure some libs will be found
ffmpeg libs
sudo -i
mkdir /usr/include/ffmpeg
cd /usr/include/ffmpeg
ln -sf /usr/include/x86_64-linux-gnu/libavcodec/*.h ./
ln -sf /usr/include/x86_64-linux-gnu/libavformat/*.h ./
ln -sf /usr/include/x86_64-linux-gnu/libswscale/*.h ./
If, during compilation, occurs any problems when trying to find some ffmpeg libs, uninstall ffmpeg and build it from source.
python bindings with opencv_contrib modules
echo "\nfind_package(HDF5)\ninclude_directories(\${HDF5_INCLUDE_DIRS})" >> ~/git/opencv/modules/python/common.cmake
Compiling
source ~/python35-opencv31/bin/activate
mkdir ~/opencv3.1.0
cd ~/git/opencv/
mkdir release
cd ./release
export CC=$(which gcc)
export CXX=$(which g++)
cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=~/opencv3.1.0 \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_EXTRA_MODULES_PATH=~/git/opencv_contrib/modules \
-D BUILD_EXAMPLES=ON \
-D CUDA_NVCC_FLAGS="-D_FORCE_INLINES" ..
The output should include this:
-- Python 2:
-- Interpreter: /home/rodrigo/anaconda/bin/python2.7 (ver 2.7.12)
-- Libraries: /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.12)
-- numpy: /home/rodrigo/anaconda/lib/python2.7/site-packages/numpy/core/include (ver 1.10.4)
-- packages path: lib/python2.7/site-packages
--
-- Python 3:
-- Interpreter: /home/rodrigo/python35-opencv/bin/python3 (ver 3.5.2)
-- Libraries: /usr/lib/x86_64-linux-gnu/libpython3.5m.so (ver 3.5.2)
-- numpy: /home/rodrigo/python35-opencv/lib/python3.5/site-packages/numpy/core/include (ver 1.11.2)
-- packages path: lib/python3.5/site-packages
--
-- Python (for build): /home/rodrigo/anaconda/bin/python2.7
Now:
make
If it succeed, then:
make install
Add OpenCV libs to your virtualenv
cd ~/python35-opencv31/lib/site-packages
ln -s ~/opencv3.1.0/lib/python3.5/site-packages/cv2.cpython-35m-x86_64-linux-gnu.so
Done!
To test if it works as expected:
cd ~
source ~/python35-opencv31/bin/activate
python
import cv2
cv2.__version__
It should import cv2 and show version number 3.1.0.