We Keep Coding

poetry install fail build with private dependency

I have a module that depends on another module (module_a) stored in my private repository (nexus) that reuqest another module (module_b) at build time, even stored in the private repository.
I add repository source on poetry.toml in order to add my private repo.
[[tool.poetry.source]]
name = "nexus"
url = "https://my_nexus_url/private_repo/simple"
secondary = true
Then I specify the dependency on th toml
[tool.poetry.dependencies]
python = ">=3.9.0,<3.11,"
module_a="1.0.0"
When I run poetry install it download module_a and build it. During the build process I get this error:
ERROR: Could not find a version that satisfies the requirement module_b==1.0.1
ERROR: No matching distribution found for module_b==1.0.1
When I try to install the module using pip with --extra-index-url <my repo> everything works fine.
pip install module_a --extra-index-url https://my_nexus_url/private_repo/simple
I guess that the problem is releted on the pip command, executed by poetry. It dose not specify the extra-index-url that point to my repo so it try to download dependency (module_b) from pypi repository instead from my repo.
There is a way to instruct poetry in order to use my private repo when source build is required?
I have already try with this:
[tool.poetry.dependencies]
python = ">=3.9.0,<3.11,"
module_b={version="1.0.1", source="nexus"}
module_a={version="1.0.0", source="nexus"}
without any success.

How to build multiple packages from a single python module using pyproject.toml and poetry?

I want to achieve a similar behavior as the library Dask does, it is possible to use pip to install dask, dask[dataframe], dask[array] and others. They do it by using the setup.py with a packages key like this. If I install only dask the dask[dataframe] is not installed and they warn you about this when executing the module.
I found this in the poetry documentation but when I execute poetry build I only get one .whl file with all of the packages within.
How can I package my module to be able to install specific parts of a library using poetry?

Actually the Dask example does not install sub packages sepparatelly, it just installs the custom dependencies sepparatelly as explained in this link.
In order to accomplish the same behavior using poetry you need to use this (as mentioned by user #sinoroc in this comment)
The example pyproject.toml from the poetry extras page is this:
[tool.poetry]
name = "awesome"
[tool.poetry.dependencies]
# These packages are mandatory and form the core of this package’s distribution.
mandatory = "^1.0"
# A list of all of the optional dependencies, some of which are included in the
# below `extras`. They can be opted into by apps.
psycopg2 = { version = "^2.7", optional = true }
mysqlclient = { version = "^1.3", optional = true }
[tool.poetry.extras]
mysql = ["mysqlclient"]
pgsql = ["psycopg2"]
By using poetry build --format wheel a single wheel file would be created.
In order to install a specific set of extra dependencies using pip and the wheel file you should use:
pip install "wheel_filename.whl[mysql]"

Python poetry install failure - invalid hashes

I'm unsure exactly what information / context to provide with this post in order for it to be
properly answered - so if I can add more information please let me know.
Whilst trying to install a package I've had an error message as follows:
Invalid hashes (sha256:01d6b62184e55367ce7d770bd87d97a6cfaf783ff7ba8328e7473a339fe19807) for numpy (1.22.2)
I don't know how to go about troubleshooting this problem.
poetry --version
(install-check-clU745_7-py3.8) ➜ install_check poetry --version
Poetry version 1.1.12
poetry env info
Output of poetry env info:
(install-check-clU745_7-py3.8) ➜ install_check poetry env info
Virtualenv
Python: 3.8.12
Implementation: CPython
Path: /Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-clU745_7-py3.8
Valid: True
System
Platform: darwin
OS: posix
Python: /Users/user_name/.pyenv/versions/3.8.12
pyproject.toml
pyproject.toml config:
(install-check-clU745_7-py3.8) ➜ install_check cat pyproject.toml
[tool.poetry]
name = "install_check"
version = "0.1.0"
description = ""
authors = ["user_name <user_name#email_add.com>"]
[tool.poetry.dependencies]
python = "^3.8"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Steps to reproduce
Here is an attempt to clearly outline a reproduction of this problem:
python --version
Python 3.8.12
which python
/Users/user_name/.pyenv/shims/python
poetry new install_check
Created package install_check in install_check
cd install_check
cat pyproject.toml| grep python
python = "^3.7"
# update pyproject.toml to 3.8 instead of 3.7.
cat pyproject.toml | grep python
python = "^3.8"
python --version
Python 3.8.12
poetry env use python3.8
which python3.8
/Users/user_name/.pyenv/shims/python3.8
poetry env use python3.8
poetry install
Updating dependencies
Resolving dependencies... (5.0s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (3.0.7)
• Installing attrs (21.4.0)
• Installing more-itertools (8.12.0)
• Installing packaging (21.3)
• Installing pluggy (0.13.1)
• Installing py (1.11.0)
• Installing wcwidth (0.2.5)
• Installing pytest (5.4.3)
Installing the current project: install_check (0.1.0)
Following this, after leaving and revisiting the directory to activate the env, I have:
(install-check-clU745_7-py3.8) ➜ install_check which python
/Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-clU745_7-py3.8/bin/python
(install-check-clU745_7-py3.8) ➜ install_check python --version
Python 3.8.12
Trying to add pandas then gives the following error:
(install-check-clU745_7-py3.8) ➜ install_check poetry add pandas
Using version ^1.4.0 for pandas
Updating dependencies
Resolving dependencies... (2.3s)
Writing lock file
Package operations: 5 installs, 0 updates, 0 removals
• Installing six (1.16.0)
• Installing numpy (1.22.2): Failed
RuntimeError
Invalid hashes (sha256:01d6b62184e55367ce7d770bd87d97a6cfaf783ff7ba8328e7473a339fe19807) for numpy (1.22.2) using archive numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl. Expected one of sha256:03ae5850619abb34a879d5f2d4bb4dcd025d6d8fb72f5e461dae84edccfe129f, sha256:076aee5a3763d41da6bef9565fdf3cb987606f567cd8b104aded2b38b7b47abf, sha256:0b536b6840e84c1c6a410f3a5aa727821e6108f3454d81a5cd5900999ef04f89, sha256:15efb7b93806d438e3bc590ca8ef2f953b0ce4f86f337ef4559d31ec6cf9d7dd, sha256:168259b1b184aa83a514f307352c25c56af111c269ffc109d9704e81f72e764b, sha256:2638389562bda1635b564490d76713695ff497242a83d9b684d27bb4a6cc9d7a, sha256:3556c5550de40027d3121ebbb170f61bbe19eb639c7ad0c7b482cd9b560cd23b, sha256:4a176959b6e7e00b5a0d6f549a479f869829bfd8150282c590deee6d099bbb6e, sha256:515a8b6edbb904594685da6e176ac9fbea8f73a5ebae947281de6613e27f1956, sha256:55535c7c2f61e2b2fc817c5cbe1af7cb907c7f011e46ae0a52caa4be1f19afe2, sha256:59153979d60f5bfe9e4c00e401e24dfe0469ef8da6d68247439d3278f30a180f, sha256:60cb8e5933193a3cc2912ee29ca331e9c15b2da034f76159b7abc520b3d1233a, sha256:6767ad399e9327bfdbaa40871be4254d1995f4a3ca3806127f10cec778bd9896, sha256:76a4f9bce0278becc2da7da3b8ef854bed41a991f4226911a24a9711baad672c, sha256:8cf33634b60c9cef346663a222d9841d3bbbc0a2f00221d6bcfd0d993d5543f6, sha256:94dd11d9f13ea1be17bac39c1942f527cbf7065f94953cf62dfe805653da2f8f, sha256:aafa46b5a39a27aca566198d3312fb3bde95ce9677085efd02c86f7ef6be4ec7, sha256:badca914580eb46385e7f7e4e426fea6de0a37b9e06bec252e481ae7ec287082, sha256:d76a26c5118c4d96e264acc9e3242d72e1a2b92e739807b3b69d8d47684b6677.
at ~/Library/Application Support/pypoetry/venv/lib/python3.7/site-packages/poetry/installation/executor.py:632 in _download_link
628│ "Invalid hashes ({}) for {} using archive {}. Expected one of {}.".format(
629│ ", ".join(sorted(archive_hashes)),
630│ package,
631│ archive_path.name,
→ 632│ ", ".join(sorted(hashes)),
633│ )
634│ )
635│
636│ return archive
• Installing python-dateutil (2.8.2)
• Installing pytz (2021.3)
Failed to add packages, reverting the pyproject.toml file to its original content.
Attempt to solve
remove existing poetry installation
curl -sSL https://install.python-poetry.org | python3 - --uninstall
install specific poetry version
Trying to following this issue comment I installed with:
curl -sSL https://install.python-poetry.org | python3 - --version 1.1.8
Try creating another env like before (Still fails)
I'm still getting failures.
python --version
Python 3.8.12
poetry --version
Poetry version 1.1.8
poetry new install_check_2
Created package install_check_2 in install_check_2
cd install_check_2
Then within the newly created project:
poetry env use python3.8
Creating virtualenv install-check-2-1m3rSWN1-py3.8 in /Users/user_name/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-2-1m3rSWN1-py3.8
poetry install
Updating dependencies
Resolving dependencies... (4.9s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (3.0.7): Failed
AttributeError
'Link' object has no attribute 'is_absolute'
at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/core/packages/file_dependency.py:33 in __init__
29│ self._path = path
30│ self._base = base or Path.cwd()
31│ self._full_path = path
32│
→ 33│ if not self._path.is_absolute():
34│ try:
35│ self._full_path = self._base.joinpath(self._path).resolve()
36│ except FileNotFoundError:
37│ raise ValueError("Directory {} does not exist".format(self._path))

There are several issue reports about invalid hashes. One common cause is running multiple Poetry instances simultaneously; another is aborted downloads (e.g. ^C). Poetry should be fixed to make corruption less likely to happen in such cases, and perhaps to try downloading corrupted files again automatically.
Until then, the workaround is to remove the offending file from the cache and let Poetry download a fresh copy.
You can find the full path to the offending file like this:
$ find ~/.cache/pypoetry -name numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl
Or, for the lazy, just blow away the entire cache:
$ rm -rf ~/.cache/pypoetry

I ctrl-C'd during a poetry install, which caused one of the cached wheels to partially download and have a hash that didn't match what was on PyPI.
On macOS with Poetry 1.1.13, the only method that worked for me was Yihua Zhou's comment:
rm -r ~/Library/Caches/pypoetry/cache
rm -r ~/Library/Caches/pypoetry/artifacts

Maybe this is a solution:
poetry cache clear . --all
rm poetry.lock
poetry install
https://github.com/python-poetry/poetry/issues/4523#issuecomment-923155835

If you're doing this in a Docker image, in addition to the answers above I also had to do:
docker builder prune
This could be necessary if taking using some of the more recent docker build caching features in the Dockerfile such as:
RUN --mount=type=cache,target=$POETRY_CACHE_DIR/cache \
--mount=type=cache,target=$POETRY_CACHE_DIR/artifacts \
poetry install
This is because the poetry cache is on the host file system and needs to be cleared for same reason as the answers above.

nrfutil - "ImportError: No module named main" on Nixos

I'm using the tool nrfutil which is implemented in Python. To be able to use it under NixOS I was using a default.nix file, that installed nrfutil into a venv. This worked for some time very well. (The last build on the build server using Nix within an alpine container could build the software I'm working on 11 days ago successfully.) When I do exactly the same things (i.e. restarting the CI server build without changes), the build fails now complaining about pip being incorrect:
$ nix-shell
New python executable in /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
Not overwriting existing python script /home/matthias/source/tbconnect/bootloader/.venv/bin/python (you must use /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7)
Installing pip, wheel...
done.
Traceback (most recent call last):
File "/home/matthias/source/tbconnect/bootloader/.venv/bin/pip", line 6, in <module>
from pip._internal.main import main
ImportError: No module named main
To me it seems that the module main should exist:
$ ls -l .venv/lib/python2.7/site-packages/pip/_internal/main.py
-rw-r--r-- 1 matthias matthias 1359 10月 15 12:27 .venv/lib/python2.7/site-packages/pip/_internal/main.py
I'm not very much into the Python environment, so I don't know any further. Has somebody any pointer for me where to continue debugging? How is Python resolving modules? Why doesn't it find the module, that seems to be present to me?
This is my default.nix that I use to install pip:
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
python27Packages.pip
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv --no-setuptools .venv
export PATH=$PWD/.venv/bin:$PATH
#pip install nrfutil
pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}
I replaced pip install nrfutil with pip help to make sure the problem is not the package I try to install itself.
I'm still using python 2.7 as the nrfutil still is not fit for Python 3.
Anyway replacing python27 with python37 did not change the error I get when trying to start pip.)
NixOS version used locally is 19.09. Nix in the CI docker container is nixos/nix:latest which is the nix package manager on Alpine Linux.
Update:
Actually it works when I replace the call to pip install nrfutil with python2.7 -m pip install nrfutil. This actually confuses me even more. python2.7 is exactly the binary that is in the shebang of pip:
[nix-shell:~/source/tbconnect/bootloader]$ type python2.7
python2.7 is /home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
[nix-shell:~/source/tbconnect/bootloader]$ type pip
pip is /home/matthias/source/tbconnect/bootloader/.venv/bin/pip
[nix-shell:~/source/tbconnect/bootloader]$ head --lines 2 .venv/bin/pip
#!/home/matthias/source/tbconnect/bootloader/.venv/bin/python2.7
# -*- coding: utf-8 -*-
Update 2:
I found out that another way to fix the problem is to edit .venv/bin/pip. This script tried the following import:
from pip._internal.main import main
Which I think is the new module path starting with pip 19.3. But I still have pip 19.2. When I change this line to:
from pip._internal import main
Running pip by typing pip is working.
The thing is I have no idea why the pip script is trying to load the new module path while NixOS still has the old version of pip.
I also opened an issue for NixOS on GitHub: https://github.com/NixOS/nixpkgs/issues/71178

I got your shell derivation to work by dropping the Python27Packages.pip,
(nix-shell) 2d [azul:/tmp/lixo12333] $
>>> pip list
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package Version
---------------- -------
behave 1.2.6
Click 7.0
crcmod 1.7
ecdsa 0.13.3
enum34 1.1.6
future 0.18.2
intelhex 2.2.1
ipaddress 1.0.23
libusb1 1.7.1
linecache2 1.0.0
nrfutil 5.2.0
parse 1.12.1
parse-type 0.5.2
pc-ble-driver-py 0.11.4
piccata 1.0.1
pip 19.3.1
protobuf 3.10.0
pyserial 3.4
pyspinel 1.0.0a3
PyYAML 4.2b4
setuptools 41.6.0
six 1.12.0
tqdm 4.37.0
traceback2 1.4.0
virtualenv 16.4.3
wheel 0.33.6
wrapt 1.11.2
(nix-shell) 2d [azul:/tmp/lixo12333] $
and my default.nix
with import <nixpkgs> {};
with pkgs.python27Packages;
stdenv.mkDerivation {
name = "impurePythonEnv";
buildInputs = [
automake
autoconf
gcc-arm-embedded-7
# these packages are required for virtualenv and pip to work:
#
python27Full
python27Packages.virtualenv
# the following packages are related to the dependencies of your python
# project.
# In this particular example the python modules listed in the
# requirements.txt require the following packages to be installed locally
# in order to compile any binary extensions they may require.
#
taglib
openssl
git
stdenv
zlib ];
src = null;
shellHook = ''
# set SOURCE_DATE_EPOCH so that we can use python wheels
SOURCE_DATE_EPOCH=$(date +%s)
virtualenv .venv
export PATH=$PWD/.venv/bin:$PATH
pip install nrfutil
#pip help
# the following is required to build micro_ecc_lib_nrf52.a in the SDK
export GNU_INSTALL_ROOT="${gcc-arm-embedded-7}/bin/"
unset CC
'';
}

Install needed libraries for Weasyprint on pipenv (Windows environment)

In order to start generating documents with Weasyprint I installed it on my Windows machine following these instructions:
https://weasyprint.readthedocs.io/en/stable/install.html#step-5-run-weasyprint
On my computer it works but I have a Django project where I want to integrate this library and I use pipenv. How to install the necessary libraries even in the virtual environment?
I tried setting the path for the pycairo package into the Pipfile like
pycairo= {path= "C:/Program Files/GTK3-Runtime Win64/bin/"}
but still it throws the error:
OSError: dlopen() failed to load a library: cairo / cairo-2 / cairo-gobject-2 / cairo.so.2
I have 64bit Windows machine and this is the Pipfile:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
pylint = "*"
[packages]
django = "*"
mysql = "*"
ipython = "*"
django-webpack = "*"
django-webpack-loader = "*"
django-livereload-server = "*"
pylint = "*"
reportlab = "*"
weasyprint = "*"
django-weasyprint = "*"
pycairo= {path= "C:/Program Files/GTK3-Runtime Win64/bin/"}
cairocffi = "*"
[requires]
python_version = "3.7"

You need install 'GTK+ 64 Bit Installer' path in this local:
C:\msys2
Source: WeasyPrint Github

I came across this error as well and followed every step mentioned in the Weasyprint docs for the installation. I was using PowerShell as my default terminal and pipenv. First I tried using the command import weasyprint inside python shell in my virtualenv, but it always returned the cairo, dlopen() etc error.
What worked for me was switching to cmd. Switched and used the same commands and now it's executing perfectly.
Also, in my pipfile, for weasyprint, which is enough to generate a report in django, I only have weasyprint installed. In the question, if it is still relevant for someone, the libraries reportlab, django-weasyprint, pycairo and cairocffi can be safely removed/uninstalled from pipenv.

Please type the following command:
WHERE libcairo-2.dll
you should be getting 'C:\msys2\mingw64\bin\libcairo-2.dll'
then open your cmd and type the following.
SET PROPER_GTK_FOLDER=
SET PATH=%PROPER_GTK_FOLDER%;%PATH%
Please follow the documentation it has everything to run on windows. It worked for me it i hope it will work for youenter link description here