I've got a couple of projects here for which I'm preparing documentation at the moment, hosted at readthedocs.org. FYI, all of them use poetry and I use custom .readthedocs.yml files with this entry:
python:
install:
- method: pip
path: .
It works fine for most projects, but it fails for two for different reasons during installation of the project via pip:
The first one uses PyGObject, which failes like this:
Package gobject-introspection-1.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `gobject-introspection-1.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gobject-introspection-1.0' found
Command '('pkg-config', '--print-errors', '--exists', 'gobject-introspection-1.0 >= 1.56.0')' returned non-zero exit status 1.
Try installing it with: 'sudo apt install libgirepository1.0-dev'
So it seems that PyGObject cannot be installed without some system packages to be installed. I could rearrange the code so that the import is not top-level. But still I need it in the dependencies. Can I tell pip install to ignore this single package somehow? Any other idea?
The second project compiles some C++ code via Cython and fails, because it's missing a library. I use a custom build script in the pyproject.toml:
[tool.poetry.build]
script = "build.py"
generate-setup-file = false
Is there some flag in pip that I could set and retrieve in build.py to skip the compilation? Or is there a better way?
I'm unsure exactly what information / context to provide with this post in order for it to be
properly answered - so if I can add more information please let me know.
Whilst trying to install a package I've had an error message as follows:
Invalid hashes (sha256:01d6b62184e55367ce7d770bd87d97a6cfaf783ff7ba8328e7473a339fe19807) for numpy (1.22.2)
I don't know how to go about troubleshooting this problem.
poetry --version
(install-check-clU745_7-py3.8) ➜ install_check poetry --version
Poetry version 1.1.12
poetry env info
Output of poetry env info:
(install-check-clU745_7-py3.8) ➜ install_check poetry env info
Virtualenv
Python: 3.8.12
Implementation: CPython
Path: /Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-clU745_7-py3.8
Valid: True
System
Platform: darwin
OS: posix
Python: /Users/user_name/.pyenv/versions/3.8.12
pyproject.toml
pyproject.toml config:
(install-check-clU745_7-py3.8) ➜ install_check cat pyproject.toml
[tool.poetry]
name = "install_check"
version = "0.1.0"
description = ""
authors = ["user_name <user_name#email_add.com>"]
[tool.poetry.dependencies]
python = "^3.8"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Steps to reproduce
Here is an attempt to clearly outline a reproduction of this problem:
python --version
Python 3.8.12
which python
/Users/user_name/.pyenv/shims/python
poetry new install_check
Created package install_check in install_check
cd install_check
cat pyproject.toml| grep python
python = "^3.7"
# update pyproject.toml to 3.8 instead of 3.7.
cat pyproject.toml | grep python
python = "^3.8"
python --version
Python 3.8.12
poetry env use python3.8
which python3.8
/Users/user_name/.pyenv/shims/python3.8
poetry env use python3.8
poetry install
Updating dependencies
Resolving dependencies... (5.0s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (3.0.7)
• Installing attrs (21.4.0)
• Installing more-itertools (8.12.0)
• Installing packaging (21.3)
• Installing pluggy (0.13.1)
• Installing py (1.11.0)
• Installing wcwidth (0.2.5)
• Installing pytest (5.4.3)
Installing the current project: install_check (0.1.0)
Following this, after leaving and revisiting the directory to activate the env, I have:
(install-check-clU745_7-py3.8) ➜ install_check which python
/Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-clU745_7-py3.8/bin/python
(install-check-clU745_7-py3.8) ➜ install_check python --version
Python 3.8.12
Trying to add pandas then gives the following error:
(install-check-clU745_7-py3.8) ➜ install_check poetry add pandas
Using version ^1.4.0 for pandas
Updating dependencies
Resolving dependencies... (2.3s)
Writing lock file
Package operations: 5 installs, 0 updates, 0 removals
• Installing six (1.16.0)
• Installing numpy (1.22.2): Failed
RuntimeError
Invalid hashes (sha256:01d6b62184e55367ce7d770bd87d97a6cfaf783ff7ba8328e7473a339fe19807) for numpy (1.22.2) using archive numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl. Expected one of sha256:03ae5850619abb34a879d5f2d4bb4dcd025d6d8fb72f5e461dae84edccfe129f, sha256:076aee5a3763d41da6bef9565fdf3cb987606f567cd8b104aded2b38b7b47abf, sha256:0b536b6840e84c1c6a410f3a5aa727821e6108f3454d81a5cd5900999ef04f89, sha256:15efb7b93806d438e3bc590ca8ef2f953b0ce4f86f337ef4559d31ec6cf9d7dd, sha256:168259b1b184aa83a514f307352c25c56af111c269ffc109d9704e81f72e764b, sha256:2638389562bda1635b564490d76713695ff497242a83d9b684d27bb4a6cc9d7a, sha256:3556c5550de40027d3121ebbb170f61bbe19eb639c7ad0c7b482cd9b560cd23b, sha256:4a176959b6e7e00b5a0d6f549a479f869829bfd8150282c590deee6d099bbb6e, sha256:515a8b6edbb904594685da6e176ac9fbea8f73a5ebae947281de6613e27f1956, sha256:55535c7c2f61e2b2fc817c5cbe1af7cb907c7f011e46ae0a52caa4be1f19afe2, sha256:59153979d60f5bfe9e4c00e401e24dfe0469ef8da6d68247439d3278f30a180f, sha256:60cb8e5933193a3cc2912ee29ca331e9c15b2da034f76159b7abc520b3d1233a, sha256:6767ad399e9327bfdbaa40871be4254d1995f4a3ca3806127f10cec778bd9896, sha256:76a4f9bce0278becc2da7da3b8ef854bed41a991f4226911a24a9711baad672c, sha256:8cf33634b60c9cef346663a222d9841d3bbbc0a2f00221d6bcfd0d993d5543f6, sha256:94dd11d9f13ea1be17bac39c1942f527cbf7065f94953cf62dfe805653da2f8f, sha256:aafa46b5a39a27aca566198d3312fb3bde95ce9677085efd02c86f7ef6be4ec7, sha256:badca914580eb46385e7f7e4e426fea6de0a37b9e06bec252e481ae7ec287082, sha256:d76a26c5118c4d96e264acc9e3242d72e1a2b92e739807b3b69d8d47684b6677.
at ~/Library/Application Support/pypoetry/venv/lib/python3.7/site-packages/poetry/installation/executor.py:632 in _download_link
628│ "Invalid hashes ({}) for {} using archive {}. Expected one of {}.".format(
629│ ", ".join(sorted(archive_hashes)),
630│ package,
631│ archive_path.name,
→ 632│ ", ".join(sorted(hashes)),
633│ )
634│ )
635│
636│ return archive
• Installing python-dateutil (2.8.2)
• Installing pytz (2021.3)
Failed to add packages, reverting the pyproject.toml file to its original content.
Attempt to solve
remove existing poetry installation
curl -sSL https://install.python-poetry.org | python3 - --uninstall
install specific poetry version
Trying to following this issue comment I installed with:
curl -sSL https://install.python-poetry.org | python3 - --version 1.1.8
Try creating another env like before (Still fails)
I'm still getting failures.
python --version
Python 3.8.12
poetry --version
Poetry version 1.1.8
poetry new install_check_2
Created package install_check_2 in install_check_2
cd install_check_2
Then within the newly created project:
poetry env use python3.8
Creating virtualenv install-check-2-1m3rSWN1-py3.8 in /Users/user_name/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-2-1m3rSWN1-py3.8
poetry install
Updating dependencies
Resolving dependencies... (4.9s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (3.0.7): Failed
AttributeError
'Link' object has no attribute 'is_absolute'
at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/core/packages/file_dependency.py:33 in __init__
29│ self._path = path
30│ self._base = base or Path.cwd()
31│ self._full_path = path
32│
→ 33│ if not self._path.is_absolute():
34│ try:
35│ self._full_path = self._base.joinpath(self._path).resolve()
36│ except FileNotFoundError:
37│ raise ValueError("Directory {} does not exist".format(self._path))
There are several issue reports about invalid hashes. One common cause is running multiple Poetry instances simultaneously; another is aborted downloads (e.g. ^C). Poetry should be fixed to make corruption less likely to happen in such cases, and perhaps to try downloading corrupted files again automatically.
Until then, the workaround is to remove the offending file from the cache and let Poetry download a fresh copy.
You can find the full path to the offending file like this:
$ find ~/.cache/pypoetry -name numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl
Or, for the lazy, just blow away the entire cache:
$ rm -rf ~/.cache/pypoetry
I ctrl-C'd during a poetry install, which caused one of the cached wheels to partially download and have a hash that didn't match what was on PyPI.
On macOS with Poetry 1.1.13, the only method that worked for me was Yihua Zhou's comment:
rm -r ~/Library/Caches/pypoetry/cache
rm -r ~/Library/Caches/pypoetry/artifacts
Maybe this is a solution:
poetry cache clear . --all
rm poetry.lock
poetry install
https://github.com/python-poetry/poetry/issues/4523#issuecomment-923155835
If you're doing this in a Docker image, in addition to the answers above I also had to do:
docker builder prune
This could be necessary if taking using some of the more recent docker build caching features in the Dockerfile such as:
RUN --mount=type=cache,target=$POETRY_CACHE_DIR/cache \
--mount=type=cache,target=$POETRY_CACHE_DIR/artifacts \
poetry install
This is because the poetry cache is on the host file system and needs to be cleared for same reason as the answers above.
I'm learning how to create conda packages, and I am having trouble with a package that will work locally, but that when downloaded from anaconda into a different server will return an error:
/lib64/libm.so.6: version `GLIBC_2.29' not found
The meta.yaml looks like the following:
package:
name: app
version: 2.4
source:
git_url: https://gitlab.com/user/repo.git
git_tag: v2.4
requirements:
build:
host:
run:
about:
home: https://gitlab.com/user/repo
license: GPL-3
license_family: GPL
summary: blabla.
The app is build with a simple build.sh script:
#!/bin/bash
set -x
echo $(pwd)
make
BIN=$PREFIX/bin
mkdir -p $BIN
cp app $BIN
I assumed that build: glibc >= 2.29 under requirements would make the job, but that results in an error when running conda build ..
How can I include GLIBC in the package? is that something meant to be done manually? from the package version I can download from anaconda, I can see other packages are downloaded as well (e.g. libgcc-ng) that I did not really mention in the meta.yaml or anywhere.
How can I include GLIBC in the package?
You can't, for reasons explained here.
Your best bet is to build on a system (or in a docker container) which has the lowest version of GLIBC you need to support (i.e. the version installed on the server(s) you will be running your package on).
Question
Is there a way to build a wheel for a package while in a different repository such that the wheel has been built exactly as it would be if you built the wheel inside of the repository containing the package?
Example
Consider the following repo:
/repo-containing-your-package
|___ your_module/
|___ setup.py
Build method A
When I run python setup.py bdist_wheel from within repo-containing-your-package it builds the wheel as expected, including your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), I can run python -m your_module.foo from the command line.
When the package is building, I get output that verifies that my module has been picked up by the wheel:
creating 'dist/your_module-#.#.#-py3-none-any.whl' and adding 'build/bar' to it
adding 'your_module/__init__.py'
etc...
Build method B
However, if I run python ../repo-containing-your-package/setup.py bdist_wheel from a repository that is a sibling to repo-containing-your-package, it does not build the wheel as expected, as it fails to include your_module. This means after I install pip install ./dist/your_module-#.#.#-py3-none-any.whl (which is successful), attempting python -m your_module.foo fails:
Error while finding module specification for 'your_module.foo' (ModuleNotFoundError: No module named 'your_module')
The fact that the module has not been properly installed with the package is confirmed by reviewing the build output, which does not include the adding 'your_module' output that method A includes.
Two solutions I know of:
change working directory in setup.py
If you can modify the setup script, you can change the working directory programmatically. Add an os.chdir call early enough in the setup script:
import os
from setuptools import setup
os.chdir(os.path.dirname(__file__))
setup(...)
You can also change the working directory with other means without having to modify the setup script, e.g. in bash:
$ pushd path/to/repo; python setup.py bdist_wheel; popd
Use pip wheel
pip has a subcommand wheel that builds a wheel from the given arg; this arg is usually the name of the package, but can be a directory containing the setup script. Pass -e in that case so the wheel has the correct name:
$ pip wheel -e path/to/repo
I want to use setuptools to create a package consisting of two files: foo.py (script) and foo.conf.
Then I want to publish the package on my devpi-server and then install the package using pip.
Suppose I that initially I have my current working directory clean
$ ls -l
total 0
Then I issue pip install (or download?) command
$ pip install -i http://mydevpi.server foo
And get a dir with my two files created
$ tree
.
|
foo
|
|\_ foo.py
|
\_ foo.conf
So questions are:
what setuptools configuration should I use?
what exact pip command should I use to install the package the way I want? Will pip install -i http://mydevpi.server --target=. do the trick?
First write somethings as setup.py in foo directory like:
import setuptools
setuptools.setup(
name='foo_pip',
version='1',
packages=[''],
url='1',
license='1',
author='1',
author_email='1',
description='1'
)
(You can use distutils or setuptools)
Then python setup.py bdist_wheel -d TARGET and there will be a whl file in target directory, copy the path.
You can now install using pip install the_wheel_file_path --prefix="the_path_to_install"
Something like this
Processing .../TARGET/foo_pip-1-py2-none-any.whl
Installing collected packages: foo-pip
Successfully installed foo-pip-1
Then use it by import foo