We Keep Coding

How to install scripts from pyproject.toml?

I have this snippet in my pyproject.toml
[tool.poetry.scripts]
devel = "uvicorn api:app --reload"
prod = "uvicorn api:app"
I install it as mentioned in poetry documentation
(wowtcg-tracker-api-dIKBwUYp-py3.10) bastakka#Karel-NTB:/m/c/U/b/D/P/I/w/wowtcg-tracker-api (main↓2|✚9…9)
» poetry install
Installing dependencies from lock file
Package operations: 38 installs, 0 updates, 0 removals
• Installing idna (3.3)
...
• Installing yapf (0.32.0)
(wowtcg-tracker-api-dIKBwUYp-py3.10) bastakka#Karel-NTB:/m/c/U/b/D/P/I/w/wowtcg-tracker-api (main↓2|✚9…9)
»
But there were no scripts generated from this command.
Poetry can't run them
…-tracker-api-dIKBwUYp-py3.10) bastakka#Karel-NTB:/m/c/U/b/D/P/I/w/wowtcg-tracker-api (main↓2|✚9…9)
» poetry run prod
ModuleOrPackageNotFound
No file/folder found for package wowtcg-tracker-api
at ~/.poetry/lib/poetry/_vendor/py3.10/poetry/core/masonry/utils/module.py:63 in __init__
59│ "from": str(src.relative_to(self._path)),
60│ }
61│ ]
62│ else:
→ 63│ raise ModuleOrPackageNotFound(
64│ "No file/folder found for package {}".format(name)
65│ )
66│
67│ for package in packages:
…-tracker-api-dIKBwUYp-py3.10) bastakka#Karel-NTB:/m/c/U/b/D/P/I/w/wowtcg-tracker-api (main↓2|✚9…9)
»
And there are none in the venv bin either
…-tracker-api-dIKBwUYp-py3.10) bastakka#Karel-NTB:/m/c/U/b/D/P/I/w/wowtcg-tracker-api (main↓2|✚9…9)
» ls /home/bastakka/.cache/pypoetry/virtualenvs/wowtcg-tracker-api-dIKBwUYp-py3.10/bin
activate dotenv* pip* pyreverse* watchgod*
activate.csh epylint* pip3* python# wheel*
activate.fish get_objgraph* pip-3.10* python3# wheel3*
activate.nu httpx* pip3.10* python3.10# wheel-3.10*
activate.ps1 isort* prisma* symilar* wheel3.10*
activate_this.py isort-identify-imports* prisma-client-py* undill* yapf*
deactivate.nu normalizer* pylint* uvicorn* yapf-diff*
…-tracker-api-dIKBwUYp-py3.10) bastakka#Karel-NTB:/m/c/U/b/D/P/I/w/wowtcg-tracker-api (main↓2|✚9…9)
»
Am I missing something needed or doing something wrong?

tool.poetry.scripts is used for "scripts or executables that will be installed when installing the package". You'd have to drop the poetry run prefix to run them.
For development scripts I'd recommend something like taskipy. e.g.
# pyproject.toml
[tool.taskipy.tasks]
devel = "uvicorn api:app --reload"
prod = "uvicorn api:app"
Then you can run something like poetry run task devel

Python poetry install failure - invalid hashes

I'm unsure exactly what information / context to provide with this post in order for it to be
properly answered - so if I can add more information please let me know.
Whilst trying to install a package I've had an error message as follows:
Invalid hashes (sha256:01d6b62184e55367ce7d770bd87d97a6cfaf783ff7ba8328e7473a339fe19807) for numpy (1.22.2)
I don't know how to go about troubleshooting this problem.
poetry --version
(install-check-clU745_7-py3.8) ➜ install_check poetry --version
Poetry version 1.1.12
poetry env info
Output of poetry env info:
(install-check-clU745_7-py3.8) ➜ install_check poetry env info
Virtualenv
Python: 3.8.12
Implementation: CPython
Path: /Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-clU745_7-py3.8
Valid: True
System
Platform: darwin
OS: posix
Python: /Users/user_name/.pyenv/versions/3.8.12
pyproject.toml
pyproject.toml config:
(install-check-clU745_7-py3.8) ➜ install_check cat pyproject.toml
[tool.poetry]
name = "install_check"
version = "0.1.0"
description = ""
authors = ["user_name <user_name#email_add.com>"]
[tool.poetry.dependencies]
python = "^3.8"
[tool.poetry.dev-dependencies]
pytest = "^5.2"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
Steps to reproduce
Here is an attempt to clearly outline a reproduction of this problem:
python --version
Python 3.8.12
which python
/Users/user_name/.pyenv/shims/python
poetry new install_check
Created package install_check in install_check
cd install_check
cat pyproject.toml| grep python
python = "^3.7"
# update pyproject.toml to 3.8 instead of 3.7.
cat pyproject.toml | grep python
python = "^3.8"
python --version
Python 3.8.12
poetry env use python3.8
which python3.8
/Users/user_name/.pyenv/shims/python3.8
poetry env use python3.8
poetry install
Updating dependencies
Resolving dependencies... (5.0s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (3.0.7)
• Installing attrs (21.4.0)
• Installing more-itertools (8.12.0)
• Installing packaging (21.3)
• Installing pluggy (0.13.1)
• Installing py (1.11.0)
• Installing wcwidth (0.2.5)
• Installing pytest (5.4.3)
Installing the current project: install_check (0.1.0)
Following this, after leaving and revisiting the directory to activate the env, I have:
(install-check-clU745_7-py3.8) ➜ install_check which python
/Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-clU745_7-py3.8/bin/python
(install-check-clU745_7-py3.8) ➜ install_check python --version
Python 3.8.12
Trying to add pandas then gives the following error:
(install-check-clU745_7-py3.8) ➜ install_check poetry add pandas
Using version ^1.4.0 for pandas
Updating dependencies
Resolving dependencies... (2.3s)
Writing lock file
Package operations: 5 installs, 0 updates, 0 removals
• Installing six (1.16.0)
• Installing numpy (1.22.2): Failed
RuntimeError
Invalid hashes (sha256:01d6b62184e55367ce7d770bd87d97a6cfaf783ff7ba8328e7473a339fe19807) for numpy (1.22.2) using archive numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl. Expected one of sha256:03ae5850619abb34a879d5f2d4bb4dcd025d6d8fb72f5e461dae84edccfe129f, sha256:076aee5a3763d41da6bef9565fdf3cb987606f567cd8b104aded2b38b7b47abf, sha256:0b536b6840e84c1c6a410f3a5aa727821e6108f3454d81a5cd5900999ef04f89, sha256:15efb7b93806d438e3bc590ca8ef2f953b0ce4f86f337ef4559d31ec6cf9d7dd, sha256:168259b1b184aa83a514f307352c25c56af111c269ffc109d9704e81f72e764b, sha256:2638389562bda1635b564490d76713695ff497242a83d9b684d27bb4a6cc9d7a, sha256:3556c5550de40027d3121ebbb170f61bbe19eb639c7ad0c7b482cd9b560cd23b, sha256:4a176959b6e7e00b5a0d6f549a479f869829bfd8150282c590deee6d099bbb6e, sha256:515a8b6edbb904594685da6e176ac9fbea8f73a5ebae947281de6613e27f1956, sha256:55535c7c2f61e2b2fc817c5cbe1af7cb907c7f011e46ae0a52caa4be1f19afe2, sha256:59153979d60f5bfe9e4c00e401e24dfe0469ef8da6d68247439d3278f30a180f, sha256:60cb8e5933193a3cc2912ee29ca331e9c15b2da034f76159b7abc520b3d1233a, sha256:6767ad399e9327bfdbaa40871be4254d1995f4a3ca3806127f10cec778bd9896, sha256:76a4f9bce0278becc2da7da3b8ef854bed41a991f4226911a24a9711baad672c, sha256:8cf33634b60c9cef346663a222d9841d3bbbc0a2f00221d6bcfd0d993d5543f6, sha256:94dd11d9f13ea1be17bac39c1942f527cbf7065f94953cf62dfe805653da2f8f, sha256:aafa46b5a39a27aca566198d3312fb3bde95ce9677085efd02c86f7ef6be4ec7, sha256:badca914580eb46385e7f7e4e426fea6de0a37b9e06bec252e481ae7ec287082, sha256:d76a26c5118c4d96e264acc9e3242d72e1a2b92e739807b3b69d8d47684b6677.
at ~/Library/Application Support/pypoetry/venv/lib/python3.7/site-packages/poetry/installation/executor.py:632 in _download_link
628│ "Invalid hashes ({}) for {} using archive {}. Expected one of {}.".format(
629│ ", ".join(sorted(archive_hashes)),
630│ package,
631│ archive_path.name,
→ 632│ ", ".join(sorted(hashes)),
633│ )
634│ )
635│
636│ return archive
• Installing python-dateutil (2.8.2)
• Installing pytz (2021.3)
Failed to add packages, reverting the pyproject.toml file to its original content.
Attempt to solve
remove existing poetry installation
curl -sSL https://install.python-poetry.org | python3 - --uninstall
install specific poetry version
Trying to following this issue comment I installed with:
curl -sSL https://install.python-poetry.org | python3 - --version 1.1.8
Try creating another env like before (Still fails)
I'm still getting failures.
python --version
Python 3.8.12
poetry --version
Poetry version 1.1.8
poetry new install_check_2
Created package install_check_2 in install_check_2
cd install_check_2
Then within the newly created project:
poetry env use python3.8
Creating virtualenv install-check-2-1m3rSWN1-py3.8 in /Users/user_name/Library/Caches/pypoetry/virtualenvs
Using virtualenv: /Users/user_name/Library/Caches/pypoetry/virtualenvs/install-check-2-1m3rSWN1-py3.8
poetry install
Updating dependencies
Resolving dependencies... (4.9s)
Writing lock file
Package operations: 8 installs, 0 updates, 0 removals
• Installing pyparsing (3.0.7): Failed
AttributeError
'Link' object has no attribute 'is_absolute'
at ~/Library/Application Support/pypoetry/venv/lib/python3.8/site-packages/poetry/core/packages/file_dependency.py:33 in __init__
29│ self._path = path
30│ self._base = base or Path.cwd()
31│ self._full_path = path
32│
→ 33│ if not self._path.is_absolute():
34│ try:
35│ self._full_path = self._base.joinpath(self._path).resolve()
36│ except FileNotFoundError:
37│ raise ValueError("Directory {} does not exist".format(self._path))

There are several issue reports about invalid hashes. One common cause is running multiple Poetry instances simultaneously; another is aborted downloads (e.g. ^C). Poetry should be fixed to make corruption less likely to happen in such cases, and perhaps to try downloading corrupted files again automatically.
Until then, the workaround is to remove the offending file from the cache and let Poetry download a fresh copy.
You can find the full path to the offending file like this:
$ find ~/.cache/pypoetry -name numpy-1.22.2-cp38-cp38-macosx_10_14_x86_64.whl
Or, for the lazy, just blow away the entire cache:
$ rm -rf ~/.cache/pypoetry

I ctrl-C'd during a poetry install, which caused one of the cached wheels to partially download and have a hash that didn't match what was on PyPI.
On macOS with Poetry 1.1.13, the only method that worked for me was Yihua Zhou's comment:
rm -r ~/Library/Caches/pypoetry/cache
rm -r ~/Library/Caches/pypoetry/artifacts

Maybe this is a solution:
poetry cache clear . --all
rm poetry.lock
poetry install
https://github.com/python-poetry/poetry/issues/4523#issuecomment-923155835

If you're doing this in a Docker image, in addition to the answers above I also had to do:
docker builder prune
This could be necessary if taking using some of the more recent docker build caching features in the Dockerfile such as:
RUN --mount=type=cache,target=$POETRY_CACHE_DIR/cache \
--mount=type=cache,target=$POETRY_CACHE_DIR/artifacts \
poetry install
This is because the poetry cache is on the host file system and needs to be cleared for same reason as the answers above.

Download dependencies declared in pyproject.toml using Pip

I have a Python project that doesn't contain requirements.txt.
But it has a pyproject.toml file.
How can I download packages (dependencies) required by this Python project and declared in pyproject.toml using the Pip package manager (instead of the build tool Poetry).
So instead of pip download -r requirements.txt, something like pip download -r pyproject.toml.

Here is an example of .toml file:
[build-system]
requires = [
"flit_core >=3.2,<4",
]
build-backend = "flit_core.buildapi"
[project]
name = "aedttest"
authors = [
{name = "Maksim Beliaev", email = "beliaev.m.s#gmail.com"},
{name = "Bo Yang", email = "boy#kth.se"},
]
readme = "README.md"
requires-python = ">=3.7"
classifiers = ["License :: OSI Approved :: MIT License"]
dynamic = ["version", "description"]
dependencies = [
"pyaedt==0.4.7",
"Django==3.2.8",
]
[project.optional-dependencies]
test = [
"black==21.9b0",
"pre-commit==2.15.0",
"mypy==0.910",
"pytest==6.2.5",
"pytest-cov==3.0.0",
]
deploy = [
"flit==3.4.0",
]
to install core dependencies you run:
pip install .
if you need test(develop) environment (we use test because it is a name defined in .toml file, you can use any):
pip install .[test]
To install from Wheel:
pip install C:\git\aedt-testing\dist\aedttest-0.0.1-py3-none-any.whl[test]

pip supports installing pyproject.toml dependencies natively.
As of version 10.0, pip supports projects declaring dependencies that are required at install time using a pyproject.toml file, in the form described in PEP 518. When building a project, pip will install the required dependencies locally, and make them available to the build process. Furthermore, from version 19.0 onwards, pip supports projects specifying the build backend they use in pyproject.toml, in the form described in PEP 517.
From the project's root, use pip's local project install:
python -m pip install .

You can export the dependencies to a requirements.txt and use pip download afterwards:
poetry export -f requirements.txt > requirements.txt
pip download -r requirements.txt

How Can I generate a requirements file from Pipfile with indexes, without locking?

I want to generate a requirements.txt file from a Pipfile.
I've tried several tools that do this (pigar, pipenv-to-requirements, pipfile-requirements) but none of them address indexes
Meaning that for the following Pipfile:
[[source]]
url = <some-url>
verify_ssl = true
name = <some-name>
[packages]
statsd = "*"
pytest = "*"
I get the following requirements file:
pytest
statsd
Instead of the expected requirements file with an index:
-i <some-url>
pytest
statsd
Another solution that was suggested to me is using "pipenv lock":
pipenv lock --requirements > requirements.txt
But this updates Pipfile.lock - which I really want to avoid
Isn't there some way to generate requirements without locking?

pipenv lock --requirements > requirements.txt isn't working.
Error: No such option: -r
pipenv version:
bash-4.2# pipenv --version
pipenv, version 2022.9.4
Following workaround worked:
pipenv run pip freeze > requirements.txt

Using an extra python package index url with setup.py

Is there a way to use an extra Python package index (ala pip --extra-index-url pypi.example.org mypackage) with setup.py so that running python setup.py install can find the packages hosted on pypi.example.org?

If you're the package maintainer, and you want to host one or more dependencies for your package somewhere other than PyPi, you can use the dependency_links option of setuptools in your distribution's setup.py file. This allows you to provide an explicit location where your package can be located.
For example:
from setuptools import setup
setup(
name='somepackage',
install_requires=[
'somedep'
],
dependency_links=[
'https://pypi.example.org/pypi/somedep/'
]
# ...
)
If you host your own index server, you'll need to provide links to the pages containing the actual download links for each egg, not the page listing all of the packages (e.g. https://pypi.example.org/pypi/somedep/, not https://pypi.example.org/)

setuptools uses easy_install under the hood.
It relies on either setup.cfg or ~/.pydistutils.cfg as documented here.
Extra paths to packages can be defined in either of these files with the find_links. You can override the registry url with index_url but cannot supply an extra-index-url. Example below inspired by the docs:
[easy_install]
find_links = http://mypackages.example.com/somedir/
http://turbogears.org/download/
http://peak.telecommunity.com/dist/
index-url = https://mypi.example.com

I wanted to post a latest answer to this since both the top answers are obsolete; use of easy_install has been deprecated by setuptools.
https://setuptools.pypa.io/en/latest/deprecated/easy_install.html
Easy Install is deprecated. Do not use it. Instead use pip. If you think you need Easy Install, please reach out to the PyPA team (a ticket to pip or setuptools is fine), describing your use-case.
Please use pip moving forward. You can do one of the following:
provide --index-url flag to pip command
define index-url in pip.conf file
define PIP_INDEX_URL environment variable
https://pip.pypa.io/en/stable/topics/configuration/

The following worked for me (develop, not install):
$ python setup.py develop --index-url https://x.com/n/r/pypi-proxy/simple
Where https://x.com/n/r/pypi-proxy/simple is a local PyPI repository.

Found solution when using Dockerfile:
RUN cd flask-mongoengine-0.9.5 && \
/bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple >> setup.cfg && \
python setup.py install
Which /bin/echo -e [easy_install]\\nindex-url = https://pypi.tuna.tsinghua.edu.cn/simple will exists in file setup.cfg:
[easy_install]
index-url = https://pypi.tuna.tsinghua.edu.cn/simple

this worked for me
PIP_INDEX_URL=<MY CUSTOM PIP INDEX URL> pip install -e .
I use setup.py and setup.cfg

As far as I know, you cant do that.
You need to tell pip this, or by passing a parameter like you mentioned, or by setting this on the user environment.
Check my ~/.pip/pip.conf:
[global]
download_cache = ~/.cache/pip
index-url = http://user:pass#localpypiserver.com:80/simple
timeout = 300
In this case, my local pypiserver also proxies all packages from pypi.python.org, so I dont need to add a 2nd entry.

You can include --extra-index-urls in a requirements.txt file. See: http://pip.readthedocs.org/en/0.8.3/requirement-format.html