We Keep Coding

Poetry: how to specify the same dependency twice with different CPU architectures (e.g x86 and x86_64)

I am evaluating using Poetry for packaging and building a desktop application.
The only roadblocker is that poetry doesn't seam to allow specifying the same package twice. For instance I couldn't do the following:
[tool.poetry.dependencies]
python = "^3.9"
lru-dict = {path = "./packages/lru_dict-1.1.6-cp39-cp39-win_amd64.whl"}
lru-dict = {path = "./packages/lru_dict-1.1.6-cp39-cp39-win32.whl"}
Notice that the lru-dict package is specified twice with the only difference being the bitness (i.e. the CPU architecture) that the package is built for.
I know I can upload the package to PyPI and pip will choose the appropriate version dynamically. But what about private or local packages?

From the poetry documentation:
Poetry supports environment markers via the markers property.
One of those markers is platform_machine, which is the output of platform.machine().
So you should be able to do something like this:
[tool.poetry.dependencies]
python = "^3.9"
lru-dict = [
{path = "./packages/lru_dict-1.1.6-cp39-cp39-win_amd64.whl", markers = "platform_machine == 'amd64'"},
{path = "./packages/lru_dict-1.1.6-cp39-cp39-win32.whl", markers = "platform_machine == 'win32'"}
]

Tox: per-platform configuration for tools like flake8 and pylint

I've got tox.ini configuration for ci both in linux and windows environments, something like that:
[tox]
envlist =
{py3,py27}-{test,lint}-{linux,windows}
flake8
check-package
skip_missing_interpreters = true
isolated_build = True
distdir = dist
[testenv]
platform =
linux: linux
windows: win
# Reuse py3 and py27 envs for pylint
envdir =
py3: {toxworkdir}/py3
py27: {toxworkdir}/py27
!py3-!py27: {toxworkdir}/{envname}
deps =
py27: setuptools < 45.0.0
# test extras must include pytest and pylint
extras = test
commands =
test: python -m pytest -c {toxinidir}/pytest.ini --junitxml=test-reports/pytest.{envname}.xml {posargs}
lint: python -m pylint --rcfile=tox.ini src/displaylink {posargs}
[testenv:flake8]
basepython = python3
skip_install = true
deps = flake8
commands = flake8 src tests
[tool:pylint]
disable = missing-docstring,
R,
C,
line-too-long
output-format = parseable
reports = no
extension-pkg-whitelist = win32api, win32gui
[flake8]
ignore = E501, E722, W503
per-file-ignores =
# imported but unused
__init__.py: F401
max-complexity = 10
Question is how may I add per-platform configuration for tools (flake and pylint) ? I need to exclude files/dirs from flake8 and pylint runs depending on os, i.e. I've got windows subdir with files that won't pass linting on linux and vice-versa
EDIT:
example of pylint errors I have on linux:
py3-lint-linux run-test: commands[0] | python -m pylint --rcfile=tox.ini src/displaylink
************* Module displaylink.qa.windows.registry
registry.py:4: [E0401(import-error), ] Unable to import 'win32con'
registry.py:74: [E0602(undefined-variable), reg_value_exists] Undefined variable 'WindowsError'
registry.py:82: [E0602(undefined-variable), reg_key_exists] Undefined variable 'WindowsError'

Maybe I miss some parts of your requirement, but running flake8 platform specific seems pretty straightforward:
[testenv]
platform =
linux: linux
windows: win
[testenv:flake8]
deps = flake8
commands =
linux: flake8 <linux specific directories>
windows: flake8 <windows specific directories>
Also compare to the official documentation.

How to use python black formatter under a project for python>=3.5 managed by poetry?

I created a python project "foo" with Poetry.
This is the content of pyproject.toml:
[tool.poetry]
name = "bar"
version = "0.1.0"
description = ""
[tool.poetry.dependencies]
python = ">=3.5"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
This package is compatible with Python3.5.
I want to black formatter, which is not compatible with Python3.5.
I think there is no problem if I use Python>=3.6 for development, but I cannot install black formatter:
$ poetry add black --dev
[SolverProblemError]
The current project's Python requirement (>=3.5) is not compatible with some of the required packages Python requirement:
- black requires Python >=3.6
Because no versions of black match >19.10b0,<20.0
and black (19.10b0) requires Python >=3.6, black is forbidden.
So, because bar depends on black (^19.10b0), version solving failed.
So I installed black directly with pip:
$ poetry run pip install black
This way doesn't sit well with me. I want to install black by poetry.
How should I do? (I don't want to modify the dependency to python>=3.6)

Seems a bit late but actually you can do what you want even if black supports only Python >=3.6.2
In your pyproject.toml you can define a restricted dependcy as documented in https://python-poetry.org/docs/dependency-specification/#python-restricted-dependencies
[tool.poetry.dependencies]
python = ">=3.5"
[tool.poetry.dev-dependencies]
black = {version = "^21.7b0", python = ">=3.6.2"}
Poetry won't complain and you won't have any problems since it is a dev dependency.

You need to edit the python value in your pyproject.toml:
[tool.poetry]
name = "bar"
version = "0.1.0"
description = ""
[tool.poetry.dependencies]
python = ">=3.6"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

In NixOS, how can I install an environment with the Python packages SpaCy, pandas, and jenks-natural-breaks?

I'm very new to NixOS, so please forgive my ignorance. I'm just trying to set up a Python environment---any kind of environment---for developing with SpaCy, the SpaCy data, pandas, and jenks-natural-breaks. Here's what I've tried so far:
pypi2nix -V "3.6" -E gcc -E libffi -e spacy -e pandas -e numpy --default-overrides, followed by nix-build -r requirements.nix -A packages. I've managed to get the first command to work, but the second fails with Could not find a version that satisfies the requirement python-dateutil>=2.5.0 (from pandas==0.23.4)
Writing a default.nix that looks like this: with import <nixpkgs> {};
python36.withPackages (ps: with ps; [ spacy pandas scikitlearn ]). This fails with collision between /nix/store/9szpqlby9kvgif3mfm7fsw4y119an2kb-python3.6-msgpack-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so and /nix/store/d08bgskfbrp6dh70h3agv16s212zdn6w-python3.6-msgpack-python-0.5.6/lib/python3.6/site-packages/msgpack/_packer.cpython-36m-x86_64-linux-gnu.so
Making a new virtualenv, and then running pip install on all these packages. Scikit-learn fails to install, with fish: Unknown command 'ar rc build/temp.linux-x86_64-3.6/liblibsvm-skl.a build/temp.linux-x86_64-3.6/sklearn/svm/src/libsvm/libsvm_template.o'
I guess ideally I'd like to install this environment with nix, so that I could enter it with nix-shell, and so other environments could reuse the same python packages. How would I go about doing that? Especially since some of these packages exist in nixpkgs, and others are only on Pypi.

Caveat
I had trouble with jenks-natural-breaks to the tune of
nix-shell ❯ poetry run python -c 'import jenks_natural_breaks'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/matt/2022/12/28-2/.venv/lib/python3.10/site-packages/jenks_natural_breaks/__init__.py", line 5, in <module>
from ._jenks_matrices import ffi as _ffi
ModuleNotFoundError: No module named 'jenks_natural_breaks._jenks_matrices'
So I'm going to use jenkspy which appears to be a bit livelier. If that doesn't scratch your itch, I'd contact the maintainer of jenks-natural-breaks for guidance
Flakes
you said:
so other environments could reuse the same python packages
Which makes me think that a flake.nix is what you need. What's cool about flakes is that you can define an environment that has spacy, pandas, and jenkspy with one flake. And then you (or somebody else) might say:
I want an env like Jonathan's, except I also want sympy
and rather than copying your env and making tweaks, they can declare your env as a build input and write a flake.nix with their modifications--which can be further modified by others.
One could imagine a sort of family-tree of environments, so you just need to pick the one that suits your task. The python community has not yet converged on this vision.
Poetry
Poetry will treat you like you're trying to publish a library when all you asked for is an environment, but a library's dependencies are pretty much an environment so there's nothing wrong with having an empty package and just using poetry as an environment factory.
Bonus: if you decide to publish a library after all, you're ready.
The Setup
nix flakes thinks in terms of git repo's, so we'll start with one:
$ git init
Then create a file called flake.nix. Usually I end up with poetry handling 90% of the python stuff, but both pandas and spacy are in that 10% that has dependencies which link to system libraries. So we ask nix to install them so that when poetry tries to install them in the nix develop shell, it has what it needs.
{
description = "Jonathan's awesome env";
inputs = {
nixpkgs.url = "github:nixos/nixpkgs";
};
outputs = { self, nixpkgs, flake-utils }: (flake-utils.lib.eachSystem [
"x86_64-linux"
"x86_64-darwin"
"aarch64-linux"
"aarch64-darwin"
] (system:
let
pkgs = nixpkgs.legacyPackages.${system};
in
rec {
packages.jonathansenv = pkgs.poetry2nix.mkPoetryApplication {
projectDir = ./.;
};
defaultPackage = packages.jonathansenv;
devShell = pkgs.mkShell {
buildInputs = [
pkgs.poetry
pkgs.python310Packages.pandas
pkgs.python310Packages.spacy
];
};
}));
}
Now we let git know about the flake and enter the environment:
❯ git add flake.nix
❯ nix develop
$
Then we initialize the poetry project. I've found that poetry, installed by nix, is kind of odd about which python it uses by default, so we'll set it explicitly
$ poetry init # follow prompts
$ poetry env use $(which python)
$ poetry run python --version
Python 3.10.9 # declared in the flake.nix
At this point, we should have a pyproject.toml:
[tool.poetry]
name = "jonathansenv"
version = "0.1.0"
description = ""
authors = ["Your Name <you#example.com>"]
readme = "README.md"
[tool.poetry.dependencies]
python = "^3.10"
jenkspy = "^0.3.2"
spacy = "^3.4.4"
pandas = "^1.5.2"
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
Usage
Now we create the venv that poetry will use, and run a command that depends on these.
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
You can also have poetry put you in a shell:
$ poetry shell
(venv)$ python -c 'import jenkspy, spacy, pandas'
It's kind of awkward to do so though, because we're two subshells deep and any shell customizations that we have the grandparent shell are not available. So I recommend using direnv, to enter the dev shell whenever I navigate to that directory and then just use poetry run ... to run commands in the environment.
Publishing the env
In addition to running nix develop with the flake.nix in your current dir, you can also do nix develop /local/path/to/repo or develop nix develop github:/githubuser/githubproject to achieve the same result.
To demonstrate the github example, I have pushed the files referenced above here. So you ought to be able to run this from any linux shell with nix installed:
❯ nix develop github:/MatrixManAtYrService/nix-flake-pandas-spacy
$ poetry install
$ poetry run python -c 'import jenkspy, spacy, pandas'
I say "ought" because if I run that command on a mac it complains about linux-headers-5.19.16 being unsupported on x86_64-darwin.
Presumably there's a way to write the flake (or fix a package) so that it doesn't insist on building linux stuff on a mac, but until I figure it out I'm afraid that this is only a partial answer.

How to run different tox command inside Jenkins environment?

I would like to use different command line arguments for py.test, depending on environment: running locally there should be only default ones, but on Jenkins I would like to add --junitxml=junit-{envname}.xml, so the test results could be published in nice format.
I know from documentation, that there is special [tox:jenkins] section, which should be used in case there is defined 'JENKINS_URL' or 'HUDSON_URL'. So now I created simple tox.ini file:
[tox]
envlist = py27, py35
[tox:jenkins]
commands = echo "We are in JENKINS!"
[testenv]
setenv =
PYTHONPATH = {toxinidir}:{toxinidir}/my_module
commands = python setup.py test
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/requirements_test.txt
And have defined JENKINS_URL environment variable:
export JENKINS_URL=true
I expect, that if I run tox, then my original command will be substituted with echo, but it doesn't work, instead I end with original command been executed.
Could someone help me with this problem?

Okay, found the solution myself:
[tox]
envlist = py27, py35
[testenv]
setenv =
PYTHONPATH = {toxinidir}:{toxinidir}/my_module
commands = py.test {env:CUSTOM_ARGS} ; <== use environment variable here
deps =
-r{toxinidir}/requirements.txt
-r{toxinidir}/requirements_test.txt
and in Jenkins just define CUSTOM_ARGS environment variable like this:
export CUSTOM_ARGS="--junitxml=junit.xml"

You could have used positional arguments.
Like this:
commands = py.test --mandatory-flag {posargs:--default-flag}
And in Jenkins you can call tox like this:
tox -- --override-flag
Note the separation with two dashes --.