I have been using tox to run the lintin packes over my code base. However I have ran into the issue of not having the dependecies up to date with my gitlab pipeline, because I was not updating my dependencies not to affect the deployed version. For this reason I wanted to switch to using requirements-dev.txt in my tox command. My current setup is as follow:
[tox]
envlist = py3, flake8, black, mypy, pylint
basepython=python3
[testenv]
deps =
-rrequirements.txt
-rrequirements-dev.txt
commands =
python -m pytest -v --cov={[vars]SOURCE} \
--cov-config=tox.ini \
--cov-report=html \
--cov-fail-under={[vars]MINIMAL_COVERAGE} \
--cov-report term-missing \
{[vars]TEST_DIR}
[testenv:pylint]
basepython = python3
deps =
-rrequirements.txt
-rrequirements-dev.txt
# ignored:
# R0903 - Too few public methods
# W3101 - missing-timeout
commands = python -m pylint {[vars]SOURCE} --rcfile=tox.ini -d R0903,W0511,W3101
# pylint configuration
[pylint.MAIN]
load-plugins=pylint_flask_sqlalchemy,pylint_flask
[pylint]
max-args = 10
[testenv:flake8]
basepython=python3
deps =
-rrequirements-dev.txt
commands= python3 -m flake8 --max-line-length=100 {[vars]SOURCE} {[vars]TEST_DIR}
whitelist_externals = /usr/bin/python3
# flake8 config
[flake8]
ignore = E722, W503, W504, N818, F841
[testenv:mypy]
deps =
-rrequirements-dev.txt
commands = mypy --install-types --non-interactive \
--ignore-missing-imports \
--disallow-untyped-defs \
--disallow-incomplete-defs \
--disallow-untyped-decorators {[vars]SOURCE} {[vars]TEST_DIR}
[testenv:black]
deps = -rrequirements-dev.txt
commands = black --check --diff {[vars]SOURCE} {[vars]TEST_DIR}
# Format code automatically using black rules
[testenv:black-format]
deps = -rrequirements-dev.txt
commands = black {[vars]SOURCE} {[vars]TEST_DIR}
As you can see, i've had to keep the requirements.txt in dependencies for testenv and pylint (I had some dependencies that I needed in both), however I am now getting a double requirement error such as ERROR: Double requirement given: click==8.1.3 (from -r requirements-dev.txt (line 13)) (already in click==8.0.3 (from -r requirements.txt (line 19)), name='click').
I am wondering what is the most elegant solution to this problem? Omitting those dependencies that are covered in requirements.txt in requirements-dev.txt? Or just keeping the version lower?
Thank you for your advice
The most elegant, and widely used solution is not to run each linter in a separate tox environment, but to have one linter environment, which runs pre-commit.
pre-commit is a linter runner and both takes care of running the linters and dependency management of the linters.
Your tox.ini would look like that:
[testenv:lint]
deps= pre-commit
commands = pre-commit run --all-files
More info on pre-commit
https://pre-commit.com/
Example configuration from one of my projects
https://github.com/jugmac00/flask-reuploaded/blob/723fe4e355cd260bc82bf4f1c712036ae3d3d4b6/tox.ini#L24
https://github.com/jugmac00/flask-reuploaded/blob/master/.pre-commit-config.yaml
Disclaimer: I am one of the tox maintainers
Related
I want to use a pre-release version of a package (https://test.pypi.org/project/delta-spark/2.1.0rc1/) in my project.
I'm using poetry to manage my pyproject.toml. How do I do this?
In other words what is the poetry equivalent of:
pip install -i https://test.pypi.org/simple/ delta-spark==2.1.0rc1
I tried:
poetry add delta-spark==2.1.0rc1
poetry add --allow-prereleases delta-spark==2.1.0rc1
Both give: Could not find a matching version of package delta-spark
$ poetry config --local repositories.test-pypi https://test.pypi.org/
$ poetry config --list | fgrep repositories
repositories.test.url = "https://test.pypi.org/"
repositories.test-pypi.url = "https://test.pypi.org/"
$ fgrep -A 3 tool.poetry.source pyproject.toml
[[tool.poetry.source]]
name = "test-pypi"
url = "https://test.pypi.org/"
secondary = true
$ poetry add --group dev delta-spark==2.1.0rc1
Could not find a matching version of package delta-spark
$
This is described here. Basically, you can add the repository via:
poetry config repositories.test https://test.pypi.org/simple/
and then make it available in pyproject.toml via:
[[tool.poetry.source]]
name = "test"
url = "https://test.pypi.org/simple/"
secondary = true
Then adding the dependency should work.
Question
How to use tox to install a the local package without it refering to a random time when I installed the pacakge from GitHub?
Context
In this PR, I'm trying to build and test my package, dbt-synapse, before publishing a new minor version to PyPI. At some point (rookie move), I think I tried installing the package directly from a specific GitHub commit with the below line.
pip install -e git+https://github.com/dbt-msft/dbt-synapse.git#345d7cafcb08bac25d23867a2e22c0e9b741f603#egg=dbt_synapse
Here's a gist of the requirements.txt, tox.ini, setup.py and some other files
When I call tox -e integration-synapse to kick off the tests, having listing the actual package in the deps as -e. compared to just ., causes the package to be installed from GitHub (see the integration-synapse installed: line below, shortened for clarity)!
when deps has just ., however it lists this as having been installed (which also doesn't seem right): dbt-synapse # file:///Users/anders.swanson/repos/dbt-synapse.
Unfortunately, neither are giving me what I know should be happening.
integration-synapse recreate: /Users/anders.swanson/repos/dbt-synapse/.tox/integration-synapse
integration-synapse installdeps: -rrequirements.txt, -e.
integration-synapse installed: [...],-e git+https://github.com/dbt-msft/dbt-synapse.git#345d7cafcb08bac25d23867a2e22c0e9b741f603#egg=dbt_synapse,[...]
integration-synapse runtests: PYTHONHASHSEED='1932234374'
integration-synapse runtests: commands[0] | /bin/bash -c '/Users/anders.swanson/repos/dbt-synapse/.tox/integration-synapse/bin/python -m pytest -v test/integration/synapse.dbtspec'
======================================================================================================================================== test session starts =========================================================================================================================================
platform darwin -- Python 3.6.12, pytest-6.2.1, py-1.10.0, pluggy-0.13.1 -- /Users/anders.swanson/repos/dbt-synapse/.tox/integration-synapse/bin/python
cachedir: .pytest_cache
rootdir: /Users/anders.swanson/repos/dbt-synapse
plugins: dbt-adapter-0.4.0
collected 1 item
tox.ini
[tox]
skipsdist = True
envlist = integration-synapse
[testenv:integration-synapse]
basepython = python3
commands = /bin/bash -c '{envpython} -m pytest -v test/integration/synapse.dbtspec'
passenv = DBT_SYNAPSE_DB DBT_SYNAPSE_PORT DBT_SYNAPSE_PWD DBT_SYNAPSE_SERVER DBT_SYNAPSE_UID
deps =
-rrequirements.txt
-e.
If you want to start from a clean state, you can you can delete the hidden .tox folder and start from scratch.
Also, you can run tox -r which will recreate the environemnts.
In order to test the code from your local development environment, you do not have to include your package in the deps list (-e.). tox will build the package from your setup.py.
Well.. it would. But in your tox.ini you specified skipsdist = True which afaik means tox should not build the package.
There are some more issues in your tox.ini, e.g. unit and flake8 envs do nothing.
Why do you use the /bin/bash -c construct?
Unfortunately, I am missing some header files, so I cannot run your tox.ini file, but I would create a tox.ini file like this...
[tox]
envlist = py36,py37,py38,py39,integration-synapse,flake8
[testenv]
deps = whatever needed for your unit tests
commands = python -m pytest ...unit tests...
[testenv:integration-synapse]
deps =
pytest
pytest-dbt-adapter
commands =
python -m pytest -v test/integration/synapse.dbtspec
passenv = DBT_SYNAPSE_DB DBT_SYNAPSE_PORT DBT_SYNAPSE_PWD DBT_SYNAPSE_SERVER DBT_SYNAPSE_UID
[testenv:flake8]
skipsdist = True
deps = flake8
commands = flake8 setup.py ... (and more)
Attempting to add shortcuts such as cov = coverage report --show-missing to the [aliases] section of setup.cfg causes python setup.py cov to fail with error: invalid command 'coverage'. What additional options are needed to support this?
Version info:
$ python -V
Python 3.7.3
$ pip list | egrep 'setuptools|pip|coverage|pytest'
coverage 5.2.1
pip 20.2.2
pytest 5.4.3
pytest-html 2.1.1
pytest-metadata 1.10.0
setuptools 50.0.0
setup.cfg:
[aliases]
test = coverage run -m pytest --html=pytest-report.html --self-contained-html
cov = coverage report --show-missing
covhtml = coverage html --title "Coverage report - xyz"
[tool:pytest]
testpaths = tests
[coverage:run]
branch = True
source = pyapp
[options]
# ...
install_requires =
Flask==1.1.*
psycopg2==2.8.*
requests==2.*
werkzeug==1.*
click==7.*
setup_requires =
pytest-runner
tests_require =
coverage
pytest
pytest-html
setup.cfg aliases creates aliases for setup.py, not general commands. In your case python setup.py cov is equivalent to
python setup.py coverage report --show-missing
and setup.py complains it doesn't have a command coverage.
In my tox.ini file, the dependencies are installed via the requirements.txt file which is also used by setup.py, as follows:
The requirements.txt file contains the acceptable range of django packages, depending on the python version installed, as follows:
Django>=1.11,<2 ; python_version == '2.7'
Django>=1.11,<3 ; python_version > '3'
For python3, I want to make sure the tests run on django 2.0 as well as the latest django 2.1+ that will be installed by default, obeying the version constraints specified in the requirements.txt file. To achieve that, I force the installation of the desired django version with commands, as follows:
[tox]
envlist = {py27,py3}-django111,py3-django{20,21}
[testenv]
deps =
-r{toxinidir}/requirements.txt
commands =
django111: pip install 'Django>=1.11,<1.12'
py3-django20: pip install 'Django>=2.0,<2.1'
py3-django21: pip install 'Django>=2.1'
pytest
Ideally I could just add to the deps variable like so:
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
But pip does not support double requirements and will throw an error even though there is no conflict in how the version constraints are specified.
The drawback of using commands to override the installation is that it needs to remove the django package version installed via requirements.txt to install the desired one. Is there a way to avoid that extra step?
One trick is to move the requirement from requirements.txt into setup.py - where it's loosely pinned so that all your django versions are possible. For example
# setup.py
from setuptools import setup, find_packages
setup(
...
install_requires=[
"Django>=1.11,<2.1",
]
)
and then use your second suggestion in tox.ini
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
... so long as the Django requirement isn't listed in requirements.txt.
This works because the pip install is split in two parts, the first from tox:deps where you specify the hard requirement, and the second from the equivalent of pip install -e . where the setup.py has the looser requirement.
i just committed a seemingly uninteresting commit, updating the release notes and setup for pypi. the travis-ci build fails however running tox with py26, py33 and pypy:
https://travis-ci.org/Turbo87/aerofiles
1.13s$ tox -e $TOX_ENV -- --cov aerofiles --cov-report term-missing
py26 create: /home/travis/build/Turbo87/aerofiles/.tox/py26
ERROR: InterpreterNotFound: python2.6
i didn't change anything to the travis.yml and tox has been fixed on the 1.7.2 version:
language: python
python: 2.7
sudo: false
env:
- TOX_ENV=py26
- TOX_ENV=py27
- TOX_ENV=py33
- TOX_ENV=py34
- TOX_ENV=pypy
install:
# Install tox and flake8 style checker
- pip install tox==1.7.2 flake8==2.1.0
script:
# Run the library through flake8
- flake8 --exclude=".git,docs" --ignore=E501 .
# Run the unit test suite
- tox -e $TOX_ENV -- --cov aerofiles --cov-report term-missing
Would be great if someone could help out. I am quite new to travis-ci (and tox) and it's quite a black box at the moment.
A few week ago I was forced to change all my .travis.yml exactly because of the problem. See my commit. Instead of
env:
- TOXENV=py27
- TOXENV=py34
write
matrix:
include:
- python: "2.7"
env: TOXENV=py27
- python: "3.4"
env: TOXENV=py34