I have been using tox to run the lintin packes over my code base. However I have ran into the issue of not having the dependecies up to date with my gitlab pipeline, because I was not updating my dependencies not to affect the deployed version. For this reason I wanted to switch to using requirements-dev.txt in my tox command. My current setup is as follow:
[tox]
envlist = py3, flake8, black, mypy, pylint
basepython=python3
[testenv]
deps =
-rrequirements.txt
-rrequirements-dev.txt
commands =
python -m pytest -v --cov={[vars]SOURCE} \
--cov-config=tox.ini \
--cov-report=html \
--cov-fail-under={[vars]MINIMAL_COVERAGE} \
--cov-report term-missing \
{[vars]TEST_DIR}
[testenv:pylint]
basepython = python3
deps =
-rrequirements.txt
-rrequirements-dev.txt
# ignored:
# R0903 - Too few public methods
# W3101 - missing-timeout
commands = python -m pylint {[vars]SOURCE} --rcfile=tox.ini -d R0903,W0511,W3101
# pylint configuration
[pylint.MAIN]
load-plugins=pylint_flask_sqlalchemy,pylint_flask
[pylint]
max-args = 10
[testenv:flake8]
basepython=python3
deps =
-rrequirements-dev.txt
commands= python3 -m flake8 --max-line-length=100 {[vars]SOURCE} {[vars]TEST_DIR}
whitelist_externals = /usr/bin/python3
# flake8 config
[flake8]
ignore = E722, W503, W504, N818, F841
[testenv:mypy]
deps =
-rrequirements-dev.txt
commands = mypy --install-types --non-interactive \
--ignore-missing-imports \
--disallow-untyped-defs \
--disallow-incomplete-defs \
--disallow-untyped-decorators {[vars]SOURCE} {[vars]TEST_DIR}
[testenv:black]
deps = -rrequirements-dev.txt
commands = black --check --diff {[vars]SOURCE} {[vars]TEST_DIR}
# Format code automatically using black rules
[testenv:black-format]
deps = -rrequirements-dev.txt
commands = black {[vars]SOURCE} {[vars]TEST_DIR}
As you can see, i've had to keep the requirements.txt in dependencies for testenv and pylint (I had some dependencies that I needed in both), however I am now getting a double requirement error such as ERROR: Double requirement given: click==8.1.3 (from -r requirements-dev.txt (line 13)) (already in click==8.0.3 (from -r requirements.txt (line 19)), name='click').
I am wondering what is the most elegant solution to this problem? Omitting those dependencies that are covered in requirements.txt in requirements-dev.txt? Or just keeping the version lower?
Thank you for your advice
The most elegant, and widely used solution is not to run each linter in a separate tox environment, but to have one linter environment, which runs pre-commit.
pre-commit is a linter runner and both takes care of running the linters and dependency management of the linters.
Your tox.ini would look like that:
[testenv:lint]
deps= pre-commit
commands = pre-commit run --all-files
More info on pre-commit
https://pre-commit.com/
Example configuration from one of my projects
https://github.com/jugmac00/flask-reuploaded/blob/723fe4e355cd260bc82bf4f1c712036ae3d3d4b6/tox.ini#L24
https://github.com/jugmac00/flask-reuploaded/blob/master/.pre-commit-config.yaml
Disclaimer: I am one of the tox maintainers
After I implemented my test, which is using pandas, my build is failing with "ModuleNotFoundError: No module named 'pandas'" error, however, I added pandas to the testenv deps and in the log file I also see that it is installed. I got the same error in case of boto3 but after I added it to the deps, it solved the problem but in case of pandas it doesn't work.
tox.ini
[tox]
envlist=flake8
py36_tests
[testenv]
deps=pytest
flake8
[testenv:py36_tests]
basepython=python3.6
deps=boto3
pandas
commands=py.test -s -v tests --junitxml=report.xml
[testenv:flake8]
commands=flake8 --exclude=.git,__pycache__,__init__.py data_collector/
flake8 tests/
flake8 setup.py
flake8 setup-cy.py
flake8 Docker/startup_scripts/
flake8 bin/data_collector
Log:
$ tox -e py36_tests
GLOB sdist-make: /builds/<path>/<my_package>/setup.py
py36_tests create: /builds/<path>/<my_package>/.tox/py36_tests
py36_tests installdeps: boto3, pandas
py36_tests inst: /builds/<path>/.tox/.tmp/package/1/<my_package>-0.0.0.zip
py36_tests installed: <many packages>, pandas==1.1.1, <many packages>
y36_tests run-test-pre: PYTHONHASHSEED='3745093701'
py36_tests run-test: commands[0] | py.test -s -v tests --junitxml=report.xml
WARNING: test command found but not installed in testenv
cmd: /usr/local/bin/py.test
env: /builds/<path>/<my_package>/.tox/py36_tests
Maybe you forgot to specify a dependency? See also the allowlist_externals envconfig setting.
DEPRECATION WARNING: this will be an error in tox 4 and above!
============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /usr/bin/python3.6
cachedir: .tox/py36_tests/.pytest_cache
rootdir: /builds/<path>/<my_package>
collecting ... collected 10 items / 1 error / 9 selected
==================================== ERRORS ====================================
________________ ERROR collecting tests/ingestion/test_task.py _________________
ImportError while importing test module '/builds/<path>/tests/ingestion/test_task.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.6/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/ingestion/test_task.py:3: in <module>
import pandas as pd
E ModuleNotFoundError: No module named 'pandas'
- generated xml file: /builds/<path>/<my_package>/report.xml -
=========================== short test summary info ============================
ERROR tests/ingestion/test_task.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.24s ===============================
ERROR: InvocationError for command /usr/local/bin/py.test -s -v tests --junitxml=report.xml (exited with code 2)
___________________________________ summary ____________________________________
ERROR: py36_tests: commands failed
While I hadn't got any import in my test it worked well. What is the problem?
See this
WARNING: test command found but not installed in testenv
cmd: /usr/local/bin/py.test
You have not installed py.test into the virtual environment so tox found a global one. And of course the global one runs with a global Python and doesn't know anything about your virtual environments in tox.
To fix it install pytest into the virtual environment:
[testenv:py36_tests]
deps=boto3
pandas
pytest
I used a virtual env after the tox call, the solution was to move the tox call when the virtual env was already activated.
Attempting to add shortcuts such as cov = coverage report --show-missing to the [aliases] section of setup.cfg causes python setup.py cov to fail with error: invalid command 'coverage'. What additional options are needed to support this?
Version info:
$ python -V
Python 3.7.3
$ pip list | egrep 'setuptools|pip|coverage|pytest'
coverage 5.2.1
pip 20.2.2
pytest 5.4.3
pytest-html 2.1.1
pytest-metadata 1.10.0
setuptools 50.0.0
setup.cfg:
[aliases]
test = coverage run -m pytest --html=pytest-report.html --self-contained-html
cov = coverage report --show-missing
covhtml = coverage html --title "Coverage report - xyz"
[tool:pytest]
testpaths = tests
[coverage:run]
branch = True
source = pyapp
[options]
# ...
install_requires =
Flask==1.1.*
psycopg2==2.8.*
requests==2.*
werkzeug==1.*
click==7.*
setup_requires =
pytest-runner
tests_require =
coverage
pytest
pytest-html
setup.cfg aliases creates aliases for setup.py, not general commands. In your case python setup.py cov is equivalent to
python setup.py coverage report --show-missing
and setup.py complains it doesn't have a command coverage.
In my tox.ini file, the dependencies are installed via the requirements.txt file which is also used by setup.py, as follows:
The requirements.txt file contains the acceptable range of django packages, depending on the python version installed, as follows:
Django>=1.11,<2 ; python_version == '2.7'
Django>=1.11,<3 ; python_version > '3'
For python3, I want to make sure the tests run on django 2.0 as well as the latest django 2.1+ that will be installed by default, obeying the version constraints specified in the requirements.txt file. To achieve that, I force the installation of the desired django version with commands, as follows:
[tox]
envlist = {py27,py3}-django111,py3-django{20,21}
[testenv]
deps =
-r{toxinidir}/requirements.txt
commands =
django111: pip install 'Django>=1.11,<1.12'
py3-django20: pip install 'Django>=2.0,<2.1'
py3-django21: pip install 'Django>=2.1'
pytest
Ideally I could just add to the deps variable like so:
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
But pip does not support double requirements and will throw an error even though there is no conflict in how the version constraints are specified.
The drawback of using commands to override the installation is that it needs to remove the django package version installed via requirements.txt to install the desired one. Is there a way to avoid that extra step?
One trick is to move the requirement from requirements.txt into setup.py - where it's loosely pinned so that all your django versions are possible. For example
# setup.py
from setuptools import setup, find_packages
setup(
...
install_requires=[
"Django>=1.11,<2.1",
]
)
and then use your second suggestion in tox.ini
[testenv]
deps =
-r{toxinidir}/requirements.txt
django111: Django>=1.11,<1.12
py3-django20: Django>=2.0,<2.1
py3-django21: Django>=2.1
commands =
pytest
... so long as the Django requirement isn't listed in requirements.txt.
This works because the pip install is split in two parts, the first from tox:deps where you specify the hard requirement, and the second from the equivalent of pip install -e . where the setup.py has the looser requirement.
subtitle: Not only sdist
I am trying to get the setup.py file of a package I'm working on to play nicely with sdist. The relevant parts of the setup.py file are:
from setuptools.command.test import test
[...]
class Tox(test):
"as described in
http://tox.readthedocs.org/en/latest/example/basic.html?highlight=setuptools#integration-with-setuptools-distribute-test-commands"
[...]
def run_tests(self):
if self.distribution.install_requires:
self.distribution.fetch_build_eggs(
self.distribution.install_requires)
if self.distribution.tox_requires:
self.distribution.fetch_build_eggs(self.distribution.tox_requires)
# import here, cause outside the eggs aren't loaded
import tox
import shlex
args = self.tox_args
if args:
args = shlex.split(self.tox_args)
else:
args = ""
errno = tox.cmdline(args=args)
sys.exit(errno)
entry_points ={}
distutils_ext = {'distutils.setup_keywords': [
"tox_requires = setuptools.dist:check_requirements", ]
}
entry_points.update(distutils_ext)
setup(
install_requires=['six', 'numpy', 'matplotlib', 'scipy', 'astropy>=1',
'Pillow', ],
cmdclass={
'test': PyTest, # this is to run python setup.py test
'tox': Tox,
},
# list of packages and data
packages=find_packages(),
# tests
tests_require=['pytest', 'pytest-cov'],
tox_requires=['tox'],
# other keywords, mostly metadata
)
If I run python setup.py sdist, I get a warning at the beginning:
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tox_requires'
warnings.warn(msg)
but then the sdist works fine and it creates a tar.gz file that I can use to install my package.
But if I run it a second time, it starts with (it's the beginning of Pillow building):
warning: no previously-included files found matching '.editorconfig'
Building using 4 processes
_imaging.c: In function ‘getink’:
and begins building all the required packages into .eggs directory.
If I remove the *egg-info directory I can rerun the command.
If I comment out the tox_requires=[...] line, I can build the sdist as many times as I want.
Now according to the setuptools documentation the command above should be the correct way to run add new arguments to the setup function.
As per the subtitle, the problem is not only with sdist but it's probably due to my non-understanding on how setuptools and requirements work.
If I run python setup.py tox in a place without tox already installed I get, after the installation of some testing package it should not install (namely pytest and pytest-cov):
Traceback (most recent call last):
[...]
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path
func()
File "setup.py", line 65, in run_tests
if self.distribution.tox_requires:
AttributeError: Distribution instance has no attribute 'tox_requires'
[Update] The tox_requires also confuse very badly pip during installation. If it is commented out I can install the package without any issue; otherwise it begins to compile the source of the packages and it systematically fails because it doesn't find numpy while building stuff like scipy
How can I get setuptools to recognize and properly use tox_requires?
Once this issue is fixed, I think that I can get rid of the spurious installations here doing a better job at implementing the Tox class, maybe overriding more things from test or deriving it directly from Command
Complete (working) solution described below consist of 8 files (incl. short
README.rst) and has in total 43 lines of code. This is less then code in your
original question.
Despite of being so short, it supports many development and testing scenarios
in very convenient way.
Anyway, it does not answer exactly your question, but I am sure, it fulfils the
requirements which were behind it.
Three lines long setup.py
Technically it may be possible to put test command including tox automation into your setup.py, however, the result may be very messy and difficult to understand.
The same result can be achieved in much simpler way:
for developer assume:
using git
having tox installed into system
for package user:
there are no special requirements to install the resulting package
(optional) if you want your users to test the package by single command and keep test reports collected in central server:
install devpi-server and give your users access to it
ask your users to install $ pip install devpi
The solution builds on following tools and packages:
pbr: simplify package creation incl. versioning via git tags and
creation of AUTHORS and ChangeLog from git commit messages.
pytest: excelent testing framework, but any other framework can
be used instead of it.
tox: excellent build and test automation tool.
coverage: tools to measure test coverage (working simpler then
pytest-cov)
Optionally you may also use:
devpi-server: private PyPi server with password protected
access. Allows simple testing and provides test reports collection.
devpi: tool similar to pip. Apart from installation also supports
running tox defined tests (install, run tests, publish reports in on step).
Authoring the package
Create new project directory and initialize git:
$ mkdir francesco
$ cd francesco
$ git init
Create a package or module
Here we create single module francesco, but the same works for more
modules or packages.
francesco.py
def main():
print("Hi, it is me, Francesco, keeping things simple.")
requirements.txt
Create list of packages for actual installation of the package:
six
test_requirements.txt
Define packages required for testing:
pytest
coverage
tests/test_it.py
Initiate the test suite:
from francesco import main
def test_this():
main()
print("All seems fine to me")
assert True
setup.py
Did you dream of stupid simple setup.py? Here it goes:
from setuptools import setup
setup(setup_requires=["pbr"], pbr=True)
setup.cfg
Metadata belong to configuration file:
[metadata]
name = francesco
author = Francesco Montesano
author-email = fm#acme.com
summary = Nice and simply installed python module supporting testing in different pythons
description-file = README.rst
[files]
modules=francesco
[entry_points]
console_scripts =
francesco = francesco:main
tox.ini
To configure tox automated builds and tests:
[tox]
envlist = py27, py34
[testenv]
commands =
coverage run --source francesco -m pytest -sv tests
coverage report
coverage html
deps =
-rtest_requirements.txt
README.rst
Never forget README.rst:
===========================================
Complex package with 3 line long `setup.py`
===========================================
Can we keep`setup.py` simple and still support automated testing?
...
tox: build sdist and run tests in all supported python versions
Being in the project directory root, just run single command tox:
$ tox
GLOB sdist-make: /home/javl/sandbox/setuppy/setup.py
py27 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py27 runtests: PYTHONHASHSEED='2409409075'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py27 runtests: commands[2] | coverage html
py34 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py34 runtests: PYTHONHASHSEED='2409409075'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py34 runtests: commands[2] | coverage html
___________________________________ summary ____________________________________
py27: commands succeeded
py34: commands succeeded
congratulations :)
Getting the sdist
ls .tox/dist
francesco-0.0.0.zip
Developing in Python 2.7 virtualenv
Activate Python 2.7 virtualenv
$ source .tox/py27/bin/activate
Run tests
(py27) $ py.test -sv tests
==============================================================================================
test session starts
===============================================================================================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
-- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7 cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile: collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things
simple. All seems fine to me PASSED
============================================================================================
1 passed in 0.01 seconds
============================================================================================
Measure test coverage
(py27)$ coverage run --source francesco -m pytest -sv tests
.....
(py27)$ coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
View coverage report in web browser
(py27)$ coverage html
(py27)$ firefox htmlcov/index.html
Release new package version
(optional) Install local devpi-server
The installation of devpi-server is not covered here, but is very
simple, especially, if you install only to your local machine for your
personal testing.
Commit source code, assign version tag
Make sure, all your source code is commited.
Assing version tag:
$ git tag -a 0.1
Rerun the tests by tox and build sdist
Make sure, you have deactivated virtualenv (otherwise it conflicts with
tox):
(py27)$ deactivate
Run the tox:
$ tox
.....
...it builds as usual, may fail, if you have forgotten to commit some changes or files...
Find the sdist for new version of your package:
$ ls .tox/dist/francesco-0.1.0.
.tox/dist/francesco-0.1.0.zip
You are done. You may distribute your new tested versions of your package
to users as usually.
(optional) Upload the sdist to devpi-server and test it locally
Following steps assume, you have devpi-server installed and running.
$ devpi login javl
...enter your password...
$ devpi upload .tox/dist/francesco-0.1.0.zip
Test the package in clean environment
(deactivate virtualenv if active) :
$ cd /tmp
$ mkdir testing
$ cd testing
$ devpi test francesco
received http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
unpacking /tmp/devpi-test6/downloads/francesco-0.1.0.zip to /tmp/devpi-test6/zip
/tmp/devpi-test6/zip/francesco-0.1.0$ tox --installpkg /tmp/devpi-test6/downloads/francesco-0.1.0.zip -i ALL=http://localhost:3141/javl/dev/+simple/ --recreate --result-json /tmp/devpi-test6/zip/toxreport.json
-c /tmp/devpi-test6/zip/francesco-0.1.0/tox.ini
py27 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27
py27 installdeps: -rtest_requirements.txt
py27 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py27 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='3916044270'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py27 runtests: commands[2] | coverage html
py34 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34
py34 installdeps: -rtest_requirements.txt
py34 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py34 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py34 runtests: PYTHONHASHSEED='3916044270'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py34 runtests: commands[2] | coverage html
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
py27: commands succeeded
py34: commands succeeded
congratulations :)
wrote json report at: /tmp/devpi-test6/zip/toxreport.json
posting tox result data to http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
successfully posted tox result data
You may check the test results in web browser:
$ firefox http://localhost:3141
then search for "francesco" package, click the package name, find in table
column named "tox results", click there to show environment set up and test
results.
Let your users test the package
Let's assume, your devpi-server is running and your user has access to it.
The user shall have devpi command installed:
$ pip install devpi
(note, this tool is not installing anything from the devpi-server)
Help your user to gain access to devpi-server (not covering here).
Then the user just runs the test:
$ devpi test francesco
After the test is run (it is automatically using tox, but user does not have
to care about that), you will find test results on the same place on devpi web
interface as you found yours before.