I'm developing a custom layer for the Keras framework and would like to extend their existing testsuite.
Since I'm working on the keras sourcecode, I have installed it like this:
pip install keras-preprocessing
pip install keras-applications
git clone https://github.com/keras-team/keras
cd keras
export PYTHONPATH=$PWD:$PYTHONPATH
After these commands, you are in the cloned keras repository, this is the working directory for the code samples below.
To begin with, I would like to run the existing tests and see what they do. It looks like they can simply be run as a python file:
if __name__ == '__main__':
pytest.main([__file__])
But this:
python tests/keras/layers/wrappers_test.py
produces the following output, and doesn't run any tests
Using TensorFlow backend.
usage: wrappers_test.py [options] [file_or_dir] [file_or_dir] [...]
wrappers_test.py: error: unrecognized arguments: -n tests/keras/layers/wrappers_test.py
inifile: /home/lhk/programming/keras/pytest.ini
rootdir: /home/lhk/programming/keras
So I tried explicitly calling pytest next:
pytest tests/keras/layers/wrappers_test.py
Exactly the same response (without the tensorflow log):
usage: pytest [options] [file_or_dir] [file_or_dir] [...]
pytest: error: unrecognized arguments: -n tests/keras/layers/wrappers_test.py
inifile: /home/lhk/programming/keras/pytest.ini
rootdir: /home/lhk/programming/keras
I'm using PyCharm, and if I run the tests from within pycharm (which I admittedly tried first), it only produces the messages above.
Apparently, I'm not configuring it correctly. Pytest is not able to pick up the test suite. To find a reference configuration, I looked at the Keras CI setup. They use Travis and the config is opensource: https://travis-ci.org/keras-team/keras/jobs/442252422/config
It looks like I have all the dependencies installed. And the actual test command is basically what I have already tried:
PYTHONPATH=$PWD:$PYTHONPATH py.test tests/ --ignore=tests/integration_tests --ignore=tests/test_documentation.py --ignore=tests/keras/legacy/layers_test.py --cov-config .coveragerc --cov=keras tests/
It produces the exact same output as above.
I assume that the problem lies with pytest. But the installation section of their automated testing simply shows pip install pytest pytest-pip8. I ran this, but sure enough, requirement already satisfied.
How can I execute the keras pytests.
I'm running ubuntu 18.04.1, python 3.6.5 with anaconda 64bit.
Running the keras tests is mentioned on the Contributing page:
Run our test suite locally. It's easy: from the Keras folder, simply run: py.test tests/.
You will need to install the test requirements as well: pip install -e .[tests].
If you don't want to install the package in editable mode, just install all of the test dependencies explicitly. Looking at keras' setup script, the command for that would be:
$ pip install pytest pytest-pep8 pytest-xdist pytest-cov pytest-timeout pandas requests
I used to run the tests for my packages from the Makefile as a way to perform three tasks in one: setup a virtual environment, install requirements and call the testing suite with the corresponding arguments:
test: venv
env/bin/pip install -r test_requirements.txt
env/bin/nosetests --rednose --with-coverage --cover-pagacke my_module
Then I read that requirements.txt files are discouraged in favor of setup.py, so I modified the setup.py file aiming to get the same result:
setup(
...
tests_require=['nose', 'rednose', 'coverage'],
test_suite=['nose.collector'])
Now I could change the Makefile with
test: venv
coverage run --source=my_module/ setup.py test
But that requires installing the testing dependencies before running the setup.py file. I'm also not sure how to include other arguments such as rednose. What is the best way to do this?
Tox is good and all, but here's how to do it without having to install any other package beforehand.
List the testing dependencies as setup_requires instead of tests_require in the setup.py file
setup(
setup_requires=['nose', 'rednose', 'coverage'],
install_requires=[] # fill in other arguments as usual
)
Optionally add test parameters to the setup.cfg file.
[nosetests]
rednose=1
detailed-errors=1
with-coverage=1
cover-package=server
cover-xml=1
verbosity=2
Run the tests with the following command
python setup.py nosetests
Source: http://nose.readthedocs.io/en/latest/api/commands.html
This is the tree structure of the module I'm writing the setup.py file for:
ls .
LICENSE
README.md
bin
examples
module
scratch
setup.py
tests
tox.ini
I configured my setup.py as follows:
from setuptools import setup, find_packages
setup(
name="package_name",
version="0.1",
packages=find_packages(),
install_requires=[
# [...]
],
extras_require={
# [...]
},
tests_require={
'pytest',
'doctest'
},
scripts=['bin/bootstrap'],
data_files=[
('license', ['LICENSE']),
],
# [...]
# could also include long_description, download_url, classifiers, etc.
)
If I install the package from my python environment (also a virtualenv)
pip install .
the LICENSE file gets correctly installed.
But running tox:
[tox]
envlist = py27, py35
[testenv]
deps =
pytest
git+https://github.com/djc/couchdb-python
docopt
commands = py.test \
{posargs}
I get this error:
running install_data
creating build/bdist.macosx-10.11-x86_64/wheel/leafline-0.1.data
creating build/bdist.macosx-10.11-x86_64/wheel/leafline-0.1.data/data
creating build/bdist.macosx-10.11-x86_64/wheel/leafline-0.1.data/data/license
error: can't copy 'LICENSE': doesn't exist or not a regular file
Removing the data_files part from the setup.py makes tox running correctly.
Your issue here is that setuptools is not able to find the 'LICENSE' file in the files that have been included for building the source distribution. You have 2 options, to tell setuptools to include that file (both have been pointed to here):
Add a MANIFEST.in file (like https://github.com/pypa/sampleproject/)
Use include_package_data=True in your setup.py file.
Using MANIFEST.in is often simpler and easier to verify due to https://pypi.org/project/check-manifest/, making it possible to use automation to verify that things are indeed correct (if you use a VCS like Git or SVN).
pip install . builds a wheel using python setup.py bdist_wheel which is installed by simply unpacking it appropriately, as defined in the Wheel Specification: https://www.python.org/dev/peps/pep-0427/
tox builds a source distribution using python setup.py sdist, which is then unpacked and installed using python setup.py install.
That might be a reason for the difference in behavior for you.
I have some resource files inside my packages which I use during the execution. To make setup store them in a package with python code, I use include_package_data=True and I access them using importlib.resources. You can use backport for an older Python version than 3.7 or another library.
Before each release I have a script which verifies, that all files I need are placed inside a bdist wheel to be sure that everything is on the place.
subtitle: Not only sdist
I am trying to get the setup.py file of a package I'm working on to play nicely with sdist. The relevant parts of the setup.py file are:
from setuptools.command.test import test
[...]
class Tox(test):
"as described in
http://tox.readthedocs.org/en/latest/example/basic.html?highlight=setuptools#integration-with-setuptools-distribute-test-commands"
[...]
def run_tests(self):
if self.distribution.install_requires:
self.distribution.fetch_build_eggs(
self.distribution.install_requires)
if self.distribution.tox_requires:
self.distribution.fetch_build_eggs(self.distribution.tox_requires)
# import here, cause outside the eggs aren't loaded
import tox
import shlex
args = self.tox_args
if args:
args = shlex.split(self.tox_args)
else:
args = ""
errno = tox.cmdline(args=args)
sys.exit(errno)
entry_points ={}
distutils_ext = {'distutils.setup_keywords': [
"tox_requires = setuptools.dist:check_requirements", ]
}
entry_points.update(distutils_ext)
setup(
install_requires=['six', 'numpy', 'matplotlib', 'scipy', 'astropy>=1',
'Pillow', ],
cmdclass={
'test': PyTest, # this is to run python setup.py test
'tox': Tox,
},
# list of packages and data
packages=find_packages(),
# tests
tests_require=['pytest', 'pytest-cov'],
tox_requires=['tox'],
# other keywords, mostly metadata
)
If I run python setup.py sdist, I get a warning at the beginning:
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'tox_requires'
warnings.warn(msg)
but then the sdist works fine and it creates a tar.gz file that I can use to install my package.
But if I run it a second time, it starts with (it's the beginning of Pillow building):
warning: no previously-included files found matching '.editorconfig'
Building using 4 processes
_imaging.c: In function ‘getink’:
and begins building all the required packages into .eggs directory.
If I remove the *egg-info directory I can rerun the command.
If I comment out the tox_requires=[...] line, I can build the sdist as many times as I want.
Now according to the setuptools documentation the command above should be the correct way to run add new arguments to the setup function.
As per the subtitle, the problem is not only with sdist but it's probably due to my non-understanding on how setuptools and requirements work.
If I run python setup.py tox in a place without tox already installed I get, after the installation of some testing package it should not install (namely pytest and pytest-cov):
Traceback (most recent call last):
[...]
File "/usr/lib/python2.7/dist-packages/setuptools/command/test.py", line 127, in with_project_on_sys_path
func()
File "setup.py", line 65, in run_tests
if self.distribution.tox_requires:
AttributeError: Distribution instance has no attribute 'tox_requires'
[Update] The tox_requires also confuse very badly pip during installation. If it is commented out I can install the package without any issue; otherwise it begins to compile the source of the packages and it systematically fails because it doesn't find numpy while building stuff like scipy
How can I get setuptools to recognize and properly use tox_requires?
Once this issue is fixed, I think that I can get rid of the spurious installations here doing a better job at implementing the Tox class, maybe overriding more things from test or deriving it directly from Command
Complete (working) solution described below consist of 8 files (incl. short
README.rst) and has in total 43 lines of code. This is less then code in your
original question.
Despite of being so short, it supports many development and testing scenarios
in very convenient way.
Anyway, it does not answer exactly your question, but I am sure, it fulfils the
requirements which were behind it.
Three lines long setup.py
Technically it may be possible to put test command including tox automation into your setup.py, however, the result may be very messy and difficult to understand.
The same result can be achieved in much simpler way:
for developer assume:
using git
having tox installed into system
for package user:
there are no special requirements to install the resulting package
(optional) if you want your users to test the package by single command and keep test reports collected in central server:
install devpi-server and give your users access to it
ask your users to install $ pip install devpi
The solution builds on following tools and packages:
pbr: simplify package creation incl. versioning via git tags and
creation of AUTHORS and ChangeLog from git commit messages.
pytest: excelent testing framework, but any other framework can
be used instead of it.
tox: excellent build and test automation tool.
coverage: tools to measure test coverage (working simpler then
pytest-cov)
Optionally you may also use:
devpi-server: private PyPi server with password protected
access. Allows simple testing and provides test reports collection.
devpi: tool similar to pip. Apart from installation also supports
running tox defined tests (install, run tests, publish reports in on step).
Authoring the package
Create new project directory and initialize git:
$ mkdir francesco
$ cd francesco
$ git init
Create a package or module
Here we create single module francesco, but the same works for more
modules or packages.
francesco.py
def main():
print("Hi, it is me, Francesco, keeping things simple.")
requirements.txt
Create list of packages for actual installation of the package:
six
test_requirements.txt
Define packages required for testing:
pytest
coverage
tests/test_it.py
Initiate the test suite:
from francesco import main
def test_this():
main()
print("All seems fine to me")
assert True
setup.py
Did you dream of stupid simple setup.py? Here it goes:
from setuptools import setup
setup(setup_requires=["pbr"], pbr=True)
setup.cfg
Metadata belong to configuration file:
[metadata]
name = francesco
author = Francesco Montesano
author-email = fm#acme.com
summary = Nice and simply installed python module supporting testing in different pythons
description-file = README.rst
[files]
modules=francesco
[entry_points]
console_scripts =
francesco = francesco:main
tox.ini
To configure tox automated builds and tests:
[tox]
envlist = py27, py34
[testenv]
commands =
coverage run --source francesco -m pytest -sv tests
coverage report
coverage html
deps =
-rtest_requirements.txt
README.rst
Never forget README.rst:
===========================================
Complex package with 3 line long `setup.py`
===========================================
Can we keep`setup.py` simple and still support automated testing?
...
tox: build sdist and run tests in all supported python versions
Being in the project directory root, just run single command tox:
$ tox
GLOB sdist-make: /home/javl/sandbox/setuppy/setup.py
py27 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py27 runtests: PYTHONHASHSEED='2409409075'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py27 runtests: commands[2] | coverage html
py34 inst-nodeps: /home/javl/sandbox/setuppy/.tox/dist/francesco-0.0.0.zip
py34 runtests: PYTHONHASHSEED='2409409075'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /home/javl/sandbox/setuppy/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py34 runtests: commands[2] | coverage html
___________________________________ summary ____________________________________
py27: commands succeeded
py34: commands succeeded
congratulations :)
Getting the sdist
ls .tox/dist
francesco-0.0.0.zip
Developing in Python 2.7 virtualenv
Activate Python 2.7 virtualenv
$ source .tox/py27/bin/activate
Run tests
(py27) $ py.test -sv tests
==============================================================================================
test session starts
===============================================================================================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
-- /home/javl/sandbox/setuppy/.tox/py27/bin/python2.7 cachedir: .cache
rootdir: /home/javl/sandbox/setuppy, inifile: collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things
simple. All seems fine to me PASSED
============================================================================================
1 passed in 0.01 seconds
============================================================================================
Measure test coverage
(py27)$ coverage run --source francesco -m pytest -sv tests
.....
(py27)$ coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
View coverage report in web browser
(py27)$ coverage html
(py27)$ firefox htmlcov/index.html
Release new package version
(optional) Install local devpi-server
The installation of devpi-server is not covered here, but is very
simple, especially, if you install only to your local machine for your
personal testing.
Commit source code, assign version tag
Make sure, all your source code is commited.
Assing version tag:
$ git tag -a 0.1
Rerun the tests by tox and build sdist
Make sure, you have deactivated virtualenv (otherwise it conflicts with
tox):
(py27)$ deactivate
Run the tox:
$ tox
.....
...it builds as usual, may fail, if you have forgotten to commit some changes or files...
Find the sdist for new version of your package:
$ ls .tox/dist/francesco-0.1.0.
.tox/dist/francesco-0.1.0.zip
You are done. You may distribute your new tested versions of your package
to users as usually.
(optional) Upload the sdist to devpi-server and test it locally
Following steps assume, you have devpi-server installed and running.
$ devpi login javl
...enter your password...
$ devpi upload .tox/dist/francesco-0.1.0.zip
Test the package in clean environment
(deactivate virtualenv if active) :
$ cd /tmp
$ mkdir testing
$ cd testing
$ devpi test francesco
received http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
unpacking /tmp/devpi-test6/downloads/francesco-0.1.0.zip to /tmp/devpi-test6/zip
/tmp/devpi-test6/zip/francesco-0.1.0$ tox --installpkg /tmp/devpi-test6/downloads/francesco-0.1.0.zip -i ALL=http://localhost:3141/javl/dev/+simple/ --recreate --result-json /tmp/devpi-test6/zip/toxreport.json
-c /tmp/devpi-test6/zip/francesco-0.1.0/tox.ini
py27 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27
py27 installdeps: -rtest_requirements.txt
py27 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py27 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py27 runtests: PYTHONHASHSEED='3916044270'
py27 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux2 -- Python 2.7.9, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py27/bin/python2.7
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py27 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py27 runtests: commands[2] | coverage html
py34 create: /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34
py34 installdeps: -rtest_requirements.txt
py34 inst: /tmp/devpi-test6/downloads/francesco-0.1.0.zip
py34 installed: coverage==4.0.3,francesco==0.1.0,py==1.4.31,pytest==2.8.7,six==1.10.0,wheel==0.24.0
py34 runtests: PYTHONHASHSEED='3916044270'
py34 runtests: commands[0] | coverage run --source francesco -m pytest -sv tests
============================= test session starts ==============================
platform linux -- Python 3.4.2, pytest-2.8.7, py-1.4.31, pluggy-0.3.1 -- /tmp/devpi-test6/zip/francesco-0.1.0/.tox/py34/bin/python3.4
cachedir: .cache
rootdir: /tmp/devpi-test6/zip/francesco-0.1.0, inifile:
collecting ... collected 1 items
tests/test_it.py::test_this Hi, it is me, Francesco, keeping things simple.
All seems fine to me
PASSED
=========================== 1 passed in 0.01 seconds ===========================
py34 runtests: commands[1] | coverage report
Name Stmts Miss Cover
----------------------------------
francesco.py 2 0 100%
py34 runtests: commands[2] | coverage html
____________________________________________________________________________________________________ summary _____________________________________________________________________________________________________
py27: commands succeeded
py34: commands succeeded
congratulations :)
wrote json report at: /tmp/devpi-test6/zip/toxreport.json
posting tox result data to http://localhost:3141/javl/dev/+f/4f7/c13fee84bb7c8/francesco-0.1.0.zip
successfully posted tox result data
You may check the test results in web browser:
$ firefox http://localhost:3141
then search for "francesco" package, click the package name, find in table
column named "tox results", click there to show environment set up and test
results.
Let your users test the package
Let's assume, your devpi-server is running and your user has access to it.
The user shall have devpi command installed:
$ pip install devpi
(note, this tool is not installing anything from the devpi-server)
Help your user to gain access to devpi-server (not covering here).
Then the user just runs the test:
$ devpi test francesco
After the test is run (it is automatically using tox, but user does not have
to care about that), you will find test results on the same place on devpi web
interface as you found yours before.
I want to use nose.collector as a test suite for setuptools, as described here. My package's source lives in mypackage/src, and I have tests in mypackage/tests. I have a setup.py that looks like this:
import setuptools
setuptools.setup(
name='mypackage',
version='1.2.3',
package_dir={'': 'src'},
packages=setuptools.find_packages('src'),
tests_require=['nose'],
test_suite='nose.collector',
provides=setuptools.find_packages('src'),
)
However, when I run python setup.py test, it doesn't test anything:
$ python setup.py test
running test
running egg_info
writing src/mypackage.egg-info/PKG-INFO
writing top-level names to src/mypackage.egg-info/top_level.txt
writing dependency_links to src/mypackage.egg-info/dependency_links.txt
reading manifest file 'src/mypackage.egg-info/SOURCES.txt'
writing manifest file 'src/mypackage.egg-info/SOURCES.txt'
running build_ext
----------------------------------------------------------------------
Ran 0 tests in 0.002s
OK
How can I tell nose where to look for tests? Up until now, I've been doing nosetests -d tests, which works fine. But I'd like to change to use setuptools so that I can follow the python setup.py test convention.
from the docs
When running under setuptools, you can configure nose settings via the
environment variables detailed in the nosetests script usage message,
or the setup.cfg, ~/.noserc or ~/.nose.cfg config files.
http://nose.readthedocs.org/en/latest/setuptools_integration.html
Fix them with chmod -x $(find tests/ -name '*.py')