We Keep Coding

How to let pytest discover and run doctests in installed modules?

I am adding unit tests and to a kind of "legacy" Python package. Some of the modules contain their own doctests embedded in docstrings. My goal is to run both those doctests and new, dedicated unit tests.
Following this Q&A ("How to make py.test run doctests as well as normal tests directory?") I'm using the --doctest-modules option to pytest. When running from the source repository, pytest indeed discovers the embedded doctests from Python modules under the src directory.
However, my goal is to test that the source distribution builds and installs at all, and then test everything against the installed package. To do this I'm using tox which automate the process of building a sdist (source distribution) tarball, installing it in a virtual environment, and running the tests against the installed version. To ensure that it is the installed version, rather than the one in the source repository, that is imported by the tests, I follow the suggestion in this article and the repository looks like this now:
repo/
src/
my_package/
__init__.py
module_a.py
module_b.py
...
tests/
test_this.py
test_that.py
requirements.txt
setup.py
tox.ini
(The test scripts under tests import the package as in import my_package, which hits the installed version, because the repository layout makes sure that the src/my_package directory is out of the module search paths.)
And in the tox configuration file, the relevant sections looks like
[tox]
envlist = py27,py36,coverage-report
[testenv]
deps =
-rrequirements.txt
commands =
coverage run -p -m pytest --
[pytest]
addopts = --doctest-modules
So far, the tests run fine, and the doctests are picked up -- from the modules under src/my_package, rather than from the package installed in tox virtual environments.
My questions related to this set-up is as follows:
Is this actually a concern? tox seems to ensure that what you install is what you have in the source repository, but does it?
How can I instruct pytest to actually run doctests from the installed modules in a sort of clean way? I can think of a few solutions such as building a dedicated documentation tree in a doc directory and let pytest find the doctests in there with the --doctest-glob option. But is there a method to do this without building the docs first?

I found the answer to my own question.
To quote pytest issue #2042:
currently doctest-modules is fundamentally incompatible with testing
against a installed package due to module search in checkout vs module
usage from site-packages
So as of now the solution does not exist.

pytest collects tests from the current directory (unless you instruct it otherwise passing an explicit directory). Add the installation directory using tox substitutions in tox.ini. I.e., either pass the directory:
[testenv]
deps =
-rrequirements.txt
commands =
coverage run -p -m pytest {envsitepackagesdir}/my_package
or change directory:
[testenv]
changedir = {envsitepackagesdir}/my_package
deps =
-rrequirements.txt
commands =
coverage run -p -m pytest --

This is how I solved the import mismatch:
$ pytest tests/ --doctest-modules --pyargs myrootpkg <other args>
The problem here is that once you start specifying source paths explicitly (in this case via --pyargs), you have to specify all other source paths as well (tests/ in this example) as pytest will stop scanning the rootdir. This shouldn't be an issue when using the src layout though, as the tests aren't usually scattered around the repository.

How do I tell pbr to use pytest when setup.py test command is invoked?

While using pbr to simplify Python packaging what do we need to configure in order to make it use pytest when python setup.py test command is executed.
Running pytest works without any problems.

In setup.py:
setup(
setup_requires=['pbr>=1.9', 'setuptools>=17.1', 'pytest-runner'],
pbr=True,
)
In setup.cfg (after standard pbr config):
[aliases]
test=pytest
In test-requirements.txt (same directory as requirements.txt):
pytest
If your tests are outside the application code, you will also need to specify your test directory using addopts in setup.cfg. For example, if your directory structure looks like the first example on this page, you should have
[tool:pytest]
addopts = tests

coverage.py can't collect any data for coveralls failing in travis

I am building a python application in travis. I have setup.py which basically runs the command:
py.test packageName
when we do: setup.py test
I am trying to run coverage plugin to push to coveralls so when i run the command:
coverage run setup.py test
it fails with :
running test
Coverage.py warning: No data was collected.
It is collecting all the test cases but just doesn't collect the data for some reason. My covezragerc looks like this:
> [run] omit = respawn/gen.py, respawn/cli.py, respawn/test/*
> data_file = .coverage
Any reasons why ??

I'll guess from the fact that you have "respawn" in your .coveragerc, that your product code is running in subprocesses. Take a look at http://coverage.readthedocs.org/en/latest/subprocess.html to see how to configure coverage.py for that case.

How to run tests without installing package?

I have some Python package and some tests. The files are layed out following http://pytest.org/latest/goodpractices.html#choosing-a-test-layout-import-rules
Putting tests into an extra directory outside your actual application
code, useful if you have many functional tests or for other reasons
want to keep tests separate from actual application code (often a good
idea):
setup.py # your distutils/setuptools Python package metadata
mypkg/
__init__.py
appmodule.py
tests/
test_app.py
My problem is, when I run the tests py.test, I get an error
ImportError: No module named 'mypkg'
I can solve this by installing the package python setup.py install but this means the tests run against the installed package, not the local one, which makes development very tedious. Whenever I make a change and want to run the tests, I need to reinstall, else I am testing the old code.
What can I do?

I know this question has been already closed, but a simple way I often use is to call pytest via python -m, from the root (the parent of the package).
$ python -m pytest tests
This works because -m option adds the current directory to the python path, and hence mypkg is detected as a local package (not as the installed).
See:
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-python-m-pytest

The normal approach for development is to use a virtualenv and use pip install -e . in the virtualenv (this is almost equivalent to python setup.py develop). Now your source directory is used as installed package on sys.path.
There are of course a bunch of other ways to get your package on sys.path for testing, see Ensuring py.test includes the application directory in sys.path for a question with a more complete answer for this exact same problem.

On my side, while developing, I prefer to run tests from the IDE (using a runner extension) rather than using the command line. However, before pushing my code or prior to a release, I like to use the command line.
Here is a way to deal with this issue, allowing you to run tests from both the test runner used by your IDE and the command line.
My setup:
IDE: Visual Studio Code
Testing: pytest
Extension (test runner): https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
Work directory structure (my solution should be easily adaptable to your context):
project_folder/
src/
mypkg/
__init__.py
appmodule.py
tests/
mypkg/
appmodule_test.py
pytest.ini <- Use so pytest can locate pkgs from ./src
.env <- Use so VsCode and its extention can locate pkgs from ./src
.env:
PYTHONPATH="${PYTHONPATH};./src;"
pytest.ini (tried with pytest 7.1.2):
[pytest]
pythonpath = . src
./src/mypkg/appmodule.py:
def i_hate_configuring_python():
return "Finally..."
./tests/mypkg/appmodule_test.py:
from mypkg import app_module
def test_demo():
print(app_module.i_hate_configuring_python())
This should do the trick

Import the package using from .. import mypkg. For this to work you will need to add (empty) __init__.py files to the tests directory and the containing directory. py.test should take care of the rest.

Does unittest allow single case/suite testing through "setup.py test"?

I'm a newbie when it comes to python unit testing, but I'm eager to learn!
I just read python setup.py test can run all suites derived from unittest classes. I wonder if I also can use setup.py to run a single suite and/or a single test case, maybe adding some modifier to the previous command like python setup.py tests suitename. If so, can you please point me to any docs/examples?

You guys are all wrong, setup.py test can be used with the -s option the same way python -m unittest does:
cd root_of_your_package
python setup.py test -s tests.TestClass.test_method

The setup.py test runner is rather limited; it only supports letting you specify a specific module. The documentation for the command-line switches is given when you use the --help switch:
python setup.py test --help
Common commands: (see '--help-commands' for more)
[ ... cut ... ]
Options for 'test' command:
--test-module (-m) Run 'test_suite' in specified module
--test-suite (-s) Test suite to run (e.g. 'some_module.test_suite')
[ ... more cut ... ]
so python setup.py test -m your.package.tests.test_module would limit running the tests from the test_module.py file only.
All the test command does, really, is make sure your egg has been built already, extract the test_suite value from setup() metadata, configure a test loader that understands about zipped eggs, then run the unittest.main() function.
If you need to run a single test only, have already built your egg, are not running this with a zipped egg, then you can also just use the unittest command line interface, which does pretty much everything else:
python -m unittest yourpackage.tests.TestClass.test_method
would instruct unittest to only run a very specific test method.

setup.py test
setup.py test is not that flexible, but here's an alternative:
The unittest module can run specific test methods
From the Documentation on unittest
The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
You can pass in a list with any combination of module names, and fully qualified class or method names.
You can run tests with more detail (higher verbosity) by passing in the -v flag:
python -m unittest -v test_module
For a list of all the command-line options:
python -m unittest -h

in case of pytest run specific test:
python setup.py test --addopts tests/jobs/test_data_monitoring.py::TestDataMonitoring::test_dataframe__bulk_update

In case you use pytest for unit testing you can run a file directly by avoiding the setup.py file.
Running all tests for me is:
python setup.py test
Running a specific test:
pytest --pyargs path/to/test.py
Hope it helps.
(I know this post is already a bit old, but I came here looking for running a way to run pytest tests alone, figured it might be of use to someone)