The Python documentation states that the -t option controls the:
Top level directory of project (defaults to start directory)
Usually people use the -s option (python -m unittest discover tests/ is equivalent to python -m unittest discover -s tests), and I have never seen anyone use -t before. The brief description in the documentation is not enlightening to me.
What does the "top level directory" mean in this particular context? What exactly does the -t option do?
My understanding is that although the top-level directory defaults to whatever the starting directory is, the starting directory must be contained by the test directory.
From the first paragraph on test discovery:
Unittest supports simple test discovery. In order to be compatible with test discovery, all of the test files must be modules or packages (including namespace packages) importable from the top-level directory of the project (this means that their filenames must be valid identifiers).
Suppose you have a directory structure like
./
tests1/
tests2/
If ./ is the top-level directory and tests1 is the starting directory, no tests will discovered under tests2, even though tests2 is importable from the top-level directory.
The purpose of -s would be to discover only a subset of tests for a particular project. The purpose of -t might be to choose a particular "subproject" to run tests for.
Related
I have a CMake based project that uses both C++ and python sources. Tests are added via add_test for all C++ tests in a folder structure like:
src
mynamespace/foo.cpp
mypyspace/mypkg/__init__.py
mypyspace/mypkg/bar.py
test
mynamespace/testFoo.cpp
mypyspace/testBar.py
In testBar.py I have a import mypkg and want to run this with add_test(... COMMAND pytest WORKING_DIRECTORY .../test)
Of course it does not find my package, but I also do not want to install it (it is required to generate some stuff during build and is actively developed as part of the other sources).
What is the correct/preferred way of executing those tests?
Notes:
I don't want to move the sources out of src or tests out of test
set_tests_properties may set environment variables.
I think setting PYTHONPATH=.../src/mypyspace should work, but that would also ignore changes to the users PYTHONPATH.
There are 2 options:
Use PYTHONPATH
add_test (NAME python-tests
COMMAND ${PYTHON_EXECUTABLE} -m pytest # Or just COMMAND pytest
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
)
set_tests_properties(python-tests
PROPERTIES ENVIRONMENT "PYTHONPATH=${PROJECT_SOURCE_DIR}/src/mypyspace:$ENV{PYTHONPATH}")
Invoke inside of the src directory
add_test (NAME python-tests
COMMAND ${PYTHON_EXECUTABLE} -m pytest ${CMAKE_CURRENT_SOURCE_DIR}
WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/src/mypyspace
)
The first approach has the disadvantage, that the user might configure and later change its environment so that PYTHONPATH is different (e.g. loading modules in HPC clusters) and find that the tests incorrectly succeed.
The 2nd is the most flexible one. It relies on information from https://docs.pytest.org/en/latest/pythonpath.html. Most important part:
the [python -m pytest] call will add the current directory to sys.path.
So we only have to set the current directory when running the tests to the source directory.
Note: For other alternatives specific to unittest (the module) you may also find something in Running unittest with typical test directory structure
I'm looking for a way of running python -m unittest discover, which will discover tests in, say, directories A, B and C. However, directories A, B and C have directories named dependencies inside each of them, in which there are also some tests which, however, I don't want to run.
Is there a way to run my tests satisfying these constraints without having to create a script for this?
I ran into the same problem and was eventually able to find these handy arguments to pass to unittest discover that resolved my issue.
It is documented here: https://docs.python.org/2/library/unittest.html#test-discovery
-s, --start-directory directory
Directory to start discovery (. default)
-p, --pattern pattern
Pattern to match test files (test*.py default)
So I modified my command to be:
python -m unittest discover -s test
since all of the tests I actually want to run are in the one module, test. You could also use the -p to in theory match regex that only hits your tests, ignoring all of the rest it may find.
I've managed to do it this way (In *NIX):
find `pwd` -name '*_test.py' -not -path '*unwanted_path*' \
| xargs python3 -m unittest -v
That is, the tests are discovered by find, which allows for options like path pattern exclusions, then they're passed to the unittest command as argument list.
Note that I had to switch to find pwd, where usually I can write find ., since relative paths in the form ./xxx aren't accepted by unittest (module not found).
It would seem that python -m unittest descends into module directories but not in other directories.
It quickly tried the following structure
temp
+ a
- test_1.py
+ dependencies
- test_a.py
With the result
>python -m unittest discover -s temp\a
test_1
.
----------------------------------------------------------------------
Ran 1 test in 0.002s
OK
However, if the directory is a module directory (containing a file __init__.py) the situation is different.
temp
+ a
- __init__.py
- test_1.py
+ dependencies
- __init__.py
- test_a.py
Here the result was
>python -m unittest discover -s temp\a
test_a
.test_1
.
----------------------------------------------------------------------
Ran 2 tests in 0.009s
OK
The usefulness of this answer for you now depends on whether it is acceptable for your folder dependencies not to be a module directory.
EDIT: After seeing your comment
Would using pytest be an option? This test runner has many command arguments, one specifically to exclude tests.
See Changing standard (Python) test discovery
From their site
Ignore paths during test collection
You can easily ignore certain test directories and modules during collection by passing the --ignore=path option on the cli. pytest allows multiple --ignore options
I am having an issue with nose.
I want to run tests in a tests directory but I don't want nose to change the working directory. This is because I am using relative imports in my library and the unittests themselves.
First the directory structure:
app_dir
app.py
library_dir
tests_dir
test_1.py
test_2.py
I am currently running my tests like this from within the app_dir directory:
python -m unittest discover -s library_dir.tests_dir
This works fine.
Now, I would like to do this with nose (nosetests).
When I try it, I get the following:
nosetests library_dir.tests_dir --collect-only -vv
Failure: SystemError (Parent module '' not loaded, cannot perform relative import) ... ok
Failure: SystemError (Parent module '' not loaded, cannot perform relative import) ... ok
I've looked at similar questions on SO already (there are many) and one suggestion for this issue that I can see is to remove the relative imports from within the unittest itself but I like this pattern. Additionally, if I do change to an absolute import, I get an
(No module named '<the_module_under_test')
Is there any way for nose to find tests from a directory (like what the -w option provides BUT keeping the working directory the same as that from the terminal which invoked nose?
Whoops, looks like all I needed to do was to add __init__.py to the library and the subdirectory tests directory. I was working off of the assumption nose would work as expected with PEP 420
Blockquote
PEP 420: Implicit Namespace Packages
Native support for package directories that don’t require init.py marker files and can automatically span multiple path segments (inspired by various third party approaches to namespace packages, as described in PEP 420)
I have some Python package and some tests. The files are layed out following http://pytest.org/latest/goodpractices.html#choosing-a-test-layout-import-rules
Putting tests into an extra directory outside your actual application
code, useful if you have many functional tests or for other reasons
want to keep tests separate from actual application code (often a good
idea):
setup.py # your distutils/setuptools Python package metadata
mypkg/
__init__.py
appmodule.py
tests/
test_app.py
My problem is, when I run the tests py.test, I get an error
ImportError: No module named 'mypkg'
I can solve this by installing the package python setup.py install but this means the tests run against the installed package, not the local one, which makes development very tedious. Whenever I make a change and want to run the tests, I need to reinstall, else I am testing the old code.
What can I do?
I know this question has been already closed, but a simple way I often use is to call pytest via python -m, from the root (the parent of the package).
$ python -m pytest tests
This works because -m option adds the current directory to the python path, and hence mypkg is detected as a local package (not as the installed).
See:
https://docs.pytest.org/en/latest/usage.html#calling-pytest-through-python-m-pytest
The normal approach for development is to use a virtualenv and use pip install -e . in the virtualenv (this is almost equivalent to python setup.py develop). Now your source directory is used as installed package on sys.path.
There are of course a bunch of other ways to get your package on sys.path for testing, see Ensuring py.test includes the application directory in sys.path for a question with a more complete answer for this exact same problem.
On my side, while developing, I prefer to run tests from the IDE (using a runner extension) rather than using the command line. However, before pushing my code or prior to a release, I like to use the command line.
Here is a way to deal with this issue, allowing you to run tests from both the test runner used by your IDE and the command line.
My setup:
IDE: Visual Studio Code
Testing: pytest
Extension (test runner): https://marketplace.visualstudio.com/items?itemName=LittleFoxTeam.vscode-python-test-adapter
Work directory structure (my solution should be easily adaptable to your context):
project_folder/
src/
mypkg/
__init__.py
appmodule.py
tests/
mypkg/
appmodule_test.py
pytest.ini <- Use so pytest can locate pkgs from ./src
.env <- Use so VsCode and its extention can locate pkgs from ./src
.env:
PYTHONPATH="${PYTHONPATH};./src;"
pytest.ini (tried with pytest 7.1.2):
[pytest]
pythonpath = . src
./src/mypkg/appmodule.py:
def i_hate_configuring_python():
return "Finally..."
./tests/mypkg/appmodule_test.py:
from mypkg import app_module
def test_demo():
print(app_module.i_hate_configuring_python())
This should do the trick
Import the package using from .. import mypkg. For this to work you will need to add (empty) __init__.py files to the tests directory and the containing directory. py.test should take care of the rest.
Python's unittest discover does not find my tests!
I have been using nose to discover my unit tests and it is working fine. From the top level of my project, if I run nosetests I get:
Ran 31 tests in 0.390s
Now that Python 2.7 unittest has discovery, I have tried using
python -m unittest discover
but I get
Ran 0 tests in 0.000s
My directory structure is:
myproj/
reporter/
__init__.py
report.py
[other app modules]
tests/
__init__.py
test-report.py
[other test modules]
Do you have any ideas why unittest's discovery algorithm can't find the tests?
I'm using Python 2.7.1 and nose 1.0.0 on Windows 7.
The behaviour is intentional, but the documentation could make this clearer. If you look at the first paragraph in the test discovery section, it says:
For a project’s tests to be compatible with test discovery they must all be importable from the top level directory of the project (in other words, they must all be in Python packages).
A corollary to that is that the file names must also be valid Python module names. test-report.py fails that test, since test-report is not a legal Python identifier.
A docs bug suggesting that this be mentioned explicitly in the documentation for the -p pattern option would probably be a good way forward.
I had this problem because some directories in a project were missing __init__.py. I thought I don't need them in Python 3.7.
Just add __init__.py to every directory and python3 -m unittest will find tests automatically.
As someone relatively new to Python, the naming convention in the docs implied the opposite. Ben's comment was very helpful: the default discovery pattern looks for test-modules prefixed with the string "test"
I thought the introspection would just look for class names and not require a specific file naming convention.
Here is what the docs say:
https://docs.python.org/3/library/unittest.html
python -m unittest discover -s project_directory -p "_test.py"
I couldn't get this to work, but by changing my file names to be "test_.py" - success!