I have a Django project with multiple apps. Each app has a set of unittests. I'm using pytest as my test runner. We have gotten to a point that we want to start writing integration tests. I was wondering if there is any way to keep the naming convention and thus the auto discovery of pytest but still be able (via flag maybe?) to run the different test types. The most intuitive solution that comes to mind is some sort of decorator on test methods or even TestCase classes (something like Category in JUnit).
something like:
#testtype('unittest')
def test_my_test(self):
# do some testing
#testtype('integration')
def test_my_integration_test(self):
# do some integration testing
and then i could run the test like:
py.test --type=integration
py.test --type=unittest
Is there such a thing?
If not, the only other solution i can think about is to add a django command and "manually" build a testsuite and run it with pytest... I would prefer not to use this option. Is there any other solution that can help me?
Thanks
You can mark test functions.
import pytest
#pytest.mark.unittest
def test_my_test(self):
# do some testing
#pytest.mark.integration
def test_my_integration_test(self):
# do some integration testing
These custom markers must be registered in your pytest.ini file.
Then use the -m flag to run the marked tests
py.test -v -m unittest
Another option would be to split your tests into unittest and integration directories. Then you can run tests in a specific directory with:
py.test -v unittest
or
py.test -v integration
Another way to do this (without any config or code)
pytest -o "python_functions=*_integration_test"
You can also do this in module/class level, e.g.,
python_files = integration_test_*.py
python_classes = IntegrationTest
Ref:
https://docs.pytest.org/en/latest/example/pythoncollection.html#changing-naming-conventions
Related
Suppose I've got a few modules with test cases derived from unittest.TestCase. All those modules reside in one package test:
test/
test_case1.py
test_case2.py
test_case3.py
I'd like to run all tests in all modules in test with one shell command. In order to do it I've added a new module test_all.py, which creates a TestSuite with all the test cases, and main:
def make_suite():
... # add test cases explicitly one by one
if __name__ == "__main__":
suite = make_suite()
unittest.TextTestRunner().run(suite)
Now I wonder if there is a way to run all the test cases in test without making a TestSuite
Starting in Python 2.7, you can do something like this:
python -m unittest discover <test_directory>
Of course, all your test directories must contain an __init__.py file.
You can also use a tool like nose to run your tests.
See also:
How do I run all Python unit tests in a directory?
for f in ./test/test_case*.py; do python $f; done
Is there a way to select pytest tests to run from a file?
For example, a file foo.txt containing a list of tests to be executed:
tests_directory/foo.py::test_001
tests_directory/bar.py::test_some_other_test
Or, is there a way to select multiple tests, having no common pattern in test name, from different directories with pytest?
pytest -k <pattern> allows a single pattern.
One option is to have a pytest.mark against each test, but my requirement is to run different combination of tests from different files.
Is there a way to specify multiple patterns and a test file name for each pattern?
Or
Is there a way to specify the exact test paths in a file and feed that file as an input to pytest?
Or
Is there a hook function that can be utilized for this purpose?
You can use -k option to run test cases with different patterns:
py.test tests_directory/foo.py tests_directory/bar.py -k 'test_001 or test_some_other_test'
This will run test cases with name test_001 and test_some_other_test deselecting the rest of the test cases.
Note: This will select any test case starting with test_001 or test_some_other_test. For example, if you have test case test_0012 it will also be selected.
Specifying tests / selecting tests
Pytest supports several ways to run and select tests from the command-line.
Run tests in a module
pytest test_mod.py
Run tests in a directory
pytest testing/
Run tests by keyword expressions
pytest -k "MyClass and not method"
This will run tests which contain names that match the given string expression, which can include Python operators that use filenames, class names and function names as variables. The example above will run TestMyClass.test_something but not TestMyClass.test_method_simple.
Run tests by node ids
Each collected test is assigned a unique nodeid which consist of the module filename followed by specifiers like class names, function names and parameters from parametrization, separated by :: characters.
To run a specific test within a module:
pytest test_mod.py::test_func
Another example specifying a test method in the command line:
pytest test_mod.py::TestClass::test_method
Run tests by marker expressions
pytest -m slow
Will run all tests which are decorated with the #pytest.mark.slow decorator.
For more information see marks.
Run tests from packages
pytest --pyargs pkg.testing
This will import pkg.testing and use its filesystem location to find and run tests from.
Source: https://docs.pytest.org/en/latest/usage.html#specifying-tests-selecting-tests
My answer provides a ways to run a subset of test in different scenarios.
Run all tests in a project
pytest
Run tests in a Single Directory
To run all the tests from one directory, use the directory as a parameter to
pytest:
pytest tests/my-directory
Run tests in a Single Test File/Module
To run a file full of tests, list the file with the relative path as a parameter to pytest:
pytest tests/my-directory/test_demo.py
Run a Single Test Function
To run a single test function, add :: and the test function name:
pytest -v tests/my-directory/test_demo.py::test_specific_function
-v is used so you can see which function was run.
Run a Single Test Class
To run just a class, do like we did with functions and add ::, then the class name to the file parameter:
pytest -v tests/my-directory/test_demo.py::TestClassName
Run a Single Test Method of a Test Class
If you don't want to run all of a test class, just one method, just add
another :: and the method name:
pytest -v tests/my-directory/test_demo.py::TestClassName::test_specific_method
Run a Set of Tests Based on Test Name
The -k option enables you to pass in an expression to run tests that have
certain names specified by the expression as a substring of the test name.
It is possible to use and, or, and not to create complex expressions.
For example, to run all of the functions that have _raises in their name:
pytest -v -k _raises
Method 1: Randomly selected tests. Long and ugly.
python -m pytest test/stress/test_performance.py::TestPerformance::test_continuous_trigger test/integration/test_config.py::TestConfig::test_valid_config
Method 2: Use Keyword Expressions.
Note: I am searching by testcase names. Same is applicable to TestClass names.
Case 1: The below will run whichever is found. Since we have used 'OR' .
python -m pytest -k 'test_password_valid or test_no_configuration'
Lets say the two above are actually correct, 2 tests will be run.
Case 2: Now an incorrect name and another correct name.
python -m pytest -k 'test_password_validzzzzzz or test_no_configuration'
Only one is found and run.
Case 3: If you want to run either all tests mentioned, or None, then use AND
python -m pytest -k 'test_password_valid and test_no_configuration'
Both will be run if correct or none.
Case 4: Run test only in one folder:
python -m pytest test/project1/integration -k 'test_password_valid or test_no_configuration'
Case 5: Run test from only one file.
python -m pytest test/integration/test_authentication.py -k 'test_password_expiry or test_incorrect_password'
Case 6: Run all tests except the match.
python -m pytest test/integration/test_authentication.py -k 'not test_incorrect_password'
If you have the same method name in two different classes and you just want to run one of them, this works:
pytest tests.py -k "TestClassName and test_method_name"
According to the doc about Run tests by node ids
since you have all node ids in foo.txt, just run
pytest $(tr '\n' ' ' <foo.txt)
this is same with below command (with file content in the question)
pytest tests_directory/foo.py::test_001 tests_directory/bar.py::test_some_other_test
Maybe using pytest_collect_file() hook you can parse the content of a .txt o .yaml file where the tests are specify as you want, and return them to the pytest core.
A nice example is shown in the pytest documentation. I think what you are looking for.
Try:
pytest path/file_test.py::test_name
Using -k as others have suggested will match any test with that substring, not just a specific test.
Say for example you have a file called test.py. In this file you have a class called TestSomeService. This class has 2 functions namely test_service1 and test_service2.
Now to run specific function test you can this
pytest test.py::TestSomeService::test_service1
pytest test.py::TestSomeService::test_service2
To run specific class
pytest test.py::TestSomeService
Here's a possible partial answer, because it only allows selecting the test scripts, not individual tests within those scripts.
And it also limited by my using legacy compatibility mode vs unittest scripts, so not guaranteeing it would work with native pytest.
Here goes:
create a new dictory, say subset_tests_directory.
ln -s tests_directory/foo.py
ln -s tests_directory/bar.py
be careful about imports which implicitly assume files are in test_directory. I had to fix several of those by running python foo.py, from within subset_tests_directory and correcting as needed.
Once the test scripts execute correctly, just cd subset_tests_directory and pytest there. Pytest will only pick up the scripts it sees.
Another possibility is symlinking within your current test directory, say as ln -s foo.py subset_foo.py then pytest subset*.py. That would avoid needing to adjust your imports, but it would clutter things up until you removed the symlinks. Worked for me as well.
Assuming the test names are unique, you have to remove the test file's name:
cut -d : -f 3 foo.txt > FAILED_TESTS.txt
As others pointed out use -k, but you have to pass the file's content (i.e., list of test names) as a single string:
pytest -k "$(awk '$1=$1' RS= OFS=' or ' FAILED_TESTS.txt)"
awk will replace the new lines with a delimiter or so that the test names are joined in a format that pytest expects.
I'm a newbie when it comes to python unit testing, but I'm eager to learn!
I just read python setup.py test can run all suites derived from unittest classes. I wonder if I also can use setup.py to run a single suite and/or a single test case, maybe adding some modifier to the previous command like python setup.py tests suitename. If so, can you please point me to any docs/examples?
You guys are all wrong, setup.py test can be used with the -s option the same way python -m unittest does:
cd root_of_your_package
python setup.py test -s tests.TestClass.test_method
The setup.py test runner is rather limited; it only supports letting you specify a specific module. The documentation for the command-line switches is given when you use the --help switch:
python setup.py test --help
Common commands: (see '--help-commands' for more)
[ ... cut ... ]
Options for 'test' command:
--test-module (-m) Run 'test_suite' in specified module
--test-suite (-s) Test suite to run (e.g. 'some_module.test_suite')
[ ... more cut ... ]
so python setup.py test -m your.package.tests.test_module would limit running the tests from the test_module.py file only.
All the test command does, really, is make sure your egg has been built already, extract the test_suite value from setup() metadata, configure a test loader that understands about zipped eggs, then run the unittest.main() function.
If you need to run a single test only, have already built your egg, are not running this with a zipped egg, then you can also just use the unittest command line interface, which does pretty much everything else:
python -m unittest yourpackage.tests.TestClass.test_method
would instruct unittest to only run a very specific test method.
setup.py test
setup.py test is not that flexible, but here's an alternative:
The unittest module can run specific test methods
From the Documentation on unittest
The unittest module can be used from the command line to run tests from modules, classes or even individual test methods:
python -m unittest test_module1 test_module2
python -m unittest test_module.TestClass
python -m unittest test_module.TestClass.test_method
You can pass in a list with any combination of module names, and fully qualified class or method names.
You can run tests with more detail (higher verbosity) by passing in the -v flag:
python -m unittest -v test_module
For a list of all the command-line options:
python -m unittest -h
in case of pytest run specific test:
python setup.py test --addopts tests/jobs/test_data_monitoring.py::TestDataMonitoring::test_dataframe__bulk_update
In case you use pytest for unit testing you can run a file directly by avoiding the setup.py file.
Running all tests for me is:
python setup.py test
Running a specific test:
pytest --pyargs path/to/test.py
Hope it helps.
(I know this post is already a bit old, but I came here looking for running a way to run pytest tests alone, figured it might be of use to someone)
How can a user of a library run his own initialization code (setting debug levels of loggers for example) before running tests supplied with the library? Python's unittest module is used as a testrunner.
You can try using pytest to run the unittests. If that works (many unittest based test suites work), then you can create a little module, for example "mymod.py", defining a pytest configuration hook:
# content of mymod.py
def pytest_configure():
import logging
logging.getLogger().setLevel(logging.WARN)
If you now execute py.test like this:
$ py.test -p mymmod --pyargs mylib
Then the "mylib" package will be searched for tests and ahead of running them the logging level is modified. The "-p" option specifies a plugin to load and the "--pyargs" tells pytest to try importing the arguments first, instead of treating them as directory or file names (the default).
For more info: http://pytest.org and http://pytest.org/latest/unittest.html
HTH,
holger
You could do the set-up work before calling unittest.main.
Or you could subclass the test suite and run a class-level setup method.
Or you could have the test setup incorporate a callback to a user-defined setup method.
Answer from similar question is more helpful here:
How to a run a specific code before & after each unit test in python
I used python setUpClass method
https://docs.python.org/2/library/unittest.html#unittest.TestCase.setUpClass
setUpClass()
A class method called before tests in an individual class are run. setUpClass is called with the class as the only argument and must be decorated as a classmethod():
#classmethod
def setUpClass(cls):
...
I run my tests like this:
python -m unittest discover -v
I am trying to get nosetests to identify my tests but it is not running any of my tests properly.
I have the following file structure
Project
+----Foo/
+----__init__.py
+----bar.py
+----test/
+----__init__.py
+----unit/
+----__init__.py
+----bar_test.py
+----functional/
+----__init__.py
+----foo_test.py
Within bar_test.py
class BarTest(unittest.TestCase):
def bar_1_test():
...
Within foo_test.py
class FooFTest.py
def foo_1_test():
...
Using -m, -i, -e options of nosetests
What is the regex I need to run only unit tests (under unit/, tests in class BarTest)
What is the regex I need run only functional tests (under functional/, tests in class FooFTest)
I've tried various combinations and can't seem to get nosetests to do what I want consistently
The easiest way to run only the tests under Project/test/unit is to use --where. For example:
nosetests --where=Project/test/unit
Then use --match (-m) or --exclude (-e) to refine the list, if needed.
If you still want to use the regex selectors, you can probably do it like this (not tested):
nosetests --match='^Foo[\b_\./-])[Tt]est'
Executing this script from the Project directory would run all tests that start with "Foo" and end in "[Tt]est".
As a general rule, you probably want to use either --match or --exclude, but not both. These parameters both specify the pattern of the function names to match. You can refine either one by using --ignore-files which, naturally, allows you to ignore whole files.
Given your directory structure, you can easily run segments of the tests using the --exclude option.
Run all tests:
nosetests Project
Run unit tests:
nosetests Project -e functional
Run functional tests:
nosetests Project -e unit
If you have more complex test execution needs, I'd recommend marking the tests and using the attrib package.