Let me say I'm still new to unit testing and using pytest.
I'm building unit tests and running them with GitHub Actions and I understand why it failed. I need to run the test modules in a specific order.
For reference, I have a following repo structure (generalized for simplicity)
mypackage/
├── __init__.py
├── foo.py
└── bar.py
tests/
├── __init__.py
├── test_foo.py
└── test_bar.py
My question (written for generalization): Is there a way to run specific test modules (not functions) in a given order with pytest? By default pytest will run in alphabetical order, i.e., test_bar before test_foo. However, in my case, test_bar uses data from test_foo, so I would like the order to be flipped. I can find other work arounds, but from my search I only found explanations of using pytest-order or pytest-ordering for specific functional orders in a given module. I thought this would be a good question for end-to-end testing while still ensuring 100% code coverage.
Thanks.
Related
We using py.test. We try to put different conftest.py files in different folders to split our fixtures:
tests/api/
├── conftest.py
├── folder1
│ └── conftest.py
├── folder2
│ └── conftest.py
But when run the tests this error occurs:
____ ERROR collecting api/folder1/conftest.py ____
import file mismatch:
imported module 'conftest' has this __file__ attribute:
/tests/api/folder2/conftest.py
which is not the same as the test file we want to collect:
/tests/api/folder1/conftest.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
Why is that? How fix it?
PS. Removing __pycache__.pyc did not help.
PPS. __init__.py files already exist in each folder.
I had the same issue. To solve this you need to create python packages instead of directories. Then pytest will look at the conftest.py in your package instead of root directory. Hope, this will help you.
tests/api/
├── conftest.py
├── package1 # not folder
│ └── conftest.py
├── package2 # not folder
│ └── conftest.py
Your use case sounds like this example in the pytest documentation. Because of that I think it's possible to use conftest.pys at different levels to override fixtures.
The errors you're seeing may be related to incorrect imports. Is your test code importing from conftest files directly? Are your conftest files importing from your tests? Are any of your imports relative instead of absolute? If any of these are true, that may be your issue. I recommend only using absolute imports, and avoid imports between conftest.pys and test files.
Rename one (or both) of the test files Pytest is complaining about. Pytest is telling you in the error message to do this (i.e. change the basename, meaning don't name all your test files conftest.py). For example, you can fix it by doing:
tests/api/
├── conftest.py
├── folder1
│ └── test_conf1.py
├── folder2
│ └── test_conf2.py
In your case, the module names conflict (you have three conftest.pys). This is a quirk of Pytest AFAIK. Pytest could get around this by managing full package/module paths: but it doesn't do this (probably for good reason, but I do not maintain/contribute to pytest so I can't shed light on the issue). Pytest is a fantastic framework (it's even telling you exactly why it can't run your tests): I'm sure they have a good reason for not supporting this behavior.
You claim that you want to:
separate tests and fixtures by different functionalities.
So do that. Separating the test fixtures/functionalities has nothing to do with what you name the files.
I commonly run into this error when splitting up unit/integration/acceptance tests. I split them up so I can run my (fast) unit tests without having to run my (potentially slow) integration/acceptance tests. I might have some module, call it Abc. And I have something like:
tests/
├── unit
│ └── test_abc.py
├── integration
│ └── test_abc.py
But then pytest barfs with the identical error you've shown, and so I just rename integration/test_abc.py to integration/test_abc_integration.py and move on with my day. Like this:
tests/
├── unit
│ └── test_abc.py
├── integration
│ └── test_abc_integration.py
Is it annoying? A little. How long does the fix take? 5 whole seconds.
P.S. You might have to remove __pycache__ directories or you .pyc files for the first run after you get the error you've posted about (if you don't you'll just get the same error again even if you rename).
P.S.S. You can stop the Cpython interpreter (and most others) from writing out __pycache__ and .pyc files by calling python -B -m pytest .... The -B option makes the interpreter not save the bytecode to your filesystem. This results in some performance penalty whenever you run your test suite, but the penalty is usually very small (milage may vary). I typically use this option because I don't like the clutter in my repositories and the performance loss is typically negligible.
I have a package with a directory "tests" in which I'm storing my unit tests. My package looks like:
.
├── LICENSE
├── models
│ └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│ ├── db
│ │ └── test_employee.py
│ └── test_tc.py
└── todo.txt
From my package directory, I want to be able to find both tests/test_tc.py and tests/db/test_employee.py. I'd prefer not to have to install a third-party library (nose or etc) or have to manually build a TestSuite to run this in.
Surely there's a way to tell unittest discover not to stop looking once it's found a test? python -m unittest discover -s tests will find tests/test_tc.py and python -m unittest discover -s tests/db will find tests/db/test_employee.py. Isn't there a way to find both?
In doing a bit of digging, it seems that as long as deeper modules remain importable, they'll be discovered via python -m unittest discover. The solution, then, was simply to add a __init__.py file to each directory to make them packages.
.
├── LICENSE
├── models
│ └── __init__.py
├── README.md
├── requirements.txt
├── tc.py
├── tests
│ ├── db
│ │ ├── __init__.py # NEW
│ │ └── test_employee.py
│ ├── __init__.py # NEW
│ └── test_tc.py
└── todo.txt
So long as each directory has an __init__.py, python -m unittest discover can import the relevant test_* module.
If you're okay with adding a __init__.py file inside tests, you can put a load_tests function there that will handle discovery for you.
If a test package name (directory with __init__.py) matches the
pattern then the package will be checked for a 'load_tests' function. If
this exists then it will be called with loader, tests, pattern.
If load_tests exists then discovery does not recurse into the package,
load_tests is responsible for loading all tests in the package.
I'm far from confident that this is the best way, but one way to write that function would be:
import os
import pkgutil
import inspect
import unittest
# Add *all* subdirectories to this module's path
__path__ = [x[0] for x in os.walk(os.path.dirname(__file__))]
def load_tests(loader, suite, pattern):
for imp, modname, _ in pkgutil.walk_packages(__path__):
mod = imp.find_module(modname).load_module(modname)
for memname, memobj in inspect.getmembers(mod):
if inspect.isclass(memobj):
if issubclass(memobj, unittest.TestCase):
print("Found TestCase: {}".format(memobj))
for test in loader.loadTestsFromTestCase(memobj):
print(" Found Test: {}".format(test))
suite.addTest(test)
print("=" * 70)
return suite
Pretty ugly, I agree.
First you add all subdirectories to the test packages's path (Docs).
Then, you use pkgutil to walk the path, looking for packages or modules.
When it finds one, it then checks the module members to see whether they're classes, and if they're classes, whether they're subclasses of unittest.TestCase. If they are, the tests inside the classes are loaded into the test suite.
So now, from inside your project root, you can type
python -m unittest discover -p tests
Using the -p pattern switch. If all goes well, you'll see what I saw, which is something like:
Found TestCase: <class 'test_tc.TestCase'>
Found Test: testBar (test_tc.TestCase)
Found Test: testFoo (test_tc.TestCase)
Found TestCase: <class 'test_employee.TestCase'>
Found Test: testBar (test_employee.TestCase)
Found Test: testFoo (test_employee.TestCase)
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.001s
OK
Which is what was expected, each of my two example files contained two tests, testFoo and testBar each.
Edit: After some more digging, it looks like you could specify this function as:
def load_tests(loader, suite, pattern):
for imp, modname, _ in pkgutil.walk_packages(__path__):
mod = imp.find_module(modname).load_module(modname)
for test in loader.loadTestsFromModule(mod):
print("Found Tests: {}".format(test._tests))
suite.addTests(test)
This uses the loader.loadTestsFromModule() method instead of the loader.loadTestsFromTestCase() method I used above. It still modifies the tests package path and walks it looking for modules, which I think is the key here.
The output looks a bit different now, since we're adding a found testsuite at a time to our main testsuite suite:
python -m unittest discover -p tests
Found Tests: [<test_tc.TestCase testMethod=testBar>, <test_tc.TestCase testMethod=testFoo>]
Found Tests: [<test_employee.TestCase testMethod=testBar>, <test_employee.TestCase testMethod=testFoo>]
======================================================================
....
----------------------------------------------------------------------
Ran 4 tests in 0.000s
OK
But we still get the 4 tests we expected, in both classes, in both subdirectories.
The point using init.py, is that one may encounters side effects, like file not being the script file path. Using FOR DOS command can help (not found of DOS commands, but sometimes it helps
setlocal
set CWD=%CD%
FOR /R %%T in (*_test.py) do (
CD %%~pT
python %%T
)
CD %CWD%
endlocal
/R allows for walkthrough the hierarchy from current folder.
(expr) allows for selecting test files (I use _test.py)
%%~pT is $(dirname $T) in shell.
I saved and restore my original directory, as the .bat leaves me where it ends
setlocal ... endlocal to not pollute my environment with CWD.
When playing with nose and trying to combine it with cython I can't quite get it all to work the way I'd like. The code is organised like this:
.
├── setup.py
└── src
├── calc
│ ├── factorial.py
│ ├── __init__.py
│ └── tests.py
└── cycalc
├── tests.py
└── triangle.pyx
Each of the tests.py contains 2 tests, one succeeds, one fails. The result of running setup.py nosetests is that only calc/tests.py are run. If I after this run nosetests3 src/cycalc the two tests in cycalc/tests.py are run. However, if I clean up all build files it fails because cycalc/triangle.pyx hasn't been built into a shared lib.
Then I tried adding the file src/cycalc/__init__.py, now setup.py nosetests picks up cycalc/tests.py but it fails to find the required module, it was placed in src.
How do I arrange my cython source and tests to make setup.py nosetests find everything it needs?
For nose to run your tests automatically you should add them into a folder called tests containing all your tests. Like this:
.
|-setup.py
|-src
|---calc
|------factorial.py
|------__init__.py
|---cycalc
|------triangle.pyx
|------__init__.py
|-tests
|---__init__.py
|---test_calc.py
|---test_cycalc.py
This way both tests will be run automatically with everything in the same path. If you remove the built files you need to run python setup.py build before the tests will work again.
I'm setting up a directory structure for my Django app to separate functional and unit tests. I am using nose as the test runner for my Django project.
At the root of the Django project, I have a folder called "tests" that has this structure:
tests
├── __init__.py
├── functional
│ ├── __init__.py
└── unit
├── __init__.py
├── data.py
├── tests.py
If I want to run just the unit tests, should I not be able to use the following from the project root:
$ nosetests tests.unit
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
As you can see, this doesn't find the tests in the tests.py file.
However, when I run using the directory structure, the tests are found as they should be:
$ nosetests tests/unit/
E
# .. Some errors I expected because settings are not initialized when called this way
-----------------
Ran 1 test in 0.001s
FAILED (errors=1)
What am I missing? My main issue is that I have a setup function in tests.unit.__init__.py that should be called for creating the data in the test DB for the upcoming tests.
Thanks
This all depends on what kind of code is in tests/unit/__init__.py
When you say
nosetests tests.unit
You are pointing to unit/__init__.py not the directory unit/ thus if you had no tests in your __init__.py module then nothing would be run. So it is understandable when you say you used the directory path and then your tests started working.
You mention
What am I missing? My main issue is that I have a setup function in
tests.unit.init.py that should be called for creating the data in
the test DB for the upcoming tests.
It is likely that although you have a setup function in __init__.py you may have not ever imported your test functions into __init__.py
One quick fix to this would be to add this line in __init__.py
from tests.unit.tests import *
That said it is really not very wise to be putting any code in __init__.py at all and if you have code that returns some kind of configuration data I would recommend creating a new library module with functions that will return configuration data to your tests
I have a module that sits in a namespace. Should tests and data the tests rely on go in the namespace or in the top level where setup.py sites?
./company/__init__.py
./company/namespace/__init__.py
./company/namespace/useful.py
./company/namespace/test_useful.py
./company/namespace/test_data/useful_data.xml
./setup.py
or
./company/__init__.py
./company/namespace/__init__.py
./company/namespace/useful.py
./test_useful.py
./test_data/useful_data.xml
./setup.py
Does the question amount to whether tests should be installed or not?
The Sample Project stores the tests outside the module.
The directory structure looks like this:
├── data
│ └── data_file
├── MANIFEST.in
├── README.rst
├── sample
│ ├── __init__.py
│ └── package_data.dat
├── setup.cfg
├── setup.py
└── tests
├── __init__.py
└── test_simple.py
Related: The Packing Guide: https://packaging.python.org/en/latest/
Hint: Don't follow the "The Hitchhiker's Guide to Packaging". It has not been updated since 2010!
(do not confuse both pages. The "The Hitchhiker’s Guide to Python" is a very solid book)
You should put your test module inside the module it tests according to The Hitchhiker's Guide to Packaging.
Here is their example:
TowelStuff/
bin/
CHANGES.txt
docs/
LICENSE.txt
MANIFEST.in
README.txt
setup.py
towelstuff/
__init__.py
location.py
utils.py
test/
__init__.py
test_location.py
test_utils.py
This way your module will be distributed with its tests and users can use them to verify that it works with their set up.
See http://the-hitchhikers-guide-to-packaging.readthedocs.org/en/latest/creation.html.
I personally create a single tests package as a sub package of the main package for a few reasons:
If tests is in parallel with the root package there's an off chance you, or a user may misconfigure setup.py and accidentally expose a global package named tests that will cause a great deal of confusion and headache until you realize what has happened. Putting it in the main module solves this as it's now under a (hopefully) globally unique namespace.
I don't like putting a test module within user package because test runners have to search through production code. This is probably not a problem for most. But, if you happen to be a hardware test engineer, you probably use the word 'test' a lot in your production code and don't want the unit test runner to pick that stuff up. It's much easier if all the tests are in one place separate from the production code.
I can further subdivide my tests folder into the types of tests, such as unit, functional and integration. My functional tests tend to have dependencies on weird proprietary hardware, data or are slow. So it's easy for me to continuously run just the fast unit test folder as I develop.
It can sometimes be convenient to have the tests be inside of the same package hierarchy as what it is testing.
Overall though, I think it's important to think for yourself about what's best for your particular problem domain after taking everyone's advice into account. 'Best practices' are great starting points, not end points, for developing a process.