I know what pytest fixtures do and how they work, but they do things before and after the test. How do you do things inside the test in a composable fashion? I need to run tests composed of various smaller functions:
#pytest.mark.django_db
def my_example_test():
# Check if user does not exist
assert not Users.objects.filter(email='foo#bar.com').exists()
# Do a bunch of things to sign up and commit a user to the db
sign_up_routine()
# Check if user exists.
assert Users.objects.filter(email='foo#bar.com').exists()
# Checkout a shopping cart
checkout_shopping_cart(item="toothpaste", qty=10)
# do some more checks
...
Now, in my case, fixture doesn't work because it runs even before the test case starts. In general, I want to compose hundreds of tests like this:
Run a bunch of assert statements
Run a composable routine <--- how? function? fixture?
Assert more conditions
Run a different routine
What is a good way to structure composable tests like this in pytest? I am thinking of just writing a bunch of functions and give them database access?
I am sorry if this is just an obvious solution to run functions but I thought there was a pytest way to do this.
I think the short answer you're looking for is just use plain old functions! There's nothing wrong with that. If you want these reusable chunks of code to have access to other fixtures, just pass them through on invocation.
#pytest.mark.fixture
def db_session():
...
def create_some_users(session):
...
def my_test(db_session):
expected = ...
create_some_users(db_session)
actual = do_thing()
assert actual == expected
I like to think of tests with the AAA pattern - Arrange, Act, & Assert. First we get the universe in order, then we fire our cannon off at it, and finally we check to see that everything is how we'd expect it to be at the end. It's an ideal world if all tests are kept simple like this. Fixtures are pytest's way of managing and sharing sets of resources and the instructions to arrange them in some way. This is why they always run at start and -b/c we often want to do some related disposal afterwards- at end. And a nice side-effect is you can more explicitly state the dependencies of a given test within its declaration and can move common surrounding (beginning & end) "Arrange" code so that the test is more easily parsed as "do this X and expect Y".
For what you're looking for, you'd have to have some way to tell pytest when to run your reusable thing as it could be at any midpoint within the test function and at that point, mind as well just use a normal function. For example, you could write a fixture that returns a callable(function) and then just invoke the fixture with (..), but there's not a ton of difference. You could also have fixtures return classes and encapsulate reusable logic in that way. Any of these things would work fine.
I restructured the test with fixtures as follows. Instead of running one test with steps in a linear fashion, I read thoroughly through the fixtures documentation to end up with this:
#pytest.fixture(scope="function")
def sign_up_user(db)
# Check if user does not exist
assert not Users.objects.filter(email='foo#bar.com').exists()
# Do a bunch of things to sign up and commit a user to the db
# that were part of the sign_up_routine() here. Just showing an example below:
client = Client()
resp = client.post('url', kwargs={form:form})
#pytest.fixture(scope="function")
def assert_user_exists(db, sign_up_user):
# Check if user exists. You can imagine a lot more things to assert here, just example from my original post here.
assert Users.objects.filter(email='foo#bar.com').exists()
#pytest.fixture(scope="function")
def checkout_shopping_cart(db, assert_user_exists):
# Checkout shopping cart with 10 quantity of toothpaste
...
def test_order_in_database(db, checkout_shopping_cart):
# Check if order exists in the database
assert Orders.objects.filter(order__user__email='foo#bar.com').exists()
# This is the final test that calls all previous fixtures.
# Now, fixtures can be used to compose tests in various ways. For example, repeat 'sign_up_user' but checkout toothbrush instead of toothpaste.
I think this is pretty clean, not sure if it this is the intended way to use pytest. I welcome feedback. I can now compose smaller bits of tests that can run as fixtures by calling other fixtures in a long chain.
This is a toy example but you can imagine testing a lot of conditions in the database in each of these fixtures. Please note db fixture is needed for django-pytest package for database to work properly in fixtures. Otherwise, you'll get errors that won't be obvious ("use django_db mark" which doesn't fix the problem), see here: pytest django: unable to access db in fixture teardown
Also, the pytest.fixture scope must be function so each fixture runs again instead of caching.
More reading here: https://docs.pytest.org/en/6.2.x/fixture.html#running-multiple-assert-statements-safely
I'm new to Robot framework and trying to better understand the concepts and usage in Suite Setup and Suite Teardown when the test's folder structure is not "flat". After a long search on the web, the Robot framework user guide - executing tests section and also this question which is similar but not exactly my situation, I still did not find any solution, so here we go.
My project now contains the following files:
_init__.robot that contains the Suite Setup & Suite Teardown "definitions", as follows:
*** Settings ***
Library /path/to/some/python/file.py
Suite Setup myCustomSuiteSetup
Suite Teardown myCustomSuiteTeardown
*** Keywords ***
myCustomSuiteSetup
${ret_code} = run keyword MySuiteSetupKeyword
should be eqaul as integers ${ret_code} 0
myCustomSuiteTeardown
${ret_code} = run keyword MySuiteTeardownKeyword
should be eqaul as integers ${ret_code} 0
Where myCustomSuiteTeardown and MySuiteTeardownKeyword are keywords "linked" to some Python functions in the file /path/to/some/python/file.py.
The 4 suite files in my project are currently arranged like so:
|--tests
|----suite_1.robot
|----suite_2.robot
|----suite_3.robot
|----suite_4.robot
|----__init__.robot
Now, the purpose (and usage) of the Suite Setup & Suite Teardown is that the Suite Setup will run at the beginning of the run of the ENTIRE tests folder, i.e.- before the first test case of the first suite, which in this case is suite_1.robot and the Suite Teardown will run after the last test case of the last suite, which in this case it is suite_4.robot.
For this to happen, I simply invoke all the suites as follows (from within one folder "above" the tests folder):
robot tests
So far so good.
Now my question is as follows: Actually I wish to "re-arrange" the folder's structure of the test files, like so:
|--tests
|----testGroup1
|--------suite_1.robot
|--------suite_2.robot
|----testGroup2
|--------suite_3.robot
|--------suite_4.robot
|--__init__.robot <----- Where the __init__.robot file should be placed now ?
Meaning, to "gather" test suites to sub-folders, nevertheless, I still wish to keep the usage of the Suite Setup & Suite Teardown as before,i.e.- upon invocation of every possible subset of the test suites under the "root" folder tests, Suite Setup & Suite Teardown MUST be the first and last (respectively) "steps" to be executed, meaning, for example, let say I wish to run suite_3.robot & suite_4.robot, then now, Suite Setup should be called before the first test case in suite_3.robot and Suite Teardown should be called after the last test case in the suite_4.robot. Also, of course, I wish to keep only a single copy of the __init__.robot file - i.e. - not keep two similar copies of the __init__.robot in each subfolder, testGroup1 & testGroup2. When I did this, it worked, but this is not the (proper) way I wish to do that.
So my questions are:
Where do I need to place the __init__.robot file?
In case I wish to run,for instance, only the two test suites within testGroup2 (i.e.- suite_3.robot & suite_4.robot), what command do I need to use?
Of course, in case it is not the "correct" way (approach) to achieve my objectives (single and unified Suite Setup & Suite Teardown for every test suites subset) - please advice how should it be done.
Note: I'm using Robot framework 3.1.2 (Python 3.5.2 on Linux)
Your placement of the initialization file in the tests folder that contains both testGroup1 and testGroup2 is correct. This way, if you run at least one test that's somewhere in tests or within a sub folder of it, the Suite Setup will run before all such tests and Suite Teardown afterwards.
To force the Suite Setup and Suite Teardown to run, you must keep the parent folder (tests) as the folder you run. It is good practice to always run the folder that contains all of your tests. That way, you don't miss any suite setups or teardowns by mistake.
To then run only select tests, use the options --include or --exclude for tags, or --suite or --test.
In your example, you would run the following:
robot --suite suite_3 --suite suite_4 tests, to run only suite_3 and suite_4
Or, you can use fully qualified suite names:
robot --suite tests.testGroup2.suite_3 --suite tests.testGroup2.suite_4 tests
I am using pytest. I like the way I call pytest (re-try the failed tests first, verbose, grab and show serial output, stop at first failure):
pytest --failed-first -v -s -x
However there is one more thing I want:
I want pytest to run the new tests (ie tests never tested before) immediately after the --failed-first ones. This way, when working with tests that are long to perform, I would get most relevant information as soon as possible.
Any way to do that?
This may not be directly what you are asking about, but, my understanding is that the test execution order is important for you when you create new tests during development.
Since you are already working with these "new" tests, the pytest-ordering plugin might be a good option to consider. It allows you to influence the execution order by decorating your tests with #pytest.mark.first, #pytest.mark.second etc decorators.
pytest-ordering is able to change the execution order by using a pytest_collection_modifyitems hook. There is also pytest-random-order plugin which also uses the same hook to control/change the order.
You can also have your own hook defined and adjusted to your specific needs. For example, here another hook is used to shuffle the tests:
Dynamically control order of tests with pytest
For anyone coming to this now, pytest added a --new-first option to run new tests before all other tests. It can be combined with --failed-first to run new and failed tests. For test-driven development, I've found it helpful to use these options with pytest-watch, which I described in my blog.
I am using nosetests --with-coverage to test and see code coverage of my unit tests. The class that I test has many external dependencies and I mock all of them in my unit test.
When I run nosetests --with-coverage, it shows a really long list of all the imports (including something I don't even know where it is being used).
I learned that I can use .coveragerc for configuration purposes but it seems like I cannot find a helpful instruction on the web.
My questions are..
1) In which directory do I need to add .coveragerc? How do I specify the directories in .coveragerc? My tests are in a folder called "tests"..
/project_folder
/project_folder/tests
2)It is going to be a pretty long list if I were to add each in omit= ...
What is the best way to only show the class that I am testing with the unittest in the coverage report?
It would be nice if I could get some beginner level code examples for .coveragerc. Thanks.
The simplest way to direct coverage.py's focus is to use the source option, usually source=. to indicate that you only want to measure code in the current working tree.
You can also use the --cover-package=PACKAGE option. For example:
nosetests --with-coverage --cover-package=module_you_are_testing
See http://nose.readthedocs.org/en/latest/plugins/cover.html for details.
I'm using Python's built-in unittest module and I want to write a few tests that are not critical.
I mean, if my program passes such tests, that's great! However, if it doesn't pass, it's not really a problem, the program will still work.
For example, my program is designed to work with a custom type "A". If it fails to work with "A", then it's broken. However, for convenience, most of it should also work with another type "B", but that's not mandatory. If it fails to work with "B", then it's not broken (because it still works with "A", which is its main purpose). Failing to work with "B" is not critical, I will just miss a "bonus feature" I could have.
Another (hypothetical) example is when writing an OCR. The algorithm should recognize most images from the tests, but it's okay if some of them fails. (and no, I'm not writing an OCR)
Is there any way to write non-critical tests in unittest (or other testing framework)?
As a practical matter, I'd probably use print statements to indicate failure in that case. A more correct solution is to use warnings:
http://docs.python.org/library/warnings.html
You could, however, use the logging facility to generate a more detailed record of your test results (i.e. set your "B" class failures to write warnings to the logs).
http://docs.python.org/library/logging.html
Edit:
The way we handle this in Django is that we have some tests we expect to fail, and we have others that we skip based on the environment. Since we can generally predict whether a test SHOULD fail or pass (i.e. if we can't import a certain module, the system doesn't have it, and so the test won't work), we can skip failing tests intelligently. This means that we still run every test that will pass, and have no tests that "might" pass. Unit tests are most useful when they do things predictably, and being able to detect whether or not a test SHOULD pass before we run it makes this possible.
Asserts in unit tests are binary: they will work or they will fail, there's no mid-term.
Given that, to create those "non-critical" tests you should not use assertions when you don't want the tests to fail. You should do this carefully so you don't compromise the "usefulness" of the test.
My advice to your OCR example is that you use something to record the success rate in your tests code and then create one assertion like: "assert success_rate > 8.5", and that should give the effect you desire.
Thank you for the great answers. No only one answer was really complete, so I'm writing here a combination of all answers that helped me. If you like this answer, please vote up the people who were responsible for this.
Conclusions
Unit tests (or at least unit tests in unittest module) are binary. As Guilherme Chapiewski says: they will work or they will fail, there's no mid-term.
Thus, my conclusion is that unit tests are not exactly the right tool for this job. It seems that unit tests are more concerned about "keep everything working, no failure is expected", and thus I can't (or it's not easy) to have non-binary tests.
So, unit tests don't seem the right tool if I'm trying to improve an algorithm or an implementation, because unit tests can't tell me how better is one version when compared to the other (supposing both of them are correctly implemented, then both will pass all unit tests).
My final solution
My final solution is based on ryber's idea and code shown in wcoenen answer. I'm basically extending the default TextTestRunner and making it less verbose. Then, my main code call two test suits: the critical one using the standard TextTestRunner, and the non-critical one, with my own less-verbose version.
class _TerseTextTestResult(unittest._TextTestResult):
def printErrorList(self, flavour, errors):
for test, err in errors:
#self.stream.writeln(self.separator1)
self.stream.writeln("%s: %s" % (flavour,self.getDescription(test)))
#self.stream.writeln(self.separator2)
#self.stream.writeln("%s" % err)
class TerseTextTestRunner(unittest.TextTestRunner):
def _makeResult(self):
return _TerseTextTestResult(self.stream, self.descriptions, self.verbosity)
if __name__ == '__main__':
sys.stderr.write("Running non-critical tests:\n")
non_critical_suite = unittest.TestLoader().loadTestsFromTestCase(TestSomethingNonCritical)
TerseTextTestRunner(verbosity=1).run(non_critical_suite)
sys.stderr.write("\n")
sys.stderr.write("Running CRITICAL tests:\n")
suite = unittest.TestLoader().loadTestsFromTestCase(TestEverythingImportant)
unittest.TextTestRunner(verbosity=1).run(suite)
Possible improvements
It should still be useful to know if there is any testing framework with non-binary tests, like Kathy Van Stone suggested. Probably I won't use it this simple personal project, but it might be useful on future projects.
Im not totally sure how unittest works, but most unit testing frameworks have something akin to categories. I suppose you could just categorize such tests, mark them to be ignored, and then run them only when your interested in them. But I know from experience that ignored tests very quickly become...just that ignored tests that nobody ever runs and are therefore a waste of time and energy to write them.
My advice is for your app to do, or do not, there is no try.
From unittest documentation which you link:
Instead of unittest.main(), there are
other ways to run the tests with a
finer level of control, less terse
output, and no requirement to be run
from the command line. For example,
the last two lines may be replaced
with:
suite = unittest.TestLoader().loadTestsFromTestCase(TestSequenceFunctions)
unittest.TextTestRunner(verbosity=2).run(suite)
In your case, you can create separate TestSuite instances for the criticial and non-critical tests. You could control which suite is passed to the test runner with a command line argument. Test suites can also contain other test suites so you can create big hierarchies if you want.
Python 2.7 (and 3.1) added support for skipping some test methods or test cases, as well as marking some tests as expected failure.
http://docs.python.org/library/unittest.html#skipping-tests-and-expected-failures
Tests marked as expected failure won't be counted as failure on a TestResult.
There are some test systems that allow warnings rather than failures, but test_unit is not one of them (I don't know which ones do, offhand) unless you want to extend it (which is possible).
You can make the tests so that they log warnings rather than fail.
Another way to handle this is to separate out the tests and only run them to get the pass/fail reports and not have any build dependencies (this depends on your build setup).
Take a look at Nose : http://somethingaboutorange.com/mrl/projects/nose/0.11.1/
There are plenty of command line options for selecting tests to run, and you can keep your existing unittest tests.
Another possibility is to create a "B" branch (you ARE using some sort of version control, right?) and have your unit tests for "B" in there. That way, you keep your release version's unit tests clean (Look, all dots!), but still have tests for B. If you're using a modern version control system like git or mercurial (I'm partial to mercurial), branching/cloning and merging are trivial operations, so that's what I'd recommend.
However, I think you're using tests for something they're not meant to do. The real question is "How important to you is it that 'B' works?" Because your test suite should only have tests in it that you care whether they pass or fail. Tests that, if they fail, it means the code is broken. That's why I suggested only testing "B" in the "B" branch, since that would be the branch where you are developing the "B" feature.
You could test using logger or print commands, if you like. But if you don't care enough that it's broken to have it flagged in your unit tests, I'd seriously question whether you care enough to test it at all. Besides, that adds needless complexity (extra variables to set debug level, multiple testing vectors that are completely independent of each other yet operate within the same space, causing potential collisions and errors, etc, etc). Unless you're developing a "Hello, World!" app, I suspect your problem set is complicated enough without adding additional, unnecessary complications.
You could write your test so that they count success rate.
With OCR you could throw at code 1000 images and require that 95% is successful.
If your program must work with type A then if this fails the test fails. If it's not required to work with B, what is the value of doing such a test ?