I am trying to check the exit code I have for a script I'm writing in python3 on a Mac (10.14.4). When I run the test it doesn't fail which I think is wrong. But I can't see what it is that I've got wrong.
The test file looks like this:
import pytest
import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '..')))
import my_script
class TestMyScript():
def test_exit(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
assert pytest_wrapped_e.type == SystemExit
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
self.assertEqual(pytest_wrapped_e.exception.code, 42)
My script looks like:
#!/usr/bin/env python3
import sys
def main():
print('Hello World!')
sys.exit(0)
if __name__ == '__main__':
main()
The output I get is:
$ py.test -v
============================= test session starts ==============================
platform darwin -- Python 3.7.3, pytest-3.10.1, py-1.8.0, pluggy-0.9.0 -- /usr/local/opt/python/bin/python3.7
cachedir: .pytest_cache
rootdir: /Users/robertpostill/software/gateway, inifile:
plugins: shutil-1.6.0
collected 2 items
test/test_git_refresh.py::TestGitRefresh::test_exit PASSED [ 50%]
test/test_git_refresh.py::TestGitRefresh::test_exit_code PASSED [100%]
=========================== 2 passed in 0.02 seconds ===========================
$
I would expect the second test(test_exit_code) to fail as the exit call is getting a code of 0, not 42. But for some reason, the assert is happy whatever value I put in the sys.exit call.
Good question, that's because your Asserts are never called (either of them). When exit() is called the program is done (at least within the with clause), it turns off the lights, packs up its bags, and goes home. No further functions will be called. To see this add an assert before and after you call main:
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
self.assertEqual(0, 1) # This will make it fail
my_script.main()
self.assertEqual(0, 1) # This will never be called because main `exits`
self.assertEqual(pytest_wrapped_e.exception.code, 42)
A test passes if no asserts fail and nothing breaks, so in your case both tests passed because the assert was never hit.
To fix this pull your asserts out of the with statement:
def test_exit_code(self):
with pytest.raises(SystemExit) as pytest_wrapped_e:
my_script.main()
self.assertEqual(pytest_wrapped_e.exception.code, 42)
Though now you will need to fix the pytest syntax because you are missing some other stuff.
See: Testing sys.exit() with pytest
Related
I have set logging.captureWarnings(True) in an application, and would like to test if warnings are logged correctly. I'm having difficulties understanding some of the behavior I'm seeing where tests are influencing each other in ways that I don't quite get.
Here is an example test suite which reproduces the behavior I'm seeing:
test_warning_logs.py
import warnings
import logging
def test_a(caplog):
logging.captureWarnings(True)
logging.basicConfig()
warnings.warn("foo")
assert "foo" in caplog.text
def test_b(caplog):
logging.captureWarnings(True)
logging.basicConfig()
warnings.warn("foo")
assert "foo" in caplog.text
Both tests are identical. When run in isolation (pytest test_warning_logs.py -k test_a, pytest test_warning_logs.py -k test_b), they each pass. When both of them are executed in the same run (pytest test_warning_logs.py), only the first one will pass:
============== test session starts ========================
platform linux -- Python 3.10.2, pytest-7.2.1, pluggy-1.0.0
rootdir: /home/me
plugins: mock-3.10.0, dependency-0.5.1
collected 2 items
test_warning_logs.py .F [100%]
==================== FAILURES =============================
_____________________ test_b ______________________________
caplog = <_pytest.logging.LogCaptureFixture object at 0x7f8041857c40>
def test_b(caplog):
logging.captureWarnings(True)
logging.basicConfig()
warnings.warn("foo")
> assert "foo" in caplog.text
E AssertionError: assert 'foo' in ''
E + where '' = <_pytest.logging.LogCaptureFixture object at 0x7f8041857c40>.text
[...]
Additional Information
First I thought that the commands logging.captureWarnings and logging.basicConfig aren't idempotent, and running them more than once is the issue. But if you remove them from test_b, it still fails.
My current assumption is that it's a pytest issue, because when the code is executed without it, both warnings are logged:
# add this block to the bottom of test_warning_logs.py
if __name__ == '__main__':
from unittest.mock import MagicMock
test_a(MagicMock(text="foo"))
test_b(MagicMock(text="foo"))
$ python test_warning_logs.py
WARNING:py.warnings:/home/me/test_warning_logs.py:9: UserWarning: foo
warnings.warn("foo")
WARNING:py.warnings:/home/me/test_warning_logs.py:17: UserWarning: foo
warnings.warn("foo")
If it is an option to log the warnings using the logging module instead of warnings, then you won't face this issue.
import logging
def test_a(caplog):
logging.captureWarnings(True)
logging.basicConfig()
logging.warning("foo")
assert "foo" in caplog.text
def test_b(caplog):
logging.captureWarnings(True)
logging.basicConfig()
# This will be deprecated eventually, you can use logging.warning() instead
logging.warn("foo")
assert "foo" in caplog.text
I don't know why it doesn't work using warnings module. According to their documentation:
Repetitions of a particular warning for the same source location are
typically suppressed.
So I assumed this is what is happening here, but even calling warnings.resetwarnings() does not change the behavior.
I am trying to generate a coverage report using the coverage-API (https://coverage.readthedocs.io/en/6.3.2/api.html#api).
The simple use-case described on the linked page tells me to wrap my executed code inside the snippet they provide. I am using unittest.main() to execute tests. The below code runs without an error but neither is any report information created nor is print("Done.") executed.
I guess unittest.main() calls a sys.exit() somewhere along the way? How does one use the API to execute all unittest-tests?
Example
import coverage
import unittest
def func(input):
return input
class testInput(unittest.TestCase):
def test_func(self):
self.assertEqual(func(1), 1)
if __name__ == '__main__':
cov = coverage.Coverage()
cov.start()
unittest.main()
cov.stop()
cov.save()
cov.html_report()
print("Done.")
Yes, it looks like unittest.main() is calling sys.exit(). Are you sure you need to use the coverage API? You can skip coverage and most of unittest:
# foo.py
import unittest
def func(input):
return input
class testInput(unittest.TestCase):
def test_func(self):
self.assertEqual(func(1), 1)
Then:
$ python -m coverage run -m unittest foo.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
$ python -m coverage report -m
Name Stmts Miss Cover Missing
--------------------------------------
foo.py 6 0 100%
--------------------------------------
TOTAL 6 0 100%
To get the example to work you can simply put the call to unittest.main() in a try/except statement so the cov.stop() will be called once the unit test has completed.
# testInput.py
import coverage
import unittest
def func(input):
return input
class testInput(unittest.TestCase):
def test_func(self):
self.assertEqual(func(1), 1)
if __name__ == '__main__':
cov = coverage.Coverage()
cov.start()
try:
unittest.main()
except: # catch-all except clause
pass
cov.stop()
cov.save()
cov.html_report()
print("Done.")
This will run as desired, including printing Done. at the end:
python3 testInput.py
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Done.
The HTML coverage report will have been created in the htmlcov directory in the current working directory.
I'm working on the Code-dojo Tennis Kata and I'm stuck with the silliest problem. My pytest stops after running the first test in the file. My test file looks like this:
from tennis import game
import re
def test_game_start(capsys):
MaxVsMimi = game()
MaxVsMimi.result()
out, err = capsys.readouterr()
assert bool(re.search(r'Love : Love', out))
def p1_scores_once(capsys):
MaxVsMimi = game()
MaxVsMimi.score("p1")
MaxVsMimi.result()
out, err = capsys.readouterr()
assert bool(re.match(r'^abc', out))
This is the code:
class game:
def __init__(self):
self.scorep1 = "Love"
self.scorep2 = "Love"
def result(self):
print(self.scorep1, ":", self.scorep2)
and the output:
:stdout:
============================= test session starts ==============================
platform linux -- Python 3.8.6, pytest-6.0.2, py-1.9.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /sandbox
collecting ... collected 1 item
test_tennis.py::test_game_start PASSED [100%]
============================== 1 passed in 0.01s ===============================
Why does it not run the second "p1_scores_once" test?
Thanks!
Pytest, like many testing libraries including unittest require all test functions to have the word test in the start of the name. You can fix your code simply by changing the name of p1_scores_once:
def test_p1_scores_once(capsys):
...
Now the automatic test-finding system will treat the function like a test case.
Suppose I have myclass_test.py with Nose tests with following lines:
import nose
class TestMyClass:
def test_method(self):
assert true
def test_another_method(self):
assert true
if __name__ == "__main__":
nose.runmodule()
So I can run python myclass_test.py and have all of my tests to pass. Everything work as expected.
What should I pass to runmodule() to run a single test (for example test_method)?
If it is impossible with Nose is it possible with some other unit-test framework on the same manner (with runner inside module)?
P.S. I actually run not python myclass_test.py but sage myclass_test.sage. These are SageMath interpreter and SageMath file. SageMath code is basically Python code and myclass_test.sage is finally preparsing to myclass_test.sage.py and run with embedded to Sage Python interpreter.
So it isn't recognizable as module by Python interpreter so passing something like sage myclass_test.sage TestMyClass.test_method or sage myclass_test.sage myclass_test.sage:TestMyClass.test_method or sage myclass_test.sage myclass_test.sage.py:TestMyClass.test_method isn't working (by the same reason one cannot directly import sage file as module). Even though Nose passing argv to runmodule() automatically.
P.P.S. Also I cannot use external runner like nosetests -q -s sage myclass_test.sage:TestMyClass.test_method.
I haven't used nose, but here is a possibility to do it with unittest:
import unittest
class TestMyClass(unittest.TestCase):
def test_method(self):
assert True
def test_another_method(self):
assert True
if __name__ == '__main__':
suite = unittest.TestSuite([TestMyClass('test_method')])
unittest.TextTestRunner().run(suite)
And here is the same with pytest:
import pytest
class TestMyClass:
def test_method(self):
assert True
def test_another_method(self):
assert True
if __name__ == '__main__':
pytest.main([f'{__file__}::TestMyClass::test_method'])
Ok, I also checked with nose, and this seems to work:
import nose
from nose.suite import ContextSuite
class TestMyClass:
def test_method(self):
assert True
def test_another_method(self):
assert True
if __name__ == '__main__':
suite = ContextSuite([TestMyClass.test_method])
nose.runmodule(suite=suite)
though the output is strange - while it certainly runs the test, it says "Ran 0 tests".
I am using py.test to write some tests and in my tests I utilize funcargs. These funcargs have their own setups and teardowns defined in the conftest.py like this:
conftest.py:
def pytest_funcarg__resource_name(request):
def setup():
# do setup
def teardown():
# do teardown
My problem is when someone uses CTRL+C to stop the test executions it leaves everything un-teardowned.
I know there is a hook pytest_keyboard_interrupt but I dont know what to do from there.
Sorry for the noobish question.
You don't provide a full example so maybe i am missing something. But here is an example of how it can work, using the request.cached_setup() helper:
def pytest_funcarg__res(request):
def setup():
print "res-setup"
def teardown(val):
print "res-teardown"
return request.cached_setup(setup, teardown)
def test_hello(res):
raise KeyboardInterrupt()
If you run this with "py.test" you get:
============================= test session starts ==============================
platform linux2 -- Python 2.7.3 -- pytest-2.2.5.dev4
plugins: xdist, bugzilla, pep8, cache
collected 1 items
tmp/test_keyboardinterrupt.py res-setup
res-teardown
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! KeyboardInterrupt !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/home/hpk/p/pytest/tmp/test_keyboardinterrupt.py:10: KeyboardInterrupt
which shows that setup and teardown are called if a KeyboardInterrupt occurs during test execution.