We Keep Coding

Using pytest fixture from common test function

I have some unittest-based code that currently looks like this:
class TestMain(TestCase):
def run_prog(self, args):
with TemporaryFile() as stdout:
old_stdout = sys.stdout
try:
main.main()
stdout.seek(0)
stdout_data = stdout.read()
finally:
sys.stdout = old_stdout
return stdout_data
def test_one(self):
out = self.run_prog(...)
def test_two(self):
out = self.run_prog(...)
def test_three(self):
out = self.run_prog(...)
run_prog invokes the "main" program under test and manually captures its stdout.
I'm in the process of converting this project to pytest, but this last piece has pushed the limits of my understanding of pytest fixtures.
I understand that pytest has full support for capturing stdout/stderr and I would like to leverage this.
The problem is, their examples work on a test-function level:
def test_myoutput(capfd):
do_something
captured = capsys.readouterr()
assert captured.out == "hello\n"
assert captured.err == "world\n"
In my case, run_prog is used 42 times, so I'm trying to use the fixture starting at run_prog -- the calling functions ideally don't need to bother with capsys/capfd.
Is there a way to "invoke" the fixture from my run_prog helper? Or do I need to add capfd to all 42 tests and pass it to run_prog?

You can define an autouse fixture that will store the CaptureFixture object (returned by the capsys fixture) as an instance property:
class TestMain(TestCase):
#pytest.fixture(autouse=True)
def inject_capsys(self, capsys):
self._capsys = capsys
def run_prog(self, args):
main.main()
return self._capsys.out
def test_out(self):
assert self.run_prog('spam') == 'eggs'
The TestMain.inject_capsys fixture will be rerun for each test, guaranteeing the test isolation (no output from test_one will be leaked in test_two etc).

Here's a slight variation on hoefling's answer that gives a little more control over the scope of the capsys fixture.
It uses request.getfixturevalue() to retrieve the fixture at function invocation time:
import pytest
import sys
class TestMain:
#pytest.fixture(autouse=True)
def inject_request(self, request):
self.request = request
def run_prog(self, message):
capfd = self.request.getfixturevalue('capfd')
sys.stdout.write(message)
captured = capfd.readouterr()
assert captured.out == message
def test_one(self):
self.run_prog("Hello world!")

How to test required argparse arguments during unittests?

I'm doing TDD tests for argparser. How can I test arguments with the option required?I need to test all options like:
too many arguments,
no arguments are given,
the wrong argument is given.
I can raise SystemExit, but this is not really what I need:
def test_no_arguments(self):
with patch.object(sys, 'exit') as mock_method:
self.parser.parse_arguments()
self.assertTrue(mock_method.called)
However, without raising system exit I have always errors like this:
zbx-check-mount.py
class CommandLine:
def __init__(self):
self.args_parser = argparse.ArgumentParser(description="Monitoring mounted filesystems",
formatter_class=argparse.RawTextHelpFormatter)
self.parsed_args = None
self.add_arguments()
def add_arguments(self):
"""
Add arguments to parser.
"""
try:
self.args_parser._action_groups.pop() # pylint: disable=protected-access
required = self.args_parser.add_argument_group('required arguments')
required.add_argument('--fs_name', required=True, help='Given filesystem')
except argparse.ArgumentError as err:
log.error('argparse.ArgumentError: %s', err)
sys.exit(1)
def parse_arguments(self, args=None):
"""
Parse added arguments. Then run private method to return values
"""
self.parsed_args = self.args_parser.parse_args()
return self.parsed_args.fs_name,
tests
from pyfakefs.fake_filesystem_unittest import TestCase
import os
import sys
try:
from StringIO import StringIO
except ImportError:
from io import StringIO
if sys.version_info[0] == 3:
from unittest.mock import MagicMock, patch
else:
from mock import MagicMock, patch
sys.path.extend([os.path.join(os.path.dirname(os.path.abspath(__file__)),'..','..', "bin")])
module_name = __import__('zbx-check-mount')
class TestCommandLine(TestCase):
def setUp(self):
"""
Method called to prepare the test fixture. This is called immediately before calling the test method
"""
self.parser = module_name.CommandLine()
def test_no_arguments(self):
opts = self.parser.parse_arguments([])
assert opts.fs_name
def tearDown(self):
"""
Method called immediately after the test method has been called and the result recorded.
"""
pass
How to avoid this situation and test other options?

In def parse_arguments(self, args=None):, you should pass args on to the parser, as in:
self.args_parser.parse_args(args)
parse_args() parses sys.argv[1:], or if the given argument is None. Otherwise it parses the provided list.
In a full distribution of python there's a unittest file for argparse (test_argparse.py). It's somewhat complex, defining a subclass of ArgumentParser that captures errors and redirects error messages.
Testing argparse is tricky because it looks at sys.argv, with the unittest scripts also use. And it usually tries to exit on errors. This has been discussed in a number of SO questions already.

If I'm interpreting your symptoms correctly, you are having problems in the test harness because your monkey patched implementation of sys.exit actually returns, which the argparse library is not expecting.
Introducing a side_effect that raises an exception, which you can then trap and verify in the unit test, may be sufficient to get around the problem.

How to use logging, pytest fixture and capsys?

I am trying to unit-test some algorithm that uses logging library.
I have a fixture that creates a logger.
In my 1st test case, I do not use this fixture and uses a print to log to stdout. This test case passes.
In my 2nd test case, I use this fixture, but not as documented in pytest doc. I just call the associated function in my test to get the logger. Then I use the logger to log to stdout. This test case passes.
In my 3rd test case, I use this fixture as documented in pytest doc. The fixture is passed as an argument to the test function. Then I use the logger to log to stdout. This test case fails! It does not find anything in stdout. But in the error message, it says that my log is in the captured stdout call.
What am I doing wrong?
import pytest
import logging
import sys
#pytest.fixture()
def logger():
logger = logging.getLogger('Some.Logger')
logger.setLevel(logging.INFO)
stdout = logging.StreamHandler(sys.stdout)
logger.addHandler(stdout)
return logger
def test_print(capsys):
print 'Bouyaka!'
stdout, stderr = capsys.readouterr()
assert 'Bouyaka!' in stdout
# passes
def test_logger_without_fixture(capsys):
logger().info('Bouyaka!')
stdout, stderr = capsys.readouterr()
assert 'Bouyaka!' in stdout
# passes
def test_logger_with_fixture(logger, capsys):
logger.info('Bouyaka!')
stdout, stderr = capsys.readouterr()
assert 'Bouyaka!' in stdout
# fails with this error:
# > assert 'Bouyaka!' in stdout
# E assert 'Bouyaka!' in ''
#
# tests/test_logging.py:21: AssertionError
# ---- Captured stdout call ----
# Bouyaka!
There is no change if I reorder the test cases by the way.

Thanks a lot for your ideas!
Reverse logger, capsys, make logger request the capsys fixture and use capfd do not change anything.
I tried pytest-catchlog plugin and it works fine!
import pytest
import logging
#pytest.fixture()
def logger():
logger = logging.getLogger('Some.Logger')
logger.setLevel(logging.INFO)
return logger
def test_logger_with_fixture(logger, caplog):
logger.info('Bouyaka!')
assert 'Bouyaka!' in caplog.text
# passes!
In my original tests, I logged to stdout and stderr and captured them.
This is an even better solution, as I do not need this tweak to check that my logs work fine.
Well, now I just need to rework all my tests to use caplog, but this is my own business ;)
The only thing left, now that I have a better solution, is to understand what is wrong in my original test case def test_logger_with_fixture(logger, capsys).

As of pytest 3.3, the functionality of capturing log message was added to pytest core. This is supported by the caplog fixture:
def test_baz(caplog):
func_under_test()
for record in caplog.records:
assert record.levelname != 'CRITICAL'
assert 'wally' not in caplog.text
More information can be found at the documentation

I'm guessing the logger gets created (via the fixture) before the capsys fixture is set up.
Some ideas:
Use the pytest-catchlog plugin
Maybe reverse logger, capsys
Make logger request the capsys fixture
Use capfd which is more lowlevel capturing without altering sys

Python: Write unittest for console print

Function foo prints to console. I want to test the console print. How can I achieve this in python?
Need to test this function, has NO return statement :
def foo(inStr):
print "hi"+inStr
My test :
def test_foo():
cmdProcess = subprocess.Popen(foo("test"), stdout=subprocess.PIPE)
cmdOut = cmdProcess.communicate()[0]
self.assertEquals("hitest", cmdOut)

You can easily capture standard output by just temporarily redirecting sys.stdout to a StringIO object, as follows:
import StringIO
import sys
def foo(inStr):
print "hi"+inStr
def test_foo():
capturedOutput = StringIO.StringIO() # Create StringIO object
sys.stdout = capturedOutput # and redirect stdout.
foo('test') # Call unchanged function.
sys.stdout = sys.__stdout__ # Reset redirect.
print 'Captured', capturedOutput.getvalue() # Now works as before.
test_foo()
The output of this program is:
Captured hitest
showing that the redirection successfully captured the output and that you were able to restore the output stream to what it was before you began the capture.
Note that the code above in for Python 2.7, as the question indicates. Python 3 is slightly different:
import io
import sys
def foo(inStr):
print ("hi"+inStr)
def test_foo():
capturedOutput = io.StringIO() # Create StringIO object
sys.stdout = capturedOutput # and redirect stdout.
foo('test') # Call function.
sys.stdout = sys.__stdout__ # Reset redirect.
print ('Captured', capturedOutput.getvalue()) # Now works as before.
test_foo()

This Python 3 answer uses unittest.mock. It also uses a reusable helper method assert_stdout, although this helper is specific to the function being tested.
import io
import unittest
import unittest.mock
from .solution import fizzbuzz
class TestFizzBuzz(unittest.TestCase):
#unittest.mock.patch('sys.stdout', new_callable=io.StringIO)
def assert_stdout(self, n, expected_output, mock_stdout):
fizzbuzz(n)
self.assertEqual(mock_stdout.getvalue(), expected_output)
def test_only_numbers(self):
self.assert_stdout(2, '1\n2\n')
Note that the mock_stdout arg is passed automatically by the unittest.mock.patch decorator to the assert_stdout method.
A general-purpose TestStdout class, possibly a mixin, can in principle be derived from the above.
For those using Python ≥3.4, contextlib.redirect_stdout also exists, but it seems to serve no benefit over unittest.mock.patch.

If you happen to use pytest, it has builtin output capturing. Example (pytest-style tests):
def eggs():
print('eggs')
def test_spam(capsys):
eggs()
captured = capsys.readouterr()
assert captured.out == 'eggs\n'
You can also use it with unittest test classes, although you need to passthrough the fixture object into the test class, for example via an autouse fixture:
import unittest
import pytest
class TestSpam(unittest.TestCase):
#pytest.fixture(autouse=True)
def _pass_fixtures(self, capsys):
self.capsys = capsys
def test_eggs(self):
eggs()
captured = self.capsys.readouterr()
self.assertEqual('eggs\n', captured.out)
Check out Accessing captured output from a test function for more info.

You can also use the mock package as shown below, which is an example from
https://realpython.com/lessons/mocking-print-unit-tests.
from mock import patch
def greet(name):
print('Hello ', name)
#patch('builtins.print')
def test_greet(mock_print):
# The actual test
greet('John')
mock_print.assert_called_with('Hello ', 'John')
greet('Eric')
mock_print.assert_called_with('Hello ', 'Eric')

The answer of #Acumenus says:
It also uses a reusable helper method assert_stdout, although this helper is specific to the function being tested.
the bold part seems a big drawback, thus I would do the following instead:
# extend unittest.TestCase with new functionality
class TestCase(unittest.TestCase):
def assertStdout(self, expected_output):
return _AssertStdoutContext(self, expected_output)
# as a bonus, this syntactical sugar becomes possible:
def assertPrints(self, *expected_output):
expected_output = "\n".join(expected_output) + "\n"
return _AssertStdoutContext(self, expected_output)
class _AssertStdoutContext:
def __init__(self, testcase, expected):
self.testcase = testcase
self.expected = expected
self.captured = io.StringIO()
def __enter__(self):
sys.stdout = self.captured
return self
def __exit__(self, exc_type, exc_value, tb):
sys.stdout = sys.__stdout__
captured = self.captured.getvalue()
self.testcase.assertEqual(captured, self.expected)
this allows for the much nicer and much more re-usable:
# in a specific test case, the new method(s) can be used
class TestPrint(TestCase):
def test_print1(self):
with self.assertStdout("test\n"):
print("test")
by using a straight forward context manager. (It might also be desirable to append "\n" to expected_output since print() adds a newline by default. See next example...)
Furthermore, this very nice variant (for an arbitrary number of prints!)
def test_print2(self):
with self.assertPrints("test1", "test2"):
print("test1")
print("test2")
is possible now.

You can also capture the standard output of a method using contextlib.redirect_stdout:
import unittest
from contextlib import redirect_stdout
from io import StringIO
class TestMyStuff(unittest.TestCase):
# ...
def test_stdout(self):
with redirect_stdout(StringIO()) as sout:
my_command_that_prints_to_stdout()
# the stream replacing `stdout` is available outside the `with`
# you may wish to strip the trailing newline
retval = sout.getvalue().rstrip('\n')
# test the string captured from `stdout`
self.assertEqual(retval, "whatever_retval_should_be")
Gives you a locally scoped solution. It is also possible to capture the standard error using contextlib.redirect_stderr().

Another variant is leaning on the logging module rather than print(). This module also has a suggestion of when to use print in the documentation:
Display console output for ordinary usage of a command line script or program
PyTest has built-in support for testing logging messages.

Python 3 unittest: How to extract results of tests?

I am using Python's (3.4.1) unittest module for my unit tests.
I load all my testing module files using imports and then run unittest.main():
import unittest
import testing_module1
import testing_module2
# [...]
if __name__ == '__main__':
unittest.main()
This works perfectly for me as it is simple and respect the command line arguments I use to control verbosity or which test(s) to run.
I want to continue to output the same information, but I would like to generate an XML file from the results. I tried xmlrunner (https://github.com/xmlrunner/unittest-xml-reporting/) but:
it does not output as much info to stdout as the standard runner;
it uses a specific format of the XML that doesn't suites me.
I would like to generate the XML (I don't mind doing it manually) with the format I need but with minimal change to how the tests are run.
What are my options?
I could write my own TestRunner but I don't want to re-write everything, I just want to add extra output to the actual runner with minimal code change.
I could inherit unittest.TextTestRunner but I fear that adding XML output to it would require re-writing every methods, loosing the advantage of inheritance in the first place.
I could try to extract the test results after the call to unittest.main() and parse it. The problem here is that unittest.main() seems to exit when it's done so any code after it is not executed.
Any suggestion?
Thanks!

I ended up writing two new classes, inheriting from unittest.TextTestResult and unittest.TextTestRunner. That way, I could run main like that:
unittest.main(testRunner=xmlrunner.XMLTestRunner(...))
I overloaded unittest.TextTestRunner's __init__ and those from unittest.TextTestResult:
addSuccess()
addError()
addFailure()
addSubTest()
For example:
def addSuccess(self, test):
super().addSuccess(test)
[... store the test into list, dictionary, whatever... ]
Since these add*() functions are called with the actual test, I can store them in a global list and parse them at the end of my XMLTestRunner.run():
def run(self, test):
result = super().run(test)
self.save_xml_report(result)
return result
Note that these functions are normally defined in /usr/lib/python3.4/unittest/runner.py.
Warning note: By using an actual object passed unittest.main()'s testRunner argument as shown, the command line arguments given when launching python are ignored. For example, increasing verbose level with -v argument is ignored. This is because the TestProgram class, defined in /usr/lib/python3.4/unittest/main.py, detects if unittest.main() was run with testRunner being a class or an object (see runTests() near the end of the file). If you give just a class like that:
unittest.main(testRunner=xmlrunner.XMLTestRunner)
then command line arguments are parsed. But you pass an instantiated object (like I need), runTests() will just use it as is. I thus had to parse arguments myself in my XMLTestRunner.__init__():
# Similar to what /usr/lib/python3.4/unittest/main.py's TestProgram._getParentArgParser() does.
import argparse
parser = argparse.ArgumentParser(add_help=False)
parser.add_argument('-v', '--verbose', dest='verbosity',
action='store_const', const=2, default=1, # Add default=1, not present in _getParentArgParser()
help='Verbose output')
parser.add_argument('-q', '--quiet', dest='verbosity',
action='store_const', const=0,
help='Quiet output')
parser.add_argument('-f', '--failfast', dest='failfast',
action='store_true',
help='Stop on first fail or error')
parser.add_argument('-c', '--catch', dest='catchbreak',
action='store_true',
help='Catch ctrl-C and display results so far')
parser.add_argument('-b', '--buffer', dest='buffer',
action='store_true',
help='Buffer stdout and stderr during tests')

How does this work for you. Capture the output of unittest, which goes to sys.stderr, in a StringIO. Continue after unittest.main by adding `exit=False'. Read the captured output and process as you want. Proof of concept:
import contextlib
import io
import sys
import unittest
class Mytest(unittest.TestCase):
def test_true(self):
self.assertTrue(True)
#contextlib.contextmanager
def err_to(file):
old_err = sys.stderr
sys.stderr = file
yield
sys.stderr = old_err
if __name__ == '__main__':
result = io.StringIO()
with err_to(result):
unittest.main(exit=False)
result.seek(0)
print(result.read())
This prints (to sys.stdout)
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Note: contextlib has redirect_stdout, but not redirect_stderr. The above is simpler that the contextlib code. The above assumes that there are no exceptions not caught by unittest. See the contextlib.contextmanager doc for adding try: except: finally. I leave that to you.

I have faced the same issue with catching FAIL events from unittest lib. Following big_gie's answer, this code appeared:
File testFileName_1.py
import unittest
class TestClassToTestSth(unittest.TestCase):
def test_One(self):
self.AssertEqual(True, False, 'Hello world')
import unittest
from io import StringIO
import testFileName_1
def suites():
return [
# your testCase classes, for example
testFileName_1.TestClassToTestSth,
testFileName_445.TestClassToTestSomethingElse,
]
class TextTestResult(unittest.TextTestResult):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.slack = Slack('data-engineering-tests')
def addFailure(self, test, err):
super().addFailure(test, err)
# Whatever you want here
print(err, test)
print(self.failures)
class TextTestRunner(unittest.TextTestRunner):
resultclass = TextTestResult
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
loader = unittest.TestLoader()
suite = unittest.TestSuite()
stream = StringIO()
for test_case in suites():
suite.addTests(loader.loadTestsFromTestCase(test_case))
runner = TextTestRunner(stream=stream)
result = runner.run(suite)
stream.seek(0)
print(stream.read())