I'm using pytest with pytest-xdist for parallel test running. It doesn't seem to honour the -s option for passing through the standard output to the terminal as the tests are run. Is there any way to make this happen? I realise this could cause the output from the different processes to be jumbled up in the terminal but I'm ok with that.
I found a workaround, although not a full solution. By redirecting stdout to stderr, the output of print statements is displayed. This can be accomplished with a single line of Python code:
sys.stdout = sys.stderr
If placed in conftest.py, it applies to all tests.
I used the followed code:
# conftest.py
import _pytest.capture
def get_capman(plugin_manager):
capman_list = filter(lambda p: isinstance(p, _pytest.capture.CaptureManager), plugin_manager._plugins)
return capman_list[0] if len(capman_list) == 1 else None
def get_xdist_slave(plugin_manager):
# TODO: have no idea how to check isinstance "__channelexec__.SlaveInteractor"
slave_list = filter(lambda p: hasattr(p, 'slaveid'), plugin_manager._plugins)
return slave_list[0] if len(slave_list) == 1 else None
def is_remote_xdist_session(plugin_manager):
return get_xdist_slave(plugin_manager) is not None
def pytest_configure(config):
if is_remote_xdist_session(config.pluginmanager) and get_capman(config.pluginmanager) is not None:
capman = get_capman(config.pluginmanager)
capman._method = "no"
capman.reset_capturings()
capman.init_capturings()
Insert it to conftest.py
The main thing is to be sure that it is remote session, and we have to reconfigure CaptureManager instance.
There one unresolved issue is how to check that remote object has "__channelexec__.SlaveInteractor" type.
Related
I am having trouble testing the output of a custom exception in pytest.
import pytest
class CustomException(Exception):
def __init__(self, extra_message: str):
self.message = extra_message
super().__init__(self.message)
# Should not need to print this as Exception already does this
# print(self.message)
def test_should_get_capsys_output(capsys):
with pytest.raises(CustomException):
raise CustomException("This should be here.")
out, err = capsys.readouterr()
# This should not be true
assert out == ''
assert err == ''
assert 'This' not in out
This example should not pass as I should be able to assert something came out in the output. If I print(self.message) I end up getting the message printed twice when actually used but only then does capsys collect stdout.
I've also tried with variations of caplog and capfd to no avail. [This SO solution] recommends adding an output to the with pytest.raises(...) as info and testing the info but I would have expected capsys to work as well.
Thank you for your time.
I'm sort of confused by what you're asking?
A python exception will stop execution of the current test, even within pytest. So by raising the exception inside a context manager (with statement) we're simulating/catching it before it gets a chance to stop our current test and go to stderr.
I wrote a package that is using multiprocessing.Pool inside one of its functions.
Due to this reason, it is mandatory (as specified in here under "Safe importing of main module") that the outermost calling function can be imported safely e.g. without starting a new process. This is usually achieved using the if __name__ == "__main__": statement as explicitly explained at the link above.
My understanding (but please correct me if I'm wrong) is that multiprocessing imports the outermost calling module. So, if this is not "import-safe", this will start a new process that will import again the outermost module and so on recursively, until everything crashes.
If the outermost module is not "import-safe" when the main function is launched it usually hangs without printing any warning, error, message, anything.
Since using if __name__ == "__main__": is not usually mandatory and the user is usually not always aware of all the modules used inside a package, I would like to check at the beginning of my function if the user complied with this requirement and, if not, raise a warning/error.
Is this possible? How can I do this?
To show this with an example, consider the following example.
Let's say I developed my_module.py and I share it online/in my company.
# my_module.py
from multiprocessing import Pool
def f(x):
return x*x
def my_function(x_max):
with Pool(5) as p:
print(p.map(f, range(x_max)))
If a user (not me) writes his own script as:
# Script_of_a_good_user.py
from my_module import my_function
if __name__ == '__main__':
my_function(10)
all is good and the output is printed as expected.
However, if a careless user writes his script as:
# Script_of_a_careless_user.py
from my_module import my_function
my_function(10)
then the process hangs, no output is produces, but no error message or warning is issued to the user.
Is there a way inside my_function, BEFORE opening Pool, to check if the user used the if __name__ == '__main__': condition in its script and, if not, raise an error saying it should do it?
NOTE: I think this behavior is only a problem on Windows machines where fork() is not available, as explained here.
You can use the traceback module to inspect the stack and find the information you're looking for. Parse the top frame, and look for the main shield in the code.
I assume this will fail when you're working with a .pyc file and don't have access to the source code, but I assume developers will test their code in the regular fashion first before doing any kind of packaging, so I think it's safe to assume your error message will get printed when needed.
Version with verbose messages:
import traceback
import re
def called_from_main_shield():
print("Calling introspect")
tb = traceback.extract_stack()
print(traceback.format_stack())
print(f"line={tb[0].line} lineno={tb[0].lineno} file={tb[0].filename}")
try:
with open(tb[0].filename, mode="rt") as f:
found_main_shield = False
for i, line in enumerate(f):
if re.search(r"__name__.*['\"]__main__['\"]", line):
found_main_shield = True
if i == tb[0].lineno:
print(f"found_main_shield={found_main_shield}")
return found_main_shield
except:
print("Coulnd't inspect stack, let's pretend the code is OK...")
return True
print(called_from_main_shield())
if __name__ == "__main__":
print(called_from_main_shield())
In the output, we see that the first called to called_from_main_shield returns False, while the second returns True:
$ python3 introspect.py
Calling introspect
[' File "introspect.py", line 24, in <module>\n print(called_from_main_shield())\n', ' File "introspect.py", lin
e 7, in called_from_main_shield\n print(traceback.format_stack())\n']
line=print(called_from_main_shield()) lineno=24 file=introspect.py
found_main_shield=False
False
Calling introspect
[' File "introspect.py", line 27, in <module>\n print(called_from_main_shield())\n', ' File "introspect.py", lin
e 7, in called_from_main_shield\n print(traceback.format_stack())\n']
line=print(called_from_main_shield()) lineno=27 file=introspect.py
found_main_shield=True
True
More concise version:
def called_from_main_shield():
tb = traceback.extract_stack()
try:
with open(tb[0].filename, mode="rt") as f:
found_main_shield = False
for i, line in enumerate(f):
if re.search(r"__name__.*['\"]__main__['\"]", line):
found_main_shield = True
if i == tb[0].lineno:
return found_main_shield
except:
return True
Now, it's not super elegant to use re.search() like I did, but it should be reliable enough. Warning: since I defined this function in my main script, I had to make sure that line didn't match itself, which is why I used ['\"] to match the quotes instead of using a simpler RE like __name__.*__main__. Whatever you chose, just make sure it's flexible enough to match all legal variants of that code, which is what I aimed for.
I think the best you can do is to try execute the code and provide a hint if it fails. Something like this:
# my_module.py
import sys # Use sys.stderr to print to the error stream.
from multiprocessing import Pool
def f(x):
return x*x
def my_function(x_max):
try:
with Pool(5) as p:
print(p.map(f, range(x_max)))
except RuntimeError as e:
print("Whoops! Did you perhaps forget to put the code in `if __name__ == '__main__'`?", file=sys.stderr)
raise e
This is of course not a 100% solution, as there might be several other reasons the code throws a RuntimeError.
If it doesn't raise a RuntimeError, an ugly solution would be to explicitly force the user to pass in the name of the module.
# my_module.py
from multiprocessing import Pool
def f(x):
return x*x
def my_function(x_max, module):
"""`module` must be set to `__name__`, for example `my_function(10, __name__)`"""
if module == '__main__':
with Pool(5) as p:
print(p.map(f, range(x_max)))
else:
raise Exception("This can only be called from the main module.")
And call it as:
# Script_of_a_careless_user.py
from my_module import my_function
my_function(10, __name__)
This makes it very explicit to the user.
I've created a pytest fixture which gets a token. When the tests which use this fixture fail, then the token will be printed in the logs. On the one hand that is not helpful, on the other hand it is a security issue.
How can I prevent the fixtures content to be printed?
MVCE
import pytest
#pytest.fixture
def token():
yield "secret"
def test_foo(token):
assert False
shows the "secret":
The easiest solution is to change the traceback format, e.g. pytest --tb=short will omit printing test function args. This can also be persisted in pytest.ini, effectively modifying the default pytest invocation:
[pytest]
addopts = --tb=short
However, you can also customize the output by extending pytest.
Technically, everything pytest prints to terminal is contained in TestReport, so you can modify the report object after the test finishes, but before the failure summary is printed. Example code, to be put in a conftest.py in the project or tests root dir:
def pytest_runtest_logreport(report):
if report.longrepr is None:
return
for tb_repr, *_ in report.longrepr.chain:
for entry in tb_repr.reprentries:
if entry.reprfuncargs is not None:
args = entry.reprfuncargs.args
for idx, (name, value) in enumerate(args):
if name == "token":
args[idx] = (name, "********")
if entry.reprlocals is not None:
lines = entry.reprlocals.lines
for idx, line in enumerate(lines):
if line.startswith("token"):
lines[idx] = "token = '*********'"
Although clumsy and untested, this demonstrates the approach: get the traceback info stored in the report, if any entry has either reprfuncargs available (this contains values for all test function arguments, including fixtures), modify the token value if present. Do the same for reprlocals (those are the f_locals of the recorded frame and are printed when you invoke e.g. pytest --showlocals).
When running the test now, you should get the modified error output like
===== FAILURES =====
_____ test_foo _____
token = ********
def test_foo(token):
> assert False
E assert False
The pytest_runtest_logreport hook is used to postprocess the report object created in pytest_runtest_makereport, before the actual reporting starts.
An approach which works well enough, adapted from here:
import pytest
class Secret:
def __init__(self, value):
self.value = value
def __repr__(self):
return "Secret(********)"
def __str___(self):
return "*******"
def get_from_vault(key):
return "something looked up in vault"
#pytest.fixture(scope='session')
def password():
return Secret(get_from_vault("key_in_value"))
def login(username, password):
pass
def test_using_password(password):
# reference the value directly in a function
login("username", password.value)
# If you use the value directly in an assert, it'll still log if it fails
assert "something looked up in vault" == password.value
# but won't be printed here
assert False
This isn't perfect but it will be simpler. This is the output:
==================================================================================== FAILURES ====================================================================================
______________________________________________________________________________ test_using_password _______________________________________________________________________________
password = Secret(********)
def test_using_password(password):
# reference the value directly in a function
login("username", password.value)
# If you use the value directly in an assert, it'll still log if it fails
assert "something looked up in vault" == password.value
# but won't be printed here
> assert False
E assert False
test_stuff.py:31: AssertionError
============================================================================ short test summary info =============================================================================
FAILED test_stuff.py::test_using_password - assert False
You can also follow an approach as suggested here: https://github.com/pytest-dev/pytest/issues/8613#issuecomment-830011874
Wrap the value in an object that won't allow it to escape unintentionally (__str__ and __repr__ return obfuscated values)
Use that wrapper in place of a raw string, unpacking it only where needed.
I have a method under test that looks like this:
def execute_update(self):
"""Execute an update."""
p = subprocess.Popen(['command'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
try:
stdout, stderr = p.communicate()
if p.returncode == 0:
# successful update, notify
self.logger.info("Successfully updated.")
else:
# failed update, notify
self.logger.error("Unable to update (code {returncode}):\n{output}".format(
returncode=p.returncode, output=stdout))
except KeyboardInterrupt as e:
# terminate the process
p.terminate()
raise e
I'm attempting to test its invocation of Popen and its invocation of the logging functions in a unittest.TestCase test method with mock:
#mock.patch.object(subprocess.Popen, 'communicate', autospec=True)
#mock.patch('updateservice.subprocess.Popen', autospec=True)
def test_fetch_metadata(self, mock_popen, mock_communicate):
"""Test that we can fetch metadata correctly."""
mock_communicate.return_value = ("OUT", "ERR")
mock_popen.returncode = 0
self.reference.execute_update()
# other asserts
The last line fails with:
stdout, stderr = p.communicate()
ValueError: need more than 0 values to unpack
What am I doing wrong? I have the following requirements:
Test that the constructor to subprocess.Popen was called with the right values.
Test that the logging calls are running with the output and return code from the process.
Number two is easy enough, I'm just injecting a MagicMock as the logger object, but I"m having trouble with number one.
I think the main problem is coming from your patch object here:
#mock.patch.object(subprocess.Popen, 'communicate', autospec=True)
Oddly enough, it seems like the type of mock that is being created is:
<class 'unittest.mock.NonCallableMagicMock'>
This is the first time I have come across a NonCallableMagicMock type before, but looking at the minimal information I found on this, the docs specify this:
The part that raises a flag for me is here:
with the exception of return_value and side_effect which have no
meaning on a non-callable mock.
It would require further investigation to determine what that exactly means. Taking that in to consideration, and maybe you have tried this already, the following change to your unittest yields successful mocking results:
#mock.patch('server.upd.subprocess.Popen', autospec=True)
def test_fetch_metadata(self, mock_popen):
"""Test that we can fetch metadata correctly."""
mock_popen.return_value = Mock()
mock_popen_obj = mock_popen.return_value
mock_popen_obj.communicate.return_value = ("OUT", "ERR")
mock_popen_obj.returncode = 0
self.reference.execute_update()
So, as you can see, we are pretty much creating our mock object per the mock_popen.return_value. From there everything else is pretty much aligned with what you were doing.
I have a test module in the standard unittest format
class my_test(unittest.TestCase):
def test_1(self):
[tests]
def test_2(self):
[tests]
etc....
My company has a proprietary test harness that will execute my module as a command line script, and which will catch any errors raised by my module, but requires that my module be mute if successful.
So, I am trying to find a way to run my test module naked, so that if all my tests pass then nothing is printed to the screen, and if a test fails with an AssertionError, that error gets piped through the standard Python error stack (just like any other error would in a normal Python script.)
The docs advocate using the unittest.main() function to run all the tests in a given module like
if __name__ == "__main__":
unittest.main()
The problem is that this wraps the test results in unittest's harness, so that even if all tests are successful, it still prints some fluff to the screen, and if there is an error, it's not simply dumped as a usual python error, but also dressed in the harness.
I've tried redirecting the output to an alternate stream using
with open('.LOG','a') as logf:
suite = unittest.TestLoader().loadTestsFromTestCase(my_test)
unittest.TextTestRunner(stream = logf).run(suite)
The problem here is that EVERYTHING gets piped to the log file (including all notice of errors). So when my companies harness runs the module, it complete's successfully because, as far as it can tell, no errors were raised (because they were all piped to the log file).
Any suggestions on how I can construct a test runner that suppresses all the fluff, and pipes errors through the normal Python error stack? As always, if you think there is a better way to approach this problem, please let me know.
EDIT:
Here is what I ended up using to resolve this. First, I added a "get_test_names()" method to my test class:
class my_test(unittest.TestCase):
etc....
#staticmethod
def get_test_names():
"""Return the names of all the test methods for this class."""
test_names = [ member[0] for memeber in inspect.getmembers(my_test)
if 'test_' in member[0] ]
Then I replaced my call to unittest.main() with the following:
# Unittest catches all errors raised by the test cases, and returns them as
# formatted strings inside a TestResult object. In order for the test
# harness to catch these errors they need to be re-raised, and so I am defining
# this CompareError class to do that.
# For each code error, a CompareError will be raised, with the original error
# stack as the argument. For test failures (i.e. assertion errors) an
# AssertionError is raised.
class CompareError(Exception):
def __init__(self,err):
self.err = err
def __str__(self):
return repr(self.err)
# Collect all tests into a TestSuite()
all_tests = ut.TestSuite()
for test in my_test.get_test_names():
all_tests.addTest(my_test(test))
# Define a TestResult object and run tests
results = ut.TestResult()
all_tests.run(results)
# Re-raise any script errors
for error in results.errors:
raise CompareError(error[1])
# Re-raise any test failures
for failure in results.failures:
raise AssertionError(failure[1])
I came up with this. If you are able to change the command line you might remove the internal io redirection.
import sys, inspect, traceback
# redirect stdout,
# can be replaced by testharness.py > /dev/null at console
class devnull():
def write(self, data):
pass
f = devnull()
orig_stdout = sys.stdout
sys.stdout = f
class TestCase():
def test_1(self):
print 'test_1'
def test_2(self):
raise AssertionError, 'test_2'
def test_3(self):
print 'test_3'
if __name__ == "__main__":
testcase = TestCase()
testnames = [ t[0] for t in inspect.getmembers(TestCase)
if t[0].startswith('test_') ]
for testname in testnames:
try:
getattr(testcase, testname)()
except AssertionError, e:
print >> sys.stderr, traceback.format_exc()
# restore
sys.stdout = orig_stdout