In my pytest suite, I run plenty of iOS UI tests. Which comes with plenty of frustrating issues. Im experimenting with the use of a hook based retry logic. Essentially, I have a pytest_runtest_call hook, where I collect the output of a test via a yield, and do extra processing with that data. Based on the failed output of the test I would like to re-trigger the test again, including the setup and teardown of the test. Is this possible? I am trying to do this without adding any extra packages (reinventing the wheel here, but in an effort to better understand pytest logic). Here's the idea of where I'm currently at with my hook:
def pytest_runtest_call(item):
output = yield
if output.excinfo:
# The retry_logic_handler.should_try will return a bool of whether or not to retry the test
if retry_logic_handler.should_retry(item):
# Re-run this test including the runtest_setup & runtest_teardown
...
_validate_expected_failures(output)
I understand I will get a lot of "just write better tests" but in my case, UI testing is rather unpredictable. And the data I am collecting helps clarify why a retry is required.
So the ultimate question is, how can I add retry logic to my pytest hooks?
Turns out this will do the trick, a different hook but the same objective.
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
report = yield
result = report.get_result()
if result.outcome == 'failed':
try:
attempt = item.runtest()
except:
attempt = 'failed'
if not attempt:
result.outcome = 'passed'
Related
I am doing some hypothesis testing on async test. My code create and alter databases real-time, and I'm facing a problem with cleanup.
Basically, most of the time, I can cleanup the database without a problem. The only time when it get a bit messy is when a test doesn't pass, but that's not really a problem as I will fix the code and still have the error, thanks to pytest.
But that's not true with Hypothesis. Here is my test:
#given(st.text(min_size=1, max_size=128))
#pytest.mark.asyncio
async def test_hypothesis_add_column(name):
assume('\x00' not in name)
database = await get_database()
project = await database.create_project('test_project')
source = await project.create_source(
'test_source',
[
lib.ColumnDefinition(
name='external_id',
type=lib.ColumnTypes.NUMERIC,
is_key=True
)
]
)
await source.add_column(lib.ColumnDefinition(
name=name,
type=lib.ColumnTypes.TEXT
))
await end_database(database)
assert len(source.columns) == 2
assert await source.column(name) is not None
assert (await source.column(name)).internal_name.isidentifier()
This test raise an error. That's ok - it means there's a bug in my code, so I should fix it. But then, on the next run of hypothesis, there is another error, at another point (basically it cannot do the "create_source" because the database is messed up).
My problem is that hypothesis keep testing stuff AFTER the initial failure, even with report_multiple_bugs=False in my profile. And then it report the bug like this:
hypothesis.errors.Flaky: Inconsistent test results! Test case was Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'asyncpg.exceptions.PostgresSyntaxError'>, 'asyncpg/protocol/protocol.pyx', 168, (), ())) on first run but Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'asyncpg.exceptions.InternalServerError'>, 'asyncpg/protocol/protocol.pyx', 201, (), ())) on second
And the worst part is that the pytest dump is related to the second test (the InternalServerError one) and I can't find the first test (the PostgresSyntaxError one). My problem is that the information I actually need to debug are the one from the first run - I don't even understand why it keeps trying when there is a fail, especially when I setup that I don't want multiple errors.
Is there a way to make it stop doing it and avoid those "Interesting" cases? I'd rather have the nice and clean explaination from hypothesis.
Thank you !
The quick-and-dirty answer is to adjust the phases setting to exclude Phase.shrink.
The real answer is that to get much out of Hypothesis, you'll need to make sure that running the same input twice has the same behavior, i.e. ensure that you clean up any corrupted database state on failure (e.g. using a context manager). This is more work, sorry, but getting reproducible tests and minimal failing examples is worth it!
I have a python function foo with a while True loop inside.
For background: It is expected do stream info from the web, do some writing and run indefinitely. The asserts test if the writing was done correctly.
Clearly I need it to stop sometime, in order to test.
What I did was to run via multirpocessing and introduce a timeout there, however when I see the test coverage, the function which ran through the multiprocessing, are not marked as covered.
Question 1: Why does pytest now work this way?
Question 2: How can I make this work?
I was thinking it's probably because I technically exit the loop, so maybe pytest does not mark this as tested....
import time
import multiprocessing
def test_a_while_loop():
# Start through multiprocessing in order to have a timeout.
p = multiprocessing.Process(
target=foo
name="Foo",
)
try:
p.start()
# my timeout
time.sleep(10)
p.terminate()
finally:
# Cleanup.
p.join()
# Asserts below
...
More info
I looked into adding a decorator such as #pytest.mark.timeout(5), but that did not work and it stops the whole function, so I never get to the asserts. (as suggested here).
If I don't find a way, I will just test the parts, but ideally I would like to find a way to test by breaking the loop.
I know I can re-write my code in order to make it have a timeout, but that would mean changing the code to make it testable, which I don't think is a good design.
Mocks I have not tried (as suggested here), because I don't believe I can mock what I do, since it writes info from the web. I need to actually see the "original" working.
Break out the functionality you want to test into a helper method. Test the helper method.
def scrape_web_info(url):
data = get_it(url)
return data
# In production:
while True:
scrape_web_info(...)
# During test:
def test_web_info():
assert scrape_web_info(...) == ...
Yes, it is possible and the code above shows one way to do it (run through a multiprocessing with a timeout).
Since the asserts were running fine, I found out that the issue was not the pytest, but the coverage report not accounting for the multiprocessing properly.
I describe how I fix this (now separate) issue question here.
Actually, I had the same problem with an endless task to test and coverage. However, In my code, there is a .run_forever() method which runs a .run_once() method inside in an infinite loop. So, I can write a unit test for the .run_once() method to test its functionality. Nevertheless, if you want to test your forever function despite the Halting Problem for getting more extent code coverage, I propose the following approach using a timeout regardless of tools you've mentioned including multiprocessing or #pytest.mark.timeout(5) which didn't work for me either:
First, install the interruptingcow PyPI package to have a nice timeout for raising an optional exception: pip install interruptingcow
Then:
import pytest
import asyncio
from interruptingcow import timeout
from <path-to-loop-the-module> import EventLoop
class TestCase:
#pytest.mark.parametrize("test_case", ['none'])
def test_events(self, test_case: list):
assert EventLoop().run_once() # It's usual
#pytest.mark.parametrize("test_case", ['none'])
def test_events2(self, test_case: list):
try:
with timeout(10, exception=asyncio.CancelledError):
EventLoop().run_forever()
assert False
except asyncio.CancelledError:
assert True
We are sucessfully using pytest (Python 3) to run a test suite testing some hardware devices (electronics).
For a subset of these tests, we need the tester to change the hardware arrangement, and afterwards change it back.
My approach was to use a module-level fixture attached to the tests in question (which are all in a separate module), with two input calls:
#pytest.fixture(scope="module")
def disconnect_component():
input('Disconnect component, then press enter')
yield # At this point all the tests with this fixture are run
input('Connect component again, then press enter')
When running this, I get OSError: reading from stdin while output is captured. I can avoid this by calling pytest with --capture=no, and have confirmed that my approach works, meaning I get the first query before the test subset in question, and the second one after they have run.
The big drawback is that this deactivates capturing stdin/stderr for the whole test suite, which some of the other test rely on.
I also tried to use capsys.disabled (docs) like this
#pytest.fixture(scope="module")
def disconnect_component(capsys):
with capsys.disabled():
input('Disconnect component, then press enter')
yield # At this point all the tests with this fixture are run
input('Connect component again, then press enter')
but when running this I get ScopeMismatch: You tried to access the 'function' scoped fixture 'capsys' with a 'module' scoped request object, involved factories.
Can I make pytest wait for user action in some other way than input? If not, can I disable capturing just for the tests using above fixture?
So, I found a hint by a pytest dev, based on which I basically do what the capsys.disable() function does:
#pytest.fixture(scope="module")
def disconnect_component(pytestconfig):
capmanager = pytestconfig.pluginmanager.getplugin('capturemanager')
capmanager.suspend_global_capture(in_=True)
input('Disconnect component, then press enter')
capmanager.resume_global_capture()
yield # At this point all the tests with this fixture are run
capmanager.suspend_global_capture(in_=True)
input('Connect component again, then press enter')
capmanager.resume_global_capture()
This works flawlessly as far as I can see. Don't forget the in_=True bit.
Edit: From pytest 3.3.0 (I think), capmanager.suspendcapture and capmanager.resumecapture were renamed to capmanager.suspend_global_capture and capmanager.resume_global_capture, respectively.
As of pytest 5, as a fixture, you can use this:
#pytest.fixture
def suspend_capture(pytestconfig):
class suspend_guard:
def __init__(self):
self.capmanager = pytestconfig.pluginmanager.getplugin('capturemanager')
def __enter__(self):
self.capmanager.suspend_global_capture(in_=True)
def __exit__(self, _1, _2, _3):
self.capmanager.resume_global_capture()
yield suspend_guard()
Example usage:
def test_input(suspend_capture):
with suspend_capture:
input("hello")
Maybe it's worth noting that above solution doesn't have to be in a fixture. I've made a helper function for that:
import pytest
def ask_user_input(msg=''):
""" Asks user to check something manually and answer a question
"""
notification = "\n\n???\tANSWER NEEDED\t???\n\n{}".format(msg)
# suspend input capture by py.test so user input can be recorded here
capture_manager = pytest.config.pluginmanager.getplugin('capturemanager')
capture_manager.suspendcapture(in_=True)
answer = raw_input(notification)
# resume capture after question have been asked
capture_manager.resumecapture()
logging.debug("Answer: {}".format(answer))
return answer
For future reference, if you need to use input with pytest. You can do this in any part of your pytest, setup_class, test_..., teardown_method, etc. This is for pytest > 3.3.x
import pytest
capture_manager = pytest.config.pluginmanager.getplugin('capturemanager')
capture_manager.suspend_global_capture(in_=True)
answer = input('My reference text here')
capture_manager.resume_global_capture()
Solutions that use the global pytest.config object no longer work. For my use case, using --capture=sys together with a custom input() that uses stdin and stdout directly works well.
def fd_input(prompt):
with os.fdopen(os.dup(1), "w") as stdout:
stdout.write("\n{}? ".format(prompt))
with os.fdopen(os.dup(2), "r") as stdin:
return stdin.readline()
I've been trying to integrate Luigi as our workflow handler. Currently we are using concourse, however many of the things we're trying to do is a hassle to get around in concourse so we made the switch to Luigi as our dependency manager. No problems so far, workflows trigger and execute properly.
The issue comes in when a task fails for whatever reason. This case specifically the requires block of a task, however all cases need to be taken care of. As of right now Luigi gracefully takes care of the error and writes it to STDOUT. It still emits and exit code 0 though, which to concourse means the job passed. A false positive.
I've been trying to get the event handling to fix this, but I cannot get it to trigger, even with an extremely simple job:
#luigi.Task.event_handler(luigi.Event.FAILURE)
def mourn_failure(task, exception):
with open('/root/luigi', 'a') as f:
f.write("we got the exception!") #testing in concourse image
sys.exit(luigi.retcodes.retcode().unhandled_exception)
class Test(luigi.Task):
def requires(self):
raise Exception()
return []
def run(self):
pass
def output(self):
return []
Then running the command in python shell
luigi.run(main_task_cls=Test, local_scheduler=True)
The exception gets raised, but the even doesn't fire or something.
The file doesn't get written and the exit code is still 0.
Also, if it makes a difference I have my luigi config at /etc/luigi/client.cfg which contains
[retcode]
already_running=10
missing_data=20
not_run=25
task_failed=30
scheduling_error=35
unhandled_exception=40
I'm at a loss as to why the event handler won't trigger, but somehow I need the process to fail on an error.
It seems like the problem is where you place the "raise Exception" call.
If you place it in the requires function - it basically runs before your Test task run method. So it's not as if your Test task failed, but the task it's dependent on (right now, empty...).
for example if you move the raise to run, you're code will behave as you expect.
def run(self):
print('start')
raise Exception()
To handle a case where your dependency fails (in this case, the exception is raised in the requires method), you can add another type of luigi event handler, BROKEN_TASK: luigi.Event.BROKEN_TASK.
This will make sure the luigi code emits the return code (different than 0) you expect.
Cheers!
If you'd like to catch exceptions in requires(), use the following:
#luigi.Task.event_handler(luigi.Event.BROKEN_TASK)
def mourn_failure(task, exception):
...
If I understand it correctly, you just want luigi to return an error code when a task fails, I had many issues with this one, but it turns out to be quite simple, you just need to run it with luigi on the command line, not with python. Like this:
luigi --module my_module MyTask
I don't know if that was your problem too, but I was running with python, and then luigi ignored the retcodes on the luigi.cfg. Hope it helps.
I am looking for a way to prevent tests from being executed when, for example, a required test server cannot be contacted.
Is essential to be able to detect this before starting to execute the tests, and failing as fast as possible.
The tests are run using py.test or using tox which calls py.test.
I do have a piece of code that detects if the tests server is up, but I don't know which is the right place to put this into.
Initially I assumed that this would be a global fixture but that's not quite true as it would mean that will run for each test and what I want is not to run them, at all.
I'm using skipif for skipping tests.
I have a function, which returns True or False based on the availability of the program I need:
def have_gpg():
try:
subprocess.call(['gpg', '--version'])
except subprocess.CalledProcessError:
return False
return True
Then for tests I need to skip if GPG is not present, I use skipif:
#pytest.mark.skipif("not have_gpg()")
def test_gpg_decode(tmpdir):
...
assert out = 'test'
See more documentation on skipif.