I am doing some hypothesis testing on async test. My code create and alter databases real-time, and I'm facing a problem with cleanup.
Basically, most of the time, I can cleanup the database without a problem. The only time when it get a bit messy is when a test doesn't pass, but that's not really a problem as I will fix the code and still have the error, thanks to pytest.
But that's not true with Hypothesis. Here is my test:
#given(st.text(min_size=1, max_size=128))
#pytest.mark.asyncio
async def test_hypothesis_add_column(name):
assume('\x00' not in name)
database = await get_database()
project = await database.create_project('test_project')
source = await project.create_source(
'test_source',
[
lib.ColumnDefinition(
name='external_id',
type=lib.ColumnTypes.NUMERIC,
is_key=True
)
]
)
await source.add_column(lib.ColumnDefinition(
name=name,
type=lib.ColumnTypes.TEXT
))
await end_database(database)
assert len(source.columns) == 2
assert await source.column(name) is not None
assert (await source.column(name)).internal_name.isidentifier()
This test raise an error. That's ok - it means there's a bug in my code, so I should fix it. But then, on the next run of hypothesis, there is another error, at another point (basically it cannot do the "create_source" because the database is messed up).
My problem is that hypothesis keep testing stuff AFTER the initial failure, even with report_multiple_bugs=False in my profile. And then it report the bug like this:
hypothesis.errors.Flaky: Inconsistent test results! Test case was Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'asyncpg.exceptions.PostgresSyntaxError'>, 'asyncpg/protocol/protocol.pyx', 168, (), ())) on first run but Conclusion(status=Status.INTERESTING, interesting_origin=(<class 'asyncpg.exceptions.InternalServerError'>, 'asyncpg/protocol/protocol.pyx', 201, (), ())) on second
And the worst part is that the pytest dump is related to the second test (the InternalServerError one) and I can't find the first test (the PostgresSyntaxError one). My problem is that the information I actually need to debug are the one from the first run - I don't even understand why it keeps trying when there is a fail, especially when I setup that I don't want multiple errors.
Is there a way to make it stop doing it and avoid those "Interesting" cases? I'd rather have the nice and clean explaination from hypothesis.
Thank you !
The quick-and-dirty answer is to adjust the phases setting to exclude Phase.shrink.
The real answer is that to get much out of Hypothesis, you'll need to make sure that running the same input twice has the same behavior, i.e. ensure that you clean up any corrupted database state on failure (e.g. using a context manager). This is more work, sorry, but getting reproducible tests and minimal failing examples is worth it!
Related
In my pytest suite, I run plenty of iOS UI tests. Which comes with plenty of frustrating issues. Im experimenting with the use of a hook based retry logic. Essentially, I have a pytest_runtest_call hook, where I collect the output of a test via a yield, and do extra processing with that data. Based on the failed output of the test I would like to re-trigger the test again, including the setup and teardown of the test. Is this possible? I am trying to do this without adding any extra packages (reinventing the wheel here, but in an effort to better understand pytest logic). Here's the idea of where I'm currently at with my hook:
def pytest_runtest_call(item):
output = yield
if output.excinfo:
# The retry_logic_handler.should_try will return a bool of whether or not to retry the test
if retry_logic_handler.should_retry(item):
# Re-run this test including the runtest_setup & runtest_teardown
...
_validate_expected_failures(output)
I understand I will get a lot of "just write better tests" but in my case, UI testing is rather unpredictable. And the data I am collecting helps clarify why a retry is required.
So the ultimate question is, how can I add retry logic to my pytest hooks?
Turns out this will do the trick, a different hook but the same objective.
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
report = yield
result = report.get_result()
if result.outcome == 'failed':
try:
attempt = item.runtest()
except:
attempt = 'failed'
if not attempt:
result.outcome = 'passed'
I have a python function foo with a while True loop inside.
For background: It is expected do stream info from the web, do some writing and run indefinitely. The asserts test if the writing was done correctly.
Clearly I need it to stop sometime, in order to test.
What I did was to run via multirpocessing and introduce a timeout there, however when I see the test coverage, the function which ran through the multiprocessing, are not marked as covered.
Question 1: Why does pytest now work this way?
Question 2: How can I make this work?
I was thinking it's probably because I technically exit the loop, so maybe pytest does not mark this as tested....
import time
import multiprocessing
def test_a_while_loop():
# Start through multiprocessing in order to have a timeout.
p = multiprocessing.Process(
target=foo
name="Foo",
)
try:
p.start()
# my timeout
time.sleep(10)
p.terminate()
finally:
# Cleanup.
p.join()
# Asserts below
...
More info
I looked into adding a decorator such as #pytest.mark.timeout(5), but that did not work and it stops the whole function, so I never get to the asserts. (as suggested here).
If I don't find a way, I will just test the parts, but ideally I would like to find a way to test by breaking the loop.
I know I can re-write my code in order to make it have a timeout, but that would mean changing the code to make it testable, which I don't think is a good design.
Mocks I have not tried (as suggested here), because I don't believe I can mock what I do, since it writes info from the web. I need to actually see the "original" working.
Break out the functionality you want to test into a helper method. Test the helper method.
def scrape_web_info(url):
data = get_it(url)
return data
# In production:
while True:
scrape_web_info(...)
# During test:
def test_web_info():
assert scrape_web_info(...) == ...
Yes, it is possible and the code above shows one way to do it (run through a multiprocessing with a timeout).
Since the asserts were running fine, I found out that the issue was not the pytest, but the coverage report not accounting for the multiprocessing properly.
I describe how I fix this (now separate) issue question here.
Actually, I had the same problem with an endless task to test and coverage. However, In my code, there is a .run_forever() method which runs a .run_once() method inside in an infinite loop. So, I can write a unit test for the .run_once() method to test its functionality. Nevertheless, if you want to test your forever function despite the Halting Problem for getting more extent code coverage, I propose the following approach using a timeout regardless of tools you've mentioned including multiprocessing or #pytest.mark.timeout(5) which didn't work for me either:
First, install the interruptingcow PyPI package to have a nice timeout for raising an optional exception: pip install interruptingcow
Then:
import pytest
import asyncio
from interruptingcow import timeout
from <path-to-loop-the-module> import EventLoop
class TestCase:
#pytest.mark.parametrize("test_case", ['none'])
def test_events(self, test_case: list):
assert EventLoop().run_once() # It's usual
#pytest.mark.parametrize("test_case", ['none'])
def test_events2(self, test_case: list):
try:
with timeout(10, exception=asyncio.CancelledError):
EventLoop().run_forever()
assert False
except asyncio.CancelledError:
assert True
When testing a single function with different inputs (some that are default), is it better practice to do:
def test_init(self):
page = HTMLGen("test", "path\\goes\\here")
self.assertEqual(page.path, "path\\goes\\here\\index.html")
page_2 = HTMLGen("test", "path\\goes\\here", "cool_page")
self.assertEqual(page_2.path, "path\\goes\\here\\cool_page.html")
or
def test_init(self):
page = HTMLGen("test", "path\\goes\\here")
self.assertEqual(page.path, "path\\goes\\here\\index.html")
def test_init_with_filename(self):
page = HTMLGen("test", "path\\goes\\here", "cool_page")
self.assertEqual(page.path, "path\\goes\\here\\cool_page.html")
The second approach is better because if the first test fails, the second one will still have a chance to run. This can give you more information for tracking down exactly where the bug is happening and what is causing it.
Additionally, any cleanup/teardown code will be run between the tests which can help to guarantee that the tests are independent.
I've been trying to integrate Luigi as our workflow handler. Currently we are using concourse, however many of the things we're trying to do is a hassle to get around in concourse so we made the switch to Luigi as our dependency manager. No problems so far, workflows trigger and execute properly.
The issue comes in when a task fails for whatever reason. This case specifically the requires block of a task, however all cases need to be taken care of. As of right now Luigi gracefully takes care of the error and writes it to STDOUT. It still emits and exit code 0 though, which to concourse means the job passed. A false positive.
I've been trying to get the event handling to fix this, but I cannot get it to trigger, even with an extremely simple job:
#luigi.Task.event_handler(luigi.Event.FAILURE)
def mourn_failure(task, exception):
with open('/root/luigi', 'a') as f:
f.write("we got the exception!") #testing in concourse image
sys.exit(luigi.retcodes.retcode().unhandled_exception)
class Test(luigi.Task):
def requires(self):
raise Exception()
return []
def run(self):
pass
def output(self):
return []
Then running the command in python shell
luigi.run(main_task_cls=Test, local_scheduler=True)
The exception gets raised, but the even doesn't fire or something.
The file doesn't get written and the exit code is still 0.
Also, if it makes a difference I have my luigi config at /etc/luigi/client.cfg which contains
[retcode]
already_running=10
missing_data=20
not_run=25
task_failed=30
scheduling_error=35
unhandled_exception=40
I'm at a loss as to why the event handler won't trigger, but somehow I need the process to fail on an error.
It seems like the problem is where you place the "raise Exception" call.
If you place it in the requires function - it basically runs before your Test task run method. So it's not as if your Test task failed, but the task it's dependent on (right now, empty...).
for example if you move the raise to run, you're code will behave as you expect.
def run(self):
print('start')
raise Exception()
To handle a case where your dependency fails (in this case, the exception is raised in the requires method), you can add another type of luigi event handler, BROKEN_TASK: luigi.Event.BROKEN_TASK.
This will make sure the luigi code emits the return code (different than 0) you expect.
Cheers!
If you'd like to catch exceptions in requires(), use the following:
#luigi.Task.event_handler(luigi.Event.BROKEN_TASK)
def mourn_failure(task, exception):
...
If I understand it correctly, you just want luigi to return an error code when a task fails, I had many issues with this one, but it turns out to be quite simple, you just need to run it with luigi on the command line, not with python. Like this:
luigi --module my_module MyTask
I don't know if that was your problem too, but I was running with python, and then luigi ignored the retcodes on the luigi.cfg. Hope it helps.
Some time ago I wrote a piece of code, a Flask route to log out users from a web application I was working on, that looked like that:
#app.route('/logout')
#login_required
def logout():
# lets get the user cookie, and if it exists, delete it
cookie = request.cookies.get('app_login')
response = make_response(redirect(url_for('login')))
if cookie:
riak_bucket = riak_connect('sessions')
riak_bucket.get(cookie).delete()
response.delete_cookie('app_login', None)
return response
return response
I did its job, and was certainly working, but now I am getting into making the app more robust by adding proper error handling, something that I havent done before on a large scale nowhere in my code. So I stumbled on this route function and I started writing its new version, when I realised I dont know how to do it 'the right way'. Here is what I came up with:
#app.route('/logout')
#login_required
def logout():
# why dont we call variables after what they are in specifics?
login_redirect = make_response(redirect(url_for('login')))
try:
cookie = request.cookies.get('app_login')
except:
return login_redirect
# if we are here, the above try/except went good, right?
try:
# perhaps sessions_bucket should really be bucket_object?
# is it valid to chain try statements like that, or should they be
# tried separately one by one?
sessions_bucket = riak_connect('sessions')
sessions_bucket.get(cookie).delete()
login_redirect.delete_cookie('app_login', None)
except:
return login_redirect
# return redirect by default, just because it seems more secure
return login_redirect
It also does it job, but still doesnt look 'right' to me. So, the question are, to all of you who have larger experience in writing really pythonic Python code, given the fact I would love the code to handle all errors nicely, be readable to others and do its job fast and well (in this particular case but also in rest of rather large codebase):
how are you calling your variables, extra specific or general: sessions_bucket vs riak_bucket vs bucket_object?
how do you handle errors, by usage of try/except one after another, or by nesting one try/except in another, or in any other way?
is it ok to do more than one thing in one try/except, or not?
and perhaps anything else, that comes to your mind to the above code examples
Thanks in advance!
I don't know the exact riak python API, so I don't know what exceptions are thrown. On the other hand, how should the web app behave on the different error conditions? Has the user to be informed?
Variable names: I prefer generic. If you change the implementation (e.g. Session store), you don't have to change the variable names.
Exceptions: Depends on the desired behavior. If you want to recover from errors, try/except one after another. (Generally, linear code is simpler.) If you don't recover from errors, I find one bigger try clause with several exception clauses very acceptable.
For me it's ok to do several things in one try/except. If there are too many try/except clauses, the code gets less readable.
More things: logging. logging.exception will log the traceback so you can know where exactly the error appeared.
Some suggestion:
import logging
log = loggin.getLogger(__name__)
#app.route('/logout')
#login_required
def logout():
login_redirect = make_response(redirect(url_for('login')))
try:
sessionid = request.cookies.get('app_login', None)
except AttributeError:
sessionid = None
log.error("Improperly configured")
if sessionid:
try:
session_store = riak_connect('sessions')
session = session_store.get(sessionid)
if session:
session.delete()
login_redirect.delete_cookie('app_login', None)
except IOError: # what errors appear when connect fails?
log.exception("during logout")
return login_redirect