In py.test there must be a function that loops over each found test and executes them sequentially.
Is there a way to overwrite this function/loop?
Because I want to run the tests sequentially, as before, but run the failed tests again a second time (or a third time) to see if they might succeed on the second (third) try.
(Some background explanation: The system to test is a GUI application which depends on many different systems, and rarely some of them fail or does not behave as expected. When you have some of these tests it becomes quite likely that at least on test will fail. Therefore I want to repeat the failed test for a couple of times. Also, that system cannot be changed).
Related
i have a pytest test suite with about 1800 tests which takes more than 10 minutes to collect and execute. i tried to create a cprofile on the test and found out that majority of the time, around 300 seconds went in {built-in method builtins.compile}
There were some other compile method calls from the regular expression package which i tried to remove and saw a reduction of about 50 seconds. but it still takes 9.5 minutes which is huge.
What i understood till now is that the builtins compile method is used to convert the script into code object and that pytest internally uses this function for creating and executing code objects. But 9-10 minutes is insanely huge amount of time for running 1800 tests. I am new to pytest and python so trying to figure out the reason for this time.
Could there be a possibility that pytest is not configured properly that it uses compile method to generate code object ? or could the other imported libraries use compile internally ?
Could there be a possibility that pytest is not configured properly that it uses compile method to generate code object ?
Though I have never looked, I would fully expect pytest to compile files to bytecode by hand, for the simple reason that it performs assertion rewriting by default in order to instrument assert statements: when an assertion fails, rather than just show the assertion message pytest shows the various intermediate values. This requires compiling either way: either they're compiling the code to bytecode and rewriting the bytecode, or they're parsing the code to the AST, updating the AST, and still compiling to bytecode.
It's possible to disable this behaviour (--assert=plain), but I would not expect there to be much gain from it (though I could be wrong): pytest simply does that instead of the interpreter performing the compilation on its own. It has to be done one way or an other for the test suite to run.
Though taking 5 minutes does sound like a lot, do you have a large amounts of very small files or something? Rough benching indicates that compile works at about 5usec/line on my machine (though it probably depends on code complexity). I've got 6kLOC worth of test, and while the test suite takes ages it's because the tests themselves are expensive, the collection is unnoticeable.
Of course it's possible you could be triggering some sort of edge case or issue in pytest e.g. maybe you have an ungodly number of assert statements which causes pytest to generate an insane amount of rewritten code? The aforementioned --assert=plain could hint at that if it makes running the test suite significantly shorter.
You could also try running e.g. --collect-only to see what that yields, though I don't know whether the assertion rewriting is performed during or after the collection. FWIW on the 6kLOC test suite above I get 216 tests collected in 1.32s.
Either way this seems like something more suitable to the pytest bug tracker.
or could the other imported libraries use compile internally ?
You could use a flamegraph-based profiler to record the entire stack. cprofile is, frankly, kinda shit.
Perhaps this is a broad question, but I haven't found an answer elsewhere, so here goes.
The Python script I'm writing needs to run constantly (in a perfect world, I recognize this may not be exactly possible) on a deployed device. I've already dedicated time to adding "try...except" statements throughout so that, should an issue arise, the script will recover and continue to work.
The issue is that I'm not sure I can (nor should) handle every single possible exception that may be thrown. As such, I've decided it may be better to allow the script to die and to use systemd to restart it.
The three options:
Making no attempt to handle any exception, and just allowing systemd to restart it whenever it dies.
Meticulously creating handlers for every possible exception to guarantee that, short of loss of power, interpreter bug, or heat death of the universe, the script will always run.
A mix of the two -- making an effort to prevent crashes in some cases while allowing them in others and letting systemd restart the script.
The third choice seems the most reasonable to me. So the question is this: What factors should be considered when optimizing between "crash-proof" code and allowing a crash and restart by systemd?
For some more application specific information: there is a small but noticeable overhead involved with starting the script, the main portion will run between 50 to 100 times per second, it is not "mission critical" in that there will be no death/damage in the event of failure (just some data loss), and I already expect intermittent issues with the network it will be on.
All known exceptional cases should be handled. Any undefined behavior is a potential security issue.
As you suggest, it is also prudent to plan for unknown exceptions. Perhaps there's also a small memory leak that will also cause the application to crash even when it's running correctly. So, it's still prudent to have systemd automatically restart it if it fails, even when all expected failure modes have been handled.
We have two machines with the purpose to split our testing across machines to make testing faster. I would like to know of a way to tell behave to run half of the tests. I am aware of the --tags argument but this is too cumbersome as, when the test suite grows, so must our --tags argument if we wish to keep it at the halfway point. I would also need to know which of the other half of tests were not run so I can run those on the other machine.
TL;DR Is there a simple way to get behave to run, dynamically, half of the tests? (that doesn't include specifying which tests through the use of --tags)
And is there a way of finding the other half of tests that were not run?
Thanks
No there is not, you would have to write your own runner to do that. But that would be complex to do as trying to piece together content of two separate test runs, which are half of each other would be rather complex if any errors are to show up.
A better and faster solution will be to write a simple bash/python script that will traverse given directory for .feature files and then fire indivisdual behave process against it. Then with properly configured outputs it should be collision free in terms of outputs and if you separate your cases give you a much better boost than running half. And of course delegate that task to other machine by some means, be it bare SSH command or queues.
I've got a large number of tests written using Python unittest (Python 2.7.8) as a large TestSuite. Many of these tests invoke other programs. Sometimes these other programs dump core. When they do, I want to discover that and ensure the test fails. If some number of cores are dumped, I want to abort the entire test environment and exit rather than continuing: my total test suite has >6000 tests and if everything is dumping core it's useless (and dangerous: disk space etc.) to continue.
In order to ensure we look for coredumps after every test (so I have the best possible idea of what program/invocation dumped core) I decided to look for cores in tearDown(), which I am doing successfully. If I find a core, I can run an assert variant in tearDown() to specify that the test failed.
But I can't figure out how to just give up on all my testing completely from within tearDown() if I find too many cores. I even tried to run sys.exit("too many cores"), but unittest case.py catches every exception thrown by tearDown() except KeyboardInterrupt (if I try to raise that by hand my script hangs until I do a real ^C).
I thought about trying to call stop(), but this is a method on the result and I can't find any way to get access to the result object from within tearDown() (!).
So far my only option seems to be to invoke os._exit() which is really annoying because it keeps any results from being reported at all!
Is there really no facility in Python unittest.TestCase to tell the test environment to just stop right now, generate what results you have but don't run anything else?
Can you check how many cores have been dumped in setUp()? If so, you could just call self.skipTest('Too many cores dumped.') when things get bad.
If you don't want to look for cores in setUp(), you could probably use a class variable to hold the core dump count and check that instead.
I have some unit tests that are timing sensitive: an action is timed and an error is triggered if it takes too long. When run individually, these tests pass, but when running nosetest recursively on my modules, they often fail. I run concurrent tests, which likely is one reason why the timing is off. Is there any way to indicate that I want this test to be run with no interruptions?
I think your problem is dependent from how you implemented the timing. The solution I would personally adopt would be to set an environment variable that controls the behaviour of the tests. Candidates could be:
if WITH_TIMING == False [turn off timing altogether]
TIME_STRETCH_FACTOR = ... [apply a time-stretching multiplier in case of concurrent test are run, so that for example a time limit of 5 would become 7.5 if TIME_STRETCH_FACTOR would be 1.5]
If this is not an option, a possible ugly workaround would be to mock the time.time() function, making it return a constant value [this would only work if you use time.time() in your tests directly of course]...
HTH