I was wondering how I could alter a pytest test outcome (from a fail to a skip) in the case that my error message includes a specific string.
Occasionally we get test failures using appium where the response from the appium server is a 500 error with the failure message: "An unknown server-side error occurred while processing the command." Its an issue that we need to solve, but for the meantime we want to basically say, if the test failed because of an error message similar to that, skip the test instead of failing it.
Ive considered and tried something like this:
def pytest_runtest_setup(item):
excinfo = None
try:
item.obj()
except Exception as e:
excinfo = sys.exc_info()
if excinfo and "An unknown server-side error occurred while processing the command." in str(excinfo[1]):
pytest.skip("Skipping test due to error message")
And this obviously won't work.
But I was hoping for a similar approach.
The successful answer:
In order for me to get this working #Teejay pointed out below I needed to use the runtest_call hook and assess the message there. Currently working well in my test suite!
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_call(item):
output = yield
if output.excinfo:
for potential_error_message in expected_failure_messages.keys():
if output._excinfo[1].stacktrace and potential_error_message in output._excinfo[1].stacktrace:
pytest.skip(reason=expected_failure_messages[potential_error_message])
I recommend utilizing a hook wrapper to inspect the exception raised and act accordingly
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_setup(item):
output = yield
if output.excinfo:
# Additional logic to target specific error
pytest.skip()
Your sketched idea is close to workable. I'd skip using sys.exc_info() and just inspect the str() value of the exception, and I'd restrict the classes of exception caught to the smallest set that covers the failures you're trying to ignore.
Something like:
try:
item.obj()
except OSError as e: # ideally a narrower subclass or tuple of classes
if 'An unknown server-side error occurred while processing the command.' in str(e):
pytest.skip(f'Skipping test due to error message {e}')
else:
raise e
The only additional logical change I've made is moving the if/skip into the exception handler and re-raising if it doesn't match the message you're expecting.
Restricting the classes of errors matched is a best practice to avoid catching situations you didn't intend to catch. It's probably harmless to over-catch here where you're inspecting the message, just a good habit to cultivate. But it might also let you identify a specific field of the exception class to check rather than just the string representation - eg OSError has the strerror attribute containing the error message from the OS, so if you've limited your except block to catching just those you know you'll have that attribute available.
I chose to include the exception in the skip message, you might decide differently if they're uninformative.
Related
I am scratching my head about what is the best-practice to get the traceback in the logfile only once. Please note that in general I know how to get the traceback into the log.
Let's assume I have a big program consisting of various modules and functions that are imported, so that it can have quite some depth and the logger is set up properly.
Whenever an exception may occur I do the following:
try:
do_something()
except MyError as err:
log.error("The error MyError occurred", exc_info=err)
raise
Note that the traceback is written to the log via the option exc_info=err.
My Problem is now that when everything gets a bit more complex and nested I loose control about how often this traceback is written to the log and it gets quite messy.
An example of the situation with my current solution for this problem is as follows:
from other_module import other_f
def main():
try:
# do something
val = other_f()
except (AlreadyLoggedError1, AlreadyLoggedError2, AlreadyLoggedError3):
# The error was caught within other_f() or deeper and
# already logged with traceback info where it occurred
# After logging it was raised like in the above example
# I do not want to log it again, so it is just raised
raise
except BroaderException as err:
# I cannot expect to have thought of all exceptions
# So in case something unexpected happened
# I want to have the traceback logged here
# since the error is not logged yet
log.error("An unecpected error occured", exc_info=err)
raise
The problem with this solution is, that I need to to keep track of all Exceptions that are already logged by myself and the line except (AlreadyLoggedError1, AlreadyLoggedError2, ...) gets arbitrary long and has to be put at any level between main() and the position the error actually occured.
So my question is: Is there some better (pythonic) way handling this? To be more specific: I want to raise the information that the exception was already logged together with the exception so that I do not have to account for that via an extra except block like in my above example.
The solution normally used for larger applications is for the low-level code to not actually do error handling itself if it's just going to be logged, but to put exception logging/handling at the highest level in the code possible, since exceptions will bubble up as far as needed. For example, libraries that send errors to a service like New Relic and Sentry don't need you to instrument each small part of your code that might throw an error, they are set up to just catch any exception and send it to a remote service for aggregation and tracking.
I am working on a simple automation script in Python, which could throw exceptions in various spots. In each of them I would like to log a specific message and exit the program. In order to do that, I raise SystemExit after catching the exception and handling it (performing specific logging operations and such).
In the top-level calling of main, I do the following:
if __name__ == "__main__":
try:
main()
except SystemExit: # handled exception
sys.exit(1)
except: # any unhandled exception
logging.error('Unexpected error: ', exc_info=True)
sys.exit(2)
However, using a bare except is something frowned upon. Is using an "exception tree" where I use a bare except to specify "anything but the exception that I've handled" a nonstandard way? Is there a better way to achieve this? I would still like to log these unhandled exceptions, even if they were not handled.
Edit: SystemExit is raised to note that an exception has been handled - no matter what the exception is in my case, I always want to stop running the scripts as any failure should result in an absolute failure.
The main reason I'm asking this is that PEP8 seems to consider using a bare except as an error, and even though I could use except BaseException, it should be just a syntactic difference. Is one way more standard than the other or is there another standard route of achieving this?
Bare exceptions trap things you do not want to trap, such as GeneratorExit. Do it this way:
except Exception as details:
logging.error('Unexpected error: {0}'.format(details))
The main issue with a bare except is that it can catch things like SystemExit and KeyboardInterrupt which are not standard 'code' errors and shouldn't usually be handled in the same way as an exception generated by your code. Using the Exception class doesn't cover those cases as they do not inherit from it, so it is more than a syntax difference.
https://docs.python.org/2/howto/doanddont.html#except
https://docs.python.org/3.1/howto/doanddont.html#except
If you want to handle those specific cases, then it is better to do so explicitly as you have done for SystemExit.
This worked for me:
try:
<code>
raise Exception("my error")
except Exception as e:
raise e
If my error occurs then the error message "my errror" is seen. If an unknown exception occurs then it displays the default exception handler's text. In either case an exception is raised and the script is halted.
Take a look at the following test case:
def test_1_check_version(self):
try:
self.version()
print('\n')
except cx_Oracle.DatabaseError as error_message:
print("Sorry Connection could not be established because "+str(error_message))
Above is the test case of many test cases I am writing in Unittest of Python, and now I am running it to check that connection of the database is connected or not.
If yes, then it will pass 'Version number of Database'.
If not, then it will throw an exception, which I have handled.
After running this rest case, the test case is showing pass in any testing framework (Robot, unittest, pytest) that I have used.
But, I want this test case to fail because it is not the result I am looking for.
Handling exceptions because I want to see the only error message rather than all those red lines of exception.
I am open for any kind of suggestion, whether it involves removing exceptions.
The behavior of passing the test is the expected.. tests only fail if you receive a result which was not expected in some assertion.
In your case, if you want the exception to be thrown, you should use:
self.assertRaises(cx_Oracle.DatabaseError, self.version())
If you want to check if the version is correct, then use:
self.assertEqual(XXX, self.version())
where XXX is the value of self.version() you're expecting
since version 2.7: Added the ability to use assertRaises() as a context manager
So the recommended usage would be
with self.assertRaises(SomeException) as cm:
self.version()
the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)
see document
I want to explicitly fail the step in behave when I encounter an exception
eg. I am writing the code according to behave documentation -
from behave import *
#when('verify test fails.*?(?P<param_dict>.*)')
def test_logger(context, param_dict):
try:
logger.info("testing the logger. this is info message")
logger.info(1/0)
except Exception as e:
logger.error("arrived at exception: "+str(e))
fail("failed with exception: "+str(e))
but it throws this error:
NameError: name 'fail' is not defined
I tried other ways too, but nothing works
eg. context.failed = True (did not work either)
If I do not try to fail explicitly, final test result becomes PASS even if it goes in exception block ...which is weird.
context.failed is only an attribute set by Behave and doesn't do anything as is. It's an information attribute, and while you can use it to determine a fail-case and throw an assertion error, it will not do anything on it's own. See context.failed
As for the fail method you've mentioned, it is probably from the unittest module, as seen here. This module is used in Behave's development tests (see their Github) as well. I'll agree though, that this should be clarified in their documentation.
To fix your error you'd need to import the unittest module. To explicitly fail the step, you'd just raise the exception after you've logged your message, something like this:
except Exception as e:
logger.error("arrived at exception: "+str(e))
fail("failed with exception: "+str(e))
raise
As #Verv mentioned in their answer, a behave step will be marked as failed whenever an exception is thrown, so you could just re-raise the exception.
However, behave will show the backtrace for normal exceptions, whereas you probably just want to show a specific failure message.
For that, raise an AssertionError. In your code, that would look like this:
except Exception as e:
logger.error("arrived at exception: " + str(e))
raise AssertionError("failed with exception: " + str(e))
If behave encounters an AssertionError, it will fail the step, display the message you instantiate the error with, but not display other exception stuff.
What the best way of handling exceptions.
class SRException(Exception):
default_exception = True
def somefun1(email):
try:
user = somefun2(email)
except Exception, e:
raise SRException, ("could not find the email", e)
def somefun2(email):
try:
user = User.objects.get(email = email)
except Exception, e:
raise SRException, ("could not find the user objecets",e )
So when exception happens I get a long list of Exception
UserProfileException('could not find
the user or service objects',
UserProfileException('could not find
the user', ServicesException('could
not find the service',
DoesNotExist('Services matching query
does not exist.',))))))))
Error and above code examples are not the same. But I guess I made my point clear.
So what the best way of handling exceptions.
Should I not raise it in every exceptions. And I am sending mail to technical team every time exceptions occurs.
Your exception handler is too broad, catch only the specific exceptions you know you can handle. There is no sense in catching an Exception only to wrap it in another exception and re-raising it; an exception object carries a Traceback that will show you what path the code is going. Just let the Exception bubble up, and catch it at a level where you can recover from the exception.
It is usually not necessary to wrap exceptions at every level of the call stack. You are better off catching exceptions somewhere high up in the call stack and dumping a stack trace into the tech-support email. This will indicate quite clearly where the problem occurred and where it was called from.
With a bit of digging around in sys.exc_info()[2], you could even dump a complete list of parameters and locals in each stack frame, which would give support staff the offending email address.
First of all, never just check for Exception! Always use the correct subtype you are actually expecting.
Also you shouldn't encapsulate already useful exceptions into other ones; and in general use the type of the exception as an identification of what happens. Then you can simply keep throwing (informative) exceptions at low level, and catch them all separately at a higher level to determine the correct error message for the end user.