I am writting a daemon server using python, sometimes there are python runtime errors, for example some variable type is not correct. That error will not cause the process to exit.
Is it possible for me to redirect such runtime error to a log file?
It looks like you are asking two questions.
To prevent your process from exiting on errors, you need to catch all exceptions that are raised using try...except...finally.
You also wish to redirect all output to a log. Happily, Python provides a comprehensive logging module for your convenience.
An example, for your delight and delectation:
#!/usr/bin/env python
import logging
logging.basicConfig(filename='warning.log', level=logging.WARNING)
try:
1/0
except ZeroDivisionError, e:
logging.warning('The following error occurred, yet I shall carry on regardless: %s', e)
This graciously emits:
% cat warning.log
WARNING:root:The following error occurred, yet I shall carry on regardless: integer division or modulo by zero
Look at the traceback module. If you catch a RuntimeError, you can write it to the log (look at the logging module for that).
Related
I have made an program in python that i converted into .exe using auto-py-to-exe and im wondering how to stop .exe stop closing it self after an error (python exception) the .exe closes it self in the speed of light and you cant read the error.
Does someone knows how to make it not close it self?
using input doesnt work if the exception happens in a pip library
Thanks
You can use a try-except based system (that you have to build suitably to catch every exception of your code) and to print the exception you can use the module traceback like so:
try:
## code with error ##
except Exception:
print("Exception in user code:")
print("-"*60)
traceback.print_exc(file=sys.stdout)
print("-"*60)
or you could use a simpler form like:
try:
## code with error ##
except Exception as e:
print(e, file='crash log.txt')
that only prints the error class (like file not found).
I should also point out that the finally keyword exists with the purpose of executing code either if an exception arose or not:
try:
## code with error ##
except: #optional
## code to execute in case of exception ##
finally:
## code executed either way ##
Something you could do on top of that is logging everything your program does with
print(status_of_program, file=open('log.txt','a'))
this is useful to see exactly at what point the program has crashed and in general to see the program in action step by step.
But a thing you should do is properly test the program while in .py form and if it works you could possibly assume the error comes from the actual exportation method and consult the documentation or try exporting simpler programs to catch the difference (and so the error).
i'd suggest to consult:
https://docs.python.org/3/library/exceptions.html
to learn the error types and the try-except construct.
If input() doesn't work try using sys.stdout().
I gather that this is a console app. If it's a GUI app, you can use a similar approach but the details will be different.
The approach I'd use is to set sys.excepthook to be your own function that prints the exception, then waits for input. You can call the existing exception hook to actually do the printing.
import sys, traceback as tb
oldhook = sys.excepthook
def waitexcepthook(type, exception, traceback):
oldhook(type, exception, traceback)
input()
sys.excepthook = waitexcepthook
How do you best handle multiple levels of methods in a call hierarchy that raise exceptions, so that if it is a fatal error the program will exit (after displaying an error dialog)?
I'm basically coming from Java. There I would simply declare any methods as throws Exception, re-throw it and catch it somewhere at the top level.
However, Python is different. My Python code basically looks like the below.
EDIT: added much simpler code...
Main entry function (plugin.py):
def main(catalog):
print "Executing main(catalog)... "
# instantiate generator
gen = JpaAnnotatedClassGenerator(options)
# run generator
try:
gen.generate_bar() # doesn't bubble up
except ValueError as error:
Utilities.show_error("Error", error.message, "OK", "", "")
return
... usually do the real work here if no error
JpaAnnotatedClassGenerator class (engine.py):
class JpaAnnotatedClassGenerator:
def generate_bar(self):
self.generate_value_error()
def generate_value_error(self):
raise ValueError("generate_value_error() raised an error!")
I'd like to return to the caller with an exception that is to be thrown back to that ones call until it reaches the outermost try-except to display an error dialog with the exception's message.
QUESTION:
How is this best done in Python? Do I really have to repeat try-except for every method being called?
BTW: I am using Python 2.6.x and I cannot upgrade due to being bound to MySQL Workbench that provides the interpreter (Python 3 is on their upgrade list).
If you don't catch an exception, it bubbles up the call stack until someone does. If no one catches it, the runtime will get it and die with the exception error message and a full traceback. IOW, you don't have to explicitely catch and reraise your exception everywhere - which would actually defeat the whole point of having exceptions. Actually, despite being primarily used for errors / unexpected conditions, exceptions are first and foremost a control flow tool allowing to break out of the normal execution flow and pass control (and some informations) to any arbitrary place up in the call stack.
From this POV your code seems mostlt correct (caveat: I didn't bother reading the whole thing, just had a quick look), except (no pun indented) for a couple points:
First, you should define your own specific exception class(es) instead of using the builtin ValueError (you can inherit from it if it makes sense to you) so you're sure you only catch the exact exceptions you expect (quite a few layers "under" your own code could raise a ValueError that you didn't expect).
Then, you may (or not, depending on how your code is used) also want to add a catch-all top-level handler in your main() function so you can properly log (using the logger module) all errors and eventually free resources, do some cleanup etc before your process dies.
As a side note, you may also want to learn and use proper string formatting, and - if perfs are an issue at least -, avoid duplicate constant calls like this:
elif AnnotationUtil.is_embeddable_table(table) and AnnotationUtil.is_secondary_table(table):
# ...
elif AnnotationUtil.is_embeddable_table(table):
# ...
elif AnnotationUtil.is_secondary_table(table):
# ...
Given Python's very dynamic nature, neither the compiler nor runtime can safely optimize those repeated calls (the method could have been dynamically redefined between calls), so you have to do it yourself.
EDIT:
When trying to catch the error in the main() function, exceptions DON'T bubble up, but when I use this pattern one level deeper, bubbling-up seems to work.
You can easily check that it works correctly with a simple MCVE:
def deeply_nested():
raise ValueError("foo")
def nested():
return deeply_nested()
def firstline():
return nested()
def main():
try:
firstline()
except ValueError as e:
print("got {}".format(e))
else:
print("you will not see me")
if __name__ == "__main__":
main()
It appears the software that supplies the Python env is somehow treating the main plugin file in a wrong way. Looks I will have to check the MySQL Workbench guys
Uhu... Even embedded, the mechanism expection should still work as expected - at least for the part of the call stack that depends on your main function (can't tell what happens upper in the call stack). But given how MySQL treats errors (what about having your data silently truncated ?), I wouldn't be specially suprised if they hacked the runtime to silently pass any error in plugins code xD
It is fine for errors to bubble up
Python's exceptions are unchecked, meaning you have no obligation to declare or handle them. Even if you know that something may raise, only catch the error if you intend to do something with it. It is fine to have exception-transparent layers, which gracefully abort as an exception bubbles through them:
def logged_get(map: dict, key: str):
result = map[key] # this may raise, but there is no state to corrupt
# the following is not meaningful if an exception occurred
# it is fine for it to be skipped by the exception bubbling up
print(map, '[%s]' % key, '=>', result)
return result
In this case, logged_get will simply forward any KeyError (and others) that are raised by the lookup.
If an outer caller knows how to handle the error, it can do so.
So, just call self.create_collection_embeddable_class_stub the way you do.
It is fine for errors to kill the application
Even if nothing handles an error, the interpreter does. You get a stack trace, showing what went wrong and where. Fatal errors of the kind "only happens if there is a bug" can "safely" bubble up to show what went wrong.
In fact, exiting the interpreter and assertions use this mechanism as well.
>>> assert 2 < 1, "This should never happen"
Traceback (most recent call last):
File "<string>", line 1, in <module>
AssertionError: This should never happen
For many services, you can use this even in deployment - for example, systemd would log that for a Linux system service. Only try to suppress errors for the outside if security is a concern, or if users cannot handle the error.
It is fine to use precise errors
Since exceptions are unchecked, you can use arbitrary many without overstraining your API. This allows to use custom errors that signal different levels of problems:
class DBProblem(Exception):
"""Something is wrong about our DB..."""
class DBEntryInconsistent(DBProblem):
"""A single entry is broken"""
class DBInconsistent(DBProblem):
"""The entire DB is foobar!"""
It is generally a good idea not to re-use builtin errors, unless your use-case actually matches their meaning. This allows to handle errors precisely if needed:
try:
gen.generate_classes(catalog)
except DBEntryInconsistent:
logger.error("aborting due to corrupted entry")
sys.exit(1)
except DBInconsistent as err:
logger.error("aborting due to corrupted DB")
Utility.inform_db_support(err)
sys.exit(1)
# do not handle ValueError, KeyError, MemoryError, ...
# they will show up as a stack trace
I am scratching my head about what is the best-practice to get the traceback in the logfile only once. Please note that in general I know how to get the traceback into the log.
Let's assume I have a big program consisting of various modules and functions that are imported, so that it can have quite some depth and the logger is set up properly.
Whenever an exception may occur I do the following:
try:
do_something()
except MyError as err:
log.error("The error MyError occurred", exc_info=err)
raise
Note that the traceback is written to the log via the option exc_info=err.
My Problem is now that when everything gets a bit more complex and nested I loose control about how often this traceback is written to the log and it gets quite messy.
An example of the situation with my current solution for this problem is as follows:
from other_module import other_f
def main():
try:
# do something
val = other_f()
except (AlreadyLoggedError1, AlreadyLoggedError2, AlreadyLoggedError3):
# The error was caught within other_f() or deeper and
# already logged with traceback info where it occurred
# After logging it was raised like in the above example
# I do not want to log it again, so it is just raised
raise
except BroaderException as err:
# I cannot expect to have thought of all exceptions
# So in case something unexpected happened
# I want to have the traceback logged here
# since the error is not logged yet
log.error("An unecpected error occured", exc_info=err)
raise
The problem with this solution is, that I need to to keep track of all Exceptions that are already logged by myself and the line except (AlreadyLoggedError1, AlreadyLoggedError2, ...) gets arbitrary long and has to be put at any level between main() and the position the error actually occured.
So my question is: Is there some better (pythonic) way handling this? To be more specific: I want to raise the information that the exception was already logged together with the exception so that I do not have to account for that via an extra except block like in my above example.
The solution normally used for larger applications is for the low-level code to not actually do error handling itself if it's just going to be logged, but to put exception logging/handling at the highest level in the code possible, since exceptions will bubble up as far as needed. For example, libraries that send errors to a service like New Relic and Sentry don't need you to instrument each small part of your code that might throw an error, they are set up to just catch any exception and send it to a remote service for aggregation and tracking.
I have been writing unittests. Although there must be a way, I find it exceptionally difficult to get any print statement in a unittest class method to print anything, but can see what is going on when unittest shows me the diffs.
Have previously tested for exceptions with assertRaises as described here, but this is about detecting not asserting.
The tests assume a certain hardware device is plugged into the USB port when they run. The program being tested raises the following if someone tries to run it without the device connected:
RuntimeError('Device not reachable. Please connect it to USB')
I just tested sys.exit(0) inside a unittest method and that didn't exit either. Still, exiting is not that important, is probably better if all the tests fail, but it would be polite if the first test detects that Exception then prints a message to the user such as "The device needs to be connected before running unittests.". Does anyone know how to achieve this?
It is the standard class as described in the docs inheriting from unittest.TestCase with a setUp and tearDown method and the tests in between.
There are a few solutions you could try:
log and raise your RuntimeError in the setUpClass method
Use the skipUnless decorator
Intercept every error raised to test its type and log if it is a RuntimeError (see below)
To intercept every exception raised:
SYS_EXCEPT_HOOK = sys.excepthook
def _excepthook(typ, value, trace):
if typ is RuntimeError:
print("{}\n{}\n{}".format(typ.__name__, value, ''.join(traceback.format_tb(trace))))
SYS_EXCEPT_HOOK(typ, value, trace)
sys.excepthook = _excepthook
Is there anyway for the warnings.warn() function to be caught be a caller while still executing the rest of the code after the warn() call? The problem I am having is that function b will warnings.warn() if something happens, and then I want the rest of that function to finish its job and return a list of what it actually did. If a warning was thrown, I want to catch it, email it to someone, and continue on when I call that function from another module, but that isn't happening. here is what it looks like in code:
import warnings
def warn_function(arg_1):
if arg_1 > 10:
warnings.warn("Your argument was greater than 10.")
return arg_1 - 5
with warnings.catch_warnings():
warnings.filterwarnings("error")
try:
answer = warn_function(20)
except Warning:
print("A warning was thrown")
finally:
print(answer)
Yes, warnings can warn without exiting out of a function. But the way you're trying to do things just isn't going to work.
Using catch_warnings with the "error" action means you're explicitly asking Python to raise every warning as an exception. And the Python exception model doesn't have any way to resume from the point where an exception was thrown.
You can reorganize your code to provide explicit ways to "do the rest" after each possible warnings, but for non-trivial cases you either end up doing a ton of work, or building a hacky continuation-passing mechanism.
The right way to handle your use case is logging.captureWarnings. This way, all warnings go to a logger named 'py.warnings' instead of through the normal warning path. You can then configure a log handler that sends these warnings to someone via email, and you're done.
And of course once you've built this, you can use the exact same handler to get emails sent from high-severity log messages to other loggers, or to add in runtime configuration so you can turn up and down the email threshold without deploying a whole new build of the server, and so on.
If you're not already using logging, it may be easier to hook warnings manually. As the warnings introduction explains:
The printing of warning messages is done by calling showwarning(), which may be overridden; the default implementation of this function formats the message by calling formatwarning(), which is also available for use by custom implementations.
Yes, Python is encouraging you to monkeypatch a stdlib module. The code to do this looks something like:
def showwarning(message, category, filename, lineno, file=None, line=None):
fmsg = warning.formatwarning(message, category, filename, lineno, line)
# send fmsg by email
warning.showwarning = showwarning