We Keep Coding

How to catch when *any* error occurs in a whole script? [duplicate]

How do you best handle multiple levels of methods in a call hierarchy that raise exceptions, so that if it is a fatal error the program will exit (after displaying an error dialog)?
I'm basically coming from Java. There I would simply declare any methods as throws Exception, re-throw it and catch it somewhere at the top level.
However, Python is different. My Python code basically looks like the below.
EDIT: added much simpler code...
Main entry function (plugin.py):
def main(catalog):
print "Executing main(catalog)... "
# instantiate generator
gen = JpaAnnotatedClassGenerator(options)
# run generator
try:
gen.generate_bar() # doesn't bubble up
except ValueError as error:
Utilities.show_error("Error", error.message, "OK", "", "")
return
... usually do the real work here if no error
JpaAnnotatedClassGenerator class (engine.py):
class JpaAnnotatedClassGenerator:
def generate_bar(self):
self.generate_value_error()
def generate_value_error(self):
raise ValueError("generate_value_error() raised an error!")
I'd like to return to the caller with an exception that is to be thrown back to that ones call until it reaches the outermost try-except to display an error dialog with the exception's message.
QUESTION:
How is this best done in Python? Do I really have to repeat try-except for every method being called?
BTW: I am using Python 2.6.x and I cannot upgrade due to being bound to MySQL Workbench that provides the interpreter (Python 3 is on their upgrade list).

If you don't catch an exception, it bubbles up the call stack until someone does. If no one catches it, the runtime will get it and die with the exception error message and a full traceback. IOW, you don't have to explicitely catch and reraise your exception everywhere - which would actually defeat the whole point of having exceptions. Actually, despite being primarily used for errors / unexpected conditions, exceptions are first and foremost a control flow tool allowing to break out of the normal execution flow and pass control (and some informations) to any arbitrary place up in the call stack.
From this POV your code seems mostlt correct (caveat: I didn't bother reading the whole thing, just had a quick look), except (no pun indented) for a couple points:
First, you should define your own specific exception class(es) instead of using the builtin ValueError (you can inherit from it if it makes sense to you) so you're sure you only catch the exact exceptions you expect (quite a few layers "under" your own code could raise a ValueError that you didn't expect).
Then, you may (or not, depending on how your code is used) also want to add a catch-all top-level handler in your main() function so you can properly log (using the logger module) all errors and eventually free resources, do some cleanup etc before your process dies.
As a side note, you may also want to learn and use proper string formatting, and - if perfs are an issue at least -, avoid duplicate constant calls like this:
elif AnnotationUtil.is_embeddable_table(table) and AnnotationUtil.is_secondary_table(table):
# ...
elif AnnotationUtil.is_embeddable_table(table):
# ...
elif AnnotationUtil.is_secondary_table(table):
# ...
Given Python's very dynamic nature, neither the compiler nor runtime can safely optimize those repeated calls (the method could have been dynamically redefined between calls), so you have to do it yourself.
EDIT:
When trying to catch the error in the main() function, exceptions DON'T bubble up, but when I use this pattern one level deeper, bubbling-up seems to work.
You can easily check that it works correctly with a simple MCVE:
def deeply_nested():
raise ValueError("foo")
def nested():
return deeply_nested()
def firstline():
return nested()
def main():
try:
firstline()
except ValueError as e:
print("got {}".format(e))
else:
print("you will not see me")
if __name__ == "__main__":
main()
It appears the software that supplies the Python env is somehow treating the main plugin file in a wrong way. Looks I will have to check the MySQL Workbench guys
Uhu... Even embedded, the mechanism expection should still work as expected - at least for the part of the call stack that depends on your main function (can't tell what happens upper in the call stack). But given how MySQL treats errors (what about having your data silently truncated ?), I wouldn't be specially suprised if they hacked the runtime to silently pass any error in plugins code xD

It is fine for errors to bubble up
Python's exceptions are unchecked, meaning you have no obligation to declare or handle them. Even if you know that something may raise, only catch the error if you intend to do something with it. It is fine to have exception-transparent layers, which gracefully abort as an exception bubbles through them:
def logged_get(map: dict, key: str):
result = map[key] # this may raise, but there is no state to corrupt
# the following is not meaningful if an exception occurred
# it is fine for it to be skipped by the exception bubbling up
print(map, '[%s]' % key, '=>', result)
return result
In this case, logged_get will simply forward any KeyError (and others) that are raised by the lookup.
If an outer caller knows how to handle the error, it can do so.
So, just call self.create_collection_embeddable_class_stub the way you do.
It is fine for errors to kill the application
Even if nothing handles an error, the interpreter does. You get a stack trace, showing what went wrong and where. Fatal errors of the kind "only happens if there is a bug" can "safely" bubble up to show what went wrong.
In fact, exiting the interpreter and assertions use this mechanism as well.
>>> assert 2 < 1, "This should never happen"
Traceback (most recent call last):
File "<string>", line 1, in <module>
AssertionError: This should never happen
For many services, you can use this even in deployment - for example, systemd would log that for a Linux system service. Only try to suppress errors for the outside if security is a concern, or if users cannot handle the error.
It is fine to use precise errors
Since exceptions are unchecked, you can use arbitrary many without overstraining your API. This allows to use custom errors that signal different levels of problems:
class DBProblem(Exception):
"""Something is wrong about our DB..."""
class DBEntryInconsistent(DBProblem):
"""A single entry is broken"""
class DBInconsistent(DBProblem):
"""The entire DB is foobar!"""
It is generally a good idea not to re-use builtin errors, unless your use-case actually matches their meaning. This allows to handle errors precisely if needed:
try:
gen.generate_classes(catalog)
except DBEntryInconsistent:
logger.error("aborting due to corrupted entry")
sys.exit(1)
except DBInconsistent as err:
logger.error("aborting due to corrupted DB")
Utility.inform_db_support(err)
sys.exit(1)
# do not handle ValueError, KeyError, MemoryError, ...
# they will show up as a stack trace

Killing cv2 read process after N time

I'm desperate.
My code reads nframe in videos, sometimes the code just stop for no reason, and no error.
So I decided to somehow raise an error.
The thing is, the code does raise an error, but it ignores it for some reason, and just works as normal.
*Ive provided a code block on which exactly the same method works.
handler:
def handler(signum,frame):
print("error") ## This is printed
raise Exception('time out') ## I guess this is getting raised
Code part i want to wrap:
for i in range(0,int(frame_count), nframe): # basicly loads every nframe from the video
try:
frame = video.set(1,i)
signal.signal(signal.SIGALRM), handler)
signal.alarm(1) # At this point, the 'handler' did raise the error, but it did not kill this 'try' block.
_n,frame = video.read() # This line sometimes gets for infinit amount of time, and i want to wrap it
except Exception as e:
print('test') # Code does not get here, yet the 'handler' does raise an exception
raise e
# Here i need to return False, or rise an error, but the code just does not get here.
An example where exactly the same method will work:
import signal
import time
def handler(signum, frame):
raise Exception('time out')
def function():
try:
signal.signal(signal.SIGALRM,handler)
signal.alarm(5) # 5 seconds till raise
time.sleep(10) # does not get here, an Exception is raised after 5 seconds
except Exception as e:
raise e # This will indeed work

My guess is that the read() call is blocked somewhere inside C code. The signal handler runs, puts an exception into the Python interpreter somewhere, but the exception isn't handled until the Python interpreter regains control. This is a limitation documented in the signal module:
A long-running calculation implemented purely in C (such as regular expression matching on a large body of text) may run uninterrupted for an arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes.
One possible workaround is to read frames on a separate process using the multiprocessing module, and return them to the main process using a multiprocessing.Queue (from which you can get with a timeout). However, there will be extra overhead in sending the frames between processes.
Another approach might be to try and avoid the root of the problem. OpenCV has different video backends (V4L, GStreamer, ffmpeg, ...); one of them might work where another doesn't. Using the second argument to the VideoCapture constructor, you can indicate a preference for which backend to use:
cv.VideoCapture(..., cv.CAP_FFMPEG)
See the documentation for the full list of backends. Depending on your platform and OpenCV build, not all of them will be available.

Python3: purge exception chain?

I'm trying to raise an exception within an except: block but the interpreter tries to be helpful and prints stack traces 'by force'. Is it possible to avoid this?
A little bit of background information:
I'm toying with urwid, a TUI library for python. The user interface is started by calling urwid.MainLoop.run() and ended by raising urwid.ExitMainLoop(). So far this works fine but what happens when another exception is raised? E.g. when I'm catching KeyboardInterrupt (the urwid MainLoop does not), I do some cleanup and want to end the user interface - by raising the appropriate exception. But this results in a screen full of stack traces.
Some little research showed python3 remembers chained exceptions and one can explicitly raise an exception with a 'cause': raise B() from A(). I learned a few ways to change or append data regarding the raised exceptions but I found no way to 'disable' this feature. I'd like to avoid the printing of stack traces and lines like The above exception was the direct cause of... and just raise the interface-ending exception within an except: block like I would outside of one.
Is this possible or am I doing something fundamentally wrong?
Edit:
Here's an example resembling my current architecture, resulting in the same problem:
#!/usr/bin/env python3
import time
class Exit_Main_Loop(Exception):
pass
# UI main loop
def main_loop():
try:
while True:
time.sleep(0.1)
except Exit_Main_Loop as e:
print('Exit_Main_Loop')
# do some UI-related clean up
# my main script
try:
main_loop()
except KeyboardInterrupt as e:
print('KeyboardInterrupt')
# do some clean up
raise Exit_Main_Loop() # signal the UI to terminate
Unfortunately I can't change main_loop to except KeyboardInterrupt as well. Is there a pattern to solve this?

I still don't quite understand your explanation, but from the code:
try:
main_loop()
except KeyboardInterrupt as e:
print('KeyboardInterrupt')
# do some clean up
raise Exit_Main_Loop() # signal the UI to terminate
There is no way that main_loop could ever see the Exit_Main_Loop() exception. By the time you get to the KeyboardInterrupt handle, main_loop is guaranteed to have already finished (in this case, because of an unhandled KeyboardInterrupt), so its exception handler is no longer active.
So, what happens is that you raise a new exception that nobody catches. And when an exception gets to the top of your code without being handled, Python handles it automatically by printing a traceback and quitting.
If you want to convert one type of exception into another so main_loop can handle it, you have to do that somewhere inside the try block.
You say:
Unfortunately I can't change main_loop to except KeyboardInterrupt as well.
If that's true, there's no real answer to your problem… but I'm not sure there's a problem in the first place, other than the one you created. Just remove the Exit_Main_Loop() from your code, and isn't it already doing what you wanted? If you're just trying to prevent Python from printing a traceback and exiting, this will take care of it for you.
If there really is a problem—e.g., the main_loop code has some cleanup code that you need to get executed no matter what, and it's not getting executed because it doesn't handle KeyboardInterrupt—there are two ways you could work around this.
First, as the signal docs explain:
The signal.signal() function allows to define custom handlers to be executed when a signal is received. A small number of default handlers are installed: … SIGINT is translated into a KeyboardInterrupt exception.
So, all you have to do is replace the default handler with a different one:
def handle_sigint(signum, frame):
raise ExitMainLoop()
signal.signal(signal.SIGINT, handle_sigint)
Just do this before you start main_loop, and you should be fine. Keep in mind that there are some limitations with threaded programs, and with Windows, but if none of those limitations apply, you're golden; a ctrl-C will trigger an ExitMainLoop exception instead of a KeyboardInterrupt, so the main loop will handle it. (You may want to also add an except ExitMainLoop: block in your wrapper code, in case there's an exception outside of main_loop. However, you could easily write a contextmanager that sets and restores the signal around the call to main_loop, so there isn't any outside code that could possibly raise it.)
Alternatively, even if you can't edit the main_loop source code, you can always monkeypatch it at runtime. Without knowing what the code looks like, it's impossible to explain exactly how to do this, but there's almost always a way to do it.

Do we use try,except in every single function?

Should we always enclose every function we write with a try...except block?
I ask this because sometimes in one function we raise Exception, and the caller that calls this function doesn't have exception
def caller():
stdout, stderr = callee(....)
def callee():
....
if stderr:
raise StandardError(....)
then our application crash. In this obvious case, I am tempted to enclose callee and caller with try..except.
But I've read so many Python code and they don't do these try..block all the time.
def cmd(cmdl):
try:
pid = Popen(cmdl, stdout=PIPE, stderr=PIPE)
except Exception, e:
raise e
stdout, stderr = pid.communicate()
if pid.returncode != 0:
raise StandardError(stderr)
return (stdout, stderr)
def addandremove(*args,**kwargs):
target = kwargs.get('local', os.getcwd())
f = kwargs.get('file', None)
vcs = kwargs.get('vcs', 'hg')
if vcs is "hg":
try:
stdout, stderr = cmd(['hg', 'addremove', '--similarity 95'])
except StandardError, e:
// do some recovery
except Exception, e:
// do something meaningful
return True
The real thing that bothers me is this:
If there is a 3rd function that calls addandremove() in one of the statements, do we also surround the call with a try..except block? What if this 3rd function has 3 lines, and each function calls itself has a try-except? I am sorry for building this up. But this is the sort of problem I don't get.

Exceptions are, as the name implies, for exceptional circumstances - things that shouldn't really happen
..and because they probably shouldn't happen, for the most part, you can ignore them. This is a good thing.
There are times where you do except an specific exception, for example if I do:
urllib2.urlopen("http://example.com")
In this case, it's reasonable to expect the "cannot contact server" error, so you might do:
try:
urllib2.urlopen("http://example.com")
except urllib2.URLError:
# code to handle the error, maybe retry the server,
# report the error in a helpful way to the user etc
However it would be futile to try and catch every possible error - there's an inenumerable amount of things that could potentially go wrong.. As a strange example, what if a module modifies urllib2 and removes the urlopen attribute - there's no sane reason to expect that NameError, and no sane way you could handle such an error, therefore you just let the exception propagate up
Having your code exit with a traceback is a good thing - it allows you to easily see where the problem originate, and what caused it (based on the exception and it's message), and fix the cause of the problem, or handle the exception in the correct location...
In short, handle exceptions only if you can do something useful with them. If not, trying to handle all the countless possible errors will only make your code buggier and harder to fix
In the example you provide, the try/except blocks do nothing - they just reraise the exception, so it's identical to the much tidier:
def cmd(cmdl):
pid = Popen(cmdl, stdout=PIPE, stderr=PIPE)
stdout, stderr = pid.communicate()
if pid.returncode != 0:
raise StandardError(stderr)
return (stdout, stderr)
# Note: Better to use actual args instead of * and **,
# gives better error handling and docs from help()
def addandremove(fname, local = None, vcs = 'hg'):
if target is None:
target = os.getcwd()
if vcs is "hg":
stdout, stderr = cmd(['hg', 'addremove', '--similarity 95'])
return True
About the only exception-handling related thing I might expect is to handle if the 'hg' command isn't found, the resulting exception isn't particularly descriptive. So for a library, I'd do something like:
class CommandNotFound(Exception): pass
def cmd(cmdl):
try:
pid = Popen(cmdl, stdout=PIPE, stderr=PIPE)
except OSError, e:
if e.errno == 2:
raise CommandNotFound("The command %r could not be found" % cmdl)
else:
# Unexpected error-number in OSError,
# so a bare "raise" statement will reraise the error
raise
stdout, stderr = pid.communicate()
if pid.returncode != 0:
raise StandardError(stderr)
return (stdout, stderr)
This just wraps the potentially confusing "OSError" exception in the clearer "CommandNotFound".
Rereading the question, I suspect you might be misunderstanding something about how Python exceptions work (the "and the caller that calls this function doesn't have exception" bit, so to be hopefully clarify:
The caller function does not need any knowledge of the exceptions that might be raised from the children function. You can just call the cmd() function and hope it works fine.
Say your code is in a mystuff module, and someone else wants to use it, they might do:
import mystuff
mystuff.addandremove("myfile.txt")
Or, maybe they want to give a nice error message and exit if the user doesn't have hg installed:
import mystuff
try:
mystuff.addandremove("myfile.txt")
except mystuff.CommandNotFound:
print "You don't appear to have the 'hg' command installed"
print "You can install it with by... etc..."
myprogram.quit("blahblahblah")

You should use a try catch block so that you can specifically locate the source of the exception. You can put these blocks around anything you want, but unless they produce some sort of useful information, there is no need to add them.

try/except clauses are really only useful if you know how to handle the error that gets raised. Take the following program:
while True:
n=raw_input("Input a number>")
try:
n=float(n)
break
except ValueError:
print ("That wasn't a number!") #Try again.
However, you may have a function like:
def mult_2_numbers(x,y):
return x*y
and the user may try to use it as:
my_new_list=mult_2_numbers([7,3],[8,7])
The user could put this in a try/except block, but then my_new_list wouldn't be defined and would probably just raise an exception later (likely a NameError). In that case, you'd make it harder to debug because the line number/information in the traceback is pointing to a piece of code which isn't the real problem.

There are a couple of programming decisions for your coding team to make regarding introspection type tools and "exception" handling.
One good place to use exception handling is with Operating System calls such as file operations. Reasoning is, for example, a file may have it's access restricted from being accessed by the client appliction. That access restriction is typically an OS-admin task, not the Python application function. So exceptions would be a good use where you application does NOT have control.
You can mutate the previous paragraph's application of exceptions to be broader, in the sense of using Exceptions for things outside the control of what your team codes, to such things as all "devices" or OS resources like OS timers, symbolic links, network connections, etc.
Another common use case is when you use a library or package that is -designed- to through lots of exceptions, and that package expects you to catch or code for that. Some packages are designed to throw as few exceptions as possible and expect you to code based on return values. Then your exceptions should be rare.
Some coding teams use exceptions as a way to log fringe cases of "events" within your application.
I find when desiding whether to use exceptions I am either programing to minimize failure by not having a lot of try/except and have calling routines expect either valid return values or expect an invalid return values. OR I program for failure. That is to say, I program for expected failure by functions, which I use alot of try/except blocks. Think of programming for "expected" failure like working with TCP; the packets aren't garenteed to get there or even in order, but there are exception handling with TCP by use of send/read retrys and such.
Personally I use try-except blocks around the smallest possible block sizes, usually one line of code.

It's up to you; the main exceptions' roles involve (a quote from this splendid book):
Error handling
Event notification
Special-case handling
Termination actions
Unusual control flows

When you know what the error will be, use try/except for debugging purpose. Otherwise, you don't have to use try/except for every function.

Overriding basic signals (SIGINT, SIGQUIT, SIGKILL??) in Python

I'm writing a program that adds normal UNIX accounts (i.e. modifying /etc/passwd, /etc/group, and /etc/shadow) according to our corp's policy. It also does some slightly fancy stuff like sending an email to the user.
I've got all the code working, but there are three pieces of code that are very critical, which update the three files above. The code is already fairly robust because it locks those files (ex. /etc/passwd.lock), writes to to a temporary files (ex. /etc/passwd.tmp), and then, overwrites the original file with the temporary. I'm fairly pleased that it won't interefere with other running versions of my program or the system useradd, usermod, passwd, etc. programs.
The thing that I'm most worried about is a stray ctrl+c, ctrl+d, or kill command in the middle of these sections. This has led me to the signal module, which seems to do precisely what I want: ignore certain signals during the "critical" region.
I'm using an older version of Python, which doesn't have signal.SIG_IGN, so I have an awesome "pass" function:
def passer(*a):
pass
The problem that I'm seeing is that signal handlers don't work the way that I expect.
Given the following test code:
def passer(a=None, b=None):
pass
def signalhander(enable):
signallist = (signal.SIGINT, signal.SIGQUIT, signal.SIGABRT, signal.SIGPIPE, signal.SIGALRM, signal.SIGTERM, signal.SIGKILL)
if enable:
for i in signallist:
signal.signal(i, passer)
else:
for i in signallist:
signal.signal(i, abort)
return
def abort(a=None, b=None):
sys.exit('\nAccount was not created.\n')
return
signalhander(True)
print('Enabled')
time.sleep(10) # ^C during this sleep
The problem with this code is that a ^C (SIGINT) during the time.sleep(10) call causes that function to stop, and then, my signal handler takes over as desired. However, that doesn't solve my "critical" region problem above because I can't tolerate whatever statement encounters the signal to fail.
I need some sort of signal handler that will just completely ignore SIGINT and SIGQUIT.
The Fedora/RH command "yum" is written is Python and does basically exactly what I want. If you do a ^C while it's installing anything, it will print a message like "Press ^C within two seconds to force kill." Otherwise, the ^C is ignored. I don't really care about the two second warning since my program completes in a fraction of a second.
Could someone help me implement a signal handler for CPython 2.3 that doesn't cause the current statement/function to cancel before the signal is ignored?
As always, thanks in advance.
Edit: After S.Lott's answer, I've decided to abandon the signal module.
I'm just going to go back to try: except: blocks. Looking at my code there are two things that happen for each critical region that cannot be aborted: overwriting file with file.tmp and removing the lock once finished (or other tools will be unable to modify the file, until it is manually removed). I've put each of those in their own function inside a try: block, and the except: simply calls the function again. That way the function will just re-call itself in the event of KeyBoardInterrupt or EOFError, until the critical code is completed.
I don't think that I can get into too much trouble since I'm only catching user provided exit commands, and even then, only for two to three lines of code. Theoretically, if those exceptions could be raised fast enough, I suppose I could get the "maximum reccurrsion depth exceded" error, but that would seem far out.
Any other concerns?
Pesudo-code:
def criticalRemoveLock(file):
try:
if os.path.isFile(file):
os.remove(file)
else:
return True
except (KeyboardInterrupt, EOFError):
return criticalRemoveLock(file)
def criticalOverwrite(tmp, file):
try:
if os.path.isFile(tmp):
shutil.copy2(tmp, file)
os.remove(tmp)
else:
return True
except (KeyboardInterrupt, EOFError):
return criticalOverwrite(tmp, file)

There is no real way to make your script really save. Of course you can ignore signals and catch a keyboard interrupt using try: except: but it is up to your application to be idempotent against such interrupts and it must be able to resume operations after dealing with an interrupt at some kind of savepoint.
The only thing that you can really to is to work on temporary files (and not original files) and move them after doing the work into the final destination. I think such file operations are supposed to be "atomic" from the filesystem prospective. Otherwise in case of an interrupt: restart your processing from start with clean data.