We Keep Coding

How to ignore all potential exceptions in Python?

I'm writing a custom backup script in Python. Sometimes the mkdir function or the print function or whatever function are failing for diverse reasons. Such exceptions stop the whole script and halt the backup in the middle, which is very frustrating. So far, I've managed these problems by adding try: ... except: ... statements and properly managing these exceptions. However, one day some other statement or function might raise an exception as well because of some other reason that hasn't been triggered yet.
Is there a way to tell a script to proceed anyway? An equivalent of wrapping every single statement of the code in a try: ... except: pass clause? A log would be better of course.
I've noticed that when programming with GUI toolkits like Tkinter, the application keeps running even if exceptions are raised. Is it possible to accomplish this type of thing with the console?

There actually is a module that is supposed to do exactly that: https://github.com/ajalt/fuckitpy.
Although it was obviously written as a joke. I cannot imagine a situation where doing something like that is a good idea. God, I can't believe I'm even suggesting that as a solution.
What you should do instead is identify what lines of code can produce what kind of errors, and handle those errors properly. There is only so much places where errors can actually happen - mostly while interfacing with outside systems, including filesystem, network, user input etc. And remember that actually failing is often better than continuing "work" and messing up your data, files and so on. Exceptions are there for a reason, they are not a result of Guido's malice.

Python has no way of doing that, and for good reasons.
It seems you're confused about what does it mean to write "robust" software: a robust program is not a program that is hard to kill and that will keep running no matter what, but a program that will handle edge cases properly. Keeping running is NOT enough... keeping running doing sensible things is the key point.
Unfortunately there's no way to do reasonable things automatically and you've to think on a case-by-case basis how to handle the error.
Beware that if a program has a lot of catch it's rarely a good program. Exceptions are meant to be raised in a lot of places and caught almost nowhere.
Note also that every catch is potentially a source of bugs... for example:
try:
print my_dict[foo()]
except KeyError:
...
cannot distinguish if the KeyError is coming for accessing a non-existing key in my_dict or if instead escaped from foo(). Rarely the two cases should be handled the same way...
Better is to write:
key = foo()
if key in my_dict:
print my_dict[key]
else:
...
so that only the side case of missing key in my_dict is handled and instead a KeyError exception will stop the program (stopping a program when you're not sure of what it's doing is the only reasonable thing to do).

Python has 'BaseException' as the base class for Exception classes. You can catch and ignore the base class Exception and that will cover all exceptions.
try
... your code here ...
except BaseException as exp:
print "A General Exception Occurred"

try:
# code segment
except:
pass
pass keyword will ignore all the exceptions

Normally, this one should catch everything:
try:
....
except:
pass
Only problem is, that you don't get the exception object with this syntax, but that was not asked for in this case.

You can add a general except block like #Kanwar Saad proposed. The question is, can you continue with your program in a valid state after the exception has been raised?
From the Zen of Python:
Errors should never pass silently.
Unless explicitly silenced.
Trying to catch all exceptions you know is in my opinion is the best way to go here. If you can not explicitly catch an exception you should not try to work around it. You (and your users) should know what exactly went wrong, otherwise your code might become a nightmare to debug.
If you are worried about losing backup data maybe you could do something like this:
def save_unfinished_backup():
# try to find a graceful exit without losing any data
try:
# some code
except OSError:
# handle oS Errors
except Exception:
save_unfinished_backup()
raise
This way you get both: A chance to fend of data loss and the exact error to debug it.
I hope this helps!
On a funny note: You could also use the fuckit module. Which silences ALL errors, including syntax errors. Do not, ever, use this in productive code though.
This should work perfectly. It will not print the "foo", but you will reach the print("bar") without a crash.
import fuckit
with fuckit:
prnt("foo")
print("bar")

New answer for new Gen...
Python now ships with suppress(), this tells the interpreter to suppress the indicated exceptions while running.
Can be easily imported used as below
from contextlib import suppress
with suppress(ValueError):
int('this wont catch')
print('yea')
The above will work, wont raise the ValueError exception of changing invalid int string to int...
It's more clean than third party libraries.
Happy Hacking

Do we use try,except in every single function?

Should we always enclose every function we write with a try...except block?
I ask this because sometimes in one function we raise Exception, and the caller that calls this function doesn't have exception
def caller():
stdout, stderr = callee(....)
def callee():
....
if stderr:
raise StandardError(....)
then our application crash. In this obvious case, I am tempted to enclose callee and caller with try..except.
But I've read so many Python code and they don't do these try..block all the time.
def cmd(cmdl):
try:
pid = Popen(cmdl, stdout=PIPE, stderr=PIPE)
except Exception, e:
raise e
stdout, stderr = pid.communicate()
if pid.returncode != 0:
raise StandardError(stderr)
return (stdout, stderr)
def addandremove(*args,**kwargs):
target = kwargs.get('local', os.getcwd())
f = kwargs.get('file', None)
vcs = kwargs.get('vcs', 'hg')
if vcs is "hg":
try:
stdout, stderr = cmd(['hg', 'addremove', '--similarity 95'])
except StandardError, e:
// do some recovery
except Exception, e:
// do something meaningful
return True
The real thing that bothers me is this:
If there is a 3rd function that calls addandremove() in one of the statements, do we also surround the call with a try..except block? What if this 3rd function has 3 lines, and each function calls itself has a try-except? I am sorry for building this up. But this is the sort of problem I don't get.

Exceptions are, as the name implies, for exceptional circumstances - things that shouldn't really happen
..and because they probably shouldn't happen, for the most part, you can ignore them. This is a good thing.
There are times where you do except an specific exception, for example if I do:
urllib2.urlopen("http://example.com")
In this case, it's reasonable to expect the "cannot contact server" error, so you might do:
try:
urllib2.urlopen("http://example.com")
except urllib2.URLError:
# code to handle the error, maybe retry the server,
# report the error in a helpful way to the user etc
However it would be futile to try and catch every possible error - there's an inenumerable amount of things that could potentially go wrong.. As a strange example, what if a module modifies urllib2 and removes the urlopen attribute - there's no sane reason to expect that NameError, and no sane way you could handle such an error, therefore you just let the exception propagate up
Having your code exit with a traceback is a good thing - it allows you to easily see where the problem originate, and what caused it (based on the exception and it's message), and fix the cause of the problem, or handle the exception in the correct location...
In short, handle exceptions only if you can do something useful with them. If not, trying to handle all the countless possible errors will only make your code buggier and harder to fix
In the example you provide, the try/except blocks do nothing - they just reraise the exception, so it's identical to the much tidier:
def cmd(cmdl):
pid = Popen(cmdl, stdout=PIPE, stderr=PIPE)
stdout, stderr = pid.communicate()
if pid.returncode != 0:
raise StandardError(stderr)
return (stdout, stderr)
# Note: Better to use actual args instead of * and **,
# gives better error handling and docs from help()
def addandremove(fname, local = None, vcs = 'hg'):
if target is None:
target = os.getcwd()
if vcs is "hg":
stdout, stderr = cmd(['hg', 'addremove', '--similarity 95'])
return True
About the only exception-handling related thing I might expect is to handle if the 'hg' command isn't found, the resulting exception isn't particularly descriptive. So for a library, I'd do something like:
class CommandNotFound(Exception): pass
def cmd(cmdl):
try:
pid = Popen(cmdl, stdout=PIPE, stderr=PIPE)
except OSError, e:
if e.errno == 2:
raise CommandNotFound("The command %r could not be found" % cmdl)
else:
# Unexpected error-number in OSError,
# so a bare "raise" statement will reraise the error
raise
stdout, stderr = pid.communicate()
if pid.returncode != 0:
raise StandardError(stderr)
return (stdout, stderr)
This just wraps the potentially confusing "OSError" exception in the clearer "CommandNotFound".
Rereading the question, I suspect you might be misunderstanding something about how Python exceptions work (the "and the caller that calls this function doesn't have exception" bit, so to be hopefully clarify:
The caller function does not need any knowledge of the exceptions that might be raised from the children function. You can just call the cmd() function and hope it works fine.
Say your code is in a mystuff module, and someone else wants to use it, they might do:
import mystuff
mystuff.addandremove("myfile.txt")
Or, maybe they want to give a nice error message and exit if the user doesn't have hg installed:
import mystuff
try:
mystuff.addandremove("myfile.txt")
except mystuff.CommandNotFound:
print "You don't appear to have the 'hg' command installed"
print "You can install it with by... etc..."
myprogram.quit("blahblahblah")

You should use a try catch block so that you can specifically locate the source of the exception. You can put these blocks around anything you want, but unless they produce some sort of useful information, there is no need to add them.

try/except clauses are really only useful if you know how to handle the error that gets raised. Take the following program:
while True:
n=raw_input("Input a number>")
try:
n=float(n)
break
except ValueError:
print ("That wasn't a number!") #Try again.
However, you may have a function like:
def mult_2_numbers(x,y):
return x*y
and the user may try to use it as:
my_new_list=mult_2_numbers([7,3],[8,7])
The user could put this in a try/except block, but then my_new_list wouldn't be defined and would probably just raise an exception later (likely a NameError). In that case, you'd make it harder to debug because the line number/information in the traceback is pointing to a piece of code which isn't the real problem.

There are a couple of programming decisions for your coding team to make regarding introspection type tools and "exception" handling.
One good place to use exception handling is with Operating System calls such as file operations. Reasoning is, for example, a file may have it's access restricted from being accessed by the client appliction. That access restriction is typically an OS-admin task, not the Python application function. So exceptions would be a good use where you application does NOT have control.
You can mutate the previous paragraph's application of exceptions to be broader, in the sense of using Exceptions for things outside the control of what your team codes, to such things as all "devices" or OS resources like OS timers, symbolic links, network connections, etc.
Another common use case is when you use a library or package that is -designed- to through lots of exceptions, and that package expects you to catch or code for that. Some packages are designed to throw as few exceptions as possible and expect you to code based on return values. Then your exceptions should be rare.
Some coding teams use exceptions as a way to log fringe cases of "events" within your application.
I find when desiding whether to use exceptions I am either programing to minimize failure by not having a lot of try/except and have calling routines expect either valid return values or expect an invalid return values. OR I program for failure. That is to say, I program for expected failure by functions, which I use alot of try/except blocks. Think of programming for "expected" failure like working with TCP; the packets aren't garenteed to get there or even in order, but there are exception handling with TCP by use of send/read retrys and such.
Personally I use try-except blocks around the smallest possible block sizes, usually one line of code.

It's up to you; the main exceptions' roles involve (a quote from this splendid book):
Error handling
Event notification
Special-case handling
Termination actions
Unusual control flows

When you know what the error will be, use try/except for debugging purpose. Otherwise, you don't have to use try/except for every function.

Bad idea to catch all exceptions in Python

Why is it a bad idea to catch all exceptions in Python ?
I understand that catching all exceptions using the except: clause will even catch the 'special' python exceptions: SystemExit, KeyboardInterrupt, and GeneratorExit. So why not just use a except Exception: clause to catch all exceptions?

Because it's terribly nonspecific and it doesn't enable you to do anything interesting with the exception. Moreover, if you're catching every exception there could be loads of exceptions that are happening that you don't even know are happening (which could cause your application to fail without you really knowing why). You should be able to predict (either through reading documentation or experimentation) specifically which exceptions you need to handle and how to handle them, but if you're blindly suppressing all of them from the beginning you'll never know.
So, by popular request, here's an example. A programmer is writing Python code and she gets an IOError. Instead of investigating further, she decides to catch all exceptions:
def foo():
try:
f = open("file.txt")
lines = f.readlines()
return lines[0]
except:
return None
She doesn't realize the issue in his ways: what if the file exists and is accessible, but it is empty? Then this code will raise an IndexError (since the list lines is empty). So she'll spend hours wondering why she's getting None back from this function when the file exists and isn't locked without realizing something that would be obvious if she had been more specific in catching errors, which is that she's accessing data that might not exist.

Because you probably want to handle each exception differently. It's not the same thing to have a KeyInterrupt than to have a Encoding problem, or an OS one... You can catch specific exceptions one after the other.
try:
XXX
except TYPE:
YYY
except TYPE:
ZZZ

Handling Exceptions with else clause

The Python Tutorial states:
The try ... except statement has an optional else clause, which,
when present, must follow all except clauses. It is useful for code
that must be executed if the try clause does not raise an exception.
For example:
for arg in sys.argv[1:]:
try:
f = open(arg, 'r')
except IOError:
print 'cannot open', arg
else:
print arg, 'has', len(f.readlines()), 'lines'
f.close()
The use of the else clause is better than adding additional code
to the try clause because it avoids accidentally catching an exception
that wasn’t raised by the code being protected by the try ... except
statement.
Question 1>　Ａｆｔｅｒ reading the above document, I still don't get the idea why we cannot simply move the code from else clause into try clause.
Question 2> How does try clause can accidentally catch an exception since all those catches are done in the except clause, right?

You could put the else code in the try suite, but then you'd catch any exceptions that might be raised there. If you didn't intend that to happen, it would be "accidental," hence the wording of the document you linked to.
Best practice is to put as little code as possible in a try block, so that when an error occurs, you know what operation caused it and can handle it appropriately. If you have five lines of code in a try block and only expect one of them to ever raise an exception, your exception-handling code will be ill-prepared when an exception occurs in a line you didn't expect it to. Better in that case to let the exception be raised than handle it the wrong way.

If you move the code from the else into the try then that becomes part of the "critical path" which can raise an exception. If f.readlines() raises some sort of exception (perhaps an I/O error while reading the file because of a bad sector on the disk) then that error will be conflated with the one error that you currently catch. (Technically the "cannot open" error message would be wrong at that point ... because opening a file can succeed when reading it later fails; in fact opening it must succeed before you can even get an I/O error while processing it).
Normally you'd use a pattern more like:
foo = None
try:
# some code to access/load/initialize foo from an external source
except ...:
# handle various types of file open/read, database access, etc errors
else:
foo = something
... so that your subsequently run code and simply check if foo is None and use it or
work around it's unavailability in whatever way you see fit.

Answer for both questions is similar,
If you move code to the try clause, then you can not be sure from where the exception is coming from. Thus if you have another line of code that produces an unexpected IOError you could end searching for a problem where there is not.
So to better disect your code you want to simplify as much as possible the lines in the try making the catch as especific as possible.

1) Of course you could just move code from the else clause into the try clause. You could move it outside of the try block entirely, but this allows extra flexibility, and further modulation of the code. Also, perhaps specificity with the errors being caught. You could list a whole load of different exceptions that might be likely to happen, each with different statements. The stuff in the else clause would still only happen if no exception was raised, after the execution of the final line in the try block. e.g. printing a successful return message.
Also, try clauses add some extra CPU overhead with regards to garbage collection management and error tracing, so anything outside of the try clause is not protected in the same manner and may run more efficiently.
2) Error catching is quite specific. e.g. The except clause in your above example will only be run if an IOError is raised when running the f = open(arg,'r') line. If you want to catch any form of exception, use except Exception:.

Python Ignore Exception and Go Back to Where I Was

I know using below code to ignore a certain exception, but how to let the code go back to where it got exception and keep executing? Say if the exception 'Exception' raises in do_something1, how to make the code ignore it and keep finishing do_something1 and process do_something2? My code just go to finally block after process pass in except block. Please advise, thanks.
try:
do_something1
do_something2
do_something3
do_something4
except Exception:
pass
finally:
clean_up
EDIT:
Thanks for the reply. Now I know what's the correct way to do it. But here's another question, can I just ignore a specific exception (say if I know the error number). Is below code possible?
try:
do_something1
except Exception.strerror == 10001:
pass
try:
do_something2
except Exception.strerror == 10002:
pass
finally:
clean_up
do_something3
do_something4

There's no direct way for the code to go back inside the try-except block. If, however, you're looking at trying to execute these different independant actions and keep executing when one fails (without copy/pasting the try/except block), you're going to have to write something like this:
actions = (
do_something1, do_something2, #...
)
for action in actions:
try:
action()
except Exception, error:
pass

update. The way to ignore specific exceptions is to catch the type of exception that you want, test it to see if you want to ignore it and re-raise it if you dont.
try:
do_something1
except TheExceptionTypeThatICanHandleError, e:
if e.strerror != 10001:
raise
finally:
clean_up
Note also, that each try statement needs its own finally clause if you want it to have one. It wont 'attach itself' to the previous try statement. A raise statement with nothing else is the correct way to re-raise the last exception. Don't let anybody tell you otherwise.
What you want are continuations which python doesn't natively provide. Beyond that, the answer to your question depends on exactly what you want to do. If you want do_something1 to continue regardless of exceptions, then it would have to catch the exceptions and ignore them itself.
if you just want do_something2 to happen regardless of if do_something1 completes, you need a separate try statement for each one.
try:
do_something1()
except:
pass
try:
do_something2()
except:
pass
etc. If you can provide a more detailed example of what it is that you want to do, then there is a good chance that myself or someone smarter than myself can either help you or (more likely) talk you out of it and suggest a more reasonable alternative.

This is pretty much missing the point of exceptions.
If the first statement has thrown an exception, the system is in an indeterminate state and you have to treat the following statement as unsafe to run.
If you know which statements might fail, and how they might fail, then you can use exception handling to specifically clean up the problems which might occur with a particular block of statements before moving on to the next section.
So, the only real answer is to handle exceptions around each set of statements that you want to treat as atomic

you could have all of the do_something's in a list, and iterate through them like this, so it's no so wordy. You can use lambda functions instead if you require arguments for the working functions
work = [lambda: dosomething1(args), dosomething2, lambda: dosomething3(*kw, **kwargs)]
for each in work:
try:
each()
except:
pass
cleanup()

Exceptions are usually raised when a performing task can not be completed in a manner intended by the code due to certain reasons. This is usually raised as exceptions. Exceptions should be handled and not ignored. The whole idea of exception is that the program can not continue in the normal execution flow without abnormal results.
What if you write a code to open a file and read it? What if this file does not exist?
It is much better to raise exception. You can not read a file where none exists. What you can do is handle the exception, let the user know that no such file exists. What advantage would be obtained for continuing to read the file when a file could not be opened at all.
In fact the above answers provided by Aaron works on the principle of handling your exceptions.

I posted this recently as an answer to another question. Here you have a function that returns a function that ignores ("traps") specified exceptions when calling any function. Then you invoke the desired function indirectly through the "trap."
def maketrap(*exceptions):
def trap(func, *args, **kwargs):
try:
return func(*args, **kwargs)
except exceptions:
return None
return trap
# create a trap that ignores all exceptions
trapall = maketrap(Exception)
# create a trap that ignores two exceptions
trapkeyattrerr = maketrap(KeyError, AttributeError)
# Now call some functions, ignoring specific exceptions
trapall(dosomething1, arg1, arg2)
trapkeyattrerr(dosomething2, arg1, arg2, arg3)
In general I'm with those who say that ignoring exceptions is a bad idea, but if you do it, you should be as specific as possible as to which exceptions you think your code can tolerate.

Python 3.4 added contextlib.suppress(), a context manager that takes a list of exceptions and suppresses them within the context:
with contextlib.suppress(IOError):
print('inside')
print(pathlib.Path('myfile').read_text()) # Boom
print('inside end')
print('outside')
Note that, just as with regular try/except, an exception within the context causes the rest of the context to be skipped. So, if an exception happens in the line commented with Boom, the output will be:
inside
outside