We Keep Coding

A DRY approach to Python try-except blocks?

Objective: I have several lines of code each capable of producing the same type of error, and warranting the same kind of response. How do I prevent a 'do not repeat yourself' problem with the try-except blocks.
Background:
I using ReGex to scrape poorly formatted data from a text file, and input it into the field of a custom object. The code works great except when the field has been left blank in which case it throws an error.
I handle this error in a try-except block. If error, insert a blank into the field of the object (i.e. '').
The problem is it turns easily readable, nice, Python code into a mess of try-except blocks that each do exact same thing. This is a 'do not repeat yourself' (a.k.a. DRY) violation.
The Code:
Before:
sample.thickness = find_field('Thickness', sample_datum)[0]
sample.max_tension = find_field('Maximum Load', sample_datum)[0]
sample.max_length = find_field('Maximum Extension', sample_datum)[0]
sample.test_type = sample_test
After:
try:
sample.thickness = find_field('Thickness', sample_datum)[0]
except:
sample.thickness = ''
try:
sample.max_tension = find_field('Maximum Load', sample_datum)[0]
except:
sample.max_tension = ''
try:
sample.max_length = find_field('Maximum Extension', sample_datum)[0]
except:
sample.max_length = ''
try:
sample.test_type = sample_test
except:
sample.test_type = ''
What I Need:
Is there some Pythonic way to write this? Some block where I can say if there is an index-out-of-range error on any of these lines (indicating the field was blank, and ReGex failed to return anything) insert a blank in the sample field.

What about extracting a function out of it?
def maybe_find_field(name, datum):
try:
return find_field(name, datum)[0]
except IndexError: # Example of specific exception to catch
return ''
sample.thickness = maybe_find_field('Thickness', sample_datum)
sample.max_tension = maybe_find_field('Maximum Load', sample_datum)
sample.max_length = maybe_find_field('Maximum Extension', sample_datum)
sample.test_type = sample_test
BTW, don't simply catch all possible exceptions with except: unless that's really what you want to do. Catching everything may hide implementation errors and become quite difficult to debug later. Whenever possible, bind your except case to the specific exception that you need.

Not quite matching the question, but Google sent me here and so others might come.
I have post functions in two separate Django-views, which call similar backend-functions and require the same exception-handling.
I solved the issue by extracting the whole except:-tree and sticking it into a decorator.
Before:
# twice this
def post(request):
try:
return backend_function(request.post)
except ProblemA as e:
return Response("Problem A has occurred, try this to fix it.", status=400)
except ProblemB as e:
return Response("Problem B has occurred, try this to fix it.", status=400)
except ProblemC as e:
return Response("Problem C has occurred, try this to fix it.", status=400)
except ProblemD as e:
return Response("Problem D has occurred, try this to fix it.", status=400)
After:
# once this
def catch_all_those_pesky_exceptions(original_function):
def decorated(*args, **kwargs):
try:
return original_function(*args, **kwargs)
except ProblemA as e:
return Response("Problem A has occurred, try this to fix it.", status=400)
except ProblemB as e:
return Response("Problem B has occurred, try this to fix it.", status=400)
except ProblemC as e:
return Response("Problem C has occurred, try this to fix it.", status=400)
except ProblemD as e:
return Response("Problem D has occurred, try this to fix it.", status=400)
return decorated
# twice this - note the #decorator matching the above function.
#catch_all_those_pesky_exceptions
def post(request):
return backend_function(request.post)
Now I can add more exception-handling in a single place.

When you find yourself repeating code, encapsulate it in a function. In this case, create a function that handles the exception for you.
def try_find_field(field_name, datum, default_value):
try:
return find_field(field_name, datum)[0]
except:
return default_value

What about something like this:
def exception_handler(exception_class):
logger = logging.getLogger('app_error')
logger.error(exception_class)
exception_name = exception_class.__name__
if exception_name == 'AuthenticationError':
raise AuthenticationError
elif exception_name == 'AuthorizationError':
raise AuthorizationError
elif exception_name == 'ConnectionError':
raise ConnectionError
else:
raise GenericError
def call_external_api()
try:
result = http_request()
except Exception as e:
exception_handler(exception_class=type(e))

You can have any number of except blocks over and over, handling different kinds of exceptions. There's also nothing wrong with having multiple statements in the same try/catch block.
try:
doMyDangerousThing()
except ValueError:
print "ValueError!"
except HurrDurrError:
print "hurr durr, there's an error"
try:
doMyDangerousThing()
doMySecondDangerousThing()
except:
print "Something went wrong!"

More pythonic way to handle nested try... except blocks?

Is there a cleaner or more pythonic way to do the following?
try:
error_prone_function(arg1)
except MyError:
try:
error_prone_function(arg2)
except MyError:
try:
another_error_prone_function(arg3)
except MyError:
try:
last_error_prone_function(arg4)
except MyError:
raise MyError("All backup parameters failed.")
Basically it's: If attempt #1 fails, try #2. If #2 fails, try #3. If #3 fails, try #4. If #4 fails, ... if #n fails, then finally raise some exception.
Note that I'm not necessarily calling the same function every time, nor am I using the same function arguments every time. I am, however, expecting the same exception MyError on each function.

Thanks to John Kugelman's post here, I decided to go with this which utilizes the lesser-known else clause of a for loop to execute code when an entire list has been exhausted without a break happening.
funcs_and_args = [(func1, "150mm"),
(func1, "100mm",
(func2, "50mm"),
(func3, "50mm"),
]
for func, arg in funcs_and_args :
try:
func(arg)
# exit the loop on success
break
except MyError:
# repeat the loop on failure
continue
else:
# List exhausted without break, so there must have always been an Error
raise MyError("Error text")
As Daniel Roseman commented below, be careful with indentation since the try statement also has an else clause.

A generator based approach might give you a little more flexibility than the data-driven approach:
def attempts_generator():
# try:
# <the code you're attempting to run>
# except Exception as e:
# # failure
# yield e.message
# else:
# # success
# return
try:
print 'Attempt 1'
raise Exception('Failed attempt 1')
except Exception as e:
yield e.message
else:
return
try:
print 'Attempt 2'
# raise Exception('Failed attempt 2')
except Exception as e:
yield e.message
else:
return
try:
print 'Attempt 3'
raise Exception('Failed attempt 3')
except Exception as e:
yield e.message
else:
return
try:
print 'Attempt 4'
raise Exception('Failed attempt 4')
except Exception as e:
yield e.message
else:
return
raise Exception('All attempts failed!')
attempts = attempts_generator()
for attempt in attempts:
print attempt + ', retrying...'
print 'All good!'
The idea is to build a generator that steps through attempt blocks via a retry loop.
Once the generator hits a successful attempt it stops its own iteration with a hard return. Unsuccessful attempts yield to the retry loop for the next fallback. Otherwise if it runs out of attempts it eventually throws an error that it couldn't recover.
The advantage here is that the contents of the try..excepts can be whatever you want, not just individual function calls if that's especially awkward for whatever reason. The generator function can also be defined within a closure.
As I did here, the yield can pass back information for logging as well.
Output of above, btw, noting that I let attempt 2 succeed as written:
mbp:scratch geo$ python ./fallback.py
Attempt 1
Failed attempt 1, retrying...
Attempt 2
All good!
If you uncomment the raise in attempt 2 so they all fail you get:
mbp:scratch geo$ python ./fallback.py
Attempt 1
Failed attempt 1, retrying...
Attempt 2
Failed attempt 2, retrying...
Attempt 3
Failed attempt 3, retrying...
Attempt 4
Failed attempt 4, retrying...
Traceback (most recent call last):
File "./fallback.py", line 47, in <module>
for attempt in attempts:
File "./fallback.py", line 44, in attempts_generator
raise Exception('All attempts failed!')
Exception: All attempts failed!
Edit:
In terms of your pseudocode, this looks like:
def attempts_generator():
try:
error_prone_function(arg1)
except MyError
yield
else:
return
try:
error_prone_function(arg2)
except MyError
yield
else:
return
try:
another_error_prone_function(arg3)
except MyError:
yield
else:
return
try:
last_error_prone_function(arg4)
except MyError:
yield
else:
return
raise MyError("All backup parameters failed.")
attempts = attempts_generator()
for attempt in attempts:
pass
It'll let any exception but MyError bubble out and stop the whole thing. You also could choose to catch different errors for each block.

Is "return False" needed after "traceback.print_exc()"?

My code about this:
try:
self.cookie = Cookie.SimpleCookie(os.environ["HTTP_COOKIE"])
tmpuid = self.cookie["uid"].value
tmpsid = self.cookie["sid"].value
except Exception as e:
if not str(e).startswith('No cookie set'):
import traceback
traceback.print_exc()
return False
Is "return False" needed after "traceback.print_exc()"?

Sure, The exception only stops on No cookie set exceptions. If it is any other exception the program will return False instead of continued to the next statement

Is it reasonable in Python to check for a specific type of exception using isinstance?

Is it reasonable in Python to catch a generic exception, then use isinstance() to detect the specific type of exception in order to handle it appropriately?
I'm playing around with the dnspython toolkit at the moment, which has a range of exceptions for things like a timeout, an NXDOMAIN response, etc. These exceptions are subclasses of dns.exception.DNSException, so I am wondering if it's reasonable, or pythonic, to catch DNSException then check for a specific exception with isinstance().
e.g.
try:
answers = dns.resolver.query(args.host)
except dns.exception.DNSException as e:
if isinstance(e, dns.resolver.NXDOMAIN):
print "No such domain %s" % args.host
elif isinstance(e, dns.resolver.Timeout):
print "Timed out while resolving %s" % args.host
else:
print "Unhandled exception"
I'm new to Python so be gentle!

That's what multiple except clauses are for:
try:
answers = dns.resolver.query(args.host)
except dns.resolver.NXDOMAIN:
print "No such domain %s" % args.host
except dns.resolver.Timeout:
print "Timed out while resolving %s" % args.host
except dns.exception.DNSException:
print "Unhandled exception"
Be careful about the order of the clauses: The first matching clause will be taken, so move the check for the superclass to the end.

From dns.resolver you can import some exceptions.
(untested code)
from dns.resolver import Resolver, NXDOMAIN, NoNameservers, Timeout, NoAnswer
try
host_record = self.resolver.query(self.host, "A")
if len(host_record) > 0:
Mylist['ERROR'] = False
# Do something
except NXDOMAIN:
Mylist['ERROR'] = True
Mylist['ERRORTYPE'] = NXDOMAIN
except NoNameservers:
Mylist['ERROR'] = True
Mylist['ERRORTYPE'] = NoNameservers
except Timeout:
Mylist['ERROR'] = True
Mylist['ERRORTYPE'] = Timeout
except NameError:
Mylist['ERROR'] = True
Mylist['ERRORTYPE'] = NameError

Nested exceptions?

Will this work?
try:
try:
field.value = filter(field.value, fields=self.fields, form=self, field=field)
except TypeError:
field.value = filter(field.value)
except ValidationError, e:
field.errors += e.args
field.value = revert
valid = False
break
Namely, if that first line throws a ValidationError, will the second except catch it?
I would have written it un-nested, but the second filter statement can fail too! And I want to use the same ValidationError block to catch that as well.
I'd test it myself, but this code is so interwoven now it's difficult to trip it properly :)
As a side note, is it bad to rely on it catching the TypeError and passing in only one arg instead? i.e., deliberately omitting some arguments where they aren't needed?

If the filter statement in the inner try raises an exception, it will first get checked against the inner set of "except" statements and then if none of those catch it, it will be checked against the outer set of "except" statements.
You can convince yourself this is the case just by doing something simple like this (this will only print "Caught the value error"):
try:
try:
raise ValueError('1')
except TypeError:
print 'Caught the type error'
except ValueError:
print 'Caught the value error!'
As another example, this one should print "Caught the inner ValueError" only:
try:
try:
raise ValueError('1')
except TypeError:
pass
except ValueError:
print 'Caught the inner ValueError!'
except ValueError:
print 'Caught the outer value error!'

To compliment Brent's answer, and test the other case:
class ValidationError(Exception): pass
def f(a): raise ValidationError()
try:
try:
f()
except TypeError:
f(1)
except ValidationError:
print 'caught1'
try:
try:
f(1)
except TypeError:
print 'oops'
except ValidationError:
print 'caught2'
Which prints:
caught1
caught2