I have a block of statements in a try: except KeyError: pass block and wanted to know how I could have Python attempt to execute the code after the line that throws the exception.
For e.g
try:
print "entering try block"
print this_var_does_not_exists
print "past exception"
except:
pass
I want to try and print "past exception".
When I run the above code it only prints 'entering try block'
Python doesn't allow you to re-enter a failed block. Instead, you can make a nested try block:
try:
print "before"
try:
print d['not-exist']
except KeyError:
pass
print "after"
except OtherError:
print "OtherError"
Note that you can often avoid KeyErrors using .get:
try:
x = d['key']
except KeyError:
x = 0
is equivalent to
x = d.get('key', 0)
In general, try to make your try blocks as short as logically possible, so you have a better chance of dealing with the errors in an appropriate, localized fashion.
You can also use 'else' to a try/except block:
d={'a':1, 'b':2, 'd':4}
for k in 'abcd':
try:
print k, d[k],
except KeyError:
print '{} not there'.format(k)
else:
print 'I did it! Found {}'.format(k)
Prints:
a 1 I did it! Found a
b 2 I did it! Found b
c c not there
d 4 I did it! Found d
In general, the full try/except/else/final set go like this:
try:
potential_error()
except ExceptionType:
handle_that_error_somehow()
else: # 'else' to the except is SUCCESS
# There was no error
# handle success!
handle_success()
finally:
# success or failure -- do this
regardless_something_we_always_need_to_do()
python supports finally blocks, which will be executed even if there is an exception in the try block:
try:
print "entering try block"
print this_var_does_not_exists
except:
pass
finally:
print "past exception"
You can't do it with that structure. You'd need to move the extra print out of the try block. The way a try/except block works is that if an exception is raised it jumps to the appropriate except block. There's no way to go back. If you want to continue, you need to put your code either in the except or a finally, or after the whole block. Here's one example:
try:
print "entering try block"
print this_var_does_not_exists
finally:
print "past exception"
Also, don't form the dangerous habit of using bare except: clauses, especially with pass as their only content. You should catch the kinds of exceptions you can handle, and then handle them, not blindly silence all exceptions.
print with literal string will not raise exception. Put try-except only in the second print statement.
print "entering try block"
try:
print this_var_does_not_exists
except:
pass
print "past exception"
Simply put it after the try-except block
I can imagine a situation, when indeed it may be useful to execute many such one-line try-excepts without a need to add try-except block each time.
Let's say that you have a dictionary d and an object o. Object has three attributes: 'a', 'b' and 'c'. Dictionary d is generated using some function generate_dict() and may have the following keys: 'x', 'y' and 'z', but you cannot be sure which of them are present in a given dictionary. You want to assign value of key 'x' to attribute 'a', 'y' to attribute 'b' etc. In such case, you have to surround each assignment with try-catch, like this:
o = Object()
d = generate_dict()
try:
o.a = d['x']
except:
pass
try:
o.b = d['y']
except:
pass
etc. You may also replace try-excepts with checking if a given key exists in the dictionary, but the problem is still there. What happens, if you have dozens of key-attribute mappings? The number of lines grows quickly.
Another way to code this is to generate two tables of attribute and key names and execute the code in a loop using exec function, like this:
o = Object()
d = generate_dict()
attributeNames = ['a', 'b', 'c']
dataDictKeys = ['x', 'y', 'z']
for (attr, key) in zip(attributeNames, dataDictKeys):
try:
exec("o.{attr} = d['{key}']".format(attr = attr, key = key))
except:
pass
While this is not a good coding practice to do such things, still this somehow may solve your problem. But use it with caution.
Related
I often have functions that return multiple outputs which are structured like so:
def f(vars):
...
if something_unexpected():
return None, None
...
# normal return
return value1, value2
In this case, there might be a infrequent problem that something_unexpected detects (say, a empty dataframe when the routine expects at least one row of data), and so I want to return a value to the caller that says to ignore the output and skip over it. If this were a single return function then returning None once would seem fine, but when I'm returning multiple values it seems sloppy to return multiple copies of None just so the caller has the right number of arguments to unpack.
What are some better ways of coding up this construct? Is simply having the caller use a try-except block and the function raising an exception the way to go, or is there another example of good practice to use here?
Edit: Of course I could return the pair of outputs into a single variable, but then I'd have to call the function like
results = f(inputs)
if results is None:
continue
varname1, varname2 = results[0], results[1]
rather than the more clean-seeming
varname1, varname2 = f(inputs)
if varname1 is None:
continue
Depends on where you want to handle this behavior, but exceptions are a pretty standard way to do this. Without exceptions, you could still return None, None:
a, b = f(inputs)
if None in (a, b):
print("Got something bad!")
continue
Though, I think it might be better to raise in your function and catch it instead:
def f():
if unexpected:
raise ValueError("Got empty values")
else:
return val1, val2
try:
a, b = f()
except ValueError:
print("bad behavior in f, skipping")
continue
The best practice is to raise an exception:
if something_unexpected():
raise ValueError("Something unexpected happened")
REFERENCES:
Explicit is better than implicit.
Errors should never pass silently.
Unless explicitly silenced.
PEP 20 -- The Zen of Python
a = []
b = [1,2,'x']
try:
for i in b:
a.append(i%4)
except:
print('Not possible')
finally:
print("It's over")
print(a)
Result:
Not possible
It's over
[1, 2]
I always thought try in python is similar to transactions; commit() and rollback() in SQL. So in a way, the operation would not return partial results, as it does in my case. This is a dummy case, for example, but does python offer solutions in a way it doesn't commit the change to a list if the error was imposed in the process? So it would return a blank list in this example.
Please note, I am aware of how to fix this problem, I am curious about solving the issue with a try & except.
Wrap your operation in a list comprehension
a = []
b = [1, 2, 'x']
try:
a += [i % 4 for i in b]
except Exception:
print("failed")
This way nothing will be appended to a since the list comprehension failed to instantiate.
Try is a catcher, patiently waiting for an exception to be thrown so that it can catch it and send the code down the Except block. The only way an exception will be thrown is if it executes code, so no, it won't automatically rollback. Note that:
The code inside the Try has no idea it exists within a Try-Except block.
Because of 1., if there was a rollback, Python would really be taking all of your content inside of the Try and storing changes or copies of it. You will certainly have nested Trys. If you import a package, some of the functions you use in it will have Trys inside. This means the stack would be keeping track of multiple copies of everything you do in the block. Note that in most (many?) DBMS's, you cannot start a second transaction without ending your first. This is not the case in python.
If you want to rollback the changes, you should do that in the first few lines of your Except statement.
In Python, anything occurring in a try will execute (as if it weren't in a try/except block at all) all the way until an Exception is encountered. At that point, it then proceeds to the except block(s). finally is executed whether except was entered or not (aka, it will always fire). try/except/finally is not meant to be atomic like SQL.
You can define a new metaclass such that every class using this metaclass implement the try/except the way you mean.
For instance, you could define:
from copy import deepcopy
import types
class TryExceptRollbackMeta(type):
def __new__(cls, name, bases, attrs):
new_attrs = {}
for name, value in attrs.items():
if name == "__init__" or not isinstance(value, types.FunctionType):
new_attrs[name] = value
continue
# We know from now on that we're dealing with a non-static function
# If for some reason, a non-static method is defined without being passed self as an argument
if value.__code__.co_argcount == 0:
new_attrs[name] = value
continue
new_attrs[f"updated_{name}"] = TryExceptRollbackMeta.generate_updated_method(value)
return super().__new__(cls, name, bases, new_attrs)
#staticmethod
def generate_updated_method(func):
def updated_method(*args, **kwargs):
original = deepcopy(args[0])
try:
result = func(*args, **kwargs)
except Exception as e:
print(f"Exception {type(e)} has occured: {e}. Reverting state...")
args[0].__dict__.update(original.__dict__)
return None
return result
return updated_method
class Test(metaclass=TryExceptRollbackMeta):
def __init__(self):
self.a = []
def correct(self):
for i in [1, 2, 3]:
self.a.append(i % 4)
def incorrect(self):
for i in [1, 2, 'x']:
self.a.append(i % 4)
Then, this would work like this:
>>> test = Test()
>>> test.updated_correct()
>>> print(test.a)
[1, 2, 3]
>>> test.updated_incorrect()
Exception <class 'TypeError'> has occured: not all arguments converted during string formatting. Reverting state...
>>> print(test.a)
[1, 2, 3]
By doing so, you have full control on the way you want to deal with an Exception, you can act according to the Exception type, print the line at which it failed, etc...
The problem is that the deepcopy can potentially be very long depending on what your object attributes are. It is still possible to target the attributes you want to revert specifically though, this is just the general case where you don't know which attribute can be affected by a method.
The problem
I have the following list in Python 3.6
Piko = {}
Piko['Name']='Luke'
I am trying to write a function that give the value of the element if it exist and is set and give None otherwise.
For example:
INPUT: isset(Piko['Name']) OUTPUT: Luke
INPUT: isset(Piko['Surname']) OUTPUT: None
What I have tried
1st try; based on my know how:
def isset1(x):
try:
x
except KeyError:
print(None)
else:
print(x)
2nd try; based on this answer:
def isset2(x):
try:
t=x
except IndexError:
print(None)
3rd try; based on this answer:
def isset3(x):
try:
x
except Exception:
print(None)
else:
print(x)
Any one of the previous gives me KeyError: 'Surname' error and does not output None as I wanted. Can anybody help me explaining how could I manage correctly the KeyError?
Piko.get('Surname')
Piko.get('Surname', None)
are identical and return None since "Surname" is not in your dictionary.
For future reference you can quickly discover this from the Python shell (eg ipython) by typing:
In[4]: help(Piku.get)
Which produces:
Help on built-in function get:
get(...)
D.get(k[,d]) -> D[k] if k in D, else d. d defaults to None.
The exception is happening before it even gets into your isset function. When you do this:
isset(Piko['Name'])
… it's basically the same as doing this:
_tmp = Piko['Name']
isset(_tmp)
No matter what code you put inside isset, it's not going to help, because that function never gets called. The only place you can put the exception handling is one level up, in the function that calls isset.
Or, alternatively, you can not try to lookup dict[key] to pass into isset, and pass the dict and the key as separate parameters:
def isset(d, key):
try:
print(d[key])
except KeyError:
print(None)
But at this point, you're just duplicating dict.get in a clumsier way. You can do this:
def isset(d, key):
print(d.get(key, None))
… or just scrap isset and do this:
print(Piko.get('Name', None))
Or, since None is the default if you don't specify anything:
print(Piko.get('Name'))
I have a Python function called plot_pdf(f) that might throw an error. I use a list comprehension to iterate over a list of files on this function:
[plot_pdf(f) for f in file_list]
I want to use try-except block to skip any possible errors during the iteration loop and continue with the next file. So is the following code correct way to do the exception handling in Python list comprehension?
try:
[plot_pdf(f) for f in file_list] # using list comprehensions
except:
print ("Exception: ", sys.exc_info()[0])
continue
Will the above code terminate the current iteration and go to the next iteration? If I can't use list comprehension to catch errors during iteration, then I have to use the normal for loop:
for f in file_list:
try:
plot_pdf(f)
except:
print("Exception: ", sys.exc_info()[0])
continue
I want to know if I can use try-except to do exception handling in list comprehension.
try:
[plot_pdf(f) for f in file_list] # using list comprehensions
except:
print ("Exception: ", sys.exc_info()[0])
continue
If plot_pdf(f) throws an error during execution of comprehension, then, it is caught in the except clause, other items in comprehension won't be evaluated.
It is not possible to handle exceptions in a list comprehension, for a list comprehension is an expression containing other expression, nothing more (i.e. no statements, and only statements can catch/ignore/handle exceptions).
Function calls are expression, and the function bodies can include all
the statements you want, so delegating the evaluation of the
exception-prone sub-expression to a function, as you've noticed, is
one feasible workaround (others, when feasible, are checks on values
that might provoke exceptions, as also suggested in other answers).
More here.
You're stuck with your for loop unless you handle the error inside plot_pdf or a wrapper.
def catch_plot_pdf(f):
try:
return plot_pdf(f)
except:
print("Exception: ", sys.exc_info()[0])
[catch_plot_pdf(f) for f in file_list]
You could create a catch object
def catch(error, default, function, *args, **kwargs):
try: return function(*args, **kwargs)
except error: return default
Then you can do
# using None as default value
result (catch(Exception, None, plot_pdf, f) for f in file_list)
And then you can do what you want with the result:
result = list(result) # turn it into a list
# or
result = [n for n in result if n is not None] # filter out the Nones
Unfortunately this will not be even remotely C speed, see my question here
This is a follow-up to Handle an exception thrown in a generator and discusses a more general problem.
I have a function that reads data in different formats. All formats are line- or record-oriented and for each format there's a dedicated parsing function, implemented as a generator. So the main reading function gets an input and a generator, which reads its respective format from the input and delivers records back to the main function:
def read(stream, parsefunc):
for record in parsefunc(stream):
do_stuff(record)
where parsefunc is something like:
def parsefunc(stream):
while not eof(stream):
rec = read_record(stream)
do some stuff
yield rec
The problem I'm facing is that while parsefunc can throw an exception (e.g. when reading from a stream), it has no idea how to handle it. The function responsible for handling exceptions is the main read function. Note that exceptions occur on a per-record basis, so even if one record fails, the generator should continue its work and yield records back until the whole stream is exhausted.
In the previous question I tried to put next(parsefunc) in a try block, but as turned out, this is not going to work. So I have to add try-except to the parsefunc itself and then somehow deliver exceptions to the consumer:
def parsefunc(stream):
while not eof(stream):
try:
rec = read_record()
yield rec
except Exception as e:
?????
I'm rather reluctant to do this because
it makes no sense to use try in a function that isn't intended to handle any exceptions
it's unclear to me how to pass exceptions to the consuming function
there going to be many formats and many parsefunc's, I don't want to clutter them with too much helper code.
Has anyone suggestions for a better architecture?
A note for googlers: in addition to the top answer, pay attention to senderle's and Jon's posts - very smart and insightful stuff.
You can return a tuple of record and exception in the parsefunc and let the consumer function decide what to do with the exception:
import random
def get_record(line):
num = random.randint(0, 3)
if num == 3:
raise Exception("3 means danger")
return line
def parsefunc(stream):
for line in stream:
try:
rec = get_record(line)
except Exception as e:
yield (None, e)
else:
yield (rec, None)
if __name__ == '__main__':
with open('temp.txt') as f:
for rec, e in parsefunc(f):
if e:
print "Got an exception %s" % e
else:
print "Got a record %s" % rec
Thinking deeper about what would happen in a more complex case kind of vindicates the Python choice of avoiding bubbling exceptions out of a generator.
If I got an I/O error from a stream object the odds of simply being able to recover and continue reading, without the structures local to the generator being reset in some way, would be low. I would somehow have to reconcile myself with the reading process in order to continue: skip garbage, push back partial data, reset some incomplete internal tracking structure, etc.
Only the generator has enough context to do that properly. Even if you could keep the generator context, having the outer block handle the exceptions would totally flout the Law of Demeter. All the important information that the surrounding block needs to reset and move on is in local variables of the generator function! And getting or passing that information, though possible, is disgusting.
The resulting exception would almost always be thrown after cleaning up, in which case the reader-generator will already have an internal exception block. Trying very hard to maintain this cleanliness in the brain-dead-simple case only to have it break down in almost every realistic context would be silly. So just have the try in the generator, you are going to need the body of the except block anyway, in any complex case.
It would be nice if exceptional conditions could look like exceptions, though, and not like return values. So I would add an intermediate adapter to allow for this: The generator would yield either data or exceptions and the adapter would re-raise the exception if applicable. The adapter should be called first-thing inside the for loop, so that we have the option of catching it within the loop and cleaning up to continue, or breaking out of the loop to catch it and and abandon the process. And we should put some kind of lame wrapper around the setup to indicate that tricks are afoot, and to force the adapter to get called if the function is adapting.
That way each layer is presented errors that it has the context to handle, at the expense of the adapter being a tiny bit intrusive (and perhaps also easy to forget).
So we would have:
def read(stream, parsefunc):
try:
for source in frozen(parsefunc(stream)):
try:
record = source.thaw()
do_stuff(record)
except Exception, e:
log_error(e)
if not is_recoverable(e):
raise
recover()
except Exception, e:
properly_give_up()
wrap_up()
(Where the two try blocks are optional.)
The adapter looks like:
class Frozen(object):
def __init__(self, item):
self.value = item
def thaw(self):
if isinstance(value, Exception):
raise value
return value
def frozen(generator):
for item in generator:
yield Frozen(item)
And parsefunc looks like:
def parsefunc(stream):
while not eof(stream):
try:
rec = read_record(stream)
do_some_stuff()
yield rec
except Exception, e:
properly_skip_record_or_prepare_retry()
yield e
To make it harder to forget the adapter, we could also change frozen from a function to a decorator on parsefunc.
def frozen_results(func):
def freezer(__func = func, *args, **kw):
for item in __func(*args, **kw):
yield Frozen(item)
return freezer
In which case we we would declare:
#frozen_results
def parsefunc(stream):
...
And we would obviously not bother to declare frozen, or wrap it around the call to parsefunc.
Without knowing more about the system, I think it's difficult to tell what approach will work best. However, one option that no one has suggested yet would be to use a callback. Given that only read knows how to deal with exceptions, might something like this work?
def read(stream, parsefunc):
some_closure_data = {}
def error_callback_1(e):
manipulate(some_closure_data, e)
def error_callback_2(e):
transform(some_closure_data, e)
for record in parsefunc(stream, error_callback_1):
do_stuff(record)
Then, in parsefunc:
def parsefunc(stream, error_callback):
while not eof(stream):
try:
rec = read_record()
yield rec
except Exception as e:
error_callback(e)
I used a closure over a mutable local here; you could also define a class. Note also that you can access the traceback info via sys.exc_info() inside the callback.
Another interesting approach might be to use send. This would work a little differently; basically, instead of defining a callback, read could check the result of yield, do a lot of complex logic, and send a substitute value, which the generator would then re-yield (or do something else with). This is a bit more exotic, but I thought I'd mention it in case it's useful:
>>> def parsefunc(it):
... default = None
... for x in it:
... try:
... rec = float(x)
... except ValueError as e:
... default = yield e
... yield default
... else:
... yield rec
...
>>> parsed_values = parsefunc(['4', '6', '5', '5h', '22', '7'])
>>> for x in parsed_values:
... if isinstance(x, ValueError):
... x = parsed_values.send(0.0)
... print x
...
4.0
6.0
5.0
0.0
22.0
7.0
On it's own this is a bit useless ("Why not just print the default directly from read?" you might ask), but you could do more complex things with default inside the generator, resetting values, going back a step, and so on. You could even wait to send a callback at this point based on the error you receive. But note that sys.exc_info() is cleared as soon as the generator yields, so you'll have to send everything from sys.exc_info() if you need access to the traceback.
Here's an example of how you might combine the two options:
import string
digits = set(string.digits)
def digits_only(v):
return ''.join(c for c in v if c in digits)
def parsefunc(it):
default = None
for x in it:
try:
rec = float(x)
except ValueError as e:
callback = yield e
yield float(callback(x))
else:
yield rec
parsed_values = parsefunc(['4', '6', '5', '5h', '22', '7'])
for x in parsed_values:
if isinstance(x, ValueError):
x = parsed_values.send(digits_only)
print x
An example of a possible design:
from StringIO import StringIO
import csv
blah = StringIO('this,is,1\nthis,is\n')
def parse_csv(stream):
for row in csv.reader(stream):
try:
yield int(row[2])
except (IndexError, ValueError) as e:
pass # don't yield but might need something
# All others have to go up a level - so it wasn't parsable
# So if it's an IOError you know why, but this needs to catch
# exceptions potentially, just let the major ones propogate
for record in parse_csv(blah):
print record
I like the given answer with the Frozen stuff. Based on that idea I came up with this, solving two aspects I did not yet like. The first was the patterns needed to write it down. The second was the loss of the stack trace when yielding an exception. I tried my best to solve the first by using decorators as good as possible. I tried keeping the stack trace by using sys.exc_info() instead of the exception alone.
My generator normally (i.e. without my stuff applied) would look like this:
def generator():
def f(i):
return float(i) / (3 - i)
for i in range(5):
yield f(i)
If I can transform it into using an inner function to determine the value to yield, I can apply my method:
def generator():
def f(i):
return float(i) / (3 - i)
for i in range(5):
def generate():
return f(i)
yield generate()
This doesn't yet change anything and calling it like this would raise an error with a proper stack trace:
for e in generator():
print e
Now, applying my decorators, the code would look like this:
#excepterGenerator
def generator():
def f(i):
return float(i) / (3 - i)
for i in range(5):
#excepterBlock
def generate():
return f(i)
yield generate()
Not much change optically. And you still can use it the way you used the version before:
for e in generator():
print e
And you still get a proper stack trace when calling. (Just one more frame is in there now.)
But now you also can use it like this:
it = generator()
while it:
try:
for e in it:
print e
except Exception as problem:
print 'exc', problem
This way you can handle in the consumer any exception raised in the generator without too much syntactic hassle and without losing stack traces.
The decorators are spelled out like this:
import sys
def excepterBlock(code):
def wrapper(*args, **kwargs):
try:
return (code(*args, **kwargs), None)
except Exception:
return (None, sys.exc_info())
return wrapper
class Excepter(object):
def __init__(self, generator):
self.generator = generator
self.running = True
def next(self):
try:
v, e = self.generator.next()
except StopIteration:
self.running = False
raise
if e:
raise e[0], e[1], e[2]
else:
return v
def __iter__(self):
return self
def __nonzero__(self):
return self.running
def excepterGenerator(generator):
return lambda *args, **kwargs: Excepter(generator(*args, **kwargs))
(I answered the other question linked in the OP but my answer applies to this situation as well)
I have needed to solve this problem a couple of times and came upon this question after a search for what other people have done.
One option- which will probably require refactoring things a little bit- would be to simply create an error handling generator, and throw the exception in the generator (to another error handling generator) rather than raise it.
Here is what the error handling generator function might look like:
def err_handler():
# a generator for processing errors
while True:
try:
# errors are thrown to this point in function
yield
except Exception1:
handle_exc1()
except Exception2:
handle_exc2()
except Exception3:
handle_exc3()
except Exception:
raise
An additional handler argument is provided to the parsefunc function so it has a place to put the errors:
def parsefunc(stream, handler):
# the handler argument fixes errors/problems separately
while not eof(stream):
try:
rec = read_record(stream)
do some stuff
yield rec
except Exception as e:
handler.throw(e)
handler.close()
Now just use almost the original read function, but now with an error handler:
def read(stream, parsefunc):
handler = err_handler()
for record in parsefunc(stream, handler):
do_stuff(record)
This isn't always going to be the best solution, but it's certainly an option, and relatively easy to understand.
About your point of propagating exception from generator to consuming function,
you can try to use an error code (set of error codes) to indicate the error.
Though not elegant that is one approach you can think of.
For example in the below code yielding a value like -1 where you were expecting
a set of positive integers would signal to the calling function that there was
an error.
In [1]: def f():
...: yield 1
...: try:
...: 2/0
...: except ZeroDivisionError,e:
...: yield -1
...: yield 3
...:
In [2]: g = f()
In [3]: next(g)
Out[3]: 1
In [4]: next(g)
Out[4]: -1
In [5]: next(g)
Out[5]: 3
Actually, generators are quite limited in several aspects. You found one: the raising of exceptions is not part of their API.
You could have a look at the Stackless Python stuff like greenlets or coroutines which offer a lot more flexibility; but diving into that is a bit out of scope here.