Related
From time to time in Python, I see the block:
try:
try_this(whatever)
except SomeException as exception:
#Handle exception
else:
return something
What is the reason for the try-except-else to exist?
I do not like that kind of programming, as it is using exceptions to perform flow control. However, if it is included in the language, there must be a good reason for it, isn't it?
It is my understanding that exceptions are not errors, and that they should only be used for exceptional conditions (e.g. I try to write a file into disk and there is no more space, or maybe I do not have permission), and not for flow control.
Normally I handle exceptions as:
something = some_default_value
try:
something = try_this(whatever)
except SomeException as exception:
#Handle exception
finally:
return something
Or if I really do not want to return anything if an exception happens, then:
try:
something = try_this(whatever)
return something
except SomeException as exception:
#Handle exception
"I do not know if it is out of ignorance, but I do not like that
kind of programming, as it is using exceptions to perform flow control."
In the Python world, using exceptions for flow control is common and normal.
Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination).
In addition, the try-except-style is used to prevent the race-conditions inherent in some of the "look-before-you-leap" constructs. For example, testing os.path.exists results in information that may be out-of-date by the time you use it. Likewise, Queue.full returns information that may be stale. The try-except-else style will produce more reliable code in these cases.
"It my understanding that exceptions are not errors, they should only
be used for exceptional conditions"
In some other languages, that rule reflects their cultural norms as reflected in their libraries. The "rule" is also based in-part on performance considerations for those languages.
The Python cultural norm is somewhat different. In many cases, you must use exceptions for control-flow. Also, the use of exceptions in Python does not slow the surrounding code and calling code as it does in some compiled languages (i.e. CPython already implements code for exception checking at every step, regardless of whether you actually use exceptions or not).
In other words, your understanding that "exceptions are for the exceptional" is a rule that makes sense in some other languages, but not for Python.
"However, if it is included in the language itself, there must be a
good reason for it, isn't it?"
Besides helping to avoid race-conditions, exceptions are also very useful for pulling error-handling outside loops. This is a necessary optimization in interpreted languages which do not tend to have automatic loop invariant code motion.
Also, exceptions can simplify code quite a bit in common situations where the ability to handle an issue is far removed from where the issue arose. For example, it is common to have top level user-interface code calling code for business logic which in turn calls low-level routines. Situations arising in the low-level routines (such as duplicate records for unique keys in database accesses) can only be handled in top-level code (such as asking the user for a new key that doesn't conflict with existing keys). The use of exceptions for this kind of control-flow allows the mid-level routines to completely ignore the issue and be nicely decoupled from that aspect of flow-control.
There is a nice blog post on the indispensibility of exceptions here.
Also, see this Stack Overflow answer: Are exceptions really for exceptional errors?
"What is the reason for the try-except-else to exist?"
The else-clause itself is interesting. It runs when there is no exception but before the finally-clause. That is its primary purpose.
Without the else-clause, the only option to run additional code before finalization would be the clumsy practice of adding the code to the try-clause. That is clumsy because it risks
raising exceptions in code that wasn't intended to be protected by the try-block.
The use-case of running additional unprotected code prior to finalization doesn't arise very often. So, don't expect to see many examples in published code. It is somewhat rare.
Another use-case for the else-clause is to perform actions that must occur when no exception occurs and that do not occur when exceptions are handled. For example:
recip = float('Inf')
try:
recip = 1 / f(x)
except ZeroDivisionError:
logging.info('Infinite result')
else:
logging.info('Finite result')
Another example occurs in unittest runners:
try:
tests_run += 1
run_testcase(case)
except Exception:
tests_failed += 1
logging.exception('Failing test case: %r', case)
print('F', end='')
else:
logging.info('Successful test case: %r', case)
print('.', end='')
Lastly, the most common use of an else-clause in a try-block is for a bit of beautification (aligning the exceptional outcomes and non-exceptional outcomes at the same level of indentation). This use is always optional and isn't strictly necessary.
What is the reason for the try-except-else to exist?
A try block allows you to handle an expected error. The except block should only catch exceptions you are prepared to handle. If you handle an unexpected error, your code may do the wrong thing and hide bugs.
An else clause will execute if there were no errors, and by not executing that code in the try block, you avoid catching an unexpected error. Again, catching an unexpected error can hide bugs.
Example
For example:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
else:
return something
The "try, except" suite has two optional clauses, else and finally. So it's actually try-except-else-finally.
else will evaluate only if there is no exception from the try block. It allows us to simplify the more complicated code below:
no_error = None
try:
try_this(whatever)
no_error = True
except SomeException as the_exception:
handle(the_exception)
if no_error:
return something
so if we compare an else to the alternative (which might create bugs) we see that it reduces the lines of code and we can have a more readable, maintainable, and less buggy code-base.
finally
finally will execute no matter what, even if another line is being evaluated with a return statement.
Broken down with pseudo-code
It might help to break this down, in the smallest possible form that demonstrates all features, with comments. Assume this syntactically correct (but not runnable unless the names are defined) pseudo-code is in a function.
For example:
try:
try_this(whatever)
except SomeException as the_exception:
handle_SomeException(the_exception)
# Handle a instance of SomeException or a subclass of it.
except Exception as the_exception:
generic_handle(the_exception)
# Handle any other exception that inherits from Exception
# - doesn't include GeneratorExit, KeyboardInterrupt, SystemExit
# Avoid bare `except:`
else: # there was no exception whatsoever
return something()
# if no exception, the "something()" gets evaluated,
# but the return will not be executed due to the return in the
# finally block below.
finally:
# this block will execute no matter what, even if no exception,
# after "something" is eval'd but before that value is returned
# but even if there is an exception.
# a return here will hijack the return functionality. e.g.:
return True # hijacks the return in the else clause above
It is true that we could include the code in the else block in the try block instead, where it would run if there were no exceptions, but what if that code itself raises an exception of the kind we're catching? Leaving it in the try block would hide that bug.
We want to minimize lines of code in the try block to avoid catching exceptions we did not expect, under the principle that if our code fails, we want it to fail loudly. This is a best practice.
It is my understanding that exceptions are not errors
In Python, most exceptions are errors.
We can view the exception hierarchy by using pydoc. For example, in Python 2:
$ python -m pydoc exceptions
or Python 3:
$ python -m pydoc builtins
Will give us the hierarchy. We can see that most kinds of Exception are errors, although Python uses some of them for things like ending for loops (StopIteration). This is Python 3's hierarchy:
BaseException
Exception
ArithmeticError
FloatingPointError
OverflowError
ZeroDivisionError
AssertionError
AttributeError
BufferError
EOFError
ImportError
ModuleNotFoundError
LookupError
IndexError
KeyError
MemoryError
NameError
UnboundLocalError
OSError
BlockingIOError
ChildProcessError
ConnectionError
BrokenPipeError
ConnectionAbortedError
ConnectionRefusedError
ConnectionResetError
FileExistsError
FileNotFoundError
InterruptedError
IsADirectoryError
NotADirectoryError
PermissionError
ProcessLookupError
TimeoutError
ReferenceError
RuntimeError
NotImplementedError
RecursionError
StopAsyncIteration
StopIteration
SyntaxError
IndentationError
TabError
SystemError
TypeError
ValueError
UnicodeError
UnicodeDecodeError
UnicodeEncodeError
UnicodeTranslateError
Warning
BytesWarning
DeprecationWarning
FutureWarning
ImportWarning
PendingDeprecationWarning
ResourceWarning
RuntimeWarning
SyntaxWarning
UnicodeWarning
UserWarning
GeneratorExit
KeyboardInterrupt
SystemExit
A commenter asked:
Say you have a method which pings an external API and you want to handle the exception at a class outside the API wrapper, do you simply return e from the method under the except clause where e is the exception object?
No, you don't return the exception, just reraise it with a bare raise to preserve the stacktrace.
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
raise
Or, in Python 3, you can raise a new exception and preserve the backtrace with exception chaining:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
raise DifferentException from the_exception
I elaborate in my answer here.
Python doesn't subscribe to the idea that exceptions should only be used for exceptional cases, in fact the idiom is 'ask for forgiveness, not permission'. This means that using exceptions as a routine part of your flow control is perfectly acceptable, and in fact, encouraged.
This is generally a good thing, as working this way helps avoid some issues (as an obvious example, race conditions are often avoided), and it tends to make code a little more readable.
Imagine you have a situation where you take some user input which needs to be processed, but have a default which is already processed. The try: ... except: ... else: ... structure makes for very readable code:
try:
raw_value = int(input())
except ValueError:
value = some_processed_value
else: # no error occured
value = process_value(raw_value)
Compare to how it might work in other languages:
raw_value = input()
if valid_number(raw_value):
value = process_value(int(raw_value))
else:
value = some_processed_value
Note the advantages. There is no need to check the value is valid and parse it separately, they are done once. The code also follows a more logical progression, the main code path is first, followed by 'if it doesn't work, do this'.
The example is naturally a little contrived, but it shows there are cases for this structure.
See the following example which illustrate everything about try-except-else-finally:
for i in range(3):
try:
y = 1 / i
except ZeroDivisionError:
print(f"\ti = {i}")
print("\tError report: ZeroDivisionError")
else:
print(f"\ti = {i}")
print(f"\tNo error report and y equals {y}")
finally:
print("Try block is run.")
Implement it and come by:
i = 0
Error report: ZeroDivisionError
Try block is run.
i = 1
No error report and y equals 1.0
Try block is run.
i = 2
No error report and y equals 0.5
Try block is run.
Is it a good practice to use try-except-else in python?
The answer to this is that it is context dependent. If you do this:
d = dict()
try:
item = d['item']
except KeyError:
item = 'default'
It demonstrates that you don't know Python very well. This functionality is encapsulated in the dict.get method:
item = d.get('item', 'default')
The try/except block is a much more visually cluttered and verbose way of writing what can be efficiently executing in a single line with an atomic method. There are other cases where this is true.
However, that does not mean that we should avoid all exception handling. In some cases it is preferred to avoid race conditions. Don't check if a file exists, just attempt to open it, and catch the appropriate IOError. For the sake of simplicity and readability, try to encapsulate this or factor it out as apropos.
Read the Zen of Python, understanding that there are principles that are in tension, and be wary of dogma that relies too heavily on any one of the statements in it.
You should be careful about using the finally block, as it is not the same thing as using an else block in the try, except. The finally block will be run regardless of the outcome of the try except.
In [10]: dict_ = {"a": 1}
In [11]: try:
....: dict_["b"]
....: except KeyError:
....: pass
....: finally:
....: print "something"
....:
something
As everyone has noted using the else block causes your code to be more readable, and only runs when an exception is not thrown
In [14]: try:
dict_["b"]
except KeyError:
pass
else:
print "something"
....:
Just because no-one else has posted this opinion, I would say
avoid else clauses in try/excepts because they're unfamiliar to most people
Unlike the keywords try, except, and finally, the meaning of the else clause isn't self-evident; it's less readable. Because it's not used very often, it'll cause people that read your code to want to double-check the docs to be sure they understand what's going on.
(I'm writing this answer precisely because I found a try/except/else in my codebase and it caused a wtf moment and forced me to do some googling).
So, wherever I see code like the OP example:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
else:
# do some more processing in non-exception case
return something
I would prefer to refactor to
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
return # <1>
# do some more processing in non-exception case <2>
return something
<1> explicit return, clearly shows that, in the exception case, we are finished working
<2> as a nice minor side-effect, the code that used to be in the else block is dedented by one level.
Whenever you see this:
try:
y = 1 / x
except ZeroDivisionError:
pass
else:
return y
Or even this:
try:
return 1 / x
except ZeroDivisionError:
return None
Consider this instead:
import contextlib
with contextlib.suppress(ZeroDivisionError):
return 1 / x
This is my simple snippet on howto understand try-except-else-finally block in Python:
def div(a, b):
try:
a/b
except ZeroDivisionError:
print("Zero Division Error detected")
else:
print("No Zero Division Error")
finally:
print("Finally the division of %d/%d is done" % (a, b))
Let's try div 1/1:
div(1, 1)
No Zero Division Error
Finally the division of 1/1 is done
Let's try div 1/0
div(1, 0)
Zero Division Error detected
Finally the division of 1/0 is done
I'm attempting to answer this question in a slightly different angle.
There were 2 parts of the OP's question, and I add the 3rd one, too.
What is the reason for the try-except-else to exist?
Does the try-except-else pattern, or the Python in general, encourage using exceptions for flow control?
When to use exceptions, anyway?
Question 1: What is the reason for the try-except-else to exist?
It can be answered from a tactical standpoint. There is of course reason for try...except... to exist. The only new addition here is the else... clause, whose usefulness boils down to its uniqueness:
It runs an extra code block ONLY WHEN there was no exception happened in the try... block.
It runs that extra code block, OUTSIDE of the try... block (meaning any potential exceptions happen inside the else... block would NOT be caught).
It runs that extra code block BEFORE the final... finalization.
db = open(...)
try:
db.insert(something)
except Exception:
db.rollback()
logging.exception('Failing: %s, db is ROLLED BACK', something)
else:
db.commit()
logging.info(
'Successful: %d', # <-- For the sake of demonstration,
# there is a typo %d here to trigger an exception.
# If you move this section into the try... block,
# the flow would unnecessarily go to the rollback path.
something)
finally:
db.close()
In the example above, you can't move that successful log line into behind the finally... block. You can't quite move it into inside the try... block, either, due to the potential exception inside the else... block.
Question 2: does Python encourage using exceptions for flow control?
I found no official written documentation to support that claim. (To readers who would disagree: please leave comments with links to evidences you found.) The only vaguely-relevant paragraph that I found, is this EAFP term:
EAFP
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.
Such paragraph merely described that, rather than doing this:
def make_some_noise(speaker):
if hasattr(speaker, "quack"):
speaker.quack()
we would prefer this:
def make_some_noise(speaker):
try:
speaker.quack()
except AttributeError:
logger.warning("This speaker is not a duck")
make_some_noise(DonaldDuck()) # This would work
make_some_noise(DonaldTrump()) # This would trigger exception
or potentially even omitting the try...except:
def make_some_noise(duck):
duck.quack()
So, the EAFP encourages duck-typing. But it does not encourage using exceptions for flow control.
Question 3: In what situation you should design your program to emit exceptions?
It is a moot conversation on whether it is anti-pattern to use exception as control flow. Because, once a design decision is made for a given function, its usage pattern would also be determined, and then the caller would have no choice but to use it that way.
So, let's go back to the fundamentals to see when a function would better produce its outcome via returning a value or via emitting exception(s).
What is the difference between the return value and the exception?
Their "blast radius" are different. Return value is only available to the immediate caller; exception can be automatically relayed for unlimited distance until it is caught.
Their distribution patterns are different. Return value is by definition one piece of data (even though you could return a compound data type such as a dictionary or a container object, it is still technically one value).
The exception mechanism, on the contrary, allows multiple values (one at a time) to be returned via their respective dedicate channel. Here, each except FooError: ... and except BarError: ... block is considered as its own dedicate channel.
Therefore, it is up to each different scenario to use one mechanism that fits well.
All normal cases should better be returned via return value, because the callers would most likely need to use that return value immediately. The return-value approach also allows nesting layers of callers in a functional programming style. The exception mechanism's long blast radius and multiple channels do not help here.
For example, it would be unintuitive if any function named get_something(...) produces its happy path result as an exception. (This is not really a contrived example. There is one practice to implement BinaryTree.Search(value) to use exception to ship the value back in the middle of a deep recursion.)
If the caller would likely forget to handle the error sentinel from the return value, it is probably a good idea to use exception's characterist #2 to save caller from its hidden bug. A typical non-example would be the position = find_string(haystack, needle), unfortunately its return value of -1 or null would tend to cause a bug in the caller.
If the error sentinel would collide with a normal value in the result namespace, it is almost certain to use an exception, because you'd have to use a different channel to convey that error.
If the normal channel i.e. the return value is already used in the happy-path, AND the happy-path does NOT have sophisicated flow control, you have no choice but to use exception for flow control. People keep talking about how Python uses StopIteration exception for iteration termination, and use it to kind of justify "using exception for flow control". But IMHO this is only a practical choice in a particular situation, it does not generalize and glorify "using exception for flow control".
At this point, if you already make a sound decision on whether your function get_stock_price() would produce only return-value or also raise exceptions, or if that function is provided by an existing library so that its behavior has long be decided, you do not have much choice in writing its caller calculate_market_trend(). Whether to use get_stock_price()'s exception to control the flow in your calculate_market_trend() is merely a matter of whether your business logic requires you to do so. If yes, do it; otherwise, let the exception bubble up to a higher level (this utilizes the characteristic #1 "long blast radius" of exception).
In particular, if you are implementing a middle-layer library Foo and you happen to be making a dependency on lower-level library Bar, you would probably want to hide your implementation detail, by catching all Bar.ThisError, Bar.ThatError, ..., and map them into Foo.GenericError. In this case, the long blast radius is actually working against us, so you might hope "only if library Bar were returning its errors via return values". But then again, that decision has long been made in Bar, so you can just live with it.
All in all, I think whether to use exception as control flow is a moot point.
OP, YOU ARE CORRECT. The else after try/except in Python is ugly. it leads to another flow-control object where none is needed:
try:
x = blah()
except:
print "failed at blah()"
else:
print "just succeeded with blah"
A totally clear equivalent is:
try:
x = blah()
print "just succeeded with blah"
except:
print "failed at blah()"
This is far clearer than an else clause. The else after try/except is not frequently written, so it takes a moment to figure what the implications are.
Just because you CAN do a thing, doesn't mean you SHOULD do a thing.
Lots of features have been added to languages because someone thought it might come in handy. Trouble is, the more features, the less clear and obvious things are because people don't usually use those bells and whistles.
Just my 5 cents here. I have to come along behind and clean up a lot of code written by 1st-year out of college developers who think they're smart and want to write code in some uber-tight, uber-efficient way when that just makes it a mess to try and read / modify later. I vote for readability every day and twice on Sundays.
I'm investigating an issue that I've narrowed down to one spot inside a method of the following format:
def method_shorthand(properties):
some stuff
some more stuff
return transaction.on_commit(
lambda: external_function_call_I_do_not_control(properties))
)
For a particular problem case I can reproduce, that external_function_call_I_do_not_control (firing a Segment event) doesn't happen/succeed, though I've determined all the info is correct right up until that point. For this codebase, 'ATOMIC_REQUESTS': True, so I'm predicting either:
the commit isn't actually happening (seems unlikely; this function is called as the very last step of a successful-until-then block of code)
the commit happens or there's no active transaction (in which case this just executes), but when this function call happens it barfs in some way- my money's on this one
So I'd like to add some kind of try/catch block in this external function line, so that if there's an exception when it runs, I can catch that, log the info, and see what's the matter so I can fix it. The problem is it's all within return transaction.on_commit, so everything I try seems to be the wrong syntax, and looking around I haven't been able to find an example of doing this.
Is there a safe way to add a try/catch block inside this return transaction.on_commit? Any other ideas for getting in there to troubleshoot, since the actual function call is external and I can't alter the code for logging/debugging?
Instead of using a lambda, use a whole function, which lets you do more complicated things than a lambda (which is usually just a direct input -> output map).
def method_shorthand(properties):
some stuff
some more stuff
def oncommit():
try:
external_function_call_I_do_not_control(properties)
except Exception as e:
# do whatever error handling needs to be done
return transaction.on_commit(oncommit)
I fully get the general principle of not catching all exception, as explained in this question, for instance (Why is "except: pass" a bad programming practice?). Yet I have found myself writing this sequence to get the source of a file like object:
try:
self.doc.source = source.geturl()
except:
try:
self.doc.source = pathlib.Path(os.path.abspath(source.name)).as_uri()
except:
self.doc.source = None
Clearly with some spelunking I could figure out which specific errors to catch with a reasonable degree of confidence. But at is explained in this question (Python: How can I know which exceptions might be thrown from a method call) you can't ever be quite certain.
So is there a better way to do what I am doing here, which is essentially to say: try this and if it doesn't work try this and if that doesn't work do this. Since this is all about setting a single variable, and there is a fallback of setting it to None, it is not obvious to me wherein the peril lies in this construct.
Maybe it'd be better to put it into a dedicated function ?
def func(source):
try:
return source.geturl()
except Exception:
try:
return pathlib.Path(os.path.abspath(source.name)).as_uri()
except Exception:
pass
Notes on swallowing exceptions
Ignoring any kind of exception is not the best pattern : it'd be better to know what call can raise what kind of exception.
Even if except ...: pass is bad, that's already what you were doing in your example, and here it's clear that we try another way if it fails.
The bad programming practice is basing the error handling on the ability to catch any kind of error. In itself, except: pass is better than except: <lots_of_things_that_wont_raise_again>.
However, the first answer in the post you referred to says something very sensible : pass semantically indicates that you won't do ANYTHING with the exception, and as you do not store in a variable, it also shows that you'll never be able to access it again (well you could, but still..). As /u/Aprillion/, you'll want to at least log these errors, because otherwise, you may be swallowing very useful debugging information, which could make your life (or others) much more difficult at some point.
For example, what happens if there's a bug hidden in geturl, that makes it raises exception in normal cases ? It may stay hidden for long, because this exception will be swallowed and never shown anywhere. Good luck to the debugger to find it !
Ways to improve this
replace except Exception with the exceptions that could be raised
check before an attempt if it is going to fail, so that you don't have to catch an exception
Additionally, you probably want to use a logger to save/show these exceptions somewhere, at least in DEBUG mode.
Clearly with some spelunking I could figure out which specific errors to catch with a reasonable degree of confidence. But at is explained in this question (Python: How can I know which exceptions might be thrown from a method call) you can't ever be quite certain.
Why would that change anything ? If geturl returns specific exception types, you'd only catch these, and you'd want other unexpected errors to bubble up to the interpreter.
The big problem with the givn approach is that if geturl is undefined, or not callable, or take more arguments, this error would not make the interpreter crash, even though it is a clear programming error. That's because except: or except Exception: will catch a lof of Python errors, should it be AttributeError, NameError, ImportError or even SyntaxError.
To conclude, I really think you'd prefer to replace Exception with the list of exceptions you can catch (and at least send the exceptions to a logger). Note that you can write it as such :
try:
return source.geturl()
except (ExceptionA, ExceptionB, ExceptionC):
pass
Is there a performance or code maintenance issue with using assert as part of the standard code instead of using it just for debugging purposes?
Is
assert x >= 0, 'x is less than zero'
better or worse than
if x < 0:
raise Exception('x is less than zero')
Also, is there any way to set a business rule like if x < 0 raise error that is always checked without the try/except/finally so, if at anytime throughout the code x is less than 0 an error is raised, like if you set assert x < 0 at the start of a function, anywhere within the function where x becomes less then 0 an exception is raised?
Asserts should be used to test conditions that should never happen. The purpose is to crash early in the case of a corrupt program state.
Exceptions should be used for errors that can conceivably happen, and you should almost always create your own Exception classes.
For example, if you're writing a function to read from a configuration file into a dict, improper formatting in the file should raise a ConfigurationSyntaxError, while you can assert that you're not about to return None.
In your example, if x is a value set via a user interface or from an external source, an exception is best.
If x is only set by your own code in the same program, go with an assertion.
"assert" statements are removed when the compilation is optimized. So, yes, there are both performance and functional differences.
The current code generator emits no code for an assert statement when optimization is requested at compile time. - Python 2 Docs Python 3 Docs
If you use assert to implement application functionality, then optimize the deployment to production, you will be plagued by "but-it-works-in-dev" defects.
See PYTHONOPTIMIZE and -O -OO
To be able to automatically throw an error when x become less than zero throughout the function. You can use class descriptors. Here is an example:
class LessThanZeroException(Exception):
pass
class variable(object):
def __init__(self, value=0):
self.__x = value
def __set__(self, obj, value):
if value < 0:
raise LessThanZeroException('x is less than zero')
self.__x = value
def __get__(self, obj, objType):
return self.__x
class MyClass(object):
x = variable()
>>> m = MyClass()
>>> m.x = 10
>>> m.x -= 20
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "my.py", line 7, in __set__
raise LessThanZeroException('x is less than zero')
LessThanZeroException: x is less than zero
The four purposes of assert
Assume you work on 200,000 lines of code with four colleagues Alice, Bernd, Carl, and Daphne.
They call your code, you call their code.
Then assert has four roles:
Inform Alice, Bernd, Carl, and Daphne what your code expects.
Assume you have a method that processes a list of tuples and the program logic can break if those tuples are not immutable:
def mymethod(listOfTuples):
assert(all(type(tp)==tuple for tp in listOfTuples))
This is more trustworthy than equivalent information in the documentation
and much easier to maintain.
Inform the computer what your code expects.
assert enforces proper behavior from the callers of your code.
If your code calls Alices's and Bernd's code calls yours,
then without the assert, if the program crashes in Alices code,
Bernd might assume it was Alice's fault,
Alice investigates and might assume it was your fault,
you investigate and tell Bernd it was in fact his.
Lots of work lost.
With asserts, whoever gets a call wrong, they will quickly be able to see it was
their fault, not yours. Alice, Bernd, and you all benefit.
Saves immense amounts of time.
Inform the readers of your code (including yourself) what your code has achieved at some point.
Assume you have a list of entries and each of them can be clean (which is good)
or it can be smorsh, trale, gullup, or twinkled (which are all not acceptable).
If it's smorsh it must be unsmorshed; if it's trale it must be baludoed;
if it's gullup it must be trotted (and then possibly paced, too);
if it's twinkled it must be twinkled again except on Thursdays.
You get the idea: It's complicated stuff.
But the end result is (or ought to be) that all entries are clean.
The Right Thing(TM) to do is to summarize the effect of your
cleaning loop as
assert(all(entry.isClean() for entry in mylist))
This statements saves a headache for everybody trying to understand
what exactly it is that the wonderful loop is achieving.
And the most frequent of these people will likely be yourself.
Inform the computer what your code has achieved at some point.
Should you ever forget to pace an entry needing it after trotting,
the assert will save your day and avoid that your code
breaks dear Daphne's much later.
In my mind, assert's two purposes of documentation (1 and 3) and
safeguard (2 and 4) are equally valuable.
Informing the people may even be more valuable than informing the computer
because it can prevent the very mistakes the assert aims to catch (in case 1)
and plenty of subsequent mistakes in any case.
In addition to the other answers, asserts themselves throw exceptions, but only AssertionErrors. From a utilitarian standpoint, assertions aren't suitable for when you need fine grain control over which exceptions you catch.
The only thing that's really wrong with this approach is that it's hard to make a very descriptive exception using assert statements. If you're looking for the simpler syntax, remember you can also do something like this:
class XLessThanZeroException(Exception):
pass
def CheckX(x):
if x < 0:
raise XLessThanZeroException()
def foo(x):
CheckX(x)
#do stuff here
Another problem is that using assert for normal condition-checking is that it makes it difficult to disable the debugging asserts using the -O flag.
The English language word assert here is used in the sense of swear, affirm, avow. It doesn't mean "check" or "should be". It means that you as a coder are making a sworn statement here:
# I solemnly swear that here I will tell the truth, the whole truth,
# and nothing but the truth, under pains and penalties of perjury, so help me FSM
assert answer == 42
If the code is correct, barring Single-event upsets, hardware failures and such, no assert will ever fail. That is why the behaviour of the program to an end user must not be affected. Especially, an assert cannot fail even under exceptional programmatic conditions. It just doesn't ever happen. If it happens, the programmer should be zapped for it.
As has been said previously, assertions should be used when your code SHOULD NOT ever reach a point, meaning there is a bug there. Probably the most useful reason I can see to use an assertion is an invariant/pre/postcondition. These are something that must be true at the start or end of each iteration of a loop or a function.
For example, a recursive function (2 seperate functions so 1 handles bad input and the other handles bad code, cause it's hard to distinguish with recursion). This would make it obvious if I forgot to write the if statement, what had gone wrong.
def SumToN(n):
if n <= 0:
raise ValueError, "N must be greater than or equal to 0"
else:
return RecursiveSum(n)
def RecursiveSum(n):
#precondition: n >= 0
assert(n >= 0)
if n == 0:
return 0
return RecursiveSum(n - 1) + n
#postcondition: returned sum of 1 to n
These loop invariants often can be represented with an assertion.
Well, this is an open question, and I have two aspects that I want to touch on: when to add assertions and how to write the error messages.
Purpose
To explain it to a beginner - assertions are statements which can raise errors, but you won't be catching them. And they normally should not be raised, but in real life they sometimes do get raised anyway. And this is a serious situation, which the code cannot recover from, what we call a 'fatal error'.
Next, it's for 'debugging purposes', which, while correct, sounds very dismissive. I like the 'declaring invariants, which should never be violated' formulation better, although it works differently on different beginners... Some 'just get it', and others either don't find any use for it, or replace normal exceptions, or even control flow with it.
Style
In Python, assert is a statement, not a function! (remember assert(False, 'is true') will not raise. But, having that out of the way:
When, and how, to write the optional 'error message'?
This acually applies to unit testing frameworks, which often have many dedicated methods to do assertions (assertTrue(condition), assertFalse(condition), assertEqual(actual, expected) etc.). They often also provide a way to comment on the assertion.
In throw-away code you could do without the error messages.
In some cases, there is nothing to add to the assertion:
def dump(something):
assert isinstance(something, Dumpable)
# ...
But apart from that, a message is useful for communication with other programmers (which are sometimes interactive users of your code, e.g. in Ipython/Jupyter etc.).
Give them information, not just leak internal implementation details.
instead of:
assert meaningless_identifier <= MAGIC_NUMBER_XXX, 'meaningless_identifier is greater than MAGIC_NUMBER_XXX!!!'
write:
assert meaningless_identifier > MAGIC_NUMBER_XXX, 'reactor temperature above critical threshold'
or maybe even:
assert meaningless_identifier > MAGIC_NUMBER_XXX, f'reactor temperature({meaningless_identifier }) above critical threshold ({MAGIC_NUMBER_XXX})'
I know, I know - this is not a case for a static assertion, but I want to point to the informational value of the message.
Negative or positive message?
This may be conroversial, but it hurts me to read things like:
assert a == b, 'a is not equal to b'
these are two contradictory things written next to eachother. So whenever I have an influence on the codebase, I push for specifying what we want, by using extra verbs like 'must' and 'should', and not to say what we don't want.
assert a == b, 'a must be equal to b'
Then, getting AssertionError: a must be equal to b is also readable, and the statement looks logical in code. Also, you can get something out of it without reading the traceback (which can sometimes not even be available).
For what it's worth, if you're dealing with code which relies on assert to function properly, then adding the following code will ensure that asserts are enabled:
try:
assert False
raise Exception('Python assertions are not working. This tool relies on Python assertions to do its job. Possible causes are running with the "-O" flag or running a precompiled (".pyo" or ".pyc") module.')
except AssertionError:
pass
Is there a performance issue?
Please remember to "make it work first before you make it work fast".
Very few percent of any program are usually relevant for its speed.
You can always kick out or simplify an assert if it ever proves to
be a performance problem -- and most of them never will.
Be pragmatic:
Assume you have a method that processes a non-empty list of tuples and the program logic will break if those tuples are not immutable. You should write:
def mymethod(listOfTuples):
assert(all(type(tp)==tuple for tp in listOfTuples))
This is probably fine if your lists tend to be ten entries long, but
it can become a problem if they have a million entries.
But rather than discarding this valuable check entirely you could
simply downgrade it to
def mymethod(listOfTuples):
assert(type(listOfTuples[0])==tuple) # in fact _all_ must be tuples!
which is cheap but will likely catch most of the actual program errors anyway.
An Assert is to check -
1. the valid condition,
2. the valid statement,
3. true logic;
of source code. Instead of failing the whole project it gives an alarm that something is not appropriate in your source file.
In example 1, since variable 'str' is not null. So no any assert or exception get raised.
Example 1:
#!/usr/bin/python
str = 'hello Python!'
strNull = 'string is Null'
if __debug__:
if not str: raise AssertionError(strNull)
print str
if __debug__:
print 'FileName '.ljust(30,'.'),(__name__)
print 'FilePath '.ljust(30,'.'),(__file__)
------------------------------------------------------
Output:
hello Python!
FileName ..................... hello
FilePath ..................... C:/Python\hello.py
In example 2, var 'str' is null. So we are saving the user from going ahead of faulty program by assert statement.
Example 2:
#!/usr/bin/python
str = ''
strNull = 'NULL String'
if __debug__:
if not str: raise AssertionError(strNull)
print str
if __debug__:
print 'FileName '.ljust(30,'.'),(__name__)
print 'FilePath '.ljust(30,'.'),(__file__)
------------------------------------------------------
Output:
AssertionError: NULL String
The moment we don't want debug and realized the assertion issue in the source code. Disable the optimization flag
python -O assertStatement.py
nothing will get print
Both the use of assert and the raising of exceptions are about communication.
Assertions are statements about the correctness of code addressed at developers: An assertion in the code informs readers of the code about conditions that have to be fulfilled for the code being correct. An assertion that fails at run-time informs developers that there is a defect in the code that needs fixing.
Exceptions are indications about non-typical situations that can occur at run-time but can not be resolved by the code at hand, addressed at the calling code to be handled there. The occurence of an exception does not indicate that there is a bug in the code.
Best practice
Therefore, if you consider the occurence of a specific situation at run-time as a bug that you would like to inform the developers about ("Hi developer, this condition indicates that there is a bug somewhere, please fix the code.") then go for an assertion. If the assertion checks input arguments of your code, you should typically add to the documentation that your code has "undefined behaviour" when the input arguments violate that conditions.
If instead the occurrence of that very situation is not an indication of a bug in your eyes, but instead a (maybe rare but) possible situation that you think should rather be handled by the client code, raise an exception. The situations when which exception is raised should be part of the documentation of the respective code.
Is there a performance [...] issue with using assert
The evaluation of assertions takes some time. They can be eliminated at compile time, though. This has some consequences, however, see below.
Is there a [...] code maintenance issue with using assert
Normally assertions improve the maintainability of the code, since they improve readability by making assumptions explicit and during run-time regularly verifying these assumptions. This will also help catching regressions. There is one issue, however, that needs to be kept in mind: Expressions used in assertions should have no side-effects. As mentioned above, assertions can be eliminated at compile time - which means that also the potential side-effects would disappear. This can - unintendedly - change the behaviour of the code.
In IDE's such as PTVS, PyCharm, Wing assert isinstance() statements can be used to enable code completion for some unclear objects.
I'd add I often use assert to specify properties such as loop invariants or logical properties my code should have, much like I'd specify them in formally-verified software.
They serve both the purpose of informing readers, helping me reason, and checking I am not making a mistake in my reasoning. For example :
k = 0
for i in range(n):
assert k == i * (i + 1) // 2
k += i
#do some things
or in more complicated situations:
def sorted(l):
return all(l1 <= l2 for l1, l2 in zip(l, l[1:]))
def mergesort(l):
if len(l) < 2: #python 3.10 will have match - case for this instead of checking length
return l
k = len(l // 2)
l1 = mergesort(l[:k])
l2 = mergesort(l[k:])
assert sorted(l1) # here the asserts allow me to explicit what properties my code should have
assert sorted(l2) # I expect them to be disabled in a production build
return merge(l1, l2)
Since asserts are disabled when python is run in optimized mode, do not hesitate to write costly conditions in them, especially if it makes your code clearer and less bug-prone
Is there a way knowing (at coding time) which exceptions to expect when executing python code?
I end up catching the base Exception class 90% of the time since I don't know which exception type might be thrown (reading the documentation doesn't always help, since many times an exception can be propagated from the deep. And many times the documentation is not updated or correct).
Is there some kind of tool to check this (like by reading the Python code and libs)?
I guess a solution could be only imprecise because of lack of static typing rules.
I'm not aware of some tool that checks exceptions, but you could come up with your own tool matching your needs (a good chance to play a little with static analysis).
As a first attempt, you could write a function that builds an AST, finds all Raise nodes, and then tries to figure out common patterns of raising exceptions (e. g. calling a constructor directly)
Let x be the following program:
x = '''\
if f(x):
raise IOError(errno.ENOENT, 'not found')
else:
e = g(x)
raise e
'''
Build the AST using the compiler package:
tree = compiler.parse(x)
Then define a Raise visitor class:
class RaiseVisitor(object):
def __init__(self):
self.nodes = []
def visitRaise(self, n):
self.nodes.append(n)
And walk the AST collecting Raise nodes:
v = RaiseVisitor()
compiler.walk(tree, v)
>>> print v.nodes
[
Raise(
CallFunc(
Name('IOError'),
[Getattr(Name('errno'), 'ENOENT'), Const('not found')],
None, None),
None, None),
Raise(Name('e'), None, None),
]
You may continue by resolving symbols using compiler symbol tables, analyzing data dependencies, etc. Or you may just deduce, that CallFunc(Name('IOError'), ...) "should definitely mean raising IOError", which is quite OK for quick practical results :)
You should only catch exceptions that you will handle.
Catching all exceptions by their concrete types is nonsense. You should catch specific exceptions you can and will handle. For other exceptions, you may write a generic catch that catches "base Exception", logs it (use str() function) and terminates your program (or does something else that's appropriate in a crashy situation).
If you really gonna handle all exceptions and are sure none of them are fatal (for example, if you're running the code in some kind of a sandboxed environment), then your approach of catching generic BaseException fits your aims.
You might be also interested in language exception reference, not a reference for the library you're using.
If the library reference is really poor and it doesn't re-throw its own exceptions when catching system ones, the only useful approach is to run tests (maybe add it to test suite, because if something is undocumented, it may change!). Delete a file crucial for your code and check what exception is being thrown. Supply too much data and check what error it yields.
You will have to run tests anyway, since, even if the method of getting the exceptions by source code existed, it wouldn't give you any idea how you should handle any of those. Maybe you should be showing error message "File needful.txt is not found!" when you catch IndexError? Only test can tell.
The correct tool to solve this problem is unittests. If you are having exceptions raised by real code that the unittests do not raise, then you need more unittests.
Consider this
def f(duck):
try:
duck.quack()
except ??? could be anything
duck can be any object
Obviously you can have an AttributeError if duck has no quack, a TypeError if duck has a quack but it is not callable. You have no idea what duck.quack() might raise though, maybe even a DuckError or something
Now supposing you have code like this
arr[i] = get_something_from_database()
If it raises an IndexError you don't know whether it has come from arr[i] or from deep inside the database function. usually it doesn't matter so much where the exception occurred, rather that something went wrong and what you wanted to happen didn't happen.
A handy technique is to catch and maybe reraise the exception like this
except Exception as e
#inspect e, decide what to do
raise
Noone explained so far, why you can't have a full, 100% correct list of exceptions, so I thought it's worth commenting on. One of the reasons is a first-class function. Let's say that you have a function like this:
def apl(f,arg):
return f(arg)
Now apl can raise any exception that f raises. While there are not many functions like that in the core library, anything that uses list comprehension with custom filters, map, reduce, etc. are affected.
The documentation and the source analysers are the only "serious" sources of information here. Just keep in mind what they cannot do.
I ran into this when using socket, I wanted to find out all the error conditions I would run in to (so rather than trying to create errors and figure out what socket does I just wanted a concise list). Ultimately I ended up grep'ing "/usr/lib64/python2.4/test/test_socket.py" for "raise":
$ grep raise test_socket.py
Any exceptions raised by the clients during their tests
raise TypeError, "test_func must be a callable function"
raise NotImplementedError, "clientSetUp must be implemented."
def raise_error(*args, **kwargs):
raise socket.error
def raise_herror(*args, **kwargs):
raise socket.herror
def raise_gaierror(*args, **kwargs):
raise socket.gaierror
self.failUnlessRaises(socket.error, raise_error,
self.failUnlessRaises(socket.error, raise_herror,
self.failUnlessRaises(socket.error, raise_gaierror,
raise socket.error
# Check that setting it to an invalid value raises ValueError
# Check that setting it to an invalid type raises TypeError
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
Which is a pretty concise list of errors. Now of course this only works on a case by case basis and depends on the tests being accurate (which they usually are). Otherwise you need to pretty much catch all exceptions, log them and dissect them and figure out how to handle them (which with unit testing wouldn't be to difficult).
There are two ways that I found informative. The first one, run the code in iPython, which will display the exception type.
n = 2
str = 'me '
str + 2
TypeError: unsupported operand type(s) for +: 'int' and 'str'
In the second way we settle for catching too much and improve on it over time. Include a try expression in your code and catch except Exception as err. Print sufficient data to know what exception was thrown. As exceptions are thrown improve your code by adding a more precise except clause. When you feel that you have caught all relevant exceptions remove the all inclusive one. A good thing to do anyway because it swallows programming errors.
try:
so something
except Exception as err:
print "Some message"
print err.__class__
print err
exit(1)
normally, you'd need to catch exception only around a few lines of code. You wouldn't want to put your whole main function into the try except clause. for every few line you always should now (or be able easily to check) what kind of exception might be raised.
docs have an exhaustive list of built-in exceptions. don't try to except those exception that you're not expecting, they might be handled/expected in the calling code.
edit: what might be thrown depends on obviously on what you're doing! accessing random element of a sequence: IndexError, random element of a dict: KeyError, etc.
Just try to run those few lines in IDLE and cause an exception. But unittest would be a better solution, naturally.
This is a copy and pasted answer I wrote for How to list all exceptions a function could raise in Python 3?, I hope that is allowed.
I needed to do something similar and found this post. I decided I
would write a little library to help.
Say hello to Deep-AST. It's very early alpha but it is pip
installable. It has all of the limitations mentioned in this post
and some additional ones but its already off to a really good start.
For example when parsing HTTPConnection.getresponse() from
http.client it parses 24489 AST Nodes. It finds 181 total raised
Exceptions (this includes duplicates) and 8 unique Exceptions were
raised. A working code example.
The biggest flaw is this it currently does work with a bare raise:
def foo():
try:
bar()
except TypeError:
raise
But I think this will be easy to solve and I plan on fixing it.
The library can handle more than just figuring out exceptions, what
about listing all Parent classes? It can handle that too!