Related
From time to time in Python, I see the block:
try:
try_this(whatever)
except SomeException as exception:
#Handle exception
else:
return something
What is the reason for the try-except-else to exist?
I do not like that kind of programming, as it is using exceptions to perform flow control. However, if it is included in the language, there must be a good reason for it, isn't it?
It is my understanding that exceptions are not errors, and that they should only be used for exceptional conditions (e.g. I try to write a file into disk and there is no more space, or maybe I do not have permission), and not for flow control.
Normally I handle exceptions as:
something = some_default_value
try:
something = try_this(whatever)
except SomeException as exception:
#Handle exception
finally:
return something
Or if I really do not want to return anything if an exception happens, then:
try:
something = try_this(whatever)
return something
except SomeException as exception:
#Handle exception
"I do not know if it is out of ignorance, but I do not like that
kind of programming, as it is using exceptions to perform flow control."
In the Python world, using exceptions for flow control is common and normal.
Even the Python core developers use exceptions for flow-control and that style is heavily baked into the language (i.e. the iterator protocol uses StopIteration to signal loop termination).
In addition, the try-except-style is used to prevent the race-conditions inherent in some of the "look-before-you-leap" constructs. For example, testing os.path.exists results in information that may be out-of-date by the time you use it. Likewise, Queue.full returns information that may be stale. The try-except-else style will produce more reliable code in these cases.
"It my understanding that exceptions are not errors, they should only
be used for exceptional conditions"
In some other languages, that rule reflects their cultural norms as reflected in their libraries. The "rule" is also based in-part on performance considerations for those languages.
The Python cultural norm is somewhat different. In many cases, you must use exceptions for control-flow. Also, the use of exceptions in Python does not slow the surrounding code and calling code as it does in some compiled languages (i.e. CPython already implements code for exception checking at every step, regardless of whether you actually use exceptions or not).
In other words, your understanding that "exceptions are for the exceptional" is a rule that makes sense in some other languages, but not for Python.
"However, if it is included in the language itself, there must be a
good reason for it, isn't it?"
Besides helping to avoid race-conditions, exceptions are also very useful for pulling error-handling outside loops. This is a necessary optimization in interpreted languages which do not tend to have automatic loop invariant code motion.
Also, exceptions can simplify code quite a bit in common situations where the ability to handle an issue is far removed from where the issue arose. For example, it is common to have top level user-interface code calling code for business logic which in turn calls low-level routines. Situations arising in the low-level routines (such as duplicate records for unique keys in database accesses) can only be handled in top-level code (such as asking the user for a new key that doesn't conflict with existing keys). The use of exceptions for this kind of control-flow allows the mid-level routines to completely ignore the issue and be nicely decoupled from that aspect of flow-control.
There is a nice blog post on the indispensibility of exceptions here.
Also, see this Stack Overflow answer: Are exceptions really for exceptional errors?
"What is the reason for the try-except-else to exist?"
The else-clause itself is interesting. It runs when there is no exception but before the finally-clause. That is its primary purpose.
Without the else-clause, the only option to run additional code before finalization would be the clumsy practice of adding the code to the try-clause. That is clumsy because it risks
raising exceptions in code that wasn't intended to be protected by the try-block.
The use-case of running additional unprotected code prior to finalization doesn't arise very often. So, don't expect to see many examples in published code. It is somewhat rare.
Another use-case for the else-clause is to perform actions that must occur when no exception occurs and that do not occur when exceptions are handled. For example:
recip = float('Inf')
try:
recip = 1 / f(x)
except ZeroDivisionError:
logging.info('Infinite result')
else:
logging.info('Finite result')
Another example occurs in unittest runners:
try:
tests_run += 1
run_testcase(case)
except Exception:
tests_failed += 1
logging.exception('Failing test case: %r', case)
print('F', end='')
else:
logging.info('Successful test case: %r', case)
print('.', end='')
Lastly, the most common use of an else-clause in a try-block is for a bit of beautification (aligning the exceptional outcomes and non-exceptional outcomes at the same level of indentation). This use is always optional and isn't strictly necessary.
What is the reason for the try-except-else to exist?
A try block allows you to handle an expected error. The except block should only catch exceptions you are prepared to handle. If you handle an unexpected error, your code may do the wrong thing and hide bugs.
An else clause will execute if there were no errors, and by not executing that code in the try block, you avoid catching an unexpected error. Again, catching an unexpected error can hide bugs.
Example
For example:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
else:
return something
The "try, except" suite has two optional clauses, else and finally. So it's actually try-except-else-finally.
else will evaluate only if there is no exception from the try block. It allows us to simplify the more complicated code below:
no_error = None
try:
try_this(whatever)
no_error = True
except SomeException as the_exception:
handle(the_exception)
if no_error:
return something
so if we compare an else to the alternative (which might create bugs) we see that it reduces the lines of code and we can have a more readable, maintainable, and less buggy code-base.
finally
finally will execute no matter what, even if another line is being evaluated with a return statement.
Broken down with pseudo-code
It might help to break this down, in the smallest possible form that demonstrates all features, with comments. Assume this syntactically correct (but not runnable unless the names are defined) pseudo-code is in a function.
For example:
try:
try_this(whatever)
except SomeException as the_exception:
handle_SomeException(the_exception)
# Handle a instance of SomeException or a subclass of it.
except Exception as the_exception:
generic_handle(the_exception)
# Handle any other exception that inherits from Exception
# - doesn't include GeneratorExit, KeyboardInterrupt, SystemExit
# Avoid bare `except:`
else: # there was no exception whatsoever
return something()
# if no exception, the "something()" gets evaluated,
# but the return will not be executed due to the return in the
# finally block below.
finally:
# this block will execute no matter what, even if no exception,
# after "something" is eval'd but before that value is returned
# but even if there is an exception.
# a return here will hijack the return functionality. e.g.:
return True # hijacks the return in the else clause above
It is true that we could include the code in the else block in the try block instead, where it would run if there were no exceptions, but what if that code itself raises an exception of the kind we're catching? Leaving it in the try block would hide that bug.
We want to minimize lines of code in the try block to avoid catching exceptions we did not expect, under the principle that if our code fails, we want it to fail loudly. This is a best practice.
It is my understanding that exceptions are not errors
In Python, most exceptions are errors.
We can view the exception hierarchy by using pydoc. For example, in Python 2:
$ python -m pydoc exceptions
or Python 3:
$ python -m pydoc builtins
Will give us the hierarchy. We can see that most kinds of Exception are errors, although Python uses some of them for things like ending for loops (StopIteration). This is Python 3's hierarchy:
BaseException
Exception
ArithmeticError
FloatingPointError
OverflowError
ZeroDivisionError
AssertionError
AttributeError
BufferError
EOFError
ImportError
ModuleNotFoundError
LookupError
IndexError
KeyError
MemoryError
NameError
UnboundLocalError
OSError
BlockingIOError
ChildProcessError
ConnectionError
BrokenPipeError
ConnectionAbortedError
ConnectionRefusedError
ConnectionResetError
FileExistsError
FileNotFoundError
InterruptedError
IsADirectoryError
NotADirectoryError
PermissionError
ProcessLookupError
TimeoutError
ReferenceError
RuntimeError
NotImplementedError
RecursionError
StopAsyncIteration
StopIteration
SyntaxError
IndentationError
TabError
SystemError
TypeError
ValueError
UnicodeError
UnicodeDecodeError
UnicodeEncodeError
UnicodeTranslateError
Warning
BytesWarning
DeprecationWarning
FutureWarning
ImportWarning
PendingDeprecationWarning
ResourceWarning
RuntimeWarning
SyntaxWarning
UnicodeWarning
UserWarning
GeneratorExit
KeyboardInterrupt
SystemExit
A commenter asked:
Say you have a method which pings an external API and you want to handle the exception at a class outside the API wrapper, do you simply return e from the method under the except clause where e is the exception object?
No, you don't return the exception, just reraise it with a bare raise to preserve the stacktrace.
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
raise
Or, in Python 3, you can raise a new exception and preserve the backtrace with exception chaining:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
raise DifferentException from the_exception
I elaborate in my answer here.
Python doesn't subscribe to the idea that exceptions should only be used for exceptional cases, in fact the idiom is 'ask for forgiveness, not permission'. This means that using exceptions as a routine part of your flow control is perfectly acceptable, and in fact, encouraged.
This is generally a good thing, as working this way helps avoid some issues (as an obvious example, race conditions are often avoided), and it tends to make code a little more readable.
Imagine you have a situation where you take some user input which needs to be processed, but have a default which is already processed. The try: ... except: ... else: ... structure makes for very readable code:
try:
raw_value = int(input())
except ValueError:
value = some_processed_value
else: # no error occured
value = process_value(raw_value)
Compare to how it might work in other languages:
raw_value = input()
if valid_number(raw_value):
value = process_value(int(raw_value))
else:
value = some_processed_value
Note the advantages. There is no need to check the value is valid and parse it separately, they are done once. The code also follows a more logical progression, the main code path is first, followed by 'if it doesn't work, do this'.
The example is naturally a little contrived, but it shows there are cases for this structure.
See the following example which illustrate everything about try-except-else-finally:
for i in range(3):
try:
y = 1 / i
except ZeroDivisionError:
print(f"\ti = {i}")
print("\tError report: ZeroDivisionError")
else:
print(f"\ti = {i}")
print(f"\tNo error report and y equals {y}")
finally:
print("Try block is run.")
Implement it and come by:
i = 0
Error report: ZeroDivisionError
Try block is run.
i = 1
No error report and y equals 1.0
Try block is run.
i = 2
No error report and y equals 0.5
Try block is run.
Is it a good practice to use try-except-else in python?
The answer to this is that it is context dependent. If you do this:
d = dict()
try:
item = d['item']
except KeyError:
item = 'default'
It demonstrates that you don't know Python very well. This functionality is encapsulated in the dict.get method:
item = d.get('item', 'default')
The try/except block is a much more visually cluttered and verbose way of writing what can be efficiently executing in a single line with an atomic method. There are other cases where this is true.
However, that does not mean that we should avoid all exception handling. In some cases it is preferred to avoid race conditions. Don't check if a file exists, just attempt to open it, and catch the appropriate IOError. For the sake of simplicity and readability, try to encapsulate this or factor it out as apropos.
Read the Zen of Python, understanding that there are principles that are in tension, and be wary of dogma that relies too heavily on any one of the statements in it.
You should be careful about using the finally block, as it is not the same thing as using an else block in the try, except. The finally block will be run regardless of the outcome of the try except.
In [10]: dict_ = {"a": 1}
In [11]: try:
....: dict_["b"]
....: except KeyError:
....: pass
....: finally:
....: print "something"
....:
something
As everyone has noted using the else block causes your code to be more readable, and only runs when an exception is not thrown
In [14]: try:
dict_["b"]
except KeyError:
pass
else:
print "something"
....:
Just because no-one else has posted this opinion, I would say
avoid else clauses in try/excepts because they're unfamiliar to most people
Unlike the keywords try, except, and finally, the meaning of the else clause isn't self-evident; it's less readable. Because it's not used very often, it'll cause people that read your code to want to double-check the docs to be sure they understand what's going on.
(I'm writing this answer precisely because I found a try/except/else in my codebase and it caused a wtf moment and forced me to do some googling).
So, wherever I see code like the OP example:
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
else:
# do some more processing in non-exception case
return something
I would prefer to refactor to
try:
try_this(whatever)
except SomeException as the_exception:
handle(the_exception)
return # <1>
# do some more processing in non-exception case <2>
return something
<1> explicit return, clearly shows that, in the exception case, we are finished working
<2> as a nice minor side-effect, the code that used to be in the else block is dedented by one level.
Whenever you see this:
try:
y = 1 / x
except ZeroDivisionError:
pass
else:
return y
Or even this:
try:
return 1 / x
except ZeroDivisionError:
return None
Consider this instead:
import contextlib
with contextlib.suppress(ZeroDivisionError):
return 1 / x
This is my simple snippet on howto understand try-except-else-finally block in Python:
def div(a, b):
try:
a/b
except ZeroDivisionError:
print("Zero Division Error detected")
else:
print("No Zero Division Error")
finally:
print("Finally the division of %d/%d is done" % (a, b))
Let's try div 1/1:
div(1, 1)
No Zero Division Error
Finally the division of 1/1 is done
Let's try div 1/0
div(1, 0)
Zero Division Error detected
Finally the division of 1/0 is done
I'm attempting to answer this question in a slightly different angle.
There were 2 parts of the OP's question, and I add the 3rd one, too.
What is the reason for the try-except-else to exist?
Does the try-except-else pattern, or the Python in general, encourage using exceptions for flow control?
When to use exceptions, anyway?
Question 1: What is the reason for the try-except-else to exist?
It can be answered from a tactical standpoint. There is of course reason for try...except... to exist. The only new addition here is the else... clause, whose usefulness boils down to its uniqueness:
It runs an extra code block ONLY WHEN there was no exception happened in the try... block.
It runs that extra code block, OUTSIDE of the try... block (meaning any potential exceptions happen inside the else... block would NOT be caught).
It runs that extra code block BEFORE the final... finalization.
db = open(...)
try:
db.insert(something)
except Exception:
db.rollback()
logging.exception('Failing: %s, db is ROLLED BACK', something)
else:
db.commit()
logging.info(
'Successful: %d', # <-- For the sake of demonstration,
# there is a typo %d here to trigger an exception.
# If you move this section into the try... block,
# the flow would unnecessarily go to the rollback path.
something)
finally:
db.close()
In the example above, you can't move that successful log line into behind the finally... block. You can't quite move it into inside the try... block, either, due to the potential exception inside the else... block.
Question 2: does Python encourage using exceptions for flow control?
I found no official written documentation to support that claim. (To readers who would disagree: please leave comments with links to evidences you found.) The only vaguely-relevant paragraph that I found, is this EAFP term:
EAFP
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.
Such paragraph merely described that, rather than doing this:
def make_some_noise(speaker):
if hasattr(speaker, "quack"):
speaker.quack()
we would prefer this:
def make_some_noise(speaker):
try:
speaker.quack()
except AttributeError:
logger.warning("This speaker is not a duck")
make_some_noise(DonaldDuck()) # This would work
make_some_noise(DonaldTrump()) # This would trigger exception
or potentially even omitting the try...except:
def make_some_noise(duck):
duck.quack()
So, the EAFP encourages duck-typing. But it does not encourage using exceptions for flow control.
Question 3: In what situation you should design your program to emit exceptions?
It is a moot conversation on whether it is anti-pattern to use exception as control flow. Because, once a design decision is made for a given function, its usage pattern would also be determined, and then the caller would have no choice but to use it that way.
So, let's go back to the fundamentals to see when a function would better produce its outcome via returning a value or via emitting exception(s).
What is the difference between the return value and the exception?
Their "blast radius" are different. Return value is only available to the immediate caller; exception can be automatically relayed for unlimited distance until it is caught.
Their distribution patterns are different. Return value is by definition one piece of data (even though you could return a compound data type such as a dictionary or a container object, it is still technically one value).
The exception mechanism, on the contrary, allows multiple values (one at a time) to be returned via their respective dedicate channel. Here, each except FooError: ... and except BarError: ... block is considered as its own dedicate channel.
Therefore, it is up to each different scenario to use one mechanism that fits well.
All normal cases should better be returned via return value, because the callers would most likely need to use that return value immediately. The return-value approach also allows nesting layers of callers in a functional programming style. The exception mechanism's long blast radius and multiple channels do not help here.
For example, it would be unintuitive if any function named get_something(...) produces its happy path result as an exception. (This is not really a contrived example. There is one practice to implement BinaryTree.Search(value) to use exception to ship the value back in the middle of a deep recursion.)
If the caller would likely forget to handle the error sentinel from the return value, it is probably a good idea to use exception's characterist #2 to save caller from its hidden bug. A typical non-example would be the position = find_string(haystack, needle), unfortunately its return value of -1 or null would tend to cause a bug in the caller.
If the error sentinel would collide with a normal value in the result namespace, it is almost certain to use an exception, because you'd have to use a different channel to convey that error.
If the normal channel i.e. the return value is already used in the happy-path, AND the happy-path does NOT have sophisicated flow control, you have no choice but to use exception for flow control. People keep talking about how Python uses StopIteration exception for iteration termination, and use it to kind of justify "using exception for flow control". But IMHO this is only a practical choice in a particular situation, it does not generalize and glorify "using exception for flow control".
At this point, if you already make a sound decision on whether your function get_stock_price() would produce only return-value or also raise exceptions, or if that function is provided by an existing library so that its behavior has long be decided, you do not have much choice in writing its caller calculate_market_trend(). Whether to use get_stock_price()'s exception to control the flow in your calculate_market_trend() is merely a matter of whether your business logic requires you to do so. If yes, do it; otherwise, let the exception bubble up to a higher level (this utilizes the characteristic #1 "long blast radius" of exception).
In particular, if you are implementing a middle-layer library Foo and you happen to be making a dependency on lower-level library Bar, you would probably want to hide your implementation detail, by catching all Bar.ThisError, Bar.ThatError, ..., and map them into Foo.GenericError. In this case, the long blast radius is actually working against us, so you might hope "only if library Bar were returning its errors via return values". But then again, that decision has long been made in Bar, so you can just live with it.
All in all, I think whether to use exception as control flow is a moot point.
OP, YOU ARE CORRECT. The else after try/except in Python is ugly. it leads to another flow-control object where none is needed:
try:
x = blah()
except:
print "failed at blah()"
else:
print "just succeeded with blah"
A totally clear equivalent is:
try:
x = blah()
print "just succeeded with blah"
except:
print "failed at blah()"
This is far clearer than an else clause. The else after try/except is not frequently written, so it takes a moment to figure what the implications are.
Just because you CAN do a thing, doesn't mean you SHOULD do a thing.
Lots of features have been added to languages because someone thought it might come in handy. Trouble is, the more features, the less clear and obvious things are because people don't usually use those bells and whistles.
Just my 5 cents here. I have to come along behind and clean up a lot of code written by 1st-year out of college developers who think they're smart and want to write code in some uber-tight, uber-efficient way when that just makes it a mess to try and read / modify later. I vote for readability every day and twice on Sundays.
It's generally accepted that using eval is bad practice. The accepted answer to this question states that there is almost always a better alternative. However, the timeit module in the standard library uses it, and I stumbled onto a case where I can't find a better alternative.
The unittest module has assertion functions of the form
self.assert*(..., msg=None)
allowing to assert something, optionally printing msg if it failed. This allows running code like
for i in range(1, 20):
self.assertEqual(foo(i), i, str(i) + ' failed')
Now consider the case where foo can raise an exception, e.g.,
def foo(i):
if i % 5 == 0:
raise ValueError()
return i
then
On the one hand, msg won't be printed, as assertEqual was technically never called for the offending iteration.
On the other hand, fundamentally, foo(i) == i failed to be true (admittedly because foo(i) never finished executing), and so this is a case where it would be useful to print msg.
I would like a version that prints out msg even if the failure cause was an exception - this will allow to understand exactly which invocation failed. Using eval, I could do this by writing a version taking strings, such as the following (which is a somewhat simplified version just to illustrate the point):
def assertEqual(lhs, rhs, msg=None):
try:
lhs_val = eval(lhs)
rhs_val = eval(rhs)
if lhs_val != rhs_val:
raise ValueError()
except:
if msg is not None:
print msg
raise
and then using
for i in range(1, 20):
self.assertEqual('foo(i)', 'i', str(i) + ' failed')
Of course technically it's possible to do it completely differently, by placing each call to assert* within a try/except/finally, but I could only think of extremely verbose alternatives (that also required duplicating msg.)
Is the use of eval legitimate here, then, or is there a better alternative?
If an exception is raised unexpectedly, that would point to a bug in your code. Exactly the case you want to discover with your unit tests. It's not simply not equal, it's a bug you discovered that you need to fix.
If you expect an exception to be raised, assert that with:
with self.assertRaises(ValueError):
foo(i)
If you expect no exception to be raised, use:
try:
foo(i)
except ValueError:
self.fail("foo() raised ValueEror unexpectedly!")
If anything, I'd suggest you write your own wrapper like:
self.assertEqualsAndCatch(foo, i, msg=...)
I.e.: pass a callback and its arguments, instead of a string to eval.
I'm thinking in a direction more advanced as well as difficult to find solutions this problem. Before coming to any decision, I thought of asking expert advice to address this problem.
The enhanced generators have new methods .send() and .throw() that allow the caller to pass messages or to raise exceptions into the generator (coroutine).
From python documentation: This can be very handy, especially the .throw() method that requests the generator to handle exceptions raised in the caller.
Request #1: Any example code for the above statement. I didn't find any code snippets for this explanation.
However, I'm considering the inverse problem as well: can a generator raise an exception, pass it to the caller, let the caller "repair" it, and continue the generator's own execution? That is what I would like to call a "reverse throw".
Request #2: Any example code for the above statement. I didn't find any code snippets for this explanation.
Simply raising exceptions in the generator is not OK. I tried "raise SomeException" in the generator, and that didn't work, because after a "raise" the generator can no longer be executed --- it simply stops, and further attempts to run the generator cause the StopIteration exception. In other words, "raise" is much more deadly than "yield": one can resume itself after yielding to the caller but a "raise" sends itself to the dead end.
I wonder if there are simple ways to do the "reverse throw" in Python? That will enable us to write coroutines that cooperate by throwing exceptions at each other. But why use exceptions? Well, I dunno... it all began as some rough idea.
CASE STUDY CODE:
class MyException(Exception):pass
def handleError(func):
''' handle an error'''
errors =[]
def wrapper(arg1):
result = func(arg1)
for err in findError(result):
errors.append(err)
print errors
return result
return wrapper
def findError(result):
'''
Find an error if any
'''
print result
for k, v in result.iteritems():
error_nr = v % 2
if error_nr ==0:
pass
elif error_nr > 0:
yield MyException
#handleError
def numGen(input):
''' This function take the input and generates 10 random numbers. 10 random numbers are saved in result dictionary with indices. Find error decorator is called based on the result dictionary'''
from random import randint
result= {}
errors = []
for i in range(9):
j = (randint(0,4))
result[i] = input + j
return result
if __name__ == '__main__':
numGen(4)
Could anyone explain please both the ideas based on case study example(Raising exception in a generator and handle it elsewhere vice versa)? I do expect pro's and con's of both methods.
Thanks in advance.
Looking for an answer drawing from credible and/or official sources.
Request #1 (Example for .throw())
I have never actually used this, but you could use it to change behaviour in the generator after the fact. You can also do this with .send of course, but then you'll need to deal with it in the line with the yield expressions (which might be in several locations in the code), rather than centralized with a try-except block.
def getstuff():
i=0
try:
while True:
yield i
i+=1
except ValueError:
while True:
yield i**2
i+=1
generator = getstuff()
print("Get some numbers...")
print(next(generator))
print(next(generator))
print(next(generator))
print("Oh, actually, I want squares!")
print(generator.throw(ValueError))
print(next(generator))
print(next(generator))
Request #1: Any example code for the above statement. I didn't find any code snippets for this explanation.
Take a look at ayscio source code
https://github.com/python/asyncio/search?utf8=%E2%9C%93&q=.throw
Request #2: Any example code for the above statement. I didn't find any code snippets for this explanation.
There is no way* to it today in python - maybe (if proven useful) can be a nice enhancement
*That is you can use yield to signal a framework to raise an exception elsewhere..
I have needed to solve this problem a couple of times and came upon this question after a search for what other people have done. I don't think I would used either of the methods suggested by the OP- they're pretty complicated.
One option- which will probably require refactoring things a little bit- would be to simply throw the exception in the generator (to another error handling generator) rather than raise it. Here is what that might look like:
def f(handler):
# the handler argument fixes errors/problems separately
while something():
try:
yield something_else()
except Exception as e:
handler.throw(e)
handler.close()
def err_handler():
# a generator for processing errors
while True:
try:
yield
except Exception1:
handle_exc1()
except Exception2:
handle_exc2()
except Exception3:
handle_exc3()
except Exception:
raise
def process():
handler = err_handler()
for item in f(handler):
do stuff
This isn't always going to be the best solution, but it's certainly an option, and relatively easy to understand.
Let me start off by saying: this is to be used for esoteric purposes - not production code. I'm playing around with doing stuff in a single line of Python code, hence my need for expressions and not statements. (EDIT: I'm working on mechanically compiling code to single line of (mostly) equivalent Python code, BitBucket - onelinepython. Note it's very work in progress, hence my reticence in initially mentioning it)
I essentially want to do two things:
Call a function that raises an exception instance of my choosing something like:
raise_exception(WhateverException())
Run a function in an enclosed environment where I can get the exception instance that is raised, if one is raised, and, otherwise, the return value of the function that was called. E.g.:
has_exception, return_or_exception = check_exception(f, param1, param2, ...)
Ideally, I want to do this with some default library or built-in function (no matter how much I have to bastardise its intended use). I don't need functions that have the exact same signatures as the examples I provided, just something I can mangle into something close enough. I do have one restriction, though: no use of eval() or equivalent functions.
EDIT: I know I could define my own functions to do this, but then they would still have to follow the restriction that they are a single expression. So solutions that use raise and try inside a function definition are out. Function definitions, raise-statement and try-blocks are unfortunately statements and not expressions.
As for any solutions I've tried. The answer is none yet. The closest I have to an idea of how to solve this is by misusing unittest's assert functionality, but I think that is a dead-end.
EDIT 2: To make it clear, I'm fine with using a module or such that uses raise-statements or try-blocks somewhere in its code. My goal is to take some code and turn it into an equivalent single line of code (which includes any helper functions I may be using). But since I want this to work on a default installation of Python I want to only use default libraries.
To raise an exception:
>>> import warnings
>>> WV = type("WV", (Warning, ValueError), {})
>>> warnings.simplefilter("error", WV)
>>> warnings.warn("wv", WV)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
__main__.WV: wv
To catch an exception:
>>> import unittest
>>> res = type("TR", (unittest.TestResult, ), dict(addError=lambda self, test, err: setattr(self, '_last_err', err)))()
>>> unittest.FunctionTestCase(lambda: [][0])(res)
>>> res._last_err
(<type 'exceptions.IndexError'>, IndexError('list index out of range',), <traceback object at 0x2b4358e69950>)
Note that the warnings method will only work for exceptions that derive from Warning, but you should always be able to multiply-inherit; here's an example:
>>> WS = type("WS", (Warning, StopIteration), {})
>>> warnings.simplefilter("error", WS)
>>> list(type("R", (object,), dict(__init__=lambda self, stop: (setattr(self, 'stop', stop), setattr(self, 'i', 0), None)[-1], __iter__=lambda self: self, next=lambda self: (self.i, setattr(self, 'i', self.i + 1))[0] if self.i < self.stop else warnings.warn("Stop", WS)))(5))
[0, 1, 2, 3, 4]
You can define your own functions to do this:
def raise_exception(ex):
raise ex
def check_exception(f, *args, **kwargs):
try:
return False, f(*args, **kwargs)
except Exception as e:
return True, e
This answer suggests that catching exceptions with an expression is not possible in general. I'm also pretty sure that it's not possible to raise an arbitrary exception without using raise. (You can generate some particular exceptions with expressions like 1/0 or dict['keyThatWillNeverExist'], but not any arbitrary exception with arbitrary exception info.)
The language reference says:
The Python interpreter raises an exception when it detects a run-time error (such as division by zero). A Python program can also explicitly raise an exception with the raise statement. Exception handlers are specified with the try ... except statement.
Although this doesn't rule out the possibility that some dark corner of the language specification allows raising exceptions in other ways, the statement is pretty straightforward: you raise exceptions with raise and catch them with try/except.
Note that using unittest, or any other Python library, is unlikely to be a real solution in your sense, because unittest contains functions written in Python that use try/except. So if you're okay with using unittest, you ought to be okay with writing your own functions.
I imagine it might be possible to achieve your goal by "cheating" and writing a C extension that provides functions doing what you want. But that's not really converting it to equivalent Python code.
You are asking how to raise an exception without using raise and catch an exception without using except. Your reluctance to use these statements is because you can't use more than one statement in a single line of code, and you have the idea to compile Python modules into oneliners.
Short answer: Well, you can't.
And even if you could, why would you? It's a completely meaningless effort. The code is not faster or even significantly smaller because it is in one line. It goes against the idea of Python as well. And if you want to obfuscate it, there are much better ways, including compiling it to bytecode.
Longer answer:
You could implement your own exception system, independent of the Python exceptions, but that would be astonishingly slow, and this would still not catch the Python exceptions, so it's not useful in your case.
For the raise-statement, you could re-implementing the raise statement as a function in C, but this you seem to think is cheating, and I also don't see how it would be possible with other statements, such as except.
You could also move out some statements into functions in a separate module, but this is of course then no longer actually a one-liner module in any meaningful way, and not all statements are easily wrapped like this, except being the most relevant case here. You'd have to wrap the whole try/except block, but the resulting function would in turn also only take expressions as parameters, so you would have to extract the blocks into functions, and you'd end up needing to basically re-implement most of Python as a statement-less language, which is silly. And you'd end up with the helper functions in a separate module, which you don't want to.
So the answer to your question of how to raise an exception without using raise and catch an exception without using except is "You don't".
Is there a way knowing (at coding time) which exceptions to expect when executing python code?
I end up catching the base Exception class 90% of the time since I don't know which exception type might be thrown (reading the documentation doesn't always help, since many times an exception can be propagated from the deep. And many times the documentation is not updated or correct).
Is there some kind of tool to check this (like by reading the Python code and libs)?
I guess a solution could be only imprecise because of lack of static typing rules.
I'm not aware of some tool that checks exceptions, but you could come up with your own tool matching your needs (a good chance to play a little with static analysis).
As a first attempt, you could write a function that builds an AST, finds all Raise nodes, and then tries to figure out common patterns of raising exceptions (e. g. calling a constructor directly)
Let x be the following program:
x = '''\
if f(x):
raise IOError(errno.ENOENT, 'not found')
else:
e = g(x)
raise e
'''
Build the AST using the compiler package:
tree = compiler.parse(x)
Then define a Raise visitor class:
class RaiseVisitor(object):
def __init__(self):
self.nodes = []
def visitRaise(self, n):
self.nodes.append(n)
And walk the AST collecting Raise nodes:
v = RaiseVisitor()
compiler.walk(tree, v)
>>> print v.nodes
[
Raise(
CallFunc(
Name('IOError'),
[Getattr(Name('errno'), 'ENOENT'), Const('not found')],
None, None),
None, None),
Raise(Name('e'), None, None),
]
You may continue by resolving symbols using compiler symbol tables, analyzing data dependencies, etc. Or you may just deduce, that CallFunc(Name('IOError'), ...) "should definitely mean raising IOError", which is quite OK for quick practical results :)
You should only catch exceptions that you will handle.
Catching all exceptions by their concrete types is nonsense. You should catch specific exceptions you can and will handle. For other exceptions, you may write a generic catch that catches "base Exception", logs it (use str() function) and terminates your program (or does something else that's appropriate in a crashy situation).
If you really gonna handle all exceptions and are sure none of them are fatal (for example, if you're running the code in some kind of a sandboxed environment), then your approach of catching generic BaseException fits your aims.
You might be also interested in language exception reference, not a reference for the library you're using.
If the library reference is really poor and it doesn't re-throw its own exceptions when catching system ones, the only useful approach is to run tests (maybe add it to test suite, because if something is undocumented, it may change!). Delete a file crucial for your code and check what exception is being thrown. Supply too much data and check what error it yields.
You will have to run tests anyway, since, even if the method of getting the exceptions by source code existed, it wouldn't give you any idea how you should handle any of those. Maybe you should be showing error message "File needful.txt is not found!" when you catch IndexError? Only test can tell.
The correct tool to solve this problem is unittests. If you are having exceptions raised by real code that the unittests do not raise, then you need more unittests.
Consider this
def f(duck):
try:
duck.quack()
except ??? could be anything
duck can be any object
Obviously you can have an AttributeError if duck has no quack, a TypeError if duck has a quack but it is not callable. You have no idea what duck.quack() might raise though, maybe even a DuckError or something
Now supposing you have code like this
arr[i] = get_something_from_database()
If it raises an IndexError you don't know whether it has come from arr[i] or from deep inside the database function. usually it doesn't matter so much where the exception occurred, rather that something went wrong and what you wanted to happen didn't happen.
A handy technique is to catch and maybe reraise the exception like this
except Exception as e
#inspect e, decide what to do
raise
Noone explained so far, why you can't have a full, 100% correct list of exceptions, so I thought it's worth commenting on. One of the reasons is a first-class function. Let's say that you have a function like this:
def apl(f,arg):
return f(arg)
Now apl can raise any exception that f raises. While there are not many functions like that in the core library, anything that uses list comprehension with custom filters, map, reduce, etc. are affected.
The documentation and the source analysers are the only "serious" sources of information here. Just keep in mind what they cannot do.
I ran into this when using socket, I wanted to find out all the error conditions I would run in to (so rather than trying to create errors and figure out what socket does I just wanted a concise list). Ultimately I ended up grep'ing "/usr/lib64/python2.4/test/test_socket.py" for "raise":
$ grep raise test_socket.py
Any exceptions raised by the clients during their tests
raise TypeError, "test_func must be a callable function"
raise NotImplementedError, "clientSetUp must be implemented."
def raise_error(*args, **kwargs):
raise socket.error
def raise_herror(*args, **kwargs):
raise socket.herror
def raise_gaierror(*args, **kwargs):
raise socket.gaierror
self.failUnlessRaises(socket.error, raise_error,
self.failUnlessRaises(socket.error, raise_herror,
self.failUnlessRaises(socket.error, raise_gaierror,
raise socket.error
# Check that setting it to an invalid value raises ValueError
# Check that setting it to an invalid type raises TypeError
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
def raise_timeout(*args, **kwargs):
self.failUnlessRaises(socket.timeout, raise_timeout,
Which is a pretty concise list of errors. Now of course this only works on a case by case basis and depends on the tests being accurate (which they usually are). Otherwise you need to pretty much catch all exceptions, log them and dissect them and figure out how to handle them (which with unit testing wouldn't be to difficult).
There are two ways that I found informative. The first one, run the code in iPython, which will display the exception type.
n = 2
str = 'me '
str + 2
TypeError: unsupported operand type(s) for +: 'int' and 'str'
In the second way we settle for catching too much and improve on it over time. Include a try expression in your code and catch except Exception as err. Print sufficient data to know what exception was thrown. As exceptions are thrown improve your code by adding a more precise except clause. When you feel that you have caught all relevant exceptions remove the all inclusive one. A good thing to do anyway because it swallows programming errors.
try:
so something
except Exception as err:
print "Some message"
print err.__class__
print err
exit(1)
normally, you'd need to catch exception only around a few lines of code. You wouldn't want to put your whole main function into the try except clause. for every few line you always should now (or be able easily to check) what kind of exception might be raised.
docs have an exhaustive list of built-in exceptions. don't try to except those exception that you're not expecting, they might be handled/expected in the calling code.
edit: what might be thrown depends on obviously on what you're doing! accessing random element of a sequence: IndexError, random element of a dict: KeyError, etc.
Just try to run those few lines in IDLE and cause an exception. But unittest would be a better solution, naturally.
This is a copy and pasted answer I wrote for How to list all exceptions a function could raise in Python 3?, I hope that is allowed.
I needed to do something similar and found this post. I decided I
would write a little library to help.
Say hello to Deep-AST. It's very early alpha but it is pip
installable. It has all of the limitations mentioned in this post
and some additional ones but its already off to a really good start.
For example when parsing HTTPConnection.getresponse() from
http.client it parses 24489 AST Nodes. It finds 181 total raised
Exceptions (this includes duplicates) and 8 unique Exceptions were
raised. A working code example.
The biggest flaw is this it currently does work with a bare raise:
def foo():
try:
bar()
except TypeError:
raise
But I think this will be easy to solve and I plan on fixing it.
The library can handle more than just figuring out exceptions, what
about listing all Parent classes? It can handle that too!