arguments for / against `raise Exception(message)` in Python - python

I'm working with a framework and the source code is raising exceptions using the Exception class (and not a subclass, either framework specific or from the stdlib) in a few places, which is is not a good idea in my opinion.
The main argument against this idiom is that it forces the caller to use except Exception: which can catch more than what is meant, and therefore hide problems at lower stack levels.
However, a quick search in the Python documentation did not come up with arguments against this practice, and there are even examples of this in the tutorial (although things which are OK in Python scripts may not be OK at all in a Python framework in my opinion).
So is raise Exception considered pythonic?

From PEP 8:
Modules or packages should define their own domain-specific base exception class, which should be subclassed from the built-in Exception class.

No, it is not. At the very minimum the framework should provide its own exception class, and probably should have several (depending on the variety of things that could go wrong).
As you said, except Exception will catch way too much and is not good practice.

Related

In python how does the caller of something know if that something would throw an exception or not?

In the Java world, we know that the exceptions are classified into checked vs runtime and whenever something throws a checked exception, the caller of that something will be forced to handle that exception, one way or another. Thus the caller would be well aware of the fact that there is an exception and be prepared/coded to handle that.
But coming to Python, given there is no concept of checked exceptions (I hope that is correct), how does the caller of something know if that something would throw an exception or not? Given this "lack of knowledge that an exception could be thrown", how does the caller ever know that it could have handled an exception until it is too late?
There are no checked exceptions in Python.
Read the module docs.
Read the source.
Discover during testing.
Catch a wide range of exception types if necessary (see below).
For example, if you need to be safe:
try:
...
except Exception:
...
Avoid using a bare except clause, as it will even catch things like a KeyboardInterrupt.
As far as I know Python (6 years) there isn't anything similar to Java's throws keyword in Python.
how does the caller of something know if that something would throw an exception or not?
By reading the documentation for that something.
Design Principle of Python: it's easier to ask forgiveness than permission
EAFP
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and
catches exceptions if the assumption proves false. This clean and fast
style is characterized by the presence of many try and except
statements. The technique contrasts with the LBYL style common to many
other languages such as C.
Basics of Unix Philosophy: Rule of Repair
Repair what you can — but when you must fail, fail noisily and as soon
as possible.
The essence of both is to use error handling that allows you to find your bugs quickly and wind up with a much more robust program over the long run.
The practical lesson is to learn what errors you should look for as you develop, and only attempt to catch those in your modules, and only use generic Exception handling as a wrapper.

Is exception handling always expensive?

I've been told time and again that exception handling for operations like determining type is bad form since exceptions are always computationally expensive. Nevertheless, I've seen posts (especially Python-related ones, such as the to reply of this one) that advise using exception handling for exactly that purpose.
I was wondering, then, if throwing and catching exceptions is to be avoided universally, because it is always computationally expensive, or whether some languages, such as Python, handle exceptions better and it is permissible to use exception handling more liberally.
You cannot give general advice such as "exceptions are expensive and therefore they should be avoided" for all programming languages.
As you suspected, in Python, Exceptions are used more liberally than in other languages such as C++. Instead of raw performance, Python puts emphasis on code readability. There is an idiom "It's easier to ask for forgiveness than for permission", meaning: It's easier to just attempt what you want to achieve and catch an exception than check for compatibility first.
Forgiveness:
try:
do_something_with(dict["key"])
except (KeyError, TypeError):
# Oh well, there is no "key" in dict, or it has the wrong type
Permission:
if hasattr(dict, "__getitem__") and "key" in dict:
do_something_with(dict["key"])
else:
# Oh well
Actually, in Python, iteration with for loops is implemented with exceptions under the hood: The iterable raises a StopIteration exception when the end is reached. So even if you try to avoid exceptions, you will use them anyway all the time.
I think a lot of it comes down to specific use cases.
In the example you posted, the poster explicitly refers to the "duck-typing" aspect of Python. Essentially, you use the exceptions generated to determine if a variable has a particular capability or set of capabilities instead of manually checking (since Python allows a lot of dynamic operations, a class might access "split" through __getattr__, which makes it impossible to check using a standard if statement, so you try to use split, then if it can't do it, we go to plan B).
In a lot of Python applications, also, we tend not to worry a lot about some of the performance details that might matter in other applications, so any overhead from exceptions is "trivial."
In coding my module tco, I encountered this question. In the version 1.0.1alpha, I emplemented three versions of the same class. The module is intended for computational purpose; thus I think I may give some answer to your question.
Computing quick operations by embedding them in the class working without exceptions was twice as quick as with the two classes working with exception. But you have to know that such a test may be meaningless if you think that computing interesting things between the exceptions will make the difference become very tiny. Nobody will seriously care about the difference of time between an empty loop and an empty system raising and catching exceptions!
For that reason, I decided to remove the first system when releasing the 1.1 version of my module. Though a little slower, I found that the system relying on exceptions was more robust and I focused on it.

Catching Python runtime errors only

I find myself handling exceptions without specifying an exception type when I call code that interacts with system libraries like shutil, http, etc, all of which can throw if the system is in an unexpected state (e.g. a file locked, network unavailable, etc)
try:
# call something
except:
print("OK, so it went wrong.")
This bothers me because it also catches SyntaxError and other exceptions based in programmer error, and I've seen recommendations to avoid such open-ended exception handlers.
Is there a convention that all runtime errors derive from some common exception base class that I can use here? Or anything that doesn't involve syntax errors, module import failures, etc.? Even KeyError I would consider a bug, because I tend to use dict.get() if I'm not 100% sure the key will be there.
I'd hate to have to list every single conceivable exception type, especially since I'm calling a lot of supporting code I have no control over.
UPDATE: OK, the answers made me realize I'm asking the wrong question -- what I'm really wondering is if there's a Python convention or explicit recommendation for library writers to use specific base classes for their exceptions, so as to separate them from the more mundane SyntaxError & friends.
Because if there's a convention for library writers, I, as a library consumer, can make general assumptions about what might be thrown, even if specific cases may vary. Not sure if that makes more sense?
UPDATE AGAIN: Sven's answer finally led me to understand that instead of giving up and catching everything at the top level, I can handle and refine exceptions at the lower levels, so the top level only needs to worry about the specific exception type from the level below.
Thanks!
Always make the try block as small as possible.
Only catch the exceptions you want to handle. Look in the documentation of the functions you are dealing with.
This ensures that you think about what exceptions may occur, and you think about what to do if they occur. If something happens you never thought about, chances are your exception handling code won't be able to correctly deal with that case anyway, so it would be better the exception gets propagated.
You said you'd "hate to have to list every single conceivable exception type", but usually it's not that bad. Opening a file? Catch IOError. Dealing with some library code? They often have their own exception hierarchies with a specific top-level exception -- just catch this one if you want to catch any of the library-specific exceptions. Be as specific as possible, otherwise errors will pass unnoticed sooner or later.
As for convention about user-defined exceptions in Python: They usually should be derived from Exception. This is also what most user-defined exceptions in the standard library derive from, so the least you should do is use
except Exception:
instead of a bare except clause, which also catches KeyboardInterrupt and SystemExit. As you have noted yourself, this would still catch a lot of exceptions you don't want to catch.
Check out this list of built-in exceptions in the Python docs. It sounds like what you mostly want is to catch StandardError although this also includes KeyError. There are also other base classes, such as ArithmeticError and EnvironmentError, that you may find useful sometimes.
I find third-party libraries do derive their custom exceptions from Exception as documented here. Programmers also tend to raise standard Python exceptions such as TypeError, ValueError, etc. in appropriate situations. Unfortunately, this makes it difficult to consistently catch library errors separately from other errors derived from Exception. I wish Python defined e.g. a UserException base class. Some libraries do declare a base class for their exceptions, but you'd have to know what these are, import the module, and catch them explicitly.
Of course, if you want to catch everything except KeyError and, say, IndexError, along with the script-stopper exceptions, you could do this:
try:
doitnow()
except (StopIteration, GeneratorExit, KeyboardInterrupt, SystemExit):
raise # these stop the script
except (KeyError, IndexError):
raise # we don't want to handle these ones
except Exception as e:
handleError(e)
Admittedly, this becomes a hassle to write each time.
How about RuntimeError: http://docs.python.org/library/exceptions.html#exceptions.RuntimeError
If that isn't what you want (and it may well not be), look at the list of exceptions on that page. If you're confused by how the hierarchy fits together, I suggest you spend ten minutes examining the __bases__ property of the exceptions you're interested in, to see what base classes they share. (Note that __bases__ isn't closed over the whole hierarchy - you may need to examine superclass bases also).

try except and programming etiquette

I'm making a GUI and I'm finding myself to be using a lot of try except statements. My question is, should I be redesigning my program to use less try except statements or is try except a good practice to be using in python programs? I like them because they're informative and make debugging, for me, easier. Just wondering what real developers think about it.
Thanks
One of Python's idioms is: It's easier to ask for forgiveness than for permission. (Python Glossary, have a look at EAFP).
So it's perfectly acceptable to structure program flow with exception handling (and reasonably fast too, compared to other languages). It fits Python's dynamic nature nicely imho.
One large consideration when deciding whether to catch an exception is what legitimate errors you could be hiding.
For example, consider this code:
try:
name = person['name']
except KeyError:
name = '<none provided>'
This is reasonable if person is known to be a dict… But if person can possibly be something more complex, for example:
class Person(object):
def __getitem__(self, key):
return do_something(key)
You run the risk of accidentally catching an exception which was the result of a legitimate bug (for example, a bug in do_something).
And I feel the need to mention: you should never, ever (except under a couple of very specific circumstances) use a "naked" except:.
My personal preference is to avoid catching exceptions when ever possible (for example, using name = person.get('name', '<none provided>')), both because I find it cleaner and I dislike the look of try/catch blocks.
It's hard to give a general answer on whether you should use less exception handling... you can definitely do too much and too little. It's almost certainly wrong to be catching every possible exception and also almost certainly wrong to be doing no exception handling.
Here are some things to think about:
It's usually a good idea to catch the exception if you can programmatically do something about the error condition. E.g. your code is trying to make a web request and if it fails, you want to retry. In that situation you want to catch the exception and then do the retry.
Think carefully about where to catch an exception. In some low-level function, can you reasonably do something about the error? E.g. let's say you have a function that writes out a file and it fails with a permissions error. Probably not much you can do about it there but maybe at a higher level you can catch the exception and display a message to the user instructing them to try to save the file somewhere else.
It almost never makes sense to catch "fatal" types of errors e.g. out of memory, stack overflow etc. At least not low down in your code - it might make sense to have a top-level handler that tries to gracefully exit.
Don't "swallow" exceptions that really should bubble up i.e. don't have an except clause that doesn't re-raise the exception if your calling function should really see it. This can hide serious bugs.
For more, do a Google search for "exception handling guidelines". Many of the results you see will be for other languages/environments, but the concepts apply just as well.

listing exceptions programmatically - Python

Is there any way to programmatically determine which exceptions an object or method might raise?
Like dir(obj) lists available methods, I'm looking for the equivalent dir_exceptions(obj).
As far as I know, the only way to achieve this would be to parse the source.
I don't think this is possible. An exception is a runtime phenomenon and you'll know what it possible (or what happens) only while running. Why would you want to do this though?
It looks like you'll have to trust the code's developers on this one: if they did a good job, the method/class documentation should list all the exceptions that could be raised.
No, there is not a practical way to do this.
Most python developers derive from Exception, so if you're not sure, just catch Exception.
try:
some_secret_code()
except Exception:
print 'oops, something happened'
If you're thinking that you can import a module and poke around looking for things derived from Exception, that won't quite work either. What about that python nut that does this ->
exec "raise SystemExit()"
I'm not sure that there is a non-practical way to accomplish this.
I don't think this is possible either, but if you trust that the programmer has named their exceptions with "Exception" or "Error" in the name, then you could do a dir on the class and search for elements that end with "Exception" or "Error". Aside from that (which is pretty hacky in itself), I don't see a straightforward/native/idiomatic way to do this

Categories