I've been told time and again that exception handling for operations like determining type is bad form since exceptions are always computationally expensive. Nevertheless, I've seen posts (especially Python-related ones, such as the to reply of this one) that advise using exception handling for exactly that purpose.
I was wondering, then, if throwing and catching exceptions is to be avoided universally, because it is always computationally expensive, or whether some languages, such as Python, handle exceptions better and it is permissible to use exception handling more liberally.
You cannot give general advice such as "exceptions are expensive and therefore they should be avoided" for all programming languages.
As you suspected, in Python, Exceptions are used more liberally than in other languages such as C++. Instead of raw performance, Python puts emphasis on code readability. There is an idiom "It's easier to ask for forgiveness than for permission", meaning: It's easier to just attempt what you want to achieve and catch an exception than check for compatibility first.
Forgiveness:
try:
do_something_with(dict["key"])
except (KeyError, TypeError):
# Oh well, there is no "key" in dict, or it has the wrong type
Permission:
if hasattr(dict, "__getitem__") and "key" in dict:
do_something_with(dict["key"])
else:
# Oh well
Actually, in Python, iteration with for loops is implemented with exceptions under the hood: The iterable raises a StopIteration exception when the end is reached. So even if you try to avoid exceptions, you will use them anyway all the time.
I think a lot of it comes down to specific use cases.
In the example you posted, the poster explicitly refers to the "duck-typing" aspect of Python. Essentially, you use the exceptions generated to determine if a variable has a particular capability or set of capabilities instead of manually checking (since Python allows a lot of dynamic operations, a class might access "split" through __getattr__, which makes it impossible to check using a standard if statement, so you try to use split, then if it can't do it, we go to plan B).
In a lot of Python applications, also, we tend not to worry a lot about some of the performance details that might matter in other applications, so any overhead from exceptions is "trivial."
In coding my module tco, I encountered this question. In the version 1.0.1alpha, I emplemented three versions of the same class. The module is intended for computational purpose; thus I think I may give some answer to your question.
Computing quick operations by embedding them in the class working without exceptions was twice as quick as with the two classes working with exception. But you have to know that such a test may be meaningless if you think that computing interesting things between the exceptions will make the difference become very tiny. Nobody will seriously care about the difference of time between an empty loop and an empty system raising and catching exceptions!
For that reason, I decided to remove the first system when releasing the 1.1 version of my module. Though a little slower, I found that the system relying on exceptions was more robust and I focused on it.
Related
In the Java world, we know that the exceptions are classified into checked vs runtime and whenever something throws a checked exception, the caller of that something will be forced to handle that exception, one way or another. Thus the caller would be well aware of the fact that there is an exception and be prepared/coded to handle that.
But coming to Python, given there is no concept of checked exceptions (I hope that is correct), how does the caller of something know if that something would throw an exception or not? Given this "lack of knowledge that an exception could be thrown", how does the caller ever know that it could have handled an exception until it is too late?
There are no checked exceptions in Python.
Read the module docs.
Read the source.
Discover during testing.
Catch a wide range of exception types if necessary (see below).
For example, if you need to be safe:
try:
...
except Exception:
...
Avoid using a bare except clause, as it will even catch things like a KeyboardInterrupt.
As far as I know Python (6 years) there isn't anything similar to Java's throws keyword in Python.
how does the caller of something know if that something would throw an exception or not?
By reading the documentation for that something.
Design Principle of Python: it's easier to ask forgiveness than permission
EAFP
Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and
catches exceptions if the assumption proves false. This clean and fast
style is characterized by the presence of many try and except
statements. The technique contrasts with the LBYL style common to many
other languages such as C.
Basics of Unix Philosophy: Rule of Repair
Repair what you can — but when you must fail, fail noisily and as soon
as possible.
The essence of both is to use error handling that allows you to find your bugs quickly and wind up with a much more robust program over the long run.
The practical lesson is to learn what errors you should look for as you develop, and only attempt to catch those in your modules, and only use generic Exception handling as a wrapper.
I'm making a GUI and I'm finding myself to be using a lot of try except statements. My question is, should I be redesigning my program to use less try except statements or is try except a good practice to be using in python programs? I like them because they're informative and make debugging, for me, easier. Just wondering what real developers think about it.
Thanks
One of Python's idioms is: It's easier to ask for forgiveness than for permission. (Python Glossary, have a look at EAFP).
So it's perfectly acceptable to structure program flow with exception handling (and reasonably fast too, compared to other languages). It fits Python's dynamic nature nicely imho.
One large consideration when deciding whether to catch an exception is what legitimate errors you could be hiding.
For example, consider this code:
try:
name = person['name']
except KeyError:
name = '<none provided>'
This is reasonable if person is known to be a dict… But if person can possibly be something more complex, for example:
class Person(object):
def __getitem__(self, key):
return do_something(key)
You run the risk of accidentally catching an exception which was the result of a legitimate bug (for example, a bug in do_something).
And I feel the need to mention: you should never, ever (except under a couple of very specific circumstances) use a "naked" except:.
My personal preference is to avoid catching exceptions when ever possible (for example, using name = person.get('name', '<none provided>')), both because I find it cleaner and I dislike the look of try/catch blocks.
It's hard to give a general answer on whether you should use less exception handling... you can definitely do too much and too little. It's almost certainly wrong to be catching every possible exception and also almost certainly wrong to be doing no exception handling.
Here are some things to think about:
It's usually a good idea to catch the exception if you can programmatically do something about the error condition. E.g. your code is trying to make a web request and if it fails, you want to retry. In that situation you want to catch the exception and then do the retry.
Think carefully about where to catch an exception. In some low-level function, can you reasonably do something about the error? E.g. let's say you have a function that writes out a file and it fails with a permissions error. Probably not much you can do about it there but maybe at a higher level you can catch the exception and display a message to the user instructing them to try to save the file somewhere else.
It almost never makes sense to catch "fatal" types of errors e.g. out of memory, stack overflow etc. At least not low down in your code - it might make sense to have a top-level handler that tries to gracefully exit.
Don't "swallow" exceptions that really should bubble up i.e. don't have an except clause that doesn't re-raise the exception if your calling function should really see it. This can hide serious bugs.
For more, do a Google search for "exception handling guidelines". Many of the results you see will be for other languages/environments, but the concepts apply just as well.
I'm doing a bit of research for my final year project. It's mostly about creating a more convenient way of dealing with Exceptions thrown in programs. It does this by creating a custom handler for each type of Exception. I was wondering how often are builtin/standard library Exceptions are dealt with in comparison to Exception by you/3rd party software?
Why I'm asking is two fold:
I would like my demonstration as to more realistic. My project has the chance to be more help than just dealing with Exceptions so given the chance, I would rather work on giving the tool far more abilities. Given this, I would like my sample handlers to be bias in the "right" direction.
It will influence how detailed I can make the API to help create more detailed Exceptions and Exception Handlers.
Thanks for taking the time to read this crap.
EDIT:
I'll break it down because I dont think I'm explaining it properly.
The nice little stack trace you get when errors thrown about? I want to try and improve it and see if something before could indicate when it all started to go wrong(for some errors might need a different strategy for others and that's where defining handlers come in). I think I could do this. In order to do this, I need to divide my time accordingly. I want to know whether I should focus on tackling builtin errors or helping people define their handlers for their Exceptions(maybe this second is pointless but I can't know until I ask people). I'll do this by asking people about their experiences.
EDIT2:
I'm a dumbass, I mean errors not exceptions. I need sleep.
Regardless of what you're trying to do with the answer, I'll answer your specific question:
how often are builtin/standard library
Exceptions are dealt with in
comparison to Exception by you/3rd
party software?
It depends on the domain. Some areas lend themselves to defining specific exceptions (e.g. web programming), and others tend to rely on builtins (e.g. mathematical and scientific computing). The amount of exceptions handled probably leans more towards "standard" exceptions like TypeError or NameError, but harder errors usually aren't contained in the builtins (it's easy to fix an incorrect argument, invalid input, or a typo, which are common causes of exceptions like NameError or TypeError, but it's hard to fix an error that doesn't come from something so simple).
So, IMO, standard exceptions are more prevalent but the ones defined by users, frameworks, etc. are more useful/important (or whatever you want to call it) because they represent more complex and relevant issues.
You can always look at some popular 3rd party Python modules (Google code is a good place to look) and see how many of them define exceptions.
Suppose I'm creating a class to validate a number, like "Social Security" in US (just as an example of a country-based id). There are some rules to validate this number that comes from an input in a html form in a website.
I thinking about creating a simple class in Python, and a public validate method. This validate returns True or False, simply. This method will call other small private methods (like for the first 'x' numbers if there is a different rule), each one returning True or False as well.
Since this is really simple, I'm thinking of using boolean status codes only (if it's valid or not, don't need meaningful messages about what is wrong).
I've been reading some articles about using exceptions, and I would like to know your opinion in my situation: would using exceptions would be a good idea?
This is a very old question but since the only answer - IMO - is not applicable to Python, here comes my take on it.
Exceptions in Python is something many programmers new to the language have difficulties dealing with. Compared to other languages, Python differs significantly in how exceptions are used: in fact Python routinely uses exceptions for flow control.
The canonical example is the for loop: you will certainly agree that there is nothing "uniquely bizarre" about the loop exhausting its iterations (indeed that's what all loops do, unless broken)... yet rather than checking in advance if there are still values to process, Python keeps on trying reading values from the iterable, and failing that, rises the StopIterator exception, which in turn is catch by the for expression and make the code exiting the loop.
Furthermore, it is idiomatic in Python to go by the EAFP (it's Easier to Ask for Forgiveness than Permission = try-except) rather than LBYL (Look Before You Leap = if not A, B or C then).
In this regard, csj's answer is correct for C or Java but is irrelevant for Python (whose exceptions are seldom "exceptional" in nature).
Another factor to consider - though - is the scenario in which user data is invalid but you fail to act on the validation function outcome:
with a return statement, failing to process the False value will result in having your non-valid data sent down the pipeline,
contrarily, if you were to raise an Exception, failing to catch it would result in the exception propagating through your stack eventually resulting in your code to halt.
While the second option might seems scary at first, it is still the right road to take: if data is invalid, there is no sense in passing it further down the line... it will most probably introduce difficult-to-track bugs later on in the flow and you will have also missed the chance to fix a bug in your code (failing to act on non-valid data).
Again. Using exceptions is the pythonic way to do (but it does not apply to most other languages) as also stated in this other answer and in the zen of python:
Errors should never pass silently.
Unless explicitly silenced.
HTH!
If an input is either valid or not, then just return the boolean. There's nothing exceptional about a validation test encountering an invalid value.
This particular example relates to Django in Python, but should apply to any language supporting exceptions:
try:
object = ModelClass.objects.get(search=value)
except DoesNotExist:
pass
if object:
# do stuff
The Django model class provides a simple method get which allows me to search for one and only one object from the database, if it finds more or less it raises an exception. If can find zero or more with an alternative filter method, which returns a list:
objects = ModelClass.objects.filter(search=value)
if len(objects) == 1:
object = objects[0]
# do stuff
Am I overly averse to exceptions? To me the exception seems a little wasteful, at a guess, a quarter-to-a-half of the time will be 'exceptional'. I'd much prefer a function that returns None on failure. Would I be better to use Django's filter method and process the list myself?
Believe it or not, this actually is an issue that is a bit different in each language. In Python, exceptions are regularly thrown for events that aren't exceptional by the language itself. Thus I think that the "you should only throw exceptions under exceptional circumstances" rule doesn't quite apply. I think the results you'll get on this forum will be slanted towards that point of view though, considering the high number of .Net programmers (see this question) for more info on that).
At a very minimum, I'd better not catch anyone who sticks to that rule ever using a generator or a for loop in Python (both of which involve throwing exceptions for non-exceptional circumstances).
There's a big schism in programming languages around the use of exceptions.
The majority view is that exceptions should be exceptional. In most languages with exceptions, transfer of control by exception is considerably more expensive than by procedure return, for example.
There is a strong minority view that exceptions are just another control-flow construct, and they should be cheap. The Standard ML of New Jersey and Objective Caml compilers subscribe to that view. If you have cheap exceptions you can code some fancy backtracking algorithms in ways that are more difficult to code cleanly using other mechanisms.
I've seen this debate repeated many times for new language designs, and almost always, the winner decides that exceptions should be expensive and rare. When you care about performanced, you'd be wise to program with this in mind.
The clue is in the name - exceptions should be exceptional.
If you always expect the item will exist then use get, but if you expect it not to exist a reasonable proportion of the time (i.e. it not existing is an expected result rather than an exceptional result) then I'd suggest using filter.
So, seeing as you indicated that between 1 in 2 and 1 in 4 are expected not to exist, I'd definitely write a wrapper around filter, as that's definitely not an exceptional case.
I agree with the other answer but I wanted to add that exception passing like this is going to give you a very noticeable performance hit. Highly recommended that you check to see if the result exists (if that's what filter does) instead of passing on exceptions.
Edit:
In response to request for numbers on this, I ran this simple test...
import time
def timethis(func, list, num):
st=time.time()
for i in xrange(0,1000000):
try:
func(list,num)
except:
pass
et = time.time()
print "Took %gs" % (et-st)
def check(list, num):
if num < len(list):
return list[num]
else:
return None
a=[1]
timethis(check, a, 1)
timethis(lambda x,y:x[y], a, 1)
And the output was..
Took 0.772558s
Took 3.4512s
HTH.
The answer will depend on the intent of the code. (I'm not sure what your code sample was meant to do, the pass in the exceptional case is confusing, what will the rest of the code do with object variable to work with?)
Whether to use exceptions or to use a method which treat the case as non-exceptional is a matter of taste in many cases. Certainly if the real code in the except clause is as complicated as the filter method you'd have to use to avoid the exception, then use the filter method. Simpler code is better code.
Aversion to excpetions is a matter of opinion - however, if there's reason to believe that a function or method is going to be called many times or called rapidly, exceptions will cause a significant slowdown. I learned this from my previous question, where I was previously relying on a thrown exception to return a default value rather than doing parameter checking to return that default.
Of course, exceptions can still exist for any reason, and you shouldn't be afraid to use or throw one if necessary - especially ones that could potentially break the normal flow of the calling function.
I disagree with the above comments that an exception is inefficient in this instance, especially since it's being used in an I/O bound operation.
Here's a more realistic example using Django with an in-memory sqlite database. Each of a 100 different queries was run, then averaged for each of a 100 runs. Although I doubt if it would matter, I also changed the order of execution.
With ObjectDoesNotExist... 0.102783939838
Without exception ........ 0.105322141647
With ObjectDoesNotExist... 0.102762134075
Without exception ........ 0.101523952484
With ObjectDoesNotExist... 0.100004930496
Without exception ........ 0.107946784496
You can instrument this in your own Django environment, but I doubt if your time is well spent avoiding this exception.