We Keep Coding

Django/Python code: readable vs fast code

Lets say I want to get a record that match a value and I have 2 approaches to do it:
First:
try:
obj = Model.objects.get(field = value)
except
pass
Second:
if Model.objects.filter(field = value).count() > 0:
obj = Model.objects.filter(field_value)[0]
Lets put the code comments aside,which way should I use or which one do you prefer to read? The first one seems faster because only 1 DB look up, but the second way seems a bit more readable, but requires 2 DB look ups.

The first is preferred in Python, based on the EAFP design principle ("Easier to Ask Forgiveness than Permission"). Aside from speed, one advantage of this system is that there is no race condition -- in the second example, if other concurrent access to the database changes the results between the execution of the first and the second line of code, then your results will be inconsistent.
Depending on your use of transactions, the race condition possibility may be a non-issue, but in general EAFP is a prominent design pattern in Python and experienced Python coders will not have any trouble reading code in that form.
ETA: Oh, and I forgot: don't use except: (you do need the colon). Use except IndexError: or whatever other specific exception you are looking for. That way if you get a totally unexpected error like a failure to reach the database, it will propagate through and not be hidden. You don't want a situation where you later write code counting on the exception being thrown to mean "there are no results" and instead the system is trying to tell you "the database is down".

How about
matches = Model.objects.filter(field=value)[:1]
obj = matches[0] if matches else None
The query will only be evaluated once (when evaluating matches in the if part of the conditional expression), and this is used both for the emptiness check and retrieving the result. Note the slicing to limit the number of returned objects.

In simple cases, is it better to except & recover or to avoid the exception?

Consider:
categories = {'foo':[4], 'mer':[2, 9, 0]}
key = 'bar'
value = 5
We could safely append to a list stored in a dictionary in either of the following ways:
Being cautious, we always check whether the list exists before appending to it.
if not somedict.has_key(key):
somedict[key] = []
somedict[key].append(value)
Being direct, we simply clean up if there is an exception.
try:
somedict[key].append(value)
except KeyError:
somedict[key] = [value]
In both cases, the result could be:
{'foo':[4], 'mer':[2, 9, 0], 'bar':[5]}
To restate my question: In simple instances like this, is it better (in terms of style, efficiency, & philosophy) to be cautious or direct?

What you'll find is that your option 1 "being cautious" is often remarkably slow. Also, it's subject to obscure errors because the test you tried to write to "avoid" the exception is incorrect.
What you'll find is that your option 2 "being direct" is often much faster. It's also more likely to be correct, as well as faster and easier for people to read.
Why? Internally, Python often implements things like "contains" or "has_key" as an exception test.
def has_key( self, some_key ):
try:
self[some_key]
except KeyError:
return False
return True
Since this is typically how a has_key type of method is implemented, there's no reason for you code do waste time doing this in addition to what Python will already do.
More fundamentally, there's a correctness issue. Many attempts to prevent or avoid an exception are incomplete are incorrect.
For example, trying to establish if a string is potentially a float-point number is fraught with numerous exceptions and special cases. About the only way to do it correctly is this.
try:
x= float( some_string )
except ValueError:
# not a floating-point value
Just do the algorithm without worrying about "preventing" or "avoiding" exceptions.

In the general case, EFAP ("easier to ask for forgiveness than permission") is preferred in Python. Of course the rule of thumb "exceptions should be for exceptional cases" still holds (if you expect an exception to occur frequently, you propably should "look before you leap") - i.e. it depends. Efficiency-wise, it shouldn't make too much of a difference in most cases - when it does, consider that try blocks without exceptions are cheap and conditions are always checked.
Note that neither is necessary (at least you don't have to do it yourself/epplicitly) some cases, including this example - here, you should just use collections.defaultdict

You don't need a strong, compelling reason to use exceptions--they're not very expensive in Python. Here are some possible reasons to prefer one or the other, for your particular example:
The exception version requires a simpler API. Any container that supports item lookup and assignment (__getitem__ and __setitem__) will work. The non-exception version additionally requires that has_key be implemented.
The exception version may be slightly faster if the key usually exists, since it only requires a single dict lookup. The has_key version requires at least two--one for has_key and one for the actual lookup.
The non-exception version has a more consistent code path: it always puts the value in the array in the same place. By comparison, the exception version has a separate code path for each case.
Unless performance is particularly important (in which case you'd be benchmarking and profiling), none of these are very strong reasons; just use whichever seems more natural.

try is fast enough, except (if it happens) may not be. If the average length of those lists is going to be 1.1, use the check-first method. If it's going to be in the thousands, use try/except. If you are really worried, benchmark the alternatives.
Ensure that you are benchmarking the best alternatives. d.has_key(k) is a slow old has_been; you don't need the attribute lookup and the function call. Use k in d instead. Also use else to save a wasted append on the first trip:
Instead of:
if not somedict.has_key(key):
somedict[key] = []
somedict[key].append(value)
do this:
if key in somedict:
somedict[key].append(value)
else:
somedict[key] = [value]

You can use setdefault for this specific case:
somedict.setdefault(key, []).append(value)
See here: http://docs.python.org/library/stdtypes.html#mapping-types-dict

It depends, for exemple if the key is a paramenter of a function that will be used by an other programer, I would use the second approach, because I can't control the input, and the exception information it's actually usefull for a programer. But if its just a process inside a function and the key it's just some input from a database for exemple, the first approach it's better, then if something goes wrong, maybe show the exception information isn't helpfull at all. Use the exception approach if you want to do someting with the exception information.

EFAP is a good habit to get into for Python.
One reason is that it avoids the race condition if someone wants to use your code in a multithreaded app

Using 'try' vs. 'if' in Python

Is there a rationale to decide which one of try or if constructs to use, when testing variable to have a value?
For example, there is a function that returns either a list or doesn't return a value. I want to check result before processing it. Which of the following would be more preferable and why?
result = function();
if (result):
for r in result:
#process items
or
result = function();
try:
for r in result:
# Process items
except TypeError:
pass;
Related discussion:
Checking for member existence in Python

You often hear that Python encourages EAFP style ("it's easier to ask for forgiveness than permission") over LBYL style ("look before you leap"). To me, it's a matter of efficiency and readability.
In your example (say that instead of returning a list or an empty string, the function were to return a list or None), if you expect that 99 % of the time result will actually contain something iterable, I'd use the try/except approach. It will be faster if exceptions really are exceptional. If result is None more than 50 % of the time, then using if is probably better.
To support this with a few measurements:
>>> import timeit
>>> timeit.timeit(setup="a=1;b=1", stmt="a/b") # no error checking
0.06379691968322732
>>> timeit.timeit(setup="a=1;b=1", stmt="try:\n a/b\nexcept ZeroDivisionError:\n pass")
0.0829463709378615
>>> timeit.timeit(setup="a=1;b=0", stmt="try:\n a/b\nexcept ZeroDivisionError:\n pass")
0.5070195056614466
>>> timeit.timeit(setup="a=1;b=1", stmt="if b!=0:\n a/b")
0.11940114974277094
>>> timeit.timeit(setup="a=1;b=0", stmt="if b!=0:\n a/b")
0.051202772912802175
So, whereas an if statement always costs you, it's nearly free to set up a try/except block. But when an Exception actually occurs, the cost is much higher.
Moral:
It's perfectly OK (and "pythonic") to use try/except for flow control,
but it makes sense most when Exceptions are actually exceptional.
From the Python docs:
EAFP
Easier to ask for forgiveness than
permission. This common Python coding
style assumes the existence of valid
keys or attributes and catches
exceptions if the assumption proves
false. This clean and fast style is
characterized by the presence of many
try and except statements. The
technique contrasts with the LBYL
style common to many other languages
such as C.

Your function should not return mixed types (i.e. list or empty string). It should return a list of values or just an empty list. Then you wouldn't need to test for anything, i.e. your code collapses to:
for r in function():
# process items

Please ignore my solution if the code I provide is not obvious at first glance and you have to read the explanation after the code sample.
Can I assume that the "no value returned" means the return value is None? If yes, or if the "no value" is False boolean-wise, you can do the following, since your code essentially treats "no value" as "do not iterate":
for r in function() or ():
# process items
If function() returns something that's not True, you iterate over the empty tuple, i.e. you don't run any iterations. This is essentially LBYL.

Generally, the impression I've gotten is that exceptions should be reserved for exceptional circumstances. If the result is expected never to be empty (but might be, if, for instance, a disk crashed, etc), the second approach makes sense. If, on the other hand, an empty result is perfectly reasonable under normal conditions, testing for it with an if statement makes more sense.
I had in mind the (more common) scenario:
# keep access counts for different files
file_counts={}
...
# got a filename somehow
if filename not in file_counts:
file_counts[filename]=0
file_counts[filename]+=1
instead of the equivalent:
...
try:
file_counts[filename]+=1
except KeyError:
file_counts[filename]=1

Which of the following would be more preferable and why?
Look Before You Leap is preferable in this case. With the exception approach, a TypeError could occur anywhere in your loop body and it'd get caught and thrown away, which is not what you want and will make debugging tricky.
(I agree with Brandon Corfman though: returning None for ‘no items’ instead of an empty list is broken. It's an unpleasant habit of Java coders that should not be seen in Python. Or Java.)

Your second example is broken - the code will never throw a TypeError exception since you can iterate through both strings and lists. Iterating through an empty string or list is also valid - it will execute the body of the loop zero times.

bobince wisely points out that wrapping the second case can also catch TypeErrors in the loop, which is not what you want. If you do really want to use a try though, you can test if it's iterable before the loop
result = function();
try:
it = iter(result)
except TypeError:
pass
else:
for r in it:
#process items
As you can see, it's rather ugly. I don't suggest it, but it should be mentioned for completeness.

As far as the performance is concerned, using try block for code that normally
doesn’t raise exceptions is faster than using if statement everytime. So, the decision depends on the probability of excetional cases.

As a general rule of thumb, you should never use try/catch or any exception handling stuff to control flow. Even though behind the scenes iteration is controlled via the raising of StopIteration exceptions, you still should prefer your first code snippet to the second.

Is it better to use an exception or a return code in Python?

You may know this recommendation from Microsoft about the use of exceptions in .NET:
Performance Considerations
...
Throw exceptions only for
extraordinary conditions, ...
In addition, do not throw an exception
when a return code is sufficient...
(See the whole text at http://msdn.microsoft.com/en-us/library/system.exception.aspx.)
As a point of comparison, would you recommend the same for Python code?

The pythonic thing to do is to raise and handle exceptions. The excellent book "Python in a nutshell" discusses this in 'Error-Checking Strategies' in Chapter 6.
The book discusses EAFP ("it's easier to ask forgiveness than permission") vs. LBYL ("look before you leap").
So to answer your question:
No, I would not recommend the same for python code. I suggest you read chapter 6 of Python in a nutshell.

The best way to understand exceptions is "if your method can't do what its name says it does, throw." My personal opinion is that this advice should be applied equally to both .NET and Python.
The key difference is where you have methods that frequently can't do what their name says they should do, for instance, parsing strings as integers or retrieving a record from a database. The C# style is to avoid an exception being thrown in the first place:
int i;
if (Int32.TryParse(myString, out i)) {
doWhatever(i);
}
else {
doWhatever(0);
}
whereas Python is much more at ease with this kind of thing:
try:
i = int(myString)
except ValueError:
i = 0
doWhatever(i);

Usually, Python is geared towards expressiveness.
I would apply the same principle here: usually, you expect a function to return a result (in line with its name!) and not an error code.
For this reason, it is usually better raising an exception than returning an error code.
However, what is stated in the MSDN article applies to Python as well, and it's not really connected to returning an error code instead of an exception.
In many cases, you can see exception handling used for normal flow control, and for handling expected situations. In certain environments, this has a huge impact on performance; in all environments it has a big impact on program expressiveness and maintainability.
Exceptions are for exceptional situations, that are outside of normal program flow; if you expect something will happen, then you should handle directly, and then raise anything that you cannot expect / handle.
Of course, this is not a recipe, but only an heuristic; the final decision is always up to the developer and onto the context and cannot be stated in a fixed set of guidelines - and this is much truer for exception handling.

In Python exceptions are not very expensive like they are in some other languages, so I wouldn't recommend trying to avoid exceptions. But if you do throw an exception you would usually want catch it somewhere in your code, the exception being if a fatal error occurs.

I think whether to return an error code or throw an exception is something very valid to think about, and a cross-linguistic comparison may be helpful and informative. I guess the very generalized answer to this concern is simply the consideration: that the set of legal return values for any function should be made as small as possible, and as large as necessary.
Generally, this will mean that if a given method returns an integer number in a single test case, users can rightfully expect the method to always return an integer number or throw an exception. But, of course, the conceptually simplest way is not always the best way to handle things.
The return-value-of-least-surprise is usually None; and if you look into it, you’ll see that it’s the very semantics of None that license its usage across the board: it is a singleton, immutable value that, in a lot of cases, evaluates to False or prohibits further computation—no concatenation, no arithmetics. So if you chose to write a frob(x) method that returns a number for a string input, and None for non-numeric strings and any other input, and you use that inside an expression like a=42+frob('foo'), you still get an exception very close to the point where bogus things happened. Of course, if you stuff frob('foo') into a database column that has not been defined with NOT NULL, you might run into problems perhaps months later. This may or may not be justifiable.
So in most cases where you e.g. want to derive a number from a string, using somwething like a bare float(x) or int(x) is the way to go, as these built-ins will raise an exception when not given a digestable input. If that doesn’t suit your use case, consider returning None from a custom method; basically, this return value tells consumers that ‘Sorry, I was unable to understand your input.’. But you only want to do this if you positively know that going on in your program does make sense from that point onwards.
You see, I just found out how to turn each notice, warning, and error message into a potentially show-stopping exception in, uhm, PHP. It just drives me crazy that a typo in a variable name generates in the standard PHP configuration, nothing but a notice to the user. This is so bad. The program just goes on doing things with a program code that does not make sense at all! I can’t believe people find this a feature.
Likewise, one should view it like this: if, at any given point in time, it can be asserted with reasonable costs that the execution of a piece of code does no more make sense — since values are missing, are out of bounds, or are of an unexpected type, or when resources like a database connection have gone down — it is imperative, to minimize debugging headaches, to break execution and hand control up to any level in the code which feels entitled to handle the mishap.
Experience shows that refraining from early action and allowing bogus values to creep into your data is good for nothing but making your code harder to debug. So are many examples of over-zealous type-casting: allowing integers to be added to floats is reasonable. To allow a string with nothing but digits to be added to a number is a bogus practice that is likely to create strange, unlocalized errors that may pop up on any given line that happens to be processing that data.

I did a simple experiment to compare the performance of raising exceptions with the following code:
from functools import wraps
from time import time
import logging
def timed(foo):
#wraps(foo)
def bar(*a, **kw):
s = time()
foo(*a, **kw)
e = time()
print '%f sec' % (e - s)
return bar
class SomeException(Exception):
pass
def somefunc(_raise=False):
if _raise:
raise SomeException()
else:
return
#timed
def test1(_reps):
for i in xrange(_reps):
try:
somefunc(True)
except SomeException:
pass
#timed
def test2(_reps):
for i in xrange(_reps):
somefunc(False)
def main():
test1(1000000)
test2(1000000)
pass
if __name__ == '__main__':
main()
With the following results:
Raising exceptions: 3.142000 sec
Using return: 0.383000 sec
Exceptions are about 8 times slower than using return.

"else" considered harmful in Python?

In an answer (by S.Lott) to a question about Python's try...else statement:
Actually, even on an if-statement, the
else: can be abused in truly terrible
ways creating bugs that are very hard
to find. [...]
Think twice about else:. It is
generally a problem. Avoid it except
in an if-statement and even then
consider documenting the else-
condition to make it explicit.
Is this a widely held opinion? Is else considered harmful?
Of course you can write confusing code with it but that's true of any other language construct. Even Python's for...else seems to me a very handy thing to have (less so for try...else).

S.Lott has obviously seen some bad code out there. Haven't we all? I do not consider else harmful, though I've seen it used to write bad code. In those cases, all the surrounding code has been bad as well, so why blame poor else?

No it is not harmful, it is necessary.
There should always be a catch-all statement. All switches should have a default. All pattern matching in an ML language should have a default.
The argument that it is impossible to reason what is true after a series of if statements is a fact of life. The computer is the biggest finite state machine out there, and it is silly to enumerate every single possibility in every situation.
If you are really afraid that unknown errors go unnoticed in else statements, is it really that hard to raise an exception there?

Saying that else is considered harmful is a bit like saying that variables or classes are harmful. Heck, it's even like saying that goto is harmful. Sure, things can be misused. But at some point, you just have to trust programmers to be adults and be smart enough not to.
What it comes down to is this: if you're willing to not use something because an answer on SO or a blog post or even a famous paper by Dijkstra told you not to, you need to consider if programming is the right profession for you.

I wouldn't say it is harmful, but there are times when the else statement can get you into trouble. For instance, if you need to do some processing based on an input value and there are only two valid input values. Only checking for one could introduce a bug.
eg:
The only valid inputs are 1 and 2:
if(input == 1)
{
//do processing
...
}
else
{
//do processing
...
}
In this case, using the else would allow all values other than 1 to be processed when it should only be for values 1 and 2.

To me, the whole concept of certain popular language constructs being inherently bad is just plain wrong. Even goto has its place. I've seen very readable, maintainable code by the likes of Walter Bright and Linus Torvalds that uses it. It's much better to just teach programmers that readability counts and to use common sense than to arbitrarily declare certain constructs "harmful".

If you write:
if foo:
# ...
elif bar:
# ...
# ...
then the reader may be left wondering: what if neither foo nor bar is true? Perhaps you know, from your understanding of the code, that it must be the case that either foo or bar. I would prefer to see:
if foo:
# ...
else:
# at this point, we know that bar is true.
# ...
# ...
or:
if foo:
# ...
else:
assert bar
# ...
# ...
This makes it clear to the reader how you expect control to flow, without requiring the reader to have intimate knowledge of where foo and bar come from.
(in the original case, you could still write a comment explaining what is happening, but I think I would then wonder: "Why not just use an else: clause?")
I think the point is not that you shouldn't use else:; rather, that an else: clause can allow you to write unclear code and you should try to recognise when this happens and add a little comment to help out any readers.
Which is true about most things in programming languages, really :-)

Au contraire... In my opinion, there MUST be an else for every if. Granted, you can do stupid things, but you can abuse any construct if you try hard enough. You know the saying "a real programer can write FORTRAN in every language".
What I do lots of time is to write the else part as a comment, describing why there's nothing to be done.

Else is most useful when documenting assumptions about the code. It ensures that you have thought through both sides of an if statement.
Always using an else clause with each if statement is even a recommended practice in "Code Complete".

The rationale behind including the else statement (of try...else) in Python in the first place was to only catch the exceptions you really want to. Normally when you have a try...except block, there's some code that might raise an exception, and then there's some more code that should only run if the previous code was successful. Without an else block, you'd have to put all that code in the try block:
try:
something_that_might_raise_error()
do_this_only_if_that_was_ok()
except ValueError:
# whatever
The issue is, what if do_this_only_if_that_was_ok() raises a ValueError? It would get caught by the except statement, when you might not have wanted it to. That's the purpose of the else block:
try:
something_that_might_raise_error()
except ValueError:
# whatever
else:
do_this_only_if_that_was_ok()
I guess it's a matter of opinion to some extent, but I personally think this is a great idea, even though I use it very rarely. When I do use it, it just feels very appropriate (and besides, I think it helps clarify the code flow a bit)

Seems to me that, for any language and any flow-control statement where there is a default scenario or side-effect, that scenario needs to have the same level of consideration. The logic in if or switch or while is only as good as the condition if(x) while(x) or for(...). Therefore the statement is not harmful but the logic in their condition is.
Therefore, as developers it is our responsibility to code with the wide scope of the else in-mind. Too many developers treat it as a 'if not the above' when in-fact it can ignore all common sense because the only logic in it is the negation of the preceding logic, which is often incomplete. (an algorithm design error itself)
I don't then consider 'else' any more harmful than off-by-ones in a for() loop or bad memory management. It's all about the algorithms. If your automata is complete in its scope and possible branches, and all are concrete and understood then there is no danger. The danger is misuse of the logic behind the expressions by people not realizing the impact of wide-scope logic. Computers are stupid, they do what they are told by their operator(in theory)
I do consider the try and catch to be dangerous because it can negate handling to an unknown quantity of code. Branching above the raise may contain a bug, highlighted by the raise itself. This is can be non-obvious. It is like turning a sequential set of instructions into a tree or graph of error handling, where each component is dependent on the branches in the parent. Odd. Mind you, I love C.

There is a so called "dangling else" problem which is encountered in C family languages as follows:
if (a==4)
if (b==2)
printf("here!");
else
printf("which one");
This innocent code can be understood in two ways:
if (a==4)
if (b==2)
printf("here!");
else
printf("which one");
or
if (a==4)
if (b==2)
printf("here!");
else
printf("which one");
The problem is that the "else" is "dangling", one can confuse the owner of the else. Of course the compiler will not make this confusion, but it is valid for mortals.
Thanks to Python, we can not have a dangling else problem in Python since we have to write either
if a==4:
if b==2:
print "here!"
else:
print "which one"
or
if a==4:
if b==2:
print "here!"
else:
print "which one"
So that human eye catches it. And, nope, I do not think "else" is harmful, it is as harmful as "if".

In the example posited of being hard to reason, it can be written explicitly, but the else is still necessary.
E.g.
if a < 10:
# condition stated explicitly
elif a > 10 and b < 10:
# condition confusing but at least explicit
else:
# Exactly what is true here?
# Can be hard to reason out what condition is true
Can be written
if a < 10:
# condition stated explicitly
elif a > 10 and b < 10:
# condition confusing but at least explicit
elif a > 10 and b >=10:
# else condition
else:
# Handle edge case with error?

I think the point with respect to try...except...else is that it is an easy mistake to use it to create inconsistent state rather than fix it. It is not that it should be avoided at all costs, but it can be counter-productive.
Consider:
try:
file = open('somefile','r')
except IOError:
logger.error("File not found!")
else:
# Some file operations
file.close()
# Some code that no longer explicitly references 'file'
It would be real nice to say that the above block prevented code from trying to access a file that didn't exist, or a directory for which the user has no permissions, and to say that everything is encapsulated because it is within a try...except...else block. But in reality, a lot of code in the above form really should look like this:
try:
file = open('somefile','r')
except IOError:
logger.error("File not found!")
return False
# Some file operations
file.close()
# Some code that no longer explicitly references 'file'
You are often fooling yourself by saying that because file is no longer referenced in scope, it's okay to go on coding after the block, but in many cases something will come up where it just isn't okay. Or maybe a variable will later be created within the else block that isn't created in the except block.
This is how I would differentiate the if...else from try...except...else. In both cases, one must make the blocks parallel in most cases (variables and state set in one ought to be set in the other) but in the latter, coders often don't, likely because it's impossible or irrelevant. In such cases, it often will make a whole lot more sense to return to the caller than to try and keep working around what you think you will have in the best case scenario.