We Keep Coding

PEP8, locals() and interpolation

Here is some code:
foo = "Bears"
"Lions, Tigers and %(foo)s" % locals()
My PEP8 linter (SublimeLinter) complains about this, because foo is "unreferenced". My question is whether PEP8 should count this type of string interpolation as "referenced", or if there is a good reason to consider this "bad style".

Well, it isn't referenced. The part that's questionable style is using locals() to access variables instead of just accessing them by name. See this previous question for why that's a dubious idea. It's not a terrible thing, but it's not good style for a program that you want to maintain in the long term.
Edit: It's true that when you use a literal format string, it seems more explicit. But part of the point of the previous post is that in a larger program, you will probably wind up not using a literal format string. If it's a small program and you don't care, go ahead and use it. But warning about things that are likely to cause maintainability problems later is also part of what style guides and linters are for.
Also, locals isn't a canonical representation of names that are explicitly referenced in the literal. It's a canonical representation of all names in the local namespace. You can still do it if you like, but it's basically a loose/sloppy alternative to explicitly using the names you're using, which is again, exactly the sort of thing linters are supposed to warn you about.

Even if you reject BrenBarn's argument that foo isn't referenced, if you accept the argument that passing locals() in string formatting should be flagged, it may not be worth writing to code to consider foo referenced.
First, in every case where that extra code would help, the construct is not acceptable anyway, and the user is going to have to ignore a lint warning anyway. Yes, there is some harm in giving the user two lint warnings to ignore when there's only actually one problem, especially if one of the warnings is somewhat misleading. But is it enough harm to justify writing very complicated code and introduce new bugs into the linter?
You also have to consider that for this to actually work, the linter has to recognize not just % formatting, but also {} formatting, and every other kind of string formatting, HTML templating, etc. that the user could be using. In practice, this means handling various very common forms, and providing some kind of hook for the user to describe anything else.
And, on top of that, even if you don't think it should work with arbitrarily-generated format strings, it surely has to at least work with l10n. How is that going to work? If the format string is generated by something like gettext, the linter has no way of knowing whether foo is referenced, unless it can check all of the translations and see that at least one of them references foo—which means it has to understand (or have hooks to be taught) every string translation mechanism, and have access to the translation database.
So, I would suggest that, even if you consider the warning spurious in this case, you leave it there anyway. At most, add something which qualifies the warning:
foo is possibly unreferenced, but in a function that uses locals()

The following wouldn't make SublimeLinter happy either, It looks up each variable name referenced in the string and substitutes the corresponding value from the namespace mapping, which defaults to the caller's locals. As such it show the inherent limitation a utility like SublimeLinter has when trying to determine if something has been referenced in Python). My advice is just ignore SublimeLinter or add code to fake it out, like foo = foo. I've had to do something like the latter to get rid of C compiler warnings about things which were both legal and intended.
import re
import sys
SUB_RE = re.compile(r"%\((.*?)\)s")
def local_vars_subst(s, namespace=None):
if namespace is None:
namespace = sys._getframe(1).f_locals
def repl(matchobj):
var = matchobj.group(1).strip()
try:
retval = namespace[var]
except KeyError:
retval = "<undefined>"
return retval
return SUB_RE.sub(repl, s)
foo = "Bears"
print local_vars_subst("Lions, Tigers and %(foo)s")

Is using Python `isinstance` ever right?

I've got a 2D array of different blocks, all inheriting from Block. I want to check if the block that I clicked on is a Dirt type block, like this:
clickedblock = getClickedBlock()
if isinstance(clickedblock, Dirt):
place a block
else:
don't place a block
I've heard that isinstance is bad, and should be avoided because it creates forks in code. What times would isinstance be good to use?
Another more cumbersome solution for my problem would be to have a field of Block called 'id' and then check if it equals some constant that means Dirt. But that sounds quite bad and more prone for mistake than the simple isinstance.

Your example seems like a legitimate use case of isinstance().
It's not that isinstance() is bad, often polymorphism can be used for the same purpose (which results in cleaner code in where the class is used).
But sometimes, isinstance() is what you need. For example, the pythonic way of detecting whether a variable is string or not is isinstance(var, basestring).

I learned the hard way against using it. The problem is that the outcome is sensitive to how the class definitions were imported:
in the place where the object was instantiated
in the place where the isinstance test is performed
If one import was relative and the other absolute - the check will fail.
Essentially it'll be like checking on equality between SomeClass vs somepackage.SomeClass. It doesn't even matter that they come from the same file and so on. Also, there will be a similar outcome if you somehow have both the root directory and somepackage directory in your PYTHONPATH - then an absolute-styled import could denote a path from either of the "source roots" so two different absolute-styled imports would result in failing instance checks.
One could argue about how good practices would anyway prevent that from happening but good practices are also largely about not taking chances. In that spirit I prefer to put some abstract method in a common ancestor class so that I can later rely on how things quack rather than on what the interpreter believes them to be.
In Java each class resolves to its fully qualified class name. These are unique within a program and tests for instances are a breeze. In Python these can be slippery.

I think I'd change it to be more like:
PLACEABLE_TYPES = [ Dirt ]
if isinstance(clickedblock, PLACEABLE_TYPES):
place the block
else:
don't place the block
but the idea in the comments of:
if clickedblock.is_placeable(that_place):
place the block
else:
don't place the block
also has merit.

If you don't want to use it, you've got other options. Traditional duck typing solution:
try:
clickedblock_place = clickedblock.place
except AttributeError:
# don't place block
else:
clickedblock_place()
Or you can use hasattr:
if hasattr(clickedblock, 'place'):
clickedblock.place()
I hardly ever use isinstance except for checking up (or is it down?) the inheritance hierarchy, say, for instance, if you need to know if a name points to a str OR a unicode:
if isinstance(str1, basestring):
blah, blah, blah

"Ask forgiveness not permission" - explain

I'm not asking for personal "religious" opinions about this philosophy, rather something a bit more technical.
I understand this phrase is one of several litmus tests to see if your code is "pythonic". But to me, pythonic means clean, simple and intuitive, not loaded with exception handlers for bad coding.
So, practical example. I define a class:
class foo(object):
bar = None
def __init__(self):
# a million lines of code
self.bar = "Spike is my favorite vampire."
# a million more lines of code
Now, coming from a procedural background, in another function I wanna do this:
if foo.bar:
# do stuff
I'll get an attribute exception if I was impatient and did not do the initial foo = None. So, "ask forgiveness not permission" suggests I should do this instead?
try:
if foo.bar:
# do stuff
except:
# this runs because my other code was sloppy?
Why would it be better for me to add additional logic in a try block just so I can leave my class definition more ambiguous? Why not define everything initially, therfore explicitly grant permission?
(Don't beat me up about using try/except blocks... I use them everywhere. I just don't think it's right to use them to catch my own errors because I wasn't a thorough programmer.)
Or... do I completely misunderstand the "Ask Forgivess" mantra?

“Ask forgiveness, not permission” opposes two programming styles. “Ask for permission” goes like this:
if can_do_operation():
perform_operation()
else:
handle_error_case()
“Ask forgiveness” goes like this:
try:
perform_operation()
except Unable_to_perform:
handle_error_case()
This is a situation where it is expected that attempting to perform the operation might fail, and you have to handle the situation where the operation is impossible, one way or another. For example, if the operation is accessing a file, the file might not exist.
There are two main reasons why it's better to ask for forgiveness:
In a concurrent world (in a multithreaded program, or if the operation involves objects that are external to the program such as files, other processes, network resources, etc.), the situation might change between the time when you run can_do_operation() and the time when you run perform_operation(). So you'd have to handle the error anyway.
You need to use exactly the right criteria for asking permission. If you get it wrong, you'll either be unable to perform an operation that you could perform, or have an error occur because you can't perform the operation after all. For example, if you test whether a file exists before opening it, it's possible that the file does exist, but you can't open it because you don't have permission. Conversely, maybe the file is created when you open it (for example because it comes over a network connection that is only brought up when you actually open the file, not when you only poke to see whether it's there).
What ask-forgiveness situations have in common is that you're attempting to perform an operation, and you know that the operation may fail.
When you write foo.bar, the non-existence of bar is not normally considered a failure of the object foo. It's usually a programmer error: attempting to use an object in a way that it wasn't designed for. The consequence of a programmer error in Python is an unhandled exception (if you're lucky: of course, some programmer errors can't be detected automatically). So if bar is an optional part of the object, the normal way to deal with this is to have a bar field that's initialized to None, and set to some other value if the optional part is present. To test whether bar is present, write
if foo.bar is not None:
handle_optional_part(foo.bar)
else:
default_handling()
You can abbreviate if foo.bar is not None: to if foo.bar: only if bar will always be true when interpreted as a boolean — if bar could be 0, [], {} or any other object that has a false truth value, you need the is not None. It's also clearer, if you're testing for an optional part (as opposed to testing between True and False).
At this point you may ask: why not omit the initialization of bar when it's not there, and test its presence with hasattr or catch it with an AttributeError handler? Because your code only makes sense in two cases:
the object has no bar field;
the object has a bar field that means what you think it means.
So when writing or deciding to use the object, you need to make sure that it doesn't have a bar field with a different meaning. If you need to use some different object that has no bar field, that's probably not the only thing you'll need to adapt, so you'll probably want to make a derived class or encapsulate the object in another one.

The classical "ask forgiveness not permission" example is accessing values from a dict that may not exist. E.g.:
names = { 'joe': 'Joe Nathan', 'jo': 'Jo Mama', 'joy': 'Joy Full' }
name = 'hikaru'
try:
print names[name]
except KeyError:
print "Sorry, don't know this '{}' person".format(name)
Here the exception that might occur (KeyError) is stated, so that you're not asking forgiveness for every error that might occur, but only the one that would naturally occur.
For comparison, the "ask permission first" approach might look like:
if name in names:
print names[name]
else:
print "Sorry, don't know this '{}' person".format(name)
or
real_name = names.get(name, None)
if real_name:
print real_name
else:
print "Sorry, don't know this '{}' person".format(name)
Such examples of "ask forgiveness" are often too simple. IMO it's not crystal clear that try/except blocks are inherently better than if/else. The real value is much clearer when performing operations that might fail in a variety of ways--such as parsing; using eval(); accessing operating system, middleware, database, or network resources; or performing complex mathematics. When there are multiple potential failure modes, being prepared to get forgiveness is hugely valuable.
Other notes about your code examples:
You do not need to ladle try/except blocks around every variable usage. That would be horrible. And you don't need to set self.bar in your __init__() since it's set in your class definition above. It is usual to define it either in the class (if it's data likely to be shared among all instances of the class) or in the __init__() (if it's instance data, specific to each instance).
A value of None is not undefined, or an error, by the way. It's a specific and legitimate value, meaning none, nil, null, or nothing. Many languages have such values so programmers don't "overload" 0, -1, '' (empty string) or similar useful values.

There's lots of good answers here, I just thought I would add a point I have not seen mentioned so far.
Often asking for forgiveness instead of permission has performance improvements.
When you ask for permission, you have to perform an extra operation
to ask for permission every time.
When asking for forgiveness,
you only have to perform an extra operation sometimes, i.e. when
it fails.
Usually the failure cases are rare, which means if you are only asking for permission, then you hardly ever have to do any extra operations.
Yes, when it fails it throws an exception, as well as performing an extra operation, but exceptions in python are very fast. You can see some timings here: https://jeffknupp.com/blog/2013/02/06/write-cleaner-python-use-exceptions/

You're right -- the purpose of try and except are not to cover your sloppy coding. That just leads to more sloppy coding.
Exception handling should be used to handle exceptional circumstances (sloppy coding is not an exceptional circumstance). However, often, it is easy to predict which exceptional circumstances might happen. (e.g. a your program accepts user input and uses that to access a dictionary, but the user's input wasn't a key in the dictionary ...)

My personal non-religious opinion is that said mantra applies mainly to documented and well understood exit conditions and edge cases (e.g. I/O errors), and should never be used as a get-out-of-jail card for sloppy programming.
That said, try/except is often used when "better" alternatives exist. For example:
# Do this
value = my_dict.get("key", None)
# instead of this
try:
value = my_dict["key"]
except KeyError:
value = None
As for your example, do use if hasattr(foo, "bar") if your have no control over foo and need to check conformance with your expectations, otherwise simply use foo.bar and let the resulting error be your guide to identifying and fixing sloppy code.

While there's already a number of high quality answers, most primarily discuss this from a stylistic stand point, as appose to a functional one.
There are certain cases where we need to ask forgiveness, not permission to insure correct code (outside of a multithreaded programs).
A canonical example being,
if file_exists:
open_it()
In this example, the file could have been deleted between the check and trying to actually open the file. This is avoided by using try:
try:
open_it()
except FileNotFoundException:
pass # file doesn't exist
This crops up in a variety of places, often working with filesystems or external APIs.

In the Python context "Ask forgiveness not permission" implies a style of programming where you don't check for things to be as you expect beforhand, but rather you handle the errors that result if they are not. The classical example is not to check that a dictionary contains a given key as in:
d = {}
k = "k"
if k in d.keys():
print d[k]
else:
print "key \"" + k + "\" missing"
But rather to handle the resulting error if the key is missing:
d = {}
k = "k"
try:
print d[k]
except KeyError:
print "key \"" + k + "\" missing"
However the point is not to replace every if in your code with a try/except; that would make your code decidedly messier. Instead you should only catch errors where you really can do something about them. Ideally this would reduce the amount of overall error handling in your code, making its actual purpose more evident.

Ask forgiveness not permission is meant to simplify code. It's meant for code to be written like this when there's a reasonable expectation that .bar might trigger an AttributeError.
try:
print foo.bar
except AttributeError as e
print "No Foo!"
Your code appears to both ask permission AND forgiveness :)
The thing is, if you reasonably expect something to fail, use a try/catch. If you don't expect something to fail, and it fails anyway, well the exception that gets thrown becomes the equivalent of a failed assertion in other languages. You see where the unexpected exception is occurring and adjust your code/assumptions accordingly.

True privateness in Python

PEP 8 states that (emphasis mine):
We don't use the term "private" here, since no attribute is really private in Python (without a generally unnecessary amount of work).
I guess it refers to defining the actual class in some other language and then exposing only the public members to the interpreter. Is there some other way to achieve true privateness in Python?
I'm asking just out of curiosity.

No, nothing is truly private in Python.
If you know the method name, you can get it.
I think you might come up with a clever hack, but it would be just that - a hack. No such functionality exists in the language.

(Note: This is not "private" in the sense of C++/C#/Java type private, but it's close)
For a class, you can prefix a variable with '__'. This will cause Python to name mangle it so you can't accidentally call it from outside the class. For example:
class Test(object):
def __init__(self):
self.__number = 5
a = Test()
print a.__number # name error!
On the other hand, this isn't really private. You can access the number with:
print a.__Test_number
But it will prevent accidental mistakes, which is all private should be used for anyway. If it's prefixed with '__' and someone uses the code, their fault if that code breaks later on (they were warned).

There is no true privateness. The best you could do is obfuscate the variable name and then create getters/setters or properties.
The only way to achieve this that I can think of this is to read/write a file every time you need to have access to that variable.
You could also write a program in another language that will only respond to certain classes. Not sure how to go about doing this, though.

What is your strategy to avoid dynamic typing errors in Python (NoneType has no attribute x)?

I'm not sure if I like Python's dynamic-ness. It often results in me forgetting to check a type, trying to call an attribute and getting the NoneType (or any other) has no attribute x error. A lot of them are pretty harmless but if not handled correctly they can bring down your entire app/process/etc.
Over time I got better predicting where these could pop up and adding explicit type checking, but because I'm only human I miss one occasionally and then some end-user finds it.
So I'm interested in your strategy to avoid these. Do you use type-checking decorators? Maybe special object wrappers?
Please share...

forgetting to check a type
This doesn't make much sense. You so rarely need to "check" a type. You simply run unit tests and if you've provided the wrong type object, things fail. You never need to "check" much, in my experience.
trying to call an attribute and
getting the NoneType (or any other)
has no attribute x error.
Unexpected None is a plain-old bug. 80% of the time, I omitted the return. Unit tests always reveal these.
Of those that remain, 80% of the time, they're plain old bugs due to an "early exit" which returns None because someone wrote an incomplete return statement. These if foo: return structures are easy to detect with unit tests. In some cases, they should have been if foo: return somethingMeaningful, and in still other cases, they should have been if foo: raise Exception("Foo").
The rest are dumb mistakes misreading the API's. Generally, mutator functions don't return anything. Sometimes I forget. Unit tests find these quickly, since basically, nothing works right.
That covers the "unexpected None" cases pretty solidly. Easy to unit test for. Most of the mistakes involve fairly trivial-to-write tests for some pretty obvious species of mistakes: wrong return; failure to raise an exception.
Other "has no attribute X" errors are really wild mistakes where a totally wrong type was used. That's either really wrong assignment statements or really wrong function (or method) calls. They always fail elaborately during unit testing, requiring very little effort to fix.
A lot of them are pretty harmless but if not handled correctly they can bring down your entire app/process/etc.
Um... Harmless? If it's a bug, I pray that it brings down my entire app as quickly as possible so I can find it. A bug that doesn't crash my app is the most horrible situation imaginable. "Harmless" isn't a word I'd use for a bug that fails to crash my app.

If you write good unit tests for all of your code, you should find the errors very quickly when testing code.

You can also use decorators to enforce the type of attributes.
>>> #accepts(int, int, int)
... #returns(float)
... def average(x, y, z):
... return (x + y + z) / 2
...
>>> average(5.5, 10, 15.0)
TypeWarning: 'average' method accepts (int, int, int), but was given
(float, int, float)
15.25
>>> average(5, 10, 15)
TypeWarning: 'average' method returns (float), but result is (int)
15
I'm not really a fan of them, but I can see their usefulness.

One tool to try to help you keep your pieces fitting together well is interfaces. zope.interface is the most notable package in the Python world for using interfaces. Check out http://wiki.zope.org/zope3/WhatAreInterfaces and http://glyph.twistedmatrix.com/2009/02/explaining-why-interfaces-are-great.html to start to get an idea how interfaces and z.i in particular work. Interfaces can prove very useful in a large Python codebases.
Interfaces are no substitute for testing. Reasonably comprehensive testing is especially important in highly dynamic languages like Python where there are types of bugs that could not exist in a statically types language. Tests will also help you catch the sorts of bugs that are not unique to dynamic languages. Fortunately, developing in Python means that testing is easy (due to the flexibility) and you have plenty of time to write them that you saved because you're using Python.

One advantage of TDD is that you end up writing code that is easier to write tests for.
Writing code first and then the tests can result in code that superficially works the same, but is much harder to write 100% coverage tests for.
Each case is likely to be different
It might make sense to have a decorator to check whether a particular parameter is None (or some other unexpected value) if you use it in a bunch of places.
Maybe it is appropriate to use the Null pattern - if the code is blowing up because you are setting the initial value to None, you could instead set the initial value to a null version of the object.
More and more wrappers can add up to quite a performance hit though, so it's always better to write code from the start that avoids the corner cases

forgetting to check a type
With duck typing, it shouldn't be necessary to check a type. But that's theory, in reality you will often want to validate input parameters (e.g. checking a UUID with a regex). For that purpose, I created myself some handy decorators for simple type and return type checking which are called like this:
#decorators.params(0, int, 2, str) # first parameter must be integer / third a string
#decorators.returnsOrNone(int, long) # must return an int/long value or None
def doSomething(integerParam, noMatterWhatParam, stringParam):
...
For everything else I mostly use assertions. Of course one often forgets to check a parameter, so it's necessary to test and to test often.
trying to call an attribute
Happens to me very seldom. Actually I often use methods instead of direct access to attributes (the "good" old getter/setter approach sometimes).
because I'm only human I miss one occasionally and then some end-user finds it
"Software is always completed at the customers'." - An anti-pattern which you should solve with unit tests that handle all possible cases in a function. Easier said than done, but it helps...
As for other common Python mistakes (mistyped names, wrong imports, ...), I'm using Eclipse with PyDev for projects (not for small scripts). PyDev warns you about most of the simple kinds of mistakes.

I haven’t done a lot of Python programming, but I’ve done no programming at all in staticly typed languages, so I don’t tend to think about things in terms of variable types. That might explain why I haven’t come across this problem much. (Although the small amount of Python programming I’ve done might explain that too.)
I do enjoy Python 3’s revised handling of strings (i.e. all strings are unicode, everything else is just a stream of bytes), because in Python 2 you might not notice TypeErrors until dealing with unusual real world string values.

You can hint your IDE via function doc, for example: http://www.pydev.org/manual_adv_type_hints.html, in JavaScript the jsDoc helps in a similar way.
But at some point you will face errors that a typed language would avoid immediately without unit tests (via the IDE compilation and the types/inference).
Of course this does not remove the benefit of unit tests, static analysis and assertions. For larger project I tend to use statically typed languages because they have very good IDE support (excellent autocompletion, heavy refactoring...). You can still use scripting or a DSL for some sub part of the project.

Something you can use to simplify your code is using the Null Object Design Pattern (to which I was introduced in Python Cookbook).
Roughly, the goal with Null objects is to provide an 'intelligent'
replacement for the often used primitive data type None in Python or
Null (or Null pointers) in other languages. These are used for many
purposes including the important case where one member of some group
of otherwise similar elements is special for whatever reason. Most
often this results in conditional statements to distinguish between
ordinary elements and the primitive Null value.
This object just eats the lack of attribute error, and you can avoid checking for their existence.
It's nothing more than
class Null(object):
def __init__(self, *args, **kwargs):
"Ignore parameters."
return None
def __call__(self, *args, **kwargs):
"Ignore method calls."
return self
def __getattr__(self, mname):
"Ignore attribute requests."
return self
def __setattr__(self, name, value):
"Ignore attribute setting."
return self
def __delattr__(self, name):
"Ignore deleting attributes."
return self
def __repr__(self):
"Return a string representation."
return "<Null>"
def __str__(self):
"Convert to a string and return it."
return "Null"
With this, if you do Null("any", "params", "you", "want").attribute_that_doesnt_exists() it won't explode, but just silently become the equivalent of pass.
Normally you'd do something like
if obj.attr:
obj.attr()
With this, you just do:
obj.attr()
and forget about it. Beware that extensive use of the Null object can potentially hide bugs in your code.

I tend to use
if x is None:
raise ValueError('x cannot be None')
But this will only work with the actual None value.
A more general approach is to test for the necessary attributes before you try to use them. For example:
def write_data(f):
# Here we expect f is a file-like object. But what if it's not?
if not hasattr(f, 'write'):
raise ValueError('write_data requires a file-like object')
# Now we can do stuff with f that assumes it is a file-like object
The point of this code is that instead of getting an error message like "NoneType has no attribute write", you get "write_data requires a file-like object". The actual bug isn't in write_data(), and isn't really a problem with NoneType at all. The actual bug is in the code that calls write_data(). The key is to communicate that information as directly as possible.