We Keep Coding

Python decorator crashing even though its code is solid, can't understand why

So I wrote a function called 'anonimize' that gets a string and returns that string with some details blacked-out, using regex.
The function works fine on its own, but I tried to turn it into a decorator and things went south and got so many crashes so many times.
I am learning decorators and would really like to understand what I am doing wrong.
In this unsuccessful version I got "TypeError: BankApplication.anonimize() missing 1 required positional argument: 'callable'".
import re
class BankApplication:
def __init__(self, bank_name):
self.name = bank_name
'''
Perform anonimization to the text data:
1. Remove possible account numbers (more than 5 digits in a row)
2. Remove possible email addresses
3. Remove possible First + Last names (two consequtive words
starting from upper case, but not divided by .)
'''
def anonimize(func):
def inner(arg):
ret = func(arg)
print(
re.sub("[A-Z][a-z]+\s[A-Z][a-z]+", "***", re.sub("\S+#\S+\.\S+", "***", re.sub("\d{5,}", "***", arg))))
return ret
return inner
#anonimize
def feedback(self, feedback_text):
print("Called feedback")
#anonimize
def log_info(self, info_to_log):
print("Called feedback")
bank = BankApplication("Bank")
bank.feedback("Name: John Doe\nE-mail: johndoe#fakemail.com\nAccount number: 911911911.")

You need to move anonimize out of the class. Why?
The first argument that gets passed to a standard method is always the owning object itself (the self argument is the standard syntax, but you can call it anything).
You cannot use it as a static method because decorators are run when the code is defined. You won't be able to reference the owning object because it won't have been instantiated yet.
import re
def anonimize(func):
def inner(arg):
ret = func(arg)
print(
re.sub("[A-Z][a-z]+\s[A-Z][a-z]+", "***", re.sub("\S+#\S+\.\S+", "***", re.sub("\d{5,}", "***", arg))))
return ret
return inner
class BankApplication:
def __init__(self, bank_name):
self.name = bank_name
'''
Perform anonimization to the text data:
1. Remove possible account numbers (more than 5 digits in a row)
2. Remove possible email addresses
3. Remove possible First + Last names (two consequtive words
starting from upper case, but not divided by .)
'''
...

anonimize shouldn't be an instance method of BankApplication, it doesn't even accept self. Define it outside of the class.
EDIT: I've also suggested #staticmethod, but it wouldn't work either, see #James's answer.

Ok, so this was the wrong version I posted:
def anonimize(func):
def inner(text):
print(re.sub("[A-Z][a-z]+\s[A-Z][a-z]+", "***", re.sub("\S+#\S+\.\S+", "***", re.sub("\d{5,}", "***", text))))
return func(text)
return inner
And this is the way it should have been:
def anonimize(func):
def inner(self, text):
text = re.sub("[A-Z][a-z]+\s[A-Z][a-z]+", "***", re.sub("\S+#\S+\.\S+", "***", re.sub("\d{5,}", "***", text)))
return func(self, text)
return inner

How to print a string static member of a class with `yield` from a class method

I am really new to python, so this might be really easy.
I want to print two strings defined in a class as static members with a class method that yields each string.
This is a simplified version of what I am trying to do:
#!/usr/bin/python
import sys
class test:
str1 = "Hello"
str2 = "World\n" #"\n" is needed for the example
def printMe(self):
yield test.str1
yield test.str2
hello = test()
print "Testing initiated:"
sys.stdout.write(hello.printMe())
sys.stdout.write(hello.printMe())
This is the output:
sys.stdout.write(hello.printMe()) TypeError: expected a character
buffer object

You are attempting to use a generator function, read about the yield keyword here
import sys
class Test:
def __init__(self): # it's possible to initialise these attributes in the __init__ method, so they are created on class instantiation(when you did hello = Test())
self.str1 = "Hello"
self.str2 = "World\n" #"\n" is needed for the example
def printMe(self):
for i in [self.str1, self.str2]:
yield i
app = Test()
print "Testing initiated:"
for i in app.printMe():
print i # is there a reason why you can't use print?
If however you want to print the lines one at a time, at specific points in the code, like in your loop you mentioned in the comment:
gen = app.printMe()
then every time you want to print:
gen.next()
this triggers the next yield statement. The generator function effectively 'holds'/remembers it's place until you call next again, until all the yield statements have been yielded.

You should do something like this
for line in hello.printMe():
print line
But really there are a lot of easier ways than using yield statements.

using yield turns your function into a generator. If this is really what you want, you will need to iterate over the generator to get the values:
gen = hello.printMe()
sys.stdout.write(gen.next())
sys.stdout.write(gen.next())
or better:
for prop in hello.printMe():
sys.stdout.write(prop)

Your printMe method is a generator function, which returns an iterable. You need to iterate over it to get the results :
for item in hello.printMe():
print item

You can do this, but I'm using print, hope this helps you:
class test:
str1 = "Hello"
str2 = "World\n" #"\n" is needed for the example
def printMe(self):
yield test.str1
yield test.str2
hello = test()
print "Testing initiated:"
out = hello.printMe()
print(out.next(),end=' ')
print(out.next(),end=' ')

python function that changes itself to list

So I'm working on a chemistry project for fun, and I have a function that initializes a list from a text file. What I want to do s make it so the function replaces itself with a list. So here's my first attempt at it which randomly will or won't work and I don't know why:
def periodicTable():
global periodicTable
tableAtoms = open('/Users/username/Dropbox/Python/Chem Project/atoms.csv','r')
listAtoms = tableAtoms.readlines()
tableAtoms.close()
del listAtoms[0]
atoms = []
for atom in listAtoms:
atom = atom.split(',')
atoms.append(Atom(*atom))
periodicTable = atoms
It gets called in in this way:
def findAtomBySymbol(symbol):
try:
periodicTable()
except:
pass
for atom in periodicTable:
if atom.symbol == symbol:
return atom
return None
Is there a way to make this work?

Don't do that. The correct thing to do would be using a decorator that ensures the function is only executed once and caches the return value:
def cachedfunction(f):
cache = []
def deco(*args, **kwargs):
if cache:
return cache[0]
result = f(*args, **kwargs)
cache.append(result)
return result
return deco
#cachedfunction
def periodicTable():
#etc
That said, there's nothing stopping you from replacing the function itself after it has been called, so your approach should generally work. I think the reason it doesn't is because an exception is thrown before you assign the result to periodicTable and thus it never gets replaced. Try removing the try/except block or replacing the blanket except with except TypeError to see what exactly happens.

This is very bad practice.
What would be better is to have your function remember if it has already loaded the table:
def periodicTable(_table=[]):
if _table:
return _table
tableAtoms = open('/Users/username/Dropbox/Python/Chem Project/atoms.csv','r')
listAtoms = tableAtoms.readlines()
tableAtoms.close()
del listAtoms[0]
atoms = []
for atom in listAtoms:
atom = atom.split(',')
atoms.append(Atom(*atom))
_table[:] = atoms
The first two lines check to see if the table has already been loaded, and if it has it simply returns it.

Can't iterate over a list class in Python

I'm trying to write a simple GUI front end for Plurk using pyplurk.
I have successfully got it to create the API connection, log in, and retrieve and display a list of friends. Now I'm trying to retrieve and display a list of Plurks.
pyplurk provides a GetNewPlurks function as follows:
def GetNewPlurks(self, since):
'''Get new plurks since the specified time.
Args:
since: [datetime.datetime] the timestamp criterion.
Returns:
A PlurkPostList object or None.
'''
offset = jsonizer.conv_datetime(since)
status_code, result = self._CallAPI('/Polling/getPlurks', offset=offset)
return None if status_code != 200 else \
PlurkPostList(result['plurks'], result['plurk_users'].values())
As you can see this returns a PlurkPostList, which in turn is defined as follows:
class PlurkPostList:
'''A list of plurks and the set of users that posted them.'''
def __init__(self, plurk_json_list, user_json_list=[]):
self._plurks = [PlurkPost(p) for p in plurk_json_list]
self._users = [PlurkUser(u) for u in user_json_list]
def __iter__(self):
return self._plurks
def GetUsers(self):
return self._users
def __eq__(self, other):
if other.__class__ != PlurkPostList: return False
if self._plurks != other._plurks: return False
if self._users != other._users: return False
return True
Now I expected to be able to do something like this:
api = plurk_api_urllib2.PlurkAPI(open('api.key').read().strip(), debug_level=1)
plurkproxy = PlurkProxy(api, json.loads)
user = plurkproxy.Login('my_user', 'my_pass')
ps = plurkproxy.GetNewPlurks(datetime.datetime(2009, 12, 12, 0, 0, 0))
print ps
for p in ps:
print str(p)
When I run this, what I actually get is:
<plurk.PlurkPostList instance at 0x01E8D738>
from the "print ps", then:
for p in ps:
TypeError: __iter__ returned non-iterator of type 'list'
I don't understand - surely a list is iterable? Where am I going wrong - how do I access the Plurks in the PlurkPostList?

When you define your own __iter__ method, you should realize that that __iter__ method should return an iterator, not an iterable. You are returning a list, not an iterator to a list, so it fails. You can fix it by doing return iter(self._plurks), for example.
If you wanted to do something a little more complex, like process each item in self._plurks as it's being iterated over, the usual trick is to make your __iter__ method be a generator. That way, the returnvalue of the call to __iter__ is the generator, which is an iterator:
def __iter__(self):
for item in self._plurks:
yield process(item)

The __iter__ method should return an object which implements the next() method.
A list does not have a next() method, but it has an __iter__ method, which returns a listiterator object. The listiterator object has a next() method.
You should write:
def __iter__(self):
return iter(self._plurks)

As an alternative, you can also define the next() function and have __iter__() return self. See Build a Basic Python Iterator for a nice example.

Real-world examples of nested functions

I asked previously how the nested functions work, but unfortunately I still don't quite get it. To understand it better, can someone please show some real-wold, practical usage examples of nested functions?
Many thanks

Your question made me curious, so I looked in some real-world code: the Python standard library. I found 67 examples of nested functions. Here are a few, with explanations.
One very simple reason to use a nested function is simply that the function you're defining doesn't need to be global, because only the enclosing function uses it. A typical example from Python's quopri.py standard library module:
def encode(input, output, quotetabs, header = 0):
...
def write(s, output=output, lineEnd='\n'):
# RFC 1521 requires that the line ending in a space or tab must have
# that trailing character encoded.
if s and s[-1:] in ' \t':
output.write(s[:-1] + quote(s[-1]) + lineEnd)
elif s == '.':
output.write(quote(s) + lineEnd)
else:
output.write(s + lineEnd)
... # 35 more lines of code that call write in several places
Here there was some common code within the encode function, so the author simply factored it out into a write function.
Another common use for nested functions is re.sub. Here's some code from the json/encode.py standard library module:
def encode_basestring(s):
"""Return a JSON representation of a Python string
"""
def replace(match):
return ESCAPE_DCT[match.group(0)]
return '"' + ESCAPE.sub(replace, s) + '"'
Here ESCAPE is a regular expression, and ESCAPE.sub(replace, s) finds all matches of ESCAPE in s and replaces each one with replace(match).
In fact, any API, like re.sub, that accepts a function as a parameter can lead to situations where nested functions are convenient. For example, in turtle.py there's some silly demo code that does this:
def baba(xdummy, ydummy):
clearscreen()
bye()
...
tri.write(" Click me!", font = ("Courier", 12, "bold") )
tri.onclick(baba, 1)
onclick expects you to pass an event-handler function, so we define one and pass it in.

Decorators are a very popular use for nested functions. Here's an example of a decorator that prints a statement before and after any call to the decorated function.
def entry_exit(f):
def new_f(*args, **kwargs):
print "Entering", f.__name__
f(*args, **kwargs)
print "Exited", f.__name__
return new_f
#entry_exit
def func1():
print "inside func1()"
#entry_exit
def func2():
print "inside func2()"
func1()
func2()
print func1.__name__

Nested functions avoid cluttering other parts of the program with other functions and variables that only make sense locally.
A function that return Fibonacci numbers could be defined as follows:
>>> def fib(n):
def rec():
return fib(n-1) + fib(n-2)
if n == 0:
return 0
elif n == 1:
return 1
else:
return rec()
>>> map(fib, range(10))
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]
EDIT: In practice, generators would be a better solution for this, but the example shows how to take advantage of nested functions.

They are useful when using functions that take other functions as input. Say you're in a function, and want to sort a list of items based on the items' value in a dict:
def f(items):
vals = {}
for i in items: vals[i] = random.randint(0,100)
def key(i): return vals[i]
items.sort(key=key)
You can just define key right there and have it use vals, a local variable.
Another use-case is callbacks.

I have only had to use nested functions when creating decorators. A nested function is basically a way of adding some behavior to a function without knowing what the function is that you are adding behavior to.
from functools import wraps
from types import InstanceType
def printCall(func):
def getArgKwargStrings(*args, **kwargs):
argsString = "".join(["%s, " % (arg) for arg in args])
kwargsString = "".join(["%s=%s, " % (key, value) for key, value in kwargs.items()])
if not len(kwargs):
if len(argsString):
argsString = argsString[:-2]
else:
kwargsString = kwargsString[:-2]
return argsString, kwargsString
#wraps(func)
def wrapper(*args, **kwargs):
ret = None
if args and isinstance(args[0], InstanceType) and getattr(args[0], func.__name__, None):
instance, args = args[0], args[1:]
argsString, kwargsString = getArgKwargStrings(*args, **kwargs)
ret = func(instance, *args, **kwargs)
print "Called %s.%s(%s%s)" % (instance.__class__.__name__, func.__name__, argsString, kwargsString)
print "Returned %s" % str(ret)
else:
argsString, kwargsString = getArgKwargStrings(*args, **kwargs)
ret = func(*args, **kwargs)
print "Called %s(%s%s)" % (func.__name__, argsString, kwargsString)
print "Returned %s" % str(ret)
return ret
return wrapper
def sayHello(name):
print "Hello, my name is %s" % (name)
if __name__ == "__main__":
sayHelloAndPrintDebug = printCall(sayHello)
name = "Nimbuz"
sayHelloAndPrintDebug(name)
Ignore all the mumbo jumbo in the "printCall" function for right now and focus only the "sayHello" function and below. What we're doing here is we want to print out how the "sayHello" function was called everytime it is called without knowing or altering what the "sayHello" function does. So we redefine the "sayHello" function by passing it to "printCall", which returns a NEW function that does what the "sayHello" function does AND prints how the "sayHello" function was called. This is the concept of decorators.
Putting "#printCall" above the sayHello definition accomplishes the same thing:
#printCall
def sayHello(name):
print "Hello, my name is %s" % (name)
if __name__ == "__main__":
name = "Nimbuz"
sayHello(name)

Yet another (very simple) example. A function that returns another function. Note how the inner function (that is returned) can use variables from the outer function's scope.
def create_adder(x):
def _adder(y):
return x + y
return _adder
add2 = create_adder(2)
add100 = create_adder(100)
>>> add2(50)
52
>>> add100(50)
150

Python Decorators
This is actually another topic to learn, but if you look at the stuff on 'Using Functions as Decorators', you'll see some examples of nested functions.

OK, besides decorators: Say you had an application where you needed to sort a list of strings based on substrings which varied from time to time. Now the sorted functions takes a key= argument which is a function of one argument: the items (strings in this case) to be sorted. So how to tell this function which substrings to sort on? A closure or nested function, is perfect for this:
def sort_key_factory(start, stop):
def sort_key(string):
return string[start: stop]
return sort_key
Simple eh? You can expand on this by encapsulating start and stop in a tuple or a slice object and then passing a sequence or iterable of these to the sort_key_factory.