In a bit of my code I'm using the nice memoized class from the Python Decorator Library.
One of the libraries I'm using uses introspection on a function to get the number of arguments it takes, and fails on the decorated function. Specifically, it checks the co_argcount variable.
if (PyInt_AsLong(co_argcount) < 1) {
PyErr_SetString(PyExc_TypeError, "This function has no parameters to mini\
mize.");
It seems the argcount isn't being transferred to the memoized function.
>>> def f(x):
... return x
...
>>> f.func_code.co_argcount
1
>>> g = memoized(f)
>>> g.func_code.co_argcount
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'memoized' object has no attribute 'func_code'
How can I modify the memoized class so that my memoized functions look, taste, and smell like the original function?
You need to create a signature-preserving decorator. The easiest way to do that is to use the library http://pypi.python.org/pypi/decorator which takes care of preserving the signature for you.
The internals of the library are quite ugly (it uses exec!) but it encapsulates them quite well.
Add that to your memoized class
def __getattr__(self, name):
if name.startswith('func_'):
return getattr(self.func, name)
raise AttributeError
So it'll pass attribute lookup for func_... to the original function.
Maybe you will also want to write a __setattr__ function to deny writing these attributes, but it's not necessary if you know you won't try to change the values.
Related
Let's say you work with a wrapper object:
class IterOrNotIter:
def __init__(self):
self.f = open('/tmp/toto.txt')
def __getattr__(self, item):
try:
return self.__getattribute__(item)
except AttributeError:
return self.f.__getattribute__(item)
This object implements __iter__, because it passes any call to it to its member f, which implements it. Case in point:
>>> x = IterOrNotIter()
>>> x.__iter__().__next__()
'Whatever was in /tmp/toto.txt\n'
According to the documentation (https://docs.python.org/3/library/stdtypes.html#iterator-types), IterOrNotIter should thus be iterable.
However, the Python interpreter does not recognize an IterOrNotIter object as actually being iterable:
>>> x = IterOrNotIter()
>>> for l in x:
... print(l)
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'IterOrNotIter' object is not iterable
Whereas this works:
>>> x = IterOrNotIter()
>>> for l in x.f:
... print(l)
...
Whatever was in /tmp/toto.txt
I don't understand why.
Basically because your class just doesn't have a real __iter__ method:
>>> hasattr(IterOrNotIter, '__iter__')
False
So it doesn't qualify as iterator because the actual check for __iter__ checks for the existence instead of assuming it's implemented. So workarounds with __getattr__ or __getattribute__ (unfortunatly) don't work.
This is actually mentioned in the documentation for __getattribute__:
Note
This method may still be bypassed when looking up special methods as the result of implicit invocation via language syntax or built-in functions. See Special method lookup.
The latter section also explains the why:
Bypassing the __getattribute__() machinery in this fashion provides significant scope for speed optimisations within the interpreter, at the cost of some flexibility in the handling of special methods (the special method must be set on the class object itself in order to be consistently invoked by the interpreter).
Emphasis mine.
I have some functions which, under normal circumstances, are called with arguments provided by user input. It is, however, valid to call some of these functions with certain series of arguments which are determined at runtime based on some system state.
I would like for the user to be able to optionally instruct my program to call these functions with all valid input and return the results of each call. I think a decorator which would work something like an activation switch for functions which have another decorator which indicates which series of arguments to use would work well.
Additionally, I need to preserve the function signature and metadata. It's vital to the operation of my program.
This is what I've tried, but it doesn't work. It is based upon this example.
>>> from decorator import decorator
>>> def the_list():
... return ["foo", "bar", "baz"]
...
>>> import itertools
>>> #decorator
... def do_all(func):
... # this will do nothing (besides wrap in a tuple) unless func is decorated with #gets_arg_from
... if hasattr(func, 'get_from'):
... return tuple(func(*args) for args in itertools.product(*(list_fun() for list_fun in func.get_from)))
... else:
... return (func(),)
...
>>> def gets_arg_from(*list_funcs):
... # this will do nothing to func unless decorated with #do_all
... def gets_arg_from(func, *args, **kwargs):
... func.get_from = list_funcs
... return func(*args, **kwargs)
... return decorator(gets_arg_from)
...
>>> #gets_arg_from(the_list)
... def print_func(word):
... # this will print its argument unless decorated with #do_all
... # at that point it will print every element returned by the_list()
... print word
...
>>> print_func("foo")
foo
>>> all = decorator(do_all, print_func)
>>> all()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: print_func() takes exactly 1 argument (0 given)
>>> print_func.get_from
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'function' object has no attribute 'get_from'
What I expected was:
>>> all()
("foo", "bar", "baz")
What I've noticed is wrong:
gets_arg_from doesn't add the get_from attribute to func.
Something about me using the notation #gets_arg_from(the_list) is wrong. It thinks I am trying to pass two arguments (but why would that be a problem anyway?)
As for my motivation, I think of decorators for this because there are literally hundreds of these routines, their implementation details (as well as their functional correctness) is subject to frequent change, and I don't want to use inspect to reason what to do based on their argument names nor do I want to hard-code the do_all functionality for each function for which it makes sense. Class methods might work, but for my purpose, they're semantically contrived. Furthermore, for the sake of others who may have to maintain my code, I think it is easier to ask them to apply a decorator and not to worry about the rest rather than to use certain argument names or place the function in a certain class or whatever. I realize this question may sound strange, so I figured this footnote might help make me look less like a madman.
Isn't this doing the thing you want?
import functools
from itertools import product
def the_list():
return ["foo", "bar", "baz"]
def do_all(func):
if hasattr(func, 'get_from'):
#functools.wraps(func)
def wrapper(*args, **kwargs):
return tuple(func(*args) for args in
product(*(lf() for lf in func.get_from)))
return wrapper
return func
def gets_arg_from(*list_funcs):
def decorator(func):
func.get_from = list_funcs
return func
return decorator
#gets_arg_from(the_list)
def print_func(word):
return word
print print_func('foo')
all = do_all(print_func)
print all()
Edit: Explanation
These two code segments are identical:
#deco
def func(...):
some code
is the same as
func = deco(lambda ...: some code)
#something is just a syntactic sugar for the function call and anonymous function creation...
I'll explain what happened in the next peace of code step by step:
#gets_arg_from(the_list)
def print_func(word):
return word
First Python creates an anonimous function that receives a parameter word and has a body that just returns this word (or does whatever the function body does)
Then the function get_arg_from gets called and the_list gets passed to it as an argument
get_arg_from creates a decorator function and returnes it
The decorator function returned from the get_arg_from is called (this is the syntactic sugar thing) passing as an argument func the anonimous function created in the step 1.
decorator just assigns the list_funcs tuple to the get_from attribute of the anonymous function and returns the anonimous function
The return value of the decorator function is assigned to a variable print_func
Similar effect can be achieved with:
def __anonimous(word):
return word
__decorator = gets_arg_from(the_list)
print_func = __decorator(__anonimous)
So basically gets_arg_from is not a decorator it's a function that returns a decorator.
do_all on the other hand is a decorator, it receives a function as an argument, and returns either the original function (if the function doesn't have the attribute get_from) or a wrapper function which replaces the original function (if it has the get_from attribute).
You can find more examples here.
I'm trying to get the hang of python 2.6, which is my introduction to a programming language. I'm following Learn Python the Hard Way and I must have missed something. I have a set of functions that I want to combine into a class. These functions are contained here. I tried to combine these into a class called "Parsers". This code is contained here.
The functions themselves work outside of the class. I'm trying to call the functions out of the class like this:
import the module:
>>> from ex48 import parser2
Assign the class:
>>> parser_class = parser2.Parsers()
Call a function:
>>> parser_class.parse_subject(word_list, ('noun', 'player'))
I get the following:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ex48/parser2.py", line 60, in parse_subject
verb = parse_verb(word_list)
NameError: global name 'parse_verb' is not defined
Not sure why it's telling me that. Can someone point me in the right direction?
You are confused as to what your code is doing.
This does not assign, a class, it creates an instance, and assigns that instance to a variable:
parser_class = parser2.Parsers()
This calls a method on that instance:
parser_class.parse_subject(word_list, ('noun', 'player'))
The following tells you that there is no global function (or indeed variable of any type) parse_verb:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "ex48/parser2.py", line 60, in parse_subject
verb = parse_verb(word_list)
NameError: global name 'parse_verb' is not defined
I expect that you want to change that to verb = self.parse_verb(word_list), which will call the parse_verb method of the same object (self).
There are two problems here, one mechanical, one conceptual. I'll deal with the latter first.
The whole point of a class is that its instances represent some kind of object in your model of the world. For example, a "sentence" is an object, so it makes sense to have a class called Sentence, but a "parsers" is not an object, so Parsers is probably going to be a confused class. Looking at things less abstractly, a class usually has some kind of state, encapsulated in data members that its methods work on.
What you really have here is a parser function that returns a Sentence object (or raises a ParserError). This function could be a method of Sentence, or it could be a free function; it could also be a method of a Parser class if there were any reason for such an object to exist in your model, but there doesn't seem to be one.
If you're just looking to encapsulate all those helper functions so they don't litter the global namespace (a good idea in itself), you don't need to abuse a class to do that. Just make them local functions inside parse_sentence, or put them in a module called parsers.
Now, on to the mechanical problem: If you have a method in a class, you normally only call it through dot syntax: someObject.methodName(params). When a method calls another method of the same object, you use the special self parameter (which you've correctly listed as the first param of each method) to do that. So:
verb = self.parse_verb(word_list)
obj = self.parse_object(word_list)
… and so on for all the other method calls in your sample.
Not every language requires an explicit self like this. For example, in C++ or related languages (Java, C#, etc.), a method can call another method without specifying a target, and it's implicitly assumed that the target is this (the C++ equivalent of self).
It looks like you are working from a python interactive session.
If you've made any changes to parser2.py after importing parser2, then you have to
reload(parser2)
to make those changes known to the interactive interpreter.
I know a ton has been written on this subject. I cannot, however, absorb much of it. Perhaps because I'm a complete novice teaching myself without the benefit of any training in computer science. Regardless, maybe if some of you big brains chime in on this specific example, you'll help other beginners like me.
So, I've written the following function which works just fine when I call it (as a module?) as it's own file called 'funky.py':
I type the following into my terminal:
python classy.py
and it runs fine.
def load_deck():
suite = ('Spades', 'Hearts')
rank = ('2', '3')
full_deck = {}
i = 0
for s in suite:
for r in rank:
full_deck[i] = "%s of %s" % (r, s)
i += 1
return full_deck
print load_deck()
When I put the same function in a class, however, I get an error.
Here's my code for 'classy.py':
class GAME():
def load_deck():
suite = ('Spades', 'Hearts')
rank = ('2', '3')
full_deck = {}
i = 0
for s in suite:
for r in rank:
full_deck[i] = "%s of %s" % (r, s)
i += 1
return full_deck
MyGame = GAME()
print MyGame.load_deck()
I get the following error:
Traceback (most recent call last):
File "classy.py", line 15, in <module>
print MyGame.load_deck()
TypeError: load_deck() takes no arguments (1 given)
So, I changed the definition line to the following and it works fine:
def load_deck(self):
What is it about putting a function in a class that demands the use of 'self'. I understand that 'self' is just a convention. So, why is any argument needed at all? Do functions behave differently when they are called from within a class?
Also, and this is almost more important, why does my class work without the benefit of using init ? What would using init do for my class?
Basically, if someone has the time to explain this to me like i'm a 6 year-old, it would help. Thanks in advance for any help.
Defining a function in a class definition invokes some magic that turns it into a method descriptor. When you access foo.method it will automatically create a bound method and pass the object instance as the first parameter. You can avoid this by using the #staticmethod decorator.
__init__ is simply a method called when your class is created to do optional setup. __new__ is what actually creates the object.
Here are some examples
>>> class Foo(object):
def bar(*args, **kwargs):
print args, kwargs
>>> foo = Foo()
>>> foo.bar
<bound method Foo.bar of <__main__.Foo object at 0x01C9FEB0>>
>>> Foo.bar
<unbound method Foo.bar>
>>> foo.bar()
(<__main__.Foo object at 0x01C9FEB0>,) {}
>>> Foo.bar()
Traceback (most recent call last):
File "<pyshell#29>", line 1, in <module>
Foo.bar()
TypeError: unbound method bar() must be called with Foo instance as first argument (got nothing instead)
>>> Foo.bar(foo)
(<__main__.Foo object at 0x01C9FEB0>,) {}
So, why is any argument needed at all?
To access attributes on the current instance of the class.
Say you have a class with two methods, load_deck and shuffle. At the end of load_deck you want to shuffle the deck (by calling the shuffle method)
In Python you'd do something like this:
class Game(object):
def shuffle(self, deck):
return random.shuffle(deck)
def load_deck(self):
# ...
return self.shuffle(full_deck)
Compare this to the roughly-equivalent C++ code:
class Game {
shuffle(deck) {
return random.shuffle(deck);
}
load_deck() {
// ...
return shuffle(full_deck)
}
}
On shuffle(full_deck) line, first it looks for a local variable called shuffle - this doesn't exist, to next it checks one level higher, and finds an instance-method called shuffle (if this doesn't exist, it would check for a global variable with the right name)
This is okay, but it's not clear if shuffle refers to some local variable, or the instance method. To address this ambiguity, instance-methods or instance-attributes can also be accessed via this:
...
load_deck() {
// ...
return this->shuffle(full_deck)
}
this is almost identical to Python's self, except it's not passed as an argument.
Why is it useful to have self as an argument useful? The FAQ lists several good reasons - these can be summarised by a line in "The Zen of Python":
Explicit is better than implicit.
This is backed up by a post in The History of Python blog,
I decided to give up on the idea of implicit references to instance variables. Languages like C++ let you write this->foo to explicitly reference the instance variable foo (in case there’s a separate local variable foo). Thus, I decided to make such explicit references the only way to reference instance variables. In addition, I decided that rather than making the current object ("this") a special keyword, I would simply make "this" (or its equivalent) the first named argument to a method. Instance variables would just always be referenced as attributes of that argument.
With explicit references, there is no need to have a special syntax for method definitions nor do you have to worry about complicated semantics concerning variable lookup. Instead, one simply defines a function whose first argument corresponds to the instance, which by convention is named "self."
If you don't intent to use self you should probably declare the method to be a staticmethod.
class Game:
#staticmethod
def load_deck():
....
This undoes the automatic default packing that ordinarily happens to turn a function in a class scope into a method taking the instance as an argument.
Passing arguments you don't use is disconcerting to others trying to read your code.
Most classes have members. Yours doesn't, so all of its methods should be static. As your project develops, you will probably find data that should be accessible to all of the functions in it, and you will put those in self, and pass it around to all of them.
In this context, where the application itself is your primary object, __init__ is just the function that would initialize all of those shared values.
This is the first step toward an object-oriented style, wherein smaller pieces of data get used as objects themselves. But this is a normal stage in moving from straight scripting to OO programming.
As I understand it, I can use the for loop construction on an object with a __iter__ method that returns an iterator. I have an object for which I implement the following __getattribute__ method:
def __getattribute__(self,name):
if name in ["read","readlines","readline","seek","__iter__","closed","fileno","flush","mode","tell","truncate","write","writelines","xreadlines"]:
return getattr(self.file,name)
return object.__getattribute__(self,name)
I have an object of this class, a for which the following happens:
>>> hasattr(a,"__iter__")
True
>>> for l in a: print l
...
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'TmpFile' object is not iterable
>>> for l in a.file: print l
...
>>>
So python sees that a has an __iter__ method, but doesn't think it is iterable. What have I done wrong? This is with python 2.6.4.
There's a subtle implementation detail getting in your way: __iter__ isn't actually an instance method, but a class method. That is, obj.__class__.__iter__(obj) is called, rather than obj.__iter__().
This is due to slots optimizations under the hood, allowing the Python runtime to set up iterators faster. This is needed since it's very important that iterators be as fast as possible.
It's not possible to define __getattribute__ for the underlying class type, so it's not possible to return this method dynamically. This applies to most __metamethods__; you'll need to write an actual wrapper.
Some of the special methods are optimised when a class is created and cannot be added later or overridden by assignment. See the documentation for __getattribute__ which says:
This method may still be bypassed when
looking up special methods as the
result of implicit invocation via
language syntax or built-in functions.
What you need to do in this case is provide a direct implementation of __iter__ that forwards the call:
def __iter__(self):
return self.file.__iter__()