We Keep Coding

How to implement a data-parallelising (multiprocessing.Pool) decorator in python?

Problem description
TL;DR
How to implement decorators not breaking the pickle process?
Goal
multiprocessing.Pool can be used to chunk data and distribute it to processes for data-parallelising a given function. I'd like to use such an approach within a decorator for an user-friendly data-parallelisation. The decorator would typically look like the following:
from multiprocessing import Pool
from functools import partial, wraps
def deco_data_parallel(func):
#wraps(func)
def to_parallel(arg, **kwargs):
part_func = partial(func, **kwargs)
tot = 0
with Pool() as p:
for output in p.imap_unordered(part_func, arg):
tot += output
return tot
return to_parallel
The above implementation imposes the following conditions on the function to be parallelized. These limitations can very likely be overcome with a better design.
arg is an iterable to be split into chunks
The fix arguments must be called as keyword arguments
Here is an example of the intended use:
#deco_data_parallel
def compute(data, arg1, arg2):
return data + arg1 + arg2
if __name__ == "__main__":
# Dummy data
data = [4]*100000
# Fix arguments must be used as keyword arguments
compute(data, arg1=1, arg2=2)
Error
The function fed to imap_unordered must be pickable.
The decorator seems to break the pickability of the original function:
_pickle.PicklingError: Can't pickle <function compute at 0x1040137a0>: it's not the same object as __main__.compute
Best solution
I first thought that #wraps was the problem: if the decorated function is identical to the original function, then the latter can't be found by the pools. But it turns out that the #wraps decorator doesn't have any effect.
Thanks to that great post, I could come up with the following non-optimal solution: create a new top-level function object by using the decorator explicitly as follows. It partially breaks the user-friendliness, and is therefore not satisfying. It nevertheless fulfils the expected purpose.
# Beware: the names should not be the same
compute_ = deco_data_parallel(compute)
if __name__ == "__main__":
...
compute_(data, arg1=1, arg2=2)
Questions
How to solve the pickability problem in an elegant way so that the user can simply decorate the function to be parallelised?
Why doesn't the #functools.wraps decorator have any effect?
What does the error it's not the same object as __main__.compute actually mean? I.e. in what sense exactly am I breaking the pickling process?
My configuration
Macport's Python 3.7.7 on OSX 10.14.6
Disclaimer
I'm quite new to the world of parallel computing in python, as well as in the world of python decorators. Heresy is very likely to have happened in this post and I apologise for that!
That's also my second question on StackOverflow, any proposition of improvement is welcome.
Detailed investigation
Convinced to make it work, I tried multiple decorating strategies. This very complete post on the decorators matter was a great guide. This blog post gave me some hope that an object-oriented decorating strategy could make things work: the author indeed claims that it fixed his/her pickability problem.
All of the following approaches have been tested, with and without #wraps, and they all lead to the same _pickle.PicklingError error. I begin to have the feeling that I tried all the non-hacky possibilities python has to offer, and that would be a great pleasure to be proven wrong!
Functional approach
The most simple approach is the one I showed above. For decorators with arguments, a "decorator factory" can be used as well. Let's use the number of processors here for the sake of the example.
def factory_data_parallel(nproc=4):
def deco_data_parallel(func):
#wraps(func)
def to_parallel(arg, **kwargs):
part_func = partial(func, **kwargs)
tot = 0
with Pool(nproc) as p:
for output in p.imap_unordered(part_func, arg):
tot += output
return tot
return to_parallel
return deco_data_parallel
# Usage: only with the argument (or parenthesis at least)
#factory_data_parallel(8)
def compute(data, arg1, arg2):
...
A hybrid form which can be used as a simple decorator or a decorator factory would be implemented as follows:
def factorydeco_data_parallel(_func=None, *, nproc=4):
def deco_data_parallel(func):
...
if _func is None:
return deco_data_parallel
else:
return deco_data_parallel(_func)
# Usage as a factory (with argument)
#factorydeco_data_parallel(8)
def compute(data, arg1, arg2):
...
# Usage as a simple decorator
#factorydeco_data_parallel
def other_compute(data, arg1, arg2):
...
Object-oriented approach
From my understanding, a decorator can be any callable object. A simple decorator using an object can be implemented as follow. The first version is called with parenthesis (explicit creation of the object when decorating), and the second one is used as a standard decorator.
class Class_data_parallel(object):
def __call__(self, func):
self.orig_func = func
#wraps(func)
def to_parallel(arg, **kwargs):
# Does it make a difference to use the argument func instead?
part_func = partial(self.orig_func, **kwargs)
tot = 0
with Pool() as p:
for output in p.imap_unordered(part_func, arg):
tot += output
return tot
return to_parallel
class Class_data_parallel_alt(object):
def __init__(self, func):
self.orig_func = func
# PB: no way I'm aware of to use #wraps
def __call__(self, arg, **kwargs):
part_func = partial(self.orig_func, **kwargs)
tot = 0
with Pool() as p:
for output in p.imap_unordered(part_func, arg):
tot += output
return tot
# Usage: with parenthesis
#Class_data_parallel()
def compute(data, arg1, arg2):
...
# Usage: without parenthesis
#Class_data_parallel_alt
def other_compute(data, arg1, arg2):
...
An obvious extension of the first case could enable to add some parameters to the constructor. The class would then play the role of a decorator factory.
Some more thinking
As I mentioned, #wraps was candidate for both being the cause and the solution to the problem. Using it or not doesn't change anything
The use of parallel for handling the constant arguments (i.e. constant across processes, arg1 and arg2 in my examples) could be a problem, but I doubt it. I could use the initializer argument of the Pool() constructor.
A. Sherman and P. Den Hartog did achieve that goal in their DECO parallel model. I'm however not able to understand how they overcame my problem. It seems to prove that what I want to do is not a fundamental limitation of decorators.

You're trying to have the workers execute partial(func, **kwargs), where func is the undecorated function. For this to work, the workers have to be able to find func by module and qualified name, but it's not where the name suggests it should be. The to_parallel wrapper is there instead. This gets detected when the master process tries to pickle func.
I wouldn't use a decorator for this. The options I see for a decorator all interfere with something else, like documentation generation or decorator composability.

Prevent other classes' methods from calling my constructor

How do I make a python "constructor" "private", so that the objects of its class can only be created by calling static methods? I know there are no C++/Java like private methods in Python, but I'm looking for another way to prevent others from calling my constructor (or other method).
I have something like:
class Response(object):
#staticmethod
def from_xml(source):
ret = Response()
# parse xml into ret
return ret
#staticmethod
def from_json(source):
# parse json
pass
and would like the following behavior:
r = Response() # should fail
r = Response.from_json(source) # should be allowed
The reason for using static methods is that I always forget what arguments my constructors take - say JSON or an already parsed object. Even then, I sometimes forget about the static methods and call the constructor directly (not to mention other people using my code). Documenting this contract won't help with my forgetfulness. I'd rather enforce it with an assertion.
And contrary to some of the commenters, I don't think this is unpythonic - "explicit is better than implicit", and "there should be only one way to do it".
How can I get a gentle reminder when I'm doing it wrong? I'd prefer a solution where I don't have to change the static methods, just a decorator or a single line drop-in for the constructor would be great. A la:
class Response(object):
def __init__(self):
assert not called_from_outside()

I think this is what you're looking for - but it's kind of unpythonic as far as I'm concerned.
class Foo(object):
def __init__(self):
raise NotImplementedError()
def __new__(cls):
bare_instance = object.__new__(cls)
# you may want to have some common initialisation code here
return bare_instance
#classmethod
def from_whatever(cls, arg):
instance = cls.__new__(cls)
instance.arg = arg
return instance
Given your example (from_json and from_xml), I assume you're retrieving attribute values from either a json or xml source. In this case, the pythonic solution would be to have a normal initializer and call it from your alternate constructors, i.e.:
class Foo(object):
def __init__(self, arg):
self.arg = arg
#classmethod
def from_json(cls, source):
arg = get_arg_value_from_json_source(source)
return cls(arg)
#classmethod
def from_xml(cls, source):
arg = get_arg_value_from_xml_source(source)
return cls(arg)
Oh and yes, about the first example: it will prevent your class from being instantiated in the usual way (calling the class), but the client code will still be able to call on Foo.__new__(Foo), so it's really a waste of time. Also it will make unit testing harder if you cannot instantiate your class in the most ordinary way... and quite a few of us will hate you for this.

I'd recommend turning the factory methods into module-level factory functions, then hiding the class itself from users of your module.
def one_constructor(source):
return _Response(...)
def another_constructor(source):
return _Response(...)
class _Response(object):
...
You can see this approach used in modules like re, where match objects are only constructed through functions like match and search, and the documentation doesn't actually name the match object type. (At least, the 3.4 documentation doesn't. The 2.7 documentation incorrectly refers to re.MatchObject, which doesn't exist.) The match object type also resists direct construction:
>>> type(re.match('',''))()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot create '_sre.SRE_Match' instances
but unfortunately, the way it does so relies upon the C API, so it's not available to ordinary Python code.

Good discussion in the comments.
For the minimal use case you describe,
class Response(object):
def __init__(self, construct_info = None):
if construct_info is None: raise ValueError, "must create instance using from_xml or from_json"
# etc
#staticmethod
def from_xml(source):
info = {} # parse info into here
return Response(info)
#staticmethod
def from_json(source):
info = {} # parse info into here
return Response(info)
It can be gotten around by a user who passes in a hand-constructed info, but at that point they'll have to read the code anyway and the static method will provide the path of least resistance. You can't stop them, but you can gently discourage them. It's Python, after all.

This might be achievable through metaclasses, but is heavily discouraged in Python. Python is not Java. There is no first-class notion of public vs private in Python; the idea is that users of the language are "consenting adults" and can use methods however they like. Generally, functions that are intended to be "private" (as in not part of the API) are denoted by a single leading underscore; however, this is mostly just convention and there's nothing stopping a user from using these functions.
In your case, the Pythonic thing to do would be to default the constructor to one of the available from_foo methods, or even to create a "smart constructor" that can find the appropriate parser for most cases. Or, add an optional keyword arg to the __init__ method that determines which parser to use.

An alternative API (and one I've seen far more in Python APIs) if you want to keep it explicit for the user would be to use keyword arguments:
class Foo(object):
def __init__(self, *, xml_source=None, json_source=None):
if xml_source and json_source:
raise ValueError("Only one source can be given.")
elif xml_source:
from_xml(xml_source)
elif json_source:
from_json(json_source)
else:
raise ValueError("One source must be given.")
Here using 3.x's * to signify keyword-only arguments, which helps enforce the explicit API. In 2.x this is recreatable with kwargs.
Naturally, this doesn't scale well to lots of arguments or options, but there are definitely cases where this style makes sense. (I'd argue bruno desthuilliers probably has it right for this case, from what we know, but I'll leave this here as an option for others).

The following is similar to what I ended up doing. It is a bit more general then what was asked in the question.
I made a function called guard_call, that checks if the current method is being called from a method of a certain class.
This has multiple uses. For example, I used the Command Pattern to implement undo and redo, and used this to ensure that my objects were only ever modified by command objects, and not random other code (which would make undo impossible).
In this concrete case, I place a guard in the constructor ensuring only Response methods can call it:
class Response(object):
def __init__(self):
guard_call([Response])
pass
#staticmethod
def from_xml(source):
ret = Response()
# parse xml into ret
return ret
For this specific case, you could probably make this a decorator and remove the argument, but I didn't do that here.
Here is the rest of the code. It's been a long time since I tested it, and can't guarentee that it works in all edge cases, so beware. It is also still Python 2. Another caveat is that it is slow, because it uses inspect. So don't use it in tight loops and when speed is an issue, but it might be useful when correctness is more important than speed.
Some day I might clean this up and release it as a library - I have a couple more of these functions, including one that asserts you are running on a particular thread. You may snear at the hackishness (it is hacky), but I did find this technique useful to smoke out some hard to find bugs, and to ensure my code still behaves during refactorings, for example.
from __future__ import print_function
import inspect
# http://stackoverflow.com/a/2220759/143091
def get_class_from_frame(fr):
args, _, _, value_dict = inspect.getargvalues(fr)
# we check the first parameter for the frame function is
# named 'self'
if len(args) and args[0] == 'self':
# in that case, 'self' will be referenced in value_dict
instance = value_dict.get('self', None)
if instance:
# return its class
return getattr(instance, '__class__', None)
# return None otherwise
return None
def guard_call(allowed_classes, level=1):
stack_info = inspect.stack()[level + 1]
frame = stack_info[0]
method = stack_info[3]
calling_class = get_class_from_frame(frame)
# print ("calling class:", calling_class)
if calling_class:
for klass in allowed_classes:
if issubclass(calling_class, klass):
return
allowed_str = ", ".join(klass.__name__ for klass in allowed_classes)
filename = stack_info[1]
line = stack_info[2]
stack_info_2 = inspect.stack()[level]
protected_method = stack_info_2[3]
protected_frame = stack_info_2[0]
protected_class = get_class_from_frame(protected_frame)
if calling_class:
origin = "%s:%s" % (calling_class.__name__, method)
else:
origin = method
print ()
print ("In %s, line %d:" % (filename, line))
print ("Warning, call to %s:%s was not made from %s, but from %s!" %
(protected_class.__name__, protected_method, allowed_str, origin))
assert False
r = Response() # should fail
r = Response.from_json("...") # should be allowed

validating argument types for different functions using a single interface (without using inspect)

I wish to check the types of the arguments for each function in the module (without making use of inspect module).
The easiest solution I have done myself is to implement the check in each function separately.
def func1( num1, num2 ): # the two params must be integers
if isinstance( num1, int ) and isinstance( num2, int ):
# ...function code...
pass
else:
return
def func2( float1 ): # The only param must be float
if isinstance( float1, float ):
# ...function code...
pass
else:
return
def func3( str1 ): # string
if isinstance( str1, str ):
# ...function code...
print 'fdfd'
pass
else:
return
# and, so on ...
But, want to do it at the module level, rather than doing it for each function. Each function can have different arguments. Please note that this is not function overloading. I was thinking of either writing a decorator or a metaclass. Here are the problems that I faced in both the approaches:-
Using a common decorator for all the functions:
In this method, I am not able to access the actual variables defined inside each function, so scrapped this idea. Here is a closure (to be used as decorator) that I was planning to write :
def dec( funcName ):
def typeChecker():
i = __import__( __name__ )
for m in map( lambda x: i.__getattribute__( x ), dir( i ) ):
if '__call__' in dir( m ): #Identifying a real function or callable which can be treated as function
## Now, that the function is identified, what should I do here to type-check the arguments??
return typeChecker
Please provide some insights here as to how I can make this work.
2.Using a metaclass for creating function
I was just wondering if I could access the arguments being sent to a function using a metaclass and then, validate each of the argument, and then, return a brand new class which would be responsible for creating the function object. But, do not know how to do it. Is it a good way of solving this problem?
1 very good suggestion Martijn Peters has given - To use annotations. Is there something in Python 2.7 that we can use?

Doing this through a decorator is fairly easy -
In Python 2 you'd have to decorate each function explicitly to annotate the type for
each parameter - or you could use some annotation using some markup on the doc-string,
and place a call on the bottom of the module that would iterate over all
objects on the module, and apply the decorator to each function
defined therein.
In both cases a decorator like this could suffice:
from functools import wraps
from itertools import count
def typechecker(*types, **kw_types):
def decorator(func):
#wraps(func)
def wrapper(*args, **kw):
for position, type_, arg in zip(count(), types, args):
if not isinstance(arg, type_):
raise TypeError("Argument in position %s should be a %s"
% (position, type_))
for key, type_ in kw_types.items():
if key in kw_types and not isinstance(kw[key], type_):
raise TypeError("Argument %s should be a %s"
% (key, type_))
return func(*args, **kw)
return wrapper
return decorator
And you can see it working like this:
>>> #typechecker(int, int)
... def mysum(a,b):
... return a + b
...
>>> mysum(2,3)
5
>>> mysum(2.0,3)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 8, in wrapper
TypeError: Argument in position 0 should be a <type 'int'>
>>>

My inclination would be something like this:
class A(object):
def func(self, arg):
result = self._get_func(arg)(arg)
def _get_func(self, arg):
# Create and return the appropriate function here.
# You have a variety of options, e.g. arg.__class__ (if it exists),
# or use a bunch of isinstance checks, etc.
# Please bear in mind that this is really unpythonic!!
if isinstance(arg, basestring):
pass
If you're going to be calling the function a lot, this is obviously inefficient, so you would want to cache your specific functions, and try to pull from the cache first inside _get_func.
def _get_func(self, arg):
if arg.__class__ in self._cache: # Or whatever
return self._cache[arg.__class__]
else:
pass # Create and return the appropriate function here.
Of course you can override the _get_func method as needed (this ought to work at module level or class-level, though I'd probably prefer classes for clarity).
Also it's worth mentioning that this is really unpythonic. It's generally much cleaner to just write separate functions out in the open if they need to do different things, rather than hiding them inside function constructors. Or at least, "better to ask forgiveness" and use try/except blocks to encapsulate operations that only apply to specific types/classes.

Make a membermethod a default parameter

Here is what I want to do:
class demo(object):
def a(self):
pass
def b(self, param=self.a): #I tried demo.a as well after making a static
param()
The problem is apparently that one can't access the class in the function declaration line.
Is there a way to add a prototype like in c(++)?
At the moment I use a ugly workarround:
def b(self, param=True): #my real function shall be able to use None, to skip the function call
if param == True:
param = self.a
if param != None: #This explainds why I can't take None as default,
#for param, I jsut needed something as default which was
#neither none or a callable function (don't want to force the user to create dummy lambdas)
param()
So is it possible to achieve something like described in the top part without this ugly workarround? Note bene: I am bound to Jython which is approximately python 2.5 (I know there is 2.7 but I can't upgrade)

Short answer: No.
I think the best way to do it, if you want to be able to pass objects like None, True, etc., is to create a custom placeholder object like so:
default_value = object()
class demo(object):
def a(self):
pass
def b(self, param=default_value):
if param is default_value:
self.a()
else:
param()
You can use the funciton a as the default value for b, like so:
def b(self, param=a):
This will work long as a is defined before b. But the function a is not the same as the bound method self.a, so you'd need to bind it before you could call it, and you would need some way to distinguish a passed callable from the default method a, so that you could bind the latter but not the former. This would obviously be much messier than the comparatively short and readable code that I suggest.

Don't tell anyone I showed you this.
class demo:
def a(self): print(self, "called 'a'")
def b(self, param): param(self)
demo.b.__defaults__ = (demo.a,)
demo().b()
(In 2.x, __defaults__ is spelled func_defaults.)

I'll answer this question again, contradicting my earlier answer:
Short answer: YES! (sort of)
With the help of a method decorator, this is possible. The code is long and somewhat ugly, but the usage is short and simple.
The problem was that we can only use unbound methods as default arguments. Well, what if we create a wrapping function -- a decorator -- which binds the arguments before calling the real function?
First we create a helper class that can perform this task.
from inspect import getcallargs
from types import MethodType
from functools import wraps
class MethodBinder(object):
def __init__(self, function):
self.function = function
def set_defaults(self, args, kwargs):
kwargs = getcallargs(self.function, *args, **kwargs)
# This is the self of the method we wish to call
method_self = kwargs["self"]
# First we build a list of the functions that are bound to self
targets = set()
for attr_name in dir(method_self):
attr = getattr(method_self, attr_name)
# For older python versions, replace __func__ with im_func
if hasattr(attr, "__func__"):
targets.add(attr.__func__)
# Now we check whether any of the arguments are identical to the
# functions we found above. If so, we bind them to self.
ret = {}
for kw, val in kwargs.items():
if val in targets:
ret[kw] = MethodType(val, method_self)
else:
ret[kw] = val
return ret
So instances of MethodBinder are associated with a method (or rather a function that will become a method). MethodBinders method set_defaults may be given the arguments used to call the associated method, and it will bind any unbound method of the self of the associated method and return a kwargs dict that may be used to call the associated method.
Now we can create a decorator using this class:
def bind_args(f):
# f will be b in the below example
binder = MethodBinder(f)
#wraps(f)
def wrapper(*args, **kwargs):
# The wrapper function will get called instead of b, so args and kwargs
# contains b's arguments. Let's bind any unbound function arguments:
kwargs = binder.set_defaults(args, kwargs)
# All arguments have been turned into keyword arguments. Now we
# may call the real method with the modified arguments and return
# the result.
return f(**kwargs)
return wrapper
Now that we've put the uglyness behind us, let's show the simple and pretty usage:
class demo(object):
def a(self):
print("{0}.a called!".format(self))
#bind_args
def b(self, param=a):
param()
def other():
print("other called")
demo().b()
demo().b(other)
This recipe uses a rather new addition to python, getcallargs from inspect. It's available only in newer versions of python2.7 and 3.1.

You can put the method name in the function definition:
class Demo(object):
def a(self):
print 'a'
def b(self, param='a'):
if param:
getattr(self, param)()
However, you'll still have to check whether param has a value of whether it is None. Note that this approach should not be used for untrusted input as it allows execution of any function of that class.

I like lazyr's answer but maybe you will like this solution better:
class Demo(object):
def a(self):
pass
def b(self, *args):
if not args:
param=self.a
elif len(args)>1:
raise TypeError("b() takes at most 1 positional argument")
else:
param=args[0]
if param is not None:
param()

I also prefer lazyr's answer (I usually use None as the default parameter), but you can also use keyword arguments to be more explicit about it:
def b(self, **kwargs):
param = kwargs.get('param', self.a)
if param: param()
You can still use None as a parameter, resulting in param not being executed. However, if you don't include the keyword argument param=, it will default to a().
demo.b() #demo.a() executed
demo.b(param=some_func) #some_func() executed
demo.b(param=None) #nothing executed.

How to get all methods of a Python class with given decorator?

How to get all methods of a given class A that are decorated with the #decorator2?
class A():
def method_a(self):
pass
#decorator1
def method_b(self, b):
pass
#decorator2
def method_c(self, t=5):
pass

Method 1: Basic registering decorator
I already answered this question here: Calling functions by array index in Python =)
Method 2: Sourcecode parsing
If you do not have control over the class definition, which is one interpretation of what you'd like to suppose, this is impossible (without code-reading-reflection), since for example the decorator could be a no-op decorator (like in my linked example) that merely returns the function unmodified. (Nevertheless if you allow yourself to wrap/redefine the decorators, see Method 3: Converting decorators to be "self-aware", then you will find an elegant solution)
It is a terrible terrible hack, but you could use the inspect module to read the sourcecode itself, and parse it. This will not work in an interactive interpreter, because the inspect module will refuse to give sourcecode in interactive mode. However, below is a proof of concept.
#!/usr/bin/python3
import inspect
def deco(func):
return func
def deco2():
def wrapper(func):
pass
return wrapper
class Test(object):
#deco
def method(self):
pass
#deco2()
def method2(self):
pass
def methodsWithDecorator(cls, decoratorName):
sourcelines = inspect.getsourcelines(cls)[0]
for i,line in enumerate(sourcelines):
line = line.strip()
if line.split('(')[0].strip() == '#'+decoratorName: # leaving a bit out
nextLine = sourcelines[i+1]
name = nextLine.split('def')[1].split('(')[0].strip()
yield(name)
It works!:
>>> print(list( methodsWithDecorator(Test, 'deco') ))
['method']
Note that one has to pay attention to parsing and the python syntax, e.g. #deco and #deco(... are valid results, but #deco2 should not be returned if we merely ask for 'deco'. We notice that according to the official python syntax at http://docs.python.org/reference/compound_stmts.html decorators are as follows:
decorator ::= "#" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
We breathe a sigh of relief at not having to deal with cases like #(deco). But note that this still doesn't really help you if you have really really complicated decorators, such as #getDecorator(...), e.g.
def getDecorator():
return deco
Thus, this best-that-you-can-do strategy of parsing code cannot detect cases like this. Though if you are using this method, what you're really after is what is written on top of the method in the definition, which in this case is getDecorator.
According to the spec, it is also valid to have #foo1.bar2.baz3(...) as a decorator. You can extend this method to work with that. You might also be able to extend this method to return a <function object ...> rather than the function's name, with lots of effort. This method however is hackish and terrible.
Method 3: Converting decorators to be "self-aware"
If you do not have control over the decorator definition (which is another interpretation of what you'd like), then all these issues go away because you have control over how the decorator is applied. Thus, you can modify the decorator by wrapping it, to create your own decorator, and use that to decorate your functions. Let me say that yet again: you can make a decorator that decorates the decorator you have no control over, "enlightening" it, which in our case makes it do what it was doing before but also append a .decorator metadata property to the callable it returns, allowing you to keep track of "was this function decorated or not? let's check function.decorator!". And then you can iterate over the methods of the class, and just check to see if the decorator has the appropriate .decorator property! =) As demonstrated here:
def makeRegisteringDecorator(foreignDecorator):
"""
Returns a copy of foreignDecorator, which is identical in every
way(*), except also appends a .decorator property to the callable it
spits out.
"""
def newDecorator(func):
# Call to newDecorator(method)
# Exactly like old decorator, but output keeps track of what decorated it
R = foreignDecorator(func) # apply foreignDecorator, like call to foreignDecorator(method) would have done
R.decorator = newDecorator # keep track of decorator
#R.original = func # might as well keep track of everything!
return R
newDecorator.__name__ = foreignDecorator.__name__
newDecorator.__doc__ = foreignDecorator.__doc__
# (*)We can be somewhat "hygienic", but newDecorator still isn't signature-preserving, i.e. you will not be able to get a runtime list of parameters. For that, you need hackish libraries...but in this case, the only argument is func, so it's not a big issue
return newDecorator
Demonstration for #decorator:
deco = makeRegisteringDecorator(deco)
class Test2(object):
#deco
def method(self):
pass
#deco2()
def method2(self):
pass
def methodsWithDecorator(cls, decorator):
"""
Returns all methods in CLS with DECORATOR as the
outermost decorator.
DECORATOR must be a "registering decorator"; one
can make any decorator "registering" via the
makeRegisteringDecorator function.
"""
for maybeDecorated in cls.__dict__.values():
if hasattr(maybeDecorated, 'decorator'):
if maybeDecorated.decorator == decorator:
print(maybeDecorated)
yield maybeDecorated
It works!:
>>> print(list( methodsWithDecorator(Test2, deco) ))
[<function method at 0x7d62f8>]
However, a "registered decorator" must be the outermost decorator, otherwise the .decorator attribute annotation will be lost. For example in a train of
#decoOutermost
#deco
#decoInnermost
def func(): ...
you can only see metadata that decoOutermost exposes, unless we keep references to "more-inner" wrappers.
sidenote: the above method can also build up a .decorator that keeps track of the entire stack of applied decorators and input functions and decorator-factory arguments. =) For example if you consider the commented-out line R.original = func, it is feasible to use a method like this to keep track of all wrapper layers. This is personally what I'd do if I wrote a decorator library, because it allows for deep introspection.
There is also a difference between #foo and #bar(...). While they are both "decorator expressons" as defined in the spec, note that foo is a decorator, while bar(...) returns a dynamically-created decorator, which is then applied. Thus you'd need a separate function makeRegisteringDecoratorFactory, that is somewhat like makeRegisteringDecorator but even MORE META:
def makeRegisteringDecoratorFactory(foreignDecoratorFactory):
def newDecoratorFactory(*args, **kw):
oldGeneratedDecorator = foreignDecoratorFactory(*args, **kw)
def newGeneratedDecorator(func):
modifiedFunc = oldGeneratedDecorator(func)
modifiedFunc.decorator = newDecoratorFactory # keep track of decorator
return modifiedFunc
return newGeneratedDecorator
newDecoratorFactory.__name__ = foreignDecoratorFactory.__name__
newDecoratorFactory.__doc__ = foreignDecoratorFactory.__doc__
return newDecoratorFactory
Demonstration for #decorator(...):
def deco2():
def simpleDeco(func):
return func
return simpleDeco
deco2 = makeRegisteringDecoratorFactory(deco2)
print(deco2.__name__)
# RESULT: 'deco2'
#deco2()
def f():
pass
This generator-factory wrapper also works:
>>> print(f.decorator)
<function deco2 at 0x6a6408>
bonus Let's even try the following with Method #3:
def getDecorator(): # let's do some dispatching!
return deco
class Test3(object):
#getDecorator()
def method(self):
pass
#deco2()
def method2(self):
pass
Result:
>>> print(list( methodsWithDecorator(Test3, deco) ))
[<function method at 0x7d62f8>]
As you can see, unlike method2, #deco is correctly recognized even though it was never explicitly written in the class. Unlike method2, this will also work if the method is added at runtime (manually, via a metaclass, etc.) or inherited.
Be aware that you can also decorate a class, so if you "enlighten" a decorator that is used to both decorate methods and classes, and then write a class within the body of the class you want to analyze, then methodsWithDecorator will return decorated classes as well as decorated methods. One could consider this a feature, but you can easily write logic to ignore those by examining the argument to the decorator, i.e. .original, to achieve the desired semantics.

To expand upon #ninjagecko's excellent answer in Method 2: Source code parsing, you can use the ast module introduced in Python 2.6 to perform self-inspection as long as the inspect module has access to the source code.
def findDecorators(target):
import ast, inspect
res = {}
def visit_FunctionDef(node):
res[node.name] = [ast.dump(e) for e in node.decorator_list]
V = ast.NodeVisitor()
V.visit_FunctionDef = visit_FunctionDef
V.visit(compile(inspect.getsource(target), '?', 'exec', ast.PyCF_ONLY_AST))
return res
I added a slightly more complicated decorated method:
#x.y.decorator2
def method_d(self, t=5): pass
Results:
> findDecorators(A)
{'method_a': [],
'method_b': ["Name(id='decorator1', ctx=Load())"],
'method_c': ["Name(id='decorator2', ctx=Load())"],
'method_d': ["Attribute(value=Attribute(value=Name(id='x', ctx=Load()), attr='y', ctx=Load()), attr='decorator2', ctx=Load())"]}

If you do have control over the decorators, you can use decorator classes rather than functions:
class awesome(object):
def __init__(self, method):
self._method = method
def __call__(self, obj, *args, **kwargs):
return self._method(obj, *args, **kwargs)
#classmethod
def methods(cls, subject):
def g():
for name in dir(subject):
method = getattr(subject, name)
if isinstance(method, awesome):
yield name, method
return {name: method for name,method in g()}
class Robot(object):
#awesome
def think(self):
return 0
#awesome
def walk(self):
return 0
def irritate(self, other):
return 0
and if I call awesome.methods(Robot) it returns
{'think': <mymodule.awesome object at 0x000000000782EAC8>, 'walk': <mymodulel.awesome object at 0x000000000782EB00>}

For those of us who just want the absolute simplest possible case - namely, a single-file solution where we have total control over both the class we're working with and the decorator we're trying to track, I've got an answer. ninjagecko linked to a solution for when you have control over the decorator you want to track, but I personally found it to be complicated and really hard to understand, possibly because I've never worked with decorators until now. So, I've created the following example, with the goal of being as straightforward and simple as possible. It's a decorator, a class with several decorated methods, and code to retrieve+run all methods that have a specific decorator applied to them.
# our decorator
def cool(func, *args, **kwargs):
def decorated_func(*args, **kwargs):
print("cool pre-function decorator tasks here.")
return_value = func(*args, **kwargs)
print("cool post-function decorator tasks here.")
return return_value
# add is_cool property to function so that we can check for its existence later
decorated_func.is_cool = True
return decorated_func
# our class, in which we will use the decorator
class MyClass:
def __init__(self, name):
self.name = name
# this method isn't decorated with the cool decorator, so it won't show up
# when we retrieve all the cool methods
def do_something_boring(self, task):
print(f"{self.name} does {task}")
#cool
# thanks to *args and **kwargs, the decorator properly passes method parameters
def say_catchphrase(self, *args, catchphrase="I'm so cool you could cook an egg on me.", **kwargs):
print(f"{self.name} says \"{catchphrase}\"")
#cool
# the decorator also properly handles methods with return values
def explode(self, *args, **kwargs):
print(f"{self.name} explodes.")
return 4
def get_all_cool_methods(self):
"""Get all methods decorated with the "cool" decorator.
"""
cool_methods = {name: getattr(self, name)
# get all attributes, including methods, properties, and builtins
for name in dir(self)
# but we only want methods
if callable(getattr(self, name))
# and we don't need builtins
and not name.startswith("__")
# and we only want the cool methods
and hasattr(getattr(self, name), "is_cool")
}
return cool_methods
if __name__ == "__main__":
jeff = MyClass(name="Jeff")
cool_methods = jeff.get_all_cool_methods()
for method_name, cool_method in cool_methods.items():
print(f"{method_name}: {cool_method} ...")
# you can call the decorated methods you retrieved, just like normal,
# but you don't need to reference the actual instance to do so
return_value = cool_method()
print(f"return value = {return_value}\n")
Running the above example gives us the following output:
explode: <bound method cool.<locals>.decorated_func of <__main__.MyClass object at 0x00000220B3ACD430>> ...
cool pre-function decorator tasks here.
Jeff explodes.
cool post-function decorator tasks here.
return value = 4
say_catchphrase: <bound method cool.<locals>.decorated_func of <__main__.MyClass object at 0x00000220B3ACD430>> ...
cool pre-function decorator tasks here.
Jeff says "I'm so cool you could cook an egg on me."
cool post-function decorator tasks here.
return value = None
Note that the decorated methods in this example have different types of return values and different signatures, so the practical value of being able to retrieve and run them all is a bit dubious. However, in cases where there are many similar methods, all with the same signature and/or type of return value (like if you're writing a connector to retrieve unnormalized data from one database, normalize it, and insert it into a second, normalized database, and you have a bunch similar methods, e.g. 15 read_and_normalize_table_X methods), being able to retrieve (and run) them all on the fly could be more useful.

Maybe, if the decorators are not too complex (but I don't know if there is a less hacky way).
def decorator1(f):
def new_f():
print "Entering decorator1", f.__name__
f()
new_f.__name__ = f.__name__
return new_f
def decorator2(f):
def new_f():
print "Entering decorator2", f.__name__
f()
new_f.__name__ = f.__name__
return new_f
class A():
def method_a(self):
pass
#decorator1
def method_b(self, b):
pass
#decorator2
def method_c(self, t=5):
pass
print A.method_a.im_func.func_code.co_firstlineno
print A.method_b.im_func.func_code.co_firstlineno
print A.method_c.im_func.func_code.co_firstlineno

I don't want to add much, just a simple variation of ninjagecko's Method 2. It works wonders.
Same code, but using list comprehension instead of a generator, which is what I needed.
def methodsWithDecorator(cls, decoratorName):
sourcelines = inspect.getsourcelines(cls)[0]
return [ sourcelines[i+1].split('def')[1].split('(')[0].strip()
for i, line in enumerate(sourcelines)
if line.split('(')[0].strip() == '#'+decoratorName]

A simple way to solve this problem is to put code in the decorator that adds each function/method, that is passed in, to a data set (for example a list).
e.g.
def deco(foo):
functions.append(foo)
return foo
now every function with the deco decorator will be added to functions.