Related
For some given class in python like:
class Foo:
def __init__(self, ...):
...
pass
def not_raising_1(self, ...):
...
pass
def not_raising_2(self, ...):
...
pass
is it possible to enforce that the user has to call not_raising_1() or not_raising_2() after creating an object of type Foo. So I'm thinking of a behavior:
foo = Foo(...) # will raise a Exception saying you need to call not_raising_1 or not_raising_1
foo = Foo(...).not_raising_1(...) # will NOT raise a Excpetion
foo = Foo(...).not_raising_2(...) # will NOT raise a Excpetion
I know that a pragmatic solution would obviously be to put what ever should happen in not_raising_1() or not_raising_2() with some parameter in the constructor of Foo. But I'm here not asking for a pragmatic solution but am just curios if someone can think of some creative solution to get the described behavior.
First, for the record: if the two methods must be called before the object is ready to be used, that means that calling them is part of the initialization of the object, and so they should be called by __init__ itself:
class Foo:
def __init__(self, ...):
...
self.not_raising_1()
self.not_raising_2()
def not_raising_1(self, ...):
...
def not_raising_2(self, ...):
...
But, moving on to the question as asked...
The problem is not well defined.
Unless you call the methods inside __init__ itself, it is trivially true that neither method has been called the instant __init__ exits.
Further, once __init__ exits, the class Foo has no visibility into what happens outside its own definition. You need some sort of explicit state that maintains what happens after Foo.__init__ exits. Foo.not_raising_1 could examine that state to determine if anything else happened before it was called.
But that raises another problem: who will update that state? Every single bit of code would have to cooperate with Foo. Is this illegal?
x = Foo()
y = 3
x.not_raising_1()
Then how are you going to make Python update your state when it executes y = 3? The hooks just aren't there.
And finally, who is going to raise the exception if x.not_raising_1 is never called?
Refine the problem.
Rather than ask if the functions are never called, you can ensure they are called inside a with statement using an appropriately defined context manager. This context manager can ensure that not_raising_1 and not_raising_2 are called before the with statement completes, as well as ensure that they are only used inside a with statement. You can't enforce that the object is used as a context manager, but you can ensure that it is only used in a with statement.
class Foo:
def __init__(self, ...):
...
self._in_with_statement = False
self._r1_called = False
self._r2_called = False
def not_raising_1(self, ...):
self._r1_called = True
if not self._in_with_statement
raise RuntimeException("Foo instance must be used with context manager")
def not_raising_2(self, ...):
self._r2_called = True
if not self._in_with_statement
raise RuntimeException("Foo instance must be used with context manager")
def something_else(self):
if not self._r1_called or not self._r2_called:
raise RuntimeException("Failed to call not_raising_1 and/or not_raising_2")
...
def __enter__(self):
self._in_with_statement = True
def __exit__(self):
self._in_with_statement = False
if not self._r1_called or not self._r2_called:
raise RuntimeException("Failed to call not_raising_1 and/or not_raising_2")
self._r1_called = False
self._r2_called = False
Here, __init__ sets the condition that neither method has yet been called, nor are we yet executing in a with statement. The instance itself acts as the external state that monitors how the instance is used.
The two required methods require themselves to be executed inside a with statement (by checking if __enter__ has been called).
Every other method can check if the required methods have been called.
The __enter__ method simply marks the object as now being in a with statement, allowing the required methods to be called.
The __exit_ method ensures that the required methods were eventually called, and resets the state of the object as being outside a context manger.
I think this is as strong a guarantee as you can enforce, short of a class that uses the inspect module to examine the script's source code looking for violations.
You could use a classmethod like this:
class Foo:
def __init__(self, flag=True):
if flag:
raise CustomError()
#classmethod
def not_raising_1(cls):
return cls(flag=False)
Thus foo = Foo() or foo = Foo(...).not_raising_1(...) would still raise the exception, but foo = Foo.not_raising_1(...) would work.
It's not possible. You could use a workaround as that one suggested by Ajay Signh Rana or chepner but I would, personally, not recommend it as it is hard to grasp when reading the code.
Your goal should be to increase readability and usability of the class for yourself and other programmers that uses this class. Use well-known patterns and concepts whenever possible and if applicable.
Reading your question, I understand that the object is not ready to use until one of the other methods is invoked. You could consider Julian Fock's answer and use a class method.
Or use any of the other Creational Design Patterns:
https://refactoring.guru/design-patterns/creational-patterns
Depending on the reason why you want to achieve this behaviour, you could consider to implement the Builder pattern:
https://refactoring.guru/design-patterns/builder/python/example
https://stackoverflow.com/a/26193004/42659
Builder pattern equivalent in Python
A third alternative would be, as you mention yourself, that you pass some parameters along when invoking the constructor and call, depending on the parameter, either of the other methods within the constructor.
Which approach is usable and applicable for your situation depends on your needs and bigger picture than the example in your Question. You should choose the approach that suits your needs best and is most readable.
I did get your question but as others suggested it cannot be done. But yeah you wann raise an exception and it should be raised if the function isn't call then you must create another function that checks if the previous functions were called or not and if not you can raise the exception.
I would approach this problem by creating a variable that changes it's value based on the funciton calls and if the functions haven't been called we can determine that as well.
Try:
class SomeError(Exception):
pass
class Foo:
def __init__(self, ...):
self.flag = False # set the flag to false for each object initially
...
pass
def not_raising_1(self, ...):
self.flag = True # set it to true once the function has been called
...
pass
def not_raising_2(self, ...):
self.flag = True # repeat for this on too
...
pass
def raise_exception(self):
if(not self.flag):
raise SomeError
obj1 = Foo()
obj1.not_raising_1()
obj1.raise_exception() # won't do anything
obj2 = Foo()
obj2.raise_exception() # would raise exception as either of the two functions weren't called
As others have suggested, it's not something that you should consider in actual code. But Just as an excercise, I tried doing something similar:
class NoError(Exception):
pass
class Foo:
def __init__(self):
pass
def not_raising_1(self):
raise NoError()
def not_raising_2(self):
raise NoError()
How to use:
try:
Foo()
raise Exception('please use either not_raising_1 or not_raising_2')
except NoError:
print('No error')
# actual code
class Package:
def __init__(self):
self.files = []
# ...
def __del__(self):
for file in self.files:
os.unlink(file)
__del__(self) above fails with an AttributeError exception. I understand Python doesn't guarantee the existence of "global variables" (member data in this context?) when __del__() is invoked. If that is the case and this is the reason for the exception, how do I make sure the object destructs properly?
I'd recommend using Python's with statement for managing resources that need to be cleaned up. The problem with using an explicit close() statement is that you have to worry about people forgetting to call it at all or forgetting to place it in a finally block to prevent a resource leak when an exception occurs.
To use the with statement, create a class with the following methods:
def __enter__(self)
def __exit__(self, exc_type, exc_value, traceback)
In your example above, you'd use
class Package:
def __init__(self):
self.files = []
def __enter__(self):
return self
# ...
def __exit__(self, exc_type, exc_value, traceback):
for file in self.files:
os.unlink(file)
Then, when someone wanted to use your class, they'd do the following:
with Package() as package_obj:
# use package_obj
The variable package_obj will be an instance of type Package (it's the value returned by the __enter__ method). Its __exit__ method will automatically be called, regardless of whether or not an exception occurs.
You could even take this approach a step further. In the example above, someone could still instantiate Package using its constructor without using the with clause. You don't want that to happen. You can fix this by creating a PackageResource class that defines the __enter__ and __exit__ methods. Then, the Package class would be defined strictly inside the __enter__ method and returned. That way, the caller never could instantiate the Package class without using a with statement:
class PackageResource:
def __enter__(self):
class Package:
...
self.package_obj = Package()
return self.package_obj
def __exit__(self, exc_type, exc_value, traceback):
self.package_obj.cleanup()
You'd use this as follows:
with PackageResource() as package_obj:
# use package_obj
The standard way is to use atexit.register:
# package.py
import atexit
import os
class Package:
def __init__(self):
self.files = []
atexit.register(self.cleanup)
def cleanup(self):
print("Running cleanup...")
for file in self.files:
print("Unlinking file: {}".format(file))
# os.unlink(file)
But you should keep in mind that this will persist all created instances of Package until Python is terminated.
Demo using the code above saved as package.py:
$ python
>>> from package import *
>>> p = Package()
>>> q = Package()
>>> q.files = ['a', 'b', 'c']
>>> quit()
Running cleanup...
Unlinking file: a
Unlinking file: b
Unlinking file: c
Running cleanup...
A better alternative is to use weakref.finalize. See the examples at Finalizer Objects and Comparing finalizers with __del__() methods.
As an appendix to Clint's answer, you can simplify PackageResource using contextlib.contextmanager:
#contextlib.contextmanager
def packageResource():
class Package:
...
package = Package()
yield package
package.cleanup()
Alternatively, though probably not as Pythonic, you can override Package.__new__:
class Package(object):
def __new__(cls, *args, **kwargs):
#contextlib.contextmanager
def packageResource():
# adapt arguments if superclass takes some!
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
yield package
package.cleanup()
def __init__(self, *args, **kwargs):
...
and simply use with Package(...) as package.
To get things shorter, name your cleanup function close and use contextlib.closing, in which case you can either use the unmodified Package class via with contextlib.closing(Package(...)) or override its __new__ to the simpler
class Package(object):
def __new__(cls, *args, **kwargs):
package = super(Package, cls).__new__(cls)
package.__init__(*args, **kwargs)
return contextlib.closing(package)
And this constructor is inherited, so you can simply inherit, e.g.
class SubPackage(Package):
def close(self):
pass
Here is a minimal working skeleton:
class SkeletonFixture:
def __init__(self):
pass
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
pass
def method(self):
pass
with SkeletonFixture() as fixture:
fixture.method()
Important: return self
If you're like me, and overlook the return self part (of Clint Miller's correct answer), you will be staring at this nonsense:
Traceback (most recent call last):
File "tests/simplestpossible.py", line 17, in <module>
fixture.method()
AttributeError: 'NoneType' object has no attribute 'method'
Hope it helps the next person.
I don't think that it's possible for instance members to be removed before __del__ is called. My guess would be that the reason for your particular AttributeError is somewhere else (maybe you mistakenly remove self.file elsewhere).
However, as the others pointed out, you should avoid using __del__. The main reason for this is that instances with __del__ will not be garbage collected (they will only be freed when their refcount reaches 0). Therefore, if your instances are involved in circular references, they will live in memory for as long as the application run. (I may be mistaken about all this though, I'd have to read the gc docs again, but I'm rather sure it works like this).
I think the problem could be in __init__ if there is more code than shown?
__del__ will be called even when __init__ has not been executed properly or threw an exception.
Source
Just wrap your destructor with a try/except statement and it will not throw an exception if your globals are already disposed of.
Edit
Try this:
from weakref import proxy
class MyList(list): pass
class Package:
def __init__(self):
self.__del__.im_func.files = MyList([1,2,3,4])
self.files = proxy(self.__del__.im_func.files)
def __del__(self):
print self.__del__.im_func.files
It will stuff the file list in the del function that is guaranteed to exist at the time of call. The weakref proxy is to prevent Python, or yourself from deleting the self.files variable somehow (if it is deleted, then it will not affect the original file list). If it is not the case that this is being deleted even though there are more references to the variable, then you can remove the proxy encapsulation.
It seems that the idiomatic way to do this is to provide a close() method (or similar), and call it explicitely.
A good idea is to combine both approaches.
To implement a context manager for explicit life-cycle handling. As well as handle cleanup in case the user forgets it or it is not convenient to use a with statement. This is best done by weakref.finalize.
This is how many libraries actually do it. And depending on the severity, you could issue a warning.
It is guaranteed to be called exactly once, so it is safe to call it at any time before.
import os
from typing import List
import weakref
class Package:
def __init__(self):
self.files = []
self._finalizer = weakref.finalize(self, self._cleanup_files, self.files)
#staticmethod
def _cleanup_files(files: List):
for file in files:
os.unlink(file)
def __enter__(self):
return self
def __exit__(self, exc_type, exc_value, traceback):
self._finalizer()
weakref.finalize returns a callable finalizer object which will be called when obj is garbage collected. Unlike an ordinary weak reference, a finalizer will always survive until the reference object is collected, greatly simplifying lifecycle management."
Unlike atexit.register the object is not held in memory until the interpreter is shut down.
And unlike object.__del__, weakref.finalize is guaranteed to be called at interpreter shutdown. So it is much more safe.
atexit.register is the standard way as has already been mentioned in ostrakach's answer.
However, it must be noted that the order in which objects might get deleted cannot be relied upon as shown in example below.
import atexit
class A(object):
def __init__(self, val):
self.val = val
atexit.register(self.hello)
def hello(self):
print(self.val)
def hello2():
a = A(10)
hello2()
a = A(20)
Here, order seems legitimate in terms of reverse of the order in which objects were created as program gives output as :
20
10
However when, in a larger program, python's garbage collection kicks in object which is out of it's lifetime would get destructed first.
Following this related question, while there are always examples of some library using a language feature in a unique way, I was wondering whether returning a value other than self in an __enter__ method should be considered an anti-pattern.
The main reason why this seems to me like a bad idea is that it makes wrapping context managers problematic. For example, in Java (also possible in C#), one can wrap an AutoCloseable class in another class which will take care of cleaning up after the inner class, like in the following code snippet:
try (BufferedReader reader =
new BufferedReader(new FileReader("src/main/resources/input.txt"))) {
return readAllLines(reader);
}
Here, BufferedReader wraps FileReader, and calls FileReader's close() method inside its own close() method. However, if this was Python, and FileReader would've returned an object other than self in its __enter__ method, this would make such an arrangement significantly more complicated. The following issues would have to be addressed by the writer of BufferedReader:
When I need to use FileReader for my own methods, do I use FileReader directly or the object returned by its __enter__ method? What methods are even supported by the returned object?
In my __exit__ method, do I need to close only the FileReader object, or the object returned in the __enter__ method?
What happens if __enter__ actually returns a different object on its call? Do I now need to keep a collection of all of the different objects returned by it in case someone calls __enter__ several times on me? How do I know which one to use when I need to use on of these objects?
And the list goes on. One semi-successful solution to all of these problems would be to simply avoid having one context manager class clean up after another context manager class. In my example, that would mean that we would need two nested with blocks - one for the FileReader, and one for the BufferedReader. However, this makes us write more boilerplate code, and seems significantly less elegant.
All in all, these issues lead me to believe that while Python does allow us to return something other than self in the __enter__ method, this behavior should simply be avoided. Is there some official or semi-official remarks about these issues? How should a responsible Python developer write code that addresses these issues?
TLDR: Returning something other than self from __enter__ is perfectly fine and not bad practice.
The introducing PEP 343 and Context Manager specification expressly list this as desired use cases.
An example of a context manager that returns a related object is the
one returned by decimal.localcontext(). These managers set the active
decimal context to a copy of the original decimal context and then
return the copy. This allows changes to be made to the current decimal
context in the body of the with statement without affecting code
outside the with statement.
The standard library has several examples of returning something other than self from __enter__. Notably, much of contextlib matches this pattern.
contextlib.contextmanager produces context managers which cannot return self, because there is no such thing.
contextlib.closing wraps a thing and returns it on __enter__.
contextlib.nullcontext returns a pre-defined constant
threading.Lock returns a boolean
decimal.localcontext returns a copy of its argument
The context manager protocol makes it clear what is the context manager, and who is responsible for cleanup. Most importantly, the return value of __enter__ is inconsequential for the protocol.
A rough paraphrasing of the protocol is this: When something runs cm.__enter__, it is responsible for running cm.__exit__. Notably, whatever code does that has access to cm (the context manager itself); the result of cm.__enter__ is not needed to call cm.__exit__.
In other words, a code that takes (and runs) a ContextManager must run it completely. Any other code does not have to care whether its value comes from a ContextManager or not.
# entering a context manager requires closing it…
def managing(cm: ContextManager):
value = cm.__enter__() # must clean up `cm` after this point
try:
yield from unmanaged(value)
except BaseException as exc:
if not cm.__exit__(type(exc), exc, exc.__traceback__):
raise
else:
cm.__exit__(None, None, None)
# …other code does not need to know where its values come from
def unmanaged(smth: Any):
yield smth
When context managers wrap others, the same rules apply: If the outer context manager calls the inner one's __enter__, it must call its __exit__ as well. If the outer context manager already has the entered inner context manager, it is not responsible for cleanup.
In some cases it is in fact bad practice to return self from __enter__. Returning self from __enter__ should only be done if self is fully initialised beforehand; if __enter__ runs any initialisation code, a separate object should be returned.
class BadContextManager:
"""
Anti Pattern: Context manager is in inconsistent state before ``__enter__``
"""
def __init__(self, path):
self.path = path
self._file = None # BAD: initialisation not complete
def read(self, n: int):
return self._file.read(n) # fails before the context is entered!
def __enter__(self) -> 'BadContextManager':
self._file = open(self.path)
return self # BAD: self was not valid before
def __exit__(self, exc_type, exc_val, tb):
self._file.close()
class GoodContext:
def __init__(self, path):
self.path = path
self._file = None # GOOD: Inconsistent state not visible/used
def __enter__(self) -> TextIO:
if self._file is not None:
raise RuntimeError(f'{self.__class__.__name__} is not re-entrant')
self._file = open(self.path)
return self._file # GOOD: value was not accessible before
def __exit__(self, exc_type, exc_val, tb):
self._file.close()
Notably, even though GoodContext returns a different object, it is still responsible to clean up. Another context manager wrapping GoodContext does not need to close the return value, it just has to call GoodContext.__exit__.
There are essentially three ways to use the with statement:
Use an existing context manager:
with manager:
pass
Create a context manager and bind its result to a variable:
with Manager() as result:
pass
Create an context manager and discard its return value:
with Manager():
pass
If we have place a function get_manager() inside the three with blocks above, is there any implementation that can return the enclosing context manager, or at least their __exit__ function?
It's obviously easy in the first case, but I can't think of a way to make it work in the other two. I doubt it's possible to get the entire context manager, since the value stack is popped immediately after the SETUP_WITH opcode. However, since the __exit__ function is stored on the block stack by SETUP_WITH, is there some way to access it?
Unfortunately, as discussed in the comments, this is not possible in all cases. When a context manager is created, the following code is run (in cPython 2.7, at least. I can't comment on other implementations):
case SETUP_WITH:
{
static PyObject *exit, *enter;
w = TOP();
x = special_lookup(w, "__exit__", &exit);
if (!x)
break;
SET_TOP(x);
/* more code follows... */
}
The __exit__ method is pushed onto a stack with the SET_TOP macro, which is defined as:
#define SET_TOP(v) (stack_pointer[-1] = (v))
The stack pointer, in turn, is set to the top of the frame's value stack at the start of frame eval:
stack_pointer = f->f_stacktop;
Where f is a frame object defined in frameobject.h. Unfortunately for us, this is where the trail stops. The python accessible frame object is defined with the following methods only:
static PyMemberDef frame_memberlist[] = {
{"f_back", T_OBJECT, OFF(f_back), RO},
{"f_code", T_OBJECT, OFF(f_code), RO},
{"f_builtins", T_OBJECT, OFF(f_builtins),RO},
{"f_globals", T_OBJECT, OFF(f_globals), RO},
{"f_lasti", T_INT, OFF(f_lasti), RO},
{NULL} /* Sentinel */
};
Which, unfortunaltey, does not include the f_valuestack that we would need. This makes sense, since f_valuestack is of the type PyObject **, which would need to be wrapped in an object to be accessible from python any way.
TL;DR: The __exit__ method we're looking for is only located in one place, the value stack of a frame object, and cPython doesn't make the value stack accessible to python code.
The difference between this case and similar-appearing cases like super is that here there is no enclosing frame to look at. A with statement is not a new scope. sys._getframe(0) (or, if you're putting the code into a function, sys._getframe(1)) will work just fine, but it'll return you the exact same frame you have before and after the with statement.
The only way you could do it would be by inspecting the bytecode. But even that won't help. For example, try this:
from contextlib import contextmanager
#contextmanager
def silly():
yield
with silly():
fr = sys._getframe(0)
dis.dis(fr.f_code)
Obviously, as SETUP_WITH explains, the method does get looked up and pushed onto the stack for WITH_CLEANUP to use later. So, even after POP_TOP removes the return value of silly(), its __exit__ is still on the stack.
But there's no way to get at that from Python. Unless you want to start munging the bytecode, or digging apart the stack with ctypes or something, it might as well not exist.
If the context manager is a class and only ever has a single instance, then you could find it on the heap:
import gc
class ConMan(object):
def __init__(self, name):
self.name = name
def __enter__(self):
print "enter %s" % self.name
def found(self):
print "You found %s!" % self.name
def __exit__(self, *args):
print "exit %s" % self.name
def find_single(typ):
single = None
for obj in gc.get_objects():
if isinstance(obj, typ):
if single is not None:
raise ValueError("Found more than one")
single = obj
return single
def foo():
conman = find_single(ConMan)
conman.found()
with ConMan('the-context-manager'):
foo()
(Disclaimer: Don't do this)
If you will accept a hacky solution, I bring you one inspired by this.
Have the context manager edit the local namespace.
class Context(object):
def __init__(self, locals_reference):
self.prev_context = locals_reference.get('context', None)
self.locals_reference = locals_reference
def __enter__(self):
self.locals_reference['context'] = self
def __exit__(self, exception_type, exception_value, traceback):
if self.prev_context is not None:
self.locals_reference['context'] = self.prev_context
else:
del self.locals_reference['context']
You can then get the context with the context variable
with Context(locals()):
print(context)
This implementation also works on nested contexts
with Context(locals()):
c_context = context
with Context(locals()):
print(c_context == context)
print(c_context == context)
However this is implementation specific, as the return value of locals may be a copy of the namespace. Tested on CPython 3.10.
Edit:
The implementation above will not work in functions from other modules (I wonder why), so here is a function that fetches the context:
def get_current_context(cls) -> "Context | None":
try:
if context is not None:
return context
except NameError:
pass
i = 0
while True:
try:
c = sys._getframe(i).f_locals.get('context',None)
except ValueError:
return None
if c is not None:
return c
i += 1
I would make it a classmethod of the context manager class.
Many years late, here's a straightforward way to do this in the second of the OP's 3 cases, where with ... as is used to bind the output of the context manager's __enter__ method to a variable. You can have the __enter__ method return the context manager itself, or its __exit__ method if that's all you're interested in.
class Manager:
def __enter__(self): return self
def __exit__(self, *args): print("Doing exit method stuff!")
with Manager() as manager:
print("Doing some stuff before manually calling the exit method...")
manager.__exit__() # call the exit method
print("Doing some more stuff before exiting for real...")
Of course, this would interfere with using with ... as to bind some other return-value from __enter__, but it would be straightforward to have __enter__ return a tuple consisting of its ordinary return value and the manager, just as you can make a function return multiple values.
As the OP noted, it's also straightforward to call the __exit__ method in the first case, where the context manager had already been assigned to a variable beforehand. So the only really tricky case is the third one where the context manager is simply created via with Manager(): but is never assigned to a variable. My advice would be: if you're going to want to refer to the manager (or its methods) later, then either (1) assign it a name beforehand, (2) have its __enter__ method return a reference to it for with ... as to capture as I did above, but (3) DO NOT create it without storing any reference to it!
How to get all methods of a given class A that are decorated with the #decorator2?
class A():
def method_a(self):
pass
#decorator1
def method_b(self, b):
pass
#decorator2
def method_c(self, t=5):
pass
Method 1: Basic registering decorator
I already answered this question here: Calling functions by array index in Python =)
Method 2: Sourcecode parsing
If you do not have control over the class definition, which is one interpretation of what you'd like to suppose, this is impossible (without code-reading-reflection), since for example the decorator could be a no-op decorator (like in my linked example) that merely returns the function unmodified. (Nevertheless if you allow yourself to wrap/redefine the decorators, see Method 3: Converting decorators to be "self-aware", then you will find an elegant solution)
It is a terrible terrible hack, but you could use the inspect module to read the sourcecode itself, and parse it. This will not work in an interactive interpreter, because the inspect module will refuse to give sourcecode in interactive mode. However, below is a proof of concept.
#!/usr/bin/python3
import inspect
def deco(func):
return func
def deco2():
def wrapper(func):
pass
return wrapper
class Test(object):
#deco
def method(self):
pass
#deco2()
def method2(self):
pass
def methodsWithDecorator(cls, decoratorName):
sourcelines = inspect.getsourcelines(cls)[0]
for i,line in enumerate(sourcelines):
line = line.strip()
if line.split('(')[0].strip() == '#'+decoratorName: # leaving a bit out
nextLine = sourcelines[i+1]
name = nextLine.split('def')[1].split('(')[0].strip()
yield(name)
It works!:
>>> print(list( methodsWithDecorator(Test, 'deco') ))
['method']
Note that one has to pay attention to parsing and the python syntax, e.g. #deco and #deco(... are valid results, but #deco2 should not be returned if we merely ask for 'deco'. We notice that according to the official python syntax at http://docs.python.org/reference/compound_stmts.html decorators are as follows:
decorator ::= "#" dotted_name ["(" [argument_list [","]] ")"] NEWLINE
We breathe a sigh of relief at not having to deal with cases like #(deco). But note that this still doesn't really help you if you have really really complicated decorators, such as #getDecorator(...), e.g.
def getDecorator():
return deco
Thus, this best-that-you-can-do strategy of parsing code cannot detect cases like this. Though if you are using this method, what you're really after is what is written on top of the method in the definition, which in this case is getDecorator.
According to the spec, it is also valid to have #foo1.bar2.baz3(...) as a decorator. You can extend this method to work with that. You might also be able to extend this method to return a <function object ...> rather than the function's name, with lots of effort. This method however is hackish and terrible.
Method 3: Converting decorators to be "self-aware"
If you do not have control over the decorator definition (which is another interpretation of what you'd like), then all these issues go away because you have control over how the decorator is applied. Thus, you can modify the decorator by wrapping it, to create your own decorator, and use that to decorate your functions. Let me say that yet again: you can make a decorator that decorates the decorator you have no control over, "enlightening" it, which in our case makes it do what it was doing before but also append a .decorator metadata property to the callable it returns, allowing you to keep track of "was this function decorated or not? let's check function.decorator!". And then you can iterate over the methods of the class, and just check to see if the decorator has the appropriate .decorator property! =) As demonstrated here:
def makeRegisteringDecorator(foreignDecorator):
"""
Returns a copy of foreignDecorator, which is identical in every
way(*), except also appends a .decorator property to the callable it
spits out.
"""
def newDecorator(func):
# Call to newDecorator(method)
# Exactly like old decorator, but output keeps track of what decorated it
R = foreignDecorator(func) # apply foreignDecorator, like call to foreignDecorator(method) would have done
R.decorator = newDecorator # keep track of decorator
#R.original = func # might as well keep track of everything!
return R
newDecorator.__name__ = foreignDecorator.__name__
newDecorator.__doc__ = foreignDecorator.__doc__
# (*)We can be somewhat "hygienic", but newDecorator still isn't signature-preserving, i.e. you will not be able to get a runtime list of parameters. For that, you need hackish libraries...but in this case, the only argument is func, so it's not a big issue
return newDecorator
Demonstration for #decorator:
deco = makeRegisteringDecorator(deco)
class Test2(object):
#deco
def method(self):
pass
#deco2()
def method2(self):
pass
def methodsWithDecorator(cls, decorator):
"""
Returns all methods in CLS with DECORATOR as the
outermost decorator.
DECORATOR must be a "registering decorator"; one
can make any decorator "registering" via the
makeRegisteringDecorator function.
"""
for maybeDecorated in cls.__dict__.values():
if hasattr(maybeDecorated, 'decorator'):
if maybeDecorated.decorator == decorator:
print(maybeDecorated)
yield maybeDecorated
It works!:
>>> print(list( methodsWithDecorator(Test2, deco) ))
[<function method at 0x7d62f8>]
However, a "registered decorator" must be the outermost decorator, otherwise the .decorator attribute annotation will be lost. For example in a train of
#decoOutermost
#deco
#decoInnermost
def func(): ...
you can only see metadata that decoOutermost exposes, unless we keep references to "more-inner" wrappers.
sidenote: the above method can also build up a .decorator that keeps track of the entire stack of applied decorators and input functions and decorator-factory arguments. =) For example if you consider the commented-out line R.original = func, it is feasible to use a method like this to keep track of all wrapper layers. This is personally what I'd do if I wrote a decorator library, because it allows for deep introspection.
There is also a difference between #foo and #bar(...). While they are both "decorator expressons" as defined in the spec, note that foo is a decorator, while bar(...) returns a dynamically-created decorator, which is then applied. Thus you'd need a separate function makeRegisteringDecoratorFactory, that is somewhat like makeRegisteringDecorator but even MORE META:
def makeRegisteringDecoratorFactory(foreignDecoratorFactory):
def newDecoratorFactory(*args, **kw):
oldGeneratedDecorator = foreignDecoratorFactory(*args, **kw)
def newGeneratedDecorator(func):
modifiedFunc = oldGeneratedDecorator(func)
modifiedFunc.decorator = newDecoratorFactory # keep track of decorator
return modifiedFunc
return newGeneratedDecorator
newDecoratorFactory.__name__ = foreignDecoratorFactory.__name__
newDecoratorFactory.__doc__ = foreignDecoratorFactory.__doc__
return newDecoratorFactory
Demonstration for #decorator(...):
def deco2():
def simpleDeco(func):
return func
return simpleDeco
deco2 = makeRegisteringDecoratorFactory(deco2)
print(deco2.__name__)
# RESULT: 'deco2'
#deco2()
def f():
pass
This generator-factory wrapper also works:
>>> print(f.decorator)
<function deco2 at 0x6a6408>
bonus Let's even try the following with Method #3:
def getDecorator(): # let's do some dispatching!
return deco
class Test3(object):
#getDecorator()
def method(self):
pass
#deco2()
def method2(self):
pass
Result:
>>> print(list( methodsWithDecorator(Test3, deco) ))
[<function method at 0x7d62f8>]
As you can see, unlike method2, #deco is correctly recognized even though it was never explicitly written in the class. Unlike method2, this will also work if the method is added at runtime (manually, via a metaclass, etc.) or inherited.
Be aware that you can also decorate a class, so if you "enlighten" a decorator that is used to both decorate methods and classes, and then write a class within the body of the class you want to analyze, then methodsWithDecorator will return decorated classes as well as decorated methods. One could consider this a feature, but you can easily write logic to ignore those by examining the argument to the decorator, i.e. .original, to achieve the desired semantics.
To expand upon #ninjagecko's excellent answer in Method 2: Source code parsing, you can use the ast module introduced in Python 2.6 to perform self-inspection as long as the inspect module has access to the source code.
def findDecorators(target):
import ast, inspect
res = {}
def visit_FunctionDef(node):
res[node.name] = [ast.dump(e) for e in node.decorator_list]
V = ast.NodeVisitor()
V.visit_FunctionDef = visit_FunctionDef
V.visit(compile(inspect.getsource(target), '?', 'exec', ast.PyCF_ONLY_AST))
return res
I added a slightly more complicated decorated method:
#x.y.decorator2
def method_d(self, t=5): pass
Results:
> findDecorators(A)
{'method_a': [],
'method_b': ["Name(id='decorator1', ctx=Load())"],
'method_c': ["Name(id='decorator2', ctx=Load())"],
'method_d': ["Attribute(value=Attribute(value=Name(id='x', ctx=Load()), attr='y', ctx=Load()), attr='decorator2', ctx=Load())"]}
If you do have control over the decorators, you can use decorator classes rather than functions:
class awesome(object):
def __init__(self, method):
self._method = method
def __call__(self, obj, *args, **kwargs):
return self._method(obj, *args, **kwargs)
#classmethod
def methods(cls, subject):
def g():
for name in dir(subject):
method = getattr(subject, name)
if isinstance(method, awesome):
yield name, method
return {name: method for name,method in g()}
class Robot(object):
#awesome
def think(self):
return 0
#awesome
def walk(self):
return 0
def irritate(self, other):
return 0
and if I call awesome.methods(Robot) it returns
{'think': <mymodule.awesome object at 0x000000000782EAC8>, 'walk': <mymodulel.awesome object at 0x000000000782EB00>}
For those of us who just want the absolute simplest possible case - namely, a single-file solution where we have total control over both the class we're working with and the decorator we're trying to track, I've got an answer. ninjagecko linked to a solution for when you have control over the decorator you want to track, but I personally found it to be complicated and really hard to understand, possibly because I've never worked with decorators until now. So, I've created the following example, with the goal of being as straightforward and simple as possible. It's a decorator, a class with several decorated methods, and code to retrieve+run all methods that have a specific decorator applied to them.
# our decorator
def cool(func, *args, **kwargs):
def decorated_func(*args, **kwargs):
print("cool pre-function decorator tasks here.")
return_value = func(*args, **kwargs)
print("cool post-function decorator tasks here.")
return return_value
# add is_cool property to function so that we can check for its existence later
decorated_func.is_cool = True
return decorated_func
# our class, in which we will use the decorator
class MyClass:
def __init__(self, name):
self.name = name
# this method isn't decorated with the cool decorator, so it won't show up
# when we retrieve all the cool methods
def do_something_boring(self, task):
print(f"{self.name} does {task}")
#cool
# thanks to *args and **kwargs, the decorator properly passes method parameters
def say_catchphrase(self, *args, catchphrase="I'm so cool you could cook an egg on me.", **kwargs):
print(f"{self.name} says \"{catchphrase}\"")
#cool
# the decorator also properly handles methods with return values
def explode(self, *args, **kwargs):
print(f"{self.name} explodes.")
return 4
def get_all_cool_methods(self):
"""Get all methods decorated with the "cool" decorator.
"""
cool_methods = {name: getattr(self, name)
# get all attributes, including methods, properties, and builtins
for name in dir(self)
# but we only want methods
if callable(getattr(self, name))
# and we don't need builtins
and not name.startswith("__")
# and we only want the cool methods
and hasattr(getattr(self, name), "is_cool")
}
return cool_methods
if __name__ == "__main__":
jeff = MyClass(name="Jeff")
cool_methods = jeff.get_all_cool_methods()
for method_name, cool_method in cool_methods.items():
print(f"{method_name}: {cool_method} ...")
# you can call the decorated methods you retrieved, just like normal,
# but you don't need to reference the actual instance to do so
return_value = cool_method()
print(f"return value = {return_value}\n")
Running the above example gives us the following output:
explode: <bound method cool.<locals>.decorated_func of <__main__.MyClass object at 0x00000220B3ACD430>> ...
cool pre-function decorator tasks here.
Jeff explodes.
cool post-function decorator tasks here.
return value = 4
say_catchphrase: <bound method cool.<locals>.decorated_func of <__main__.MyClass object at 0x00000220B3ACD430>> ...
cool pre-function decorator tasks here.
Jeff says "I'm so cool you could cook an egg on me."
cool post-function decorator tasks here.
return value = None
Note that the decorated methods in this example have different types of return values and different signatures, so the practical value of being able to retrieve and run them all is a bit dubious. However, in cases where there are many similar methods, all with the same signature and/or type of return value (like if you're writing a connector to retrieve unnormalized data from one database, normalize it, and insert it into a second, normalized database, and you have a bunch similar methods, e.g. 15 read_and_normalize_table_X methods), being able to retrieve (and run) them all on the fly could be more useful.
Maybe, if the decorators are not too complex (but I don't know if there is a less hacky way).
def decorator1(f):
def new_f():
print "Entering decorator1", f.__name__
f()
new_f.__name__ = f.__name__
return new_f
def decorator2(f):
def new_f():
print "Entering decorator2", f.__name__
f()
new_f.__name__ = f.__name__
return new_f
class A():
def method_a(self):
pass
#decorator1
def method_b(self, b):
pass
#decorator2
def method_c(self, t=5):
pass
print A.method_a.im_func.func_code.co_firstlineno
print A.method_b.im_func.func_code.co_firstlineno
print A.method_c.im_func.func_code.co_firstlineno
I don't want to add much, just a simple variation of ninjagecko's Method 2. It works wonders.
Same code, but using list comprehension instead of a generator, which is what I needed.
def methodsWithDecorator(cls, decoratorName):
sourcelines = inspect.getsourcelines(cls)[0]
return [ sourcelines[i+1].split('def')[1].split('(')[0].strip()
for i, line in enumerate(sourcelines)
if line.split('(')[0].strip() == '#'+decoratorName]
A simple way to solve this problem is to put code in the decorator that adds each function/method, that is passed in, to a data set (for example a list).
e.g.
def deco(foo):
functions.append(foo)
return foo
now every function with the deco decorator will be added to functions.