Ive been on a tear of writing some decorators recently.
One of the ones I just wrote allows you to put the decorator just before a class definition, and it will cause every method of the class to print some logigng info when its run (more for debugging/initial super basic speed tests during a build)
def class_logit(cls):
class NCls(object):
def __init__(self, *args, **kwargs):
self.instance = cls(*args, **kwargs)
#staticmethod
def _class_logit(original_function):
def arg_catch(*args, **kwargs):
start = time.time()
result = original_function(*args, **kwargs)
print('Called: {0} | From: {1} | Args: {2} | Kwargs: {3} | Run Time: {4}'
''.format(original_function.__name__, str(inspect.getmodule(original_function)),
args, kwargs, time.time() - start))
return result
return arg_catch
def __getattribute__(self, s):
try:
x = super(NCls, self).__getattribute__(s)
except AttributeError:
pass
else:
return x
x = self.instance.__getattribute__(s)
if type(x) == type(self.__init__):
return self._class_logit(x)
else:
return x
return NCls
This works great when applied to a very basic class i create.
Where I start to encounter issues is when I apply it to a class that is inheriting another - for instance, using QT:
#scld.class_logit
class TestWindow(QtGui.QDialog):
def __init__(self):
print self
super(TestWindow, self).__init__()
a = TestWindow()
Im getting the following error... and im not entirely sure what to do about it!
self.instance = cls(*args, **kwargs)
File "<string>", line 15, in __init__
TypeError: super(type, obj): obj must be an instance or subtype of type
Any help would be appreciated!
(Apologies in advance, no matter WHAT i do SO is breaking the formatting on my first bit of code... Im even manually spending 10 minutes adding spaces but its coming out incorrectly... sorry!)
You are being a bit too intrusive with your decorator.
While if you want to profile methods defined on the Qt framework itself, a somewhat aggressive approach is needed, your decorator replaces the entire class by a proxy.
Qt bindings are somewhat complicated indeed, and it is hard to tell why it is erroring when being instantiated in this case.
So - first things first - if your intent would be to apply the decorator to a class hierarchy defined by yourself, or at least one defined in pure Python, a good approach there could be using metaclasses: with a metaclass you could decorate each method when a class is created, and do not mess anymore at runtime, when methods are retrieved from each class.
but Qt, as some other libraries, have its methods and classes defined in native code, and that will prevent you from wrapping existing methods in a new class. So, wrapping the methods on attribute retrieval on __getattribute__ could work.
Here is a simpler approach that instead of using a Proxy, just plug-in a foreign __getattribute__ that does the wrap-with-logger thing you want.
Your mileage may vary with it. Specially, it won't be triggered if one method of the class is called by other method in native code - as this won't go through Python's attribute retrieval mechanism (instead, it will use C++ method retrieval directly).
from PyQt5 import QtWidgets, QtGui
def log_dec(func):
def wraper(*args, **kwargs):
print(func.__name__, args, kwargs)
return func(*args, **kwargs)
return wraper
def decorate(cls):
def __getattribute__(self, attr):
attr = super(cls, self).__getattribute__(attr)
if callable(attr):
return log_dec(attr)
return attr
cls.__getattribute__ = __getattribute__
return cls
#decorate
class Example(QtGui.QWindow):
pass
app = QtWidgets.QApplication([])
w = Example()
w.show()
(Of course, just replace the basic logger by your fancy logger above)
Related
Consider this small example:
import datetime as dt
class Timed(object):
def __init__(self, f):
self.func = f
def __call__(self, *args, **kwargs):
start = dt.datetime.now()
ret = self.func(*args, **kwargs)
time = dt.datetime.now() - start
ret["time"] = time
return ret
class Test(object):
def __init__(self):
super(Test, self).__init__()
#Timed
def decorated(self, *args, **kwargs):
print(self)
print(args)
print(kwargs)
return dict()
def call_deco(self):
self.decorated("Hello", world="World")
if __name__ == "__main__":
t = Test()
ret = t.call_deco()
which prints
Hello
()
{'world': 'World'}
Why is the self parameter (which should be the Test obj instance) not passed as first argument to the decorated function decorated?
If I do it manually, like :
def call_deco(self):
self.decorated(self, "Hello", world="World")
it works as expected. But if I must know in advance if a function is decorated or not, it defeats the whole purpose of decorators. What is the pattern to go here, or do I misunderstood something?
tl;dr
You can fix this problem by making the Timed class a descriptor and returning a partially applied function from __get__ which applies the Test object as one of the arguments, like this
class Timed(object):
def __init__(self, f):
self.func = f
def __call__(self, *args, **kwargs):
print(self)
start = dt.datetime.now()
ret = self.func(*args, **kwargs)
time = dt.datetime.now() - start
ret["time"] = time
return ret
def __get__(self, instance, owner):
from functools import partial
return partial(self.__call__, instance)
The actual problem
Quoting Python documentation for decorator,
The decorator syntax is merely syntactic sugar, the following two function definitions are semantically equivalent:
def f(...):
...
f = staticmethod(f)
#staticmethod
def f(...):
...
So, when you say,
#Timed
def decorated(self, *args, **kwargs):
it is actually
decorated = Timed(decorated)
only the function object is passed to the Timed, the object to which it is actually bound is not passed on along with it. So, when you invoke it like this
ret = self.func(*args, **kwargs)
self.func will refer to the unbound function object and it is invoked with Hello as the first argument. That is why self prints as Hello.
How can I fix this?
Since you have no reference to the Test instance in the Timed, the only way to do this would be to convert Timed as a descriptor class. Quoting the documentation, Invoking descriptors section,
In general, a descriptor is an object attribute with “binding behavior”, one whose attribute access has been overridden by methods in the descriptor protocol: __get__(), __set__(), and __delete__(). If any of those methods are defined for an object, it is said to be a descriptor.
The default behavior for attribute access is to get, set, or delete the attribute from an object’s dictionary. For instance, a.x has a lookup chain starting with a.__dict__['x'], then type(a).__dict__['x'], and continuing through the base classes of type(a) excluding metaclasses.
However, if the looked-up value is an object defining one of the descriptor methods, then Python may override the default behavior and invoke the descriptor method instead.
We can make Timed a descriptor, by simply defining a method like this
def __get__(self, instance, owner):
...
Here, self refers to the Timed object itself, instance refers to the actual object on which the attribute lookup is happening and owner refers to the class corresponding to the instance.
Now, when __call__ is invoked on Timed, the __get__ method will be invoked. Now, somehow, we need to pass the first argument as the instance of Test class (even before Hello). So, we create another partially applied function, whose first parameter will be the Test instance, like this
def __get__(self, instance, owner):
from functools import partial
return partial(self.__call__, instance)
Now, self.__call__ is a bound method (bound to Timed instance) and the second parameter to partial is the first argument to the self.__call__ call.
So, all these effectively translate like this
t.call_deco()
self.decorated("Hello", world="World")
Now self.decorated is actually Timed(decorated) (this will be referred as TimedObject from now on) object. Whenever we access it, the __get__ method defined in it will be invoked and it returns a partial function. You can confirm that like this
def call_deco(self):
print(self.decorated)
self.decorated("Hello", world="World")
would print
<functools.partial object at 0x7fecbc59ad60>
...
So,
self.decorated("Hello", world="World")
gets translated to
Timed.__get__(TimedObject, <Test obj>, Test.__class__)("Hello", world="World")
Since we return a partial function,
partial(TimedObject.__call__, <Test obj>)("Hello", world="World"))
which is actually
TimedObject.__call__(<Test obj>, 'Hello', world="World")
So, <Test obj> also becomes a part of *args, and when self.func is invoked, the first argument will be the <Test obj>.
You first have to understand how function become methods and how self is "automagically" injected.
Once you know that, the "problem" is obvious: you are decorating the decorated function with a Timed instance - IOW, Test.decorated is a Timed instance, not a function instance - and your Timed class does not mimick the function type's implementation of the descriptor protocol. What you want looks like this:
import types
class Timed(object):
def __init__(self, f):
self.func = f
def __call__(self, *args, **kwargs):
start = dt.datetime.now()
ret = self.func(*args, **kwargs)
time = dt.datetime.now() - start
ret["time"] = time
return ret
def __get__(self, instance, cls):
return types.MethodType(self, instance, cls)
Assume I have to unit test methodA, defined in the following class:
class SomeClass(object):
def wrapper(fun):
def _fun(self, *args, **kwargs):
self.b = 'Original'
fun(self, *args, **kwargs)
return _fun
#wrapper
def methodA(self):
pass
My test class is as follows:
from mock import patch
class TestSomeClass(object):
def testMethodA(self):
def mockDecorator(f):
def _f(self, *args, **kwargs):
self.b = 'Mocked'
f(self, *args, **kwargs)
return _f
with patch('some_class.SomeClass.wrapper', mockDecorator):
from some_class import SomeClass
s = SomeClass()
s.methodA()
assert s.b == 'Mocked', 's.b is equal to %s' % s.b
If I run the test, I hit the assertion:
File "/home/klinden/workinprogress/mockdecorators/test_some_class.py", line 17, in testMethodA
assert s.b == 'Mocked', 's.b is equal to %s' % s.b
AssertionError: s.b is equal to Original
If I stick a breakpoint in the test, after patching, this is I can see wrapper has been mocked out just fine, but that methodA still references the old wrapper:
(Pdb) p s.wrapper
<bound method SomeClass.mockDecorator of <some_class.SomeClass object at 0x7f9ed1bf60d0>>
(Pdb) p s.methodA
<bound method SomeClass._fun of <some_class.SomeClass object at 0x7f9ed1bf60d0>>
Any idea of what the problem is here?
After mulling over, I've found a solution.
Since monkey patching seems not to be effective (and I've also tried a few
other solutions), I dug into the function internals and that proved to be fruitful.
Python 3
You're lucky - just use the wraps decorator, which creates a __wrapped__ attribute, which in turn contains the wrapped function. See the linked answers above for more details.
Python 2
Even if you use #wraps, no fancy attribute is created.
However, you just need to realise that the wrapper method does nothing but a closure: so you'll be able to find your wrapped function in its func_closure attribute.
In the original example, the wrapped function would be at: s.methodA.im_func.func_closure[0].cell_contents
Wrapping up (ha!)
I created a getWrappedFunction helper along this lines, to ease my testing:
#staticmethod
def getWrappedFunction(wrapper):
return wrapper.im_func.func_closure[0].cell_contents
YMMV, especially if you do fancy stuff and include other objects in the closure.
I'm writing a Python class to wrap/decorate/enhance another class from a package called petl, a framework for ETL (data movement) workflows. Due to design constraints I can't just subclass it; every method call has to be sent through my own class so I can control what kind of objects are being passed back. So in principle this is a proxy class, but I'm having some trouble using existing answers/recipes out there. This is what my code looks like:
from functools import partial
class PetlTable(object):
"""not really how we construct petl tables, but for illustrative purposes"""
def hello(name):
print('Hello, {}!'.format(name)
class DatumTable(object):
def __init__(self, petl_tbl):
self.petl_tbl = petl_tbl
def __getattr__(self, name):
"""this returns a partial referencing the child method"""
petl_attr = getattr(self.petl_tbl, name, None)
if petl_attr and callable(petl_attr):
return partial(self.call_petl_method, func=petl_attr)
raise NotImplementedError('Not implemented')
def call_petl_method(self, func, *args, **kwargs):
func(*args, **kwargs)
Then I try to instantiate a table and call something:
# create a petl table
pt = PetlTable()
# wrap it with our own class
dt = DatumTable(pt)
# try to run the petl method
dt.hello('world')
This gives a TypeError: call_petl_method() got multiple values for argument 'func'.
This only happens with positional arguments; kwargs seem to be fine. I'm pretty sure it has to do with self not being passed in, but I'm not sure what the solution is. Can anyone think of what I'm doing wrong, or a better solution altogether?
This seems to be a common issue with mixing positional and keyword args:
TypeError: got multiple values for argument
To get around it, I took the positional arg func out of call_petl_method and put it in a kwarg that's unlikely to overlap with the kwargs of the child function. A little hacky, but it works.
I ended up writing a Proxy class to do all this generically:
class Proxy(object):
def __init__(self, child):
self.child = child
def __getattr__(self, name):
child_attr = getattr(self.child, name)
return partial(self.call_child_method, __child_fn__=child_attr)
#classmethod
def call_child_method(cls, *args, **kwargs):
"""
This calls a method on the child object and wraps the response as an
object of its own class.
Takes a kwarg `__child_fn__` which points to a method on the child
object.
Note: this can't take any positional args or they get clobbered by the
keyword args we're trying to pass to the child. See:
https://stackoverflow.com/questions/21764770/typeerror-got-multiple-values-for-argument
"""
# get child method
fn = kwargs.pop('__child_fn__')
# call the child method
r = fn(*args, **kwargs)
# wrap the response as an object of the same class
r_wrapped = cls(r)
return r_wrapped
This will also solve the problem. It doesn't use partial at all.
class PetlTable(object):
"""not really how we construct petl tables, but for illustrative purposes"""
def hello(name):
print('Hello, {}!'.format(name))
class DatumTable(object):
def __init__(self, petl_tbl):
self.petl_tbl = petl_tbl
def __getattr__(self, name):
"""Looks-up named attribute in class of the petl_tbl object."""
petl_attr = self.petl_tbl.__class__.__dict__.get(name, None)
if petl_attr and callable(petl_attr):
return petl_attr
raise NotImplementedError('Not implemented')
if __name__ == '__main__':
# create a petl table
pt = PetlTable()
# wrap it with our own class
dt = DatumTable(pt)
# try to run the petl method
dt.hello('world') # -> Hello, world!
I want to force object instantiation via class context manager. So make it impossible to instantiate directly.
I implemented this solution, but technically user can still instantiate object.
class HessioFile:
"""
Represents a pyhessio file instance
"""
def __init__(self, filename=None, from_context_manager=False):
if not from_context_manager:
raise HessioError('HessioFile can be only use with context manager')
And context manager:
#contextmanager
def open(filename):
"""
...
"""
hessfile = HessioFile(filename, from_context_manager=True)
Any better solution ?
If you consider that your clients will follow basic python coding principles then you can guarantee that no method from your class will be called if you are not within the context.
Your client is not supposed to call __enter__ explicitly, therefore if __enter__ has been called you know your client used a with statement and is therefore inside context (__exit__ will be called).
You just need to have a boolean variable that helps you remember if you are inside or outside context.
class Obj:
def __init__(self):
self._inside_context = False
def __enter__(self):
self._inside_context = True
print("Entering context.")
return self
def __exit__(self, *exc):
print("Exiting context.")
self._inside_context = False
def some_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some stuff with", name)
def some_other_stuff(self, name):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
print("Doing some other stuff with", name)
with Obj() as inst_a:
inst_a.some_stuff("A")
inst_a.some_other_stuff("A")
inst_b = Obj()
with inst_b:
inst_b.some_stuff("B")
inst_b.some_other_stuff("B")
inst_c = Obj()
try:
inst_c.some_stuff("c")
except Exception:
print("Instance C couldn't do stuff.")
try:
inst_c.some_other_stuff("c")
except Exception:
print("Instance C couldn't do some other stuff.")
This will print:
Entering context.
Doing some stuff with A
Doing some other stuff with A
Exiting context.
Entering context.
Doing some stuff with B
Doing some other stuff with B
Exiting context.
Instance C couldn't do stuff.
Instance C couldn't do some other stuff.
Since you'll probably have many methods that you want to "protect" from being called from outside context, then you can write a decorator to avoid repeating the same code to test for your boolean:
def raise_if_outside_context(method):
def decorator(self, *args, **kwargs):
if not self._inside_context:
raise Exception("This method should be called from inside context.")
return method(self, *args, **kwargs)
return decorator
Then change your methods to:
#raise_if_outside_context
def some_other_stuff(self, name):
print("Doing some other stuff with", name)
I suggest the following approach:
class MainClass:
def __init__(self, *args, **kwargs):
self._class = _MainClass(*args, **kwargs)
def __enter__(self):
print('entering...')
return self._class
def __exit__(self, exc_type, exc_val, exc_tb):
# Teardown code
print('running exit code...')
pass
# This class should not be instantiated directly!!
class _MainClass:
def __init__(self, attribute1, attribute2):
self.attribute1 = attribute1
self.attribute2 = attribute2
...
def method(self):
# execute code
if self.attribute1 == "error":
raise Exception
print(self.attribute1)
print(self.attribute2)
with MainClass('attribute1', 'attribute2') as main_class:
main_class.method()
print('---')
with MainClass('error', 'attribute2') as main_class:
main_class.method()
This will outptut:
entering...
attribute1
attribute2
running exit code...
---
entering...
running exit code...
Traceback (most recent call last):
File "scratch_6.py", line 34, in <module>
main_class.method()
File "scratch_6.py", line 25, in method
raise Exception
Exception
None that I am aware of. Generally, if it exists in python, you can find a way to call it. A context manager is, in essence, a resource management scheme... if there is no use-case for your class outside of the manager, perhaps the context management could be integrated into the methods of the class? I would suggest checking out the atexit module from the standard library. It allows you to register cleanup functions much in the same way that a context manager handles cleanup, but you can bundle it into your class, such that each instantiation has a registered cleanup function. Might help.
It is worth noting that no amount of effort will prevent people from doing stupid things with your code. Your best bet is generally to make it as easy as possible for people to do smart things with your code.
You can think of hacky ways to try and enforce this (like inspecting the call stack to forbid direct calls to your object, boolean attribute that is set upon __enter__ that you check before allowing other actions on the instance) but that will eventually become a mess to understand and explain to others.
Irregardless, you should also be certain that people will always find ways to bypass it if wanted. Python doesn't really tie your hands down, if you want to do something silly it lets you do it; responsible adults, right?
If you need an enforcement, you'd be better off supplying it as a documentation notice. That way if users opt to instantiate directly and trigger unwanted behavior, it's their fault for not following guidelines for your code.
I'm trying to write a library that will register an arbitrary list of service calls from multiple service endpoints to a container. I intend to implement the service calls in classes written one per service. Is there a way to maintain the boundedness of the methods from the service classes when registering them to the container (so they will still have access to the instance data of their owning object instance), or must I register the whole object then write some sort of pass through in the container class with __getattr__ or some such to access the methods within instance context?
container:
class ServiceCalls(object):
def __init__(self):
self._service_calls = {}
def register_call(self, name, call):
if name not in self._service_calls:
self._service_calls[name] = call
def __getattr__(self, name):
if name in self._service_calls:
return self._service_calls[name]
services:
class FooSvc(object):
def __init__(self, endpoint):
self.endpoint = endpoint
def fooize(self, *args, **kwargs):
#call fooize service call with args/kwargs utilizing self.endpoint
def fooify(self, *args, **kwargs):
#call fooify service call with args/kwargs utilizing self.endpoint
class BarSvc(object):
def __init__(self, endpoint):
self.endpoint = endpoint
def barize(self, *args, **kwargs):
#call barize service call with args/kwargs utilizing self.endpoint
def barify(self, *args, **kwargs):
#call barify service call with args/kwargs utilizing self.endpoint
implementation code:
foosvc = FooSvc('fooendpoint')
barsvc = BarSvc('barendpoint')
calls = ServiceCalls()
calls.register('fooize', foosvc.fooize)
calls.register('fooify', foosvc.fooify)
calls.register('barize', barsvc.barize)
calls.register('barify', barsvc.barify)
calls.fooize(args)
I think this answers your question:
In [2]: f = 1 .__add__
In [3]: f(3)
Out[3]: 4
You won't need the staticmethod function when adding these functions to classes, because they are effectively already "staticed".
What you are trying to do will work fine, as you can see by running your own code. :)
The object foosvc.fooize is called a "bound method" in Python, and it contains both, a reference to foosvc and to the function FooSvc.fooize. If you call the bound method, the reference to self will be passed implicitly as the first paramater.
On a side note, __getattr__() shouldn't silently return None for invalid attribute names. Better use this:
def __getattr__(self, name):
try:
return self._service_calls[name]
except KeyError:
raise AttributeError
I don't understand the use case for this -- it seems to me that the easy, simple, idiomatic way to accomplish this is to just pass in an object.
But: program to the interface, not the implementation. Only assume that the object has the method you need -- don't touch the internals or any other methods.