We Keep Coding

How python records information about all method calls on a class object(Includes built-in methods eg.__getattr__)? [duplicate]

I'm implementing a RESTful web service in python and would like to add some QOS logging functionality by intercepting function calls and logging their execution time and so on.
Basically i thought of a class from which all other services can inherit, that automatically overrides the default method implementations and wraps them in a logger function. What's the best way to achieve this?

Something like this? This implictly adds a decorator to your method (you can also make an explicit decorator based on this if you prefer that):
class Foo(object):
def __getattribute__(self,name):
attr = object.__getattribute__(self, name)
if hasattr(attr, '__call__'):
def newfunc(*args, **kwargs):
print('before calling %s' %attr.__name__)
result = attr(*args, **kwargs)
print('done calling %s' %attr.__name__)
return result
return newfunc
else:
return attr
when you now try something like:
class Bar(Foo):
def myFunc(self, data):
print("myFunc: %s"% data)
bar = Bar()
bar.myFunc(5)
You'll get:
before calling myFunc
myFunc: 5
done calling myFunc

What if you write a decorator on each functions ? Here is an example on python's wiki.
Do you use any web framework for doing your webservice ? Or are you doing everything by hand ?

Class Decorator when Inheriting from another class

Ive been on a tear of writing some decorators recently.
One of the ones I just wrote allows you to put the decorator just before a class definition, and it will cause every method of the class to print some logigng info when its run (more for debugging/initial super basic speed tests during a build)
def class_logit(cls):
class NCls(object):
def __init__(self, *args, **kwargs):
self.instance = cls(*args, **kwargs)
#staticmethod
def _class_logit(original_function):
def arg_catch(*args, **kwargs):
start = time.time()
result = original_function(*args, **kwargs)
print('Called: {0} | From: {1} | Args: {2} | Kwargs: {3} | Run Time: {4}'
''.format(original_function.__name__, str(inspect.getmodule(original_function)),
args, kwargs, time.time() - start))
return result
return arg_catch
def __getattribute__(self, s):
try:
x = super(NCls, self).__getattribute__(s)
except AttributeError:
pass
else:
return x
x = self.instance.__getattribute__(s)
if type(x) == type(self.__init__):
return self._class_logit(x)
else:
return x
return NCls
This works great when applied to a very basic class i create.
Where I start to encounter issues is when I apply it to a class that is inheriting another - for instance, using QT:
#scld.class_logit
class TestWindow(QtGui.QDialog):
def __init__(self):
print self
super(TestWindow, self).__init__()
a = TestWindow()
Im getting the following error... and im not entirely sure what to do about it!
self.instance = cls(*args, **kwargs)
File "<string>", line 15, in __init__
TypeError: super(type, obj): obj must be an instance or subtype of type
Any help would be appreciated!
(Apologies in advance, no matter WHAT i do SO is breaking the formatting on my first bit of code... Im even manually spending 10 minutes adding spaces but its coming out incorrectly... sorry!)

You are being a bit too intrusive with your decorator.
While if you want to profile methods defined on the Qt framework itself, a somewhat aggressive approach is needed, your decorator replaces the entire class by a proxy.
Qt bindings are somewhat complicated indeed, and it is hard to tell why it is erroring when being instantiated in this case.
So - first things first - if your intent would be to apply the decorator to a class hierarchy defined by yourself, or at least one defined in pure Python, a good approach there could be using metaclasses: with a metaclass you could decorate each method when a class is created, and do not mess anymore at runtime, when methods are retrieved from each class.
but Qt, as some other libraries, have its methods and classes defined in native code, and that will prevent you from wrapping existing methods in a new class. So, wrapping the methods on attribute retrieval on __getattribute__ could work.
Here is a simpler approach that instead of using a Proxy, just plug-in a foreign __getattribute__ that does the wrap-with-logger thing you want.
Your mileage may vary with it. Specially, it won't be triggered if one method of the class is called by other method in native code - as this won't go through Python's attribute retrieval mechanism (instead, it will use C++ method retrieval directly).
from PyQt5 import QtWidgets, QtGui
def log_dec(func):
def wraper(*args, **kwargs):
print(func.__name__, args, kwargs)
return func(*args, **kwargs)
return wraper
def decorate(cls):
def __getattribute__(self, attr):
attr = super(cls, self).__getattribute__(attr)
if callable(attr):
return log_dec(attr)
return attr
cls.__getattribute__ = __getattribute__
return cls
#decorate
class Example(QtGui.QWindow):
pass
app = QtWidgets.QApplication([])
w = Example()
w.show()
(Of course, just replace the basic logger by your fancy logger above)

Python initializing a memoizing decorator with settings

I have a memoizer decorator class in a library, as such:
class memoizer(object):
def __init__(self, f):
"some code here"
def __call__(self, *args, **kwargs):
"some code here"
When I use it for functions in the library, I use #memoizer. However, I'd like to have the client (ie the programmer using the library) initialize the memoization class from outside the library with some arguments so that it holds for all uses of the decorator used in the client program. Particularly, this particular memoization class saves the results to a file, and I want the client to be able to specify how much memory the files can take. Is this possible?

You can achieve this using decorator factory:
class DecoratorFactory(object):
def __init__(self, value):
self._value = value
def decorator(self, function):
def wrapper(*args, **kwargs):
print(self._value)
return function(*args, **kwargs)
return wrapper
factory = DecoratorFactory("shared between all decorators")
#factory.decorator
def dummy1():
print("dummy1")
#factory.decorator
def dummy2():
print("dummy2")
# prints:
# shared between all decorators
# dummy1
dummy1()
# prints:
# shared between all decorators
# dummy2
dummy2()
If you don't like factories you can create global variables within some module and set them before usage of our decorators (not nice solution, IMO factory is more clean).

Wrapping function calls to xmlrpc server

I have a xmlrpc server running looking like the following
from SimpleXMLRPCServer import SimpleXMLRPCServer
def add(x,y):
return x+y
server = SimpleXMLRPCServer(("localhost", 8000))
server.register_function(add, 'add.numbers')
server.serve_forever()
which is called used within the following code:
import xmlrpclib
class DeviceProxy(object):
def __init__(self, uri):
self.rpc = xmlrpclib.ServerProxy(uri)
def __getattr__(self, attr):
return getattr(self.rpc, attr)
original = DeviceProxy.__getattr__
def mygetattr(device, attr):
def wrapper(*args, **kw):
print('called with %r and %r' % (args, kw))
return original(device, attr)(*args, **kw)
return wrapper
DeviceProxy.__getattr__ = mygetattr
dev = DeviceProxy("http://localhost:8000/RPC2")
print dev.add.numbers(4,6)
As you can see, the Proxy class wraps the xmlrpc proxy for reasons outside the scope of this question, forwarding arbitrary calls via the __getattr__ method . For further reasons outside the scope for this question, I need to wrap/replace this __getattr__ method by a different method to e.g. print out the name of the function called, the arguments etc. (see related question here).
But this approach does not work, it gives the following error:
AttributeError: 'function' object has no attribute 'numbers'
The example works as expected when I
do not replace DeviceProxy.__getattr__ with something else
replace DeviceProxy.__getattr__ with the function
def dummy(instance, attr):
return original(device,attr)
replace the name of the xmlrpc function by a zero-dotted name (e.g. just sum instead of sum.numbers)
You can verify yourself that the following, direct call via the xmlrpc proxy will work as expected:
dev = xmlrpclib.ServerProxy("http://localhost:8000/RPC2")
print dev.add.numbers(4,6)
My question: How to solve my problem, i.e. how to be able to wrap/overwrite the DeviceProxy.__getattr__ correctly to be able to see the function called, all arguments etc WITHOUT making changes in the xmlrpc server or the DeviceProxy class?

I can see two problems here:
Are all attributes of a DeviceProxy functions? If they're not, then you're sometimes returning a function when an object is expected
When you wrap the function, you're not copying across members - use functools.wraps to achieve that.
This ought to work
from functools import wraps
#wraps(original) # probably not needed, but sensible
def mygetattr(device, key):
attr = original(device, key)
if callable(attr):
#wraps(attr) # copy across __name__, __dict__ etc
def wrapper(*args, **kw):
print('called with %r and %r' % (args, kw))
return attr(*args, **kw)
return wrapper
else: # handle (or rather, don't) non-callable attributes
return attr

Is there a __enter__ and __exit__ equivalent for all functions in a class?

One functionality of python that I found very handy when working with databases (or files) are the __enter__ and __exit__ functions you can give to a class. Now by using the with statement you can make sure that in this block __enter__ is first called (and you can open the database or file) and after it's done __exit__ is called (and you can close a database or file.
I want to open and close a sqlite transaction every time a function from my Database class is called. I can do it at the start and end of every function, but since it has to be done for every function is that class, I was wondering, are there methods that get called before and after each function call? Like SetUp and TearDown in unittesting.

You can decorate every member function with a pie decorator, something like
#transaction
def insertData(self):
# code
and transaction is a decorator you define to wrap the function with a pre and post.
Yes, you have to do it for every function. Here is an example
def transaction(f):
def pre():
print "pre transaction"
def post():
print "post transaction"
def wrapped(*args):
pre()
f(*args)
post()
return wrapped
class Foo(object):
def __init__(self):
print "instantiating"
def doFoo(self):
print "doing foo"
#transaction
def doBar(self, value):
print "doing bar "+str(value)
#transaction
def foofunc():
print "hello"
foofunc()
f=Foo()
f.doFoo()
f.doBar(5)
.
stefanos-imac:python borini$ python decorator.py
pre transaction
hello
post transaction
instantiating
doing foo
pre transaction
doing bar 5
post transaction
The alternative is that you use a metaclass, like this:
import types
class DecoratedMetaClass(type):
def __new__(meta, classname, bases, classDict):
def pre():
print "pre transaction"
def post():
print "post transaction"
newClassDict={}
for attributeName, attribute in classDict.items():
if type(attribute) == types.FunctionType:
def wrapFunc(f):
def wrapper(*args):
pre()
f(*args)
post()
return wrapper
newAttribute = wrapFunc(attribute)
else:
newAttribute = attribute
newClassDict[attributeName] = newAttribute
return type.__new__(meta, classname, bases, newClassDict)
class MyClass(object):
__metaclass__ = DecoratedMetaClass
def __init__(self):
print "init"
def doBar(self, value):
print "doing bar "+str(value)
def doFoo(self):
print "doing foo"
c = MyClass()
c.doFoo()
c.doBar(4)
This is pure black magic, but it works
stefanos-imac:python borini$ python metaclass.py
pre transaction
init
post transaction
pre transaction
doing foo
post transaction
pre transaction
doing bar 4
post transaction
You normally don't want to decorate the __init__, and you may want to decorate only those methods with a special name, so you may want to replace
for attributeName, attribute in classDict.items():
if type(attribute) == types.FunctionType:
with something like
for attributeName, attribute in classDict.items():
if type(attribute) == types.FunctionType and "trans_" in attributeName[0:6]:
This way, only methods called trans_whatever will be transactioned.