I am trying to understand deeper how metaclasses work in python. My problem is the following, I want to use metaclasses in order to define a method for each class which would use a class attribute defined within the metaclass. For instance, this has application for registration.
Here is a working example:
import functools
def dec_register(func):
#functools.wraps(func)
def wrapper_register(*args, **kwargs):
(args[0].__class__.list_register_instances).append(args[0])
return func(*args, **kwargs)
return wrapper_register
dict_register_classes = {}
class register(type):
def __new__(meta, name, bases, attrs):
dict_register_classes[name] = cls = type.__new__(meta, name, bases, attrs) # assigniation from right to left
cls.list_register_instances = []
cls.print_register = meta.print_register
return cls
def print_register(self):
for element in self.list_register_instances:
print(element)
def print_register_class(cls):
for element in cls.list_register_instances:
print(element)
#
class Foo(metaclass=register):
#dec_register
def __init__(self):
pass
def print_register(self):
pass
class Boo(metaclass=register):
#dec_register
def __init__(self):
pass
def print_register(self):
pass
f = Foo()
f_ = Foo()
b = Boo()
print(f.list_register_instances)
print(b.list_register_instances)
print(dict_register_classes)
print("1")
f.print_register()
print("2")
Foo.print_register_class()
print("3")
f.print_register_class()
print("4")
Foo.print_register()
The test I am making at the end do not work as I was expected. I apologize in advance if what I am saying is not using the proper syntax, I am trying to be as clear as possible :
I was thinking that the line cls.print_register = meta.print_register is defining a method within the class using the method defined within the metaclass. Thus it is a method that I can use on an object. I can also use it a class method since it is defined in the metaclass. However, though the following works :
print("1")
f.print_register()
this do not work correctly :
print("4")
Foo.print_register()
with error :
Foo.print_register()
TypeError: print_register() missing 1 required positional argument: 'self'
Same for test 2 and 3, where I was expecting that if a method is defined on the class level, it should also be defined on the object level. However, test 3 is raising an error.
print("2")
Foo.print_register_class()
print("3")
f.print_register_class()
Hence, can you please explain me how come my understanding of class methods is wrong ? I would like to be able to call the method print_register either on the class or on the object.
Perhaps it could help to know that in fact I was trying to reproduce the following very simple example :
# example without anything fancy:
class Foo:
list_register_instances = []
def __init__(self):
self.__class__.list_register_instances.append(self)
#classmethod
def print_register(cls):
for element in cls.list_register_instances:
print(element)
Am I not doing the exact same thing with a metaclass ? A classmethod can be used either on a class or on objects.
Also if you have any tips about code structure I would greatly appreciate it. I must be very bad at the syntax of metaclasses.
Fundamentally, because you have shadowed print_register on your instance of the metaclass (your class).
So when you do Foo.print_register, it finds the print_register you defined in
class Foo(metaclass=register):
...
def print_register(self):
pass
Which of course, is just the plain function print_register, which requires the self argument.
This is (almost) the same thing that would happen with just a regular class and it's instances:
class Foo:
def bar(self):
print("I am a bar")
foo = Foo()
foo.bar = lambda x: print("I've hijacked bar")
foo.bar()
Note:
In [1]: class Meta(type):
...: def print_register(self):
...: print('hi')
...:
In [2]: class Foo(metaclass=Meta):
...: pass
...:
In [3]: Foo.print_register()
hi
In [4]: class Foo(metaclass=Meta):
...: def print_register(self):
...: print('hello')
...:
In [5]: Foo.print_register()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-5-a42427fde947> in <module>
----> 1 Foo.print_register()
TypeError: print_register() missing 1 required positional argument: 'self'
However, you do this in your metaclass constructor as well!
cls.print_register = meta.print_register
Which is effectively like defining that function in your class definition... not sure why you are doing this though.
You are not doing the exact same thing as using a classmethod, which is a custom descriptor that handles the binding of methods to instances in just the way you'd need to be able to call it on a class or on an instance. That is not the same as defining a method on the class and on the instance! You could just do this in your metaclass __new__, i.e. cls.print_register = classmethod(meta.print_register) and leave def print_register(self) out of your class definitions:
import functools
def dec_register(func):
#functools.wraps(func)
def wrapper_register(*args, **kwargs):
(args[0].__class__.list_register_instances).append(args[0])
return func(*args, **kwargs)
return wrapper_register
dict_register_classes = {}
class register(type):
def __new__(meta, name, bases, attrs):
dict_register_classes[name] = cls = type.__new__(meta, name, bases, attrs) # assigniation from right to left
cls.list_register_instances = []
cls.print_register = classmethod(meta.print_register) # just create the classmethod manually!
return cls
def print_register(self):
for element in self.list_register_instances:
print(element)
def print_register_class(cls):
for element in cls.list_register_instances:
print(element)
#
class Foo(metaclass=register):
#dec_register
def __init__(self):
pass
Note, print_register doesn't have to be defined inside your metaclass, indeed, in this case, I would just define it at the module level:
def print_register(self):
for element in self.list_register_instances:
print(element)
...
class register(type):
def __new__(meta, name, bases, attrs):
dict_register_classes[name] = cls = type.__new__(meta, name, bases, attrs) # assigniation from right to left
cls.list_register_instances = []
cls.print_register = classmethod(print_register)
return cls
...
I think you understand metaclasses sufficiently, actually, it is your understanding of classmethod that is incorrect, as far as I can tell. If you want to understand how classmethod works, indeed, how method-instance binding works for regular functions, you need to understand descriptors. Here's an enlightening link. Function objects are descriptors, they bind the instance as the first argument to themselves when called on an instance (rather, they create a method object and return that, but it is basically partial application). classmethod objects are another kind of descriptor, one that binds the class to the first argument to the function it decorates when called on either the class or the instance. The link describes how you could write classmethod using pure python.
Related
Is there a way to access a class (where function is defined as a method) before there is an instance of that class?
class MyClass:
def method(self):
print("Calling me")
m1 = MyClass.method
instance = MyClass()
m2 = instance.method
print(m2.__self__.__class__) # <class 'MyClass'>
# how to access `MyClass` from `m1`?
For example I have m1 variable somewhere in my code and want to have a reference to MyClass the same way I can access it from bound method m2.__self__.__class__.
print(m1.__qualname__) # 'MyClass.method'
The only option I was able to find is __qualname__ which is a string containing name of the class.
The attribute __self__ itself is annotated by Python when the function is bound to an instance and become a method. (The code to that is run somewhere when running the __get__ code in the function, but passing an instance different than None).
So, as people pointed out, you have the option of getting the classname as a string by going through __qualname__. Otherwise, if the functions/methods for which you will need this feature are known beforehand, it is possible to create a decorator that will annotate their class when they are retrieved as a class attribute (in contrast to the native annotation which only takes place when retrieving then as an instance attribute):
class unboundmethod:
def __init__(self, func, cls):
self.__func__ = func
self.class_ = cls
self.__self__ = None
def __call__(self, instance, *args, **kw):
if not isinstance(instance, self.class_):
# This check is actually optional fancy stuff, since we are here! :-)
raise TypeError(f"First parameter fo {self.__func__.__name__} must be an instance of {self.class_}")
return self.__func__(instance, *args, **kw)
def __repr__(self):
return f"Unbound method {self.__func__!r} related to {self.class_}"
class clsbind:
def __init__(self, func):
self.func = func
def __get__(self, instance, owner):
if instance is None:
# the function is being retrieved from the class:
return unboundmethod(self.func, owner)
# return control to usual method creation codepath:
return self.func.__get__(instance, owner)
class MyClass:
#clsbind
def method(self):
print("Calling me")
And on the REPL you can have this:
In [136]: m1 = MyClass.method
In [137]: m1.class_
Out[137]: __main__.MyClass
In [138]: m1(MyClass())
Calling me
You can get the class instance using the __qualname__
my_class = eval(m1.__qualname__.split('.')[-2])
print(my_class)
Not the most generic and safest approach, but should work for this simple scenario.
I started learning Selenium and I am curious how to realize behavior from the PythonOrgSearch class which inherits from unittest.TestCase. Namely, each method which starts with test_ will be called automatically after initialization. I know this behavior is implemented in TestCase but I am interested in how to make something similar. Is there a design pattern that will take care of this?
And one bonus question, what is the point of assert True, since the condition is always True
import unittest
from selenium import webdriver
class PythonOrgSearch(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Chrome("C:\chorme\chromedriver.exe")
self.driver.get("http://www.python.org")
def test_example(self):
print("Test")
assert True
def not_test(self):
print("Not a test")
def tearDown(self):
self.driver.close()
if __name__ == "__main__":
unittest.main()
You can do what you want with a metaclass which can customize the construction of your own classes. This a very powerful and general technique and arguably a Python design pattern.
Below is an example of it being applied to what you want to do. The metaclass' __new__() method looks through the contents of the class being defined—which is when it gets called—and looks for callable attributes whose names start with test_. After doing that, it defines __init__() and post_init() methods and makes them part of the class. The former calls the latter method which then iteratively calls all the methods defined that had matching names.
class MyMetaClass(type):
""" Create class that calls an added post_init() method which in turn calls
all method's whose names start with "test_".
"""
def __new__(meta, classname, bases, classdict):
# Get any class __init__() method defined.
class_init = classdict.get('__init__', lambda *_, **__: None)
test_funcs = [value for key, value in classdict.items()
if key.startswith('test_') and callable(value)]
def __init__(self, *args, **kwargs):
print('In metaclass generated __init__()')
class_init(self, *args, **kwargs) # Call class' __init__() method.
self.post_init()
def post_init(self):
print('In metaclass generated post_init()')
for method in test_funcs:
print(f'calling {classname}.{method.__name__}()')
method(self)
classdict.update({'__init__': __init__, # Attach methods to class.
'post_init': post_init})
return type.__new__(meta, classname, bases, classdict)
class Example(metaclass=MyMetaClass):
def __init__(self, arg, macnab=None):
print(f'in Example.__init__({arg!r}, macnab={macnab!r})')
def setUp(self):
pass
def test_example1(self):
print("Test1")
def test_example2(self):
print("Test2")
def not_test(self):
print("Not a test")
def tearDown(self):
print("Also not a test")
pass
print('Creating instance of Example')
Example = Example(42, macnab='keyword')
Output:
Creating instance of Example
In metaclass generated __init__()
in Example.__init__(42, macnab='keyword')
In metaclass generated post_init()
calling Example.test_example1()
Test1
calling Example.test_example2()
Test2
For the first question, you can use dir() on self to get a list of its member (Ufficial Documentation for dir).
After that, you can test the name pattern in some simple way, and if it is callable you can call it:
for name in dir(self):
if name[:5] == 'test_' and callable(getattr(self, name)):
res = getattr(self, name)()
print(res)
Concerning you bonus question, it is a common practice to force the function to be overloaded.
I would imagine they're simply finding callable methods that begin with "test_" using the dir() function. Something you could achieve pretty easily like:
class CustomTestCaseRunner:
def run(self):
methods = [
m for m in dir(self)
if callable(getattr(self, m))
and m.startswith("test_")
]
for m in methods:
print(f"Running {self.__class__.__name__}.{m}")
getattr(self, m)()
class MyTest(CustomTestCaseRunner):
def test_foo(self):
assert True
def test_bar(self):
assert 1
MyTest().run()
# Running MyTest.test_bar
# Running MyTest.test_foo
As for your second question about assert True, it is unlikely you'd ever actually assert True in live code. That function appears to just be an example. assert is typically used on the response from a function. Here are a few examples:
assert isinstance(1, int)
assert isinstance("foo", str)
When the condition evaluates to False, it will raise an AssertionError which will fail your test case.
I have a code like this:
class Base:
def __init__(self):
pass
def new_obj(self):
return Base() # ← return Derived()
class Derived(Base):
def __init__(self):
pass
In the line with a comment I actually want not exactly the Derived object, but any object of class that self really is.
Here is a real-life example from Mercurial.
How to do that?
def new_obj(self):
return self.__class__()
I can't think of a really good reason to do this, but as D.Shawley pointed out:
def new_obj(self):
return self.__class__()
will do it.
That's because when calling a method on a derived class, if it doesn't exist on that class, it will use the method resolution order to figure out which method to call on its inheritance chain. In this case, you've only got one, so it's going to call Base.new_obj and pass in the instance as the first argument (i.e. self).
All instances have a __class__ attribute, that refers to the class that they are an instance of. So given
class Base:
def new_obj(self):
return self.__class__()
class Derived(Base): pass
derived = Derived()
The following lines are functionally equivalent:
derived.new_obj()
# or
Base.new_obj(derived)
You may have encountered a relative of this if you've either forgotten to add the self parameter to your function declaration, or not provided enough arguments to a function and seen a stack trace that looks like this:
>>> f.bar()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: bar() takes exactly 2 arguments (1 given)
You can use a classmethod:
class Base:
def __init__(self):
pass
#classmethod
def new_obj(cls):
return cls()
class Derived(Base):
def __init__(self):
pass
>>> b = Base()
>>> b.new_obj()
<__main__.Base at 0x10fc12208>
>>> d = Derived()
>>> d.new_obj()
<__main__.Derived at 0x10fdfce80>
You can also do this with a class method, which you create with a decorator.
In [1]: class Base:
...: #classmethod
...: def new_obj(cls):
...: return cls()
...:
In [2]: class Derived(Base): pass
In [3]: print type(Base.new_obj())
<type 'instance'>
In [4]: print Base.new_obj().__class__
__main__.Base
In [5]: print Derived.new_obj().__class__
__main__.Derived
Incidentally (you may know this), you don't have to create __init__ methods if you don't do anything with them.
I'm implementing several constructors using #classobj. I'm not only setting variables, but also calling methods in the new class:
class Example:
def __init__(self):
pass
#classmethod
def constructor1(cls,x,y):
self=cls
self.__x = x
self.__somemethod(self,y)
...
I get the following error:
unbound method __somemethod() must be called with Example instance as
first argument (got classobj instance instead)
How do I resolve this problem?
If you're wanting your class method to be a constructor, you probably want to be creating an instance of the class you get passed in as cls. I suspect you're trying to do that with your self = cls line, but you're not actually creating a new instance because you've neglected to put parentheses. There are some other issues too, but I think that is the key one. Here's a fixed constructor:
#classmethod
def constructor1(cls,x,y):
self=cls() # parentheses added, to make it a call
self.__x = x
self.__somemethod(y) # self is not needed as a parameter here
return self # return the new instance
looks like __somemethod is not a classmethod, but a "normal" method.
And normal methods expect an actual instance as the first parameter, not a class.
And because constructor1 is decorated as a #classmethod, cls is the class itself - which you pass to __somemethod.
That cannot work.
You should reconsider your design approach.
Addendum:
Maybe you meant something like this?
#classmethod
def constructor1(cls, x, y):
newinst = cls()
newinst.__x = x
cls.__somemethod(newinst, y)
That'd be better written as followed, though:
#classmethod
def constructor1(cls, x, y):
newinst = cls()
newinst.__x = x
newinst.__somemethod(y)
actually, I like neighter approach - seems like a codesmell of overcomplexity to me.
This may be a template of what I think you're trying to achieve...
import random
class Something(object):
def __init__(self, value, **kwargs):
self.value = value
for k, v in kwargs.iteritems():
setattr(self, k, v)
#classmethod
def from_iterable(cls, iterable):
return cls(sum(iterable), description='came from from_iterable')
#classmethod
def make_random(cls):
return cls(random.randint(1,1000), is_random=True)
a = Something.from_iterable([1, 2, 3])
b = Something.make_random()
c = Something(56)
for obj in (a, b, c):
print type(obj), obj.value
<class '__main__.Something'> 6
<class '__main__.Something'> 308
<class '__main__.Something'> 56
Thanks to ch3ka's answer and Tim Pietzcker's comment, I found my error: I used the factory method from http://jjinux.blogspot.co.at/2008/11/python-class-methods-make-good.html and forgot the () in the line self=cls(). Now it works just fine:
class Example:
def __init__(self):
pass
#classmethod
def constructor1(cls,x,y):
self=cls()
self.__x = x
self.__somemethod(self,y)
...
class Liquid(object):
def foo(self):
pass
def bar(self):
pass
class Water(Liquid):
Say, I have the two classes above, Water inherits from Liquid. Is there any way I can restrict Water from inheriting one of the parent's methods, say bar()?
Sort of. But don't do it.
class Liquid(object):
def foo(self):
pass
def bar(self):
pass
class Water(Liquid):
def __getattribute__(self, name):
if name == 'bar':
raise AttributeError("'Water' object has no attribute 'bar'")
l = Liquid()
l.bar()
w = Water()
w.bar()
You can override the method to be a no-op, but you can't remove it. Doing so would violate one of the core principles of object-oriented design, namely that any object that inherits from some parent should be able to be used anywhere the parent is used. This is known as the Liskov Substitution Principle.
You can, as the other answers, say, break one of the inherited methods.
The alternative is to refactor out the "optional" methods, and inherit from a baseclass that doesn't have them:
class BaseLiquid(object):
def foo(self):
pass
class Barised(object):
def bar(self):
pass
class Liquid(BaseLiquid, Barised): pass
class Water(BaseLiquid):
def drip(self):
pass
This is probably not a good idea, but you could always use metaclasses to implement private attributes:
def private_attrs(name, bases, attrs):
def get_base_attrs(base):
result = {}
for deeper_base in base.mro()[1:]:
result.update( get_base_attrs(deeper_base) )
priv = []
if "__private__" in base.__dict__:
priv = base.__private__
for attr in base.__dict__:
if attr not in priv:
result.update( {attr: base.__dict__[attr]} )
return result
final_attrs = {}
for base in bases:
final_attrs.update( get_base_attrs(base) )
final_attrs.update(attrs)
return type(name, (), final_attrs)
class Liquid(object):
__metaclass__ = private_attrs
__private__ = ['bar']
def foo(self):
pass
def bar(self):
pass
class Water(Liquid):
__metaclass__ = private_attrs
print Water.foo
print Water.bar
Output is:
<unbound method Water.foo>
Traceback (most recent call last):
File "testing-inheritance.py", line 41, in <module>
print Water.bar
AttributeError: type object 'Water' has no attribute 'bar'
EDIT: This will mess up isinstance() because it doesn't modify bases of the class.
http://docs.python.org/release/2.5.2/ref/slots.html
I suspect you can do this, by using the slots attr.
It might be possible implementing a getattr method and throwing the appropriate exception if bar is called.
However, I agree, you don't want to do this in practice, since its a sign of bad design.