This has been bugging me for a while and appears hard to google.
In the following code can anyone explain the pragmatic difference between FirstChild and SecondChild. It's clear from experiments that both "work" and arguably SecondChild is marginally more efficient. But is there something that I'm missing about the way these two behave? Are they different and how are they different?
import collections
class Parent:
def send_message(self, message: str):
pass
class FirstChild(Parent):
def __init__(self):
self.message_queue = collections.deque()
def send_message(self, message: str):
self.message_queue.append(message)
class SecondChild(Parent):
def __init__(self):
self.message_queue = collections.deque()
self.send_message = self.message_queue.append
FirstChild creates a descriptor in the class called send_message. When you do instance.send_message, the interpreter first searches the instance __dict__ for the name, then the class. When it's found in the class, the function is bound to the instance to create a method object that doesn't accept self. It happens every time you do the lookup, and it looks something like
method = type(instance).send_message.__get__(type(instance), instance)
SecondChild assigns a bound method as the attribute send_message in the instance. It cuts out the lookup in its own class object, as well as the lookup in the deque class object, and binding. That is probably why it appears marginally more efficient.
A major practical difference between these approaches is that send_message in SecondChild is not overridable. Since functions are non data descriptors (they have a __get__ method but not __set__ (and yes, functions have a class type and methods, like any other object)), the instance attribute send_message in SecondChild will always trump any class-level function. This means that a child of SecondChild that calls the parent __init__ method will hide any implementation of send_message it creates.
You will likely find the official descriptor guide to be quite informative: https://docs.python.org/3/howto/descriptor.html
Related
TL;DR: Python; I have Parent, Child classes. I have an instance of Parent class, parent. Can I make a Child class instance whose super() is parent?
Somewhat specific use case (workaround available) is as follows: I'd like to make an instance of Logger class (from Python logging module), with _log method overloaded. Methods like logger.info or logger.error call this method with a level specified as either INFO or ERROR etc., I'd like to replace this one method, touch nothing else, and make it all work seamlessly.
Here's some things that don't work (well):
I can't just inherit from logging.Logger instance and overload this one method and constructor, because Logger instances tend to be created via a factory method, logging.getLogger(name). So I can't just overload the constructor of the wrapper like:
class WrappedLogger(logging.Logger):
def __init__(self, ...):
super().__init__(...)
def _log(self, ...):
and expect it to all work OK.
I could make a wrapper class, which provides the methods I'd like to call on the resulting instance, like .info or .error - but then I have to manually come up with all the cases. It also doesn't work well when the _log method is buried a few calls down the stack - there is basically no way to guarantee that any use of the wrapped class will call my desired _log method
I can make a little kludge like so:
class WrappedLogger(logging.Logger):
def __init__(self, parent):
self._parent = parent
def _log(...): # overload
def __getattr__(self, method_name):
return getattr(self._parent, method_name)
now whenever I have an instance of this class, and call, say, wrapped.info(...), it will retrieve the parent method of info, call it, which will then call self._log which in turn points to my wrapped instance. But this feels very ugly.
Similarly, I could take a regular instance of Logger and manually swap out the method; this is maybe a bit less "clever", and less ugly than the above, but similarly underwhelming.
This question has been asked a few times, but in slightly different contexts, where other solutions were proposed. Rather than looking for a workaround, I'm interested in whether there is a native way of constructing a child class instance with the parent instance specified.
Related questions:
Create child class instances from parent class instance, and call parent methods from child class instance - here effectively a workaround is suggested
Python construct child class from parent - here the parent can be created in the child's constructor
If your goal is to supply a custom logger class that is used by getLogger, you can "register" the custom class with the logging manager.
So, let's define a custom logger class
from logging import Logger
class MyLogger(Logger):
def _log(self, level, msg, *args, **kwargs) -> None:
print("my logger wants to log:", msg)
super()._log(level, msg, *args, **kwargs)
Then we tell the global logging manager to use this class instead.
from logging import setLoggerClass
setLoggerClass(MyLogger)
Thank you #Daniil Fajnberg, for pointing out, that setLoggerClass exists.
Now getLogger will instantiate your custom class.
from logging import getLogger
logger = getLogger(__file__)
logger.error("Dummy Error")
This will log the error as normal and also print "my logger wants to log: ...".
Note: The _log method you are overloading is undocumented. Maybe there is a better way to achieve what you want.
If i am understanding correctly, what #Bennet wants is - he has some custom logger classes derived from Logger(Logger acts as interface) like Logger1, Logger2 etc(which implementation gets chosen will vary at runtime). On top of each of this he wants to add some functionality which modifies only the _log function of each of these implementations.
IMO there shouldn't be any direct way to do it, since what you are attempting is trying to modify(not extend) the behaviour of an existing class which is not recommended for OOP paradigm.
The hacky way is clever (found it cool).
def __getattr__(self, method_name):
return getattr(self._parent, method_name)
(I don't think you can do the same in Java)
P.S. Wanted to comment this but i am poor in SO it seems :)
From the way you keep re-phrasing your more general question, it seems you misunderstand how object creation works. You are asking for a
way of constructing a child class instance with the parent instance specified.
There is no such concept as a "parent instance". Inheritance refers to classes, not objects. This means you need to define yourself, what that term is supposed to mean. How would you define a "parent instance"? What should it be and when and how should it be created?
Just to demonstrate that there is no mechanism for creating "parent instances", when a child class instance is created, consider this:
class Foo:
instances = []
def __new__(cls):
print(f"{cls.__name__}.__new__()")
instance = super().__new__(cls)
Foo.instances.append(instance)
return instance
class Bar(Foo):
pass
bar = Bar()
assert len(Foo.instances) == 1
assert Foo.instances[0] is bar
assert type(bar) is Bar
assert isinstance(bar, Foo)
The output is Bar.__new__() and obviously the assertions are passed. This goes to show that when we create an instance of Bar it delegates construction further up the MRO chain (because it doesn't implement its own __new__ method), which results in a call to Foo.__new__. It creates the object (by calling object.__new__) and puts it into its instances list. Foo does not also create another instance of class Foo.
You also seem to misunderstand, what calling the super class does, so I suggest checking out the documentation. In short, it is just an elegant tool to access a related class (again: not an instance).
So, again, your question is ill defined.
If you mean (as #Barmar suggested) that you want a way to copy all the attributes of an instance of Foo over to an instance of Bar, that is another story. In that case, you still need to be careful to define, what exactly you mean by "all attributes".
Typically this would refer to the instances __dict__. But do you also want its __slots__ copied? What about methods? Do you want them copied, too? And do you want to just replace everything on the Bar instance or only update those attributes set on the Foo instance?
I hope you see, what I am getting at. I guess the simplest way is just update the instances __dict__ with values from the other one:
...
class Bar(Foo):
def update_from(self, obj):
self.__dict__.update(obj.__dict__)
foo = Foo()
foo.n = 1
foo.text = "Hi"
bar = Bar()
bar.update_from(foo)
print(bar.n, bar.text) # output: `1 Hi`
And you could of course do that in the __init__ method of Bar, if you wanted. If the initialization of Foo is deterministic and instances keep the initial arguments laying around somewhere, you could instead just call the inherited super().__init__ from Bar.__init__ and pass those initial arguments to it from the instance. Something like this:
class Foo:
def __init__(self, x, y):
self.x = x
self.y = y
self.z = x + y
class Bar(Foo):
def __init__(self, foo_obj):
super().__init__(foo_obj.x, foo_obj.y)
foo = Foo(2, 3)
bar = Bar(foo)
print(bar.z) # output: `5`
I hope this makes things clearer for you.
I have a class
class A:
def sample_method():
I would like to decorate class A sample_method() and override the contents of sample_method()
class DecoratedA(A):
def sample_method():
The setup above resembles inheritance, but I need to keep the preexisting instance of class A when the decorated function is used.
a # preexisting instance of class A
decorated_a = DecoratedA(a)
decorated_a.functionInClassA() #functions in Class A called as usual with preexisting instance
decorated_a.sample_method() #should call the overwritten sample_method() defined in DecoratedA
What is the proper way to go about this?
There isn't a straightforward way to do what you're asking. Generally, after an instance has been created, it's too late to mess with the methods its class defines.
There are two options you have, as far as I see it. Either you create a wrapper or proxy object for your pre-existing instance, or you modify the instance to change its behavior.
A proxy defers most behavior to the object itself, while only adding (or overriding) some limited behavior of its own:
class Proxy:
def __init__(self, obj):
self.obj = obj
def overridden_method(self): # add your own limited behavior for a few things
do_stuff()
def __getattr__(self, name): # and hand everything else off to the other object
return getattr(self.obj, name)
__getattr__ isn't perfect here, it can only work for regular methods, not special __dunder__ methods that are often looked up directly in the class itself. If you want your proxy to match all possible behavior, you probably need to add things like __add__ and __getitem__, but that might not be necessary in your specific situation (it depends on what A does).
As for changing the behavior of the existing object, one approach is to write your subclass, and then change the existing object's class to be the subclass. This is a little sketchy, since you won't have ever initialized the object as the new class, but it might work if you're only modifying method behavior.
class ModifiedA(A):
def overridden_method(self): # do the override in a normal subclass
do_stuff()
def modify_obj(obj): # then change an existing object's type in place!
obj.__class__ = ModifiedA # this is not terribly safe, but it can work
You could also consider adding an instance variable that would shadow the method you want to override, rather than modifying __class__. Writing the function could be a little tricky, since it won't get bound to the object automatically when called (that only happens for functions that are attributes of a class, not attributes of an instance), but you could probably do the binding yourself (with partial or lambda if you need to access self.
First, why not just define it from the beginning, how you want it, instead of decorating it?
Second, why not decorate the method itself?
To answer the question:
You can reassign it
class A:
def sample_method(): ...
pass
A.sample_method = DecoratedA.sample_method;
but that affects every instance.
Another solution is to reassign the method for just one object.
import functools;
a.sample_method = functools.partial(DecoratedA.sample_method, a);
Another solution is to (temporarily) change the type of an existing object.
a = A();
a.__class__ = DecoratedA;
a.sample_method();
a.__class__ = A;
I'd like a particular function to be callable as a classmethod, and to behave differently when it's called on an instance.
For example, if I have a class Thing, I want Thing.get_other_thing() to work, but also thing = Thing(); thing.get_other_thing() to behave differently.
I think overwriting the get_other_thing method on initialization should work (see below), but that seems a bit hacky. Is there a better way?
class Thing:
def __init__(self):
self.get_other_thing = self._get_other_thing_inst()
#classmethod
def get_other_thing(cls):
# do something...
def _get_other_thing_inst(self):
# do something else
Great question! What you seek can be easily done using descriptors.
Descriptors are Python objects which implement the descriptor protocol, usually starting with __get__().
They exist, mostly, to be set as a class attribute on different classes. Upon accessing them, their __get__() method is called, with the instance and owner class passed in.
class DifferentFunc:
"""Deploys a different function accroding to attribute access
I am a descriptor.
"""
def __init__(self, clsfunc, instfunc):
# Set our functions
self.clsfunc = clsfunc
self.instfunc = instfunc
def __get__(self, inst, owner):
# Accessed from class
if inst is None:
return self.clsfunc.__get__(None, owner)
# Accessed from instance
return self.instfunc.__get__(inst, owner)
class Test:
#classmethod
def _get_other_thing(cls):
print("Accessed through class")
def _get_other_thing_inst(inst):
print("Accessed through instance")
get_other_thing = DifferentFunc(_get_other_thing,
_get_other_thing_inst)
And now for the result:
>>> Test.get_other_thing()
Accessed through class
>>> Test().get_other_thing()
Accessed through instance
That was easy!
By the way, did you notice me using __get__ on the class and instance function? Guess what? Functions are also descriptors, and that's the way they work!
>>> def func(self):
... pass
...
>>> func.__get__(object(), object)
<bound method func of <object object at 0x000000000046E100>>
Upon accessing a function attribute, it's __get__ is called, and that's how you get function binding.
For more information, I highly suggest reading the Python manual and the "How-To" linked above. Descriptors are one of Python's most powerful features and are barely even known.
Why not set the function on instantiation?
Or Why not set self.func = self._func inside __init__?
Setting the function on instantiation comes with quite a few problems:
self.func = self._funccauses a circular reference. The instance is stored inside the function object returned by self._func. This on the other hand is stored upon the instance during the assignment. The end result is that the instance references itself and will clean up in a much slower and heavier manner.
Other code interacting with your class might attempt to take the function straight out of the class, and use __get__(), which is the usual expected method, to bind it. They will receive the wrong function.
Will not work with __slots__.
Although with descriptors you need to understand the mechanism, setting it on __init__ isn't as clean and requires setting multiple functions on __init__.
Takes more memory. Instead of storing one single function, you store a bound function for each and every instance.
Will not work with properties.
There are many more that I didn't add as the list goes on and on.
Here is a bit hacky solution:
class Thing(object):
#staticmethod
def get_other_thing():
return 1
def __getattribute__(self, name):
if name == 'get_other_thing':
return lambda: 2
return super(Thing, self).__getattribute__(name)
print Thing.get_other_thing() # 1
print Thing().get_other_thing() # 2
If we are on class, staticmethod is executed. If we are on instance, __getattribute__ is first to be executed, so we can return not Thing.get_other_thing but some other function (lambda in my case)
Consider:
class Parent():
def __init__(self, last_name, eye_color):
self.last_name = last_name
self.eye_color = eye_color
def show_info(self):
print("Last Name - "+self.last_name)
print("Eye Color - "+self.eye_color)
billy_cyrus = Parent("Cyrus", "blue")
The above is from the Udacity Python course. I discovered I'm able to call show_info for instance billy_cyrus using either of the following:
billy_cyrus.show_info()
Parent.show_info(billy_cyrus)
I'm curious as to why. Is there a difference between the two methods? If so when would one be used vs. the other? I'm using Python 3.6 if that matters.
In terms of just calling the method, there is no difference most of the time. In terms of how the underlying machinery, works, there is a bit of a difference.
Since show_info is a method, it is a descriptor in the class. That means that when you access it through an instance in which it is not shadowed by another attribute, the . operator calls __get__ on the descriptor to create a bound method for that instance. A bound method is basically a closure that passes in the self parameter for you before any of the other arguments you supply. You can see the binding happen like this:
>>> billy_cyrus.show_info
<bound method Parent.show_info of <__main__.Parent object at 0x7f7598b14be0>>
A different closure is created every time you use the . operator on a class method.
If you access the method through the class object, on the other hand, it does not get bound. The method is a descriptor, which is just a regular attribute of the class:
>>> Parent.show_info
<function __main__.Parent.show_info>
You can simulate the exact behavior of binding a method before calling it by calling its __get__ yourself:
>>> bound_meth = Parent.show_info.__get__(billy_cyrus, type(billy_cyrus))
>>> bound_meth
<bound method Parent.show_info of <__main__.Parent object at 0x7f7598b14be0>>
Again, this will not make any difference to you in 99.99% of cases, since functionally bound_meth() and Parent.bound_meth(billy_cyrus) end up calling the same underlying function object with the same parameters.
Where it matters
There are a couple of places where it matters how you call a class method. One common use case is when you override a method, but want to use the definition provided in the parent class. For example, say I have a class that I made "immutable" by overriding __setattr__. I can still set attributes on the instance, as in the __init__ method shown below:
class Test:
def __init__(self, a):
object.__setattr__(self, 'a', a)
def __setattr__(self, name, value):
raise ValueError('I am immutable!')
If I tried to do a normal call to __setattr__ in __init__ by doing self.a = a, a ValueError would be raised every time. But by using object.__setattr__, I can bypass this limitation. Alternatively, I could do super().__setattr__('a', a) for the same effect, or self.__dict__['a'] = a for a very similar one.
#Silvio Mayolo's answer has another good example, where you would deliberately want to use the class method as a function that could be applied to many objects.
Another place it matters (although not in terms of calling methods), is when you use other common descriptors like property. Unlike methods, properties are data-descriptors. This means that they define a __set__ method (and optionally __delete__) in addition to __get__. A property creates a virtual attribute whose getter and setter are arbitrarily complex functions instead of just simple assignments. To properly use a property, you have to do it through the instance. For example:
class PropDemo:
def __init__(self, x=0):
self.x = x
#property
def x(self):
return self.__dict__['x']
#x.setter
def x(self, value):
if value < 0:
raise ValueError('Not negatives, please!')
self.__dict__['x'] = value
Now you can do something like
>>> inst = PropDemo()
>>> inst.x
0
>>> inst.x = 3
>>> inst.x
3
If you try to access the property through the class, you can get the underlying descriptor object since it will be an unbound attribute:
>>> PropDemo.x
<property at 0x7f7598af00e8>
On a side note, hiding attributes with the same name as a property in __dict__ is a neat trick that works because data descriptors in a class __dict__ trump entries in the instance __dict__, even though instance __dict__ entries trump non-data-descriptors in a class.
Where it can Get Weird
You can override a class method with an instance method in Python. That would mean that type(foo).bar(foo) and foo.bar() don't call the same underlying function at all. This is irrelevant for magic methods because they always use the former invocation, but it can make a big difference for normal method calls.
There are a few ways to override a method on an instance. The one I find most intuitive is to set the instance attribute to a bound method. Here is an example of a modified billy_cyrus, assuming the definition of Parent in the original question:
def alt_show_info(self):
print('Another version of', self)
billy_cyrus.show_info = alt_show_info.__get__(billy_cyrus, Parent)
In this case, calling the method on the instance vs the class would have completely different results. This only works because methods are non-data descriptors by the way. If they were data descriptors (with a __set__ method), the assignment billy_cyrus.show_info = alt_show_info.__get__(billy_cyrus, Parent) would not override anything but would instead just redirect to __set__, and manually setting it in b
billy_cyrus's __dict__ would just get it ignored, as happens with a property.
Additional Resources
Here are a couple of resources on descriptors:
Python Reference - Descriptor Protocol: http://python-reference.readthedocs.io/en/latest/docs/dunderdsc/
(Official?) Descriptor HowTo Guide: https://docs.python.org/3/howto/descriptor.html
There is no semantic difference between the two. It's entirely a matter of style. You would generally use billy_cyrus.show_info() in normal use, but the fact that the second approach is allowed permits you to use Parent.show_info to get the method as a first-class object itself. If that was not allowed, then it would not be possible (or at least, it would be fairly difficult) to do something like this.
function = Parent.show_info
so_many_billy_cyrus = [billy_cyrus, billy_cyrus, billy_cyrus]
map(function, so_many_billy_cyrus)
How can I quickly disable all methods in a class instance based on a condition? My naive solution is to override using the __getattr__ but this is not called when the function name exists already.
class my():
def method1(self):
print 'method1'
def method2(self):
print 'method2'
def __getattr__(self, name):
print 'Fetching '+str(name)
if self.isValid():
return getattr(self, name)
def isValid(self):
return False
if __name__ == '__main__':
m=my()
m.method1()
The equivalent of what you want to do is actually to override __getattribute__, which is going to be called for every attribute access. Besides it being very slow, take care: by definition of every, that includes e.g. the call to self.isValid within __getattribute__'s own body, so you'll have to use some circuitous route to access that attribute (type(self).isValid(self) should work, for example, as it gets the attribute from the class, not from the instance).
This points to a horrible terminological confusion: this is not disabling "method from a class", but from an instance, and in particular has nothing to do with classmethods. If you do want to work in a similar way on a class basis, rather than an instance basis, you'll need to make a custom metaclass and override __getattribute__ on the metaclass (that's the one that's called when you access attributes on the class -- as you're asking in your title and text -- rather than on the instance -- as you in fact appear to be doing, which is by far the more normal and usual case).
Edit: a completely different approach might be to use a peculiarly Pythonic pathway to implementing the State design pattern: class-switching. E.g.:
class _NotValid(object):
def isValid(self):
return False
def setValid(self, yesno):
if yesno:
self.__class__ = TheGoodOne
class TheGoodOne(object):
def isValid(self):
return True
def setValid(self, yesno):
if not yesno:
self.__class__ = _NotValid
# write all other methods here
As long as you can call setValid appropriately, so that the object's __class__ is switched appropriately, this is very fast and simple -- essentially, the object's __class__ is where all the object's methods are found, so by switching it you switch, en masse, the set of methods that exist on the object at a given time. However, this does not work if you absolutely insist that validity checking must be performed "just in time", i.e. at the very instant the object's method is being looked up.
An intermediate approach between this and the __getattribute__ one would be to introduce an extra level of indirection (which is popularly held to be the solution to all problems;-), along the lines of:
class _Valid(object):
def __init__(self, actualobject):
self._actualobject = actualobject
# all actual methods go here
# keeping state in self._actualobject
class Wrapit(object):
def __init__(self):
self._themethods = _Valid(self)
def isValid(self):
# whatever logic you want
# (DON'T call other self. methods!-)
return False
def __getattr__(self, n):
if self.isValid():
return getattr(self._themethods, n)
raise AttributeError(n)
This is more idiomatic than __getattribute__ because it relies on the fact that __getattr__ is only called for attributes that aren't found in other ways -- so the object can hold normal state (data) in its __dict__, and that will be accessed without any big overhead; only method calls pay the extra overhead of indiretion. The _Valid class instances can keep some or all state in their respective self._actualobject, if any of the state needs to stay accessible on invalid objects (so that the invalid state disable methods, but not data attributes access; it's not clear from your Q if that's needed, but it's a free extra possibility offered by this approach). This idiom is less error-prone than __getattribute__, since state can be accessed more directly in the methods (without triggering validity checks).
As presented, the solution creates a circular reference loop, which may impose a bit of overhead in terms of garbage collection. If that's a problem in your application, use the weakref module from the standard Python library, of course -- that module is generally the simplest way to remove circular loops of references, if and when they're a problem.
(E.g., make the _actualobject attribute of _Valid class instances a weak reference to the object that holds that instance as its _themethods attribute).