We Keep Coding

Best practice for providing optional functions for a class in Python

Currently I am writing a Python program with a plugin system. To develop a new plugin a new class must be created and inherit from a base plugin class. Now it should be possible to add optional functions via mixins. Some mixins provide new functions others access builtin types of the base class and can act with them or change them.
In the following a simplified structure:
import abc
import threading
class Base:
def __init__(self):
self.config = dict()
if hasattr(self, "edit_config"):
self.edit_config()
def start(self):
"""Starts the Plugin"""
if hasattr(self, "loop"):
self._loop()
class AMixin:
def edit_config(self):
self.config["foo"] = 123
class BMixin(abc.ABC):
def _loop(self):
thread = threading.Thread(target=self.loop, daemon=True)
thread.start()
#abc.abstractmethod
def loop(self):
"""Override this method with a while true loop to establish a ongoing loop
"""
pass
class NewPlugin(Base, AMixin, BMixin):
def loop(self):
while True:
print("Hello")
plugin = NewPlugin()
plugin.start()
What is the best way to tackle this problem?
EDIT: I need to make my question more specific. The question is whether the above is the Pythonic way and is it possible to ensure that the mixin are inherited exclusively in conjunction with the Base class. Additionally it would be good in an IDE like VSCode to get support for e.g. autocomplete when accessing builtin types of the Base class, like in AMixin, without inheriting from it of course.

If you want to allow but not require subclasses to define some behaviour in a method called by the base class, the simplest way is to declare the method in the base class, have an empty implementation, and just call the method unconditionally. This way you don't have to check whether the method exists before calling it.
class Base:
def __init__(self):
self.config = dict()
self.edit_config()
def start(self):
self.loop()
def edit_config(self):
pass
def loop(self):
pass
class AMixin:
def edit_config(self):
self.config["foo"] = 123
class NewPlugin(AMixin, Base):
def loop(self):
for i in range(10):
print("Hello")
Note that you have to write AMixin before Base in the list of superclasses, so that its edit_config method overrides the one from Base, and not the other way around. You can avoid this by writing class AMixin(Base): so that AMixin.edit_config always overrides Base.edit_config in the method resolution order.
If you want to require subclasses to implement one of the methods, then you can raise TypeError() instead of pass in the base class's method.

I would move the calls to the methods provided by the mix-ins to __init__ methods defined by those classes.
import abc
import threading
class Base:
def __init__(self, **kwargs):
super.__init__(**kwargs)
self.config = dict()
class AMixin:
def __init__(self, **kwargs):
super().__init__(**kwargs)
self.edit_config()
def edit_config(self):
self.config["foo"] = 123
class BMixin(abc.ABC):
def __init__(self, **kwargs):
super().__init__(**kwargs):
self.loop()
def _loop(self):
thread = threading.Thread(target=self.loop, daemon=True)
thread.start()
#abc.abstractmethod
def loop(self):
"""Override this method with a while true loop to establish a ongoing loop
"""
pass
class NewPlugin(Base, AMixin, BMixin):
pass
When you instantiate a concrete subclass of NewPlugin, Base.__init__, AMixin.__init__, and BMixin.__init__ will be called in that order.

Get decorated class from its name in the decorator?

I decorated some methods with #bot_thinking, which stores some information about the decorated method in the functions attribute.
One piece of information is 'class_name', but my program needs the class type as a variable, e.g. RandomBot. I would like to get this class.
Here is some sample code:
class DepthPrunedMinimaxAgent(Agent):
#bot_thinking(associated_name="minimax profondeur")
def select_move(self, game_state: GameState):
Above is the decorated part of the code.
The decorator:
functions = {}
def bot_thinking(associated_name, active=True):
def _(func):
if active:
class_name = func.__qualname__.rsplit('.')[-2]
import sys
# class_name_2=getattr(sys.modules[__name__], class_name)
# module=importlib.import_module('sources.agent')
functions[associated_name] = (associated_name, class_name,
globals()[class_name], func)
else:
functions.pop(associated_name)
return _
bot_thinking isn't a real decorator, it's a decorator factory.
From the func function, I get the class_name, but I can't use the accepted answer by #m.kocikowski, to find the correct class because this class is decorated, so it already imports the annotation module, so importing from the module of the annotation the annotated module would result in a cyclic import, which python does not seem to permit.
Do you see a method to get the class from its name?
ps:
ps:
to be clearer : the annotation part of the code need an import to the annotated classes(to retrieve the class from its name), which also need an importation of the annotation (for the annotation to work).

You can do what you want if you use a descriptor class, rather than a function, as the decorator, at least if you're using Python 3.6 or newer. That's because there's a new method added to the descriptor protocol, __set_name__. It gets called when the descriptor object is saved as a class variable. While most descriptors will use it to record the name they're being saved as, you can use it to get the class you're in.
You do need to make your decorator object wrap the real function (implementing calling and descriptor lookup methods), rather than being able to return the unmodified function you were decorating. Here's my attempt at a quick and dirty implementation. I don't really understand what you're doing with functions, so I may not have put the right data in it, but it should be close enough to get the idea across (owner is the class the method stored in).
functions = {}
def bot_thinking(associated_name, active=True):
class decorator:
def __init__(self, func):
self.func = func
def __set_name__(self, owner, name):
if active:
functions[associated_name] = (associated_name, owner.__name__,
owner, self.func)
else:
functions.pop(associated_name)
def __get__(self, obj, owner):
return self.func.__get__(obj, owner)
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
return decorator

The problem is the class hasn't been defined yet when the bot_thinking() decorator factory (and decorator itself) are executing. The only workaround I can think of would be to patch things up after the class is defined, as illustrated below:
from pprint import pprint, pformat
functions = {}
def bot_thinking(associated_name, active=True):
def _(func):
if active:
class_name = func.__qualname__.split(".")[-2]
functions[associated_name] = (associated_name, class_name, class_name, func)
else:
functions.pop(associated_name, None)
return func # Decorators must return a callable.
return _
class Agent: pass
class GameState: pass
class DepthPrunedMinimaxAgent(Agent):
#bot_thinking(associated_name="minimax profondeur")
def select_move(self, game_state: GameState):
pass
# After class is defined, update data put into functions dictionary.
for associated_name, info in functions.items():
functions[associated_name] = (info[0], info[1], globals()[info[2]], info[3])
pprint(functions)
Output:
{'minimax profondeur': ('minimax profondeur',
'DepthPrunedMinimaxAgent',
<class '__main__.DepthPrunedMinimaxAgent'>,
<function DepthPrunedMinimaxAgent.select_move at 0x00F158A0>)}

Call another method in a class when the given method does not exist?

Say I have a class which contains several functions.
class Person:
def __init__(self): pass
def say(self, speech): pass
def walk(self, destination): pass
def jump(self): pass
When the user instantiates a Person, I'd like them to be able to call any method of the class. If the requested method does not exist (e.g. Person.dance()), a default function should be called instead.
I imagine that this could be done via a theoretical magic method -
class Person:
def __init__(self): pass
def say(self, speech): pass
def walk(self, destination): pass
def jump(self): pass
def sleep(self): print("Zzz")
def __method__(self, func):
if func.__name__ not in ['say','walk','jump']:
return self.sleep
else
return func
billy = Person()
billy.dance()
>> "Zzz"
However, I know of no such magic method.
Is there a way to make non-existent methods within a class redirect to another class?
It's important that the end-user doesn't have to do anything - from their perspective, it should just work.

The standard way to catch an undefined attribute is to use __getattr__:
# But see the end of the answer for an afterthought
def __getattr__(self, attr):
return self.sleep
Python does not differentiate between "regular" attributes and methods; a method call starts with an ordinary attribute lookup, whose result just happens to be callable. That is,
billy.run()
is the same as
f = billy.run
f()
This means that __getattr__ will be invoked for any undefined attribute; there is no way to tell at lookup time whether the result is going to be called or not.
However, if all you want is to define "aliases" for a common method, you can do that with a loop after the class statement.
class Person:
def __init__(self): pass
def say(self, speech): pass
def walk(self, destination): pass
def jump(self): pass
def do_all(self): pass
for alias in ["something", "something_else", "other"]:
setattr(Person, alias, Person.do_all)
You can also make hard-coded assignments in the class statement, but that would be unwieldy if there are, as you mention, hundreds of such cases:
class Person:
def do_all(self): pass
something = do_all
something_else = do_all
(I did not experiment with using exec to automate such assignments in a loop; it might be possible, though not recommended.)
You can also embed the list of aliases in the definition of __getattr__, come to think of it:
def __getattr__(self, attr):
if attr in ["something", "something_else", "other"]:
return self.sleep
else:
raise AttributeError(f"type object 'Person' has no attribute '{attr}'")

Your users might find the API behavior confusing. However, if you're sure you need this pattern, you can try something like
# getattr will get the function attribute by a string name version
# None is returned if no function is found
my_func = getattr(self, 'my_func', None)
# callable ensures `my_func` is actually a function and not a generic attribute
# Add your if-else logic here
if callable(my_func):
my_func(args)
else:
...

You could nest your "default" function inside __getattr__ in order to gain access to the called non-existent method's name and arguments.
class Test:
def __getattr__(self, attr):
def default(*args, **kwargs):
return attr, args, kwargs
return default
test = Test()
print(test.test('test'))
# ('test', ('test',), {})

Python tracking sub-classes that are in scope

I'm trying to write a tracker class where the instances of the tracker class track the sub-classes of another class that are in the scope of the tracker instance.
More concretely, the following is an example of what I am trying to achieve:
class Foo(object): pass
class FooTracker(object):
def __init__(self):
# use Foo.__subclasses__() or a metaclass to track subclasses
# - but how do I filter this to only get the ones in scope?
self.inscope = <something magic goes here>
ft1 = FooTracker()
assert ft1.inscope == []
class Bar(Foo): pass
ft2 = FooTracker()
assert ft2.inscope == [<class '__main__.Bar'>]
def afunction():
class Baz(Foo): pass # the global definition of Bar is now hidden
class Bar(Foo): pass
ft3 = FooTracker()
assert (set(ft3.inscope) == set([<class '__main__.afunction.<locals>.Baz'>,
<class '__main__.afunction.<locals>.Bar'>])
ft4 = FooTracker() # afunction.Baz and afunction.Bar are no longer in scope
assert ft4.inscope == [<class '__main__.Bar'>]
So I want the instances of FooTracker to track the sub-classes of Foo that were in scope at the time the FooTracker object was created.
I've tried a few different things, such as parsing the qualified names of the Foo sub-classes and using exec() to do the name resolution but the fundamental problem is that it always works out the sub-classes relative to the scope within FooTracker.__init__() and not where it was called.
My only other thought was to try something with inspect.currentframe() but even if this were possible it would probably be too much of a hack and would make the code too brittle (e.g., there is a comment in the docs that not all Python implementations will have frame support in the interpreter").

There's no easy way to do exactly what you're asking for. But you might be able to use some Python features to get something with a roughly similar API, without as much hassle.
One option would be to require each subclass to be decorated with a method of your Tracker class. This would make it really easy to keep track of them, since you'd just append each caller of the method to a list:
class Tracker:
def __init__(self):
self.subclasses = []
def register(self, cls):
self.subclasses.append(cls)
return cls
class Foo(): pass
foo_tracker = Tracker()
#foo_tracker.register
class FooSubclass1(Foo): pass
#foo_tracker.register
class FooSubclass2(Foo): pass
print(foo_tracker.subclasses)
This doesn't actually require that the classes being tracked are subclasses of Foo, all classes (and even non-class objects) can be tracked if you pass them to the register method. Decorator syntax makes it a little nicer than just appending each class to a list after you define it, but not by a whole lot (you still repeat yourself a fair amount, which may be annoying unless you make the tracker and method names very short).
A slightly trickier version might get passed the base class, so that it would detect subclasses automatically (via Foo.__subclasses__). To limit the subclasses it detects (rather than getting all subclasses of the base that have ever existed), you could make it behave as a context manager, and only track new subclasses defined within a with block:
class Tracker:
def __init__(self, base):
self.base = base
self._exclude = set()
self.subclasses = set()
def __enter__(self):
self._exclude = set(self.base.__subclasses__())
return self
def __exit__(self, *args):
self.subclasses = set(self.base.__subclasses__()) - self._exclude
return False
class Foo(): pass
class UntrackedSubclass1(Foo): pass
with Tracker(Foo) as foo_tracker:
class TrackedSubclass1(Foo): pass
class TrackedSubclass2(Foo): pass
class UntrackedSubclass2(Foo): pass
print(foo_tracker.subclasses)
If you're using Python 3.6 or later, you can do the tracking a different way by injecting an __init_subclass__ class method into the tracked base class, rather than relying upon __subclasses__. If you don't need to support class hierarchies that are already using __init_subclass__ for their own purposes (and you don't need to support nested trackers), it can be quite elegant:
class Tracker:
def __init__(self, base):
self.base = base
self.subclasses = []
def __enter__(self):
#classmethod
def __init_subclass__(cls, **kwargs):
self.subclasses.append(cls)
self.base.__init_subclass__ = __init_subclass__
return self
def __exit__(self, *args):
del self.base.__init_subclass__
return False
class Foo(): pass
class UntrackedSubclass1(Foo): pass
with Tracker(Foo) as foo_tracker:
class TrackedSubclass1(Foo): pass
class TrackedSubclass2(Foo): pass
class UntrackedSubclass2(Foo): pass
print(foo_tracker.subclasses)
One nice feature of this version is that it automatically tracks deeper inheritance hierarchies. If a subclass of a subclass is created within the with block, that "grandchild" class will still be tracked. We could make the previous __subclasses__ based version work this way too, if you wanted, by adding another function to recursively expand out the subclasses of each class we find.
If you do want to play nice with existing __init_subclass__ methods, or want to be able to nest trackers, you need to make the code a bit more complicated. Injecting a well behaved classmethod in a reversible way is tricky since you need handle both the case where the base class has its own method, and the case where it's inheriting a version from its parents.
class Tracker:
def __init__(self, base):
self.base = base
self.subclasses = []
def __enter__(self):
if '__init_subclass__' in self.base.__dict__:
self.old_init_subclass = self.base.__dict__['__init_subclass__']
else:
self.old_init_subclass = None
#classmethod
def __init_subclass__(cls, **kwargs):
if self.old_init_subclass is not None:
self.old_init_subclass.__get__(None, cls)(**kwargs)
else:
super(self.base, cls).__init_subclass__(**kwargs)
self.subclasses.append(cls)
self.base.__init_subclass__ = __init_subclass__
return self
def __exit__(self, *args):
if self.old_init_subclass is not None:
self.base.__init_subclass__ = self.old_init_subclass
else:
del self.base.__init_subclass__
return False
class Foo:
def __init_subclass__(cls, **kwargs):
super().__init_subclass__(**kwargs)
print("Foo!")
class Bar(Foo): pass # every class definition from here on prints "Foo!" when it runs
with Tracker(Bar) as tracker1:
class Baz(Bar): pass
with Tracker(Foo) as tracker2:
class Quux(Foo): pass
with Tracker(Bar) as tracker3:
class Plop(Bar): pass
# four Foo! lines will have be printed by now by Foo.__init_subclass__
print(tracker1.subclasses) # will describe Baz and Plop, but not Quux
print(tracker2.subclasses) # will describe Quux and Plop
print(tracker3.subclasses) # will describe only Plop

Refer to a class outside its method?

I need to deliver something like this in my program
class the_class_name(Parent):
the_attribute = self.parent_class_method()
#the parent class method will return a value
#but I cannot use self here since there's no self
How can I carry this out? Is there any other alternative that can do the job for me?
I have tried using __init__ like this:
def __init__(self):
Parent.__init__(self)
self.attribute = self.the_method()
But then I have problem creating the object, it won't receive any parameters that the Parent class normally receives anymore

Sounds like you are looking for __init__:
class TheClassName(Parent):
def __init__(self):
# Set attribute to the result of the parent method
self.attribute = super(TheClassName, self).the_method()
EDIT
If your parent class has parameters in it's own __init__ function, include them in the child class:
class Parent(object):
def __init__(self, foo, bar):
...
#classmethod
def the_method(cls):
...
class TheClassName(Parent):
def __init__(self, foo, bar):
super(TheClassName, self).__init__(foo, bar)
self.attribute = super(TheClassName, self).the_method()
I don't quite understand why you don't just call the parent method on your child object when you need the value though.

There is no self at that point of the creation of the subclass, nor is there an instance of the Parent class. That means the only Parent class methods you could call would have to be either static or class methods.
To demonstrate:
class Parent(object):
#staticmethod
def static_method():
return 42
#classmethod
def class_method(cls):
return 43
class TheClassName(Parent):
the_attribute = Parent.static_method()
another_attribute = Parent.class_method()
print(TheClassName.the_attribute) # -> 42
print(TheClassName.another_attribute) # -> 43

You must use class methods, declared with the #classmethod decorator, or a #staticmethod. The #classmethod decorator is preferable so that inheritance is handled correctly, i.e. the method is invoked on the derived class (a bit of a technicality, if you are still learning this).
class Alpha(object):
#classmethod
def method1(cls):
return 'method1 has been called on {}'.format(cls)
class Beta(Alpha):
def __init__(self):
self.myattr = Beta.method1()
print(Beta().myattr)
method1 has been called on class <'__main__.Beta'>

Use
super(ClassName, self).methodname(arg)
inside a method
def child_method(self, arg):
super(ClassName, self).methodname(arg)
You cannot use self outside a method.