My python module uses some functions from another module, but I have several implementations of that module interface. How to point out, which one to use?
Simple example:
A.py:
import B
def say_hi()
print "Message: " + B.greeting()
main.py:
import A(B=my_B_impl)
A.say_hi()
my_B_impl.py:
def greeting():
return "Hallo!"
output:
Message: Hallo!
In python this could be most elegantly done with inheritance:
A.py:
import B
class SayHi(object):
b = B
def say_hi(self):
print "Message: " + self.b.greeting()
my_B_impl.py:
class AlternativeHi(object):
def greeting(self):
return "Hallo!"
main.py:
import A
from my_B_impl.py import AlternativeHi
class MyHi(SayHi):
b=AlternativeHi
a=MyHi()
MyHi.say_hi()
output:
Message: Hallo!
You can also use the factory pattern to avoid explicit declaration of class AlternativeHi and MyHi:
A.py
from B import greeting
class SayHi(object):
def __init__(self,*args,**kwargs):
self.greeting = greeting
def say_hi(self):
print "Message: " + self.greeting()
def hi_factory(func):
class CustomHi(SayHi):
def __init__(self,*args,**kwargs):
result = super(CustomHi, self).__init__(*args, **kwargs)
self.greeting = func
return CustomHi
my_B_impl.py:
def greeting(self):
return "Hallo!"
main.py:
form A import hi_factory
from my_B_impl import greeting
a = hi_factory(greeting)
a.say_hi()
What you ask is not directly possible. There is no parameterisation capability built in to Python's module system. If you think about it, it's not clear how such a proposal ought to work: if modules A and B both import module M, but they supply different parameters, which parameter is used when M is imported? Is it imported twice? What would that mean for module-level configuration (as in logging)? It gets worse if a third module C attempts to import M without parameters. Also, the "open-world" idea that you could override any import statement from the outside violates the language-design principle that "the code you wrote is the code that ran".
Other languages have incorporated parameterised modules in a variety of ways (compare Scala's object model, ML's modules and signatures, and - stretching it - C++'s templates), but it's not clear that such a feature would be a good fit for Python. (That said, you could probably hack something resembling parameterised modules using importlib if you were determined and masochistic enough.)
Python does have very powerful and flexible capabilities for dynamic dispatch, however. Python's standard, day-to-day features like functions, classes, parameters and overriding provide the basis for this support.
There are lots of ways to cut the cake on your example of a function whose behaviour is configurable by its client.
A function parameterised by a value:
def say_hi(greeting):
print("Message: " + greeting)
def main():
say_hi("Hello")
A class parameterised by a value:
class Greeter:
def __init__(self, greeting):
self.greeting = greeting
def say_hi(self):
print("Message: " + self.greeting)
def main():
Greeter("Hello").say_hi()
A class with a virtual method:
class Greeter:
def say_hi(self):
print("Message: " + self.msg())
class MyGreeter(Greeter):
def msg(self):
return "Hello"
A function parameterised by a function:
def say_hi(greeting):
print("Message: " + greeting())
def make_greeting():
return "Hello"
def main():
say_hi(make_greeting)
There are more options (I'm avoiding the Java-y example of objects invoking other objects) but you get the idea. In each of these cases, the selection of the behaviour (the passing of the parameter, the overriding of the method) is decoupled from the code which uses it and could be put in a different file. The right one to choose depends on your situation (though here's a hint: the right one is always the simplest one that works).
Update: in a comment you mention that you'd like an API which sets up the dependency at the module-level. The main problem with this is that the dependency would be global - modules are singletons, so anyone who imports the module has to use the same implementation of the dependency.
My advice is to provide an object-oriented API with "proper" (per-instance) dependency injection, and provide top-level convenience functions which use a (configurable) "default" set-up of the dependency. Then you have the option of not using the globally-configured version. This is roughly how asyncio does it.
# flexible object with dependency injection
class Greeter:
def __init__(self, msg):
self.msg = msg
def say_hi(self):
print("Message: " + self.msg)
# set up a default configuration of the object to be used by the high-level API
_default_greeter = Greeter("Hello")
def configure(msg):
global _default_greeter
_default_greeter = Greeter(msg)
# delegate to whatever default has been configured
def say_hi():
_default_greeter.say_hi()
Related
I have a factory as shown in the following code:
class ClassFactory:
registry = {}
#classmethod
def register(cls, name):
def inner_wrapper(wrapped_class):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = wrapped_class
return wrapped_class
return inner_wrapper
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
#ClassFactory.register('Class 1')
class M1():
def __init__(self):
print ("Starting Class 1")
#ClassFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
This works fine and when I do
if __name__ == '__main__':
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
I get the expected result of dict_keys(['Class 1', 'Class 2']) Starting Class 2
Now the problem is that I want to isolate classes M1 and M2 to their own files m1.py and m2.py, and in the future add other classes using their own files in a plugin manner.
However, simply placing it in their own file
m2.py
from test_ import ClassFactory
#MethodFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
gives the result dict_keys(['Class 1']) since it never gets to register the class.
So my question is: How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class? How to self register in this way? Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Thanks
How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class?
I'm playing around with a similar problem, and I've found a possible solution. It seems too much of a 'hack' though, so set your critical thinking levels to 'high' when reading my suggestion below :)
As you've mentioned in one of your comments above, the trick is to force the loading of the individual *.py files that contain individual class definitions.
Applying this to your example, this would involve:
Keeping all class implementations in a specific folders, e.g., structuring the files as follows:
.
└- factory.py # file with the ClassFactory class
└─ classes/
└- __init__.py
└- m1.py # file with M1 class
└- m2.py # file with M2 class
Adding the following statement to the end of your factory.py file, which will take care of loading and registering each individual class:
from classes import *
Add a piece of code like the snippet below to your __init__.py within the classes/ foder, so that to dynamically load all classes [1]:
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
If I then run your test code, I get the desired result:
# test.py
from factory import ClassFactory
if __name__ == "__main__":
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
$ python test.py
dict_keys(['Class 1', 'Class 2'])
Starting Class 2
Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Unfortunately, I'm not experienced enough to answer this question. However, when searching for answers to this problem, I've came across the following sources that may be helpful to you:
[2] : this presents a method for registering class existence based on Python Metaclasses. As far as I understand, it relies on the registering of subclasses, so I don't know how well it applies to your case. I did not follow this approach, as I've noticed that the new edition of the book suggests the use of another technique (see bullet below).
[3], item 49 : this is the 'current' suggestion for subclass registering, which relies on the definition of the __init_subclass__() function in a base class.
If I had to apply the __init_subclass__() approach to your case, I'd do the following:
Add a Registrable base class to your factory.py (and slightly re-factor ClassFactory), like this:
class Registrable:
def __init_subclass__(cls, name:str):
ClassFactory.register(name, cls)
class ClassFactory:
registry = {}
#classmethod
def register(cls, name:str, sub_class:Registrable):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = sub_class
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
from classes import *
Slightly modify your concrete classes to inherit from the Registrable base class, e.g.:
from factory import Registrable
class M2(Registrable, name='Class 2'):
def __init__(self):
print ("Starting Class 2")
I am currently using argparse in a class much like this:
class MyClass:
P_OPT = "my-opt"
"""This is an option to my script."""
P_OPT_HELP = "This is an option to my script"
"""Description for argparse"""
def __init__(self, **kwargs):
# do stuff here
pass
#classmethod
def parse_arguments(cls):
parser = argparse.ArgumentParser()
parser.add_argument('--' + cls.P_OPT, help=cls.P_OPT_HELP)
def main():
MyClass(**MyClass.parse_arguments())
Works fine, but I have a lot of options and arguments, there is redundancy between the P_OPT docstring and the content of P_OPT_HELP. Also I have to make a docstring for P_OPT_HELP too otherwise my project linter will remain unhappy. PEP 224 decided that there would be no built-in way to get to class and instance attributes docstring at runtime.
Is there a module or a clever python trick to do this ?
What I am considering so far is just using dict, it does not prevent the redundancy but does not pollute my class namespace so much:
P_OPT = {"name": "my-opt", "help": "This is an option to my script"}
"""This is an option to my script"""
And then:
parser.add_argument('-' + P_OPT["name"], help=P_OPT["help"])
My Python package depends on an external library for a few of it's functions. This is a non-Python package and can be difficult to install, so I'd like users to still be able to use my package but have it fail when using any functions that depend on this non-Python package.
What is the standard practice for this? I could only import the non-Python package inside the methods that use it, but I really hate doing this
My current setup:
myInterface.py
myPackage/
--classA.py
--classB.py
The interfaces script myInterface.py imports classA and classB and classB imports the non-Python package. If the import fails I print a warning. If myMethod is called and the package isn't installed there will be some error downstream but I do not catch it anywhere, nor do I warn the user.
classB is imported every time the interface script is called so I can't have anything fail there, which is why I included the pass. Like I said above, I could import inside the method and have it fail there, but I really like keeping all of my imports in one place.
From classB.py
try:
import someWeirdPackage
except ImportError:
print("Cannot import someWeirdPackage")
pass
class ClassB():
...
def myMethod():
swp = someWeirdPackage()
...
If you are only importing one external library, I would go for something along these lines:
try:
import weirdModule
available = True
except ImportError:
available = False
def func_requiring_weirdmodule():
if not available:
raise ImportError('weirdModule not available')
...
The conditional and error checking is only needed if you want to give more descriptive errors. If not you can omit it and let python throw the corresponding error when trying to calling a non-imported module, as you do in your current setup.
If multiple functions do use weirdModule, you can wrap the checking into a function:
def require_weird_module():
if not available:
raise ImportError('weirdModule not available')
def f1():
require_weird_module()
...
def f2():
require_weird_module()
...
On the other hand, if you have multiple libraries to be imported by different functions, you can load them dynamically. Although it doesn't look pretty, python caches them and there is nothing wrong with it. I would use importlib
import importlib
def func_requiring_weirdmodule():
weirdModule = importlib.import_module('weirdModule')
Again, if multiple of your functions import complicated external modules you can wrap them into:
def import_external(name):
return importlib.import_module(name)
def f1():
weird1 = import_external('weirdModule1')
def f2():
weird2 = import_external('weirdModule2')
And last, you could create a handler to prevent importing the same module twice, something along the lines of:
class Importer(object):
__loaded__ = {}
#staticmethod
def import_external(name):
if name in Importer.__loaded__:
return Importer.__loaded__[name]
mod = importlib.import_module(name)
Importer.__loaded__[name] = mod
return mod
def f1():
weird = Importer.import_external('weird1')
def f2():
weird = Importer.import_external('weird1')
Although I'm pretty sure that importlib does caching behing the scenes and you don't really need for manual caching.
In short, although it does look ugly, there is nothing wrong with importing modules dynamically in python. In fact, a lot of libraries rely on this. On the other hand, if it is just for an special case of 3 methods accessing 1 external function, do use your approach or my first one in case you cant to add custom sception handling.
I'm not really sure that there's any best practice in this situation, but I would redefine the function if it's not supported:
def warn_import():
print("Cannot import someWeirdPackage")
try:
import someWeirdPackage
external_func = someWeirdPackage
except ImportError:
external_func = warn_import
class ClassB():
def myMethod(self):
swp = external_func()
b = ClassB()
b.myMethod()
You can create two separate classes for the two cases. The first will be used when the the package exist . The second will used when the package does not exist.
class ClassB1():
def myMethod(self):
print("someWeirdPackage exist")
# do something
class ClassB2(ClassB1):
def myMethod(self):
print("someWeirdPackage does not exist")
# do something or raise Exception
try:
import someWeirdPackage
class ClassB(ClassB1):
pass
except ImportError:
class ClassB(ClassB2):
pass
You can also use given below approach to overcome the problem that you're facing.
class UnAvailableName(object):
def __init__(self, name):
self.target = name
def __getattr_(self, attr):
raise ImportError("{} is not available.".format(attr))
try:
import someWeirdPackage
except ImportError:
print("Cannot import someWeirdPackage")
someWeirdPackage = someWeirdPackage("someWeirdPackage")
class ClassB():
def myMethod():
swp = someWeirdPackage.hello()
a = ClassB()
a.myMethod()
Context
I'm trying to implement some variant of strategy pattern in Python 2.7.
I want to be able to instantiate a 'my_strategy' base class, but switch between different implementations of a 'score' method at run-time.
I will have many common methods in 'my_strategy' but a bunch of 'score' implementations.
The main illustrates how I want to use it.
Here the scoring implementation is dummy of course.
What I tried (i.e. My code so far)
strategy.py:
from algo_one import *
#from algo_two import *
class my_strategy ( object ):
def __init__(self, candidate = ""):
self.candidate = candidate
self.method = 'default'
self.no = 10
self._algo = algo_one
def set_strategy(self, strategy='default'):
self.strategy = strategy
if self.strategy == 'algo_one':
self._algo = algo_one
elif self.strategy == 'algo_two':
# self._algo = algo_two
pass
else:
self._algo = None
def score(self, *args):
if len(args) > 0:
self.candidate = args[0]
self._algo.score(self.candidate)
if __name__ == "__main__":
s = my_strategy()
s.strategy = 'algo_one'
s.candidate = "hello world"
print s.score()
print s.score("hi")
# s.set_method('algo_two')
# print s.score("hi")
I want to save the selected strategy in some sort of private pointer to the sub-class method.
algo_one.py:
from strategy import my_strategy
class algo_one ( my_strategy ):
def score(self, candidate):
return len(candidate)*self.no
I could have a class-less method, but later I'll need to access public variables of the base class.
algo_two.py:
from strategy import my_strategy
class algo_two ( my_strategy ):
def score(self, candidate):
return len(candidate)*3
I have an empty init.py too.
The errors
1.
in score self._algo.score(self.candidate)
TypeError: unbound method score() must be called with algo_one
instance as first argument (got str instance instead)
2.
If I uncomment the import of the second strategy:
from algo_two import *
I get the following error.
ImportError: cannot import name my_strategy
My guess is that I run into some sort of circular dependency.
3.
from algo_one import *
This is obviously not pretty (unable to detect undefined names), but if I
from algo_one import algo_one
I get
ImportError: cannot import name algo_one
Question
I think the errors are intertwined and that my approach, as a whole, may be flawed. If not just addressing the error, I'm looking for suggestions to improve the design. Or any comment, really. Also I'm open to suggestions regarding the title of this question. Thank you!
You make it much more complicated than it needs to be. Python functions are first class objects so the simplest way to implement the strategy pattern in Python is to pass a 'strategy' function to your "context" object (the one that uses the strategy). The fine part is that any callable object (ie: any object implementing the __call__ method) will work.
def default_score_strategy(scorer):
return len(scorer.candidate) * 3
def universal_answer_score_strategy(scorer):
return 42 # definitly the universal answer <g>
class ComplicatedStrategy(object):
def __init__(self, factor):
self.factor = factor
def __call__(self, scorer):
return len(scorer.candidate) * self.factor
class Scorer(object):
def __init__(self, candidate="", strategy=default_score_strategy):
self.candidate = candidate
self.strategy = strategy
def score(self):
return self.strategy(self)
s1 = Scorer("foo")
s2 = Scorer("bar", strategy=universal_answer_score_strategy)
s3 = Scorer("baaz", strategy=ComplicatedStrategy(365))
Note that your strategies dont have to be in the same module as the Scorer class (well, except the default one of course), and that the module containing the Scorer class doesn't have to import the stratgeies modules - nor know anything about where the strategies are defined:
# main.py
from mylib.scores import Scorer
from myapp.strategies import my_custom_strategy
s = Scorer("yadda", my_custom_strategy)
You don't instantiate your algo object in the __init__ method. Remember, to instantiate a class object, you need to call it:
self._algo = algo_one()
Yes, that's a circular dependency. I don't see however why algo_one and algo_two need to inherit from my_strategy at all. Just make them plain objects, or inherit a base class stored somewhere else. Or, keep them all in the same file - there's no reason to necessarily have classes in separate files in Python.
This is the same problem as 2.
One of your main problems are that your algorithms try to subclass from your base class, which is a huge design flaw (you already noticed that). Use a simple method binding instead, which deals with all the necessary things:
def algo_one(candidate):
# do stuff
return "A fluffy unicorn"
def algo_two(candidate):
# do some other stuff
return "Awesome rabbits"
# not really necessary, just to make it easier to add new algorithms
STRATEGIES = { "one": algo_one, "two": algo_two }
class Strategy(object):
def __init__(self):
...
def set_strategy(self, which):
if which not in STRATEGIES:
raise ValueError("'%s' is an unknown strategy" % which)
# compatibility checks about the entries in STRATEGIES omitted here
self._algo = STRATEGIES[which]
def score(self, *args):
# ...
return self._algo(...)
If you need a more complex approach (this however depends on your requirements), in which everyone knows about each other, split the algorithms and strategy chooser into different classes referencing each other (shortened version below):
class ScoreAlgo(object):
def __init__(self, parent):
self._strategy = parent # if you need a back-reference, just be aware of circular dependencies in the garbage collection
def __del__(self):
self._strategy = None # resolve circular dependency for the GC
def score(self, candidate):
return None
class Strategy(object):
def __init__(self):
...
def set_strategy(self, ...):
...
self._algo = ScoreAlgo(self)
def score(self, ...):
return self._algo.score(...)
(If you need a huge variety of algorithms, you should make ScoreAlgo an abstract base class, for which subclasses have to implement the score() method).
You also could use a mixin pattern (which is a bit more formal than the method binding) or several other ways. This however depends on your overall requirements.
EDIT: I just added a returnto both def score(): stubs to avoid confusion about why those might not return anything.
I'm currently writing some kind of tiny api to support extending module classes. Users should be able to just write their class name in a config and it gets used in our program. The contract is, that the class' module has a function called create(**kwargs) to return an instance of our base module class, and is placed in a special folder. But the isinstance check Fails as soon as the import is made dynamically.
modules are placed in lib/services/name
module base class (in lib/services/service)
class Service:
def __init__(self, **kwargs):
#some initialization
example module class (in lib/services/ping)
class PingService(Service):
def __init__(self, **kwargs):
Service.__init__(self,**kwargs)
# uninteresting init
def create(kwargs):
return PingService(**kwargs)
importing function
import sys
from lib.services.service import Service
def doimport( clazz, modPart, kw, class_check):
path = "lib/" + modPart
sys.path.append(path)
mod = __import__(clazz)
item = mod.create(kw)
if class_check(item):
print "im happy"
return item
calling code
class_check = lambda service: isinstance(service, Service)
s = doimport("ping", "services", {},class_check)
print s
from lib.services.ping import create
pingService = create({})
if isinstance(pingService, Service):
print "why this?"
what the hell am I doing wrong
here is a small example zipped up, just extract and run test.py without arguments
zip example
The problem was in your ping.py file. I don't know exactly why, but when dinamically importing it was not accepting the line from service import Service, so you just have to change it to the relative path: from lib.services.service import Service. Adding lib/services to the sys.path could not make it work the inheritance, which I found strange...
Also, I am using imp.load_source which seems more robust:
import os, imp
def doimport( clazz, modPart, kw, class_check):
path = os.path.join('lib', modPart, clazz + '.py')
mod = imp.load_source( clazz, path )
item = mod.create(kw)
if class_check(item):
print "im happy"
return item