I have an existing python (python v2.7) application that imports external py files on the fly which contain specifically named classes to processes data. The external py file loaded is chosen based on the type of post-processing of the data that is needed.
So I have this collection of classes, each in their own file. The files are named in a specific fashion based on the type of processing so that the main program knows what file to import from the upstream request.
Keep in mind that I and others are always tweaking these class files, but we can not change the code on the main application.
What I would like to do is to import a "template" of the common functions into the class scope which can provide the standard set of controls that the main program expects without needing to copy/paste them into each file. I hate it when I find a bug and make a correction in one of these main class i/o function which I then have to replicate in thirty-some other files.
Now, I understand from googling that my import here is bad... I get the message:
TestClassFile.py:5: SyntaxWarning: import * only allowed at module level
But this method is the only way I have found to import the functions so that they come into the namespace of the class itself. I have an example below...
What method (if any) is the appropriate way to do this in Python?
Example
main.py
import TestClassFile
print "New TestClass - Init"
oTest = TestClassFile.TestClass("foo")
print "Should run... Function A"
oTest.funcA()
print "Should run... Function b"
oTest.funcB()
TestClassFile.py
class TestClass:
from TestClassImport import *
def __init__(self, str):
print "INIT! and be ... ", str
def funcA(self):
print "Function A"
TestClassImport.py
def funcB(self):
print "Function B"
Much appreciated!
Update
Many thanks to everyone for the contributions. From researching MixIns, these appear to be the proper python way to extend a class.
TestClassImport.py
class ImportClass:
def funcB(self):
print "Function B"
TestClassFile.py
from TestClassImport import ImportClass
class TestClass(ImportClass):
def __init__(self, str):
print "INIT! and be ... ", str
def funcA(self):
print "Function A"
It sounds like you should make the imported functions into mixins, which you can inherit from. So:
TestClassImport.py
class ClassB(object):
def funcB(self):
print "Function B"
TestClassFile.py
from TestClassImport import ClassB
from OtherClassImport import ClassX
class TestClass(ClassB, ClassX):
...
This appears to work:
import types
from TestClassImport import funcB
class TestClass:
def __init__(self, str):
print "INIT! and be ... ", str
setattr(self, 'funcB', types.MethodType(funcB, self, TestClass))
def funcA(self):
print "Function A"
When I run it I get the following output:
INIT! and be ... foo
Should run... Function A
Function A
Should run... Function b
Function B
I don't know if this is by any means a good solution, but you can write a function to construct a metaclass to dynamically add properties to your classes.
def make_meta(*import_classes):
class TestMeta(type):
def __new__(meta, name, bases, dct):
new_class = super(TestMeta, meta).__new__(meta, name, bases, dct)
for import_class in import_classes:
for name in vars(import_class):
if not name.startswith('__'):
prop = getattr(import_class, name)
setattr(new_class, name, prop)
return new_class
return TestMeta
class TestClass:
import TestClassImport
__metaclass__ = make_meta(TestClassImport)
# other functions defined as normal...
This will add everything in the global scope of TestClassImport.py that doesn't start with '__' as a property on TestClass.
Or, you can use a class decorator to add properties dynamically in the same fashion.
def add_imports(*import_classes):
def augment_class(cls):
for import_class in import_classes:
for name in vars(import_class):
if not name.startswith('__'):
prop = getattr(import_class, name)
setattr(cls, name, prop)
return cls
return augment_class
import TestClassImport
#add_imports(TestClassImport)
class TestClass:
# normal class body
But mixins do seem like a better approach.
You can use importlib for this, e.g.:
import importlib
class TestClass:
def __init__(self, module_name):
_tmp = importlib.import_module(module_name)
for elem in _tmp.__dir__():
if not elem.startswith('_'):
prop = getattr(_tmp, elem)
setattr(self, elem, prop)
def funcA(self):
print("function A")
tc = TestClass('some_module')
tc.funcB()
>>> prints "function B"
With this approach, you can create function load_module(module_name) instead of __init__() to load modules independently of each other (e.g. to prevent names collision).
Related
I have a factory as shown in the following code:
class ClassFactory:
registry = {}
#classmethod
def register(cls, name):
def inner_wrapper(wrapped_class):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = wrapped_class
return wrapped_class
return inner_wrapper
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
#ClassFactory.register('Class 1')
class M1():
def __init__(self):
print ("Starting Class 1")
#ClassFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
This works fine and when I do
if __name__ == '__main__':
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
I get the expected result of dict_keys(['Class 1', 'Class 2']) Starting Class 2
Now the problem is that I want to isolate classes M1 and M2 to their own files m1.py and m2.py, and in the future add other classes using their own files in a plugin manner.
However, simply placing it in their own file
m2.py
from test_ import ClassFactory
#MethodFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
gives the result dict_keys(['Class 1']) since it never gets to register the class.
So my question is: How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class? How to self register in this way? Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Thanks
How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class?
I'm playing around with a similar problem, and I've found a possible solution. It seems too much of a 'hack' though, so set your critical thinking levels to 'high' when reading my suggestion below :)
As you've mentioned in one of your comments above, the trick is to force the loading of the individual *.py files that contain individual class definitions.
Applying this to your example, this would involve:
Keeping all class implementations in a specific folders, e.g., structuring the files as follows:
.
└- factory.py # file with the ClassFactory class
└─ classes/
└- __init__.py
└- m1.py # file with M1 class
└- m2.py # file with M2 class
Adding the following statement to the end of your factory.py file, which will take care of loading and registering each individual class:
from classes import *
Add a piece of code like the snippet below to your __init__.py within the classes/ foder, so that to dynamically load all classes [1]:
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
If I then run your test code, I get the desired result:
# test.py
from factory import ClassFactory
if __name__ == "__main__":
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
$ python test.py
dict_keys(['Class 1', 'Class 2'])
Starting Class 2
Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Unfortunately, I'm not experienced enough to answer this question. However, when searching for answers to this problem, I've came across the following sources that may be helpful to you:
[2] : this presents a method for registering class existence based on Python Metaclasses. As far as I understand, it relies on the registering of subclasses, so I don't know how well it applies to your case. I did not follow this approach, as I've noticed that the new edition of the book suggests the use of another technique (see bullet below).
[3], item 49 : this is the 'current' suggestion for subclass registering, which relies on the definition of the __init_subclass__() function in a base class.
If I had to apply the __init_subclass__() approach to your case, I'd do the following:
Add a Registrable base class to your factory.py (and slightly re-factor ClassFactory), like this:
class Registrable:
def __init_subclass__(cls, name:str):
ClassFactory.register(name, cls)
class ClassFactory:
registry = {}
#classmethod
def register(cls, name:str, sub_class:Registrable):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = sub_class
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
from classes import *
Slightly modify your concrete classes to inherit from the Registrable base class, e.g.:
from factory import Registrable
class M2(Registrable, name='Class 2'):
def __init__(self):
print ("Starting Class 2")
I have scenario where I am passing a file name and checking if it has argument start as constructor if it has then I have to create instance of that class.
Consider the example where I have a file named test.py which have three class namely A,B,C now only class A has start parameter others have other different parameter or extra parameter.
#test.py
class A:
def __init__(self, start=""):
pass
class B:
def __init__(self, randomKeyword, start=""):
pass
class C:
def __init__(self):
pass
Now I want to write a script which takes test.py as an argument and create instance of A. Till now my progress is
detail = importlib.util.spec_from_file_location('test.py', '/path/to/test.py')
module = importlib.util.module_from_spec(detail)
spec.loader.exec_module(mod)
Bacially I need to write a program which finds init argument of all class in file and create an instance of file with start as init argument.
As mentioned by #deceze it's not a good idea to instantiate a class on the basis of it's init parameter as we're not sure what is there. But it's possible to do it. So I am posting this answer just so that you know how it can be done.
#test.py
class A:
def __init__(self, start=""):
pass
class B:
def __init__(self, randomKeyword, start=""):
pass
class C:
def __init__(self):
pass
One of the possibility is
#init.py
import importlib.util
from inspect import getmembers, isclass, signature
detail = importlib.util.spec_from_file_location('test.py', '/path/to/test.py')
module = importlib.util.module_from_spec(detail)
spec.loader.exec_module(module)
for name, data in getmembers(mod, isclass):
cls = getattr(mod, name)
parameter = signature(cls.__init__).parameters.keys()
# parameter start
if len(parameter) == 2 and 'start' in parameter:
object = cls(start="Whatever you want")
Ofcourse it's not the best approach so more answer are welcome and if you are in this scenario consider #deceze comment and define a builder.
I'm writing a GUI library, and I'd like to let the programmer provide meta-information about their program which I can use to fine-tune the GUI. I was planning to use function decorators for this purpose, for example like this:
class App:
#Useraction(description='close the program', hotkey='ctrl+q')
def quit(self):
sys.exit()
The problem is that this information needs to be bound to the respective class. For example, if the program is an image editor, it might have an Image class which provides some more Useractions:
class Image:
#Useraction(description='invert the colors')
def invert_colors(self):
...
However, since the concept of unbound methods has been removed in python 3, there doesn't seem to be a way to find a function's defining class. (I found this old answer, but that doesn't work in a decorator.)
So, since it looks like decorators aren't going to work, what would be the best way to do this? I'd like to avoid having code like
class App:
def quit(self):
sys.exit()
Useraction(App.quit, description='close the program', hotkey='ctrl+q')
if at all possible.
For completeness' sake, the #Useraction decorator would look somewhat like this:
class_metadata= defaultdict(dict)
def Useraction(**meta):
def wrap(f):
cls= get_defining_class(f)
class_metadata[cls][f]= meta
return f
return wrap
You are using decorators to add meta data to methods. That is fine. It can be done e.g. this way:
def user_action(description):
def decorate(func):
func.user_action = {'description': description}
return func
return decorate
Now, you want to collect that data and store it in a global dictionary in form class_metadata[cls][f]= meta. For that, you need to find all decorated methods and their classes.
The simplest way to do that is probably using metaclasses. In metaclass, you can define what happens when a class is created. In this case, go through all methods of the class, find decorated methods and store them in the dictionary:
class UserActionMeta(type):
user_action_meta_data = collections.defaultdict(dict)
def __new__(cls, name, bases, attrs):
rtn = type.__new__(cls, name, bases, attrs)
for attr in attrs.values():
if hasattr(attr, 'user_action'):
UserActionMeta.user_action_meta_data[rtn][attr] = attr.user_action
return rtn
I have put the global dictionary user_action_meta_data in the meta class just because it felt logical. It can be anywhere.
Now, just use that in any class:
class X(metaclass=UserActionMeta):
#user_action('Exit the application')
def exit(self):
pass
Static UserActionMeta.user_action_meta_data now contains the data you want:
defaultdict(<class 'dict'>, {<class '__main__.X'>: {<function exit at 0x00000000029F36C8>: {'description': 'Exit the application'}}})
I've found a way to make decorators work with the inspect module, but it's not a great solution, so I'm still open to better suggestions.
Basically what I'm doing is to traverse the interpreter stack until I find the current class. Since no class object exists at this time, I extract the class's qualname and module instead.
import inspect
def get_current_class():
"""
Returns the name of the current module and the name of the class that is currently being created.
Has to be called in class-level code, for example:
def deco(f):
print(get_current_class())
return f
def deco2(arg):
def wrap(f):
print(get_current_class())
return f
return wrap
class Foo:
print(get_current_class())
#deco
def f(self):
pass
#deco2('foobar')
def f2(self):
pass
"""
frame= inspect.currentframe()
while True:
frame= frame.f_back
if '__module__' in frame.f_locals:
break
dict_= frame.f_locals
cls= (dict_['__module__'], dict_['__qualname__'])
return cls
Then in a sort of post-processing step, I use the module and class names to find the actual class object.
def postprocess():
global class_metadata
def findclass(module, qualname):
scope= sys.modules[module]
for name in qualname.split('.'):
scope= getattr(scope, name)
return scope
class_metadata= {findclass(cls[0], cls[1]):meta for cls,meta in class_metadata.items()}
The problem with this solution is the delayed class lookup. If classes are overwritten or deleted, the post-processing step will find the wrong class or fail altogether. Example:
class C:
#Useraction(hotkey='ctrl+f')
def f(self):
print('f')
class C:
pass
postprocess()
The scenerio is I'm using an arg parser to get a command line argument auth_application.
auth_application command can have many values, for example:
cheese
eggs
noodles
pizza
These values are related to a programmable class.
I'd like a way to name the class, possible using a decorator.
So I can say
if auth_application is Cheese.__name__:
return Cheese()
Currently I maintain a tuple of auth_application names and have to expose that to my arg parser class as well as import the classes I need.
Anyways to make this better? Is there a decorator for classes to name them?
I'm looking for a python 2.7 solution, but a python 3 solution might be useful to know.
Easy peasy.
class command(object):
map = {}
def __init__(self, commandname):
self.name = commandname
def __call__(self, cls):
command.map[self.name] = cls
return cls
class NullCommand(object):
pass
#command('cheese')
class Cheese(object):
pass
#command('eggs')
class Eggs(object):
pass
def func(auth_application):
return command.map.get(auth_application, command.NullCommand)()
You can just keep a sinlge list of all of your "allowed classes" and iterate over that to find the class being referred to from the command line.
allow_classes = [Cheese,Eggs,Noodles,Pizza]
for cls in allow_classes:
if auth_application.lower() is cls.__name__.lower():
return cls()
Absolutely you can! You need to understand class attributes.
class NamedClass(object):
name = "Default"
class Cheese(NamedClass):
name = "Cheese"
print(Cheese.name)
> Cheese
You can use the standard Inspect Library to get the real class names, without having to augment your classes with any extra data - and this works for any class, in any module - even if you don't have the source code.
For instance - to list all the classes defined in mymodule :
import mymodule
import inspect
for name, obj in inspect.getmembers(mymodule, inspect.isclass):
print name
the obj variable is a real class object - which you can use to declare an instance, access class methods etc.
To get the definition of a class by it's name string - you can write a simple search function :
import mymodule
import inspect
def find_class(name):
"""Find a named class in mymodule"""
for this_name, _cls_ in inspect.getmembers(mymodule, inspect.isclass):
if this_name = name:
return _cls_
return None
....
# Create an instance of the class named in auth_application
find_class(auth_application)(args, kwargs)
NB: Code snippets not tested
EDIT: Note that this is a REALLY BAD idea to do in production code. This was just an interesting thing for me. Don't do this at home!
Is it possible to modify __metaclass__ variable for whole program (interpreter) in Python?
This simple example is working:
class ChattyType(type):
def __init__(cls, name, bases, dct):
print "Class init", name
super(ChattyType, cls).__init__(name, bases, dct)
__metaclass__= ChattyType
class Data:
pass
data = Data() # prints "Class init Data"
print data
but I would love to be able change of __metaclass__ to work even in submodules. So for example (file m1.py):
class A:
pass
a=A()
print a
file main.py:
class ChattyType(type):
def __init__(cls, name, bases, dct):
print "Class init", name
super(ChattyType, cls).__init__(name, bases, dct)
__metaclass__= ChattyType
import m1 # and now print "Class init A"
class Data:
pass
data = Data() # print "Class init Data"
print data
I understand that global __metaclass__ is no longer working in Python 3.X, but that is not my concern (my code if proof of concept). So is there any way to accomplish this in Python-2.x?
The "global __metaclass__" feature of Python 2 is designed to work per-module, only (just think what havoc it would wreak, otherwise, by forcing your own metaclass on all library and third-party modules that you imported from that point onwards -- shudder!). If it's very important to you to "secretly" alter the behavior of all modules you're importing from a certain point onwards, for whatever cloak-and-dagger reason, you could play very very dirty tricks with an import hook (at worst by first copying the sources to a temporary location while altering them...) but the effort would be proportionate to the enormity of the deed, which seems appropriate;-)
Okay; IMO this is gross, hairy, dark magic. You shouldn't use it, perhaps ever, but especially not in production code. It is kind of interesting just for curiosity's sake, however.
You can write a custom importer using the mechanisms described in PEP 302, and further discussed in Doug Hellmann's PyMOTW: Modules and Imports. That gives you the tools to accomplish the task you contemplated.
I implemented such an importer, just because I was curious. Essentially, for the modules you specify by means of the class variable __chatty_for__, it will insert a custom type as a __metaclass__ variable in the imported module's __dict__, before the code is evaluated. If the code in question defines its own __metaclass__, that will replace the one pre-inserted by the importer. It would be inadvisable to apply this importer to any modules before carefully considering what it would do to them.
I haven't written many importers, so I may have done one or more silly things while writing this one. If anyone notices flaws / corner cases I missed in the implementation, please leave a comment.
source file 1:
# foo.py
class Foo: pass
source file 2:
# bar.py
class Bar: pass
source file 3:
# baaz.py
class Baaz: pass
and the main event:
# chattyimport.py
import imp
import sys
import types
class ChattyType(type):
def __init__(cls, name, bases, dct):
print "Class init", name
super(ChattyType, cls).__init__(name, bases, dct)
class ChattyImporter(object):
__chatty_for__ = []
def __init__(self, path_entry):
pass
def find_module(self, fullname, path=None):
if fullname not in self.__chatty_for__:
return None
try:
if path is None:
self.find_results = imp.find_module(fullname)
else:
self.find_results = imp.find_module(fullname, path)
except ImportError:
return None
(f,fn,(suf,mode,typ)) = self.find_results
if typ == imp.PY_SOURCE:
return self
return None
def load_module(self, fullname):
#print '%s loading module %s' % (type(self).__name__, fullname)
(f,fn,(suf,mode,typ)) = self.find_results
data = f.read()
if fullname in sys.modules:
module = sys.modules[fullname]
else:
sys.modules[fullname] = module = types.ModuleType(fullname)
module.__metaclass__ = ChattyType
module.__file__ = fn
module.__name__ = fullname
codeobj = compile(data, fn, 'exec')
exec codeobj in module.__dict__
return module
class ChattyImportSomeModules(ChattyImporter):
__chatty_for__ = 'foo bar'.split()
sys.meta_path.append(ChattyImportSomeModules(''))
import foo # prints 'Class init Foo'
import bar # prints 'Class init Bar'
import baaz
Nope. (This is a feature!)