We Keep Coding

Python: Using import to add common functions into a class

I have an existing python (python v2.7) application that imports external py files on the fly which contain specifically named classes to processes data. The external py file loaded is chosen based on the type of post-processing of the data that is needed.
So I have this collection of classes, each in their own file. The files are named in a specific fashion based on the type of processing so that the main program knows what file to import from the upstream request.
Keep in mind that I and others are always tweaking these class files, but we can not change the code on the main application.
What I would like to do is to import a "template" of the common functions into the class scope which can provide the standard set of controls that the main program expects without needing to copy/paste them into each file. I hate it when I find a bug and make a correction in one of these main class i/o function which I then have to replicate in thirty-some other files.
Now, I understand from googling that my import here is bad... I get the message:
TestClassFile.py:5: SyntaxWarning: import * only allowed at module level
But this method is the only way I have found to import the functions so that they come into the namespace of the class itself. I have an example below...
What method (if any) is the appropriate way to do this in Python?
Example
main.py
import TestClassFile
print "New TestClass - Init"
oTest = TestClassFile.TestClass("foo")
print "Should run... Function A"
oTest.funcA()
print "Should run... Function b"
oTest.funcB()
TestClassFile.py
class TestClass:
from TestClassImport import *
def __init__(self, str):
print "INIT! and be ... ", str
def funcA(self):
print "Function A"
TestClassImport.py
def funcB(self):
print "Function B"
Much appreciated!
Update
Many thanks to everyone for the contributions. From researching MixIns, these appear to be the proper python way to extend a class.
TestClassImport.py
class ImportClass:
def funcB(self):
print "Function B"
TestClassFile.py
from TestClassImport import ImportClass
class TestClass(ImportClass):
def __init__(self, str):
print "INIT! and be ... ", str
def funcA(self):
print "Function A"

It sounds like you should make the imported functions into mixins, which you can inherit from. So:
TestClassImport.py
class ClassB(object):
def funcB(self):
print "Function B"
TestClassFile.py
from TestClassImport import ClassB
from OtherClassImport import ClassX
class TestClass(ClassB, ClassX):
...

This appears to work:
import types
from TestClassImport import funcB
class TestClass:
def __init__(self, str):
print "INIT! and be ... ", str
setattr(self, 'funcB', types.MethodType(funcB, self, TestClass))
def funcA(self):
print "Function A"
When I run it I get the following output:
INIT! and be ... foo
Should run... Function A
Function A
Should run... Function b
Function B

I don't know if this is by any means a good solution, but you can write a function to construct a metaclass to dynamically add properties to your classes.
def make_meta(*import_classes):
class TestMeta(type):
def __new__(meta, name, bases, dct):
new_class = super(TestMeta, meta).__new__(meta, name, bases, dct)
for import_class in import_classes:
for name in vars(import_class):
if not name.startswith('__'):
prop = getattr(import_class, name)
setattr(new_class, name, prop)
return new_class
return TestMeta
class TestClass:
import TestClassImport
__metaclass__ = make_meta(TestClassImport)
# other functions defined as normal...
This will add everything in the global scope of TestClassImport.py that doesn't start with '__' as a property on TestClass.
Or, you can use a class decorator to add properties dynamically in the same fashion.
def add_imports(*import_classes):
def augment_class(cls):
for import_class in import_classes:
for name in vars(import_class):
if not name.startswith('__'):
prop = getattr(import_class, name)
setattr(cls, name, prop)
return cls
return augment_class
import TestClassImport
#add_imports(TestClassImport)
class TestClass:
# normal class body
But mixins do seem like a better approach.

You can use importlib for this, e.g.:
import importlib
class TestClass:
def __init__(self, module_name):
_tmp = importlib.import_module(module_name)
for elem in _tmp.__dir__():
if not elem.startswith('_'):
prop = getattr(_tmp, elem)
setattr(self, elem, prop)
def funcA(self):
print("function A")
tc = TestClass('some_module')
tc.funcB()
>>> prints "function B"
With this approach, you can create function load_module(module_name) instead of __init__() to load modules independently of each other (e.g. to prevent names collision).

Python classmethod and(?) instancemethod

I have written a Python class for parsing a specialized text format.
class Parser(object):
def __init__(self):
# Initialize parser instance
def parseFile(self , filename):
pass
def modifyParser(self , *args , **kwargs):
pass
#Classmethod has same name as instance method - this does not work.
#classmethod
def parseFile(cls , filename)
parser = Parser( )
return parser.parseFile( filename )
As indicated the parser can be modified with the modifyParser method - but in most cases I will just use the Parser instance as it comes from the Parser.__init__(). I would like to be able to do this:
# Parse file using 'custom' parser:
parser = Parser( )
parser.modifyParser( ... )
result = parser.parseFile("file.input")
# Parse using the default parser - do not explicitly instantiate an object:
result = Parser.parseFile("file.input")
This requires that the parseFile( ) method can be called both as an instance method - with a self - and as a classmethod. Is this possible? Bad form?

If I were you, I'd offer two distinct functions:
mymodule.Parser().parseFile() (instance method), and
mymodule.parseFile() (module-level function that uses the default instance).
This is what happens for example with the standard json module, where you have json.JSONDecoder().decode() and json.loads(). Offering two distinct functions makes the code less ambiguous, more explicit and more predictable (in my opinion).
However, yes: what you want to do is possible. You have to implement your own descriptor using __get__. Here's an example:
from functools import partial
class class_and_instance_method(object):
def __init__(self, func):
self.func = func
def __get__(self, obj, type=None):
first_arg = obj if obj is not None else type
return partial(self.func, first_arg)
class Parser(object):
#class_and_instance_method
def parseFile(self):
if isinstance(self, type):
print('using default parser')
else:
print('using the current instance')
>>> Parser.parseFile()
using default parser
>>> p = Parser()
>>> p.parseFile()
using the current instance

You'll have to use two separate names. In python due to it's dynamic nature there's no operator overloading as in C++, when one function has same name with different arguments.
When you say def in your script, you tell Python "set the following object(function object) to this name". So in your code you just redefine the name to reference classmethod and your instance method function object is lost.
Solution: use different names for instace method and class method.

Dependency injection in imported python module

My python module uses some functions from another module, but I have several implementations of that module interface. How to point out, which one to use?
Simple example:
A.py:
import B
def say_hi()
print "Message: " + B.greeting()
main.py:
import A(B=my_B_impl)
A.say_hi()
my_B_impl.py:
def greeting():
return "Hallo!"
output:
Message: Hallo!

In python this could be most elegantly done with inheritance:
A.py:
import B
class SayHi(object):
b = B
def say_hi(self):
print "Message: " + self.b.greeting()
my_B_impl.py:
class AlternativeHi(object):
def greeting(self):
return "Hallo!"
main.py:
import A
from my_B_impl.py import AlternativeHi
class MyHi(SayHi):
b=AlternativeHi
a=MyHi()
MyHi.say_hi()
output:
Message: Hallo!
You can also use the factory pattern to avoid explicit declaration of class AlternativeHi and MyHi:
A.py
from B import greeting
class SayHi(object):
def __init__(self,*args,**kwargs):
self.greeting = greeting
def say_hi(self):
print "Message: " + self.greeting()
def hi_factory(func):
class CustomHi(SayHi):
def __init__(self,*args,**kwargs):
result = super(CustomHi, self).__init__(*args, **kwargs)
self.greeting = func
return CustomHi
my_B_impl.py:
def greeting(self):
return "Hallo!"
main.py:
form A import hi_factory
from my_B_impl import greeting
a = hi_factory(greeting)
a.say_hi()

What you ask is not directly possible. There is no parameterisation capability built in to Python's module system. If you think about it, it's not clear how such a proposal ought to work: if modules A and B both import module M, but they supply different parameters, which parameter is used when M is imported? Is it imported twice? What would that mean for module-level configuration (as in logging)? It gets worse if a third module C attempts to import M without parameters. Also, the "open-world" idea that you could override any import statement from the outside violates the language-design principle that "the code you wrote is the code that ran".
Other languages have incorporated parameterised modules in a variety of ways (compare Scala's object model, ML's modules and signatures, and - stretching it - C++'s templates), but it's not clear that such a feature would be a good fit for Python. (That said, you could probably hack something resembling parameterised modules using importlib if you were determined and masochistic enough.)
Python does have very powerful and flexible capabilities for dynamic dispatch, however. Python's standard, day-to-day features like functions, classes, parameters and overriding provide the basis for this support.
There are lots of ways to cut the cake on your example of a function whose behaviour is configurable by its client.
A function parameterised by a value:
def say_hi(greeting):
print("Message: " + greeting)
def main():
say_hi("Hello")
A class parameterised by a value:
class Greeter:
def __init__(self, greeting):
self.greeting = greeting
def say_hi(self):
print("Message: " + self.greeting)
def main():
Greeter("Hello").say_hi()
A class with a virtual method:
class Greeter:
def say_hi(self):
print("Message: " + self.msg())
class MyGreeter(Greeter):
def msg(self):
return "Hello"
A function parameterised by a function:
def say_hi(greeting):
print("Message: " + greeting())
def make_greeting():
return "Hello"
def main():
say_hi(make_greeting)
There are more options (I'm avoiding the Java-y example of objects invoking other objects) but you get the idea. In each of these cases, the selection of the behaviour (the passing of the parameter, the overriding of the method) is decoupled from the code which uses it and could be put in a different file. The right one to choose depends on your situation (though here's a hint: the right one is always the simplest one that works).
Update: in a comment you mention that you'd like an API which sets up the dependency at the module-level. The main problem with this is that the dependency would be global - modules are singletons, so anyone who imports the module has to use the same implementation of the dependency.
My advice is to provide an object-oriented API with "proper" (per-instance) dependency injection, and provide top-level convenience functions which use a (configurable) "default" set-up of the dependency. Then you have the option of not using the globally-configured version. This is roughly how asyncio does it.
# flexible object with dependency injection
class Greeter:
def __init__(self, msg):
self.msg = msg
def say_hi(self):
print("Message: " + self.msg)
# set up a default configuration of the object to be used by the high-level API
_default_greeter = Greeter("Hello")
def configure(msg):
global _default_greeter
_default_greeter = Greeter(msg)
# delegate to whatever default has been configured
def say_hi():
_default_greeter.say_hi()

How to give a class a referencable string name?

The scenerio is I'm using an arg parser to get a command line argument auth_application.
auth_application command can have many values, for example:
cheese
eggs
noodles
pizza
These values are related to a programmable class.
I'd like a way to name the class, possible using a decorator.
So I can say
if auth_application is Cheese.__name__:
return Cheese()
Currently I maintain a tuple of auth_application names and have to expose that to my arg parser class as well as import the classes I need.
Anyways to make this better? Is there a decorator for classes to name them?
I'm looking for a python 2.7 solution, but a python 3 solution might be useful to know.

Easy peasy.
class command(object):
map = {}
def __init__(self, commandname):
self.name = commandname
def __call__(self, cls):
command.map[self.name] = cls
return cls
class NullCommand(object):
pass
#command('cheese')
class Cheese(object):
pass
#command('eggs')
class Eggs(object):
pass
def func(auth_application):
return command.map.get(auth_application, command.NullCommand)()

You can just keep a sinlge list of all of your "allowed classes" and iterate over that to find the class being referred to from the command line.
allow_classes = [Cheese,Eggs,Noodles,Pizza]
for cls in allow_classes:
if auth_application.lower() is cls.__name__.lower():
return cls()

Absolutely you can! You need to understand class attributes.
class NamedClass(object):
name = "Default"
class Cheese(NamedClass):
name = "Cheese"
print(Cheese.name)
> Cheese

You can use the standard Inspect Library to get the real class names, without having to augment your classes with any extra data - and this works for any class, in any module - even if you don't have the source code.
For instance - to list all the classes defined in mymodule :
import mymodule
import inspect
for name, obj in inspect.getmembers(mymodule, inspect.isclass):
print name
the obj variable is a real class object - which you can use to declare an instance, access class methods etc.
To get the definition of a class by it's name string - you can write a simple search function :
import mymodule
import inspect
def find_class(name):
"""Find a named class in mymodule"""
for this_name, _cls_ in inspect.getmembers(mymodule, inspect.isclass):
if this_name = name:
return _cls_
return None
....
# Create an instance of the class named in auth_application
find_class(auth_application)(args, kwargs)
NB: Code snippets not tested

How to define a static utility class correctly in Python

I wanted to write a utility class to read from a config file in python.
import os,ConfigParser
class WebPageTestConfigUtils:
configParser = ConfigParser.RawConfigParser()
configFilePath = (os.path.join(os.getcwd(),'webPageTestConfig.cfg'))
#staticmethod
def initializeConfig():
configParser.read(self.configFilePath)
#staticmethod
def getConfigValue(key):
return configParser.get('WPTConfig', key)
def main():
WebPageTestConfigUtils.initializeConfig()
print WebPageTestConfigUtils.getConfigValue('testStatus')
if __name__ =='__main__':
main()
Upon execution this throws the error.
NameError: global name 'configParser' is not defined
Why is python not able to recognize the static member.

In general, it is almost always better to use #classmethod over #staticmethod.
Then, configParser is an attribute of the cls argument:
class WebPageTestConfigUtils(object):
configParser = ConfigParser.RawConfigParser()
configFilePath = (os.path.join(os.getcwd(),'webPageTestConfig.cfg'))
#classmethod
def initializeConfig(cls):
cls.configParser.read(cls.configFilePath)
#classmethod
def getConfigValue(cls, key):
return cls.configParser.get('WPTConfig', key)
Also note your usage of self is replaced by cls.

Class and instance attributes do not participate in the variable resolution process within a method. If you want to access them, you need to use ordinary attribute lookup syntax:
WebPageTestConfigUtils.configParser.read(self.configFilePath)
That said, you shouldn't be using a class at all for this. You seem to be used to a language where everything has to be in a class. Python doesn't work that way; you should just be using a module with ordinary functions in it.

If you want to create static variable in your file, create before class definition. Generally in python static variable declare as UPPERCASE variable name.
For your example you can use
CONFIGPARSER = ConfigParser.RawConfigParser()
CONFIGFILEPATH = (os.path.join(os.getcwd(),'webPageTestConfig.cfg'))
...
...
#staticmethod
def initializeConfig():
CONFIGPARSER.read(CONFIGFILEPATH)
...
...