We Keep Coding

Dynamically create class instance with complex Class input parameters in Python using reflection

I am trying to make design metadata driven data pipelines, so I define in external textual metadata (json, yaml) things like:
dataFunctionSequence = DataFunctionSequence(
functions=[
Function(
functionClass="Function1",
functionPackage="pipelines.experiments.meta_driven.function_1",
parameters=[
Parameter(name="param1", dataType="str", value="this is my function str value")
]
)
]
)
Now with help of importlib i can get class:
functionMainClass = getattr(
importlib.import_module(
functionMeta.functionPackage), functionMeta.functionClass)
But I want real instance, so I got it working with this piece of code which utilizes on top of importlib as well eval builtin function:
def create_instance(class_str:str):
"""
Create a class instance from a full path to a class constructor
:param class_str: module name plus '.' plus class name and optional parens with arguments for the class's
__init__() method. For example, "a.b.ClassB.ClassB('World')"
:return: an instance of the class specified.
"""
try:
if "(" in class_str:
full_class_name, args = class_name = class_str.rsplit('(', 1)
args = '(' + args
else:
full_class_name = class_str
args = ()
# Get the class object
module_path, _, class_name = full_class_name.rpartition('.')
mod = importlib.import_module(module_path)
klazz = getattr(mod, class_name)
# Alias the the class so its constructor can be called, see the following link.
# See https://www.programiz.com/python-programming/methods/built-in/eval
alias = class_name + "Alias"
instance = eval(alias + args, { alias: klazz})
return instance
except (ImportError, AttributeError) as e:
raise ImportError(class_str)
And I can construct a string that will construct the class and it works like charm.
Now my problem is that the class requires another parameter which is complex Spark DataFrame object which is not loaded from metadata but from some database or s3 bucket for example. Here I fail to be able to create dynamically instance with non-string variable.
I am failing here:
instance = eval(alias + args, { alias: klazz})
I tried to extend the create_instance() fnc with **kwargs, so I can dynamically search for parameter by name eg kwargs["dataFrame"], but how to assign it dynamically to init?
evel is not the right way probably or my expression is not correct?
NOTE: Another possible approach was to iterate somehoe over init object where I will get all params of constructor, but still I don't know what python reflection fuction to use to make a real instance.
Workaround: What I can do is that I can simply remove the dataFrame from class constructor, create instance only on simple params (eg. strings) and call method:
instance.setDataFrame(dataFrame)
And it will work. but I wanted some reflection base approach if possible.
Thank you very much.
Ladislav

How do I pickle a dictionary containing a module & class?

I need to assign a module & class to a dictionary key. Then pickle that dictionary to file. Then later, load the pkl file, then import & instantiate the class, based on that dictionary key value.
I've tried this:
import module_example
from module_example import ClassExample
dictionary = {'module': module_example, 'class': ClassExample)
Yet it won't store a reference to module_exmaple.py in the pkl file.
I've tried a workaround of using a string instead of the module & class name. But that's going to lead to a mess if the module name gets refactored or location is changed down the road.
Is there anyway do this directly? Somehow store a reference to the module & class in a dictionary, then later import & instantiate based on that reference?

This works for single class. If you want to do this in multiple modules and classes, you can extend the following code.
module_class_writer.py
import module_example
from module_example import ClassExample
included_module = ["module_example"]
d = {}
for name, val in globals().items():
if name in included_module:
if "__module__" in dir(val):
d["module"] = val.__module__
d["class"] = name
#d = {'module': module_example, 'class': ClassExample}
import pickle
filehandler = open("imports.pkl","wb")
pickle.dump(d, filehandler)
filehandler.close()
module_class_reader.py
import pickle
filehandler = open("imports.pkl",'rb')
d = pickle.load(filehandler)
filehandler.close()
def reload_class(module_name, class_name):
mod = __import__(module_name, fromlist=[class_name])
reload(mod)
return getattr(mod, class_name)
if "class" in d and "module" in d:
reload(__import__(d["module"]))
ClassExample = reload_class(d["module"], d["class"])

If you want the unpickled class to be the exact same object that was pickled, you'd have to store the class's code (using an external library like dill, for example).
Otherwise, there is no way with standard pickle to store a reference to a class that will 'survive' a refactoring.

Error insert a value into a list-Python

I want to insert an object into a list and I gives me an error saying:
Archive.insertdoc(d)
TypeError: insertdoc() missing 1 required positional argument: 'd'
This is in my Main module:
doc = Document(name, author, file)
Archive.insertdoc(doc)
Archive module:
def __init__(self):
self.listdoc = []
def insertdoc(self, d):
self.listdoc.append(d)

You need to create an instance of the Archive class; you are accessing the unbound method instead.
This should work:
archive = Archive()
doc = Document(name, author, file)
archive.insertdoc(doc)
This assumes you have:
class Archive():
def __init__(self):
self.listdoc = []
def insertdoc(self, d):
self.listdoc.append(d)
If you put two functions at module level instead, you cannot have a self reference in a function and have it bind to the module; functions are not bound to modules.
If your archive is supposed to be a global to your application, create a single instance of the Archive class in the module instead, and use that one instance only.

It looks like Archive.insertdoc is an instance method of the class Archive. Meaning, it must be invoked on an instance of Archive:
doc = Document(name, author, file)
archive = Archive() # Make an instance of class Archive
archive.insertdoc(doc) # Invoke the insertdoc method of that instance

How to change ConfigParser.RawConfigParser module behaviour

I want to change default output of ConfigParser.RawConfigParser.items(section) and also set ConfigParser.RawConfigParser.optionxform = str in ConfigParser module, so I created my own module MyConfigParser inheriting ConfigParser.RawConfigParser as base class. Below is its code which I think is incorrect. Also I am unable to use it.
import os
import ConfigParser
class MyConfigParser(ConfigParser.RawConfigParser):
"""
"""
ConfigParser.RawConfigParser.optionxform = str
def items(section):
items_list = ConfigParser.RawConfigParser.items(section)
for item in item_list:
index = item_list.index(item)
if not (os.path.exists(item[1]) and os.path.isfile(item[1])):
item_list.pop(index)
return items_list

Declaring classes inside a python module

I have a python module that I've made that contains regular function defintions as well as classes. For some reason when I call the constructor and pass a value, it's not updating the class variable.
My module is called VODControl (VODControl.py). The class I have declared inside the module is called DRMPath. The DRMPath class has two instance vairables: logfile and results. logfile is a string and results is a dictionary.
My constructor looks like this:
def __init__(self, file):
self.logilfe = file
self.results['GbE1'] = ""
self.results['GbE2'] = ""
self.results['NetCrypt'] = ""
self.results['QAM'] = ""
when I import it from my other python script I do:
import VODControl
The call i use is the following:
d = VODControl.DRMPath('/tmp/adk.log')
However, when I print the value of the logfile instance variable, it isn't updated with what I passed to the constructor:
print d.logfile
After printing, it's still an empty string. What gives?

self.logilfe = file is not the same as self.logfile = file In addition, it is likely returning None, not an empty string.