We Keep Coding

Dynamically create class instance with complex Class input parameters in Python using reflection

I am trying to make design metadata driven data pipelines, so I define in external textual metadata (json, yaml) things like:
dataFunctionSequence = DataFunctionSequence(
functions=[
Function(
functionClass="Function1",
functionPackage="pipelines.experiments.meta_driven.function_1",
parameters=[
Parameter(name="param1", dataType="str", value="this is my function str value")
]
)
]
)
Now with help of importlib i can get class:
functionMainClass = getattr(
importlib.import_module(
functionMeta.functionPackage), functionMeta.functionClass)
But I want real instance, so I got it working with this piece of code which utilizes on top of importlib as well eval builtin function:
def create_instance(class_str:str):
"""
Create a class instance from a full path to a class constructor
:param class_str: module name plus '.' plus class name and optional parens with arguments for the class's
__init__() method. For example, "a.b.ClassB.ClassB('World')"
:return: an instance of the class specified.
"""
try:
if "(" in class_str:
full_class_name, args = class_name = class_str.rsplit('(', 1)
args = '(' + args
else:
full_class_name = class_str
args = ()
# Get the class object
module_path, _, class_name = full_class_name.rpartition('.')
mod = importlib.import_module(module_path)
klazz = getattr(mod, class_name)
# Alias the the class so its constructor can be called, see the following link.
# See https://www.programiz.com/python-programming/methods/built-in/eval
alias = class_name + "Alias"
instance = eval(alias + args, { alias: klazz})
return instance
except (ImportError, AttributeError) as e:
raise ImportError(class_str)
And I can construct a string that will construct the class and it works like charm.
Now my problem is that the class requires another parameter which is complex Spark DataFrame object which is not loaded from metadata but from some database or s3 bucket for example. Here I fail to be able to create dynamically instance with non-string variable.
I am failing here:
instance = eval(alias + args, { alias: klazz})
I tried to extend the create_instance() fnc with **kwargs, so I can dynamically search for parameter by name eg kwargs["dataFrame"], but how to assign it dynamically to init?
evel is not the right way probably or my expression is not correct?
NOTE: Another possible approach was to iterate somehoe over init object where I will get all params of constructor, but still I don't know what python reflection fuction to use to make a real instance.
Workaround: What I can do is that I can simply remove the dataFrame from class constructor, create instance only on simple params (eg. strings) and call method:
instance.setDataFrame(dataFrame)
And it will work. but I wanted some reflection base approach if possible.
Thank you very much.
Ladislav

Instance attributes in Python and __init__ method

I am trying to writing a program to read a configuration file but while testing it am having this error:
self.connection_attempts = self.config_file.get('CONNECTION_ATTEMPTS', 'TIME')
AttributeError: 'list' object has no attribute 'get'
I ma pretty sure it is something I don't get, but it is few hours I am trying to understand where the problem is.
My __init__ method looks like this:
import simpleconfigparser
class ReportGenerator:
def __init__(self):
self.config_parser = simpleconfigparser.configparser()
self.config_file = config_parser.read('config.ini')
self.connection_attempts = config_file.get('CONNECTION_ATTEMPTS', 'TIME')
self.connection_timeout = config_file.get('CONNECTION_TIMEOUT', 'TIMEOUT')
self.report_destination_path = config_file.get('REPORT', 'REPORT_PATH')
This code uses the SimpleConfigParser package.

You want config_parser.get() not config_file.get(). config_parser.read() simply returns the list of config files successfully read after populating the config object. (Usually it is called config or cfg, not config_parser).
This list (config_file) serves no purpose in your code and you might as well not capture it at all.
from simpleconfigparser import simpleconfigparser
TIME = 5
TIMEOUT = 10
REPORT_PATH = '/tmp/'
class ReportGenerator:
def __init__(self):
self.config = simpleconfigparser()
config.read('config.ini')
self.connection_attempts = config.get('CONNECTION_ATTEMPTS', TIME)
self.connection_timeout = config.get('CONNECTION_TIMEOUT', TIMEOUT)
self.report_destination_path = config.get('REPORT', REPORT_PATH)
My guess would also be, that you use the default value in .get() the wrong way, but i cannot be certain with the information you have given.

How to instance objects from a unicode string in Python

I have the following problem:
# testcase is a list object
i=0
for e in testcase:
testclass = e['class'] #unicode Object
properties = e['properties'] #dict Object
probertie = testcase[i]['properties']
tester = test.ScreenshotCounterTest.ScreenshotCounterTest(probertie)
#tester = InstanceClass(testclass,propertie)
tester.checkResults()
i=i+1
The unicode object testclass contains several unicode strings like test.ScreenshotCounterTest or FileExistTest
I need a way to initialize these Objects with the property argument
a way to dynamically create objects from this testclass list object.
Thanks for your help

If I get what you need, then this should help you:
import importlib
qual_name = "my.module.MyClass"
splitted = qual_name.split(".")
class_name = splitted[-1]
module_name = ".".join(splitted[:-1])
module_obj = importlib.import_module(module_name)
class_obj = getattr(module_obj, class_name)
class_inst = class_obj() # this is instance of my.module.MyClass; assuming that this class has no-argument constructor; in other case use constructor arguments here
importlib interface may have changed between p2.7 and p3k, I'm not sure. Also, there is (was?) __import__ built-in function, but I find importlib most reliable.
In your case testclass/e['class'] will be used instead of qual_name, and tester should be assigned like class_inst in this example.

Declaring classes inside a python module

I have a python module that I've made that contains regular function defintions as well as classes. For some reason when I call the constructor and pass a value, it's not updating the class variable.
My module is called VODControl (VODControl.py). The class I have declared inside the module is called DRMPath. The DRMPath class has two instance vairables: logfile and results. logfile is a string and results is a dictionary.
My constructor looks like this:
def __init__(self, file):
self.logilfe = file
self.results['GbE1'] = ""
self.results['GbE2'] = ""
self.results['NetCrypt'] = ""
self.results['QAM'] = ""
when I import it from my other python script I do:
import VODControl
The call i use is the following:
d = VODControl.DRMPath('/tmp/adk.log')
However, when I print the value of the logfile instance variable, it isn't updated with what I passed to the constructor:
print d.logfile
After printing, it's still an empty string. What gives?

self.logilfe = file is not the same as self.logfile = file In addition, it is likely returning None, not an empty string.

python google closure compiler Source class question

This code is copy from http://code.google.com/p/closure-library/source/browse/trunk/closure/bin/build/source.py
The Source class's __str
__method referred self._path
Is it a special property for self?
Cuz, i couldn't find the place define this variable at Source Class
import re
_BASE_REGEX_STRING = '^\s*goog\.%s\(\s*[\'"](.+)[\'"]\s*\)'
_PROVIDE_REGEX = re.compile(_BASE_REGEX_STRING % 'provide')
_REQUIRES_REGEX = re.compile(_BASE_REGEX_STRING % 'require')
# This line identifies base.js and should match the line in that file.
_GOOG_BASE_LINE = (
'var goog = goog || {}; // Identifies this file as the Closure base.')
class Source(object):
"""Scans a JavaScript source for its provided and required namespaces."""
def __init__(self, source):
"""Initialize a source.
Args:
source: str, The JavaScript source.
"""
self.provides = set()
self.requires = set()
self._source = source
self._ScanSource()
def __str__(self):
return 'Source %s' % self._path #!!!!!! what is self_path !!!!
def GetSource(self):
"""Get the source as a string."""
return self._source
def _ScanSource(self):
"""Fill in provides and requires by scanning the source."""
# TODO: Strip source comments first, as these might be in a comment
# block. RegExes can be borrowed from other projects.
source = self.GetSource()
source_lines = source.splitlines()
for line in source_lines:
match = _PROVIDE_REGEX.match(line)
if match:
self.provides.add(match.group(1))
match = _REQUIRES_REGEX.match(line)
if match:
self.requires.add(match.group(1))
# Closure's base file implicitly provides 'goog'.
for line in source_lines:
if line == _GOOG_BASE_LINE:
if len(self.provides) or len(self.requires):
raise Exception(
'Base files should not provide or require namespaces.')
self.provides.add('goog')
def GetFileContents(path):
"""Get a file's contents as a string.
Args:
path: str, Path to file.
Returns:
str, Contents of file.
Raises:
IOError: An error occurred opening or reading the file.
"""
fileobj = open(path)
try:
return fileobj.read()
finally:
fileobj.close()

No, _path is just an attribute that may or me not be set on an object like any other attribute. The leading underscore simply means that the author felt it was an internal detail of the object and didn't want it regarded as part of the public interface.
In this particular case, unless something is setting the attribute from outside that source file, it looks like it's simply a mistake. It won't do any harm unless anyone ever tries to call str() on a Source object and probably nobody ever does.
BTW, you seem to be thinking there is something special about self. The name self isn't special in any way: it's a convention to use this name for the first parameter of a method, but it is just a name like any other that refers to the object being processed. So if you could access self._path without causing an error you could access it equally well through any other name for the object.