I need to assign a module & class to a dictionary key. Then pickle that dictionary to file. Then later, load the pkl file, then import & instantiate the class, based on that dictionary key value.
I've tried this:
import module_example
from module_example import ClassExample
dictionary = {'module': module_example, 'class': ClassExample)
Yet it won't store a reference to module_exmaple.py in the pkl file.
I've tried a workaround of using a string instead of the module & class name. But that's going to lead to a mess if the module name gets refactored or location is changed down the road.
Is there anyway do this directly? Somehow store a reference to the module & class in a dictionary, then later import & instantiate based on that reference?
This works for single class. If you want to do this in multiple modules and classes, you can extend the following code.
module_class_writer.py
import module_example
from module_example import ClassExample
included_module = ["module_example"]
d = {}
for name, val in globals().items():
if name in included_module:
if "__module__" in dir(val):
d["module"] = val.__module__
d["class"] = name
#d = {'module': module_example, 'class': ClassExample}
import pickle
filehandler = open("imports.pkl","wb")
pickle.dump(d, filehandler)
filehandler.close()
module_class_reader.py
import pickle
filehandler = open("imports.pkl",'rb')
d = pickle.load(filehandler)
filehandler.close()
def reload_class(module_name, class_name):
mod = __import__(module_name, fromlist=[class_name])
reload(mod)
return getattr(mod, class_name)
if "class" in d and "module" in d:
reload(__import__(d["module"]))
ClassExample = reload_class(d["module"], d["class"])
If you want the unpickled class to be the exact same object that was pickled, you'd have to store the class's code (using an external library like dill, for example).
Otherwise, there is no way with standard pickle to store a reference to a class that will 'survive' a refactoring.
Related
Suppose in "./data_writers/excel_data_writer.py", I have:
from generic_data_writer import GenericDataWriter
class ExcelDataWriter(GenericDataWriter):
def __init__(self, config):
super().__init__(config)
self.sheet_name = config.get('sheetname')
def write_data(self, pandas_dataframe):
pandas_dataframe.to_excel(
self.get_output_file_path_and_name(), # implemented in GenericDataWriter
sheet_name=self.sheet_name,
index=self.index)
In "./data_writers/csv_data_writer.py", I have:
from generic_data_writer import GenericDataWriter
class CSVDataWriter(GenericDataWriter):
def __init__(self, config):
super().__init__(config)
self.delimiter = config.get('delimiter')
self.encoding = config.get('encoding')
def write_data(self, pandas_dataframe):
pandas_dataframe.to_csv(
self.get_output_file_path_and_name(), # implemented in GenericDataWriter
sep=self.delimiter,
encoding=self.encoding,
index=self.index)
In "./datawriters/generic_data_writer.py", I have:
import os
class GenericDataWriter:
def __init__(self, config):
self.output_folder = config.get('output_folder')
self.output_file_name = config.get('output_file')
self.output_file_path_and_name = os.path.join(self.output_folder, self.output_file_name)
self.index = config.get('include_index') # whether to include index column from Pandas' dataframe in the output file
Suppose I have a JSON config file that has a key-value pair like this:
{
"__comment__": "Here, user can provide the path and python file name of the custom data writer module she wants to use."
"custom_data_writer_module": "./data_writers/excel_data_writer.py"
"there_are_more_key_value_pairs_in_this_JSON_config_file": "for other input parameters"
}
In "main.py", I want to import the data writer module based on the custom_data_writer_module provided in the JSON config file above. So I wrote this:
import os
import importlib
def main():
# Do other things to read and process data
data_writer_class_file = config.get('custom_data_writer_module')
data_writer_module = importlib.import_module\
(os.path.splitext(os.path.split(data_writer_class_file)[1])[0])
dw = data_writer_module.what_should_this_be? # <=== Here, what should I do to instantiate the right specific data writer (Excel or CSV) class instance?
for df in dataframes_to_write_to_output_file:
dw.write_data(df)
if __name__ == "__main__":
main()
As I asked in the code above, I want to know if there's a way to retrieve and instantiate the class defined in a Python module assuming that there is ONLY ONE class defined in the module. Or if there is a better way to refactor my code (using some sort of pattern) without changing the structure of JSON config file described above, I'd like to learn from Python experts on StackOverflow. Thank you in advance for your suggestions!
You can do this easily with vars:
cls1,=[v for k,v in vars(data_writer_module).items()
if isinstance(v,type)]
dw=cls1(config)
The comma enforces that exactly one class is found. If the module is allowed to do anything like from collections import deque (or even foo=str), you might need to filter based on v.__module__.
I have a class named flower with two attributes, name and price
I made a list where i have a few flower objects and i wrote them into a file, how do I create a list of objects by reading the file?
This is what I have in my file:
[lily;5, rose;10]
Python provides a couple of standard ways for serializing objects to file for later retrieval. The two most common are pickle and JSON (using the simplejson module).
If you're only going to use the serialized data within Python programs then pickle is the simpler choice. If you need to be able to share data across multiple programs, then JSON is the more natural choice.
You can read more about the pickle module here: https://docs.python.org/2/library/pickle.html
As a simple example of using a pickle:
class Flower(object):
def __init__(self, type, value):
super(Flower, self).__init__()
self.type = type
self.value = value
def __str__(self):
return "I'm a {} and cost ${}!".format(self.type, self.value)
if __name__== "__main__":
flowers = [ Flower("lily",5), Flower("rose",10) ]
for f in flowers:
print f
pfile = open("flower_list.pickle", mode="wb")
pickle.dump(flowers, pfile)
pfile.close()
pfile = open("flower_list.pickle", mode="rb")
recalled_flowers = pickle.load(pfile)
pfile.close()
for f in recalled_flowers:
print f
if a directory contains multiple .py files
say, a.py , b.py, c.py
How can i fetch all the module name and their class name and eventually all the method names of the corresponding class, contaned in that directory
To filter the files in a directory you can use('.' refers to the current directory):
from os import listdir,chdir
default_path = r'c:\dev\python'
modules = [ fl for fl in listdir(default_path) if fl.endswith('.py') ]
To get the classes and methods I know you can use inspect though you can't use it without importing the module:
e.g.
from inspect import getmembers, isfunction, ismethod, isclass
your_module = __import__(modules[0].split('.')[0])
class_funcs = {}
funcs = [ obj for obj in getmembers(your_module) if isfunction(obj[1]) ]
classes = [ obj for obj in getmembers(your_module) if isclass(obj[1]) ]
for cls in classes:
class_funcs[cls[0]] = [ obj for obj in getmembers(cls[1]) if ismethod(obj[1]) ]
print funcs
[('function1', <function your_module.function1>),
('function2', <function your_module.function2>)]
print class_funcs
{'ClassName': [('__init__', <unbound method ClassName.__init__>),
('Method_name', <unbound method ClassName.Method_name>)]}
That will give you all the functions, classes, classes methods in your module.
If you don't want to import the module I'm not familiar with a way to get the methods(except for reading the file and using regex and etc.).
If you don't want to import the modules you could use AST to inspect them.
import ast
import _ast
def inspect_module(module_path):
with open(module_path) as mf:
tree = ast.parse(mf.read())
module_classes = [_ for _ in tree.body if isinstance(_, _ast.ClassDef)]
module_classes = [(c.name, [_.name for _ in c.body if isinstance(_, _ast.FunctionDef)]) for c in module_classes]
return module_classes
It returns a list containing a tuple of the class name and a list of the function names. Unfortunately it does not work for inhereted functions.
It could easily be extended to give you all the arguments for the functions.
I have the following problem:
# testcase is a list object
i=0
for e in testcase:
testclass = e['class'] #unicode Object
properties = e['properties'] #dict Object
probertie = testcase[i]['properties']
tester = test.ScreenshotCounterTest.ScreenshotCounterTest(probertie)
#tester = InstanceClass(testclass,propertie)
tester.checkResults()
i=i+1
The unicode object testclass contains several unicode strings like test.ScreenshotCounterTest or FileExistTest
I need a way to initialize these Objects with the property argument
a way to dynamically create objects from this testclass list object.
Thanks for your help
If I get what you need, then this should help you:
import importlib
qual_name = "my.module.MyClass"
splitted = qual_name.split(".")
class_name = splitted[-1]
module_name = ".".join(splitted[:-1])
module_obj = importlib.import_module(module_name)
class_obj = getattr(module_obj, class_name)
class_inst = class_obj() # this is instance of my.module.MyClass; assuming that this class has no-argument constructor; in other case use constructor arguments here
importlib interface may have changed between p2.7 and p3k, I'm not sure. Also, there is (was?) __import__ built-in function, but I find importlib most reliable.
In your case testclass/e['class'] will be used instead of qual_name, and tester should be assigned like class_inst in this example.
I'm trying to do a dynamic import of a python module in django. I have two different apps that I want to import from, and I want to replace these import statements:
from app1.forms import App1ProfileForm
from app2.forms import App2ProfileForm
I am dynamically able to create the strings App1ProfileForm and App2ProfileForm and then instantiate them like so:
globals()[form]()
I tried following some of the instructions in this post: Dynamically import class by name for static access
and so I tried doing this:
theModule = __import__("app1.forms.App1ProfileForm")
but I'm getting an error that says No module named App1ProfileForm
EDIT:::
Ok I tried this code:
theModule = __import__("app1")
print theModule
theClass = getattr(theModule,'forms')
print theClass
theForm = getattr(theClass,'App1ProfileForm')
print theForm
theForm.initialize()
but I get an error that type object 'App1ProfileForm' has no attribute 'initialize'
You don't want to do this. Imports are done when the relevant code is first executed - in the case of module-level imports, it's when the module itself is imported. If you're depending on something in the request, or some other run-time element, to determine what class you want, then this will not work.
Instead, just import them both, and get the code to choose which one you need:
from app1.forms import App1ProfileForm
from app2.forms import App2ProfileForm
forms = {'app1': App1ProfileForm,
'app2': App2ProfileForm}
relevant_form = forms[whatever_the_dependent_value_is]
I don't quite know how you're generting the string to import. I'll assume you generate the whole "path". Try this:
def import_from_strings(paths):
ret = []
for path in paths:
module_name, class_name = path.rsplit('.', 1)
module = __import__(module_name, globals(), locals(), [class_name], -1)
ret.append(getattr(module, class_name))
return ret
Aren't you trying to import a class, and not a module ? I'm not an expert, but I think you must import the module using __import__, then select it's App1ProfileForm class with something like yourmodule.App1ProfileForm
I figured it out. Here's how to do it:
theModule = __import__(module_name+".forms") # for some reason need the .forms part
theClass = getattr(theModule,'forms')
theForm = getattr(theClass,form_name)
then to initialize:
theForm() or theForm(request.POST)