I have a class instance I want to access in other modules. This class loads config values using configParser to update an class instance __dict__ attribute as per this post:
I want to access this instance in other module. The instance is only created in the main.py file where it has access to the required parameters, which come via command line arguments.
I have three files: main.py, config.py and file.py. I don't know the best way to access the instance in the file.py. I only have access to it in main.py and not other modules.
I've looked at the following answers, here and here but they don't fully answer my scenario.
#config.py
class Configuration():
def __init__(self, *import_sections):
#use configParser, get config for relevant sections, update self.__dict__
#main.py
from config import Configuration
conf = Configuration('general', 'dev')
# other lines of code use conf instance ... e.g. config.log_path in log setup
#file.py
#I want to use config instance like this:
class File():
def __init__(self, conf.feed_path):
# other code here...
Options considered:
Initialise Configuration in config.py module
In config.py after class definition I could add:
conf = Configuration('general', 'dev')
and in file.py and main.py:
from config import conf
but the general and dev variables are only found in main.py so doesn't look like it will work.
Make Configuration class a function
I could make it a function and create a module-level dictionary and import data into other modules:
#config.py
conf = {}
def set_config(*import_section):
# use configParser, update conf dictionary
conf.update(...)
This would mean referring to it as config.conf['log_path'] for example. I'd prefer conf.log_path as it's used multiple times.
Pass via other instances
I could pass the conf instance as parameters via other class instances from main.py, even if the intermediate instances don't use it. Seems very messy.
Other options?
Can I use Configuration as an instance somehow?
By changing your Configuration class into a Borg, you are guaranteed to get a common state from wherever you want. You can either provide initialization through a specific __init__:
#config.py
class Configuration:
__shared_state = {}
def __init__(self, *import_sections):
self.__dict__ = self.__shared_state
if not import_sections: # we are not initializing this time
return
#your old code verbatim
initialization is donne as usual with a c = config.Configuration('general','dev') and any call to conf = config.Configuration() will get the state that c created.
or you can provide an initialization method to avoid tampering with the shared state in the __init__:
#config.py
class Configuration:
__shared_state = {}
def __init__(self):
self.__dict__ = self.__shared_state
def import(self, *import_sections):
#your old __init__
that way there is only one meaning to the __init__ method, which is cleaner.
In both cases, you can get the shared state, once initialized, from anywhere in your code by using config.Configuration().
Related
Let's say I have an app.py like this
class myClassA :
def __init__(self):
self.id = 100
class myClassB :
def __init__(self, objA, id):
pass
Is there a way to use hydra to have a config file like below work like it intuitively should ?
myClassA:
_target_: myapp.myClassA
myclassB:
_target_: myapp.myClassB
param1: ${myClassA}
param2: ${myclassB.param1.id}
My issue is that in order to instanciate my class B, I need an attribute from the class A object but this attribute is set in the init function of classA and cannot be set in the config file.
I've tried putting id: ??? but it didn't work
Thank a lot !
The following does the trick:
# app.py
import hydra
from hydra.utils import instantiate
from omegaconf import OmegaConf
class myClassA:
def __init__(self):
self.id = 100
class myClassB:
def __init__(self, objA, objA_id):
assert isinstance(objA, myClassA)
assert objA_id == 100
print("myClassB __init__ ran")
#hydra.main(config_name="conf.yaml", config_path=".", version_base="1.2")
def app(cfg):
instantiate(cfg)
if __name__ == "__main__":
app()
# conf.yaml
myClassA:
_target_: __main__.myClassA
myClassB:
_target_: __main__.myClassB
objA: ${myClassA}
objA_id:
_target_: builtins.getattr
_args_:
- ${myClassA}
- "id"
$ python app.py
myClassB __init__ ran
How does this work? Using builtins.getattr as a target allows for looking up the "id" attribute on an instance of myClassA.
NOTE: Several instances of myClassA will be created here. There is an open feature request in Hydra regarding support for a singleton pattern in recursive instantiation, which would enable re-using the same instance of myClassA in several places.
Is it possible use a variable as a container for a Python module, and then adding another one to the same variable?
So for example, if I would have a Python file called general_config.py containing a general config of some kind:
class GeneralConfig:
attribute1 = "Some attribute"
attribute2 = "Some other attribute"
And if I would import this Python module as a variable containing a general config, I would do:
import general_config.py as Config
Then I can access its attributes by doing:
generalParameter = Config.GeneralConfig.attribute1
But what if I want to add some specific parameters to my config (say from specific_config.py), while keeping the general one as part of the entire config? So it would do something like that:
if someSpecificCondition:
Config += import specific_config.py
else:
Config += import other_config.py
While keeping the Config in the original scope? Thanks in advance.
If you want your general config to inherit your other configs for whatever reason, you could do something like this. But Tom's answer makes more sense, since there's no runtime class creation.
class BaseConfig:
att = "hello world"
def inherit_configs(some_condition):
if some_condition:
from config1 import Config1
class Config(BaseConfig, Config1):
pass
return Config
else:
from config2 import Config2
class Config(BaseConfig, Config2):
pass
return Config
config = inherit_configs(some_condition)()
I have a factory as shown in the following code:
class ClassFactory:
registry = {}
#classmethod
def register(cls, name):
def inner_wrapper(wrapped_class):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = wrapped_class
return wrapped_class
return inner_wrapper
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
#ClassFactory.register('Class 1')
class M1():
def __init__(self):
print ("Starting Class 1")
#ClassFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
This works fine and when I do
if __name__ == '__main__':
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
I get the expected result of dict_keys(['Class 1', 'Class 2']) Starting Class 2
Now the problem is that I want to isolate classes M1 and M2 to their own files m1.py and m2.py, and in the future add other classes using their own files in a plugin manner.
However, simply placing it in their own file
m2.py
from test_ import ClassFactory
#MethodFactory.register('Class 2')
class M2():
def __init__(self):
print("Starting Class 2")
gives the result dict_keys(['Class 1']) since it never gets to register the class.
So my question is: How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class? How to self register in this way? Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Thanks
How can I ensure that the class is registered when placed in a file different from the factory, without making changes to the factory file whenever I want to add a new class?
I'm playing around with a similar problem, and I've found a possible solution. It seems too much of a 'hack' though, so set your critical thinking levels to 'high' when reading my suggestion below :)
As you've mentioned in one of your comments above, the trick is to force the loading of the individual *.py files that contain individual class definitions.
Applying this to your example, this would involve:
Keeping all class implementations in a specific folders, e.g., structuring the files as follows:
.
└- factory.py # file with the ClassFactory class
└─ classes/
└- __init__.py
└- m1.py # file with M1 class
└- m2.py # file with M2 class
Adding the following statement to the end of your factory.py file, which will take care of loading and registering each individual class:
from classes import *
Add a piece of code like the snippet below to your __init__.py within the classes/ foder, so that to dynamically load all classes [1]:
from inspect import isclass
from pkgutil import iter_modules
from pathlib import Path
from importlib import import_module
# iterate through the modules in the current package
package_dir = Path(__file__).resolve().parent
for (_, module_name, _) in iter_modules([package_dir]):
# import the module and iterate through its attributes
module = import_module(f"{__name__}.{module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
# Add the class to this package's variables
globals()[attribute_name] = attribute
If I then run your test code, I get the desired result:
# test.py
from factory import ClassFactory
if __name__ == "__main__":
print(ClassFactory.registry.keys())
foo = ClassFactory.create_type("Class 2")
$ python test.py
dict_keys(['Class 1', 'Class 2'])
Starting Class 2
Also, is this decorator way a good way to do this kind of thing, or are there better practices?
Unfortunately, I'm not experienced enough to answer this question. However, when searching for answers to this problem, I've came across the following sources that may be helpful to you:
[2] : this presents a method for registering class existence based on Python Metaclasses. As far as I understand, it relies on the registering of subclasses, so I don't know how well it applies to your case. I did not follow this approach, as I've noticed that the new edition of the book suggests the use of another technique (see bullet below).
[3], item 49 : this is the 'current' suggestion for subclass registering, which relies on the definition of the __init_subclass__() function in a base class.
If I had to apply the __init_subclass__() approach to your case, I'd do the following:
Add a Registrable base class to your factory.py (and slightly re-factor ClassFactory), like this:
class Registrable:
def __init_subclass__(cls, name:str):
ClassFactory.register(name, cls)
class ClassFactory:
registry = {}
#classmethod
def register(cls, name:str, sub_class:Registrable):
if name in cls.registry:
print(f'Class {name} already exists. Will replace it')
cls.registry[name] = sub_class
#classmethod
def create_type(cls, name):
exec_class = cls.registry[name]
type = exec_class()
return type
from classes import *
Slightly modify your concrete classes to inherit from the Registrable base class, e.g.:
from factory import Registrable
class M2(Registrable, name='Class 2'):
def __init__(self):
print ("Starting Class 2")
Problem
Many people say their variable-sharing problems are resolved with the approach provided here and here, but neither of them works in my use case, where all configurations have to be written in a class like following.
# config.py
class Config(object):
var = None
The structure of my project looks like
├── config.py
├── main.py
├── subfolder
| ├── change_config.py
In this project, main.py will invoke change_config.py and they both have access to variables defined in config.py. Importantly, change_config.py will modify the variables (aka., var) whose value is only known during runtime.
But it is not clear how I should share the instantiated Config() (aka. opt) across main.py and change_config.py. I tried following but no luck. The issue is
If I instantiated another Config() in change_config.py, the one in main.py will be wiped out.
If I do not do the first one, then name opt would not be resolved in change_config.py
# main.py
import config
from subfolder import change_config
opt = config.Config()
print(opt.var)
change_config.change_config()
print(opt.var)
# change_config.py
import config
def change_config():
opt.var = 10
More information
If I run main.py, then I will have
NameError: name 'opt' is not defined
which is expected since opt is never declared in change_config.py.
If I change change_config.py into
# change_config.py
import config
opt = config.Config()
def change_config():
opt.var = 10
where I declared another opt. There is no error but returned
None
None
which is also expected since opt declared in main.py is wiped by the one in change_config.py. But the expected output should be
None
10
So the question is how to share opt in main.py with change_config.py
Let's start by looking at what the differences between Config.var and Config().var are in python terms. Like everything else in Python, a class is an object. A class has a namespace, so Config.var is a variable in that namespace.
When you instantiate Config, say with opt = Config(), opt is now an object, also with a namespace. At first, when you request the value opt.var, it returns Config.var because of how inheritance works in general: if an object does not have an attribute, you look in the class, and then in the parent class, etc. But when you assign opt.var = 10, you are assigning into the namespace of the instance. Now, when you request the value of opt.var, it does not look at Config.var any more, because opt has its own var attribute.
Now keep in mind that opt in main and opt in change_config are completely different instances that do not affect each other except that they share the same class. So when you set change_config.opt.var = 10, requesting main.opt.var still returns None, because the actual attribute main.opt.var does not exist as such: it is returning Config.var from the class namespace.
So now you have a couple of options available to you.
The first is not to bother instantiating anything at all. You can keep your class variable config.Config.var, and just update in the class namespace. The code would look like this:
main.py
from config import Config
from subfolder.change_config import change_config
print(Config.var)
change_config()
print(Config.var)
config.py
class Config:
var = None
subfolder/change_config.py
from config import Config
def change_config():
Config.var = 10
Aside from changing your imports to relative imports and removing the explicit inheritance from object, your code is only modified to use Config as a namespace, without instantiating it.
A second approach would be to instantiate Config, but put the reference in a place that can be accessed by everyone that needs it. At that point, config.py should really be maintaining the shared reference, since that is what stores the actual configuration. I would also recommend removing the class variable Config.var entirely, to avoid exactly the type of confusion you are having now:
main.py
from config import opt
from subfolder.change_config import change_config
print(opt.var)
change_config()
print(opt.var)
config.py
class Config:
def __init__(self):
self.var = None
opt = Config()
subfolder/change_config.py
from config import opt
def change_config():
opt.var = 10
You could commit to making your configuration an instance of Config even deeper by adding del Config to the end of config.py. This will make it much more difficult to create another, conflicting instance of the class, since it will not be readily available through the module. The only reference to Config will be through the inheritance hierarchy of opt at that point.
you can change your application like below, this will help you
in config.py
x = 0
in change_config.py
import config
def change_config():
config.x = 10 #change the global config variable
and in main.py
import config
from subfolder import change_config
print(config.x)
change_config.change_config()
print(config.x)
you will get output like below after you run main.py
0
10
EDIT
using singleton class
in config.py
class Config(object):
var = None
def __new__(cls):
if not hasattr(cls, 'instance'):
cls.instance = super(Config, cls).__new__(cls)
return cls.instance
in main.py
from config import Config
from subfolder import change_config
opt = Config()
print('memory address :%s'%id(opt))
print(opt.var)
change_config.change_config()
print(opt.var)
in change_config.py
from config import Config
opt = Config()
print('memory address :%s'%id(opt))
def change_config():
opt.var = 10
You will get output like below
memory address :140185708587536
memory address :140185708587536
None
10
I wanted to write a utility class to read from a config file in python.
import os,ConfigParser
class WebPageTestConfigUtils:
configParser = ConfigParser.RawConfigParser()
configFilePath = (os.path.join(os.getcwd(),'webPageTestConfig.cfg'))
#staticmethod
def initializeConfig():
configParser.read(self.configFilePath)
#staticmethod
def getConfigValue(key):
return configParser.get('WPTConfig', key)
def main():
WebPageTestConfigUtils.initializeConfig()
print WebPageTestConfigUtils.getConfigValue('testStatus')
if __name__ =='__main__':
main()
Upon execution this throws the error.
NameError: global name 'configParser' is not defined
Why is python not able to recognize the static member.
In general, it is almost always better to use #classmethod over #staticmethod.
Then, configParser is an attribute of the cls argument:
class WebPageTestConfigUtils(object):
configParser = ConfigParser.RawConfigParser()
configFilePath = (os.path.join(os.getcwd(),'webPageTestConfig.cfg'))
#classmethod
def initializeConfig(cls):
cls.configParser.read(cls.configFilePath)
#classmethod
def getConfigValue(cls, key):
return cls.configParser.get('WPTConfig', key)
Also note your usage of self is replaced by cls.
Class and instance attributes do not participate in the variable resolution process within a method. If you want to access them, you need to use ordinary attribute lookup syntax:
WebPageTestConfigUtils.configParser.read(self.configFilePath)
That said, you shouldn't be using a class at all for this. You seem to be used to a language where everything has to be in a class. Python doesn't work that way; you should just be using a module with ordinary functions in it.
If you want to create static variable in your file, create before class definition. Generally in python static variable declare as UPPERCASE variable name.
For your example you can use
CONFIGPARSER = ConfigParser.RawConfigParser()
CONFIGFILEPATH = (os.path.join(os.getcwd(),'webPageTestConfig.cfg'))
...
...
#staticmethod
def initializeConfig():
CONFIGPARSER.read(CONFIGFILEPATH)
...
...