We Keep Coding

In Python, how can I add another module to an already existing variable that contains one?

Is it possible use a variable as a container for a Python module, and then adding another one to the same variable?
So for example, if I would have a Python file called general_config.py containing a general config of some kind:
class GeneralConfig:
attribute1 = "Some attribute"
attribute2 = "Some other attribute"
And if I would import this Python module as a variable containing a general config, I would do:
import general_config.py as Config
Then I can access its attributes by doing:
generalParameter = Config.GeneralConfig.attribute1
But what if I want to add some specific parameters to my config (say from specific_config.py), while keeping the general one as part of the entire config? So it would do something like that:
if someSpecificCondition:
Config += import specific_config.py
else:
Config += import other_config.py
While keeping the Config in the original scope? Thanks in advance.

If you want your general config to inherit your other configs for whatever reason, you could do something like this. But Tom's answer makes more sense, since there's no runtime class creation.
class BaseConfig:
att = "hello world"
def inherit_configs(some_condition):
if some_condition:
from config1 import Config1
class Config(BaseConfig, Config1):
pass
return Config
else:
from config2 import Config2
class Config(BaseConfig, Config2):
pass
return Config
config = inherit_configs(some_condition)()

Sharing variables across nested folders

Problem
Many people say their variable-sharing problems are resolved with the approach provided here and here, but neither of them works in my use case, where all configurations have to be written in a class like following.
# config.py
class Config(object):
var = None
The structure of my project looks like
├── config.py
├── main.py
├── subfolder
|  ├── change_config.py
In this project, main.py will invoke change_config.py and they both have access to variables defined in config.py. Importantly, change_config.py will modify the variables (aka., var) whose value is only known during runtime.
But it is not clear how I should share the instantiated Config() (aka. opt) across main.py and change_config.py. I tried following but no luck. The issue is
If I instantiated another Config() in change_config.py, the one in main.py will be wiped out.
If I do not do the first one, then name opt would not be resolved in change_config.py
# main.py
import config
from subfolder import change_config
opt = config.Config()
print(opt.var)
change_config.change_config()
print(opt.var)
# change_config.py
import config
def change_config():
opt.var = 10
More information
If I run main.py, then I will have
NameError: name 'opt' is not defined
which is expected since opt is never declared in change_config.py.
If I change change_config.py into
# change_config.py
import config
opt = config.Config()
def change_config():
opt.var = 10
where I declared another opt. There is no error but returned
None
None
which is also expected since opt declared in main.py is wiped by the one in change_config.py. But the expected output should be
None
10
So the question is how to share opt in main.py with change_config.py

Let's start by looking at what the differences between Config.var and Config().var are in python terms. Like everything else in Python, a class is an object. A class has a namespace, so Config.var is a variable in that namespace.
When you instantiate Config, say with opt = Config(), opt is now an object, also with a namespace. At first, when you request the value opt.var, it returns Config.var because of how inheritance works in general: if an object does not have an attribute, you look in the class, and then in the parent class, etc. But when you assign opt.var = 10, you are assigning into the namespace of the instance. Now, when you request the value of opt.var, it does not look at Config.var any more, because opt has its own var attribute.
Now keep in mind that opt in main and opt in change_config are completely different instances that do not affect each other except that they share the same class. So when you set change_config.opt.var = 10, requesting main.opt.var still returns None, because the actual attribute main.opt.var does not exist as such: it is returning Config.var from the class namespace.
So now you have a couple of options available to you.
The first is not to bother instantiating anything at all. You can keep your class variable config.Config.var, and just update in the class namespace. The code would look like this:
main.py
from config import Config
from subfolder.change_config import change_config
print(Config.var)
change_config()
print(Config.var)
config.py
class Config:
var = None
subfolder/change_config.py
from config import Config
def change_config():
Config.var = 10
Aside from changing your imports to relative imports and removing the explicit inheritance from object, your code is only modified to use Config as a namespace, without instantiating it.
A second approach would be to instantiate Config, but put the reference in a place that can be accessed by everyone that needs it. At that point, config.py should really be maintaining the shared reference, since that is what stores the actual configuration. I would also recommend removing the class variable Config.var entirely, to avoid exactly the type of confusion you are having now:
main.py
from config import opt
from subfolder.change_config import change_config
print(opt.var)
change_config()
print(opt.var)
config.py
class Config:
def __init__(self):
self.var = None
opt = Config()
subfolder/change_config.py
from config import opt
def change_config():
opt.var = 10
You could commit to making your configuration an instance of Config even deeper by adding del Config to the end of config.py. This will make it much more difficult to create another, conflicting instance of the class, since it will not be readily available through the module. The only reference to Config will be through the inheritance hierarchy of opt at that point.

you can change your application like below, this will help you
in config.py
x = 0
in change_config.py
import config
def change_config():
config.x = 10 #change the global config variable
and in main.py
import config
from subfolder import change_config
print(config.x)
change_config.change_config()
print(config.x)
you will get output like below after you run main.py
0
10
EDIT
using singleton class
in config.py
class Config(object):
var = None
def __new__(cls):
if not hasattr(cls, 'instance'):
cls.instance = super(Config, cls).__new__(cls)
return cls.instance
in main.py
from config import Config
from subfolder import change_config
opt = Config()
print('memory address :%s'%id(opt))
print(opt.var)
change_config.change_config()
print(opt.var)
in change_config.py
from config import Config
opt = Config()
print('memory address :%s'%id(opt))
def change_config():
opt.var = 10
You will get output like below
memory address :140185708587536
memory address :140185708587536
None
10

Pytest mock: how to patch an import statement inside a function [duplicate]

Is it possible to mock a module in python using unittest.mock? I have a module named config, while running tests I want to mock it by another module test_config. how can I do that ? Thanks.
config.py:
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
test_config.py:
CONF_VAR1 = "test_VAR1"
CONF_VAR2 = "test_VAR2"
All other modules read config variables from the config module. While running tests I want them to read config variables from test_config module instead.

If you're always accessing the variables in config.py like this:
import config
...
config.VAR1
You can replace the config module imported by whatever module you're actually trying to test. So, if you're testing a module called foo, and it imports and uses config, you can say:
from mock import patch
import foo
import config_test
....
with patch('foo.config', new=config_test):
foo.whatever()
But this isn't actually replacing the module globally, it's only replacing it within the foo module's namespace. So you would need to patch it everywhere it's imported. It also wouldn't work if foo does this instead of import config:
from config import VAR1
You can also mess with sys.modules to do this:
import config_test
import sys
sys.modules["config"] = config_test
# import modules that uses "import config" here, and they'll actually get config_test
But generally it's not a good idea to mess with sys.modules, and I don't think this case is any different. I would favor all of the other suggestions made over it.

foo.py:
import config
VAR1 = config.CONF_VAR1
def bar():
return VAR1
test.py:
import unittest
import unittest.mock as mock
import test_config
class Test(unittest.TestCase):
def test_one(self):
with mock.patch.dict('sys.modules', config=test_config):
import foo
self.assertEqual(foo.bar(), 'test_VAR1')
As you can see, the patch works even for code executed during import foo.

If you want to mock an entire module just mock the import where the module is used.
myfile.py
import urllib
test_myfile.py
import mock
import unittest
class MyTest(unittest.TestCase):
#mock.patch('myfile.urllib')
def test_thing(self, urllib):
urllib.whatever.return_value = 4

Consider this following setup
configuration.py:
import os
class Config(object):
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
class TestConfig(object):
CONF_VAR1 = "test_VAR1"
CONF_VAR2 = "test_VAR2"
if os.getenv("TEST"):
config = TestConfig
else:
config = Config
now everywhere else in your code you can use:
from configuration import config
print config.CONF_VAR1, config.CONF_VAR2
And when you want to mock your coniguration file just set the environment variable "TEST".
Extra credit:
If you have lots of configuration variables that are shared between your testing and non-testing code, then you can derive TestConfig from Config and simply overwrite the variables that need changing:
class Config(object):
CONF_VAR1 = "VAR1"
CONF_VAR2 = "VAR2"
CONF_VAR3 = "VAR3"
class TestConfig(Config):
CONF_VAR2 = "test_VAR2"
# CONF_VAR1, CONF_VAR3 remain unchanged

If your application ("app.py" say) looks like
import config
print config.var1, config.var2
And gives the output:
$ python app.py
VAR1 VAR2
You can use mock.patch to patch the individual config variables:
from mock import patch
with patch('config.var1', 'test_VAR1'):
import app
This results in:
$ python mockimport.py
test_VAR1 VAR2
Though I'm not sure if this is possible at the module level.

Access instance in other modules

I have a class instance I want to access in other modules. This class loads config values using configParser to update an class instance __dict__ attribute as per this post:
I want to access this instance in other module. The instance is only created in the main.py file where it has access to the required parameters, which come via command line arguments.
I have three files: main.py, config.py and file.py. I don't know the best way to access the instance in the file.py. I only have access to it in main.py and not other modules.
I've looked at the following answers, here and here but they don't fully answer my scenario.
#config.py
class Configuration():
def __init__(self, *import_sections):
#use configParser, get config for relevant sections, update self.__dict__
#main.py
from config import Configuration
conf = Configuration('general', 'dev')
# other lines of code use conf instance ... e.g. config.log_path in log setup
#file.py
#I want to use config instance like this:
class File():
def __init__(self, conf.feed_path):
# other code here...
Options considered:
Initialise Configuration in config.py module
In config.py after class definition I could add:
conf = Configuration('general', 'dev')
and in file.py and main.py:
from config import conf
but the general and dev variables are only found in main.py so doesn't look like it will work.
Make Configuration class a function
I could make it a function and create a module-level dictionary and import data into other modules:
#config.py
conf = {}
def set_config(*import_section):
# use configParser, update conf dictionary
conf.update(...)
This would mean referring to it as config.conf['log_path'] for example. I'd prefer conf.log_path as it's used multiple times.
Pass via other instances
I could pass the conf instance as parameters via other class instances from main.py, even if the intermediate instances don't use it. Seems very messy.
Other options?
Can I use Configuration as an instance somehow?

By changing your Configuration class into a Borg, you are guaranteed to get a common state from wherever you want. You can either provide initialization through a specific __init__:
#config.py
class Configuration:
__shared_state = {}
def __init__(self, *import_sections):
self.__dict__ = self.__shared_state
if not import_sections: # we are not initializing this time
return
#your old code verbatim
initialization is donne as usual with a c = config.Configuration('general','dev') and any call to conf = config.Configuration() will get the state that c created.
or you can provide an initialization method to avoid tampering with the shared state in the __init__:
#config.py
class Configuration:
__shared_state = {}
def __init__(self):
self.__dict__ = self.__shared_state
def import(self, *import_sections):
#your old __init__
that way there is only one meaning to the __init__ method, which is cleaner.
In both cases, you can get the shared state, once initialized, from anywhere in your code by using config.Configuration().

How to use a 'global' variable in python?

In my problem I have a python code that is being started by a user, like:
# file main.py
import sys
import mymodule
config = sys.argv[1]
which imports another module containing functions, classes etc. like
# module.py
def check_config():
# do something with the content of 'config'
class Module(object):
# does something with the content of 'config'
How can I access the value of 'config' from within module.py? Should I use a 'global' variable here? Or is there a more sophisticated, pythonic way to solve this problem?
Also, I do not want to define an argument 'config' to each function and class I use in other modules...
Further remark: main.py imports other modules, not the other way around...

Instead of trying to wrangle global into performing this you should pass config as a parameter.
file main.py
import sys
import mymodule
config = sys.argv[1]
checked = mymodule.check_config(config)
mod = mymodule.Module(config)
module.py
def check_config(config):
# do something with the content of 'config'
class Module(object):
# does something with the content of 'config'
def __init__(self, config):
# initialise with config
Always avoid usingglobal when you can. If you need to modify config just have a module function return it.
config = change_config(config)
module.py
def change_config(config):
...
return config
However, an alternative method is to define a value within module.py which will store this information that holds nothing by default. Then as soon as file main.py has imported module.py and the config data is ready, you could assign the data to module.py's config name. Like this:
file main.py
import sys
import mymodule
config = sys.argv[1]
mymodule.config = config
mymodule.test_config()
mymodule.check_config()
mymodule.Module()
module.py
config = None
def test_config():
print config
# this will refer to the value supplied from file main.py
Note however, that the values in the module and main file will not be joined. If you reassign config in file main.py for any reason you have to pass that value to the module again. However if you pass a mutable value like a dict or list then you can modify it in file main.py and the values will be shared.

I don't recommend using a global variable, but here's the design you should use if you do. config needs to be defined in mymodule; after you import the module, you can set the value of mymodule.config the way you are currently setting config.
# file main.py
import sys
import mymodule
mymodule.config = sys.argv[1]
# module.py
# The exact value doesn't matter, as long as we create the name.
# None is good as it conveys the lack of a value; it's part of your
# module's contract, presumably, that a proper value must be assigned
# before you can use the rest of the module.
config = None
def check_config():
# do something with the content of 'config'
class Module(object):
# does something with the content of 'config'

A global variable is almost never the answer. Just allow the functions and classes in your "library" (module.py or mymodule.py, you seem to use both) to accept arguments. So:
mymodule.py
def check_config(configuration):
pass
class Module(object):
def __init__(self, configuration):
self.config = configuration
class ConfigError(Exception):
pass
Then when you want to use them in your "application" code:
main.py
import sys
import mymodule
config = sys.argv[1]
if mymodule.check_config(config):
myobject = mymodule.Module(config)
else:
raise mymodule.ConfigError('Unrecognized configuration format.')

Could you describe that you app should to do? Because now it's not clear, why you want it.
Maybe environment variable could help you?
Btw, you can read config file in one place (module), and import all stuff you need from it.
config.py
import os
if os.environ['sys'] == 'load_1':
import load_1 as load
i = 12
else:
import load_2 as load
i = 13
main.py
import config
config.load("some_data")
print config.i