I am fairly new to Python. I have been modifying our existing python modules for improved logging. I came across with a requirement to tag each method call(and exit) with a specific key (context) that needs to be logged and entry and exit points. That key is supposed to be generated at root method and passed down to further method calls.
I am confused over the approach I should follow. Is there a way to dynamically insert an extra parameter in existing method interface may be using AOP or annotations ?
Or I rephrase this question to short -
How can I pass an extra argument to subsequent method calls ?
Thanks
Panks
If you have a single logger instance for all those methods, maybe you could have the root method set the context on the logger on entry and remove it on exit.
#contextlib.contextmanager
def setup_login(logger, context):
old_name = logger.name
logger.name = context
yield
logger.name = old_name
And the root method becomes :
def root():
context = compute_context()
with setup_login(logger, context):
# call other methods
I used logger.name, but you could use any other attribute of logger, or use any other singleton to share the global state.
Related
I have the below context manager inside a class method, which I would like to mock for unit testing.
def load_yaml_config(self) -> dict:
"""
Load the config based on the arguments provided.
Returns: dict
dictionary which will be used for configuring the logger, handlers, etc.
"""
with open(self.yaml_path, 'r') as config_yaml:
return yaml.safe_load(config_yaml.read())
How could I achieve it?
EDIT:
As #chepner suggested (I can't accept his/her answer since it was via comment), the best way to go seems to be by using the unittest's mock_open functionality.
This way, I can simply go:
import unittest.mock as um
with um.patch('builtins.open', um.mock_open(read_data=YAML_TEST)):
h = MyClass.load_yaml_config()
If you wanted to refactor this code to be able to test the safe_load part without having to actually open a file or patch builtins.open, you could do:
def load_yaml_config(self) -> dict:
"""
Load the config based on the arguments provided.
Returns: dict
dictionary which will be used for configuring the logger, handlers, etc.
"""
with open(self.yaml_path, 'r') as config_yaml:
return self._load_yaml_config(config_yaml.read())
def _load_yaml_config(self, yaml_text: str) -> dict:
return yaml.safe_load(yaml_text)
and then in your test:
TEST_YAML_DATA = """
stuff:
other_stuff
"""
def test_load_yaml_config():
assert WhateverMyClassIs()._load_yaml_config(TEST_YAML_DATA) == {
'stuff': 'other_stuff'
}
Modify to use actual appropriate YAML formatting and the correct expected dict output.
Note that all this is really testing is yaml.safe_load (which should have its own unit tests already) and the fact that your code calls it. Other than a typo in a variable name (which is easier to catch with a linter or static type analyzer), it's hard to imagine what type of bug this test might catch/prevent.
Practically speaking I probably wouldn't bother covering this function in a unit test at all, but would instead try to have some sort of larger integration test (using a real file) that involved loading a config as part of some larger test scenario.
The easy way
Just create the appropriate yaml file in your testing code. But you probably don't want that, since you're making this post.
A hack with mocking
You can override open with your mock in the module scope:
# test_YourClass.py
builtin_open = open
class open:
def __init__(self, *args, **kwargs):
pass
def __enter__(self):
pass
def __exit__(self, exc_type, exc_value, exc_traceback):
pass
def read(self):
return 'hardcoded file contents for testing'
# Test here
open = builtin_open
This code is just a general idea, I haven't run it. It might require some additional work, such as parameterizing the mock file contents.
Dependency injection
The "proper" way is to unhardcode open() call in the class and inject your context manager, I suppose. It's up to you. I personally don't like injecting everything just for the purpose of unit testing.
I have the following problem. I have multiple unittests, each one having a context manager to open a browser and do some selenium testing.
I want to ensure that I can run the tests in parallel, and that the browser window is closed in case of error, so I use a context manager:
def test_xxx():
with webapp() as p:
file_loader = FileLoader(p, id="loader").upload(path)
As you can see, I use a class FileLoader, which takes the context manager web application (basically a wrapper of the selenium driver) and uses it to encapsulate the rigmarole required to upload the file.
My objective would be not to have to specify the p parameter to FileLoader(), so that I could write
def test_xxx():
with webapp():
file_loader = FileLoader(id="loader").upload(path)
I could use a global that is assigned when the context manager is opened, but this would prevent any isolation when tests run in parallel. Suppose that one test connects to site A, and another test connects to site B. I need two drivers, each connected to a different site.
In other words, how can I design FileLoader to be aware of its enclosing context manager without passing the context variable?
By using the inspect module, a code can read the local variables of its caller. It is a rather unusual if not dangerous use, because it actually boils down to a non standard and non conventional way of passing a parameter to a function. But if you really want to go that way, this other SO question gives a possible way.
Demo:
class ContextAware:
""" Class that will copy the x local variable of its caller if any"""
def __init__(self):
# uncomment next line for debugging
# print("ContextAware", inspect.currentframe().f_back.f_locals)
self.x = inspect.currentframe().f_back.f_locals.get('x')
def foo(x):
# the value of x is not explicitely passed, yet it will be used...
return ContextAware()
The object created in foo is aware of the x variable of its caller:
>>> a = foo(4)
>>> a.x
4
>>> a = foo(6)
>>> a.x
6
That means you you could write something close to:
def test_xxx():
with webapp() as ctx_p:
file_loader = FileLoader(id="loader").upload(path)
provided the __init__ method of FileLoader spies on the ctx_p local variable of its caller.
For example, i have these files with classes that i want to use (only example, maybe not working)
# helper_one.py
import logging
logger = logging.getLogger('HelperOne')
class HelperOne:
def __init__(self, data):
self.data = data
def run(self):
logger.debug('HelperOne::Run function with data: ' + self.data)
return True
.
# controller_one.py
import logging
from helper_one import HelperOne
logger = logging.getLogger('ControllerOne')
class ControllerOne:
def __init__(self, data):
self.data = data
def run(self):
logger.debug('ControllerOne::Run function with data: ' + self.data)
ho = HelperOne(self.data + '_data')
return ho.run()
And i have a file which creates Threads
import logging
from controller_one import ControllerOne
# this function creates threads (its just an example)
def run_task(task_id, data):
logging.basicConfig(
filename=os.path.join('logs', str(task_id) + '.log'),
level=logging.DEBUG,
format='%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s'
)
result = ControllerOne(data)
logging.debug('run_task: ' + result)
If i create logging instance like in my example, all log are written in a single file.
How can I, for every thread, create its own logging instance with logging to its own file ?
Regards, Alex.
The logger name (the string passed to the getLogger function) can be anything you want. As the docs explain:
The name is potentially a period-separated hierarchical value, like foo.bar.baz (though it could also be just plain foo, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of foo, loggers with names of foo.bar, foo.bar.baz, and foo.bam are all descendants of foo. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise your loggers on a per-module basis using the recommended construction logging.getLogger(__name__). That’s because in a module, __name__ is the module’s name in the Python package namespace.
So, using __name__ as the name is recommended, but not required. And in fact you're already explicitly violating it:
logger = logging.getLogger('ControllerOne')
So, you could just use a separate name per thread, by putting a thread ID or name into the logger name. For example:
logger = logging.getLogger('ControllerOne.{}'.format(threading.get_ident())
Or, in your case, since you seem to have a unique task_id for each thread, you can just use that instead of the thread ID.
Of course this means you have to actually use the logger; you can't just call logging.debug. And you can't rely on the basicConfig; you're going to need to explicitly configure the logger in each thread. Since each thread creates its own independent ControllerOne instance, the obvious place to do that is in ControllerOne.__init__; in that function, you can call getLogger with a logger name that incorporates the thread or task ID, create and set a FileHandler that also incorporates the thread or task ID in the name, and store it as self.logger. Then, whenever you need to log something, you just do self.logger.debug(…).
If that all sounds like gobbledegook to you because you have no idea what a FileHandler is, you probably need to read at least the basic tutorial in the HOWTOs, but the advanced tutorial and cookbook are also very useful.
Is there any equivalent of strict mocks in python? Some mechanism to report unintended call of mocked methods (action.step2() in this example), just like this in GoogleMock framework.
class Action:
def step1(self, arg):
return False
def step2(self, arg):
return False
def algorithm(action):
action.step1('111')
action.step2('222')
return True
class TestAlgorithm(unittest.TestCase):
def test_algorithm(self):
actionMock = mock.create_autospec(Action)
self.assertTrue(algorithm(actionMock))
actionMock.step1.assert_called_once_with('111')
Looks like it's not supported out of the box. However there are at least two approaches on how to achieve the same result.
Passing list of allowed members
According to mock documentation
spec: This can be either a list of strings or an existing object (a class or instance) that acts as the specification for the mock object. If you pass in an object then a list of strings is formed by calling dir on the object (excluding unsupported magic attributes and methods). Accessing any attribute not in this list will raise an AttributeError.
So, in order to fail your test example just replace
actionMock = mock.create_autospec(Action)
to
actionMock = mock.Mock(spec=['step1'])
Such an approach have certain drawbacks compared to passing class or instance as spec argument, as you have to pass all the allowed methods and than set up expectations on them, effectively registering them twice. Also, if you need to restrict a subset of methods you have to pass list of all methods execept those. This can be achieved as follows:
all_members = dir(Action) # according to docs this is what's happening behind the scenes
all_members.remove('step2') # remove all unwanted methods
actionMock = mock.Mock(spec=all_members)
Setting exceptions on restricted methods
Alternative approach would be to excplicitly set failures on methods you don't want to be called:
def test_algorithm(self):
actionMock = mock.create_autospec(Action)
actionMock.step2.side_effect = AttributeError("Called step2") # <<< like this
self.assertTrue(algorithm(actionMock))
actionMock.step1.assert_called_once_with('111')
This have some limitations as well: you've got to set errors as well as expectations.
As a final note, one radical solution to the problem would be to patch mock to add strict parameter to Mock constructor and send a pull request. Than either it would be accepted or mock maintainers will point out on how to achieve that. :)
Yes, this is possible using the spec= and autospec= arguments. See the mock documentation on Autospeccing for more information. In your example it would become:
action_mock = mock.Mock(spec=Action)
or:
action_mock = mock.Mock('Action', autospec=True)
Another possibility:
Checking call_count individually on restricted methods
Ensure that call_count is zero on methods that should not be called.
class TestAlgorithm(unittest.TestCase):
def test_algorithm(self):
actionMock = mock.create_autospec(Action)
self.assertTrue(algorithm(actionMock))
actionMock.step1.assert_called_once_with('111')
self.assertEqual(actionMock.step2.call_count, 0) # <<< like this
The drawback is that you have to check all unexpected calls one by one.
I've decorated a method in Python. And when I import the module that contains the method, the decorator autoruns.
I realize that this is how decorators were made however Is there a way to have decorators NOT do this?
It sounds like what you want to do is to choose what decorator to apply at run time. Something like this might work:
to_decorate = []
def decorate_later(func):
to_decorate.append(func)
return func
#decorate_later
def do_stuff(*args, **kw):
print('I am doing stuff')
#decorate_later
def do_more_stuff(*args, **kw):
print('Even more stuff')
def apply_decorator(decorator):
for func in to_decorate:
globals()[func.func_name] = decorator(func)
Then you can import the module and all the functions will be defined as normal. decorate_later returns the original function unmodified. You can call apply_decorator() to apply a specified decorator to all of the functions in the module that were registered by #decorate_later
This is exactly what the venusian library does; you define your decorators according to their API, but the actual behavior isn't triggered until you do a "scan" of the containing module or package.
You don't even need to have a global app object to use venusian decorators; you can pass in the app object as part of the scan, and it'll get passed along to the decorator implementations. So, for example, the same functions can be shared among multiple owners with only a single decorator, just by doing more than one scan.
This is what the Pyramid web framework uses for e.g. event registration, so that merely importing a module doesn't expect to need an app instance. A good example is their event subscriber.
Use
if __name__ == "__main__":
#code
in the file, where code is all outside a method or class ( that runs when you import it).