We Keep Coding

Module or singleton that uses logging and config

I've been trying to figure out how best to set this up. Cutting it down as much as I can. I have 4 python files: core.py (main), logger_controler.py, config_controller.py, and a 4th as a module or singleton well just call it tool.py.
The way I have it setup is logging has an init function that setup pythons built in logging with the necessary levels, formatter, directory location, etc. I call this init function in main.
import logging
import logger_controller
def main():
logger_controller.init_log()
logger = logging.getLogger(__name__)
if __name__ == "__main__":
main()
config_controller is using configparser and is mainly a singleton as a controller for my config.
import configparser
import logging
logger = logging.getLogger(__name__)
class ConfigController(object):
def __init__(self, *file_names):
self.config_parser = configparser.ConfigParser()
found_files = self.config_parser.read(file_names)
if not found_files:
raise ValueError("No config file found.")
self._validate()
def _validate(self):
...
def read_config(self, section, field):
try:
data = self.config_parser.get(section, field)
except (configparser.NoSectionError, configparser.NoOptionError) as e:
logger.error(e)
data = None
return data
config = ConfigController("config.ini")
And then my problem is trying to create the 4th file and making sure both my logger and config parser are running before it. I'm also wanting this 4th one to be a singleton so it's following a similar format as the config_controller.
So tool.py uses config_controller to pull anything it needs from the config file. It also has some error checking for if config_controller's read_config returns None as that isn't validated in _validate. I did this as I wanted my logging to have a general layer for error checking and a more specific layer. So _validate just checks if required fields and sections are in the config file. Then wherever the field is read will handle extra error checking.
So my main problem is this:
How do I have it where my logger and configparser are both running and available before anything else. I'm very much willing to rework all of this, but I'd like to keep the functionality of it all.
One attempt I tried that works, but seems very messy is making my logger_controler a singleton that just returns python's logging object.
import logging
import os
class MyLogger(object):
def __new__(cls, *args, **kwargs):
init_log()
return logging
def init_log():
...
mylogger = MyLogger()
Then in core.py
from logger_controller import mylogger
logger = mylogger.getLogger(__name__)
I feel like there should be a better way to do the above, but I'm honestly not sure how.
A few ideas:
Would I be able to extend the logging class instead of just using that init_log function?
Maybe there's a way I can make all 3 individual modules such that they each initialize in a correct order? My attempts here didn't quite work as I also have some internal data that I wouldn't want exposed to classes using the module, just the functionality.
I'd like to have it where all 3, logging, configparsing, and the tool, available anywhere I import them.
How I have it setup now "works" but if I were to import the tool.py anywhere in core.py and an error occurs that I need to catch, then my logger won't be able to log it as this tool is loading before the init of my logger.

Customize log format in python Azure Functions

I am writing many Python Azure Functions. I want every line in logs to be prefixed with invocation-id from context to segregate and correlate the logs easily.
I know there are multiple ways to do this for a normal/stand-alone python application. Here Azure Function runtime provides an environment where it invokes my code. I don't-want-to/prefer-not-to:
mess around with existing handlers/formatters registered by Azure Function runtime or
write my own handlers/formatters
(because whatever is registered by default sends the logs to Azure Log Analytics workspace and powers my dashboards etc)
E.g. following code:
import logging
from azure import functions as func
def main(msg: func.QueueMessage, ctx: func.Context) -> None:
logging.info('entry')
logging.info('invocation id of this run: %s', ctx.invocation_id)
logging.debug('doing something...')
logging.info('exit with success')
will produce logs like:
entry
invocation id of this run: 111-222-33-4444
doing something...
exit with success
what I want instead is:
(111-222-33-4444) entry
(111-222-33-4444) invocation id of this run: 111-222-33-4444
(111-222-33-4444) doing something...
(111-222-33-4444) exit with success
I've seen some docs on Azure, seem useless.

You can use a LoggerAdapter to do this, as shown by the following runnable program:
import logging
class Adapter(logging.LoggerAdapter):
def process(self, msg, kwargs):
return '(%s) %s' % (self.extra['context'], msg), kwargs
def main(msg, ctx):
logger = Adapter(logging.getLogger(), {'context': ctx})
logger.info('entry')
logger.info('invocation id of this run: %s', ctx)
logger.debug('doing something ...')
logger.info('exit with success')
if __name__ == '__main__':
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
main('hello', '111-222-33-4444')
Obviously I've removed the Azure references so that I can run it locally, but you should get the gist. The preceding script prints
(111-222-33-4444) entry
(111-222-33-4444) invocation id of this run: 111-222-33-4444
(111-222-33-4444) doing something ...
(111-222-33-4444) exit with success
Update: If you don't want to/can't use LoggerAdapter, then you can subclass Logger as documented here or use a Filter as documented here. But in the latter case you'd still have to attach the filter to all loggers (or handlers, which would be easier) of interest.

Python remove logging file after running tests

I am testing the logger's functionality and it requires me to create a log file, but at the end I want to remove it so I tried to os.remove in the tearDownClass
#classmethod
def tearDownClass(cls) -> None:
log = logging.getLogger('client_logger')
try:
log.removeHandler(log.handlers.pop())
except:
pass
os.remove('client_logger.log')
I read that the RotatingFileHandler is the cause of it and once I remove it the handler list is empty, but it still gives me PermissionError: [WinError 32].

Log files work like any other ordinary files. It must be opened and closed.
The same goes for the log files. The log handlers opens a file to write into.
This means it before the file can be removed it should be disconnected to the log handler.
Example with Django TestCase:
import os
import logging
from django.test import TestCase
_logger = logging.getLogger("logger_name")
class CustomTestCase(TestCase):
def setUp(self) -> None:
... # stuff
def test_more_stuff(self) -> None:
... # more stuff
def tearDown(self) -> None:
# First close logger file before removing.
_logger.handlers[0].close() # <FileHandler D:\path\to\logfile\test.log (DEBUG)>
if os.path.exists("test.log"):
os.remove("test.log")
The following StackOverflow page helped me find a solution.
Reference: python does not release filehandles to logfile

Access pytest session or arguments in pytest_runtest_logreport

I am trying to build a pytest plugin that utilizes the pytest_runtest_logreport in order to invoke some code everytime a test fails. I'd like to gate this plugin with a CLI argument I have added using the pytest_addoption hook. Unfortunately I can't seem to figure out how to access the pytest session state or arguments inside the pytest_runtest_logreport hook. Is there a way to do this? I don't see it in the hookspec.

You can't get the session from the standard TestReport object. However, you can introduce a custom wrapper around the pytest_runtest_makereport hook (the one that creates the report object), where you can attach the session yourself. Example:
import pytest
#pytest.hookimpl(hookwrapper=True)
def pytest_runtest_makereport(item, call):
out = yield
report = out.get_result()
report.session = item.session
def pytest_runtest_logreport(report):
print(report.session)
Another example of passing state between hooks is a plugin class. Example with accessing config object in pytest_runtest_logreport:
import pytest
#pytest.mark.tryfirst
def pytest_configure(config):
p = MyPlugin(config)
config.pluginmanager.register(p, 'my_plugin')
class MyPlugin:
def __init__(self, config):
self.config = config
def pytest_runtest_logreport(self, report):
print(report, self.config)

How to configure/initialize logging using logger for multiple modules only once in Python for entire project?

I have python project with multiple modules with logging. I perform initialization (reading log configuration file and creating root logger and enable/disable logging) in every module before start of logging the messages. Is it possible to perform this initialization only once in one place (like in one class may be called as Log) such that the same settings are reused by logging all over the project?
I am looking for a proper solution to have only once to read the configuration file and to only once get and configure a logger, in a class constructor, or perhaps in the initializer (__init__.py). I don't want to do this at client side (in __main__ ). I want to do this configuration only once in separate class and call this class in other modules when logging is required.

setup using #singleton pattern
#log.py
import logging.config
import yaml
from singleton_decorator import singleton
#singleton
class Log:
def __init__(self):
configFile = 'path_to_my_lof_config_file'/logging.yaml
with open(configFile) as f:
config_dict = yaml.load(f)
logging.config.dictConfig(config_dict)
self.logger = logging.getLogger('root')
def info(self, message):
self.logger.info(message)
#module1.py
from Log import Log
myLog = Log()
myLog.info('Message logged successfully)
#module2.py
from Log import Log
myLog = Log() #config read only once and only one object is created
myLog.info('Message logged successfully)

From the documentation,
Note that Loggers should NEVER be instantiated directly, but always through the module-level function logging.getLogger(name). Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.

You can initialize and configure logging in your main entry point. See Logging from multiple modules in this Howto (Python 2.7).

I had the same problem and I don't have any classes or anything, so I solved it with just using global variable
utils.py:
existing_loggers = {}
def get_logger(name='my_logger', level=logging.INFO):
if name in existing_loggers:
return existing_loggers[name]
# Do the rest of initialization, handlers, formatters etc...