When using logging in python like this
import logging
logging.basicConfig(level=logging.INFO)
logger = None
if <some condition>:
logger = logging.getLogger(__name__)
... some code ...
logger.debug('message')
is it possible to avoid calling logger.debug if it does not exist without if statement?
That's not how you use logging: it's missing the point entirely. The point is to always just log, and then (at the app level) configure handlers appropriately so that they either record the debug level or not. Your app-level configuration can even set the handler for loggers in each module. See the Python logging docs for more info.
You could construct a wrapper object and use that everywhere:
class LoggerWrapper(object):
def __init__(self, logger):
self.logger = logger
def debug(self, *args, **kwargs):
if self.logger is None:
return
self.logger.debug(*args, **kwargs)
my_wrapper = LoggerWrapper(logger)
my_wrapper.debug('message')
That being said, this is not specific to Python's logging module. Any variable var that can also be None will need to be checked before you call a method on it (var.method()).
It may be easier to always define a logger and to simply disable the logging through the logging configuration / settings. That keeps the code clean, all logger calls will succeed but the output won't go anywhere if that particular logger is disabled.
Lastly, you could opt to indeed use an if statement as it's not inherently that much more code:
if logger:
logger.debug('message')
is it possible to avoid calling logger.debug if it does not exist without if statement?
Well, you could (ab)use an if expression, or an or expression, or various other things, but otherwise, not really.
You can, however, always write a wrapper function:
def logdebug(*args):
if logger:
logger.debug(*args)
logdebug('message')
logdebug('other message')
You can do other, similar tricks. For example, instead of having logger = None, you can create an object that has a debug method that does nothing:
class Dummy(object):
def debug(self, *args): pass
logger = Dummy()
Or, even better, always create a logger and configure it with a null handler, or—simplest of all—just set its logging level higher than debug.
But note that none of these will completely replace the if statement. Consider, for example:
if logger:
logger.debug(very_expensive_function())
logdebug(very_expensive_function())
The first one will only call very_expensive_function to create the log message if you actually need it; the second one will always call it and then just throw it away if you don't need it.
Related
I get a regular non-root logger of the default type:
logger = logging.getLogger('a')
Now I want to change the type and get a new logger for the same name:
logging.setLoggerClass(type('NewLoggerClass', (logging.Logger,), {}))
logger = logging.getLogger('a')
I would like the second call to getLogger to respect the new logger class I set. However, logger remains the same instance of logging.Logger instead of __main__.NewLoggerClass.
How do I either remove an existing logger for a given name, or replace it with one of a different type?
You can't do this (other than low-level hackery) once a logger has been created. You'll need to call setLoggerClass() before you instantiate any loggers via getLogger().
If in doubt ... "Use the source, Luke!"
So spelunking into logging/__init__.py, one can see that a call to logging.getLogger actually invokes Logger.manager.getLogger.
Logger.manager is itself an instance of Manager that was instatiated on module load.
Hunting in the Manager class shows that it checks for existing logger objects in self.loggerDict. This attribute acts as the 'cache' for instantiated logger class.
So if you want to replace a certain logger with a new logger class, you probably can do it by deleting it first. Maybe like this:
del logging.Logger.manager.loggerDict['a']
logging.setLoggerClass(type('NewLoggerClass', (logging.Logger,), {}))
logger = logging.getLogger('a')
Try it out!
I have the following application structure:
./utils.py
def do_something(logger=None):
if not logger:
logger = logging.getLogger(__name__)
print('hello')
logger.debug('test')
./one.py
from utils import do_something
logger = logging.getLogger(__name__)
do_something(logger=logger)
./two.py
from utils import do_something
logger = logging.getLogger(__name__)
do_something(logger=logger)
Now, then this runs, the logging output will show the names of the respective modules that are using the functions (one and two) rather than utils. I then use this information (the logger's name) to filter the messages.
Is there a way to do this without having to pass the logger though as an argument? Would I have to basically introspect the caller function and initialize a logger based on that? What's the common solution for this type of recipe, when one function is used all over the code base but the loggers should be of those calling the function?
Variable logger being a global, can be accessed from inside do_something() function like this:
logger = logging.getLogger(__name__)
def do_something():
x = logger
After reading this carefully:
What's the common solution for this type of recipe, when one function
is used all over the code base but the loggers should be of those
calling the function?
In simple english would be:
How to access global variable logger from imported function do_something()?
I conclude that there is no other way!
You have to pass logger as an argument for this particular case.
from utils import do_something
logger = logging.getLogger(__name__)
do_something() # logger global is not visible in do_something()
do_something(logger=logger) # the only way
Passing the logger as an argument looks fine to me. But if you have many calls to do_something then passing the same logger in each call is not only too verbose but prone to errors --you might pass the wrong logger by mistake. Also, while you do need to make do_something aware of the logger, doing it at the point where you are calling it probably does not make sense --you want to do something, not mess with loggers. OOP to the rescue: make do_something a method of a class and pass the logger when instantiating the class, then you do not need to pass it to the method.
# utils.py
class UsefulObject:
def __init__(self, logger):
self._logger = logger
def do_something(self):
print('hello')
self._logger.debug('test')
Now in the client module you create an instance of UsefulObject and you can call .do_something on it without passing the logger every time.
# one.py
from utils import UsefulObject
logger = logging.getLogger(__name__)
useful_object = UsefulObject(logger) # do this once
# ...
useful_object.do_something() # do this many times, do not pass the logger
If you don't like the existing solutions, here's a magic way to do it:
def do_something():
logger = inspect.currentframe().f_back.f_globals['log']
...
The calling function must then have a global variable called log.
I have the following application structure:
./utils.py
def do_something(logger=None):
if not logger:
logger = logging.getLogger(__name__)
print('hello')
logger.debug('test')
./one.py
from utils import do_something
logger = logging.getLogger(__name__)
do_something(logger=logger)
./two.py
from utils import do_something
logger = logging.getLogger(__name__)
do_something(logger=logger)
Now, then this runs, the logging output will show the names of the respective modules that are using the functions (one and two) rather than utils. I then use this information (the logger's name) to filter the messages.
Is there a way to do this without having to pass the logger though as an argument? Would I have to basically introspect the caller function and initialize a logger based on that? What's the common solution for this type of recipe, when one function is used all over the code base but the loggers should be of those calling the function?
Variable logger being a global, can be accessed from inside do_something() function like this:
logger = logging.getLogger(__name__)
def do_something():
x = logger
After reading this carefully:
What's the common solution for this type of recipe, when one function
is used all over the code base but the loggers should be of those
calling the function?
In simple english would be:
How to access global variable logger from imported function do_something()?
I conclude that there is no other way!
You have to pass logger as an argument for this particular case.
from utils import do_something
logger = logging.getLogger(__name__)
do_something() # logger global is not visible in do_something()
do_something(logger=logger) # the only way
Passing the logger as an argument looks fine to me. But if you have many calls to do_something then passing the same logger in each call is not only too verbose but prone to errors --you might pass the wrong logger by mistake. Also, while you do need to make do_something aware of the logger, doing it at the point where you are calling it probably does not make sense --you want to do something, not mess with loggers. OOP to the rescue: make do_something a method of a class and pass the logger when instantiating the class, then you do not need to pass it to the method.
# utils.py
class UsefulObject:
def __init__(self, logger):
self._logger = logger
def do_something(self):
print('hello')
self._logger.debug('test')
Now in the client module you create an instance of UsefulObject and you can call .do_something on it without passing the logger every time.
# one.py
from utils import UsefulObject
logger = logging.getLogger(__name__)
useful_object = UsefulObject(logger) # do this once
# ...
useful_object.do_something() # do this many times, do not pass the logger
If you don't like the existing solutions, here's a magic way to do it:
def do_something():
logger = inspect.currentframe().f_back.f_globals['log']
...
The calling function must then have a global variable called log.
For example, i have these files with classes that i want to use (only example, maybe not working)
# helper_one.py
import logging
logger = logging.getLogger('HelperOne')
class HelperOne:
def __init__(self, data):
self.data = data
def run(self):
logger.debug('HelperOne::Run function with data: ' + self.data)
return True
.
# controller_one.py
import logging
from helper_one import HelperOne
logger = logging.getLogger('ControllerOne')
class ControllerOne:
def __init__(self, data):
self.data = data
def run(self):
logger.debug('ControllerOne::Run function with data: ' + self.data)
ho = HelperOne(self.data + '_data')
return ho.run()
And i have a file which creates Threads
import logging
from controller_one import ControllerOne
# this function creates threads (its just an example)
def run_task(task_id, data):
logging.basicConfig(
filename=os.path.join('logs', str(task_id) + '.log'),
level=logging.DEBUG,
format='%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s'
)
result = ControllerOne(data)
logging.debug('run_task: ' + result)
If i create logging instance like in my example, all log are written in a single file.
How can I, for every thread, create its own logging instance with logging to its own file ?
Regards, Alex.
The logger name (the string passed to the getLogger function) can be anything you want. As the docs explain:
The name is potentially a period-separated hierarchical value, like foo.bar.baz (though it could also be just plain foo, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of foo, loggers with names of foo.bar, foo.bar.baz, and foo.bam are all descendants of foo. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise your loggers on a per-module basis using the recommended construction logging.getLogger(__name__). That’s because in a module, __name__ is the module’s name in the Python package namespace.
So, using __name__ as the name is recommended, but not required. And in fact you're already explicitly violating it:
logger = logging.getLogger('ControllerOne')
So, you could just use a separate name per thread, by putting a thread ID or name into the logger name. For example:
logger = logging.getLogger('ControllerOne.{}'.format(threading.get_ident())
Or, in your case, since you seem to have a unique task_id for each thread, you can just use that instead of the thread ID.
Of course this means you have to actually use the logger; you can't just call logging.debug. And you can't rely on the basicConfig; you're going to need to explicitly configure the logger in each thread. Since each thread creates its own independent ControllerOne instance, the obvious place to do that is in ControllerOne.__init__; in that function, you can call getLogger with a logger name that incorporates the thread or task ID, create and set a FileHandler that also incorporates the thread or task ID in the name, and store it as self.logger. Then, whenever you need to log something, you just do self.logger.debug(…).
If that all sounds like gobbledegook to you because you have no idea what a FileHandler is, you probably need to read at least the basic tutorial in the HOWTOs, but the advanced tutorial and cookbook are also very useful.
The following dev enviroment is to be considered.
Small number of python modules, each contanining one or more classes. A main module, that calls those classes. A custom logging module called logger with a class called Logger. Assume that I call the main execution class with a logging level of debug. How may I make this sufficient to be that log level inherited to every other call including the rest of the classes, methods in those classes, functions in the main module and so forth...
The Logger objects are called like log=Logger(0) for example (logging level is an int, to use the same syntax that we use in other scripts (shell scripts, not python).
My final picture is to have the code filled with log.log_debug('debug message') and log.log_error('error message') but only actually printing a message when the right log_level is choosen. And if possible just one
from logger import Logger
call within the main module.
Thanks for your time.
==================================
Edit
In the main execution module there is a main() function in which a parser.parse_args() object is returned with an argument --log_level to globally define (at least is my intention) the log_level. There is a default log_level handling (i.e, it is always defined)
I will try to mock a minimum example
import argparse
from logfile from Logfile
from logger import Logger
def argument_parser():
parser=argparse.ArgumentParser()
stuff
return parser.parse_args()
def log_file_stuff():
log_file=Logfile()
log_file.methods() [*]
def main():
args=argument_parser()
# Here log_level is defined with args.log_level
global log
log=log(log_level)
log_file_stuff()
main()
[*] One of those methods may call a Logger object, which I want to be exactly the same as the one defined in the main() function the question is how may I achieve this? without a log return and argument waterfall
Well to answer my own question... I actually implemented a cross module variable dictionary to use it as settings. Those settings have the log_level variable son only with
from main import settings
from logger import Logger
log=Logger(settings['log_level'])
then I have my custom logging class with the user input log_level. Of course settings is constructed with argparse, mainly as vars(args) following the notation in the question.