We Keep Coding

Difference between logger and __name__ in logger = logging.getLogger(__name__)

According to the official logging documentation,
logging.getLogger([name])
Return a logger with the specified name or, if no name is specified, return a logger which is the root logger of the hierarchy. If specified, the name is typically a dot-separated hierarchical name like “a”, “a.b” or “a.b.c.d”. Choice of these names is entirely up to the developer who is using logging.
What is the difference between the name we are passing as argument and the name of the object we are actually catching the logger into?
ie difference between logger and __name__ in the statement
logger = logging.getLogger(__name__)

What is the difference between the name we are passing as argument and the name of the object we are actually catching the logger into?
That's absolutely different things:
logger in your case is the way to access logger object later in your code (most probably, in scope of single file). logger acts as local variable pointing to logger object. You can, actually, call your .log with
getLogger(__name__).info('My data')
but it's just less practical that referencing it by variable name
__name__ is derived name of current module, normally this will be dotted path like project.module.submodule — this can be any string that makes sense to you. __name__ makes a lot of sense usually, so it became sort of convention

One is just a reference - logger, what we call a variable and has nothing to do with the logging process. You could call it babapaloopa and that would not make any difference at all, except make your program cooler when referencing the logging object (though something could be said about good design here). I would stress this is not the name of the object, just the name of a variable that happens to reference the object - you could:
another_reference = logger
and now both another_reference and logger are both pointing to the same logging object, and clearly the variable name is unrelated to the object. Objects do have ids, but that is not something you should tamper with usually.
The name argument will become the actual name associated with the logger, a property of the logger object itself, not just a name of a reference to it - likely stored in the initialization of the object. The fact you use __name__ which is a special string, or some other string is just a detail, the point is this will be the name of the logger from a programming viewpoint.
logger.name
will hold this name after the logger is defined (assuming logger is reference to said logging object of course).

Logging based on caller module [duplicate]

I have the following application structure:
./utils.py
def do_something(logger=None):
if not logger:
logger = logging.getLogger(__name__)
print('hello')
logger.debug('test')
./one.py
from utils import do_something
logger = logging.getLogger(__name__)
do_something(logger=logger)
./two.py
from utils import do_something
logger = logging.getLogger(__name__)
do_something(logger=logger)
Now, then this runs, the logging output will show the names of the respective modules that are using the functions (one and two) rather than utils. I then use this information (the logger's name) to filter the messages.
Is there a way to do this without having to pass the logger though as an argument? Would I have to basically introspect the caller function and initialize a logger based on that? What's the common solution for this type of recipe, when one function is used all over the code base but the loggers should be of those calling the function?

Variable logger being a global, can be accessed from inside do_something() function like this:
logger = logging.getLogger(__name__)
def do_something():
x = logger
After reading this carefully:
What's the common solution for this type of recipe, when one function
is used all over the code base but the loggers should be of those
calling the function?
In simple english would be:
How to access global variable logger from imported function do_something()?
I conclude that there is no other way!
You have to pass logger as an argument for this particular case.
from utils import do_something
logger = logging.getLogger(__name__)
do_something() # logger global is not visible in do_something()
do_something(logger=logger) # the only way

Passing the logger as an argument looks fine to me. But if you have many calls to do_something then passing the same logger in each call is not only too verbose but prone to errors --you might pass the wrong logger by mistake. Also, while you do need to make do_something aware of the logger, doing it at the point where you are calling it probably does not make sense --you want to do something, not mess with loggers. OOP to the rescue: make do_something a method of a class and pass the logger when instantiating the class, then you do not need to pass it to the method.
# utils.py
class UsefulObject:
def __init__(self, logger):
self._logger = logger
def do_something(self):
print('hello')
self._logger.debug('test')
Now in the client module you create an instance of UsefulObject and you can call .do_something on it without passing the logger every time.
# one.py
from utils import UsefulObject
logger = logging.getLogger(__name__)
useful_object = UsefulObject(logger) # do this once
# ...
useful_object.do_something() # do this many times, do not pass the logger

If you don't like the existing solutions, here's a magic way to do it:
def do_something():
logger = inspect.currentframe().f_back.f_globals['log']
...
The calling function must then have a global variable called log.

Avoid check if logger exists

When using logging in python like this
import logging
logging.basicConfig(level=logging.INFO)
logger = None
if <some condition>:
logger = logging.getLogger(__name__)
... some code ...
logger.debug('message')
is it possible to avoid calling logger.debug if it does not exist without if statement?

That's not how you use logging: it's missing the point entirely. The point is to always just log, and then (at the app level) configure handlers appropriately so that they either record the debug level or not. Your app-level configuration can even set the handler for loggers in each module. See the Python logging docs for more info.

You could construct a wrapper object and use that everywhere:
class LoggerWrapper(object):
def __init__(self, logger):
self.logger = logger
def debug(self, *args, **kwargs):
if self.logger is None:
return
self.logger.debug(*args, **kwargs)
my_wrapper = LoggerWrapper(logger)
my_wrapper.debug('message')
That being said, this is not specific to Python's logging module. Any variable var that can also be None will need to be checked before you call a method on it (var.method()).
It may be easier to always define a logger and to simply disable the logging through the logging configuration / settings. That keeps the code clean, all logger calls will succeed but the output won't go anywhere if that particular logger is disabled.
Lastly, you could opt to indeed use an if statement as it's not inherently that much more code:
if logger:
logger.debug('message')

is it possible to avoid calling logger.debug if it does not exist without if statement?
Well, you could (ab)use an if expression, or an or expression, or various other things, but otherwise, not really.
You can, however, always write a wrapper function:
def logdebug(*args):
if logger:
logger.debug(*args)
logdebug('message')
logdebug('other message')
You can do other, similar tricks. For example, instead of having logger = None, you can create an object that has a debug method that does nothing:
class Dummy(object):
def debug(self, *args): pass
logger = Dummy()
Or, even better, always create a logger and configure it with a null handler, or—simplest of all—just set its logging level higher than debug.
But note that none of these will completely replace the if statement. Consider, for example:
if logger:
logger.debug(very_expensive_function())
logdebug(very_expensive_function())
The first one will only call very_expensive_function to create the log message if you actually need it; the second one will always call it and then just throw it away if you don't need it.

How can i create own logging instance for each thread?

For example, i have these files with classes that i want to use (only example, maybe not working)
# helper_one.py
import logging
logger = logging.getLogger('HelperOne')
class HelperOne:
def __init__(self, data):
self.data = data
def run(self):
logger.debug('HelperOne::Run function with data: ' + self.data)
return True
.
# controller_one.py
import logging
from helper_one import HelperOne
logger = logging.getLogger('ControllerOne')
class ControllerOne:
def __init__(self, data):
self.data = data
def run(self):
logger.debug('ControllerOne::Run function with data: ' + self.data)
ho = HelperOne(self.data + '_data')
return ho.run()
And i have a file which creates Threads
import logging
from controller_one import ControllerOne
# this function creates threads (its just an example)
def run_task(task_id, data):
logging.basicConfig(
filename=os.path.join('logs', str(task_id) + '.log'),
level=logging.DEBUG,
format='%(asctime)s\t%(name)s\t%(levelname)s\t%(message)s'
)
result = ControllerOne(data)
logging.debug('run_task: ' + result)
If i create logging instance like in my example, all log are written in a single file.
How can I, for every thread, create its own logging instance with logging to its own file ?
Regards, Alex.

The logger name (the string passed to the getLogger function) can be anything you want. As the docs explain:
The name is potentially a period-separated hierarchical value, like foo.bar.baz (though it could also be just plain foo, for example). Loggers that are further down in the hierarchical list are children of loggers higher up in the list. For example, given a logger with a name of foo, loggers with names of foo.bar, foo.bar.baz, and foo.bam are all descendants of foo. The logger name hierarchy is analogous to the Python package hierarchy, and identical to it if you organise your loggers on a per-module basis using the recommended construction logging.getLogger(__name__). That’s because in a module, __name__ is the module’s name in the Python package namespace.
So, using __name__ as the name is recommended, but not required. And in fact you're already explicitly violating it:
logger = logging.getLogger('ControllerOne')
So, you could just use a separate name per thread, by putting a thread ID or name into the logger name. For example:
logger = logging.getLogger('ControllerOne.{}'.format(threading.get_ident())
Or, in your case, since you seem to have a unique task_id for each thread, you can just use that instead of the thread ID.
Of course this means you have to actually use the logger; you can't just call logging.debug. And you can't rely on the basicConfig; you're going to need to explicitly configure the logger in each thread. Since each thread creates its own independent ControllerOne instance, the obvious place to do that is in ControllerOne.__init__; in that function, you can call getLogger with a logger name that incorporates the thread or task ID, create and set a FileHandler that also incorporates the thread or task ID in the name, and store it as self.logger. Then, whenever you need to log something, you just do self.logger.debug(…).
If that all sounds like gobbledegook to you because you have no idea what a FileHandler is, you probably need to read at least the basic tutorial in the HOWTOs, but the advanced tutorial and cookbook are also very useful.

Custom logging class python scope

The following dev enviroment is to be considered.
Small number of python modules, each contanining one or more classes. A main module, that calls those classes. A custom logging module called logger with a class called Logger. Assume that I call the main execution class with a logging level of debug. How may I make this sufficient to be that log level inherited to every other call including the rest of the classes, methods in those classes, functions in the main module and so forth...
The Logger objects are called like log=Logger(0) for example (logging level is an int, to use the same syntax that we use in other scripts (shell scripts, not python).
My final picture is to have the code filled with log.log_debug('debug message') and log.log_error('error message') but only actually printing a message when the right log_level is choosen. And if possible just one
from logger import Logger
call within the main module.
Thanks for your time.
==================================
Edit
In the main execution module there is a main() function in which a parser.parse_args() object is returned with an argument --log_level to globally define (at least is my intention) the log_level. There is a default log_level handling (i.e, it is always defined)
I will try to mock a minimum example
import argparse
from logfile from Logfile
from logger import Logger
def argument_parser():
parser=argparse.ArgumentParser()
stuff
return parser.parse_args()
def log_file_stuff():
log_file=Logfile()
log_file.methods() [*]
def main():
args=argument_parser()
# Here log_level is defined with args.log_level
global log
log=log(log_level)
log_file_stuff()
main()
[*] One of those methods may call a Logger object, which I want to be exactly the same as the one defined in the main() function the question is how may I achieve this? without a log return and argument waterfall

Well to answer my own question... I actually implemented a cross module variable dictionary to use it as settings. Those settings have the log_level variable son only with
from main import settings
from logger import Logger
log=Logger(settings['log_level'])
then I have my custom logging class with the user input log_level. Of course settings is constructed with argparse, mainly as vars(args) following the notation in the question.