I have this piece of code to set my logger in Python:
#Configure logging
logging.basicConfig(format = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M',
filename= "log.txt",
level = logging.getLevelName('DEBUG'))
print(logging.getLogger().getEffectiveLevel())
But the output from the print statement sometimes is this:
30
And other times is this (which is correct):
10
But often even when the the logging level is set to the correct number, it is not logging anything to the file, but other times it works. What do I need to do to make sure my logging level is set correctly?
*Edit: Below is my solution based off the recommendation of #randomir.
**Edit: I had to make a second change where I set the level after I call logging.basicConfig() or else the logging level still was not getting called consistently. The line `logging.getLogger().setLevel(...) now seems to work.
I created a new class: Logger.py.
import logging
class Logger(object):
def __init__(self):
logging.basicConfig(format = '%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M',
filename= "log.txt")
logging.getLogger().setLevel(logging.getLevelName("DEBUG"))
print(logging.getLogger().getEffectiveLevel())
And now instead of configuring the basic config directly in my startup class I just instantiate the logger class once:
from Logger import Logger
import logging
class LaunchPython(object):
#Configure Logging
Logger()
logging.info("Application has started")
On all subsequent classes that call the logger I just put import logging on the top and then do logging.info(...) or logging.debug(....). There is no need to import the Logger.py class and reinstantiate it.
The logging.basicConfig() creates a root logger for you application (that's a logger with the name root, you can get it with logging.getLogger('root'), or logging.getLogger()).
The trick is that the root logger gets created with defaults (like level=30) on the first call to any logging function (like logging.info()) if it doesn't exist already. So, make sure you call your basicConfig() before any logging in any other part of the application.
You can do that by extracting your logger config to a separate module, like logger.py, and then import that module in each of your modules. Or, if your application has a central entry point, just do the configuration there. Note that the 3rd party functions you call will also create the root logger if it doesn't exist.
Also note, if you application is multi-threaded:
Note
This function should be called from the main thread before other threads are started. In versions of Python prior to 2.7.1 and 3.2, if this function is called from multiple threads, it is possible (in rare circumstances) that a handler will be added to the root logger more than once, leading to unexpected results such as messages being duplicated in the log.
Related
Context: I would like to create a single stream to stdout of log events formatted in JSON without any duplicate events. Within my own applications, this is relatively simple. I simply configure the root logger to have a single StreamHandler and use python-json-logger as the Formatter.
included_attributes = '%(asctime)s %(filename)s %(funcName)s %(lineno)d %(message)s %(levelname)s %(module)s %(name)s %(pathname)s %(process)d %(thread)s %(threadName)s'
json_formatter = jsonlogger.JsonFormatter(included_attributes)
def change_root_logger_format():
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_handler = logging.StreamHandler()
root_handler.setFormatter(json_formatter)
root_logger.handlers = [root_handler]
For various modules I add the standard logger = logging.getLogger(__name__).
All logs roll up the hierarchy to root with my single StreamHandler() and output to stdout as JSON.
As I understand it, best practice for distributed packages is that they don't declare their own handlers (or at least check if they are __main__ before doing so). I have found that this is not universally applied, however.
I am using at least one package that adds its own handlers. That is, the package __init__.py contains something like
logger = logging.getLogger(__name__)
logger.addHandler(StreamHandler())
As a result, I end up with duplicate logs for everything logged by this package. One line from its own handler, and one from the root logger.
The only solution I've found to this is to remove the handlers after the fact. That is, loop through logging.root.manager.loggerDict and removing the handlers of any Logger that isn't root. The problem with this approach is that any new import could cause new Handlers to appear. Thus, all of my modules would need the same boilerplate handler removal after their imports.
Does anyone know of a way to prevent this problem pre-emptively and exhuastively? That is, preventing all loggers but root from having Handlers?
I don't believe this question is a duplicate. There are many questions on disabling package/module loggers (for example, this one). These questions and answers don't apply to removing handlers while keeping the logger active.
You can disable logging from imported modules using the logging module. You can configure it to not log messages unless they are at least warnings using the following code: import logging logging.getLogger("imported_module").setLevel(logging.WARNING)
When debugging a python function, I set up my logging like this:
import logging
logger_name = 'test.log'
log_format = "%(funcName)s:%(lineno)d -%(message)s"
logging.basicConfig(filename=logger_name,
level=logging.DEBUG,
format = log_format,
filemode='w')
My program consists of multiple functions that are located in separate files, so I get the logger object at each file where I take functions from using the following:
import logging
logger = logging.getLogger(__name__)
However, when executing the function for finding out its execution time, I do not want to include my logs. Is it enough to just remove the set up of logging in the file where the main function is located? Or should I also remove logger = logging.getLogger(__name__) from each file, or should I do something else?
The logger main idea beside logging useful information to your console or file is to disable certains level of log easily.
To disable only the debug logging just set the level=logging.DEBUGto level=logging.INFO or level=logging.WARNING.
In that way your are keeping your log to warning when in production, and if you want to use debug, reverse it back
I want to print all logging messages from all imported modules. Certain imported modules are not logging.
Note that all the files in the libraries I care about have calls
logger = logging.getLogger(__name__) at the top.
Sorry if this is an easy problem. I've looked through a lot of posts without success.
I observe that the loggers for some modules are not being updated by the call to basicConfig
import logging
import local_util # a local util file
from transformers import T5ForConditionalGeneration
for n in logging.root.manager.loggerDict:
print(logging.getLogger(n))
logging.basicConfig(level=logging.DEBUG)
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
formatter = logging.Formatter("[%(asctime)s] %(levelname)s::%(module)s::%(funcName)s() %(message)s")
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)
for n in logging.root.manager.loggerDict:
print(logging.getLogger(n))
In the first call to print(logging.getLogger(n)) almost all loggers are set with level WARNING.
In the second call, most loggers are set to DEBUG (including the one for local_util), except for the transformers library which all remain at level WARNING.
I can get transformers messages to print if I manually cycle through all loggers and reset their levels. Even if I use force=True in the call to basicConfig, the loggers for the transformers library do not get updated.
basicConfig means to affect the root logger. If other child loggers don't set their level (which is default to NOTSET), they use the level of the root logger.
So if transformers has set a WARNING level on its own, it won't use DEBUG level of root.
You can set its level directly as the following:
transformers_logger = logging.getLogger('transformers')
transformers_logger.setLevel(logging.DEBUG)
Does anyone know of any good examples of modules with nice logging implementations?
I have been doing logging a couple of different ways, but I'm not sure which is most Pythonic.
For scripts, this is what I've been doing:
import logging
LOGGER = logging.getLogger(__program__)
STREAM = logging.StreamHandler()
FORMATTER = logging.Formatter('(%(name)s) %(levelname)s: %(message)s')
STREAM.setFormatter(FORMATTER)
LOGGER.addHandler(STREAM)
def main():
LOGGER.warning('This is a warning message.')
That is executed in the global namespace and I can call LOGGER from anywhere.
The aforementioned solution is not such a good idea for modules because the code is executed upon import. So for modules, I have been calling this _logging() function early on to set up the logging.
def _logging():
import logging
global logger
logger = logging.getLogger(__program__)
stream = logging.StreamHandler()
formatter = logging.Formatter('(%(name)s) %(levelname)s: %(message)s')
stream.setFormatter(formatter)
logger.addHandler(stream)
def main():
_logging()
logger.warning('This is a warning message.')
Since logger is global, I can just call it anywhere it is needed. However pylint barks out a global-variable-undefined warning. It is defined as Global variable logger undefined at the module level Used when a variable is defined through the “global” statement but the variable is not defined in the module scope but I'm not really sure why that is an issue here.
Or should I call the _logger() function early on (minus the global) and then create the logger everywhere that it is needed?
def _logging():
import logging
logger = logging.getLogger(__program__)
stream = logging.StreamHandler()
formatter = logging.Formatter('(%(name)s) %(levelname)s: %(message)s')
stream.setFormatter(formatter)
logger.addHandler(stream)
def main():
_logging()
logger = logging.getLogger(__program__)
logger.warning('This is a warning message.')
The last technique seems to be the cleanest, albeit the most tedious, especially since I am often logging from within dozens of small classes, functions, methods, et cetera. Are there any examples from people/modules who have already blazed a path through this territory?
If I understood correctly, you are configuring logging in each module separately. That would be unnecessary and against the design of logging module.
I think that the key to logging is understanding that logging module is a stateful object in you Python process. At least for me after that insight there was only one obvious way to do logging in most situations.
You should configure logging at the beginning of your program. Define handlers, formatters, etc., and the configuration will remain throughout the program, as long as it isn't explicitly overridden.
All modules that do logging can define a global logger right after logging is imported. There is no need to put this into a function. As is also recommended by the documentation, it is good practice to name each logger according to module name (including package path):
import logging
logger = logging.getLogger(__name__)
It is also important to understand that loggers in your program form a hierarchy. By default, loggers propagate records to their parents. This means that at the bottom there is one logger (root) which gets all the records unless you configure some loggers to prevent this. Often it may be enough to configure only the root logger.
To be a little more concrete, let's make a program with two modules, one.py and two.py. one.py contains a function main which will be the entry point to the program. We'll configure logging using dictConfig, which lets us separate the logging configuration nicely from the rest of the code. We'll put the configuration dict in a separate YAML file, like this:
# logging_config.yaml
version: 1
formatters:
brief:
format: '%(message)s'
default:
format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
datefmt: '%Y-%m-%d %H:%M:%S'
handlers:
console:
class: logging.StreamHandler
formatter: brief
stream: ext://sys.stdout
file:
class: logging.handlers.RotatingFileHandler
formatter: default
filename: example.log
maxBytes: 1024
backupCount: 3
loggers:
two:
level: INFO
handlers: [file]
propagate: False
root:
level: INFO
handlers: [console]
These snippets are mostly adapted from the documentation. In this configuration we define that everything above level INFO from logger two is logged into a file. Records from two are also not propagated further. Everything that comes to the root logger is fed to console if it is above level INFO.
The definition of module one could then be like this:
# one.py
import logging
import logging.config
import yaml
def configure_logging(filename):
with open(filename) as f:
config = yaml.load(f)
logging.config.dictConfig(config)
def main():
configure_logging('logging_config.yaml')
from two import func
logger = logging.getLogger(__name__)
logger.info('Starting the program')
func()
logger.info('Finished')
One tricky detail here is that we import module two and define the logger only after the configuration is set. This is done because by default dictConfig disables all existing loggers.
And finally, here is the definition for module two:
# two.py
import logging
logger = logging.getLogger(__name__)
def func():
logger.info('Doing stuff')
Now if we run the program, we get the following output:
>>> import one
>>> one.main()
Starting the program
Finished
And the log file example.log contains the following line:
2015-06-07 15:04:15 INFO two Doing stuff
Excellent examples of logging can be found in Python documentation:
Logging HOWTO
Logging Cookbook
Logging API reference
I have my main.py as follows:
import logging
import os
import web
def is_test():
if 'WEBPY_ENV' in os.environ:
return os.environ['WEBPY_ENV'] == 'test'
app = web.application(router.urls, globals())
logging.basicConfig(filename="log/debug.log", level=logging.INFO)
global logger
logger = logging.getLogger("debug")
if (not is_test()) and __name__ == "__main__":
app.run()
Now, here I have defined a variable named logger as global. So, can I access this variable anywhere in my application w/o redefining it? I am using web.py.
Basically what i need is something like this. I want to initialize the logger once and I should be able to use it anywhere in my whole application. How can i do that?
You don't need to access any global variable since the logging module already provides access to logger objects anywhere in your code, that is, if you use:
logger = logging.getLogger("debug")
in other modules, you'll get the same logger object as in the main module.
I don't fully understand why you're using a "debug" logger when you can adjust the level of your logger to get the debug messages, but maybe yours was just an example or you really need that, so let's get to the point:
From the official documentation: The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log can include your own messages integrated with messages from third-party modules.
This means that you just need to configure your logging at the beginning of your application and after that when you need a logger you call logging.getLogger(name) and you get that logger.
So there's no need of "tracking" your logger variable with:
global logger
logger = logging.getLogger("debug")
because whenever you need to log from your "debug" logger (in the middle of whatever you want) you just do something like:
my_debug_logger = logging.getLogger("debug")
my_debug_logger.info('some message')
At the end the point is that when you import the logging module import logging you have access to each and every logger previously defined (and of course you can define new ones).