My project is composed by a main.py main script and a module aaa.py . I've succesfully setup a log procedure by using the logging package in main.py, also specifying a log filename.
Which is the cleanest/"correct" way to let the functions contained in aaa.py to write in the same log file?
Use the root logger in main.py by defining
logger = logging.getLogger()
fh = logging.FileHandler("path/to/file")
logger.addHandler(fh)
Then use a module logger in aaa.py by defining
logger = logging.getLogger(__name__)
The logger does not have to have the same name as the module, but it is common practice. The submodule logger will automatically bubble up to the root logger and be sent to any handlers.
To quote from the docs
Child loggers propagate messages up to the handlers associated with their ancestor loggers. Because of this, it is unnecessary to define and configure handlers for all the loggers an application uses. It is sufficient to configure handlers for a top-level logger and create child loggers as needed.
If you call only a single python script at once (main.py)
then you simply can define your logging config once; for exemple:
logging.basicConfig(filename=logfilepath, format='%(levelname)s:%(message)s')
and call it wherever you want in the modules, for exemple
logging.<level>("log_string")
Related
I want to print all logging messages from all imported modules. Certain imported modules are not logging.
Note that all the files in the libraries I care about have calls
logger = logging.getLogger(__name__) at the top.
Sorry if this is an easy problem. I've looked through a lot of posts without success.
I observe that the loggers for some modules are not being updated by the call to basicConfig
import logging
import local_util # a local util file
from transformers import T5ForConditionalGeneration
for n in logging.root.manager.loggerDict:
print(logging.getLogger(n))
logging.basicConfig(level=logging.DEBUG)
console = logging.StreamHandler()
console.setLevel(logging.DEBUG)
formatter = logging.Formatter("[%(asctime)s] %(levelname)s::%(module)s::%(funcName)s() %(message)s")
console.setFormatter(formatter)
logging.getLogger('').addHandler(console)
for n in logging.root.manager.loggerDict:
print(logging.getLogger(n))
In the first call to print(logging.getLogger(n)) almost all loggers are set with level WARNING.
In the second call, most loggers are set to DEBUG (including the one for local_util), except for the transformers library which all remain at level WARNING.
I can get transformers messages to print if I manually cycle through all loggers and reset their levels. Even if I use force=True in the call to basicConfig, the loggers for the transformers library do not get updated.
basicConfig means to affect the root logger. If other child loggers don't set their level (which is default to NOTSET), they use the level of the root logger.
So if transformers has set a WARNING level on its own, it won't use DEBUG level of root.
You can set its level directly as the following:
transformers_logger = logging.getLogger('transformers')
transformers_logger.setLevel(logging.DEBUG)
I've imported a python module, apex (from NVIDIA), which uses its own logger. Unfortunately those logged messages are not getting caught by my main logger, which also writes to a file.
import logging
logger = logging.getLogger(__name__) # my main logger
what I need to do is (somehow) run logging.getLogger("apex.amp") so that I can attach the apex logger to my main logger, and catch the corresponding warnings, etc.
How do I combine the loggers?
You can assign your logger to the 'apex.amp' key of the loggerDict dict of the manager object:
logging.manager.loggerDict['apex.amp'] = logger
I write big program with many modules. In same module I wish use logging. What best practice for logging in Python?
Should I use import standart logging module and use it in every my file:
#proxy_control.py
#!/usr/bin/env python3
import logging
class ProxyClass():
def check_proxy():
pass
logging.basicConfig(filename='proxy.log', level=logging.INFO)
logging.info('Proxy work fine')
Or maybe i should write one MyLog() class inherit from default logging and use it from all my other modules?
#proxy_control.py
#!/usr/bin/env python3
import MyLog
class ProxyClass():
def check_proxy():
pass
Mylog.send_log('Proxy work fine')
#my_log.py
#!/usr/env/bin python3
import logging
class MyLog():
def send_log(value):
pass
A typical convention is to define a logger at the top of every module that requires logging and then use that logger object throughout the module.
logger = logging.getLogger(__name__)
This way, loggers will follow your package names (ie. package.subpackage.module). This is useful in the logging module because loggers propagate messages upwards based on the logger name (ie. parent.child will pass messages up to parent). This means that you can do all your configuration at the top level logger and it will get messages from all the sub-loggers in your package. Or, if someone else is using your library, it will be very easy for them to configure which logging messages they get from your library because it will have a consistent namespace.
For a library, you typically don't want to show logging messages unless the user explicitly enables them. Python logging will automatically log to stderr unless you set up a handler, so you should add a NullHandler to your top-level logger. This will prevent the messages from automatically being printed to stderr.
logger = logging.getLogger(__name__)
logger.addHandler(logging.NullHandler())
NOTE - The NullHandler was added in Python2.7, for earlier versions, you'll have to implement it yourself.
Use the logging module, and leave logging configuration to your application's entry point (modules should not configure logging by themselves).
Does anyone know of any good examples of modules with nice logging implementations?
I have been doing logging a couple of different ways, but I'm not sure which is most Pythonic.
For scripts, this is what I've been doing:
import logging
LOGGER = logging.getLogger(__program__)
STREAM = logging.StreamHandler()
FORMATTER = logging.Formatter('(%(name)s) %(levelname)s: %(message)s')
STREAM.setFormatter(FORMATTER)
LOGGER.addHandler(STREAM)
def main():
LOGGER.warning('This is a warning message.')
That is executed in the global namespace and I can call LOGGER from anywhere.
The aforementioned solution is not such a good idea for modules because the code is executed upon import. So for modules, I have been calling this _logging() function early on to set up the logging.
def _logging():
import logging
global logger
logger = logging.getLogger(__program__)
stream = logging.StreamHandler()
formatter = logging.Formatter('(%(name)s) %(levelname)s: %(message)s')
stream.setFormatter(formatter)
logger.addHandler(stream)
def main():
_logging()
logger.warning('This is a warning message.')
Since logger is global, I can just call it anywhere it is needed. However pylint barks out a global-variable-undefined warning. It is defined as Global variable logger undefined at the module level Used when a variable is defined through the “global” statement but the variable is not defined in the module scope but I'm not really sure why that is an issue here.
Or should I call the _logger() function early on (minus the global) and then create the logger everywhere that it is needed?
def _logging():
import logging
logger = logging.getLogger(__program__)
stream = logging.StreamHandler()
formatter = logging.Formatter('(%(name)s) %(levelname)s: %(message)s')
stream.setFormatter(formatter)
logger.addHandler(stream)
def main():
_logging()
logger = logging.getLogger(__program__)
logger.warning('This is a warning message.')
The last technique seems to be the cleanest, albeit the most tedious, especially since I am often logging from within dozens of small classes, functions, methods, et cetera. Are there any examples from people/modules who have already blazed a path through this territory?
If I understood correctly, you are configuring logging in each module separately. That would be unnecessary and against the design of logging module.
I think that the key to logging is understanding that logging module is a stateful object in you Python process. At least for me after that insight there was only one obvious way to do logging in most situations.
You should configure logging at the beginning of your program. Define handlers, formatters, etc., and the configuration will remain throughout the program, as long as it isn't explicitly overridden.
All modules that do logging can define a global logger right after logging is imported. There is no need to put this into a function. As is also recommended by the documentation, it is good practice to name each logger according to module name (including package path):
import logging
logger = logging.getLogger(__name__)
It is also important to understand that loggers in your program form a hierarchy. By default, loggers propagate records to their parents. This means that at the bottom there is one logger (root) which gets all the records unless you configure some loggers to prevent this. Often it may be enough to configure only the root logger.
To be a little more concrete, let's make a program with two modules, one.py and two.py. one.py contains a function main which will be the entry point to the program. We'll configure logging using dictConfig, which lets us separate the logging configuration nicely from the rest of the code. We'll put the configuration dict in a separate YAML file, like this:
# logging_config.yaml
version: 1
formatters:
brief:
format: '%(message)s'
default:
format: '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'
datefmt: '%Y-%m-%d %H:%M:%S'
handlers:
console:
class: logging.StreamHandler
formatter: brief
stream: ext://sys.stdout
file:
class: logging.handlers.RotatingFileHandler
formatter: default
filename: example.log
maxBytes: 1024
backupCount: 3
loggers:
two:
level: INFO
handlers: [file]
propagate: False
root:
level: INFO
handlers: [console]
These snippets are mostly adapted from the documentation. In this configuration we define that everything above level INFO from logger two is logged into a file. Records from two are also not propagated further. Everything that comes to the root logger is fed to console if it is above level INFO.
The definition of module one could then be like this:
# one.py
import logging
import logging.config
import yaml
def configure_logging(filename):
with open(filename) as f:
config = yaml.load(f)
logging.config.dictConfig(config)
def main():
configure_logging('logging_config.yaml')
from two import func
logger = logging.getLogger(__name__)
logger.info('Starting the program')
func()
logger.info('Finished')
One tricky detail here is that we import module two and define the logger only after the configuration is set. This is done because by default dictConfig disables all existing loggers.
And finally, here is the definition for module two:
# two.py
import logging
logger = logging.getLogger(__name__)
def func():
logger.info('Doing stuff')
Now if we run the program, we get the following output:
>>> import one
>>> one.main()
Starting the program
Finished
And the log file example.log contains the following line:
2015-06-07 15:04:15 INFO two Doing stuff
Excellent examples of logging can be found in Python documentation:
Logging HOWTO
Logging Cookbook
Logging API reference
I have my main.py as follows:
import logging
import os
import web
def is_test():
if 'WEBPY_ENV' in os.environ:
return os.environ['WEBPY_ENV'] == 'test'
app = web.application(router.urls, globals())
logging.basicConfig(filename="log/debug.log", level=logging.INFO)
global logger
logger = logging.getLogger("debug")
if (not is_test()) and __name__ == "__main__":
app.run()
Now, here I have defined a variable named logger as global. So, can I access this variable anywhere in my application w/o redefining it? I am using web.py.
Basically what i need is something like this. I want to initialize the logger once and I should be able to use it anywhere in my whole application. How can i do that?
You don't need to access any global variable since the logging module already provides access to logger objects anywhere in your code, that is, if you use:
logger = logging.getLogger("debug")
in other modules, you'll get the same logger object as in the main module.
I don't fully understand why you're using a "debug" logger when you can adjust the level of your logger to get the debug messages, but maybe yours was just an example or you really need that, so let's get to the point:
From the official documentation: The key benefit of having the logging API provided by a standard library module is that all Python modules can participate in logging, so your application log can include your own messages integrated with messages from third-party modules.
This means that you just need to configure your logging at the beginning of your application and after that when you need a logger you call logging.getLogger(name) and you get that logger.
So there's no need of "tracking" your logger variable with:
global logger
logger = logging.getLogger("debug")
because whenever you need to log from your "debug" logger (in the middle of whatever you want) you just do something like:
my_debug_logger = logging.getLogger("debug")
my_debug_logger.info('some message')
At the end the point is that when you import the logging module import logging you have access to each and every logger previously defined (and of course you can define new ones).