We Keep Coding

pytest: selective log levels on a per-module basis

I'm using pytest-3.7.1 which has good support for logging, including live logging to stdout during tests. I'm using --log-cli-level=DEBUG to dump all debug-level logging to the console as it happens.
The problem I have is that --log-cli-level=DEBUG turns on debug logging for all modules in my test program, including third-party dependencies, and it floods the log with a lot of uninteresting output.
Python's logging module has the ability to set logging levels per module. This enables selective logging - for example, in a normal Python program I can turn on debugging for just one or two of my own modules, and restrict the log output to just those, or set different log levels for each module. This enables turning off debug-level logging for noisy libraries.
So what I'd like to do is apply the same concept to pytest's logging - i.e. specify a logging level, from the command line, for specific non-root loggers. For example, if I have a module called test_foo.py then I'm looking for a way to set the log level for this module from the command line.
I'm prepared to roll-my-own if necessary (I know how to add custom arguments to pytest), but before I do that I just want to be sure that there isn't already a solution. Is anyone aware of one?

I had the same problem, and found a solution in another answer:
Instead of --log-cli-level=DEBUG, use --log-level DEBUG. It disables all third-party module logs (in my case, I had plenty of matplotlib logs), but still outputs your app logs for each test that fails.

I got this working by writing a factory class and using it to set the level of the root logger to logger.INFO and use the logging level from the command line for all the loggers obtained from the factory. If the logging level from the command line is higher than the minimum global log level you specify in the class (using constant MINIMUM_GLOBAL_LOG_LEVEL), the global log level isn't changed.
import logging
MODULE_FIELD_WIDTH_IN_CHARS = '20'
LINE_NO_FIELD_WIDTH_IN_CHARS = '3'
LEVEL_NAME_FIELD_WIDTH_IN_CHARS = '8'
MINIMUM_GLOBAL_LOG_LEVEL = logging.INFO
class EasyLogger():
root_logger = logging.getLogger()
specified_log_level = root_logger.level
format_string = '{asctime} '
format_string += '{module:>' + MODULE_FIELD_WIDTH_IN_CHARS + 's}'
format_string += '[{lineno:' + LINE_NO_FIELD_WIDTH_IN_CHARS + 'd}]'
format_string += '[{levelname:^' + LEVEL_NAME_FIELD_WIDTH_IN_CHARS + 's}]: '
format_string += '{message}'
level_change_warning_sent = False
#classmethod
def get_logger(cls, logger_name):
if not EasyLogger._logger_has_format(cls.root_logger, cls.format_string):
EasyLogger._setup_root_logger()
logger = logging.getLogger(logger_name)
logger.setLevel(cls.specified_log_level)
return logger
#classmethod
def _setup_root_logger(cls):
formatter = logging.Formatter(fmt=cls.format_string, style='{')
if not cls.root_logger.hasHandlers():
handler = logging.StreamHandler()
cls.root_logger.addHandler(handler)
for handler in cls.root_logger.handlers:
handler.setFormatter(formatter)
cls.root_logger.setLevel(MINIMUM_GLOBAL_LOG_LEVEL)
if (cls.specified_log_level < MINIMUM_GLOBAL_LOG_LEVEL and
cls.level_change_warning_sent is False):
cls.root_logger.log(
max(cls.specified_log_level, logging.WARNING),
"Setting log level for %s class to %s, all others to %s" % (
__name__,
cls.specified_log_level,
MINIMUM_GLOBAL_LOG_LEVEL
)
)
cls.level_change_warning_sent = True
#staticmethod
def _logger_has_format(logger, format_string):
for handler in logger.handlers:
return handler.format == format_string
return False
The above class is then used to send logs normally as you would with a logging.logger object as follows:
from EasyLogger import EasyLogger
class MySuperAwesomeClass():
def __init__(self):
self.logger = EasyLogger.get_logger(__name__)
def foo(self):
self.logger.debug("debug message")
self.logger.info("info message")
self.logger.warning("warning message")
self.logger.critical("critical message")
self.logger.error("error message")

Enable/Disable/Modify the log level of any module in Python:
logging.getLogger("module_name").setLevel(logging.log_level)

How to configure/initialize logging using logger for multiple modules only once in Python for entire project?

I have python project with multiple modules with logging. I perform initialization (reading log configuration file and creating root logger and enable/disable logging) in every module before start of logging the messages. Is it possible to perform this initialization only once in one place (like in one class may be called as Log) such that the same settings are reused by logging all over the project?
I am looking for a proper solution to have only once to read the configuration file and to only once get and configure a logger, in a class constructor, or perhaps in the initializer (__init__.py). I don't want to do this at client side (in __main__ ). I want to do this configuration only once in separate class and call this class in other modules when logging is required.

setup using #singleton pattern
#log.py
import logging.config
import yaml
from singleton_decorator import singleton
#singleton
class Log:
def __init__(self):
configFile = 'path_to_my_lof_config_file'/logging.yaml
with open(configFile) as f:
config_dict = yaml.load(f)
logging.config.dictConfig(config_dict)
self.logger = logging.getLogger('root')
def info(self, message):
self.logger.info(message)
#module1.py
from Log import Log
myLog = Log()
myLog.info('Message logged successfully)
#module2.py
from Log import Log
myLog = Log() #config read only once and only one object is created
myLog.info('Message logged successfully)

From the documentation,
Note that Loggers should NEVER be instantiated directly, but always through the module-level function logging.getLogger(name). Multiple calls to getLogger() with the same name will always return a reference to the same Logger object.

You can initialize and configure logging in your main entry point. See Logging from multiple modules in this Howto (Python 2.7).

I had the same problem and I don't have any classes or anything, so I solved it with just using global variable
utils.py:
existing_loggers = {}
def get_logger(name='my_logger', level=logging.INFO):
if name in existing_loggers:
return existing_loggers[name]
# Do the rest of initialization, handlers, formatters etc...

How to extend the logging.Logger Class?

I would like to start with a basic logging class that inherits from Python's logging.Logger class. However, I am not sure about how I should be constructing my class so that I can establish the basics needed for customising the inherited logger.
This is what I have so far in my logger.py file:
import sys
import logging
from logging import DEBUG, INFO, ERROR
class MyLogger(object):
def __init__(self, name, format="%(asctime)s | %(levelname)s | %(message)s", level=INFO):
# Initial construct.
self.format = format
self.level = level
self.name = name
# Logger configuration.
self.console_formatter = logging.Formatter(self.format)
self.console_logger = logging.StreamHandler(sys.stdout)
self.console_logger.setFormatter(self.console_formatter)
# Complete logging config.
self.logger = logging.getLogger("myApp")
self.logger.setLevel(self.level)
self.logger.addHandler(self.console_logger)
def info(self, msg, extra=None):
self.logger.info(msg, extra=extra)
def error(self, msg, extra=None):
self.logger.error(msg, extra=extra)
def debug(self, msg, extra=None):
self.logger.debug(msg, extra=extra)
def warn(self, msg, extra=None):
self.logger.warn(msg, extra=extra)
This is the main myApp.py:
import entity
from core import MyLogger
my_logger = MyLogger("myApp")
def cmd():
my_logger.info("Hello from %s!" % ("__CMD"))
entity.third_party()
entity.another_function()
cmd()
And this is the entity.py module:
# Local modules
from core import MyLogger
# Global modules
import logging
from logging import DEBUG, INFO, ERROR, CRITICAL
my_logger = MyLogger("myApp.entity", level=DEBUG)
def third_party():
my_logger.info("Initial message from: %s!" % ("__THIRD_PARTY"))
def another_function():
my_logger.warn("Message from: %s" % ("__ANOTHER_FUNCTION"))
When I run the main app, I get this:
2016-09-14 12:40:50,445 | INFO | Initial message from: __THIRD_PARTY!
2016-09-14 12:40:50,445 | INFO | Initial message from: __THIRD_PARTY!
2016-09-14 12:40:50,445 | WARNING | Message from: __ANOTHER_FUNCTION
2016-09-14 12:40:50,445 | WARNING | Message from: __ANOTHER_FUNCTION
2016-09-14 12:40:50,445 | INFO | Hello from __CMD!
2016-09-14 12:40:50,445 | INFO | Hello from __CMD!
Everything is printed twice, as probably I have failed to set the logger class properly.
Update 1
Let me clarify my goals.
(1) I would like to encapsulate the main logging functionality in one single location so I can do this:
from mylogger import MyLogger
my_logger = MyLogger("myApp")
my_logger.info("Hello from %s!" % ("__CMD"))
(2) I am planning to use CustomFormatter and CustomAdapter classes. This bit does not require a custom logging class, those can be plugged in straight away.
(3) I probably do not need to go very deep in terms of customisation of the underlying logger class (records etc.), intercepting logger.info, loggin.debug etc. should be enough.
So referring back to this python receipt that has been circulated many times on these forums:
I am trying to the find the sweet point between having a Logger Class, yet still be able to use the built in functions like assigning Formatters and Adapters etc. So everything stays compatible with the logging module.
class OurLogger(logging.getLoggerClass()):
def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
# Don't pass all makeRecord args to OurLogRecord bc it doesn't expect "extra"
rv = OurLogRecord(name, level, fn, lno, msg, args, exc_info, func)
# Handle the new extra parameter.
# This if block was copied from Logger.makeRecord
if extra:
for key in extra:
if (key in ["message", "asctime"]) or (key in rv.__dict__):
raise KeyError("Attempt to overwrite %r in LogRecord" % key)
rv.__dict__[key] = extra[key]
return rv
Update 2
I have created a repo with a simple python app demonstrating a possible solution. However, I am keen to improve this.
xlog_example
This example effectively demonstrates the technique of overriding the logging.Logger class and the logging.LogRecord class through inheritance.
Two external items are mixed into the log stream: funcname and username without using any Formatters or Adapters.

At this stage, I believe that the research I have made so far and the example provided with the intention to wrap up the solution is sufficient to serve as an answer to my question. In general, there are many approaches that may be utilised to wrap a logging solution. This particular question aimed to focus on a solution that utilises logging.Logger class inheritance so that the internal mechanics can be altered, yet the rest of the functionality kept as it is since it is going to be provided by the original logging.Logger class.
Having said that, class inheritance techniques should be used with great care. Many of the facilities provided by the logging module are already sufficient to maintain and run a stable logging workflow. Inheriting from the logging.Logger class is probably good when the goal is some kind of a fundamental change to the way the log data is processed and exported.
To summarise this, I see that there are two approaches for wrapping logging functionality:
1) The Traditional Logging:
This is simply working with the provided logging methods and functions, but wrap them in a module so that some of the generic repetitive tasks are organised in one place. In this way, things like log files, log levels, managing custom Filters, Adapters etc. will be easy.
I am not sure if a class approach can be utilised in this scenario (and I am not talking about a super class approach which is the topic of the second item) as it seems that things are getting complicated when the logging calls are wrapped inside a class. I would like to hear about this issue and I will definitely prepare a question that explores this aspect.
2) The Logger Inheritance:
This approach is based on inheriting from the original logging.Logger class and adding to the existing methods or entirely hijacking them by modifying the internal behaviour. The mechanics are based on the following bit of code:
# Register our logger.
logging.setLoggerClass(OurLogger)
my_logger = logging.getLogger("main")
From here on, we are relying on our own Logger, yet we are still able to benefit from all of the other logging facilities:
# We still need a loggin handler.
ch = logging.StreamHandler()
my_logger.addHandler(ch)
# Confgure a formatter.
formatter = logging.Formatter('LOGGER:%(name)12s - %(levelname)7s - <%(filename)s:%(username)s:%(funcname)s> %(message)s')
ch.setFormatter(formatter)
# Example main message.
my_logger.setLevel(DEBUG)
my_logger.warn("Hi mom!")
This example is crucial as it demonstrates the injection of two data bits username and funcname without using custom Adapters or Formatters.
Please see the xlog.py repo for more information regarding this solution. This is an example that I have prepared based on other questions and bits of code from other sources.

This line
self.logger = logging.getLogger("myApp")
always retrieves a reference to the same logger, so you are adding an additional handler to it every time you instantiate MyLogger. The following would fix your current instance, since you call MyLogger with a different argument both times.
self.logger = logging.getLogger(name)
but note that you will still have the same problem if you pass the same name argument more than once.
What your class needs to do is keep track of which loggers it has already configured.
class MyLogger(object):
loggers = set()
def __init__(self, name, format="%(asctime)s | %(levelname)s | %(message)s", level=INFO):
# Initial construct.
self.format = format
self.level = level
self.name = name
# Logger configuration.
self.console_formatter = logging.Formatter(self.format)
self.console_logger = logging.StreamHandler(sys.stdout)
self.console_logger.setFormatter(self.console_formatter)
# Complete logging config.
self.logger = logging.getLogger(name)
if name not in self.loggers:
self.loggers.add(name)
self.logger.setLevel(self.level)
self.logger.addHandler(self.console_logger)
This doesn't allow you to re-configure a logger at all, but I leave it as an exercise to figure out how to do that properly.
The key thing to note, though, is that you can't have two separately configured loggers with the same name.
Of course, the fact that logging.getLogger always returns a reference to the same object for a given name means that your class is working at odds with the logging module. Just configure your loggers once at program start-up, then get references as necessary with getLogger.

Registering multiple observers for twisted logger

I have written a custom tcp observer that sends message over network to be parsed by logstash.
In my application I'd like to log stuffs like this:
"acme" namespace [info] goes to one logstash server
"acme" namespace [debug, critical, warning] goes to another
"" namespace [info, debug] (twisted logs) goes to info file
"" namespace [warning, critical] (twisted errors) goes to error file
Is it possible to achieve this with one single logger ? or should I register multiple loggers ?
The problem seems to be that the observers can be registered in two places:
twisted.Logger.globalLogBeginner but this registers obs for ALL loggers
logger.Logger(observer=) but this can register only a single observer
Can I achieve what I want to do with one single logger or should I register multiple loggers ? For points 1 and 2 I will have two loggers, a "customer" and a "debug", but if possible I'd rather avoid the 3 and 4 to be a different logger than the debug one.
Here what I have tried actually:
from twisted import logger
from twisted.internet import reactor
from txlab.logger import LogstashLogObserver
# Global object to control globally namespace logging
logLevelFilterPredicate = logger.LogLevelFilterPredicate(defaultLogLevel=logger.LogLevel.info)
def startLogging(console=True, filepath=None, errpath=None):
"""
Starts the global Twisted logger subsystem with maybe
stdout and/or a file specified in the config file
"""
global logLevelFilterPredicate
observers = []
if console:
observers.append(logger.FilteringLogObserver(observer=logger.textFileLogObserver(sys.stdout),
predicates=[logLevelFilterPredicate]))
if filepath is not None and filepath != "":
observers.append(logger.FilteringLogObserver(observer=logger.textFileLogObserver(open(filepath, 'a')),
predicates=[logLevelFilterPredicate]))
if errpath is not None and errpath != "":
_logLevelFilterPredicate = logger.LogLevelFilterPredicate(defaultLogLevel=logger.LogLevel.error)
observers.append(logger.FilteringLogObserver(observer=logger.textFileLogObserver(open(errpath, 'a')),
predicates=[_logLevelFilterPredicate]))
logger.globalLogBeginner.beginLoggingTo(observers)
def setLogLevel(namespace=None, levelStr='info'):
"""
Set a new log level for a given namespace
LevelStr is: 'critical', 'error', 'warn', 'info', 'debug'
"""
level = logger.LogLevel.levelWithName(levelStr)
logLevelFilterPredicate.setLogLevelForNamespace(namespace=namespace, level=level)
if __name__ == '__main__':
import sys
logInfo = logger.Logger(observer=LogstashLogObserver(b'127.0.0.1', 5001, version=1))
logDebug = logger.Logger(observer=LogstashLogObserver(b'127.0.0.1', 5002, version=1))
startLogging(filepath='/code/txacme/_logs/info.log',
errpath='/code/txacme/_logs/errors.log')
setLogLevel(namespace='', levelStr='debug')
"""
The problem here is that when logging, I cannot use the logDebug to log twisted events,
I need to create another logger without an observer to be able to use the globally registered ones
"""
logTwisted = logger.Logger() # I want to avoid this, using logDebug instead
try:
raise IOError()
except:
log.failure('errrrrr')
log.info("Some values: {values!r}", values=[1234, 5678])
log.info("Some values: {values!r}", values=[9876, 5432])
log.info('{a}, {b}, {c}, that\'s all :)', a=1, b=2, c=3)
reactor.run()

You can register multiple Log Observers to a single logger through twisted.logger.globalLogPublisher. globalLogPublisher is designed by Composite pattern. That is, it itself is an observer, and can have another observers. Logger will register globalLogPublisher unless an observer is explicitly specified in the constructor. You can do something like
import sys
from twisted.logger import globalLogPublisher, textFileLogObserver, Logger
from txlab.logger import LogstashLogObserver
log = Logger()
globalLogPublisher.addObserver(LogstashLogObserver(b'127.0.0.1', 5001, version=1))
globalLogPublisher.addObserver(LogstashLogObserver(b'127.0.0.1', 5002, version=1))
globalLogPublisher.addObserver(textFileLogObserver(sys.stdout))
# this will publish logging event to all observers registered in globalLobPublisher.
log.info('some message')

logging in continouous loop

What would be a good way to create logs (with python logging module) inside a constant running loop, without producing a large amount of useless log-files?
An example would be a loop that constant list a folder, and does some action when it sees a file of a specific type.
I want to log that no files were found, or files were found but of a wrong type, without logging that same line constantly for each folder check, as it might run many times a second.

Create a Handler that subclasses whatever other functionality you need. Store either the last, or all the previously logged messages that you don't want to emit again:
def make_filetype_aware_handler(handler_class):
class DontRepeatFiletypeHandler(handler_class):
def __init__(self, *args, **kwds):
super().__init__(*args, **kwds)
self.previous_types = set()
def emit(self, record):
if not record.file_type in self.previous_types:
self.previous_types.add(record.file_type)
super().emit(record)
return DontRepeatFiletypeHandler
FiletypeStreamHandler = make_filetype_aware_handler(logging.StreamHandler)
logger = logging.getLogger()
logger.addHandler(FiletypeStreamHandler(sys.stderr))
logger.debug('Found file of type %(file_type)', file_type='x-type/zomg')

import logging
logger = logging.getLogger(test)
# logging to a file
hdlr = logging.FileHandler(test.log)
formatter = logging.Formatter('%(asctime)s %(filename)s %(lineno)s %(levelname)s % (message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.DEBUG)
Then in the loop, you have to check for file type and if file is present or not.
Then add :
logger.debug('File Type : %s ' % file_type)
also
if file_present:
logger.debug('File Present : %s ' % present)

Log the less important events with a lower precedence, like DEBUG. See setLevel and SysLogHandler.
At development time set the level to DEBUG, and as your application matures, set it to more reasonable values like INFO or ERROR.
Your app should do something about the errors, like remove files of the wrong type and/or create lacking files; or move the wrongfully configured directories from the job polling to a quarantine location, so your log will not be flooded.

My understanding is that you are trying to limit logging the same message over and over again.
If this is your issue, I would create a a set of file_types you have already logged. However you need to be careful, if this is going to run forever, you will eventually crash..
from sets import Set
logged = Set()
while yourCondition:
file_type = get_next_file_type()
needToLog = #determine if you need to log this thing
if needToLog and (not file_type in logged):
logger.info("BAH! " + file_type)
logged.add(file_type)

A bit of a hack but much easier is "misusing" functools lru_cache.
from functools import lru_cache
from logging import getLogger
# Keep track of 10 different messages and then warn again
#lru_cache(10)
def warn_once(logger: Logger, msg: str):
logger.warning(msg)
You can increase the 10 to suppress more if required or set it to None to store suppress everything duplicated.