i try to use python logging module to create a timed rotating handler to some sort cache to a file.
eventually i need to know what is the cache file name.
i cannot find any property in logger to retrieve it.
so i create kind of monkey job to overload the logger for later retrieve.
anybody know a better solution?
def srvcacheconfig(**kwargs):
srv_name = kwargs['server_name']
# cache_path = CACHEPATH
# ans=dict()
cache_filename = srv_name+CACHE_EXT
cache_logger = logging.getLogger(srv_name)
cache_logger.propagate = False
#monkey job
cache_logger.cache_name = cache_filename
cache_format = logging.Formatter("%(message)s")
# cache_handlers = logging.handlers.TimedRotatingFileHandler(cache_filename, when = 'midnight', backupCount = 366)
cache_handlers = logging.handlers.TimedRotatingFileHandler(cache_filename, when = 'M', interval = 1, backupCount = 200)
cache_handlers.setFormatter(cache_format)
cache_logger.addHandler(cache_handlers)
# consoleHandler = logging.StreamHandler()
# cache_logger.addHandler(consoleHandler)
cache_logger.setLevel(logging.DEBUG)
return cache_logger
thank you very much
The TimedRotatingFileHandler has a property named baseFilename, which holds the name of the logfile. However, this is not documented and may therefore change without further notice.
From this property you can get the logfile's name:
import logging
from logging.handlers import TimedRotatingFileHandler
my_logger = logging.getLogger('my_logger')
my_handler = TimedRotatingFileHandler('logfile.log')
my_logger.addHandler(my_handler)
def get_filename(some_logger):
return [some_handler.baseFilename
for some_handler in some_logger.handlers
if isinstance(some_handler, TimedRotatingFileHandler)]
print(get_filename(my_logger)) # ['/home/finwood/temp/so/logfile.log']
Related
I'm trying to achieve a behavior available in log4j, which is described in this SO Q&A
Instead of the logger producing the following output (notice the full module name)
DEBUG aproject.bpackage.cclass:filename.py:79 log message 1
I'd like to only have the initials of the logger name:
DEBUG a.b.c:filename.py:79 log message 1
Trying to figure out the correct format string with the manual, but no mention of initials for the logger name
Note that loggers are initialized with the module name as convention:
logger = logging.getLogger(__name__)
There's no out-of-the-box functionality for this, but you can achieve the desired result using something like this:
import logging
class CustomFormatter(logging.Formatter):
def format(self, record):
saved_name = record.name # save and restore for other formatters if desired
parts = saved_name.split('.')
# import pdb; pdb.set_trace()
record.name = '.'.join(p[0] for p in parts)
result = super().format(record)
record.name = saved_name
return result
h = logging.StreamHandler()
f = CustomFormatter('%(name)-6s %(message)s')
h.setFormatter(f)
root = logging.getLogger()
root.addHandler(h)
root.setLevel(logging.DEBUG)
logging.getLogger('alpha.beta.gamma').debug('foo')
logging.getLogger('delta.epsilon.zeta').debug('bar')
logging.getLogger('eta.theta').debug('baz')
When run, the above script outputs
a.b.g foo
d.e.z bar
e.t baz
I have been using a custom formatter for logging to the terminal in my code. Lately I have been changing stuff in the code an I can't find why now in some parts of the code the log is printed twice.
This is the code for the custom formatter:
import logging
class MyFormatter(logging.Formatter):
debug_format = "[%(levelname)s] (%(module)s::%(funcName)s::%(lineno)d) %(message)s"
normal_format = "[%(levelname)s] %(message)s"
blue = "\x1b[36;21m"
grey = "\x1b[38;21m"
yellow = "\x1b[33;21m"
red = "\x1b[31;21m"
bold_red = "\x1b[31;1m"
reset = "\x1b[0m"
def __init__(self):
super().__init__(fmt="%(levelno)d: %(msg)s", datefmt=None, style="%")
def format(self, record):
# Save the original format configured by the user
# when the logger formatter was instantiated
format_orig = self._style._fmt
# Replace the original format with one customized by logging level
if record.levelno == logging.DEBUG:
self._style._fmt = MyFormatter.debug_format
format = MyFormatter.debug_format
else:
self._style._fmt = MyFormatter.normal_format
format = MyFormatter.normal_format
self.FORMATS = {
logging.DEBUG: MyFormatter.grey + format + MyFormatter.reset,
logging.INFO: MyFormatter.blue + format + MyFormatter.reset,
logging.WARNING: MyFormatter.yellow + format + MyFormatter.reset,
logging.ERROR: MyFormatter.red + format + MyFormatter.reset,
logging.CRITICAL: MyFormatter.bold_red + format + MyFormatter.reset,
}
log_fmt = self.FORMATS.get(record.levelno)
# Restore the original format configured by the user
self._style._fmt = format_orig
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
This is how I create my logger:
from src.logs import set_logger, logging
logger = set_logger(__name__, logging.DEBUG)
This is set_logger function code:
import logging
from .custom_formatter import MyFormatter
def set_logger(module_name: str, level=logging.DEBUG) -> logging.Logger:
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler()
formatter = MyFormatter()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
return logger
Now when I call this logger from main for example or at the top of a module which is imported, there is no problem, and it logs perfectly only once. However when calling the logger from inside a function in the same module it is printed twice.
I have notice by debugging that what is doing is going to the end of the format method in MyFormatter class and then it returns again to this format method, I have no clue what is going on here. Do you have any ideas on what could be happening?
PD: Also if I also call a print when the logger prints twice I only get one print, so that code runs only once for sure.
Thanks for your time!
Andrés
In set_logger(), it calls addHandler() but the logger (or an ancestor logger) will already have a handler, which you're not removing, so you'll have multiple handlers.
Have a look at the docs for Logger.propagate: https://docs.python.org/3/library/logging.html#logging.Logger.propagate
I am getting duplicate log output from a logger that only has 1 handler attached.
How can I debug this?
Code (should be copy-and-paste-able):
import io
import logging
import os
import sys
from typing import Optional
DEFAULT_LOG_LEVEL = logging.INFO
def get_sub_loggers():
return [
logging.getLogger('a'),
logging.getLogger('b'),
logging.getLogger('c')
]
def safe_add_handler(logger: logging.Logger, h: logging.Handler) -> None:
if h not in logger.handlers:
logger.addHandler(h)
def safe_add_filter(logger: logging.Logger, f: logging.Filter) -> None:
if f not in logger.filters:
logger.addFilter(f)
def configure_logging(main_logger: logging.Logger,
log_level = DEFAULT_LOG_LEVEL,
log_file = None,
fmt: Optional[str] = None,
force_add_handler: bool = False) -> None:
main_logger.setLevel(log_level)
logging.captureWarnings(True) # send all warnings to be logged
if force_add_handler or not main_logger.hasHandlers():
if log_file is None or log_file == '-':
log_file = sys.stderr
if isinstance(log_file, (str, os.PathLike)):
handler = logging.FileHandler(log_file)
else:
handler = logging.StreamHandler(log_file)
handler.setFormatter(logging.Formatter(fmt if fmt is not None else logging.BASIC_FORMAT))
main_logger.addHandler(handler)
log_output_location = handler.stream.name if isinstance(handler, logging.StreamHandler) else handler.baseFilename
main_logger.info('Writing logs to {}'.format(log_output_location))
else:
main_logger.info('Using existing log handlers')
main_logger.info('Using log level {}'.format(log_level))
for logger in get_sub_loggers():
logger.setLevel(log_level)
for h in main_logger.handlers:
safe_add_handler(logger, h)
for f in main_logger.filters:
safe_add_filter(logger, f)
main_logger.debug('Configured logger {}'.format(logger.name))
main_logger.info('Configured submodule loggers.')
configure_logging(logging.getLogger())
a_logger = logging.getLogger('a')
assert len(a_logger.handlers) == 1
# Should only output "hi" once, but it does it twice
a_logger.info('hi')
Loggers propagate log events to their ancestors' handlers by default. a_logger may only have one handler, but its parent, the root logger, also has a handler (actually the same handler). a_logger.info('hi') is handled by both handlers (actually the same handler twice).
You don't need to attach the same handler to every logger. Attaching it to the root logger is enough.
Let's simplify this. My goal is to make color output in terminal using logging module in Python. I want info has a green prefix, warnings have a yellow prefix and errors have a red prefix. To make it simple let's use *** as the prefix i.e.
*** log text
*** another message with another prefix color
What I have done so far
# declaration of function (global scope)
log = None
warn = None
error = None
def build_log_funcs():
# why I initialize it inside the function ?
# because script doesnt have to know about logging method
# the function just provide log functions
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
sh = logging.StreamHandler()
# LOG_FORMAT just global variable with pattern including %(levelmarks)s
# it will be replaced with ** with proper color
formatter = logging.Formatter(LOG_FORMAT)
sh.setFormatter(formatter)
logger.addHandler(sh)
def make_log_func(func, color, is_exit = False):
color_string = "\x1b[{};1m***\x1b[0m".format(color)
def newfunc(*args, **kwargs):
func(*args, extra={'levelmarks':color_string}, **kwargs)
if is_exit:
sys.exit(-1)
return newfunc
# 32, 33, 31 are color codes
log = make_log_func(logger.info, 32)
warn = make_log_func(logger.warning, 33)
error = make_log_func(logger.error, 31, is_exit = True)
return log, warn, error
And use it as
log, warn, error = build_log_funcs()
It works but what I don't like: (from small to big problems)
I hide the capabilities of logging module. For example enabling/disabling debug messages
I should use global declaration of functions before their initialization because I can't call a function before its declaration.
It's too difficult to read and maintain the code. I believe that everything should be as simple as possible.
Why don't I just make simple log, warn, simple function? I don't know. logging is the very comprehensive module so may be I will need its features in the future.
My question is how would you solve this problem? May be there is a simple obvious way which I don't know.
Thanks Dominic Kexel for this link. I saw this but did not pay attention to the answer.
The following code is more or less suitable for me
def setup_logger(logger):
logger.setLevel(logging.DEBUG)
sh = logging.StreamHandler()
formatter = logging.Formatter(LOG_FORMAT)
sh.setFormatter(formatter)
def decorate_emit(fn):
# add methods we need to the class
def new(*args):
levelno = args[0].levelno
if(levelno >= logging.CRITICAL):
color = '\x1b[31;1m'
elif(levelno >= logging.ERROR):
color = '\x1b[31;1m'
elif(levelno >= logging.WARNING):
color = '\x1b[33;1m'
elif(levelno >= logging.INFO):
color = '\x1b[32;1m'
elif(levelno >= logging.DEBUG):
color = '\x1b[35;1m'
else:
color = '\x1b[0m'
# add colored *** in the beginning of the message
args[0].msg = "{0}***\x1b[0m {1}".format(color, args[0].msg)
# new feature i like: bolder each args of message
args[0].args = tuple('\x1b[1m' + arg + '\x1b[0m' for arg in args[0].args)
return fn(*args)
return new
sh.emit = decorate_emit(sh.emit)
logger.addHandler(sh)
There is one flaw in this: I can't control the position of *** in the pattern but as I said it's suitable.
I have been trying for some time to figure out how to create a custom function for the python logging module. My goal is, with the usual function such as logging.debug(...) a log message over several channels, such as Telegram or MQTT, to publishing. So my idea is to add extra arguments to the normal log methode. For example logging.debug ("a log", telegram=True, mqtt=False) and maybe other arguments. All I find is the inheritance of the class logging.StreamingHandler and then using the method emit, but this only passes the argument record. So how can I implement my problem in a meaningful way? Do I have a thinking error or the wrong approach?
I solved my problem by creating a interface for the logging module.
A small view on my code:
# ulogging.py
import logging
import telegram
def uloggingConfig(*args, **kwargs):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# general logging config section
fmt = kwargs.pop("fmt", "%(asctime)s %(levelname)s %(message)s")
datefmt = kwargs.pop("datefmt", "%m.%d.%Y %I:%M:%S %p")
streamHandler = logging.StreamHandler()
streamHandler.setLevel(logging.DEBUG)
formater = logging.Formatter(fmt=fmt, datefmt=datefmt)
streamHandler.setFormatter(formater)
logger.addHandler(streamHandler)
# telegram config section
telegramBot = kwargs.pop("teleBot", None)
telegramChatID = kwargs.pop("teleChatID", None)
telegramLevel = kwargs.pop("teleLevel", logging.INFO)
telegramFmt = kwargs.pop("telefmt", "%(message)s")
telegramDatefmt = kwargs.pop("teledatefmt", None)
if telegramBot is not None and telegramChatID is not None:
telegramStream = TelegramStream(telegramBot, telegramChatID)
formater = logging.Formatter(fmt=telegramFmt, datefmt=telegramDatefmt)
telegramStream.setFormatter(formater)
telegramStream.setLevel(telegramLevel)
logger.addHandler(telegramStream)
elif (telegramBot is not None and telegramChatID is None) or (telegramBot is None and telegramChatID is not None):
raise KeyError("teleBot and teleChatID have to be both given")
if kwargs:
keys = ', '.join(kwargs.keys())
raise ValueError('Unrecognised argument(s): %s' % keys)
return logger
def getLogger():
return logging.getLogger(__name__)
class TelegramStream(logging.StreamHandler):
def __init__(self, telegramBot, telegramChatID):
logging.StreamHandler.__init__(self)
self._bot = telegramBot
self._chatID = telegramChatID
def emit(self, record):
if record.levelname == "DEBUG":
self._bot.send_message(self._chatID, record.levelname + ": " + record.msg)
else:
self._bot.send_message(self._chatID, record.msg)
With the uloggingConfig() method I can now pass all settings for the different handlers (at the moment only for telegram, further handlers should follow). The uloggingConfig() method then takes over the configuration and returns a logger with which log messages can be created as usual.
A simple example:
logger = ulogging.uloggingConfig(fmt="format, datefmt="dateformat, teleBot=telegramBot, teleChatID=telegramChatID, teleLevel=logging.DEBUG)
logger.debug("A log message")