My current format string is:
formatter = logging.Formatter('%(asctime)s : %(message)s')
and I want to add a new field called app_name which will have a different value in each script that contains this formatter.
import logging
formatter = logging.Formatter('%(asctime)s %(app_name)s : %(message)s')
syslog.setFormatter(formatter)
logger.addHandler(syslog)
But I'm not sure how to pass that app_name value to the logger to interpolate into the format string. I can obviously get it to appear in the log message by passing it each time but this is messy.
I've tried:
logging.info('Log message', app_name='myapp')
logging.info('Log message', {'app_name', 'myapp'})
logging.info('Log message', 'myapp')
but none work.
You could use a LoggerAdapter so you don't have to pass the extra info with every logging call:
import logging
extra = {'app_name':'Super App'}
logger = logging.getLogger(__name__)
syslog = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s %(app_name)s : %(message)s')
syslog.setFormatter(formatter)
logger.setLevel(logging.INFO)
logger.addHandler(syslog)
logger = logging.LoggerAdapter(logger, extra)
logger.info('The sky is so blue')
logs (something like)
2013-07-09 17:39:33,596 Super App : The sky is so blue
Filters can also be used to add contextual information.
import logging
class AppFilter(logging.Filter):
def filter(self, record):
record.app_name = 'Super App'
return True
logger = logging.getLogger(__name__)
logger.addFilter(AppFilter())
syslog = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s %(app_name)s : %(message)s')
syslog.setFormatter(formatter)
logger.setLevel(logging.INFO)
logger.addHandler(syslog)
logger.info('The sky is so blue')
produces a similar log record.
Python3
As of Python3.2 you can now use LogRecordFactory
import logging
logging.basicConfig(format="%(custom_attribute)s - %(message)s")
old_factory = logging.getLogRecordFactory()
def record_factory(*args, **kwargs):
record = old_factory(*args, **kwargs)
record.custom_attribute = "my-attr"
return record
logging.setLogRecordFactory(record_factory)
>>> logging.info("hello")
my-attr - hello
Of course, record_factory can be customized to be any callable and the value of custom_attribute could be updated if you keep a reference to the factory callable.
Why is that better than using Adapters / Filters?
You do not need to pass your logger around the application
It actually works with 3rd party libraries that use their own logger (by just calling logger = logging.getLogger(..)) would now have the same log format. (this is not the case with Filters / Adapters where you need to be using the same logger object)
You can stack/chain multiple factories
You need to pass the dict as a parameter to extra to do it that way.
logging.info('Log message', extra={'app_name': 'myapp'})
Proof:
>>> import logging
>>> logging.basicConfig(format="%(foo)s - %(message)s")
>>> logging.warning('test', extra={'foo': 'bar'})
bar - test
Also, as a note, if you try to log a message without passing the dict, then it will fail.
>>> logging.warning('test')
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 846, in emit
msg = self.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 723, in format
return fmt.format(record)
File "/usr/lib/python2.7/logging/__init__.py", line 467, in format
s = self._fmt % record.__dict__
KeyError: 'foo'
Logged from file <stdin>, line 1
Another way is to create a custom LoggerAdapter. This is particularly useful when you can't change the format OR if your format is shared with code that does not send the unique key (in your case app_name):
class LoggerAdapter(logging.LoggerAdapter):
def __init__(self, logger, prefix):
super(LoggerAdapter, self).__init__(logger, {})
self.prefix = prefix
def process(self, msg, kwargs):
return '[%s] %s' % (self.prefix, msg), kwargs
And in your code, you would create and initialize your logger as usual:
logger = logging.getLogger(__name__)
# Add any custom handlers, formatters for this logger
myHandler = logging.StreamHandler()
myFormatter = logging.Formatter('%(asctime)s %(message)s')
myHandler.setFormatter(myFormatter)
logger.addHandler(myHandler)
logger.setLevel(logging.INFO)
Finally, you would create the wrapper adapter to add a prefix as needed:
logger = LoggerAdapter(logger, 'myapp')
logger.info('The world bores you when you are cool.')
The output will look something like this:
2013-07-09 17:39:33,596 [myapp] The world bores you when you are cool.
I found this SO question after implementing it myself. Hope it helps someone. In the code below, I'm inducing an extra key called claim_id in the logger format. It will log the claim_id whenever there is a claim_id key present in the environment. In my use case, I needed to log this information for an AWS Lambda function.
import logging
import os
LOG_FORMAT = '%(asctime)s %(name)s %(levelname)s %(funcName)s %(lineno)s ClaimID: %(claim_id)s: %(message)s'
class AppLogger(logging.Logger):
# Override all levels similarly - only info overriden here
def info(self, msg, *args, **kwargs):
return super(AppLogger, self).info(msg, extra={"claim_id": os.getenv("claim_id", "")})
def get_logger(name):
""" This function sets log level and log format and then returns the instance of logger"""
logging.setLoggerClass(AppLogger)
logging.basicConfig(level=logging.INFO, format=LOG_FORMAT)
logger = logging.getLogger(name)
logger.setLevel(logging.INFO)
return logger
LOGGER = get_logger(__name__)
LOGGER.info("Hey")
os.environ["claim_id"] = "12334"
LOGGER.info("Hey")
Gist: https://gist.github.com/ygivenx/306f2e4e1506f302504fb67abef50652
If you need a default extra mapping, and you want to customize it for ad-hoc log messages, this works in Python 2.7+ by creating a LoggerAdapter that merges a default extra dictionary with any extra from a given message.
import logging
import os
import sys
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s %(levelname)-8s Py%(python)-4s pid:%(pid)-5s %(message)s',
)
_logger = logging.getLogger("my-logger")
_logger.setLevel(logging.DEBUG)
class DefaultExtrasAdapter(logging.LoggerAdapter):
def __init__(self, logger, extra):
super(DefaultExtrasAdapter, self).__init__(logger, extra)
def process(self, msg, kwargs):
# Speed gain if no extras are present
if "extra" in kwargs:
copy = dict(self.extra).copy()
copy.update(kwargs["extra"])
kwargs["extra"] = copy
else:
kwargs["extra"] = self.extra
return msg, kwargs
LOG = DefaultExtrasAdapter(_logger, {"python": sys.version_info[0], "pid": os.getpid()})
if __name__ == "__main__":
LOG.info("<-- With defaults")
LOG.info("<-- With my version", extra={"python": 3.10})
LOG.info("<-- With my pid", extra={"pid": 0})
LOG.info("<-- With both", extra={"python": 2.7, "pid": -1})
Results:
2021-08-05 18:58:27,308 INFO Py2 pid:8435 <-- With defaults
2021-08-05 18:58:27,309 INFO Py3.1 pid:8435 <-- With my version
2021-08-05 18:58:27,309 INFO Py2 pid:0 <-- With my pid
2021-08-05 18:58:27,309 INFO Py2.7 pid:-1 <-- With both
The accepted answer did not log the format in logfile, whereas the format was reflected in sys output.
Alternatively I used a simpler approach and worked as;
logging.basicConfig(filename="mylogfile.test",
filemode="w+",
format='%(asctime)s: ' +app_name+': %(message)s ',
level=logging.DEBUG)
Using mr2ert's answer, I came up with this comfortable solution (Though I guess it's not recommended) - Override the built-in logging methods to accept the custom argument and create the extra dictionary inside the methods:
import logging
class CustomLogger(logging.Logger):
def debug(self, msg, foo, *args, **kwargs):
extra = {'foo': foo}
if self.isEnabledFor(logging.DEBUG):
self._log(logging.DEBUG, msg, args, extra=extra, **kwargs)
*repeat for info, warning, etc*
logger = CustomLogger('CustomLogger', logging.DEBUG)
formatter = logging.Formatter('%(asctime)s [%(foo)s] %(message)s')
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.debug('test', 'bar')
Output:
2019-03-02 20:06:51,998 [bar] test
This is the built in function for reference:
def debug(self, msg, *args, **kwargs):
"""
Log 'msg % args' with severity 'DEBUG'.
To pass exception information, use the keyword argument exc_info with
a true value, e.g.
logger.debug("Houston, we have a %s", "thorny problem", exc_info=1)
"""
if self.isEnabledFor(DEBUG):
self._log(DEBUG, msg, args, **kwargs)
import logging;
class LogFilter(logging.Filter):
def __init__(self, code):
self.code = code
def filter(self, record):
record.app_code = self.code
return True
logging.basicConfig(format='[%(asctime)s:%(levelname)s]::[%(module)s -> %(name)s] - APP_CODE:%(app_code)s - MSG:%(message)s');
class Logger:
def __init__(self, className):
self.logger = logging.getLogger(className)
self.logger.setLevel(logging.ERROR)
#staticmethod
def getLogger(className):
return Logger(className)
def logMessage(self, level, code, msg):
self.logger.addFilter(LogFilter(code))
if level == 'WARN':
self.logger.warning(msg)
elif level == 'ERROR':
self.logger.error(msg)
else:
self.logger.info(msg)
class Test:
logger = Logger.getLogger('Test')
if __name__=='__main__':
logger.logMessage('ERROR','123','This is an error')
Related
I want to use different log levels. For example I need to log actions as INFO if everyting is OK, and WARNING when the required action is NOT proceeded OK (or ERROR, FATAl etc. it doesn't matter right now). How can I reach it? Basically I need just 2 levels, but unfortunately if I am trying to use 2 levels then my log file created as empty. Below is my code but it doesn't work and log file created empty.
class Logger:
#staticmethod
def info_logger(logLevel=log.INFO):
logger_name = inspect.stack()[1][3]
logger = log.getLogger(logger_name)
logger.setLevel(logLevel)
fh = log.FileHandler(log_file)
formatter = log.Formatter("%(asctime)s - %(levelname)s - %(message)s", datefmt='%m/%d/%Y %I: %M: %S %p')
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
#staticmethod
def warning_logger(logLevel=log.WARNING):
logger_name = inspect.stack()[1][3]
logger = log.getLogger(logger_name)
logger.setLevel(logLevel)
fh = log.FileHandler(error_log_file)
formatter = log.Formatter("%(asctime)s - %(levelname)s - %(message)s", datefmt='%m/%d/%Y %I: %M: %S %p')
fh.setFormatter(formatter)
logger.addHandler(fh)
return logger
I'm trying to achieve a behavior available in log4j, which is described in this SO Q&A
Instead of the logger producing the following output (notice the full module name)
DEBUG aproject.bpackage.cclass:filename.py:79 log message 1
I'd like to only have the initials of the logger name:
DEBUG a.b.c:filename.py:79 log message 1
Trying to figure out the correct format string with the manual, but no mention of initials for the logger name
Note that loggers are initialized with the module name as convention:
logger = logging.getLogger(__name__)
There's no out-of-the-box functionality for this, but you can achieve the desired result using something like this:
import logging
class CustomFormatter(logging.Formatter):
def format(self, record):
saved_name = record.name # save and restore for other formatters if desired
parts = saved_name.split('.')
# import pdb; pdb.set_trace()
record.name = '.'.join(p[0] for p in parts)
result = super().format(record)
record.name = saved_name
return result
h = logging.StreamHandler()
f = CustomFormatter('%(name)-6s %(message)s')
h.setFormatter(f)
root = logging.getLogger()
root.addHandler(h)
root.setLevel(logging.DEBUG)
logging.getLogger('alpha.beta.gamma').debug('foo')
logging.getLogger('delta.epsilon.zeta').debug('bar')
logging.getLogger('eta.theta').debug('baz')
When run, the above script outputs
a.b.g foo
d.e.z bar
e.t baz
I have been using a custom formatter for logging to the terminal in my code. Lately I have been changing stuff in the code an I can't find why now in some parts of the code the log is printed twice.
This is the code for the custom formatter:
import logging
class MyFormatter(logging.Formatter):
debug_format = "[%(levelname)s] (%(module)s::%(funcName)s::%(lineno)d) %(message)s"
normal_format = "[%(levelname)s] %(message)s"
blue = "\x1b[36;21m"
grey = "\x1b[38;21m"
yellow = "\x1b[33;21m"
red = "\x1b[31;21m"
bold_red = "\x1b[31;1m"
reset = "\x1b[0m"
def __init__(self):
super().__init__(fmt="%(levelno)d: %(msg)s", datefmt=None, style="%")
def format(self, record):
# Save the original format configured by the user
# when the logger formatter was instantiated
format_orig = self._style._fmt
# Replace the original format with one customized by logging level
if record.levelno == logging.DEBUG:
self._style._fmt = MyFormatter.debug_format
format = MyFormatter.debug_format
else:
self._style._fmt = MyFormatter.normal_format
format = MyFormatter.normal_format
self.FORMATS = {
logging.DEBUG: MyFormatter.grey + format + MyFormatter.reset,
logging.INFO: MyFormatter.blue + format + MyFormatter.reset,
logging.WARNING: MyFormatter.yellow + format + MyFormatter.reset,
logging.ERROR: MyFormatter.red + format + MyFormatter.reset,
logging.CRITICAL: MyFormatter.bold_red + format + MyFormatter.reset,
}
log_fmt = self.FORMATS.get(record.levelno)
# Restore the original format configured by the user
self._style._fmt = format_orig
formatter = logging.Formatter(log_fmt)
return formatter.format(record)
This is how I create my logger:
from src.logs import set_logger, logging
logger = set_logger(__name__, logging.DEBUG)
This is set_logger function code:
import logging
from .custom_formatter import MyFormatter
def set_logger(module_name: str, level=logging.DEBUG) -> logging.Logger:
logger = logging.getLogger(module_name)
logger.setLevel(level)
stream_handler = logging.StreamHandler()
formatter = MyFormatter()
stream_handler.setFormatter(formatter)
logger.addHandler(stream_handler)
return logger
Now when I call this logger from main for example or at the top of a module which is imported, there is no problem, and it logs perfectly only once. However when calling the logger from inside a function in the same module it is printed twice.
I have notice by debugging that what is doing is going to the end of the format method in MyFormatter class and then it returns again to this format method, I have no clue what is going on here. Do you have any ideas on what could be happening?
PD: Also if I also call a print when the logger prints twice I only get one print, so that code runs only once for sure.
Thanks for your time!
Andrés
In set_logger(), it calls addHandler() but the logger (or an ancestor logger) will already have a handler, which you're not removing, so you'll have multiple handlers.
Have a look at the docs for Logger.propagate: https://docs.python.org/3/library/logging.html#logging.Logger.propagate
I am getting duplicate log output from a logger that only has 1 handler attached.
How can I debug this?
Code (should be copy-and-paste-able):
import io
import logging
import os
import sys
from typing import Optional
DEFAULT_LOG_LEVEL = logging.INFO
def get_sub_loggers():
return [
logging.getLogger('a'),
logging.getLogger('b'),
logging.getLogger('c')
]
def safe_add_handler(logger: logging.Logger, h: logging.Handler) -> None:
if h not in logger.handlers:
logger.addHandler(h)
def safe_add_filter(logger: logging.Logger, f: logging.Filter) -> None:
if f not in logger.filters:
logger.addFilter(f)
def configure_logging(main_logger: logging.Logger,
log_level = DEFAULT_LOG_LEVEL,
log_file = None,
fmt: Optional[str] = None,
force_add_handler: bool = False) -> None:
main_logger.setLevel(log_level)
logging.captureWarnings(True) # send all warnings to be logged
if force_add_handler or not main_logger.hasHandlers():
if log_file is None or log_file == '-':
log_file = sys.stderr
if isinstance(log_file, (str, os.PathLike)):
handler = logging.FileHandler(log_file)
else:
handler = logging.StreamHandler(log_file)
handler.setFormatter(logging.Formatter(fmt if fmt is not None else logging.BASIC_FORMAT))
main_logger.addHandler(handler)
log_output_location = handler.stream.name if isinstance(handler, logging.StreamHandler) else handler.baseFilename
main_logger.info('Writing logs to {}'.format(log_output_location))
else:
main_logger.info('Using existing log handlers')
main_logger.info('Using log level {}'.format(log_level))
for logger in get_sub_loggers():
logger.setLevel(log_level)
for h in main_logger.handlers:
safe_add_handler(logger, h)
for f in main_logger.filters:
safe_add_filter(logger, f)
main_logger.debug('Configured logger {}'.format(logger.name))
main_logger.info('Configured submodule loggers.')
configure_logging(logging.getLogger())
a_logger = logging.getLogger('a')
assert len(a_logger.handlers) == 1
# Should only output "hi" once, but it does it twice
a_logger.info('hi')
Loggers propagate log events to their ancestors' handlers by default. a_logger may only have one handler, but its parent, the root logger, also has a handler (actually the same handler). a_logger.info('hi') is handled by both handlers (actually the same handler twice).
You don't need to attach the same handler to every logger. Attaching it to the root logger is enough.
I have been trying for some time to figure out how to create a custom function for the python logging module. My goal is, with the usual function such as logging.debug(...) a log message over several channels, such as Telegram or MQTT, to publishing. So my idea is to add extra arguments to the normal log methode. For example logging.debug ("a log", telegram=True, mqtt=False) and maybe other arguments. All I find is the inheritance of the class logging.StreamingHandler and then using the method emit, but this only passes the argument record. So how can I implement my problem in a meaningful way? Do I have a thinking error or the wrong approach?
I solved my problem by creating a interface for the logging module.
A small view on my code:
# ulogging.py
import logging
import telegram
def uloggingConfig(*args, **kwargs):
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
# general logging config section
fmt = kwargs.pop("fmt", "%(asctime)s %(levelname)s %(message)s")
datefmt = kwargs.pop("datefmt", "%m.%d.%Y %I:%M:%S %p")
streamHandler = logging.StreamHandler()
streamHandler.setLevel(logging.DEBUG)
formater = logging.Formatter(fmt=fmt, datefmt=datefmt)
streamHandler.setFormatter(formater)
logger.addHandler(streamHandler)
# telegram config section
telegramBot = kwargs.pop("teleBot", None)
telegramChatID = kwargs.pop("teleChatID", None)
telegramLevel = kwargs.pop("teleLevel", logging.INFO)
telegramFmt = kwargs.pop("telefmt", "%(message)s")
telegramDatefmt = kwargs.pop("teledatefmt", None)
if telegramBot is not None and telegramChatID is not None:
telegramStream = TelegramStream(telegramBot, telegramChatID)
formater = logging.Formatter(fmt=telegramFmt, datefmt=telegramDatefmt)
telegramStream.setFormatter(formater)
telegramStream.setLevel(telegramLevel)
logger.addHandler(telegramStream)
elif (telegramBot is not None and telegramChatID is None) or (telegramBot is None and telegramChatID is not None):
raise KeyError("teleBot and teleChatID have to be both given")
if kwargs:
keys = ', '.join(kwargs.keys())
raise ValueError('Unrecognised argument(s): %s' % keys)
return logger
def getLogger():
return logging.getLogger(__name__)
class TelegramStream(logging.StreamHandler):
def __init__(self, telegramBot, telegramChatID):
logging.StreamHandler.__init__(self)
self._bot = telegramBot
self._chatID = telegramChatID
def emit(self, record):
if record.levelname == "DEBUG":
self._bot.send_message(self._chatID, record.levelname + ": " + record.msg)
else:
self._bot.send_message(self._chatID, record.msg)
With the uloggingConfig() method I can now pass all settings for the different handlers (at the moment only for telegram, further handlers should follow). The uloggingConfig() method then takes over the configuration and returns a logger with which log messages can be created as usual.
A simple example:
logger = ulogging.uloggingConfig(fmt="format, datefmt="dateformat, teleBot=telegramBot, teleChatID=telegramChatID, teleLevel=logging.DEBUG)
logger.debug("A log message")