While subclassing logging.Handler, I can make a custom handler by doing something like:
import requests
import logging
class RequestsHandler(logging.Handler):
def emit(self, record):
res = requests.get('http://google.com')
print (res, record)
handler = RequestsHandler()
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.warning('ok!')
# <Response [200]> <LogRecord: __main__, 30, <stdin>, 1, "ok!">
What would be the simplest RequestHandler (i.e., what methods would it need?) if it was just a base class without subclassing logging.Handler ?
In general, you can find out which attributes of a class is getting accessed externally by overriding the __getattribue__ method with a wrapper function
that adds the name of the attribute being accessed to a set if the caller's class is not the same as the current class:
import logging
import sys
class MyHandler(logging.Handler):
def emit(self, record):
pass
def show_attribute(self, name):
caller_locals = sys._getframe(1).f_locals
if ('self' not in caller_locals or
object.__getattribute__(caller_locals['self'], '__class__') is not
object.__getattribute__(self, '__class__')):
attributes.add(name)
return original_getattribute(self, name)
attributes = set()
original_getattribute = MyHandler.__getattribute__
MyHandler.__getattribute__ = show_attribute
so that:
handler = MyHandler()
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.warning('ok!')
print(attributes)
outputs:
{'handle', 'level'}
Demo: https://repl.it/#blhsing/UtterSoupyCollaborativesoftware
As you see from the result above, handle and level are the only attributes needed for a basic logging handler. In other words, #jirassimok is correct in that handle is the only method of the Handler class that is called externally, but one also needs to implement the level attribute as well since it is also directly accessed in the Logger.callHandlers method:
if record.levelno >= hdlr.level:
where the level attribute has to be an integer, and should be 0 if records of all logging levels are to be handled.
A minimal implementation of a Handler class should therefore be something like:
class MyHandler:
def __init__(self):
self.level = 0
def handle(self, record):
print(record.msg)
so that:
handler = MyHandler()
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.warning('ok!')
outputs:
ok!
Looking at the source for Logger.log leads me to Logger.callHandlers, which calls only handle on the handlers. So that might be the minimum you need if you're injecting the fake handler directly into a logger instance.
If you want to really guarantee compatibility with the rest of the logging module, the only thing you can do is go through the module's source to figure out how it works. The documentation is a good starting place, but that doesn't get into the internals much at all.
If you're just trying to write a dummy handler for a small use case, you could probably get away with skipping a lot of steps; try something, see where it fails, and build on that.
Otherwise, you won't have much choice but to dive into the source code (though trying things and seeing what breaks can also be a good way to find places to start reading).
A quick glance at the class' source tells me that the only gotchas in the class are related to the module's internal management of its objects; Handler.__init__ puts the handler into a global handler list, which the module could use in any number of places. But beyond that, the class is quite straightforward; it shouldn't be too hard to read.
Related
I have a root logging class that I created, which I'd like to use for each micro-service function that I'm deploying.
Example output log: [2023-01-01 13:46:26] - INFO - [utils.logger.<module>:5] - testaaaaa
The logger is defined in utils.logger so that's why it's showing that in the log, hence %(name)s.
Instead of using the same root logger name which is set with logger = logging.getLogger(__name__), how can I get the same structure in dot notation where the logger is being instantiated and called?
Even if I have to modify my logger class to accept a name parameter when initializing the object, that is fine. But I like the dot notation because I will have files like routes.users.functiona, routes.users.functionb, routes.database.functiona and so on.
So I want to show which module the logging came from. Can't seem to follow how logging is capturing the path when using __name__.
Also if you have any suggestions about making the following more robust :)
Here is my class:
import typing
import logging
class GlobalLogger:
MINIMUM_GLOBAL_LEVEL = logging.DEBUG
GLOBAL_HANDLER = logging.StreamHandler()
LOG_FORMAT = "[%(asctime)s] - %(levelname)s - [%(name)s.%(funcName)s:%(lineno)d] - %(message)s"
LOG_DATETIME_FORMAT = "%Y-%m-%d %H:%M:%S"
def __init__(self, level: typing.Union[int, str] = MINIMUM_GLOBAL_LEVEL):
self.level = level
self.logger = self._get_logger()
self.log_format = self._log_formatter()
self.GLOBAL_HANDLER.setFormatter(self.log_format)
def _get_logger(self):
logger = logging.getLogger(__name__)
logger.setLevel(self.level)
logger.addHandler(self.GLOBAL_HANDLER)
return logger
def _log_formatter(self):
return logging.Formatter(fmt=self.LOG_FORMAT, datefmt=self.LOG_DATETIME_FORMAT)
I think you misunderstood the logging concept.
The variable __name__ holds the name of Python package where this variable is used. I think in your case it will have the value where class GlobalLogger is defined. And it does not matter from where you will create instance of your class. => Check it with print command.
At the top of every of your Python module (*.py), initialize module logger simply by logger = logging.getLogger(__name__). In your access the logging methods via module variable logger.
The logging data are automatically propagated to the parent logger based on the dot notation. If you have module A.B.C, the data are propagated C -> B -> A.
The output handlers, formatting can be defined for every logger or just for root logger e.g.: A or just (empty) "".
I usually have function InitLogging, where I get handle to the root logger and setup the output handlers. You do not need setup logging handlers in every module.
Conclusion:
If you still want use your class, give name of your logger as named parameter into the __init__. When you create instance of your class use variable __name__ during initialization.
I think there is not needed to create so many specific logger classes. You can define the logger properties later.
A library that I use emits warnings and errors through the logging module (logging.Logger's warn() and error() methods). I would like to implement an option to turn the warnings into errors (i.e., fail on warnings).
Is there an easy way to achieve this?
From looking at the documentation, I cannot see a ready-made solution. I assume it is possible by adding a custom Handler object, but I am not sure how to do it "right". Any pointers?
#hoefling's answer is close, but I would change it like so:
class LevelRaiser(logging.Filter):
def filter(self, record):
if record.levelno == logging.WARNING:
record.levelno = logging.ERROR
record.levelname = logging.getLevelName(logging.ERROR)
return True
def configure_library_logging():
library_root_logger = logging.getLogger(library.__name__)
library_root_logger.addFilter(LevelRaiser())
The reason is that filters are used to change LogRecord attributes and filter stuff out, whereas handlers are used to do I/O. What you're trying to do here isn't I/O, and so doesn't really belong in a handler.
Update: I like the proposal of Vinay made in this answer, injecting a custom Filter instead of a Handler is a much cleaner way. Please check it out!
You are on the right track with implementing own Handler. This is pretty easy to implement. I would do it like that: write a handler that edits the LogRecord in-place and attach one handler instance to the library's root loggers. Example:
# library.py
import logging
_LOGGER = logging.getLogger(__name__)
def library_stuff():
_LOGGER.warning('library stuff')
This is a script that uses the library:
import logging
import library
class LevelRaiser(logging.Handler):
def emit(self, record: logging.LogRecord):
if record.levelno == logging.WARNING:
record.levelno = logging.ERROR
record.levelname = logging.getLevelName(logging.ERROR)
def configure_library_logging():
library_root_logger = logging.getLogger(library.__name__)
library_root_logger.addHandler(LevelRaiser())
if __name__ == '__main__':
# do some example global logging config
logging.basicConfig(level=logging.INFO)
# additional configuration for the library logging
configure_library_logging()
# play with different loggers
our_logger = logging.getLogger(__name__)
root_logger = logging.getLogger()
root_logger.warning('spam')
our_logger.warning('eggs')
library.library_stuff()
root_logger.warning('foo')
our_logger.warning('bar')
library.library_stuff()
Run the script:
WARNING:root:spam
WARNING:__main__:eggs
ERROR:library:library stuff
WARNING:root:foo
WARNING:__main__:bar
ERROR:library:library stuff
Note that warning level is elevated to error level only on library's logging prints, all the rest remains unchanged.
You can assign logging.warn to logging.error before calling methods from your library:
import logging
warn_log_original = logging.warn
logging.warn = logging.error
library_call()
logging.warn = warn_log_original
I would like to start with a basic logging class that inherits from Python's logging.Logger class. However, I am not sure about how I should be constructing my class so that I can establish the basics needed for customising the inherited logger.
This is what I have so far in my logger.py file:
import sys
import logging
from logging import DEBUG, INFO, ERROR
class MyLogger(object):
def __init__(self, name, format="%(asctime)s | %(levelname)s | %(message)s", level=INFO):
# Initial construct.
self.format = format
self.level = level
self.name = name
# Logger configuration.
self.console_formatter = logging.Formatter(self.format)
self.console_logger = logging.StreamHandler(sys.stdout)
self.console_logger.setFormatter(self.console_formatter)
# Complete logging config.
self.logger = logging.getLogger("myApp")
self.logger.setLevel(self.level)
self.logger.addHandler(self.console_logger)
def info(self, msg, extra=None):
self.logger.info(msg, extra=extra)
def error(self, msg, extra=None):
self.logger.error(msg, extra=extra)
def debug(self, msg, extra=None):
self.logger.debug(msg, extra=extra)
def warn(self, msg, extra=None):
self.logger.warn(msg, extra=extra)
This is the main myApp.py:
import entity
from core import MyLogger
my_logger = MyLogger("myApp")
def cmd():
my_logger.info("Hello from %s!" % ("__CMD"))
entity.third_party()
entity.another_function()
cmd()
And this is the entity.py module:
# Local modules
from core import MyLogger
# Global modules
import logging
from logging import DEBUG, INFO, ERROR, CRITICAL
my_logger = MyLogger("myApp.entity", level=DEBUG)
def third_party():
my_logger.info("Initial message from: %s!" % ("__THIRD_PARTY"))
def another_function():
my_logger.warn("Message from: %s" % ("__ANOTHER_FUNCTION"))
When I run the main app, I get this:
2016-09-14 12:40:50,445 | INFO | Initial message from: __THIRD_PARTY!
2016-09-14 12:40:50,445 | INFO | Initial message from: __THIRD_PARTY!
2016-09-14 12:40:50,445 | WARNING | Message from: __ANOTHER_FUNCTION
2016-09-14 12:40:50,445 | WARNING | Message from: __ANOTHER_FUNCTION
2016-09-14 12:40:50,445 | INFO | Hello from __CMD!
2016-09-14 12:40:50,445 | INFO | Hello from __CMD!
Everything is printed twice, as probably I have failed to set the logger class properly.
Update 1
Let me clarify my goals.
(1) I would like to encapsulate the main logging functionality in one single location so I can do this:
from mylogger import MyLogger
my_logger = MyLogger("myApp")
my_logger.info("Hello from %s!" % ("__CMD"))
(2) I am planning to use CustomFormatter and CustomAdapter classes. This bit does not require a custom logging class, those can be plugged in straight away.
(3) I probably do not need to go very deep in terms of customisation of the underlying logger class (records etc.), intercepting logger.info, loggin.debug etc. should be enough.
So referring back to this python receipt that has been circulated many times on these forums:
I am trying to the find the sweet point between having a Logger Class, yet still be able to use the built in functions like assigning Formatters and Adapters etc. So everything stays compatible with the logging module.
class OurLogger(logging.getLoggerClass()):
def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
# Don't pass all makeRecord args to OurLogRecord bc it doesn't expect "extra"
rv = OurLogRecord(name, level, fn, lno, msg, args, exc_info, func)
# Handle the new extra parameter.
# This if block was copied from Logger.makeRecord
if extra:
for key in extra:
if (key in ["message", "asctime"]) or (key in rv.__dict__):
raise KeyError("Attempt to overwrite %r in LogRecord" % key)
rv.__dict__[key] = extra[key]
return rv
Update 2
I have created a repo with a simple python app demonstrating a possible solution. However, I am keen to improve this.
xlog_example
This example effectively demonstrates the technique of overriding the logging.Logger class and the logging.LogRecord class through inheritance.
Two external items are mixed into the log stream: funcname and username without using any Formatters or Adapters.
At this stage, I believe that the research I have made so far and the example provided with the intention to wrap up the solution is sufficient to serve as an answer to my question. In general, there are many approaches that may be utilised to wrap a logging solution. This particular question aimed to focus on a solution that utilises logging.Logger class inheritance so that the internal mechanics can be altered, yet the rest of the functionality kept as it is since it is going to be provided by the original logging.Logger class.
Having said that, class inheritance techniques should be used with great care. Many of the facilities provided by the logging module are already sufficient to maintain and run a stable logging workflow. Inheriting from the logging.Logger class is probably good when the goal is some kind of a fundamental change to the way the log data is processed and exported.
To summarise this, I see that there are two approaches for wrapping logging functionality:
1) The Traditional Logging:
This is simply working with the provided logging methods and functions, but wrap them in a module so that some of the generic repetitive tasks are organised in one place. In this way, things like log files, log levels, managing custom Filters, Adapters etc. will be easy.
I am not sure if a class approach can be utilised in this scenario (and I am not talking about a super class approach which is the topic of the second item) as it seems that things are getting complicated when the logging calls are wrapped inside a class. I would like to hear about this issue and I will definitely prepare a question that explores this aspect.
2) The Logger Inheritance:
This approach is based on inheriting from the original logging.Logger class and adding to the existing methods or entirely hijacking them by modifying the internal behaviour. The mechanics are based on the following bit of code:
# Register our logger.
logging.setLoggerClass(OurLogger)
my_logger = logging.getLogger("main")
From here on, we are relying on our own Logger, yet we are still able to benefit from all of the other logging facilities:
# We still need a loggin handler.
ch = logging.StreamHandler()
my_logger.addHandler(ch)
# Confgure a formatter.
formatter = logging.Formatter('LOGGER:%(name)12s - %(levelname)7s - <%(filename)s:%(username)s:%(funcname)s> %(message)s')
ch.setFormatter(formatter)
# Example main message.
my_logger.setLevel(DEBUG)
my_logger.warn("Hi mom!")
This example is crucial as it demonstrates the injection of two data bits username and funcname without using custom Adapters or Formatters.
Please see the xlog.py repo for more information regarding this solution. This is an example that I have prepared based on other questions and bits of code from other sources.
This line
self.logger = logging.getLogger("myApp")
always retrieves a reference to the same logger, so you are adding an additional handler to it every time you instantiate MyLogger. The following would fix your current instance, since you call MyLogger with a different argument both times.
self.logger = logging.getLogger(name)
but note that you will still have the same problem if you pass the same name argument more than once.
What your class needs to do is keep track of which loggers it has already configured.
class MyLogger(object):
loggers = set()
def __init__(self, name, format="%(asctime)s | %(levelname)s | %(message)s", level=INFO):
# Initial construct.
self.format = format
self.level = level
self.name = name
# Logger configuration.
self.console_formatter = logging.Formatter(self.format)
self.console_logger = logging.StreamHandler(sys.stdout)
self.console_logger.setFormatter(self.console_formatter)
# Complete logging config.
self.logger = logging.getLogger(name)
if name not in self.loggers:
self.loggers.add(name)
self.logger.setLevel(self.level)
self.logger.addHandler(self.console_logger)
This doesn't allow you to re-configure a logger at all, but I leave it as an exercise to figure out how to do that properly.
The key thing to note, though, is that you can't have two separately configured loggers with the same name.
Of course, the fact that logging.getLogger always returns a reference to the same object for a given name means that your class is working at odds with the logging module. Just configure your loggers once at program start-up, then get references as necessary with getLogger.
I use a LoggerAdapter to let my python logging output Linux TIDs instead of the long unique IDs. But this way I don't modify an existing logger but I create a new object:
new_logger = logging.LoggerAdapter(
logger=logging.getLogger('mylogger'),
extra=my_tid_extractor())
Now I want this LoggerAdapter be used by certain modules. As long as I know a global variable being used as logger I can do something like this:
somemodule.logger = new_logger
But this is not nice - it works only in a couple of cases and you need to know the logger variables used by the modules.
Do you know a way to make a LoggerAdapter available globally e.g. by calling s.th. like
logging.setLogger('mylogger', new_logger)
Or alternatively: is there some other way to let Python logging output Linux thread IDs like printed by ps?
Alternatively, you can to implement custom logger, and make it default in logging module.
Here is example:
import logging
import ctypes
SYS_gettid = 186
libc = ctypes.cdll.LoadLibrary('libc.so.6')
FORMAT = '%(asctime)-15s [thread=%(tid)s] %(message)s'
logging.basicConfig(level=logging.DEBUG, format=FORMAT)
def my_tid_extractor():
tid = libc.syscall(SYS_gettid)
return {'tid': tid}
class CustomLogger(logging.Logger):
def _log(self, level, msg, args, exc_info=None, extra=None):
if extra is None:
extra = my_tid_extractor()
super(CustomLogger, self)._log(level, msg, args, exc_info, extra)
logging.setLoggerClass(CustomLogger)
logger = logging.getLogger('test')
logger.debug('test')
Output sample:
2015-01-20 19:24:09,782 [thread=5017] test
I think you need override LoggerAdapter.process() method
Because the default LoggerAdapter.process method will do nothing, Here is example：
import logging
import random
L=logging.getLogger('name')
class myLogger(logging.LoggerAdapter):
def process(self,msg,kwargs):
return '(%d),%s' % (self.extra['name1'](1,1000),msg) ,kwargs
#put the randint function object
LA=myLogger(L,{'name1':random.randint})
#now,do some logging
LA.debug('some_loging_messsage')
out>>DEBUG:name:(167),some_loging_messsage
I had a similar problem. My solution might be a bit more generic than the accepted one.
I’ve also used a custom logger class, but I did a generic extension that allows me to register adapters after it’s instantiated.
class AdaptedLogger(logging.Logger):
"""A logger that allows you to register adapters on a instance."""
def __init__(self, name):
"""Create a new logger instance."""
super().__init__(name)
self.adapters = []
def _log(self, level, msg, *args, **kwargs):
"""Let adapters modify the message and keyword arguments."""
for adapter in self.adapters:
msg, kwargs = adapter.process(msg, kwargs)
return super()._log(level, msg, *args, **kwargs)
To make you logger use the class you have to instantiate it before it is used elsewhere. For example using:
original_class = logging.getLoggerClass()
logging.setLoggerClass(AdaptedLogger)
logcrm_logger = logging.getLogger("test")
logging.setLoggerClass(original_class)
Then you can register adapters on the instance at any time later on.
logger = logging.getLogger("test")
adapter = logging.LoggerAdapter(logger, extra=my_tid_extractor())
logger.adapters.append(adapter)
Actually the “adapters” can be any object now as long as they have a process-method with a signature compatible with logging.LoggingAdapter.process().
I was wondering what the standard set up is for performing logging from within a Python app.
I am using the Logging class, and I've written my own logger class that instantiates the Logging class. My main then instantiates my logger wrapper class. However, my main instantiates other classes and I want those other classes to also be able to write to he log file via the logger object in the main.
How do I make that logger object such that it can be called by other classes? It's almost like we need some sort of static logger object to get this to work.
I guess the long and short of the question is: how do you implement logging within your code structure such that all classes instantiated from within main can write to the same log file? Do I just have to create a new logging object in each of the classes that points to the same file?
I don't know what you mean by the Logging class - there's no such class in Python's built-in logging. You don't really need wrappers: here's an example of how to do logging from arbitrary classes that you write:
import logging
# This class could be imported from a utility module
class LogMixin(object):
#property
def logger(self):
name = '.'.join([__name__, self.__class__.__name__])
return logging.getLogger(name)
# This class is just there to show that you can use a mixin like LogMixin
class Base(object):
pass
# This could be in a module separate from B
class A(Base, LogMixin):
def __init__(self):
# Example of logging from a method in one of your classes
self.logger.debug('Hello from A')
# This could be in a module separate from A
class B(Base, LogMixin):
def __init__(self):
# Another example of logging from a method in one of your classes
self.logger.debug('Hello from B')
def main():
# Do some work to exercise logging
a = A()
b = B()
with open('myapp.log') as f:
print('Log file contents:')
print(f.read())
if __name__ == '__main__':
# Configure only in your main program clause
logging.basicConfig(level=logging.DEBUG,
filename='myapp.log', filemode='w',
format='%(name)s %(levelname)s %(message)s')
main()
Generally it's not necessary to have loggers at class level: in Python, unlike say Java, the unit of program (de)composition is the module. However, nothing stops you from doing it, as I've shown above. The script, when run, displays:
Log file contents:
__main__.A DEBUG Hello from A
__main__.B DEBUG Hello from B
Note that code from both classes logged to the same file, myapp.log. This would have worked even with A and B in different modules.
Try using logging.getLogger() to get your logging object instance:
http://docs.python.org/3/library/logging.html#logging.getLogger
All calls to this function with a given name return the same logger instance. This means that logger instances never need to be passed between different parts of an application.
UPDATE:
The recommended way to do this is to use the getLogger() function and configure it (setting a handler, formatter, etc...):
# main.py
import logging
import lib
def main():
logger = logging.getLogger('custom_logger')
logger.setLevel(logging.INFO)
logger.addHandler(logging.FileHandler('test.log'))
logger.info('logged from main module')
lib.log()
if __name__ == '__main__':
main()
# lib.py
import logging
def log():
logger = logging.getLogger('custom_logger')
logger.info('logged from lib module')
If you really need to extend the logger class take a look at logging.setLoggerClass(klass)
UPDATE 2:
Example on how to add a custom logging level without changing the Logging class:
# main.py
import logging
import lib
# Extend Logger class
CUSTOM_LEVEL_NUM = 9
logging.addLevelName(CUSTOM_LEVEL_NUM, 'CUSTOM')
def custom(self, msg, *args, **kwargs):
self._log(CUSTOM_LEVEL_NUM, msg, args, **kwargs)
logging.Logger.custom = custom
# Do global logger instance setup
logger = logging.getLogger('custom_logger')
logger.setLevel(logging.INFO)
logger.addHandler(logging.FileHandler('test.log'))
def main():
logger = logging.getLogger('custom_logger')
logger.custom('logged from main module')
lib.log()
if __name__ == '__main__':
main()
Note that adding custom level is not recommended: http://docs.python.org/2/howto/logging.html#custom-levels
Defining a custom handler and maybe using more than one logger may do the trick for your other requirement: optional output to stderr.