When I run the below piece of code, the logging is working fine. But when I comment #1) and #2) under setup_logger the log is not displayed.
What does #1) and #2) do here?
import logging
import sys
def get_logger(name):
print('get_logger -- ', name)
log = logging.getLogger("hello.{}".format(name))
return log
def setup_logger():
print('setup_logger')
root = logging.getLogger("")
root.setLevel(logging.ERROR)
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter(
fmt="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
))
root.addHandler(handler)
logger = logging.getLogger("hello") #1
logger.setLevel(logging.DEBUG) #2
LOG = get_logger(__name__)
setup_logger()
print(LOG)
def main():
LOG.debug('hello')
if __name__ == '__main__':
main()
The first line you marked gets a logger with given name. If there currently is no logger with the name you provided, a new one will be created (Also known as Singleton). You can read more about logging.getLogger in the Python documentation :
Return a logger with the specified name or, if name is None, return a logger
which is the root logger of the hierarchy. If
specified, the name is typically a dot-separated hierarchical name
like ‘a’, ‘a.b’ or ‘a.b.c.d’. Choice of these names is entirely up to
the developer who is using logging.
All calls to this function with a given name return the same logger instance.
This means that logger instances never need to be
passed between different parts of an application.
The second line you marked sets the loglevel on the logger instance. If you set the loglevel to DEBUG, all messages you log will be printed to the console / file. If you set it to INFO, all messages except for
debug messages will be logged. And so on. Quote from the docs again:
Sets the threshold for this logger to level. Logging messages which
are less severe than level will be ignored; logging messages which
have severity level or higher will be emitted by whichever handler or
handlers service this logger, unless a handler’s level has been set to
a higher severity level than level.
When a logger is created, the level is set to NOTSET (which causes all
messages to be processed when the logger is the root logger, or
delegation to the parent when the logger is a non-root logger). Note
that the root logger is created with level WARNING.
If you still have questions, I can edit my post to answer them.
Overview
I want to use httpimport as a logging library common to several scripts. This module generates logs of its own which I do not know how to silence.
In other cases such as this one, I would have used
logging.getLogger('httpimport').setLevel(logging.ERROR)
but it did not work.
Details
The following code is a stub of the "common logging code" mentioned above:
# toconsole.py
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s %(message)s')
handler_console = logging.StreamHandler()
level = logging.DEBUG if 'DEV' in os.environ else logging.INFO
handler_console.setLevel(level)
handler_console.setFormatter(formatter)
log.addHandler(handler_console)
# disable httpimport logging except for errors+
logging.getLogger('httpimport').setLevel(logging.ERROR)
A simple usage such as
import httpimport
httpimport.INSECURE = True
with httpimport.remote_repo(['githublogging'], 'http://localhost:8000/') :
from toconsole import log
log.info('yay!')
gives the following output
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
2019-08-25 13:56:48,671 yay!
yay!
The second (bare) yay! must be coming from httpimport, namely from its logging setup.
How can I disable the logging for such a module, or better - raise its level so that only errors+ are logged?
Note: this question was initially asked at the Issues section of the GitHub repository for httpimport but the author did not know either how to fix that.
Author of httpimport here.
I totally forgot I was using the basicConfig logger thing.
It is fixed in master right now (0.7.2) - will be included in next PyPI release:
https://github.com/operatorequals/httpimport/commit/ff2896c8f666c3f16b0f27716c732d68be018ef7
The reason why this is happening is because when you do import httpimport they do the initial configuration for the logging machinery. This happens right here. What this means is that the root logger already has a StreamHandler attached to it. Because of this, and the fact that all loggers inherit from the root logger, when you do log.info('yay') it not only uses your Handler and Formatter, but it also propagates all they way to the root logger, which also emits the message.
Remember that whoever calls basicConfig first when an application starts that sets up the default configuration for the root logger, which in turn, is inherited by all loggers, unless otherwise specified.
If you have a complex logging configuration you need to ensure that you call it before you do any third-party imports which might call basicConfig. basicConfig is idempotent meaning the first call seals the deal, and subsequent calls have no effect.
Solutions
You could do log.propagate = False and you will see that the 2nd yay will not show.
You could attach the Formatter directly to the already existent root Handler by doing something like this (without adding another Handler yourself)
root = logging.getLogger('')
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler = root.handlers[0]
root_handler.setFormatter(formatter)
You could do a basicConfig call when you initialize your application (if you had such a config available, with initial Formatters and Handlers, etc. that will elegantly attach everything to the root logger neatly) and then you would only do something like logger = logging.getLogger(__name__) and logger.info('some message') that would work the way you'd expect because it would propagate all the way to the root logger which already has your configuration.
You could remove the initial Handler that's present on the root logger by doing something like
root = logging.getLogger('')
root.handlers = []
... and many more solutions, but you get the idea.
Also do note that logging.getLogger('httpimport').setLevel(logging.ERROR) this works perfectly fine. No messages below logging.ERROR would be logged by that logger, it's just that the problem wasn't from there.
If you however want to completely disable a logger you can just do logger.disabled = True (also do note, again, that the problem wasn't from the httpimport logger, as aforementioned)
One example demonstrated
Change your toconsole.py with this and you won't see the second yay.
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
root_logger = logging.getLogger('')
root_handler = root_logger.handlers[0]
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler.setFormatter(formatter)
# or you could just keep your old code and just add log.propagate = False
# or any of the above solutions and it would work
logging.getLogger('httpimport').setLevel(logging.ERROR)
The below code is copied from the documentation. I am supposed to be able to see all the info logs. But I don't. I am only able to see the warn and above even though I've set setLevel to INFO.
Why is this happening? foo.py:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
Output:
workingDirectory$ python foo.py
warn message
error message
critical message
Where did the info and debug messages go??
Replace the line
logger.setLevel(logging.DEBUG)
with
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
and it should work as expected. If you don't configure logging with any handlers (as in your post - you only configure a level for your logger, but no handlers anywhere), you'll get an internal handler "of last resort" which is set to output just the message (with no other formatting) at the WARNING level.
Try running logging.basicConfig() in there. Of note, I see you mention INFO, but use DEBUG. As written, it should show all five messages. Swap out DEBUG with INFO, and you should see four messages.
import logging
logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.debug('debug message')
logger.info('info message')
logger.warn('warn message')
logger.error('error message')
logger.critical('critical message')
edit: Do you have logging set up elsewhere in your code already? Can't reproduce the exact behavior you note with the specific code provided.
As pointed by some users, using:
logging.basicConfig(level=logging.DEBUG, format='%(message)s')
like written in the accepted answer is not a good option because it sets the log level for the root logger, so it may lead to unexpected behaviours (eg. third party libraries may start to log debug messages if you set loglevel=logging.DEBUG)
In my opinion the best solution is to set log level just for your logger, like this:
import logging
logger = logging.getLogger('MyLogger')
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.DEBUG)
Not really intuitive solution, but is necessary if you want to set log level only for 'MyLogger' and leave the root logger untouched.
So, why is logging.basicConfig(level=logging.DEBUG, format='%(message)s') setting the log level globally?
Well, actually it doesn't. As said, it's just changing the configuration of the root logger and, as described in the python documentation:
Loggers should NEVER be instantiated directly, but always through the
module-level function logging.getLogger(name). Multiple calls to
getLogger() with the same name will always return a reference to the
same Logger object.
So, logging.basicConfig is creating a StreamHandler with a default Formatter and adding it to the root logger.
The point is that if any other library is using the "root logger", you're going to set that log level for that library too so it can happen that you start to see debug logs from third party libraries.
This is why I think it's better to create your own logger and set your own formatters and handlers, so you can leave the root logger untouched.
This is technically also an "answer", because it can "solve" the problem. BUT I definitely DO NOT like it. It is not intuitive, and I lost 2+ hours over it.
Before:
import logging
logger = logging.getLogger('foo')
logger.setLevel(logging.INFO)
logger.info('You can not see me')
# Or you can just use the following one-liner in command line.
# $ python -c "import logging; logger = logging.getLogger('foo'); logger.setLevel(logging.INFO); logger.info('You can not see me')"
After:
import logging
logging.debug('invisible magic') # <-- magic
logger = logging.getLogger('foo')
logger.setLevel(logging.INFO)
logger.info('But now you can see me')
# Or you can just use the following one-liner in command line.
$ python -c "import logging; logging.debug('invisible magic'); logger = logging.getLogger('foo'); logger.setLevel(logging.INFO); logger.info('But now you see me')"
PS: Comparing it to the current chosen answer, and #Vinay-Sajip's explanation, I can kind of understand why. But still, I wish it was not working that way.
If you want this to work WITHOUT basicConfig, you have to first set up the lowest possible level you'll log onto the logger. Since the logger sets a minimum threshold, handlers which have a lower threshold but belong to the same logger won't get those lower threshold messages since they're ignored by the logger in the first place. Intuitive, but not obvious.
We start by doing this:
lgr = logging.getLogger(name)
lgr.setLevel(logging.DEBUG)
Then, set up the handlers with the different levels you need, in my case I want DEBUG logging on stdout and INFO logging to a rotating file, so I do the following:
rot_hndlr = RotatingFileHandler('filename.log',
maxBytes=log_size,
backupCount=3)
rot_hndlr.setFormatter(formatter)
rot_hndlr.setLevel(logging.INFO)
lgr.addHandler(rot_hndlr)
stream_hndlr = logging.StreamHandler()
stream_hndlr.setFormatter(stream_formatter)
lgr.addHandler(stream_hndlr)
Then, to test, I do this:
lgr.debug("Hello")
lgr.info("There")
My stdout (console) will look like this:
Hello
There
and my filename.log file will look like this:
There
In short, change the level in logging.basicConfig will influence the global settings.
You should better set level for each logger and the specific handler in the logger.
The following is an example that displays all levels on the console and only records messages >= errors in log_file.log. Notice the level for each handler is different.
import logging
# Define logger
logger = logging.getLogger('test')
# Set level for logger
logger.setLevel(logging.DEBUG)
# Define the handler and formatter for console logging
consoleHandler = logging.StreamHandler() # Define StreamHandler
consoleHandler.setLevel(logging.DEBUG) # Set level
concolsFormatter = logging.Formatter('%(name)s - %(levelname)s - %(message)s') # Define formatter
consoleHandler.setFormatter(concolsFormatter) # Set formatter
logger.addHandler(consoleHandler) # Add handler to logger
# Define the handler and formatter for file logging
log_file = 'log_file'
fileHandler = logging.FileHandler(f'{log_file}.log') # Define FileHandler
fileHandler.setLevel(logging.ERROR) # Set level
fileFormatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') # Define formatter
fileHandler.setFormatter(fileFormatter) # Set formatter
logger.addHandler(fileHandler) # Add handler to logger
# Test
logger.debug('This is a debug')
logger.info('This is an info')
logger.warning('This is a warning')
logger.error('This is an error')
logger.critical('This is a critical')
Console output:
# Test
test - DEBUG - This is a debug
test - INFO - This is an info
test - WARNING - This is a warning
test - ERROR - This is an error
test - CRITICAL - This is a critical
File log_file.log content:
2021-09-22 12:50:50,938 - test - ERROR - This is an error
2021-09-22 12:50:50,938 - test - CRITICAL - This is a critical
To review your logger's level:
logger.level
The result should be one of the following:
10 # DEBUG
20 # INFO
30 # WARNING
40 # ERROR
50 # CRITICAL
To review your handlers's levels:
logger.handlers
[<StreamHandler stderr (DEBUG)>,
<FileHandler ***/log_file.log (ERROR)>]
The accepted answer does not work for me on Win10, Python 3.7.2.
My solution:
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
It's order sensitive.
You have to set the basicConfig of the root logger to DEBUG, then you can set the level of your individual loggers to more restrictive levels.
This is not what I expected. Here is what I had to do:
#!/usr/bin/env python3
import logging
# by default this is WARNING. Leaving it as WARNING here overrides
# whatever setLevel-ing you do later so it seems they are ignored.
logging.basicConfig(level=logging.DEBUG)
l = logging.getLogger(__name__)
l.setLevel(level=logging.INFO)
# if I hadn't called basicConfig with DEBUG level earlier,
# info messages would STILL not be shown despite calling
# setLevel above. However now debug messages will not be shown
# for l because setLevel set it to INFO
l.warning('A warning message will be displayed')
l.info('A friendly info message will be displayed')
l.debug('A friendly debug message will not be displayed')
Most of the answers that I've found for this issue uses the basicConfig of the root logger.
It's not helpful for those who intend to use multiple independent loggers that were not initialised with basicConfig. The use of basicConfig implies that the loglevels of ALL loggers will be changed. It also had the unfortunate side effect of generating duplicate logs.
So I tried over several days experimenting with different ways to manipulate the loglevels and came up with one that finally worked.
The trick was to not only change the log levels of all the handlers but also the all the handlers of the parent of the logger.
def setLevel(self, infoLevel):
# To dynamically reset the loglevel, you need to also change the parent levels as well as all handlers!
self.logger.parent.setLevel(infoLevel)
for handler in self.logger.parent.handlers:
handler.setLevel(infoLevel)
self.logger.setLevel(infoLevel)
for handler in self.logger.handlers:
handler.setLevel(infoLevel)
The inspiration came from the fact that the basicConfig changes the root logger settings, so I was trying to do the same without using basicConfig.
For those that are interested, I did a little Python project on Github that illustrates the different issues with setting loglevel of the logger (it works partially), proves the SLogger (Sample Logger) implementation works, and also illustrates the duplicate log issue with basicConfig when using multiple loggers not initialised with it.
https://github.com/FrancisChung/python-logging-playground
TLDR: If you're only interested in a working sample code for the logger, the implentation is listed below
import logging
CRITICAL = 50
FATAL = CRITICAL
ERROR = 40
WARNING = 30
WARN = WARNING
INFO = 20
DEBUG = 10
NOTSET = 0
class SLogger():
"""
SLogger : Sample Logger class using the standard Python logging Library
Parameters:
name : Name of the Logger
infoLevel : logging level of the Logger (e.g. logging.DEBUG/INFO/WARNING/ERROR)
"""
def __init__(self, name: str, infoLevel=logging.INFO):
try:
if name is None:
raise ValueError("Name argument not specified")
logformat = '%(asctime)s %(levelname)s [%(name)s %(funcName)s] %(message)s'
self.logformat = logformat
self.name = name.upper()
self.logger = logging.getLogger(self.name)
self.logger.setLevel(infoLevel)
self.add_consolehandler(infoLevel, logformat)
except Exception as e:
if self.logger:
self.logger.error(str(e))
def error(self, message):
self.logger.error(message)
def info(self, message):
self.logger.info(message)
def warning(self, message):
self.logger.warning(message)
def debug(self, message):
self.logger.debug(message)
def critical(self, message):
self.logger.critical(message)
def setLevel(self, infoLevel):
# To dynamically reset the loglevel, you need to also change the parent levels as well as all handlers!
self.logger.parent.setLevel(infoLevel)
for handler in self.logger.parent.handlers:
handler.setLevel(infoLevel)
self.logger.setLevel(infoLevel)
for handler in self.logger.handlers:
handler.setLevel(infoLevel)
return self.logger.level
def add_consolehandler(self, infoLevel=logging.INFO,
logformat='%(asctime)s %(levelname)s [%(name)s %(funcName)s] %(message)s'):
sh = logging.StreamHandler()
sh.setLevel(infoLevel)
formatter = logging.Formatter(logformat)
sh.setFormatter(formatter)
self.logger.addHandler(sh)
Create object the right way, e.g. inspired by Google:
import logging
formatter = logging.Formatter('%(asctime)s %(threadName)s: %(message)s')
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
log.addHandler(handler)
log.debug('debug message')
log.info('info message')
log.warn('warn message')
log.error('error message')
log.critical('critical message')
2022-11-22 23:17:59,342 MainThread: debug message
2022-11-22 23:17:59,342 MainThread: info message
2022-11-22 23:17:59,342 MainThread: warn message
2022-11-22 23:17:59,342 MainThread: error message
2022-11-22 23:17:59,342 MainThread: critical message
As pointed out by #ManuelFedele, logging.basicConfig is not a good solution.
#VinaySajip explained that the setLevel is ignored because the logger is using the internal handler "of last resort", whose level is set to WARNING.
This explanation was also helpful:
The Handler.setLevel() method, just as in logger objects, specifies the lowest severity that will be dispatched to the appropriate destination. Why are there two setLevel() methods? The level set in the logger determines which severity of messages it will pass to its handlers. The level set in each handler determines which messages that handler will send on.
So a good solution is to add a handler to the logger, with the appropriate level:
import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG) # or whatever level should be displayed on the console
logger.addHandler(ch)
Output
>>> logger.debug('debug message')
debug message
>>> logger.info('info message')
info message
>>> logger.warn('warn message')
<stdin>:1: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
warn message
>>> logger.error('error message')
error message
>>> logger.critical('critical message')
critical message
It seems recently there were some changes of python logging package. Some code work previously doesn't work now. And I am confused. My python version is Python 2.7.15.
The first example I don't understand is that below only prints "WARNING:root:hello from warn". If I understand correctly, "logging.info" actually calls the root logger, and root logger default to warn level. So the first "hello from info" is ignored, which is fine. But why the second "hello from info" is also not printed?
import logging
logging.info("hello from info")
logging.warn("hello from warn")
logging.basicConfig(level=logging.INFO)
logging.info("hello from info")
The second question is the logging level of Handler versus logger. If we set the log level for both the handler and logger, which one is working? Or what if we just set level for Handler? Take the example as below. We already set the log level for StreamHandler, but the "hello from info" is not printed to stdout. Only the "hello from warn" (Besides, it is not "WARNING:t:hello from warn"). Why is that?
import logging
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
logger = logging.getLogger('t')
logger.addHandler(ch)
logger.info("hello from info")
logger.warn("hello from warn")
But why the second "hello from info" is also not printed?
Because
logging.info / warn / error / debug call logging.basicConfig under the hood. Example:
def info(msg, *args, **kwargs):
if len(root.handlers) == 0:
basicConfig()
root.info(msg, *args, **kwargs)
logging.basicConfig does not do anything if the root logger is already configured. Quote from the docs:
This function does nothing if the root logger already has handlers configured for it.
So, in your code, the root logger is configured with WARN level when logging.info("hello from info") is executed. The subsequent call of logging.basicConfig(level=logging.INFO) has no effect.
Rule of thumb: configure the loggers (no matter if manually or via logging.basicConfig()) as early as possible in your code.
If we set the log level for both the handler and logger, which one is working?
Both! Logger level and handler level are two different stages of filtering records. The logger level defines what records are actually passed to its handlers, while the handler level defines what records will be handled by the particular handler. Examples:
logger.setLevel(logging.INFO)
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.info('spam')
Since logger has level INFO, it will process spam record and pass it to its handlers. However, handler has level ERROR, so the record will not be processed by handler.
logger.setLevel(logging.WARN)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
logger.info('spam')
Now the handler will process almost any record, including spam record since its level is INFO, thus greater than DEBUG. However, the handler will never receive spam to process because the logger will not process it, thus not passing spam to its handlers.
logger.setLevel(logging.INFO)
h1 = logging.StreamHandler()
h1.setLevel(logging.CRITICAL)
h2 = logging.FileHandler('some.log')
h2.setLevel(logging.DEBUG)
logger.addHandler(h1)
logger.addHandler(h2)
logger.info('spam')
Now the logger has two handlers, h1 printing records to terminal, h2 writing them to file. The logger will pass only records of level INFO or greater to its handlers. However, you will see only records with level CRITICAL in terminal, but all records in log file.
Is there anything in this code that would explain why my info messages aren't going into the log. Properly formatted warnings and above are going into both log files.
Initializing logger:
logger = logging.getLogger()
f = logging.Formatter('%(asctime)s\n%(levelname)s: %(funcName)s %(message)s')
out = logging.handlers.RotatingFileHandler(filename=self.f_stdout, maxBytes=1048576, backupCount=99)
err = logging.handlers.RotatingFileHandler(filename=self.f_stderr, maxBytes=1048576, backupCount=99)
out.setLevel(logging.INFO)
err.setLevel(logging.WARNING)
err.setFormatter(f)
logger.addHandler(out)
logger.addHandler(err)
Usage:
logging.info('this doesnt get logged')
logging.warning('this gets logged to stdout and stderr with respective formatting')
You never actually set the log level of the root logger object itself (the logger variable in your code). Both handlers and loggers have log levels; lines only reach the output if they're above both thresholds.
Since you don't set the root log level, it uses its default (warning). Try adding a call to logger.setLevel(logging.INFO) to change that.