Change level logged to IPython/Jupyter notebook - python

I have a package that relies on several different modules, each of which sets up its own logger. That allows me to log where each log message originates from, which is useful.
However, when using this code in an IPython/Jupyter notebook, I was having trouble controlling what got printed to the screen. Specifically, I was getting a lot of DEBUG-level messages that I didn't want to see.
How do I change the level of logs that get printed to the notebook?
More info:
I've tried to set up a root logger in the notebook as follows:
# In notebook
import logging
logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Import the module
import mymodule
And then at the top of my modules, I have
# In mymodule.py
import logging
logger = logging.getLogger('mypackage.' + __name__)
logger.setLevel(logging.DEBUG)
logger.propagate = True
# Log some messages
logger.debug('debug')
logger.info('info')
When the module code is called in the notebook, I would expect the logs to get propagated up, and then for the top logger to only print the info log statement. But both the debug and the info log statement get shown.
Related links:
From this IPython issue, it seems that there are two different levels of logging that one needs to be aware of. One, which is set in the ipython_notebook_config file, only affects the internal IPython logging level. The other is the IPython logger, accessed with get_ipython().parent.log.
https://github.com/ipython/ipython/issues/8282
https://github.com/ipython/ipython/issues/6746

With current ipython/Jupyter versions (e.g. 6.2.1), the logging.getLogger().handlers list is empty after startup and logging.getLogger().setLevel(logging.DEBUG) has no effect, i.e. no info/debug messages are printed.
Inside ipython, you have to change an ipython configuration setting (and possibly work around ipython bugs), as well. For example, to lower the logging threshold to debug messages:
# workaround via specifying an invalid value first
%config Application.log_level='WORKAROUND'
# => fails, necessary on Fedora 27, ipython3 6.2.1
%config Application.log_level='DEBUG'
import logging
logging.getLogger().setLevel(logging.DEBUG)
log = logging.getLogger()
log.debug('Test debug')
For just getting the debug messages of one module (cf. the __name__ value in that module) you can replace the above setLevel() call with a more specific one:
logging.getLogger('some.module').setLevel(logging.DEBUG)

The root cause of this issue (from https://github.com/ipython/ipython/issues/8282) is that the Notebook creates a root logger by default (which is different from IPython default behavior!). The solution is to get at the handler of the notebook logger, and set its level:
# At the beginning of the notebook
import logging
logger = logging.getLogger()
assert len(logger.handlers) == 1
handler = logger.handlers[0]
handler.setLevel(logging.INFO)
With this, I don't need to set logger.propagate = True in the modules and it works.

Adding another solution because the solution was easier for me. On startup of the Ipython kernel:
import logging
logging.basicConfig(level=20)
Then this works:
logging.getLogger().info("hello")
>> INFO:root:hello
logging.info("hello")
>> INFO:root:hello
And if I have similar logging code in a function that I import and run, the message will display as well.

Related

Logging level not applied

it seems that in python setting the logging level for loggers (including the root logger) is not applied, until you use one of the logging module's logging functions. Here's some code to show what I mean (I used python 3.7):
import logging
if __name__ == "__main__":
# Create a test logger and set its logging level to DEBUG
test_logger = logging.getLogger("test")
test_logger.setLevel(logging.DEBUG)
# Log some debug messages
# THESE ARE NOT PRINTED
test_logger.debug("test debug 1")
# Now use the logging module directly to log something
logging.debug("whatever, you won't see this anyway")
# Apparently the line above "fixed" the logging for the custom logger
# and you should be able to see the message below
test_logger.debug("test debug 2")
Output:
DEBUG:test:test debug 2
Maybe there's something I misunderstood about the configuration of the loggers, in that case I'd appreciate to know the correct way of doing it.
You didn't (explicitly) call logging.basicConfig, so the handler isn't configured correctly.
test_logger initially has no handler, because you didn't add one and the root logger doesn't have one yet. So although the message is "logged", nothing defines what that actually means.
When you call logging.debug, logging.basicConfig is called for you, because the root logger has no handler. At this time, a StreamHandler is created, but the root logger remains at the default level of INFO, and so nothing is sent to the new handler.
Now when you call test_logger.debug again, it has the inherited StreamHandler to actually output the long message to standard error.

Python Logging Module Inconsistent Behaviour

I've seen some very odd behaviour with the logger module. It started with a relatively complex project, but now I've seen it with the following script:
import logging
import os
# Uncomment the following line to remove handlers
# logging.getLogger().handlers = []
filePath = os.environ['userprofile'] + r'\Documents\log.txt'
logging.basicConfig(filename=filePath)
logging.debug('Gleep')
logging.shutdown()
This should simply write 'Gleep' to the log.txt file to your documents. Currently it is writing the file but not writing anything to it, however, I've inconsistently seen the following behaviour:
List item
No log file being written at all.
Log file created, but nothing written to it.
Everything working fine.
The only way I've got it working before is to remove existing handlers (commented out in the example above).
This is on several machines in different locations.
So...am I doing something grotesquely wrong here? Why is the logging module acting this way?
I'm not sure how to prove/disprove/debug your 'other' situations, but maybe the following can help clarify what is happening in the code from your question:
First, setting logging.getLogger().handlers = [] should not be necessary, since logging.getLogger() is the root logger by default and has no handlers. Here is a fresh Python 3.7 shell:
>>> import logging
>>> logging.getLogger()
<RootLogger root (WARNING)>
>>> logging.getLogger().handlers
[]
(Note that in the absence of any handlers, a logger will fall back to lastResort, but that should be irrelevant here since you add a handler implicitly via basicConfig().)
Which brings you to logging.basicConfig(filename=filePath): this adds a FileHandler handler to the root logger. It does not touch the level of the root logger, which is WARNING by default, so your message won't pass the 'level test' and won't be emitted as a result.
>>> logging.root.getEffectiveLevel()
30
(This uses .getEffectiveLevel() rather than just the plain attribute because a logger will walk its hierarchy until it finds a level if its level is NOTSET.)
All that is to say: as you currently have it, you are logging from the root logger (level WARNING) a message object that has level DEBUG, so the message will go nowhere.

Python logging to console window randomly on different machines

I have a Python script that utilizes the logging module to log messages to a file. And in most cases it works just fine, but on some of my co-worker's machines it is logging every message from every imported module to the console window. There doesn't seem to be any pattern as to why this is working on some machines and not on others.
Here is how I set up the logger
logger = logging.getLogger(__name__)
if __name__ == "__main__":
args = parser.parse_args()
logging.basicConfig(handlers=[logging.NullHandler()]) #do this so that imported modules don't screw with logging
logger.setLevel(logging.DEBUG)
handler = logging.FileHandler(args.log)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter('[%(asctime)s] %(levelname)s: %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.info(str(args))
main(args)
So what's happening is first I set up my logger as a global variable, then I use basicConfig to set the root handler as the NullHandler(), this is so that any imported modules won't log anything to my log file or my console window, and finally I set my logger to log to a log file provided in the program arguments.
This works exactly how I want to on my machine, the log file is only written to when I use my logger object. No log messages are passed to my console window and no log messages from imported modules go to my log file.
However, on a few random machines, every single call to logging or logger goes to the console window. The logs in the log file are ok, they only show the logs I expect them to (the ones called with logger).
For instance, when I do logger.info(str(args)),
INFO: Namespace(arg1='Foo', arg2='Bar', arg3='foobar')
is outputted to the console window.
I can't think of any reason that this would behave differently on different machines. It is the exact same script, the same version of python, and the same version of logging. Any ideas?

Python logging levels are behaving inconsistently

I can't understand why the following code does not produce my debug message even though effective level is appropriate (output is just 10)
import logging
l = logging.getLogger()
l.setLevel(logging.DEBUG)
l.debug("Debug Mess!")
l.error(l.getEffectiveLevel())
while when I add this line after the import: logging.debug("Start...")
import logging
logging.debug("Start...")
l = logging.getLogger()
l.setLevel(logging.DEBUG)
l.debug("Debug Mess!")
l.error(l.getEffectiveLevel())
it produces following output:
DEBUG:root:Debug Mess!
ERROR:root:10
so even though "Start..." is not shown, it starts to log. Why?
It's on Python 3.5. Thanks
The top-level logging.debug(..) call calls the logging.basicConfig() function for you if no handlers have been configured yet on the root logger.
Because using a call to logging.getLogger().debug() does not trigger that call, you don't see any output because there are no handlers to show the output on.
The Python 3 version of logger does have a logging.lastResort handler, used for when no logging configuration exists, but this handler is configured to only show messages of level WARNING and up, which is why you see your ERROR level message (10) printed to STDERR, but not your DEBUG level message. In Python 2 you would get the message No handlers could be found for logger "root" printed instead, just once for the first attempt to log anything. I'd not rely on the lastResort handler however; instead properly configure your logging hierarchy with a decent handler configured for your own needs.
Either call logging.basicConfig() yourself, or manually add a handler on the root logger:
l = logging.getLogger()
l.addHandler(logging.StreamHandler())
The above basically does the same thing as a logging.basicConfig() call with no further arguments. The StreamHandler() created this way logs to STDERR and does not further filter on the message level. Note that a logging.basicConfig() call can also set the logging level for you.
The root logger(default logger, top level) and all other logger's default log level is warning(order 3) in all 5 log levels: debug < info < warning < error < fatal in order.
So at your first logging.debug('starting...'), you haven't set the root log level to debug as following code and you can't get the starting...output.
import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug('starting...')
see python logging HOW TO for detail

Python logging before you run logging.basicConfig?

It appears that if you invoke logging.info() BEFORE you run logging.basicConfig, the logging.basicConfig call doesn't have any effect. In fact, no logging occurs.
Where is this behavior documented? I don't really understand.
You can remove the default handlers and reconfigure logging like this:
# if someone tried to log something before basicConfig is called, Python creates a default handler that
# goes to the console and will ignore further basicConfig calls. Remove the handler if there is one.
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
Yes.
You've asked to log something. Logging must, therefore, fabricate a default configuration. Once logging is configured... well... it's configured.
"With the logger object configured,
the following methods create log
messages:"
Further, you can read about creating handlers to prevent spurious logging. But that's more a hack for bad implementation than a useful technique.
There's a trick to this.
No module can do anything except logging.getlogger() requests at a global level.
Only the if __name__ == "__main__": can do a logging configuration.
If you do logging at a global level in a module, then you may force logging to fabricate it's default configuration.
Don't do logging.info globally in any module. If you absolutely think that you must have logging.info at a global level in a module, then you have to configure logging before doing imports. This leads to unpleasant-looking scripts.
This answer from Carlos A. Ibarra is in principle right, however that implementation might break since you are iterating over a list that might be changed by calling removeHandler(). This is unsafe.
Two alternatives are:
while len(logging.root.handlers) > 0:
logging.root.removeHandler(logging.root.handlers[-1])
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
or:
logging.root.handlers = []
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
where the first of these two using the loop is the safest (since any destruction code for the handler can be called explicitly inside the logging framework). Still, this is a hack, since we rely on logging.root.handlers to be a list.
Here's the one piece of the puzzle that the above answers didn't mention... and then it will all make sense: the "root" logger -- which is used if you call, say, logging.info() before logging.basicConfig(level=logging.DEBUG) -- has a default logging level of WARNING.
That's why logging.info() and logging.debug() don't do anything: because you've configured them not to, by... um... not configuring them.
Possibly related (this one bit me): when NOT calling basicConfig, I didn't seem to be getting my debug messages, even though I set my handlers to DEBUG level. After a bit of hair-pulling, I found you have to set the level of the custom logger to be DEBUG as well. If your logger is set to WARNING, then setting a handler to DEBUG (by itself) won't get you any output on logger.info() and logger.debug().
Ran into this same issue today and, as an alternative to the answers above, here's my solution.
import logging
import sys
logging.debug('foo') # IRL, this call is from an imported module
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO, force=True)
logging.info('bar') # without force=True, this is not printed to the console
Here's what the docs say about the force argument.
If this keyword argument is specified as true, any existing handlers
attached to the root logger are removed and closed, before carrying
out the configuration as specified by the other arguments.
A cleaner version of the answer given by #paul-kremer is:
while len(logging.root.handlers):
logging.root.removeHandler(logging.root.handlers[-1])
Note: it is generally safe to assume logging.root.handlers will always be a list (see: https://github.com/python/cpython/blob/cebe9ee988837b292f2c571e194ed11e7cd4abbb/Lib/logging/init.py#L1253)
Here is what I did.
I wanted to log to a file which has a name configured in a config-file and also get the debug-logs of the config-parsing.
TL;DR; This logs into a buffer until everything to configure the logger is available
# Log everything into a MemoryHandler until the real logger is ready.
# The MemoryHandler never flushes (flushLevel 100 is above CRITICAL) automatically but only on close.
# If the configuration was loaded successfully, the real logger is configured and set as target of the MemoryHandler
# before it gets flushed by closing.
# This means, that if the log gets to stdout, it is unfiltered by level
root_logger = logging.getLogger()
root_logger.setLevel(logging.NOTSET)
stdout_logging_handler = logging.StreamHandler(sys.stderr)
tmp_logging_handler = logging.handlers.MemoryHandler(1024 * 1024, 100, stdout_logging_handler)
root_logger.addHandler(tmp_logging_handler)
config: ApplicationConfig = ApplicationConfig.from_filename('config.ini')
# because the records are already logged, unwanted ones need to be removed
filtered_buffer = filter(lambda record: record.levelno >= config.main_config.log_level, tmp_logging_handler.buffer)
tmp_logging_handler.buffer = filtered_buffer
root_logger.removeHandler(tmp_logging_handler)
logging.basicConfig(filename=config.main_config.log_filename, level=config.main_config.log_level, filemode='wt')
logging_handler = root_logger.handlers[0]
tmp_logging_handler.setTarget(logging_handler)
tmp_logging_handler.close()
stdout_logging_handler.close()

Categories