I am testing the python logger with the jupyter notebook.
When I run the following example code in a freshly started kernel, it works and create the log file with the right content.
import logging
logging.basicConfig(filename='/home/depot/wintergreen/example.log',level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
However if I try to rerun the same code with say, for instance, the filename changed from example.logto example.log2, nothing happens, the file example.log2 is not created.
I ended up devising that test as it seemed to me that when trying to run the logging, it would only function the very first time I am running it. What am I doin wrong here?
You are right, .basicConfig() uses your kwargs only once. Because after first time you got handlers logging.root.handlers, one handler actually, so if you look in source code
def basicConfig(**kwargs):
...
_acquireLock()
try:
if len(root.handlers) == 0:
...
finally:
_releaseLock()
So since your len(root.handlers) != 0 actual assignment of the provided arguments is not happening.
HOW TO CHANGE WITHOUT RESTARTING:
The only solution i came up with is for changing basic Config with calling .basicConfig() without restarting kernel is to:
for handler in logging.root.handlers:
logging.root.removeHandler(handler)
Which will remove all handlers from root logger and after that you are good to set anything you want.
The issue is that basicConfig() function is designed to only be run once.
Per the docs: The first time it runs, it "creates a StreamHandler with a default Formatter and adding it to the root logger". However on the second time, the "function does nothing if the root logger already has handlers configured for it".
One possible solution is clear the previous handler with using logging.root.removeHandler. Alternatively, you can directly access stream attribute for open stream used by the StreamHandler instance:
>>> import logging
>>> logging.basicConfig(filename='abc.txt') # 1st call to basicConfig
>>> h = logging.root.handlers[0] # get the handler
>>> h.stream.close() # close the current stream
>>> h.stream = open('def.txt', 'a') # set-up a new stream
FWIW, basicConfig() was a late addition to the logging module and was intended as a simplified short-cut API for common cases. In general, whenever you have problems with basicConfig(), it means that it is time to use the full API which is a little less convenient but gives you more control:
import logging
# First pass
h = logging.StreamHandler(open('abc.txt', 'a'))
h.setLevel(logging.DEBUG)
h.setFormatter(logging.Formatter('%(asctime)s | %(message)s'))
logging.root.addHandler(h)
logging.critical('The GPU is melting')
# Later passes
logging.root.removeHandler(h)
h = logging.StreamHandler(open('def.txt', 'a'))
h.setLevel(logging.DEBUG)
h.setFormatter(logging.Formatter('%(asctime)s | %(message)s'))
logging.root.addHandler(h)
logging.critical('The CPU is getting hot too')
Related
Apologies if this question is similar to others posted on SO, but I have tried many of the answers given and could not achieve what I am attempting to do.
I have some code that calls an external module:
import trafilatura
# after obtaining article_html
text = trafilatura.extract(article_html, language=en)
This sometimes prints out a warning on the console, which comes from the following code in the trafilatura module:
# at the top of the file
LOGGER = logging.getLogger(__name__)
# in the method that I'm calling
LOGGER.warning('HTML lang detection failed')
I'd like to not print this and other messages produced by the module directly to the console, but to store them somewhere such that I can edit the messages and decide what to do with them. (Specifically, I want to save the messages in slightly modified form but only given certain circumstances.) I am not using the logging library in my own code.
I have tried the following solution suggestions:
buf = io.StringIO()
with contextlib.redirect_stderr(buf): # I also tried redirect_stdout
text = trafilatura.extract(article_html, language=en)
and
buf = io.StringIO()
sysout = sys.stdout
syserr = sys.stderr
sys.stdout = sys.stderr = buf
text = trafilatura.extract(article_html, language=en)
sys.stdout = sysout
sys.stderr = syserr
However, in both cases buf remains empty and trafilatura still prints its logging messages to the console. Testing the redirects above with other calls (e.g. print("test")) they seem to catch those just fine, so apparently LOGGER.warning() from trafilatura is just not printing to stderr or stdout?
I thought I could set a different output stream target for trafilatura's LOGGER, but it is using a NullHandler so I could neither figure out its stream target nor do I know how to change it:
# from trafilatura's top-level __init__.py
logging.getLogger(__name__).addHandler(NullHandler())
Any ideas? Thanks in advance.
The idea here is to work within the standard logging lib of python. Adding a NullHandler is actually standard recommended practice for libraries that add a logger because it prevents falling back to stderr if no logging configuration is present.
What is likely happening here is that those logs are propagating to the root logger which got some handler attached somewhere else. You can stop that by getting the logger of the module in your code and setting it to not propagate:
# assuming that "trafilatura" is the __name__ of the module:
logger = logging.getLogger("trafilatura")
logger.propagate = False
I've seen some very odd behaviour with the logger module. It started with a relatively complex project, but now I've seen it with the following script:
import logging
import os
# Uncomment the following line to remove handlers
# logging.getLogger().handlers = []
filePath = os.environ['userprofile'] + r'\Documents\log.txt'
logging.basicConfig(filename=filePath)
logging.debug('Gleep')
logging.shutdown()
This should simply write 'Gleep' to the log.txt file to your documents. Currently it is writing the file but not writing anything to it, however, I've inconsistently seen the following behaviour:
List item
No log file being written at all.
Log file created, but nothing written to it.
Everything working fine.
The only way I've got it working before is to remove existing handlers (commented out in the example above).
This is on several machines in different locations.
So...am I doing something grotesquely wrong here? Why is the logging module acting this way?
I'm not sure how to prove/disprove/debug your 'other' situations, but maybe the following can help clarify what is happening in the code from your question:
First, setting logging.getLogger().handlers = [] should not be necessary, since logging.getLogger() is the root logger by default and has no handlers. Here is a fresh Python 3.7 shell:
>>> import logging
>>> logging.getLogger()
<RootLogger root (WARNING)>
>>> logging.getLogger().handlers
[]
(Note that in the absence of any handlers, a logger will fall back to lastResort, but that should be irrelevant here since you add a handler implicitly via basicConfig().)
Which brings you to logging.basicConfig(filename=filePath): this adds a FileHandler handler to the root logger. It does not touch the level of the root logger, which is WARNING by default, so your message won't pass the 'level test' and won't be emitted as a result.
>>> logging.root.getEffectiveLevel()
30
(This uses .getEffectiveLevel() rather than just the plain attribute because a logger will walk its hierarchy until it finds a level if its level is NOTSET.)
All that is to say: as you currently have it, you are logging from the root logger (level WARNING) a message object that has level DEBUG, so the message will go nowhere.
Ok so in my environment.py file I am able to log stuff by:
logging.basicConfig(level=logging.DEBUG, filename="example.log")
def before_feature(context, feature):
logging.info("test logging")
but when I am inside the steps file I cannot perform logging:
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
#given("we have a step")
def step_impl(context):
logger.debug("Test logging 2")
The logging message inside the step does not show up. I am using the python behave module. Any ideas?
I have tried enabling and disabling logcapture when I run behave but it makes no difference.
By default, behave tends to capture logs during feature execution, and only display them in cases of failure.
To disable this, you can set
log_capture=false
in behave.ini
Or, you can use the --no-logcapture command line option
Further Reading : Behave API Reference, Behave LogCapture
what worked for me:
behave --no-capture --no-capture-stderr --no-logcapture
and add in the environment.py the follwing snipest:
def after_step(context, step):
print("")
Why: I disovered that behave does not log the last print statement of a step. So I just added a empty print after each step with the previous snipest.
Hope it helped
Importing logging from environment.py in steps.py solved problem for me.
from features.environment import logging
I am not sure but I guess the problem is that every time you import logging it rewrites your previous configs because disable_existing_loggers is True by default. (Here is the documentation paragraph explaining this)
I tried to build a logger that also loggs the full output from the python interpreter, what you normally see on the console.
This is my suggestion, it did not work.
logger = logging.getLogger()
stdoutStreamHandler = logging.StreamHandler(sys.stdout)
stderrStreamHandler = logging.StreamHandler(sys.stderr)
fileHandler = logging.FileHandler(logpath)
logger.addHandler(stdoutStreamHandler)
logger.addHandler(stderrStreamHandler)
Unsure what you mean by logging the full output from the Python Interpreter, do you mean the command-line interpreter or the background interpreter at run-time?
Either way this may help: trace
This will allow you to execute any code with the trace, and will also allow you to write it out to a file:
import sys
import trace
# create a Trace object, telling it what to ignore, and whether to
# do tracing or line-counting or both.
tracer = trace.Trace(
ignoredirs=[sys.prefix, sys.exec_prefix],
trace=0,
count=1)
# run the new command using the given tracer
tracer.run('main()')
# make a report, placing output in the current directory
r = tracer.results()
r.write_results(show_missing=True, coverdir=".")
I have not yet used it myself, but it could be possible to execute
logger.debug(r)
Which would create a record in your log file of the debug log level, you could, of course, change that to info, warning, error or critical.
It appears that if you invoke logging.info() BEFORE you run logging.basicConfig, the logging.basicConfig call doesn't have any effect. In fact, no logging occurs.
Where is this behavior documented? I don't really understand.
You can remove the default handlers and reconfigure logging like this:
# if someone tried to log something before basicConfig is called, Python creates a default handler that
# goes to the console and will ignore further basicConfig calls. Remove the handler if there is one.
root = logging.getLogger()
if root.handlers:
for handler in root.handlers:
root.removeHandler(handler)
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
Yes.
You've asked to log something. Logging must, therefore, fabricate a default configuration. Once logging is configured... well... it's configured.
"With the logger object configured,
the following methods create log
messages:"
Further, you can read about creating handlers to prevent spurious logging. But that's more a hack for bad implementation than a useful technique.
There's a trick to this.
No module can do anything except logging.getlogger() requests at a global level.
Only the if __name__ == "__main__": can do a logging configuration.
If you do logging at a global level in a module, then you may force logging to fabricate it's default configuration.
Don't do logging.info globally in any module. If you absolutely think that you must have logging.info at a global level in a module, then you have to configure logging before doing imports. This leads to unpleasant-looking scripts.
This answer from Carlos A. Ibarra is in principle right, however that implementation might break since you are iterating over a list that might be changed by calling removeHandler(). This is unsafe.
Two alternatives are:
while len(logging.root.handlers) > 0:
logging.root.removeHandler(logging.root.handlers[-1])
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
or:
logging.root.handlers = []
logging.basicConfig(format='%(asctime)s %(message)s',level=logging.DEBUG)
where the first of these two using the loop is the safest (since any destruction code for the handler can be called explicitly inside the logging framework). Still, this is a hack, since we rely on logging.root.handlers to be a list.
Here's the one piece of the puzzle that the above answers didn't mention... and then it will all make sense: the "root" logger -- which is used if you call, say, logging.info() before logging.basicConfig(level=logging.DEBUG) -- has a default logging level of WARNING.
That's why logging.info() and logging.debug() don't do anything: because you've configured them not to, by... um... not configuring them.
Possibly related (this one bit me): when NOT calling basicConfig, I didn't seem to be getting my debug messages, even though I set my handlers to DEBUG level. After a bit of hair-pulling, I found you have to set the level of the custom logger to be DEBUG as well. If your logger is set to WARNING, then setting a handler to DEBUG (by itself) won't get you any output on logger.info() and logger.debug().
Ran into this same issue today and, as an alternative to the answers above, here's my solution.
import logging
import sys
logging.debug('foo') # IRL, this call is from an imported module
if __name__ == '__main__':
logging.basicConfig(level=logging.INFO, force=True)
logging.info('bar') # without force=True, this is not printed to the console
Here's what the docs say about the force argument.
If this keyword argument is specified as true, any existing handlers
attached to the root logger are removed and closed, before carrying
out the configuration as specified by the other arguments.
A cleaner version of the answer given by #paul-kremer is:
while len(logging.root.handlers):
logging.root.removeHandler(logging.root.handlers[-1])
Note: it is generally safe to assume logging.root.handlers will always be a list (see: https://github.com/python/cpython/blob/cebe9ee988837b292f2c571e194ed11e7cd4abbb/Lib/logging/init.py#L1253)
Here is what I did.
I wanted to log to a file which has a name configured in a config-file and also get the debug-logs of the config-parsing.
TL;DR; This logs into a buffer until everything to configure the logger is available
# Log everything into a MemoryHandler until the real logger is ready.
# The MemoryHandler never flushes (flushLevel 100 is above CRITICAL) automatically but only on close.
# If the configuration was loaded successfully, the real logger is configured and set as target of the MemoryHandler
# before it gets flushed by closing.
# This means, that if the log gets to stdout, it is unfiltered by level
root_logger = logging.getLogger()
root_logger.setLevel(logging.NOTSET)
stdout_logging_handler = logging.StreamHandler(sys.stderr)
tmp_logging_handler = logging.handlers.MemoryHandler(1024 * 1024, 100, stdout_logging_handler)
root_logger.addHandler(tmp_logging_handler)
config: ApplicationConfig = ApplicationConfig.from_filename('config.ini')
# because the records are already logged, unwanted ones need to be removed
filtered_buffer = filter(lambda record: record.levelno >= config.main_config.log_level, tmp_logging_handler.buffer)
tmp_logging_handler.buffer = filtered_buffer
root_logger.removeHandler(tmp_logging_handler)
logging.basicConfig(filename=config.main_config.log_filename, level=config.main_config.log_level, filemode='wt')
logging_handler = root_logger.handlers[0]
tmp_logging_handler.setTarget(logging_handler)
tmp_logging_handler.close()
stdout_logging_handler.close()