Python's logging facility comes with several levels of severity:
CRITICAL
ERROR
WARNING*
INFO
DEBUG
NOTSET
*The root logger defaults to WARNING.
How does one deal with regular messages in this setup?
With "regular messages" I mean those that are not warnings but should always be shown to the user.
An example could be:
Loop over a list of files and apply an accumulating operation to them. A "regular message" could be "currently working on {filename}". The INFO level might hold the status of individual sub-steps of the file treatment that the user may not be interested in knowing most of the time. On the WARNING level could be potential problems like two files containing differing entries for the same key.
I think that I want a level between WARNING and INFO, which might be called NOTICE, NOTE, REGULAR or similar. This level would be my default instead of WARNING. Why does logging not provide such a level?
I understand that I can add such a level fairly easily but the logging manual strongly argues against custom levels. Thus, I assume there must be a canonical way of logging messages like the above ...
EDIT: Just to clarify regarding the vote to close for being "opinion-based": I would like to know how logging is supposed to be used sticking to the given set of levels and conventions. Thus, I would think that the correct answer is not necessarily an opinion but should be the best practice of using this module.
The manual states "Defining your own levels is possible, but should not be necessary, as the existing levels have been chosen on the basis of practical experience". In my example above, I seem to be lacking one level: either "regular messages" or -- if I switch to INFO meaning "regular messages" -- something like "verbose info" for the user.
Log messages are not really indented for the user. What you are describing sounds more like regular output. However there are at least two possible ways to solve this. Either make the regular message INFO level and move the not-as-interesting messages to the DEBUG level, or use a log filter. To quote the documentation:
Filters can be used by Handlers and Loggers for more sophisticated
filtering than is provided by levels.
Related
The python logging library allows to log based on different levels:
https://docs.python.org/3/howto/logging.html#logging-levels
But I would like to use it to log based on custom tags, for example "show_intermediate_results" or "display_waypoints_in_the_code" or "report_time_for_each_module" and so on...
Those tags cannot be measured in a severity ladder, during development i would sometimes want to see them and sometimes not depending on what i am developing/debugging at the moment.
So the question is if I can use the logging library to do that?
Btw, i DO want to use the library and not write something by myself because i want it to be thread safe.
As per the documentation, you can use logging.Filter objects with Logger and Handler instances
for more sophisticated filtering than is provided by levels.
One of my logger.debug() statements takes input that's fairly labour-intensive.
I know that I should do logger.debug("The spam is %s", spam_temperature) rather than logger.debug("The spam is {}".format(spam_temperature)). The problem is that the operations I need to perform to actually find out the spam_temperature are quite CPU-intensive, and I have no use for them if the logging level is, say, INFO.
What is the best practice in a case like this?
Found the answer myself - I added
python
if not logging.getLogger().isEnabledFor(logging.DEBUG):
return
before the section I wanted to avoid.
(I got the inspiration from https://stackoverflow.com/a/27849836/3061818)
I want logging messages from my program, but not from the libraries it uses. I can disable / change the logging level of individual libraries like this:
logging.getLogger('alibrary').setLevel(logging.ERROR)
The problem is, my program uses lots and lots of libraries, which use lots themselves. So doing this individually for every library is a big chore. Is there a better way to do this?
You could set the root logger's level to e.g. ERROR and then selectively set more verbose levels for your own code:
logging.getLogger().setLevel(logging.ERROR)
then assuming the libraries you use are well-behaved with regard to logging, the effective levels of their loggers should effectively be ERROR just as if you had set each one individually.
Is there some best practice for interpolating values into a log message? My google-fu is not effective here, but I get examples like this which say:
#Change all logging from this
logger.warning("This is a %s" % var)
#to
logger.warning("This is a %s", var)
like its a best practice everybody must follow. Is there a reason? Will any values of var (like an object of a class) fail here?
Use the latter. The logging module will only interpolate the value when the message is processed.
This mean you save CPU cycles for debug messages, for example, if the log level excludes them.
This is documented in the logging HOWTO, "Optimization" chapter:
Formatting of message arguments is deferred until it cannot be avoided.
You may want to follow the rest of the advice given in that chapter to avoid expensive calculations just for logging, as well.
Internally, the logging module uses the first form, so any value that cannot be converted to a string will fail.
I think second form is better, because logger has a chance to check whether warning level is enabled before doing substitution.
long-time lurker here, finally emerging from the woodwork.
Essentially, what I'm trying to do is have my logger write data like this to the logfile:
Connecting to database . . . Done.
I'd like the 'Connecting to database . . . ' to be written when the function is called, and the 'Done' written after the function has successfully executed.
I'm using Python 2.6 and the logging module. Also, I'd really like to avoid using decorators for this. Any help would be most appreciated!
Writing to a log is, and must be, an atomic action -- this is crucial, and a key feature of any logging package (including the one in Python's standard library) that distinguishes logging from the simple appending of information to files (where bits of things being written by different processes and threads might well "interleave" -- one of them writing some part of a line but not the line-end, just as you desire, and then maybe another one interposing something right afterwards, before the first task writes what it thinks will be the last part of the line but actually ends up on another line... utter confusion often results;-).
It's not inevitable that the atomic unit be "a line" (logs can be recorded elsewhere than to a text file, of course, and some of the things that are acceptable "sinks" for logs won't even have the concept of "a line"!), but, for logging, atomic units there must be. If you want something entire non-atomic then you don't really want logging but simple appends to a file or other stream (and, watch out for the likely confusion mentioned in the first paragraph;-).
For transient updates about what your task is doing (in the middle of X, X done, starting Y, etc), you could think of a specialized log-handler that (for example) interprets such streams of updates by taking the first word as a subtask-identifier (incrementally building up and displaying somewhere the composite message about the "current subtask", recognizing when the subtask identifier changes that the previous subtask is finished or taking an explicit "subtask finished" message, and only writing persistent log entries on subtask-finished events).
It's a pretty specialized requirement so you're not likely to find a pre-made tool for this, but rather you'll have to roll your own. To help you with that, it's crucial to understand exactly what you're trying to accomplish (why would you want non-atomic logging entries, if such a concept even made any sense -- what deployment or system administration task are you trying to ameliorate by using such a hypothetical tool?) so that the specialized subsystem can be tailored to your actual needs. So, can you please expand on this?
I don't believe Python's logger supports that.
However, would it not be better to aggree on a Log format so that the log file can be easily parsed when you want analyse the data where ; is any deliminator you want:
DateTime;LogType;string
That could be parsed easiily by a script and then you could do analysis on the logs
If you use:
Connecting to database . . . Done.
Then you won't be able to analyse how long the transaction took
I don't think you should go down this route. A logging methodolgy with entry:
Time;functionName()->
And exit logging is more useful for troubleshooting:
Time;functionName()<-