I want logging messages from my program, but not from the libraries it uses. I can disable / change the logging level of individual libraries like this:
logging.getLogger('alibrary').setLevel(logging.ERROR)
The problem is, my program uses lots and lots of libraries, which use lots themselves. So doing this individually for every library is a big chore. Is there a better way to do this?
You could set the root logger's level to e.g. ERROR and then selectively set more verbose levels for your own code:
logging.getLogger().setLevel(logging.ERROR)
then assuming the libraries you use are well-behaved with regard to logging, the effective levels of their loggers should effectively be ERROR just as if you had set each one individually.
Related
I have a project that uses 3rd party packages (which in turn might be using other packages). I want to apply a common xxxHandler to all packages logging. Since all packages define logger = logging.getLogger(__name__), this is the solution I have implemented:
handler = RotatingFileHandler(filename='app.log')
for name in logging.root.manager.loggerDict:
logging.getLogger(name).addHandler(handler)
This works, but it creates multiple entries per log (probably because of logging propagation). So, I enhanced this a bit by providing a list of names that should get the handler:
def add_handler_to_loggers(loggers=None):
if loggers:
for name in loggers:
logging.getLogger(name).addHandler(handler)
add_handler_to_loggers(['requests', 'sleekxmpp']
And I am happy with this. However, I am not sure I will be missing any important log from some other package.
Questions
Is this a good approach for my problem?
Is there a better approach?
Thank you!
Python's logging facility comes with several levels of severity:
CRITICAL
ERROR
WARNING*
INFO
DEBUG
NOTSET
*The root logger defaults to WARNING.
How does one deal with regular messages in this setup?
With "regular messages" I mean those that are not warnings but should always be shown to the user.
An example could be:
Loop over a list of files and apply an accumulating operation to them. A "regular message" could be "currently working on {filename}". The INFO level might hold the status of individual sub-steps of the file treatment that the user may not be interested in knowing most of the time. On the WARNING level could be potential problems like two files containing differing entries for the same key.
I think that I want a level between WARNING and INFO, which might be called NOTICE, NOTE, REGULAR or similar. This level would be my default instead of WARNING. Why does logging not provide such a level?
I understand that I can add such a level fairly easily but the logging manual strongly argues against custom levels. Thus, I assume there must be a canonical way of logging messages like the above ...
EDIT: Just to clarify regarding the vote to close for being "opinion-based": I would like to know how logging is supposed to be used sticking to the given set of levels and conventions. Thus, I would think that the correct answer is not necessarily an opinion but should be the best practice of using this module.
The manual states "Defining your own levels is possible, but should not be necessary, as the existing levels have been chosen on the basis of practical experience". In my example above, I seem to be lacking one level: either "regular messages" or -- if I switch to INFO meaning "regular messages" -- something like "verbose info" for the user.
Log messages are not really indented for the user. What you are describing sounds more like regular output. However there are at least two possible ways to solve this. Either make the regular message INFO level and move the not-as-interesting messages to the DEBUG level, or use a log filter. To quote the documentation:
Filters can be used by Handlers and Loggers for more sophisticated
filtering than is provided by levels.
I am new to python and just trying to learn and find better ways to write code. I want to create a custom class for logging and use package logging inside it. I want the function in this class to be reusable and do my logging from other scripts rather than writing custom code in each and every scripts. Is there a good link you guys can share? Is this the right way to handle logging? I just want to avoid writing the same code in every script if I can reuse it from one module.
I would highly appreciate any reply.
You can build a custom class that utilizes the built in python logging library. There isn't really any right way to handle logging as the library allows you to use 5 standard levels indicating the severity of events (DEBUG, INFO, WARNING, ERROR, and CRITICAL). The way you use these levels are application specific. Here's another good explanation of the package.
It's indeed a good idea to keep all your logging configuration (formatters, level, handlers) in one place.
create a class wrapping a custom logger with your configuration
expose methods for logging with different levels
import this class wherever you want
create an instance of this class to log where you want
To make sure all you custom logging objects have the same config, you should make logging class own the configuration.
I don't think there's any links I can share for the whole thing but you can find links for the individual details I mentioned easily enough.
The python logging library allows to log based on different levels:
https://docs.python.org/3/howto/logging.html#logging-levels
But I would like to use it to log based on custom tags, for example "show_intermediate_results" or "display_waypoints_in_the_code" or "report_time_for_each_module" and so on...
Those tags cannot be measured in a severity ladder, during development i would sometimes want to see them and sometimes not depending on what i am developing/debugging at the moment.
So the question is if I can use the logging library to do that?
Btw, i DO want to use the library and not write something by myself because i want it to be thread safe.
As per the documentation, you can use logging.Filter objects with Logger and Handler instances
for more sophisticated filtering than is provided by levels.
I am currently developing a Python application which I continually performance test, simply by recording the runtime of various parts.
A lot of the code is related only to the testing environment and would not exist in the real world application, I have these separated into functions and at the moment I comment out these calls when testing. This requires me to remember which calls refer to test only components (they are quite interleaved so I cannot group the functionality).
I was wondering if there was a better solution to this, the only idea I have had so far is creation of a 'mode' boolean and insertion of If statements, though this feels needlessly messy. I was hoping there might be some more standardised testing method that I am naive of.
I am new to python so I may have overlooked some simple solutions.
Thank you in advance
There are libraries for testing like those in the development-section of the standard library. If you did not use such tools yet, you should start to do so - they help a lot with testing. (especially unittest).
Normally Python runs programs in debug mode with __debug__ set to True (see docs on assert) - you can switch off debug mode by setting the command-line switches -O or -OO for optimization (see docs).
There is something about using specifically assertions in the Python Wiki
I'd say if you're commenting out several parts of your code when switching between debug&release mode I think you're doing wrong. Take a look for example to the logging library, as you can see, with that library you can specify the logging level you want to use only by changing a single parameter.
Try to avoid commenting specific parts of your debug code by having one or more variables which controls the mode (debug, release, ...) your script will run. You could also use some builtin ones python already provides