We Keep Coding

Python logging stdout and stderr based on level

Using Python 3 logging, how can I specify that logging.debug() and logging.info() should go to stdout and logging.warning() and logging.error() go to stderr?

You can create separate loggers for each stream like this:
import logging
import sys
logging.basicConfig(format="%(levelname)s %(message)s")
stdout_logger = logging.Logger(name="stdout_logger", level=logging.DEBUG)
stderr_logger = logging.Logger(name="stderr_logger", level=logging.DEBUG)
stdout_handler = logging.StreamHandler(stream=sys.stdout)
stderr_handler = logging.StreamHandler(stream=sys.stderr)
stdout_logger.addHandler(hdlr=stdout_handler)
stderr_logger.addHandler(hdlr=stderr_handler)
stdout_logger.info("this will output to stdout")
stderr_logger.info("this will output to stderr")
Then if you want to log something on 'debug' or 'info' level you can just use stdout_logger. For 'warning' and 'error' level messages use stderr_logger.

How to do this kind of thing is documented in the logging cookbook, and though the example levels/destinations in the cookbook recipe are slightly different from those in your question, there should be enough information there for you to arrive at a solution. The key thing is using a filter function and attaching it to a handler.

Making logging module write all logs to only stderr

I am using pytest to test a CLI that produces some output. While running the test, I want to set my CLI's log level to DEBUG. However, I don't want CLI logs to interfere with tests that are parsing the output of the CLI.
How can I make logging module send all the logs to only stderr? I looked at this post but it talks about sending logs to stderr in addition to stdout.

You can direct the log output for a given logger to stderr as follows. This defaults to stderr for output, but you can use sys.stdout instead if you prefer.
import logging
import sys
DEFAULT_LOGGER_NAME = 'default_logger'
def init_logging(logger_name=DEFAULT_LOGGER_NAME,
log_level=logging.DEBUG,
stream=None):
# logging
logger = logging.getLogger(logger_name)
logger.setLevel(log_level)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# create console handler and set level to debug
if stream is None:
stream = sys.stderr
ch = logging.StreamHandler(stream=stream)
ch.setLevel(log_level)
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
return logger
This function needs to be called at the beginning of your program (e.g. beginning of main()).
Then within the code, you just need to call the following:
logger = logging.getLogger(LOGGER_NAME)

Do the same that the linked answer does but replace stdout with stderr. So you would create a handler with logging.StreamHandler(sys.stderr) and make sure this is the only active handler if you want to exclusively have logs go to stderr.
As #Tomerikoo correctly points out you don't need to do anything though as logging defaults to using a StreamHandler with stderr. The only real value of the code below is that it sets a different level than the default. Just logging.warning('log') with no other setup will send a log to stderr.
Addendum: you can also achieve this using basicConfig to have less boilerplate code.
import logging
import sys
logging.basicConfig(stream=sys.stderr, level=logging.INFO)
logging.info('test') # sends log to stderr

Python2.7 log level of logging doesn't work

It seems recently there were some changes of python logging package. Some code work previously doesn't work now. And I am confused. My python version is Python 2.7.15.
The first example I don't understand is that below only prints "WARNING:root:hello from warn". If I understand correctly, "logging.info" actually calls the root logger, and root logger default to warn level. So the first "hello from info" is ignored, which is fine. But why the second "hello from info" is also not printed?
import logging
logging.info("hello from info")
logging.warn("hello from warn")
logging.basicConfig(level=logging.INFO)
logging.info("hello from info")
The second question is the logging level of Handler versus logger. If we set the log level for both the handler and logger, which one is working? Or what if we just set level for Handler? Take the example as below. We already set the log level for StreamHandler, but the "hello from info" is not printed to stdout. Only the "hello from warn" (Besides, it is not "WARNING:t:hello from warn"). Why is that?
import logging
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
logger = logging.getLogger('t')
logger.addHandler(ch)
logger.info("hello from info")
logger.warn("hello from warn")

But why the second "hello from info" is also not printed?
Because
logging.info / warn / error / debug call logging.basicConfig under the hood. Example:
def info(msg, *args, **kwargs):
if len(root.handlers) == 0:
basicConfig()
root.info(msg, *args, **kwargs)
logging.basicConfig does not do anything if the root logger is already configured. Quote from the docs:
This function does nothing if the root logger already has handlers configured for it.
So, in your code, the root logger is configured with WARN level when logging.info("hello from info") is executed. The subsequent call of logging.basicConfig(level=logging.INFO) has no effect.
Rule of thumb: configure the loggers (no matter if manually or via logging.basicConfig()) as early as possible in your code.
If we set the log level for both the handler and logger, which one is working?
Both! Logger level and handler level are two different stages of filtering records. The logger level defines what records are actually passed to its handlers, while the handler level defines what records will be handled by the particular handler. Examples:
logger.setLevel(logging.INFO)
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.info('spam')
Since logger has level INFO, it will process spam record and pass it to its handlers. However, handler has level ERROR, so the record will not be processed by handler.
logger.setLevel(logging.WARN)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
logger.info('spam')
Now the handler will process almost any record, including spam record since its level is INFO, thus greater than DEBUG. However, the handler will never receive spam to process because the logger will not process it, thus not passing spam to its handlers.
logger.setLevel(logging.INFO)
h1 = logging.StreamHandler()
h1.setLevel(logging.CRITICAL)
h2 = logging.FileHandler('some.log')
h2.setLevel(logging.DEBUG)
logger.addHandler(h1)
logger.addHandler(h2)
logger.info('spam')
Now the logger has two handlers, h1 printing records to terminal, h2 writing them to file. The logger will pass only records of level INFO or greater to its handlers. However, you will see only records with level CRITICAL in terminal, but all records in log file.

python: logging sending stdout to log [duplicate]

This question already has answers here:
Redirect Python 'print' output to Logger
(7 answers)
Closed 6 years ago.
import logging
import sys
log_fmt = 'brbuild: %(message)s'
# Initilaize log here
# TODO may need to flush
logging.basicConfig(filename="logtest",
level=logging.DEBUG,
format=log_fmt,
datefmt='%H:%M:%S',
filemode='a')
# capture stdout to log
ch = logging.StreamHandler(sys.stdout)
ch.setLevel(logging.DEBUG)
log_fmt = logging.Formatter(log_fmt)
ch.setFormatter(log_fmt)
logging.getLogger("logtest").addHandler(ch)
logging.info("using logging")
print "using stdout"
logtest
brbuild: using logging
how can i get "using stdout" to be written in the log as well?

That's a kind of a hack but you could redefine print in the current module, and others module could perform a from foo import print to be able to use it.
For simplicity's sake, I haven't used file handles in that example, but stdout/stderr. If you use files, you can still add a sys.stdout.write(msg+os.linesep) statement to your new print function.
my new print may not be as powerful as the original print but it supports multiple arguments as well.
import logging,sys
def print(*args):
logger.info(" ".join([str(x) for x in args]))
if __name__ == '__main__':
logger = logging.getLogger('foo')
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.addHandler(logging.StreamHandler(sys.stderr))
logger.setLevel(logging.INFO)
a=12
logger.info('1. This should appear in both stdout and stderr.')
print("logging works!",a)
(you have to use it with parentheses). Result:
1. This should appear in both stdout and stderr.
1. This should appear in both stdout and stderr.
logging works! 12
logging works! 12

If your intention is to redirect the print output (i.e. redirect sys.stdout) to logger, or to both the logger and the standard output, you will need to create a class that mimics a file-like object to do that and assign an instance of that file-like object class to sys.stdout.

Extending the Python Logger

I'm looking for a simple way to extend the logging functionality defined in the standard python library. I just want the ability to choose whether or not my logs are also printed to the screen.
Example: Normally to log a warning you would call:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s', filename='log.log', filemode='w')
logging.warning("WARNING!!!")
This sets the configurations of the log and puts the warning into the log
I would like to have something along the lines of a call like:
logging.warning("WARNING!!!", True)
where the True statement signifys if the log is also printed to stdout.
I've seen some examples of implementations of overriding the logger class
but I am new to the language and don't really follow what is going on, or how to implement this idea. Any help would be greatly appreciated :)

The Python logging module defines these classes:
Loggers that emit log messages.
Handlers that put those messages to a destination.
Formatters that format log messages.
Filters that filter log messages.
A Logger can have Handlers. You add them by invoking the addHandler() method. A Handler can have Filters and Formatters. You similarly add them by invoking the addFilter() and setFormatter() methods, respectively.
It works like this:
import logging
# make a logger
main_logger = logging.getLogger("my logger")
main_logger.setLevel(logging.INFO)
# make some handlers
console_handler = logging.StreamHandler() # by default, sys.stderr
file_handler = logging.FileHandler("my_log_file.txt")
# set logging levels
console_handler.setLevel(logging.WARNING)
file_handler.setLevel(logging.INFO)
# add handlers to logger
main_logger.addHandler(console_handler)
main_logger.addHandler(file_handler)
Now, you can use this object like this:
main_logger.info("logged in the FILE")
main_logger.warning("logged in the FILE and on the CONSOLE")
If you just run python on your machine, you can type the above code into the interactive console and you should see the output. The log file will get crated in your current directory, if you have permissions to create files in it.
I hope this helps!

It is possible to override logging.getLoggerClass() to add new functionality to loggers. I wrote simple class which prints green messages in stdout.
Most important parts of my code:
class ColorLogger(logging.getLoggerClass()):
__GREEN = '\033[0;32m%s\033[0m'
__FORMAT = {
'fmt': '%(asctime)s %(levelname)s: %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S',
}
def __init__(self, format=__FORMAT):
formatter = logging.Formatter(**format)
self.root.setLevel(logging.INFO)
self.root.handlers = []
(...)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
self.root.addHandler(handler)
def info(self, message):
self.root.info(message)
(...)
def info_green(self, message):
self.root.info(self.__GREEN, message)
(...)
if __name__ == '__main__':
logger = ColorLogger()
logger.info("This message has default color.")
logger.info_green("This message is green.")

Handlers send the log records (created by loggers) to the appropriate
destination.
(from the docs: http://docs.python.org/library/logging.html)
Just set up multiple handlers with your logging object, one to write to file, another to write to the screen.
UPDATE
Here is an example function you can call in your classes to get logging set up with a handler.
def set_up_logger(self):
# create logger object
self.log = logging.getLogger("command")
self.log.setLevel(logging.DEBUG)
# create console handler and set min level recorded to debug messages
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# add the handler to the log object
self.log.addHandler(ch)
You would just need to set up another handler for files, ala the StreamHandler code that's already there, and add it to the logging object. The line that says ch.setLevel(logging.DEBUG) means that this particular handler will take logging messages that are DEBUG or higher. You'll likely want to set yours to WARNING or higher, since you only want the more important things to go to the console. So, your logging would work like this:
self.log.info("Hello, World!") -> goes to file
self.log.error("OMG!!") -> goes to file AND console