I am using pytest to test a CLI that produces some output. While running the test, I want to set my CLI's log level to DEBUG. However, I don't want CLI logs to interfere with tests that are parsing the output of the CLI.
How can I make logging module send all the logs to only stderr? I looked at this post but it talks about sending logs to stderr in addition to stdout.
You can direct the log output for a given logger to stderr as follows. This defaults to stderr for output, but you can use sys.stdout instead if you prefer.
import logging
import sys
DEFAULT_LOGGER_NAME = 'default_logger'
def init_logging(logger_name=DEFAULT_LOGGER_NAME,
log_level=logging.DEBUG,
stream=None):
# logging
logger = logging.getLogger(logger_name)
logger.setLevel(log_level)
# create formatter
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
# create console handler and set level to debug
if stream is None:
stream = sys.stderr
ch = logging.StreamHandler(stream=stream)
ch.setLevel(log_level)
# add formatter to ch
ch.setFormatter(formatter)
# add ch to logger
logger.addHandler(ch)
return logger
This function needs to be called at the beginning of your program (e.g. beginning of main()).
Then within the code, you just need to call the following:
logger = logging.getLogger(LOGGER_NAME)
Do the same that the linked answer does but replace stdout with stderr. So you would create a handler with logging.StreamHandler(sys.stderr) and make sure this is the only active handler if you want to exclusively have logs go to stderr.
As #Tomerikoo correctly points out you don't need to do anything though as logging defaults to using a StreamHandler with stderr. The only real value of the code below is that it sets a different level than the default. Just logging.warning('log') with no other setup will send a log to stderr.
Addendum: you can also achieve this using basicConfig to have less boilerplate code.
import logging
import sys
logging.basicConfig(stream=sys.stderr, level=logging.INFO)
logging.info('test') # sends log to stderr
Related
Using Python 3 logging, how can I specify that logging.debug() and logging.info() should go to stdout and logging.warning() and logging.error() go to stderr?
You can create separate loggers for each stream like this:
import logging
import sys
logging.basicConfig(format="%(levelname)s %(message)s")
stdout_logger = logging.Logger(name="stdout_logger", level=logging.DEBUG)
stderr_logger = logging.Logger(name="stderr_logger", level=logging.DEBUG)
stdout_handler = logging.StreamHandler(stream=sys.stdout)
stderr_handler = logging.StreamHandler(stream=sys.stderr)
stdout_logger.addHandler(hdlr=stdout_handler)
stderr_logger.addHandler(hdlr=stderr_handler)
stdout_logger.info("this will output to stdout")
stderr_logger.info("this will output to stderr")
Then if you want to log something on 'debug' or 'info' level you can just use stdout_logger. For 'warning' and 'error' level messages use stderr_logger.
How to do this kind of thing is documented in the logging cookbook, and though the example levels/destinations in the cookbook recipe are slightly different from those in your question, there should be enough information there for you to arrive at a solution. The key thing is using a filter function and attaching it to a handler.
My logging setup is:
import coloredlogs
import logging
import sys
# Create a logger object.
# logger = logging.getLogger(__name__)
# By default the install() function installs a handler on the root logger,
# this means that log messages from your code and log messages from the
# libraries that you use will all show up on the terminal.
coloredlogs.install(level='DEBUG')
logging.basicConfig(
format='%(asctime)s %(levelname)-8s %(message)s',
level=logging.INFO,
stream=sys.stdout,
datefmt='%Y-%m-%d %H:%M:%S')
If I configure the console to use python for output, all lines start at column 0, but all output is red. But if I specify use terminal, the colors are there, but the lines don't start at column 1. They all start with the end column of the line before.
How can I get all log messages starting at column 0 AND in color?
Try adding isatty to your call to install. This overrides the whatever auto-detection is doing to attempt at determining what type of terminal is being used.
import coloredlogs
import logging
import sys
logger = logging.getLogger(__name__)
coloredlogs.install(level=logging.DEBUG, logger=logger, isatty=True,
fmt="%(asctime)s %(levelname)-8s %(message)s",
stream=sys.stdout,
datefmt='%Y-%m-%d %H:%M:%S')
logger.debug("this is a debugging message")
logger.info("this is an informational message")
logger.warning("this is a warning message")
logger.error("this is an error message")
logger.critical("this is a critical message")
explanation:
If I understood the problem correctly, you are seeing the logging handler default to using sys.stderr instead of sys.stdout. (this is why the text would appear in red)
There are likely two issues going on.
coloredlogs.install needs to be told what type of rules you want in the handler, not logging.basicConfig.
The automatic terminal detection would fail unless you either force the formatter to think it's a terminal or you set pycharm to 'simulate' a terminal. Fixed by passing in stream=sys.stdout
I've got a sagemaker instance running a jupyter notebook. I'd like to use python's logging module to write to a log file, but it doesn't work.
My code is pretty straightforward:
import logging
logger = logging.getLogger()
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(name)s - %(message)s", datefmt="%y/%m/%d %H:%M:%S")
fhandler = logging.FileHandler("taxi_training.log")
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.debug("starting log...")
This should write a line to my file taxi_training.log but it doesn't.
I tried using the reload function from importlib, I also tried setting the output stream to sys.stdout explicitly. Nothing is logging to the file or in cloudwatch.
Do I need to add anything to my Sagemaker instance for this to work properly?
The Python logging module requires a logging level and one or more handlers to process output. By default, the logging level is set to WARNING (30) with a STDOUT handler for that level. If a logging level and/or handler is not explicitly defined, these settings are inherited from the parent root logger settings. These settings can be verified by adding the following lines to the bottom of your code:
# Verify levels and handlers
print("Parent Logger: "+logger.parent.name)
print("Parent Level: "+str(logger.parent.level))
print("Parent Handlers: "+str(logger.parent.handlers))
print("Logger Level: "+str(logger.level))
print("Logger Handlers: "+str(logger.handlers))
The easiest way to instantiate a handler and set a logging level is by running the logging.basicConfig() function (documentation). This will set a logging level and STDOUT handler at the root logger level which will propagate to any child loggers created in the same code. Here is an example using the code provided:
import logging
logger = logging.getLogger('log')
logging.basicConfig(level=logging.INFO) # Set logging level and STDOUT handler
logger.info(5)
Is there anything in this code that would explain why my info messages aren't going into the log. Properly formatted warnings and above are going into both log files.
Initializing logger:
logger = logging.getLogger()
f = logging.Formatter('%(asctime)s\n%(levelname)s: %(funcName)s %(message)s')
out = logging.handlers.RotatingFileHandler(filename=self.f_stdout, maxBytes=1048576, backupCount=99)
err = logging.handlers.RotatingFileHandler(filename=self.f_stderr, maxBytes=1048576, backupCount=99)
out.setLevel(logging.INFO)
err.setLevel(logging.WARNING)
err.setFormatter(f)
logger.addHandler(out)
logger.addHandler(err)
Usage:
logging.info('this doesnt get logged')
logging.warning('this gets logged to stdout and stderr with respective formatting')
You never actually set the log level of the root logger object itself (the logger variable in your code). Both handlers and loggers have log levels; lines only reach the output if they're above both thresholds.
Since you don't set the root log level, it uses its default (warning). Try adding a call to logger.setLevel(logging.INFO) to change that.
I'm looking for a simple way to extend the logging functionality defined in the standard python library. I just want the ability to choose whether or not my logs are also printed to the screen.
Example: Normally to log a warning you would call:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s', filename='log.log', filemode='w')
logging.warning("WARNING!!!")
This sets the configurations of the log and puts the warning into the log
I would like to have something along the lines of a call like:
logging.warning("WARNING!!!", True)
where the True statement signifys if the log is also printed to stdout.
I've seen some examples of implementations of overriding the logger class
but I am new to the language and don't really follow what is going on, or how to implement this idea. Any help would be greatly appreciated :)
The Python logging module defines these classes:
Loggers that emit log messages.
Handlers that put those messages to a destination.
Formatters that format log messages.
Filters that filter log messages.
A Logger can have Handlers. You add them by invoking the addHandler() method. A Handler can have Filters and Formatters. You similarly add them by invoking the addFilter() and setFormatter() methods, respectively.
It works like this:
import logging
# make a logger
main_logger = logging.getLogger("my logger")
main_logger.setLevel(logging.INFO)
# make some handlers
console_handler = logging.StreamHandler() # by default, sys.stderr
file_handler = logging.FileHandler("my_log_file.txt")
# set logging levels
console_handler.setLevel(logging.WARNING)
file_handler.setLevel(logging.INFO)
# add handlers to logger
main_logger.addHandler(console_handler)
main_logger.addHandler(file_handler)
Now, you can use this object like this:
main_logger.info("logged in the FILE")
main_logger.warning("logged in the FILE and on the CONSOLE")
If you just run python on your machine, you can type the above code into the interactive console and you should see the output. The log file will get crated in your current directory, if you have permissions to create files in it.
I hope this helps!
It is possible to override logging.getLoggerClass() to add new functionality to loggers. I wrote simple class which prints green messages in stdout.
Most important parts of my code:
class ColorLogger(logging.getLoggerClass()):
__GREEN = '\033[0;32m%s\033[0m'
__FORMAT = {
'fmt': '%(asctime)s %(levelname)s: %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S',
}
def __init__(self, format=__FORMAT):
formatter = logging.Formatter(**format)
self.root.setLevel(logging.INFO)
self.root.handlers = []
(...)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
self.root.addHandler(handler)
def info(self, message):
self.root.info(message)
(...)
def info_green(self, message):
self.root.info(self.__GREEN, message)
(...)
if __name__ == '__main__':
logger = ColorLogger()
logger.info("This message has default color.")
logger.info_green("This message is green.")
Handlers send the log records (created by loggers) to the appropriate
destination.
(from the docs: http://docs.python.org/library/logging.html)
Just set up multiple handlers with your logging object, one to write to file, another to write to the screen.
UPDATE
Here is an example function you can call in your classes to get logging set up with a handler.
def set_up_logger(self):
# create logger object
self.log = logging.getLogger("command")
self.log.setLevel(logging.DEBUG)
# create console handler and set min level recorded to debug messages
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# add the handler to the log object
self.log.addHandler(ch)
You would just need to set up another handler for files, ala the StreamHandler code that's already there, and add it to the logging object. The line that says ch.setLevel(logging.DEBUG) means that this particular handler will take logging messages that are DEBUG or higher. You'll likely want to set yours to WARNING or higher, since you only want the more important things to go to the console. So, your logging would work like this:
self.log.info("Hello, World!") -> goes to file
self.log.error("OMG!!") -> goes to file AND console