We Keep Coding

Python logging stdout and stderr based on level

Using Python 3 logging, how can I specify that logging.debug() and logging.info() should go to stdout and logging.warning() and logging.error() go to stderr?

You can create separate loggers for each stream like this:
import logging
import sys
logging.basicConfig(format="%(levelname)s %(message)s")
stdout_logger = logging.Logger(name="stdout_logger", level=logging.DEBUG)
stderr_logger = logging.Logger(name="stderr_logger", level=logging.DEBUG)
stdout_handler = logging.StreamHandler(stream=sys.stdout)
stderr_handler = logging.StreamHandler(stream=sys.stderr)
stdout_logger.addHandler(hdlr=stdout_handler)
stderr_logger.addHandler(hdlr=stderr_handler)
stdout_logger.info("this will output to stdout")
stderr_logger.info("this will output to stderr")
Then if you want to log something on 'debug' or 'info' level you can just use stdout_logger. For 'warning' and 'error' level messages use stderr_logger.

How to do this kind of thing is documented in the logging cookbook, and though the example levels/destinations in the cookbook recipe are slightly different from those in your question, there should be enough information there for you to arrive at a solution. The key thing is using a filter function and attaching it to a handler.

Python logging module doesn't work in Sagemaker

I've got a sagemaker instance running a jupyter notebook. I'd like to use python's logging module to write to a log file, but it doesn't work.
My code is pretty straightforward:
import logging
logger = logging.getLogger()
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(name)s - %(message)s", datefmt="%y/%m/%d %H:%M:%S")
fhandler = logging.FileHandler("taxi_training.log")
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.debug("starting log...")
This should write a line to my file taxi_training.log but it doesn't.
I tried using the reload function from importlib, I also tried setting the output stream to sys.stdout explicitly. Nothing is logging to the file or in cloudwatch.
Do I need to add anything to my Sagemaker instance for this to work properly?

The Python logging module requires a logging level and one or more handlers to process output. By default, the logging level is set to WARNING (30) with a STDOUT handler for that level. If a logging level and/or handler is not explicitly defined, these settings are inherited from the parent root logger settings. These settings can be verified by adding the following lines to the bottom of your code:
# Verify levels and handlers
print("Parent Logger: "+logger.parent.name)
print("Parent Level: "+str(logger.parent.level))
print("Parent Handlers: "+str(logger.parent.handlers))
print("Logger Level: "+str(logger.level))
print("Logger Handlers: "+str(logger.handlers))
The easiest way to instantiate a handler and set a logging level is by running the logging.basicConfig() function (documentation). This will set a logging level and STDOUT handler at the root logger level which will propagate to any child loggers created in the same code. Here is an example using the code provided:
import logging
logger = logging.getLogger('log')
logging.basicConfig(level=logging.INFO) # Set logging level and STDOUT handler
logger.info(5)

Python logger DEBUG is not shown

Given this setup, I cannot see any debug logs:
import logging
import coloredlogs
coloredlogs.install(isatty=True)
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# Show the current debug level
logger.critical(logger.level)
logger.critical("critical")
logger.error("error")
logger.warning("warning")
logger.info("info")
logger.debug("debug") <-- NOT SHOWN!
Yet if I change to
coloredlogs.install(level=logging.DEBUG,isatty=True)
I get all the DEBUG notices but they include other libs debug messages (botocore, urllib..) which is unwanted since I want to see only my debug notices.
What did I miss?

Have you tried:
logging.basicConfig(level=logging.DEBUG)
instead of logger.setLevel? I assume you only set the level of that logger, but the corresponding handlers still have a higher level, so the message is ignored.

How to use RotatingFileHandler with the root logger

I configure the root logger:
logging.basicConfig(filename='logfile.log', level=logging.DEBUG)
Then I put log messages in my code like this:
logging.debug("This is a log message")
Question: How do I add a RotatingFileHandler such that my logs will be rotated?
Note: I do not want a logger instance which I then have to pass around everywhere.

You can do this by using the handlers kwarg of basicConfig. Be aware that this needs to be an iterable and you can not use the filename argument together with it.
import logging
import logging.handlers
rot_handler = logging.handlers.RotatingFileHandler('filename.txt')
logging.basicConfig(level=logging.DEBUG, handlers=[rot_handler])
Link to relevant part of documentation: https://docs.python.org/3/library/logging.html#logging.basicConfig

python logging set level in basicConfig

python logging set level in basicConfig:
import logging
def show(level):
logging.basicConfig(level=level)
logging.info('info')
logging.debug('debug')
logging.warn('warn')
logging.error('error')
logging.fatal('fatal')
logging.warning('warning')
logging.critical('critical')
logging.exception('exception')
show(logging.WARNING)
show(logging.DEBUG)
The two results are the same, how to get what I expects?

According to logging.basicConfig documentation, the second call to logging.basicConfig does not take effect.
This function does nothing if the root logger already has handlers
configured for it.
def show(level):
logger = logging.getLogger()
logger.setLevel(level)
logging.info('info')
....