We Keep Coding

Logging to file and console

I have a Python script and I want that the info method write the messages in the console. But the warning, critical or error writes the messages to a file. How can I do that?
I tried this:
import logging
console_log = logging.getLogger("CONSOLE")
console_log.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
console_log.addHandler(stream_handler)
file_log = logging.getLogger("FILE")
file_log.setLevel(logging.WARNING)
file_handler = logging.FileHandler('log.txt')
file_handler.setLevel(logging.WARNING)
file_log.addHandler(file_handler)
def log_to_console(message):
console_log.info(message)
def log_to_file(message):
file_log.warning(message)
log_to_console("THIS SHOULD SHOW ONLY IN CONSOLE")
log_to_file("THIS SHOULD SHOW ONLY IN FILE")
but the message that should be only in the file is going to the console too, and the message that should be in the console, is duplicating. What am I doing wrong here?

What happens is that the two loggers you created propagated the log upwards to the root logger. The root logger does not have any handlers by default, but will use the lastResort handler if needed:
A "handler of last resort" is available through this attribute. This
is a StreamHandler writing to sys.stderr with a level of WARNING, and
is used to handle logging events in the absence of any logging
configuration. The end result is to just print the message to
sys.stderr.
Source from the Python documentation.
Inside the Python source code, you can see where the call is done.
Therefore, to solve your problem, you could set the console_log and file_log loggers' propagate attribute to False.
On another note, I think you should refrain from instantiating several loggers for you use case. Just use one custom logger with 2 different handlers that will each log to a different destination.
Create a custom StreamHandler to log only the specified level:
import logging
class MyStreamHandler(logging.StreamHandler):
def emit(self, record):
if record.levelno == self.level:
# this ensures this handler will print only for the specified level
super().emit(record)
Then, use it:
my_custom_logger = logging.getLogger("foobar")
my_custom_logger.propagate = False
my_custom_logger.setLevel(logging.INFO)
stream_handler = MyStreamHandler()
stream_handler.setLevel(logging.INFO)
file_handler = logging.FileHandler("log.txt")
file_handler.setLevel(logging.WARNING)
my_custom_logger.addHandler(stream_handler)
my_custom_logger.addHandler(file_handler)
my_custom_logger.info("THIS SHOULD SHOW ONLY IN CONSOLE")
my_custom_logger.warning("THIS SHOULD SHOW ONLY IN FILE")
And it works without duplicate and without misplaced log.

Python2.7 log level of logging doesn't work

It seems recently there were some changes of python logging package. Some code work previously doesn't work now. And I am confused. My python version is Python 2.7.15.
The first example I don't understand is that below only prints "WARNING:root:hello from warn". If I understand correctly, "logging.info" actually calls the root logger, and root logger default to warn level. So the first "hello from info" is ignored, which is fine. But why the second "hello from info" is also not printed?
import logging
logging.info("hello from info")
logging.warn("hello from warn")
logging.basicConfig(level=logging.INFO)
logging.info("hello from info")
The second question is the logging level of Handler versus logger. If we set the log level for both the handler and logger, which one is working? Or what if we just set level for Handler? Take the example as below. We already set the log level for StreamHandler, but the "hello from info" is not printed to stdout. Only the "hello from warn" (Besides, it is not "WARNING:t:hello from warn"). Why is that?
import logging
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
logger = logging.getLogger('t')
logger.addHandler(ch)
logger.info("hello from info")
logger.warn("hello from warn")

But why the second "hello from info" is also not printed?
Because
logging.info / warn / error / debug call logging.basicConfig under the hood. Example:
def info(msg, *args, **kwargs):
if len(root.handlers) == 0:
basicConfig()
root.info(msg, *args, **kwargs)
logging.basicConfig does not do anything if the root logger is already configured. Quote from the docs:
This function does nothing if the root logger already has handlers configured for it.
So, in your code, the root logger is configured with WARN level when logging.info("hello from info") is executed. The subsequent call of logging.basicConfig(level=logging.INFO) has no effect.
Rule of thumb: configure the loggers (no matter if manually or via logging.basicConfig()) as early as possible in your code.
If we set the log level for both the handler and logger, which one is working?
Both! Logger level and handler level are two different stages of filtering records. The logger level defines what records are actually passed to its handlers, while the handler level defines what records will be handled by the particular handler. Examples:
logger.setLevel(logging.INFO)
handler.setLevel(logging.ERROR)
logger.addHandler(handler)
logger.info('spam')
Since logger has level INFO, it will process spam record and pass it to its handlers. However, handler has level ERROR, so the record will not be processed by handler.
logger.setLevel(logging.WARN)
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
logger.info('spam')
Now the handler will process almost any record, including spam record since its level is INFO, thus greater than DEBUG. However, the handler will never receive spam to process because the logger will not process it, thus not passing spam to its handlers.
logger.setLevel(logging.INFO)
h1 = logging.StreamHandler()
h1.setLevel(logging.CRITICAL)
h2 = logging.FileHandler('some.log')
h2.setLevel(logging.DEBUG)
logger.addHandler(h1)
logger.addHandler(h2)
logger.info('spam')
Now the logger has two handlers, h1 printing records to terminal, h2 writing them to file. The logger will pass only records of level INFO or greater to its handlers. However, you will see only records with level CRITICAL in terminal, but all records in log file.

Python logging setLevel not logging

Is there anything in this code that would explain why my info messages aren't going into the log. Properly formatted warnings and above are going into both log files.
Initializing logger:
logger = logging.getLogger()
f = logging.Formatter('%(asctime)s\n%(levelname)s: %(funcName)s %(message)s')
out = logging.handlers.RotatingFileHandler(filename=self.f_stdout, maxBytes=1048576, backupCount=99)
err = logging.handlers.RotatingFileHandler(filename=self.f_stderr, maxBytes=1048576, backupCount=99)
out.setLevel(logging.INFO)
err.setLevel(logging.WARNING)
err.setFormatter(f)
logger.addHandler(out)
logger.addHandler(err)
Usage:
logging.info('this doesnt get logged')
logging.warning('this gets logged to stdout and stderr with respective formatting')

You never actually set the log level of the root logger object itself (the logger variable in your code). Both handlers and loggers have log levels; lines only reach the output if they're above both thresholds.
Since you don't set the root log level, it uses its default (warning). Try adding a call to logger.setLevel(logging.INFO) to change that.

Python redirecting log

I am running a web server, tornado, and I am trying to redirect all the log output to a file using the following command. But I don't see the output in the file.
/usr/bin/python -u index.py 2>&1 >> /tmp/tornado.log
I pass -u option to python interpreter. I still don't see any output logged to my log file.
However, I see the output on stdout when I do the following
/usr/bin/python index.py

Tornado uses the built-in logging module. You can easily attach a file handler to the root logger and set its level to NOTSET so it records everything, or some other level if you want to filter.
Reference docs: logging, logging.handlers
Example that works with Tornado's logging:
import logging
# the root logger is created upon the first import of the logging module
# create a file handler to add to the root logger
filehandler = logging.FileHandler(
filename = 'test.log',
mode = 'a',
encoding = None,
delay = False
)
# set the file handler's level to your desired logging level, e.g. INFO
filehandler.setLevel(logging.INFO)
# create a formatter for the file handler
formatter = logging.Formatter('%(asctime)s.%(msecs)d [%(name)s](%(process)d): %(levelname)s: %(message)s')
# add filters if you want your handler to only handle events from specific loggers
# e.g. "main.sub.classb" or something like that. I'll leave this commented out.
# filehandler.addFilter(logging.Filter(name='root.child'))
# set the root logger's level to be at most as high as your handler's
if logging.root.level > filehandler.level:
logging.root.setLevel = filehandler.level
# finally, add the handler to the root. after you do this, the root logger will write
# records to file.
logging.root.addHandler(filehandler)
More often than not, I actually wish to suppress tornado's loggers (because I have my own, and catch their exceptions anyway, and they just end up polluting my logs,) and this is where adding a filter on your filehandlers can come in very handy.

Extending the Python Logger

I'm looking for a simple way to extend the logging functionality defined in the standard python library. I just want the ability to choose whether or not my logs are also printed to the screen.
Example: Normally to log a warning you would call:
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(levelname)s: %(message)s', filename='log.log', filemode='w')
logging.warning("WARNING!!!")
This sets the configurations of the log and puts the warning into the log
I would like to have something along the lines of a call like:
logging.warning("WARNING!!!", True)
where the True statement signifys if the log is also printed to stdout.
I've seen some examples of implementations of overriding the logger class
but I am new to the language and don't really follow what is going on, or how to implement this idea. Any help would be greatly appreciated :)

The Python logging module defines these classes:
Loggers that emit log messages.
Handlers that put those messages to a destination.
Formatters that format log messages.
Filters that filter log messages.
A Logger can have Handlers. You add them by invoking the addHandler() method. A Handler can have Filters and Formatters. You similarly add them by invoking the addFilter() and setFormatter() methods, respectively.
It works like this:
import logging
# make a logger
main_logger = logging.getLogger("my logger")
main_logger.setLevel(logging.INFO)
# make some handlers
console_handler = logging.StreamHandler() # by default, sys.stderr
file_handler = logging.FileHandler("my_log_file.txt")
# set logging levels
console_handler.setLevel(logging.WARNING)
file_handler.setLevel(logging.INFO)
# add handlers to logger
main_logger.addHandler(console_handler)
main_logger.addHandler(file_handler)
Now, you can use this object like this:
main_logger.info("logged in the FILE")
main_logger.warning("logged in the FILE and on the CONSOLE")
If you just run python on your machine, you can type the above code into the interactive console and you should see the output. The log file will get crated in your current directory, if you have permissions to create files in it.
I hope this helps!

It is possible to override logging.getLoggerClass() to add new functionality to loggers. I wrote simple class which prints green messages in stdout.
Most important parts of my code:
class ColorLogger(logging.getLoggerClass()):
__GREEN = '\033[0;32m%s\033[0m'
__FORMAT = {
'fmt': '%(asctime)s %(levelname)s: %(message)s',
'datefmt': '%Y-%m-%d %H:%M:%S',
}
def __init__(self, format=__FORMAT):
formatter = logging.Formatter(**format)
self.root.setLevel(logging.INFO)
self.root.handlers = []
(...)
handler = logging.StreamHandler()
handler.setFormatter(formatter)
self.root.addHandler(handler)
def info(self, message):
self.root.info(message)
(...)
def info_green(self, message):
self.root.info(self.__GREEN, message)
(...)
if __name__ == '__main__':
logger = ColorLogger()
logger.info("This message has default color.")
logger.info_green("This message is green.")

Handlers send the log records (created by loggers) to the appropriate
destination.
(from the docs: http://docs.python.org/library/logging.html)
Just set up multiple handlers with your logging object, one to write to file, another to write to the screen.
UPDATE
Here is an example function you can call in your classes to get logging set up with a handler.
def set_up_logger(self):
# create logger object
self.log = logging.getLogger("command")
self.log.setLevel(logging.DEBUG)
# create console handler and set min level recorded to debug messages
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
# add the handler to the log object
self.log.addHandler(ch)
You would just need to set up another handler for files, ala the StreamHandler code that's already there, and add it to the logging object. The line that says ch.setLevel(logging.DEBUG) means that this particular handler will take logging messages that are DEBUG or higher. You'll likely want to set yours to WARNING or higher, since you only want the more important things to go to the console. So, your logging would work like this:
self.log.info("Hello, World!") -> goes to file
self.log.error("OMG!!") -> goes to file AND console