We Keep Coding

Why isn't my Python logging setup working?

My code looks like this
import logging
formatter = logging.Formatter("{asctime} - {name} - {levelname} - {message}",
"%d-%b-%y %H:%M", "{")
def _add_handler(handler: logging.StreamHandler) -> logging.StreamHandler:
handler.setFormatter(formatter)
handler.setLevel(20)
return handler
logging.basicConfig(
handlers={
_add_handler(logging.FileHandler("filename.log")),
_add_handler(logging.StreamHandler())
})
logging.info("hello world")
What this is supposed to do is log "hello world" both to the console and a file named filename.log, with a severity of INFO, which is what the 20 in the setLevel method is for. However, nothing is being logged at all. Where have I gone wrong?

You need to set the logging level for the logger as well. By default, it's set to logging.WARNING, so neither handler is seeing the message, let alone determining if it should be handled.
logging.basicConfig(
level=logging.INFO,
handlers={
_add_handler(logging.FileHandler("filename.log")),
_add_handler(logging.StreamHandler())
})

Logger doesn't print debug messages after setting handler and log level

I have the following minimalist example of a logging test based on the Logging Cookbook:
import logging
logger = logging.getLogger('test')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - '
'%(message)s')
ch.setFormatter(formatter)
logger.addHandler(ch)
print(logger.handlers)
logger.debug('hello world')
The above produces the following output:
$ python test_log.py
[<StreamHandler <stderr> (DEBUG)>]
As I've defined a handler and set the log level to debug, I was expecting the hello world log message to show up in the sample above.

If a logger's level isn't explicitly set, the system looks up the level of ancestor loggers until it gets to a logger whose level is explicitly set. In this case, it's the root logger which is the parent of the logger named 'test'. Setting the level of either this logger or the root logger to DEBUG will result in the log message being output. See this part of the documentation for the event information flow in Python logging.

So if we call logger.debug('hello world') we are generating a DEBUG event. However the logger acts as first filter before passing the event to the handlers. So if the logger level is at higher severity (like INFO) it will filter the event and the handler won't receive it, even if the event was matching the handler's level (DEBUG).
So the logger's level should be as low as the lowest level of its handlers. If you don't want the rest of the handlers to log DEBUG messages you would need to explicitly set their level higher, instead of leaving it unset and therefor inherited from the logger.

Printing stderr with logging module

I am using python logging module. With this logging module I used to print statements with logging.info func
But problem is that I am not getting stderr messages in the log file. Is there a way I can print stderr with logging module
Following code I am using
import logging
def initialize_logger(output_dir):
'''
This initialise the logger
:param output_dir:
:return:
'''
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
# create console handler and set level to info
handler = logging.StreamHandler()
handler.setLevel(logging.INFO)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
handler = logging.FileHandler(output_dir, "w")
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(levelname)s - %(message)s")
handler.setFormatter(formatter)
logger.addHandler(handler)
In the above code, I am sending only logging statements in the log. It means that I am not getting stderr in the same log file as logging file. Is there a way I can use the same file to send stderr. For example, If my code fails, then along-with logging messages it should also send stderr message.
Right now I don't know why this is happening but seems like my framework has suppressed stderr and stdin messages to print on the console.
Is there a way I can log stderr via logging module?

Python - Logging from multiple modules to rotating files without printing to console

I am trying to add logging to a medium size Python project with minimal disruption. I would like to log from multiple modules to rotating files silently (without printing messages to the terminal). I have tried to modify this example and I almost have the functionality I need except for one issue.
Here is how I am attempting to configure things in my main script:
import logging
import logging.handlers
import my_module
LOG_FILE = 'logs\\logging_example_new.out'
#logging.basicConfig(level=logging.DEBUG)
# Define a Handler which writes messages to rotating log files.
handler = logging.handlers.RotatingFileHandler(LOG_FILE, maxBytes=100000, backupCount=1)
handler.setLevel(logging.DEBUG) # Set logging level.
# Create message formatter.
formatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s')
# Tell the handler to use this format
handler.setFormatter(formatter)
# Add the handler to the root logger
logging.getLogger('').addHandler(handler)
# Now, we can log to the root logger, or any other logger. First the root...
logging.debug('Root debug message.')
logging.info('Root info message.')
logging.warning('Root warning message.')
logging.error('Root error message.')
logging.critical('Root critical message.')
# Use a Logger in another module.
my_module.do_something() # Call function which logs a message.
Here is a sample of what I am trying to do in modules:
import logging
def do_something():
logger = logging.getLogger(__name__)
logger.debug('Module debug message.')
logger.info('Module info message.')
logger.warning('Module warning message.')
logger.error('Module error message.')
logger.critical('Module critical message.')
Now, here is my problem. I currently get the messages logged into rotating files silently. But I only get warning, error, and critical messages. Despite setting handler.setLevel(logging.DEBUG).
If I uncomment logging.basicConfig(level=logging.DEBUG), then I get all the messages in the log files but I also get the messages printed to the terminal.
How do I get all messages above the specified threshold to my log files without outputing them to the terminal?
Thanks!
Update:
Based on this answer, it appears that calling logging.basicConfig(level=logging.DEBUG) automatically adds a StreamHandler to the Root logger and you can remove it. When I did remove it leaving only my RotatingFileHandler, messages no longer printed to the terminal. I am still wondering why I have to use logging.basicConfig(level=logging.DEBUG) to set the message level threshold, when I am setting handler.setLevel(logging.DEBUG). If anyone can shed a little more light on these issues it would still be appreciated.

You need to call set the logging level on the logger itself as well. I believe by default, the logging level on the logger is logging.WARNING
Ex.
root_logger = logging.getLogger('')
root_logger.setLevel(logging.DEBUG)
# Add the handler to the root logger
root_logger.addHandler(handler)
The loggers log level determines what the logger will actually log (i.e what messages will actually get handed to the handlers). The handlers log level determines what it will actually handle (i.e. what messages actually are output to file, stream, etc). So you could potentially have multiple handlers attached to a logger each handling a different log level.
Here's an SO answer that explains the way this works

django/python logging

I have a pretty weird problem with my logging facility i use inside django/python. The logging is not working anymore since i upgraded to django 1.3. It seems to be related to the logging level and the 'debug=' setting in the settings.py file.
1) When i log INFO messages and debug=False, the logging won't happen, my file doesn't get appended.
2) When i log WARNING messages and debug=False, the logging works perfectly like i want it to, the file gets appended
3) When i log INFO messages and debug=True, the logging seems to work, the file get appended.
How could i log INFO messages with debug=False? It worked before django 1.3... is there somewhere a mysterious setting which do the trick? Underneath there is a sample code:
views.py:
import logging
logging.basicConfig(level=logging.INFO,
format='%(asctime)s %(levelname)s %(message)s',
filename='/opt/general.log',
filemode='a')
def create_log_file(filename, log_name, level=logging.INFO):
handler = logging.handlers.TimedRotatingFileHandler(filename, 'midnight', 7, backupCount=10)
handler.setLevel(level)
formatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-8s %(message)s', '%a, %Y-%m-%d %H:%M:%S')
handler.setFormatter(formatter)
logging.getLogger(log_name).addHandler(handler)
create_log_file('/opt/test.log', 'testlog')
logger_test = logging.getLogger('testlog')
logger_test.info('testing the logging functionality')
With this code the logging does not work in Django 1.3 with debug set to False in the settings.py file. When i should do like this:
logger_test.warning('testing the logging functionality')
This works perfectly when debug is set to False. The levels DEBUG and INFO aint logging but WARNING,ERROR and CRITICAL are doing their job...
Does anyone have an idea?

Since Django 1.3 contains its own logging configuration, you need to ensure that anything you're doing doesn't clash with it. For example, if the root logger has handlers already configured by Django by the time your module first gets imported, your basicConfig() call won't have any effect.
What you're describing is the normal logging situation - WARNINGs and above get handled, while INFO and DEBUG are suppressed by default. It looks as if your basicConfig() is not having any effect; You should consider replacing your basicConfig() call with the appropriate logging configuration in settings.py, or at least investigate the root logging level and what handlers are attached to it, at the time of your basicConfig() call.