I configure the root logger:
logging.basicConfig(filename='logfile.log', level=logging.DEBUG)
Then I put log messages in my code like this:
logging.debug("This is a log message")
Question: How do I add a RotatingFileHandler such that my logs will be rotated?
Note: I do not want a logger instance which I then have to pass around everywhere.
You can do this by using the handlers kwarg of basicConfig. Be aware that this needs to be an iterable and you can not use the filename argument together with it.
import logging
import logging.handlers
rot_handler = logging.handlers.RotatingFileHandler('filename.txt')
logging.basicConfig(level=logging.DEBUG, handlers=[rot_handler])
Link to relevant part of documentation: https://docs.python.org/3/library/logging.html#logging.basicConfig
Related
I tried to enable logging for my python library. I want all logs to go into my custom subset my_logger instead of root. So this is what I tried:
import logging
my_logger = logging.getLogger('my_logger')
my_logger.warning("my hello!")
logging.warning("hello!")
And for some reason my custom logger didn't output subset name (my_logger) in front of it.
my hello!
WARNING:root:hello!
A simple change of the order of root and my loggers fixed the issue:
import logging
logging.warning("hello!")
my_logger = logging.getLogger('my_logger')
my_logger.warning("my hello!")
Output
WARNING:root:hello!
WARNING:my_logger:my hello!
I do not ever want to use a root logger at all. Is it possible to get WARNING:my_logger prefix to my output without logging to the root logger first?
Seems like the logging module is doing first-time configuration during the first call. You can do that first to get it out of the way before using your logger:
import logging
logging.basicConfig() # do the configuration
my_logger = logging.getLogger('my_logger')
my_logger.warning("my hello!")
logging.warning("hello!")
Result:
WARNING:my_logger:my hello!
WARNING:root:hello!
For a library that you expect others to import, this should be done by the code importing the module instead of the library.
I've got a sagemaker instance running a jupyter notebook. I'd like to use python's logging module to write to a log file, but it doesn't work.
My code is pretty straightforward:
import logging
logger = logging.getLogger()
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(name)s - %(message)s", datefmt="%y/%m/%d %H:%M:%S")
fhandler = logging.FileHandler("taxi_training.log")
fhandler.setFormatter(formatter)
logger.addHandler(fhandler)
logger.debug("starting log...")
This should write a line to my file taxi_training.log but it doesn't.
I tried using the reload function from importlib, I also tried setting the output stream to sys.stdout explicitly. Nothing is logging to the file or in cloudwatch.
Do I need to add anything to my Sagemaker instance for this to work properly?
The Python logging module requires a logging level and one or more handlers to process output. By default, the logging level is set to WARNING (30) with a STDOUT handler for that level. If a logging level and/or handler is not explicitly defined, these settings are inherited from the parent root logger settings. These settings can be verified by adding the following lines to the bottom of your code:
# Verify levels and handlers
print("Parent Logger: "+logger.parent.name)
print("Parent Level: "+str(logger.parent.level))
print("Parent Handlers: "+str(logger.parent.handlers))
print("Logger Level: "+str(logger.level))
print("Logger Handlers: "+str(logger.handlers))
The easiest way to instantiate a handler and set a logging level is by running the logging.basicConfig() function (documentation). This will set a logging level and STDOUT handler at the root logger level which will propagate to any child loggers created in the same code. Here is an example using the code provided:
import logging
logger = logging.getLogger('log')
logging.basicConfig(level=logging.INFO) # Set logging level and STDOUT handler
logger.info(5)
Overview
I want to use httpimport as a logging library common to several scripts. This module generates logs of its own which I do not know how to silence.
In other cases such as this one, I would have used
logging.getLogger('httpimport').setLevel(logging.ERROR)
but it did not work.
Details
The following code is a stub of the "common logging code" mentioned above:
# toconsole.py
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s %(message)s')
handler_console = logging.StreamHandler()
level = logging.DEBUG if 'DEV' in os.environ else logging.INFO
handler_console.setLevel(level)
handler_console.setFormatter(formatter)
log.addHandler(handler_console)
# disable httpimport logging except for errors+
logging.getLogger('httpimport').setLevel(logging.ERROR)
A simple usage such as
import httpimport
httpimport.INSECURE = True
with httpimport.remote_repo(['githublogging'], 'http://localhost:8000/') :
from toconsole import log
log.info('yay!')
gives the following output
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
2019-08-25 13:56:48,671 yay!
yay!
The second (bare) yay! must be coming from httpimport, namely from its logging setup.
How can I disable the logging for such a module, or better - raise its level so that only errors+ are logged?
Note: this question was initially asked at the Issues section of the GitHub repository for httpimport but the author did not know either how to fix that.
Author of httpimport here.
I totally forgot I was using the basicConfig logger thing.
It is fixed in master right now (0.7.2) - will be included in next PyPI release:
https://github.com/operatorequals/httpimport/commit/ff2896c8f666c3f16b0f27716c732d68be018ef7
The reason why this is happening is because when you do import httpimport they do the initial configuration for the logging machinery. This happens right here. What this means is that the root logger already has a StreamHandler attached to it. Because of this, and the fact that all loggers inherit from the root logger, when you do log.info('yay') it not only uses your Handler and Formatter, but it also propagates all they way to the root logger, which also emits the message.
Remember that whoever calls basicConfig first when an application starts that sets up the default configuration for the root logger, which in turn, is inherited by all loggers, unless otherwise specified.
If you have a complex logging configuration you need to ensure that you call it before you do any third-party imports which might call basicConfig. basicConfig is idempotent meaning the first call seals the deal, and subsequent calls have no effect.
Solutions
You could do log.propagate = False and you will see that the 2nd yay will not show.
You could attach the Formatter directly to the already existent root Handler by doing something like this (without adding another Handler yourself)
root = logging.getLogger('')
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler = root.handlers[0]
root_handler.setFormatter(formatter)
You could do a basicConfig call when you initialize your application (if you had such a config available, with initial Formatters and Handlers, etc. that will elegantly attach everything to the root logger neatly) and then you would only do something like logger = logging.getLogger(__name__) and logger.info('some message') that would work the way you'd expect because it would propagate all the way to the root logger which already has your configuration.
You could remove the initial Handler that's present on the root logger by doing something like
root = logging.getLogger('')
root.handlers = []
... and many more solutions, but you get the idea.
Also do note that logging.getLogger('httpimport').setLevel(logging.ERROR) this works perfectly fine. No messages below logging.ERROR would be logged by that logger, it's just that the problem wasn't from there.
If you however want to completely disable a logger you can just do logger.disabled = True (also do note, again, that the problem wasn't from the httpimport logger, as aforementioned)
One example demonstrated
Change your toconsole.py with this and you won't see the second yay.
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
root_logger = logging.getLogger('')
root_handler = root_logger.handlers[0]
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler.setFormatter(formatter)
# or you could just keep your old code and just add log.propagate = False
# or any of the above solutions and it would work
logging.getLogger('httpimport').setLevel(logging.ERROR)
Looking at the python docs, if I set my logger level to INFO, it should print out all logs at level INFO and above.
However, the code snipper below only prints "error"
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.info("Info")
logger.error("error")
logger.info("info")
Output
error
What could be the reason for this?
Use logging.basicConfig to set a default level and a default handler:
import logging
logger = logging.getLogger()
logging.basicConfig(level=logging.INFO)
logger.info("Info")
logger.error("error")
logger.info("info")
prints:
INFO:root:Info
ERROR:root:error
INFO:root:info
The logging module is powerful yet confusing. Look into the HOWTO in the docs for a tutorial. I've made my own helper function that logs to stderr and a file that I've detailed on my blog. You might like to adapt it to your needs.
The reason for this behaviour is that there are no logging handlers defined.
In this case the handler "logging.lastResort" is used.
Per default this handler is "<_StderrHandler (WARNING)>".
It logs to stderr but only starting from level "WARNING".
For your example you could do the following (logging to stdout):
import logging
import sys
logger = logging.getLogger()
logger.setLevel(logging.INFO)
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.info("info")
Output
info
In the howto you can find other useful handlers.
python logging set level in basicConfig:
import logging
def show(level):
logging.basicConfig(level=level)
logging.info('info')
logging.debug('debug')
logging.warn('warn')
logging.error('error')
logging.fatal('fatal')
logging.warning('warning')
logging.critical('critical')
logging.exception('exception')
show(logging.WARNING)
show(logging.DEBUG)
The two results are the same, how to get what I expects?
According to logging.basicConfig documentation, the second call to logging.basicConfig does not take effect.
This function does nothing if the root logger already has handlers
configured for it.
def show(level):
logger = logging.getLogger()
logger.setLevel(level)
logging.info('info')
....