I'm trying to do some basic logging. I'm doing the following:
log = logging.getLogger('test')
log.error('test') # 'test' gets logged
log.info('test') # Nothing gets logged, as expected
log.setLevel(logging.INFO) # Now info should be logged, right?
log.info('test') # No output
log.setLevel(logging.INFO) # Getting desperate
log.info('test') # Still nothing
What aspect of the logging module am I missing?
You have forgot to add handlers i.e. logging.basicConfig(), can you check if following code works for you.
import logging
logging.basicConfig()
log = logging.getLogger('test')
log.setLevel(logging.INFO)
log.error('test')
log.info('test')
You need to configure logging before you use it like below. You can configure it beautifully in python.
logging.basicConfig(format='%(asctime)s %(name)25s %(lineno)4d %(levelname)8s: %(message)s', level=40)
above statement will print timestamp, name of the file, line number, level type, message
level of logging also can be set during basic Config itself (40 = ERROR)
It is not necessary to call basicConfig.
However, the issue is about handlers (as pointed out by #Mahesh Karia).
You see, if you don't define some handlers you are dependent on whatever is already "in place" for you and that raises lots of confusion.
Check this:
You create your "test" logger
>>> log = logging.getLogger('test')
which prints WARNING messages but does not print INFO messages even after setting the level to INFO:
>>> log.warning('hi')
hi
>>> log.setLevel(logging.INFO)
>>> log.info('hi')
>>>
(personally, I think this is very wrong behaviour)
It can be "solved" by manipulating "handlers".
Your "test" logger has none
>>> log.handlers
[]
nor does it parent, the "root logger"
>>> log.parent
<RootLogger root (WARNING)>
>>> log.parent.handlers
[]
however, one can add a handler easy enough (to the root logger)
>>> log.parent.addHandler(logging.StreamHandler())
and now your "test" logger "works":
>>> log.info('hi')
hi
PS. Was using Python 3.8.2 prompt after importing logging module.
Related
I have a Python script and I want that the info method write the messages in the console. But the warning, critical or error writes the messages to a file. How can I do that?
I tried this:
import logging
console_log = logging.getLogger("CONSOLE")
console_log.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
console_log.addHandler(stream_handler)
file_log = logging.getLogger("FILE")
file_log.setLevel(logging.WARNING)
file_handler = logging.FileHandler('log.txt')
file_handler.setLevel(logging.WARNING)
file_log.addHandler(file_handler)
def log_to_console(message):
console_log.info(message)
def log_to_file(message):
file_log.warning(message)
log_to_console("THIS SHOULD SHOW ONLY IN CONSOLE")
log_to_file("THIS SHOULD SHOW ONLY IN FILE")
but the message that should be only in the file is going to the console too, and the message that should be in the console, is duplicating. What am I doing wrong here?
What happens is that the two loggers you created propagated the log upwards to the root logger. The root logger does not have any handlers by default, but will use the lastResort handler if needed:
A "handler of last resort" is available through this attribute. This
is a StreamHandler writing to sys.stderr with a level of WARNING, and
is used to handle logging events in the absence of any logging
configuration. The end result is to just print the message to
sys.stderr.
Source from the Python documentation.
Inside the Python source code, you can see where the call is done.
Therefore, to solve your problem, you could set the console_log and file_log loggers' propagate attribute to False.
On another note, I think you should refrain from instantiating several loggers for you use case. Just use one custom logger with 2 different handlers that will each log to a different destination.
Create a custom StreamHandler to log only the specified level:
import logging
class MyStreamHandler(logging.StreamHandler):
def emit(self, record):
if record.levelno == self.level:
# this ensures this handler will print only for the specified level
super().emit(record)
Then, use it:
my_custom_logger = logging.getLogger("foobar")
my_custom_logger.propagate = False
my_custom_logger.setLevel(logging.INFO)
stream_handler = MyStreamHandler()
stream_handler.setLevel(logging.INFO)
file_handler = logging.FileHandler("log.txt")
file_handler.setLevel(logging.WARNING)
my_custom_logger.addHandler(stream_handler)
my_custom_logger.addHandler(file_handler)
my_custom_logger.info("THIS SHOULD SHOW ONLY IN CONSOLE")
my_custom_logger.warning("THIS SHOULD SHOW ONLY IN FILE")
And it works without duplicate and without misplaced log.
i'm trying to reduce the amount of logging that the napalm library sends to syslog, but also allow for info logs to be sent from other parts of the code. I set up logging.basicConfig to be INFO but then i'd like the napalm function to be WARNING and above.
So i have code like this:
from napalm import get_network_driver
import logging
import getpass
logging.basicConfig(
filename="/var/log/myscripts/script.log", level=logging.INFO, format="%(asctime)s %(message)s")
def napalm(device):
logging.getLogger().setLevel(logging.WARNING)
username = getpass.getuser()
driver = get_network_driver("junos")
router = driver(str(device), username, "", password="", timeout=120)
router.open()
return router
router = napalm('myrouter')
config = "hostname foobar"
router.load_merge_candidate(config=config)
show = router.compare_config()
logging.info(show)
The issue is the logging.info output never makes it to the log file. If i do logging.warning(show) it does, but i'd like this to be info. The reason i want the function to be WARNING is that it generates so much other logging at the info level that is just noise. So trying to cut down on that.
A nice trick from the book Effective Python. See if it helps your situation.
def napalm(count):
for x in range(count):
logger.info('useless log line')
#contextmanager
def debug_logging(level):
logger = logging.getLogger()
old_level = logger.getEffectiveLevel()
logger.setLevel(level)
try:
yield
finally:
logger.setLevel(old_level)
napalm(5)
with debug_logging(logging.WARNING):
napalm(5)
napalm(5)
By calling logging.getLogger() without a parameter you are currently retrieving the root logger, and overriding the level for it also affects all other loggers.
You should instead retrieve the library's logger and override level only for that specific one.
The napalm library executes the following in its __init__.py:
logger = logging.getLogger("napalm")
i.e. the library logger's name is "napalm".
You should thus be able to override the level of that specific logger by putting the following line in your script:
logging.getLogger("napalm").setLevel(logging.WARNING)
Generic example:
import logging
logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
A = logging.getLogger("A")
B = logging.getLogger("B")
A.debug("#1 from A gets printed")
B.debug("#1 from B gets printed")
logging.getLogger("A").setLevel(logging.CRITICAL)
A.debug("#2 from A doesn't get printed") # because we've increased the level
B.debug("#2 from B gets printed")
Output:
DEBUG: #1 from A gets printed
DEBUG: #1 from B gets printed
DEBUG: #2 from B gets printed
Edit:
Since this didn't work well for you, it's probably because there's a bunch of various other loggers in this library:
$ grep -R 'getLogger' .
napalm/junos/junos.py:log = logging.getLogger(__file__)
napalm/base/helpers.py:logger = logging.getLogger(__name__)
napalm/base/clitools/cl_napalm.py:logger = logging.getLogger("napalm")
napalm/base/clitools/cl_napalm_validate.py:logger = logging.getLogger("cl_napalm_validate.py")
napalm/base/clitools/cl_napalm_test.py:logger = logging.getLogger("cl_napalm_test.py")
napalm/base/clitools/cl_napalm_configure.py:logger = logging.getLogger("cl-napalm-config.py")
napalm/__init__.py:logger = logging.getLogger("napalm")
napalm/pyIOSXR/iosxr.py:logger = logging.getLogger(__name__)
napalm/iosxr/iosxr.py:logger = logging.getLogger(__name__)
I would then resort to something like this (related: How to list all existing loggers using python.logging module):
# increase level for all loggers
for name, logger in logging.root.manager.loggerDict.items():
logger.setLevel(logging.WARNING)
or perhaps it will suffice if you just print the various loggers, identify the most noisy ones, and silence them one by one.
Overview
I want to use httpimport as a logging library common to several scripts. This module generates logs of its own which I do not know how to silence.
In other cases such as this one, I would have used
logging.getLogger('httpimport').setLevel(logging.ERROR)
but it did not work.
Details
The following code is a stub of the "common logging code" mentioned above:
# toconsole.py
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
formatter = logging.Formatter('%(asctime)s %(message)s')
handler_console = logging.StreamHandler()
level = logging.DEBUG if 'DEV' in os.environ else logging.INFO
handler_console.setLevel(level)
handler_console.setFormatter(formatter)
log.addHandler(handler_console)
# disable httpimport logging except for errors+
logging.getLogger('httpimport').setLevel(logging.ERROR)
A simple usage such as
import httpimport
httpimport.INSECURE = True
with httpimport.remote_repo(['githublogging'], 'http://localhost:8000/') :
from toconsole import log
log.info('yay!')
gives the following output
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
2019-08-25 13:56:48,671 yay!
yay!
The second (bare) yay! must be coming from httpimport, namely from its logging setup.
How can I disable the logging for such a module, or better - raise its level so that only errors+ are logged?
Note: this question was initially asked at the Issues section of the GitHub repository for httpimport but the author did not know either how to fix that.
Author of httpimport here.
I totally forgot I was using the basicConfig logger thing.
It is fixed in master right now (0.7.2) - will be included in next PyPI release:
https://github.com/operatorequals/httpimport/commit/ff2896c8f666c3f16b0f27716c732d68be018ef7
The reason why this is happening is because when you do import httpimport they do the initial configuration for the logging machinery. This happens right here. What this means is that the root logger already has a StreamHandler attached to it. Because of this, and the fact that all loggers inherit from the root logger, when you do log.info('yay') it not only uses your Handler and Formatter, but it also propagates all they way to the root logger, which also emits the message.
Remember that whoever calls basicConfig first when an application starts that sets up the default configuration for the root logger, which in turn, is inherited by all loggers, unless otherwise specified.
If you have a complex logging configuration you need to ensure that you call it before you do any third-party imports which might call basicConfig. basicConfig is idempotent meaning the first call seals the deal, and subsequent calls have no effect.
Solutions
You could do log.propagate = False and you will see that the 2nd yay will not show.
You could attach the Formatter directly to the already existent root Handler by doing something like this (without adding another Handler yourself)
root = logging.getLogger('')
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler = root.handlers[0]
root_handler.setFormatter(formatter)
You could do a basicConfig call when you initialize your application (if you had such a config available, with initial Formatters and Handlers, etc. that will elegantly attach everything to the root logger neatly) and then you would only do something like logger = logging.getLogger(__name__) and logger.info('some message') that would work the way you'd expect because it would propagate all the way to the root logger which already has your configuration.
You could remove the initial Handler that's present on the root logger by doing something like
root = logging.getLogger('')
root.handlers = []
... and many more solutions, but you get the idea.
Also do note that logging.getLogger('httpimport').setLevel(logging.ERROR) this works perfectly fine. No messages below logging.ERROR would be logged by that logger, it's just that the problem wasn't from there.
If you however want to completely disable a logger you can just do logger.disabled = True (also do note, again, that the problem wasn't from the httpimport logger, as aforementioned)
One example demonstrated
Change your toconsole.py with this and you won't see the second yay.
import logging
import os
log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)
root_logger = logging.getLogger('')
root_handler = root_logger.handlers[0]
formatter = logging.Formatter('%(asctime)s %(message)s')
root_handler.setFormatter(formatter)
# or you could just keep your old code and just add log.propagate = False
# or any of the above solutions and it would work
logging.getLogger('httpimport').setLevel(logging.ERROR)
I have written following code to enable Cloudwatch support.
import logging
from boto3.session import Session
from watchtower import CloudWatchLogHandler
logging.basicConfig(level=logging.INFO,format='[%(asctime)s.%(msecs).03d] [%(name)s,%(funcName)s:%(lineno)s] [%(levelname)s] %(message)s',datefmt='%d/%b/%Y %H:%M:%S')
log = logging.getLogger('Test')
boto3_session = Session(aws_access_key_id=AWS_ACCESS_KEY_ID,
aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
region_name=REGION_NAME)
cw_handler = CloudWatchLogHandler(log_group=CLOUDWATCH_LOG_GROUP_NAME,stream_name=CLOUDWATCH_LOG_STREAM_NAME,boto3_session=boto3_session)
log.addHandler(cw_handler)
Whenever i try to print any logger statement, i am getting different output on my local system and cloudwatch.
Example:
log.info("Hello world")
Output of above logger statement on my local system (terminal) :
[24/Feb/2019 15:25:06.969] [Test,<module>:1] [INFO] Hello world
Output of above logger statement on cloudwatch (log stream) :
Hello world
Is there something i am missing ?
In the Lambda execution environment, the root logger is already preconfigured. You'll have to work with it or work around it. You could do some of the following:
You can set the formatting directly on the root logger:
root = logging.getLogger()
root.setLevel(logging.INFO)
root.handlers[0].setFormatter(logging.Formatter(fmt='[%(asctime)s.%(msecs).03d] [%(name)s,%(funcName)s:%(lineno)s] [%(levelname)s] %(message)s', datefmt='%d/%b/%Y %H:%M:%S'))
You could add the Watchtower handler to it (disclaimer: I have not tried this approach):
root = logging.getLogger()
root.addHandler(cw_handler)
However I'm wondering if you even need to use Watchtower. In Lambda, every line you print to stdout (so even just using print) get logged to Cloudwatch. So using the standard logging interface might be sufficient.
This worked for me
import logging
import watchtower
watch = watchtower.CloudWatchLogHandler()
watch.setFormatter(fmt = logging.Formatter('%(levelname)s - %(module)s - %(message)s'))
logger = logging.getLogger()
logger.addHandler(watch)
As per comment in https://stackoverflow.com/a/45624044/1021819 (and another answer there),
just add force=True to logging.basicConfig(), so in your case you need
logging.basicConfig(level=logging.INFO, force=True, format='[%(asctime)s.%(msecs).03d] [%(name)s,%(funcName)s:%(lineno)s] [%(levelname)s] %(message)s',datefmt='%d/%b/%Y %H:%M:%S')
log = logging.getLogger('Test')
This function does nothing if the root logger already has handlers configured, unless the keyword argument force is set to True.
(i.e. AWS case)
Re: force:
If this keyword argument is specified as true, any existing handlers attached to the root logger are removed and closed, before carrying out the configuration as specified by the other arguments.
REF: https://docs.python.org/3/library/logging.html#logging.basicConfig
THANKS:
https://stackoverflow.com/a/72054516/1021819
https://stackoverflow.com/a/45624044/1021819
I'm seeing extra logging messages after I've imported a module I need to use. I'm trying to work out the correct way to stop this happening. The following code shows the issue best:
import os
import logging
import flickrapi
class someObject:
def __init__(self):
self.value = 1
logger = logging.getLogger(__name__)
print logger.handlers
logger.info("value = " + str(self.value))
def main():
# Set up logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter('[%(asctime)-15s] %(name)-8s %(levelname)-6s %message)s')
fh = logging.FileHandler(os.path.splitext(os.path.basename(__file__))[0]+".log")
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
logger.addHandler(fh)
ch = logging.StreamHandler()
ch.setLevel(logging.INFO)
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.debug("Debug message")
logger.info("Info message")
thingy = someObject()
if __name__ == "__main__":
main()
With the flickrapi import I see the following output:
DEBUG:__main__:Debug message
[2013-05-03 12:10:47,755] __main__ INFO Info message
INFO:__main__:Info message
[<logging.FileHandler instance at 0x1676dd0>, <logging.StreamHandler instance at 0x1676ea8>]
[2013-05-03 12:10:47,755] __main__ INFO value = 1
INFO:__main__:value = 1
With the flickrapi import removed I see the correct output:
[2013-05-03 12:10:47,755] __main__ INFO Info message
[<logging.FileHandler instance at 0x1676dd0>, <logging.StreamHandler instance at 0x1676ea8>]
[2013-05-03 12:10:47,755] __main__ INFO value = 1
This is my first time of using logging and it's got me a little stumped. I've read the documentation a couple of times but I think I'm missing something in my understanding.
Looking at logging.Logger.manager.loggerDict, there are other loggers but each of their .handlers is empty. The __main__ logger only has the two handlers I've added so where do these messages come from?
Any pointers as to how I can solve this would be much appreciated as I've hit a wall.
Thanks
This is a bug in the flickrapi library you are using. It is calling logging.basicConfig() in it's __init__.py which is the wrong thing to do for a library since it adds a StreamHandler defaulting to stderr to the root logger.
You should probably open a bug report with the author. There is a HOWTO in the python logging docs on how libraries should configure logging.
To work around this issue until the bug is fixed you should be able to do the following:
# at the top of your module before doing anything else
import flickrapi
import logging
try:
logging.root.handlers.pop()
except IndexError:
# once the bug is fixed in the library the handlers list will be empty - so we need to catch this error
pass