We Keep Coding

Logging to file and console

I have a Python script and I want that the info method write the messages in the console. But the warning, critical or error writes the messages to a file. How can I do that?
I tried this:
import logging
console_log = logging.getLogger("CONSOLE")
console_log.setLevel(logging.INFO)
stream_handler = logging.StreamHandler()
stream_handler.setLevel(logging.INFO)
console_log.addHandler(stream_handler)
file_log = logging.getLogger("FILE")
file_log.setLevel(logging.WARNING)
file_handler = logging.FileHandler('log.txt')
file_handler.setLevel(logging.WARNING)
file_log.addHandler(file_handler)
def log_to_console(message):
console_log.info(message)
def log_to_file(message):
file_log.warning(message)
log_to_console("THIS SHOULD SHOW ONLY IN CONSOLE")
log_to_file("THIS SHOULD SHOW ONLY IN FILE")
but the message that should be only in the file is going to the console too, and the message that should be in the console, is duplicating. What am I doing wrong here?

What happens is that the two loggers you created propagated the log upwards to the root logger. The root logger does not have any handlers by default, but will use the lastResort handler if needed:
A "handler of last resort" is available through this attribute. This
is a StreamHandler writing to sys.stderr with a level of WARNING, and
is used to handle logging events in the absence of any logging
configuration. The end result is to just print the message to
sys.stderr.
Source from the Python documentation.
Inside the Python source code, you can see where the call is done.
Therefore, to solve your problem, you could set the console_log and file_log loggers' propagate attribute to False.
On another note, I think you should refrain from instantiating several loggers for you use case. Just use one custom logger with 2 different handlers that will each log to a different destination.
Create a custom StreamHandler to log only the specified level:
import logging
class MyStreamHandler(logging.StreamHandler):
def emit(self, record):
if record.levelno == self.level:
# this ensures this handler will print only for the specified level
super().emit(record)
Then, use it:
my_custom_logger = logging.getLogger("foobar")
my_custom_logger.propagate = False
my_custom_logger.setLevel(logging.INFO)
stream_handler = MyStreamHandler()
stream_handler.setLevel(logging.INFO)
file_handler = logging.FileHandler("log.txt")
file_handler.setLevel(logging.WARNING)
my_custom_logger.addHandler(stream_handler)
my_custom_logger.addHandler(file_handler)
my_custom_logger.info("THIS SHOULD SHOW ONLY IN CONSOLE")
my_custom_logger.warning("THIS SHOULD SHOW ONLY IN FILE")
And it works without duplicate and without misplaced log.

python logging - different level for specific function

i'm trying to reduce the amount of logging that the napalm library sends to syslog, but also allow for info logs to be sent from other parts of the code. I set up logging.basicConfig to be INFO but then i'd like the napalm function to be WARNING and above.
So i have code like this:
from napalm import get_network_driver
import logging
import getpass
logging.basicConfig(
filename="/var/log/myscripts/script.log", level=logging.INFO, format="%(asctime)s %(message)s")
def napalm(device):
logging.getLogger().setLevel(logging.WARNING)
username = getpass.getuser()
driver = get_network_driver("junos")
router = driver(str(device), username, "", password="", timeout=120)
router.open()
return router
router = napalm('myrouter')
config = "hostname foobar"
router.load_merge_candidate(config=config)
show = router.compare_config()
logging.info(show)
The issue is the logging.info output never makes it to the log file. If i do logging.warning(show) it does, but i'd like this to be info. The reason i want the function to be WARNING is that it generates so much other logging at the info level that is just noise. So trying to cut down on that.

A nice trick from the book Effective Python. See if it helps your situation.
def napalm(count):
for x in range(count):
logger.info('useless log line')
#contextmanager
def debug_logging(level):
logger = logging.getLogger()
old_level = logger.getEffectiveLevel()
logger.setLevel(level)
try:
yield
finally:
logger.setLevel(old_level)
napalm(5)
with debug_logging(logging.WARNING):
napalm(5)
napalm(5)

By calling logging.getLogger() without a parameter you are currently retrieving the root logger, and overriding the level for it also affects all other loggers.
You should instead retrieve the library's logger and override level only for that specific one.
The napalm library executes the following in its __init__.py:
logger = logging.getLogger("napalm")
i.e. the library logger's name is "napalm".
You should thus be able to override the level of that specific logger by putting the following line in your script:
logging.getLogger("napalm").setLevel(logging.WARNING)
Generic example:
import logging
logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
A = logging.getLogger("A")
B = logging.getLogger("B")
A.debug("#1 from A gets printed")
B.debug("#1 from B gets printed")
logging.getLogger("A").setLevel(logging.CRITICAL)
A.debug("#2 from A doesn't get printed") # because we've increased the level
B.debug("#2 from B gets printed")
Output:
DEBUG: #1 from A gets printed
DEBUG: #1 from B gets printed
DEBUG: #2 from B gets printed
Edit:
Since this didn't work well for you, it's probably because there's a bunch of various other loggers in this library:
$ grep -R 'getLogger' .
napalm/junos/junos.py:log = logging.getLogger(__file__)
napalm/base/helpers.py:logger = logging.getLogger(__name__)
napalm/base/clitools/cl_napalm.py:logger = logging.getLogger("napalm")
napalm/base/clitools/cl_napalm_validate.py:logger = logging.getLogger("cl_napalm_validate.py")
napalm/base/clitools/cl_napalm_test.py:logger = logging.getLogger("cl_napalm_test.py")
napalm/base/clitools/cl_napalm_configure.py:logger = logging.getLogger("cl-napalm-config.py")
napalm/__init__.py:logger = logging.getLogger("napalm")
napalm/pyIOSXR/iosxr.py:logger = logging.getLogger(__name__)
napalm/iosxr/iosxr.py:logger = logging.getLogger(__name__)
I would then resort to something like this (related: How to list all existing loggers using python.logging module):
# increase level for all loggers
for name, logger in logging.root.manager.loggerDict.items():
logger.setLevel(logging.WARNING)
or perhaps it will suffice if you just print the various loggers, identify the most noisy ones, and silence them one by one.

Python logging module not logging info even after level is set

I'm trying to do some basic logging. I'm doing the following:
log = logging.getLogger('test')
log.error('test') # 'test' gets logged
log.info('test') # Nothing gets logged, as expected
log.setLevel(logging.INFO) # Now info should be logged, right?
log.info('test') # No output
log.setLevel(logging.INFO) # Getting desperate
log.info('test') # Still nothing
What aspect of the logging module am I missing?

You have forgot to add handlers i.e. logging.basicConfig(), can you check if following code works for you.
import logging
logging.basicConfig()
log = logging.getLogger('test')
log.setLevel(logging.INFO)
log.error('test')
log.info('test')

You need to configure logging before you use it like below. You can configure it beautifully in python.
logging.basicConfig(format='%(asctime)s %(name)25s %(lineno)4d %(levelname)8s: %(message)s', level=40)
above statement will print timestamp, name of the file, line number, level type, message
level of logging also can be set during basic Config itself (40 = ERROR)

It is not necessary to call basicConfig.
However, the issue is about handlers (as pointed out by #Mahesh Karia).
You see, if you don't define some handlers you are dependent on whatever is already "in place" for you and that raises lots of confusion.
Check this:
You create your "test" logger
>>> log = logging.getLogger('test')
which prints WARNING messages but does not print INFO messages even after setting the level to INFO:
>>> log.warning('hi')
hi
>>> log.setLevel(logging.INFO)
>>> log.info('hi')
>>>
(personally, I think this is very wrong behaviour)
It can be "solved" by manipulating "handlers".
Your "test" logger has none
>>> log.handlers
[]
nor does it parent, the "root logger"
>>> log.parent
<RootLogger root (WARNING)>
>>> log.parent.handlers
[]
however, one can add a handler easy enough (to the root logger)
>>> log.parent.addHandler(logging.StreamHandler())
and now your "test" logger "works":
>>> log.info('hi')
hi
PS. Was using Python 3.8.2 prompt after importing logging module.

How should I configure logging in a script and then use that configuration in only my modules?

I want to find out how logging should be organised given that I write many scripts and modules that should feature similar logging. I want to be able to set the logging appearance and the logging level from the script and I want this to propagate the appearance and level to my modules and only my modules.
An example script could be something like the following:
import logging
import technicolor
import example_2_module
def main():
verbose = True
global log
log = logging.getLogger(__name__)
logging.root.addHandler(technicolor.ColorisingStreamHandler())
# logging level
if verbose:
logging.root.setLevel(logging.DEBUG)
else:
logging.root.setLevel(logging.INFO)
log.info("example INFO message in main")
log.debug("example DEBUG message in main")
example_2_module.function1()
if __name__ == '__main__':
main()
An example module could be something like the following:
import logging
log = logging.getLogger(__name__)
def function1():
print("printout of function 1")
log.info("example INFO message in module")
log.debug("example DEBUG message in module")
You can see that in the module there is minimal infrastructure written to import the logging of the appearance and the level set in the script. This has worked fine, but I've encountered a problem: other modules that have logging. This can result in output being printed twice, and very detailed debug logging from modules that are not my own.
How should I code this such that the logging appearance/level is set from the script but then used only by my modules?

You need to set the propagate attribute to False so that the log message does not propagate to ancestor loggers. Here is the documentation for Logger.propagate -- it defaults to True. So just:
import logging
log = logging.getLogger(__name__)
log.propagate = False

Setting the log level causes fabric to complain with 'No handlers could be found for logger "ssh.transport"'

The following script:
#!/usr/bin/env python
from fabric.api import env, run
import logging
logging.getLogger().setLevel(logging.INFO)
env.host_string = "%s#%s:%s" % ('myuser', 'myhost', '22')
res = run('date', pty = False)
Produces the following output:
[myuser#myhost:22] run: date
No handlers could be found for logger "ssh.transport"
[myuser#myhost:22] out: Thu Mar 29 16:15:15 CEST 2012
I would like to get rid of this annoying error message: No handlers could be found for logger "ssh.transport"
The problem happens when setting the log level (setLevel).
How can I solve this? I need to set the log level, so skipping that won't help.

You need to initialize the logging system. You can make the error go away by doing so in your app thusly:
import logging
logging.basicConfig( level=logging.INFO )
Note: this uses the default Formatter, which is not terribly useful. You might consider something more like:
import logging
FORMAT="%(name)s %(funcName)s:%(lineno)d %(message)s"
logging.basicConfig(format=FORMAT, level=logging.INFO)

My hack is ugly but works:
# This is here to avoid the mysterious messages: 'No handlers could be found for logger "ssh.transport"'
class MyNullHandler(logging.Handler):
def emit(self, record):
pass
bugfix_loggers = { }
def bugfix(name):
global bugfix_loggers
if not name in bugfix_loggers:
# print "Setting dummy logger for '%s'" % (name)
logging.getLogger(name).addHandler(MyNullHandler())
bugfix_loggers[name] = True