Unexpected python logger output when using several handlers with different log levels - python

I am trying to log data to stderr and into a file. The file should contain all log messages, and to stderr should go only the log level configured on the command line. This is described several times in the logging howto - but it does not seem to work for me. I have created a small test script which illustrates my problem:
#!/usr/bin/env python
import logging as l
l.basicConfig(level=100)
logger = l.getLogger("me")
# ... --- === SEE THIS LINE === --- ...
logger.setLevel(l.CRITICAL)
sh = l.StreamHandler()
sh.setLevel(l.ERROR)
sh.setFormatter(l.Formatter('%(levelname)-8s CONSOLE %(message)s'))
logger.addHandler(sh)
fh = l.FileHandler("test.dat", "w")
fh.setLevel(l.DEBUG)
fh.setFormatter(l.Formatter('%(levelname)-8s FILE %(message)s'))
logger.addHandler(fh)
logger.info("hi this is INFO")
logger.error("well this is ERROR")
In line 5th code line I can go for logger.setLevel(l.CRITICAL) or logger.setLevel(l.DEBUG). Both results are unsatisfying.
With logger.setLevel(l.CRITICAL) I get ...
$ python test.py
$ cat test.dat
$
Now with logger.setLevel(l.DEBUG) I get ...
$ python test.py
INFO:me:hi this is INFO
ERROR CONSOLE well this is ERROR
ERROR:me:well this is ERROR
$ cat test.dat
INFO FILE hi this is INFO
ERROR FILE well this is ERROR
$
In one case I see nothing nowhere, in the other I see everything everywhere, and one message is being displayed even twice on the console.
Now I get where the ERROR CONSOLE and ERROR FILE outputs come from, those I expect. I don't get where the INFO:me... or ERROR:me... outputs are coming from, and I would like to get rid of them.
Things I already tried:
Creating a filter as described here: https://stackoverflow.com/a/7447596/902327 (does not work)
Emptying handlers from the logger with logger.handlers = [] (also does not work)
Can somebody help me out here? It seems like a straightforward requirement and I really don't seem to get it.

You can set the root level to DEBUG, set propagate to False and then set the appropriate level for the other handlers.
import logging as l
l.basicConfig()
logger = l.getLogger("me")
# ... --- === SEE THIS LINE === --- ...
logger.setLevel(l.DEBUG)
logger.propagate = False
sh = l.StreamHandler()
sh.setLevel(l.ERROR)
sh.setFormatter(l.Formatter('%(levelname)-8s CONSOLE %(message)s'))
logger.addHandler(sh)
fh = l.FileHandler("test.dat", "w")
fh.setLevel(l.INFO)
fh.setFormatter(l.Formatter('%(levelname)-8s FILE %(message)s'))
logger.addHandler(fh)
logger.info("hi this is INFO")
logger.error("well this is ERROR")
Output:
~$ python test.py
ERROR CONSOLE well this is ERROR
~$ cat test.dat
INFO FILE hi this is INFO
ERROR FILE well this is ERROR

Related

Python: How to color logs while printing to a file?

A little context to what I am doing. I am running some python scripts through a different programming language on an industrial controller. Since I am not running the python scripts directly I can't watch any print or log statements from the terminal so I need to send the detailed logs to a log file.
Since we are logging a lot of information when debugging, I wanted to find a way to color the log file such as coloredlogs does to logs printed to terminal. I looked at coloredlogs but it appears that it can only print colored logs to files when using VIM. Does anyone know a way to print colored logs to a file using python that can be opened with a program such as wordpad? (maybe a .rtf file).
It can be a solution to use the Windows PowerShell Get-Content function to print a file which contains ANSI escape sequences to color the log.
For example:
import coloredlogs
import logging
# Create a logger object.
logger = logging.getLogger(__name__)
# Create a filehandler object
fh = logging.FileHandler('spam.log')
fh.setLevel(logging.DEBUG)
# Create a ColoredFormatter to use as formatter for the FileHandler
formatter = coloredlogs.ColoredFormatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
logger.addHandler(fh)
# Install the coloredlogs module on the root logger
coloredlogs.install(level='DEBUG')
logger.debug("this is a debugging message")
logger.info("this is an informational message")
logger.warning("this is a warning message")
logger.error("this is an error message")
logger.critical("this is a critical message")
When opening a Windows PowerShell you can use Get-Content .\spam.log to print the logs in color.

Python logging does not produce newline after each log (Linux)

I have used the same Python script on Windows which worked fine and produced several logs during each time it was run. The problem is when I ran the script on Linux the logging produced all of the logs onto one line.
I have tried \n in different places such as in the formatter, in each line itself.
This is how the logging is set up:
# This is the setup of the logging system for the program.
logger = logging.getLogger(__name__)
# Sets the name of the log file to 'login.log'
handler = logging.FileHandler(config.path['logger'])
# Sets up the format of the log file: Time, Function Name, Error Level (e.g. Warning Info, Critical),
# and then the message that follows the format.
formatter = logging.Formatter('%(asctime)-5s %(funcName)-20s %(levelname)-10s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
# Sets the lowest level of messages to info.
logger.setLevel(logging.INFO)
And here is how each log is made:
logger.warning('%-15s' % client + ' Failed: Logout Error')
Thanks in advance

python logging not saving to file

I have been stuck on this for the past hour. I had plugged logging into my tkinter gui, but could not get it to work. I then started removing parts until I got at the bare bones example in the very python docs and it will not work. At this point I have nothing else to remove.
The code is as follows:
import logging
LOG_FILENAME = r'logging_example.out'
logging.basicConfig(filename=LOG_FILENAME ,level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
f = open(LOG_FILENAME, 'rt')
try:
body = f.read()
finally:
f.close()
print('FILE:')
print (body)
The warning is printed to stdout, but the file is not generated.
I am runing python 3.4, x64 on a windows 7. It is a anacondas distribution, so this is running in Ipython inside spyder.
I guess this should be working
As Jonas Byström noted, this does work outside Ipython. It seems that Ipython configures a logging handler before I get the chance to do so. Also, basicConfig will do nothing if a handler is already present. So, in order to have it working in Ipython, one must do one of three things: 1) Add a new handler, OR 2)reload logging, OR 3) remove existing handlers. I did number 2 bellow.
import logging
from imp import reload
reload(logging)
LOG_FILENAME = r'logging_example.out'
logging.basicConfig(filename=LOG_FILENAME ,level=logging.DEBUG)
logging.debug('This message should go to the log file')
logging.info('So should this')
logging.warning('And this, too')
f = open(LOG_FILENAME, 'rt')
try:
body = f.read()
finally:
f.close()
print('FILE:')
print (body)
See theese for more information:
Logging in ipython;
More on the same

Selenium unit tests in Python -- where is my log file?

So I exported some unit tests from the Selenium IDE to Python. Now I'm trying to debug something, and I've noticed that Selenium uses the logging module. There is one particular line in selenium.webdriver.remote.remote_connection that I would really like to see the output of. It is:
LOGGER.debug('%s %s %s' % (method, url, data))
At the top of the file is another line that reads:
LOGGER = logging.getLogger(__name__)
So where is this log file? I want to look at it.
In your unit test script, place
import logging
logging.basicConfig(filename = log_filename, level = logging.DEBUG)
where log_filename is a path to wherever you'd like the log file written to.
Without the call to logging.basicConfig or some such call to setup a logging handler, the LOGGER.debug command does nothing.

Python logging module logs on Mac, but not Linux

I am experiencing an issue where I am using the logging module in my app. I am working in Eclipse against the LDT Python (Py 2.7) interface (rather than Pydev) on my MacBook Pro. The logging module works through Eclipse; however, when I transfer my app over to a RHEL5 2.7, logging does not seem to be working at all. It is not throwing any exceptions, it is just not logging anything to console or file (it creates the file though).
Code:
# Initialize logging
log = logging.getLogger('pepPrep')
# Log to stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# Log to file
logname = 'pepPrep.' + datetime.datetime.now().strftime("%Y%m%d_%H:%M") + '.log'
filelog = logging.FileHandler(logname)
filelog.setLevel(logging.DEBUG)
# set a format
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
filelog.setFormatter(formatter)
# add the handler to the root logger
log.addHandler(console)
log.addHandler(filelog)
log.INFO('This is a test')
log.DEBUG('This is a test2')
Any pointers on how I can make this work?
The default threshold for logging is WARNING, so INFO and DEBUG messages are not output by default. To do so, add e.g.
logging.getLogger().setLevel(logging.DEBUG)
to get DEBUG and INFO messages.
You can confirm this is your problem by doing
log.warning('This is a test3')
before adding that setLevel, and confirming that the warning is actually output.

Categories