I am python beginner. My python script logs output to a file (say example.log) using the basic python logging module. However, my python script also makes some 3rd party API calls (for example, parse_the_file) over which I don't have any control. I want to capture the output (usually on console) produced by the API into my example.log. The following example code works partially but the problem is that the contents get overwritten as soon as I start logging output of the API into my log file.
#!/usr/bin/env python
import logging
import sys
import os
from common_lib import * # import additional modules
logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
logging.debug('This is a log message.') # message goes to log file.
sys.stdout = open('example.log','a')
metadata_func=parse_the_file('/opt/metadata.txt') # output goes to log file but OVERWRITES the content
sys.stdout = sys.__stdout__
logging.debug('This is a second log message.') # message goes to log file.
I know there have been post suggesting to similar question on this site but I haven't a workaround/solution for this problem that will work in this scenario.
Try:
log_file = open('example.log', 'a')
logging.basicConfig(stream=log_file, level=logging.DEBUG)
logging.debug("Test")
sys.stdout = log_file
sys.stderr = log_file
stdout_fd = os.dup(1)
stderr_fd = os.dup(2)
os.dup2(log_file.fileno(), 1)
os.dup2(log_file.fileno(), 2)
os.system("echo foo")
os.dup2(stdout_fd, 1)
os.dup2(stderr_fd, 2)
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__
However, this will not format it accordingly. If you want that, you can try something like http://plumberjack.blogspot.com/2009/09/how-to-treat-logger-like-output-stream.html
Related
I have problems with logging in Python 3.7 I use Spyder as editor
This code is working normally and creates a file and writes in it.
import logging
LOG_FORMAT="%(Levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="C:\\Users\\MOHAMED\\Desktop\\Learn python\\tst.log",
level=logging.DEBUG)
logger=logging.getLogger()
logger.info("Our first message.")
The problem is when I add the format in my file this code does not write anything in tst file.
import logging
LOG_FORMAT="%(Levelname)s %(asctime)s - %(message)s"
logging.basicConfig(filename="C:\\Users\\MOHAMED\\Desktop\\Learn python\\tst.log",
level=logging.DEBUG,
format=LOG_FORMAT)
logger=logging.getLogger()
logger.info("Our first message.")
You are specifying logging variable Levelname but you do not use the extra to populate the variable.
try it with
logger.info("Our first message.", extra={"Levelname":"test"})
also, recommend the docs https://docs.python.org/3/library/logging.html
I'm running python script in which I intend to save tensorflow logs (regardless of their severity) generated during the execution of the script, to a file so that they don't pop up on the console.
Here's the code snippet:
# Setting up Info Logger
import logging, os, sys
from logging.handlers import RotatingFileHandler
def setup_logger(fileName=None, level=logging.NOTSET):
if fileName is None:
fileName = u'./test.log'
FORMAT = '%(asctime)s -- %(name)s -- %(levelname)s -- %(message)s'
logger_file_handler = RotatingFileHandler(fileName, 'w')
logger_file_handler.setLevel(level)
formatter = logging.Formatter(FORMAT)
logger_file_handler.setFormatter(formatter)
tf_logger = logging.getLogger('tensorflow')
tf_logger.addHandler(logger_file_handler)
tf_logger.setLevel(logging.NOTSET)
tf_logger.propagate = False
logging.captureWarnings(True)
logger = logging.getLogger(__name__.split('.')[0])
warnings_logger = logging.getLogger('py.warnings')
logger.addHandler(logger_file_handler)
logger.setLevel(level)
warnings_logger.addHandler(logger_file_handler)
# Redirect all logs to disk
setup_logger(level=logging.DEBUG)
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
Problem
1) The script saves all logs except the ones of tensorflow.
2) If I comment out os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2', then I get the following line on the console:
2019-12-10 11:51:07.501527: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_100.dll
What I've tried
I found out that the TF_CPP environment variable sends these logs to sys.stderr (correct me if I'm wrong) which gets printed on the console.
But it may lead to instability as it'll redirect even the tracebacks to log file, if I try to log stderr.
What I expect
I intend to get the tensorflow log (like the above one) to appear in the log file.
Kindly give your valuable opinion/solution regarding this problem.
Thanks!
Preetkaran
Python Ver: 3.7.5
Tensorflow Ver: 2.0.0
Currently, this is what I have (testlog.py):
import logging
import logging.handlers
filename = "example.log"
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler = logging.handlers.RotatingFileHandler(filename, mode = 'w', backupCount = 5)
handler.setLevel(logging.DEBUG)
handler.setFormatter(formatter)
logger.addHandler(handler)
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
for i in range(10):
logger.debug("testx") #where I alternate x from 1 thru 9 to see output
It currently successfully prints out to the console and to example.log, which is what I want.
Every time I run it, it makes a new file and replaces the old example.log like so:
run with logger.debug("test1") - example.log will contain test1 10 times like it should.
run with logger.debug("test2") - it rewrites example.log to contain test2 10 times.
etc...
However, I would like for the code to make a new log file every time I run the program so that I have:
example.log
example.log1
example.log2
...
example.log5
In conclusion, I'd like for this file to print the log message to the console, to the log file, and I would like a new log file (up to *.5) whenever I run the program.
logging.handlers.RotatingFileHandler rotates your logs based either on size or on date, but you can force it to rotate using RotatingFileHandler.doRollover() so something like:
import logging.handlers
import os
filename = "example.log"
# your logging setup
should_roll_over = os.path.isfile(filename)
handler = logging.handlers.RotatingFileHandler(filename, mode='w', backupCount=5)
if should_roll_over: # log already exists, roll over!
handler.doRollover()
# the rest of your setup...
Should work like a charm.
If you change the mode to 'a' in the solution posted by zwer, then you get a single empty log file at the first run, after that it will work as intended. Just increase the backupCount by +1 :-)
Attempt [see it running here]:
from sys import stdout, stderr
from cStringIO import StringIO
from logging import getLogger, basicConfig, StreamHandler
basicConfig(format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M')
log = getLogger(__name__)
sio = StringIO()
console = StreamHandler(sio)
log.addHandler(console)
log.addHandler(StreamHandler(stdout))
log.info('Jackdaws love my big sphinx of quartz.')
print 'console.stream.read() = {!r}'.format(console.stream.read())
Output [stdout]:
console.stream.read() = ''
Expected output [stdout]:
[date] [filename] INFO Jackdaws love my big sphinx of quartz.
console.stream.read() = 'Jackdaws love my big sphinx of quartz.'
Two things going on here.
Firstly, the root logger is instantiated with a level of WARNING, which means that no messages with a level lower than WARNING will be processed. You can set the level using Logger.setLevel(level), where levels are defined here - https://docs.python.org/2/library/logging.html#levels.
As suggested in comments, the log level can also be set with:
basicConfig(level='INFO', ...)
Secondly, when you write to the StringIO object, the position in the stream is set at the end of the current stream. You need to rewind the StringIO object to then be able to read from it.
console.stream.seek(0)
console.stream.read()
Even simpler, just call:
console.stream.getvalue()
Full code:
from sys import stdout, stderr
from cStringIO import StringIO
from logging import getLogger, basicConfig, StreamHandler
basicConfig(format='%(asctime)s %(name)-12s %(levelname)-8s %(message)s',
datefmt='%m-%d %H:%M')
log = getLogger(__name__)
log.setLevel("INFO")
sio = StringIO()
console = StreamHandler(sio)
log.addHandler(console)
log.addHandler(StreamHandler(stdout))
log.info('Jackdaws love my big sphinx of quartz.')
console.stream.seek(0)
print 'console.stream.read() = {!r}'.format(console.stream.read())
I am experiencing an issue where I am using the logging module in my app. I am working in Eclipse against the LDT Python (Py 2.7) interface (rather than Pydev) on my MacBook Pro. The logging module works through Eclipse; however, when I transfer my app over to a RHEL5 2.7, logging does not seem to be working at all. It is not throwing any exceptions, it is just not logging anything to console or file (it creates the file though).
Code:
# Initialize logging
log = logging.getLogger('pepPrep')
# Log to stderr
console = logging.StreamHandler()
console.setLevel(logging.INFO)
# Log to file
logname = 'pepPrep.' + datetime.datetime.now().strftime("%Y%m%d_%H:%M") + '.log'
filelog = logging.FileHandler(logname)
filelog.setLevel(logging.DEBUG)
# set a format
formatter = logging.Formatter('%(name)-12s: %(levelname)-8s %(message)s')
# tell the handler to use this format
console.setFormatter(formatter)
filelog.setFormatter(formatter)
# add the handler to the root logger
log.addHandler(console)
log.addHandler(filelog)
log.INFO('This is a test')
log.DEBUG('This is a test2')
Any pointers on how I can make this work?
The default threshold for logging is WARNING, so INFO and DEBUG messages are not output by default. To do so, add e.g.
logging.getLogger().setLevel(logging.DEBUG)
to get DEBUG and INFO messages.
You can confirm this is your problem by doing
log.warning('This is a test3')
before adding that setLevel, and confirming that the warning is actually output.