I want to save my results as a log file, so I am thinking to import logging module. I understand that to output a file, the code is very straightforward.
logging.basicConfig(filename='logger.log', level=logging.INFO)
logging.debug('debug message')
logging.info('info message')
logging.warn('warn message')
logging.error('error message')
logging.critical('critical message')
However, what if I want to output multiple log files? for example, in the following for loop, each iteration will output a log file, how should I do this?
for i in range(1,10):
print (i)
#output a log file to save I value
I tried to use these code, but it's not working.
for i in range(1,10):
filename = str.format('mylog%d.txt' % i)
logging.basicConfig(format=log_fmt, level=logging.DEBUG, filename=filename)
logging.debug('This is debug message')
logging.info('This is info message')
logging.warning('This is warning message')
You're using the format function of strings incorrectly. You're trying to use string interpolation, which is an entirely different method of formatting strings. You should try something like this:
filename = 'mylog{0}.txt'.format(i)
The {0} explicitly states that you should take the first value passed to format. You could leave it as {} if you'd like - it makes no real difference.
About file name:
filename = str.format('mylog%d.txt' % i)
is equal to:
filename = 'mylog%d.txt' % i
For output to multiple files you can use multiple handlers.
Logging class that handle logging.
root_logger = logging.getLogger()
Return you root handler. You can add or remove handlers to logger.
root_logger.handlers
Contains list of handlers of root logger.
first = root_logger.handlers[0]
first.close()
root_logger.removeHandler(first)
remove first handler.
new_handler = logging.FileHandler(file_name)
formatter = logging.Formatter('%(asctime)s ' + ' %(message)s', '%H:%M:%S')
new_handler.setFormatter(formatter)
root_logger.addHandler(new_handler)
Add new formatter to root_handler.
You can output log to any files at the same time.
For more info read:
https://docs.python.org/2/library/logging.html
https://docs.python.org/2/howto/logging-cookbook.html
Related
I am trying to use a helper function for python logging. But this printed the wrong line no and function name in my log file. Is there a way to get the correct line from where the helper log fn was called?
log_format = "%(asctime)s %(lineno)d %(message)s"
helper log fn Looks like
def helper_log(msg, test_id):
print(test_id, msg)
logger.info(msg, extra={"test_id": test_id})
this shows the same line number in my log file, Irrespective of from where I am calling the helper_log function
i would like to add an dictionary as an additional object in the logs.
I have an dictionary which contains the 'message".
this is my approach:
import logging
message = {'message': body_dict['message']}
logger_with_message_details = logging.getLogger()
handler = logging.StreamHandler()
json_formatter = logging.Formatter({"message": "%(message)s"})
handler.setFormatter(json_formatter)
logger_with_message_details.addHandler(handler)
logger_with_message_details = logging.LoggerAdapter(log, message)
logger_with_message_details.info("Message details.")
logger_with_message_details.info("Message details extracted.", extra=message)
I tried with two logs, but no output is containing the object :(
fh = logging.FileHandler('example.log',delay = True)
fh.setLevel(logging.INFO)
Since delay is True, the file will never be written unless something is logged.
At that point, the first line in the file is the first record, and it will contain the asctime, levelname etc elements
Using python 2.7.10, is there a sane way to add a line (or two) the first time a record is written that don't include those elements?
I can just write to the file before using it for logging, but if I do that, I end up with logs empty but for the header.
The desired output might look like:
Using test.fil with option 7
2015-11-01 13:57:58,045 :log_example: INFO fn:main result:process 4 complete --7 knights said ni
2015-11-01 13:57:58,045 :log_example: INFO fn:main result:process 3 complete --3 bunnies attacked
Thanks,
Sub class the FileHandler to create your own custom FileHandleWithHeader as shown below:
import os
import logging
# Create a class that extends the FileHandler class from logging.FileHandler
class FileHandlerWithHeader(logging.FileHandler):
# Pass the file name and header string to the constructor.
def __init__(self, filename, header, mode='a', encoding=None, delay=0):
# Store the header information.
self.header = header
# Determine if the file pre-exists
self.file_pre_exists = os.path.exists(filename)
# Call the parent __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
# Write the header if delay is False and a file stream was created.
if not delay and self.stream is not None:
self.stream.write('%s\n' % header)
def emit(self, record):
# Create the file stream if not already created.
if self.stream is None:
self.stream = self._open()
# If the file pre_exists, it should already have a header.
# Else write the header to the file so that it is the first line.
if not self.file_pre_exists:
self.stream.write('%s\n' % self.header)
# Call the parent class emit function.
logging.FileHandler.emit(self, record)
# Create a logger and set the logging level.
logger = logging.getLogger("example")
logger.setLevel(logging.INFO)
# Create a file handler from our new FileHandlerWith Header class and set the
# logging level.
fh = FileHandlerWithHeader('example.log', 'This is my header', delay=True)
fh.setLevel(logging.INFO)
# Add formatter to the file handler.
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
# Add the handler to the logger.
logger.addHandler(fh)
# Since the constructor of the FileHandlerWithHeader was passed delay=True
# the file should not exist until the first log as long as the log file did
# not pre-exist.
print "Ready to write to the the example.log file."
raw_input("Press Enter to continue...")
# Send 3 logs to the logger.
logger.info("First line in the file")
logger.info("Second line in the file")
logger.info("Third line in the file")
# The log file should now be created and only have a header at the begining of
# the file.
print "The example.log file should exist and have a header."
This script should run as is in Python 2.7. If the "example.log" file already exists, it will not recreate the header.
This solution required knowledge of the logging source code found here
and general use of the python logging package found here.
I had a simpler idea. The following just uses a custom formatter. The first message formatted spits out a header record then after that just does normal formatting.
import logging
class FormatterWithHeader(logging.Formatter):
def __init__(self, header, fmt=None, datefmt=None, style='%'):
super().__init__(fmt, datefmt, style)
self.header = header # This is hard coded but you could make dynamic
# Override the normal format method
self.format = self.first_line_format
def first_line_format(self, record):
# First time in, switch back to the normal format function
self.format = super().format
return self.header + "\n" + self.format(record)
def test_logger():
logger = logging.getLogger("test")
logger.setLevel(logging.DEBUG)
formatter = FormatterWithHeader('First Line Only')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.info("This line will kick out a header first.")
logger.info("This line will *not* kick out a header.")
Currently I have everything getting logged to one logfile but I want to separate it out to multiple log files. I look at the logging in python documentation but they don't discuss about this.
log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=(os.path.join(OUT_DIR, + '-user.log')),
format=log_format, level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S')
Currently this is how I do the logging. what I want to do have different type of errors or information get log into different log files. At the moment when I do logging.info('Logging IN') and logging.error('unable to login') will go to same logfile. I want to seperate them. Do I need to create another logging object to support the logging into another file?
What you /could/ do (I haven't dug into the logging module too much so there may be a better way to do this) is maybe use a stream rather than a file object:
In [1]: class LogHandler(object):
...: def write(self, msg):
...: print 'a :%s' % msg
...: print 'b :%s' % msg
...:
In [3]: import logging
In [4]: logging.basicConfig(stream=LogHandler())
In [5]: logging.critical('foo')
a :CRITICAL:root:foo
b :CRITICAL:root:foo
In [6]: logging.warn('bar')
a :WARNING:root:bar
b :WARNING:root:bar
Edit with further handling:
Assuming your log files already exist, you could do something like this:
import logging
class LogHandler(object):
format = '%(levelname)s %(message)s'
files = {
'ERROR': 'error.log',
'CRITICAL': 'error.log',
'WARN': 'warn.log',
}
def write(self, msg):
type_ = msg[:msg.index(' ')]
with open(self.files.get(type_, 'log.log'), 'r+') as f:
f.write(msg)
logging.basicConfig(format=LogHandler.format, stream=LogHandler())
logging.critical('foo')
This would allow you to split your logging into various files based on conditions in your log messages. If what you're looking for isn't found, it simply defaults to log.log.
I created this solution from docs.python.org/2/howto/logging-cookbook.html
Simply create two logging file handlers, assign their logging level and add them to your logger.
import os
import logging
current_path = os.path.dirname(os.path.realpath(__file__))
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
#to log debug messages
debug_log = logging.FileHandler(os.path.join(current_path, 'debug.log'))
debug_log.setLevel(logging.DEBUG)
#to log errors messages
error_log = logging.FileHandler(os.path.join(current_path, 'error.log'))
error_log.setLevel(logging.ERROR)
logger.addHandler(debug_log)
logger.addHandler(error_log)
logger.debug('This message should go in the debug log')
logger.info('and so should this message')
logger.warning('and this message')
logger.error('This message should go in both the debug log and the error log')
I'm configuring my Python logging from a file (see http://www.python.org/doc//current/library/logging.html#configuration-file-format ).
From the example on that page, i have a formatter in the config file that looks like:
[formatter_form01]
format=F1 %(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
How do i put a newline in the "format" string that specifies the formatter? Neither \n nor \\n work (e.g. format=F1\n%(asctime)s %(levelname)s %(message)s does not work). Thanks
The logging.config module reads config files with ConfigParser, which has support for multiline values.
So you can specify your format string like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
Multilines values are continued by indenting the following lines (one or more spaces or tabs count as an indent).
The logging configuration file is based on the ConfigParser module. There you'll find you can solve it like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
My best bet would be using a custom formatter (instead of logging.Formatter)... For reference, here's the source code for logging.Formatter.format:
def format(self, record):
record.message = record.getMessage()
if string.find(self._fmt,"%(asctime)") >= 0:
record.asctime = self.formatTime(record, self.datefmt)
s = self._fmt % record.__dict__
if record.exc_info:
# Cache the traceback text to avoid converting it multiple times
# (it's constant anyway)
if not record.exc_text:
record.exc_text = self.formatException(record.exc_info)
if record.exc_text:
if s[-1:] != "\n":
s = s + "\n"
s = s + record.exc_text
return s
It's pretty clear to me that, if self._fmt is read from a text file (single line), no escapping of any kind would be possible. Maybe you can extend from logging.Formatter, override this method and substitute the 4th line for something like:
s = self._fmt.replace('\\n', '\n') % record.__dict__
or something more general, if you want other things to be escaped as well.
EDIT: alternatively, you can do that in the init method, once (instead of every time a message is formatted). But as others already pointed out, the ConfigParser support multiple lines, so no need to go this route...
This might be an easy way:
import logging
logformat = """%(asctime)s ... here you get a new line
... %(thread)d .... here you get another new line
%(message)s"""
logging.basicConfig(format=logformat, level=logging.DEBUG)
I tested, the above setting gives two new lines for each logging message, as it shown in the codes. Note: %(asctime)s and things like this is python logging formatting strings.
import logging
logformat = "%(asctime)s %(message)s\n\r"
logging.basicConfig(level=logging.DEBUG, format=logformat,filename='debug.log', filemode='w')
logging.debug (Your String here)
Debug text in the file will be written with new line.
Just add "\n" before the closing apostrophe of basicConfig function
logging.basicConfig(level=logging.DEBUG, format=' %(levelname)s - %(message)s\n')