Currently I have everything getting logged to one logfile but I want to separate it out to multiple log files. I look at the logging in python documentation but they don't discuss about this.
log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=(os.path.join(OUT_DIR, + '-user.log')),
format=log_format, level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S')
Currently this is how I do the logging. what I want to do have different type of errors or information get log into different log files. At the moment when I do logging.info('Logging IN') and logging.error('unable to login') will go to same logfile. I want to seperate them. Do I need to create another logging object to support the logging into another file?
What you /could/ do (I haven't dug into the logging module too much so there may be a better way to do this) is maybe use a stream rather than a file object:
In [1]: class LogHandler(object):
...: def write(self, msg):
...: print 'a :%s' % msg
...: print 'b :%s' % msg
...:
In [3]: import logging
In [4]: logging.basicConfig(stream=LogHandler())
In [5]: logging.critical('foo')
a :CRITICAL:root:foo
b :CRITICAL:root:foo
In [6]: logging.warn('bar')
a :WARNING:root:bar
b :WARNING:root:bar
Edit with further handling:
Assuming your log files already exist, you could do something like this:
import logging
class LogHandler(object):
format = '%(levelname)s %(message)s'
files = {
'ERROR': 'error.log',
'CRITICAL': 'error.log',
'WARN': 'warn.log',
}
def write(self, msg):
type_ = msg[:msg.index(' ')]
with open(self.files.get(type_, 'log.log'), 'r+') as f:
f.write(msg)
logging.basicConfig(format=LogHandler.format, stream=LogHandler())
logging.critical('foo')
This would allow you to split your logging into various files based on conditions in your log messages. If what you're looking for isn't found, it simply defaults to log.log.
I created this solution from docs.python.org/2/howto/logging-cookbook.html
Simply create two logging file handlers, assign their logging level and add them to your logger.
import os
import logging
current_path = os.path.dirname(os.path.realpath(__file__))
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
#to log debug messages
debug_log = logging.FileHandler(os.path.join(current_path, 'debug.log'))
debug_log.setLevel(logging.DEBUG)
#to log errors messages
error_log = logging.FileHandler(os.path.join(current_path, 'error.log'))
error_log.setLevel(logging.ERROR)
logger.addHandler(debug_log)
logger.addHandler(error_log)
logger.debug('This message should go in the debug log')
logger.info('and so should this message')
logger.warning('and this message')
logger.error('This message should go in both the debug log and the error log')
Related
i would like to add an dictionary as an additional object in the logs.
I have an dictionary which contains the 'message".
this is my approach:
import logging
message = {'message': body_dict['message']}
logger_with_message_details = logging.getLogger()
handler = logging.StreamHandler()
json_formatter = logging.Formatter({"message": "%(message)s"})
handler.setFormatter(json_formatter)
logger_with_message_details.addHandler(handler)
logger_with_message_details = logging.LoggerAdapter(log, message)
logger_with_message_details.info("Message details.")
logger_with_message_details.info("Message details extracted.", extra=message)
I tried with two logs, but no output is containing the object :(
I have a Python library which I want to use structured logging for. I've been targeting python-json-logger, which automatically converts a dict passed to the extra kwarg of a logger to JSON:
In [1]: import logging
In [2]: from pythonjsonlogger import jsonlogger
In [3]: logger = logging.getLogger()
In [4]: handler = logging.StreamHandler()
In [5]: formatter = jsonlogger.JsonFormatter()
In [6]: handler.setFormatter(formatter)
In [7]: logger.addHandler(handler)
In [8]: logger.warning("test", extra={"username": "jashugan"})
{"message": "test", "username": "jashugan"}
This means that if the application using my library would like JSON-formatted logs, they can configure their logger like above.
However, if the application using my library doesn't do any configuration, then the information I'm storing in extra gets completely lost:
In [1]: import logging
In [2]: logger = logging.getLogger()
In [3]: logger.warning("test", extra={"username": "jashugan"})
test
Is there a way that I could use extra to create structured logs and have the information show up automatically if the application doesn't configure logging to render it specifically?
I want to save my results as a log file, so I am thinking to import logging module. I understand that to output a file, the code is very straightforward.
logging.basicConfig(filename='logger.log', level=logging.INFO)
logging.debug('debug message')
logging.info('info message')
logging.warn('warn message')
logging.error('error message')
logging.critical('critical message')
However, what if I want to output multiple log files? for example, in the following for loop, each iteration will output a log file, how should I do this?
for i in range(1,10):
print (i)
#output a log file to save I value
I tried to use these code, but it's not working.
for i in range(1,10):
filename = str.format('mylog%d.txt' % i)
logging.basicConfig(format=log_fmt, level=logging.DEBUG, filename=filename)
logging.debug('This is debug message')
logging.info('This is info message')
logging.warning('This is warning message')
You're using the format function of strings incorrectly. You're trying to use string interpolation, which is an entirely different method of formatting strings. You should try something like this:
filename = 'mylog{0}.txt'.format(i)
The {0} explicitly states that you should take the first value passed to format. You could leave it as {} if you'd like - it makes no real difference.
About file name:
filename = str.format('mylog%d.txt' % i)
is equal to:
filename = 'mylog%d.txt' % i
For output to multiple files you can use multiple handlers.
Logging class that handle logging.
root_logger = logging.getLogger()
Return you root handler. You can add or remove handlers to logger.
root_logger.handlers
Contains list of handlers of root logger.
first = root_logger.handlers[0]
first.close()
root_logger.removeHandler(first)
remove first handler.
new_handler = logging.FileHandler(file_name)
formatter = logging.Formatter('%(asctime)s ' + ' %(message)s', '%H:%M:%S')
new_handler.setFormatter(formatter)
root_logger.addHandler(new_handler)
Add new formatter to root_handler.
You can output log to any files at the same time.
For more info read:
https://docs.python.org/2/library/logging.html
https://docs.python.org/2/howto/logging-cookbook.html
I have this formatter for django
FORMAT = "[%(filename)s:%(lineno)s - %(funcName)20s() ] %(message)s"
The file name i get is
views.py
Now that is confusing as its hard to see from which module is that views.py.
Is there any to get appname in logger formatter
Use pathname instead of filename in your logging configuration.
FORMAT = "[%(pathname)s:%(lineno)s - %(funcName)20s() ] %(message)s"
There are also other variables you can use — check the logging module documentation for a list.
Note that if you're acquiring a Logger instance using logger = logging.getLogger(__name__) (which is a common way to do it), you can also retrieve the module name (e.g. myapp.views) using name.
This is (arguably) better practice but will not work if you're doing e.g. logger = logging.getLogger("mylogger") or logger = logging.getLogger()
I'm configuring my Python logging from a file (see http://www.python.org/doc//current/library/logging.html#configuration-file-format ).
From the example on that page, i have a formatter in the config file that looks like:
[formatter_form01]
format=F1 %(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
How do i put a newline in the "format" string that specifies the formatter? Neither \n nor \\n work (e.g. format=F1\n%(asctime)s %(levelname)s %(message)s does not work). Thanks
The logging.config module reads config files with ConfigParser, which has support for multiline values.
So you can specify your format string like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
Multilines values are continued by indenting the following lines (one or more spaces or tabs count as an indent).
The logging configuration file is based on the ConfigParser module. There you'll find you can solve it like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
My best bet would be using a custom formatter (instead of logging.Formatter)... For reference, here's the source code for logging.Formatter.format:
def format(self, record):
record.message = record.getMessage()
if string.find(self._fmt,"%(asctime)") >= 0:
record.asctime = self.formatTime(record, self.datefmt)
s = self._fmt % record.__dict__
if record.exc_info:
# Cache the traceback text to avoid converting it multiple times
# (it's constant anyway)
if not record.exc_text:
record.exc_text = self.formatException(record.exc_info)
if record.exc_text:
if s[-1:] != "\n":
s = s + "\n"
s = s + record.exc_text
return s
It's pretty clear to me that, if self._fmt is read from a text file (single line), no escapping of any kind would be possible. Maybe you can extend from logging.Formatter, override this method and substitute the 4th line for something like:
s = self._fmt.replace('\\n', '\n') % record.__dict__
or something more general, if you want other things to be escaped as well.
EDIT: alternatively, you can do that in the init method, once (instead of every time a message is formatted). But as others already pointed out, the ConfigParser support multiple lines, so no need to go this route...
This might be an easy way:
import logging
logformat = """%(asctime)s ... here you get a new line
... %(thread)d .... here you get another new line
%(message)s"""
logging.basicConfig(format=logformat, level=logging.DEBUG)
I tested, the above setting gives two new lines for each logging message, as it shown in the codes. Note: %(asctime)s and things like this is python logging formatting strings.
import logging
logformat = "%(asctime)s %(message)s\n\r"
logging.basicConfig(level=logging.DEBUG, format=logformat,filename='debug.log', filemode='w')
logging.debug (Your String here)
Debug text in the file will be written with new line.
Just add "\n" before the closing apostrophe of basicConfig function
logging.basicConfig(level=logging.DEBUG, format=' %(levelname)s - %(message)s\n')