We Keep Coding

How to implement json format logs in python

I have a below piece of code for python logging and would want to convert the logs into json format for better accessibility of information. How can I convert them into JSON format?
import os
import logging
log_fmt = ("%(asctime)-s %(levelname)-s %(message)s")
logger = logging.getLogger()
logger.setLevel(os.environ.get('LOG_LEVEL', 'INFO'))
logger.info(f"this is a test")
And the output looks like "2022-04-20 17:40:31,332 INFO this is a test"
How can I format this into a json object so I can access by keys?
Desired output:
{
"time": "2022-04-20 17:40:31,332",
"level": "INFO",
"message": "this is a test"
}

You could use the Python JSON Logger
But if you don't want to, or can't do that, then your log format string should be...
log_fmt = ("{\"time\": %(asctime)-s, \"level\": %(levelname)-s, \"message\": %(message)s},")
You'll end up with an extra comma at the end of the log file that you can programatically remove later. Or, you can do this if you want the comma at the top of the file...
log_fmt = (",{\"time\": %(asctime)-s, \"level\": %(levelname)-s, \"message\": %(message)s}")
But the json will look better in an editor with the comma at the end of every line.
If you provide a mechanism for users to download, or otherwise access log files, then you can do the trailing comma cleanup there, before you send the log file to the user.

python logging: is it possible to add module name to formatter

It is possible to get a logger by module name. Like this:
logging.getLogger(module_name)
I would like to add module_name to every log record. Is it possible to set up a Formatter object which adds module_name?

You are looking for the %(name)s parameter; add that to your formatter pattern:
FORMAT = "%(name)s: %(message)s"
logging.basicConfig(format=FORMAT)
or when creating a Formatter():
FORMAT = "%(name)s: %(message)s"
formatter = logging.Formatter(fmt=FORMAT)
See the LogRecord attributes reference:
Attribute name: name
Format: %(name)s
Description: Name of the logger used to log the call.

when you initialize the logger (only need to do this once for the app) try this config
logging.basicConfig(
filename='var/bpextract.log',
level=logging.INFO,
format='%(asctime)s %(process)-7s %(module)-20s %(message)s',
datefmt='%m/%d/%Y %H:%M:%S'
)
...later in your code...
log = logging.getLogger("bpextract")
log.info('###### Starting BPExtract App #####')

In logging.basicConfig, you can specify the format:
logging.basicConfig(format='%(name)s\t%(message)s')

How to Customize the time format using logging.config.fileConfig()

I am using python logging module to write to my log files:
logging.config.fileConfig(log_conf_file)
The formatter section in my log_conf_file "logging.conf" looks like:
[formatter_mylog]
format: %(asctime)s %(message)s
This makes my log looks like:
2013-05-02 13:39:55,325 mesagebody blablablabla
How I can customize the asctime format to output something like:
May 02 13:39:55 mesage_body blablablabla
I know if it's in python code, you can do something like:
# create formatter
formatter = logging.Formatter("%(asctime)s;%(message)s",
"%Y-%m-%d %H:%M:%S")
But how do you do it via the logging configuration file ?

Try this:
[formatter_mylog]
format: %(asctime)s %(message)s
datefmt=%m-%d %H:%M:%S
Not sure if this will change the month to the verbal style.
Update:
To implement the verbal style of the month try to use %b (source)

Below should give your expected output
[formatter_mylog]
format: %(asctime)s %(message)s
datefmt=%b %d %H:%M:%S

Python: Log to multiple log files

Currently I have everything getting logged to one logfile but I want to separate it out to multiple log files. I look at the logging in python documentation but they don't discuss about this.
log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=(os.path.join(OUT_DIR, + '-user.log')),
format=log_format, level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S')
Currently this is how I do the logging. what I want to do have different type of errors or information get log into different log files. At the moment when I do logging.info('Logging IN') and logging.error('unable to login') will go to same logfile. I want to seperate them. Do I need to create another logging object to support the logging into another file?

What you /could/ do (I haven't dug into the logging module too much so there may be a better way to do this) is maybe use a stream rather than a file object:
In [1]: class LogHandler(object):
...: def write(self, msg):
...: print 'a :%s' % msg
...: print 'b :%s' % msg
...:
In [3]: import logging
In [4]: logging.basicConfig(stream=LogHandler())
In [5]: logging.critical('foo')
a :CRITICAL:root:foo
b :CRITICAL:root:foo
In [6]: logging.warn('bar')
a :WARNING:root:bar
b :WARNING:root:bar
Edit with further handling:
Assuming your log files already exist, you could do something like this:
import logging
class LogHandler(object):
format = '%(levelname)s %(message)s'
files = {
'ERROR': 'error.log',
'CRITICAL': 'error.log',
'WARN': 'warn.log',
}
def write(self, msg):
type_ = msg[:msg.index(' ')]
with open(self.files.get(type_, 'log.log'), 'r+') as f:
f.write(msg)
logging.basicConfig(format=LogHandler.format, stream=LogHandler())
logging.critical('foo')
This would allow you to split your logging into various files based on conditions in your log messages. If what you're looking for isn't found, it simply defaults to log.log.

I created this solution from docs.python.org/2/howto/logging-cookbook.html
Simply create two logging file handlers, assign their logging level and add them to your logger.
import os
import logging
current_path = os.path.dirname(os.path.realpath(__file__))
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
#to log debug messages
debug_log = logging.FileHandler(os.path.join(current_path, 'debug.log'))
debug_log.setLevel(logging.DEBUG)
#to log errors messages
error_log = logging.FileHandler(os.path.join(current_path, 'error.log'))
error_log.setLevel(logging.ERROR)
logger.addHandler(debug_log)
logger.addHandler(error_log)
logger.debug('This message should go in the debug log')
logger.info('and so should this message')
logger.warning('and this message')
logger.error('This message should go in both the debug log and the error log')

Python logging: how to represent newlines in the format string in a logging config file?

I'm configuring my Python logging from a file (see http://www.python.org/doc//current/library/logging.html#configuration-file-format ).
From the example on that page, i have a formatter in the config file that looks like:
[formatter_form01]
format=F1 %(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
How do i put a newline in the "format" string that specifies the formatter? Neither \n nor \\n work (e.g. format=F1\n%(asctime)s %(levelname)s %(message)s does not work). Thanks

The logging.config module reads config files with ConfigParser, which has support for multiline values.
So you can specify your format string like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
Multilines values are continued by indenting the following lines (one or more spaces or tabs count as an indent).

The logging configuration file is based on the ConfigParser module. There you'll find you can solve it like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter

My best bet would be using a custom formatter (instead of logging.Formatter)... For reference, here's the source code for logging.Formatter.format:
def format(self, record):
record.message = record.getMessage()
if string.find(self._fmt,"%(asctime)") >= 0:
record.asctime = self.formatTime(record, self.datefmt)
s = self._fmt % record.__dict__
if record.exc_info:
# Cache the traceback text to avoid converting it multiple times
# (it's constant anyway)
if not record.exc_text:
record.exc_text = self.formatException(record.exc_info)
if record.exc_text:
if s[-1:] != "\n":
s = s + "\n"
s = s + record.exc_text
return s
It's pretty clear to me that, if self._fmt is read from a text file (single line), no escapping of any kind would be possible. Maybe you can extend from logging.Formatter, override this method and substitute the 4th line for something like:
s = self._fmt.replace('\\n', '\n') % record.__dict__
or something more general, if you want other things to be escaped as well.
EDIT: alternatively, you can do that in the init method, once (instead of every time a message is formatted). But as others already pointed out, the ConfigParser support multiple lines, so no need to go this route...

This might be an easy way:
import logging
logformat = """%(asctime)s ... here you get a new line
... %(thread)d .... here you get another new line
%(message)s"""
logging.basicConfig(format=logformat, level=logging.DEBUG)
I tested, the above setting gives two new lines for each logging message, as it shown in the codes. Note: %(asctime)s and things like this is python logging formatting strings.

import logging
logformat = "%(asctime)s %(message)s\n\r"
logging.basicConfig(level=logging.DEBUG, format=logformat,filename='debug.log', filemode='w')
logging.debug (Your String here)
Debug text in the file will be written with new line.

Just add "\n" before the closing apostrophe of basicConfig function
logging.basicConfig(level=logging.DEBUG, format=' %(levelname)s - %(message)s\n')