We Keep Coding

Customize key value in python structured (json) logging from config file

I have to output my python job's logs as structured (json) format for our downstream datadog agent to pick them up. Crucially, I have requirements about what specific log fields are named, e.g. there must be a timestamp field which cannot be called e.g. asctime. So a desired log looks like:
{"timestamp": "2022-11-10 00:28:58,557", "name": "__main__", "level": "INFO", "message": "my message"}
I can get something very close to that with the following code:
import logging.config
from pythonjsonlogger import jsonlogger
logging.config.fileConfig("logging_config.ini", disable_existing_loggers=False)
logger = logging.getLogger(__name__)
logger.info("my message")
referencing the following logging_config.ini file:
[loggers]
keys = root
[handlers]
keys = consoleHandler
[formatters]
keys=json
[logger_root]
level=DEBUG
handlers=consoleHandler
[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=json
[formatter_json]
class = pythonjsonlogger.jsonlogger.JsonFormatter
format=%(asctime)s %(name)s - %(levelname)s:%(message)s
...however, this doesn't allow flexibility about the keys in the outputted log json objects. e.g. my timestamp object is called "asctime" as below:
{"asctime": "2022-11-10 00:28:58,557", "name": "__main__", "levelname": "INFO", "message": "my message"}
I still want that asctime value (e.g. 2022-11-10 00:28:58,557), but need it to be referenced by a key called "timestamp" instead of "asctime". If at all possible I would strongly prefer a solution that adapts the logging.config.ini file (or potentially a yaml logging config file) with relatively minimal extra python code itself.
I also tried this alternative python json logging library which I thought provided very simple and elegant code, but unfortunately when I tried to use that, I didn't get my log statement to output at all...

You'll need to have a small, minimal amount of Python code, something like
# in mymodule.py, say
class CustomJsonFormatter(jsonlogger.JsonFormatter):
def add_fields(self, log_record, record, message_dict):
super(CustomJsonFormatter, self).add_fields(log_record, record, message_dict)
log_record['timestamp'] = datetime.datetime.fromtimestamp(record.created).strftime('%Y-%m-%d %H:%M:%S') + f',{int(record.msecs)}'
and then change the configuration to reference it:
[formatter_json]
class = mymodule.CustomJsonFormatter
format=%(timestamp)s %(name)s - %(levelname)s:%(message)s
which would then output e.g.
{"timestamp": "2022-11-10 11:37:25,153", "name": "root", "levelname": "DEBUG", "message": "foo"}

Logging in python with exta

I have logging configuration in conf file, idea is to log to two different files in different formats (one is plain text, other is in json)
[handler_plainTextFileHandler]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=plainTextFormatter
args=('app/logs/log.log', 'midnight', 1, 10, None, False, False)
[handler_jsonFileHandler]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=jsonFormatter
args=('app/logs/json/jlog.log', 'midnight', 1, 10, None, False, False)
[formatter_jsonFormatter]
class=pythonjsonlogger.jsonlogger.JsonFormatter
format=%(asctime)s - %(name)s %(lineno)d - %(levelname)s - %(message)s
datefmt =%d.%m.%Y %H:%M:%S
[formatter_plainTextFormatter]
format=%(asctime)s - %(name)s(%(lineno)d) - %(levelname)s - [%(trace)s] %(message)s
datefmt =%d.%m.%Y %H:%M:%S
When I log something from my app, I send this trace parameter as extra
logger.info(f'Health check from {req.client.host}:{req.client.port}', extra={"trace": self.trace})
And everything is working as intended, logs are formatted as they should be and it's all as intended. When I have no trace parameter, I just send extra={"trace":None}
Problem happens when there is unhandled exception in my code that then gets logged by root logger. Since there is no extra in that call, error gets logged in json file but not in plain text file (I get error that param trace is missing)
Any ideas how to handle this, I would like to keep things as they are in normal logging, only add ability to log other errors in that same text file

How to get the app name in django logger

I have this formatter for django
FORMAT = "[%(filename)s:%(lineno)s - %(funcName)20s() ] %(message)s"
The file name i get is
views.py
Now that is confusing as its hard to see from which module is that views.py.
Is there any to get appname in logger formatter

Use pathname instead of filename in your logging configuration.
FORMAT = "[%(pathname)s:%(lineno)s - %(funcName)20s() ] %(message)s"
There are also other variables you can use — check the logging module documentation for a list.
Note that if you're acquiring a Logger instance using logger = logging.getLogger(__name__) (which is a common way to do it), you can also retrieve the module name (e.g. myapp.views) using name.
This is (arguably) better practice but will not work if you're doing e.g. logger = logging.getLogger("mylogger") or logger = logging.getLogger()

How to create daily log folder in python logging

I want to make the log file output into daily folder in python.
I can make the log path in hander like "../myapp/logs/20150514/xx.log" through current date.
But the problem is that the log path doesn't change when the date changes.
I create the log instance while i start my long-running python script xx.py, and now the instance's log path is "../myapp/logs/20150514/xx.log". But on tomorrow, as the instance is not changed, so its path is still "../myapp/logs/20150514/xx.log" which should be "../myapp/logs/20150515/xx.log".
How can i make the log output into daily folder?
My get log instance codes:
import os
import utils
import logging
from logging.handlers import RotatingFileHandler
import datetime
def getInstance(file=None):
global logMap
if file is None:
file = 'other/default.log'
else:
file = file + '.log'
if(logMap.has_key(file)):
return logMap.get(file)
else:
visit_date = datetime.date.today().strftime('%Y-%m-%d')
date_file = os.path.join(visit_date,file)
log_path = utils.read_from_ini('log_path').strip()
log_path = os.path.join(log_path,date_file);
if not os.path.isdir(os.path.dirname(log_path)):
os.makedirs(os.path.dirname(log_path))
logging.basicConfig(datefmt='%Y-%m-%d %H:%M:%S',level=logging.INFO)
log_format = '[%(asctime)s][%(levelname)s]%(filename)s==> %(message)s'
formatter = logging.Formatter(log_format)
log_file = RotatingFileHandler(log_path, maxBytes=10*1024*1024,backupCount=5)
log_file.setLevel(logging.INFO)
log_file.setFormatter(formatter)
instance = logging.getLogger(file)
instance.addHandler(log_file)
logMap[file] = instance
return instance

Your RotatingFileHandler doesn't rotate on a time basis, but rather a size basis. That's what the maxBytes argument is for. If you want to rotate based on time, use a TimedRotatingFileHandler instead. Note that this works with filenames, but not paths (as far as I know). You can have 20150505.log, 20150506.log, but not 20150505/mylog.log, 20150506/mylog.log.
If you want to rotate folder names you could probably do it by subclassing the TimedRotatingFileHandler and adding your own logic.

Python logging: how to represent newlines in the format string in a logging config file?

I'm configuring my Python logging from a file (see http://www.python.org/doc//current/library/logging.html#configuration-file-format ).
From the example on that page, i have a formatter in the config file that looks like:
[formatter_form01]
format=F1 %(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
How do i put a newline in the "format" string that specifies the formatter? Neither \n nor \\n work (e.g. format=F1\n%(asctime)s %(levelname)s %(message)s does not work). Thanks

The logging.config module reads config files with ConfigParser, which has support for multiline values.
So you can specify your format string like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter
Multilines values are continued by indenting the following lines (one or more spaces or tabs count as an indent).

The logging configuration file is based on the ConfigParser module. There you'll find you can solve it like this:
[formatter_form01]
format=F1
%(asctime)s %(levelname)s %(message)s
datefmt=
class=logging.Formatter

My best bet would be using a custom formatter (instead of logging.Formatter)... For reference, here's the source code for logging.Formatter.format:
def format(self, record):
record.message = record.getMessage()
if string.find(self._fmt,"%(asctime)") >= 0:
record.asctime = self.formatTime(record, self.datefmt)
s = self._fmt % record.__dict__
if record.exc_info:
# Cache the traceback text to avoid converting it multiple times
# (it's constant anyway)
if not record.exc_text:
record.exc_text = self.formatException(record.exc_info)
if record.exc_text:
if s[-1:] != "\n":
s = s + "\n"
s = s + record.exc_text
return s
It's pretty clear to me that, if self._fmt is read from a text file (single line), no escapping of any kind would be possible. Maybe you can extend from logging.Formatter, override this method and substitute the 4th line for something like:
s = self._fmt.replace('\\n', '\n') % record.__dict__
or something more general, if you want other things to be escaped as well.
EDIT: alternatively, you can do that in the init method, once (instead of every time a message is formatted). But as others already pointed out, the ConfigParser support multiple lines, so no need to go this route...

This might be an easy way:
import logging
logformat = """%(asctime)s ... here you get a new line
... %(thread)d .... here you get another new line
%(message)s"""
logging.basicConfig(format=logformat, level=logging.DEBUG)
I tested, the above setting gives two new lines for each logging message, as it shown in the codes. Note: %(asctime)s and things like this is python logging formatting strings.

import logging
logformat = "%(asctime)s %(message)s\n\r"
logging.basicConfig(level=logging.DEBUG, format=logformat,filename='debug.log', filemode='w')
logging.debug (Your String here)
Debug text in the file will be written with new line.

Just add "\n" before the closing apostrophe of basicConfig function
logging.basicConfig(level=logging.DEBUG, format=' %(levelname)s - %(message)s\n')