We Keep Coding

How to set dynamic log filename using yaml config

I have a config file in my python project:
version: 1
formatters:
logstashFormatter:
(): project.formatter.LogstashFormatter
message_type: logstash
tags: null
domainName: false
handlers:
file:
class: logging.handlers.WatchedFileHandler
level: DEBUG
formatter: logstashFormatter
filename: app.log
mode: a
root:
level: DEBUG
handlers:
- file
propagate: true
Is there any way to set dynamic filename (using timestamp or not)
to make each user write in his own file?

Properly configuring file paths and names for python logger

I have the following .yaml file defining my configurations for the logger I want to use.
version: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s - %(module)12s - %(name)12s - %(levelname)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: debug_format
stream: ext://sys.stdout
info_file_handler:
class: logging.handlers.TimedRotatingFileHandler
level: INFO
formatter: simple
interval: 1
backupCount: 20
encoding: utf8
when: midnight
filename: Log/info.log
error_file_handler:
class: logging.handlers.RotatingFileHandler
level: ERROR
formatter: error_format
maxBytes: 10485760 # 10MB
backupCount: 20
encoding: utf8
filename: Log/errors.log
loggers:
dedalus_logger:
level: DEBUG
handlers: [console]
propagate: false
root:
level: INFO
handlers: [console, info_file_handler, error_file_handler]
The problem with this is that my RotatingFileHandler, and TimedRotatingFileHandler will name the files as info.log2, and info.log.2017.10.19,
How can I make it so the names are not set automatically like that but instead have a form that makes more sense, like info.2.log, and info.2017.10.19.log.
Lastly I am defining the filename as Log/info.log if I run my app like this
>>cd Module/src
>>python run.py
It will create the log files where I expect them inside Module/src/Log. If instead I run it like this though
>> python Module/src/run.py
it will try to create the log files in my current directory, which is not what I want, how can I properly configure the logger to fix these two issues?

Using program variables in python logging configuration file

I am trying to enable my python logging using the following:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import logging
import logging.config
import os
test_filename = 'my_log_file.txt'
try:
logging.config.fileConfig('loggingpy.conf', disable_existing_loggers=False)
except Exception as e:
# try to set up a default logger
logging.error("No loggingpy.conf to parse", exc_info=e)
logging.basicConfig(level=logging.WARNING, format="%(asctime)-15s %(message)s")
test1_log = logging.getLogger("test1")
test1_log.critical("test1_log crit")
test1_log.error("test1_log error")
test1_log.warning("test1_log warning")
test1_log.info("test1_log info")
test1_log.debug("test1_log debug")
I would like to use a loggingpy.conf file to control the logging like the following:
[loggers]
keys=root
[handlers]
keys=handRoot
[formatters]
keys=formRoot
[logger_root]
level=INFO
handlers=handRoot
[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(test_filename,)
[formatter_formRoot]
format=%(asctime)s:%(name)s:%(process)d:%(lineno)d %(levelname)s %(message)s
datefmt=
class=logging.Formatter
Here I am trying to route the logging to the file named by the local "test_filename". When I run this, I get:
ERROR:root:No loggingpy.conf to parse
Traceback (most recent call last):
File "logging_test.py", line 8, in <module>
logging.config.fileConfig('loggingpy.conf', disable_existing_loggers=False)
File "/usr/lib/python2.7/logging/config.py", line 85, in fileConfig
handlers = _install_handlers(cp, formatters)
File "/usr/lib/python2.7/logging/config.py", line 162, in _install_handlers
args = eval(args, vars(logging))
File "<string>", line 1, in <module>
NameError: name 'test_filename' is not defined
CRITICAL:test1:test1_log crit
ERROR:test1:test1_log error
WARNING:test1:test1_log warning
Reading the docs, it seems that the "args" value in the config is eval'd in the context of the logging package namespace rather than the context when fileConfig is called. Is there any decent way to try to get the logging to behave this way through a configuration file so I can configure a dynamic log filename (usually like "InputFile.log"), but still have the flexibility to use the logging config file to change it?

Even though it's an old question, I think this still has relevance. An alternative to the above mentioned solutions would be to use logging.config.dictConfig(...) and manipulating the dictionary.
MWE:
log_config.yml
version: 1
disable_existing_loggers: false
formatters:
default:
format: "%(asctime)s:%(name)s:%(process)d:%(lineno)d %(levelname)s %(message)s"
handlers:
console:
class: logging.StreamHandler
formatter: default
stream: ext://sys.stdout
level: DEBUG
file:
class: logging.FileHandler
formatter: default
filename: "{path}/service.log"
level: DEBUG
root:
level: DEBUG
handlers:
- file
- console
example.py
import logging.config
import sys
import yaml
log_output_path = sys.argv[1]
log_config = yaml.load(open("log_config.yml"))
log_config["handlers"]["file"]["filename"] = log_config["handlers"]["file"]["filename"].format(path = log_output_path)
logging.config.dictConfig(log_config)
logging.debug("test")
Executable as follows:
python example.py .
Result:
service.log file in current working directory contains one line of log message.
Console outputs one line of log message.
Both state something like this:
2016-06-06 20:56:56,450:root:12232:11 DEBUG test

You could place the filename in the logging namespace with:
logging.test_filename = 'my_log_file.txt'
Then your existing loggingpy.conf file should work
You should be able to pollute the logging namespace with anything you like (within reason - i wouldn't try logging.config = 'something') in your module and that should make it referencable by the the config file.

The args statement is parsed using eval at logging.config.py _install_handlers. So you can add code into the args.
[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(os.getenv("LOG_FILE","default_value"),)
Now you only need to populate the environment variable.

This is very hacky so I wouldn't recommend it. But if you for some reason did not want to add to the logging namespace you could pass the log file name through a command line argument and then use sys.argv[1] to access it (sys.argv[0] is the script name).
[handler_handRoot]
class=FileHandler
level=INFO
formatter=formRoot
args=(sys.argv[1],)

PyYaml parsing of the Environment variable in the Yaml configuration file

I need to read follow yaml-formatted configuration file:
version: 1
disable_existing_loggers: False
formatters:
precise:
format: "%(name)-15s # %(levelname)-8s # %(asctime)s # [Line: %(lineno)-3d]: %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: <%= ENV['ENV_PROJECT'] %>/target/tracing.log
encoding: utf-8
maxBytes : 10737418244
backupCount: 7
formatter: precise
loggers:
utility:
handlers: [file]
level: INFO
propagate: True
root:
handlers: [file]
level: INFO
But, instead off <%= ENV['ENV_PROJECT'] %> in the result string I need to get the relevant path, for example.
For the loading this file I use follow code:
from yaml import load
with open('test.yml', 'rt') as stream:
configuration = load(stream)
How can I get the required result?
Tnx.

You need to use 'resolver' and 'constructor' to achieve this. Here is one way to achieve this.
import yaml, os, re
#define the regex pattern that the parser will use to 'implicitely' tag your node
pattern = re.compile( r'^\<%= ENV\[\'(.*)\'\] %\>(.*)$' )
#now define a custom tag ( say pathex ) and associate the regex pattern we defined
yaml.add_implicit_resolver ( "!pathex", pattern )
#at this point the parser will associate '!pathex' tag whenever the node matches the pattern
#you need to now define a constructor that the parser will invoke
#you can do whatever you want with the node value
def pathex_constructor(loader,node):
value = loader.construct_scalar(node)
envVar, remainingPath = pattern.match(value).groups()
return os.environ[envVar] + remainingPath
#'register' the constructor so that the parser will invoke 'pathex_constructor' for each node '!pathex'
yaml.add_constructor('!pathex', pathex_constructor)
#that is it
data = """
version: 1
disable_existing_loggers: False
formatters:
precise:
format: "%(name)-15s # %(levelname)-8s # %(asctime)s # [Line: %(lineno)-3d]: %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: <%= ENV['ENV_PROJECT'] %>/target/tracing.log
encoding: utf-8
maxBytes : 10737418244
backupCount: 7
formatter: precise
loggers:
utility:
handlers: [file]
level: INFO
propagate: True
root:
handlers: [file]
level: INFO
"""
deSerializedData = yaml.load(data)
print(deSerializedData [ 'handlers'] [ 'file' ] ['filename'] )

Multiple instances and multiple destinations

I have multiple Python modules, each of which has separate log config files. I am using Yaml, so I just do a
log_config_dict=yaml.load(open(config_file1, 'r'))
logging.config.dictConfig(log_config_dict)
self.main_logger=logging.getLogger('Main_Logger')
In another module, I have something like
log_config_dict=yaml.load(open(config_file2, 'r'))
logging.config.dictConfig(log_config_dict)
self.main_logger=logging.getLogger('Poller_Main_Logger')
The 2 loggers are writing to separate log files. Then in the code for each separate module, I do logging as -
self.main_logger.info(log_str)
However this is not working as expected. Say I do logging from module1,then from module2, then from module1 again, the log messages are either written to module2's destination, or not written at all.
Any idea what is happening? Is the problem that each time I do a dictConfig call, previous loggers are disabled? Is there any way around this?
Below - one of the log config files
version: 1
formatters:
default_formatter:
format: '%(asctime)s : %(levelname)s : %(message)s'
datefmt: '%d-%b-%Y %H:%M:%S'
plain_formatter:
format: '%(message)s'
handlers:
console_default_handler:
class: logging.StreamHandler
level: INFO
formatter: default_formatter
stream: ext://sys.stdout
console_error_handler:
class: logging.StreamHandler
level: WARNING
formatter: default_formatter
stream: ext://sys.stderr
logfile_handler:
class: logging.FileHandler
filename: logger.txt
mode: a
formatter: default_formatter
level: DEBUG
errfile_handler:
class: logging.FileHandler
filename: error.txt
mode: a
formatter: default_formatter
level: WARNING
plain_handler:
class: logging.StreamHandler
level: DEBUG
formatter: plain_formatter
stream: ext://sys.stdout
loggers:
Poller_Main_Logger:
level: DEBUG
handlers: [console_default_handler,logfile_handler]
propagate: no
Plain_Logger:
level: DEBUG
handlers: [plain_handler]
propagate: no
Error_Logger:
level: WARNING
handlers: [errfile_handler,console_error_handler,logfile_handler]
propagate: no
root:
level: INFO
handlers: [console_default_handler]

logging doesn't support the usage pattern you want here. By default, rerunning logging.config.dictConfig() blows away all existing loggers, handlers, and formatters. There are the incremental and disable_existing_loggers options available for use in the config dict, but using incremental, you can't load in new handlers or formatters. You either need to combine you configuration in to a single file for your whole program, or use the mechanisms provided in the module to manually construct and add formatters and handlers to loggers.