We Keep Coding

How to set dynamic log filename using yaml config

I have a config file in my python project:
version: 1
formatters:
logstashFormatter:
(): project.formatter.LogstashFormatter
message_type: logstash
tags: null
domainName: false
handlers:
file:
class: logging.handlers.WatchedFileHandler
level: DEBUG
formatter: logstashFormatter
filename: app.log
mode: a
root:
level: DEBUG
handlers:
- file
propagate: true
Is there any way to set dynamic filename (using timestamp or not)
to make each user write in his own file?

logging - merging multiple configuration files

I am working on a project where we have a core application that loads multiple plugins.
Each plugin has its own configuration file, and the core application has one as well.
We are using the excellent logging module from python's standard library.
The logging module includes the ability to load the logging configuration from an .ini file.
However, if you load another configuration file, the other files are discarded and only the new configuration is used.
What I would like to do is to split my logging configuration into multiple files, so that the application can load its own configuration file and then load each plugin's merging their logging configuration into the main one.
Note: fileConfig has an option called disable_existing_loggers that can be set to False. However, this only keeps existing loggers alive, but it still clears the internal map of handlers (which means that a plugin's configuration cannot use a handler defined in the application's config file).
I could merge the files manually to produce my own config, but I'd rather avoid that.
Thanks.
To make it clearer, I'd like to do something like this:
# application.ini
[loggers]
keys=root,app
[handlers]
keys=rootHandler,appHandler
[formatters]
keys=myformatter
[logger_root]
# stuff
[handler_rootHandler]
# stuff
[formatter_myformatter]
# stuff
...
# plugin.ini
[loggers]
keys=pluginLogger # no root logger
[handlers]
keys=pluginHandler # no root handler
# no formatters section
[logger_pluginLogger]
# stuff
formatter=myformatter # using the formatter from application.ini

I usually do this with logging.config.dictConfig and the pyYaml package. The package allows you to load the content of a configuration file as a dict object.
The only additional thing needed is a small helper class to handle configuration overwrites/add-ons:
import yaml
class Configuration(dict):
def __init__(self,
config_file_path=None,
overwrites=None):
with open(config_file_path) as config_file:
config = yaml.load(config_file)
super(Configuration, self).__init__(config)
if overwrites is not None:
for overwrite_key, value in overwrites.items():
self.apply_overwrite(self, overwrite_key, value)
def apply_overwrite(self, node, key, value):
if isinstance(value, dict):
for item in value:
self.apply_overwrite(node[key], item, value[item])
else:
node[key] = value
For example, if your main configuration is:
logger:
version: 1
disable_existing_loggers: False
formatters:
simple:
format: '%(levelname)s: Module: %(name)s Msg: %(message)s'
handlers:
file:
level: DEBUG
class: logging.handlers.RotatingFileHandler
maxBytes: 10000000
backupCount: 50
formatter: simple
filename: '/tmp/log1.log'
root:
handlers: [file]
level: DEBUG
and your overwrite is:
logger:
handlers:
file:
filename: '/tmp/log2.log'
you can get your overwritten logger like that:
from configuration import Configuration
from logging.config import dictConfig
import logging
if __name__ == '__main__':
config = Configuration('standard.yml', overwrites=Configuration('overwrite.yml'))
dictConfig(config['logger'])
logger = logging.getLogger(__name__)
logger.info('I logged it')

I couldn't find a way to do what I wanted, so I ended up rolling a class to do it.
Here it is as a convenient github gist.

Python logging and Pydev debugger?

Edit :
Using Liclipse 1.2.1 instead of 1.3.0 or 1.4.0 is working fine. Changelog indicate both Pydev 3.9.1 and Eclipse 4.4.1 updates for 1.3.0. Seems to break logging debug.
Using Liclipse and Pydev debugger (and CPython) with the following code sample, getting that error :
logging.config.dictConfig(config)
File "C:\Python27\lib\logging\config.py", line 794, in dictConfig
dictConfigClass(config).configure()
File "C:\Python27\lib\logging\config.py", line 576, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': 'DictConfigurator' object has no attribute 'startswith'
There is no problem without debugging, is the logging module require run environment and will only work on it ?
Here is the code sample used :
import logging.config
import yaml
def setup_logging():
default_path = 'logger.conf'
default_level = logging.DEBUG
if os.path.exists(default_path):
with open(default_path, 'rt') as f:
config = yaml.load(f.read())
logging.config.dictConfig(config)
else:
logging.basicConfig(level=default_level)
And here is my logger.conf :
version: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
lineInfo:
format: "%(asctime)s - Line: %(lineno)d - %(name)s - %(levelname)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: lineInfo
stream: ext://sys.stdout
debug_file_handler:
class: logging.handlers.RotatingFileHandler
level: DEBUG
formatter: lineInfo
filename: logs/debug.log
maxBytes: 10485760 # 10MB
backupCount: 10
encoding: utf8
info_file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: simple
filename: logs/info.log
maxBytes: 10485760 # 10MB
backupCount: 10
encoding: utf8
error_file_handler:
class: logging.handlers.RotatingFileHandler
level: ERROR
formatter: simple
filename: logs/errors.log
maxBytes: 10485760 # 10MB
backupCount: 10
encoding: utf8
root:
level: DEBUG
handlers: [console, info_file_handler, error_file_handler, debug_file_handler]
Thanks

There is now a relatively obscure (because they don't include the OP's error string in their webpage for good search indexing) fix implemented in PyCharm for this:
https://www.jetbrains.com/pycharm/help/python-debugger.html
In PyCharm, go to:
File | Settings | Build, Execution, Deployment | Python Debugger
Then uncheck 'PyQt compatible'

same problem with PyCharm.
possible workaround - to comment out pydev_monkey_qt.patch_qt() line in pycharm/helpers/pydev/pydevd.py, for Eclipse it should be located somewhere else

Ensure to import PyQt4 before importing logging

PyYaml parsing of the Environment variable in the Yaml configuration file

I need to read follow yaml-formatted configuration file:
version: 1
disable_existing_loggers: False
formatters:
precise:
format: "%(name)-15s # %(levelname)-8s # %(asctime)s # [Line: %(lineno)-3d]: %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: <%= ENV['ENV_PROJECT'] %>/target/tracing.log
encoding: utf-8
maxBytes : 10737418244
backupCount: 7
formatter: precise
loggers:
utility:
handlers: [file]
level: INFO
propagate: True
root:
handlers: [file]
level: INFO
But, instead off <%= ENV['ENV_PROJECT'] %> in the result string I need to get the relevant path, for example.
For the loading this file I use follow code:
from yaml import load
with open('test.yml', 'rt') as stream:
configuration = load(stream)
How can I get the required result?
Tnx.

You need to use 'resolver' and 'constructor' to achieve this. Here is one way to achieve this.
import yaml, os, re
#define the regex pattern that the parser will use to 'implicitely' tag your node
pattern = re.compile( r'^\<%= ENV\[\'(.*)\'\] %\>(.*)$' )
#now define a custom tag ( say pathex ) and associate the regex pattern we defined
yaml.add_implicit_resolver ( "!pathex", pattern )
#at this point the parser will associate '!pathex' tag whenever the node matches the pattern
#you need to now define a constructor that the parser will invoke
#you can do whatever you want with the node value
def pathex_constructor(loader,node):
value = loader.construct_scalar(node)
envVar, remainingPath = pattern.match(value).groups()
return os.environ[envVar] + remainingPath
#'register' the constructor so that the parser will invoke 'pathex_constructor' for each node '!pathex'
yaml.add_constructor('!pathex', pathex_constructor)
#that is it
data = """
version: 1
disable_existing_loggers: False
formatters:
precise:
format: "%(name)-15s # %(levelname)-8s # %(asctime)s # [Line: %(lineno)-3d]: %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: <%= ENV['ENV_PROJECT'] %>/target/tracing.log
encoding: utf-8
maxBytes : 10737418244
backupCount: 7
formatter: precise
loggers:
utility:
handlers: [file]
level: INFO
propagate: True
root:
handlers: [file]
level: INFO
"""
deSerializedData = yaml.load(data)
print(deSerializedData [ 'handlers'] [ 'file' ] ['filename'] )

Multiple instances and multiple destinations

I have multiple Python modules, each of which has separate log config files. I am using Yaml, so I just do a
log_config_dict=yaml.load(open(config_file1, 'r'))
logging.config.dictConfig(log_config_dict)
self.main_logger=logging.getLogger('Main_Logger')
In another module, I have something like
log_config_dict=yaml.load(open(config_file2, 'r'))
logging.config.dictConfig(log_config_dict)
self.main_logger=logging.getLogger('Poller_Main_Logger')
The 2 loggers are writing to separate log files. Then in the code for each separate module, I do logging as -
self.main_logger.info(log_str)
However this is not working as expected. Say I do logging from module1,then from module2, then from module1 again, the log messages are either written to module2's destination, or not written at all.
Any idea what is happening? Is the problem that each time I do a dictConfig call, previous loggers are disabled? Is there any way around this?
Below - one of the log config files
version: 1
formatters:
default_formatter:
format: '%(asctime)s : %(levelname)s : %(message)s'
datefmt: '%d-%b-%Y %H:%M:%S'
plain_formatter:
format: '%(message)s'
handlers:
console_default_handler:
class: logging.StreamHandler
level: INFO
formatter: default_formatter
stream: ext://sys.stdout
console_error_handler:
class: logging.StreamHandler
level: WARNING
formatter: default_formatter
stream: ext://sys.stderr
logfile_handler:
class: logging.FileHandler
filename: logger.txt
mode: a
formatter: default_formatter
level: DEBUG
errfile_handler:
class: logging.FileHandler
filename: error.txt
mode: a
formatter: default_formatter
level: WARNING
plain_handler:
class: logging.StreamHandler
level: DEBUG
formatter: plain_formatter
stream: ext://sys.stdout
loggers:
Poller_Main_Logger:
level: DEBUG
handlers: [console_default_handler,logfile_handler]
propagate: no
Plain_Logger:
level: DEBUG
handlers: [plain_handler]
propagate: no
Error_Logger:
level: WARNING
handlers: [errfile_handler,console_error_handler,logfile_handler]
propagate: no
root:
level: INFO
handlers: [console_default_handler]

logging doesn't support the usage pattern you want here. By default, rerunning logging.config.dictConfig() blows away all existing loggers, handlers, and formatters. There are the incremental and disable_existing_loggers options available for use in the config dict, but using incremental, you can't load in new handlers or formatters. You either need to combine you configuration in to a single file for your whole program, or use the mechanisms provided in the module to manually construct and add formatters and handlers to loggers.