I need to read follow yaml-formatted configuration file:
version: 1
disable_existing_loggers: False
formatters:
precise:
format: "%(name)-15s # %(levelname)-8s # %(asctime)s # [Line: %(lineno)-3d]: %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: <%= ENV['ENV_PROJECT'] %>/target/tracing.log
encoding: utf-8
maxBytes : 10737418244
backupCount: 7
formatter: precise
loggers:
utility:
handlers: [file]
level: INFO
propagate: True
root:
handlers: [file]
level: INFO
But, instead off <%= ENV['ENV_PROJECT'] %> in the result string I need to get the relevant path, for example.
For the loading this file I use follow code:
from yaml import load
with open('test.yml', 'rt') as stream:
configuration = load(stream)
How can I get the required result?
Tnx.
You need to use 'resolver' and 'constructor' to achieve this. Here is one way to achieve this.
import yaml, os, re
#define the regex pattern that the parser will use to 'implicitely' tag your node
pattern = re.compile( r'^\<%= ENV\[\'(.*)\'\] %\>(.*)$' )
#now define a custom tag ( say pathex ) and associate the regex pattern we defined
yaml.add_implicit_resolver ( "!pathex", pattern )
#at this point the parser will associate '!pathex' tag whenever the node matches the pattern
#you need to now define a constructor that the parser will invoke
#you can do whatever you want with the node value
def pathex_constructor(loader,node):
value = loader.construct_scalar(node)
envVar, remainingPath = pattern.match(value).groups()
return os.environ[envVar] + remainingPath
#'register' the constructor so that the parser will invoke 'pathex_constructor' for each node '!pathex'
yaml.add_constructor('!pathex', pathex_constructor)
#that is it
data = """
version: 1
disable_existing_loggers: False
formatters:
precise:
format: "%(name)-15s # %(levelname)-8s # %(asctime)s # [Line: %(lineno)-3d]: %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
file:
class: logging.handlers.RotatingFileHandler
filename: <%= ENV['ENV_PROJECT'] %>/target/tracing.log
encoding: utf-8
maxBytes : 10737418244
backupCount: 7
formatter: precise
loggers:
utility:
handlers: [file]
level: INFO
propagate: True
root:
handlers: [file]
level: INFO
"""
deSerializedData = yaml.load(data)
print(deSerializedData [ 'handlers'] [ 'file' ] ['filename'] )
Related
I would like to add celery TaskFormatter to my login handlers.
It does work when I configure it in the below code:
from celery.app.log import TaskFormatter
LOGGER = logging.getLogger()
sh = logging.StreamHandler()
sh.setFormatter(TaskFormatter('%(asctime)s - %(task_id)s - %(task_name)s - %(name)s - %(levelname)s - %(message)s'))
LOGGER.setLevel(logging.INFO)
LOGGER.addHandler(sh)
But when I try to configure it in the yaml file (I use pyyaml)
I got:
ValueError: Unable to configure formatter 'celery_formatter'
My logger.yaml
version: 1
formatters:
celery_formatter:
class: 'celery.app.log.TaskFormatter'
format: '%(asctime)s - %(task_id)s - %(task_name)s - %(name)s - %(levelname)s - %(message)s'
datefmt: '%Y-%m-%d %H:%M'
handlers:
stdout_handler:
class: logging.StreamHandler
level: INFO
formatter: celery_formatter
stream: ext://sys.stdout
loggers:
app_logger:
level: DEBUG
handlers: [file, stdout_handler, stderr_handler]
propagate: no
What am I missing?
According to the documentation, you can only specify format and datefmt keys to formatters item that will be used to construct Formatter instance. It seems that you can't provide a custom formatter class using class.
You can however provide '()' to provide your custom class, see here. The relevant part of you config would then become
formatters:
celery_formatter:
(): celery.app.log.TaskFormatter
format: '%(asctime)s - %(task_id)s - %(task_name)s - %(name)s - %(levelname)s - %(message)s'
datefmt: '%Y-%m-%d %H:%M'
I have a config file in my python project:
version: 1
formatters:
logstashFormatter:
(): project.formatter.LogstashFormatter
message_type: logstash
tags: null
domainName: false
handlers:
file:
class: logging.handlers.WatchedFileHandler
level: DEBUG
formatter: logstashFormatter
filename: app.log
mode: a
root:
level: DEBUG
handlers:
- file
propagate: true
Is there any way to set dynamic filename (using timestamp or not)
to make each user write in his own file?
I have the following .yaml file defining my configurations for the logger I want to use.
version: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s - %(module)12s - %(name)12s - %(levelname)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: debug_format
stream: ext://sys.stdout
info_file_handler:
class: logging.handlers.TimedRotatingFileHandler
level: INFO
formatter: simple
interval: 1
backupCount: 20
encoding: utf8
when: midnight
filename: Log/info.log
error_file_handler:
class: logging.handlers.RotatingFileHandler
level: ERROR
formatter: error_format
maxBytes: 10485760 # 10MB
backupCount: 20
encoding: utf8
filename: Log/errors.log
loggers:
dedalus_logger:
level: DEBUG
handlers: [console]
propagate: false
root:
level: INFO
handlers: [console, info_file_handler, error_file_handler]
The problem with this is that my RotatingFileHandler, and TimedRotatingFileHandler will name the files as info.log2, and info.log.2017.10.19,
How can I make it so the names are not set automatically like that but instead have a form that makes more sense, like info.2.log, and info.2017.10.19.log.
Lastly I am defining the filename as Log/info.log if I run my app like this
>>cd Module/src
>>python run.py
It will create the log files where I expect them inside Module/src/Log. If instead I run it like this though
>> python Module/src/run.py
it will try to create the log files in my current directory, which is not what I want, how can I properly configure the logger to fix these two issues?
Edit :
Using Liclipse 1.2.1 instead of 1.3.0 or 1.4.0 is working fine. Changelog indicate both Pydev 3.9.1 and Eclipse 4.4.1 updates for 1.3.0. Seems to break logging debug.
Using Liclipse and Pydev debugger (and CPython) with the following code sample, getting that error :
logging.config.dictConfig(config)
File "C:\Python27\lib\logging\config.py", line 794, in dictConfig
dictConfigClass(config).configure()
File "C:\Python27\lib\logging\config.py", line 576, in configure
'%r: %s' % (name, e))
ValueError: Unable to configure handler 'console': 'DictConfigurator' object has no attribute 'startswith'
There is no problem without debugging, is the logging module require run environment and will only work on it ?
Here is the code sample used :
import logging.config
import yaml
def setup_logging():
default_path = 'logger.conf'
default_level = logging.DEBUG
if os.path.exists(default_path):
with open(default_path, 'rt') as f:
config = yaml.load(f.read())
logging.config.dictConfig(config)
else:
logging.basicConfig(level=default_level)
And here is my logger.conf :
version: 1
disable_existing_loggers: False
formatters:
simple:
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
lineInfo:
format: "%(asctime)s - Line: %(lineno)d - %(name)s - %(levelname)s - %(message)s"
handlers:
console:
class: logging.StreamHandler
level: DEBUG
formatter: lineInfo
stream: ext://sys.stdout
debug_file_handler:
class: logging.handlers.RotatingFileHandler
level: DEBUG
formatter: lineInfo
filename: logs/debug.log
maxBytes: 10485760 # 10MB
backupCount: 10
encoding: utf8
info_file_handler:
class: logging.handlers.RotatingFileHandler
level: INFO
formatter: simple
filename: logs/info.log
maxBytes: 10485760 # 10MB
backupCount: 10
encoding: utf8
error_file_handler:
class: logging.handlers.RotatingFileHandler
level: ERROR
formatter: simple
filename: logs/errors.log
maxBytes: 10485760 # 10MB
backupCount: 10
encoding: utf8
root:
level: DEBUG
handlers: [console, info_file_handler, error_file_handler, debug_file_handler]
Thanks
There is now a relatively obscure (because they don't include the OP's error string in their webpage for good search indexing) fix implemented in PyCharm for this:
https://www.jetbrains.com/pycharm/help/python-debugger.html
In PyCharm, go to:
File | Settings | Build, Execution, Deployment | Python Debugger
Then uncheck 'PyQt compatible'
same problem with PyCharm.
possible workaround - to comment out pydev_monkey_qt.patch_qt() line in pycharm/helpers/pydev/pydevd.py, for Eclipse it should be located somewhere else
Ensure to import PyQt4 before importing logging
I have multiple Python modules, each of which has separate log config files. I am using Yaml, so I just do a
log_config_dict=yaml.load(open(config_file1, 'r'))
logging.config.dictConfig(log_config_dict)
self.main_logger=logging.getLogger('Main_Logger')
In another module, I have something like
log_config_dict=yaml.load(open(config_file2, 'r'))
logging.config.dictConfig(log_config_dict)
self.main_logger=logging.getLogger('Poller_Main_Logger')
The 2 loggers are writing to separate log files. Then in the code for each separate module, I do logging as -
self.main_logger.info(log_str)
However this is not working as expected. Say I do logging from module1,then from module2, then from module1 again, the log messages are either written to module2's destination, or not written at all.
Any idea what is happening? Is the problem that each time I do a dictConfig call, previous loggers are disabled? Is there any way around this?
Below - one of the log config files
version: 1
formatters:
default_formatter:
format: '%(asctime)s : %(levelname)s : %(message)s'
datefmt: '%d-%b-%Y %H:%M:%S'
plain_formatter:
format: '%(message)s'
handlers:
console_default_handler:
class: logging.StreamHandler
level: INFO
formatter: default_formatter
stream: ext://sys.stdout
console_error_handler:
class: logging.StreamHandler
level: WARNING
formatter: default_formatter
stream: ext://sys.stderr
logfile_handler:
class: logging.FileHandler
filename: logger.txt
mode: a
formatter: default_formatter
level: DEBUG
errfile_handler:
class: logging.FileHandler
filename: error.txt
mode: a
formatter: default_formatter
level: WARNING
plain_handler:
class: logging.StreamHandler
level: DEBUG
formatter: plain_formatter
stream: ext://sys.stdout
loggers:
Poller_Main_Logger:
level: DEBUG
handlers: [console_default_handler,logfile_handler]
propagate: no
Plain_Logger:
level: DEBUG
handlers: [plain_handler]
propagate: no
Error_Logger:
level: WARNING
handlers: [errfile_handler,console_error_handler,logfile_handler]
propagate: no
root:
level: INFO
handlers: [console_default_handler]
logging doesn't support the usage pattern you want here. By default, rerunning logging.config.dictConfig() blows away all existing loggers, handlers, and formatters. There are the incremental and disable_existing_loggers options available for use in the config dict, but using incremental, you can't load in new handlers or formatters. You either need to combine you configuration in to a single file for your whole program, or use the mechanisms provided in the module to manually construct and add formatters and handlers to loggers.