We Keep Coding

Disable logger of specific python module in logging config file

I am using the below logging file and load it via fileConfig. I would like to configure the logging behavior of modules that are imported. For example, I want to disable (or set it to some higher log level) logging from urllib3. How would I do that?
[loggers]
keys = root
[logger_root]
level = DEBUG
handlers = root
[handlers]
keys = root
[handler_root]
class = StreamHandler
level = DEBUG
formatter = json
[formatters]
keys = json
[formatter_json]
format = %(name)s%(message)s%(asctime)s%*(levelname)s%*(module)s
class = pythonjsonlogger.jsonlogger.JsonFormatter

Change your configuration to something like (these are just additions/changes to you what you posted):
[loggers]
keys = root,urllib3
[logger_urllib3]
level=CRITICAL
handlers=root
qualname=urllib3

Python Logging: Specifying converter attribute of a log formatter in config file

I'd like to have all timestamps in my log file to be UTC timestamp. When specified through code, this is done as follows:
import logging
import time
myHandler = logging.FileHandler('mylogfile.log', 'a')
formatter = logging.Formatter('%(asctime)s %(levelname)-8s %(name)-15s:%(lineno)4s: %(message)-80s')
formatter.converter = time.gmtime
myHandler.setFormatter(formatter)
myLogger = logging.getLogger('MyApp')
myLogger.addHandler(myHandler)
myLogger.setLevel(logging.DEBUG)
myLogger.info('here we are')
I'd like to move away from the above 'in-code' configuration to a config file based mechanism.
Here's the config file section for the formatter:
[handler_MyLogHandler]
args=("mylogfile.log", "a",)
class=FileHandler
level=DEBUG
formatter=simpleFormatter
Now, how do I specify the converter attribute (time.gmtime) in the above section?
Edit: The above config file is loaded thus:
logging.config.fileConfig('myLogConfig.conf')

Sadly, there is no way of doing this using the configuration file, other than having e.g. a
class UTCFormatter(logging.Formatter):
converter = time.gmtime
and then using a UTCFormatter in the configuration.

Here Vinay's solution applied to the logging.basicConfig:
import logging
import time
logging.basicConfig(filename='junk.log', level=logging.DEBUG, format='%(asctime)s: %(levelname)s:%(message)s')
logging.Formatter.converter = time.gmtime
logging.info('A message.')

Pyramid Project: "ValueError: Variable assignment..." when trying to pserve

I get the following error when trying to run a Pyramid project. As far as I'm aware, this appeared overnight, and I've no idea how to begin to debug this:
C:\mas\mas>..\Scripts\pserve.exe serve development.ini --reload
Starting subprocess with file monitor
Traceback (most recent call last):
File "C:\mas\Scripts\pserve-script.py", line 8, in <module>
load_entry_point('pyramid==1.3.2', 'console_scripts', 'pserve')()
File "C:\mas\lib\site-packages\pyramid-1.3.2-py2.7.egg\pyramid\scripts\pserve.
py", line 47, in main
return command.run()
File "C:\mas\lib\site-packages\pyramid-1.3.2-py2.7.egg\pyramid\scripts\pserve.
py", line 221, in run
vars = self.parse_vars(restvars)
File "C:\mas\lib\site-packages\pyramid-1.3.2-py2.7.egg\pyramid\scripts\pserve.
py", line 330, in parse_vars
% arg)
ValueError: Variable assignment 'development.ini' invalid (no "=")
What is the problem, or how a should I go about determining it? Sorry my question is rather vague, but if I had any more an idea of what I was asking I may have found the answer on Google :).
Line 328 - 330:
raise ValueError(
'Variable assignment %r invalid (no "=")'
% arg)
development.ini
[app:main]
use = egg:mas
pyramid.reload_templates = true
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid.includes = pyramid_debugtoolbar
pyramid_tm
pyramid_beaker
sqlalchemy.url = sqlite:///%(here)s/mas.db
# Cache settings
cache.regions = default_term, second, short_term, long_term
cache.type = memory
cache.second.expire = 1
cache.short_term.expire = 60
cache.default_term.expire = 300
cache.long_term.expire = 3600
# Beaker sessions
#session.type = file
#session.data_dir = %(here)s/data/sessions/data
#session.lock_dir = %(here)s/data/sessions/lock
session.type = memory
session.key = akhet_demo
session.secret = 0cb243f53ad865a0f70099c0414ffe9cfcfe03ac
[server:main]
use = egg:waitress#main
host = 0.0.0.0
port = 6543
# Begin logging configuration
[loggers]
keys = root, mas, sqlalchemy
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = INFO
handlers = console
[logger_mas]
level = DEBUG
handlers =
qualname = mas
[logger_sqlalchemy]
level = INFO
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither. (Recommended for production systems.)
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s][%(threadName)s] %(message)s
# End logging configuration

Ich, I'e worked it out, I was looking for the problem in the wrong place. the problem is this command:
..\Scripts\pserve.exe serve development.ini --reload
Should be this command:
..\Scripts\pserve.exe development.ini --reload
I have no idea how or when my batch file got changed, but if someone has a similar problem hopefully this will help.

It appears that you have the order of the arguments backwards - it should be:
..\Scripts\pserve.exe development.ini --reload

what is the python(2.5.2)'s logging mechanism? is there log4j look a like library or standard class of python/geodjango will do?

I am planning to add logging mechanism in my python&geodjango web service.
Is there log4j look a like logging mechanism in python/geodjango?
I am looking for log4j's dailyrollingfileappender equivalent. Then automatically delete all 1month old log files.
Any guidance is appreciated.
UPDATES1
I am thinking of the below format.
datetime(ms)|log level|current thread name|client ip address|logged username|source file|source file line number|log message

Yes - Python 2.5 includes the 'logging' module. One of the handlers it supports is handlers.TimedRotatingFileHandler, this is what you're looking for. 'logging' is very easy to use:
example:
import logging
import logging.config
logging.fileConfig('mylog.conf')
logger = logging.getLogger('root')
The following is your config file for logging
#======================
# mylog.conf
[loggers]
keys=root
[handlers]
keys=default
[formatters]
keys=default
[logger_root]
level=INFO
handlers=default
qualname=(root) # note - this is used in non-root loggers
propagate=1 # note - this is used in non-root loggers
channel=
parent=
[handler_default]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=default
args=('try.log', 'd', 1)
[formatter_default]
format=%(asctime)s %(pathname)s(%(lineno)d): %(levelname)s %(message)s

How do I configure the Python logging module in Django?

I'm trying to configure logging for a Django app using the Python logging module. I have placed the following bit of configuration code in my Django project's settings.py file:
import logging
import logging.handlers
import os
date_fmt = '%m/%d/%Y %H:%M:%S'
log_formatter = logging.Formatter(u'[%(asctime)s] %(levelname)-7s: %(message)s (%(filename)s:%(lineno)d)', datefmt=date_fmt)
log_dir = os.path.join(PROJECT_DIR, "var", "log", "my_app")
log_name = os.path.join(log_dir, "nyrb.log")
bytes = 1024 * 1024 # 1 MB
if not os.path.exists(log_dir):
os.makedirs(log_dir)
handler = logging.handlers.RotatingFileHandler(log_name, maxBytes=bytes, backupCount=7)
handler.setFormatter(log_formatter)
handler.setLevel(logging.DEBUG)
logging.getLogger().setLevel(logging.DEBUG)
logging.getLogger().addHandler(handler)
logging.getLogger(__name__).info("Initialized logging subsystem")
At startup, I get a couple Django-related messages, as well as the "Initialized logging subsystem", in the log files, but then all the log messages end up going to the web server logs (/var/log/apache2/error.log, since I'm using Apache), and use the standard log format (not the formatter I designated). Am I configuring logging incorrectly?

Kind of anti-climactic, but it turns out there was a third-party app installed in the project that had its own logging configuration that was overriding the one I set up (it modified the root logger, for some reason -- not very kosher for a Django app!). Removed that code and everything works as expected.

See this other answer. Note that settings.py is usually imported twice, so you should avoid creating multiple handlers. Better logging support is coming to Django in 1.3 (hopefully), but for now you should ensure that if your setup code is called more than once, there are no adverse effects.
I'm not sure why your logged messages are going to the Apache logs, unless you've (somewhere else in your code) added a StreamHandler to your root logger with sys.stdout or sys.stderr as the stream. You might want to print out logging.getLogger().handlers just to see it's what you'd expect to see.

I used this with success (although it does not rotate):
# in settings.py
import logging
logging.basicConfig(
level = logging.DEBUG,
format = '%(asctime)s %(levelname)s %(funcName)s %(lineno)d \
\033[35m%(message)s\033[0m',
datefmt = '[%d/%b/%Y %H:%M:%S]',
filename = '/tmp/my_django_app.log',
filemode = 'a'
)
I'd suggest to try an absolute path, too.

I guess logging stops when Apache forks the process. After that happened, because all file descriptors were closed during daemonization, logging system tries to reopen log file and as far as I understand uses relative file path:
log_dir = os.path.join(PROJECT_DIR, "var", "log", "my_app")
log_name = os.path.join(log_dir, "nyrb.log")
But there is no “current directory” when process has been daemonized. Try to use absolute log_dir path. Hope that helps.