I have a really strange problem with the standard logging module used in django views. Sometimes it works perfectly and sometimes it does not log messages.
Here is the structure of my code :
/mysite/ (Django root)
my_logging.py (logging configuration)
settings.py
views.py (global views)
data_objects.py (objects only containing data, similar to POJO)
uploader/ (application)
views.py (uploader views) --> This is where I have problems
Here is the code of my_logging.py :
import logging
import logging.handlers
from django.conf import settings
is_initialized = False
def init_logger():
"""
Initializes the logging for the application. Configure the root
logger and creates the handlers following the settings. This function should
not be used directly from outside the module and called only once.
"""
# Create the logger
server_logger = logging.getLogger()
server_logger.setLevel(logging.DEBUG)
# Set the logging format for files
files_formatter = logging.Formatter(settings.LOGGING_FORMAT_FILE)
# Rotating file handler for errors
error_handler = logging.handlers.RotatingFileHandler(
settings.LOGGING_ERROR_FILE,
maxBytes=settings.LOGGING_ERROR_FILE_SIZE,
backupCount=settings.LOGGING_ERROR_FILE_COUNT,
)
error_handler.setLevel(logging.WARNING)
error_handler.setFormatter(files_formatter)
# Rotating file handler for info
info_handler = logging.handlers.RotatingFileHandler(
settings.LOGGING_INFO_FILE,
maxBytes=settings.LOGGING_INFO_FILE_SIZE,
backupCount=settings.LOGGING_INFO_FILE_COUNT,
)
info_handler.setLevel(logging.INFO)
info_handler.setFormatter(files_formatter)
# Add the handlers to the logger
server_logger.addHandler(info_handler)
server_logger.addHandler(error_handler)
# Init once at first import
if not is_initialized:
init_logger()
is_initialized = True
Here are parts of uploader/views.py (#... = code skipped):
#...
import mysite.my_logging
import logging
#...
# The messages in the following view are written correctly :
#login_required
def delete(request, file_id):
"""
Delete the file corresponding to the given ID and confirm the deletion to
the user.
#param request: the HTTP request object
#type request: django.http.HttpRequest
#return: django.http.HttpResponse - the response to the client (html)
"""
# Get the file object form the database and raise a 404 if not found
f = get_object_or_404(VideoFile, pk=file_id)
# TODO: check if the deletion is successful
# Get the video directory
dir_path = os.path.dirname(f.file.path)
# Delete the file
f.delete()
try:
# Delete the video directory recursively
shutil.rmtree(dir_path)
logging.info("File \"%(file)s\" and its directory have been deleted by %(username)s",{'file': f.title,'username': request.user.username})
messages.success(request, _('The video file "%s" has been successfully deleted.') % f.title)
except OSError:
logging.warning("File \"%(id)d\" directory cannot be completely deleted. Some files may still be there.",{'id': f.id,})
messages.warning(request, _("The video file \"%s\" has been successfully deleted, but not its directory. There should not be any problem but useless disk usage.") % f.title)
return HttpResponseRedirect(reverse('mysite.uploader.views.list'))
#...
# The messages in the following view are NOT written at all:
#csrf_exempt
def get_thumblist(request,file_id):
"""
This view can be called only by POST and with the id of a video
file ready for the scene editor.
#param request: the HTTP request object. Must have POST as method.
#type request: django.http.HttpRequest
#return: django.http.HttpResponse - the response to the client (json)
"""
#TODO: Security, TEST
logging.info("Demand of metadata for file %(id)d received.",{'id': file_id,})
if request.method == 'POST':
if file_id:
# Get the video file object form the database and raise a 404 if not found
vid = get_object_or_404(VideoFile, pk=file_id)
# ...
try:
# ... file operations
except IOError:
logging.error("Error when trying to read index file for file %(id)d !",{'id': file_id,})
except TypeError:
logging.error("Error when trying to parse index file JSON for file %(id)d !",{'id': file_id,})
# ...
logging.info("Returning metadata for file %(id)d.",{'id': file_id,})
return HttpResponse(json,content_type="application/json")
else:
logging.warning("File %(id)d is not ready",{'id': file_id,})
return HttpResponseBadRequest('file_not_ready')
else:
logging.warning("bad POST parameters")
return HttpResponseBadRequest('bad_parameters')
else:
logging.warning("The GET method is not allowed")
return HttpResponseNotAllowed(['POST'])
and the interesting part of settings.py:
# ---------------------------------------
# Logging settings
# ---------------------------------------
#: Minimum level for logging messages. If logging.NOTSET, logging is disabled
LOGGING_MIN_LEVEL = logging.DEBUG
#: Error logging file path. Can be relative to the root of the project or absolute.
LOGGING_ERROR_FILE = os.path.join(DIRNAME,"log/error.log")
#: Size (in bytes) of the error files
LOGGING_ERROR_FILE_SIZE = 10485760 # 10 MiB
#: Number of backup error logging files
LOGGING_ERROR_FILE_COUNT = 5
#: Info logging file path. Can be relative to the root of the project or absolute.
LOGGING_INFO_FILE = os.path.join(DIRNAME,"log/info.log")
#: Size (in bytes) of the info files
LOGGING_INFO_FILE_SIZE = 10485760 # 10 MiB
#: Number of backup error info files
LOGGING_INFO_FILE_COUNT = 5
#: Format for the log files
LOGGING_FORMAT_FILE = "%(asctime)s:%(name)s:%(levelname)s:%(message)s"
Note that except logging everything is working fine. The data can be returned correctly in JSON format. I think there is no error in the rest of the code.
Please ask if you need more information. I'm sorry about the code I removed, but I have to because of confidentiality.
Instead of using the logging.info('My statement') syntax, I suggest you use something like the following:
import logging
logger = logging.getLogger('MySite')
logger.info('My statement')
That is, call your log statements against a logger object, instead of the logging module directly. Likewise, you'll have to tweak my_logging.py to configure that logger:
# Create the logger
server_logger = logging.getLogger('MySite')
server_logger.setLevel(logging.DEBUG)
In your views, you can log against logging.getLogger('MySite') or logging.getLogger('MySite.views'), etc.. Any logger which starts with 'MySite' will inherit your configuration.
Also, while you have the right idea by setting and checking is_initialized, I don't believe that approach will work. Each time my_logging.py is imported, that variable will be set to False, thus defeating its purpose. You can use the following in settings.py to ensure that logging is only configured once:
# Init once at first import
if not hasattr(my_logging, 'is_initialized'):
my_logging.is_initialized = False
if not my_logging.is_initialized:
my_logging.init_logger()
my_logging.is_initialized = True
I start all of my modules (except settings.py) with the following two lines:
import logging
logging.getLogger('MySite.ModuleInit').debug('Initializing %s' % str(__name__))
If you are still having trouble, please add those lines and then post the module initialization order for your site. There may be a strange quirk based on the order of your imports.
Related
I am using Python's logging to log execution of functions and other actions within an application. The log files are stored in a remote folder, which is accessible automatically when I connect to VPN (let's say \remote\directory). That is normal situation, 99% of the time there is a connection and log is stored without errors.
I need a solution for a situation when either the VPN connection or Internet connection is lost and the logs are temporarily stored locally. I think that on each time something is attempted to be logged, I need to run a check if the remote folder is accessible. I couldn't really find a solution, but I guess I need to modify the FileHandler somehow.
TLDR: You can already scroll down to blues' answer and my UPDATE section - there is my latest attempt to solve the issue.
Currently my handler is set like this:
log = logging.getLogger('general')
handler_error = logging.handlers.RotatingFileHandler(log_path+"\\error.log", 'a', encoding="utf-8")
log.addHandler(handler_error)
Here is a condition that sets the log path but only once - when logging is initialized. If I think correctly, I would like to run this condition each time the
if (os.path.isdir(f"\\\\remote\\folder\\")): # if remote is accessible
log_path = f"\\\\remote\\folder\\dev\\{d.year}\\{month}\\"
os.makedirs(os.path.dirname(log_path), exist_ok=True) # create this month dir if it does not exist, logging does not handle that
else: # if remote is not accesssible
log_path = f"localFiles\\logs\\dev\\{d.year}\\{month}\\"
log.debug("Cannot access the remote directory. Are you connected to the internet and the VPN?")
I have found a related thread, but was not able to adjust it to my own needs: Dynamic filepath & filename for FileHandler in logger config file in python
Should I dig deeper into custom Handler or is there some other way? Would be enough if I could call my own function that changed the logging path if needed (or change logger to one with a proper path) when logging is being executed.
UPDATE:
Per blues's answer, I have tried modifying a handler to suit my needs. Unfortunately, the code below, in which I try to switch baseFilename between local and remote paths, does not work. The logger always saves the log to local log file (that has been set while initializing logger). Thus, I think that my attempts to modify the baseFilename do not work?
class HandlerCheckBefore(RotatingFileHandler):
print("handler starts")
def emit(self, record):
calltime = date.today()
if os.path.isdir(f"\\\\remote\\Path\\"): # if remote is accessible
print("handler remote")
# create remote folders if not yet existent
os.makedirs(os.path.dirname(f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\"), exist_ok=True)
if (self.level >= 20): # if error or above
self.baseFilename = f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\error.log"
else:
self.baseFilename = f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\{calltime.strftime('%d')}-{calltime.strftime('%m')}.log"
super().emit(record)
else: # save to local
print("handler local")
if (self.level >= 20): # error or above
self.baseFilename = f"localFiles\\logs\\{calltime.year}\\{calltime.strftime('%m')}\\error.log"
else:
self.baseFilename = f"localFiles\\logs\\{calltime.year}\\{calltime.strftime('%m')}\\{calltime.strftime('%d')}-{calltime.strftime('%m')}.log"
super().emit(record)
# init the logger
handler_error = HandlerCheckBefore(f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\error.log", 'a', encoding="utf-8")
handler_error.setLevel(logging.ERROR)
handler_error.setFormatter(fmt)
log.addHandler(handler_error)
The best way to solve this is indeed to create a custom Handler for this. You can either check before each write that the directory is still there, or you could attempt to write the log and handle the resulting error in handleError which all loggers call when an exception occurs during emit(). I recommend the former. The code below shows how both could be implemented:
import os
import logging
from logging.handlers import RotatingFileHandler
class GrzegorzRotatingFileHandlerCheckBefore(RotatingFileHandler):
def emit(self, record):
if os.path.isdir(os.path.dirname(self.baseFilename)): # put appropriate check here
super().emit(record)
else:
logging.getLogger('offline').error('Cannot access the remote directory. Are you connected to the internet and the VPN?')
class GrzegorzRotatingFileHandlerHandleError(RotatingFileHandler):
def handleError(self, record):
logging.getLogger('offline').error('Something went wrong when writing log. Probably remote dir is not accessible')
super().handleError(record)
log = logging.getLogger('general')
log.addHandler(GrzegorzRotatingFileHandlerCheckBefore('check.log'))
log.addHandler(GrzegorzRotatingFileHandlerHandleError('handle.log'))
offline_logger = logging.getLogger('offline')
offline_logger.addHandler(logging.FileHandler('offline.log'))
log.error('test logging')
It's my first time using the logging module in Python(3.7). My code uses imported modules that also have their own log statements. When I first added log statements to my code, I didn't use getLogger(). I just used logging.basicConfig(filename) and called logger.debug() directly to log statements. When I did this, all the logs from both my script and also all the imported modules was output to the same file together.
Now I need to convert my code to save logs to s3 instead of a file. I tried the solution mentioned in How Can I Write Logs Directly to AWS S3 from Memory Without First Writing to stdout? (Python, boto3) - Stack Overflow but I have two issues with it:
None of the 'prefixes' are present in the output when I check on s3.
Only INFO statements are showing up. I was under the impression that logging.basicConfig(level=logging.INFO) would make it would output all logs at or above level INFO, but I'm only seeing INFO. Also, only INFO logs get printed to stdout, when before all levels were. I don't know why the 'prefixes' are missing.
from psaw import PushshiftAPI
api = PushshiftAPI()
import time
import logging
import boto3
import io
import atexit
def write_logs(body, bucket, key):
s3 = boto3.client("s3")
s3.put_object(Body=body.getvalue(), Bucket=bucket, Key=key)
logging.basicConfig(level=logging.INFO)
log = logging.getLogger()
log_stringio = io.StringIO()
handler = logging.StreamHandler(log_stringio)
log.addHandler(handler)
def collectRange(sub,start,end):
atexit.register(write_logs, body=log_stringio, bucket="<...>", key=f'{sub}/log.txt')
s3 = boto3.resource('s3')
object = s3.Object('<...>', f'{sub}/{sub}#{start}-{end}.csv')
now = time.time()
logging.info(f'Start Time:{now}')
logging.debug('First request')
gen = api.search_comments(after=start, before=end,<...>, subreddit=sub)
r=next(gen)
<...>
quit()
Output:
Found credentials in shared credentials file: ~/.aws/credentials
Start Time:1591310443.7060978
https://api.pushshift.io/reddit/comment/search?<...>
https://api.pushshift.io/reddit/comment/search?<...>
Desired output:
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:root:Start Time:1591310443.7060978
DEBUG:root:First request
INFO:psaw.PushshiftAPI:https://api.pushshift.io/reddit/comment/search?<...>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
INFO:psaw.PushshiftAPI:https://api.pushshift.io/reddit/comment/search?<...>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
Any help is appreciated. Thanks.
You can at least add the level name (or time) by following this documentation:
Changing the format of displayed messages.
And to get DEBUG as well you need to use the following instead of INFO
logging.basicConfig(..., level=logging.DEBUG)
I am having trouble loading a function in a python file that i created. The entire code in the python file runs without any problems in a Jupyter Notebook. Now that i have put the code, 1:1 into a python file, i get an error "cannot import name 'get' from 'connector'" where connector is my python file, and get is the function nested inside.
I suspect that the format from Jupyter Notebook changes and into a regular python file changes the way the file is read somehow?
I wanted to move the code into a python file, as this function is usefull for mulitple projects i am doing in python when scraping webpages. I can call the function ratelimit function, but i can't seem to figure out what is wrong with my get command.
# Imports
import scraping_class
import pandas as pd
import requests,os,time
logfile="trustpilot.txt"
Connector = scraping_class.Connector(logfile)
def ratelimit(x):
"A function that handles the rate of your calls."
time.sleep(x) # sleep x seconds.
class Connector():
def __init__(self,logfile,overwrite_log=False,connector_type='requests',session=False,path2selenium='',n_tries = 5,timeout=30):
"""This Class implements a method for reliable connection to the internet and monitoring.
It handles simple errors due to connection problems, and logs a range of information for basic quality assessments
Keyword arguments:
logfile -- path to the logfile
overwrite_log -- bool, defining if logfile should be cleared (rarely the case).
connector_type -- use the 'requests' module or the 'selenium'. Will have different since the selenium webdriver does not have a similar response object when using the get method, and monitoring the behavior cannot be automated in the same way.
session -- requests.session object. For defining custom headers and proxies.
path2selenium -- str, sets the path to the geckodriver needed when using selenium.
n_tries -- int, defines the number of retries the *get* method will try to avoid random connection errors.
timeout -- int, seconds the get request will wait for the server to respond, again to avoid connection errors.
"""
## Initialization function defining parameters.
self.n_tries = n_tries # For avoiding triviel error e.g. connection errors, this defines how many times it will retry.
self.timeout = timeout # Defining the maximum time to wait for a server to response.
## not implemented here, if you use selenium.
if connector_type=='selenium':
assert path2selenium!='', "You need to specify the path to you geckodriver if you want to use Selenium"
from selenium import webdriver
## HIN download the latest geckodriver here: https://github.com/mozilla/geckodriver/releases
assert os.path.isfile(path2selenium),'You need to insert a valid path2selenium the path to your geckodriver. You can download the latest geckodriver here: https://github.com/mozilla/geckodriver/releases'
self.browser = webdriver.Firefox(executable_path=path2selenium) # start the browser with a path to the geckodriver.
self.connector_type = connector_type # set the connector_type
if session: # set the custom session
self.session = session
else:
self.session = requests.session()
self.logfilename = logfile # set the logfile path
## define header for the logfile
header = ['id','project','connector_type','t', 'delta_t', 'url', 'redirect_url','response_size', 'response_code','success','error']
if os.path.isfile(logfile):
if overwrite_log==True:
self.log = open(logfile,'w')
self.log.write(';'.join(header))
else:
self.log = open(logfile,'a')
else:
self.log = open(logfile,'w')
self.log.write(';'.join(header))
## load log
with open(logfile,'r') as f: # open file
l = f.read().split('\n') # read and split file by newlines.
## set id
if len(l)<=1:
self.id = 0
else:
self.id = int(l[-1][0])+1
def get(self,url,project_name):
"""Method for connector reliably to the internet, with multiple tries and simple error handling, as well as default logging function.
Input url and the project name for the log (i.e. is it part of mapping the domain, or is it the part of the final stage in the data collection).
Keyword arguments:
url -- str, url
project_name -- str, Name used for analyzing the log. Use case could be the 'Mapping of domain','Meta_data_collection','main data collection'.
"""
project_name = project_name.replace(';','-') # make sure the default csv seperator is not in the project_name.
if self.connector_type=='requests': # Determine connector method.
for _ in range(self.n_tries): # for loop defining number of retries with the requests method.
ratelimit()
t = time.time()
try: # error handling
response = self.session.get(url,timeout = self.timeout) # make get call
err = '' # define python error variable as empty assumming success.
success = True # define success variable
redirect_url = response.url # log current url, after potential redirects
dt = t - time.time() # define delta-time waiting for the server and downloading content.
size = len(response.text) # define variable for size of html content of the response.
response_code = response.status_code # log status code.
## log...
call_id = self.id # get current unique identifier for the call
self.id+=1 # increment call id
#['id','project_name','connector_type','t', 'delta_t', 'url', 'redirect_url','response_size', 'response_code','success','error']
row = [call_id,project_name,self.connector_type,t,dt,url,redirect_url,size,response_code,success,err] # define row to be written in the log.
self.log.write('\n'+';'.join(map(str,row))) # write log.
return response,call_id # return response and unique identifier.
except Exception as e: # define error condition
err = str(e) # python error
response_code = '' # blank response code
success = False # call success = False
size = 0 # content is empty.
redirect_url = '' # redirect url empty
dt = t - time.time() # define delta t
## log...
call_id = self.id # define unique identifier
self.id+=1 # increment call_id
row = [call_id,project_name,self.connector_type,t,dt,url,redirect_url,size,response_code,success,err] # define row
self.log.write('\n'+';'.join(map(str,row))) # write row to log.
else:
t = time.time()
ratelimit()
self.browser.get(url) # use selenium get method
## log
call_id = self.id # define unique identifier for the call.
self.id+=1 # increment the call_id
err = '' # blank error message
success = '' # success blank
redirect_url = self.browser.current_url # redirect url.
dt = t - time.time() # get time for get method ... NOTE: not necessarily the complete load time.
size = len(self.browser.page_source) # get size of content ... NOTE: not necessarily correct, since selenium works in the background, and could still be loading.
response_code = '' # empty response code.
row = [call_id,project_name,self.connector_type,t,dt,url,redirect_url,size,response_code,success,err] # define row
self.log.write('\n'+';'.join(map(str,row))) # write row to log file.
# Using selenium it will not return a response object, instead you should call the browser object of the connector.
## connector.browser.page_source will give you the html.
return call_id
logfile="trustpilot.txt" ## name your log file.
connector = Connector(logfile)
I expected to be able to load my 'get' function from 'connector.py'
The following line may be the issue due to naming conflict as you Connector variable and Connector class:
Connector = scraping_class.Connector(logfile)
I am using the Python logger mechanism for keeping a record of my logs. I have two types of logs,
one is the Rotating log (log1, log2, log3...) and a non-rotating log called json.log (which has json logs in it as the name suggests).
The log files are created when the server is started and close when the app is closed.
What I am trying to do in general is: When I press the import button on my page, to have all json logs saved on the sqlite db.
The problem I am facing is:
When I try to rename the json.log file like this:
source_file = "./logs/json.log"
snapshot_file = "./logs/json.snapshot.log"
try:
os.rename(source_file, snapshot_file)
I get the windowsError: [Error 32] The process cannot access the file because it is being used by another process
and this is because the file is being used by the logger continuously. Therefore, I need to "close" the file somehow so I can do my I/O operation successfully.
The thing is that this is not desirable because logs might be lost until the file is closed, then renamed and then "re-created".
I was wondering if anyone came across such scenario again and if any practical solution was found.
I have tried something which works but does not seem convenient and not sure if it is safe so that any logs are not lost.
My code is this:
source_file = "./logs/json.log"
snapshot_file = "./logs/json.snapshot.log"
try:
logger = get_logger()
# some hackish way to remove the handler for json.log
if len(logger.handlers) > 2:
logger.removeHandler(logger.handlers[2])
if not os.path.exists(snapshot_file):
os.rename(source_file, snapshot_file)
try:
if type(logger.handlers[2]) == RequestLoggerHandler:
del logger.handlers[2]
except IndexError:
pass
# re-adding the logs file handler so it continues writing the logs
json_file_name = configuration["brew.log_file_dir"] + os.sep + "json.log"
json_log_level = logging.DEBUG
json_file_handler = logging.FileHandler(json_file_name)
json_file_handler.setLevel(json_log_level)
json_file_handler.addFilter(JSONLoggerFiltering())
json_file_handler.setFormatter(JSONFormatter())
logger.addHandler(json_file_handler)
... code continues to write the logs to the db and then delete the json.snapshot.file
until the next time the import button is pressed; then the snapshot is created again
only for writing the logs to the db.
Also for reference my log file has this format:
{'status': 200, 'actual_user': 1, 'resource_name': '/core/logs/process', 'log_level': 'INFO', 'request_body': None, ... }
Thanks in advance :)
The native python logger used by our flask app seems to stop writing to the log after an exception happens. The last entry logged before each stoppage is a message describing the exception. Typically the next message is one written by code in after_request but for cases where the logger stops, the after_request message is never written out.
Any idea what could be causing this?
Note: I originally posted this question on Serverfault (https://serverfault.com/questions/655683/python-logger-stops-logging) thinking it was an infrastructure issue. But now that we have narrowed the issue down to it occurring after an exception, this issue may be better suited for Stackoverflow.
Update [12/22/2015]:
Logger instantiation:
logging.addLevelName(Config.LOG_AUDIT_LEVEL_NUM, Config.LOG_AUDIT_LEVEL_NAME)
logger = logging.getLogger(Config.LOGGER_NAME)
logger.setLevel(Config.LOG_LEVEL)
handler = SysLogHandler(address='/dev/log', facility=SysLogHandler.LOG_LOCAL3)
handler.setLevel(Config.LOG_LEVEL)
formatter = log_formatter()
handler.setFormatter(formatter)
logger.addHandler(handler)
log_formatter:
class log_formatter(logging.Formatter):
def __init__(self,
fmt=None,
datefmt=None,
json_cls=None,
json_default=_default_json_default):
"""
:param fmt: Config as a JSON string, allowed fields;
extra: provide extra fields always present in logs
source_host: override source host name
:param datefmt: Date format to use (required by logging.Formatter
interface but not used)
:param json_cls: JSON encoder to forward to json.dumps
:param json_default: Default JSON representation for unknown types,
by default coerce everything to a string
"""
if fmt is not None:
self._fmt = json.loads(fmt)
else:
self._fmt = {}
self.json_default = json_default
self.json_cls = json_cls
if 'extra' not in self._fmt:
self.defaults = {}
else:
self.defaults = self._fmt['extra']
try:
self.source_host = socket.gethostname()
except:
self.source_host = ""
def format(self, record):
"""
Format a log record to JSON, if the message is a dict
assume an empty message and use the dict as additional
fields.
"""
fields = record.__dict__.copy()
aux_fields = [
'relativeCreated', 'process', 'args', 'module', 'funcName', 'name',
'thread', 'created', 'threadName', 'msecs', 'filename', 'levelno',
'processName', 'pathname', 'lineno', 'levelname'
]
for k in aux_fields:
del fields[k]
if isinstance(record.msg, dict):
fields.update(record.msg)
fields.pop('msg')
msg = ""
else:
msg = record.getMessage()
if 'msg' in fields:
fields.pop('msg')
if 'exc_info' in fields:
if fields['exc_info']:
formatted = tb.format_exception(*fields['exc_info'])
fields['exception'] = formatted
fields.pop('exc_info')
if 'exc_text' in fields and not fields['exc_text']:
fields.pop('exc_text')
logr = self.defaults.copy()
logr = {
'timestamp': datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ'),
'host': self.source_host,
}
logr.update(self._build_fields(logr, fields))
if msg:
logr['message'] = msg
something = json.dumps(logr, default=self.json_default, cls=self.json_cls)
return something
def _build_fields(self, defaults, fields):
return dict(defaults.get('fields', {}).items() + fields.items())
Update [01/03/2015]:
Answering questions posted:
Is the application still working after the exception?
Yes, the application continues to run.
Which type of exception is raised and what's the cause of it?
Internal/custom exception. Logger has stopped due to different types of exceptions.
Are you using threads in you application?
Yes, the app is threaded by gunicorn.
How is the logging library used?
We are using default FileHandler, SysLogHandler and a custom formatter (outputs JSON)
Does it log on a file? Does it use log rotation?
Yes, it logs to a file, but no rotation.
In regards to after_request, from the docs:
As of Flask 0.7 this function might not be executed at the end of the
request in case an unhandled exception occurred.
And as for your logging issue, it may be that your debug flag is set to true, which would cause the debugger to kick in and possibly stop the logging.
References:
(http://flask.pocoo.org/docs/0.10/api/#flask.Flask.after_request)
(http://flask.pocoo.org/docs/0.10/errorhandling/#working-with-debuggers)
You didn't provide enough information.
Is the application still working after the exception?
Which type of exception is raised and what's the cause of it?
Are you using threads in you application?
How is the logging library used? Does it log on a file? Does it use log rotation?
Supposing you're using threads in you application, the explanation is that the exception causes the Thread to shut down, therefore you won't see any activity from that specific thread. You should notice issues with the application as well.
If the application is still working but becomes silent, my guess is that the logging library is not configured properly. As you reported on Serverfault, the issue seemed to appear after adding fluentd which might not play well with the way your application uses the logging library.