We Keep Coding

Dynamically switching log storage folder

I am using Python's logging to log execution of functions and other actions within an application. The log files are stored in a remote folder, which is accessible automatically when I connect to VPN (let's say \remote\directory). That is normal situation, 99% of the time there is a connection and log is stored without errors.
I need a solution for a situation when either the VPN connection or Internet connection is lost and the logs are temporarily stored locally. I think that on each time something is attempted to be logged, I need to run a check if the remote folder is accessible. I couldn't really find a solution, but I guess I need to modify the FileHandler somehow.
TLDR: You can already scroll down to blues' answer and my UPDATE section - there is my latest attempt to solve the issue.
Currently my handler is set like this:
log = logging.getLogger('general')
handler_error = logging.handlers.RotatingFileHandler(log_path+"\\error.log", 'a', encoding="utf-8")
log.addHandler(handler_error)
Here is a condition that sets the log path but only once - when logging is initialized. If I think correctly, I would like to run this condition each time the
if (os.path.isdir(f"\\\\remote\\folder\\")): # if remote is accessible
log_path = f"\\\\remote\\folder\\dev\\{d.year}\\{month}\\"
os.makedirs(os.path.dirname(log_path), exist_ok=True) # create this month dir if it does not exist, logging does not handle that
else: # if remote is not accesssible
log_path = f"localFiles\\logs\\dev\\{d.year}\\{month}\\"
log.debug("Cannot access the remote directory. Are you connected to the internet and the VPN?")
I have found a related thread, but was not able to adjust it to my own needs: Dynamic filepath & filename for FileHandler in logger config file in python
Should I dig deeper into custom Handler or is there some other way? Would be enough if I could call my own function that changed the logging path if needed (or change logger to one with a proper path) when logging is being executed.
UPDATE:
Per blues's answer, I have tried modifying a handler to suit my needs. Unfortunately, the code below, in which I try to switch baseFilename between local and remote paths, does not work. The logger always saves the log to local log file (that has been set while initializing logger). Thus, I think that my attempts to modify the baseFilename do not work?
class HandlerCheckBefore(RotatingFileHandler):
print("handler starts")
def emit(self, record):
calltime = date.today()
if os.path.isdir(f"\\\\remote\\Path\\"): # if remote is accessible
print("handler remote")
# create remote folders if not yet existent
os.makedirs(os.path.dirname(f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\"), exist_ok=True)
if (self.level >= 20): # if error or above
self.baseFilename = f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\error.log"
else:
self.baseFilename = f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\{calltime.strftime('%d')}-{calltime.strftime('%m')}.log"
super().emit(record)
else: # save to local
print("handler local")
if (self.level >= 20): # error or above
self.baseFilename = f"localFiles\\logs\\{calltime.year}\\{calltime.strftime('%m')}\\error.log"
else:
self.baseFilename = f"localFiles\\logs\\{calltime.year}\\{calltime.strftime('%m')}\\{calltime.strftime('%d')}-{calltime.strftime('%m')}.log"
super().emit(record)
# init the logger
handler_error = HandlerCheckBefore(f"\\\\remote\\Path\\{calltime.year}\\{calltime.strftime('%m')}\\error.log", 'a', encoding="utf-8")
handler_error.setLevel(logging.ERROR)
handler_error.setFormatter(fmt)
log.addHandler(handler_error)

The best way to solve this is indeed to create a custom Handler for this. You can either check before each write that the directory is still there, or you could attempt to write the log and handle the resulting error in handleError which all loggers call when an exception occurs during emit(). I recommend the former. The code below shows how both could be implemented:
import os
import logging
from logging.handlers import RotatingFileHandler
class GrzegorzRotatingFileHandlerCheckBefore(RotatingFileHandler):
def emit(self, record):
if os.path.isdir(os.path.dirname(self.baseFilename)): # put appropriate check here
super().emit(record)
else:
logging.getLogger('offline').error('Cannot access the remote directory. Are you connected to the internet and the VPN?')
class GrzegorzRotatingFileHandlerHandleError(RotatingFileHandler):
def handleError(self, record):
logging.getLogger('offline').error('Something went wrong when writing log. Probably remote dir is not accessible')
super().handleError(record)
log = logging.getLogger('general')
log.addHandler(GrzegorzRotatingFileHandlerCheckBefore('check.log'))
log.addHandler(GrzegorzRotatingFileHandlerHandleError('handle.log'))
offline_logger = logging.getLogger('offline')
offline_logger.addHandler(logging.FileHandler('offline.log'))
log.error('test logging')

Using Python Logging module to save logs to S3, how to capture all levels of logs for all modules?

It's my first time using the logging module in Python(3.7). My code uses imported modules that also have their own log statements. When I first added log statements to my code, I didn't use getLogger(). I just used logging.basicConfig(filename) and called logger.debug() directly to log statements. When I did this, all the logs from both my script and also all the imported modules was output to the same file together.
Now I need to convert my code to save logs to s3 instead of a file. I tried the solution mentioned in How Can I Write Logs Directly to AWS S3 from Memory Without First Writing to stdout? (Python, boto3) - Stack Overflow but I have two issues with it:
None of the 'prefixes' are present in the output when I check on s3.
Only INFO statements are showing up. I was under the impression that logging.basicConfig(level=logging.INFO) would make it would output all logs at or above level INFO, but I'm only seeing INFO. Also, only INFO logs get printed to stdout, when before all levels were. I don't know why the 'prefixes' are missing.
from psaw import PushshiftAPI
api = PushshiftAPI()
import time
import logging
import boto3
import io
import atexit
def write_logs(body, bucket, key):
s3 = boto3.client("s3")
s3.put_object(Body=body.getvalue(), Bucket=bucket, Key=key)
logging.basicConfig(level=logging.INFO)
log = logging.getLogger()
log_stringio = io.StringIO()
handler = logging.StreamHandler(log_stringio)
log.addHandler(handler)
def collectRange(sub,start,end):
atexit.register(write_logs, body=log_stringio, bucket="<...>", key=f'{sub}/log.txt')
s3 = boto3.resource('s3')
object = s3.Object('<...>', f'{sub}/{sub}#{start}-{end}.csv')
now = time.time()
logging.info(f'Start Time:{now}')
logging.debug('First request')
gen = api.search_comments(after=start, before=end,<...>, subreddit=sub)
r=next(gen)
<...>
quit()
Output:
Found credentials in shared credentials file: ~/.aws/credentials
Start Time:1591310443.7060978
https://api.pushshift.io/reddit/comment/search?<...>
https://api.pushshift.io/reddit/comment/search?<...>
Desired output:
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:root:Start Time:1591310443.7060978
DEBUG:root:First request
INFO:psaw.PushshiftAPI:https://api.pushshift.io/reddit/comment/search?<...>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
INFO:psaw.PushshiftAPI:https://api.pushshift.io/reddit/comment/search?<...>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
DEBUG:psaw.PushshiftAPI:<whatever is usually here>
Any help is appreciated. Thanks.

You can at least add the level name (or time) by following this documentation:
Changing the format of displayed messages.
And to get DEBUG as well you need to use the following instead of INFO
logging.basicConfig(..., level=logging.DEBUG)

Write header to a python log file, but only if a record gets written

fh = logging.FileHandler('example.log',delay = True)
fh.setLevel(logging.INFO)
Since delay is True, the file will never be written unless something is logged.
At that point, the first line in the file is the first record, and it will contain the asctime, levelname etc elements
Using python 2.7.10, is there a sane way to add a line (or two) the first time a record is written that don't include those elements?
I can just write to the file before using it for logging, but if I do that, I end up with logs empty but for the header.
The desired output might look like:
Using test.fil with option 7
2015-11-01 13:57:58,045 :log_example: INFO fn:main result:process 4 complete --7 knights said ni
2015-11-01 13:57:58,045 :log_example: INFO fn:main result:process 3 complete --3 bunnies attacked
Thanks,

Sub class the FileHandler to create your own custom FileHandleWithHeader as shown below:
import os
import logging
# Create a class that extends the FileHandler class from logging.FileHandler
class FileHandlerWithHeader(logging.FileHandler):
# Pass the file name and header string to the constructor.
def __init__(self, filename, header, mode='a', encoding=None, delay=0):
# Store the header information.
self.header = header
# Determine if the file pre-exists
self.file_pre_exists = os.path.exists(filename)
# Call the parent __init__
logging.FileHandler.__init__(self, filename, mode, encoding, delay)
# Write the header if delay is False and a file stream was created.
if not delay and self.stream is not None:
self.stream.write('%s\n' % header)
def emit(self, record):
# Create the file stream if not already created.
if self.stream is None:
self.stream = self._open()
# If the file pre_exists, it should already have a header.
# Else write the header to the file so that it is the first line.
if not self.file_pre_exists:
self.stream.write('%s\n' % self.header)
# Call the parent class emit function.
logging.FileHandler.emit(self, record)
# Create a logger and set the logging level.
logger = logging.getLogger("example")
logger.setLevel(logging.INFO)
# Create a file handler from our new FileHandlerWith Header class and set the
# logging level.
fh = FileHandlerWithHeader('example.log', 'This is my header', delay=True)
fh.setLevel(logging.INFO)
# Add formatter to the file handler.
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
fh.setFormatter(formatter)
# Add the handler to the logger.
logger.addHandler(fh)
# Since the constructor of the FileHandlerWithHeader was passed delay=True
# the file should not exist until the first log as long as the log file did
# not pre-exist.
print "Ready to write to the the example.log file."
raw_input("Press Enter to continue...")
# Send 3 logs to the logger.
logger.info("First line in the file")
logger.info("Second line in the file")
logger.info("Third line in the file")
# The log file should now be created and only have a header at the begining of
# the file.
print "The example.log file should exist and have a header."
This script should run as is in Python 2.7. If the "example.log" file already exists, it will not recreate the header.
This solution required knowledge of the logging source code found here
and general use of the python logging package found here.

I had a simpler idea. The following just uses a custom formatter. The first message formatted spits out a header record then after that just does normal formatting.
import logging
class FormatterWithHeader(logging.Formatter):
def __init__(self, header, fmt=None, datefmt=None, style='%'):
super().__init__(fmt, datefmt, style)
self.header = header # This is hard coded but you could make dynamic
# Override the normal format method
self.format = self.first_line_format
def first_line_format(self, record):
# First time in, switch back to the normal format function
self.format = super().format
return self.header + "\n" + self.format(record)
def test_logger():
logger = logging.getLogger("test")
logger.setLevel(logging.DEBUG)
formatter = FormatterWithHeader('First Line Only')
ch = logging.StreamHandler()
ch.setLevel(logging.DEBUG)
ch.setFormatter(formatter)
logger.addHandler(ch)
logger.info("This line will kick out a header first.")
logger.info("This line will *not* kick out a header.")

Django 1.2 : strange logging behavior

I have a really strange problem with the standard logging module used in django views. Sometimes it works perfectly and sometimes it does not log messages.
Here is the structure of my code :
/mysite/ (Django root)
my_logging.py (logging configuration)
settings.py
views.py (global views)
data_objects.py (objects only containing data, similar to POJO)
uploader/ (application)
views.py (uploader views) --> This is where I have problems
Here is the code of my_logging.py :
import logging
import logging.handlers
from django.conf import settings
is_initialized = False
def init_logger():
"""
Initializes the logging for the application. Configure the root
logger and creates the handlers following the settings. This function should
not be used directly from outside the module and called only once.
"""
# Create the logger
server_logger = logging.getLogger()
server_logger.setLevel(logging.DEBUG)
# Set the logging format for files
files_formatter = logging.Formatter(settings.LOGGING_FORMAT_FILE)
# Rotating file handler for errors
error_handler = logging.handlers.RotatingFileHandler(
settings.LOGGING_ERROR_FILE,
maxBytes=settings.LOGGING_ERROR_FILE_SIZE,
backupCount=settings.LOGGING_ERROR_FILE_COUNT,
)
error_handler.setLevel(logging.WARNING)
error_handler.setFormatter(files_formatter)
# Rotating file handler for info
info_handler = logging.handlers.RotatingFileHandler(
settings.LOGGING_INFO_FILE,
maxBytes=settings.LOGGING_INFO_FILE_SIZE,
backupCount=settings.LOGGING_INFO_FILE_COUNT,
)
info_handler.setLevel(logging.INFO)
info_handler.setFormatter(files_formatter)
# Add the handlers to the logger
server_logger.addHandler(info_handler)
server_logger.addHandler(error_handler)
# Init once at first import
if not is_initialized:
init_logger()
is_initialized = True
Here are parts of uploader/views.py (#... = code skipped):
#...
import mysite.my_logging
import logging
#...
# The messages in the following view are written correctly :
#login_required
def delete(request, file_id):
"""
Delete the file corresponding to the given ID and confirm the deletion to
the user.
#param request: the HTTP request object
#type request: django.http.HttpRequest
#return: django.http.HttpResponse - the response to the client (html)
"""
# Get the file object form the database and raise a 404 if not found
f = get_object_or_404(VideoFile, pk=file_id)
# TODO: check if the deletion is successful
# Get the video directory
dir_path = os.path.dirname(f.file.path)
# Delete the file
f.delete()
try:
# Delete the video directory recursively
shutil.rmtree(dir_path)
logging.info("File \"%(file)s\" and its directory have been deleted by %(username)s",{'file': f.title,'username': request.user.username})
messages.success(request, _('The video file "%s" has been successfully deleted.') % f.title)
except OSError:
logging.warning("File \"%(id)d\" directory cannot be completely deleted. Some files may still be there.",{'id': f.id,})
messages.warning(request, _("The video file \"%s\" has been successfully deleted, but not its directory. There should not be any problem but useless disk usage.") % f.title)
return HttpResponseRedirect(reverse('mysite.uploader.views.list'))
#...
# The messages in the following view are NOT written at all:
#csrf_exempt
def get_thumblist(request,file_id):
"""
This view can be called only by POST and with the id of a video
file ready for the scene editor.
#param request: the HTTP request object. Must have POST as method.
#type request: django.http.HttpRequest
#return: django.http.HttpResponse - the response to the client (json)
"""
#TODO: Security, TEST
logging.info("Demand of metadata for file %(id)d received.",{'id': file_id,})
if request.method == 'POST':
if file_id:
# Get the video file object form the database and raise a 404 if not found
vid = get_object_or_404(VideoFile, pk=file_id)
# ...
try:
# ... file operations
except IOError:
logging.error("Error when trying to read index file for file %(id)d !",{'id': file_id,})
except TypeError:
logging.error("Error when trying to parse index file JSON for file %(id)d !",{'id': file_id,})
# ...
logging.info("Returning metadata for file %(id)d.",{'id': file_id,})
return HttpResponse(json,content_type="application/json")
else:
logging.warning("File %(id)d is not ready",{'id': file_id,})
return HttpResponseBadRequest('file_not_ready')
else:
logging.warning("bad POST parameters")
return HttpResponseBadRequest('bad_parameters')
else:
logging.warning("The GET method is not allowed")
return HttpResponseNotAllowed(['POST'])
and the interesting part of settings.py:
# ---------------------------------------
# Logging settings
# ---------------------------------------
#: Minimum level for logging messages. If logging.NOTSET, logging is disabled
LOGGING_MIN_LEVEL = logging.DEBUG
#: Error logging file path. Can be relative to the root of the project or absolute.
LOGGING_ERROR_FILE = os.path.join(DIRNAME,"log/error.log")
#: Size (in bytes) of the error files
LOGGING_ERROR_FILE_SIZE = 10485760 # 10 MiB
#: Number of backup error logging files
LOGGING_ERROR_FILE_COUNT = 5
#: Info logging file path. Can be relative to the root of the project or absolute.
LOGGING_INFO_FILE = os.path.join(DIRNAME,"log/info.log")
#: Size (in bytes) of the info files
LOGGING_INFO_FILE_SIZE = 10485760 # 10 MiB
#: Number of backup error info files
LOGGING_INFO_FILE_COUNT = 5
#: Format for the log files
LOGGING_FORMAT_FILE = "%(asctime)s:%(name)s:%(levelname)s:%(message)s"
Note that except logging everything is working fine. The data can be returned correctly in JSON format. I think there is no error in the rest of the code.
Please ask if you need more information. I'm sorry about the code I removed, but I have to because of confidentiality.

Instead of using the logging.info('My statement') syntax, I suggest you use something like the following:
import logging
logger = logging.getLogger('MySite')
logger.info('My statement')
That is, call your log statements against a logger object, instead of the logging module directly. Likewise, you'll have to tweak my_logging.py to configure that logger:
# Create the logger
server_logger = logging.getLogger('MySite')
server_logger.setLevel(logging.DEBUG)
In your views, you can log against logging.getLogger('MySite') or logging.getLogger('MySite.views'), etc.. Any logger which starts with 'MySite' will inherit your configuration.
Also, while you have the right idea by setting and checking is_initialized, I don't believe that approach will work. Each time my_logging.py is imported, that variable will be set to False, thus defeating its purpose. You can use the following in settings.py to ensure that logging is only configured once:
# Init once at first import
if not hasattr(my_logging, 'is_initialized'):
my_logging.is_initialized = False
if not my_logging.is_initialized:
my_logging.init_logger()
my_logging.is_initialized = True
I start all of my modules (except settings.py) with the following two lines:
import logging
logging.getLogger('MySite.ModuleInit').debug('Initializing %s' % str(__name__))
If you are still having trouble, please add those lines and then post the module initialization order for your site. There may be a strange quirk based on the order of your imports.

Python Cherrypy Access Log Rotation

If I want the access log for Cherrypy to only get to a fixed size, how would I go about using rotating log files?
I've already tried http://www.cherrypy.org/wiki/Logging, which seems out of date, or has information missing.

Look at http://docs.python.org/library/logging.html.
You probably want to configure a RotatingFileHandler
http://docs.python.org/library/logging.html#rotatingfilehandler

I've already tried http://www.cherrypy.org/wiki/Logging, which seems
out of date, or has information missing.
Try adding:
import logging
import logging.handlers
import cherrypy # you might have imported this already
and instead of
log = app.log
maybe try
log = cherrypy.log

The CherryPy documentation of the custom log handlers shows this very example.
Here is the slightly modified version that I use on my app:
import logging
from logging import handlers
def setup_logging():
log = cherrypy.log
# Remove the default FileHandlers if present.
log.error_file = ""
log.access_file = ""
maxBytes = getattr(log, "rot_maxBytes", 10000000)
backupCount = getattr(log, "rot_backupCount", 1000)
# Make a new RotatingFileHandler for the error log.
fname = getattr(log, "rot_error_file", "log\\error.log")
h = handlers.RotatingFileHandler(fname, 'a', maxBytes, backupCount)
h.setLevel(logging.DEBUG)
h.setFormatter(cherrypy._cplogging.logfmt)
log.error_log.addHandler(h)
# Make a new RotatingFileHandler for the access log.
fname = getattr(log, "rot_access_file", "log\\access.log")
h = handlers.RotatingFileHandler(fname, 'a', maxBytes, backupCount)
h.setLevel(logging.DEBUG)
h.setFormatter(cherrypy._cplogging.logfmt)
log.access_log.addHandler(h)
setup_logging()

Cherrypy does its logging using the standard Python logging module. You will need to change it to use a RotatingFileHandler. This handler will take care of everything for you including rotating the log when it reaches a set maximum size.