We Keep Coding

create multiple log files python

I want to save my results as a log file, so I am thinking to import logging module. I understand that to output a file, the code is very straightforward.
logging.basicConfig(filename='logger.log', level=logging.INFO)
logging.debug('debug message')
logging.info('info message')
logging.warn('warn message')
logging.error('error message')
logging.critical('critical message')
However, what if I want to output multiple log files? for example, in the following for loop, each iteration will output a log file, how should I do this?
for i in range(1,10):
print (i)
#output a log file to save I value
I tried to use these code, but it's not working.
for i in range(1,10):
filename = str.format('mylog%d.txt' % i)
logging.basicConfig(format=log_fmt, level=logging.DEBUG, filename=filename)
logging.debug('This is debug message')
logging.info('This is info message')
logging.warning('This is warning message')

You're using the format function of strings incorrectly. You're trying to use string interpolation, which is an entirely different method of formatting strings. You should try something like this:
filename = 'mylog{0}.txt'.format(i)
The {0} explicitly states that you should take the first value passed to format. You could leave it as {} if you'd like - it makes no real difference.

About file name:
filename = str.format('mylog%d.txt' % i)
is equal to:
filename = 'mylog%d.txt' % i
For output to multiple files you can use multiple handlers.
Logging class that handle logging.
root_logger = logging.getLogger()
Return you root handler. You can add or remove handlers to logger.
root_logger.handlers
Contains list of handlers of root logger.
first = root_logger.handlers[0]
first.close()
root_logger.removeHandler(first)
remove first handler.
new_handler = logging.FileHandler(file_name)
formatter = logging.Formatter('%(asctime)s ' + ' %(message)s', '%H:%M:%S')
new_handler.setFormatter(formatter)
root_logger.addHandler(new_handler)
Add new formatter to root_handler.
You can output log to any files at the same time.
For more info read:
https://docs.python.org/2/library/logging.html
https://docs.python.org/2/howto/logging-cookbook.html

How to get the app name in django logger

I have this formatter for django
FORMAT = "[%(filename)s:%(lineno)s - %(funcName)20s() ] %(message)s"
The file name i get is
views.py
Now that is confusing as its hard to see from which module is that views.py.
Is there any to get appname in logger formatter

Use pathname instead of filename in your logging configuration.
FORMAT = "[%(pathname)s:%(lineno)s - %(funcName)20s() ] %(message)s"
There are also other variables you can use — check the logging module documentation for a list.
Note that if you're acquiring a Logger instance using logger = logging.getLogger(__name__) (which is a common way to do it), you can also retrieve the module name (e.g. myapp.views) using name.
This is (arguably) better practice but will not work if you're doing e.g. logger = logging.getLogger("mylogger") or logger = logging.getLogger()

Python logger stops logging after uncaught exception

The native python logger used by our flask app seems to stop writing to the log after an exception happens. The last entry logged before each stoppage is a message describing the exception. Typically the next message is one written by code in after_request but for cases where the logger stops, the after_request message is never written out.
Any idea what could be causing this?
Note: I originally posted this question on Serverfault (https://serverfault.com/questions/655683/python-logger-stops-logging) thinking it was an infrastructure issue. But now that we have narrowed the issue down to it occurring after an exception, this issue may be better suited for Stackoverflow.
Update [12/22/2015]:
Logger instantiation:
logging.addLevelName(Config.LOG_AUDIT_LEVEL_NUM, Config.LOG_AUDIT_LEVEL_NAME)
logger = logging.getLogger(Config.LOGGER_NAME)
logger.setLevel(Config.LOG_LEVEL)
handler = SysLogHandler(address='/dev/log', facility=SysLogHandler.LOG_LOCAL3)
handler.setLevel(Config.LOG_LEVEL)
formatter = log_formatter()
handler.setFormatter(formatter)
logger.addHandler(handler)
log_formatter:
class log_formatter(logging.Formatter):
def __init__(self,
fmt=None,
datefmt=None,
json_cls=None,
json_default=_default_json_default):
"""
:param fmt: Config as a JSON string, allowed fields;
extra: provide extra fields always present in logs
source_host: override source host name
:param datefmt: Date format to use (required by logging.Formatter
interface but not used)
:param json_cls: JSON encoder to forward to json.dumps
:param json_default: Default JSON representation for unknown types,
by default coerce everything to a string
"""
if fmt is not None:
self._fmt = json.loads(fmt)
else:
self._fmt = {}
self.json_default = json_default
self.json_cls = json_cls
if 'extra' not in self._fmt:
self.defaults = {}
else:
self.defaults = self._fmt['extra']
try:
self.source_host = socket.gethostname()
except:
self.source_host = ""
def format(self, record):
"""
Format a log record to JSON, if the message is a dict
assume an empty message and use the dict as additional
fields.
"""
fields = record.__dict__.copy()
aux_fields = [
'relativeCreated', 'process', 'args', 'module', 'funcName', 'name',
'thread', 'created', 'threadName', 'msecs', 'filename', 'levelno',
'processName', 'pathname', 'lineno', 'levelname'
]
for k in aux_fields:
del fields[k]
if isinstance(record.msg, dict):
fields.update(record.msg)
fields.pop('msg')
msg = ""
else:
msg = record.getMessage()
if 'msg' in fields:
fields.pop('msg')
if 'exc_info' in fields:
if fields['exc_info']:
formatted = tb.format_exception(*fields['exc_info'])
fields['exception'] = formatted
fields.pop('exc_info')
if 'exc_text' in fields and not fields['exc_text']:
fields.pop('exc_text')
logr = self.defaults.copy()
logr = {
'timestamp': datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S.%fZ'),
'host': self.source_host,
}
logr.update(self._build_fields(logr, fields))
if msg:
logr['message'] = msg
something = json.dumps(logr, default=self.json_default, cls=self.json_cls)
return something
def _build_fields(self, defaults, fields):
return dict(defaults.get('fields', {}).items() + fields.items())
Update [01/03/2015]:
Answering questions posted:
Is the application still working after the exception?
Yes, the application continues to run.
Which type of exception is raised and what's the cause of it?
Internal/custom exception. Logger has stopped due to different types of exceptions.
Are you using threads in you application?
Yes, the app is threaded by gunicorn.
How is the logging library used?
We are using default FileHandler, SysLogHandler and a custom formatter (outputs JSON)
Does it log on a file? Does it use log rotation?
Yes, it logs to a file, but no rotation.

In regards to after_request, from the docs:
As of Flask 0.7 this function might not be executed at the end of the
request in case an unhandled exception occurred.
And as for your logging issue, it may be that your debug flag is set to true, which would cause the debugger to kick in and possibly stop the logging.
References:
(http://flask.pocoo.org/docs/0.10/api/#flask.Flask.after_request)
(http://flask.pocoo.org/docs/0.10/errorhandling/#working-with-debuggers)

You didn't provide enough information.
Is the application still working after the exception?
Which type of exception is raised and what's the cause of it?
Are you using threads in you application?
How is the logging library used? Does it log on a file? Does it use log rotation?
Supposing you're using threads in you application, the explanation is that the exception causes the Thread to shut down, therefore you won't see any activity from that specific thread. You should notice issues with the application as well.
If the application is still working but becomes silent, my guess is that the logging library is not configured properly. As you reported on Serverfault, the issue seemed to appear after adding fluentd which might not play well with the way your application uses the logging library.

Python: Log to multiple log files

Currently I have everything getting logged to one logfile but I want to separate it out to multiple log files. I look at the logging in python documentation but they don't discuss about this.
log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=(os.path.join(OUT_DIR, + '-user.log')),
format=log_format, level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S')
Currently this is how I do the logging. what I want to do have different type of errors or information get log into different log files. At the moment when I do logging.info('Logging IN') and logging.error('unable to login') will go to same logfile. I want to seperate them. Do I need to create another logging object to support the logging into another file?

What you /could/ do (I haven't dug into the logging module too much so there may be a better way to do this) is maybe use a stream rather than a file object:
In [1]: class LogHandler(object):
...: def write(self, msg):
...: print 'a :%s' % msg
...: print 'b :%s' % msg
...:
In [3]: import logging
In [4]: logging.basicConfig(stream=LogHandler())
In [5]: logging.critical('foo')
a :CRITICAL:root:foo
b :CRITICAL:root:foo
In [6]: logging.warn('bar')
a :WARNING:root:bar
b :WARNING:root:bar
Edit with further handling:
Assuming your log files already exist, you could do something like this:
import logging
class LogHandler(object):
format = '%(levelname)s %(message)s'
files = {
'ERROR': 'error.log',
'CRITICAL': 'error.log',
'WARN': 'warn.log',
}
def write(self, msg):
type_ = msg[:msg.index(' ')]
with open(self.files.get(type_, 'log.log'), 'r+') as f:
f.write(msg)
logging.basicConfig(format=LogHandler.format, stream=LogHandler())
logging.critical('foo')
This would allow you to split your logging into various files based on conditions in your log messages. If what you're looking for isn't found, it simply defaults to log.log.

I created this solution from docs.python.org/2/howto/logging-cookbook.html
Simply create two logging file handlers, assign their logging level and add them to your logger.
import os
import logging
current_path = os.path.dirname(os.path.realpath(__file__))
logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)
#to log debug messages
debug_log = logging.FileHandler(os.path.join(current_path, 'debug.log'))
debug_log.setLevel(logging.DEBUG)
#to log errors messages
error_log = logging.FileHandler(os.path.join(current_path, 'error.log'))
error_log.setLevel(logging.ERROR)
logger.addHandler(debug_log)
logger.addHandler(error_log)
logger.debug('This message should go in the debug log')
logger.info('and so should this message')
logger.warning('and this message')
logger.error('This message should go in both the debug log and the error log')

Django 1.2 : strange logging behavior

I have a really strange problem with the standard logging module used in django views. Sometimes it works perfectly and sometimes it does not log messages.
Here is the structure of my code :
/mysite/ (Django root)
my_logging.py (logging configuration)
settings.py
views.py (global views)
data_objects.py (objects only containing data, similar to POJO)
uploader/ (application)
views.py (uploader views) --> This is where I have problems
Here is the code of my_logging.py :
import logging
import logging.handlers
from django.conf import settings
is_initialized = False
def init_logger():
"""
Initializes the logging for the application. Configure the root
logger and creates the handlers following the settings. This function should
not be used directly from outside the module and called only once.
"""
# Create the logger
server_logger = logging.getLogger()
server_logger.setLevel(logging.DEBUG)
# Set the logging format for files
files_formatter = logging.Formatter(settings.LOGGING_FORMAT_FILE)
# Rotating file handler for errors
error_handler = logging.handlers.RotatingFileHandler(
settings.LOGGING_ERROR_FILE,
maxBytes=settings.LOGGING_ERROR_FILE_SIZE,
backupCount=settings.LOGGING_ERROR_FILE_COUNT,
)
error_handler.setLevel(logging.WARNING)
error_handler.setFormatter(files_formatter)
# Rotating file handler for info
info_handler = logging.handlers.RotatingFileHandler(
settings.LOGGING_INFO_FILE,
maxBytes=settings.LOGGING_INFO_FILE_SIZE,
backupCount=settings.LOGGING_INFO_FILE_COUNT,
)
info_handler.setLevel(logging.INFO)
info_handler.setFormatter(files_formatter)
# Add the handlers to the logger
server_logger.addHandler(info_handler)
server_logger.addHandler(error_handler)
# Init once at first import
if not is_initialized:
init_logger()
is_initialized = True
Here are parts of uploader/views.py (#... = code skipped):
#...
import mysite.my_logging
import logging
#...
# The messages in the following view are written correctly :
#login_required
def delete(request, file_id):
"""
Delete the file corresponding to the given ID and confirm the deletion to
the user.
#param request: the HTTP request object
#type request: django.http.HttpRequest
#return: django.http.HttpResponse - the response to the client (html)
"""
# Get the file object form the database and raise a 404 if not found
f = get_object_or_404(VideoFile, pk=file_id)
# TODO: check if the deletion is successful
# Get the video directory
dir_path = os.path.dirname(f.file.path)
# Delete the file
f.delete()
try:
# Delete the video directory recursively
shutil.rmtree(dir_path)
logging.info("File \"%(file)s\" and its directory have been deleted by %(username)s",{'file': f.title,'username': request.user.username})
messages.success(request, _('The video file "%s" has been successfully deleted.') % f.title)
except OSError:
logging.warning("File \"%(id)d\" directory cannot be completely deleted. Some files may still be there.",{'id': f.id,})
messages.warning(request, _("The video file \"%s\" has been successfully deleted, but not its directory. There should not be any problem but useless disk usage.") % f.title)
return HttpResponseRedirect(reverse('mysite.uploader.views.list'))
#...
# The messages in the following view are NOT written at all:
#csrf_exempt
def get_thumblist(request,file_id):
"""
This view can be called only by POST and with the id of a video
file ready for the scene editor.
#param request: the HTTP request object. Must have POST as method.
#type request: django.http.HttpRequest
#return: django.http.HttpResponse - the response to the client (json)
"""
#TODO: Security, TEST
logging.info("Demand of metadata for file %(id)d received.",{'id': file_id,})
if request.method == 'POST':
if file_id:
# Get the video file object form the database and raise a 404 if not found
vid = get_object_or_404(VideoFile, pk=file_id)
# ...
try:
# ... file operations
except IOError:
logging.error("Error when trying to read index file for file %(id)d !",{'id': file_id,})
except TypeError:
logging.error("Error when trying to parse index file JSON for file %(id)d !",{'id': file_id,})
# ...
logging.info("Returning metadata for file %(id)d.",{'id': file_id,})
return HttpResponse(json,content_type="application/json")
else:
logging.warning("File %(id)d is not ready",{'id': file_id,})
return HttpResponseBadRequest('file_not_ready')
else:
logging.warning("bad POST parameters")
return HttpResponseBadRequest('bad_parameters')
else:
logging.warning("The GET method is not allowed")
return HttpResponseNotAllowed(['POST'])
and the interesting part of settings.py:
# ---------------------------------------
# Logging settings
# ---------------------------------------
#: Minimum level for logging messages. If logging.NOTSET, logging is disabled
LOGGING_MIN_LEVEL = logging.DEBUG
#: Error logging file path. Can be relative to the root of the project or absolute.
LOGGING_ERROR_FILE = os.path.join(DIRNAME,"log/error.log")
#: Size (in bytes) of the error files
LOGGING_ERROR_FILE_SIZE = 10485760 # 10 MiB
#: Number of backup error logging files
LOGGING_ERROR_FILE_COUNT = 5
#: Info logging file path. Can be relative to the root of the project or absolute.
LOGGING_INFO_FILE = os.path.join(DIRNAME,"log/info.log")
#: Size (in bytes) of the info files
LOGGING_INFO_FILE_SIZE = 10485760 # 10 MiB
#: Number of backup error info files
LOGGING_INFO_FILE_COUNT = 5
#: Format for the log files
LOGGING_FORMAT_FILE = "%(asctime)s:%(name)s:%(levelname)s:%(message)s"
Note that except logging everything is working fine. The data can be returned correctly in JSON format. I think there is no error in the rest of the code.
Please ask if you need more information. I'm sorry about the code I removed, but I have to because of confidentiality.

Instead of using the logging.info('My statement') syntax, I suggest you use something like the following:
import logging
logger = logging.getLogger('MySite')
logger.info('My statement')
That is, call your log statements against a logger object, instead of the logging module directly. Likewise, you'll have to tweak my_logging.py to configure that logger:
# Create the logger
server_logger = logging.getLogger('MySite')
server_logger.setLevel(logging.DEBUG)
In your views, you can log against logging.getLogger('MySite') or logging.getLogger('MySite.views'), etc.. Any logger which starts with 'MySite' will inherit your configuration.
Also, while you have the right idea by setting and checking is_initialized, I don't believe that approach will work. Each time my_logging.py is imported, that variable will be set to False, thus defeating its purpose. You can use the following in settings.py to ensure that logging is only configured once:
# Init once at first import
if not hasattr(my_logging, 'is_initialized'):
my_logging.is_initialized = False
if not my_logging.is_initialized:
my_logging.init_logger()
my_logging.is_initialized = True
I start all of my modules (except settings.py) with the following two lines:
import logging
logging.getLogger('MySite.ModuleInit').debug('Initializing %s' % str(__name__))
If you are still having trouble, please add those lines and then post the module initialization order for your site. There may be a strange quirk based on the order of your imports.