I'm using the Python logging module to log what's going on in my application on both on a file and into the terminal.
My code is:
import logging
logging.basicConfig(level=logging.DEBUG, handlers=[ # stream=sys.stdout,
logging.FileHandler("debug.log", 'a'),
logging.StreamHandler()
], format='%(asctime)s %(levelname)s %(message)s', )
logging.info("This is a LOG INFO message")
Now, as soon as the program runs, it saves the log on a file.
I would like to know how the logging method save this info on the file. Does it opens -> write -> close the file evry time a logging line is called? Do keep the file open until the program ends?
I'm asking because if my software crash for some reasons or the PC reboot, the already written log file is safe or it could be corrupted?
As the names suggests the handlers you have set here, stream their log messages to the specified sinks. Logging Handler Documentation states that it:
"[...] sends logging output to streams such as sys.stdout, sys.stderr or any file-like object (or, more precisely, any object which supports write() and flush() methods)."
Looking at the Source Code and considering this GitHub Gist since you apparently could use a logging instance with a context manager if you really wanted to, I would make the assumption that your intuition was exactly correct (regarding open > write > close).
Related
How would you dynamically change the file where logs are written to in Python, using the standard logging package?
I have a single process multi-threaded application that processes tasks for specific logical bins. To help simplify debugging and searching the logs, I want each bin to have its own separate log file. Due to memory usage and scaling concerns, I don't want to split the process into multiple processes whose output I could otherwise easily redirect to a separate log. However, by default, Python's logging package only outputs to a single location, either stdout/stderr or or some other single file.
My question's similar to this question except I'm not trying to change the logging level, just the logging output destination.
you will need to create a different logger for each thread and configure each logger to it's own file.
You can call something like this function in each thread, with the appropiate bin_name:
def create_logger(bin_name, level=logging.INFO):
handler = logging.FileHandler(f'{bin_name}.log')
handler.setFormatter(logging.Formatter('%(asctime)s %(levelname)s %(message)s'))
bin_logger = logging.getLogger(bin_name)
bin_logger .setLevel(level)
bin_logger .addHandler(handler)
return bin_logger
I'm using the ConcurrentLogHandler to log my Python application log messages to a file. Tornado's HTTPServer is used in production as the server, and in certain condition (user form submit), I would like to force the roll over of the file.
The issue I'm having is, when running logger.handlers[0].doRollover(), the roll over does not happen, and I don't see an error either.
Many modules of my apps import the "logger" object from the following module to write to the log:
import logging
import logging.handlers
from cloghandler import ConcurrentRotatingFileHandler
from my_app import app
logger = logging.getLogger('my_app')
hdlr = ConcurrentRotatingFileHandler(app.config['LOG_PATH'], maxBytes=5e+8, backupCount=10)
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
Prior to using ConcurrentLogHandler, I was using the RotatingFileHandler, and performing a logger.handlers[0].doRollover() would result in a "File busy" error triggered on the os.rename down the stack.
Is there a way I could force a rollover with this set up, or are there changes I should make in order to make this possible? I'm using Centos as a server, so the filesystem is UNIX/Linux. I can see a [filename].lock file on the log file at all times while my application is running.
Thank you,
ConcurrentLogHandler can be rolled over by logger.handlers[0].doRollover() indeed. The issue I had was that I used the RotatingLogHandler previously and a .lock file was left in the directory of the log, preventing it from being rolled over. Removing the lock file and keeping the ConcurrentLogHandler as the handler of choice (rather than RotatingLogHandler) solves the problem.
def fn(filename):
import logging
logger = logging.getLogger(filename)
formatter = logging.Formatter('[%(asctime)s:%(msecs)d], [%(levelname)s], %(message)s, ')
handler = logging.FileHandler(filename + ".log")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.error("that's an error")
Basically I'm just using the python logging library for logging. The code is working fine if I run it as a normal python function.
However if I use it in celery/supervisord:
fn.delay("loggingFile")
The file is opened and never closed (by the user used in supervisord). Of course, after some time, I have hundreds of opened files which is very bad and causes other troubles. Why is this happening with celery tasks only and how to avoid it?
I know the files are never closed by monitoring the system using lsof to see which users are opening which files.
Appreciate any help
For some reason, the daemon does not close the file handler. you will need to it manually using after you finish logging:
logger.removeHandler(handler)
I am using the logging library in python. I am using several logger objects to output the information. I got to the point where the output is too much, and I need to send it to a logfile instead of the console.
I can't find a function that configures the output stream of the logger object, and the basicConf function doesn't seem to do the trick.
This is what I have:
import logging # Debug logging framework
logging.basicConfig(filename='loggggggmeee.txt')
logger = logging.getLogger('simulation')
logger.setLevel(logging.INFO)
#--------------- etc-------------------#
logger.info('This is sent to the console')
Any ideas? Thanks!
Try to add a file location like here it will solve your problem:
import logging # Debug logging framework
logging.basicConfig(filename='c:\\loggggggmeee.txt')
logger = logging.getLogger('simulation')
logger.setLevel(logging.INFO)
#--------------- etc-------------------#
logger.info('This is sent to the console')
By the way the file without a specific location will be at the module library...
I really recommend you to read the Logging HOWTO It's very simple and useful.
Also the Cookbook has lots of useful examples.
Another good tip about nameing the loggers from the docs:
a good convention to use when naming loggers is to use a module-level
logger, in each module which uses logging, named as follows:
logger = logging.getLogger(__name__)
This means that logger names
track the package/module hierarchy, and it’s intuitively obvious where
events are logged just from the logger name.
I have an application written in python that logs errors in a log file, I am using the python logging module
Example:
import logging
logging.basicConfig(filename='\logs\filename.log',
level=logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s')
logging.error('error connecting to server')
This is working fine and the log file is logging the errors, which mean the last error is logged on the last line of the file.
is there some kind of setting where I can tell the logging module to always write at the top of the file, this way the last error is always on the first line.
It is very inefficient to write to the front of a flle as others have said. Every single write will have to first seek to the front of the file and then insert the new data before the other data. The underlying I/O of your operating system is designed to make appends cheap.
That said, if you insist on doing it, look at the docs here Logging Handlers. You can implement your own version of logging.handlers.FileHandler that will seek to the beginning before each write. Then you could call logging.addHandler() and place an instance of your class. If you only care about the most recent 10 or so log entries you could even truncate the file before you write.