def fn(filename):
import logging
logger = logging.getLogger(filename)
formatter = logging.Formatter('[%(asctime)s:%(msecs)d], [%(levelname)s], %(message)s, ')
handler = logging.FileHandler(filename + ".log")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.error("that's an error")
Basically I'm just using the python logging library for logging. The code is working fine if I run it as a normal python function.
However if I use it in celery/supervisord:
fn.delay("loggingFile")
The file is opened and never closed (by the user used in supervisord). Of course, after some time, I have hundreds of opened files which is very bad and causes other troubles. Why is this happening with celery tasks only and how to avoid it?
I know the files are never closed by monitoring the system using lsof to see which users are opening which files.
Appreciate any help
For some reason, the daemon does not close the file handler. you will need to it manually using after you finish logging:
logger.removeHandler(handler)
Related
I'm using the Python logging module to log what's going on in my application on both on a file and into the terminal.
My code is:
import logging
logging.basicConfig(level=logging.DEBUG, handlers=[ # stream=sys.stdout,
logging.FileHandler("debug.log", 'a'),
logging.StreamHandler()
], format='%(asctime)s %(levelname)s %(message)s', )
logging.info("This is a LOG INFO message")
Now, as soon as the program runs, it saves the log on a file.
I would like to know how the logging method save this info on the file. Does it opens -> write -> close the file evry time a logging line is called? Do keep the file open until the program ends?
I'm asking because if my software crash for some reasons or the PC reboot, the already written log file is safe or it could be corrupted?
As the names suggests the handlers you have set here, stream their log messages to the specified sinks. Logging Handler Documentation states that it:
"[...] sends logging output to streams such as sys.stdout, sys.stderr or any file-like object (or, more precisely, any object which supports write() and flush() methods)."
Looking at the Source Code and considering this GitHub Gist since you apparently could use a logging instance with a context manager if you really wanted to, I would make the assumption that your intuition was exactly correct (regarding open > write > close).
I'm working with the Python logging configuration below
import logging
import os
#Setup logging
logger = logging.getLogger('/logs/some-log')
logger.setLevel(logging.INFO)
handler = RotatingFileHandler('./logs/some-log.log', maxBytes=10240, backupCount=5)
logger.addHandler(handler)
#End Logging
#Set Logging Permissions
os.chmod('/logs/some*.log', 0o755)
#End Set Logging Permissions
I have a rolling log of 5 files. Let's say this program kicks off for the first time and there is just one log file. It's fine. The permissions are set at the start of the application.
However, this code is at the beginning of an application that's started once a month. I don't think some-log1.log, some-log2.log, some-log3.log etc will have their permissions set correctly when they are generated because the chmod command only runs at the start of the application.
I don't want to put the chmod command in my main program loop because it will run every 30 seconds. I also don't want to make some other timer just for this one command which would cause me to maybe have access to the new file depending on when it's generated.
Is there a way to set permissions to the log file in the python logging library? Is there a better way to code this?
Dears,
I face an issue with Python: I am creating a log file but once I run my program there is still a link between python and my log file: meaning I can't delete the log file and next log messages will be sent to this log file even if I want to send them elsewhere.
My workaround is to shutdown the kernel and restart but I would like to program it instead of doing it manually. Could you please advise?
My code:
import logging
#initialize the log settings
logging.basicConfig(filename='address.log',level=logging.INFO)
You need to close the FileHandler after using it. See the related article python does not release filehandles to logfile
Simply use:
logging.shutdown()
We can do that this way:
log = logging.getLogger()
for hdle in log.handlers[:]:
if isinstance(hdle,logging.FileHandler):
hdle.close()
log.removeHandler(hdle)
I tried with logging.shutdown() but that doesnt work all the times.
You may like to visit this thread as well for more explanation.
I'm using the ConcurrentLogHandler to log my Python application log messages to a file. Tornado's HTTPServer is used in production as the server, and in certain condition (user form submit), I would like to force the roll over of the file.
The issue I'm having is, when running logger.handlers[0].doRollover(), the roll over does not happen, and I don't see an error either.
Many modules of my apps import the "logger" object from the following module to write to the log:
import logging
import logging.handlers
from cloghandler import ConcurrentRotatingFileHandler
from my_app import app
logger = logging.getLogger('my_app')
hdlr = ConcurrentRotatingFileHandler(app.config['LOG_PATH'], maxBytes=5e+8, backupCount=10)
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logger.setLevel(logging.INFO)
Prior to using ConcurrentLogHandler, I was using the RotatingFileHandler, and performing a logger.handlers[0].doRollover() would result in a "File busy" error triggered on the os.rename down the stack.
Is there a way I could force a rollover with this set up, or are there changes I should make in order to make this possible? I'm using Centos as a server, so the filesystem is UNIX/Linux. I can see a [filename].lock file on the log file at all times while my application is running.
Thank you,
ConcurrentLogHandler can be rolled over by logger.handlers[0].doRollover() indeed. The issue I had was that I used the RotatingLogHandler previously and a .lock file was left in the directory of the log, preventing it from being rolled over. Removing the lock file and keeping the ConcurrentLogHandler as the handler of choice (rather than RotatingLogHandler) solves the problem.
I am running my python application in apache environment and using timedRotatingFileHandler to log.
I have setup logger in a way that It is supposed to rotate midnight everyday. My all processes writes into the same logger file. Somehow logger is missing to log info at times. And sometimes I see logger writing into two files (old file and rotated file) at the same time.
I couldn't able to understand why is this happening? Doesn't TimedrotatingFileHandler work in multiprocess enivironment? If not why is that so?
Please help me to understand..
You can't log (and rotate) to the same file from multiple processes naively because OS wouldn't know how to serialize the write and rotate instructions from 2 different processes. What you are experiencing is known as a race condition as 2 processes are competing to write to the same file and close it and open with a new file handle at the same time at rotation time. Only 1 process will win a new file handle when you rotate, so that may explain the missing log event.
Here's a recipe from Python's documentation with hints about how to log to the same place.
http://docs.python.org/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
Essentially you will want to have a separate process listening to logging events coming from multiple places and then that process will log the events to a single file. You can configure rotation in that listener process too.
If you are not sure how to write this, you can try using a package such as Sentry or Facebook's Scribe. I recommend Sentry because Scribe is not trivial to setup.