Dears,
I face an issue with Python: I am creating a log file but once I run my program there is still a link between python and my log file: meaning I can't delete the log file and next log messages will be sent to this log file even if I want to send them elsewhere.
My workaround is to shutdown the kernel and restart but I would like to program it instead of doing it manually. Could you please advise?
My code:
import logging
#initialize the log settings
logging.basicConfig(filename='address.log',level=logging.INFO)
You need to close the FileHandler after using it. See the related article python does not release filehandles to logfile
Simply use:
logging.shutdown()
We can do that this way:
log = logging.getLogger()
for hdle in log.handlers[:]:
if isinstance(hdle,logging.FileHandler):
hdle.close()
log.removeHandler(hdle)
I tried with logging.shutdown() but that doesnt work all the times.
You may like to visit this thread as well for more explanation.
Related
I'm working with the Python logging configuration below
import logging
import os
#Setup logging
logger = logging.getLogger('/logs/some-log')
logger.setLevel(logging.INFO)
handler = RotatingFileHandler('./logs/some-log.log', maxBytes=10240, backupCount=5)
logger.addHandler(handler)
#End Logging
#Set Logging Permissions
os.chmod('/logs/some*.log', 0o755)
#End Set Logging Permissions
I have a rolling log of 5 files. Let's say this program kicks off for the first time and there is just one log file. It's fine. The permissions are set at the start of the application.
However, this code is at the beginning of an application that's started once a month. I don't think some-log1.log, some-log2.log, some-log3.log etc will have their permissions set correctly when they are generated because the chmod command only runs at the start of the application.
I don't want to put the chmod command in my main program loop because it will run every 30 seconds. I also don't want to make some other timer just for this one command which would cause me to maybe have access to the new file depending on when it's generated.
Is there a way to set permissions to the log file in the python logging library? Is there a better way to code this?
In the AWSIoTPythonSDK, in MqttCore.py, there is logger define '''class MqttCore(object):
_logger = logging.getLogger(name)'''
Everytime "subscribe timed out" is getting printed in the console, is there any way to forcefully stop the logger of the IoT SDK. I know it is not recommended but i badly needed to stop it forcefully.
I dont know if it recommended, but I did it:
sdk_logger = logging.getLogger('AWSIoTPythonSDK.core.protocol.mqtt_core')
sdk_logger.setLevel(logging.WARNING)
you can see here an example that they printed it to a different file:
https://docs.aws.amazon.com/code-samples/latest/catalog/python-iotthingsgraph-camera.py.html
this is a very simple explanation on how logging works in general when you import a module (If you need) https://www.youtube.com/watch?v=jxmzY9soFXg&ab_channel=CoreySchafer
credit: How do I define a different logger for an imported module in Python?
i have another python problem related to logging. I want to have a message logged to my log file no matter which log level i'm using. For example i always want to print the time when the program was last executed and also the git version hash of the script.
Is there a way to log something to the log file no matter what log level is set?
Thanks in advance!
def fn(filename):
import logging
logger = logging.getLogger(filename)
formatter = logging.Formatter('[%(asctime)s:%(msecs)d], [%(levelname)s], %(message)s, ')
handler = logging.FileHandler(filename + ".log")
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.error("that's an error")
Basically I'm just using the python logging library for logging. The code is working fine if I run it as a normal python function.
However if I use it in celery/supervisord:
fn.delay("loggingFile")
The file is opened and never closed (by the user used in supervisord). Of course, after some time, I have hundreds of opened files which is very bad and causes other troubles. Why is this happening with celery tasks only and how to avoid it?
I know the files are never closed by monitoring the system using lsof to see which users are opening which files.
Appreciate any help
For some reason, the daemon does not close the file handler. you will need to it manually using after you finish logging:
logger.removeHandler(handler)
I am running my python application in apache environment and using timedRotatingFileHandler to log.
I have setup logger in a way that It is supposed to rotate midnight everyday. My all processes writes into the same logger file. Somehow logger is missing to log info at times. And sometimes I see logger writing into two files (old file and rotated file) at the same time.
I couldn't able to understand why is this happening? Doesn't TimedrotatingFileHandler work in multiprocess enivironment? If not why is that so?
Please help me to understand..
You can't log (and rotate) to the same file from multiple processes naively because OS wouldn't know how to serialize the write and rotate instructions from 2 different processes. What you are experiencing is known as a race condition as 2 processes are competing to write to the same file and close it and open with a new file handle at the same time at rotation time. Only 1 process will win a new file handle when you rotate, so that may explain the missing log event.
Here's a recipe from Python's documentation with hints about how to log to the same place.
http://docs.python.org/howto/logging-cookbook.html#logging-to-a-single-file-from-multiple-processes
Essentially you will want to have a separate process listening to logging events coming from multiple places and then that process will log the events to a single file. You can configure rotation in that listener process too.
If you are not sure how to write this, you can try using a package such as Sentry or Facebook's Scribe. I recommend Sentry because Scribe is not trivial to setup.