I came across rotating file handler in order to rotate log file in Python, I have this piece of code.
My requirement is my log file is getting appended every few seconds, and there should be log rotation, whenever the file is beyond 1 GB of size.
This is my pseudo code, where I am trying only for 40 bytes, to check if my code works.
import logging
import time
from logging.handlers import RotatingFileHandler
#----------------------------------------------------------------------
def create_rotating_log(path):
"""
Creates a rotating log
"""
logger = logging.getLogger("Rotating Log")
logger.setLevel(logging.INFO)
# add a rotating handler
handler = RotatingFileHandler(path, maxBytes=40,
backupCount=1)
logger.addHandler(handler)
#----------------------------------------------------------------------
if __name__ == "__main__":
log_file = "test.log"
create_rotating_log(log_file)
Now as per this code, whenever my log file reaches 40 bytes, there should be a copy of the log file as test.log.1 and my older log file should have new content.
So, I manually opened my test.log file, added contents to it to make sure size > 40 bytes and ran this piece of code, but nothing happened.My requirement is my log file is getting appended every few seconds, and there should be log rotation, whenever the file is beyond 1 GB of size.
What is the correct way of running the code for my requirement?? I am stuck here.
Can someone please help?
As it was adviced at the comment of #jordanm, I've tried
import logging
from logging.handlers import RotatingFileHandler
#----------------------------------------------------------------------
def create_rotating_log(path):
"""
Creates a rotating log
"""
logger = logging.getLogger("Rotating Log")
logger.setLevel(logging.INFO)
# add a rotating handler
handler = RotatingFileHandler(path, maxBytes=40,
backupCount=1)
logger.addHandler(handler)
#----------------------------------------------------------------------
if __name__ == "__main__":
log_file = "test.log"
create_rotating_log(log_file)
logger = logging.getLogger("Rotating Log")
logger.info("Hello!")
It works.
Related
logger configuration to log to file and print to stdout would not work with me so:
I want to both print logging details in prog_log.txt AND in the console, I have:
# Debug Settings
Import logging
#logging.disable() #when program ready, un-indent this line to remove all logging messages
logger = logging.getLogger('')
logging.basicConfig(level=logging.DEBUG,
filename = 'prog_log.txt',
format='%(asctime)s - %(levelname)s - %(message)s',
filemode = 'w')
logger.addHandler(logging.StreamHandler())
logging.debug('Start of Program')
The above does print into the prog_log.txt file the logging details properly formated but it prints unformated logging messages multiple times as I re-run the program in the console like below..
In [3]: runfile('/Volumes/GoogleDrive/Mon Drive/MAC_test.py', wdir='/Volumes/GoogleDrive/Mon Drive/')
Start of Program
Start of Program
Start of Program #after third running
Any help welcome ;)
I need your help in solving this problem. So I am using Python's timedrotatingfilehandler lib to append logs for 24 hours in a log file and then rotate the log file. This works as expected but every time I run my script, timedrotatingfilehandler only logs till certain point. For example: If I have 10 log statement, each run only logs till 5 and then nothing gets logged in the log file. Appending happens correctly though. Not sure whats happening. Here is the python code:
#Configuring logger
LOG = logging.getLogger(__name__)
def config_Logs():
# format the log entries
global LOG
formatter = logging.Formatter('[%(asctime)s] : {%(pathname)s:%(lineno)d} %(levelname)s - %(message)s')
handler = TimedRotatingFileHandler('collecting_logs.log',
when='h', interval=24, backupCount=0)
handler.setFormatter(formatter)
LOG.addHandler(handler)
LOG.setLevel(logging.DEBUG)
I am calling config_Logs() at the beginning of the script.
I want to log a line describing the logging information for each log file created by the logger.
Currently I'm using a separate logger process which will be running all the time. It receives the information from a queue and writes it to the log. A lot of modules will be passing information to this logging queue.
My current sample code is:
import logging
import time
from logging.handlers import TimedRotatingFileHandler as rotate
def info_log(log_queue):
logger = logging.getLogger("My Log")
logger.setLevel(logging.INFO)
handler = rotate("log/info.log", when="D", interval=30, backupCount=13)
logger.addHandler(handler)
desc_string = "yyyy/mm/dd-HH:MM:SS \t name \t country \n"
logger.info(desc_string)
while True:
result=log_queue.get().split("#")
logger.info(result[0] + "\t" result[1] + "\t" + result[2] + "\n")
Each time the log is rotated, I want desc_string to be written 1st to the log file.How can I do that?
Or in other words, How to know in the program when a log is rotated?
Maybe you can simply override the doRollover method from TimedRotatingFileHandler ?
class CustomFileHandler(TimedRotatingFileHandler):
def doRollover(self):
super().doRollover()
self.stream.write(desc_string)
# to use it
handler = CustomFileHandler("log/info.log", when="D", interval=30, backupCount=13)
logger.addHandler(handler)
Hi I am trying a sample program using logger in python
import logging
import time,sys
import os
logger = logging.getLogger('myapp')
hdlr = logging.FileHandler('myapp1234.log')
formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')
hdlr.setFormatter(formatter)
logger.addHandler(hdlr)
logging.getLogger().setLevel(logging.DEBUG)
logger.error('We have a problem')
logger.info('While this is just chatty')
logger.debug("Sample")
hdlr.flush()
time.sleep(10)
logger.error('We have a problem')
logger.info('While this is just chatty')
logger.debug("Sample")
hdlr.close()
This code is not dynamically printing. I tried even handler.flush, sys.exit(0), sys.stdout.
When I try to open a file even by killing I am getting following error. Log is only printing after 120-200 seconds (sometimes it's taking even more).
How can I print immediately (at least by end of program)?
Did I miss any Handel closing.
Try removing the following statement.
time.sleep(10)
I am learning the logging module in Python.
However, if I log like this
logging.basicConfig(filename='mylog.log',format='%(asctime)s - %(levelname)s - %(message)s', level=logging.DEBUG)
while 1:
logging.debug("something")
time.sleep(1)
and interrupt the process with control-C event(or the process is killed), nothing I can got from the log file.
Can I save the most logs whatever happens?
————
EDIT
the question seem become more complex:
I have imported scipy, numpy, pyaudio in my script, and I got:
forrtl: error (200): program aborting due to control-C event
instead of KeyboardInterrupt
I have read this question: Ctrl-C crashes Python after importing scipy.stats
and add these line to my script:
import _thread
import win32api
def handler(dwCtrlType, hook_sigint=_thread.interrupt_main):
if dwCtrlType == 0: # CTRL_C_EVENT
hook_sigint()
return 1 # don't chain to the next handler
return 0 # chain to the next handler
then:
try:
main()
except KeyboardInterrupt:
print("exit manually")
exit()
Now, the script stops without any info if I use ctrl+C. print("exit manually") does not appear. Of course, no logs.
Solved
A stupid mistake!
I run the script when working directory is System32 and want to find log in the script's path.
After I change the route like this, all is well.
logging.basicConfig(filename=os.path.dirname(sys.argv[0])+os.sep+'mylog.log',format='%(asctime)s - %(levelname)s - %(message)s', level=logging.DEBUG)
When you log using logging.debug, logging.info, ..., logging.critical, you're using the root logger. I assume you're not doing anything to configure logging that you haven't shown, so you're running with the out-of-the-box, default configuration. (This is set up for you by the first call to logging.debug, which calls logging.basicConfig()).
The default logging level of the root logger is logging.WARNING (as mentioned in e.g. https://docs.python.org/3/howto/logging.html#logging-basic-tutorial). Thus, nothing you log with logging.debug or logging.info will appear :) If you change logging.debug to logging.warning (or .error or .critical), you will see logging output.
For your code to work as is, set the logging level of the root logger to logging.DEBUG before the loop:
import logging
import time
# logging.getLogger() returns the root logger
logging.getLogger().setLevel(logging.DEBUG)
while 1:
logging.debug("something")
time.sleep(1)
For the CTRL + C event use a try-except to catch the KeyboardInterrupt exception.