We Keep Coding

using run_id as name of the log file instead of timestamp but run_id is generated during the run

I'm following the widely used approach of naming log file by timestamp. The log file is created in the beginning of the run. My tool generates a run_id during each run which is also logged in the log file.
This worked well, but now as the log files are increasing, it becomes hard whenever a run fails and I need to investigate the log files. I'll be notified which run_id failed, but finding the corresponding log file is hard as I need to do grep -inr <run_id> over all log files to find the relevant log file, which takes some time.
If I could name the log file by run_id, it would have been super simple to just do vim <run_id> whenever a run fails. But the run_id is not know at the time of log file creation and is rather generated during the run by a sequence generator in backend database.
What would be the ideal solution in this case?
Should I rename the file at the end of each run? Or is there any other approach I am missing.

As you said renaming the file at the end of the run seems like the simplest solution. You'll just have to watch out for the possibility that the run fails before run_id is created and the log retains its initial name.
If you want you could append run_id to the timestamp if you still want it in the filename.

How do I integrate logging in Python with actual text output?

The intuitive way to do this would be to have a logging level that just always prints to any log file for actual program output. But there is no such level - the top level is logging.critical, which should correspond to a fatal error rather than the successful output of the program.
As far as I can tell, the only setting for system output is logging.info which actually has low priority. If I used it, then the log file would also contain exception warnings. What is the right way to get "program output" into a file using logger?

Python - how to disconnect from a log file

Dears,
I face an issue with Python: I am creating a log file but once I run my program there is still a link between python and my log file: meaning I can't delete the log file and next log messages will be sent to this log file even if I want to send them elsewhere.
My workaround is to shutdown the kernel and restart but I would like to program it instead of doing it manually. Could you please advise?
My code:
import logging
#initialize the log settings
logging.basicConfig(filename='address.log',level=logging.INFO)

You need to close the FileHandler after using it. See the related article python does not release filehandles to logfile

Simply use:
logging.shutdown()

We can do that this way:
log = logging.getLogger()
for hdle in log.handlers[:]:
if isinstance(hdle,logging.FileHandler):
hdle.close()
log.removeHandler(hdle)
I tried with logging.shutdown() but that doesnt work all the times.
You may like to visit this thread as well for more explanation.

Python2: How to parse a logfile that is held open in another process reliably?

I'm trying to write a Python script that will parse a logfile produced by another daemon. This is being done on Linux. I want to be able to parse the log file reliably.
In other words, periodically, we run a script that reads the log file, line by line, and does something with each line. The logging script would need to see every line that may end up in the log file. It could run say once per minute via cron.
Here's the problem that I'm not sure exactly how to solve. Since the other process has a write handle to the file, it could write to the while at the same time that I am reading from the same log file.
Also, every so often we would want to clear this logfile so its size does not get out of control. But the process producing the log file has no way to clear the file other than regularly stopping, truncating or deleting the file, and then restarting. (I feel like logrotate has some method of doing this, but I don't know if logrotate depends on the daemon being aware, or if it's actually closing and restarting daemons, etc. Not to mention I don't want other logs rotated, just this one specific log; and I don't want this script to require other possible users to setup logrotate.)
Here's the problems:
Since the logger process could write to the file while I already have an open file handle, I feel like I could easily miss records in the log file.
If the logger process were to decide to stop, clear the log file, and restart, and the log analyzer didn't run at exactly the same time, log entries would be lost. Similarly, if the log analyzer causes the logger to stop logging while it analyzes, information could also be lost that is dropped because the logger daemon isn't listening.
If I were to use a method like "note the size of the file since last time and seek there if the file is larger", then what would happen if, for some reason, between runs, the logger reset the logfile, but then had reason to log even more than it contained last time? E.g. We execute a log analyze loop. We get 50 log entries, so we set a mark that we have read 50 entries. Next time we run, we see 60 entries. But, all 60 are brand new; the file had been cleared and restarted since the last log run. Instead we end up seeking to entry 51 and missing 50 entries! Either way it doesn't solve the problem of needing to periodically clear the log.
I have no control over the logger daemon. (Imagine we're talking about something like syslog here. It's not syslog but same idea - a process that is pretty critical holds a logfile open.) So I have no way to change its logging method. It starts at init time, opens a log file, and writes to it. We want to be able to clear that logfile AND analyze it, making sure we get every log entry through the Python script at some point.
The ideal scenario would be this:
The log daemon runs at system init.
Via cron, the Python log analyzer runs once per minute (or once per 5 minutes or whatever is deemed appropriate)
The log analyzer collects every single line from the current log file and immediately truncates it, causing the log file to be blanked out. Python maintains the original contents in a list.
The logger then continues to go about its business, with the now blank file. In the mean time, Python can continue to parse the entries at its leisure from the Python list in memory.
I've very, very vaguely studied fifo's, but am not sure if that would be appropriate. In that scenario the log analyzer would run as a daemon itself, while the original logger writes to a FIFO. I have very little knowledge in this area however and don't know if it'd be a solution or not.
So I guess the question really is twofold:
How to reliably read EVERY entry written to the log from Python? Including if the log grows, is reset, etc.
How, if possible to truncate a file that has an open write handle? (Ideally, this would be something I could do from Python; I could do something like logfile.readlines(); logfile.truncate so that way no entries would get lost. But this seems like unless the logger process was well aware of this, it'd end up causing more problems than it solves.)
Thanks!

I don’t see any particular reason why you should be not able to read log file created by syslogd. You are saying that you are using some process similar to syslog, and process is keeping your log file open? Since you are asking rather for ideas, I would recommend you to use syslog! http://pic.dhe.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=%2Fcom.ibm.ztpf-ztpfdf.doc_put.cur%2Fgtpc1%2Fhsyslog.html
It is working anyway – use it. Some easy way to write to log is to use logger command:
logger “MYAP: hello”
In python script you can do it like:
import os
os.system(‘logger “MYAP: hello”’)
Also remember you can actually configure syslogd. http://pic.dhe.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=%2Fcom.ibm.ztpf-ztpfdf.doc_put.cur%2Fgtpc1%2Fconstmt.html
Also about your problem with empty logs – sysclog is not clearing logs. There are other tools for it – on debian for example logrotate is used. In this scenario if your log is empty – you can check backup file created by logrotate.
Since it looks like your problem is in logging tool, my advise would be to use syslog for logging. And other tool for rotating logs. Then you can easily parse logs. And if by any means (I don’t know if it is even possible with syslog) you miss some data – remember you will get it in next iteration anyway ;)
Some other idea would be to copy your logfile and work with copy...

Python log file in descending order

I have an application written in python that logs errors in a log file, I am using the python logging module
Example:
import logging
logging.basicConfig(filename='\logs\filename.log',
level=logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s')
logging.error('error connecting to server')
This is working fine and the log file is logging the errors, which mean the last error is logged on the last line of the file.
is there some kind of setting where I can tell the logging module to always write at the top of the file, this way the last error is always on the first line.

It is very inefficient to write to the front of a flle as others have said. Every single write will have to first seek to the front of the file and then insert the new data before the other data. The underlying I/O of your operating system is designed to make appends cheap.
That said, if you insist on doing it, look at the docs here Logging Handlers. You can implement your own version of logging.handlers.FileHandler that will seek to the beginning before each write. Then you could call logging.addHandler() and place an instance of your class. If you only care about the most recent 10 or so log entries you could even truncate the file before you write.