The intuitive way to do this would be to have a logging level that just always prints to any log file for actual program output. But there is no such level - the top level is logging.critical, which should correspond to a fatal error rather than the successful output of the program.
As far as I can tell, the only setting for system output is logging.info which actually has low priority. If I used it, then the log file would also contain exception warnings. What is the right way to get "program output" into a file using logger?
Related
I am trying to capture the logs of my python-behave tests (produced by the logging module) by using this command to run them:
behave -f pretty --logging-level INFO --capture "features/file_system_operations.feature"
This executes the tests, but even though they are passing, all my info lines are printed to sceen instead of being captured, eg.:
When I change the logging level to warning, the code properly responds and no lines are printed:
behave -f pretty --logging-level WARNING --capture "features/file_system_operations.feature"
Results a clean printout:
How can I ask behave to have INFO logging lines printed only if a test fails?
I think I managed to find was going on, and it seems like logging imported multiple times is the root cause. I am not sure if this is a feature or a bug, but will try to patch around the issue by extracting logging functionality into its own module.
i have another python problem related to logging. I want to have a message logged to my log file no matter which log level i'm using. For example i always want to print the time when the program was last executed and also the git version hash of the script.
Is there a way to log something to the log file no matter what log level is set?
Thanks in advance!
According to Python's Logging tutorial, logging.error is to be used to "report suppression of an error without raising an exception.", whereas if I wanted to report an error I'm supposed to just raise an exception and not use logging at all.
What if I want to report an error and then raise an exception?
For example:
try:
os.path.getsize('/nonexistent')
except os.error as error:
logging.error(str(error))
logging.error('something went wrong')
raise SystemExit(1)
This results in the following lines being printed to the standard error:
ERROR:root:[Errno 2] No such file or directory: '/nonexistent'
ERROR:root:something went wrong
It seems sensible. Or:
try:
os.path.getsize('/nonexistent')
except os.error as error:
logging.error(str(error))
raise SomeUserDefinedException('something went wrong')
Is this bad practice?
logging is used to report.
raise is an action which interrupts the normal flow of the program.
Typically, you raise exceptions where there is an actual flow to interrupt, and when you want to report, you can use logging, which will log errors and exceptions by default.
The logging part usually comes as a wrapper around a module, class, or middleware in your program.
logging within the code should only be used for DEBUG purposes, or for INFO purpose if you wanted to keep track of some information in your server logs for example. Not for ERROR.
I do not believe that is a bad practice. Exceptions and logging serve two different purposes and in any given situation you may want one, the other or both. Exceptions are for programmatic consumption and control flow within your code, while logging is for external consumption, by either a human or some sort of automated log watcher (Splunk for example). For instance, you may log the values of certain variables to help you figure out what happened and improve the code, though in that case, a level of debug would be more appropriate. Or you may want a systems administrator to be aware of the issue and potentially take some action. In that case, setting the log level to error and using something like a syslog logging handler makes sense.
I'm trying to write a Python script that will parse a logfile produced by another daemon. This is being done on Linux. I want to be able to parse the log file reliably.
In other words, periodically, we run a script that reads the log file, line by line, and does something with each line. The logging script would need to see every line that may end up in the log file. It could run say once per minute via cron.
Here's the problem that I'm not sure exactly how to solve. Since the other process has a write handle to the file, it could write to the while at the same time that I am reading from the same log file.
Also, every so often we would want to clear this logfile so its size does not get out of control. But the process producing the log file has no way to clear the file other than regularly stopping, truncating or deleting the file, and then restarting. (I feel like logrotate has some method of doing this, but I don't know if logrotate depends on the daemon being aware, or if it's actually closing and restarting daemons, etc. Not to mention I don't want other logs rotated, just this one specific log; and I don't want this script to require other possible users to setup logrotate.)
Here's the problems:
Since the logger process could write to the file while I already have an open file handle, I feel like I could easily miss records in the log file.
If the logger process were to decide to stop, clear the log file, and restart, and the log analyzer didn't run at exactly the same time, log entries would be lost. Similarly, if the log analyzer causes the logger to stop logging while it analyzes, information could also be lost that is dropped because the logger daemon isn't listening.
If I were to use a method like "note the size of the file since last time and seek there if the file is larger", then what would happen if, for some reason, between runs, the logger reset the logfile, but then had reason to log even more than it contained last time? E.g. We execute a log analyze loop. We get 50 log entries, so we set a mark that we have read 50 entries. Next time we run, we see 60 entries. But, all 60 are brand new; the file had been cleared and restarted since the last log run. Instead we end up seeking to entry 51 and missing 50 entries! Either way it doesn't solve the problem of needing to periodically clear the log.
I have no control over the logger daemon. (Imagine we're talking about something like syslog here. It's not syslog but same idea - a process that is pretty critical holds a logfile open.) So I have no way to change its logging method. It starts at init time, opens a log file, and writes to it. We want to be able to clear that logfile AND analyze it, making sure we get every log entry through the Python script at some point.
The ideal scenario would be this:
The log daemon runs at system init.
Via cron, the Python log analyzer runs once per minute (or once per 5 minutes or whatever is deemed appropriate)
The log analyzer collects every single line from the current log file and immediately truncates it, causing the log file to be blanked out. Python maintains the original contents in a list.
The logger then continues to go about its business, with the now blank file. In the mean time, Python can continue to parse the entries at its leisure from the Python list in memory.
I've very, very vaguely studied fifo's, but am not sure if that would be appropriate. In that scenario the log analyzer would run as a daemon itself, while the original logger writes to a FIFO. I have very little knowledge in this area however and don't know if it'd be a solution or not.
So I guess the question really is twofold:
How to reliably read EVERY entry written to the log from Python? Including if the log grows, is reset, etc.
How, if possible to truncate a file that has an open write handle? (Ideally, this would be something I could do from Python; I could do something like logfile.readlines(); logfile.truncate so that way no entries would get lost. But this seems like unless the logger process was well aware of this, it'd end up causing more problems than it solves.)
Thanks!
I don’t see any particular reason why you should be not able to read log file created by syslogd. You are saying that you are using some process similar to syslog, and process is keeping your log file open? Since you are asking rather for ideas, I would recommend you to use syslog! http://pic.dhe.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=%2Fcom.ibm.ztpf-ztpfdf.doc_put.cur%2Fgtpc1%2Fhsyslog.html
It is working anyway – use it. Some easy way to write to log is to use logger command:
logger “MYAP: hello”
In python script you can do it like:
import os
os.system(‘logger “MYAP: hello”’)
Also remember you can actually configure syslogd. http://pic.dhe.ibm.com/infocenter/tpfhelp/current/index.jsp?topic=%2Fcom.ibm.ztpf-ztpfdf.doc_put.cur%2Fgtpc1%2Fconstmt.html
Also about your problem with empty logs – sysclog is not clearing logs. There are other tools for it – on debian for example logrotate is used. In this scenario if your log is empty – you can check backup file created by logrotate.
Since it looks like your problem is in logging tool, my advise would be to use syslog for logging. And other tool for rotating logs. Then you can easily parse logs. And if by any means (I don’t know if it is even possible with syslog) you miss some data – remember you will get it in next iteration anyway ;)
Some other idea would be to copy your logfile and work with copy...
I have an application written in python that logs errors in a log file, I am using the python logging module
Example:
import logging
logging.basicConfig(filename='\logs\filename.log',
level=logging.DEBUG,
format='%(asctime)s - %(levelname)s - %(message)s')
logging.error('error connecting to server')
This is working fine and the log file is logging the errors, which mean the last error is logged on the last line of the file.
is there some kind of setting where I can tell the logging module to always write at the top of the file, this way the last error is always on the first line.
It is very inefficient to write to the front of a flle as others have said. Every single write will have to first seek to the front of the file and then insert the new data before the other data. The underlying I/O of your operating system is designed to make appends cheap.
That said, if you insist on doing it, look at the docs here Logging Handlers. You can implement your own version of logging.handlers.FileHandler that will seek to the beginning before each write. Then you could call logging.addHandler() and place an instance of your class. If you only care about the most recent 10 or so log entries you could even truncate the file before you write.