I want to add a log viewer tab to my website. The tab is supposed to print the whole log file, and after that print new lines (such as tail -F command in Linux) only. The client Side is in HTML and Javascript, and the server side is in Python.
Here is my tail Python function (I found it in the web):
#cherrypy.expose
def tail(self):
filename = '/opt/abc/logs/myLogFile.log'
f = subprocess.Popen(['tail','-F',filename],\
stdout=subprocess.PIPE,stderr=subprocess.PIPE)
p = select.poll()
p.register(f.stdout)
while True:
if p.poll(1):
print f.stdout.readline()
time.sleep(1)
This code is indeed printing the whole log file. However, each time I add new lines to the file, the file has been printed from the beginning, instead of printing the new lines.
Any suggestions how to fix it? I'm pretty new in Python, so I would appreciate any kind of help.
Check out pytailer
https://github.com/six8/pytailer
Specifically the follow command:
# Follow the file as it grows
for line in tailer.follow(open('/opt/abc/logs/myLogFile.log')):
print line
Related
I'm trying to read a log file, written line by line, via readline.
I'm surprised to observe the following behaviour (code executed in the interpreter, but same happens when variations are executed from a file):
f = open('myfile.log')
line = readline()
while line:
print(line)
line = f.readline()
# --> This displays all lines the file contains so far, as expected
# At this point, I open the log file with a text editor (Vim),
# add a line, save and close the editor.
line = f.readline()
print(line)
# --> I'm expecting to see the new line, but this does not print anything!
Is this behaviour standard? Am I missing something?
Note: I know there are better way to deal with an updated file for instance with generators as pointed here: Reading from a frequently updated file. I'm just interested in understanding the issue with this precise use case.
For your specific use case, the explanation is that Vim uses a write-to-temp strategy. This means that all writing operations are performed on a temporary file.
On the contrary, your scripts reads from the original file, so it does not see any change on it.
To further test, instead of Vim, you can try to directly write on the file using:
echo "Hello World" >> myfile.log
You should see the new line from python.
for following your file, you can use this code:
f = open('myfile.log')
while True:
line = readline()
if not line:
print(line)
I have a problem with figuring out how to add text for each line of getoutput method in python.
The code look like this:
...
def mloop():
for i in csv2hash(SLIST):
...
if i.get('IP'):
r = getoutput(CMD3 % (i['IP'])
for line in iter(r):
print i['IP']
...
The problem is that if I try this code, it will show me syntax error for a ":" in iter(r): and if i delete ":"(which i guess i should not do...) it will show syntax error for print. The CMD3 variable is a bash script for listing all files from folder on ssh server. And after all processing it all goes into csv (which code i did not include here)
I could just add IP in bash script, however I want to do this in python.
Thanks for help!
I think this code will solve your problem. Try this.
def mloop():
for i in csv2hash(SLIST):
...
if i.get('IP'):
r = getoutput(CMD3 % (i['IP']))
for line in iter(r):
print (i['IP'])
I am making a game in python 2.7 for fun and am trying to make a map to go along with it. I am using file I/O to read and write the map and have also got notepad ++ set to silent update, however I can only see the changes once my program has fully run and want to view the file as it is updated.
I have this code which i am testing with:
from time import sleep
map = open('C:\Users\Ryan\Desktop\Codes\Python RPG\Maps\map.txt', 'r+')
map.truncate()
print "file deleted"
sleep(1)
worldMap = open('C:\Users\Ryan\Desktop\Codes\Python RPG\Maps\worldMap.txt', 'r')
for line in worldMap:
map.write(line)
print "file updated"
worldMap.close()
map.close()
Any help is greatly appricated :)
By default Python uses buffered I/O. This means that written data is stored in memory before actually written to the file. Calling file's flush method causes the data to be written to the file.
I am trying to use Pastebin to host two text files for me to allow any copy of my script to update itself through the internet. My code is working, but the resultant .py file has a blank line added between each line. Here is my script...
import os, inspect, urllib2
runningVersion = "1.00.0v"
versionUrl = "http://pastebin.com/raw.php?i=3JqJtUiX"
codeUrl = "http://pastebin.com/raw.php?i=GWqAQ0Xj"
scriptFilePath = (os.path.abspath(inspect.getfile(inspect.currentframe()))).replace("\\", "/")
def checkUpdate(silent=1):
# silently attempt to update the script file by default, post messages if silent==0
# never update if "No_Update.txt" exists in the same folder
if os.path.exists(os.path.dirname(scriptFilePath)+"/No_Update.txt"):
return
try:
versionData = urllib2.urlopen(versionUrl)
except urllib2.URLError:
if silent==0:
print "Connection failed"
return
currentVersion = versionData.read()
if runningVersion!=currentVersion:
if silent==0:
print "There has been an update.\nWould you like to download it?"
try:
codeData = urllib2.urlopen(codeUrl)
except urllib2.URLError:
if silent==0:
print "Connection failed"
return
currentCode = codeData.read()
with open(scriptFilePath.replace(".py","_UPDATED.py"), mode="w") as scriptFile:
scriptFile.write(currentCode)
if silent==0:
print "Your program has been updated.\nChanges will take effect after you restart"
elif silent==0:
print "Your program is up to date"
checkUpdate()
I stripped the GUI (wxpython) and set the script to update another file instead of the actual running one. The "No_Update" bit is for convenience while working.
I noticed that opening the resultant file with Notepad does not show the skipped lines, opening with Wordpad gives a jumbled mess, and opening with Idle shows the skipped lines. Based on that, this seems to be a formatting problem even though the "raw" Pastebin file does not appear to have any formatting.
EDIT: I could just strip all blank lines or leave it as is without any problems, (that I've noticed) but that would greatly reduce readability.
Try adding the binary qualifier in your open():
with open(scriptFilePath.replace(".py","_UPDATED.py"), mode="wb") as scriptFile:
I notice that your file on pastebin is in DOS format, so it has \r\n in it. When you call scriptFile.write(), it translates \r\n to \r\r\n, which is terribly confusing.
Specifying "b" in the open() will cause scriptfile to skip that translate and write the file is DOS format.
In the alternative, you could ensure that the pastebin file has only \n in it, and use mode="w" in your script.
I am trying to set up a mail log parser that will pull out specific lines into another file, which will then get rsync'd to a remote server. The problem I am having is that when rsync reads the file being written, it seems to cause the parser to stop functioning. I believe this is because the parser is emulating a tail -f as maillog is being written consistently.
So: How do I allow rsync to touch the file I'm writing with this code (result_file), while still allowing it to follow the end of the maillog looking for new files:
#! /usr/bin/python
import time, re, sys
result_file = open('/var/log/mrp_mail_parsed.log', 'a+')
def tail(logfile):
logfile.seek(0,2)
while True:
line = logfile.readline()
if not line:
time.sleep(0.1)
continue
yield line
if __name__ == '__main__':
logfile = open('/var/log/maillog', 'r')
logline = tail(logfile)
for line in logline:
match = re.search(r'.+postfix-mrp.+', line)
if match:
result_file.write(line,)
result_file.flush()
I don't know who's writing the file, or how, so I can't be sure, but I'd give better than even odds that your problem is this:
If the file isn't being appended to in-place, but is instead being rewritten, your code will stop tracking the file. To test this:
import sys
import time
def tail(logfile):
logfile.seek(0,2)
while True:
line = logfile.readline()
if not line:
time.sleep(0.1)
continue
yield line
with open(sys.argv[1]) as f:
for line in tail(f):
print(line.rstrip())
Now:
$ touch foo
$ python tailf.py foo &
$ echo "hey" >> foo
foo
$ echo "hey" > foo
To see what's happening better, try checking the inode and size via stat. As soon as the path refers to a different file than the one your script has open, your script is now watching a file that nobody else will ever touch again.
It's also possible that someone is truncating and rewriting the file in-place. This won't change the inode, but it will still mean that you won't read anything, because you're trying to read from a position past the end of the file.
I have no idea whether the file being rsync'd is causing this, or whether that's just a coincidence. Without knowing what rsync command you're running, or seeing whether the file is being replaced or the file is being truncated and rewritten when that command runs, all we can do is guess.
I don't believe rsync is causing your problems: A separate process reading the file shouldn't affect the writer. You can easily test this by pausing rsync.
I'm guessing the problem is with python's handling of file reads when you hit end of file. A crude way that's guaranteed to work is to read to remember the offest at the last EOF (using tell()). For each new read, reopen the file and seek to the remembered offset.