I'm trying to build an application that displays in a GUI the contents of a log file, written by a separate program that I call through subprocess. The application runs in Windows, and is a binary that I have no control over. Also, this application (Actel Designer if anyone cares) will write its output to a log file regardless of how I redirect the output of subprocess, so using a pipe for the output doesn't seem to be an option. The bottom line is that I seem to be forced into reading from a log file at the same time another thread may be writing to it. My question is if there is a way that I can keep the GUI's display of the log file's contents up to date in a robust way?
I've tried the following:
Naively opening the file for reading periodically while the child
process is running causes Python to crash (I'm guessing because the
child thread is writing to the file while I'm attempting to read its
contents)
Next I tried to open a file handle to the log filename before invoking the child process with GENERIC_READ, and SHARED_READ | SHARED_WRITE | SHARED_DELETE and reading back from that file. With this approach, the file appears empty
Thanks for any help you can provide - I'm not a professional programmer and I've been pulling my hair out over this for a week.
You should register for notifications on file change, the way tail -f does (you can find out what system calls it uses by executing strace tail -f logfile).
pyinotify provides a Python interface for these file change notifications.
Related
I've read that there is a way to open a file in python with..
os.startfile('file.exe')
is there a way to close the same file when open?
Thank you in advance!
From the os.startfile() doc:
startfile() returns as soon as the associated application is launched. There is no option to wait for the application to close, and no way to retrieve the application’s exit status.
So, basically, no, there isn't a way to close a file opened with startfile.
It isn't clear from the question is whether you want to launch a file, or to open it (for reading/writing).
If you want to launch a process, subprocess is a better candidate for running other processes and controlling them through a subshell (including killing them.)
If you want to open a file for read/write, then open() would be a good choice to start with.
As the Python Wiki says this function does the following:
Start a file with its associated application.
So the best idea would be to use os.kill to kill the application in which it is opened. The problem lies in identyfying what application is associated with the file of following extension and finding pid of exact instance which opened that file.
You have used .exe file in example which is executable file extension, so you probably misunderstood what this function does. What are you trying to accomplish? Are you sure it is the correct way of doing it?
If you really want to launch executable file, you should probably use os.system(). If you want to create a new file, write something in it and close it, look for python file operations, here are good examples: http://www.tutorialspoint.com/python/python_files_io.htm
Question: Is there a way, using Python, to access the stdout of a running process? This process has not been started by Python.
Context: There is a program called mayabatch, that renders out images from 3D Maya scene files. If I were to run the program from the command line I would see progress messages from mayabatch. Sometimes, artists close these windows, leaving the progress untracable until the program finishes. That led me along this route of trying to read its stdout after it's been spawned by a foreign process.
Background:
OS: Windows 7 64-bit
My research so far: I have only found questions and answers of how to do this if it was a subprocess, using the subprocess module. I also looked briefly into psutil, but I could not find any way to read a process' stdout.
Any help would be really appreciated. Thank you.
I don't think you can get to the stdout of a process outside of the code that created it
The lazy way to is just to pipe the output of mayabatch to a text file, and then poll the text file periodically in your own code so it's under your control, rather than forcing you to wait on the pipe (which is especially hard on Windows, since Windows select doesn't work with the pipes used by subprocess.
I think this is what maya does internally too: by default mayaBatch logs its results to a file called mayaRenderLog.txt in the user's Maya directory.
If you're running mayabatch from the command line or a bat file, you can funnel stdout to a file with a > character:
mayabatch.exe "file.ma" > log.txt
You should be able to poll that text file from the outside using standard python as long as you only open it for reading. The advantage of doing it this way is that you control the frequency at which you check the file.
OTOH If you're doing it from python, it's a little tougher unless you don't mind having your python script idled until the mayabatch completes. The usual subprocess recipe, which uses popen.communicate() is going to wait for an end-of-process return code:
test = subprocess.Popen(["mayabatch.exe","filename.mb"], stdout=subprocess.PIPE)
print test.communicate()[0]
works but won't report until the process dies. But you calling readlines on the process's stdout will trigger the process and report it one line at a time:
test = subprocess.Popen(["mayabatch.exe","filename.mb"], stdout=subprocess.PIPE)
reader = iter(test.subprocess.readlines, "")
for line in reader:
print line
More discussion here
I have a library that interacts with a configuration file. When the library is imported, the initialization code reads the configuration file, possibly updates it, and then writes the updated contents back to the file (even if nothing was changed).
Very occasionally, I encounter a problem where the contents of the configuration file simply disappear. Specifically, this happens when I run many invocations of a short script (using the library), back-to-back, thousands of times. It never occurs during the same directories, which leads me to believe it's a somewhat random problem--specifically a race condition with IO.
This is a pain to debug, since I can never reliably reproduce the problem and it only happens on some systems. I have a suspicion about what might happen, but I wanted to see if my picture of file I/O in Python is correct.
So the question is, when does a Python program actually write file contents to a disk? I thought that the contents would make it to disk by the time that the file closed, but then I can't explain this error. When python closes a file, does it flush the contents to the disk itself, or simply queue it up to the filesystem? Is it possible that file contents can be written to disk after Python terminates? And can I avoid this issue by using fp.flush(); os.fsync(fp.fileno()) (where fp is the file handle)?
If it matters, I'm programming on a Unix system (Mac OS X, specifically). Edit: Also, keep in mind that the processes are not running concurrently.
Appendix: Here is the specific race condition that I suspect:
Process #1 is invoked.
Process #1 opens the configuration file in read mode and closes it when finished.
Process #1 opens the configuration file in write mode, erasing all of its contents. The erasing of the contents is synced to the disk.
Process #1 writes the new contents to the file handle and closes it.
Process #1: Upon closing the file, Python tells the OS to queue writing these contents to disk.
Process #1 closes and exits
Process #2 is invoked
Process #2 opens the configuration file in read mode, but new contents aren't synced yet. Process #2 sees an empty file.
The OS finally finishes writing the contents to disk, after process 2 reads the file
Process #2, thinking the file is empty, sets defaults for the configuration file.
Process #2 writes its version of the configuration file to disk, overwriting the last version.
It is almost certainly not python's fault. If python closes the file, OR exits cleanly (rather than killed by a signal), then the OS will have the new contents for the file. Any subsequent open should return the new contents. There must be something more complicated going on. Here are some thoughts.
What you describe sounds more likely to be a filesystem bug than a Python bug, and a filesystem bug is pretty unlikely.
Filesystem bugs are far more likely if your files actually reside in a remote filesystem. Do they?
Do all the processes use the same file? Do "ls -li" on the file to see its inode number, and see if it ever changes. In your scenario, it should not. Is it possible that something is moving files, or moving directories, or deleting directories and recreating them? Are there symlinks involved?
Are you sure that there is no overlap in the running of your programs? Are any of them run from a shell with "&" at the end (i.e. in the background)? That could easily mean that a second one is started before the first one is finished.
Are there any other programs writing to the same file?
This isn't your question, but if you need atomic changes (so that any program running in parallel only sees either the old version or the new one, never the empty file), the way to achieve it is to write the new content to another file (e.g. "foo.tmp"), then do os.rename("foo.tmp", "foo"). Rename is atomic.
I've read that there is a way to open a file in python with..
os.startfile('file.exe')
is there a way to close the same file when open?
Thank you in advance!
From the os.startfile() doc:
startfile() returns as soon as the associated application is launched. There is no option to wait for the application to close, and no way to retrieve the application’s exit status.
So, basically, no, there isn't a way to close a file opened with startfile.
It isn't clear from the question is whether you want to launch a file, or to open it (for reading/writing).
If you want to launch a process, subprocess is a better candidate for running other processes and controlling them through a subshell (including killing them.)
If you want to open a file for read/write, then open() would be a good choice to start with.
As the Python Wiki says this function does the following:
Start a file with its associated application.
So the best idea would be to use os.kill to kill the application in which it is opened. The problem lies in identyfying what application is associated with the file of following extension and finding pid of exact instance which opened that file.
You have used .exe file in example which is executable file extension, so you probably misunderstood what this function does. What are you trying to accomplish? Are you sure it is the correct way of doing it?
If you really want to launch executable file, you should probably use os.system(). If you want to create a new file, write something in it and close it, look for python file operations, here are good examples: http://www.tutorialspoint.com/python/python_files_io.htm
When my Python script is writing a large amount of logs to a text file line by line using the Python built-in logging library, in my Delphi-powered Windows program I want to effectively read all newly added logs (lines).
When the Python scripting is logging
to the file, my Windows program will
keep a readonly file handle to
that log file;
I'll use the Windows API to get
informed when the log file is
changed; Once the file is changed, it'll read the newly appended lines.
I'm new to Python, do you see any possible problem with this approach? Does the Python logging lib lock the entire log? Thanks!
It depends on the logging handler you use, of course, but as you can see from the source code, logging.FileHandler does not currently create any file locks. By default, it opens files in 'a' (append) mode, so as long as your Windows calls can handle that, you should be fine.
As ʇsәɹoɈ commented, the standard FileHandler logger does not lock the file, so it should work. However, if for some reason you cannot keep you lock on the file - then I'd recommend having your other app open the file periodically, record the position it's read to and then seek back to that point later. I know the Linux DenyHosts program uses this approach when dealing with log files that it has to monitor for a long period of time. In those situations, simply holding a lock isn't feasible, since directories may move, the file get rotated out, etc. Though it does complicate things in that then you have to store filename + read position in persistent state somewhere.