I want my program to wait until a specific file will contain text instead of empty string. Another program writes data to the file. When I run the first program my computer starts overheating because of the while loop that continously checks the file content. What can I do instead of that loop?
A better solution would be to start that process from within your Python script:
from subprocess import call
retcode = call(['myprocess', 'arg1', 'arg2', 'argN'])
Check if retcode is zero, this means success--your process ran successfully with no problems. You could also use os.system instead of subprocess.call. Once the process is finished, you would know now you can read the file.
Why this method is better than monitoring files?
The process might fail and there might be no output in the file you're trying to read from.
In this case scenario, your process will check the file again and again, looking for data, this wastes kernel I/O operation time. There's nothing that could guarantee that the process will succeed at all times.
The process may receive signals, (i,e. STOP and CONT), if the process received the STOP signal, the kernel will stop the process and there might be nothing that you could read from the output file, especially if you intend to read all the data at once like when you're sorting a file. Once the process receives CONT signal, there the process will start again. Basically, this means your Python script will be trying to read simultaneously from the file while the process is stopped.
The disadvantage of this method is that, the process needs to finish first before your Python script process the output from the file. The subprocess.call blocks, the next line won't be executed by Python interpreter until the spawned process finishes first, you could instead use subprocess.Popen which is non-blocking. Even better and if possible, redirect the output of the process to stdout and use Popen to read the output of your process from its stdout and then write the output from the Python script to a file.
Related
I have a program that produces a csv file and right at the end I am using os.startfile(fileName) but then due to the program finishing execution the opening file just closes also, same happens if I add a sleep after also, file loads up then once the sleep ends it closes again?
Any help would be appreciated.
From the documentation for os.startfile:
startfile() returns as soon as the associated application is launched. There is no option to wait for the application to close, and no way to retrieve the application’s exit status.
When using this function, there is no way to make your script wait for the program to complete because you have no way of knowing when it is complete. Because the program is being launched as a subprocess of your python script, the program will exit when the python script exits.
Since you don't say in your question exactly what the desired behavior is, I'm going to guess that you want the python script to block until the program finishes execution (as opposed to detaching the subprocess). There are multiple ways to do this.
Use the subprocess module
The subprocess module allows you to make a subprocess call that will not return until the subprocess completes. The exact call you make to launch the subprocess depends heavily on your specific situation, but this is a starting point:
subprocess.Popen(['start', fileName], shell=True)
Use input to allow user to close script
You can have your script block until the user tells the python script that the external program has closed. This probably requires the least modification to your code, but I don't think it's a good solution, as it depends on user input.
os.startfile(fileName)
input('Press enter when external program has completed...')
Question: Is there a way, using Python, to access the stdout of a running process? This process has not been started by Python.
Context: There is a program called mayabatch, that renders out images from 3D Maya scene files. If I were to run the program from the command line I would see progress messages from mayabatch. Sometimes, artists close these windows, leaving the progress untracable until the program finishes. That led me along this route of trying to read its stdout after it's been spawned by a foreign process.
Background:
OS: Windows 7 64-bit
My research so far: I have only found questions and answers of how to do this if it was a subprocess, using the subprocess module. I also looked briefly into psutil, but I could not find any way to read a process' stdout.
Any help would be really appreciated. Thank you.
I don't think you can get to the stdout of a process outside of the code that created it
The lazy way to is just to pipe the output of mayabatch to a text file, and then poll the text file periodically in your own code so it's under your control, rather than forcing you to wait on the pipe (which is especially hard on Windows, since Windows select doesn't work with the pipes used by subprocess.
I think this is what maya does internally too: by default mayaBatch logs its results to a file called mayaRenderLog.txt in the user's Maya directory.
If you're running mayabatch from the command line or a bat file, you can funnel stdout to a file with a > character:
mayabatch.exe "file.ma" > log.txt
You should be able to poll that text file from the outside using standard python as long as you only open it for reading. The advantage of doing it this way is that you control the frequency at which you check the file.
OTOH If you're doing it from python, it's a little tougher unless you don't mind having your python script idled until the mayabatch completes. The usual subprocess recipe, which uses popen.communicate() is going to wait for an end-of-process return code:
test = subprocess.Popen(["mayabatch.exe","filename.mb"], stdout=subprocess.PIPE)
print test.communicate()[0]
works but won't report until the process dies. But you calling readlines on the process's stdout will trigger the process and report it one line at a time:
test = subprocess.Popen(["mayabatch.exe","filename.mb"], stdout=subprocess.PIPE)
reader = iter(test.subprocess.readlines, "")
for line in reader:
print line
More discussion here
If a series of commands are piped in linux, it handles it efficiently, ie. it terminates the previous subprocess if the last subprocess has terminated. For instance,
cat filename | head -n 1
zcat filename | head -n 1
hadoop fs -cat /some/path | head -n 1
In each of the above, the cat command would take considerable time, but the combined command performs fast. How is it done internally? Are the first commands (cat commands) given SIGTERM, SIGKILL by the OS as soon as the head terminates?
I wanted to do something similar in Python and was wondering what should be the best way to do it. I am trying to do the following:
p1 = Popen(['hadoop','fs','-cat',path], stdout=PIPE)
p2 = Popen(['head','-n',str(num_lines)], stdin=p1.stdout,stdout=PIPE)
p2.communicate()
p1.kill() or p1.terminate()
Is this efficient?
Actually, I believe that the process is being sent SIGPIPE when head closes. From Wikipedia:
SIGPIPE
The SIGPIPE signal is sent to a process when it attempts to write to a pipe without a process connected to the other end.
Also, from a few answers from a question on SIGPIPE:
...
You see, when the file descriptor with the pending write is closed, the SIGPIPE happens right then. While the write will return -1 eventually, the whole point of the signal is to notify you asynchronously that the write is no longer possible. This is part of what makes the whole elegant co-routine structure of pipes work in UNIX.
...
https://stackoverflow.com/a/8369516/2334407
...
https://www.gnu.org/software/libc/manual/html_mono/libc.html
This link says:
A pipe or FIFO has to be open at both ends simultaneously. If you read from a pipe or FIFO file that doesn't have any processes writing to it (perhaps because they have all closed the file, or exited), the read returns end-of-file. Writing to a pipe or FIFO that doesn't have a reading process is treated as an error condition; it generates a SIGPIPE signal, and fails with error code EPIPE if the signal is handled or blocked.
...
https://stackoverflow.com/a/18971899/2334407
I think it is to get the error handling correct without requiring a lot of code in everything writing to a pipe.
Some programs ignore the return value of write(); without SIGPIPE they would uselessly generate all output.
Programs that check the return value of write() likely print an error message if it fails; this is inappropriate for a broken pipe as it is not really an error for the whole pipeline.
https://stackoverflow.com/a/8370870/2334407
Now, to answer your question on what the best way to do it would be, I'd say not to send any signals. Instead, read as much data as you need to, and then simply close the pipe. The OS kernel will then automatically clean up after you and send SIGPIPE to the necessary processes.
I am trying to communicate with a command-line chat bot with Python using the subprocess module. (http://howie.sourceforge.net/ using the compiled win32 binary, I have my reasons!)
This works:
proc = Popen('Howie/howie.exe', stdout=PIPE,stderr=STDOUT,stdin=PIPE)
output = proc.communicate()
But Popen.communicate waits for the process to terminate (and sends it EOF?), I want to be able to interact with it. The apparent solution for this was to read stdout / write stdin like so:
This doesn't work:
proc = Popen('Howie/howie.exe', stdout=PIPE,stderr=STDOUT,stdin=PIPE)
while True: print proc.stdout.readline()
(Note that I am actually using more complex code based on http://code.activestate.com/recipes/440554/ but the issue is the same.)
The problem is, the second approach works perfectly for communicating to cmd, but when I run the chatbot, nothing. So my question is, how is this different in capturing output to using Popen.communicate()?
i.e. I can use the second approach to use the command line as per normal, until I run the chatbot, at which point I stop receiving output. Using the first approach correctly displays the first few lines of output from the bot, but leaves me unable to interact with it.
One major difference between the two is that communicate() closes stdin after sending the data. I don't know about your particular case, but in many cases this means that if a process is awaiting the end of the user input, he will get it when communicate() is used, and will never get it when the code blocks on read() or readline().
Try adding Popen.stdin.close() first and see if it affects your case.
If you want to interact with the program after sending the EOF, rather than using Popen.stdin.close(), you can manually send the command-line End Of File character, which has the same effect but leaves stdin open.
In Python this character's escape sequence is '\x1a'.
I need to launch an external process that is to be controlled via messages sent back and forth via stdin and stdout. Using subprocess.Popen I am able to start the process but am unable to control the execution via stdin as I need to.
The flow of what I'm trying to complete is to:
Start the external process
Iterate for some number of steps
Tell the external process to complete the next processing step by writing a new-line character to it's stdin
Wait for the external process to signal it has completed the step by writing a new-line character to it's stdout
Close the external process's stdin to indicate to the external process that execution has completed.
I have come up with the following so far:
process = subprocess.Popen([PathToProcess], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
for i in xrange(StepsToComplete):
print "Forcing step # %s" % i
process.communicate(input='\n')
When I run the above code the '\n' is not communicated to the external process, and I never get beyond step #0. The code blocks at process.communicate() and does not proceed any further. I am using the communicate() method incorrectly?
Also how would I implement the "wait until the external process writes a new line" piece of functionality?
process.communicate(input='\n') is wrong. If you will notice from the Python docs, it writes your string to the stdin of the child, then reads all output from the child until the child exits. From doc.python.org:
Popen.communicate(input=None) Interact
with process: Send data to stdin. Read
data from stdout and stderr, until
end-of-file is reached. Wait for
process to terminate. The optional
input argument should be a string to
be sent to the child process, or None,
if no data should be sent to the
child.
Instead, you want to just write to the stdin of the child. Then read from it in your loop.
Something more like:
process=subprocess.Popen([PathToProcess],stdin=subprocess.PIPE,stdout=subprocess.PIPE);
for i in xrange(StepsToComplete):
print "Forcing step # %s"%i
process.stdin.write("\n")
result=process.stdout.readline()
This will do something more like what you want.
You could use Twisted, by using reactor.spawnProcess and LineReceiver.