I am using below code for running unix commands:
cmd = 'ls -l'
(status,output) = commands.getstatusoutput(cmd)
print output
But the problem is that it shows output only after the command completed, but i want to see the output printed as the execution progresses.
ls -l is just dummy command, i am using some complex command in actual program.
Thanks!!
Since this is homework, here's what to do instead of the full solution:
Use the subprocess.Popen class to call the executable. Note that the constructor takes a named stdout argument, and take a look at subprocess.PIPE.
Read from the Popen object's STDOUT pipe in a separate thread to avoid dead locks. See the threading module.
Wait until the subprocess has finished (see Popen.wait).
Wait until the thread has finished processing the output (see Thread.join). Note that this may very well happen after the subprocess has finished.
If you need more help please describe your precise problem.
Unless there are simpler ways in Python which I'm not aware of, I believe you'll have to dig into the slightly more complex os.fork and os.pipe functions.
Basically, the idea is to fork your process, have the child execute your command, while having its standard output redirected to a pipe which will be read by the parent. You'll easily find examples of this kind of pattern.
Most programs will use block buffered output if they are not connected to a tty, so you need to run the program connected to a pty; the easiest way is to use pexpect:
for line in pexpect.spawn('command arg1 arg2'):
print line
Related
I am using a 3rd-party python module which is normally called through terminal commands. When called through terminal commands it has a verbose option which prints to terminal in real time.
I then have another python program which calls the 3rd-party program through subprocess. Unfortunately, when called through subprocess the terminal output no longer flushes, and is only returned on completion (the process takes many hours so I would like real-time progress).
I can see the source code of the 3rd-party module and it does not set printing to be flushed such as print('example', flush=True). Is there a way to force the flushing through my module without editing the 3rd-party source code? Furthermore, can I send this output to a log file (again in real time)?
Thanks for any help.
The issue is most likely that many programs work differently if run interactively in a terminal or as part of a pipe line (i.e. called using subprocess). It has very little to do with Python itself, but more with the Unix/Linux architecture.
As you have noted, it is possible to force a program to flush stdout even when run in a pipe line, but it requires changes to the source code, by manually applying stdout.flush calls.
Another way to print to screen, is to "trick" the program to think it is working with an interactive terminal, using a so called pseudo-terminal. There is a supporting module for this in the Python standard library, namely pty. Using, that, you will not explicitly call subprocess.run (or Popen or ...). Instead you have to use the pty.spawn call:
def prout(fd):
data = os.read(fd, 1024)
while(data):
print(data.decode(), end="")
data = os.read(fd, 1024)
pty.spawn("./callee.py", prout)
As can be seen, this requires a special function for handling stdout. Here above, I just print it to the terminal, but of course it is possible to do other thing with the text as well (such as log or parse...)
Another way to trick the program, is to use an external program, called unbuffer. Unbuffer will take your script as input, and make the program think (as for the pty call) that is called from a terminal. This is arguably simpler if unbuffer is installed or you are allowed to install it on your system (it is part of the expect package). All you have to do then, is to change your subprocess call as
p=subprocess.Popen(["unbuffer", "./callee.py"], stdout=subprocess.PIPE)
and then of course handle the output as usual, e.g. with some code like
for line in p.stdout:
print(line.decode(), end="")
print(p.communicate()[0].decode(), end="")
or similar. But this last part I think you have already covered, as you seem to be doing something with the output.
I am creating a subprocess using this line of code:
p = subprocess.Popen(["doesItemExist.exe", id], shell=False)
and when I run the script while I have the Task Manager open, I can see that it creates two processes and not one. The issue is that when I go to kill it, it kills one (using p.kill()), but not the other. I've tried looking online but the only examples I find are about shell=True and their solutions don't work for me. I've confirmed that that line only gets called once.
What can I do? Popen is only giving me back the one pid so I don't understand how to get the other so I can kill both.
I ended up being able to deal with this issue by creating a clean up function which just uses the following:
subprocess.run(["taskkill", "/IM", "doesItemExist.exe", "/F"], shell=True)
This will kill any leftover tasks. If anyone uses this, be careful that your exe has a unique name to prevent you from killing anything you don't mean to. If you want to hide the output/errors, just set the stdout and stderr to subprocess.PIPE.
Also, if there is no process to kill it will report that as an error.
I've console application on Python. I try to use xdg-open and run it in background, but I can't. I tried
os.system('xdg-open http://google.com &')
subprocess.call('xdg-open http://google.com &', shell=True)
I don't know what you mean by
but I can't
because it works for me. I imagine, however, that you're complaining that the parent process does not close until the child has.
That code is, however, an outdated practice (if it ever was in favour); the modern equivalent would be
process = subprocess.Popen(['xdg-open', 'Dunno.png'])
Instead of asking the shell to fork for you, this runs in the background from the start without ever passing through a shell. This should deal with the problem above, too.
If you want to capture sys.stdout, you can use
process = subprocess.Popen(['xdg-open', 'Dunno.png'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
which redirects the output of the process' stdout and stderr to buffers. (You can access those buffers with process.stdout and process.stderr and communicate either by reading and writing to those or using process.communicate.
You can get the return code with process.returncode.
If your problem is not this, a problem description (traceback?) would be useful. It's also worth checking that the behaviour of using xdg-open in the shell is what you expect.
I know this has been asked a lot of times but I've yet to find a proper way of doing this. If I want to run a local command the docs say I have to use subprocess as it's replacing all other methods such as os.system/peopen etc.
If I call subprocess.Popen(command, shell=True, stdout=subprocess.PIPE) in my program and the command is for example a openvpn directive which connects my computer to a VPN the process will hang indefinitely since openvpn returns it's output ending with a new line but hangs in there while connected and so does my program (frozen).
Some say I should remove the stdout=subprocess.PIPE which indeed works in a non-blocking way but then everything gets printed to the console instead of me having some sort of control over the output (maybe I dont want to print it).
So is there some sort of proper way of doing this, an example maybe of executing commands in a non-blocking way and also having control over the output.?
If you specify stdout=PIPE, then your subprocess will write to the pipe and hang when the pipe buffer is full. The python program shoudn't hang - Popen is asynchronous which is why Popen.wait() can be called later to wait for the subprocess to exit. Read from Popen.stdout in order to keep the subprocess happy, and print, discard, or process the output as you see fit.
Consider running your process within a terminal. For example,
subprocess.Popen("xterm -e /bin/bash -c '/path/to/openvpn'", shell=True)
or even, you could try,
import shlex
subprocess.Poen(shlex.split("xterm -e /bin/bash -c '/path/to/openvpn'"), shell=False)
I have to make two programs (for example a "script A" (.py) and "script B"(.exe)) communicate.
Both programs are on a infinite loop: Script A needs to write to the stdin of script B and afterwards read the stdout of script B thereafter write again etc. Script B I cannot change.
Both files are on my hard disk, so I there must be a better way to solve this than networking. I can, however, write files with script A.
This is not course homework, I am writing a GUI for a game and I have a few AI's preprogrammed.
I have thought of piping (python scripta.py | scriptb.exe), but that seemed to require script A to finish before script B could execute. Then again, as I've never used piping, I might have missed something obvious.
I would prefer if the tools needed would be part of standard library, but if they're not, too bad.
The solution would have to work on both Linux and Windows.
Could any of you point me in the right direction?
Thank you for your time.
If you start "Script B" from within "script A" using the subprocess module, you will be able to directly interact with its stdin and stdout. For example:
from subprocess import Popen, PIPE
prog = Popen("scriptA.exe", shell=True, stdin=PIPE, stdout=PIPE)
prog.stdin.write("This will go to script A\n")
print prog.stdout.read()
prog.wait() # Wait for scriptA to finish
Just be careful, as calls to read will block, meaning if the script doesn't have anything to print, the call will hang until it does. The easiest way to avoid this is to use threading.
You might be interested in taking a look at Interprocess Communication and Networking.