Piping cmd.exe with a subprocess in order of embedding a console works fine in most cases. When using a stdout.read(1)-thread of course. However this thread is getting nothing for few commands (i spotted this for python itself as well as for python programms).
I know there are lots of questions about output from children, but this is about loosing a child's child's (and so on) output. The output of cmd.exe itself as well as most of commands is easily tracked out. Also I can assume that the same occurs for input as the interactive python shell within cmd.exe is not closing when exit() is entered.
This could be a buffering issue - but that would be strange as buffering is disabled for Popen (and p.stdin.flush() is used as python won't start within p elseways). Also this could be caused because of bad inheritance of the processes and their standard i/o streams, but i actually hope it's not.
I can see there could be good use for example code but this is actually as I mentioned embedded. So if someone sees a theoretical problem I can skip the process of exporting that code :) However I should add an example even tough this board gives lots of examples relating to cmd.exe and popen. It would take less time than I spent already on googling for the solution.
The basic problem is
subprocess.Popen("cmd.exe", stdin = -1, stdout = -1, stderr = -2, bufsize=0)
that not all children of that subprocess seem to be using the pipes. The cmd starts accepting further commands as soon as it's child has been killed (what's probably ensuring the loss of it's output).
Related
I am using a 3rd-party python module which is normally called through terminal commands. When called through terminal commands it has a verbose option which prints to terminal in real time.
I then have another python program which calls the 3rd-party program through subprocess. Unfortunately, when called through subprocess the terminal output no longer flushes, and is only returned on completion (the process takes many hours so I would like real-time progress).
I can see the source code of the 3rd-party module and it does not set printing to be flushed such as print('example', flush=True). Is there a way to force the flushing through my module without editing the 3rd-party source code? Furthermore, can I send this output to a log file (again in real time)?
Thanks for any help.
The issue is most likely that many programs work differently if run interactively in a terminal or as part of a pipe line (i.e. called using subprocess). It has very little to do with Python itself, but more with the Unix/Linux architecture.
As you have noted, it is possible to force a program to flush stdout even when run in a pipe line, but it requires changes to the source code, by manually applying stdout.flush calls.
Another way to print to screen, is to "trick" the program to think it is working with an interactive terminal, using a so called pseudo-terminal. There is a supporting module for this in the Python standard library, namely pty. Using, that, you will not explicitly call subprocess.run (or Popen or ...). Instead you have to use the pty.spawn call:
def prout(fd):
data = os.read(fd, 1024)
while(data):
print(data.decode(), end="")
data = os.read(fd, 1024)
pty.spawn("./callee.py", prout)
As can be seen, this requires a special function for handling stdout. Here above, I just print it to the terminal, but of course it is possible to do other thing with the text as well (such as log or parse...)
Another way to trick the program, is to use an external program, called unbuffer. Unbuffer will take your script as input, and make the program think (as for the pty call) that is called from a terminal. This is arguably simpler if unbuffer is installed or you are allowed to install it on your system (it is part of the expect package). All you have to do then, is to change your subprocess call as
p=subprocess.Popen(["unbuffer", "./callee.py"], stdout=subprocess.PIPE)
and then of course handle the output as usual, e.g. with some code like
for line in p.stdout:
print(line.decode(), end="")
print(p.communicate()[0].decode(), end="")
or similar. But this last part I think you have already covered, as you seem to be doing something with the output.
I am creating a subprocess using this line of code:
p = subprocess.Popen(["doesItemExist.exe", id], shell=False)
and when I run the script while I have the Task Manager open, I can see that it creates two processes and not one. The issue is that when I go to kill it, it kills one (using p.kill()), but not the other. I've tried looking online but the only examples I find are about shell=True and their solutions don't work for me. I've confirmed that that line only gets called once.
What can I do? Popen is only giving me back the one pid so I don't understand how to get the other so I can kill both.
I ended up being able to deal with this issue by creating a clean up function which just uses the following:
subprocess.run(["taskkill", "/IM", "doesItemExist.exe", "/F"], shell=True)
This will kill any leftover tasks. If anyone uses this, be careful that your exe has a unique name to prevent you from killing anything you don't mean to. If you want to hide the output/errors, just set the stdout and stderr to subprocess.PIPE.
Also, if there is no process to kill it will report that as an error.
I've console application on Python. I try to use xdg-open and run it in background, but I can't. I tried
os.system('xdg-open http://google.com &')
subprocess.call('xdg-open http://google.com &', shell=True)
I don't know what you mean by
but I can't
because it works for me. I imagine, however, that you're complaining that the parent process does not close until the child has.
That code is, however, an outdated practice (if it ever was in favour); the modern equivalent would be
process = subprocess.Popen(['xdg-open', 'Dunno.png'])
Instead of asking the shell to fork for you, this runs in the background from the start without ever passing through a shell. This should deal with the problem above, too.
If you want to capture sys.stdout, you can use
process = subprocess.Popen(['xdg-open', 'Dunno.png'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
which redirects the output of the process' stdout and stderr to buffers. (You can access those buffers with process.stdout and process.stderr and communicate either by reading and writing to those or using process.communicate.
You can get the return code with process.returncode.
If your problem is not this, a problem description (traceback?) would be useful. It's also worth checking that the behaviour of using xdg-open in the shell is what you expect.
I am using below code for running unix commands:
cmd = 'ls -l'
(status,output) = commands.getstatusoutput(cmd)
print output
But the problem is that it shows output only after the command completed, but i want to see the output printed as the execution progresses.
ls -l is just dummy command, i am using some complex command in actual program.
Thanks!!
Since this is homework, here's what to do instead of the full solution:
Use the subprocess.Popen class to call the executable. Note that the constructor takes a named stdout argument, and take a look at subprocess.PIPE.
Read from the Popen object's STDOUT pipe in a separate thread to avoid dead locks. See the threading module.
Wait until the subprocess has finished (see Popen.wait).
Wait until the thread has finished processing the output (see Thread.join). Note that this may very well happen after the subprocess has finished.
If you need more help please describe your precise problem.
Unless there are simpler ways in Python which I'm not aware of, I believe you'll have to dig into the slightly more complex os.fork and os.pipe functions.
Basically, the idea is to fork your process, have the child execute your command, while having its standard output redirected to a pipe which will be read by the parent. You'll easily find examples of this kind of pattern.
Most programs will use block buffered output if they are not connected to a tty, so you need to run the program connected to a pty; the easiest way is to use pexpect:
for line in pexpect.spawn('command arg1 arg2'):
print line
I am trying to communicate with a command-line chat bot with Python using the subprocess module. (http://howie.sourceforge.net/ using the compiled win32 binary, I have my reasons!)
This works:
proc = Popen('Howie/howie.exe', stdout=PIPE,stderr=STDOUT,stdin=PIPE)
output = proc.communicate()
But Popen.communicate waits for the process to terminate (and sends it EOF?), I want to be able to interact with it. The apparent solution for this was to read stdout / write stdin like so:
This doesn't work:
proc = Popen('Howie/howie.exe', stdout=PIPE,stderr=STDOUT,stdin=PIPE)
while True: print proc.stdout.readline()
(Note that I am actually using more complex code based on http://code.activestate.com/recipes/440554/ but the issue is the same.)
The problem is, the second approach works perfectly for communicating to cmd, but when I run the chatbot, nothing. So my question is, how is this different in capturing output to using Popen.communicate()?
i.e. I can use the second approach to use the command line as per normal, until I run the chatbot, at which point I stop receiving output. Using the first approach correctly displays the first few lines of output from the bot, but leaves me unable to interact with it.
One major difference between the two is that communicate() closes stdin after sending the data. I don't know about your particular case, but in many cases this means that if a process is awaiting the end of the user input, he will get it when communicate() is used, and will never get it when the code blocks on read() or readline().
Try adding Popen.stdin.close() first and see if it affects your case.
If you want to interact with the program after sending the EOF, rather than using Popen.stdin.close(), you can manually send the command-line End Of File character, which has the same effect but leaves stdin open.
In Python this character's escape sequence is '\x1a'.