I'm trying to run a shell program through python. I need to run a command, then while it's still running and waiting for input to continue, I need to take the output received by the program, and process that data as a string. Then I need to parse some data into that program, and simulate an enter pressing.
What would be the best way to achieve this?
subprocess.Popen will work for this, but to read and then write and then read again you can't use communicate (because this will cause the process to end).
Instead, you'll need to work with the process's output pipe (process.stdout below). This is tricky to get right, because reading on the process's stdout is blocking, so you sort of need to know when to stop trying to read (or know how much output the process is going to produce).
In this example, the subprocess is a shell script that writes a line of output, and then echoes whatever you give it until it reads EOF.
import subprocess
COMMAND_LINE = 'echo "Hello World!" ; cat'
process = subprocess.Popen(COMMAND_LINE, shell=True,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE)
s = process.stdout.readline().strip()
print(s)
s2 = process.communicate(s)[0]
print(s2)
Gives:
Hello World!
Hello World!
For more complicated cases, you might think about looking at something like pexpect.
Use subprocess.Popen to run your shell application and use communicate to interact with it.
Related
Well the first problem I run into is that I have no idea how to respond to a command prompt.
bat_location = "F:/SteamLibrary/steamapps/common/Terraria"
os.chdir(bat_location)
os.system("TerrariaServer.exe -steam -lobby friends -config serverconfig.txt")
all of this works, but then when I want to respond to the command prompt which asks me which world I want to run ( the worlds are indexed by numbers from 1 - n (amount of worlds)) I dont know how to respond to it.
I've looked all over google but the code doesnt seem to work.
So basically what I need is when a cmd asks me for example :
Choose World:
I want to automatically respond with the number 10.
os.system("10")
this doesnt seem to do anything, I've also tried a lot with subprocesses but im clearly lost.
Any help is apriciated!
EDIT NR.1 :
Welp, now I've tried this :
bat_location = r'F:\SteamLibrary\steamapps\common\Terraria'
with Popen('TerrariaServer.exe -steam -lobby friends -config serverconfig.txt',
cwd=f'{bat_location}', stdin=PIPE, shell=True) as proc:
proc.stdin.write(b'10\n')
and all it does, im guessing by the response, it just loops around and around.
EDIT NR.2 :
I WILL CLOSE THIS AND START A NEW THREAD, SINCE MY PROBLEM COMPLETELY DERIVED FROM THE ORIGINAL.
From your last few comments, I realized the problem you were having with Popen. When you pass stdout=PIPE and stderr=PIPE, the outputs of the process are captured by the pipes, so you won't see them unless you read from the pipes.
So here's a simple example that you should be able to work with:
import subprocess
from subprocess import PIPE
from textwrap import dedent
with open('tmp.py', 'w') as f:
f.write(dedent("""
print(input())
print(input())
"""))
with subprocess.Popen(['python3', 'tmp.py'], stdin=PIPE) as proc:
proc.stdin.write(b'Hello, world!\n') # write to the process' input
proc.stdin.write(b'Good bye, world!\n') # write to the process' input
If you want to read the data from the function in Python, you can use stdout=PIPE, then use proc.stdout.read and the like, but you may have to be careful about how you get data from the blocking read functions.
Is there any way to use Popen with interactive commands? I mean nslookup, ftp, powershell... I read the whole subprocess documentation several times but I can't find the way.
What I have (removing the parts of the project which aren't of interest here) is:
from subprocess import call, PIPE, Popen
command = raw_input('>>> ')
command = command.split(' ')
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
execution = process.stdout.read()
error = process.stderr.read()
output = execution + error
process.stderr.close()
process.stdout.close()
print(output)
Basically, when I try to print the output with a command like dir, the output is a string, so I can work with the .read() on it. But when I try to use nslookup for example, the output isn't a string, so it can't be read, and the script enters in a deadlock.
I know that I can invoke nslookup in non-interactive mode, but that's not the point. I want to remove all the chances of a deadlock, and make it works with every command you can run in a normal cmd.
The real way the project works is through sockets, so the raw_input is a s.recv() and the output is sending back the output, but I have simplified it to focus on the problem.
I am trying the following:
#!/usr/bin/python
import os, subprocess
func = 'print("Hello World")'
x = subprocess.Popen(['mongo', '--eval', func], stdout=subprocess.PIPE,
stderr=subprocess.PIPE, stdin=subprocess.PIPE)
print x.stdout.read()
print x.stderr.read()
But all I am getting is:
MongoDB shell version: 2.2.3
followed by two new lines. How do I capture the output of function execution?
Reading the pipes gets whatever is currently inside said pipe. Your mongo is waiting to connect to the localhost. Since it doesn't return quickly enough, your read command is not getting the results. This may be because you don't have mongo running locally, but you will run into this problem repeatedly if you don't wait for the subprocess to complete.
Also, keep in mind that subprocess.Popen, to my knowledge, doesn't block. You would probably need to make a x.wait() call if you want the function to complete before trying to grab the output.
My problem is this--I need to get output from a subprocess and I am using the following code to call it-- (Feel free to ignore the long arguments. The importing thing is the stdout= subprocess.PIPE)
(stdout, stderr) = subprocess.Popen([self.ChapterToolPath, "-x", book.xmlPath , "-a", book.aacPath , "-o", book.outputPath+ "/" + fileName + ".m4b"], stdout= subprocess.PIPE).communicate()
print stdout
Thanks to an answer below, I've been able to get the output of the program, but I still end up waiting for the process to terminate before I get anything. The interesting thing is that in my debugger, there is all sorts of text flying by in the console and it is all ignored. But the moment that anything is written to the console in black (I am using pycharm) the program continues without a problem. Could the main program be waiting for some kind of output in order to move on? This would make sense because I am trying to communicate with it.... Is there a difference between text that I can see in the console and actual text that makes it to the stdout? And how would I collect the text written to the console?
Thanks!
The first line of the documentation for subprocess.call() describes it as such:
Run the command described by args. Wait for command to complete, then return the returncode attribute.
Thus, it necessarily waits for the subprocess to exit.
subprocess.Popen(), by contrast, does not do this, returning a handle on a process with which one than then communicate().
To get all output from a program:
from subprocess import check_output as qx
output = qx([program, arg1, arg2, ...])
To get output while the program is running:
from subprocess import Popen, PIPE
p = Popen([program, arg1, ...], stdout=PIPE)
for line in iter(p.stdout.readline, ''):
print line,
There might be a buffering issue on the program' side if it prints line-by-line when run interactively but buffers its output if run as a subprocess. There are various solutions depending on your OS or the program e.g., you could run it using pexpect module.
I'm having troubles getting this to work. Basically I have a python program that expect some data in stdin, that is reading it as sys.stdin.readlines() I have tested this and it is working without problems with things like echo "" | myprogram.py
I have a second program that using the subprocess module calls on the first program with the following code
proc = subprocess.Popen(final_shell_cmd,
stderr=subprocess.PIPE, stdout=subprocess.PIPE,
shell=False), env=shell_env)
f = ' '.join(shell_cmd_args)
#f.append('\4')
return proc.communicate(f)
The second program is a daemon and i have discovered that the second program works well as long as I hit ctrl-d after calling it from the first program.
So it seems there is something wrong with subprocess not closing the file and my first program expecting more input when nothing more should be sending.
anyone has any idea how I can get this working?
The main problem here is that "shell_cmd_args" may contain passwords and other sensitive information that we do not want to pass in as the command name as it will show in tools like "ps".
You want to redirect the subprocess's stdin, so you need stdin=subprocess.PIPE.
You should not need to write Control-D ('\4') to the file object. Control-D tells the shell to close the standard input that's connected to the program. The program doesn't see a Control-D character in that context.