I'm using subprocess to spawn a conda create command and capture the resulting stdout for later use. I also immediately print the stdout to the console so the user can still see the progress of the subprocess:
p = subprocess.Popen('conda create -n env1 python', stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in iter(p.stdout.readline, b''):
sys.stdout.write(line.decode(sys.stdout.encoding))
This works fine until half way through the execution when the conda create requires user input: it prompts Proceed (n/y)? and waits for the user to input an option. However, the code above doesn't print the prompt and instead just waits for input "on a blank screen". Once an input is received the prompt is printed afterwards and then execution continues as expected.
I assume this is because the input somehow blocks the prompt being written to stdout and so the readline doesn't receive new data until after the input block has been lifted.
Is there a way I can ensure the input prompt is printed before the subprocess waits for user input? Note I'm running on Windows.
Although I'm sure pexpect would have worked in this case, I decided it would be overkill. Instead I used MisterMiyagi's insight and replaced readline with read.
The final code is as so:
p = subprocess.Popen('conda create -n env1 python', stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while p.poll() is None:
sys.stdout.write(p.stdout.read(1).decode(sys.stdout.encoding))
sys.stdout.flush()
Note the read(1) as just using read() would block the while loop until an EOF is found. Given no EOF is found before the input prompt, nothing will be printed at all! In addition, flush is called which ensures the text written to sys.stdout is actually visible on screen.
For this use case I recommend using pexpect. stdin != stdout
Example use case where it conditionally sends to stdin on prompts on stdout
def expectgit(alog):
TMPLOG = "/tmp/pexpect.log"
cmd = f'''
ssh -T git#github.com ;\
echo "alldone" ;
'''
with open(TMPLOG, "w") as log:
ch = pexpect.spawn(f"/bin/bash -c \"{cmd}\"", encoding='utf-8', logfile=log)
while True:
i = ch.expect(["successfully authenticated", "Are you sure you want to continue connecting"])
if i == 0:
alog.info("Git all good")
break
elif i == 1:
alog.info("Fingerprint verification")
ch.send("yes\r")
ch.expect("alldone")
i = ch.expect([pexpect.EOF], timeout=5)
ch.close()
alog.info("expect done - EOF")
with open(TMPLOG) as log:
for l in log.readlines():
alog.debug(l.strip())
Related
I'm making a shell with python. So far I have gotten cd to work (not pretty I know, but it's all I need for now). When I su root (for example) I get a root shell, but I can't capture the output I receive after running a command. However the shell does accept my commands, as when I type exit it exits. Is there a way to capture the output of a 'new' shell?
import os, subprocess
while True:
command = input("$ ")
if len(command.split(" ")) >= 2:
print(command.split(" ")[0]) #This line is for debugging
if command.split(" ")[0] == "cd" or command.split(" ")[1] == "cd":
os.chdir(command.split(" ")[command.split(" ").index("cd") + 1])
continue
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, universal_newlines=True)
output, error = process.communicate()
print(output.strip("\n"))
EDIT: To make my request a bit more precise, I'd like a way to authenticate as another user from a python script, basically catching the authentication, doing it in the background and then starting a new subprocess.
You really need to understand how subprocess.Popen works. This command executes a new sub-process (on a Unix machine, calls fork and then exec). The new sub-process is a separate process. Your code just calls communicate once and then discards of it.
If you just create a new shell by calling subprocess.Popen and then running su <user> inside of it, the shell will be closed right after that and the next time, you'll be running the command using the same (original) user again.
What you want is probably to create a single subprocess at the beginning of your application and then be a sort of a proxy between the user and the underlying process, and then just keep writing to its stdin and reading from stdout.
Here's an example:
import os, subprocess
process = subprocess.Popen(["bash"], stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
while True:
command = input("$ ")
process.stdin.write(command + "\n")
process.stdin.flush()
output = process.stdout.readline()
print(output.strip("\n"))
(I removed the cd command parsing bit because it wasn't constructive to understanding the solution here, but you can definitely add specific handlers for specific inputs that wrap the underlying shell)
Is there any way to use Popen with interactive commands? I mean nslookup, ftp, powershell... I read the whole subprocess documentation several times but I can't find the way.
What I have (removing the parts of the project which aren't of interest here) is:
from subprocess import call, PIPE, Popen
command = raw_input('>>> ')
command = command.split(' ')
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
execution = process.stdout.read()
error = process.stderr.read()
output = execution + error
process.stderr.close()
process.stdout.close()
print(output)
Basically, when I try to print the output with a command like dir, the output is a string, so I can work with the .read() on it. But when I try to use nslookup for example, the output isn't a string, so it can't be read, and the script enters in a deadlock.
I know that I can invoke nslookup in non-interactive mode, but that's not the point. I want to remove all the chances of a deadlock, and make it works with every command you can run in a normal cmd.
The real way the project works is through sockets, so the raw_input is a s.recv() and the output is sending back the output, but I have simplified it to focus on the problem.
I have one sh file, I need to install it in target linux box. So I'm in the process of writing automatic installation for the sh file which required lot of input from user. Example, first thing I made ./file.sh it will show a big paragaraph and ask user to press Enter. I'm stuck in this place. How to send key data to the sub process. Here is what I've tried.
import subprocess
def runProcess(exe):
global p
p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
retcode = p.poll() #returns None while subprocess is running
line = p.stdout.readline()
yield line
if(retcode is not None):
break
for line in runProcess('./file.sh'.split()):
if '[Enter]' in line:
print line + 'got it'
p.communicate('\r')
Correct me if my understanding is wrong, pardon me if it is duplicate.
If you need to send a bunch of newlines and nothing else, you need to:
Make sure the stdin for the Popen is a pipe
Send the newlines without causing a deadlock
Your current code does neither. Something that might work (assuming they're not using APIs that require direct interaction in a tty, rather than just reading stdin):
import subprocess
import threading
def feednewlines(f):
try:
# Write as many newlines as it will take
while True:
f.write(b'\n') # Write newline, not carriage return
f.flush() # Flush to ensure it's sent as quickly as possible
except OSError:
return # Done when pipe closed/process exited
def runProcess(exe):
global p
# Get stdin as pipe too
p = subprocess.Popen(exe, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# Use thread to just feed as many newlines as needed to stdin of subprocess
feeder = threading.Thread(target=feednewlines, args=(p.stdin,))
feeder.daemon = True
feeder.start()
# No need to poll, just read until it closes stdout or exits
for line in p.stdout:
yield line
p.stdin.close() # Stop feeding (causes thread to error and exit)
p.wait() # Cleanup process
# Iterate output, and echo when [Enter] seen
for line in runProcess('./file.sh'.split()):
if '[Enter]' in line:
print line + 'got it'
For the case where you need to customize the responses, you're going to need to add communication between parent and feeder thread, which makes this uglier, and it only works if the child process is properly flushing its output when it prompts you, even when not connected to a terminal. You might do something like this to define a global queue:
import queue # Queue on Python 2
feederqueue = queue.Queue()
then change the feeder function to:
def feednewlines(f):
try:
while True:
f.write(feederqueue.get())
f.flush()
except OSError:
return
and change the global code lower down to:
for line in runProcess('./file.sh'.split()):
if '[Enter]' in line:
print line + 'got it'
feederqueue.put(b'\n')
elif 'THING THAT REQUIRES YOU TO TYPE FOO' in line:
feederqueue.put(b'foo\n')
etc.
Command line programs run differently when they are run in a terminal verses when they are run in the background. If the program is attached to a terminal, they run in an interactive line mode expecting user interaction. If stdin is a file or a pipe, they run in block mode where writes are delayed until a certain block size is buffered. Your program will never see the [Enter] prompt because it uses pipes and the data is still in the subprocesses output buffer.
The python pexpect module solves this problem by emulating a terminal and allowing you to interact with the program with a series of "expect" statements.
Suppose we want to run a test program
#!/usr/bin/env python3
data = input('[Enter]')
print(data)
its pretty boring. It prompts for data, prints it, then exits. We can run it with pexpect
#!/usr/bin/env python3
import pexpect
# run the program
p = pexpect.spawn('./test.py')
# we don't need to see our input to the program echoed back
p.setecho(False)
# read lines until the desired program output is seen
p.expect(r'\[Enter\]')
# send some data to the program
p.sendline('inner data')
# wait for it to exit
p.expect(pexpect.EOF)
# show everything since the previous expect
print(p.before)
print('outer done')
I have a executable, called tsfoil2.exe, and I want to operate this .exe from my python environment.
I'm running Python 2.7.3, with Spyder 2.1.11 on Windows 7.
In order to operate the .exe, it requires some input, the default hard drive ('I:\'), a name for the outputfile ('test'), and a name for the input file ('SC20610.inp').
One of my colleagues advised me to use os.system, and supply this with a temporary input file, that contains all the arguments.
f = open('temp', 'w')
f.write('I:\ \n')
f.write('test \n')
f.write('SC20610.inp\n')
f.close()
I then supply this file with arguments to the .exe in the following way:
os.system("tsfoil2.exe < temp")
This all works, but the program requires a 'ENTER' to close. For some reason, the .exe is repeatedly asking to 'Press the ENTER key to exit'. Even, when I press the enter key in my Spyder-console, the program does not terminate.
Is there a way to give the 'ENTER' key as an interactive input to .exe?
I've tried to use the SendKeys class, but as the program does not terminate, it does not reach the line of code that contains the SendKeys command.
I've also tried to include it in the arguments-file, but this does not work either.
Furthermore I've also found out that it might be beneficial to switch to subprocesses command, as it might give me more command over the execution, but I haven't been able to run the executable with the input files.
Is it possible to provide the necessary 'ENTER' using os.system, or should I switch to subprocess, and if so, how can I construct a method similar to the os.system("tsfoil2.exe < temp") I'm using now.
I've tried this:
import subprocess as sub
f = open('temp', 'w')
f.write('I:\ \n')
f.write('test \n')
f.write('SC20610.inp\n')
f.close()
proc=sub.Popen(["tsfoil2.exe","temp"], shell=True)
and this
import subprocess as sub
p=sub.Popen('tsfoil2.exe')
p.communicate(input='I:' )
But, the program does not respond to the arguments given.
MWE:
import os
f = open('temp', 'w')
f.write('I:\ \n')
f.write('test \n')
f.write('SC20610.inp\n')
f.close()
os.system("tsfoil2.exe < temp")
Both the program can be found via http://www.dept.aoe.vt.edu/~mason/Mason_f/tsfoil2.exe, the input file can be found via http://www.dept.aoe.vt.edu/~mason/Mason_f/SC20610.inp.
I hope everything is clear, and you can help me out.
'Press the ENTER key to exit' means that the programs expects a newline.
I see no blank line at the end of the temp file. Also, you might have meant 'I:\\\n' -- you need to use '\\' in a Python string literal if you want \ in the output.
The question is what tsfoil2.exe considers a newline e.g., b'\r\n' or just b'\n' and where it expects to receive it: from stdin (getchar()) or directly from console (getch()).
Assuming that the program expects b'\r\n' from stdin on Windows:
import os
from subprocess import CalledProcessError, Popen, PIPE
cmd = "tsfoil2.exe"
input_data = os.linesep.join(['I:\\', 'test', 'SC20610.inp', os.linesep])
p = Popen(cmd, stdin=PIPE, bufsize=0)
p.communicate(input_data.encode('ascii'))
if p.returncode != 0:
raise CalledProcessError(p.returncode, cmd)
How it works
os.linesep == "\r\n" on Windows. "\n".join(["a", "b"]) == "a\nb".
Each process may have three standard streams: stdin, stdout, stderr. In Python, they are represented as sys.stdin, sys.stdout, sys.stderr. You can read input from stdin and write to stdout, stderr e.g., input function reads from stdin and print writes to stdout by default. stderr may be used to write error messages.
stdin=PIPE tells Popen to create a pipe between the parent process where it is called and the new child process ("tsfoil2.exe") and redirect the subprocess' stdin. p.communicate(data) writes data to p.stdin, closes it and waits for the child process to finish. p.returncode contains the exit status of the child process. Usually non-zero status means failure.
It emulates the shell pipeline without actually spawning the shell:
$ echo input data | tsfoil2.exe
If input is expected directly from console, you could try SendKeys module or its pure Python implementation SendKeys-ctypes:
from SendKeys import SendKeys
SendKeys(r"""
{LWIN}
{PAUSE .25}
r
C:\Path\To\tsfoil2.exe{ENTER}
{PAUSE 1}
I:\{ENTER}
{PAUSE 1}
test{ENTER}
{PAUSE 1}
SC20610.inp{ENTER}
{PAUSE 1}
{ENTER}
""")
I haven't tested it.
I'm trying to run a Powershell subprocess from Python. I need to send Powershell code from Python to the child process. I've got this far:
import subprocess
import time
args = ["powershell", "-NoProfile", "-InputFormat None", "-NonInteractive"]
startTime = time.time()
process = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.stdin.write("Write-Host 'FINISHED';".encode("utf-8"))
result = ''
while 'FINISHED' not in result:
result += process.stdout.read(32).decode('utf-8')
if time.time() > startTime + 5:
raise TimeoutError(result)
print(result)
This times out, because nothing ever gets written to stdout. I think the Write-Host cmdlet never gets executed. Even the simple bash/Cygwin code echo "Write-Host 'FINISHED';" | powershell doesn't seem to do the job.
For comparison, sending the code block using the -Command flag works correctly.
How can I convince Powershell to run the code which I'm sending to stdin?
There a couple of things you can consider:
Invoke PowerShell in a mode where you provide it with a script file which it should execute. Write this script file prior to calling the subprocess. Use the -File <FilePath> parameter for PowerShell (cf. the docs)
If you really want to go with the stdin technique, you might be missing a newline character after the command. If this does not help, you might need to send another control character that tells PowerShell that input EOF is reached. You definitely need to consult the PowerShell docs for finding out how to 'terminate' commands on stdin. One thing you definitely need is the -Command - arguments: The value of Command can be "-", a string. or a script block. If the value of Command is "-", the command text is read from standard input. You may also want to look at this little hack: https://stackoverflow.com/a/13877874/145400
If you only want to execute one command, you can simplify your code by using out, err = subprocess.communicate(in)
I had trouble with a similar task, but I was able to solve it.
First my example code:
import subprocess
args = ["powershell.exe", "-Command", r"-"]
process = subprocess.Popen(args, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
process.stdin.write(b"$data = Get-ChildItem C:\\temp\r\n")
process.stdin.write(b"Write-Host 'Finished 1st command'\r\n")
process.stdin.write(b"$data | Export-Clixml -Path c:\\temp\state.xml\r\n")
process.stdin.write(b"Write-Host 'Finished 2nd command'\r\n")
output = process.communicate()[0]
print(output.decode("utf-8"))
print("done")
The main issue was the correct argument list args. It is required to start the powershell with the -Command-flag, followed by "-" as indicated by Jan-Philipp.
Another mystery was the end-of-line character that is required to get the stuff executed. \r\n works quite well.
Getting the output of the Powershell is still an issue. But if you don't care about realtime, you can collect the output after finishing all executions by calling
output = process.communicate()[0]
However, the active Powershell will be terminated afterwards.