We Keep Coding

Subprocess will not run any commands after initial args in Popen constructor, even after flushing all pipes [duplicate]

I want to achieve something which is very similar to this.
My actual goal is to run Rasa from within python.
Taken from Rasa's site:
Rasa is a framework for building conversational software: Messenger/Slack bots, Alexa skills, etc. We’ll abbreviate this as a bot in this documentation.
It is basically a chatbot which runs in the command prompt. This is how it works on cmd :
Now I want to run Rasa from python so that I can integrate it with my Django-based website. i.e. I want to keep taking inputs from the user, pass it to rasa, rasa processes the text and gives me an output which I show back to the user.
I have tried this (running it from cmd as of now)
import sys
import subprocess
from threading import Thread
from queue import Queue, Empty # python 3.x
def enqueue_output(out, queue):
for line in iter(out.readline, b''):
queue.put(line)
out.close()
def getOutput(outQueue):
outStr = ''
try:
while True: #Adds output from the Queue until it is empty
outStr+=outQueue.get_nowait()
except Empty:
return outStr
p = subprocess.Popen('command_to_run_rasa',
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
shell=False,
universal_newlines=True,
)
outQueue = Queue()
outThread = Thread(target=enqueue_output, args=(p.stdout, outQueue))
outThread.daemon = True
outThread.start()
someInput = ""
while someInput != "stop":
someInput = input("Input: ") # to take input from user
p.stdin.write(someInput) # passing input to be processed by the rasa command
p.stdin.flush()
output = getOutput(outQueue)
print("Output: " + output + "\n")
p.stdout.flush()
But it works fine only for the first line of output. Not for successive input/output cycles. See output below.
How do I get it working for multiple cycles?
I've referred to this, and I think I understand the problem in my code from it but I dont know how to solve it.
EDIT: I'm using Python 3.6.2 (64-bit) on Windows 10

You need to keep interacting with your subprocess - at the moment once you pick the output from your subprocess you're pretty much done as you close its STDOUT stream.
Here is the most rudimentary way to continue user input -> process output cycle:
import subprocess
import sys
import time
if __name__ == "__main__": # a guard from unintended usage
input_buffer = sys.stdin # a buffer to get the user input from
output_buffer = sys.stdout # a buffer to write rasa's output to
proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."], # start the process
stdin=subprocess.PIPE, # pipe its STDIN so we can write to it
stdout=output_buffer, # pipe directly to the output_buffer
universal_newlines=True)
while True: # run a main loop
time.sleep(0.5) # give some time for `rasa` to forward its STDOUT
print("Input: ", end="", file=output_buffer, flush=True) # print the input prompt
print(input_buffer.readline(), file=proc.stdin, flush=True) # forward the user input
You can replace input_buffer with a buffer coming from your remote user(s) and output_buffer with a buffer that forwards the data to your user(s) and you'll get essentially what you're looking for - the sub-process will be getting the input directly from the user (input_buffer) and print its output to the user (output_buffer).
If you need to perform other tasks while all this is running in the background, just run everything under the if __name__ == "__main__": guard in a separate thread, and I'd suggest adding a try..except block to pick up KeyboardInterrupt and exit gracefully.
But... soon enough you'll notice that it doesn't exactly work properly all the time - if it takes longer than half a second of wait for rasa to print its STDOUT and enter the wait for STDIN stage, the outputs will start to mix. This problem is considerably more complex than you might expect. The main issue is that STDOUT and STDIN (and STDERR) are separate buffers and you cannot know when a subprocess is actually expecting something on its STDIN. This means that without a clear indication from the subprocess (like you have the \r\n[path]> in Windows CMD prompt on its STDOUT for example) you can only send data to the subprocesses STDIN and hope it will be picked up.
Based on your screenshot, it doesn't really give a distinguishable STDIN request prompt because the first prompt is ... :\n and then it waits for STDIN, but then once the command is sent it lists options without an indication of its end of STDOUT stream (technically making the prompt just ...\n but that would match any line preceding it as well). Maybe you can be clever and read the STDOUT line by line, then on each new line measure how much time has passed since the sub-process wrote to it and once a threshold of inactivity is reached assume that rasa expects input and prompt the user for it. Something like:
import subprocess
import sys
import threading
# we'll be using a separate thread and a timed event to request the user input
def timed_user_input(timer, wait, buffer_in, buffer_out, buffer_target):
while True: # user input loop
timer.wait(wait) # wait for the specified time...
if not timer.is_set(): # if the timer was not stopped/restarted...
print("Input: ", end="", file=buffer_out, flush=True) # print the input prompt
print(buffer_in.readline(), file=buffer_target, flush=True) # forward the input
timer.clear() # reset the 'timer' event
if __name__ == "__main__": # a guard from unintended usage
input_buffer = sys.stdin # a buffer to get the user input from
output_buffer = sys.stdout # a buffer to write rasa's output to
proc = subprocess.Popen(["path/to/rasa", "arg1", "arg2", "etc."], # start the process
stdin=subprocess.PIPE, # pipe its STDIN so we can write to it
stdout=subprocess.PIPE, # pipe its STDIN so we can process it
universal_newlines=True)
# lets build a timer which will fire off if we don't reset it
timer = threading.Event() # a simple Event timer
input_thread = threading.Thread(target=timed_user_input,
args=(timer, # pass the timer
1.0, # prompt after one second
input_buffer, output_buffer, proc.stdin))
input_thread.daemon = True # no need to keep the input thread blocking...
input_thread.start() # start the timer thread
# now we'll read the `rasa` STDOUT line by line, forward it to output_buffer and reset
# the timer each time a new line is encountered
for line in proc.stdout:
output_buffer.write(line) # forward the STDOUT line
output_buffer.flush() # flush the output buffer
timer.set() # reset the timer
You can use a similar technique to check for more complex 'expected user input' patterns. There is a whole module called pexpect designed to deal with this type of tasks and I wholeheartedly recommend it if you're willing to give up some flexibility.
Now... all this being said, you are aware that Rasa is built in Python, installs as a Python module and has a Python API, right? Since you're already using Python why would you call it as a subprocess and deal with all this STDOUT/STDIN shenanigans when you can directly run it from your Python code? Just import it and interact with it directly, they even have a very simple example that does exactly what you're trying to do: Rasa Core with minimal Python.

subprocess get result before terminate

Get Result of a SubProcess in Real Time
I would like to get each result (sys.stdout) in real time before the subprocess terminates.
Suppose we have the following file.py.
import time,sys
sys.stdout.write('something')
while True:
sys.stdout.write('something else')
time.sleep(4)
Well, i made some tries with modules of subprocess, asyncio and threading, although all methods gives me the result when the process is finished. Ideally i would like to terminate the process myself and get each result (stdout, stderr) in real time and not when the process it completes.
import subprocess
proc = sp.Popen([sys.executable, "/Users/../../file.py"], stdout = subprocess.PIPE, stderr= subproces.STDOUT)
proc.communicate() #This one received the result after finish
I tried as well with readline proc.stdout.readline() in a different thread with threading module and with asyncio, but it waits as well until the process completes.
The only usefull that i found is the usage of psutil.Popen(*args, **kwargs) with this one i could terminate whenever i want the process and get some stats for that. But the main issue still remains to get in real time (asynchronously) each sys.stdout or print of file.py, at the moment of each printing.
*preferred solution for python3.6

As noted in the comments, the first and foremost thing is to ensure that your file.py program actually writes the data the way you think it does.
For example, the program you have shown will write nothing for about 40 minutes, because that's how long it takes for 14-byte prints issued at 4-second intervals to fill up the 8-kilobyte IO buffer. Even more confusingly, some programs will appear to write data if you test them on a TTY (i.e. just run them), but not when you start them as subprocesses. This is because on a TTY stdout is line-buffered, and on a pipe it is fully buffered. When the output is not flushed, there is simply no way for another program to detect the output because it is stuck inside the subprocess's buffer that it never bothered to share with anyone.
In other words, don't forget to flush:
while True:
# or just print('something else', flush=True)
sys.stdout.write('something else')
sys.stdout.flush()
time.sleep(4)
With that out of the way, let's examine how to read that output. Asyncio provides a nice stream-based interface to subprocesses that is quite capable of accessing arbitrary output as it arrives. For example:
import asyncio
async def main():
loop = asyncio.get_event_loop()
proc = await asyncio.create_subprocess_exec(
"python", "file.py",
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE
)
# loop.create_task() rather than asyncio.create_task() because Python 3.6
loop.create_task(display_as_arrives(proc.stdout, 'stdout'))
loop.create_task(display_as_arrives(proc.stderr, 'stderr'))
await proc.wait()
async def display_as_arrives(stream, where):
while True:
# 1024 chosen arbitrarily - StreamReader.read will happily return
# shorter chunks - this allows reading in real-time.
output = await stream.read(1024)
if output == b'':
break
print('got', where, ':', output)
# run_until_complete() rather than asyncio.run() because Python 3.6
asyncio.get_event_loop().run_until_complete(main())

How to enter an input,read an output and then enter output based on the received output to an exe file in python?

I have been trying to create a python script which opens a exe file,
enters an input, then reads an output, and based on the output that was received enter another input.
I've been trying to use Python's subprocess library, but the problem is that the communicate() method can only be used once. so it is impossible to enter 2 inputs unless you enter them both in the communicate() method, which doesn't work in this case. because the second input is based upon the output which is generated after the first input, so you can't enter both inputs at the same time.
Also, I searched for 3rd party libraries for python but I didn't find any good libraries for windows.
Can someone show me a way of performing this action using the subprocess library or suggest me a good library for windows?

Consider a simple app that has some basic execution control flow and just echoes its STDIN in reverse to the STDOUT - this can be any executable but we'll stick with Python for simplicity - say, app.py:
#!/usr/bin/env python
import sys
sys.stdout.write("::BEGIN::\n") # tell our listener that we're listening...
sys.stdout.flush() # flush the STDOUT buffer
while True: # a simple event loop
line = sys.stdin.readline().rstrip() # read a line from STDIN
if line: # ignore empty lines
if line == "::END::": # just a convenient way to shut down the app
sys.stdout.write("::END::\n") # tell our listener that we're done
sys.stdout.flush() # flush the STDOUT buffer
break # we're finished here
sys.stdout.write(line[::-1]) # write the reversed line to STDOUT
sys.stdout.write("\n") # add a new line to the STDOUT
sys.stdout.flush() # flush the STDOUT buffer
Then if you want to open this app and communicate with it from your Python script all you need to do is control the subprocesses STDOUT and STDIN and you can do this indefinitely, for example:
import subprocess
# start our subprocess, forward its STDOUT and STDIN to the internal buffers
proc = subprocess.Popen(["python", "app.py"], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
# lets define our data to be sent one by one to our app.py, including ::END:: to exit...
items = ["test", "data", "to", "run", "sequentially", "::END::"]
# a convenience function to get the next item and write it to the passed buffer
def send_next_item(buf):
item = items.pop(0) # pop the first element from `items`
print("REQ: {}".format(item))
buf.write(item) # write it to the passed buffer
buf.write("\n") # write a new line to the passed buffer
buf.flush() # flush the passed buffer
while True: # wait for a prompt by our app
line = proc.stdout.readline().rstrip()
if line == "::BEGIN::":
print("BEGIN!")
send_next_item(proc.stdin) # send the first item to the processes' STDIN
elif line == "::END::":
print("END!")
break # nothing more to do
elif line: # ignore empty lines
print("RES: {}".format(line))
send_next_item(proc.stdin) # send the next item to the processes' STDIN
When you run this you'd get an output like:
BEGIN!
REQ: test
RES: tset
REQ: data
RES: atad
REQ: to
RES: ot
REQ: run
RES: nur
REQ: sequentially
RES: yllaitneuqes
REQ: ::END::
END!
Of course, you can do further processing to decide on how to properly respond to the called application's input request, this is just a basic example.

Communicate with process send key in subprocess linux

I have one sh file, I need to install it in target linux box. So I'm in the process of writing automatic installation for the sh file which required lot of input from user. Example, first thing I made ./file.sh it will show a big paragaraph and ask user to press Enter. I'm stuck in this place. How to send key data to the sub process. Here is what I've tried.
import subprocess
def runProcess(exe):
global p
p = subprocess.Popen(exe, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while(True):
retcode = p.poll() #returns None while subprocess is running
line = p.stdout.readline()
yield line
if(retcode is not None):
break
for line in runProcess('./file.sh'.split()):
if '[Enter]' in line:
print line + 'got it'
p.communicate('\r')
Correct me if my understanding is wrong, pardon me if it is duplicate.

If you need to send a bunch of newlines and nothing else, you need to:
Make sure the stdin for the Popen is a pipe
Send the newlines without causing a deadlock
Your current code does neither. Something that might work (assuming they're not using APIs that require direct interaction in a tty, rather than just reading stdin):
import subprocess
import threading
def feednewlines(f):
try:
# Write as many newlines as it will take
while True:
f.write(b'\n') # Write newline, not carriage return
f.flush() # Flush to ensure it's sent as quickly as possible
except OSError:
return # Done when pipe closed/process exited
def runProcess(exe):
global p
# Get stdin as pipe too
p = subprocess.Popen(exe, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# Use thread to just feed as many newlines as needed to stdin of subprocess
feeder = threading.Thread(target=feednewlines, args=(p.stdin,))
feeder.daemon = True
feeder.start()
# No need to poll, just read until it closes stdout or exits
for line in p.stdout:
yield line
p.stdin.close() # Stop feeding (causes thread to error and exit)
p.wait() # Cleanup process
# Iterate output, and echo when [Enter] seen
for line in runProcess('./file.sh'.split()):
if '[Enter]' in line:
print line + 'got it'
For the case where you need to customize the responses, you're going to need to add communication between parent and feeder thread, which makes this uglier, and it only works if the child process is properly flushing its output when it prompts you, even when not connected to a terminal. You might do something like this to define a global queue:
import queue # Queue on Python 2
feederqueue = queue.Queue()
then change the feeder function to:
def feednewlines(f):
try:
while True:
f.write(feederqueue.get())
f.flush()
except OSError:
return
and change the global code lower down to:
for line in runProcess('./file.sh'.split()):
if '[Enter]' in line:
print line + 'got it'
feederqueue.put(b'\n')
elif 'THING THAT REQUIRES YOU TO TYPE FOO' in line:
feederqueue.put(b'foo\n')
etc.

Command line programs run differently when they are run in a terminal verses when they are run in the background. If the program is attached to a terminal, they run in an interactive line mode expecting user interaction. If stdin is a file or a pipe, they run in block mode where writes are delayed until a certain block size is buffered. Your program will never see the [Enter] prompt because it uses pipes and the data is still in the subprocesses output buffer.
The python pexpect module solves this problem by emulating a terminal and allowing you to interact with the program with a series of "expect" statements.
Suppose we want to run a test program
#!/usr/bin/env python3
data = input('[Enter]')
print(data)
its pretty boring. It prompts for data, prints it, then exits. We can run it with pexpect
#!/usr/bin/env python3
import pexpect
# run the program
p = pexpect.spawn('./test.py')
# we don't need to see our input to the program echoed back
p.setecho(False)
# read lines until the desired program output is seen
p.expect(r'\[Enter\]')
# send some data to the program
p.sendline('inner data')
# wait for it to exit
p.expect(pexpect.EOF)
# show everything since the previous expect
print(p.before)
print('outer done')

How to get asynchronous input AND output using subprocess.POPEN

I've been working on this for a few hours and haven't been able to come up with a good solution. A little background, I'm running a password cracking program that's closed source from the command line but have to constantly pause it when my gpu temperature gets too hot.
I do other manipulations in python with this program so that's the language I'd prefer. Anyways, the password program gives periodic updates on how well it's doing, the gpu temperature, etc. and allows me to pause it at any time.
I'm getting the temperature fine but because of blocking issues I'm guessing I can't send the pause command. It's not doing anything at least. I've seen several examples of threading the output, but haven't seen something that that uses threading input and output without causing any issues.
I mean for all I know this could be impossible under current POPEN constraints but would appreciate some direction.
popen = Popen(command, stdout=PIPE, stdin=PIPE, shell=True)
lines_iterator = iter(popen.stdout.readline, b"")
while 1:
for line in lines_iterator:
cleanLine = line.replace("\n", "")
p = re.compile('[0-9][0-9]c Temp')
m = p.search(cleanLine)
print cleanLine
if m:
temperature = m.group(0)
if int(temperature[:2]) > 80:
overheating = True
print "overheating"
if overheating:
if "[s]tatus [p]ause [r]esume [b]ypass [q]uit" in line:
#It's not doing anything right here, it just continues
print popen.communicate("p")[0]
This is the gist of my code. It's still kind of through the hacky phase so I know that it might not be following best coding practices.

EDIT: Sorry, I was confused in the initial answer about the scope of overheating. I deleted the first part of my answer since it's not relevant anymore.
communicate will wait for the process to exit so it might not be what you're looking for in this case. If you want the process to keep going
you can use something like popen.stdin.write("p"). You might also need to send a "\n" along if that's required by your process.
Also, if you're OK with an extra dependency you might be interested in the pexpect module that was designed to control interactive processes.

A simple portable solution is to use threads here. It is enough if there are no block buffering issues.
To read output and stop input if overheating is detected (not tested):
#!/usr/bin/env python
from subprocess import Popen, PIPE, CalledProcessError
from threading import Event, Thread
def detect_overheating(pipe, overheating):
with pipe: # read output here
for line in iter(pipe.readline, ''):
if detected_overheating(line.rstrip('\n')):
overheating.set() # overheating
elif paused: #XXX global
overheating.clear() # no longer overheating
process = Popen(args, stdout=PIPE, stdin=PIPE, bufsize=1,
universal_newlines=True) # enable text mode
overheating = Event()
t = Thread(target=detect_overheating, args=[process.stdout, overheating])
t.daemon = True # close pipe if the process dies
t.start()
paused = False
with process.stdin: # write input here
while process.poll() is None:
if overheating.wait(1): # Python 2.7+
# overheating
if not paused:
process.stdin.write('p\n') # pause
process.stdin.flush()
paused = True
elif paused: # no longer overheating
pass #XXX unpause here
paused = False
if process.wait() != 0: # non-zero exit status may indicate failure
raise CalledProcessError(process.returncode, args)