We Keep Coding

Python: program hangs when trying to read from subprocess stdout while it is running

I am trying to communicate with a c++ script (let's call it script A) using the python subprocess module. Script A is running alongside the python program and is constantly interacted with. My goal is to send script A input commands, and capture the outputs that are being printed to STDOUT afterwards by script A. I'm working on windows 10.
Here is a snippet describing the logic:
proc = subprocess.Popen([".\\build\\bin.exe"], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
terminate = False
while not terminate:
command = input("Enter your command here: ")
if command == "q":
terminate = True
else:
proc.stdin.write(command.encode()) # send input to script A
output = proc.stdout.readline().decode() # problematic line, trying to capture output from script A
print(f"Output is: {output}")
The problem is that while script A is writing output to STDOUT after each command like I expect it to, the python script hangs when it reaches the line highlighted above. I tried to capture the output using proc.stdout.read(1) with bufsize=0 on the call to Popen and for line in iter(proc.stdout.readlines()) and some other ways but the problem persists.
Would appreciate any help on this because nothing I tried is working for me.
Thanks in advance!

You already suggested to use bufsize=0, which seems the right solution. However, this only affects buffering at the Python side. If the executable you are calling uses buffered input or output, I don't think there's anything you can do about it (as also mentioned here].
If both programs are under your own control, then you can easily make this work. Here is an example. For simplicity I created two Python scripts that interact with each other in a similar way you are doing. Note that this doesn't differ very much from the situation with a C++ application, since in both cases an executable is started as subprocess.
File pong.py (simple demo application that reads input and responds to it - similar to your "script A"):
while True:
try:
line = input()
except EOFError:
print('EOF')
break
if line == 'ping':
print('pong')
elif line == 'PING':
print('PONG')
elif line in ['exit', 'EXIT', 'quit', 'QUIT']:
break
else:
print('what?')
print('BYE!')
File main.py (the main program that communicates with pong.py):
import subprocess
class Test:
def __init__(self):
self.proc = subprocess.Popen(['python.exe', 'pong.py'], bufsize=0, encoding='ascii',
stdout=subprocess.PIPE, stdin=subprocess.PIPE)
def talk(self, tx):
print('TX: ' + tx)
self.proc.stdin.write(tx + '\n')
rx = self.proc.stdout.readline().rstrip('\r\n')
print('RX: ' + rx)
def main():
test = Test()
test.talk('ping')
test.talk('test')
test.talk('PING')
test.talk('exit')
if __name__ == '__main__':
main()
Output of python main.py:
TX: ping
RX: pong
TX: test
RX: what?
TX: PING
RX: PONG
TX: exit
RX: BYE!
Of course there are other solutions as well. For example, you might use a socket to communicate between the two applications. However, this is only applicable if you can modify both application (e.g. if you are developing both applications), not if the executable you are calling is a third-party application.

first, buffersize=0, which is the right solution. However, this is not enough.
In your executeable program, you should set the stdout buffersize to 0. or flush in time.
In C/C++ program, you can add
setbuf(stdout, nullptr);
to your source code.

python terminal close when I click enter

I click enter to enter input data and it closes.
I am using windows ten, new computer, not glitchy.
code is
import os
printput = input("Do: ")
if printput == "send message":
myinput = input("s:")
print (myinput)
pass
elif printput == "clear term":
os.system("clear")
pass
what I do, "Do: send message, Enter, s:random stuff, Enter, #window closes#"

I don't know how you are running your file, but probably at the end of the execution the terminal is closing automatically, to avoid that you can open the windows terminal, navigate to where your file is stored using cd for example cd Desktop and then start your file using the command python filename.py that way the terminal won't close even after your file finishes executing. A better solution that will help you in developing what seems to be a shell that executes your commands is using a while True: at the beginning. This way no matter how you execute your file, terminal won't close. and your script will keep running your commands until you close it, And your code will become like this :
import os
while True:
printput = input("Do: ")
if printput == "send message":
myinput = input("s:")
print (myinput)
pass
elif printput == "clear term":
os.system("cls")
pass
this way your code will keep running forever and asking you for commands and executing them, and i changed os.system("clear") because it doesn't work, to os.system("cls") the command that clears the terminal.
another thing you can do to avoid terminal from closing at the end of execution is to make it wait for a keypress by adding at the end of your code
input("Press Enter to continue...")

Self Restarting a Python Script

I have created a watchdog timer for my script (Python 3), which allows me to halt execution if anything goes wrong (not shown in code below). However, I would like to have the ability to restart the script automatically using only Python (no external scripts). The code needs to be cross platform compatible.
I have tried subprocess and execv (os.execv(sys.executable, ['python'] + sys.argv)), however I am seeing very weird functionality on Windows. I open the command line, and run the script ("python myscript.py"). The script stops but does not exit (verified through Task Manager), and it will not restart itself unless I press enter twice. I would like it to work automatically.
Any suggestions? Thanks for your help!
import threading
import time
import subprocess
import os
import sys
if __name__ == '__main__':
print("Starting thread list: " + str(threading.enumerate()))
for _ in range(3):
time.sleep(1)
print("Sleeping")
''' Attempt 1 with subprocess.Popen '''
# child = subprocess.Popen(['python',__file__], shell=True)
''' Attempt 2 with os.execv '''
args = sys.argv[:]
args.insert(0, sys.executable)
if sys.platform == 'win32':
args = ['"%s"' % arg for arg in args]
os.execv(sys.executable, args)
sys.exit()

Sounds like you are using threading in your original script, which explains why your can't break your original script when simply pressing Ctrl+C. In that case, you might want to add a KeyboardInterrupt exception to your script, like this:
from time import sleep
def interrupt_this()
try:
while True:
sleep(0.02)
except KeyboardInterrupt as ex:
# handle all exit procedures and data cleaning
print("[*] Handling all exit procedures...")
After this, you should be able to automatically restart your relevant procedure (even from within the script itself, without any external scripts). Anyway, it's a bit hard to know without seeing the relevant script, so maybe I can be of more help if you share some of it.

Sending to the stdin of a program in python3

I have to files, main.py and child.py.
I am trying to send a string to the stdin of main.py.
This is my incomplete code:
main.py
from subprocess import *
import time
def main():
program = Popen(['python.exe'. 'child.py', 'start'])
while True: #waiting for'1' to be sent to the stdin
if sys.stdin == '1':
print('text)
if __name__ == '__main__':
main()
child.py
import sys
if sys.argv[1] == 'start':
inp = input('Do you want to send the argument?\n').lower()
if inp == 'no':
sys.exit()
elif inp == 'yes':
#Somehow send '1' to the stdin of 1.py while it is running
I have no idea how to do this.
I am running windows 10 with python 3.5.1
-Thanks
EDIT:
When I am sending the argument back to main.py, I can not re-open the program. os.system re-opens the program which is not useful in my case.
These programs are a small demo of what I am trying to do. In my actual program, I am not able to do that as the two programs are "communicating" with each other an need to be open at all times.
What I need answered is a way to send an argument to main.py perhaps using stdin but when I am sending my argument, It can not re-open the program. Some examples like os.system re-open the program which is not what I am trying to do. I need main.py open at all times.
I have my new current code which is not working. A window pops up and then closes.
main.py
from subprocess import Popen, PIPE, STDOUT
x = Popen(['python.exe', '2.py', 'start'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
while x.poll() is None:
if b'Do you want to send the argument?' in x.stdout.read():
x.stdin.write(b'yes\n')
child.py
import sys
import time
time.sleep(1)
if 1 = 1:
inp = input('Do you want to send the argument?\n').lower()
if inp == 'no':
sys.exit()
elif inp == 'yes':
sys.stdout.write('1')
sys.stdout.flush()
That is my code.

What you need is something along the lines of (in main.py):
from subprocess import Popen, PIPE, STDOUT
x = Popen(['some_child.exe', 'parameter'], stdout=PIPE, stdin=PIPE, stderr=STDOUT)
while x.poll() is None:
child_output = x.stdout.readline()
print(child_output)
if b'Do you want to send the argument?' in child_output:
x.stdin.write(b'yes\n')
x.stdin.flush()
x.stdout.close()
x.stdin.close()
You're assuming child.exe (in your mockup demo, python.exe) is communicating with main.py via sys.stdin/stdout, however these I/O's are used to communicate with the shell that spawned the process.
Much like the childs stdout/stdin will be communicating with the shell that spawned that process, in this case Popen().
Each spawned child process of subprocess.Popen(...) will be isolated with it's own stdout/stdin/stderr, otherwise every subprocess would make a huge mess of your main process stdout/stdin. This means you'll have to check for output on that particular subprocess and write to it accordingly as done in the above example.
One way to look at it is this:
You're starting main.py, and you communicate with it via sys.stdout and sys.stdin. Each input() in main.py will output something to sys.stdout so you can read it.
Exactly the same logic applies to child.exe where every input() will output something to it's sys.stdout (- But remember - sys is not a shared variable across processes).
import sys
if sys.argv[1] == 'start':
inp = input('Do you want to send the argument?\n').lower()
if inp == 'no':
sys.exit()
elif inp == 'yes':
#Somehow send '1' to the stdin of 1.py while it is running
sys.stdout.write('1')
sys.stdout.flush()
But a simple print(1) would do the same because it will essentially output the 1 to sys.stdout for you.
Edit 2018: Don't forget to close your inputs and outputs, as they might leave open file descriptors on your file system, hogging resources and causing problems later in life.
Other conveyers of information
Assuming you have control of the code to child.exe and you can modify the communication pipe in any way, some other options are:
sockets - Use regular sockets to communicate, on *nix the most efficient would be Unix sockets.
Some other solutions can be found here: Best way to return a value from a python script
More cautionary tails!
.readline() will assume there's a \n somewhere in your data, most likely at the end. I switched to .readline() for two reasons, .read() will hang and wait for EOF unless you specify exactly how many bytes to read, if I'm not out on a bicycle. To be able to read all kinds of output you need to incorporate select.select() into your code - or a buffer of some sort where you call x.stdout.read(1) to read one byte at a time. Because if you try to read .read(1024) and there's not 1024 bytes in the buffer, your read will hang until there are 1024 characters.
I left a bug in your child.py code on purpose (mine works) - It's trivial and basic Python - in hopes that it's a learning experience on how to debug errors (you mentioned you're not good at it, this is a way to learn).

Python terminal emulator: Cannot use "cd" command.

I found this code snippet online for a python terminal emulator, I thought it looked cool so I went ahead and attempted to use it. I noticed that I'm not able to use the "cd" command, I'm stuck in the directory I ran the file in. Why is this? What's going on? And how can I modify this code to make it run like a perfect native terminal? I'm still very new to programming and have only played with the subprocess module once in my life. Please help!
import subprocess
import re
while True:
# prevents lots of python error output
try:
s = raw_input('> ')
except:
break
# check if you should exit
if s.strip().lower() == 'exit':
break
# try to run command
try:
cmd = subprocess.Popen(re.split(r'\s+', s), stdout=subprocess.PIPE)
cmd_out = cmd.stdout.read()
# Process output
print cmd_out
except OSError:
print 'Invalid command'

When you open a new process you change the current directory for the new process and not for the calling process. You should use os.chdir instead to change the directory of your program. So you need to parse the command line and check if the command is cd and then decide not to call Popen but os.chdir instead.