We Keep Coding

Read from pty without endless hanging

I have a script, that prints colored output if it is on tty. A bunch of them executes in parallel, so I can't put their stdout to tty. I don't have control over the script code either (to force coloring), so I want to fake it via pty. My code:
invocation = get_invocation()
master, slave = pty.openpty()
subprocess.call(invocation, stdout=slave)
print string_from_fd(master)
And I can't figure out, what should be in string_from_fd. For now, I have something like
def string_from_fd(fd):
return os.read(fd, 1000)
It works, but that number 1000 looks strange . I think output can be quiet large, and any number there could be not sufficient. I tried a lot of solutions from stack overflow, but none of them works (it prints nothing or hanging forever).
I am not very familiar with file descriptors and all that, so any clarification if I'm doing something wrong would be much appreciated.
Thanks!

This won't work for long outputs: subprocess.call will block once the PTY's buffer is full. That's why subprocess.communicate exists, but that won't work with a PTY.
The standard/easiest solution is to use the external module pexpect, which uses PTYs internally: For example,
pexpect.spawn("/bin/ls --color=auto").read()
will give you the ls output with color codes.
If you'd like to stick to subprocess, then you must use subprocess.Popen for the reason stated above. You are right in your assumption that by passing 1000, you read at most 1000 bytes, so you'll have to use a loop. os.read blocks if there is nothing to read and waits for data to appear. The catch is how to recognize when the process terminated: In this case, you know that no more data will arrive. The next call to os.read will block forever. Luckily, the operating system helps you detect this situation: If all file descriptors to the pseudo terminal that could be used for writing are closed, then os.read will either return an empty string or return an error, depending on the OS. You can check for this condition and exit the loop when this happens. Now the final piece to understanding the following code is to understand how open file descriptors and subprocess go together: subprocess.Popen internally calls fork(), which duplicates the current process including all open file descriptors, and then within one of the two execution paths calls exec(), which terminates the current process in favour of a new one. In the other execution path, control returns to your Python script. So after calling subprocess.Popen there are two valid file descriptors for the slave end of the PTY: One belongs to the spawned process, one to your Python script. If you close yours, then the only file descriptor that could be used to send data to the master end belongs to the spawned process. Upon its termination, it is closed, and the PTY enters the state where calls to read on the master end fail.
Here's the code:
import os
import pty
import subprocess
master, slave = pty.openpty()
process = subprocess.Popen("/bin/ls --color", shell=True, stdout=slave,
stdin=slave, stderr=slave, close_fds=True)
os.close(slave)
output = []
while True:
try:
data = os.read(master, 1024)
except OSError:
break
if not data:
break
output.append(data) # In Python 3, append ".decode()" to os.read()
output = "".join(output)

Stop a running process started from Python

I have a ROS code rostopic pub toggle_led std_msgs/Empty that basically starts once and keeps running until CTRL+C is pressed.
Now, I would like to automate this command from Python. I checked Calling an external command in Python but it only shows how to start the command.
How would I start and stop running this process as and when I want?

How would I start and stop running this process as and when I want?
Well, you already know how to start it, as you said in the previous sentence.
How do you stop it? If you want to stop it exactly like a Ctrl-C,* you do that by calling send_signal on it, using CTRL_C_EVENT on Windows, or SIGTERM on Unix.** So:
import signal
import subprocess
try:
sig = signal.CTRL_C_EVENT
except NameError:
sig = signal.SIGTERM
p = subprocess.Popen(['/path/to/prog', '-opt', '42', 'arg'])
# ... later
p.send_signal(sig)
If you only care about Linux (or *nix in general), you can make this even simpler: terminate is guaranteed to do the same thing as send_signal(SIGTERM). So:
import subprocess
p = subprocess.Popen(['/path/to/prog', '-opt', '42', 'arg'])
# ... later
p.terminate()
Since you asked in a comment "Could you please explain the various parameters to subprocess.Popen()": Well, there are a whole lot of them (see Popen Constructor and Frequently Used Arguments in the docs, but I'm only using one, the args parameter.
Normally, you pass a list to args, with the name of the program as the first element in the list, and each separate command-line argument as a separate element. But if you want to use the shell, you pass a string for args, and add a shell=True as another argument.
* Note that "exactly like a Ctrl-C" may not actually be what you want on Windows, unless the program has a console and is a process group owner. This may mean you'll need to add creationflags=subprocess.CREATE_NEW_PROCESS_GROUP to the Popen call. Or it may not—e.g.., if you use shell=True.
** In Python, you can usually ignore the platform differences between CTRL_C_EVENT and SIGTERM and always use the latter, but subprocess.send_signal is one of the few places you can't. On Windows, send_signal(SIGTERM) will call terminate instead of sending a Ctrl-C. If you don't actually care exactly how the process gets stopped, just that it gets stopped somehow, then of course you can use SIGTERM… but in that case, you might as well just call terminate.

In Linux launch a c program from Python handing the c program a large text string as an argument

In Linux. I have a c program that reads a 2048Byte text file as an input. I'd like to launch the c program from a Python script. I'd like the Python script to hand the c program the text string as an argument, instead of writing the text string to a file for the c program to then read.
How can a Python program launch a c program handing it a ~2K (text) data structure?
Also note, I cannot use "subprocess.check_output()". I have to use "os.system()". That's because the latter allows my c-program direct access to terminal input/output. The former does not.

You can pass it as an argument by just… passing it as an argument. Presumably you want to quote it rather than passing it as an arbitrary number of arguments that need to be escaped and so on, but that's easy with shlex.quote. For example:
with open('bigfile.txt', 'rb') as infile:
biginput = infile.read(2048)
os.system('cprogram {}'.format(shlex.quote(biginput)))
If you get an error about the argument or the command line being too long for the shell… then you can't do it. Python can't make the shell do things it can't do, and you refuse to go around the shell (I think because of a misunderstanding, but let's ignore that for the moment). So, you will need some other way to pass the data.
But that doesn't mean you have to store it in a file. You can use the shell from subprocess just as easily as from os.system, which means you can pass it to your child process's stdin:
with subprocess.Popen('cprogram {}'.format(shlex.quote(biginput)),
shell=True, stdin=subprocess.PIPE) as p:
p.communicate(biginput)
Since you're using shell=True, and not replacing either stdout or stderr, it will get the exact same terminal that it would get with os.system. So, for example, if it's doing, say, isatty(fileno(stdout)), it will be true if your Python script is running in a tty, false otherwise.
As a side note, storing it in a tempfile.NamedTemporaryFile may not cost nearly as much as you expect it to. In particular, the child process will likely be able to read the data you wrote right out of the in-memory disk cache instead of waiting for it to be flushed to disk (and it may never get flushed to disk).
I suspect that the reason you thought you couldn't use subprocess is that you were using check_output when you wanted check_call.
If you use check_output (or if you explicit pass stdout=PIPE to most other subprocess functions), the child process's stdout is the pipe that you're reading from, so it's obviously not a tty.
This makes sense: either you want to capture the output, in which case the C program can't output to the tty, or you want to let the C program output to the tty, in which case you can't capture it.* So, just don't capture the output, and everything will be fine.
If I'm right, this means you have no reason to use the shell in the first place, which makes everything a whole lot easier. Of course your data might still be larger than the maximum system argument size** or resource limits***, even without the shell. On most modern systems, you can count on at least 64KB, so definitely try it first:
subprocess.check_call(['cprogram', biginput])
But if you get an E2BIG error:
with subprocess.Popen(['cprogram', biginput], stdin=subprocess.PIPE) as p:
p.communicate(biginput)
* Unless, of course, you want to fake a tty for your child process, in which case you need to look at os.forkpty and related functions, or the pty module.
** On most *BSD and related systems, sysctl kern.argmax and/or getconf ARG_MAX will give you the system limit, or sysconf(_SC_ARG_MAX) from C. There may also be a constant ARG_MAX accessible through <limits.h>. On linux, things are a bit more complicated, because there are a number of different limits (most of which are very, very high) rather than just one single limit. Check your platform's manpage for execve for the details.
*** On some platforms, including recent linux, RLIMIT_STACK affects the max arg size that you can pass. Again, see your platform's execve manpage.

Is there a method for reading characters from an instance of subprocess.Popen when the process it called has not yet issued a newline?

I am attempting to wrap a program that is routinely used at work. When called with an insufficient number of arguments, or with a misspelled argument, the program issues a prompt to the user, asking for the needed input. As a consequence, when calling the routine with subprocess.Popen, the routine never sends any information to stdout or stderr when wrong parameters are passed. subprocess.Popen.communicate() and subprocess.Popen.read(1) both wait for a newline character before any information becomes available.
Is there any way to retrieve information from subprocess.Popen.stdout before the newline character is issued? If not, is there any method that can be used to determine whether the subprocess is waiting for input?

First thing to try: use the bufsize argument to Popen, and set it to 0:
subprocess.Popen(args, bufsize=0, ...)
Unfortunately, whether or not this works also depends upon how the subprocess flushes its output, and I presume you don't have much control over that.
On some platforms, when data written to stdout is flushed will actually change depending on whether the underlying I/O library detects an interactive terminal or a pipe. So while you might think the data is there waiting to be read — because that's how it works in a terminal window — it might actually be line buffered when you're running the same program as a subprocess from another within Python.
Added: I just realised that bufsize=0 is the default anyway. Nuts.

After asking around quite a bit, someone pointed me to the solution. Use pexpect.spawn and pexpect.expect. For example:
Bash "script" in a file titled prompt.sh to emulate the problem - read cannot be called directly from pexpect.spawn.
#!/bin/bash
read -p "This is a prompt: "
This will hang when called by subprocess.Popen. It can be handled by pexpect.spawn, though:
import pexpect
child = pexpect.spawn('./prompt.sh')
child.expect(search)
>>> 0
print child.after #Prints the matched text
>>> 'This is a prompt: '
A list, compiled regex, or list of compiled regex can also be used in place of the string in pexpect.expect to deal with differing prompts.

How to capture Python interpreter's and/or CMD.EXE's output from a Python script?

Is it possible to capture Python interpreter's output from a Python script?
Is it possible to capture Windows CMD's output from a Python script?
If so, which librar(y|ies) should I look into?

If you are talking about the python interpreter or CMD.exe that is the 'parent' of your script then no, it isn't possible. In every POSIX-like system (now you're running Windows, it seems, and that might have some quirk I don't know about, YMMV) each process has three streams, standard input, standard output and standard error. Bu default (when running in a console) these are directed to the console, but redirection is possible using the pipe notation:
python script_a.py | python script_b.py
This ties the standard output stream of script a to the standard input stream of script B. Standard error still goes to the console in this example. See the article on standard streams on Wikipedia.
If you're talking about a child process, you can launch it from python like so (stdin is also an option if you want two way communication):
import subprocess
# Of course you can open things other than python here :)
process = subprocess.Popen(["python", "main.py"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
x = process.stderr.readline()
y = process.stdout.readline()
process.wait()
See the Python subprocess module for information on managing the process. For communication, the process.stdin and process.stdout pipes are considered standard file objects.
For use with pipes, reading from standard input as lassevk suggested you'd do something like this:
import sys
x = sys.stderr.readline()
y = sys.stdin.readline()
sys.stdin and sys.stdout are standard file objects as noted above, defined in the sys module. You might also want to take a look at the pipes module.
Reading data with readline() as in my example is a pretty naïve way of getting data though. If the output is not line-oriented or indeterministic you probably want to look into polling which unfortunately does not work in windows, but I'm sure there's some alternative out there.

I think I can point you to a good answer for the first part of your question.
1. Is it possible to capture Python interpreter's output from a Python
script?
The answer is "yes", and personally I like the following lifted from the examples in the PEP 343 -- The "with" Statement document.
from contextlib import contextmanager
import sys
#contextmanager
def stdout_redirected(new_stdout):
saved_stdout = sys.stdout
sys.stdout = new_stdout
try:
yield None
finally:
sys.stdout.close()
sys.stdout = saved_stdout
And used like this:
with stdout_redirected(open("filename.txt", "w")):
print "Hello world"
A nice aspect of it is that it can be applied selectively around just a portion of a script's execution, rather than its entire extent, and stays in effect even when unhandled exceptions are raised within its context. If you re-open the file in append-mode after its first use, you can accumulate the results into a single file:
with stdout_redirected(open("filename.txt", "w")):
print "Hello world"
print "screen only output again"
with stdout_redirected(open("filename.txt", "a")):
print "Hello world2"
Of course, the above could also be extended to also redirect sys.stderr to the same or another file. Also see this answer to a related question.

Actually, you definitely can, and it's beautiful, ugly, and crazy at the same time!
You can replace sys.stdout and sys.stderr with StringIO objects that collect the output.
Here's an example, save it as evil.py:
import sys
import StringIO
s = StringIO.StringIO()
sys.stdout = s
print "hey, this isn't going to stdout at all!"
print "where is it ?"
sys.stderr.write('It actually went to a StringIO object, I will show you now:\n')
sys.stderr.write(s.getvalue())
When you run this program, you will see that:
nothing went to stdout (where print usually prints to)
the first string that gets written to stderr is the one starting with 'It'
the next two lines are the ones that were collected in the StringIO object
Replacing sys.stdout/err like this is an application of what's called monkeypatching. Opinions may vary whether or not this is 'supported', and it is definitely an ugly hack, but it has saved my bacon when trying to wrap around external stuff once or twice.
Tested on Linux, not on Windows, but it should work just as well. Let me know if it works on Windows!

You want subprocess. Look specifically at Popen in 17.1.1 and communicate in 17.1.2.

In which context are you asking?
Are you trying to capture the output from a program you start on the command line?
if so, then this is how to execute it:
somescript.py | your-capture-program-here
and to read the output, just read from standard input.
If, on the other hand, you're executing that script or cmd.exe or similar from within your program, and want to wait until the script/program has finished, and capture all its output, then you need to look at the library calls you use to start that external program, most likely there is a way to ask it to give you some way to read the output and wait for completion.