We Keep Coding

How to run a unix command and which requires dynamic console input in python and how to run this without manual intervention [duplicate]

This is certainly answered as part of a long discussion about subprocess elsewhere. But the answer is so simple it should be broken out.
How do I pass a string "foo" to a program expecting it on stdin if I use Python 3's subprocess.run()?

Pass input="whatever string you want" and text=True to subprocess.run:
import subprocess
subprocess.run("cat", input="foo\n", text=True)
Per the docs for subprocess.run:
The input argument is passed to Popen.communicate() and thus to the subprocess’s stdin. If used it must be a byte sequence, or a string if encoding or errors is specified or text is true. When used, the internal Popen object is automatically created with stdin=PIPE, and the stdin argument may not be used as well.
To also get the output of the command as a string, add capture_output=True:
subprocess.run("cat", input="foo\n", capture_output=True, text=True)

Simplest possible example, send foo to cat and let it print to the screen.
import subprocess
subprocess.run(['cat'],input=b'foo\n')
Notice that you send binary data and the carriage return.

subprocess popen function get lock when calling it with muscle or mafft in a pipeline

I'm trying to include sequence alignment using muscle or mafft, depending of the user in a pipeline.
To do so, i'm using the subprocess package, but sometimes, the subprocess never terminates and my script doesn't continue. Here is how I call the subprocess:
child = subprocess.Popen(str(muscle_cline), stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
child.wait()
The command muscle_cline looks like this:
./tools/muscle/muscle5.1.win64.exe -align C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\genes-fasta\12S_tmp.fasta -output C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\alignement\12S_tmp_muscle_align.fasta
I'm calling this line in a function that just creates the command line and calls the subprocess, and converts the output.
I'm then calling this function in a for loop
for file in getFastaFile(my_dir):
alignSequenceWithMuscle(file)
The issue is that sometimes, for unknown reasons, the subprocess never finishes and get locked...
I tried to check the returncode of the child, or print stuff to see where it gets locked, and it's getting locked when I'm calling the subprocess.
Any ideas?

You generally want to avoid bare Popen, especially if you don't have a good understanding of its requirements. This is precisely why Python offers you subprocess.check_output and other higher-level functions which take care of the nitty-gritty of managing a subprocess.
output = subprocess.check_output(
["./tools/muscle/muscle5.1.win64.exe",
"-align", r"C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\genes-fasta\12S_tmp.fasta",
"-output", r"C:\Users\alexis\Desktop\git-repo\MitoSplitter\results\alignement\12S_tmp_muscle_align.fasta"],
text=True)
Notice also the raw strings r"..." to avoid having to double the backslashes, and the text=True keyword argument to instruct Python to implicitly decode the bytes you receive from the subprocess.

Passing enterKey to exe file after execution using python

I need to run a external exe file inside a python script. I need two things out of this.
Get whatever the exe outputs to the stdout (stderr).
exe stops executing only after I press the enter Key. I can't change this behavior. I need the script the pass the enter Key input after it gets the output from the previous step.
This is what I have done so far and I am not sure how to go after this.
import subprocess
first = subprocess.Popen(["myexe.exe"],shell=True,stdout=subprocess.PIPE)

from subprocess import Popen, PIPE, STDOUT
first = Popen(['myexe.exe'], stdout=PIPE, stderr=STDOUT, stdin=PIPE)
while first.poll() is None:
data = first.stdout.read()
if b'press enter to' in data:
first.stdin.write(b'\n')
first.stdin.close()
first.stdout.close()
This pipes stdin as well, do not forget to close your open file handles (stdin and stdout are also file handles in a sense).
Also avoid shell=True if at all possible, I use it a lot my self but best practices say you shouldn't.
I assumed python 3 here and stdin and stdout assumes bytes data as input and output.
first.poll() will poll for a exit code of your exe, if none is given it means it's still running.
Some other tips
one tedious thing to do can be to pass arguments to Popen, one neat thing to do is:
import shlex
Popen(shlex.split(cmd_str), shell=False)
It preserves space separated inputs with quotes around them, for instance python myscript.py debug "pass this parameter somewhere" would result in three parameters from sys.argv, ['myscript.py', 'debug', 'pass this parameter somewhere'] - might be useful in the future when working with Popen
Another thing that would be good is to check if there's output in stdout before reading from it, otherwise it might hang the application. To do this you could use select.
Or you could use pexpect which is often used with SSH since it lives in another user space than your application when it asks for input, you need to either fork your exe manually and read from that specific pid with os.read() or use pexpect.

Reading Command line output Python

I have a problem where I am issuing a command using python and then taking in the values to create a list of services.
serviceList = subprocess.Popen(command, shell=True, stdout =subprocess.PIPE).stdout.read()
print serviceList
command is a working command that works perfectly when I copy and paste it into cmd, giving me a list of services and their status.
If I run this command it just returns nothing. When I print out serviceList it is blank.
I am using python 2.7

You must use communicate() method instead of stdout.read() to get the value of serviceList.
Even the Python docs recommend it.
Warning: Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.
Try this:
proc = subprocess.Popen(command, shell=True, stdout =subprocess.PIPE)
serviceList = proc.communicate()[0]
print serviceList
communicate() returns a tuple (stdoutdata, stderrdata). Here, i assign the first element of the tuple to serviceList.

If the program simply prints out a bunch of information then exits, an easier way (also no way for it to deadlock due to full buffer) to read output would be to call:
process = subprocess.Popen(command) # only call shell=True if you *really need it
stdoutdata, stderrdata = process.communicate() # blocks until process terminates
docs:
*Calling shell=True with external input opens your code to shell injection attacks, and should be used with caution

To save the standard output, add output = serviceList.stdout.readlines() to your code.

There's also the subprocess function check_output() which blocks and returns the output of the process as a byte-string. If you want to avoid blocking, you could make a function that calls this and use it as that target for a new Thread() e.g.
import subprocess
import threading
def f():
print subprocess.check_output([command])
threading.Thread(target=f).start()

How do I write to a Python subprocess' stdin?

I'm trying to write a Python script that starts a subprocess, and writes to the subprocess stdin. I'd also like to be able to determine an action to be taken if the subprocess crashes.
The process I'm trying to start is a program called nuke which has its own built-in version of Python which I'd like to be able to submit commands to, and then tell it to quit after the commands execute. So far I've worked out that if I start Python on the command prompt like and then start nuke as a subprocess then I can type in commands to nuke, but I'd like to be able to put this all in a script so that the master Python program can start nuke and then write to its standard input (and thus into its built-in version of Python) and tell it to do snazzy things, so I wrote a script that starts nuke like this:
subprocess.call(["C:/Program Files/Nuke6.3v5/Nuke6.3", "-t", "E:/NukeTest/test.nk"])
Then nothing happens because nuke is waiting for user input. How would I now write to standard input?
I'm doing this because I'm running a plugin with nuke that causes it to crash intermittently when rendering multiple frames. So I'd like this script to be able to start nuke, tell it to do something and then if it crashes, try again. So if there is a way to catch a crash and still be OK then that'd be great.

It might be better to use communicate:
from subprocess import Popen, PIPE, STDOUT
p = Popen(['myapp'], stdout=PIPE, stdin=PIPE, stderr=PIPE)
stdout_data = p.communicate(input='data_to_write')[0]
"Better", because of this warning:
Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

To clarify some points:
As jro has mentioned, the right way is to use subprocess.communicate.
Yet, when feeding the stdin using subprocess.communicate with input, you need to initiate the subprocess with stdin=subprocess.PIPE according to the docs.
Note that if you want to send data to the process’s stdin, you need to create the Popen object with stdin=PIPE. Similarly, to get anything other than None in the result tuple, you need to give stdout=PIPE and/or stderr=PIPE too.
Also qed has mentioned in the comments that for Python 3.4 you need to encode the string, meaning you need to pass Bytes to the input rather than a string. This is not entirely true. According to the docs, if the streams were opened in text mode, the input should be a string (source is the same page).
If streams were opened in text mode, input must be a string. Otherwise, it must be bytes.
So, if the streams were not opened explicitly in text mode, then something like below should work:
import subprocess
command = ['myapp', '--arg1', 'value_for_arg1']
p = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = p.communicate(input='some data'.encode())[0]
I've left the stderr value above deliberately as STDOUT as an example.
That being said, sometimes you might want the output of another process rather than building it up from scratch. Let's say you want to run the equivalent of echo -n 'CATCH\nme' | grep -i catch | wc -m. This should normally return the number characters in 'CATCH' plus a newline character, which results in 6. The point of the echo here is to feed the CATCH\nme data to grep. So we can feed the data to grep with stdin in the Python subprocess chain as a variable, and then pass the stdout as a PIPE to the wc process' stdin (in the meantime, get rid of the extra newline character):
import subprocess
what_to_catch = 'catch'
what_to_feed = 'CATCH\nme'
# We create the first subprocess, note that we need stdin=PIPE and stdout=PIPE
p1 = subprocess.Popen(['grep', '-i', what_to_catch], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We immediately run the first subprocess and get the result
# Note that we encode the data, otherwise we'd get a TypeError
p1_out = p1.communicate(input=what_to_feed.encode())[0]
# Well the result includes an '\n' at the end,
# if we want to get rid of it in a VERY hacky way
p1_out = p1_out.decode().strip().encode()
# We create the second subprocess, note that we need stdin=PIPE
p2 = subprocess.Popen(['wc', '-m'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# We run the second subprocess feeding it with the first subprocess' output.
# We decode the output to convert to a string
# We still have a '\n', so we strip that out
output = p2.communicate(input=p1_out)[0].decode().strip()
This is somewhat different than the response here, where you pipe two processes directly without adding data directly in Python.
Hope that helps someone out.

Since subprocess 3.5, there is the subprocess.run() function, which provides a convenient way to initialize and interact with Popen() objects. run() takes an optional input argument, through which you can pass things to stdin (like you would using Popen.communicate(), but all in one go).
Adapting jro's example to use run() would look like:
import subprocess
p = subprocess.run(['myapp'], input='data_to_write', capture_output=True, text=True)
After execution, p will be a CompletedProcess object. By setting capture_output to True, we make available a p.stdout attribute which gives us access to the output, if we care about it. text=True tells it to work with regular strings rather than bytes. If you want, you might also add the argument check=True to make it throw an error if the exit status (accessible regardless via p.returncode) isn't 0.
This is the "modern"/quick and easy way to do to this.

One can write data to the subprocess object on-the-fly, instead of collecting all the input in a string beforehand to pass through the communicate() method.
This example sends a list of animals names to the Unix utility sort, and sends the output to standard output.
import sys, subprocess
p = subprocess.Popen('sort', stdin=subprocess.PIPE, stdout=sys.stdout)
for v in ('dog','cat','mouse','cow','mule','chicken','bear','robin'):
p.stdin.write( v.encode() + b'\n' )
p.communicate()
Note that writing to the process is done via p.stdin.write(v.encode()). I tried using
print(v.encode(), file=p.stdin), but that failed with the message TypeError: a bytes-like object is required, not 'str'. I haven't figured out how to get print() to work with this.

You can provide a file-like object to the stdin argument of subprocess.call().
The documentation for the Popen object applies here.
To capture the output, you should instead use subprocess.check_output(), which takes similar arguments. From the documentation:
>>> subprocess.check_output(
... "ls non_existent_file; exit 0",
... stderr=subprocess.STDOUT,
... shell=True)
'ls: non_existent_file: No such file or directory\n'