I am attempting to follow this answer here: https://stackoverflow.com/a/5087695/343381
I have a need to execute multiple bash commands within a single environment. My test case is simple:
import subprocess
cmd = subprocess.Popen(['bash'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
# Write the first command
command = "export greeting=hello\n"
cmd.stdin.write(command)
cmd.stdin.flush() # Must include this to ensure data is passed to child process
result = cmd.stdout.read()
print result
# Write the second command
command = "echo $greeting world\n"
cmd.stdin.write(command)
cmd.stdin.flush() # Must include this to ensure data is passed to child process
result = cmd.stdout.read()
print result
What I expected to happen (based on the referenced answer) is that I see "hello world" printed. What actually happens is that it hangs on the first cmd.stdout.read(), and never returns.
Can anyone explain why cmd.stdout.read() never returns?
Notes:
It is absolutely essential that I run multiple bash commands from python within the same environment. Thus, subprocess.communicate() does not help because it waits for the process to terminate.
Note that in my real test case, it is not a static list of bash commands to execute. The logic is more dynamic. I don't have the option of running all of them at once.
You have two problems here:
Your first command does not produce any output. So the first read blocks waiting for some.
You are using read() instead of readline() -- read() will block until enough data is available.
The following modified code (updated with Martjin's polling suggestion) works fine:
import subprocess
import select
cmd = subprocess.Popen(['bash'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
poll = select.poll()
poll.register(cmd.stdout.fileno(),select.POLLIN)
# Write the first command
command = "export greeting=hello\n"
cmd.stdin.write(command)
cmd.stdin.flush() # Must include this to ensure data is passed to child process
ready = poll.poll(500)
if ready:
result = cmd.stdout.readline()
print result
# Write the second command
command = "echo $greeting world\n"
cmd.stdin.write(command)
cmd.stdin.flush() # Must include this to ensure data is passed to child process
ready = poll.poll(500)
if ready:
result = cmd.stdout.readline()
print result
The above has a 500ms timeout - adjust to your needs.
Related
I'm making a shell with python. So far I have gotten cd to work (not pretty I know, but it's all I need for now). When I su root (for example) I get a root shell, but I can't capture the output I receive after running a command. However the shell does accept my commands, as when I type exit it exits. Is there a way to capture the output of a 'new' shell?
import os, subprocess
while True:
command = input("$ ")
if len(command.split(" ")) >= 2:
print(command.split(" ")[0]) #This line is for debugging
if command.split(" ")[0] == "cd" or command.split(" ")[1] == "cd":
os.chdir(command.split(" ")[command.split(" ").index("cd") + 1])
continue
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE, universal_newlines=True)
output, error = process.communicate()
print(output.strip("\n"))
EDIT: To make my request a bit more precise, I'd like a way to authenticate as another user from a python script, basically catching the authentication, doing it in the background and then starting a new subprocess.
You really need to understand how subprocess.Popen works. This command executes a new sub-process (on a Unix machine, calls fork and then exec). The new sub-process is a separate process. Your code just calls communicate once and then discards of it.
If you just create a new shell by calling subprocess.Popen and then running su <user> inside of it, the shell will be closed right after that and the next time, you'll be running the command using the same (original) user again.
What you want is probably to create a single subprocess at the beginning of your application and then be a sort of a proxy between the user and the underlying process, and then just keep writing to its stdin and reading from stdout.
Here's an example:
import os, subprocess
process = subprocess.Popen(["bash"], stdin=subprocess.PIPE,
stdout=subprocess.PIPE, universal_newlines=True)
while True:
command = input("$ ")
process.stdin.write(command + "\n")
process.stdin.flush()
output = process.stdout.readline()
print(output.strip("\n"))
(I removed the cd command parsing bit because it wasn't constructive to understanding the solution here, but you can definitely add specific handlers for specific inputs that wrap the underlying shell)
I am attempting to call a bash script via the subprocess Popen function passes in a for loop. My intent is that with each iteration, a new string commit from an array out is passed as an argument to the Popen command. The command invokes a bash script that outputs a text identified by the variable commit and greps certain lines from that particular text. However, I can't get the output to flush out in the Python for loop. Right now, only the grepped data from the final commit in out is being passed into my final data structure (a pandas dataframe).
accuracy_dictionary = {}
for commit in out:
accuracy_dictionary.setdefault(commit, {})
p2 = subprocess.Popen(['~/Desktop/find_accuracies.sh', commit], encoding='utf-8', shell=True, stdout=subprocess.PIPE)
outputstring = p2.stdout.read()
# This part below is less critical to the problem at hand
# I'm putting the data from each file in a dictionary
for acc_type_line in outputstring.split('\n'):
accuracy = acc_type_line.split(': ')
if accuracy != ['']:
acc_type = accuracy[0]
value = accuracy[1]
accuracy_dictionary[commit][acc_type] = float(value)
acc_data = pd.DataFrame.from_dict(accuracy_dictionary).T
Here is the bash script that is being called:
"find_accuracies.sh":
#!/bin/sh
COMMIT=$1
git show $COMMIT:blahblahfolder/blahblah.txt | grep --line-buffered 'accuracy'
acc_data returns a dataframe of nrows=len(out) populated by unique commits, but the value is the exact same for all rows for each acc_type
For example, my output looks like this:
How can I call the file "find_accuracies.sh" with the subprocess command and have it flush the unique values of each file for each commit?
I hope this help addressing the immediate problem you're seeing: Here you should really use communicate with subprocess.PIPE as it waits for the command to finish and give give you all of its output:
outputstring = p2.communicate()[0]
You can also use convenient method like check_output to the same effect:
outputstring = subprocess.check_output(['~/Desktop/find_accuracies.sh', commit],
encoding='utf-8', shell=True)
Or also in py3 use run should also do:
p2 = subprocess.run(['~/Desktop/find_accuracies.sh', commit],
encoding='utf-8', shell=True, stdout=subprocess.PIPE)
outputstring = p2.stdout
Now few more comments, hints and suggestions:
I am a little surprised it works for you as using shell=True and list of arguments should (see the paragraph starting with "On POSIX with shell=True") make your commit argument of the underlying sh wrapped around your script call and not of the script itself. In any case you can (and I would suggest to) actually drop the shell and leave HOME resolution to python:
from pathlib import Path
executable = Path.home().joinpath('Desktop/find_accuracies.sh')
p2 = subprocess.run([executable, commit],
encoding='utf-8', stdout=subprocess.PIPE)
outputstring = p2.stdout
You can (or must for py <3.5) also use os.path.expanduser('~/Desktop/find_accuracies.sh') instead of Path.home() to get script executable. On the other hand for >=3.7 you could replace stdout=subprocess.PIPE with capture_output=True.
And last but not least. It seems a bit unnecessary to call a bash script (esp. double wrapped in sh call like in the original example) just to run git through grep when we already have a python script to process the information. I would actually try to run the corresponding git command directly getting the bulk of its output and process its output in the python script itself to get the bits of interest.
I want to run a program, wait for it's output, send inputs to it and repeat until a condition.
All I could find was questions about waiting for a program to finish, which is NOT the case. The process will still be running, it just won't be giving any (new) outputs.
Program output is in stdout and in a log file, either can be used.
Using linux.
Code so far:
import subprocess
flag = True
vsim = subprocess.popen(['./run_vsim'],
stdin=subprocess.pipe,
shell=true,
cwd='path/to/program')
while flag:
with open(log_file), 'r') as f:
for l in f:
if condition:
break
vsim.stdin.write(b'do something\n')
vsim.stdin.flush()
vsim.stdin.write(b'do something else\n')
vsim.stdin.flush()
As is, the "do something" input is being sent multiple times even before the program finished starting up. Also, the log file is read before the program finishes running the command from the last while iteraction. That causes it to buffer the inputs, so I keeps executing the commands even after the condition as been met.
I could use time.sleep after each stdin.write but since the time needed to execute each command is variable, I would need to use times longer than necessary making the python script slower. Also, that's a dumb solution to this.
Thanks!
If you are using python3, you can try updating your code to use subprocess.run instead. It should wait for your task to complete and return the output.
As of 2019, you can use subprocess.getstatusoutput() to run a process and wait for the output, i.e.:
import subprocess
args = "echo 'Sleep for 5 seconds' && sleep 5"
status_output = subprocess.getstatusoutput(args)
if status_output[0] == 0: # exitcode 0 means NO error
print("Ok:", status_output[1])
else:
print("Error:", status_output[1])
Python Demo
From python docs:
subprocess.getstatusoutput(_cmd_)
Return (exitcode, output) of executing cmd in a shell.
Execute the string cmd in a shell with Popen.check_output() and return a 2-tuple (exitcode, output). The locale encoding is used; see the notes on Frequently Used Arguments for more details.
A trailing newline is stripped from the output. The exit code for the command can be interpreted as the return code of subprocess. Example:
>>> subprocess.getstatusoutput('ls /bin/ls')
(0, '/bin/ls')
>>> subprocess.getstatusoutput('cat /bin/junk')
(1, 'cat: /bin/junk: No such file or directory')
>>> subprocess.getstatusoutput('/bin/junk')
(127, 'sh: /bin/junk: not found')
>>> subprocess.getstatusoutput('/bin/kill $$')
(-15, '')
You can use commands instead of subprocess. Here is an example with ls command:
import commands
status_output = commands.getstatusoutput('ls ./')
print status_output[0] #this will print the return code (0 if everything is fine)
print status_output[1] #this will print the output (list the content of the current directory)
I am trying to learn how to write a script control.py, that runs another script test.py in a loop for a certain number of times, in each run, reads its output and halts it if some predefined output is printed (e.g. the text 'stop now'), and the loop continues its iteration (once test.py has finished, either on its own, or by force). So something along the lines:
for i in range(n):
os.system('test.py someargument')
if output == 'stop now': #stop the current test.py process and continue with next iteration
#output here is supposed to contain what test.py prints
The problem with the above is that, it does not check the output of test.py as it is running, instead it waits until test.py process is finished on its own, right?
Basically trying to learn how I can use a python script to control another one, as it is running. (e.g. having access to what it prints and so on).
Finally, is it possible to run test.py in a new terminal (i.e. not in control.py's terminal) and still achieve the above goals?
An attempt:
test.py is this:
from itertools import permutations
import random as random
perms = [''.join(p) for p in permutations('stop')]
for i in range(1000000):
rand_ind = random.randrange(0,len(perms))
print perms[rand_ind]
And control.py is this: (following Marc's suggestion)
import subprocess
command = ["python", "test.py"]
n = 10
for i in range(n):
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = p.stdout.readline().strip()
print output
#if output == '' and p.poll() is not None:
# break
if output == 'stop':
print 'sucess'
p.kill()
break
#Do whatever you want
#rc = p.poll() #Exit Code
You can use subprocess module or also the os.popen
os.popen(command[, mode[, bufsize]])
Open a pipe to or from command. The return value is an open file object connected to the pipe, which can be read or written depending on whether mode is 'r' (default) or 'w'.
With subprocess I would suggest
subprocess.call(['python.exe', command])
or the subprocess.Popen --> that is similar to os.popen (for instance)
With popen you can read the connected object/file and check whether "Stop now" is there.
The os.system is not deprecated and you can use as well (but you won't get a object from that), you can just check if return at the end of execution.
From subprocess.call you can run it in a new terminal or if you want to call multiple times ONLY the test.py --> than you can put your script in a def main() and run the main as much as you want till the "Stop now" is generated.
Hope this solve your query :-) otherwise comment again.
Looking at what you wrote above you can also redirect the output to a file directly from the OS call --> os.system(test.py *args >> /tmp/mickey.txt) then you can check at each round the file.
As said the popen is an object file that you can access.
What you are hinting at in your comment to Marc Cabos' answer is Threading
There are several ways Python can use the functionality of other files. If the content of test.py can be encapsulated in a function or class, then you can import the relevant parts into your program, giving you greater access to the runnings of that code.
As described in other answers you can use the stdout of a script, running it in a subprocess. This could give you separate terminal outputs as you require.
However if you want to run the test.py concurrently and access variables as they are changed then you need to consider threading.
Yes you can use Python to control another program using stdin/stdout, but when using another process output often there is a problem of buffering, in other words the other process doesn't really output anything until it's done.
There are even cases in which the output is buffered or not depending on if the program is started from a terminal or not.
If you are the author of both programs then probably is better using another interprocess channel where the flushing is explicitly controlled by the code, like sockets.
You can use the "subprocess" library for that.
import subprocess
command = ["python", "test.py", "someargument"]
for i in range(n):
p = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
while True:
output = p.stdout.readline()
if output == '' and p.poll() is not None:
break
if output == 'stop now':
#Do whatever you want
rc = p.poll() #Exit Code
I'm trying to run a Powershell subprocess from Python. I need to send Powershell code from Python to the child process. I've got this far:
import subprocess
import time
args = ["powershell", "-NoProfile", "-InputFormat None", "-NonInteractive"]
startTime = time.time()
process = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
process.stdin.write("Write-Host 'FINISHED';".encode("utf-8"))
result = ''
while 'FINISHED' not in result:
result += process.stdout.read(32).decode('utf-8')
if time.time() > startTime + 5:
raise TimeoutError(result)
print(result)
This times out, because nothing ever gets written to stdout. I think the Write-Host cmdlet never gets executed. Even the simple bash/Cygwin code echo "Write-Host 'FINISHED';" | powershell doesn't seem to do the job.
For comparison, sending the code block using the -Command flag works correctly.
How can I convince Powershell to run the code which I'm sending to stdin?
There a couple of things you can consider:
Invoke PowerShell in a mode where you provide it with a script file which it should execute. Write this script file prior to calling the subprocess. Use the -File <FilePath> parameter for PowerShell (cf. the docs)
If you really want to go with the stdin technique, you might be missing a newline character after the command. If this does not help, you might need to send another control character that tells PowerShell that input EOF is reached. You definitely need to consult the PowerShell docs for finding out how to 'terminate' commands on stdin. One thing you definitely need is the -Command - arguments: The value of Command can be "-", a string. or a script block. If the value of Command is "-", the command text is read from standard input. You may also want to look at this little hack: https://stackoverflow.com/a/13877874/145400
If you only want to execute one command, you can simplify your code by using out, err = subprocess.communicate(in)
I had trouble with a similar task, but I was able to solve it.
First my example code:
import subprocess
args = ["powershell.exe", "-Command", r"-"]
process = subprocess.Popen(args, stdin = subprocess.PIPE, stdout = subprocess.PIPE)
process.stdin.write(b"$data = Get-ChildItem C:\\temp\r\n")
process.stdin.write(b"Write-Host 'Finished 1st command'\r\n")
process.stdin.write(b"$data | Export-Clixml -Path c:\\temp\state.xml\r\n")
process.stdin.write(b"Write-Host 'Finished 2nd command'\r\n")
output = process.communicate()[0]
print(output.decode("utf-8"))
print("done")
The main issue was the correct argument list args. It is required to start the powershell with the -Command-flag, followed by "-" as indicated by Jan-Philipp.
Another mystery was the end-of-line character that is required to get the stuff executed. \r\n works quite well.
Getting the output of the Powershell is still an issue. But if you don't care about realtime, you can collect the output after finishing all executions by calling
output = process.communicate()[0]
However, the active Powershell will be terminated afterwards.