I may not at all understand this correctly, but I am trying to allow a Python program to interface with a subprocess that runs commands as if on a Linux shell.
For example, I want to be able to run "cd /" and then "pwd later in the program and get "/".
I am currently trying to use subprocess.Popen and the communicate() method to send and receive data. The first command, sent with the Popen constructor, runs fine and gives proper output. But I cannot send another command via communicate(input="pwd").
My code so far:
from subprocess i
term=Popen("pwd", stdout=PIPE, stdin=PIPE)
print(flush(term.communicate()))
term.communicate(input="cd /")
print(flush(term.communicate(input="pwd")))
Is there a better way to do this? Thanks.
Also, I am running Python 3.
First of all, you need to understand that running a shell command and running a program aren't the same thing.
Let me give you an example:
>>> import subprocess
>>> subprocess.call(['/bin/echo', '$HOME'])
$HOME
0
>>> subprocess.call(['/bin/echo $HOME'], shell=True)
/home/kkinder
0
Notice that without the shell=True parameter, the text of $HOME is not expanded. That's because the /bin/echo program doesn't parse $HOME, Bash does. What's really happening in the second call is something analogous to this:
>>> subprocess.call(['/bin/bash', '-c', '/bin/echo $HOME'])
/home/kkinder
0
Using the shell=True parameter basically says to the subprocess module, go interpret this text using a shell.
So, you could add shell=True, but then the problem is that once the command finishes, its state is lost. Each application in the stack has its own working directory. So what the directory is will be something like this:
bash - /foo/bar
python - /foo
bash via subprocess - /
After your command executes, the python process's path stays the same and the subprocess's path is discarded once the shell finishes your command.
Basically, what you're asking for isn't practical. What you would need to do is, open a pipe to Bash, interactively feed it commands your user types, then read the output in a non-blocking way. That's going to involve a complicated pipe, threads, etc. Are you sure there's not a better way?
Related
I have a python script which takes an input, formats it into a command which calls another script on the server, and then executes using subprocess:
import sys, subprocess
thingy = sys.argv[1]
command = 'usr/local/bin/otherscript.pl {0} &'.format(thingy)
command_list = command.split()
subprocess.call(command_list)
I append & to the end because otherscript.pl takes some time to execute, and I prefer to have run in the background. However, the script still seems to execute without giving me back control to the shell, and I have to wait until execution finishes to get back to my prompt. Is there another way to use subprocess to fully run the script in background?
& is a shell feature. If you want it to work with subprocess, you must specify shell=True like:
subprocess.call(command, shell=True)
This will allow you to run command in background.
Notes:
Since shell=True, the above uses command, not command_list.
Using shell=True enables all of the shell's features. Don't do this unless command including thingy comes from sources that you trust.
Safer Alternative
This alternative still lets you run the command in background but is safe because it uses the default shell=False:
p = subprocess.Popen(command_list)
After this statement is executed, the command will run in background. If you want to be sure that it has completed, run p.wait().
If you want to execute it in Background I recommend you to use nohup output that would normally go to the terminal goes to a file called nohup.out
import subprocess
subprocess.Popen("nohup usr/local/bin/otherscript.pl {0} >/dev/null 2>&1 &", shell=True)
>/dev/null 2>&1 & will not create output and will redirect to background
I want to run multiple Terminal commands from Python using subprocess and simultaneously not only execute the commands but also print the output that appears in Terminal in full to my stdout, so I can see it in real-time (as I would if making the commands directly in Terminal).
Now, using the advice here I was able to run multiple Bash commands from Python:
def subprocess_cmd(command):
process = subprocess.Popen(command,stdout=subprocess.PIPE, shell=True)
proc_stdout = process.communicate()[0].strip()
print(proc_stdout)
subprocess_cmd('echo a; echo b; cd /home/; ls')
Output:
b'a\nb\n<Files_in_my_home_folder>'
So far so good. But if I try to run ls -w (which should raise an error),
subprocess_cmd('echo a; echo b; cd /home/; ls -w')
output:
b'a\nb'
whereas the error message should be shown as it would in Terminal:
ls: option requires an argument -- 'w'
Try 'ls --help' for more information.
I would like to print out whatever is in Terminal (simultaneously with running the command) for whatever the command is, be it running some executable, or a shell command like ls.
I am using Python 3.7+ so any solution using subprocess.run or similar is also welcome. However, I'm not sure this takes multiple commands together nor does using capture_output=True, text=True print error messages.
The stdout=subprocess.PIPE (or the shorthand capture_output=True which subsumes this and a few related settings) says that you want Python to read the output. If you simply want the subprocess to spill whatever it prints directly to standard output and/or standard error, you can simply leave out this keyword argument.
As always, don't use Popen if you can avoid it (and usually avoid shell=True if you can, though that is not possible in your example).
subprocess.check_call('echo a; echo b; cd /home/; ls', shell=True)
To briefly reiterate, this bypasses Python entirely, and lets the subprocess write to its (and Python's) standard output and/or standard error without Python's involvement or knowledge. If you need for Python to know what's printed, you'll need to have your script capture it, and have Python print it if required.
I have a python script which takes an input, formats it into a command which calls another script on the server, and then executes using subprocess:
import sys, subprocess
thingy = sys.argv[1]
command = 'usr/local/bin/otherscript.pl {0} &'.format(thingy)
command_list = command.split()
subprocess.call(command_list)
I append & to the end because otherscript.pl takes some time to execute, and I prefer to have run in the background. However, the script still seems to execute without giving me back control to the shell, and I have to wait until execution finishes to get back to my prompt. Is there another way to use subprocess to fully run the script in background?
& is a shell feature. If you want it to work with subprocess, you must specify shell=True like:
subprocess.call(command, shell=True)
This will allow you to run command in background.
Notes:
Since shell=True, the above uses command, not command_list.
Using shell=True enables all of the shell's features. Don't do this unless command including thingy comes from sources that you trust.
Safer Alternative
This alternative still lets you run the command in background but is safe because it uses the default shell=False:
p = subprocess.Popen(command_list)
After this statement is executed, the command will run in background. If you want to be sure that it has completed, run p.wait().
If you want to execute it in Background I recommend you to use nohup output that would normally go to the terminal goes to a file called nohup.out
import subprocess
subprocess.Popen("nohup usr/local/bin/otherscript.pl {0} >/dev/null 2>&1 &", shell=True)
>/dev/null 2>&1 & will not create output and will redirect to background
I have a python app that has lots of outputs on the screen which can be used for debugging. out of all the logging techniques, "script" command works well for me because I can see the output on the screen as well as logging it. I want to include that at the beginning of my python app to run automatically and log everything, when I do, however, the python program doesn't run. as soon as I type exit at the terminal (which stops script logging) the app starts working. The command I'm using is:
command="script /tmp/appdebug/debug.txt"
os.system(command)
I have also tried script -q but the same issue is there. Would appreciate any help.
Cheers
Well, I did find the answer for anyone who is interested:
https://stackoverflow.com/questions/15507602/logging-all-bash-in-and-out-with-script-command
and
Bash script: Using "script" command from a bash script for logging a session
I will keep this question as others might have the same issue and finding those answers wasn't exactly easy :)
Cheers
Try to use subprocess, like so:
from subprocess import Popen, PIPE
p = Popen(['script', '/tmp/appdebug/debug.txt'], stderr=PIPE, stdout=PIPE)
stdout, stderr = p.communicate()
script is a wrapper for a session of interactions. Even if it appears to terminate quickly after being started in a shell, this is not so; instead it starts a new shell in which you can interact so that everything is logged to a file.
What does this mean for you?
Your approach of using script cannot work. You start script using os.system which will wait for script to terminate before the next Python statement is executed. script's work will only happen before it terminates (i. e. during the uninteresting waiting period of your Python program).
I propose to use script -c yourprog.py yourprog.log instead. This will execute and wrap the yourprog.py and the session will be stored in yourprog.log.
I have what seems to be a simple use case: I launch a script (python or bash) which runs an emulator from command prompt and then the emulator takes commands until I type ctrl-c or exit. I want to do this same thing from a shell and my code below isn't working. What I am trying to do is test automation so I want to issue commands directly to the application from command shell. In python, I have the following:
import os
import subprocess
command = ['/usr/local/bin/YCTV-SIM.sh', '-Latest'] #emulator for yahoo widgets
process = subprocess.Popen( command, shell=True, stdin=subprocess.PIPE )
time.sleep(12) #wait for launch to finish
print '/widgets 1' #first command to issue
print '/key enter' #second command to issue
process.wait()
As you can see, this is some pretty simple stuff. When 'YCTV-SIM.sh' is launched from the command shell, I am put into an input mode and my key entries are sent to the application shell (YCTV-SIM.sh reads raw input) so ideally, I would be able to pipe text directly to this application shell. So far tho, nothing happens; test outputs to the console window but the application does not respond to the commands that I attempt to issue. I am using python 2.6.3, if that matters, but Python is not required..
Language is immaterial at this point so PERL, Python, Bash, TCL... whatever you can suggest that might help.
You need to redirect stdin of the child process and write into it. See e.g. subprocess.Popen.communicate.