I want to run multiple Terminal commands from Python using subprocess and simultaneously not only execute the commands but also print the output that appears in Terminal in full to my stdout, so I can see it in real-time (as I would if making the commands directly in Terminal).
Now, using the advice here I was able to run multiple Bash commands from Python:
def subprocess_cmd(command):
process = subprocess.Popen(command,stdout=subprocess.PIPE, shell=True)
proc_stdout = process.communicate()[0].strip()
print(proc_stdout)
subprocess_cmd('echo a; echo b; cd /home/; ls')
Output:
b'a\nb\n<Files_in_my_home_folder>'
So far so good. But if I try to run ls -w (which should raise an error),
subprocess_cmd('echo a; echo b; cd /home/; ls -w')
output:
b'a\nb'
whereas the error message should be shown as it would in Terminal:
ls: option requires an argument -- 'w'
Try 'ls --help' for more information.
I would like to print out whatever is in Terminal (simultaneously with running the command) for whatever the command is, be it running some executable, or a shell command like ls.
I am using Python 3.7+ so any solution using subprocess.run or similar is also welcome. However, I'm not sure this takes multiple commands together nor does using capture_output=True, text=True print error messages.
The stdout=subprocess.PIPE (or the shorthand capture_output=True which subsumes this and a few related settings) says that you want Python to read the output. If you simply want the subprocess to spill whatever it prints directly to standard output and/or standard error, you can simply leave out this keyword argument.
As always, don't use Popen if you can avoid it (and usually avoid shell=True if you can, though that is not possible in your example).
subprocess.check_call('echo a; echo b; cd /home/; ls', shell=True)
To briefly reiterate, this bypasses Python entirely, and lets the subprocess write to its (and Python's) standard output and/or standard error without Python's involvement or knowledge. If you need for Python to know what's printed, you'll need to have your script capture it, and have Python print it if required.
Related
I have a python script which takes an input, formats it into a command which calls another script on the server, and then executes using subprocess:
import sys, subprocess
thingy = sys.argv[1]
command = 'usr/local/bin/otherscript.pl {0} &'.format(thingy)
command_list = command.split()
subprocess.call(command_list)
I append & to the end because otherscript.pl takes some time to execute, and I prefer to have run in the background. However, the script still seems to execute without giving me back control to the shell, and I have to wait until execution finishes to get back to my prompt. Is there another way to use subprocess to fully run the script in background?
& is a shell feature. If you want it to work with subprocess, you must specify shell=True like:
subprocess.call(command, shell=True)
This will allow you to run command in background.
Notes:
Since shell=True, the above uses command, not command_list.
Using shell=True enables all of the shell's features. Don't do this unless command including thingy comes from sources that you trust.
Safer Alternative
This alternative still lets you run the command in background but is safe because it uses the default shell=False:
p = subprocess.Popen(command_list)
After this statement is executed, the command will run in background. If you want to be sure that it has completed, run p.wait().
If you want to execute it in Background I recommend you to use nohup output that would normally go to the terminal goes to a file called nohup.out
import subprocess
subprocess.Popen("nohup usr/local/bin/otherscript.pl {0} >/dev/null 2>&1 &", shell=True)
>/dev/null 2>&1 & will not create output and will redirect to background
I have a Python script which is running a shell script using the subprocess library. It has to run on any platform so I have 2 shell scripts, one for Linux/MacOS (cm) and one for Windows (cm.cmd).
Let's say they both contain just a single command example_command -param.
The code which is running the shell script looks like the following:
json = subprocess.run(['cm'], shell=True)
This way, thanks to the shell handling the execution of the script (shell=True), it runs the script cm on Linux/MacOS platforms and cm.cmd on Windows.
The output of the script is a JSON and it works properly on Linux/MacOS platforms, the only problem is with Windows where the output contains the shell prompt which breaks the JSON obviously.
The captured output in the json variable may look like this:
My prompt c:\ $ example_command -param
{ "json_data": ... }
How to avoid printing of the prompt to the subprocess output?
It's caused by the feature called command echoing which is enabled by default but it may be disabled using the echo command. From the documentation:
Syntax
echo [on | off]
Parameters
[on | off] Turns on or off the command echoing feature. Command echoing is on by default.
If you add echo off at the first line of the script, it will disable the command echoing for all subsequent commands but it will echo the echo off command itself. To suppress even echoing of that command, simply prefix it with #.
At sign (#) as a command prefix has the same effect as echo off but only for a single command.
So to summarize it: Simply add #echo off at the first line of the shell script (or batch in Windows terminology) and that's it. Only the output of command(s) executed in the script will be sent to stdout.
The Issue
I have a Python script that when I run it from the command line I do not want to record anything within .bash_history.
The reason for this is that the script uses the Python argparse library which allows me to pass in arguments to the python code directly from the command line.
For example I could write the script so that it would use "123456" as a value in the script:
$ ./scriptname.py -n 123456
The issue is that I don't want the value 123456 stored in .bash_history. In fact, I'd rather the entire command was never stored into the .bash_history file in the first place.
What I've Tried
Subprocess & history -c
I've added the subprocess library to the top of my script and then included this directly after to attempt to proactively clear the current history of the shell I am working in:
subprocess.call("history -c", shell=True)
Theoretically this should clear the history of the current shell. I don't see errors from this so I'm assuming that it runs in some other shell. When I run it outside of the script (directly after running the command to invoke the script) it works properly.
Subprocess & unset HISTFILE
I have also used subprocess with the following with no success:
subprocess.call("unset HISTFILE", shell=True)
os.system & history -c
I've also used the os library for Python and included the following in the script:
os.system("history -c")
os.system and unset HISTFILE
I've also tried unset HISTFILE with os.system to no avail.
os.system("unset HISTFILE")
Preferred Solution Characteristics
I realize that I could simply type in unset HISTFILE or history -c after using the command. But I want this to be as much as possible a self-contained script.
Ideally the solution would prevent the ./scomescript.py command from ever being recorded within .bash_history.
I need this script to output text to the terminal based on the input so I can't close the terminal immediately afterwards either.
I imagine there must be a way to do this from within the python script itself - this is my preference.
This really isn't very feasible... Adding the entry to the history file is performed by the interactive shell, and it occurs after the command has completed and the parent shell exits. It is, strictly speaking, possible, if you were to make your python program execute spawn a hacky background process that did something like read the history file in a loop re-writing it. I really can't advocate anything like this, but you could append your script with something like:
os.system("nohup bash -ic 'while :; do read -d \"\" history < \"$HISTFILE\"; echo \"$history\" | sed -e\"s#^%s.*##\" -e\"/^$/d\" > \"$HISTFILE\"; sleep 1; done &' >/dev/null 2>&1" % sys.argv[0])
I think a much better way to accomplish your goal of not recording any arguments would be to use something like var = raw_input("") instead of passing sensitive argument on the command line.
You could also perhaps create a shell function to wrap your script, something like my_script(){ set +o history; python_script.py "$#; set -o history ;}?
I am using terminal command
while ! echo exit | nc 10.0.2.11 9445; do sleep 10; done
in my commandline to lookup port in my remote machine.( it is working fine). I want to do this operation inside my python script. I found subprocess and I want to know that how can I do this with subprocess ?
from subprocess import call
call(["while xxxxxxxxxxxxxxxxxxxxxxxxxxx"])
subprocess.call does not by default use a shell to run its commands. Therefore, things like while are unknown commands. Instead, you could pass shell=True to call (security risk with dynamic data and user input*) or call the shell directly (the same advice applies):
from subprocess import call
call("while ! echo exit | nc 10.0.2.11 9445; do sleep 10; done", shell="True")
or directly with the shell, this is (a) less portable (because it assumes a specific shell) and (b) more secure (because you can specify what shell is to be used as syntax is not unified over different shells, e.g. csh vs. bash, and usage on other shells may lead to undefined or unwanted behaviour):
from subprocess import call
call(["bash", "-c", "while ! echo exit | nc 10.0.2.11 9445; do sleep 10; done"])
The exact argument to the shell to execute a command (here -c) depends on your shell.
You may want to have a look at the subprocess docs, especially for other ways of invoking processes. See e.g. check_call as a way of checking the return code for success, check_output to get the standard output of the process and Popen for advanced input/output interaction with the process.
Alternatively, you could use os.system, which implicitly launches a shell and returns the return code (subprocess.check_call with shell=True is a more flexible alternative to this)
* This link is to the Python 2 docs instead of the Python 3 docs used otherwise because it better outlines the security problems
I may not at all understand this correctly, but I am trying to allow a Python program to interface with a subprocess that runs commands as if on a Linux shell.
For example, I want to be able to run "cd /" and then "pwd later in the program and get "/".
I am currently trying to use subprocess.Popen and the communicate() method to send and receive data. The first command, sent with the Popen constructor, runs fine and gives proper output. But I cannot send another command via communicate(input="pwd").
My code so far:
from subprocess i
term=Popen("pwd", stdout=PIPE, stdin=PIPE)
print(flush(term.communicate()))
term.communicate(input="cd /")
print(flush(term.communicate(input="pwd")))
Is there a better way to do this? Thanks.
Also, I am running Python 3.
First of all, you need to understand that running a shell command and running a program aren't the same thing.
Let me give you an example:
>>> import subprocess
>>> subprocess.call(['/bin/echo', '$HOME'])
$HOME
0
>>> subprocess.call(['/bin/echo $HOME'], shell=True)
/home/kkinder
0
Notice that without the shell=True parameter, the text of $HOME is not expanded. That's because the /bin/echo program doesn't parse $HOME, Bash does. What's really happening in the second call is something analogous to this:
>>> subprocess.call(['/bin/bash', '-c', '/bin/echo $HOME'])
/home/kkinder
0
Using the shell=True parameter basically says to the subprocess module, go interpret this text using a shell.
So, you could add shell=True, but then the problem is that once the command finishes, its state is lost. Each application in the stack has its own working directory. So what the directory is will be something like this:
bash - /foo/bar
python - /foo
bash via subprocess - /
After your command executes, the python process's path stays the same and the subprocess's path is discarded once the shell finishes your command.
Basically, what you're asking for isn't practical. What you would need to do is, open a pipe to Bash, interactively feed it commands your user types, then read the output in a non-blocking way. That's going to involve a complicated pipe, threads, etc. Are you sure there's not a better way?