What is the difference between subprocess.Popen() and os.system()?
If you check out the subprocess section of the Python docs, you'll notice there is an example of how to replace os.system() with subprocess.Popen():
sts = os.system("mycmd" + " myarg")
...does the same thing as...
sts = Popen("mycmd" + " myarg", shell=True).wait()
The "improved" code looks more complicated, but it's better because once you know subprocess.Popen(), you don't need anything else. subprocess.Popen() replaces several other tools (os.system() is just one of those) that were scattered throughout three other Python modules.
If it helps, think of subprocess.Popen() as a very flexible os.system().
subprocess.Popen() is strict super-set of os.system().
os.system is equivalent to Unix system command, while subprocess was a helper module created to provide many of the facilities provided by the Popen commands with an easier and controllable interface. Those were designed similar to the Unix Popen command.
system() executes a command specified in command by calling /bin/sh -c command, and returns after the command has been completed
Whereas:
The popen() function opens a process by creating a pipe, forking, and
invoking the shell.
If you are thinking which one to use, then use subprocess definitely because you have all the facilities for execution, plus additional control over the process.
Subprocess is based on popen2, and as such has a number of advantages - there's a full list in the PEP here, but some are:
using pipe in the shell
better newline support
better handling of exceptions
When running python (cpython) on windows the <built-in function system> os.system will execute under the curtains _wsystem while if you're using a non-windows os, it'll use system.
On contrary, Popen should use CreateProcess on windows and _posixsubprocess.fork_exec in posix-based operating-systems.
That said, an important piece of advice comes from os.system docs, which says:
The subprocess module provides more powerful facilities for spawning
new processes and retrieving their results; using that module is
preferable to using this function. See the Replacing Older Functions
with the subprocess Module section in the subprocess documentation for
some helpful recipes.
Related
When trying to write script with python, I have a fundamental hole of knowledge.
Update: Thanks to the answers I corrected the word shell to process/subprocess
Nomenclature
Starting with a Bash prompt, lets call this BASH_PROCESS
Then within BASH_PROCESS I run python3 foo.py, the python script runs in say PYTHON_SUBPROCESS
Within foo.py is a call to subprocess.run(...), this subprocess command runs in say `SUBPROCESS_SUBPROCESS
Within foo.py is subprocess.run(..., shell=True), this subprocess command runs in say SUBPROCESS_SUBPROCESS=True
Test for if a process/subprocess is equal
Say SUBPROCESS_A starts SUBPROCESS_B. In the below questions, when I say is SUBPROCESS_A == SUBPROCESS_B, what I means is if SUBPROCESS_B sets an env variable, when it runs to completion, will they env variable be set in SUBPROCESS_A? If one runs eval "$(ssh-agent -s)" in SUBPROCESS_B, will SUBPROCESS_A now have an ssh agent too?
Question
Using the above nomenclature and equality tests
Is BASH_PROCESS == PYTHON_SUBPROCESS?
Is PYTHON_SUBPROCESS == SUBPROCESS_SUBPROCESS?
Is PYTHON_SUBPROCESS == SUBPROCESS_SUBPROCESS=True?
If SUBPROCESS_SUBPROCESS=True is not equal to BASH_PROCESS, then how does one alter the executing environment (e.g. eval "$(ssh-agent -s)") so that a python script can set up the env for the calller?
You seem to be confusing several concepts here.
TLDR No, there is no way for a subprocess to change its parent's environment. See also Global environment variables in a shell script
You really don't seem to be asking about "shells".
Instead, these are subprocesses; if you run python foo.py in a shell, the Python process is a subprocess of the shell process. (Many shells let you exec python foo.py which replaces the shell process with a Python process; this process is now a subprocess of whichever process started the shell. On Unix-like systems, ultimately all processes are descendants of process 1, the init process.)
subprocess runs a subprocess, simply. If shell=True then the immediate subprocess of Python is the shell, and the command(s) you run are subprocesses of that shell. The shell will be the default shell (cmd on Windows, /bin/sh on Unix-like systems) though you can explicitly override this with e.g. executable="/bin/bash"
Examples:
subprocess.Popen(['printf', '%s\n', 'foo', 'bar'])
Python is the parent process, printf is a subprocess whose parent is the Python process.
subprocess.Popen(r"printf '%s\n' foo bar", shell=True)
Python is the parent process of /bin/sh, which in turn is the parent process of printf. When printf terminates, so does sh, as it has reached the end of its script.
Perhaps notice that the shell takes care of parsing the command line and splitting it up into the four tokens we ended up explicitly passing directly to Popen in the previous example.
The commands you run have access to shell features like wildcard expansion, pipes, redirection, quoting, variable expansion, background processing, etc.
In this isolated example, none of those are used, so you are basically adding an unnecessary process. (Maybe use shlex.split() if you want to avoid the minor burden of splitting up the command into tokens.) See also Actual meaning of 'shell=True' in subprocess
subprocess.Popen(r"printf '%s\n' foo bar", shell=True, executable="/bin/bash")
Python is the parent process of Bash, which in turn is the parent process of printf. Except for the name of the shell, this is identical to the previous example.
There are situations where you really need the slower and more memory-hungry Bash shell, when the commands you want to execute require features which are available in Bash, but not in the Bourne shell. In general, a better solution is nearly always to run as little code as possible in a subprocess, and instead replace those Bash commands with native Python constructs; but if you know what you are doing (or really don't know what you are doing, but need to get the job done rather than solve the problem properly), the facility can be useful.
(Separately, you should probably avoid bare Popen when you can, as explained in the subprocess documentation.)
Subprocesses inherit the environment of their parent when they are started. On Unix-like systems, there is no way for a process to change its parent's environment (though the parent may participate in making that possible, as in your eval example).
To perhaps accomplish what you may ultimately be asking about, you can set up an environment within Python and then start your other command as a subprocess, perhaps then with an explicit env= keyword argument to point to the environment you want it to use:
import os
...
env = os.environ.copy()
env["PATH"] = "/opt/foo:" + env["PATH"]
del env["PAGER"]
env["secret_cookie"] = "xyzzy"
subprocess.Popen(["otherprogram"], env=env)
or have Python print out values in a form which can safely be passed to eval in the Bourne shell. (Caution: this requires you to understand the perils of eval in general and the target shell's quoting conventions in particular; also, you will perhaps need to support the syntax of more than one shell, unless you are only targeting a very limited audience.)
... Though in many situations, the simplest solution by far is to set up the environment in the shell, then run Python as a subprocess of that shell instance (or exec python if you want to get rid of the shell instance after it has performed its part; see also What are the uses of the exec command in shell scripts?)
Python without an argument starts the Python REPL, which could be regarded as a "shell", though we would commonly not use that term (perhaps instead call it "interactive interpreter" - see also below); but python foo.py simply runs the script foo.py and exits, so there is no shell there.
The definition of "shell" is slightly context-dependent, but you don't really seem to be asking about shells here. (Some GUIs have a concept of "graphical shell" etc but we are already out of the scope of what you were trying to ask about.) Some programs are command interpreters (the Python executable interprets and executes commands in the Python language; the Bourne shell interprets and executes shell scripts) but generally only those whose primary purposes include running other programs are called "shells".
None of those equalities are true, and half of those "shells" aren't actually shells.
Your bash shell is a shell. When you launch your Python script from that shell, the Python process that runs the script is a child process of the bash shell process. When you launch a subprocess from the Python script, that subprocess is a child process of the Python process. If you launch the subprocess with shell=True, Python invokes a shell to parse and run the command, but otherwise, no shell is involved in running the subprocess.
Child processes inherit environment variables from their parent on startup (unless you take specific steps to avoid that), but they cannot set environment variables for their parent. You cannot run a Python script to set environment variables in your shell, or run a subprocess from Python to set your Python script's environment variables.
I know you can run Linux terminal commands through Python scripts using subprocess
subprocess.call(['ls', '-l']) # for linux
But I can't find a way to do the same thing on windows
subprocess.call(['dir']) # for windows
is it possible using Python without heavy tinkering?
Should I stick to good old fashioned batch files?
dir is not a file, it is an internal command, so the shell keyword must be set to True.
subprocess.call(["dir"], shell=True)
Try this
import os
os.system("windows command")
ex: for date
os.system("date")
Almost everyone's answers are right but it seems I can do what I need using os.popen -- varStr = os.popen('dir /b *.py').read()
First of all, to get a directory listing, you should rather use os.listdir(). If you invoke dir instead, you'll have to parse its output to make any use of it, which is lots of unnecessary work and is error-prone.
Now,
dir is a cmd.exe built-in command, it's not a standalone executable. cmd.exe itself is the executable that implements it.
So, you have two options (use check_output instead of check_call if you need to get the output instead of just printing it):
use cmd's /C switch (execute a command and quit):
subprocess.check_call(['cmd','/c','dir','/s'])
use shell=True Popen() option (execute command line through the system shell):
subprocess.check_call('dir /s', shell=True)
The first way is the recommended one. That's because:
In the 2nd case, cmd, will do any shell transformations that it normally would (e.g. splitting the line into arguments, unquoting, environment variable expansion etc). So, your arguments may suddenly become something else and potentially harmful. In particular, if they happen to contain any spaces and cmd special characters and/or keywords.
shell=True uses the "default system shell" (pointed to via COMSPEC environment variable in the case of Windows), so if the user has redefined it, your program will behave unexpectedly.
I need to give Gdb commands after I have started running a shell script which invokes gdb and halts to the Gdb prompt. So, to load and execute the image (.elf) file I invoke the following subprocess:
import subprocess
os.chdir(r"/project/neptune_psv/fw/")
print os.getcwd()
proc = subprocess.Popen('./Execute.sh -i TestList_new.in -m 135.20.230.160 -c mpu',shell = True,stdin = subprocess.PIPE)
After Execute.sh halts to the Gdb prompt I need to give two Gdb commands:
Set *0x44880810 = 3 (Set a register value)
Continue
Can anyone help me how to give these two commands through stdin ?
I think the simplest way is to use something like pexpect, an expect clone in Python. This provides a way to control an otherwise interactive process programmatically.
However, since you're specifically using gdb, note that this is not the best approach to driving gdb. There are two better ways. First, you can program gdb directly using its built-in Python API. This is the best way, if it is possible for your use case, as it is a lot simpler than trying to parse the output. However, if you must parse the output, then you should investigate "MI". This is gdb's "Machine Interface", which presents a vaguely JSON-like (it predates the real JSON) way to control gdb. There are MI parsing libraries available, though offhand I don't recall if there is one written in Python.
I have successfully run several Python scripts, calling them from a base script using the subprocess module:
subprocess.popen([sys.executable, 'script.py'], shell=True)
However, each of these scripts executes some simulations (.exe files from a C++ application) that generate some output to the shell. All these outputs are written to the base shell from where I've launched those scripts. I'd like to generate a new shell for each script. I've tried to generate new shells using the shell=True attribute when calling subprocess.call (also tried with popen), but it doesn't work.
How do I get a new shell for each process generated with the subprocess.call?
I was reading the documentation about stdin and stdout as suggested by Spencer and found a flag the solved the problem: subprocess.CREATE_NEW_CONSOLE. Maybe redirecting the pipes does the job too, but this seems to be the simplest solution (at least for this specific problem). I've just tested it and worked perfectly:
subprocess.popen([sys.executable, 'script.py'], creationflags = subprocess.CREATE_NEW_CONSOLE)
To open in a different console, do (tested on Windows 7 / Python 3):
from sys import executable
from subprocess import Popen, CREATE_NEW_CONSOLE
Popen([executable, 'script.py'], creationflags=CREATE_NEW_CONSOLE)
input('Enter to exit from this launcher script...')
Popen already generates a sub process to handle things. You just need to redirect the output pipes. Look at the subprocess documentation, specifically the section on popen stdin, stdout and stderr redirection.
If you don't redirect these pipes, it inherits them from the parent. Just be careful about deadlocking your processes.
You wanted additional windows for each subprocess. This is handled as well. Look at the startupinfo section of subprocess. It explains what options to set on windows to spawn a new terminal for each subprocess. Note that it requires the use of the shell=True option.
This doesn't actually answer your question. But I've had my problems with subprocess too, and pexpect turned out to be really helpful.
I was wondering if it was possible to write a GUI in python, and then somewhere in the python script, insert a script switch to temporarily change the language to accomodate for the batch snippet.
I know this can be done in html and vbscript but what about Python?
You can control other processes, written with any language, including bash using the subprocess module.
The subprocess module is the most powerful and complete method for executing other processes. However, there's also a very simple method using the os module: os.system(command) runs command just as if you were to type it into a command line.