I know you can run Linux terminal commands through Python scripts using subprocess
subprocess.call(['ls', '-l']) # for linux
But I can't find a way to do the same thing on windows
subprocess.call(['dir']) # for windows
is it possible using Python without heavy tinkering?
Should I stick to good old fashioned batch files?
dir is not a file, it is an internal command, so the shell keyword must be set to True.
subprocess.call(["dir"], shell=True)
Try this
import os
os.system("windows command")
ex: for date
os.system("date")
Almost everyone's answers are right but it seems I can do what I need using os.popen -- varStr = os.popen('dir /b *.py').read()
First of all, to get a directory listing, you should rather use os.listdir(). If you invoke dir instead, you'll have to parse its output to make any use of it, which is lots of unnecessary work and is error-prone.
Now,
dir is a cmd.exe built-in command, it's not a standalone executable. cmd.exe itself is the executable that implements it.
So, you have two options (use check_output instead of check_call if you need to get the output instead of just printing it):
use cmd's /C switch (execute a command and quit):
subprocess.check_call(['cmd','/c','dir','/s'])
use shell=True Popen() option (execute command line through the system shell):
subprocess.check_call('dir /s', shell=True)
The first way is the recommended one. That's because:
In the 2nd case, cmd, will do any shell transformations that it normally would (e.g. splitting the line into arguments, unquoting, environment variable expansion etc). So, your arguments may suddenly become something else and potentially harmful. In particular, if they happen to contain any spaces and cmd special characters and/or keywords.
shell=True uses the "default system shell" (pointed to via COMSPEC environment variable in the case of Windows), so if the user has redefined it, your program will behave unexpectedly.
Related
When trying to write script with python, I have a fundamental hole of knowledge.
Update: Thanks to the answers I corrected the word shell to process/subprocess
Nomenclature
Starting with a Bash prompt, lets call this BASH_PROCESS
Then within BASH_PROCESS I run python3 foo.py, the python script runs in say PYTHON_SUBPROCESS
Within foo.py is a call to subprocess.run(...), this subprocess command runs in say `SUBPROCESS_SUBPROCESS
Within foo.py is subprocess.run(..., shell=True), this subprocess command runs in say SUBPROCESS_SUBPROCESS=True
Test for if a process/subprocess is equal
Say SUBPROCESS_A starts SUBPROCESS_B. In the below questions, when I say is SUBPROCESS_A == SUBPROCESS_B, what I means is if SUBPROCESS_B sets an env variable, when it runs to completion, will they env variable be set in SUBPROCESS_A? If one runs eval "$(ssh-agent -s)" in SUBPROCESS_B, will SUBPROCESS_A now have an ssh agent too?
Question
Using the above nomenclature and equality tests
Is BASH_PROCESS == PYTHON_SUBPROCESS?
Is PYTHON_SUBPROCESS == SUBPROCESS_SUBPROCESS?
Is PYTHON_SUBPROCESS == SUBPROCESS_SUBPROCESS=True?
If SUBPROCESS_SUBPROCESS=True is not equal to BASH_PROCESS, then how does one alter the executing environment (e.g. eval "$(ssh-agent -s)") so that a python script can set up the env for the calller?
You seem to be confusing several concepts here.
TLDR No, there is no way for a subprocess to change its parent's environment. See also Global environment variables in a shell script
You really don't seem to be asking about "shells".
Instead, these are subprocesses; if you run python foo.py in a shell, the Python process is a subprocess of the shell process. (Many shells let you exec python foo.py which replaces the shell process with a Python process; this process is now a subprocess of whichever process started the shell. On Unix-like systems, ultimately all processes are descendants of process 1, the init process.)
subprocess runs a subprocess, simply. If shell=True then the immediate subprocess of Python is the shell, and the command(s) you run are subprocesses of that shell. The shell will be the default shell (cmd on Windows, /bin/sh on Unix-like systems) though you can explicitly override this with e.g. executable="/bin/bash"
Examples:
subprocess.Popen(['printf', '%s\n', 'foo', 'bar'])
Python is the parent process, printf is a subprocess whose parent is the Python process.
subprocess.Popen(r"printf '%s\n' foo bar", shell=True)
Python is the parent process of /bin/sh, which in turn is the parent process of printf. When printf terminates, so does sh, as it has reached the end of its script.
Perhaps notice that the shell takes care of parsing the command line and splitting it up into the four tokens we ended up explicitly passing directly to Popen in the previous example.
The commands you run have access to shell features like wildcard expansion, pipes, redirection, quoting, variable expansion, background processing, etc.
In this isolated example, none of those are used, so you are basically adding an unnecessary process. (Maybe use shlex.split() if you want to avoid the minor burden of splitting up the command into tokens.) See also Actual meaning of 'shell=True' in subprocess
subprocess.Popen(r"printf '%s\n' foo bar", shell=True, executable="/bin/bash")
Python is the parent process of Bash, which in turn is the parent process of printf. Except for the name of the shell, this is identical to the previous example.
There are situations where you really need the slower and more memory-hungry Bash shell, when the commands you want to execute require features which are available in Bash, but not in the Bourne shell. In general, a better solution is nearly always to run as little code as possible in a subprocess, and instead replace those Bash commands with native Python constructs; but if you know what you are doing (or really don't know what you are doing, but need to get the job done rather than solve the problem properly), the facility can be useful.
(Separately, you should probably avoid bare Popen when you can, as explained in the subprocess documentation.)
Subprocesses inherit the environment of their parent when they are started. On Unix-like systems, there is no way for a process to change its parent's environment (though the parent may participate in making that possible, as in your eval example).
To perhaps accomplish what you may ultimately be asking about, you can set up an environment within Python and then start your other command as a subprocess, perhaps then with an explicit env= keyword argument to point to the environment you want it to use:
import os
...
env = os.environ.copy()
env["PATH"] = "/opt/foo:" + env["PATH"]
del env["PAGER"]
env["secret_cookie"] = "xyzzy"
subprocess.Popen(["otherprogram"], env=env)
or have Python print out values in a form which can safely be passed to eval in the Bourne shell. (Caution: this requires you to understand the perils of eval in general and the target shell's quoting conventions in particular; also, you will perhaps need to support the syntax of more than one shell, unless you are only targeting a very limited audience.)
... Though in many situations, the simplest solution by far is to set up the environment in the shell, then run Python as a subprocess of that shell instance (or exec python if you want to get rid of the shell instance after it has performed its part; see also What are the uses of the exec command in shell scripts?)
Python without an argument starts the Python REPL, which could be regarded as a "shell", though we would commonly not use that term (perhaps instead call it "interactive interpreter" - see also below); but python foo.py simply runs the script foo.py and exits, so there is no shell there.
The definition of "shell" is slightly context-dependent, but you don't really seem to be asking about shells here. (Some GUIs have a concept of "graphical shell" etc but we are already out of the scope of what you were trying to ask about.) Some programs are command interpreters (the Python executable interprets and executes commands in the Python language; the Bourne shell interprets and executes shell scripts) but generally only those whose primary purposes include running other programs are called "shells".
None of those equalities are true, and half of those "shells" aren't actually shells.
Your bash shell is a shell. When you launch your Python script from that shell, the Python process that runs the script is a child process of the bash shell process. When you launch a subprocess from the Python script, that subprocess is a child process of the Python process. If you launch the subprocess with shell=True, Python invokes a shell to parse and run the command, but otherwise, no shell is involved in running the subprocess.
Child processes inherit environment variables from their parent on startup (unless you take specific steps to avoid that), but they cannot set environment variables for their parent. You cannot run a Python script to set environment variables in your shell, or run a subprocess from Python to set your Python script's environment variables.
I guess I have to use either use os.system or subprocess.call, but I can't figure out how to use it.
I don't have the permission to edit the original folder.
subprocess.Popen('file.py', cwd=dirName) gives me 'The system cannot find the file specified' even though the file clearly exists
If I was typing in cmd,
cd directory
file.py -arg
Edit: Just to be clear I want to run another script using a python script
As you have tagged the question with cmd, I assume that you use Windows. Windows is kind enough to automatically use the appropriate command when you type a document name in cmd, but Python subprocess is not. So you have 2 possible ways here
use shell=True to ask a cmd interpretor to execute the command:
subprocess.Popen('file.py', cwd=dirName, shell=True)
pass explicitely the path of the Python interpretor (or the name if it is in the path)
subprocess.Popen([python_path, 'file.py'], cwd=dirName, shell=True)
At first you need to add the python in windows environment.
Then you can add the file path (the file that you want to run it on command line).
Then you can go to command line page and type the file name and use it like the line below:
FileName commands
for example :
pip install datetime
Its clear you are probably using python 3 rather than python 2. Otherwise you might use os.system as already suggested or commands. Commands is now obsolete as of python 3. To get this working I would use instead:
statusAndOutputText = subprocess.getstatusoutput( os.path.join( dirName, 'file.py' ) )
This will definitely work. I've used it many times in python 3 and it will give you the status in statusAndOutputText[0] and output buffer in statusAndOutputText[1] which are both very useful to have.
Could you help to check what's going on with subprocess, it performs differently on different machines with same Python version, but one is on Ubuntu docker and one is on Windows.
Ubuntu docker
I use subprocess to execute an external Python script with parameter shell=True, actually it opens a new process for me without executing the specified script, so I have to remove the parameter shell=True and then everything works as expected.
You can see from the screenshot below, I need to exit() after executing the first subprocess, and ran the second subprocess without shell=True.
Windows
In Windows, shell=True works same as I execute subprocess in Ubuntu without shell=True parameter.
Quoting https://docs.python.org/3/library/subprocess.html#popen-constructor:
On POSIX with shell=True, the shell defaults to /bin/sh. If args is a
string, the string specifies the command to execute through the shell.
This means that the string must be formatted exactly as it would be
when typed at the shell prompt. This includes, for example, quoting or
backslash escaping filenames with spaces in them. If args is a
sequence, the first item specifies the command string, and any
additional items will be treated as additional arguments to the shell
itself.
(emphasis mine)
That means, in your first example with run(['python', 'script.py'], shell=True) you are actually only starting an interactive Python session and not passing the script to the interpreter.
Further:
The only time you need to specify shell=True on Windows is when the
command you wish to execute is built into the shell (e.g. dir or
copy). You do not need shell=True to run a batch file or console-based
executable.
Conclusion: Whenever possible, pass the arguments as a list (as you did), but do not use shell=True.
I am trying to write a python script that will execute a bash script I have on my Windows machine. Up until now I have been using the Cygwin terminal so executing the bash script RunModels.scr has been as easy as ./RunModels.scr. Now I want to be able to utilize subprocess of Python, but because Windows doesn't have the built in functionality to handle bash I'm not sure what to do.
I am trying to emulate ./RunModels.scr < validationInput > validationOutput
I originally wrote this:
os.chdir(atm)
vin = open("validationInput", 'r')
vout = open("validationOutput", 'w')
subprocess.call(['./RunModels.scr'], stdin=vin, stdout=vout, shell=True)
vin.close()
vout.close()
os.chdir(home)
But after spending a while trying to figure out why my access was denied, I realized my issue wasn't the file permissions but the fact that I was trying to execute a bash file on Windows in general. Can someone please explain how to execute a bash script with directed input/output on windows using a python script?
Edit (Follow up Question):
Thanks for the responses, I needed the full path to my bash.exe as the first param. Now however, command line calls from within RunModels.scr come back in the python output as command not found. For example, ls, cp, make. Any suggestions for this?
Follow up #2:
I updated my call to this:
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
The error I now get is /usr/bin/bash: RunModels.scr: No such file or directory.
Using cwd does not seem to have any effect on this error, either way the subprocess is looking in /usr/bin/bash for RunModels.scr.
SELF-ANSWERED
I needed to specify the path to RunModels.scr in the call as well as using cwd.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'C:\\path\\dir_where_RunModels\\RunModels.scr'], stdin=vin, stdout=vout, cwd='C:\\path\\dir_where_RunModels\\')
But another problem...
Regardless of specifying cwd, the commands executed by RunModels.scr are throwing errors as if RunModels.scr is in the wrong directory. The script executes, but cp and cd throw the error no such file or directory. If I navigate to where RunModels.scr is through the command line and execute it the old fashioned way I don't get these errors.
Python 3.4 and below
Just put bash.exe in first place in your list of subprocess.call arguments. You can remove shell=True, that's not necessary in this case.
subprocess.call(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Depending on how bash is installed (is it in the PATH or not), you might have to use the full path to the bash executable.
Python 3.5 and above
subprocess.call() has been effectively replaced by subprocess.run().
subprocess.run(['C:\\cygwin64\\bin\\bash.exe', '-l', 'RunModels.scr'],
stdin=vin, stdout=vout,
cwd='C:\\path\\dir_where_RunModels\\')
Edit:
With regard to the second question, you might need to add the -l option to the shell invocation to make sure it reads all the restart command files like /etc/profile. I presume these files contain settings for the $PATH in bash.
Edit 2:
Add something like pwd to the beginning of RunModels.scr so you can verify that you are really in the right directory. Check that there is no cd command in the rc-files!
Edit 3:
The error /usr/bin/bash: RunModels.scr: No such file or directory can also be generated if bash cannot find one of the commands that are called in the script. Try adding the -v option to bash to see if that gives more info.
A better solution than the accepted answer is to use the executable keyword argument to specify the path to your shell. Behind the curtain, Python does something like
exec([executable, '-c`, subprocess_arg_string])
So, concretely, in this case,
subprocess.call(
'./RunModels.scr',
stdin=vin, stdout=vout,
shell=True,
executable="C:/cygwin64/bin/bash.exe")
(Windows thankfully lets you use forward slashes instead of backslashes, so you can avoid the requirement to double the backslashes or use a raw string.)
I use TextMate to debug python script, as I like the feature of using 'Command-R' for running python from TextMate, and I learned that emacs provide similar feature.
I need to know if the python is run from command line or from TextMate/emacs. How can I do that?
ADDED
I use TextMate for python coding/debugging, and it's pretty useful. But, sometimes I need to run the test using command line. I normally turn on debugging/logging mode with TextMate, and off with command line mode. This is the reason I asked the question. Also, I plan to use emacs for python debugging, so I wanted to ask the case for emacs.
I got an answer in the case with emacs, and I happen to solve this issue with TextMate.
Set variables in Preferences -> Advanced -> Shell Variables, and I found that TM_ORGANIZATION_NAME is already there to be used. So, I'll just use this variable.
Use this variable, if os.environ['TM_ORGANIZATION_NAME']: return True
I guess the shell variable from TextMate disappear when I'm done using it.
For Emacs: If python is run as an inferior process, then the environment variable INSIDE_EMACS will be set.
From docs:
Emacs sets the environment variable
INSIDE_EMACS in the subshell to a
comma-separated list including the
Emacs version. Programs can check this
variable to determine whether they are
running inside an Emacs subshell.
sys.argv will tell you how Python was invoked. I don't know about TextMate, but when I tell Emacs to eval buffer, its value is ['-c']. That means it's executing a specified command, according to the man page. If Python's run directly from the command line with no parameters, sys.argv will be []. If you run a python script, it will have the script name and whatever arguments you pass it. You might want to set up your python-mode in Emacs and whatever the equivalent in TextMate is to put something special like -t in the command line.
That's pretty hackish though. Maybe there's a better way.
From the docs for sys.path:
As initialized upon program startup,
the first item of this list, path[0],
is the directory containing the script
that was used to invoke the Python
interpreter. If the script directory
is not available (e.g. if the
interpreter is invoked interactively
or if the script is read from standard
input), path[0] is the empty string,
which directs Python to search modules
in the current directory first. Notice
that the script directory is inserted
before the entries inserted as a
result of PYTHONPATH.
So
if sys.path[0]:
# python was run interactively
else:
# python is running a script.
Or, for example, from the IPython prompt (inside Emacs):
In [65]: sys.path
Out[65]:
['', <-------------------- first entry is empty string
'/usr/bin',
'/usr/local/lib/python2.6/dist-packages/scikits.statsmodels-0.2.0-py2.6.egg',
'/usr/local/lib/python2.6/dist-packages/pyinterval-1.0b21-py2.6-linux-i686.egg',
... ]
Use Command-R to run the script directly
Use Shift-Command-R to run the script from terminal.