I am trying to use Python's Popen to change my working directory and execute a command.
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
However, powershell returns "The system cannot find the path specified." The path does exist.
If I run
pg = subprocess.Popen("cd c:/mydirectory ;", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
it returns the same thing.
However, if i run this: (without the semicolon)
pg = subprocess.Popen("cd c:/mydirectory",stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
The command returns without an error. This leads me to believe that the semicolon is issue. What is the cause for this behavior and how can I get around it?
I know I can just do c:/mydirectory/runExecutable.exe --help, but I would like to know why this is happening.
UPDATE :
I have tested passing the path to powershell as the argument for Popen's executable parameter. Just powershell.exe may not be enough. To find the true absolute path of powershell, execute where.exe powershell. Then you can pass it into Popen. Note that shell is still true. It will use the default shell but pass the command to powershell.exe
powershell = C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, executable=powershell)
buff,buffErr = pg.communicate()
//It works!
In your subprocess.Popen() call, shell=True means that the platform's default shell should be used.
While the Windows world is - commendably - moving from CMD (cmd.exe) to PowerShell, Python determines what shell to invoke based on the COMSPEC environment variable, which still points to cmd.exe, even in the latest W10 update that has moved toward PowerShell in terms of what the GUI offers as the default shell.
For backward compatibility, this will not change anytime soon, and will possibly never change.
Therefore, your choices are:
Use cmd syntax, as suggested in Maurice Meyer's answer.
Do not use shell = True and invoke powershell.exe explicitly - see below.
Windows only: Redefine environment variable COMSPEC before using shell = True - see below.
A simple Python example of how to invoke the powershell binary directly, with command-line switches followed by a single string containing the PowerShell source code to execute:
import subprocess
args = 'powershell', '-noprofile', '-command', 'set-location /; $pwd'
subprocess.Popen(args)
Note that I've deliberately used powershell instead of powershell.exe, because that opens up the possibility of the command working on Unix platforms too, once PowerShell Core is released.
Windows only: An example with shell = True, after redefining environment variable COMSPEC to point to PowerShell first:
import os, subprocess
os.environ["COMSPEC"] = 'powershell'
subprocess.Popen('Set-Location /; $pwd', shell=True)
Note:
COMSPEC is only consulted on Windows; on Unix platforms, the shell executable is invariably /bin/sh
As of Windows PowerShell v5.1 / PowerShell Core v6-beta.3, invoking powershell with just -c (interpreted as -Command) still loads the profiles files by default, which can have unexpected side effects (with the explicit invocation of powershell used above, -noprofile suppresses that).
Changing the default behavior to not loading the profiles is the subject of this GitHub issue, in an effort to align PowerShell's CLI with that of POSIX-like shells.
You can concat multiple commands using '&' character instead of a semicolon. Try this:
pg = subprocess.Popen("cd c:/mydirectory & ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
Related
I am using an anaconda environment both for the python code and the terminal.
When I want to execute a program in the shell (Windows CMD) with the environment activated. The program ogr2ogr returns the correct output with the given parameter. The tool ogr2ogr has been installed via a conda package.
But when I execute the my python code, the ogr2ogr returns an errors output. I thought it might be to different installations used due to usage of different environments (without my knowledge), but this is ownly a guess.
The python code goes as follows:
from pathlib import Path
from subprocess import check_call, STDOUT
...
file_path = Path(file_name)
destination = str(file_path.with_suffix(".gpkg"))
command = f"ogr2ogr -f GPKG -s_srs EPSG:25833 -t_srs EPSG:25833 {destination} GMLAS:{file_name} -oo REMOVE_UNUSED_LAYERS=YES"
check_call(command, stderr=STDOUT, shell=True)
ogr2ogr translates a file into another format. Which is also done, but when I open the file, I see, it's not done 100 % correctly.
When I copy the value of the string command and copy it to the shell and execute the command the execution is done correctly!
How can I correct the behaviour of using subprocess.check_call
I have been working on an issue that requires a Python script to run via the PowerShell command line. The script should pass the command to the command line and save the output. However, I'm running into an issue where some command line arguments are not recognized.
import subprocess
try:
output = subprocess.check_output\
(["Write-Output 'Hello world'"], shell=True)
# (["dir"], shell=True)
except subprocess.CalledProcessError as e:
print(e.output)
print('^Error Output^')
If I use the current command with the check_output command, I get an error stating that:
'"Write-Output 'Hello world'"' is not recognized as an internal or external command,
operable program or batch file.
If I just use the "dir" line, the script runs just fine. I'm at odds here as to why this would be happening. This is not the exact script that I'm running, but it produces the same problem on my machine. If I just type the problem command into the command line, it would output "Hello world" onto the new line just as expected.
Any insight as to why this would be happening would be greatly appreciated. If it's of relevance, I would like to not use any sort of admin privilege workaround.
I believe this is because in Windows your default Shell is not PowerShell, you could Execute a Powershell command, calling the executable by executing Powershell with the arguments you need.
For Example
POWERSHELL_COMMAND = r'C:\WINDOWS\system32\WindowsPowerShell\v1.0\powershell.exe'
subprocess.Popen([POWERSHELL_COMMAND,
'-ExecutionPolicy', 'Unrestricted',
'Write-Output', 'Hello World'],
stdout = subprocess.PIPE,
stderr = subprocess.PIPE)
if powershell is not in path you could use the full path for the executable
or if it's in path you could use just POWERSHELL_COMMAND = "powershell" as command, becareful, with the backslashed windows paths, to avoid errors you could use raw strings.
To verify that you have powershell in path, you could go to the configurations and check, or you could just open a cmd and type powershell and if It works, then you could assume that powershell is in path.
From the docs:
On Windows with shell=True, the COMSPEC environment variable specifies the default shell.
So set COMSPEC=powershell allows to make shell=True use powershell as default instead of cmd
I need to use the wget in a Python script with the subprocess.call function, but it seems the "wget" command cannot be identified by the bash subprocess opened by python.
I have added the environment variable (the path where wget is):
export PATH=/usr/local/bin:$PATH
to the ~/.bashrc file and the ~/.bash_profile file on my mac and guaranteed to have sourced them.
And the python script looks like:
import subprocess as sp
cmd = 'wget'
process = sp.Popen(cmd ,stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE, shell=True ,executable='/bin/bash')
(stdoutdata, stderrdata) = process.communicate()
print stdoutdata, stderrdata
The expected output should be like
wget: missing URL
Usage: wget [OPTION]... [URL]...
But the result is always
/bin/bash: wget: command not found
Interestingly I can get the help output if I type in wget directly in a bash terminal, but it never works in the python script. How could it be?
PS:
If I change the command to
cmd = '/usr/local/bin/wget'
then it works. So I am sure I got wget installed.
You can pass an env= argument to the subprocess functions.
import os
myenv = os.environ.copy
myenv['PATH'] = '/usr/local/bin:' + myenv['PATH']
subprocess.run(..., env=myenv)
However, you probably want to avoid running a shell at all, and instead augment the PATH that Python uses to find the binary to run in the subprocess call.
import subprocess as sp
import os
os.environ['PATH'] = '/usr/local/bin:' + os.environ['PATH']
cmd = 'wget'
# use run instead of Popen
# don't needlessly use a shell
# and thus put [cmd] as a list
process = sp.run([cmd], stdout=sp.PIPE, stdin=sp.PIPE,
stderr=sp.PIPE,
universal_newlines=True)
print(process.stdout, process.stderr)
Running Bash commands in Python explains the changes I made in more detail.
However, there is no good reason to use an external utility for this; Python requests does pretty everything wget does, often more naturally and with more control over what exactly it does.
Typing the command in my ubuntu terminal recognizes the parameter t in my command:
/home/daniel/Downloads/SALOME-7.6.0-UB14.04/salome start -t
What is the difference when starting the same process in python via Popen?
command ='/home/daniel/Downloads/SALOME-7.6.0-UB14.04/salome'
commandargs= 'start -t'
import subprocess
subprocess.Popen([command, commandargs], shell=True).wait()
My parameter stands for terminal mode but running my application (salome) via python Popen opens the GUI.
See: https://docs.python.org/3.5/library/subprocess.html#subprocess.Popen
args should be a sequence of program arguments or else a single string.
See also: https://stackoverflow.com/a/11309864/2776376
subprocess.Popen([command, commandargs.split(' ')], shell=True).wait()
alternatively you could do, although less recommended:
subprocess.Popen(command + commandargs, shell=True).wait()
should do the trick
I'm trying to run the command from a python file (2.7):
p=subprocess.Popen("sha256sum file1.zip >> file2.sha")
But i got an error that file '>>' does not exist.
I tried:
p=subprocess.Popen("sha256sum file1.zip >> file2.sha".split())
But still the >> is a problem.
Of course that if I run the command in the prompt line it run Ok and put the output into the file file2.sha.
I know I can add stdout to the Popen but I was wonder if there is a way to run it as simple as runing from the command line.
Thanks.
You can pass values for the stdin and stdout of the child process to Popen like so:
subprocess.Popen("sha256sum file1.zip", stdout = file("file2.sha", "a"))
Note the file needs to be opened in append mode to achieve the same behaviour as >>.
I think you should use shell=True argument to Popen:
If shell is True, the specified command will be executed through the
shell. This can be useful if you are using Python primarily for the
enhanced control flow it offers over most system shells and still want
convenient access to other shell features such as shell pipes,
filename wildcards, environment variable expansion, and expansion of ~
to a user’s home directory.
subprocess.Popen("sha256sum file1.zip >> file2.sha", shell=True)