We Keep Coding

Pythons subprocess check_call doesn't give the same result as the same command executed in the command line

I am using an anaconda environment both for the python code and the terminal.
When I want to execute a program in the shell (Windows CMD) with the environment activated. The program ogr2ogr returns the correct output with the given parameter. The tool ogr2ogr has been installed via a conda package.
But when I execute the my python code, the ogr2ogr returns an errors output. I thought it might be to different installations used due to usage of different environments (without my knowledge), but this is ownly a guess.
The python code goes as follows:
from pathlib import Path
from subprocess import check_call, STDOUT
...
file_path = Path(file_name)
destination = str(file_path.with_suffix(".gpkg"))
command = f"ogr2ogr -f GPKG -s_srs EPSG:25833 -t_srs EPSG:25833 {destination} GMLAS:{file_name} -oo REMOVE_UNUSED_LAYERS=YES"
check_call(command, stderr=STDOUT, shell=True)
ogr2ogr translates a file into another format. Which is also done, but when I open the file, I see, it's not done 100 % correctly.
When I copy the value of the string command and copy it to the shell and execute the command the execution is done correctly!
How can I correct the behaviour of using subprocess.check_call

WinError 6 - The handle is invalid with executable but code works on debug mode

I'm using a software that handle video processing in a powershell subprocess.
With PyCharm, when I run my software (in debug mode) everything works as expected.
When I use pyinstaller and inno setup to make an executable and install it on windows I got this error when Sub process start :
[WinError 6] The handle is invalid
I suppose it’s due to an error in a subprocess like this piece of code :
try:
psa1_path = EnvValues.powershell_path().format(project=project)
# using this powershell : C:/Users/${USERNAME}\projects\demo\cmd\powershell.ps1 -m -v 'CC 2018' -wait windowstyle hidden
dc_logger.info(f'using this powershell : {psa1_path}')
if project:
dc_logger.info("PowerShell Rendering Started")
with open(EnvValues.frame_path(), 'a') as f:
p = subprocess.Popen(['C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
psa1_path],
stdout=f)
p.communicate()
dc_logger.info("PowerShell Done Rendering")
return True
else:
dc_logger.info("no project to render")
return False
Is the bug related to arguments passed to the subprocess ? Why executable version of the code is the only one not working ? Why I don't have the bug in the development version ?
Here's my pyinstaller cmds :
pyinstaller --onefile -w -F -i "C:\Users\my_project\icon.ico" Project.py
pyinstaller --onefile -w -F -i "C:\Users\my_project\icon.ico" Project.spec
Then I put this in InnoSetup and I install the output to my windows machine.

The problem is with psa1_path variable
C:/Users/${USERNAME}\projects\demo\cmd\powershell.ps1 -m -v 'CC 2018' -wait windowstyle hidden
This variable has parameters. And the subprocess.Popen use it as a string, then you have to set shell=True so Popen will use this string as a complete shell cmd.
p = subprocess.Popen(['C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
psa1_path],
stdout=f, shell=True, stdin=subprocess.DEVNULL)
Don't forget to add stdin arg because it's also throw the [WinError 6] The handle is invalid
So why the code works with debug code and not with executable :
It's mainly because PyCharm does additionnal configuration and setup behind the scenes when it comes to running a program.
Because when you’re going from IDE to runtime, you need additional hooks to get things going. pyinstaller doesn’t do the subprocess the same way as pyCharm for the shell part

Python Popen fails in compound command (PowerShell)

I am trying to use Python's Popen to change my working directory and execute a command.
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()
However, powershell returns "The system cannot find the path specified." The path does exist.
If I run
pg = subprocess.Popen("cd c:/mydirectory ;", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
it returns the same thing.
However, if i run this: (without the semicolon)
pg = subprocess.Popen("cd c:/mydirectory",stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
The command returns without an error. This leads me to believe that the semicolon is issue. What is the cause for this behavior and how can I get around it?
I know I can just do c:/mydirectory/runExecutable.exe --help, but I would like to know why this is happening.
UPDATE :
I have tested passing the path to powershell as the argument for Popen's executable parameter. Just powershell.exe may not be enough. To find the true absolute path of powershell, execute where.exe powershell. Then you can pass it into Popen. Note that shell is still true. It will use the default shell but pass the command to powershell.exe
powershell = C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe
pg = subprocess.Popen("cd c:/mydirectory ; ./runExecutable.exe", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, executable=powershell)
buff,buffErr = pg.communicate()
//It works!

In your subprocess.Popen() call, shell=True means that the platform's default shell should be used.
While the Windows world is - commendably - moving from CMD (cmd.exe) to PowerShell, Python determines what shell to invoke based on the COMSPEC environment variable, which still points to cmd.exe, even in the latest W10 update that has moved toward PowerShell in terms of what the GUI offers as the default shell.
For backward compatibility, this will not change anytime soon, and will possibly never change.
Therefore, your choices are:
Use cmd syntax, as suggested in Maurice Meyer's answer.
Do not use shell = True and invoke powershell.exe explicitly - see below.
Windows only: Redefine environment variable COMSPEC before using shell = True - see below.
A simple Python example of how to invoke the powershell binary directly, with command-line switches followed by a single string containing the PowerShell source code to execute:
import subprocess
args = 'powershell', '-noprofile', '-command', 'set-location /; $pwd'
subprocess.Popen(args)
Note that I've deliberately used powershell instead of powershell.exe, because that opens up the possibility of the command working on Unix platforms too, once PowerShell Core is released.
Windows only: An example with shell = True, after redefining environment variable COMSPEC to point to PowerShell first:
import os, subprocess
os.environ["COMSPEC"] = 'powershell'
subprocess.Popen('Set-Location /; $pwd', shell=True)
Note:
COMSPEC is only consulted on Windows; on Unix platforms, the shell executable is invariably /bin/sh
As of Windows PowerShell v5.1 / PowerShell Core v6-beta.3, invoking powershell with just -c (interpreted as -Command) still loads the profiles files by default, which can have unexpected side effects (with the explicit invocation of powershell used above, -noprofile suppresses that).
Changing the default behavior to not loading the profiles is the subject of this GitHub issue, in an effort to align PowerShell's CLI with that of POSIX-like shells.

You can concat multiple commands using '&' character instead of a semicolon. Try this:
pg = subprocess.Popen("cd c:/mydirectory & ./runExecutable.exe --help", stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True)
buff,buffErr = pg.communicate()

Python: executing a complex command on windows

I am having a problem when running a command on Windows whereas it works perfectly on Linux.
I give you the context, but this is not necessary to understand my issue: I am using gimp in batch mode.
I have a Python script callPythonFuScript.py which calls another Python script, pythonFu.py, which executes a python-fu call.
In callPythonFuScript.py, I construct the command line when I call the function inside pythonFu.py to be executed. This is the command line:
gimp-console-2.8 -idf --batch-interpreter python-fu-eval -b 'import sys;sys.path=['.']+sys.path;import pythonFu;pythonFu.myFunction("arg1","arg2","arg3") ' -b 'pdb.gimp_quit(1)'
This command works perfectly on Linux but when I try to run it on Windows, it does not work.
Error messages are:
The opening of C:\Users\myRep\sys; failed : no such file or directory
The opening of C:\Users\myRep\sys.path=['.']+sys.path; failed : no such file or directory
The opening of C:\Users\myRep\"arg1","arg2","arg3")' failed no such file or directory
I am assuming that Windows interprets characters differently than Linux. Is this correct? How can I fix this problem?

As mentioned in the comments, you are having an escaping issue between what the command prompt sees as arguments, and what is being passed as a literal string for python to eval:
-b 'import sys;sys.path=["."]+sys.path;import pythonFu;pythonFu.myFunction("arg1","arg2","arg3")'
If that still gives you errors, it is possible you might need to escape the double quotes:
-b 'import sys;sys.path=[\".\"]+sys.path;import pythonFu;pythonFu.myFunction(\"arg1\",\"arg2\",\"arg3\")'

Using ruby gem/command from python

I've installed the Ruby gem 'haml' on my mac, which I can use to compile haml files into html files using the following command at the terminal:
haml 'path/to/haml/file.haml' 'desired/html/path/file.html'
This command simply creates an html file at the second path, and gives no output in the terminal. So for example, this command:
haml "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.haml" "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.html"
Creates a sugar.html file at the given path. Now I'm trying to use this functionality from a python script. When I type this into IDLE's interactive python shell:
>>>import subprocess
>>>subprocess.Popen('haml "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.haml" "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.html"', shell=True, executable='/bin/bash')
<subprocess.Popen object at 0x159d6f0>
I get output suggesting that the process has been run, however, there is no file outputted. Why is this happening? I even put in the Shell argument, but no interactive shell shows up. Also, I read somewhere that the default shell used is not bash, which is what the Mac terminal uses, so I put that in too for good measure.
Following icktoofay's advice, I ran check_call. Here is the traceback I received:
Traceback (most recent call last):
File
"/Users/neil/Desktop/subprocesstest.py",
line 7, in
p = subprocess.check_call(x, shell=True) File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py",
line 504, in check_call
raise CalledProcessError(retcode, cmd) CalledProcessError: Command 'haml
"/Volumes/Macintosh
HD/Users/neil/Sites/ICSP/sugar.haml"
"/Volumes/Macintosh
HD/Users/neil/Sites/ICSP/sugar.html"'
returned non-zero exit status 127
According to the bash reference manual, while searching for a command to be executed,
If the name is neither a shell
function nor a builtin, and contains
no slashes, Bash searches each element
of $PATH for a directory containing an
executable file by that name. ... If
that function is not defined, the
shell prints an error message and
returns an exit status of 127.
However, I thought it was indeed finding the haml command after adding the shell and executable arguments, because before that, it was giving a 'file or directory not found error', which indicates that the function is not executable directly but rather in a shell only.
Now how do I make python find this haml command? Or would I have to use some ugly workaround like an applescript which then invokes the haml command.

I see that you are using, shell=True, so I would have expected things to just work. Checked it locally here with Python 2.7.1 and haml 3.1.1 and I had no problems executing it. There are also some python implementations you might be interested in, PyHAML, HamlPy, djaml or django-haml.
import subprocess
subprocess.Popen(['haml', 'hello.haml', 'hello.html'], shell=True)
% cat hello.html
<strong class='code' id='message'>Hello, World!</strong>

shlex.split() is your friend, when you want to build args list suitable for Popen and its ilk.
>>> import subprocess
>>> import shlex
>>> p = subprocess.Popen(shlex.split('haml "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.haml" "/Volumes/Macintosh HD/Users/me/Sites/ICSP/sugar.html"'))
>>> p.wait()
0