We Keep Coding

Python subprocess permissionerror

When I try to execute a file which is located inside the Program Files directory, I get a PermissionError execption.
excat error
Traceback (most recent call last):
File "Build.py", line 24, in <module>
subprocess.Popen([buildCMD], stdin=subprocess.PIPE)
File "subprocess.py", line 854, in __init__
File "subprocess.py", line 1307, in _execute_child
PermissionError: [WinError 5] Zugriff verweigert
[23948] Failed to execute script 'Build' due to unhandled exception!
Code:
import subprocess
buildCMD = '"C:/Program Files/Microchip/xc8/v2.32/bin/xc8-cc.exe" -mcpu=16f1787 -Wl,-Map=.build/main.build.map -DXPRJ_default=default -Wl,--defsym=__MPLAB_BUILD=1 -mdfp=C:/Program Files/Microchip/MPLABX/v5.50/packs/Microchip/PIC12-16F1xxx_DFP/1.2.63/xc8 -fno-short-double -fno-short-float -fasmfile -maddrqual=ignore -xassembler-with-cpp -mwarn=-3 -Wa,-a -msummary=-psect,-class,+mem,-hex,-file -ginhx32 -Wl,--data-init -mno-keep-startup -mno-osccal -mno-resetbits -mno-save-resetbits -mno-download -mno-stackcall -std=c99 -gdwarf-3 -mstack=compiled:auto:auto -Wl,--memorysummary,.build/memoryfile.xml -o .build/main.build.hex main.c'
subprocess.Popen([buildCMD], stdin=subprocess.PIPE)

Passing a string as a list is doubly wrong, though Windows is somewhat more forgiving than real computers here. You want either
subprocess.run([
"C:/Program Files/Microchip/xc8/v2.32/bin/xc8-cc.exe",
"-mcpu=16f1787", "-Wl,-Map=.build/main.build.map",
"-DXPRJ_default=default", "-Wl,--defsym=__MPLAB_BUILD=1",
"-mdfp=C:/Program Files/Microchip/MPLABX/v5.50/packs/Microchip/PIC12-16F1xxx_DFP/1.2.63/xc8",
"-fno-short-double", "-fno-short-float", "-fasmfile",
"-maddrqual=ignore", "-xassembler-with-cpp", "-mwarn=-3",
"-Wa,-a", "-msummary=-psect,-class,+mem,-hex,-file",
"-ginhx32", "-Wl,--data-init", "-mno-keep-startup",
"-mno-osccal", "-mno-resetbits", "-mno-save-resetbits",
"-mno-download", "-mno-stackcall", "-std=c99", "-gdwarf-3",
"-mstack=compiled:auto:auto",
"-Wl,--memorysummary,.build/memoryfile.xml",
"-o", ".build/main.build.hex", "main.c"],
stdin=subprocess.PIPE,
check=True)
or the same as a string (but then with correct quoting around the arguments with spaces in them, notably -mdfp=C:/Program Files/...) and with shell=True (which however you usually want to avoid.)
Notice also the addition of check=True to have Python raise an exception if the subprocess fails, and the preference for subprocess.run() over subprocess.Popen unless you specifically require the subprocess to run alongside with your Python script, and then commit to managing the process object until it is terminated.

Running .exe without a console - OSError: [WinError 6] The handle is invalid

EDIT: Please note, I have seen other topics concerning this issue and already tried most of the suggestions there
I use pyinstaller to run an .exe file and now I am tryng to run it without a console (using the -w command).
One of my key libraries, patool, uses subprocesses, which gives the following error:
Traceback (most recent call last):
File "apscheduler\executors\base.py", line 125, in run_job
File "script.py", line 478, in Archiver
File "patoolib\__init__.py", line 521, in _create_archive
File "patoolib\__init__.py", line 421, in run_archive_cmdlist
File "patoolib\util.py", line 227, in run_checked
File "patoolib\util.py", line 219, in run
File "subprocess.py", line 339, in call
File "subprocess.py", line 753, in __init__
File "subprocess.py", line 1090, in _get_handles
OSError: [WinError 6] The handle is invalid
Here is the part of the patool util.py code which has the subprocesses.call() that gives the error:
def run (cmd, verbosity=0, **kwargs):
"""Run command without error checking.
#return: command return code"""
# Note that shell_quote_nt() result is not suitable for copy-paste
# (especially on Unix systems), but it looks nicer than shell_quote().
if verbosity >= 0:
log_info("running %s" % " ".join(map(shell_quote_nt, cmd)))
if kwargs:
if verbosity >= 0:
log_info(" with %s" % ", ".join("%s=%s" % (k, shell_quote(str(v)))\
for k, v in kwargs.items()))
if kwargs.get("shell"):
# for shell calls the command must be a string
cmd = " ".join(cmd)
if verbosity < 1:
# hide command output on stdout
with open(os.devnull, 'wb') as devnull:
kwargs['stdout'] = devnull
res = subprocess.call(cmd, **kwargs) <------------- ERROR
else:
res = subprocess.call(cmd, **kwargs)
return res
This is a common error, so I tried reading about subprocesses module and also dug out every single possible suggestion online, including:
adding kwargs['stdin'] = devnull, as suggested here: Python running as Windows Service: OSError: [WinError 6] The handle is invalid
adding shell=True to the call() method
adding subprocess._cleanup() at the start of the run() function
Neither of these works, the handler is still invalid. The programme works fine with the console active.
I am using Python 3.7, Anaconda3, 64-bit Windows 10 OS.
Later in the util.py there is a subprocess.popen() that I suspect will cause my the same problem.
I was trying to run the .exe by having the console active and then hiding it, but then I encounter other problems (it does not start upon system start up). I guess the console being present is pretty important, but I would love to get rid off it for better user experience.

Fixed it by adding: stdin=subprocess.PIPE in this file for python 3.8:
C:\Program Files (x86)\Python38-32\Lib*os.py* line 983 to 991
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, **stdin=subprocess.PIPE**, stderr=subprocess.STDOUT,bufsize=buffering)

Python - unable to call system command

I have some python code, from which I want to call another program. This program will
Print some output to STDOUT
Write a file to disk
Using call I get the following behavior;
from subprocess import call
call(['./tango_x86_64_release', 'VTS1 ct="N" nt="N" ph="7.2" te="303" io="0.02" seq="MKHPYEEFPTGSKSPYNMSRGAHPGAV"'])
34, File not properly written, try writing it up again,
1
This happens regardless if if the arguments are split into a list or not;
call(['./tango_x86_64_release', 'VTS1', 'ct="N"', 'nt="N"', 'ph="7.2"', 'te="303"', 'io="0.02"', 'seq="MKHPYEEFPTGSKSPYNMSRGAHPGAV"'])
34, File not properly written, try writing it up again,
1
I can call this same command from the my terminal
./tango_x86_64_release VTS1 ct="N" nt="N" ph="7.2" te="303" io="0.02" seq="MKHPYEEFPTGSKSPYNMSRGAHPGAV"
Which works and gives an exit status of 0.
It seems like its the writing to disk which is causing issues, if I break the command then I get the appropriate warning message (i.e. remove an argument, it warns me that the argument is missing).
Using subprocess.Popen() gives an OSError;
import subprocess as sub
output = sub.Popen('./tango_x86_64_release VTS1 ct="N" nt="N" ph="7.2" te="303" io="0.02" seq="MKHPYEEFPTGSKSPYNMSRGAHPGAV"', stdout=sub.PIPE, stderr=sub.PIPE)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Any help greatly appreciated

Use shlex.split to split the command for you:
import shlex
call(shlex.split('./tango_x86_64_release VTS1 ct="N" nt="N" ph="7.2" te="303" io="0.02" seq="MKHPYEEFPTGSKSPYNMSRGAHPGAV"'))
Note that although you might be able to solve your problem by adding shell=True, you should avoid it if possible, since it can be a security risk (search for "shell injection").

Try to add shell=True to the Popen call.
Also see:
Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
Popen error: [Errno 2] No such file or directory
Documentation (and why Passing shell=True can be a security hazard)

python errno 24 on cgi script using subprocess

I have a python cgi script that runs an application via subprocess over and over again (several thousand times). I keep getting the same error...
Traceback (most recent call last):
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 413, in <module>
webpage()
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 406, in main
displayOmpResult(form['odfFile'].value)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 342, in displayContainerDiv
makeSection(position,sAoiInput)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 360, in displayData
displayTable(i,j,lAmpAndVars,dOligoSet[key],position)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 247, in displayTable
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1039, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
The function causing it is below.
def displayTable(sData):
# convert the data to the proper format
sFormattedData = convertToFormat(sData)
# write the formatted data to file
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
sOutputFileLoc = sInputFile.replace('In_','Out_')
# run app, requires two files; an input and an output
# temp file to holds stdout stderr of subprocess
fh = tempfile.TemporaryFile(mode='w',dir=tempfile.gettempdir())
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
p.communicate()
fh.close()
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
displayTableHeader(lOutputData)
displaySimpleTable(lOutputData)
As far as I can tell, I'm closing the files properly. When I run...
import resource
print resource.getrlimit(resource.RLIMIT_NOFILE)
I get...
(1024, 1024)
Do I have to increase this value? I read that subprocess opens several file descriptors. I tried adding "close_fds = True" and I tried using the with statement when creating my file but the result was the same. I suspect the problem may be with the application that I'm subprocessing, pDat, but this program was made by someone else. It requires two inputs; an input file and the location of where you want the output file written to. I suspect it may not be closing the output file that it creates. Aside from this, I can't see what I might be doing wrong. Any suggestions? Thanks.
EDIT:
I'm on ubuntu 10.04 running python 2.6.5 and apache 2.2.14

Instead of this...
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
I should have done this...
iFileHandle,sInputFile = tempfile.mkstemp(prefix='In_')
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
os.close(iFileHandle)
The mkstemp function makes OS level handles to a file and I wasn't closing them. The solution is described in more detail here...
http://www.logilab.org/blogentry/17873

You want to add close_fds=True to the popen call (just in case).
Then, here:
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
...I might remember wrong, but unless you use the with syntax, I do not think that the output file descriptor has been closed.
UPDATE: the main problem is that you need to know which files are open. On Windows this would require something like Process Explorer. In Linux it's a bit simpler; you just have to invoke the CGI from command line, or be sure that there is only one instance of the CGI running, and fetch its pid with ps command.
Once you have the pid, run a ls -la on the content of the /proc/<PID>/fd directory. All open file descriptors will be there, with the name of the files they point to. Knowing that file so-and-so is opened 377 times, that goes a long way towards finding out where exactly that file is opened (but not closed).

checking the return code using python (MAC)

I have written a script that checks if an SVN Repo is up and running, the result is based on the return value.
import subprocess
url = " validurl"
def check_svn_status():
subprocess.call(['svn info'+url],shell=True)
def get_status():
subprocess.call('echo $?',shell=True)
def main():
check_svn_status()
get_status()
if __name__ == '__main__':
main()
The problem I'm facing is that if I change the url to something that does't exist I still get the return value as 0, but if I were to run this outside the script, i.e go to the terminal type svn info wrong url and then do a echo $? I get a return value of 1. But I can't re-create this in the python. Any guidelines ?
TraceBack after updating
Traceback (most recent call last):
File "svn_status.py", line 21, in <module>
main()
File "svn_status.py", line 15, in main
check_svn_status()
File "svn_status.py", line 8, in check_svn_status
p = sp.Popen(['svn info'], stdout=sp.PIPE, stderr=sp.PIPE)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1202, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or director
y

Why your approach does not work:
You invoke two independent subshells. The second shell does not know of the first shell and therefore does not have any information about the returncode of the process that was executed in the first shell.
Solution:
Use the subprocess module, spawn your subprocess directly (not through a subshell) and retrieve the returncode. Help yourself by reading the documentation of the module: http://docs.python.org/library/subprocess.html
There are several ways to achieve your goal. One simple way could be:
import subprocess as sp
p = sp.Popen(['command', 'arg1', 'arg2'], stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate()
returncode = p.returncode
This way, you don't go through a subshell (shell=False by default), which is the recommended approach for various reasons. You directly catch the returncode of the spawned subprocess and you have full access to the subprocess' standard output and standard error.

subprocess.call returns the retcode, just store the result of your subprocess.call(['svn info'+url],shell=True)
http://docs.python.org/library/subprocess.html