We Keep Coding

Running .exe without a console - OSError: [WinError 6] The handle is invalid

EDIT: Please note, I have seen other topics concerning this issue and already tried most of the suggestions there
I use pyinstaller to run an .exe file and now I am tryng to run it without a console (using the -w command).
One of my key libraries, patool, uses subprocesses, which gives the following error:
Traceback (most recent call last):
File "apscheduler\executors\base.py", line 125, in run_job
File "script.py", line 478, in Archiver
File "patoolib\__init__.py", line 521, in _create_archive
File "patoolib\__init__.py", line 421, in run_archive_cmdlist
File "patoolib\util.py", line 227, in run_checked
File "patoolib\util.py", line 219, in run
File "subprocess.py", line 339, in call
File "subprocess.py", line 753, in __init__
File "subprocess.py", line 1090, in _get_handles
OSError: [WinError 6] The handle is invalid
Here is the part of the patool util.py code which has the subprocesses.call() that gives the error:
def run (cmd, verbosity=0, **kwargs):
"""Run command without error checking.
#return: command return code"""
# Note that shell_quote_nt() result is not suitable for copy-paste
# (especially on Unix systems), but it looks nicer than shell_quote().
if verbosity >= 0:
log_info("running %s" % " ".join(map(shell_quote_nt, cmd)))
if kwargs:
if verbosity >= 0:
log_info(" with %s" % ", ".join("%s=%s" % (k, shell_quote(str(v)))\
for k, v in kwargs.items()))
if kwargs.get("shell"):
# for shell calls the command must be a string
cmd = " ".join(cmd)
if verbosity < 1:
# hide command output on stdout
with open(os.devnull, 'wb') as devnull:
kwargs['stdout'] = devnull
res = subprocess.call(cmd, **kwargs) <------------- ERROR
else:
res = subprocess.call(cmd, **kwargs)
return res
This is a common error, so I tried reading about subprocesses module and also dug out every single possible suggestion online, including:
adding kwargs['stdin'] = devnull, as suggested here: Python running as Windows Service: OSError: [WinError 6] The handle is invalid
adding shell=True to the call() method
adding subprocess._cleanup() at the start of the run() function
Neither of these works, the handler is still invalid. The programme works fine with the console active.
I am using Python 3.7, Anaconda3, 64-bit Windows 10 OS.
Later in the util.py there is a subprocess.popen() that I suspect will cause my the same problem.
I was trying to run the .exe by having the console active and then hiding it, but then I encounter other problems (it does not start upon system start up). I guess the console being present is pretty important, but I would love to get rid off it for better user experience.

Fixed it by adding: stdin=subprocess.PIPE in this file for python 3.8:
C:\Program Files (x86)\Python38-32\Lib*os.py* line 983 to 991
proc = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, **stdin=subprocess.PIPE**, stderr=subprocess.STDOUT,bufsize=buffering)

Python subprocess FileNotFoundError

I am trying to follow this blog on how to execute an R script from Python. I have the R script working fine from the command line using Rscript.
Here's my Python code:
import subprocess
import os
command = "C:\Program Files\R\R-3.4.4\bin\Rscript"
path2script = os.getcwd() + "\max.R" # gives me the absolute path to the R script
args = ["11", "3", "9", "42"]
cmd = [command, path2script] + args
x = subprocess.check_output(cmd, universal_newlines = True)
Which gives me this error:
FileNotFoundError: [WinError 2] The system cannot find the file specified
I've read a lot of SO posts on this error and in most cases it seems to be a problem with trying to invoke system commands like dir or passing arguments to check_output in the wrong order but in my case I really don't see what should be going wrong.
Following some of the advice I've tried building a string for cmd instead of a list, and then passing it to check_output using the argument shell = True - when I do that I get a CalledProcessError: returned non-zero exit status 1.
I'm assuming this code, which is exactly as it appeared on the blog other than adding the absolute path to the file, is failing now because the behaviour of check_output has changed since 2015...
Can anyone help?
Here's the stack trace:
Traceback (most recent call last):
File "<ipython-input-2-3a0151808726>", line 1, in <module>
runfile('C:/Users/TomWagstaff/Documents/Raising IT/Projects/15 AdWords/Python_R_test/run_max.py', wdir='C:/Users/TomWagstaff/Documents/Raising IT/Projects/15 AdWords/Python_R_test')
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)
File "C:/Users/TomWagstaff/Documents/Raising IT/Projects/15 AdWords/Python_R_test/run_max.py", line 31, in <module>
x = subprocess.check_output(cmd, universal_newlines = True)
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\subprocess.py", line 336, in check_output
**kwargs).stdout
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\site-packages\spyder\utils\site\sitecustomize.py", line 210, in __init__
super(SubprocessPopen, self).__init__(*args, **kwargs)
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\subprocess.py", line 709, in __init__
restore_signals, start_new_session)
File "C:\Users\TomWagstaff\Anaconda3\envs\adwords\lib\subprocess.py", line 997, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

check that you have a right path for command and script
print(os.path.exists(command))
print(os.path.exists(path2script))
note that writing path with backslashes may be dangerous as you can create escape sequence that way which will be interpreted in different way. You can write windows paths with forward slashes and then call os.path.normpath on them, turning them into safe form
(also in command you can use forward slashes only, Python interpret doesn't really care. In path to your R script that would be probably problem though)

Why subprocess.call () is showing error in linux?

Let us consider Linux platform where I need to execute a program called smart.exe which uses input.dat file. Both the files are placed in the same directory with each file having the same file permission 777.
Now if I run the following command in the terminal window smart.exe is fully executed without any error.
$./smart.exe input.dat
On the other hand, if I use the following python script called my_script.py placed in the same directory, then I get an error.
my_script.py has the following code:
#!/usr/bin/python
import os, subprocess
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
The error is as follows:
File "my_script.py", line 4, in <module>
exit_code = subprocess.call("./smart.exe input.dat", shell = False)
File "/usr/lib64/python2.6/subprocess.py", line 478, in call
p = Popen(*popenargs, **kwargs)
File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
errread, errwrite)
File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Can someone please tell me why this is happening. Please note that the smart.exe should take around 10 sec to fully complete. This may be a clue for the problem.
Please also advise if there is any other way to run smart.exe from my_script.py. Your solution is much appreciated!

You should decide if you want shell support or not.
If you want the shell to be used (which is not necessary here), you should use exit_code = subprocess.call("./smart.exe input.dat", shell=True). Then the shell interprets your command line.
If you don't want it (as you don't need it and want to avoid unnecessary complexity), you should do exit_code = subprocess.call(["./smart.exe", "input.dat"], shell=False).
(And there is no point naming your binarys .exe under Linux.)

python errno 24 on cgi script using subprocess

I have a python cgi script that runs an application via subprocess over and over again (several thousand times). I keep getting the same error...
Traceback (most recent call last):
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 413, in <module>
webpage()
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 406, in main
displayOmpResult(form['odfFile'].value)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 342, in displayContainerDiv
makeSection(position,sAoiInput)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 360, in displayData
displayTable(i,j,lAmpAndVars,dOligoSet[key],position)
File "/home/linuser/Webpages/cgi/SnpEdit.py", line 247, in displayTable
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1039, in _execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files
The function causing it is below.
def displayTable(sData):
# convert the data to the proper format
sFormattedData = convertToFormat(sData)
# write the formatted data to file
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
sOutputFileLoc = sInputFile.replace('In_','Out_')
# run app, requires two files; an input and an output
# temp file to holds stdout stderr of subprocess
fh = tempfile.TemporaryFile(mode='w',dir=tempfile.gettempdir())
p = subprocess.Popen(['/usr/bin/pDat',sInputFileLoc,sOutputFileLoc],stdout=fh, stderr=fh)
p.communicate()
fh.close()
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
displayTableHeader(lOutputData)
displaySimpleTable(lOutputData)
As far as I can tell, I'm closing the files properly. When I run...
import resource
print resource.getrlimit(resource.RLIMIT_NOFILE)
I get...
(1024, 1024)
Do I have to increase this value? I read that subprocess opens several file descriptors. I tried adding "close_fds = True" and I tried using the with statement when creating my file but the result was the same. I suspect the problem may be with the application that I'm subprocessing, pDat, but this program was made by someone else. It requires two inputs; an input file and the location of where you want the output file written to. I suspect it may not be closing the output file that it creates. Aside from this, I can't see what I might be doing wrong. Any suggestions? Thanks.
EDIT:
I'm on ubuntu 10.04 running python 2.6.5 and apache 2.2.14

Instead of this...
sInputFile = tempfile.mkstemp(prefix='In_')[1]
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
I should have done this...
iFileHandle,sInputFile = tempfile.mkstemp(prefix='In_')
fOpen = open(sInputFile,'w')
fOpen.write(sFormattedData)
fOpen.close()
os.close(iFileHandle)
The mkstemp function makes OS level handles to a file and I wasn't closing them. The solution is described in more detail here...
http://www.logilab.org/blogentry/17873

You want to add close_fds=True to the popen call (just in case).
Then, here:
# open output file and print parsed data into a list of dictionaries
sOutput = open(sOutputFileLoc).read()
lOutputData = parseOutput(sOutput)
...I might remember wrong, but unless you use the with syntax, I do not think that the output file descriptor has been closed.
UPDATE: the main problem is that you need to know which files are open. On Windows this would require something like Process Explorer. In Linux it's a bit simpler; you just have to invoke the CGI from command line, or be sure that there is only one instance of the CGI running, and fetch its pid with ps command.
Once you have the pid, run a ls -la on the content of the /proc/<PID>/fd directory. All open file descriptors will be there, with the name of the files they point to. Knowing that file so-and-so is opened 377 times, that goes a long way towards finding out where exactly that file is opened (but not closed).

checking the return code using python (MAC)

I have written a script that checks if an SVN Repo is up and running, the result is based on the return value.
import subprocess
url = " validurl"
def check_svn_status():
subprocess.call(['svn info'+url],shell=True)
def get_status():
subprocess.call('echo $?',shell=True)
def main():
check_svn_status()
get_status()
if __name__ == '__main__':
main()
The problem I'm facing is that if I change the url to something that does't exist I still get the return value as 0, but if I were to run this outside the script, i.e go to the terminal type svn info wrong url and then do a echo $? I get a return value of 1. But I can't re-create this in the python. Any guidelines ?
TraceBack after updating
Traceback (most recent call last):
File "svn_status.py", line 21, in <module>
main()
File "svn_status.py", line 15, in main
check_svn_status()
File "svn_status.py", line 8, in check_svn_status
p = sp.Popen(['svn info'], stdout=sp.PIPE, stderr=sp.PIPE)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 672, in __init__
errread, errwrite)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1202, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or director
y

Why your approach does not work:
You invoke two independent subshells. The second shell does not know of the first shell and therefore does not have any information about the returncode of the process that was executed in the first shell.
Solution:
Use the subprocess module, spawn your subprocess directly (not through a subshell) and retrieve the returncode. Help yourself by reading the documentation of the module: http://docs.python.org/library/subprocess.html
There are several ways to achieve your goal. One simple way could be:
import subprocess as sp
p = sp.Popen(['command', 'arg1', 'arg2'], stdout=sp.PIPE, stderr=sp.PIPE)
stdout, stderr = p.communicate()
returncode = p.returncode
This way, you don't go through a subshell (shell=False by default), which is the recommended approach for various reasons. You directly catch the returncode of the spawned subprocess and you have full access to the subprocess' standard output and standard error.

subprocess.call returns the retcode, just store the result of your subprocess.call(['svn info'+url],shell=True)
http://docs.python.org/library/subprocess.html