We Keep Coding

WinError 6 - The handle is invalid with executable but code works on debug mode

I'm using a software that handle video processing in a powershell subprocess.
With PyCharm, when I run my software (in debug mode) everything works as expected.
When I use pyinstaller and inno setup to make an executable and install it on windows I got this error when Sub process start :
[WinError 6] The handle is invalid
I suppose it’s due to an error in a subprocess like this piece of code :
try:
psa1_path = EnvValues.powershell_path().format(project=project)
# using this powershell : C:/Users/${USERNAME}\projects\demo\cmd\powershell.ps1 -m -v 'CC 2018' -wait windowstyle hidden
dc_logger.info(f'using this powershell : {psa1_path}')
if project:
dc_logger.info("PowerShell Rendering Started")
with open(EnvValues.frame_path(), 'a') as f:
p = subprocess.Popen(['C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
psa1_path],
stdout=f)
p.communicate()
dc_logger.info("PowerShell Done Rendering")
return True
else:
dc_logger.info("no project to render")
return False
Is the bug related to arguments passed to the subprocess ? Why executable version of the code is the only one not working ? Why I don't have the bug in the development version ?
Here's my pyinstaller cmds :
pyinstaller --onefile -w -F -i "C:\Users\my_project\icon.ico" Project.py
pyinstaller --onefile -w -F -i "C:\Users\my_project\icon.ico" Project.spec
Then I put this in InnoSetup and I install the output to my windows machine.

The problem is with psa1_path variable
C:/Users/${USERNAME}\projects\demo\cmd\powershell.ps1 -m -v 'CC 2018' -wait windowstyle hidden
This variable has parameters. And the subprocess.Popen use it as a string, then you have to set shell=True so Popen will use this string as a complete shell cmd.
p = subprocess.Popen(['C:\\WINDOWS\\system32\\WindowsPowerShell\\v1.0\\powershell.exe',
'-ExecutionPolicy',
'Unrestricted',
psa1_path],
stdout=f, shell=True, stdin=subprocess.DEVNULL)
Don't forget to add stdin arg because it's also throw the [WinError 6] The handle is invalid
So why the code works with debug code and not with executable :
It's mainly because PyCharm does additionnal configuration and setup behind the scenes when it comes to running a program.
Because when you’re going from IDE to runtime, you need additional hooks to get things going. pyinstaller doesn’t do the subprocess the same way as pyCharm for the shell part

Pyinstaller executable keeps opening

Background
I am working on face recognition following this link and I would like to build a standalone application using Python. My main.py script looks like the following.
# main.py
# Import packages and other scripts
import tkinter as tk
...
# Some functions
def initialization():
# OpenCV and sklearn are used here
...
def registration():
# OpenCV, and sklearn are used here
...
def face_recognition():
# OpenCV and sklearn are used here
...
# Start the Tkinter GUI
window = tk.Tk()
# Initialize the face recognition model
initialization()
# Input name for registration
tk.Label(window, text = "Name").grid(...)
entry1 = tk.Entry(window)
entry1.grid(row=0, column=1)
# When the button is clicked, different command is executed
tk.Button(window, text='Registration', command=registeration).grid(...)
tk.Button(window, text='Face Recognition', command=face_recognition).grid(...)
window.mainloop()
Using python interpret to run the script (python main.py), everything works fine.
Problem
I use Pyinstaller to convert the scripts to a single exe with this command:
pyinstaller --onefile \
--windowed \
--hidden-import sklearn.neighbors.typedefs \
main.py
Then, I have two exe generated. The first one is in dist/main and the second one is in dist/main.app/Contents/MacOS/main
When running the exe, the exe keeps duplicate itself. I display the running process and found this result
$ ps
PID TTY TIME CMD
... ... ... /path/to/main -B -s -S -E -c from multiprocessing.semaphore_tracker import main:main(8)
... ... ... /path/to/main -B -s -S -E -c from multiprocessing.semaphore_tracker import main:main(8)
... ... ... /path/to/main -B -s -S -E -c from multiprocessing.semaphore_tracker import main:main(7)
... ... ... /path/to/main -B -s -S -E -c from multiprocessing.semaphore_tracker import main:main(7)
... ... ... /path/to/main -B -s -S -E -c from multiprocessing.semaphore_tracker import main:main(8)
I have no idea what happens to the exe since I do not import multiprocessing packages in my scripts. Any idea how to fix this problem? Thanks
Update 1
I added a --log-level ERROR when using Pyinstaller and gives this result. Not sure if it is related to my problem.
Traceback (most recent call last):
File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'Crypto.Math'
174598 INFO: MKL libraries found when importing numpy. Adding MKL to binaries
176282 ERROR: Can not find path ./libtbb.dylib (needed by /Users/user/anaconda3/lib/libmkl_tbb_thread.dylib)
Update 2
I found out that one of the packages that I am using -- imultis VideoStream involves threading. I guess it is the reason for observing from multiprocessing.semaphore_tracker as shown above even though I do not import multiprocessing explicitly.
Then I come across with this post, suggesting to add multiprocessing.freeze_support(). It can suppress the GUI window from keep duplicating, but the background processes as shown using $ ps still keep duplicating. The problem has not fixed yet.
Update 3 (Issue located but not fix)
After debugging the code for quite a while, it turns out that the causes of this infinite looping is NOT due to the threading of imultis VideoStream (I write another script to test imultis and it is completely OK). But the problem comes from importing sklearn!!! This problem has been reported in GitHub in this link.
This GitHub link also suggest to include multiprocessing.freeze_support(). Moreover one of the responses suggests to import sklearn after multiprocessing.freeze_support() is called. I try it with a simple script but the problem is still here.
Conclusion: sklearn causes the executable keeps opening. But I have no idea why it is the case, and I do not know how to solve it. Any help would be appreciated. Thanks.

This answer does a good job explaining why this happens: https://stackoverflow.com/a/55382641/109525
First try setting this before starting your program:
export JOBLIB_MULTIPROCESSING=0
If it works, you can add a runtime hook to your program to set the environment variable automatically. See: https://pythonhosted.org/PyInstaller/when-things-go-wrong.html#changing-runtime-behavior

Starting my main entry point with:
from multiprocessing import freeze_support
freeze_support()
Worked for me!

I met the same issue. Scikit-learn provides a workaround.
https://scikit-learn.org/stable/faq.html#why-do-i-sometime-get-a-crash-freeze-with-n-jobs-1-under-osx-or-linux
I added below codes at the beginning of my script and fixed my issue:
import multiprocessing
multiprocessing.set_start_method('forkserver', force=True)
multiprocessing.freeze_support()
Hope this can also fix yours.

Why is $PATH in remote deployment path different from $PATH in remote system?

I'm currently working on Pycharm with remote python Interpreter(miniconda3/bin/python).
So when I type echo $PATH in remote server, it prints
/home/woosung/bin:/home/woosung/.local/bin:/home/woosung/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
I created project in Pycharm and set remote python Interpreter as miniconda3 python, it works well when I just run some *.py files.
But when I typed some os.system() lines, weird things happened.
For instance, in test.py from Pycharm project
import os
os.system('echo $PATH')
os.system('python --version')
Output is
ssh://woosung#xxx.xxx.xxx.xxx:xx/home/woosung/miniconda3/bin/python -u /tmp/pycharm_project_203/test.py
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
Python 2.7.12
Process finished with exit code 0
I tried same command in remote server,
woosung#test-pc:~$ echo $PATH
/home/woosung/bin:/home/woosung/.local/bin:/home/woosung/miniconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
woosung#test-pc:~$ python --version
Python 3.6.6 :: Anaconda, Inc.
PATH and the version of python are totally different! How can I fix this?
I've already tried add os.system('export PATH="$PATH:$HOME/miniconda3/bin"') to test.py. But it still gives same $PATH.(/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games)
EDIT
Thanks to the comment of #Dietrich Epp, I successfully add interpreter path to the shell $PATH.
(os.environ["PATH"] += ":/home/woosung/miniconda3/bin")
But I stuck the more basic problem. When I add the path and execute command the some *.py file including import library which is only in miniconda3, the shell gives ImportError.
For instance, in test.py
import matplotlib
os.environ["PATH"] += ":/home/woosung/miniconda3/bin"
os.system("python import_test.py")
and import_test.py
import matplotlib
And when I run test.py,
Traceback (most recent call last):
File "import_test.py", line 1, in <module>
import matplotlib
ImportError: No module named matplotlib
Looks like the shell doesn't understand how to utilize modified $PATH.

I find the solution.
It is not direct but quite simple.
I changed os.system("python import_test.py") to os.system(sys.executable + ' import_test.py').
This makes the shell uses the Pycharm remote interpreter(miniconda3), not original.

shebang not working for python script

I've been looking over many answers here on stackoverflow. I've tried absolutely everything. I have this at the top of my AddressConversion.py python script.
#!/usr/bin/env python
import argparse
The objective is to run this as a command utility, meaning I could type
AddressConversion [options][address]
As of now I would settle for being able to type
./AddressConversion [options][address]
I have done the chmod so the file is executable
I've ran dos2unix on the file to eliminate any random windows characters(which wouldn't seem possible because the file has only been used on Ubuntu.
I've checked the python install with which python with the results
/usr/bin/python
I've also checked which env and get a similar path
The script will work fine when I use the traditional python command. It also works fine when I type:
usr/bin/env python
It will open up the python interpreter.
These steps seem to be the solutions suggested anytime someone asks this question. I am getting this error:
./AddressConversion.py: line 1: import: command not found
./AddressConversion.py: line 3: syntax error near unexpected token `('
./AddressConversion.py: line 3: `def init_parser():'
which seems like it is trying to run it as a shell script or something.
Any suggestions?

created one file executeme.py
#!/usr/bin/env python
print("hello")
make it as executable (optional)
chmod a+x executeme.py
reanme the file
mv executeme.py executeme
Execute now
./executeme
OUTPUT
hello
Another option to create one setup.py file, for more you can read here
in entry_points a key name console_script in which you can give the name of executor and target module in format
'name=target'
from setuptools import setup, find_packages
setup(
name='executor',
packages=find_packages(),
install_requires=[,
],
entry_points = {
'console_scripts': [
'executeme=executeme:main',
],
},
)
then run the command
pip install -e /path/to/setup.py
Installing from local src in Development Mode, i.e. in such a way that
the project appears to be installed, but yet is still editable from
the src tree.
pipdoc

I had a similar issue and it ended being because of the CRLF at the end of lines. These were added when the script was created on a windows machine. To check if this is the case use the file command.
file script.py
It will give you an output like this. "Python script, ASCII text executable, with CRLF line terminators"
To remove the CRLF line terminators do the following.
dos2unix script.py

Python sh import causing Unresolved Import error in PyDev

I'm using sh in python 2.7.5 to call shell programs like curl and mkdir, but in PyDev plugin 2.7.5 under Eclipse 4.3.0. the following line gives an Unresolved Import error:
from sh import curl, printenv, mkdir, cat
I'm able to run the above code in a python shell. I do have the path to sh included in the Libraries pane in the Interpreter - Python window in Preferences, so I don't think that's the problem.

Try using the subprocess module to call console commands. For example:
from subprocess import call
dir_name = '/foo/bar/'
call('mkdir %s'%dir_name, shell=True)

Like Bill said, subprocess is a good choice here. I'd personally recommend using the Popen because it doesn't block, and allows you to wait for commands to finish with its communicate() method, which also returns stdout and stderr. Also, avoid using shell=True when possible. Usage:
import subprocess
testSubprocess = subprocess.Popen(['mkdir', dir_name], stdout=subprocess.PIPE)
testOut, testErr = testSubprocess.communicate()