I have a very peculiar bug with unittest (and pytest, which has the same behaviour), only on Windows and only on Python 3.8 and later.
My program depends on a 3rd party application that has to be installed separately. This application adds a DLL to a folder on the PATH, and I have an extension module that depends on that DLL. My program also uses the multiprocessing pool - so this DLL is loaded from the main process as well as all the spawned processes.
Starting with Python 3.8, DLLs are not loaded from everywhere in the PATH, but just from a specific set of directories. I call os.add_dll_directory with the proper directory.
My program works when I run it. It finds the DLL, loads my extension module, fires up child processes which all do their jobs, and everything is great. Until I try the same in a unit test.
When I try the same code in a unit test, it fails. The spawned child processes do not find the DLL. It is as if os.add_dll_directory is not carried over to the child processes. Again, only when running the unit test. It is carried over when running independently.
The unit tests also work with Python 3.7 or on Linux - where os.add_dll_directory is not required.
I'm rather stuck. I really want to have a test that tests this parallel functionality, and it seems as if I can't.
Related
I compiled a python program using PyInstaller. But when I execute the .exe file on a client, two instances of the .exe is running on the task manager.
How can i eliminate the other .exe?
This question didn't have any clear answer.
I'm using python 2.7 and I'm running the .exe on Windows XP.
I recommend reading this section:
https://pythonhosted.org/PyInstaller/#id74
Bootloader
The bootloader prepares everything for running Python code. It begins the setup and then returns itself in another process. This approach of using two processes allows a lot of flexibility and is used in all bundles except one-folder mode in Windows. So do not be surprised if you will see your bundled app as two processes in your system task manager.
In general I'm curious, does using a utility that converts a non-exe to an exe increase speed? It's my understanding that they just package the interpreter inside the exe.
In specific, if you have a python script and use py2exe on it, does the resulting executable run faster than the .py? My boss seems to have the assumption it does but I'm not so sure.
Especially when dealing with multiple modules. For example say you have modules first.py and second.py. You compile them all to executables. When they were .py they second.py could be called as
from second import main
main()
Now that they're executables you have to start a new process, which surely is slower?
subproccess.call(["second.exe"], shell=True)
Do I understand this correctly? Or does importing from another python module actually start a new instance of the python interpreter or something?
In our case the target platform is always Windows.
Your boss is misinformed. All py2exe does is package your program into a self-contained package capable of running without dependencies. It is still the same bytecode running on the same interpreter (well, whatever one is packaged).
See this other answer for about all of the "optimization" you can get out of using -o flags.
Also, yes, definitely run some benchmarks to confirm for yourself.
I was to make a launcher application, I haven't found a way to detach a sub-process entirely from the spawning python process.
When I launch a program with my desktop's (cinnamon's) launcher the process tree goes:
/sbin/init -> mdm -> mdm -> cinnamon-session -> cinnamon -> the-app-i-launched
Of the threads I read, this one was the most insightful/helpful: Launch a completely independent process. But gets muddied answers as the OP is looking to run python code, which can often be achieved in many usually-preferred ways than by spawning an independent process.
From other posts from stack overflow that do not answer how to launch a detatched python process:
Running daemonalized python code: Applicable to running python code/module as a daemon, (not another process/application) detached from the python instance.
subprocess.call: Process spawns as a child of the python process.
os.system: Process spawns as a child of the python process.
close_fds: (Apparently) Windows(R)-only solution, need portable solution (primary target is Debian linux). Attempting to use close_fds=True on linux, process spawns as a child of the python process.
creationflags: Windows(R)-only solution. On linux raises: ValueError: creationflags is only supported on Windows platforms.
prefix launched process with nohup: Process spawns as a child of the python process. As far as I know, nohup or equivalent is not available on all platforms, making it a linux-only solution.
os.fork: Same as "Running daemonalized python code".
multiprocessing: Same problem as "Running daemonalized python code": useful only for running python code/module.
os.spawnl* + os.P_NOWAIT: Deprecated functions are undesirable to use for new code. In my testing I was not able to see my process actually spawned at all.
os.spawnl* + os.P_DETACH: Windows(R)-only, seemingly removed in current python 2.X versions: AttributeError: 'module' object has no attribute 'P_DETACH'.
os.system + shell fork: I was able to actually see my process run detatched from the python process with this, however I am woried that it has the faults:
Relies on running commands in a shell, which is more vulnerable to maliciousness, intentional or otherwise?.
Relies on non-portable? POSIX/shell? syntaxies that may not be interpenetrate on non-Linux platforms. Which I haven't dug up any good reference for portability on Partial Ref.
subprocess.Popen Alt: I still only observed the sub-process running as a child of the python process.
A working solution can be found in JonMc's answer here. I use it to open documents using 'xdg-open'.
You can change the stderr argument to stderr=open('/dev/null', 'w'), if you do not want a logfile.
The only workable solution I have, and that may be non-portable Linux-only is to use shell evaluation with shell-detach-ampersand syntax.
#!/usr/bin/env python2
import os
os.system('{cmd} &'.format('firefox'))
This might go too-far up the process tree though, outside of the window manager session, potentially not-exiting with your desktop session.
/sbin/init -> firefox
I am looking into daemonization of my Python script and I have found some libraries that can help: daemonic, daemonize and daemon. Each of them have some issues:
daemonic and daemonize will terminate when they cannot create the PID file. daemonic will not even log or print anything. Looking into the code, they actually call os.exit(). I would like an exception or other error message, so I can fallback to running my code in the foreground.
daemon doesn't even install correctly for Python 3. And seeing the last commit was in 2010, I don't expect any updates soon (if ever).
How can I portably (both Python2 and 3) and optionally (falling back to running in foreground) create a daemonized Python script? Of course I could fallback to using the & operator when starting it, but I would like to implement PEP3143.
I am using two solutions
based on zdaemon
based on supervisor
Both packages are written in Python and are daemonizing anything, what can be run from command line. The requirement is, that the command to be run is running in foreground and not trying to daemonize itself.
supervisor is even part of Linux distributions and even though it comes in a bit outdated version, it is very well usable.
Note, that as it controls general command line driven program, it does not require python version being matched with the controlled code.
I'm currently working on a small python script, for controlling my home PC (really just a hobby project - nothing serious).
Inside the script, there is two threads running at the same time using thread (might start using threading instead) like this:
thread.start_new_thread( Function, (Args) )
Its works as intended when testing the script... but after compiling the code using Pyinstaller there are two processes (One for each thread - I think).
How do I fix this?
Just kill the loader from the main program if it really bothers you. Here's one way to do it.
import os
import win32com.client
proc_name = 'MyProgram.exe'
my_pid = os.getpid()
wmi = win32com.client.GetObject('winmgmts:')
all_procs = wmi.InstancesOf('Win32_Process')
for proc in all_procs:
if proc.Properties_("Name").Value == proc_name:
proc_pid = proc.Properties_("ProcessID").Value
if proc_pid != my_pid:
print "killed my loader %s\n" % (proc_pid)
os.kill(proc_pid, 9)
Python code does not need to be "compiled with pyinstaller"
Products like "Pyinstaller" or "py2exe" are usefull to create a single executable file that you can distribute to third parties, or relocate inside your computer without worrying about the Python instalation - however, they don add "speed" nor is the resulting binary file any more "finished" than your original .py (or .pyw on Windows) file.
What these products do is to create another copy of the Python itnrepreter, alogn with all the modules your porgram use, and pack them inside a single file. It is likely the Pyinstaller keep a second process running to check things on the main script (like launching it, maybe there are options on it to keep the script running and so on). This is not part of a standard Python program.
It is not likely Pyinstaller splits the threads into 2 separate proccess as that would cause compatibility problems - thread run on the same process and can transparently access the same data structures.
How a "canonical" Python program runs: the main process, seen by the O.S. is the Python binary (Python.exe on Windows) - it finds the Python script it was called for - if there is a ".pyc" file for it, that is loaded - else, it loads your ".py" file and compiles that to Python byte code (not to windwos executable). This compilation is authomatic and transparent to people running the program. It is analogous to a Java compile from a .java file to a .class - but there is no explicit step needed by the programmer or user - it is made in place - and other factors control wether Python will store the resulting bytecode as .pyc file or not.
To sum up: there is no performance impact in running the ".py" script directly instead of generating an .exe file with Pyinstaller or other product. You have a disk-space usage inpact if you do, though, as you will have one copy of the Python interpreter and libraries for each of your scripts.
The URL pointeded by Janne Karila on the comment nails it - its even worse than I thought:
in order to run yioru script, pyinstaller unpacks Python DLLs and modules in a temporary directory. The time and system resources needed todo that, compared with a single script run is non-trivial.
http://www.pyinstaller.org/export/v2.0/project/doc/Manual.html?format=raw#how-one-file-mode-works