We Keep Coding

Python Multiprocessing weird behavior when NOT USING time.sleep()

This is the exact code from Python.org. If you comment out the time.sleep(), it crashes with a long exception traceback. I would like to know why.
And, I do understand why Python.org included it in their example code. But artificially creating "working time" via time.sleep() shouldn't break the code when it's removed. It seems to me that the time.sleep() is affording some sort of spin up time. But as I said, I'd like to know from people who might actually know the answer.
A user comment asked me to fill in more details on the environment this was happening in. It was on OSX Big Sur 11.4. Using a clean install of Python 3.95 from Python.org (no Homebrew, etc). Run from within Pycharm inside a venv. I hope that helps add to understanding the situation.
import time
import random
from multiprocessing import Process, Queue, current_process, freeze_support
#
# Function run by worker processes
#
def worker(input, output):
for func, args in iter(input.get, 'STOP'):
result = calculate(func, args)
output.put(result)
#
# Function used to calculate result
#
def calculate(func, args):
result = func(*args)
return '%s says that %s%s = %s' % \
(current_process().name, func.__name__, args, result)
#
# Functions referenced by tasks
#
def mul(a, b):
#time.sleep(0.5*random.random()) # <--- time.sleep() commented out
return a * b
def plus(a, b):
#time.sleep(0.5*random.random()). # <--- time.sleep() commented out
return a + b
#
#
#
def test():
NUMBER_OF_PROCESSES = 4
TASKS1 = [(mul, (i, 7)) for i in range(20)]
TASKS2 = [(plus, (i, 8)) for i in range(10)]
# Create queues
task_queue = Queue()
done_queue = Queue()
# Submit tasks
for task in TASKS1:
task_queue.put(task)
# Start worker processes
for i in range(NUMBER_OF_PROCESSES):
Process(target=worker, args=(task_queue, done_queue)).start()
# Get and print results
print('Unordered results:')
for i in range(len(TASKS1)):
print('\t', done_queue.get())
# Add more tasks using `put()`
for task in TASKS2:
task_queue.put(task)
# Get and print some more results
for i in range(len(TASKS2)):
print('\t', done_queue.get())
# Tell child processes to stop
for i in range(NUMBER_OF_PROCESSES):
task_queue.put('STOP')
if __name__ == '__main__':
freeze_support()
test()
This is the traceback if it helps anyone:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
Traceback (most recent call last):
File "<string>", line 1, in <module>
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/synchronize.py", line 110, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory

Here's a technical breakdown.
This is a race condition where the main process finishes, and exits before some of the children have a chance to fully start up. As long as a child fully starts, there are mechanisms in-place to ensure they shut down smoothly, but there's an unsafe in-between time. Race conditions can be very system dependent, as it is up to the OS and the hardware to schedule the different threads, as well as how fast they chew through their work.
Here's what's going on when a process is started... Early on in the creation of a child process, it registers itself in the main process so that it will be either joined or terminated when the main process exits depending on if it's daemonic (multiprocessing.util._exit_function). This exit function was registered with the atexit module on import of multiprocessing.
Also during creation of the child process, a pair of Pipes are opened which will be used to pass the Process object to the child interpreter (which includes what function you want to execute and its arguments). This requires 2 file handles to be shared with the child, and these file handles are also registered to be closed using atexit.
The problem arises when the main process exits before the child has a chance to read all the necessary data from the pipe (un-pickling the Process object) during the startup phase. If the main process first closes the pipe, then waits for the child to join, then we have a problem. The child will continue spinning up the new python instance until it gets to the point when it needs to read in the Process object containing your function and arguments it should run. It will try to read from a pipe which has already been closed, which is an error.
If all the children get a chance to fully start-up you won't see this ever, because that pipe is only used for startup. Putting in a delay which will in some way guarantee that all the children have some time to fully start up is what solves this problem. Manually calling join will provide this delay by waiting for the children before any of the atexit handlers are called. Additionally, any amount of processing delay means that q.get in the main thread will have to wait a while which also gives the children time to start up before closing. I was never able to reproduce the problem you encountered, but presumably you saw the output from all the TASKS (" Process-1 says that mul(19, 7) = 133 "). Only one or two of the child processes ended up doing all the work, allowing the main process to get all the results, and finish up before the other children finished startup.
EDIT:
The error is unambiguous as to what's happening, but I still can't figure how it happens... As far as I can tell, the file handles should be closed when calling _run_finalizers() in _exit_function after joining or terminating all active_children rather than before via _run_finalizers(0)
EDIT2:
_run_finalizers will seemingly actually never call Popen.finalizer to close the pipes, because exitpriority is None. I'm very confused as to what's going on here, and I think I need to sleep on it...

Apparently #user2357112supportsMonica was on the right track. It totally solves the problem if you join the processes before exiting the program. Also #Aaron's answer has the deep knowledge as to why this fixes the issue!
I added the following bits of code as was suggested and it totally fixed the need to have time.sleep() in there.
First I gathered all the processes when they were started:
processes: list[Process] = []
# Start worker processes
for i in range(NUMBER_OF_PROCESSES):
p = Process(target=worker, args=(task_queue, done_queue))
p.start()
processes.append(p)
Then at the end of the program I joined them as follows:
# Join the processes
for p in processes:
p.join()
Totally solved the issues. Thanks for the advice.

python: how to kill Excel session in separate thread after timeout?

I am executing an Excel macro in python and would like to kill it after timeout. However, existing timeout and kill Excel process method are not working for me. Can you please help?
import threading
import win32com.client as win;
import pythoncom;
Excel = None;
workbook = None;
def worker(e):
global Excel;
global workbook;
pythoncom.CoInitialize();
Excel = win.DispatchEx('Excel.Application');
Excel.DisplayAlerts = False;
Excel.Visible = True;
workbook = Excel.Workbooks.Open("sample.xlsm", ReadOnly = True);
Excel.Calculation = -4135;
print "Run";
Excel.Run('Module2.Refresh');
e = threading.Event()
t = threading.Thread(target=worker, args=(e,));
t.start()
# wait 5 seconds for the thread to finish its work
t.join(5)
if t.is_alive():
print "thread is not done, setting event to kill thread."
e.set();
print "1";
workbook.Close();
print "2";
Excel.Quit();
else:
print "thread has already finished."
workbook.Close();
Excel.Quit();
I got error:
Run
thread is not done, setting event to kill thread.
1
Traceback (most recent call last):
File "check_odbc.py", line 62, in <module>
workbook.Close();
File "C:\Users\honwang\AppData\Local\conda\conda\envs\py27_32\lib\site-package
s\win32com\client\dynamic.py", line 527, in __getattr__
raise AttributeError("%s.%s" % (self._username_, attr))
AttributeError: Open.Close

Unfortunately it is not possible to kill threads. All you can do is ask them nicely to suicide, then hope for the best. Just passing an event object is not enough, you have to actively check for that event inside the thread and suicide when it is set. Since your thread is blocked while running excel code it can't check for the event - that means you can yell at it to suicide as much as you want but there's no code to make it listen.
If you need this kind of parallelism on inherently blocking code, I strongly suggest you use processes instead, because those can be killed. Otherwise if possible use asynchronous programming.

python, COM and multithreading issue

I'm trying to take a look at IE's DOM from a separate thread that dispatched IE, and for some properties I'm getting a "no such interface supported" error. I managed to reduce the problem to this script:
import threading, time
import pythoncom
from win32com.client import Dispatch, gencache
gencache.EnsureModule('{3050F1C5-98B5-11CF-BB82-00AA00BDCE0B}', 0, 4, 0) # MSHTML
def main():
pythoncom.CoInitializeEx(0)
ie = Dispatch('InternetExplorer.Application')
ie.Visible = True
ie.Navigate('http://www.Rhodia-ecommerce.com/')
while ie.Busy:
time.sleep(1)
def printframes():
pythoncom.CoInitializeEx(0)
document = ie.Document
frames = document.getElementsByTagName(u'frame')
for frame in frames:
obj = frame.contentWindow
thr = threading.Thread(target=printframes)
thr.start()
thr.join()
if __name__ == '__main__':
thr = threading.Thread(target=main)
thr.start()
thr.join()
Everything is fine until the frame.contentWindow. Then bam:
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\python22\lib\threading.py", line 414, in __bootstrap
self.run()
File "C:\python22\lib\threading.py", line 402, in run
apply(self.__target, self.__args, self.__kwargs)
File "testie.py", line 42, in printframes
obj = frame.contentWindow
File "C:\python22\lib\site-packages\win32com\client\__init__.py", line 455, in __getattr__
return self._ApplyTypes_(*args)
File "C:\python22\lib\site-packages\win32com\client\__init__.py", line 446, in _ApplyTypes_
return self._get_good_object_(
com_error: (-2147467262, 'No such interface supported', None, None)
Any hint ?

The correct answer is to marshal stuff by hand. That's not a workaround it is what you are supposed to do here. You shouldn't have to use apartment threading though.
You initialised as multithreaded apartment - that tells COM that it can call your interfaces on any thread. It does not allow you to call other interfaces on any thread, or excuse you from marshalling interfaces provided by COM. That will only work "by accident" - E.g. if the object you are calling happens to be an in-process MTA object, it won't matter.
CoMarshalInterThreadInterfaceInStream/CoGetInterfaceAndReleaseStream does the business.
The reason for this is that objects can provide their own proxies, which may or may not be free-threaded. (Or indeed provide custom marshalling). You have to marshal them to tell them they are moving between threads. If the proxy is free threaded, you may get the same pointer back.

python thread exception errors when exit the whole program

HI, guys,
I am developing a GUI with python 2.4.3 and wxpython. Everything works fine except when I exit main program(close the main window of GUI). The wierd thing is that sometimes there is such error, sometimes there is no error at all. Although I found the same error reports from the python mailing list(the link is http://bugs.python.org/issue1722344, I am not sure if my case is the same as this one). I do not know how it is finally solved and what I should do to overcome this problem.
The error message from the console is as follows.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
File "/usr/lib/python2.4/threading.py", line 442, in __bootstrap
File "/opt/company/workspace/username/application/src/mainwidget.py", line 1066, in run
File "/usr/lib/python2.4/Queue.py", line 89, in put
File "/usr/lib/python2.4/threading.py", line 237, in notify
exceptions.TypeError: exceptions must be classes, instances, or strings (deprecated), not NoneType
Unhandled exception in thread started by
Error in sys.excepthook:
Original exception was:
The following is part of my code(the thread related code is complete, I extract the main operations for the rest). when I use the GUI to launch an external subprocess, at the same time, a wx.TextCtrl object is created. This wx.TextCtrl object is used to give input and print output of the external subprocess
class BashProcessThread(threading.Thread):
def __init__(self, readlineFunc):
threading.Thread.__init__(self)
self.readlineFunc = readlineFunc
self.lines = []
self.outputQueue = Queue.Queue()
self.setDaemon(True)
def run(self):
while True:
line = self.readlineFunc()
self.outputQueue.put(line)
if (line==""):
break
return ''.join(self.lines)
def getOutput(self):
""" called from other thread """
while True:
try:
line = self.outputQueue.get_nowait()
lines.append(line)
except Queue.Empty:
break
return ''.join(self.lines)
class ExternalProcWindow(wx.Window):
def __init__(self, parent, externapp):
wx.Window.__init__(self, parent, -1, pos=wx.DefaultPosition, size = wx.Size(1200, 120))
self.externapp=externapp
self.prompt = externapp.name.lower() + '>>'
self.textctrl = wx.TextCtrl(self, -1, '', size= wx.Size(1200, 120), style=wx.TE_PROCESS_ENTER|wx.TE_MULTILINE)
self.default_txt = self.textctrl.GetDefaultStyle()
self.textctrl.AppendText(self.prompt)
self.outputThread = BashProcessThread(self.externapp.sub_process.stdout.readline)
self.outputThread.start()
self.textctrl.SetFocus()
self.__bind_events()
self.Fit()
def __bind_events(self):
self.Bind(wx.EVT_TEXT_ENTER, self.__enter)
def __enter(self, e):
nl=self.textctrl.GetNumberOfLines()
ln = self.textctrl.GetLineText(nl-1)
ln = ln[len(self.prompt):]
self.externapp.sub_process.stdin.write(ln+"\n")
time.sleep(.3)
self.textctrl.AppendText(self.outputThread.getOutput())
class ExternApp:
def launch(self):
self.sub_process = subprocess.Popen(launchcmd, stdin=subprocess.PIPE,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)

The problem is caused by the use of threading.Thread.setDaemon. Threads set daemonic don't prevent the Python intepreter from exiting, but they still keep running. Because Python cleans up the environment before the process is terminated, the threads can run into trouble when stuff is removed from under them. That raises an exception, which the thread class tries to print for your convenience -- but that, then, too fails because the process is exiting.
You could try to silence the exception, but that's tricky (and if the thread does anything substantial, it might hide a real problem. Not the case here, though.) Or you could ask the thread to stop before exiting, and not set the thread daemonic. Or you can simply avoid using threads altogether. I do not remember if wxPython has a convenient mechanism for getting a process's output or even of doing asynchronous I/O, but many GUI toolkits do. And there's always Twisted, which does it all for you.

Why can't I create a COM object in a new thread in Python?

I'm trying to create a COM Object from a dll in a new thread in Python - so I can run a message pump in that thread:
from comtypes.client import CreateObject
import threading
class MessageThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.daemon = True
def run(self):
print "Thread starting"
connection = CreateObject("IDMessaging.IDMMFileConnection")
print "connection created"
a = CreateObject("IDMessaging.IDMMFileConnection")
print "aConnection created"
t = MessageThread()
t.start()
this is the error trace I get:
aConnection created
Thread starting
>>> Exception in thread Thread-1:
Traceback (most recent call last):
File "c:\python26\lib\threading.py", line 532, in __bootstrap_inner
self.run()
File "fred.py", line 99, in run
self.connection = CreateObject("IDMessaging.IDMMFileConnection")
File "c:\python26\lib\site-packages\comtypes\client\__init__.py", line 235, in CreateObject
obj = comtypes.CoCreateInstance(clsid, clsctx=clsctx, interface=interface)
File "c:\python26\lib\site-packages\comtypes\__init__.py", line 1145, in CoCreateInstance
_ole32.CoCreateInstance(byref(clsid), punkouter, clsctx, byref(iid), byref(p))
File "_ctypes/callproc.c", line 925, in GetResult
WindowsError: [Error -2147221008] CoInitialize has not been called
Any ideas?

You need to have called CoInitialize() (or CoInitializeEx()) on a thread before you can create COM objects on that thread.
from win32com.client.pythoncom import CoInitialize
CoInitialize()

As far as I remember (long time ago I'e programmed a lot with COM Components) you have to call CoInitialize on each thread if your COM Object uses STA.
http://msdn.microsoft.com/en-us/library/ms678543(VS.85).aspx
But I've no idea how to call that function in python.
Here is the MSDN Doc
http://msdn.microsoft.com/en-us/library/ms678543(VS.85).aspx

Just to update with current experience using PyCharm and Python 2.7:
You need to import:
from pythoncom import CoInitializeEx
from pythoncom import CoUninitialize
then for running the thread:
def run(self):
res = CoInitializeEx(0)
#<your code>
CoUninitialize()
PyCharm get confused with STA apartment, you need to enable true multithreading.
It is important that each CoInitialize() is terminated with a CoUninitialize(), so be sure your code follows this rule in case of errors, too.

As another answer has said you need to run
CoInitialize()
However it is possible that the COMObject cannot just be passed to the threads directly. You will have to use CoMarshalInterThreadInterfaceInStream() and CoGetInterfaceAndReleaseStream() to pass instance between threads
https://stackoverflow.com/a/27966218/18052428