When initializer throw Error like below, script won't stop.
I would like to abort before starting main process(do not run 'do_something').
from multiprocessing import Pool
import contextlib
def initializer():
raise Exception("init failed")
def do_something(args):
# main process
pass
pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
try:
pool.map_async(do_something, [1]).get(100)
except:
pool.terminate()
The never stopping stacktrace on console is below
...
Exception: init failed
Process ForkPoolWorker-18:
Traceback (most recent call last):
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 249, in _bootstrap
self.run()
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "/home/hoge/anaconda3/lib/python3.6/multiprocessing/pool.py", line 103, in worker
initializer(*initargs)
File "hoge.py", line 5, in initializer
raise Exception("init failed")
Exception: init failed
...
My workaround is suppressing initializer error and return at the beginning of the main process by using global flag like below.
But I would like to learn better one.
def initializer():
try:
raise Exception("init failed")
except:
global failed
failed = True
def do_something(args):
global failed
if failed:
# skip when initializer failed
return
# main process
After navigating through the implementation of multiprocessing using PyCharm, I'm convinced that there is no better solution, because Pool started a thread to _maintain_pool() by _repopulate_pool() if any worker process exists--either accidentally or failed to initialize.
Check this out: Lib/multiprocessing/pool.py line 244
I just came across the same woe. My first solution was to catch the exception and raise it in the worker function (see below). But on second thought it really means that initializer support of multiprocessing.Pool is broken and sould not be used. So I now prefer to do the initialization stuff directly in the worker.
from multiprocessing import Pool
import contextlib, sys
_already_inited = False
def initializer():
global _already_inited
if _already_inited:
return
_already_inited = True
raise Exception("init failed")
def do_something(args):
initializer()
# main process
pool = Pool(1)
with contextlib.closing(pool):
pool.map_async(do_something, [1]).get(100)
Both the code and the stacktrace are simpler.
Off course all your worker function need to call initializer().
My initial solution was to defer the exception to the worker function.
from multiprocessing import Pool
import contextlib, sys
failed = None
def initializer():
try:
raise Exception("init failed")
except:
global failed
failed = sys.exc_info()[1]
def do_something(args):
global failed
if failed is not None:
raise RuntimeError(failed) from failed
# main process
pool = Pool(1, initializer=initializer)
with contextlib.closing(pool):
pool.map_async(do_something, [1]).get(100)
That way the caller still gets access to the exception.
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/tmp/try.py", line 15, in do_something
raise RuntimeError(failed)
RuntimeError: init failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/try.py", line 20, in <module>
pool.map_async(do_something, [1]).get(100)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: init failed
(venv) kmkaplan#dev1:~/src/options$ python3 /tmp/try.py
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/tmp/try.py", line 7, in initializer
raise Exception("init failed")
Exception: init failed
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.5/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/tmp/try.py", line 15, in do_something
raise RuntimeError(failed) from failed
RuntimeError: init failed
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/tmp/try.py", line 20, in <module>
pool.map_async(do_something, [1]).get(100)
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
RuntimeError: init failed
Related
This is my current code:
import psutil
count = 0
while count < 3000000000:
for process in psutil.process_iter():
if process.name().lower() == 'chrome.exe':
print(process)
process.terminate()
count = count + 1
else:
print('no')
print(count)
its scanning all of he processes running. It then crashes when it scans:
' WindowsInternal.ComposableShell.Experiences.TextInput.Inpu...'
This is the windows service for the on screen keyboard. I tried to stop it running however it still runs. I was wondering if i could give my python file full permissions so this stops happening.
This is the error code from the terminal:
Traceback (most recent call last):
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil_common.py", line 449, in wrapper
ret = self._cache[fun]
AttributeError: _cache
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil_pswindows.py", line 679, in wrapper
return fun(self, *args, **kwargs)
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil_common.py", line 452, in wrapper
return fun(self)
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil_pswindows.py", line 766, in exe
exe = cext.proc_exe(self.pid)
PermissionError: [WinError 24] The program issued a command but the command length is incorrect: '(originated from NtQuerySystemInformation)'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:/Users/benmi/PycharmProjects/HelloWorld/TerminateProgram.py", line 5, in
if process.name().lower() == 'chrome.exe':
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil__init__.py", line 630, in name
name = self._proc.name()
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil_pswindows.py", line 750, in name
return os.path.basename(self.exe())
File "C:\Users\benmi\PycharmProjects\HelloWorld\venv\lib\site-packages\psutil_pswindows.py", line 681, in wrapper
raise convert_oserror(err, pid=self.pid, name=self._name)
psutil.AccessDenied: psutil.AccessDenied (pid=8612)
I am unable to access the disk drives on the Windows system using infi.manager package found on PyPi.
Tried the below :
from infi.devicemanager import DeviceManager
dm = DeviceManager()
dm.root.rescan()
disks = dm.disk_drives
names = [disk.friendly_name for disk in disks]
Error messages :
Traceback (most recent call last):
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\functions.py", line 56, in callee
yield decorated_func(*args, **kwargs)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\functions.py", line 70, in SetupDiEnumDeviceInfo
interface(device_info_set, index, device_info_buffer)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\cwrap\__init__.py", line 138, in __new__
return_value = function(*args[1:], **kwargs)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\__init__.py", line 35, in errcheck
raise WindowsException(GetLastError())
infi.devicemanager.setupapi.WindowsException: 259, No more data is available.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\setupapi\functions.py", line 60, in callee
raise StopIteration
StopIteration
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\__init__.py", line 215, in disk_drives
for controller in self.storage_controllers:
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\__init__.py", line 227, in storage_controllers
return self.get_devices_from_handle(handle)
File "C:\Users\rsushmit\AppData\Local\Programs\Python\Python37-32\lib\site-packages\infi\devicemanager\__init__.py", line 198, in get_devices_from_handle
for devinfo in functions.SetupDiEnumDeviceInfo(handle):
RuntimeError: generator raised StopIteration
from infi.devicemanager import DeviceManager
dm = DeviceManager()
dm.root.rescan()
devices = dm.all_devices
for device in devices:
print(device)
I'm encountering a weird error when running the script in PyCharm's debug mode or inside a Flask app. I have narrowed down my code to the following
import virtualbox
import threading
class ThreadExecutor(threading.Thread):
def __init__(self):
super().__init__()
def run(self):
vbox = virtualbox.VirtualBox()
if __name__ == '__main__':
th = ThreadExecutor()
th.start()
Running this as a module produces no errors and executes perfectly fine but in debug mode it produces the following error message
Connected to pydev debugger (build 181.5087.37)
Exception in thread Thread-6:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\site-packages\vboxapi\__init__.py", line 449, in __init__
None)
pywintypes.com_error: (-2147221008, 'CoInitialize has not been called.', None, None)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Program Files\Python36\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:/Users/.../dev/debug.py", line 13, in run
vbox = virtualbox.VirtualBox()
File "C:\Program Files\Python36\lib\site-packages\virtualbox\library_ext\vbox.py", line 22, in __init__
manager = virtualbox.Manager()
File "C:\Program Files\Python36\lib\site-packages\virtualbox\__init__.py", line 143, in __init__
self.manager = vboxapi.VirtualBoxManager(mtype, mparams)
File "C:\Program Files\Python36\lib\site-packages\vboxapi\__init__.py", line 991, in __init__
self.platform = PlatformMSCOM(dPlatformParams)
File "C:\Program Files\Python36\lib\site-packages\vboxapi\__init__.py", line 455, in __init__
print("Warning: CoInitializeSecurity failed: ", oXctp);
NameError: name 'oXctp' is not defined
Going into \vboxapi__init__.py we find
try:
pythoncom.CoInitializeSecurity(None,
None,
None,
pythoncom.RPC_C_AUTHN_LEVEL_DEFAULT,
pythoncom.RPC_C_IMP_LEVEL_IMPERSONATE,
None,
pythoncom.EOAC_NONE,
None)
except:
_, oXcpt, _ = sys.exc_info();
if isinstance(oXcpt, pythoncom.com_error) and self.xcptGetStatus(oXcpt) == -2147417831: # RPC_E_TOO_LATE
print("Warning: CoInitializeSecurity was already called");
else:
print("Warning: CoInitializeSecurity failed: ", oXctp);
What's causing this error? Does sys.exc_info() behave differently inside a thread inside a debugger?
There seems to be some known, unresolved issues in the interaction between threading and virtualbox, see here. I would suggest using multiprocessing instead as I have not experienced any of the previous issues with it.
I'm using the Pool function of the multiprocessing module in order to run the same code in parallel on different data.
It turns out that on some data my code raises an exception, but the precise line in which this happens is not given:
Traceback (most recent call last):
File "my_wrapper_script.py", line 366, in <module>
main()
File "my_wrapper_script.py", line 343, in main
results = pool.map(process_function, folders)
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 148, in map
return self.map_async(func, iterable, chunksize).get()
File "/usr/lib64/python2.6/multiprocessing/pool.py", line 422, in get
raise self._value
KeyError: 'some_key'
I am aware of multiprocessing.log_to_stderr() , but it seems that it is useful when concurrency issues arise, which is not my case.
Any ideas?
If you're using a new enough version of Python, you'll actually see the real exception get printed prior to that one. For example, here's a sample that fails:
import multiprocessing
def inner():
raise Exception("FAIL")
def f():
print("HI")
inner()
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Here's the exception when running this with python 3.4:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "test.py", line 9, in f
inner()
File "test.py", line 4, in inner
raise Exception("FAIL")
Exception: FAIL
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "test.py", line 13, in <module>
p.apply(f)
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 253, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python3.4/multiprocessing/pool.py", line 599, in get
raise self._value
Exception: FAIL
If using a newer version isn't an option, the easiest thing to do is to wrap your worker function in a try/except block that will print the exception prior to re-raising it:
import multiprocessing
import traceback
def inner():
raise Exception("FAIL")
def f():
try:
print("HI")
inner()
except Exception:
print("Exception in worker:")
traceback.print_exc()
raise
p = multiprocessing.Pool()
p.apply(f)
p.close()
p.join()
Output:
HI
Exception in worker:
Traceback (most recent call last):
File "test.py", line 11, in f
inner()
File "test.py", line 5, in inner
raise Exception("FAIL")
Exception: FAIL
Traceback (most recent call last):
File "test.py", line 18, in <module>
p.apply(f)
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 244, in apply
return self.apply_async(func, args, kwds).get()
File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 558, in get
raise self._value
Exception: FAIL
You need to implement your own try/except block in the worker. Depending on how you want to organize your code, you could log to stderr as you mention above, log to some other place like a file, return some sort of error code or even tag the exception with the current traceback and re-raise. Here's an example of the last technique:
import traceback
import multiprocessing as mp
class MyError(Exception):
pass
def worker():
try:
# your real code here
raise MyError("boom")
except Exception, e:
e.traceback = traceback.format_exc()
raise
def main():
pool = mp.Pool()
try:
print "run worker"
result = pool.apply_async(worker)
result.get()
# handle exceptions you expect
except MyError, e:
print e.traceback
# re-raise the rest
except Exception, e:
print e.traceback
raise
if __name__=="__main__":
main()
It returns
run worker
Traceback (most recent call last):
File "doit.py", line 10, in worker
raise MyError("boom")
MyError: boom
tl;dr I spawn 3 threads, each thread throws an exception, most pythonic way to raise all 3 exceptions?
Below is a code example that is similar to what I am doing.
from multiprocessing.pool import ThreadPool
def fail_func(host):
raise Exception('{} FAILED!!!'.format(host))
hosts = ['172.1.1.1', '172.1.1.2', '172.1.1.3']
pool = ThreadPool(processes=5)
workers = [pool.apply_async(fail_func(host)) for host in hosts]
# join and close thread pool
pool.join(); pool.close()
# get the exceptions
[worker.get() for worker in workers if not worker.successful()]
What it ends up doing is just failing on the 1st host with the following traceback:
Traceback (most recent call last):
File "thread_exception_example.py", line 8, in <module>
workers = [pool.apply_async(fail_func(host)) for host in hosts]
File "thread_exception_example.py", line 4, in fail_func
raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.1 FAILED!!!
But what I want it to do is raise multiple exceptions for each thread that failed, like so:
Traceback (most recent call last):
File "thread_exception_example.py", line 8, in <module>
workers = [pool.apply_async(fail_func(host)) for host in hosts]
File "thread_exception_example.py", line 4, in fail_func
raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.1 FAILED!!!
Traceback (most recent call last):
File "thread_exception_example.py", line 8, in <module>
workers = [pool.apply_async(fail_func(host)) for host in hosts]
File "thread_exception_example.py", line 4, in fail_func
raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.2 FAILED!!!
Traceback (most recent call last):
File "thread_exception_example.py", line 8, in <module>
workers = [pool.apply_async(fail_func(host)) for host in hosts]
File "thread_exception_example.py", line 4, in fail_func
raise Exception('{} FAILED!!!'.format(host))
Exception: 172.1.1.3 FAILED!!!
is there any pythonic way of doing this? or do I need to wrap everything in a try/except, collect all the messages, then re-raise a single Exception?
There is no way to "raise multiple exceptions". In a given exception context, there is either an exception, or not.
So yes, you will have to create a wrapper exception that holds all of the exceptions, and raise that. But you've almost got all the code you need:
def get_exception():
try:
worker.get()
except Exception as e:
return e
Now, instead of:
[worker.get() for worker in workers if not worker.successful()]
… you can just do:
[get_exception(worker.get) for worker in workers if not worker.successful()]
And that's a list of exceptions.
Personally, I've always thought AsyncResult should have an exception method, similar to the one in concurrent.futures.Future. But then I would have used futures here in the first place (installing the backport if I were forced to use Python 2.x).