I am executing a Tornado server with ProcessPoolExecutor to handle multiple requests in parallel.
The problem is that, in one particular case, when an exception is raised in one of the processes it doesn't propagates, but instead the process crashes with this error:
concurrent.futures.process._RemoteTraceback:
\n'''\nTraceback (most recent call last):
\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\concurrent\\futures\\process.py\", line 367, in _queue_management_worker\n result_item = result_reader.recv()
\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\multiprocessing\\connection.py\", line 251, in recv
\n return _ForkingPickler.loads(buf.getbuffer())\nTypeError: __init__() missing 1 required positional argument: 'is_local'\n'''\n\nThe above exception was the direct cause of the following exception:
\n
\nTraceback (most recent call last):\n File \"C:\\S1\\Product\\Baseline\\PYTHON\\lab\\controller.py\", line 558, in get\n output = exec_future.result()
\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\concurrent\\futures\\_base.py\", line 428, in result\n return self.__get_result()\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\concurrent\\futures\\_base.py\", line 384, in __get_result
\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n
I have tried it in debugger, and found that the problem is executing this
def _send_bytes(self, buf):
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
try:
if err == _winapi.ERROR_IO_PENDING:
waitres = _winapi.WaitForMultipleObjects(
[ov.event], False, INFINITE)
assert waitres == WAIT_OBJECT_0
except:
ov.cancel()
raise
finally:
nwritten, err = ov.GetOverlappedResult(True)
assert err == 0
assert nwritten == len(buf)
This is called when the process tries to propagate the exception to the corresponding Future object.
In the first line, when calling _winapi.WriteFile, everything crashes in debugger, and I can't understand why. Any idea?
I have resolved with a workaround: I have wrapped internally the function inside separate process in try except, then copied the old exception in a new exception and raised it. I don't know why... but it works.
def _execute_tuning(tune_parameters: TuneParameters):
# function to parallelize todo to be refactored
# execute scenario, then write result or error in output
try:
config.generate_project_config(
project_name=tune_parameters.project_name,
scenario_name=tune_parameters.scenario_name
)
config.generate_session_log_config(project_name=tune_parameters.project_name,
scenario_name=tune_parameters.scenario_name)
tree = DecisionTreeGenerator(tune_parameters.project_name, tune_parameters.scenario_name)
tree.fit(
# todo refactor
auto_tune=True if tune_parameters == 'true' else False,
max_depth=tune_parameters.max_depth,
columns=tune_parameters.columns,
min_samples_leaf=tune_parameters.min_samples_per_leaf,
max_leaf_nodes=tune_parameters.max_leaf_nodes
)
kpi = KPICalculator(tune_parameters.project_name, tune_parameters.scenario_name)
kpi.run(do_optimization_kpi=False)
except Exception as exc:
Loggers.application.exception(exc)
exc_final = Exception(str(exc))
exc_final.__traceback__ = exc.__traceback__
raise exc_final
Related
For example, I am using multiprocessing pool to process files:
with Pool(5) as pool:
results = pool.starmap(self.process_file, zip(files, repeat(channel)))
When an exception occurs inside the function process_file, the exception message indicates that it occurs at the pool.starmap line, but not the actual place inside process_file function.
I am using PyCharm to develop and debug. Is there a way to change this behavior? The current error message doesn't give the correct position of the error occurred.
Multiprocessing transfers the errors between processes using the pickle module, but pickle doesn't know how to preserve the tracebacks of exceptions by default.
I found tblib to be a very convenient way to address this shortcoming. Based on this example I suggest you try adding this code to the main module of your code:
from tblib import pickling_support
# all your setup code
pickling_support.install()
if __name__ == "__main__":
# your pool work
The exception has the original exception info but PyCharm is not ferreting it out.
Assuming there are no PyCharm configuration options to enhance its ability to ferret out all the exception information, and not just the outer exception as your are seeing, you need to programmatically extract it out.
For good in-program error handling, you probably want to do that anyway. Especially with sub-processes, I very often will catch Exception and log it and re-raise if not considered handled (depends), or if I catch a specific exception I'm expecting and consider it handled, I won't re-raise it.
Note, it's not only PyCharm showing the outer exception... I see the same thing with other tools.
Below will show you the original problem (see "Line 7" below) and re-raise. Again, re-raising or not is really context dependent so below is just an example. But the point is that the exception you are seeing has more data that PyCharm by default is apparently not showing you.
from itertools import repeat
from multiprocessing import Pool
import traceback
def process(a,b):
print(a,b)
raise Exception("not good")
if __name__ == '__main__':
with Pool(5) as pool:
try:
results = pool.starmap(process, zip([1,2,3,4,5], repeat('A')))
except Exception as ex:
print("starmap failure:")
for error_line in traceback.format_exception(ex, ex, ex.__traceback__):
error_line = error_line.strip()
if not error_line:
continue
print(f" {error_line}")
raise # re-raise if we do not consider this handled.
Gives me this output:
starmap failure:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\...\multiprocessing\pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "C:\Users\...\multiprocessing\pool.py", line 51, in starmapstar
return list(itertools.starmap(args[0], args[1]))
File "...\starmap_exception.py", line 7, in process
raise Exception("not good")
Exception: not good
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "...\starmap_exception.py", line 12, in <module>
results = pool.starmap(process, zip([1,2,3,4,5], repeat('A')))
File "C:\Users\...\multiprocessing\pool.py", line 372, in starmap
return self._map_async(func, iterable, starmapstar, chunksize).get()
File "C:\Users\...\multiprocessing\pool.py", line 771, in get
raise self._value
Exception: not good
I am writing a program in python, and part of the program reads lines from a file and prints them. However, while it is printing, if I press a random key I get a KeyboardInterrupt error. This is because I have a try and except case, however, some output shows up that I do not want to show up. This is my code that executes inside the main function:
for line in output.stdout:
line = str(line)
print(line)
This is my code that contains the try and except.
if __name__ == '__main__':
try:
if os.path.exists('drop.json') == True:
drops()
except KeyboardInterrupt:
sys.stderr.close()
except (BrokenPipeError, IOError):
if os.path.exists('drop.txt') == True:
file = open("drop.txt","a+")
file.write('BrokenPipeError | IOError Exception occurred')
file.close()
except Exception as e:
if os.path.exists('drop.txt') == True:
file = open("drops.txt","a+")
file.write(' interrupted: {}'.format(str(e)))
file.close()
While the file is being printed, if I press a random key, it stops the program and outputs this error message which I do not want. The preferred behavior is for the printing to end, the program to stop, and there to be no further output at all. This is the error message that shows up when I press a random key:
Traceback (most recent call last):
File "drop.py", line 100, in <module> drops()
File "drop.py", line 321, in drops print(line)
BrokenPipeError: [Err 56] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "drop.py", line 756, in <module>file = open ("drop.txt","a+")
File "boot.py", line 68, in decoding
def decoding(do_setlocale=True):
KeyboardInterrupt
Instead of all these error messages showing up, if I press a random key during the printing of lines, I would the printing to end, the program to stop, and no error messages to be displayed. I would really appreciate if anyone could help me out here. Thanks a lot :)
Im trying to raise a custom exception message when I get an exception but I get the following error -
try:
query_start_time = time.time()
execute_sql_alchemy_query
except Exception as ex:
elapsed_time = (time.time() - query_start_time)/60
print(type(ex))
raise type(ex)("Query elapsed time(in mins) - {0}".format(elapsed_time))
Error:-
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/profiles/sources/impact/test.py", line 110, in _handle_future_exception
future.result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 428, in result
return self.__get_result()
File "/usr/local/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
raise self._exception
TypeError: __init__() missing 2 required positional arguments: 'params' and 'orig'
The exception being raised is probably a subclass of DBAPIError (https://docs.sqlalchemy.org/en/13/core/exceptions.html#sqlalchemy.exc.DBAPIError) which takes 3 positional/mandatory parameters in its constructor (statement, params and orig), by trying to raise it like this, you are only providing a “statement” but not the 2 other arguments.
You could probably manage to make it work by providing the .params and .orig from the initial exception but I’d discourage you to go that way because it’s error prone and a bit fragile (it will break if the exception is not a DBAPIError), instead I’d suggest to simply log the time separately from the error (using logging.warning)
I don't think I encountered this problem working on this in Python 2.7, but while debugging in 3.7, Python throws an exception when pool.close() is called. This is the relevant part of the function:
pool = multiprocessing.Pool(6)
iterator = pool.imap_unordered(worker_func, worker_input)
while True:
try:
t0, t1 = next(iterator)
except multiprocessing.TimeoutError:
continue
except StopIteration:
break
else:
dbinserts1(t0)
dbinserts2(t1)
pool.close()
pool.join()
The only change made by 2to3 was rewriting iterator.next() as next(iterator). The function only fails while debugging (in PyCharm), otherwise it runs successfully. This is (probably) the most relevant part of the stack trace:
Error in atexit._run_exitfuncs: Traceback (most recent call last):
File
"/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/util.py",
line 322, in _exit_function
p.join() File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py",
line 138, in join
assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
Which PyCharm version do you use? This seems to be fixed in 2019.1.2 by https://youtrack.jetbrains.com/issue/PY-34436
I am trying to execute an external process in Python 3.4. When my "command" is wrong the program just crashes. How can I handle the errors gracefully and recover?
# Call a system process
try:
pipe = subprocess.Popen(command,
stdout=subprocess.PIPE)
except (OSError,
subprocess.CalledProcessError) as error:
print("Failed to execute command")
print(command)
print("Error: " + error.output)
while True:
output = pipe.stdout.readline()
if not output:
print("Finished...")
break
output = output.decode()
output = output.strip("\r")
output = output.strip("\n")
print(output)
When I call an invalid command. I get a crash like:
C:\SDKs\Python34\python.exe C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\main.py
Traceback (most recent call last):
Executing: windeployqt.exe C:\Users\user\Documents\GitHub\SpaCentralSoftBin\GUIController.exe
File "C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\forms\maindialog.py", line 81, in execute_windeployqt
Failed to execute command
stdout=subprocess.PIPE)
windeployqt.exe C:\Users\user\Documents\GitHub\SpaCentralSoftBin\GUIController.exe
File "C:\SDKs\Python34\lib\subprocess.py", line 858, in __init__
restore_signals, start_new_session)
File "C:\SDKs\Python34\lib\subprocess.py", line 1111, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\forms\maindialog.py", line 159, in on_buttonBox_clicked
self.execute_windeployqt()
File "C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\forms\maindialog.py", line 86, in execute_windeployqt
print("Error: " + error.output)
AttributeError: 'FileNotFoundError' object has no attribute 'output'
The larger problem is printing an error message does not "handle" an error in any meaningful sense of "handle" (although many people think it does).
Log an error message if you want and then exit your program. There are few instances where you can handle an error, for example if a configuration file can't be read, see if you can read a global configuration file or KeyboardInterrupt can be passed up to a main loop that is reading commands.
In your example, if you didn't fault in the except block, the output = pipe.stdout... would yield:
NameError: name 'pipe' is not defined
and if you caught that, there'd still be nothing meaningful to do to recover the state your code expects to be in. You'll also move the traceback you do get to a place where the fault isn't.
A good idiom for your situation is
except SomeError:
log what you want
raise
where an empty raise re-raises the last exception which will propagate up, forming a full stacktrace which will give you a better idea of where the fault is. If you don't do this, you are discarding useful information.
FileNotFoundError' object has no attribute 'output'
Use
print("Error: " + error)
FileNotFoundErrr has these properties which might be useful:
args
characters_written
errno
filename
filename2