We Keep Coding

Python ProcessPoolExecutor cannot handle exception

I am executing a Tornado server with ProcessPoolExecutor to handle multiple requests in parallel.
The problem is that, in one particular case, when an exception is raised in one of the processes it doesn't propagates, but instead the process crashes with this error:
concurrent.futures.process._RemoteTraceback:
\n'''\nTraceback (most recent call last):
\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\concurrent\\futures\\process.py\", line 367, in _queue_management_worker\n result_item = result_reader.recv()
\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\multiprocessing\\connection.py\", line 251, in recv
\n return _ForkingPickler.loads(buf.getbuffer())\nTypeError: __init__() missing 1 required positional argument: 'is_local'\n'''\n\nThe above exception was the direct cause of the following exception:
\n
\nTraceback (most recent call last):\n File \"C:\\S1\\Product\\Baseline\\PYTHON\\lab\\controller.py\", line 558, in get\n output = exec_future.result()
\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\concurrent\\futures\\_base.py\", line 428, in result\n return self.__get_result()\n File \"C:\\Users\\ActionICT\\anaconda3\\lib\\concurrent\\futures\\_base.py\", line 384, in __get_result
\n raise self._exception\nconcurrent.futures.process.BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.\n
I have tried it in debugger, and found that the problem is executing this
def _send_bytes(self, buf):
ov, err = _winapi.WriteFile(self._handle, buf, overlapped=True)
try:
if err == _winapi.ERROR_IO_PENDING:
waitres = _winapi.WaitForMultipleObjects(
[ov.event], False, INFINITE)
assert waitres == WAIT_OBJECT_0
except:
ov.cancel()
raise
finally:
nwritten, err = ov.GetOverlappedResult(True)
assert err == 0
assert nwritten == len(buf)
This is called when the process tries to propagate the exception to the corresponding Future object.
In the first line, when calling _winapi.WriteFile, everything crashes in debugger, and I can't understand why. Any idea?

I have resolved with a workaround: I have wrapped internally the function inside separate process in try except, then copied the old exception in a new exception and raised it. I don't know why... but it works.
def _execute_tuning(tune_parameters: TuneParameters):
# function to parallelize todo to be refactored
# execute scenario, then write result or error in output
try:
config.generate_project_config(
project_name=tune_parameters.project_name,
scenario_name=tune_parameters.scenario_name
)
config.generate_session_log_config(project_name=tune_parameters.project_name,
scenario_name=tune_parameters.scenario_name)
tree = DecisionTreeGenerator(tune_parameters.project_name, tune_parameters.scenario_name)
tree.fit(
# todo refactor
auto_tune=True if tune_parameters == 'true' else False,
max_depth=tune_parameters.max_depth,
columns=tune_parameters.columns,
min_samples_leaf=tune_parameters.min_samples_per_leaf,
max_leaf_nodes=tune_parameters.max_leaf_nodes
)
kpi = KPICalculator(tune_parameters.project_name, tune_parameters.scenario_name)
kpi.run(do_optimization_kpi=False)
except Exception as exc:
Loggers.application.exception(exc)
exc_final = Exception(str(exc))
exc_final.__traceback__ = exc.__traceback__
raise exc_final

Python program with psutil fails after 8 hours of running with PermissionError: [WinError 5] Access is denied - why?

I have a Python 3.x program running in Windows that uses psutil. The job of this program is simple - to periodically check if a different program is running. If it finds the program, it does nothing. If it doesn't find the program, it starts the program. Then the program sleeps and does the same thing again in a loop. Here's a minimal version of the code:
import psutil, os, time
found = 0
theprogram = 'hiimaprogram.exe'
while True:
for process in psutil.process_iter():
if process.name() == theprogram: # this is line 14
found = 1
print ('process found, doing nothing')
if found == 0:
print ('process not found, starting the process')
filepath = theprogram
os.startfile(filepath)
found = 0
time.sleep(10)
print ('sleeping')
continue
The code works great and does the job nicely. However, it also fails randomly, never straight away, usually at the 8 hour mark although sometimes it'll survive just a little longer. Here's the traceback:
Traceback (most recent call last):
File "site-packages\psutil\_pswindows.py", line 707, in wrapper
File "site-packages\psutil\_pswindows.py", line 791, in exe
PermissionError: [WinError 5] Access is denied
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages\psutil\_pswindows.py", line 782, in name
File "site-packages\psutil\_pswindows.py", line 709, in wrapper
psutil.AccessDenied: psutil.AccessDenied (pid=31216)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "site-packages\psutil\_pswindows.py", line 707, in wrapper
File "site-packages\psutil\_pswindows.py", line 784, in name
ProcessLookupError: [Errno 3] No such process
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "programchecker.py", line 14, in <module>
File "site-packages\psutil\__init__.py", line 731, in name
File "site-packages\psutil\_pswindows.py", line 709, in wrapper
psutil.NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=31216)
Line 14 where the process dies is marked in the code. Easy enough for me to write a try/except to ignore the error, but I'd really like to know why it is happening. Try/except code:
while True:
for process in psutil.process_iter():
try:
if process.name() == theprogram: # this is line 14
found = 1
print ('process found')
except:
print ('error')
time.sleep(100)
continue
Reading up on psutil and other people who have had the same error (but for different reasons) all I can find that seems to fit my situation is that I need to be in administrator mode, but running the program in administrator mode appears to make no difference at all and still shows the same error. The other thing I thought of is perhaps some automatic windows process is interfering at the eight hour mark due to sleep/hibernation but I've made sure that the computer is on an "always on" power scheme where nothing should be interfering (at least in theory). Any help appreciated, thank you!

I found the solution to this, which is completely ludicrous, but here it is.
The actual program that my program is checking for the existence of, has a "security feature" built into it where it force-times itself out after the 8 hour mark, even if the program is active and being used. This is direct from the devs of that program. Insane, I know. Adding the try/except above fixes the issue, the sleep timeout of 100 is sufficient window for the master program to stop caring and reset the clock.

Catching the exception tracebacks in a loop and then raising the error(s) at the end of the script

I'm trying to catch all exception error(s) and then at the end of the script for it to raise/show all tracebacks...
I have a main script that calls my subscripts for example:
errors = open('MISC/ERROR(S).txt', 'a')
try:
execfile("SUBSCRIPTS/Test1.py", {})
except Exception:
## Spread over two calls of errors.write for readability in code...
errors.write(strftime('%d/%m/%Y %H:%M:%S') + "\n")
errors.write(traceback.format_exc() + '\n\n')
try:
execfile("SUBSCRIPTS/Test2.py", {})
except Exception:
## Spread over two calls of errors.write for readability in code...
errors.write(strftime('%d/%m/%Y %H:%M:%S') + "\n")
errors.write(traceback.format_exc() + '\n\n')
errors.close()
This script uses the traceback module to retrieve the error from the script...
In the next example this is what my current script sort of looks like:
for x in y:
if "example" in x:
for tweet in tweetlist:
# Try
try:
twitter.update_status(status=tweet)
# Do some other stuff here if it suceeeds like...
print "oo example worked"
# Damn it failed, grab the whole traceback?
except Exception as reason:
FailedTweet = True
# Do some other stuff here like...
print "I did other stuff"
if FailedTweet:
print reason # Printing the reason because I don't know how to throw the exception error (full error)
Basically there is a big loop where it could potentially go wrong around the line: twitter.update_status(status=tweet) and if it does I want it to catch the traceback error(s) could be more than one because it's in a loop and then when the script finishes I want it to send all the traceback errors back to the main script so it can write them all to the error file.
Example of the error writing to file from the first bit of code:
# 17/08/2014 12:30:00
# Traceback (most recent call last):
# File "C:\Main.py", line 117, in execute_subscripts
# execfile("SUBSCRIPTS/Test1.py", {})
# File "SUBSCRIPTS/Test1.py", line 440, in <module>
# twitter.update_status(status=string)
# File "C:\Python27\lib\site-packages\twython\endpoints.py", line 90, in update_status
# return self.post('statuses/update', params=params)
# File "C:\Python27\lib\site-packages\twython\api.py", line 234, in post
# return self.request(endpoint, 'POST', params=params, version=version)
# File "C:\Python27\lib\site-packages\twython\api.py", line 224, in request
# content = self._request(url, method=method, params=params, api_call=url)
# File "C:\Python27\lib\site-packages\twython\api.py", line 194, in _request
# retry_after=response.headers.get('retry-after'))
# TwythonError: Twitter API returned a 403 (Forbidden), This request looks like it might be automated. To protect our users from spam and other malicious activity, we can't complete this action right now. Please try again later.
How would I achieve this, it's kind of hard to explain so if something doesn't make sense please ask.

Just save the traceback data in a list, and then print the contents of the list after the loop is complete.
import traceback
reasons = []
for x in y:
if "example" in x:
for tweet in tweetlist:
# Try
try:
twitter.update_status(status=tweet)
# Do some other stuff here if it suceeeds like...
print "oo example worked"
# Damn it failed, grab the whole traceback?
except Exception:
reasons.append(traceback.format_exc())
# Do some other stuff here like...
print "I did other stuff"
for reason in reasons:
print reason
# If you want to raise a single exception that shows the traceback for
# each exception, you can do this:
class ChainedException(Exception):
def __init__(self, msg):
msg = "The following exceptions occurred:\n\n{}".format(msg)
if reasons:
raise ChainedException('\n'.join(reasons))
Example usage of ChainedException:
reasons = []
for i in range(5):
try:
raise Exception("Blah {}".format(i))
except Exception:
reasons.append(traceback.format_exc())
if reasons:
raise ChainedException("\n".join(reasons))
Output:
Traceback (most recent call last):
File "ok.py", line 17, in <module>
raise ChainedException("\n".join(reasons))
__main__.ChainedException: The following exceptions occurred:
Traceback (most recent call last):
File "ok.py", line 12, in <module>
raise Exception("Blah {}".format(i))
Exception: Blah 0
Traceback (most recent call last):
File "ok.py", line 12, in <module>
raise Exception("Blah {}".format(i))
Exception: Blah 1
Traceback (most recent call last):
File "ok.py", line 12, in <module>
raise Exception("Blah {}".format(i))
Exception: Blah 2
Traceback (most recent call last):
File "ok.py", line 12, in <module>
raise Exception("Blah {}".format(i))
Exception: Blah 3
Traceback (most recent call last):
File "ok.py", line 12, in <module>
raise Exception("Blah {}".format(i))
Exception: Blah 4
Edit:
If you really only care about raising a single exception out of the entire list of exceptions, you can do it like this:
import traceback
reason = None
for x in y:
if "example" in x:
for tweet in tweetlist:
# Try
try:
twitter.update_status(status=tweet)
# Do some other stuff here if it suceeeds like...
print "oo example worked"
# Damn it failed, grab the whole traceback?
except Exception:
reason = sys.exc_info() # We're not putting it in a list because you only care about one.
# Do some other stuff here like...
print "I did other stuff"
if reason:
raise reason[0], reason[1], reason[2]
Note that this only works in Python 2.x. If you were using Python 3.x, you would need to use this:
if reason:
raise reason[1].with_traceback(reason[2])

Calling a subprocess in Python 3.4 and handling errors if a command is invalid?

I am trying to execute an external process in Python 3.4. When my "command" is wrong the program just crashes. How can I handle the errors gracefully and recover?
# Call a system process
try:
pipe = subprocess.Popen(command,
stdout=subprocess.PIPE)
except (OSError,
subprocess.CalledProcessError) as error:
print("Failed to execute command")
print(command)
print("Error: " + error.output)
while True:
output = pipe.stdout.readline()
if not output:
print("Finished...")
break
output = output.decode()
output = output.strip("\r")
output = output.strip("\n")
print(output)
When I call an invalid command. I get a crash like:
C:\SDKs\Python34\python.exe C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\main.py
Traceback (most recent call last):
Executing: windeployqt.exe C:\Users\user\Documents\GitHub\SpaCentralSoftBin\GUIController.exe
File "C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\forms\maindialog.py", line 81, in execute_windeployqt
Failed to execute command
stdout=subprocess.PIPE)
windeployqt.exe C:\Users\user\Documents\GitHub\SpaCentralSoftBin\GUIController.exe
File "C:\SDKs\Python34\lib\subprocess.py", line 858, in __init__
restore_signals, start_new_session)
File "C:\SDKs\Python34\lib\subprocess.py", line 1111, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\forms\maindialog.py", line 159, in on_buttonBox_clicked
self.execute_windeployqt()
File "C:\Users\user\Documents\GitHub\PyQtApps\QtDeploy\src\forms\maindialog.py", line 86, in execute_windeployqt
print("Error: " + error.output)
AttributeError: 'FileNotFoundError' object has no attribute 'output'

The larger problem is printing an error message does not "handle" an error in any meaningful sense of "handle" (although many people think it does).
Log an error message if you want and then exit your program. There are few instances where you can handle an error, for example if a configuration file can't be read, see if you can read a global configuration file or KeyboardInterrupt can be passed up to a main loop that is reading commands.
In your example, if you didn't fault in the except block, the output = pipe.stdout... would yield:
NameError: name 'pipe' is not defined
and if you caught that, there'd still be nothing meaningful to do to recover the state your code expects to be in. You'll also move the traceback you do get to a place where the fault isn't.
A good idiom for your situation is
except SomeError:
log what you want
raise
where an empty raise re-raises the last exception which will propagate up, forming a full stacktrace which will give you a better idea of where the fault is. If you don't do this, you are discarding useful information.

FileNotFoundError' object has no attribute 'output'
Use
print("Error: " + error)
FileNotFoundErrr has these properties which might be useful:
args
characters_written
errno
filename
filename2

How can I get better error information with try/except in Python

Consider this try/except block I use for checking error message stored in e.
Try/Catch to get the e
queryString = "SELECT * FROM benchmark WHERE NOC = 2"
try:
res = db.query(queryString)
except SQLiteError, e:
# `e` has the error info
print `e`
The e object here contains nothing more than the above string. When python reports an unhandled error, however, it shows a pretty detailed info as below:
Traceback (most recent call last):
File "fool.py", line 1, in
open("abc.zyz", "r")
IOError: [Errno 2] No such file or directory: 'abc.zyz'
My question is, how can I get the information such as above (the file and the line number etc.)? Or, if e contains this info, how is it stored inside it?

This will show the trace to the error.
import traceback
try:
res = db.query(queryString)
except SQLiteError, e:
# `e` has the error info
print `e`
for tb in traceback.format_tb(sys.exc_info()[2]):
print tb

Like the first 2 answers, use traceback. Here is a more complete example.
import traceback
def foo():
raise RuntimeError('we have a problem')
try:
foo()
except:
traceback.print_exc()
When you run it, you'll see
Traceback (most recent call last):
File "C:\0\tmp\x.py", line 6, in <module>
foo()
File "C:\0\tmp\x.py", line 3, in foo
raise RuntimeError('we have a problem')
RuntimeError: we have a problem

See the traceback library.
If you want to just pass errors up the chain instead of modifying them, you can just use raise within an except block, which will then act like the except block isn't there (aside from any conditional logic/side effects you may have done before the raise).