I've been using concurrent.futures as it has a simple interface and let user easily control the max number of threads/processes. However, it seems like concurrent.futures hides failed tasks and continue the main thread after all tasks finished/failed.
import concurrent.futures
def f(i):
return (i + 's')
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
fs = [executor.submit(f, i ) for i in range(10)]
concurrent.futures.wait(fs)
Calling f on any integer leads an TypeError. However, the whole script runs just fine and exits with code 0. Is there any way to make it throw an exception/error when any thread failed?
Or, is there a better way to limit number of threads/processes without using concurrent.futures?
concurrent.futures.wait will ensure all the tasks completed, but it doesn't check success (something return-ed) vs. failure (exception raised and not caught in worker function). To do that, you need to call .result() on each Future (which will cause it to either re-raise the exception from the task, or produce the return-ed value). There are other methods to check without actually raising in the main thread (e.g. .exception()), but .result() is the most straightforward method.
If you want to make it re-raise, the simplest approach is just to replace the wait() call with:
for fut in concurrent.futures.as_completed(fs):
fut.result()
which will process results as Futures complete, and promptly raise an Exception if one occurred. Alternatively, you continue to use wait so all tasks finish before you check for exceptions on any of them, then iterate over fs directly and call .result() on each.
There is another way to do the same with multiprocessing.Pool (for processes) or multiprocessing.pool.ThreadPool (for threads). As far as I know it rethrows any caught exceptions.
Related
My script has started many threads. It has kept count. The maximum number of threads are now running. There are more to be run. How can the script wait for any one of the running threads to end so it can safely start another one? It is using threading.Thread() to create each thread but that can be changed if there is a better module. I am using Python 3.6.x.
To create a pool of processes and pass tasks to them:
def processing_task(arg1, arg2):
...
from multiprocessing import Pool
with Pool() as worker_pool: # By default creates processes == number of CPUs
while True:
task = some_queue_implementation.get() # Some blocking method that receives tasks
worker_pool.apply_async(processing_task, task.arg1, task.arg2)
This will create child processes that will be idle until they are passed a task
Here is a snippet of code I always use when using threads.
sets a certain amount of threads;
ensures that no code coming after the context manager block executes until all threads complete; and
kills and raises an exception, if one of the child threads throws an exception.
with concurrent.futures.ThreadPoolExecutor(max_workers=5)\
as executor:
futures.append(executor.submit(function, args))
for future in concurrent.futures.as_completed(futures):
if future.exception():
for child in futures:
child.cancel()
raise future.exception()
It has kept count
I hope you are using .lock() and .acquire(), ;-)
I am using Pools to kick off worker processes in python3.6. The workers will return True or False after completion, and I was wondering what the difference is between using the AsyncResult returned object or using a callback function to check if the worker returned True or False. From my understanding the callback is called in the main process, the same place I would do the checking anyway.
#Using the AsyncResult way
def check_result(result):
if result:
#Successful do something
else:
#Failed
with Pool() as pool:
result = pool.apply_async(upload, (args, ))
check_result(result.get())
#Using callbacks
def check_result(result):
if result:
#Successful do something
def err_result(result):
#Do something
with Pool() as pool:
pool.appy_async(upload, (args,), callback=check_result, error_callback=err_result)
I see that in python3.6 they allow error_callback, so are these two bits of code equivalent? What are the pros and cons of both?
Thanks
The comparison between AsyncResult and callback is somewhat unlucky.
Note that you only have callbacks available for asynchronous methods (returning AsyncResult objects), so there is no 'versus' in this story regarding these things.
When you write check_result(result.get()), you don't pass some AsyncResult-object into check_result, but an already awaited normal result, in your case a boolean value (if not an exception). So it's not a difference between AsyncResult and callback, but between manually calling check_result on a result or registering a callback beforehand.
I see that in python3.6 they allow error_callback, so are these two bits of code equivalent? What are the pros and cons of both?
No, these two snippets are not equivalent. error_callback is an exception handler, your possible False-result won't trigger that, but an exception will.
Your result argument within err_result will be filled with an exception instance in such a case. The difference with your upper snippet is, that an exception there will blow up in your face as soon as you call result.get() and you have not enclosed it within an try-except-block.
The obvious 'pro' of an error_callback is the omitted try-except-block, the 'pro' of the regular callback also is reduced code length. Use both only for immediately returning tasks like checking and logging, to prevent blocking the thread your pool runs in.
I have a parent function which should run 2 tests on a data set.
if any of these tests fail parent function should return fail. I want to run these 2 tests asynchronously with asyncio and as soon as one of the tests failed, parent function should return fail and cancel the other test.
I'm new to asyncio and read some examples with the condition here but couldn't figure out how to write asyncio with conditions.
so far I could handle it by throwing exceptions in any test that has been failed.
here is my basic code:
async def test1(data):
# run some test on data and return true on pass and throw exception on fail
async def test2(data):
# run some test on data and return true on pass and throw exception on fail
ioloop = asyncio.get_event_loop()
tasks = [ioloop.create_task(test1(data)), ioloop.create_task(test2(data))]
finished, unfinished = ioloop.run_until_complete(asyncio.wait(tasks, return_when=asyncio.FIRST_EXCEPTION))
but I don't think it's a proper way to handle conditions.
so I want a basic example of how to create and handle conditions with ayncio.
as soon as one of the tests failed, parent function should return fail and cancel the other test.
asyncio.gather does that automatically:
loop = asyncio.get_event_loop()
tasks = [loop.create_task(test1(data)), loop.create_task(test2(data))]
try:
loop.run_until_complete(asyncio.gather(*tasks))
except FailException: # use exception raised by the task that fails
print('failed')
When any task executed in asyncio.gather raises an exception, all other tasks will be canceled using Task.cancel, and the exception will be propagated to the awaiter of gather. You don't need a Condition at all, cancellation will automatically interrupt whatever blocking operation the tasks were waiting on.
Conditions are needed when a task that is otherwise idle (or many such tasks) needs to wait for an event that can happen in some other task. In that case it waits on a condition and is notified of it occurring. If the task is just going about its business, you can cancel it any time you like, or let functions like asyncio.gather or asyncio.wait_for do it for you.
I have a problem with silently passing exceptions in tornado while using futures in situations, when I am not explicitly waiting result from future coroutines (yield some_future_obj), such as infinite loop coroutine:
#gen.coroutine
def base_func():
#gen.coroutine
def tail_something():
raise
while True:
yield some_other_coroutine
base_func()
I have also noticed that this topic was already discussed: refer to here or here.
The problem is that if we don't wait for future completion explicitly, future.result() never be called and exception will never be raised. But tornado.concurrent is committed to use concurrent.futures package.
Now I just hang ioloop.add_future on current loop and simply execute log.exception(future.result()). But I don't like this approach, since it is a bit noisy (redundant lines in production code).
Please, contribute your ideas or may be a real answer.
The reason Futures "hide" exceptions is that you have to decide where you want the exception to show up. If you want to be able to handle the exception in your code, you must access its result somewhere (which in a Tornado coroutine means yielding it). If you just want to log the exception, you can ask the IOLoop to do it for you:
IOLoop.instance().add_future(fut, lambda fut: fut.result())
Note that I'm just calling result() instead of logging its value. This ensures that we don't log anything when there is no error, and the exception (with traceback) is logged by IOLoop's normal unhandled-exception machinery.
Let's say if we have a main thread which launches two threads for test modules - " test_a" and " test_b".
Both the test module threads maintain their state whether they are done performing test or if they encountered any error, warning or if they want to update some other information.
How main thread can get access to this information and act accordingly.
For example, if " test_a" raised an error flag; How "main" will know and stop rest of the tests before existing with error ?
One way to do this is using global variables but that gets very ugly.. Very soon.
The obvious solution is to share some kind of mutable variable, by passing it in to the thread objects/functions at constructor/start.
The clean way to do this is to build a class with appropriate instance attributes. If you're using a threading.Thread subclass, instead of just a thread function, you can usually use the subclass itself as the place to stick those attributes. But I'll show it with a list just because it's shorter:
def test_a_func(thread_state):
# ...
thread_state[0] = my_error_state
# ...
def main_thread():
test_states = [None]
test_a = threading.Thread(target=test_a_func, args=(test_states,))
test_a.start()
You can (and usually want to) also pack a Lock or Condition into the mutable state object, so you can properly synchronize between main_thread and test_a.
(Another option is to use a queue.Queue, an os.pipe, etc. to pass information around, but you still need to get that queue or pipe to the child thread—which you do in the exact same way as above.)
However, it's worth considering whether you really need to do this. If you think of test_a and test_b as "jobs", rather than "thread functions", you can just execute those jobs on a pool, and let the pool handle passing results or errors back.
For example:
try:
with concurrent.futures.ThreadPoolExecutor(workers=2) as executor:
tests = [executor.submit(job) for job in (test_a, test_b)]
for test in concurrent.futures.as_completed(tests):
result = test.result()
except Exception as e:
# do stuff
Now, if the test_a function raises an exception, the main thread will get that exception—and, because that means exiting the with block, and all of the other jobs get cancelled and thrown away, and the worker threads shut down.
If you're using 2.5-3.1, you don't have concurrent.futures built in, but you can install the backport off PyPI, or you can rewrite things around multiprocessing.dummy.Pool. (It's slightly more complicated that way, because you have to create a sequence of jobs and call map_async to get back an iterator over AsyncResult objects… but really that's still pretty simple.)