We Keep Coding

Set function timeout without having to use contextlib [duplicate]

I looked online and found some SO discussing and ActiveState recipes for running some code with a timeout. It looks there are some common approaches:
Use thread that run the code, and join it with timeout. If timeout elapsed - kill the thread. This is not directly supported in Python (used private _Thread__stop function) so it is bad practice
Use signal.SIGALRM - but this approach not working on Windows!
Use subprocess with timeout - but this is too heavy - what if I want to start interruptible task often, I don't want fire process for each!
So, what is the right way? I'm not asking about workarounds (eg use Twisted and async IO), but actual way to solve actual problem - I have some function and I want to run it only with some timeout. If timeout elapsed, I want control back. And I want it to work on Linux and Windows.

A completely general solution to this really, honestly does not exist. You have to use the right solution for a given domain.
If you want timeouts for code you fully control, you have to write it to cooperate. Such code has to be able to break up into little chunks in some way, as in an event-driven system. You can also do this by threading if you can ensure nothing will hold a lock too long, but handling locks right is actually pretty hard.
If you want timeouts because you're afraid code is out of control (for example, if you're afraid the user will ask your calculator to compute 9**(9**9)), you need to run it in another process. This is the only easy way to sufficiently isolate it. Running it in your event system or even a different thread will not be enough. It is also possible to break things up into little chunks similar to the other solution, but requires very careful handling and usually isn't worth it; in any event, that doesn't allow you to do the same exact thing as just running the Python code.

What you might be looking for is the multiprocessing module. If subprocess is too heavy, then this may not suit your needs either.
import time
import multiprocessing
def do_this_other_thing_that_may_take_too_long(duration):
time.sleep(duration)
return 'done after sleeping {0} seconds.'.format(duration)
pool = multiprocessing.Pool(1)
print 'starting....'
res = pool.apply_async(do_this_other_thing_that_may_take_too_long, [8])
for timeout in range(1, 10):
try:
print '{0}: {1}'.format(duration, res.get(timeout))
except multiprocessing.TimeoutError:
print '{0}: timed out'.format(duration)
print 'end'

If it's network related you could try:
import socket
socket.setdefaulttimeout(number)

I found this with eventlet library:
http://eventlet.net/doc/modules/timeout.html
from eventlet.timeout import Timeout
timeout = Timeout(seconds, exception)
try:
... # execution here is limited by timeout
finally:
timeout.cancel()

For "normal" Python code, that doesn't linger prolongued times in C extensions or I/O waits, you can achieve your goal by setting a trace function with sys.settrace() that aborts the running code when the timeout is reached.
Whether that is sufficient or not depends on how co-operating or malicious the code you run is. If it's well-behaved, a tracing function is sufficient.

An other way is to use faulthandler:
import time
import faulthandler
faulthandler.enable()
try:
faulthandler.dump_tracebacks_later(3)
time.sleep(10)
finally:
faulthandler.cancel_dump_tracebacks_later()
N.B: The faulthandler module is part of stdlib in python3.3.

If you're running code that you expect to die after a set time, then you should write it properly so that there aren't any negative effects on shutdown, no matter if its a thread or a subprocess. A command pattern with undo would be useful here.
So, it really depends on what the thread is doing when you kill it. If its just crunching numbers who cares if you kill it. If its interacting with the filesystem and you kill it , then maybe you should really rethink your strategy.
What is supported in Python when it comes to threads? Daemon threads and joins. Why does python let the main thread exit if you've joined a daemon while its still active? Because its understood that someone using daemon threads will (hopefully) write the code in a way that it wont matter when that thread dies. Giving a timeout to a join and then letting main die, and thus taking any daemon threads with it, is perfectly acceptable in this context.

I've solved that in that way:
For me is worked great (in windows and not heavy at all) I'am hope it was useful for someone)
import threading
import time
class LongFunctionInside(object):
lock_state = threading.Lock()
working = False
def long_function(self, timeout):
self.working = True
timeout_work = threading.Thread(name="thread_name", target=self.work_time, args=(timeout,))
timeout_work.setDaemon(True)
timeout_work.start()
while True: # endless/long work
time.sleep(0.1) # in this rate the CPU is almost not used
if not self.working: # if state is working == true still working
break
self.set_state(True)
def work_time(self, sleep_time): # thread function that just sleeping specified time,
# in wake up it asking if function still working if it does set the secured variable work to false
time.sleep(sleep_time)
if self.working:
self.set_state(False)
def set_state(self, state): # secured state change
while True:
self.lock_state.acquire()
try:
self.working = state
break
finally:
self.lock_state.release()
lw = LongFunctionInside()
lw.long_function(10)
The main idea is to create a thread that will just sleep in parallel to "long work" and in wake up (after timeout) change the secured variable state, the long function checking the secured variable during its work.
I'm pretty new in Python programming, so if that solution has a fundamental errors, like resources, timing, deadlocks problems , please response)).

solving with the 'with' construct and merging solution from -
Timeout function if it takes too long to finish
this thread which work better.
import threading, time
class Exception_TIMEOUT(Exception):
pass
class linwintimeout:
def __init__(self, f, seconds=1.0, error_message='Timeout'):
self.seconds = seconds
self.thread = threading.Thread(target=f)
self.thread.daemon = True
self.error_message = error_message
def handle_timeout(self):
raise Exception_TIMEOUT(self.error_message)
def __enter__(self):
try:
self.thread.start()
self.thread.join(self.seconds)
except Exception, te:
raise te
def __exit__(self, type, value, traceback):
if self.thread.is_alive():
return self.handle_timeout()
def function():
while True:
print "keep printing ...", time.sleep(1)
try:
with linwintimeout(function, seconds=5.0, error_message='exceeded timeout of %s seconds' % 5.0):
pass
except Exception_TIMEOUT, e:
print " attention !! execeeded timeout, giving up ... %s " % e

Replace 'While-True'-Loop with something more efficient

Problem
It's very common for beginners to solve IO waiting while concurrent processing in an similar way like here:
#!/usr/bin/env python3
"""Loop example."""
from time import sleep
WAITING: bool = True
COUNTER: int = 10
def process() -> None:
"""Non-blocking routine, that needs to be invoked periodically."""
global COUNTER # pylint: disable=global-statement
print(f"Done in {COUNTER}.")
COUNTER -= 1
sleep(1)
# Mimicking incoming IO callback
if COUNTER <= 0:
event()
def event() -> None:
"""Incoming IO callback routine."""
global WAITING # pylint: disable=global-statement
WAITING = False
try:
while WAITING:
process()
except KeyboardInterrupt:
print("Canceled.")
Possible applications might be servers, what are listening for incomming messages, while still processing some other internal stuff.
Possible Solution 1
Threading might in some cases a good solution.
But after some research it seems that threading adds a lot of overheading for the communcation between the threads.
One example for this might be the 'Warning' in the osc4py3 package documentation below the headline 'No thread'.
Also i have read somewhere the thumb rule, that 'Threading suits not for slow IO' (sorry, lost the source of this rule).
Possible Solution 2
Asynchronous processing (with the asyncio package) might be another solution.
Especially because the ominous thumb rule also says that 'For slow IO is asyncio efficient'.
What i tried
So i tried to rewrite this example with asyncio but failed completely, even after reading about Tasks, Futures and Awaitables in general in the Python asyncio documentation.
My problem was to solve the perodically (instead of one time) call while waiting.
Of course there are infinite loops possible, but all examples i found in the internet are still using 'While-True'-Loops what does not look like an improvement to me.
For example this snippet:
import asyncio
async def work():
while True:
await asyncio.sleep(1)
print("Task Executed")
loop = asyncio.get_event_loop()
try:
asyncio.ensure_future(work())
loop.run_forever()
except KeyboardInterrupt:
pass
finally:
print("Closing Loop")
loop.close()
Source: https://tutorialedge.net/python/concurrency/asyncio-event-loops-tutorial/#the-run_forever-method
What i want
To know the most elegant and efficient way of rewriting these stupid general 'While-True'-Loop from my first example code.
If my 'While-True'-Loop is still the best way to solve it (beside my global variables), then it's also okay to me.
I just want to improve my code, if possible.

What you describe is some kind of polling operation and is similar to busy waiting. You should rarely rely on those methods as they can incur a serious performance penalty if used incorrectly. Instead, you should rely on concurrency primitives provided by the OS of a concurrency library.
As said in a comment, you could rely on a condition or an event (and more broadly on mutexes) to schedule some come to run after an event occurs. For I/O operations you can also rely on low-level OS facilities such as select, poll and signals/interruptions.
Possible applications might be servers, what are listening for
incomming messages, while still processing some other internal stuff.
For such use cases you should really use a dedicated library to do that efficiently. For instance, here is an example of a minimal server developed with AsyncIO's low-level socket operations. Internally, AsyncIO probably uses the select system call and exposes a friendly interface with async-await.

Solution with asyncio:
#!/usr/bin/env python3
"""Asyncronous loop example."""
from typing import Callable
from asyncio import Event, get_event_loop
DONE = Event()
def callback():
"""Incoming IO callback routine."""
DONE.set()
def process():
"""Non-blocking routine, that needs to be invoked periodically."""
print('Test.')
try:
loop = get_event_loop()
run: Callable = lambda loop, processing: (
processing(),
loop.call_soon(run, loop, processing)
)
loop.call_soon(run, loop, process)
loop.call_later(1, callback) # Mimicking incoming IO callback after 1 sec
loop.run_until_complete(DONE.wait())
except KeyboardInterrupt:
print("Canceled.")
finally:
loop.close()
print("Bye.")

Auto-generate full stack trace to STDERR on thread crash

Problem
I have a library which I must use, written in Python 2.7. There are several bugs in it, and one of them occasionally causes the calling thread to crash (rarely, however). I would like to generate the stack trace so I can determine which thread is dying when the library crashes. I get a trace dumped to STDERR of what went wrong in the library, i.e.:
A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.
I've also tried GDB (which works wonders for my C/C++ projects), using a guide I found on StackOverflow to get "Python plus GDB" working (so I can attach to a running Python application). However, I don't see anything helpful that relates to the (now dead) thread.
Question
Is it possible, in Python 2.7, to force a thread (when it crashes) to report a full stack trace to STDOUT, STDERR, or a log file, when this sort of issue (i.e. a library call crashing the calling thread) occurs?
Thank you.

If you have access to the thread definition -- you can write a wrapper thread
import logger
log = logger.getLogger(__name__)
class WrapperThread(threading.Thread):
def __init__(self, innerThread):
self.innerThread = innerThread
def start(self):
try:
# run the thread in the current context with run.
self.innerThread.run()
except Exception as e:
log.error("%s has crashed.", self, exc_info=True) #Exec info makes loggin print the stack trace.
Depending on the library you are using you may be able to apply a decorator to the thread definition. Though I don't recommend code like this ever being included in released code
import threading
import logging
logging.basicConfig()
def loggingThread(clazz, logger):
class _Thread(clazz):
def __init__(self, *args, **kwargs):
clazz.__init__(self, *args, **kwargs)
def run(self):
try:
clazz.run(self)
except Exception as e:
logger.error("%s has Crashed!", self, exc_info=True)
return _Thread
threading.Thread = loggingThread(threading.Thread, logging)
import random
def ohNo(range1, range2):
for x in xrange(1, range1):
if x % random.randint(1, range2) == 0:
raise ValueError("Oh no. %d is an illeagal value!" % (x,))
test = threading.Thread(target=ohNo, args=(500,100))
test.start()

Catching exception from timer

I am trying to create a watchdog class, that will throw an exception after specified time:
from threading import Timer
from time import sleep
class watchdog():
def _timeout(self):
#raise self
raise TypeError
def __init__(self):
self.t = Timer(1, self._timeout)
def start(self):
self.t.start()
try:
w = watchdog()
w.start()
sleep(2)
except TypeError, e:
print "Exception caught"
else:
print "Of course I didn't catch the exception"
This exception is not caught, as the exception is thrown from completely different context, hence we will see the last message.
My question is, how can I modify the code, so the exception will be caught?

This is not possible, as you suggested, and there is no api for abruptly stopping a thread, either, which rules out other potential solutions.
I believe your best solution is to let the watchdog set a flag, and let the test read it at certain points. Similarly, your test can simply check the duration from time to time.
Note that if the "flag" would set in a way that will cause the main thread to raise an exception (for example, deleting attributes from objects), it'll be just as effective.
The other possibility is to use multiprocessing instead of multythreading, if it is possible for your application.

How to force timeout functions in python, windows platform

All I want to do is timeout a function if it does not return before that
It all started because urllib2 supports timeout for urlopen, but not for reading part
and my program hangs. Changing defaulttimeout for sockets does not work. Using signal.sigalrm
does not work. I can't switch to requests because then I will have to rewrite and test a lot more.
I DON'T want to make a thread run the function and then timeout the thread, I want to timeout the function. Any ideas how?

I like to use David's class here in my projects. I find it's very effective and I like that it provides a simple way to implement in existing code via a decorator. For example:
# Timeout after 30 seconds
#timeout(30)
def your_function():
...
CAUTION: This is not thread-safe! If you're using multithreading, the signal will get caught by a random thread. For single-threaded programs however, this is the easiest solution.

Yes, it can be done in windows without signal and it will also work in other os as well. This is using thread but not to run the function but to raise a signal for timeout. The logic is to create a new thread and wait for a given time and raise an exception using _thread(in python3 and thread in python2). This exception will be thrown in the main thread and the with block will get exit if any exception occurs.
import threading
import _thread # import thread in python2
class timeout():
def __init__(self, time):
self.time= time
self.exit=False
def __enter__(self):
threading.Thread(target=self.callme).start()
def callme(self):
time.sleep(self.time)
if self.exit==False:
_thread.interrupt_main() # use thread instead of _thread in python2
def __exit__(self, a, b, c):
self.exit=True
Usuage Example :-
with timeout(2):
func()
The program in the with block should exit within 2 seconds otherise it will be exited after 2 seconds.