We Keep Coding

How can control be returned from a signal handler function that has been defined within a function?

I'm writing a small, single function that is designed to request user input with a time delay. When the time delay runs out, the function should return None instead of a user's response and then should continue with the rest of the script.
In the current implementation, the user input works and the timeout works, with the timeout message printed by a signal handler function which is defined within the function (I'm aiming to have this outer function fairly self-contained). However, processing then halts (rather than exiting the while loop defined in the main function) and I'm not sure why.
How can I get processing to continue? Am I misusing signal in some way? Could a lambda be used in place of an explicitly-defined function for the handler function?
#!/usr/bin/env python
from __future__ import print_function
import signal
import propyte
def main():
response = "yes"
while response is not None:
response = get_input_nonblocking(
prompt = "ohai? ",
timeout = 5
)
print("start non-response procedures")
# do things
def get_input_nonblocking(
prompt = "",
timeout = 5,
message_timeout = "prompt timeout"
):
def timeout_manager(signum, frame):
print(message_timeout)
#signal.signal(signal.SIGALRM, lambda: print(message_timeout))
signal.signal(signal.SIGALRM, timeout_manager)
signal.alarm(timeout)
try:
response = propyte.get_input(prompt)
return response
except:
return None
if __name__ == '__main__':
main()

What you've got is almost there, but you need to raise an exception inside your signal handler. raw_input will block until something happens, either input or an exception. If you raise an exception in the signal handler, that will then interrupt raw_input and execution will fall into the except in your get_input_non_blocking function. Here's a toy example.
import signal
def timeout(signum, frame):
raise IOError("bye!")
signal.signal(signal.SIGALRM, timeout)
def input():
try:
print("omgplz: ")
return raw_input()
except IOError:
return None
signal.alarm(5)
txt = input()
signal.alarm(0)
print(txt)
There's some more discussion and an alternative approach using select in this answer here: Keyboard input with timeout in Python
Hope that helps!

how to close python script without just activating "except" in try/except statement

I have a python script running over some 50,000 items in a database, it takes about 10 seconds per item and I want to stop it.
But, if i just press ctrl + C while the code is in the try except part, then my program enter the expect statement and delete that item then continue. The only way to close the program is to repeatedly delete items this way until I, just be sheer luck, catch it not in the try statement.
How do I exit the script without just killing a try statement?
EDIT 1
I have a statement that looks something like this:
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except:
data = (item['id'])
db.delete('...')
EDIT 2
The top of my connect to db code looks like:
#!/usr/bin/python
import MySQLdb
import sys
class Database:
....

The issue is because you are using a blanket except which is always a bad idea, if you catch specific exceptions then when you KeyboardInterrupt your script will stop:
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except MySQLdb.Error as e:
print(e)
data = (item['id'])
db.delete('...')
If you have other exceptions to catch you can use multiple in the except:
except (KeyError, MySQLdb.Error) as e
At the very least you could catch Exception as e and print the error.
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except Exception as e:
print(e)
data = (item['id'])
db.delete('...')
The moral of the story is don't use a blanket except, catch what you expect and logging the errors might also be a good idea. The exception-hierarchy is also worth reading to see exactly what you are catching.

I would rewrite it like this:
try:
# ...
except KeyboardInterrupt:
sys.exit(1)
except Exception: # I would use a more specific exception here if you can.
data = (item['id'])
db.delete('...')
Or just
try:
# ...
except Exception: # I would use an even more specific exception here if you can.
data = (item['id'])
db.delete('...')
Python 2's exception hiearchy can be found here: https://docs.python.org/2/library/exceptions.html#exception-hierarchy
Also, many scripts are written with something like this boiler plate at the bottom:
def _main(args):
...
if __name__ == '__main__':
try:
sys.exit(_main(sys.argv[1:]))
except KeyboardInterrupt:
print >> sys.stderr, '%s: interrupted' % _SCRIPT_NAME
sys.exit(1)
When you use except: instead of except Exception: in your main body, you are not allowing this top level exception block to catch the KeyboardInterrupt. (You don't have to actually catch it for Ctrl-C to work -- this just makes it prettier when KeyboardInterrupt is raised.)

import signal
import MySQLdb
import sys
## Handle process signals:
def sig_handler(signal, frame):
global connection
global cursor
cursor.close()
connection.close()
exit(0)
## Register a signal to the handler:
signal.signal(signal.SIGINT, sig_handler)
class Database:
# Your class stuf here.
# Note that in sig_handler() i close the cursor and connection,
# perhaps your class has a .close call? If so use that.
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except MySQLdb.Error as e:
data = (item['id'])
db.delete('...')
This would register and catch for instance Ctrl+c at any given time. The handler would terminate a socket accordingly, then exit with exit code 0 which is a good exit code.
This is a more "global" approach to catching keyboard interrupts.

Error when using twisted and greenlets

I'm trying to use twisted with greenlets, so I can write synchronous looking code in twisted without using inlineCallbacks.
Here is my code:
import time, functools
from twisted.internet import reactor, threads
from twisted.internet.defer import Deferred
from functools import wraps
import greenlet
def make_async(func):
#wraps(func)
def wrapper(*pos, **kwds):
d = Deferred()
def greenlet_func():
try:
rc = func(*pos, **kwds)
d.callback(rc)
except Exception, ex:
print ex
d.errback(ex)
g = greenlet.greenlet(greenlet_func)
g.switch()
return d
return wrapper
def sleep(t):
print "sleep(): greenelet:", greenlet.getcurrent()
g = greenlet.getcurrent()
reactor.callLater(t, g.switch)
g.parent.switch()
def wait_one(d):
print "wait_one(): greenelet:", greenlet.getcurrent()
g = greenlet.getcurrent()
active = True
def callback(result):
if not active:
g.switch(result)
else:
reactor.callLater(0, g.switch, result)
def errback(failure):
if not active:
g.throw(failure)
else:
reactor.callLater(0, g.throw, failure)
d.addCallback(callback)
d.addErrback(errback)
active = False
rc = g.parent.switch()
return rc
#make_async
def inner():
print "inner(): greenelet:", greenlet.getcurrent()
import random, time
interval = random.random()
print "Sleeping for %s seconds..." % interval
sleep(interval)
print "done"
return interval
#make_async
def outer():
print "outer(): greenelet:", greenlet.getcurrent()
print wait_one(inner())
print "Here"
reactor.callLater(0, outer)
reactor.run()
There are 5 main parts:
A sleep function, that starts a timer, then switches back to the parent greenlet. When the timer goes off, it switches back to the greenlet that is sleeping.
A make_async decorator. This takes some synchronous looking code and runs it in a greenlet. IT also returns a deferred so the caller can register callbacks when the code completes.
A wait_one function, which blocks the greenlet until the deferred being waited on resolves.
The inner function, which (when wrapped) returns a deferred, sleeps for a random time, and then passes the time it slept for to the deferred.
The outer function, which calls inner(), waits for it to return, then prints the return value.
When I run this code I get this output (Note the error on the last two lines):
outer(): greenelet: <greenlet.greenlet object at 0xb729cc5c>
inner(): greenelet: <greenlet.greenlet object at 0xb729ce3c>
Sleeping for 0.545666723422 seconds...
sleep(): greenelet: <greenlet.greenlet object at 0xb729ce3c>
wait_one(): greenelet: <greenlet.greenlet object at 0xb729cc5c>
done
0.545666723422
Here
Exception twisted.python.failure.Failure: <twisted.python.failure.Failure <class 'greenlet.GreenletExit'>> in <greenlet.greenlet object at 0xb729ce3c> ignored
GreenletExit did not kill <greenlet.greenlet object at 0xb729ce3c>
Doing a bit of research I've found that:
The last line is logged by greenlet.c
The previous line is logged by python itself, as it's ignoring an exception raised in a del method.
I'm having real trouble debugging this as I can't access the GreenletExit or twisted.python.failure.Failure exceptions to get their stack traces.
Does anyone have any ideas what I'm doing wrong, or how I get debug the exceptions that are being thrown?
One other data point: If I hack wait_one() to just return immediately (and not to register anything on the deferred it is passed), the errors go away. :-/

Rewrite your error callback in wait_one like this:
def errback(failure):
## new code
if g.dead:
return
##
if not active:
g.throw(failure)
else:
reactor.callLater(0, g.throw, failure)
If greenlet is dead (finished running), there is no point throwing exceptions
in it.

mguijarr's answer fixed the problem, but I wanted to write up how I got into this situation.
I have three greenlets:
{main} that's runing the reactor.
{outer} that's running outer().
{inner} that's rrunning inner().
When the sleep finishes the {main} switches to {inner} which switches to {outer}. Outer then returns and raises GreenletExit in {inner}. This propogates back to twisted. It sees an exception being raised from callback(), and so invokes errback(). This tries to throw the exception into {outer} (which has already exited), and I hit the error.

thread.start_new_thread: transfer exception to main-thread [duplicate]

This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
Catch a thread’s exception in the caller thread in Python
I have a given code and there is a
thread.start_new_thread()
As I just read in python doc: "When the function terminates with an unhandled exception, a stack trace is printed and then the thread exits (but other threads continue to run)."
But I want to terminate also the main-thread when the (new) function terminates with an exception - So the exception shall be transfered to the main-thread. How can I do this?
edit:
here is part of my code:
def CaptureRegionAsync(region=SCREEN, name="Region", asyncDelay=None, subDir="de"):
if asyncDelay is None:
CaptureRegion(region, name, subDir)
else:
thread.start_new_thread(_CaptureRegionAsync, (region, name, asyncDelay, subDir))
def _CaptureRegionAsync(region, name, asyncDelay, subDir):
time.sleep(max(0, asyncDelay))
CaptureRegion(region, name, subDir)
def CaptureRegion(region=SCREEN, name="Region", subDir="de"):
...
if found:
return
else:
raise Exception(u"[warn] Screenshot has changed: %s" % filename)
CaptureRegionAsync(myregion,"name",2)

UPD: It is not the best solution as your question is different from what I thought it is: I expected you were trying to deal with an exception in thread code you can not modify. However, i decided not to delete the answer.
It's hard to catch an exception from another thread. Usually it should be transferred manually.
Can you wrap a thread function? If you wrap the worker thread that crashes into try...except, then you may perform actions needed to exit main thread in except block (sys.exit seems to be useless, though it's a surprise to me, but there is thread.interrupt_main() and os.abort(), though you may need something more graceful like setting a flag the main thread is checking regularly).
However, if you can not wrap the function (can not modify the third-party code calling start_new_thread), you may try monkey patching the thread module (or the third-party module itself). Patched version of start_new_thread() should wrap the function you worry about:
import thread
import sys
import time
def t():
print "Thread started"
raise Exception("t Kaboom")
def t1():
print "Thread started"
raise Exception("t1 Kaboom")
def decorate(s_n_t):
def decorated(f, a, kw={}):
if f != t:
return s_n_t(f, a, kw)
def thunk(*args, **kwargs):
try:
f(*args, **kwargs)
except:
print "Kaboom"
thread.interrupt_main() # or os.abort()
return s_n_t(thunk, a, kw)
return decorated
# Now let's do monkey patching:
thread.start_new_thread = decorate(thread.start_new_thread)
thread.start_new_thread(t1, ())
time.sleep(5)
thread.start_new_thread(t, ())
time.sleep(5)
Here an exception in t() causes thread.interrupt_main()/os.abort(). Other thread functions in the application are not affected.

In Python try until no error

I have a piece of code in Python that seems to cause an error probabilistically because it is accessing a server and sometimes that server has a 500 internal server error. I want to keep trying until I do not get the error. My solution was:
while True:
try:
#code with possible error
except:
continue
else:
#the rest of the code
break
This seems like a hack to me. Is there a more Pythonic way to do this?

It won't get much cleaner. This is not a very clean thing to do. At best (which would be more readable anyway, since the condition for the break is up there with the while), you could create a variable result = None and loop while it is None. You should also adjust the variables and you can replace continue with the semantically perhaps correct pass (you don't care if an error occurs, you just want to ignore it) and drop the break - this also gets the rest of the code, which only executes once, out of the loop. Also note that bare except: clauses are evil for reasons given in the documentation.
Example incorporating all of the above:
result = None
while result is None:
try:
# connect
result = get_data(...)
except:
pass
# other code that uses result but is not involved in getting it

Here is one that hard fails after 4 attempts, and waits 2 seconds between attempts. Change as you wish to get what you want form this one:
from time import sleep
for x in range(0, 4): # try 4 times
try:
# msg.send()
# put your logic here
str_error = None
except Exception as str_error:
pass
if str_error:
sleep(2) # wait for 2 seconds before trying to fetch the data again
else:
break
Here is an example with backoff:
from time import sleep
sleep_time = 2
num_retries = 4
for x in range(0, num_retries):
try:
# put your logic here
str_error = None
except Exception as e:
str_error = str(e)
if str_error:
sleep(sleep_time) # wait before trying to fetch the data again
sleep_time *= 2 # Implement your backoff algorithm here i.e. exponential backoff
else:
break

Maybe something like this:
connected = False
while not connected:
try:
try_connect()
connected = True
except ...:
pass

When retrying due to error, you should always:
implement a retry limit, or you may get blocked on an infinite loop
implement a delay, or you'll hammer resources too hard, such as your CPU or the already distressed remote server
A simple generic way to solve this problem while covering those concerns would be to use the backoff library. A basic example:
import backoff
#backoff.on_exception(
backoff.expo,
MyException,
max_tries=5
)
def make_request(self, data):
# do the request
This code wraps make_request with a decorator which implements the retry logic. We retry whenever our specific error MyException occurs, with a limit of 5 retries. Exponential backoff is a good idea in this context to help minimize the additional burden our retries place on the remote server.

The itertools.iter_except recipes encapsulates this idea of "calling a function repeatedly until an exception is raised". It is similar to the accepted answer, but the recipe gives an iterator instead.
From the recipes:
def iter_except(func, exception, first=None):
""" Call a function repeatedly until an exception is raised."""
try:
if first is not None:
yield first() # For database APIs needing an initial cast to db.first()
while True:
yield func()
except exception:
pass
You can certainly implement the latter code directly. For convenience, I use a separate library, more_itertools, that implements this recipe for us (optional).
Code
import more_itertools as mit
list(mit.iter_except([0, 1, 2].pop, IndexError))
# [2, 1, 0]
Details
Here the pop method (or given function) is called for every iteration of the list object until an IndexError is raised.
For your case, given some connect_function and expected error, you can make an iterator that calls the function repeatedly until an exception is raised, e.g.
mit.iter_except(connect_function, ConnectionError)
At this point, treat it as any other iterator by looping over it or calling next().

Here's an utility function that I wrote to wrap the retry until success into a neater package. It uses the same basic structure, but prevents repetition. It could be modified to catch and rethrow the exception on the final try relatively easily.
def try_until(func, max_tries, sleep_time):
for _ in range(0,max_tries):
try:
return func()
except:
sleep(sleep_time)
raise WellNamedException()
#could be 'return sensibleDefaultValue'
Can then be called like this
result = try_until(my_function, 100, 1000)
If you need to pass arguments to my_function, you can either do this by having try_until forward the arguments, or by wrapping it in a no argument lambda:
result = try_until(lambda : my_function(x,y,z), 100, 1000)

Maybe decorator based?
You can pass as decorator arguments list of exceptions on which we want to retry and/or number of tries.
def retry(exceptions=None, tries=None):
if exceptions:
exceptions = tuple(exceptions)
def wrapper(fun):
def retry_calls(*args, **kwargs):
if tries:
for _ in xrange(tries):
try:
fun(*args, **kwargs)
except exceptions:
pass
else:
break
else:
while True:
try:
fun(*args, **kwargs)
except exceptions:
pass
else:
break
return retry_calls
return wrapper
from random import randint
#retry([NameError, ValueError])
def foo():
if randint(0, 1):
raise NameError('FAIL!')
print 'Success'
#retry([ValueError], 2)
def bar():
if randint(0, 1):
raise ValueError('FAIL!')
print 'Success'
#retry([ValueError], 2)
def baz():
while True:
raise ValueError('FAIL!')
foo()
bar()
baz()
of course the 'try' part should be moved to another funcion becouse we using it in both loops but it's just example;)

Like most of the others, I'd recommend trying a finite number of times and sleeping between attempts. This way, you don't find yourself in an infinite loop in case something were to actually happen to the remote server.
I'd also recommend continuing only when you get the specific exception you're expecting. This way, you can still handle exceptions you might not expect.
from urllib.error import HTTPError
import traceback
from time import sleep
attempts = 10
while attempts > 0:
try:
#code with possible error
except HTTPError:
attempts -= 1
sleep(1)
continue
except:
print(traceback.format_exc())
#the rest of the code
break
Also, you don't need an else block. Because of the continue in the except block, you skip the rest of the loop until the try block works, the while condition gets satisfied, or an exception other than HTTPError comes up.

what about the retrying library on pypi?
I have been using it for a while and it does exactly what I want and more (retry on error, retry when None, retry with timeout). Below is example from their website:
import random
from retrying import retry
#retry
def do_something_unreliable():
if random.randint(0, 10) > 1:
raise IOError("Broken sauce, everything is hosed!!!111one")
else:
return "Awesome sauce!"
print do_something_unreliable()

e = ''
while e == '':
try:
response = ur.urlopen('https://https://raw.githubusercontent.com/MrMe42/Joe-Bot-Home-Assistant/mac/Joe.py')
e = ' '
except:
print('Connection refused. Retrying...')
time.sleep(1)
This should work. It sets e to '' and the while loop checks to see if it is still ''. If there is an error caught be the try statement, it prints that the connection was refused, waits 1 second and then starts over. It will keep going until there is no error in try, which then sets e to ' ', which kills the while loop.

Im attempting this now, this is what i came up with;
placeholder = 1
while placeholder is not None:
try:
#Code
placeholder = None
except Exception as e:
print(str(datetime.time(datetime.now()))[:8] + str(e)) #To log the errors
placeholder = e
time.sleep(0.5)
continue

Here is a short piece of code I use to capture the error as a string. Will retry till it succeeds. This catches all exceptions but you can change this as you wish.
start = 0
str_error = "Not executed yet."
while str_error:
try:
# replace line below with your logic , i.e. time out, max attempts
start = raw_input("enter a number, 0 for fail, last was {0}: ".format(start))
new_val = 5/int(start)
str_error=None
except Exception as str_error:
pass
WARNING: This code will be stuck in a forever loop until no exception occurs. This is just a simple example and MIGHT require you to break out of the loop sooner or sleep between retries.