Sorry I'm new to programming, and don't really understand how this Thread thing works. My goal was for this input to be timed, and I found some code that does that. However, I'm confused about the structure of this Thread because if you are "too slow", the program never continues on to print "checkpoint" as desired. It just sort of... freezes... Why is it getting stuck?
import time
from threading import Thread
answer = None
def check():
# waits for user input for 3 seconds
for i in range(3):
time.sleep(1)
if answer != None:
return
print('too slow')
Thread(target = check).start()
answer = input("Input something: ")
print('checkpoint')
One thing I tried is:
t = Thread(target = check)
t.start()
answer = input("Input something: ")
# also tried t.join()
if t.is_alive:
print('hi')
I tried to solve this program by trying to raise and catch an exception. However, I couldn't catch the exception. How do I catch it? (Or is there another solution to the problem I am having?)
import time
from threading import Thread
answer = None
def check():
# waits for user input for 3 seconds
for i in range(3):
time.sleep(1)
if answer != None:
return
print('too slow')
# was hoping to catch this as an exception
raise TimeoutError
# starts new thread
Thread(target = check).start()
# prompts user for an input
answer = input("Input something: ")
print('checkpoint')
What's good:
When you type something into the input prompt within 3 seconds, it prints "checkpoint" and continues on with code.
What's bad:
If you take "too long", the program prints "too slow!" as expected, BUT then it stops executing code and just sort of... freezes. So to try to fix this, I was hoping to raise a Timeout Error and then catch it, but I don't know how to catch it. This didn't catch the error:
try:
Thread(target = check).start()
except:
pass
This didn't either:
try:
answer = input("Input something: ")
except:
pass
Could I get some help? Thank you!
Edit: Forgot to mention that I am using linux so a lot of the solutions for my application did not work for me like msvcrt or keyboard. And modules that do work for Linux seem not to be "non-blocking."
You should think of the two threads as two separate programs, but sharing the same variables.
Thread 1 consists of everything that isn't indented in your code. It launches a thread, then it waits for user input, then it prints "checkpoint". Then it's done.
Thread 2 consists of the function check. It checks to see if the variable isn't None. If that happens it's done. If that doesn't happen in three seconds, it prints "too slow" and now it's done.
Neither thread "knows" what the other one is doing, except they share one variable, answer.
The program as a whole will exit when all its threads are finished.
That's it. That's what you've written. So if you type something, the program exits because Thread 1 will always exit after you type something. Thread 2 exits once it sees the variable isn't None.
If you don't type anything, Thread 1 will just sit there and wait for you, forever. That's how the input function works. Thread 2 will exit after 3 seconds or less, but that doesn't affect Thread 1.
You can't throw an Exception from one Thread to another. So you can't throw an exception from Thread 2 and have Thread 1 handle it.
Have you tried typing something AFTER the message "too slow" appears? When you do, Thread 1 (and therefore your program) will exit.
The bottom line is that you can't use the input function in cases like this, because that function blocks the flow of its thread until the user types something. There is nothing any other thread can do to make it continue.
DISCLAIMER: THIS DOESN'T ANSWER THE QUESTION BUT IN CASE YOU WANT TO KNOW HOW I GOT AROUND THE "INPUT" THING, HERE IS MY SOLUTION TO THE PROBLEM.
Actually I found something that works! It's a little strange but it works for what I am trying to do, thanks to #rayryeng 's answer here: detect key press in python?.
Problem Statement: Continue the program when 'enter' is pressed, and timeout if input takes too long. This does exactly that, although it displays strangely to the console... PS. I had to run this in my terminal as 'sudo' or it wouldn't work in my scratch file for whatever reason.
import curses
import os
from time import time, sleep
def main(win):
win.nodelay(True) # True turns on "non-blocking"
key=""
win.clear()
win.addstr("Please press 'Enter' to continue:")
start_time = time()
while 1:
end_time = time()
try:
if end_time-start_time > 5: # 5 seconds
return 'You are too slow!'
else:
key = win.getkey()
if key == os.linesep:
return 'OK. Continuing on...'
except Exception as e:
# No input
pass
p = curses.wrapper(main)
print(p) #-> either 'You are too slow!' or 'OK. Continuing on...'
I guess if you actually wanted to store the input you can modify it to be something like this:
def main(win):
win.nodelay(True) # True turns on "non-blocking"
key=""
win.clear()
win.addstr("Please press 'Enter' to continue:")
start_time = time()
while 1:
end_time = time()
try:
if end_time-start_time > 5: # 5 seconds
return 'You are too slow!'
else:
key = win.getkey() # gets a single char
if key: # == os.linesep:
return str(key) # returns that single char
except Exception as e:
# No input
pass
p = curses.wrapper(main)
print(p) #-> either 'You are too slow!' or character entered
And if you want to store more characters you could do something like this (just be aware that it also stores the "enter" key in the resulting string):
import curses
import os
from time import time, sleep
def main(win):
win.nodelay(True) # True turns on "non-blocking"
key=""
win.clear()
win.addstr("Please press 'Enter' to continue:")
start_time = time()
result = key # empty string
while 1:
end_time = time()
try:
if end_time-start_time > 5: # 5 seconds
return 'You are too slow!'
else:
key = win.getkey() # gets single char
result = result + str(key) # adds characters to the empty string
if key == os.linesep: # "new line"
return result
except Exception as e:
# No input
pass
p = curses.wrapper(main)
print(p) #-> either 'You are too slow!' or characters entered
I have nested exceptions handling in my code, when inner blocks do some stuff before re-raising the exception to upper layers.
Traceback always reports the line number that started the exception handling.
However, when running in UI debugger (Pydev/Eclipse) it stops on the outer exception block in some cases.
Consider the following code for example:
import sys
def f(a):
c=5/a
return c
def sub(d):
print("Entered sub(%d)" % d)
try:
print("Entered try #sub")
e = f(d)/(d-1)
return e
except:
print("Do some staff before re-raising the exception upwards")
raise
def main():
try:
print("Entered try #main")
d = int(sys.argv[1])
sub(d)
except:
print("Reached except block #main")
raise
if __name__ == '__main__':
main()
When running with argument = 0, the exception is caused at line#4 and the debugger stops on that line:
However, when running with argument = 1, the exception is caused at line#11 (as reported in the printed traceback) but the debugger stops at line#15.
Once the debugger stops at the incorrect location it is very difficult to watch internal variables and handle the error, especially when the code in the try block uses loops.
In Pydev->Manage Exceptions, I checked only the "suspend on uncaught exceptions".
There is a checkbox "Skip exceptions caught in same function" that seem related (because it seems like the debugger skipped the first exception in sub and stopped on "raise" statement which can be considered another exception in same function, although the documentation says that it should re-raise the same exception).
This checkbox is grayed out unless I first check "Suspend on caught exceptions*", but once enabling it the debugger gets stuck and does not stop anywhere...
Will appreciate your help.
-Moshe
I have a python script running over some 50,000 items in a database, it takes about 10 seconds per item and I want to stop it.
But, if i just press ctrl + C while the code is in the try except part, then my program enter the expect statement and delete that item then continue. The only way to close the program is to repeatedly delete items this way until I, just be sheer luck, catch it not in the try statement.
How do I exit the script without just killing a try statement?
EDIT 1
I have a statement that looks something like this:
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except:
data = (item['id'])
db.delete('...')
EDIT 2
The top of my connect to db code looks like:
#!/usr/bin/python
import MySQLdb
import sys
class Database:
....
The issue is because you are using a blanket except which is always a bad idea, if you catch specific exceptions then when you KeyboardInterrupt your script will stop:
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except MySQLdb.Error as e:
print(e)
data = (item['id'])
db.delete('...')
If you have other exceptions to catch you can use multiple in the except:
except (KeyError, MySQLdb.Error) as e
At the very least you could catch Exception as e and print the error.
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except Exception as e:
print(e)
data = (item['id'])
db.delete('...')
The moral of the story is don't use a blanket except, catch what you expect and logging the errors might also be a good idea. The exception-hierarchy is also worth reading to see exactly what you are catching.
I would rewrite it like this:
try:
# ...
except KeyboardInterrupt:
sys.exit(1)
except Exception: # I would use a more specific exception here if you can.
data = (item['id'])
db.delete('...')
Or just
try:
# ...
except Exception: # I would use an even more specific exception here if you can.
data = (item['id'])
db.delete('...')
Python 2's exception hiearchy can be found here: https://docs.python.org/2/library/exceptions.html#exception-hierarchy
Also, many scripts are written with something like this boiler plate at the bottom:
def _main(args):
...
if __name__ == '__main__':
try:
sys.exit(_main(sys.argv[1:]))
except KeyboardInterrupt:
print >> sys.stderr, '%s: interrupted' % _SCRIPT_NAME
sys.exit(1)
When you use except: instead of except Exception: in your main body, you are not allowing this top level exception block to catch the KeyboardInterrupt. (You don't have to actually catch it for Ctrl-C to work -- this just makes it prettier when KeyboardInterrupt is raised.)
import signal
import MySQLdb
import sys
## Handle process signals:
def sig_handler(signal, frame):
global connection
global cursor
cursor.close()
connection.close()
exit(0)
## Register a signal to the handler:
signal.signal(signal.SIGINT, sig_handler)
class Database:
# Your class stuf here.
# Note that in sig_handler() i close the cursor and connection,
# perhaps your class has a .close call? If so use that.
for item in items:
try:
if is_item_gone(content) == 1:
data = (item['id'])
db.update('UPDATE items SET empty = 0 WHERE id = (%s);', data)
else:
do alot of stuff
except MySQLdb.Error as e:
data = (item['id'])
db.delete('...')
This would register and catch for instance Ctrl+c at any given time. The handler would terminate a socket accordingly, then exit with exit code 0 which is a good exit code.
This is a more "global" approach to catching keyboard interrupts.
Consider:
def raiseMe( text="Test error" ):
raise Exception( text )
def break_in_finally_test():
for i in range(5):
if i==2:
try:
raiseMe()
except:
raise
else:
print "succeeded!"
finally:
print "testing this!"
break
if __name__=='__main__':
break_in_finally_test()
I expected to see Exception( "Test error" ) to be raised, but instead only "testing this" is printed. The intention, of course, was to call raiseMe() only once, no matter if we succeed or not - but if it raises an exception, I would have wanted to see that!
Why does break swallow the exception that I explicitly raise?
From https://docs.python.org/2.7/reference/compound_stmts.html#finally:
If finally is present, it specifies a ‘cleanup’ handler. The try clause is executed, including any except and else clauses. If an exception occurs in any of the clauses and is not handled, the exception is temporarily saved. The finally clause is executed. If there is a saved exception, it is re-raised at the end of the finally clause. If the finally clause raises another exception or executes a return or break statement, the saved exception is discarded
This also reflects the behaviour expected from the try...finally statement before PEP341:
This is how a try except finally block looked like pre PEP341:
try:
try:
raiseMe()
except:
raise
finally:
#here is where cleanup is supposed to happen before raising error
break
#after finally code: raise error
As the raising of errors never happens in the finally block it is never actually raised.
To maintain backwards compatibility with Python<=2.4, it had to be done in this way.
From the docs Error Handling Docs:
A finally clause is always executed before leaving the try statement, whether an exception has occurred or not.
Your exception never gets raised because you break before the try statement gets fully evaluated.
I think upon reflection that it is due to the fact that break actually raises a StopIteration to "break" out of the for-loop. This is really not very intuitive and not particularly well documented (not mentioned on 1, for example). Could maybe someone confirm/explain it better?
Have the following code structure:
def func():
try:
driver.do("something")
except TimeoutException:
pass
finally:
result = driver.do("something else")
return result
Got the exception by pylint:
return statement in finally block may swallow exception (lost-exception)
Solution was to put return out of the finally statement:
def func():
try:
driver.do("something")
except TimeoutException:
pass
finally:
result = driver.do("something else")
return result # <<
I have a piece of code in Python that seems to cause an error probabilistically because it is accessing a server and sometimes that server has a 500 internal server error. I want to keep trying until I do not get the error. My solution was:
while True:
try:
#code with possible error
except:
continue
else:
#the rest of the code
break
This seems like a hack to me. Is there a more Pythonic way to do this?
It won't get much cleaner. This is not a very clean thing to do. At best (which would be more readable anyway, since the condition for the break is up there with the while), you could create a variable result = None and loop while it is None. You should also adjust the variables and you can replace continue with the semantically perhaps correct pass (you don't care if an error occurs, you just want to ignore it) and drop the break - this also gets the rest of the code, which only executes once, out of the loop. Also note that bare except: clauses are evil for reasons given in the documentation.
Example incorporating all of the above:
result = None
while result is None:
try:
# connect
result = get_data(...)
except:
pass
# other code that uses result but is not involved in getting it
Here is one that hard fails after 4 attempts, and waits 2 seconds between attempts. Change as you wish to get what you want form this one:
from time import sleep
for x in range(0, 4): # try 4 times
try:
# msg.send()
# put your logic here
str_error = None
except Exception as str_error:
pass
if str_error:
sleep(2) # wait for 2 seconds before trying to fetch the data again
else:
break
Here is an example with backoff:
from time import sleep
sleep_time = 2
num_retries = 4
for x in range(0, num_retries):
try:
# put your logic here
str_error = None
except Exception as e:
str_error = str(e)
if str_error:
sleep(sleep_time) # wait before trying to fetch the data again
sleep_time *= 2 # Implement your backoff algorithm here i.e. exponential backoff
else:
break
Maybe something like this:
connected = False
while not connected:
try:
try_connect()
connected = True
except ...:
pass
When retrying due to error, you should always:
implement a retry limit, or you may get blocked on an infinite loop
implement a delay, or you'll hammer resources too hard, such as your CPU or the already distressed remote server
A simple generic way to solve this problem while covering those concerns would be to use the backoff library. A basic example:
import backoff
#backoff.on_exception(
backoff.expo,
MyException,
max_tries=5
)
def make_request(self, data):
# do the request
This code wraps make_request with a decorator which implements the retry logic. We retry whenever our specific error MyException occurs, with a limit of 5 retries. Exponential backoff is a good idea in this context to help minimize the additional burden our retries place on the remote server.
The itertools.iter_except recipes encapsulates this idea of "calling a function repeatedly until an exception is raised". It is similar to the accepted answer, but the recipe gives an iterator instead.
From the recipes:
def iter_except(func, exception, first=None):
""" Call a function repeatedly until an exception is raised."""
try:
if first is not None:
yield first() # For database APIs needing an initial cast to db.first()
while True:
yield func()
except exception:
pass
You can certainly implement the latter code directly. For convenience, I use a separate library, more_itertools, that implements this recipe for us (optional).
Code
import more_itertools as mit
list(mit.iter_except([0, 1, 2].pop, IndexError))
# [2, 1, 0]
Details
Here the pop method (or given function) is called for every iteration of the list object until an IndexError is raised.
For your case, given some connect_function and expected error, you can make an iterator that calls the function repeatedly until an exception is raised, e.g.
mit.iter_except(connect_function, ConnectionError)
At this point, treat it as any other iterator by looping over it or calling next().
Here's an utility function that I wrote to wrap the retry until success into a neater package. It uses the same basic structure, but prevents repetition. It could be modified to catch and rethrow the exception on the final try relatively easily.
def try_until(func, max_tries, sleep_time):
for _ in range(0,max_tries):
try:
return func()
except:
sleep(sleep_time)
raise WellNamedException()
#could be 'return sensibleDefaultValue'
Can then be called like this
result = try_until(my_function, 100, 1000)
If you need to pass arguments to my_function, you can either do this by having try_until forward the arguments, or by wrapping it in a no argument lambda:
result = try_until(lambda : my_function(x,y,z), 100, 1000)
Maybe decorator based?
You can pass as decorator arguments list of exceptions on which we want to retry and/or number of tries.
def retry(exceptions=None, tries=None):
if exceptions:
exceptions = tuple(exceptions)
def wrapper(fun):
def retry_calls(*args, **kwargs):
if tries:
for _ in xrange(tries):
try:
fun(*args, **kwargs)
except exceptions:
pass
else:
break
else:
while True:
try:
fun(*args, **kwargs)
except exceptions:
pass
else:
break
return retry_calls
return wrapper
from random import randint
#retry([NameError, ValueError])
def foo():
if randint(0, 1):
raise NameError('FAIL!')
print 'Success'
#retry([ValueError], 2)
def bar():
if randint(0, 1):
raise ValueError('FAIL!')
print 'Success'
#retry([ValueError], 2)
def baz():
while True:
raise ValueError('FAIL!')
foo()
bar()
baz()
of course the 'try' part should be moved to another funcion becouse we using it in both loops but it's just example;)
Like most of the others, I'd recommend trying a finite number of times and sleeping between attempts. This way, you don't find yourself in an infinite loop in case something were to actually happen to the remote server.
I'd also recommend continuing only when you get the specific exception you're expecting. This way, you can still handle exceptions you might not expect.
from urllib.error import HTTPError
import traceback
from time import sleep
attempts = 10
while attempts > 0:
try:
#code with possible error
except HTTPError:
attempts -= 1
sleep(1)
continue
except:
print(traceback.format_exc())
#the rest of the code
break
Also, you don't need an else block. Because of the continue in the except block, you skip the rest of the loop until the try block works, the while condition gets satisfied, or an exception other than HTTPError comes up.
what about the retrying library on pypi?
I have been using it for a while and it does exactly what I want and more (retry on error, retry when None, retry with timeout). Below is example from their website:
import random
from retrying import retry
#retry
def do_something_unreliable():
if random.randint(0, 10) > 1:
raise IOError("Broken sauce, everything is hosed!!!111one")
else:
return "Awesome sauce!"
print do_something_unreliable()
e = ''
while e == '':
try:
response = ur.urlopen('https://https://raw.githubusercontent.com/MrMe42/Joe-Bot-Home-Assistant/mac/Joe.py')
e = ' '
except:
print('Connection refused. Retrying...')
time.sleep(1)
This should work. It sets e to '' and the while loop checks to see if it is still ''. If there is an error caught be the try statement, it prints that the connection was refused, waits 1 second and then starts over. It will keep going until there is no error in try, which then sets e to ' ', which kills the while loop.
Im attempting this now, this is what i came up with;
placeholder = 1
while placeholder is not None:
try:
#Code
placeholder = None
except Exception as e:
print(str(datetime.time(datetime.now()))[:8] + str(e)) #To log the errors
placeholder = e
time.sleep(0.5)
continue
Here is a short piece of code I use to capture the error as a string. Will retry till it succeeds. This catches all exceptions but you can change this as you wish.
start = 0
str_error = "Not executed yet."
while str_error:
try:
# replace line below with your logic , i.e. time out, max attempts
start = raw_input("enter a number, 0 for fail, last was {0}: ".format(start))
new_val = 5/int(start)
str_error=None
except Exception as str_error:
pass
WARNING: This code will be stuck in a forever loop until no exception occurs. This is just a simple example and MIGHT require you to break out of the loop sooner or sleep between retries.