I learnt about python decorators recently, so I tried and designed a decorator like this:
def retry(repeat,error_return=None):
def decorator(function):
def wrapper(*args,**kwargs):
nonlocal repeat
while repeat>0:
try:
return function(*args,**kwargs)
except Exception as e:
repeat=repeat-1
print(e)
return error_return
return wrapper
return decorator
If an exception occurred while executing the function decorated, no exception will be raised. And the function will be retried for certain times before the exception was printed.
I used the decorator on a simple downloader, like this:
#retry(3)
def download_single(url,path):
........
content=requests.get(url,timeout=10).content
with open(path,'w') as f:
f.write(content)
print('Downloaded'+path)
def download_all(alist):
for url in alist:
download_single(url,PATH)
Normally, and as I have expected, it should work like this:
However, in some cases, the program just stops running after an exception is printed:
So why on earth does it stop by itself?? (The url list has more than 1000 elements, and it stopped at 37)
And this had happened many times, on different elements. So it is not the problem of a certain url.
And the it is the same problem when i run the program directly from console. So it is not because IDLE either.
Is it a problem with the decorator, or the downloader?
It only quits in this way right after an exception, so I think the exception caused the silent quit.
Has anyone met similar cases before?
I guess your program prints 3 exceptions in total, i.e. not per URL. This is because you declare repeat as nonlocal and modify that variable. The wrapper defines a closure over this name (repeat), so for each function call you access the very same object. This mean whenever you call download_single and it fails it will modify the very same repeat object. So this gives a total of 3 retries for the whole program, instead of per URL. Then repeat is zero and the wrapper just doesn't do any work anymore (it directly returns error_return). This means your program doesn't quit prematurely but it exhausts the list of all files without doing any work beyond the third timeout.
Instead you can just loop over range(repeat) inside the wrapper:
for __ in range(repeat):
try:
return function(*args,**kwargs)
except Exception as e:
print(e)
Related
I am writing a library that allows people to play music live, from the REPL. It is based on some sort of recursion that allows a function to reschedule itself at some point in the future, usually as soon as it ends. I would like to simplify the syntax from this:
#run()
def bip():
Sound("bip").out()
sleep(0.25)
Loop(bip)
to
#run()
def bip():
Sound("bip").out()
sleep(0.25)
The library is meant to be used by throwing code in a Python REPL, without writing down loops and beeps in a source file. Everything is meant to be used at runtime.
run() is a decorator that will transfer the function definition to a scheduler and start it immediately as soon as the code is doubled-checked.
Loop() is an alias for a scheduler method that will reschedule the function as soon as possible, following tempo and time constraints.
How could I get rid of Loop()? Is it possible to programmatically add Loop() to any function decorated by run()? Is it possible to alter the source code of a function evaluated directly in a Python REPL?
It is a nice programming problem that I haven't found a solution to yet..
EDIT: Here is the definition of the run() decorator.
def run(beat=1, bar=1):
def run_inner(func):
#wraps(func)
def func_wrapper(*args, **kwargs):
"""
We need to catch errors before they stop the program when the user
submits incorrect code. This ugly decorator will catch 90% of the
current errors made by users when submitting new functions to the
scheduler.
"""
try:
return func(*args, **kwargs)
except NameError as e:
print("You named something incorrectly: {}.".format(e))
Stop(func)
return None
except ValueError as e:
print("Check your numbers: {}.".format(e))
Stop(func)
return None
except TypeError as e:
print("Something ended up in the wrong pipe: {}.".format(e))
Stop(func)
return None
except AttributeError as e:
print("Whoops. Correct your code please : {}.".format(e))
Stop(func)
return None
"""
Track._memory is a dictionary holding the name, bytecode, etc...
of all the functions currently running on the scheduler. A function
can only be started if it is not already registered so we check here.
"""
if func.__name__ not in list(Track._memory.keys()):
# Start will register / schedule the function for immediate
# execution.
Start(func_wrapper, beat=0, bar=1)
return func_wrapper
return run_inner
A bit of a odd problem, I have a long script which I want to error our in certain conditions. If I error out I want to stop there and then.
exit() doesnt work because lambda gets re triggered as it didn't get a response to say the script has ended successfully.
python 'return' doesn't work for me because it only returns to the parent function. At which point the code will continue running.
I am wondering how I can exit or 'return' all the way back to lambda that the script is considered complete without letting the script end?
My current approach is to throw an exception in my sub functions and wrap my main function in a try catch where the sub functions trigger that exception, that cant be the best way.
def kill_script():
raise Exception(msg)
def do_something():
kill_script() # pretend something has gone wrong.
def lambda_handler(event, context):
try:
do_something()
except:
print("this is how I currently do it, cant be the best way?")
As mentioned by #HoboProber, you can raise another exception in your do_something(), and you can actually ultimately raise another exception in your lambda_handler's try/except block. The AWS documentation mentions:
"If your Lambda function raises an exception, AWS Lambda recognizes the failure and serializes the exception information into JSON and returns it." See here https://docs.aws.amazon.com/lambda/latest/dg/python-exceptions.html
I am new to python and I am writing code that uses OAuth to authenticate and when the token expires after 60 minutes, it needs to get a new one.
try:
if uploadedContent is not None:
thing.action(uploadedContent)
except LoginOrScopeRequired:
print("Logging in...")
set_access_credentials({"identity", "submit"}, get_access_token())
I currently have this code to handle getting a new token if it expires, but the problem is that if there was an exception it skips over the action it needed to do. I understand that I could take what was inside the try block and append it to end of the except block, but it there a more elegant way to do this?
Some of my research led to the with statement, but I didn't understand with well enough to know if it would solve my problem. So is appending it to the end the best solution or is there something better?
It is considered Idiomatic Python to do this with a function decorator/wrapper:
Example:
#!/usr/bin/env python
from functools import wraps
def retry_on_error(ntries=1):
"""
A decorator that returns a wrapper function that calls
the wrapped function repeatedly up to ntries if an
exception is encountered.
"""
def decorator(f): # wrapping the original function
#wraps(f) # make the wrapped function look like the original
def wrapper(*args, **kwargs): # our function wrapped that calls the original
for i in xrange(ntries):
try:
return f(*args, **kwargs)
except Exception as e:
print("Error executing {0:s} retrying {1:d}/{2:d}".format(f.__name__, i, ntries))
print("Error was {0:s}".format(e))
return wrapper
return decorator # returning the new wrapped function
#retry_on_error()
def f():
n = getattr(f, "n", 0)
try:
if not n:
raise ValueError("n < 0")
finally:
setattr(f, "n", n + 1)
Output:
$ python -i foo.py
>>> f()
Error executing f retrying 0/1
Error was n < 0
>>> f()
>>>
See: Python Decorators for other examples.
Update: There is also a nice library that implements this functionality with a few more features: retrying as well as several other related/similar questions How to retry after exception in python? and Pythonic way of retry running a function
Update #2: I've commented the decorator a bit so hopefully you can understand what's going on in each step of the process. Admittedly decorators aren't that easy to understand at first so I recommend you read Understanding Python Decorators in 12 easy step
Some languages like ruby let you put a retry statement in the exception catch block that makes this incredibly simple. Unfortunately in Python you will need to wrap this in a while statement:
success = False
while not success
try:
if uploadedContent is not None:
thing.action(uploadedContent)
success = True
except LoginOrScopeRequired:
print("Logging in...")
set_access_credentials({"identity", "submit"}, get_access_token())
Note that the line success = True will only be reached if no exception occurs.
EDIT
You will also want to keep track of the number of attempts in a counter to ensure this doesn't loop forever and exits after 3 retries for example.
When using multiprocessing.Pool's apply_async(), what happens to breaks in code? This includes, I think, just exceptions, but there may be other things that make the worker functions fail.
import multiprocessing as mp
pool = mp.Pool(mp.cpu_count())
for f in files:
pool.apply_async(workerfunct, args=(*args), callback=callbackfunct)
As I understand it right now, the process/worker fails (all other processes continue) and anything past a thrown error is not executed, EVEN if I catch the error with try/except.
As an example, usually I'd except Errors and put in a default value and/or print out an error message, and the code continues. If my callback function involves writing to file, that's done with default values.
This answerer wrote a little about it:
I suspect the reason you're not seeing anything happen with your example code is because all of your worker function calls are failing. If a worker function fails, callback will never be executed. The failure won't be reported at all unless you try to fetch the result from the AsyncResult object returned by the call to apply_async. However, since you're not saving any of those objects, you'll never know the failures occurred. If I were you, I'd try using pool.apply while you're testing so that you see errors as soon as they occur.
If you're using Python 3.2+, you can use the error_callback keyword argument to to handle exceptions raised in workers.
pool.apply_async(workerfunct, args=(*args), callback=callbackfunct, error_callback=handle_error)
handle_error will be called with the exception object as an argument.
If you're not, you have to wrap all your worker functions in a try/except to ensure your callback is executed. (I think you got the impression that this wouldn't work from my answer in that other question, but that's not the case. Sorry!):
def workerfunct(*args):
try:
# Stuff
except Exception as e:
# Do something here, maybe return e?
pool.apply_async(workerfunct, args=(*args), callback=callbackfunct)
You could also use a wrapper function if you can't/don't want to change the function you actually want to call:
def wrapper(func, *args):
try:
return func(*args)
except Exception as e:
return e
pool.apply_async(wrapper, args=(workerfunct, *args), callback=callbackfunct)
I know using below code to ignore a certain exception, but how to let the code go back to where it got exception and keep executing? Say if the exception 'Exception' raises in do_something1, how to make the code ignore it and keep finishing do_something1 and process do_something2? My code just go to finally block after process pass in except block. Please advise, thanks.
try:
do_something1
do_something2
do_something3
do_something4
except Exception:
pass
finally:
clean_up
EDIT:
Thanks for the reply. Now I know what's the correct way to do it. But here's another question, can I just ignore a specific exception (say if I know the error number). Is below code possible?
try:
do_something1
except Exception.strerror == 10001:
pass
try:
do_something2
except Exception.strerror == 10002:
pass
finally:
clean_up
do_something3
do_something4
There's no direct way for the code to go back inside the try-except block. If, however, you're looking at trying to execute these different independant actions and keep executing when one fails (without copy/pasting the try/except block), you're going to have to write something like this:
actions = (
do_something1, do_something2, #...
)
for action in actions:
try:
action()
except Exception, error:
pass
update. The way to ignore specific exceptions is to catch the type of exception that you want, test it to see if you want to ignore it and re-raise it if you dont.
try:
do_something1
except TheExceptionTypeThatICanHandleError, e:
if e.strerror != 10001:
raise
finally:
clean_up
Note also, that each try statement needs its own finally clause if you want it to have one. It wont 'attach itself' to the previous try statement. A raise statement with nothing else is the correct way to re-raise the last exception. Don't let anybody tell you otherwise.
What you want are continuations which python doesn't natively provide. Beyond that, the answer to your question depends on exactly what you want to do. If you want do_something1 to continue regardless of exceptions, then it would have to catch the exceptions and ignore them itself.
if you just want do_something2 to happen regardless of if do_something1 completes, you need a separate try statement for each one.
try:
do_something1()
except:
pass
try:
do_something2()
except:
pass
etc. If you can provide a more detailed example of what it is that you want to do, then there is a good chance that myself or someone smarter than myself can either help you or (more likely) talk you out of it and suggest a more reasonable alternative.
This is pretty much missing the point of exceptions.
If the first statement has thrown an exception, the system is in an indeterminate state and you have to treat the following statement as unsafe to run.
If you know which statements might fail, and how they might fail, then you can use exception handling to specifically clean up the problems which might occur with a particular block of statements before moving on to the next section.
So, the only real answer is to handle exceptions around each set of statements that you want to treat as atomic
you could have all of the do_something's in a list, and iterate through them like this, so it's no so wordy. You can use lambda functions instead if you require arguments for the working functions
work = [lambda: dosomething1(args), dosomething2, lambda: dosomething3(*kw, **kwargs)]
for each in work:
try:
each()
except:
pass
cleanup()
Exceptions are usually raised when a performing task can not be completed in a manner intended by the code due to certain reasons. This is usually raised as exceptions. Exceptions should be handled and not ignored. The whole idea of exception is that the program can not continue in the normal execution flow without abnormal results.
What if you write a code to open a file and read it? What if this file does not exist?
It is much better to raise exception. You can not read a file where none exists. What you can do is handle the exception, let the user know that no such file exists. What advantage would be obtained for continuing to read the file when a file could not be opened at all.
In fact the above answers provided by Aaron works on the principle of handling your exceptions.
I posted this recently as an answer to another question. Here you have a function that returns a function that ignores ("traps") specified exceptions when calling any function. Then you invoke the desired function indirectly through the "trap."
def maketrap(*exceptions):
def trap(func, *args, **kwargs):
try:
return func(*args, **kwargs)
except exceptions:
return None
return trap
# create a trap that ignores all exceptions
trapall = maketrap(Exception)
# create a trap that ignores two exceptions
trapkeyattrerr = maketrap(KeyError, AttributeError)
# Now call some functions, ignoring specific exceptions
trapall(dosomething1, arg1, arg2)
trapkeyattrerr(dosomething2, arg1, arg2, arg3)
In general I'm with those who say that ignoring exceptions is a bad idea, but if you do it, you should be as specific as possible as to which exceptions you think your code can tolerate.
Python 3.4 added contextlib.suppress(), a context manager that takes a list of exceptions and suppresses them within the context:
with contextlib.suppress(IOError):
print('inside')
print(pathlib.Path('myfile').read_text()) # Boom
print('inside end')
print('outside')
Note that, just as with regular try/except, an exception within the context causes the rest of the context to be skipped. So, if an exception happens in the line commented with Boom, the output will be:
inside
outside