We Keep Coding

Try/except to cleanup a large script [duplicate]

This question already has answers here:
Exception handling when errors may occur in main program or in cleanup
(3 answers)
a cleaner way to approach try except in python
(6 answers)
Closed 9 months ago.
I am developing a python application. This is formed by the main.py script (~153 lines of code) full of functions that are written in other scripts.
There is some cleanup code at the end, but I would like to run this cleanup even if an error is raised. Searching about this, I have seen approaches like this
try:
# your program goes here
# you're calling your functions from here, etc
except SystemExit:
# do cleanup
raise
This means I have to introduce basically all scripts (from the first function to the end) between try and except. Is this the correct way to do this?

To run something regardless of error or not.
you can use the finally
try:
print(x)
except:
print("Something went wrong")
finally:
print("The 'try except' is finished")

Reading suggestions. So far the best approach I have found is this
try:
run_application()
# This is a function that takes all functions contain in main.py
except:
print('something went wrong')
finally:
do cleanup
# This is at the end of the main.py script

import sys
try:
print('my code')
# "type" of exception depends of Your script for example StandardError
except StandardError as err:
print('ERROR: {}'.format(err))
sys.exit(1)
#do cleanup
print('all OK')
print('cleanup code')
sys.exit(0)

API Exception Handling (Python)

I request data from an API in a while/try/except loop.
The general structure is like this:
while (True):
try:
apiloop
except:
print(error)
else:
apiloop
If I try it like this it continues the loop after the first exceptions, as expected.
But in the next round of exceptions, it crashes again.
So my question is how do I get to catch the exception every time and still continue with my loop. Do I need to set maybe:
finally:
continue
I appreciate every help.
Regards,
Angelo (Germany)

In python, how do I re-run a function if it throws an error

Below are snippets of my codes, what I wanted to know is hypothetically if function main2() throws out an error for some reason, how do I get my exception to run the same function again say 3 times before it breaks?
Just to add here, any of the functions could throw an error (not just main2()) also I might have not just 3 but many more functions
import numpy as np
def main():
np.load('File.csv')
def main1():
np.load('File1.csv')
def main2():
np.load('File2.csv')
for i in range(1, 10):
try:
main()
main2()
main3()
except Exception as e:
print e
else:
break

You could do it with python retry decorator
#retry((Exception), tries=3, delay=0, backoff=0)
def main2():
np.load('File2.csv')
This would work the same way as if you would write:
error_counter = 0
def main2():
try:
np.load('File2.csv')
except:
if error_counter < 3
error_counter += 1
main2()
raise Exception("Will not try again, have tried 3 times")
error_counter = 0
If you want to make it robust and clean, you should go for the first solution. The first solution you can reuse in a large enterprise project and due to the backoff time it can take disk load,user load network issues into consideration with backoff/delay time.
If you don't use a time delay, you will make all the 3 tries in just a second. That is fine for some kind of exceptions but when having network issues, or disk issues you want a more sophisticated solution.
Also, consider to not catch all exceptions, it is a bad practice to cache all. More info, why it is bad

Here's an idiom you can try:
for _ in range(3): # try 3 times
try:
main2()
break # as soon as it works, break out of the loop
except Exception as e:
print e
continue # otherwise, try again
else: # if the loop exited normally, e.g. if all 3 attempts failed
pass
# do_something...
Note the indentation. The else here is attached to the for, not the try.

Unfortunately, implementing a custom retry decorator can often be a bit of a pain. If you want to tweak their logic or adjust them, it can actually get quite complicated quite fast. There's a Python library out there called Backoff-Utils that supports very robust and easily-extensible retry / backoff strategies (full disclosure: I'm biased, since I'm the author of that library).
In your hypothetical question, you could use the library in a decorator-based strategy:
from backoff_utils import backoff, apply_backoff, strategies
#apply_backoff(strategies.Fixed, max_tries = 3, catch_exceptions = [type(ValueError)])
def main2():
# np.load('File2.csv')
raise ValueError
print("In main2")
or you could use it in a function-based strategy when calling main2():
result = backoff(main2,
max_tries = 3,
catch_exceptions = [type(ValueError)],
strategy = strategies.Fixed)
Of course, the code snippet above is specifically designed to do exactly what you described up above. It uses a linear strategy (just retrying 3 times, with a default delay of 1 second between attempts).
Using the library, you can employ any number of other retry / delay strategies, including Exponential Backoff, Fibonnaci, Linear Progression, and Polynomial. You can also customize and create your own delay strategies, as well. And you can incorporate custom success / failure handlers, and different alternative paths for different types of situations.
Of course, all of this flexibility is overkill for your particular use case - you don't need that much. But it may be easier than worrying about copying/pasting/maintaining your own retry decorator and gives you additional built-in options if you find you need more sophisticated retry strategies down the road.
In case it's helpful, you can find some pretty detailed documentation here: https://backoff-utils.readthedocs.io/en/latest/index.html
Hope this helps!

You can try
for i in range(1, 10):
error_max = 3
error_counter = 0
try:
main()
try:
main2()
except Exception as e:
counter += 1
if counter == 3:
raise e
else:
continue
main3()
except Exception as e:
print e
else:
break
This code will run function main2() until it will get 3 errors and on the first 2 it will make the loop run again.

You should handle all the errors within specific functions, otherwise if handling errors of all the functions together, the function coming prior to other functions will throw error and the control will execute the except block skipping rest of the code below it in the try block.
Try it yourself:
def main():
# np.load('File.csv')
raise ValueError
print("In main")
def main1():
# np.load('File1.csv')
raise ValueError
print("In main1")
def main2():
# np.load('File2.csv')
raise ValueError
print("In main2")
for i in range(1, 10):
try:
main()
main2()
main3()
except Exception as e:
print(e)
else:
break
Try commenting raised errors in the functions in different order.
And when the errors are handled within each function then every function is executed without any skipping of the remaining functions in the loop

Python: How to try again when experiencing a timeout

I have a Python program that connects to a server to send it some commands, but occasionally I get this error:
TimeoutError: [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
In previous situations like this, I would use something like this:
try:
Do something
except KeyError:
do something else
Could I do the same thing in this same situation? I.e.,
try:
Do something
except TimeoutError:
Do something again
And if so, what would I do after the except TimeoutError? Would I just do the same command again?

Could I do the same thing in this same situtation
Yes! You can use try/except for any exception, and TimeoutError is nothing special.
and if so what would I do after the except TimeoutError? would I just do the same command again?
If you only want to retry once, and let a second timeout count as a real error, yes:
try:
do something
except TimeoutError:
do something
(If "do something" is more than a simple statement, you probably want to factor out the code so you don't repeat yourself.)
However, if you want to retry multiple times, you probably want a loop:
for _ in range(max_retries):
try:
do something
break
except TimeoutError:
pass
You may want to add an else clause to the for to distinguish the two cases (succeeded, and did a break, vs. never succeeded and just ran out of tries).
Since the idea here is usually to deal with possibly-transient errors, there are additional things you might want to add, such as:
Re-raising the error, or raising a different one, after max_retries.
Logging at progressively higher levels (e.g., a debug message for a single failure, but a warning for `max_retries).
Retrying with exponential backoff (wait 1 second, then 2, 4, 8, …).
Pushing the URL to the end of the work queue instead of retrying immediately. You can use (URL, retry_count) pairs if you also want max_retries logic, (URL, timestamp) pairs if you also want exponential backoff, or both if you want both. (Of course this only works if you don't care about the order of responses, or can reorder them at the end.)
Different rules for different exceptions (e.g., 500, 502, 503, and 504 errors can all be caused by overload on a server or proxy, but the best retry logic may be different—and the best heuristics for 2018 may be different from 2010 or 2025).
For complicated combinations, a retry decorator, like the one linked in jterrace's helpful answer, is a great way to wrap up the behavior.

You can catch the TimeoutError like you mentioned:
import socket
import sys
try:
dosomething()
except socket.TimeoutError:
print >> sys.stderr, 'Retrying after TimeoutError'
dosomething()
You could also use the retry decorator pattern on a function:
#retry(socket.TimeoutError)
def dosomething():
# code that causes a TimeoutError
...

def f():
pass #insert code here
To repeat once after the error:
try:
f()
except TimeoutError:
f()
Or to loop until success:
while True:
try:
f()
break
except TimeoutError:
pass
Or with a limited number:
attempts = 3
while attempts:
try:
f()
break
except TimeoutError:
attempts -= 1

Please run each example. They are ready!
Example 1
import sys
try:
incorrect.syntaxThatIJustMadeUP()
except:
print((sys.exc_info()[0])) #Now you know what to except and CATCH
else:
print("You will never see this message")
Example 2
import sys
try:
incorrect.syntaxThatIJustMadeUP()
except NameError:
print("There is a problem with your SYNTAX, Dude!")
except:
print((sys.exc_info()[0])) # In case another uncontrollable network problem occurs, User rages-Snaps IJ45
else:
print("You will never see this message unless TRY succeeds")
print("Why not put this try in a loop?")

Handling exceptions from urllib2 and mechanize in Python

I am a novice at using exception handling. I am using the mechanize module to scrape several websites. My program fails frequently because the connection is slow and because the requests timeout. I would like to be able to retry the website (on a timeout, for instance) up to 5 times after 30 second delays between each try.
I looked at this stackoverflow answer and can see how I can handle various exceptions. I also see (although it looks very clumsy) how I can put the try/exception inside a while loop to control the 5 attempts ... but I do not understand how to break out of the loop, or "continue" when the connection is successful and no exception has been thrown.
from mechanize import Browser
import time
b = Browser()
tried=0
while tried < 5:
try:
r=b.open('http://www.google.com/foobar')
except (mechanize.HTTPError,mechanize.URLError) as e:
if isinstance(e,mechanize.HTTPError):
print e.code
tried += 1
sleep(30)
if tried > 4:
exit()
else:
print e.reason.args
tried += 1
sleep(30)
if tried > 4:
exit()
print "How can I get to here after the first successful b.open() attempt????"
I would appreciate advice about (1) how to break out of the loop on a successful open, and (2) how to make the whole block less clumsy/more elegant.

Your first question can be done with break:
while tried < 5:
try:
r=b.open('http://www.google.com/foobar')
break
except #etc...
The real question, however, is do you really want to: this is what is known as "Spaghetti code": if you try to graph execution through the program, it looks like a plate of spaghetti.
The real (imho) problem you are having, is that your logic for exiting the while loop is flawed. Rather than trying to stop after a number of attempts (a condition that never occurs because you're already exiting anyway), loop until you've got a connection:
#imports etc
tried=0
connected = False
while not Connected:
try:
r = b.open('http://www.google.com/foobar')
connected = true # if line above fails, this is never executed
except mechanize.HTTPError as e:
print e.code
tried += 1
if tried > 4:
exit()
sleep(30)
except mechanize.URLError as e:
print e.reason.args
tried += 1
if tried > 4:
exit()
sleep(30)
#Do stuff

You don't have to repeat things in the except block that you do in either case.
from mechanize import Browser
import time
b = Browser()
tried=0
while True:
try:
r=b.open('http://www.google.com/foobar')
except (mechanize.HTTPError,mechanize.URLError) as e:
tried += 1
if isinstance(e,mechanize.HTTPError):
print e.code
else:
print e.reason.args
if tried > 4:
exit()
sleep(30)
continue
break
Also, you may be able to use while not r: depending on what Browser.open returns.
Edit: roadierich showed a more elegant way with
try:
doSomething()
break
except:
...
Because an error skips to the except block.

For your first question, you simply want the "break" keyword, which breaks out of a loop.
For the second question, you can have several "except" clauses for one "try", for different kinds of exceptions. This replaces your isinstance() check and will make your code cleaner.