Python: gather exception statistics - python

I am repeating a sequence of experiments (with random initialization) like this in Python:
num_rounds = 1e3
result_list = []
for i in range(num_rounds):
result = run_experiment() # with random initialization inside the call of `run_experiment`
result_list.append(result)
However, my code can be buggy and exceptions might be thrown halfway, which will then interrupt the whole loop.
I'm wondering if there a way to:
let go the exception and continue the loop
meanwhile, store the condition (input parameters) that causes the exception(s) somewhere
gather basic statistics of different exceptions, for example the frequency of each exception type.
Or is there already a tool to do that?

1) Let go exception and continue loop:
try:
<whatever code can cause exception>
except <Exceptions you want to catch>:
pass
that will catch the exception and let it slip. Alternatively you can put a continue in the except block:
try:
<whatever code can cause exception>
except <Exceptions you want to catch>:
pass
2) and 3) With the code from 1. you can collect any data you want in the except block and then do a continue:
statistics_list = []
for <something>:
try:
<whatever code can cause exception>
except <Exceptions you want to catch>:
statistics_list.append((exception_type, input params....))
continue
after your loop you can process the data in your statistics list to your liking

There is try..except..else:
num_rounds = 1e3
result_list = []
exceptions = {}
for i in range(num_rounds):
values = 1,2,3, ... # with random initialization outside the call of `run_experiment`
try:
result = run_experiment(*values)
except Exception as e:
exceptions[values] = e
else:
result_list.append(result)

I think you will be able to do your statistics if modify your code like this
num_rounds = 1e3
result_list = []
for i in range(num_rounds):
try:
result = run_experiment()
result_list.append(result)
except Exception as ex:
# do statistics
That way you have the exception as ex, you can check and store the type.
If you also want to store the random input parameters than you should put the same try-except in your run_experiment function

Related

How to stop an Error from being caught by more than one except blocks?

I have some code inside a try block that throws a ValueError. I have an except block that catches this ValueError and handles it. Once this is done I do not want other except blocks to catch this ValueError.
In the code below, there is another try except block later in the code and this ValueError is being caught by the second except too. How can I make sure the ValueError is caught only by the first except and not the second one?
# Get features to plot
try: #First Try Block
raw_features_list_1 = fsobject_1.genFeature(gmm.FftClassifier.fft_features)
except ValueError: #First Except Block
pass
filtered_features_list_1 = np.array([row[0] for row in raw_features_list_1 if row is not None]).reshape(-1, 1)
try: #Second Try Block
raw_features_list_2 = fsobject_2.genFeature(gmm.FftClassifier.fft_features)
except ValueError: #Second Except Block
pass
print("I am here")
filtered_features_list_2 = np.array([row[0] for row in raw_features_list_2 if row is not None]).reshape(-1, 1)
The above code gives me the following result:
I am here
NameError Traceback (most recent call last)
<ipython-input-16-ff0067cb5362> in <module>
11 pass
12 print("I am here")
---> 13 filtered_features_list_2 = np.array([row[0] for row in raw_features_list_2 if row is not None]).reshape(-1, 1)
NameError: name 'raw_features_list_2' is not defined
This is because due to the ValueError generated by the first Try Block the second try block is not being evaluated and the the second Except block is being evaluated directly.
I would like to evaluate the first Try Blocks, handle the ValueError in the First Except Block. Then evaluate the second Try Blocks and, if it generates a ValueError, handle that in the second Except Block.
Sorry looks like what I am asking is already the default behavior of python.
I had another issue where the function generating raw_features_list_2 was throwing a ValueError and aborting before returning raw_features_list_2. So I felt as if the second try block wasn't being evaluated at all. I fixed my problem by handling the exception inside the function that returns raw_features_list_2.
As can be seen below, python catches an exception only once.
In [26]: try:
...: 1/0
...: except ZeroDivisionError:
...: print("Caught")
...: try:
...: pass
...: except ZeroDivisionError:
...: print("Caught Again")
Caught
So my question is..kind of irrelevant.

Python - better solution for loops - Re-run after getting a error & ignore that error after 3 attempts

I created below for loop to run a function to get price data from pandas for a list of tickers. Basically, the loop will re-run the function if getting RemoteDataError and ignore that error after 3 times attempts.
Even though below for loop is working fine for this purpose, I do think there have a better solution since I can not define the times of attempts from below loop, like putting a while loop for times of attempt outside the for loop. I tried to define a variable named attempts = 0, every time it re-run, one attempts will be added. The logic is attempts += 1. If attempts reached 3, use continue to ignore the error. However, it didn't work. Probably I set something wrongly.
for ticker in tickers:
print(ticker)
try:
get_price_for_ticker()
except RemoteDataError:
print('No information for {}'.format(ticker))
try:
get_price_for_ticker()
print('Got data')
except RemoteDataError:
print('1st with no data')
try:
get_price_for_ticker()
print('Got data')
except RemoteDataError:
print('2nd with no data')
try:
get_price_for_ticker()
print('Got data')
except RemoteDataError:
print('3rd with no data (should have no data in the database)')
continue
Is there a better method for this purpose?
Is there a better method for this purpose?
Yes, there is. Use a while loop and a counter.
count = 0
while count < 3:
try:
get_price_for_ticker()
break # reach on success
except RemoteDataError:
print('Retrying {}'.format(count + 1))
count += 1 # increment number of failed attempts
if count == 3:
... # if count equals 3, the read was not successful
This code should go inside your outer for loop. Alternatively, you could define a function with the while + error handling code that accepts a ticker parameter, and you can call that function at each iteration of the for loop. It's a matter of style, and upto you.

python break a function nested inside a loop

I have the following piece of code:
for x in Listofurls:
function(urlquery)
function(htmlmining)
how the statement in function should be written
so that i can continue the loop moving to the next item
when the query does not match my research like
def(urlquery):
url=urlquery
Urlopen = urllib.request.urlopen(url)
Url_read = parse(Urlopen)
if 'text' not in Url_read.read():
#here is where i want a statement to stop and go to the next
#item in the loop like 'continue' in a for loop
You can use StopIteration to exit from all loops until it's caught;
try:
for i in range(10):
if i == 5:
raise StopIteration
else:
print i
except StopIteration:
print "Caught"
gives:
0
1
2
3
4
Caught
The StopIteration Exception is exactly that, an exception, not an error;
Raised by an iterator‘s next() method to signal that there are no
further values. This is derived from Exception rather than
StandardError, since this is not considered an error in its normal
application.
You can put it as deeply as you want in your nested loops, but you have to catch it at the level you want to break out of (i.e. the level at which you want to stop iteration) to.
Looking at your question, and trying to make sense of what you've written, it looks like you want to do something like this (maybe?)
for url in listOfURLs:
if urlquery(url):
htmlmining(url)
def urlquery(url):
page = parse(urllib.request.urlopen(url))
return 'text' in page.read():
#here is where i want a statement to stop and go to the next
#item in the loop like 'continue' in a for loop
This will then only run htmlmining(url) when 'text' is in the page you're parsing. If it's not, it will skip that entry and move onto the next one.
Have the inner function return True if you want to continue:
def urlquery(url):
urlopen = urllib.request.urlopen(url)
url_read = parse(urlopen)
if 'text' not in url_read.read():
# here is where I want a statement to stop and go to the next
return True
Then, the outer function can be:
for x in list_of_urls:
if urlquery(x):
continue
htmlmining(x)
Note that the code you posted was not valid Python. The above is my best guess as to what you meant.
Also, please read the Python style guide.
i finally found a solution to the question:
def urlquery(url):
urlopen = urllib.request.urlopen(url)
url_read = parse(urlopen)
if 'text' not in url_read.read():
return
else:
myurl='text' in url_read.read()
return myurl
and the for loop as follows:
for x in Listofurls:
TextAvailable=function(urlquery)
if not TextAvailable:
continue
function(htmlmining)
i am not sure this is the cleanest way to proceed but it works.

Try statement - multiple conditions - Python 2

I have little problem with try statement along with multiple conditions. When there is error at 2nd condition, it asks for 1st condition. What I want from it to do is to repeat the same condition, not the whole cycle. I hope you understand me, since my English isn't very good and also I'm newbie to Python so I also don't know how to describe it in my native language.
I hope the following example will help you to better understand my thought.
while True:
try:
zacatek = float(raw_input("Zacatek: "))
konec = float(raw_input("Konec: "))
except Exception:
pass
else:
break
it does following:
Zacatek: 1
Konec: a
Zacatek:
but I want it to do this:
Zacatek: 1
Konec: a
Konec:
Thanks in advance for any help.
Write a function to query for a single float, and call it twice:
def input_float(msg):
while True:
try:
return float(raw_input(msg))
except ValueError:
pass
zacatek = input_float("Zacatek: ")
konec = input_float("Konec: ")
What's happening is that your except clause is catching a ValueError exception on your answer to Konec and returning to the top of the loop.
Your float function is trying to cast a non-numeric response "a" to a float and it throwing the exception.
Alternatively, you could write a different loop for each input:
zacatek = None
while not zacatek:
try:
zacatek = float(raw_input("Zacatek: "))
except Exception:
continue
konec = None
while not konec:
try:
konec = float(raw_input("Konec: "))
except Exception:
continue

Execute if no exception thrown

I have some code I want to execute if an exception is not thrown.
Currently I'm doing this:
try:
return type, self.message_handlers[type](self, length - 1)
finally:
if not any(self.exc_info()):
self.last_recv_time = time.time()
Can this be improved on? Is this the best way to do it?
Update0
The optional else clause is executed if and when control flows off the
end of the try clause.
Currently, control “flows off the end” except in the case of an
exception or the execution of a return, continue, or break statement.
try:
tmp = type, self.message_handlers[type](self, length - 1)
except Exception:
pass #or handle error, or just "raise" to re-raise
else:
self.last_recv_time = time.time()
return tmp
Your code suggests that you don't want to catch the exception if it occurs, so why not simply
result = type, self.message_handlers[type](self, length - 1)
self.last_recv_time = time.time()
return result
(Am I missing anything?)

Categories