I have a nested loop to get certain JSON elements the way I want, but occasionally, the API I'm fetching from gets messy and it breaks some of the fields - I am not exactly sure how to handle this since It seems to be different each time, so I'm wondering if there is a way to continue a nested for loop even if an exception occurs inside it, or at least go back to the first loop and continue again.
My code is like this:
fields = ['email', 'displayname', 'login']
sub_fields = ['level', 'name']
all_data = []
for d in data:
login_value = d['login']
if login_value.startswith('3b3'):
continue
student = fetched_student.student_data(login_value)
student = json.loads(student)
final_json = dict()
try:
for field in fields:
#print ("Student field here: %s" % student[field])
final_json[field] = student[field]
except Exception as e:
print (e) # this is where I get a random KeyValue Error
#print ("Something happening here: %s " % final_json[field])
finally:
for sub_field in sub_fields:
for element in student['users']:
if element.get(sub_field):
final_json[sub_field] = element.get(sub_field)
for element in student['campus']:
if element.get(sub_field):
final_json[sub_field] = element.get(sub_field)
all_data.append(final_json)
print (all_data)
Is there a way to just go back to the first try block and continue after the exception has occurred or simply just ignore it and continue?
Because as things are now, if the exception ever occurs it breaks everything.
EDIT1: I have tried putting continue like so:
try:
for field in fields:
#print ("Student field here: %s" % student[field])
final_json[field] = student[field]
except Exception as e:
print (e)
continue
for sub_field in sub_fields:
for element in student['users']:
But it still fails regardless.
Use this for the try block:
for field in fields:
try:
#print ("Student field here: %s" % student[field])
final_json[field] = student[field]
except Exception as e:
print (e)
continue
for sub_field in sub_fields:
for element in student['users']:
The issue is due to the indentation level of the try block, the continue was affecting the outer most loop. Changing the try block to be inside of the loop will catch the error in that loop and continue the iteration of that specific loop.
Possibly you can use dict's get method like this in your try block:
try:
for field in fields:
#print ("Student field here: %s" % student[field])
final_json[field] = student.get(field, "") # 2nd arg is fallback object
Depending on what is needed, you can pass in an fresh dict (aka JSON object), fresh list (aka JSON array), or a str like above to suit your downstream needs.
Related
I loop though a list of currencies in order to download price series from an API and it happens that some of them are not supported so that it raises a module defined exception class : ExchangeError: This currency pair is not supported.
When it occurs I would like to continue the loop to the next currency but for some reason I'm unable to handle the module exception.
Here is an example that works fine with a built-in exception :
f = [1,2,3,4,'A',5]
def foo(nb):
return nb /2
for i in f :
try:
print(foo(i))
except TypeError :
continue
As expected it returns :
0.5
1
1.5
2
2.5
But as soon as it is a module (or user defined) exception it throws an error saying the exception is not defined :
#retry(wait_exponential_multiplier=1000, wait_exponential_max=10000)
def apiFetchOHLC(obj, currency, timeframe, option):
ohlcv = obj().fetch_ohlcv(currency, timeframe, since = option)
return ohlc
for c in currencies_list :
...
try :
# Download data
ohlc = apiFetchOHLC(obj, c, tf, maxCandlesLimit)
# except : # works fine
except ExchangeError : # doesn't work
print("Oops! That was no valid currency. Continue...")
continue
This is the error I get when I run the loop :
except ExchangeError:
NameError: name 'ExchangeError' is not defined
To make it works I need to remove the exception type ExchangeError but to me it is not a workaround because it will continue the loop whatever the exception is, and sometimes I need to retry the download.
How can I achieve this with try and except or with the retrying package ? (link)
def foo(count):
try:
while(count < 10):
if(count%2 == 1):
raise Exception()
print(count)
count = count+1
except:
print( str(count) + ' -> Exception')
foo(count+1)
foo(2)
Whenever an exception occurs in a try block, handle it in except block as follows -
To continue the process you are doing in try block, you should reach the try block from an except block - keep the try block in a function, name it foo, so that you could call it from the except block
To continue from the next iteration, you need to know the previous iteration where the exception has been raised - pass an argument to that function
After identifying the problem better I have found that I needed to give the full name space of the exception class I want to catch:
for c in currencies_list :
...
try :
# Download data
ohlc = apiFetchOHLC(obj, c, tf, maxCandlesLimit)
except ccxt.ExchangeError :
print("Oops! That was no valid currency. Continue...")
continue
Thats my first question on Stackoverflow and im a totally Python beginner.
I want to write, to get firm with python, a small Backup-Programm, the main part is done, but now i want to make it a bit "portable" and use a Config file, which i want to Validate.
My class "getBackupOptions" should be give Back a validate dict which should be enriched with "GlobalOptions" and "BackupOption" so that i finally get an fully "BackupOption" dict when i call "getBackupOptions.BackupOptions".
My Question now is, (in this Example is it easy, because its only the Function which check if the Path should be Recursive searched or not) how to simplify my Code?
For each (possible) Error i must write a new "TryExcept" Block - Can i Simplify it?
Maybe is there another way to Validate Config Files/Arrays?
class getBackupOptions:
def __init__(self,BackupOption,GlobalOptions):
self.BackupOption = BackupOption
self.GlobalOptions = GlobalOptions
self.getRecusive()
def getRecusive(self):
try:
if self.BackupOption['recursive'] != None:
pass
else:
raise KeyError
except KeyError:
try:
if self.GlobalOptions['recursive'] != None:
self.BackupOption['recursive'] = self.GlobalOptions['recursive']
else:
raise KeyError
except KeyError:
print('Recusive in: ' + str(self.BackupOption) + ' and Global is not set!')
exit()
Actually i only catch an KeyError, but what if the the Key is there but there is something else than "True" or "False"?
Thanks a lot for you help!
You may try this
class getBackupOptions:
def __init__(self,BackupOption,GlobalOptions):
self.BackupOption = BackupOption
self.GlobalOptions = GlobalOptions
self.getRecusive()
def getRecusive(self):
if self.BackupOption.get('recursive') == 'True' and self.GlobalOptions.get('recursive') == 'True':
self.BackupOption['recursive'] = self.GlobalOptions['recursive']
else:
print('Recusive in: ' + str(self.BackupOption) + ' and Global is not set!')
exit()
Here get method is used, therefore KeyError will not be faced.
If any text other than True comes in the field it will be considered as False.
I have a dictionary list of size ~250k in python (i.e 250k dictionaries in a list), which I try to process as shown below. The aim is to clean up the dictionary and return an iterable at the end. So, I have something like this:
def check_qs(dict_list_in):
try:
del_id=[]
for i in dict_list_in:
tmp=i["get_url"][0]
if i["from"][0]=="var0":
try:
URLValidator()(tmp)
except:
del_id.append( i["id"] )
elif i["from"][0]=="var1":
try:
URLValidator()( tmp.split("\"")[1] )
except:
del_id.append( i["id"] )
elif i["from"][0]=="var2":
try:
URLValidator()( tmp.split("\'")[1] )
except:
del_id.append( i["id"] )
else:
del_id.append( i["id"] )
gc.collect()
result = filter(lambda x: x['id'] not in del_id,dict_list_in)
return result
except:
return dict_list_in
What I am doing above, is checking each dictionary in ths list for some condition, and if this fails, I get the id and then use filter to delete those dictionaries specific from the list.
At the moment, this takes a long time to run - and I was wondering if there were any obvious optimizations I am missing out on. I think at the moment the above code is too naive.
I made a couple changes. I put the validation instance out of the loop so that you don't have to initialize it every time. If it's required to instantiate every time, just move it into the try accept block. I also changed from deleting items in the original list, to appending the items to a new list that you want, removing the need for a filter. I also moved the validation out of the if statements so that if you hit the else statement you don't have to run the validation. Look at the logic of the if statements, it is the same as yours. It appears that you are using django, but if you aren't change the except to except Exception.
from django.core.exceptions import ValidationError
def check_qs(dict_list_in):
new_dict_list = []
validate = URLValidator()
for i in dict_list_in:
test_url = i["get_url"][0]
if i["from"][0] == "var0":
pass
elif i["from"][0] == "var1":
test_url = test_url.split("\"")[1]
elif i["from"][0] == "var2":
test_url = test_url.split("\'")[1]
else:
continue
try:
validate(test_url)
# If you aren't using django you can change this to 'Exception'
except ValidationError:
continue
new_dict_list.append(i)
return new_dict_list
I have a request with JSON data, it may or may not contain 'items' key, if it does it has to be a list of objects, that I want to process individually. So I have to write something like:
json_data = request.get_json()
for item in json_data['items']:
process_item(item)
But, since presence of the 'items' key is not mandatory, an additional measure needs to be taken. I would like to follow EAFP approach, so wrapping it up into try ... except statement:
json_data = request.get_json()
try:
for item in json_data['items']:
process_item(item)
except KeyError as e:
pass
Let's assume that a KeyError exception can happened inside the process_item(...) function, that may indicate a code error, thus it should not go unnoticed, so I want to make sure that I will catch only exceptions coming from for statement predicate, as a workaround I came up with:
json_data = request.get_json()
try:
for item in json_data['items']:
process_item(item)
except KeyError as e:
if e.message != 'items':
raise e
pass
But
It looks ugly
It relies on knowledge of the process_item(...) implementation, assuming that KeyError('items') cannot be raised inside of it.
If the for statement becomes more complex e.g. for json_data['raw']['items'] so will the except clause making it even less readable and maintainable.
Update:
The suggested alternative
json_data = request.get_json()
try:
items = json_data["items"]
except KeyError:
items = []
for item in items:
process_item(item)
is essentially the same as
json_data = request.get_json()
if json_data.has('items')
items = json_data['items']
else:
items = []
for item in items:
process_item(item)
So we check before we loop. I would like to know if there is any more pythonic/EAFP approach?
You can catch the exception only when accessing "items":
json_data = request.get_json()
try:
items = json_data["items"]
except KeyError:
items = []
for item in items:
process_item(item)
However, we can replace the try-block with a call to the .get() function, making it much cleaner:
for item in request.get_json().get("items", []):
process_item(item)
I think the cleanest option is to use atryblock around only the code that attempts to retrieve the data associated with the'items'key:
json_data = request.get_json()
try:
items = json_data['items']
except KeyError:
print "no 'items' to process" # or whatever you want to...
else:
for item in items:
process_item(item)
This layout will allow to you clearly separate the error handling as you see fit. You can add a separate independenttry/exceptaround theforloop if desired.
Is there any way to return back and repeat the instruction that was handling an exception in Python?
E.g. if we get some data by input() method, and for some reason is caused an exception (e.g. when trying to convert the input string into int), we raised the exception, but after the exception, I would like again to go to the same line where the input() is.
Just note, "continue" is not an option, even if it is in a loop, because it could be several different input() assigning them to a different variables in different parts of the loop.
So the question again is:
while 1:
try:
foo = int(input(">")
...some other code here...
bar = int(input(">")
...some other code here...
fred = int(input(">")
...some other code here...
except Exception:
... do something for error handling and ...
jump_back_and_repeat_last_line_that_caused_the_exception
Imagine that the above code could be in a loop, and the exception can be caused in any instruction (foo... bar... fred...etc, or even can be any other line). So, if it fails in the "bar" line, it should try again the "bar" line.
Is there any reserved word to do this in python?
Define a function; Handle exception there.
def read_int():
while 1:
try:
value = int(input('>'))
except ValueError:
# Error handling + Jump back to input line.
continue
else:
return value
while 1:
foo = read_int()
bar = read_int()
fred = read_int()
There might be a way to do that, but it will probably result with a very poor design.
If I understand you correctly, then your problem is with the exception caused by calling input.
If that is indeed the case, then you should simply implement it in a separate method, which will handle the exception properly:
foo = getUserInput()
...some other code here...
bar = getUserInput()
...some other code here...
fred = getUserInput()
...some other code here...
def getUserInput():
while 1:
try:
return int(input(">"))
except Exception:
pass
don't do nothing in except:
while 1:
try:
a=int(raw_input('input an integer: ')) #on python2 it's "raw_input" instead of "input"
break
except ValueError, err:
# print err
pass
print 'user input is:', a
output is:
D:\U\ZJ\Desktop> py a.py
input an integer: a
input an integer: b
input an integer: c
input an integer: 123
user input is: 123