I have a request with JSON data, it may or may not contain 'items' key, if it does it has to be a list of objects, that I want to process individually. So I have to write something like:
json_data = request.get_json()
for item in json_data['items']:
process_item(item)
But, since presence of the 'items' key is not mandatory, an additional measure needs to be taken. I would like to follow EAFP approach, so wrapping it up into try ... except statement:
json_data = request.get_json()
try:
for item in json_data['items']:
process_item(item)
except KeyError as e:
pass
Let's assume that a KeyError exception can happened inside the process_item(...) function, that may indicate a code error, thus it should not go unnoticed, so I want to make sure that I will catch only exceptions coming from for statement predicate, as a workaround I came up with:
json_data = request.get_json()
try:
for item in json_data['items']:
process_item(item)
except KeyError as e:
if e.message != 'items':
raise e
pass
But
It looks ugly
It relies on knowledge of the process_item(...) implementation, assuming that KeyError('items') cannot be raised inside of it.
If the for statement becomes more complex e.g. for json_data['raw']['items'] so will the except clause making it even less readable and maintainable.
Update:
The suggested alternative
json_data = request.get_json()
try:
items = json_data["items"]
except KeyError:
items = []
for item in items:
process_item(item)
is essentially the same as
json_data = request.get_json()
if json_data.has('items')
items = json_data['items']
else:
items = []
for item in items:
process_item(item)
So we check before we loop. I would like to know if there is any more pythonic/EAFP approach?
You can catch the exception only when accessing "items":
json_data = request.get_json()
try:
items = json_data["items"]
except KeyError:
items = []
for item in items:
process_item(item)
However, we can replace the try-block with a call to the .get() function, making it much cleaner:
for item in request.get_json().get("items", []):
process_item(item)
I think the cleanest option is to use atryblock around only the code that attempts to retrieve the data associated with the'items'key:
json_data = request.get_json()
try:
items = json_data['items']
except KeyError:
print "no 'items' to process" # or whatever you want to...
else:
for item in items:
process_item(item)
This layout will allow to you clearly separate the error handling as you see fit. You can add a separate independenttry/exceptaround theforloop if desired.
Related
I have a nested loop to get certain JSON elements the way I want, but occasionally, the API I'm fetching from gets messy and it breaks some of the fields - I am not exactly sure how to handle this since It seems to be different each time, so I'm wondering if there is a way to continue a nested for loop even if an exception occurs inside it, or at least go back to the first loop and continue again.
My code is like this:
fields = ['email', 'displayname', 'login']
sub_fields = ['level', 'name']
all_data = []
for d in data:
login_value = d['login']
if login_value.startswith('3b3'):
continue
student = fetched_student.student_data(login_value)
student = json.loads(student)
final_json = dict()
try:
for field in fields:
#print ("Student field here: %s" % student[field])
final_json[field] = student[field]
except Exception as e:
print (e) # this is where I get a random KeyValue Error
#print ("Something happening here: %s " % final_json[field])
finally:
for sub_field in sub_fields:
for element in student['users']:
if element.get(sub_field):
final_json[sub_field] = element.get(sub_field)
for element in student['campus']:
if element.get(sub_field):
final_json[sub_field] = element.get(sub_field)
all_data.append(final_json)
print (all_data)
Is there a way to just go back to the first try block and continue after the exception has occurred or simply just ignore it and continue?
Because as things are now, if the exception ever occurs it breaks everything.
EDIT1: I have tried putting continue like so:
try:
for field in fields:
#print ("Student field here: %s" % student[field])
final_json[field] = student[field]
except Exception as e:
print (e)
continue
for sub_field in sub_fields:
for element in student['users']:
But it still fails regardless.
Use this for the try block:
for field in fields:
try:
#print ("Student field here: %s" % student[field])
final_json[field] = student[field]
except Exception as e:
print (e)
continue
for sub_field in sub_fields:
for element in student['users']:
The issue is due to the indentation level of the try block, the continue was affecting the outer most loop. Changing the try block to be inside of the loop will catch the error in that loop and continue the iteration of that specific loop.
Possibly you can use dict's get method like this in your try block:
try:
for field in fields:
#print ("Student field here: %s" % student[field])
final_json[field] = student.get(field, "") # 2nd arg is fallback object
Depending on what is needed, you can pass in an fresh dict (aka JSON object), fresh list (aka JSON array), or a str like above to suit your downstream needs.
I have a simple for loop which iterates through all objects in a local database. For each object, I reference a presalesEngineer and pass that ID to an API call to retrieve a JSON response. However, there are records in the database for which there is no value for presalesEngineer. When this is the case, the empty string throws a HttpError when a bad URL is passed to the API call. How can I handle when presalesEngineer does not exist, so the API is not passed an empty value?
views.py
objects = Opportunity.objects.all()
for object in objects:
try:
ps_e = object.presalesEngineer
if ps_e:
presales_engineers = [cwObj.get_member_by_id(ps_e) for object in objects]
else:
presales_engineers = 'None'
except NameError:
presales_engineers = 'None'
This codeblock should try to grab the presalesEngineer of your object or return None (note that the string 'None' does not equal the pytohon object None)
for object in objects:
try:
ps_e = object.presalesEngineer
# Do stuff with an object you know for sure will not trigger an exception
# Something like:
# if ps_e != '': < the object is not an empty string
# or
# if ps_e: < the object is not None
# after you pass whatever checks you deem necessary, you launch your API call.
except AttributeError:
# You can either pass here or return a None object/Empty list
ps_e = None
Possible implementation below:
# Empty list of whatever you are searching for
engineers = []
for my_object in objects:
try:
ps_e = my_object.presalesEngineer
# This is here to avoid none values in your API call
if ps_e:
# Just in case your API call falls
# It will fail silently in this try codeblock
try:
# Assuming cwObj is your driver/API endpoint builder
# And that you only get one single string as response
# And that string is not some data structure that you need to split
my_desired_id = cwObj.get_member_by_id(ps_e)
engineers.append(my_desired_id)
# Using bare except statements is not a good idea
# Use HttpError here if you don't want to pass on any exception
except:
pass
except AttributeError:
# You can either pass here or return a None object/Empty list
pass
print engineers
I have a dictionary list of size ~250k in python (i.e 250k dictionaries in a list), which I try to process as shown below. The aim is to clean up the dictionary and return an iterable at the end. So, I have something like this:
def check_qs(dict_list_in):
try:
del_id=[]
for i in dict_list_in:
tmp=i["get_url"][0]
if i["from"][0]=="var0":
try:
URLValidator()(tmp)
except:
del_id.append( i["id"] )
elif i["from"][0]=="var1":
try:
URLValidator()( tmp.split("\"")[1] )
except:
del_id.append( i["id"] )
elif i["from"][0]=="var2":
try:
URLValidator()( tmp.split("\'")[1] )
except:
del_id.append( i["id"] )
else:
del_id.append( i["id"] )
gc.collect()
result = filter(lambda x: x['id'] not in del_id,dict_list_in)
return result
except:
return dict_list_in
What I am doing above, is checking each dictionary in ths list for some condition, and if this fails, I get the id and then use filter to delete those dictionaries specific from the list.
At the moment, this takes a long time to run - and I was wondering if there were any obvious optimizations I am missing out on. I think at the moment the above code is too naive.
I made a couple changes. I put the validation instance out of the loop so that you don't have to initialize it every time. If it's required to instantiate every time, just move it into the try accept block. I also changed from deleting items in the original list, to appending the items to a new list that you want, removing the need for a filter. I also moved the validation out of the if statements so that if you hit the else statement you don't have to run the validation. Look at the logic of the if statements, it is the same as yours. It appears that you are using django, but if you aren't change the except to except Exception.
from django.core.exceptions import ValidationError
def check_qs(dict_list_in):
new_dict_list = []
validate = URLValidator()
for i in dict_list_in:
test_url = i["get_url"][0]
if i["from"][0] == "var0":
pass
elif i["from"][0] == "var1":
test_url = test_url.split("\"")[1]
elif i["from"][0] == "var2":
test_url = test_url.split("\'")[1]
else:
continue
try:
validate(test_url)
# If you aren't using django you can change this to 'Exception'
except ValidationError:
continue
new_dict_list.append(i)
return new_dict_list
I have a feeling that this sequence might be written shorter:
dim = Dimension.objects.get(pk=rows['pk'])
try:
dim.name = rows['name']
except KeyError:
pass
try:
dim.external_flg = rows['external_flg']
except:
pass
try:
dim.ext_owner = rows['ext_owner']
except KeyError:
pass
try:
dim.ext_table_name = rows['ext_table_name']
except KeyError:
pass
try:
dim.ext_start_date_column_name = rows['ext_start_date_column_name']
except KeyError:
pass
try:
dim.ext_end_date_column_name = rows['ext_end_date_column_name']
except KeyError:
pass
I've never had any experience in Python code optimization and working with exceptions but I'd be glad to have an alternative examples how it could be shortened.
Thank you!
Use the dict.get() method:
dim.name = rows.get('name', dim.name)
Or, for more DRY solution, put all assignments into the loop:
for field_name in ('name', 'external_flg', 'ext_owner', 'ext_table_name',
'ext_start_date_column_name', 'ext_end_date_column_name'):
if field_name in rows:
setattr(dim, field_name, rows[field_name])
To handle the list data you can make the similar loop:
for i, field_name in enumerate(('name', 'external_flg', 'ext_owner',
'ext_table_name', 'ext_start_date_column_name',
'ext_end_date_column_name')):
if i < len[rows]:
setattr(dim, field_name, rows[i])
Something like this should work — this answer includes try/catch block to catch KeyError :-)
attributes = ['name', 'external_flg', 'ext_owner', 'ext_table_name',
'ext_start_date_column_name', 'ext_end_date_column_name']
dim = Dimension.objects.get(pk=rows['pk'])
for attr in attributes:
try:
setattr(dim, attr, rows[attr])
except KeyError:
pass
I have this long list of try except statement:
try:
uri = entry_obj['media$group']['media$content'][0]['url']
except (KeyError, IndexError):
uri = None
try:
position = entry_obj['yt$position']['$t']
except KeyError:
position = None
try:
description = entry_obj['content']['$t']
except KeyError:
description = None
try:
seconds = entry_obj['media$group']['yt$duration']['seconds']
except KeyError:
seconds = None
try:
thumbnails = entry_obj['media$group']['media$thumbnail']
except KeyError:
thumbnails = None
Is there a more concise way to write this?
If you tire of figuring out what to use for default values in get() calls, just write a helper function:
def resolve(root, *keys):
for key in keys:
try:
root = root[key]
except (KeyError, IndexError):
return None
return root
Then you just write, e.g.:
uri = resolve(entry_obj, 'media$group', 'media$content', 0, 'url')
To simplify the calls a little, you might beef up the helper function to take a single string for the keys and split on spaces; that way you don't have to type so many quotes, and we can also add a default value argument:
def resolve(root, keys, default=None):
for key in keys.split():
try:
root = root[key]
except (TypeError, KeyError):
try:
root = root[int(key)]
except (IndexError, ValueError, KeyError):
return default
uri = resolve(entry_obj, 'media$group media$content 0 url', '')
I thought of another good way to do this, not sure how it compares to kindall's method. We first define a method property:
def res(self, property):
try:
return property()
except (KeyError, IndexError):
return None
Then replace the try-except statements with:
url = res(lambda: entry_obj['media$group']['media$content'][0]['url'])
position = res(lambda: entry_obj['yt$position']['$t'])
description = res(lambda: entry_obj['content']['$t'])
duration = res(lambda: entry_obj['media$group']['yt$duration']['seconds'])
thumbnails = res(lambda: entry_obj['media$group']['media$thumbnail'])
Use the get method of dictionaries instead:
position = entry_object.get('yt$position').get('$t')
get will handle the case of a key not existing for you, and give you a (changable) fallback value instead in that case. You'll still need to handle the first IndexError manually, but all the ones that are just except KeyError: will disappear.