I am trying to check if I have an entry in my database using this code:
def device_update(request):
json_data = json.loads(request.body)
email = json_data['email']
imei = json_data['imei']
sdk_version = json_data['sdk_version']
date = json_data['updateDate']
rule = json_data['ruleName']
group_name = json_data['group']
if Group.objects.filter(group=group_name).exists():
print("group does exists")
else:
print("group doesn't exists")
return HttpResponse("Successful")
However, when the code reaches the if statement to check if the group exists, it returns error 500.
I tried to check with two groups one that exists and another one that doesn't, in both cases I got error 500.
How can I fix this and why is this happening?
The logic for checking if a Group exists, i.e. the line:
if Group.objects.filter(group=group_name).exists()
is not throwing the error here. It is likely that json_data is missing one of the keys you expect it to have, for example, 'group'.
I'd recommend using the get method that dictionaries have. This provides default values when the specified key is not present in the dictionary. You should also have error handling for when the request body is not in valid JSON format.
Here's an example:
def device_update(request):
try:
json_data = json.loads(request.body)
except json.JSONDecodeError:
return HttpResponse('Request body must be in valid JSON format')
email = json_data.get('email', '')
imei = json_data.get('imei', '')
sdk_version = json_data.get('sdk_version', '')
date = json_data.get('updateDate', '')
rule = json_data.get('ruleName', '')
group_name = json_data.get('group', '')
if Group.objects.filter(group=group_name).exists():
print("group does exists")
else:
print("group doesn't exists")
return HttpResponse("Successful")
I set the defaults to the empty string '', but you may want to change that.
Your view doesn't have any error handling. Looking at it quickly, at least two things could go wrong. The request body might not be valid json, and if it is valid json, it might not contain the required keys.
def device_update(request):
try:
json_data = json.loads(request.body)
except ValueError:
return HttpResponse("Invalid json")
try:
email = json_data['email']
imei = json_data['imei']
sdk_version = json_data['sdk_version']
date = json_data['updateDate']
rule = json_data['ruleName']
group_name = json_data['group']
except KeyError as e:
return HttpResponse("Missing Key %s" % e[0])
...
Writing your own validation for a single view like this is ok. As it gets more complicated, you might want to look at django rest framework. It has serializers which will help you manage validation.
Alasdair/Keselme, looks that your view is correct.
Try to put the ipdb into your code in order to debug your code, and than you can print the request.data and see what is comming in the request.
Related
I'm fairly new to Python so bear with me please.
I have a function that takes two parameters, an api response and an output object, i need to assign some values from the api response to the output object:
def map_data(output, response):
try:
output['car']['name'] = response['name']
output['car']['color'] = response['color']
output['car']['date'] = response['date']
#other mapping
.
.
.
.
#other mapping
except KeyError as e:
logging.error("Key Missing in api Response: %s", str(e))
pass
return output
Now sometimes, the api response is missing some keys i'm using to generate my output object, so i used the KeyError exception to handle this case.
Now my question is, in a case where the 'color' key is missing from the api response, how can i catch the exception and continue to the line after it output['car']['date'] = response['date'] and the rest of the instructions.
i tried the pass instruction but it didn't have any affect.
Ps: i know i can check the existence of the key using:
if response.get('color') is not None:
output['car']['color'] = response['color']
and then assign the values but seeing that i have about 30 values i need to map, is there any other way i can implement ? Thank you
A few immediate ideas
(FYI - I'm not going to explain everything in detail - you can check out the python docs for more info, examples etc - that will help you learn more, rather than trying to explain everything here)
Google 'python handling dict missing keys' for a million methods/ideas/approaches - it's a common use case!
Convert your response dict to a defaultdict. In that case you can have a default value returned (eg None, '', 'N/A' ...whatever you like) if there is no actual value returned.
In this case you could do away with the try and every line would be executed.
from collections import defaultdict
resp=defaultdict(lambda: 'NA', response)
output['car']['date'] = response['date'] # will have value 'NA' if 'date' isnt in response
Use the in syntax, perhaps in combination with a ternary else
output['car']['color'] = response['color'] if 'color' in response
output['car']['date'] = response['date'] if 'date' in response else 'NA'
Again you can do away with the try block and every line will execute.
Use the dictionary get function, which allows you to specify a default if there is no value for that key:
output['car']['color'] = response.get('car', 'no car specified')
You can create a utility function that gets the value from the response and if the value is not found, it returns an empty string. See example below:
def get_value_from_response_or_null(response, key):
try:
value = response[key]
return value
except KeyError as e:
logging.error("Key Missing in api Response: %s", str(e))
return ""
def map_data(output, response):
output['car']['name'] = get_value_from_response_or_null(response, 'name')
output['car']['color'] = get_value_from_response_or_null(response, 'color')
output['car']['date'] = get_value_from_response_or_null(response, 'date')
# other mapping
# other mapping
return output
I try to get data from a webpage. This page contains several release information, but allow values not to be set. I.e. the date for testing from/to might be an empty string.
Now I try to deserialize all my data sucked from the page to insert it to a database and face problems handling empty dates.
from marshmallow import fields, Schema, ValidationError
class TestSchema(Schema):
training_necessary = fields.Function(deserialize=lambda x: True if x == 'Yes' else False)
test_from = fields.Date()
test_to = fields.Date()
data = dict(training_necessary='Yes', test_from='', test_to='')
try:
validated = TestSchema().load(data)
except ValidationError as err:
print(f"{err}")
Result:
{'test_to': ['Not a valid date.'], 'test_from': ['Not a valid date.']}
I already tried several combinations of allow_none=True or default='' but none of them helped my to get through. So, how to manage to allow empty dates? Setting a default to somewhat like 1970-01-01 won't help in that case.
Any hints?
Regards, Thomas
+++ EDIT: SOLUTION +++
Here's the working code I ended up after Jérômes helpful tipp:
from marshmallow import fields, Schema, ValidationError, pre_load
class TestSchema(Schema):
training_necessary = fields.Function(deserialize=lambda x: True if x == 'Yes' else False)
test_from = fields.Date(allow_none=True)
test_to = fields.Date(allow_none=True)
#pre_load(pass_many=False)
def string_to_none(self, data, many, **kwargs):
turn_to_none = lambda x: None if x == '' else x
for k, v in data.items():
data[k] = turn_to_none(v)
return data
data = dict(training_necessary='Yes', test_from='', test_to='')
try:
validated = TestSchema().load(data)
except ValidationError as err:
print(f"{err}")
I would pass no value at all.
data = dict(training_necessary='Yes')
Or I'd make the date fields allow_none and I'd pass None, not an empty string.
data = dict(training_necessary='Yes', test_from=None, test_to=None)
If the issue is that your input contains empty strings, I'd say this is a client issue, but you can add a pre_load method to delete empty strings from the input before deserializing. This is more or less equivalent to modifying the values you scrape from the page before feeding them to marshmallow.
I have a django form that checks if a postcode is valid and if valid it will perform a check against postcodes that are permitted for delivery. I cant get the second condition working in the nested if structure (works fine independently). This is the code:
from django import forms
import requests
class PostCodeForm (forms.Form):
pcode = forms.CharField()
def clean_pcode(self):
pcode = self.cleaned_data['pcode'].lower()
permitted = {'gu15','GF34','FG34','BT25'}
url = 'https://api.postcodes.io/postcodes/{}/validate'.format(pcode)
r = requests.get(url)
is_correct = r.json()['result']
if not is_correct:
raise forms.ValidationError("Your postcode is invalid. Please re-enter a valid entry.")
if not pcode[:4] in (permitted):
raise forms.ValidationError("Apologies, but does not currently deliver to you postcode.")
return pcode
Fix your indentation, so that the second if statment and return statement are outside of the first if statement.
def clean_pcode(self):
pcode = self.cleaned_data['pcode'].lower()
permitted = {'gu15','GF34','FG34','BT25'}
url = 'https://api.postcodes.io/postcodes/{}/validate'.format(pcode)
r = requests.get(url)
is_correct = r.json()['result']
if not is_correct:
raise forms.ValidationError("Your postcode is invalid. Please re-enter a valid entry.")
if not pcode[:4] in (permitted):
raise forms.ValidationError("Apologies, but does not currently deliver to you postcode.")
return pcode
As an aside, you should change the slicing from pcode[:4] to pcode[:-3]. This means that your code will still work if you add 3-character post districts (e.g. GU1) to your permitted set.
Also, since you call lower(), you need to make sure all your entries in permitted are lowercase. Personally, I would use upper() for normalising postcodes.
Im working on a small project of retrieving information about books from the Google Books API using Python 3. For this i make a call to the API, read out the variables and store those in a list. For a search like "linkedin" this works perfectly. However when i enter "Google", it reads the second title from the JSON input. How can this happen?
Please find my code below (Google_Results is the class I use to initialize the variables):
import requests
def Book_Search(search_term):
parms = {"q": search_term, "maxResults": 3}
r = requests.get(url="https://www.googleapis.com/books/v1/volumes", params=parms)
print(r.url)
results = r.json()
i = 0
for result in results["items"]:
try:
isbn13 = str(result["volumeInfo"]["industryIdentifiers"][0]["identifier"])
isbn10 = str(result["volumeInfo"]["industryIdentifiers"][1]["identifier"])
title = str(result["volumeInfo"]["title"])
author = str(result["volumeInfo"]["authors"])[2:-2]
publisher = str(result["volumeInfo"]["publisher"])
published_date = str(result["volumeInfo"]["publishedDate"])
description = str(result["volumeInfo"]["description"])
pages = str(result["volumeInfo"]["pageCount"])
genre = str(result["volumeInfo"]["categories"])[2:-2]
language = str(result["volumeInfo"]["language"])
image_link = str(result["volumeInfo"]["imageLinks"]["thumbnail"])
dict = Google_Results(isbn13, isbn10, title, author, publisher, published_date, description, pages, genre,
language, image_link)
gr.append(dict)
print(gr[i].title)
i += 1
except:
pass
return
gr = []
Book_Search("Linkedin")
I am a beginner to Python, so any help would be appreciated!
It does so because there is no publisher entry in volumeInfo of the first entry, thus it raises a KeyError and your except captures it. If you're going to work with fuzzy data you have to account for the fact that it will not always have the expected structure. For simple cases you can rely on dict.get() and its default argument to return a 'valid' default entry if an entry is missing.
Also, there are a few conceptual problems with your function - it relies on a global gr which is bad design, it shadows the built-in dict type and it captures all exceptions guaranteeing that you cannot exit your code even with a SIGINT... I'd suggest you to convert it to something a bit more sane:
def book_search(search_term, max_results=3):
results = [] # a list to store the results
parms = {"q": search_term, "maxResults": max_results}
r = requests.get(url="https://www.googleapis.com/books/v1/volumes", params=parms)
try: # just in case the server doesn't return valid JSON
for result in r.json().get("items", []):
if "volumeInfo" not in result: # invalid entry - missing volumeInfo
continue
result_dict = {} # a dictionary to store our discovered fields
result = result["volumeInfo"] # all the data we're interested is in volumeInfo
isbns = result.get("industryIdentifiers", None) # capture ISBNs
if isinstance(isbns, list) and isbns:
for i, t in enumerate(("isbn10", "isbn13")):
if len(isbns) > i and isinstance(isbns[i], dict):
result_dict[t] = isbns[i].get("identifier", None)
result_dict["title"] = result.get("title", None)
authors = result.get("authors", None) # capture authors
if isinstance(authors, list) and len(authors) > 2: # you're slicing from 2
result_dict["author"] = str(authors[2:-2])
result_dict["publisher"] = result.get("publisher", None)
result_dict["published_date"] = result.get("publishedDate", None)
result_dict["description"] = result.get("description", None)
result_dict["pages"] = result.get("pageCount", None)
genres = result.get("authors", None) # capture genres
if isinstance(genres, list) and len(genres) > 2: # since you're slicing from 2
result_dict["genre"] = str(genres[2:-2])
result_dict["language"] = result.get("language", None)
result_dict["image_link"] = result.get("imageLinks", {}).get("thumbnail", None)
# make sure Google_Results accepts keyword arguments like title, author...
# and make them optional as they might not be in the returned result
gr = Google_Results(**result_dict)
results.append(gr) # add it to the results list
except ValueError:
return None # invalid response returned, you may raise an error instead
return results # return the results
Then you can easily retrieve as much info as possible for a term:
gr = book_search("Google")
And it will be far more tolerant of data omissions, provided that your Google_Results type makes most of the entries optional.
Following #Coldspeed's recommendation it became clear that missing information in the JSON file caused the exception to run. Since I only had a "pass" statement there it skipped the entire result. Therefore I will have to adapt the "Try and Except" statements so errors do get handled properly.
Thanks for the help guys!
I am using ConfigObj and Validator to parse a configuration file in python. While I like this tool a lot, I am having trouble with validation using a configSpec file. I am using the option() configSpec type that forces the value to be chosen from a controlled vocabulary:
output_mode = option("Verbose", "Terse", "Silent")
I want my code to know when the user enters an option that's not in the CV. From what I have fond, Validator only seems to say which config key failed validation, but not why it failed:
from configobj import ConfigObj, flatten_errors
from validate import Validator
config = ConfigObj('config.ini', configspec='configspec.ini')
validator = Validator()
results = config.validate(validator)
if results != True:
for (section_list, key, _) in flatten_errors(config, results):
if key is not None:
print 'The "%s" key in the section "%s" failed validation' % (key, ', '.join(section_list))
else:
print 'The following section was missing:%s ' % ', '.join(section_list)
That code snippet works but there are any number of reasons why a key might have failed validation, from not being in an integer range to not being in the CV. I don't want to have to interrogate the key name and raise a different kind of exception depending on the failure cases for that key. Is there a cleaner way to handle specific types of validation errors?
Long time stackoverflow reader, first time poster :-)
Update: I think this does what I want to do. The key is that config obj stores errors as Exceptions which can then be checked against those that subclass ValidateError. Then you just have to do one check per subclass rather than one check per parameter value. It might be nicer if validate just threw an exception if validation failed but maybe you would lose other functionality.
self.config = configobj.ConfigObj(configFile, configspec=self.getConfigSpecFile())
validator = Validator()
results = self.config.validate(validator, preserve_errors=True)
for entry in flatten_errors(self.config, results):
[sectionList, key, error] = entry
if error == False:
msg = "The parameter %s was not in the config file\n" % key
msg += "Please check to make sure this parameter is present and there are no mis-spellings."
raise ConfigException(msg)
if key is not None:
if isinstance(error, VdtValueError):
optionString = self.config.configspec[key]
msg = "The parameter %s was set to %s which is not one of the allowed values\n" % (key, self.config[key])
msg += "Please set the value to be in %s" % optionString
raise ConfigException(msg)
OptionString is just a string of the form option("option 1", "option 2") rather than a list so to get this to look nice, you need to grab the substring in the ()'s.
For future reference for anyone interested, you could also check for extraneous data. This can be handled with the get_extra_values function. The complete example shown below hence does:
load the configuration with validator
look for all the validated errors
verify extra values
from configobj import ConfigObj, ConfigObjError, flatten_errors, get_extra_values
from validate import Validator, VdtValueError
def load_config(configfile, configspec, raise_exception=True):
"Load and check configvale acccording to spec"
config = ConfigObj(configfile, file_error=True, configspec=configspec)
validator = Validator()
results = config.validate(validator, preserve_errors=True)
msg = ""
fatalerr = False
for entry in flatten_errors(config, results):
[sectionList, key, error] = entry
if error is False:
msg += f"\n{key:>30s} missing in section [{']['.join(sectionList)}]"
fatalerr = True
if key is not None:
if isinstance(error, VdtValueError):
optionString = config.configspec[key]
msg += f"\nThe parameter {key} was set to {[config[s][key] for s in sectionList]} which is not one of the allowed values\n"
msg += " Please set the value to be in %s" % optionString
fatalerr = True
# verifying extra values below
wmsg = ""
for sections, name in get_extra_values(config):
# this code gets the extra values themselves
the_section = config
for section in sections:
the_section = the_section[section]
# the_value may be a section or a value
the_value = the_section[name]
section_or_value = 'value'
if isinstance(the_value, dict):
# Sections are subclasses of dict
section_or_value = 'section'
section_string = '[' + (']['.join(sections) or "TOP LEVEL") + ']'
wmsg += f"\n{name:>30s}: Extra {section_or_value} on section {section_string}"
if wmsg != "":
print(f"\nWARNINGS found in configuration file {configfile}")
print(wmsg)
if fatalerr:
print(f"\nERRORS found in configuration file {configfile}")
if raise_exception:
raise RuntimeError(msg)
else:
print("Fatal errors found, but no exception raised, as requested")
print(msg)
print(f'Configuration {configfile} validated successfully')
return config
if __name__ == "__main__":
configfile="xt_default.cfg"
configspec="xt_default_spec.cfg"
config = load_config(configfile, configspec)