I get data in JSON from an API and it may be that the received data is not complete (= some fields are missing). I am not sure either that the structure of the data follows JSON standards.
The solution for the second problem is simple: I will try: to decode the JSON and act accordingly on ValueError and TypeError exceptions.
For the first problem, my solution would also be to
d = {'a': 1}
try:
d['a']
d['b']
d['x']['shouldbethere']
except KeyError:
(...)
that is to list all the keys I need to have in the dict created from a successful JSON conversion.
This made me think that there may be a method to declare the expected keys (and possibly values types) and match the retrieved JSON against it, an unsuccessful match raising a specific exception?
Standard way to validate JSON structure is to use JSON Schema.
Basic characteristics (quoted from official webpage) are:
JSON Schema:
describes your existing data format
clear, human- and machine-readable documentation
complete structural validation, useful for
automated testing
validating client-submitted data
There is no built-in package to validate JSON object against schema, although you may use jsonschema from pypi.
Sample usage (paraphrased from official docs) may be:
import jsonschema
schema = {
"type": "object",
"properties": {
"price": {"type": "number"},
"name": {"type": "string"},
},
}
jsonschema.validate({"name": "Eggs", "price": 34.99}, schema)
# No exception from line above - document is valid
jsonschema.validate({"name": "Eggs", "price": "Invalid"}, schema)
# ValidationError: 'Invalid' is not of type 'number'
JSON parsers aren't terribly easy to use for error correction, so if the data isn't JSON I think it would be very difficult to apply any kind of auto-correction to allow you to parse it, so your solution for the invalid JSON is probably the most reasonable decision.
A function to verify that a dict contains a particular set of keys is relatively easy to implement. I'm not aware of any JSON object methods to perform that test, but given a JSON object j you could check it as follows (it might also be sensible to check that it's a dict, since JSON objects can also be of other types):
def has_all_keys(j, keylist):
return all(key in j for key in keylist)
Using this interactively suggests is might work (in this example I rely on the fact that iteration over a string yields the individual characters, but obviously you will need a real list of string key values).
>>> has_all_keys({}, "abc")
False
>>> has_all_keys({'a':1, 'b':1, 'c':1}, "abc")
True
Related
I have a JSON Schema file like this one, which contains a couple of intentional bugs:
{
"$schema": "http://json-schema.org/schema#",
"type": "object",
"description": "MWE for JSON Schema Validation",
"properties": {
"valid_prop": {
"type": ["string", "number"],
"description": "This can be either a string or a number."
},
"invalid_prop": {
// NOTE: "type:" here should have been "type" (without the colon)
"type:": ["string", "null"],
"description": "Note the extra colon in the name of the type property above"
}
},
// NOTE: Reference to a non-existent property
"required": ["valid_prop", "nonexistent_prop"]
}
I'd like to write a Python script (or, even better, install a CLI with PiP) that can find those bugs.
I've seen this answer, which suggests doing the following (modified for my use case):
import json
from jsonschema import Draft4Validator
with open('./my-schema.json') as schemaf:
schema = json.loads('\n'.join(schemaf.readlines()))
Draft4Validator.check_schema(my_schema)
print("OK!") # on invalid schema we don't get here
but the above script doesn't detect either of the errors in the schema file. I would have suspected it to detect at least the extra colon in the "type:" property.
Am I using the library incorrectly? How do I write a validation script that detects this error?
You say the schema is invalid, but that isn't the case with the example you've provided.
Unknown keywords are ignored. This is to allow for extensions to be created. If unknown keywords were prevented, we wouldn't have the ecosystem of extensions that various people and groups have created, like form generation.
You say that the value in required is a "Reference to a non-existent property". The required keyword has no link to the properties keyword.
required determins which keys an object must have.
properties determines how a subschema should be applied to values in an object.
There's no need for values in required to also be included in properties. In fact it's common that they do not when building complex modular schemas.
In terms of validating if a schema is valid, you can use the JSON Schema meta schema.
In terms of checking for additional things that you consider non desireable, that's down to you, given the examples you've provided are valid.
Some libraries may provide a sanity check, but such is unlikely to pick up on the examples you've provided, as they aren't errors.
I am looking to parse some JSON into a dictionary but need to preserve order for one particular part of the dictionary.
I know that I can parse the entire JSON file into an ordered dictionary (ex. Can I get JSON to load into an OrderedDict?) but this is not quite what I'm looking for.
{
"foo": "bar",
"columns":
{
"col_1": [],
"col_2": []
}
}
In this example, I would want to parse the entire file in as a dictionary with the "columns" portion being an OrderedDict. Is it possible to get that granular with the JSON parsing tools while guaranteeing that order is preserved throughout? Thank you!
From the comments meanwhile, I gathered that a complete, nested OrderedDict is fine as well, but this could be a solution too, if you don't mind using some knowledge about the names of the columns:
import json
from collections import OrderedDict
def hook(partialjson):
if "col_1" in partialjson:
return OrderedDict(partialjson)
return dict(partialjson)
result = json.loads("YOUR JSON STRING", object_hook=hook)
Hope this helps!
I'm downloading tweets from Twitter Streaming API using Tweepy. I manage to check if downloaded data has keys as 'extended_tweet', but I'm struggling with an specific key inside another key.
def on_data(self, data):
savingTweet = {}
if not "retweeted_status" in data:
dataJson = json.loads(data)
if 'extended_tweet' in dataJson:
savingTweet['text'] = dataJson['extended_tweet']['full_text']
else:
savingTweet['text'] = dataJson['text']
if 'coordinates' in dataJson:
if 'coordinates' in dataJson['coordinates']:
savingTweet['coordinates'] = dataJson['coordinates']['coordinates']
else:
savingTweet['coordinates'] = 'null'
I'm checking 'extended_key' propertly, but when I try to do the same with ['coordinates]['coordinates] I get the following error:
TypeError: argument of type 'NoneType' is not iterable
Twitter documentation says that key 'coordinates' has the following structure:
"coordinates":
{
"coordinates":
[
-75.14310264,
40.05701649
],
"type":"Point"
}
I achieved to solve it by just putting the conflictive check in a try, except, but I think this is not the most suitable approach to the problem. Any other idea?
So the twitter API docs are probably lying a bit about what they return (shock horror!) and it looks like you're getting a None in place of the expected data structure. You've already decided against using try, catch, so I won't go over that, but here are a few other suggestions.
Using dict get() default
There are a couple of options that occur to me, the first is to make use of the default ability of the dict get command. You can provide a fall back if the expected key does not exist, which allows you to chain together multiple calls.
For example you can achieve most of what you are trying to do with the following:
return {
'text': data.get('extended_tweet', {}).get('full_text', data['text']),
'coordinates': data.get('coordinates', {}).get('coordinates', 'null')
}
It's not super pretty, but it does work. It's likely to be a little slower that what you are doing too.
Using JSONPath
Another option, which is likely overkill for this situation is to use a JSONPath library which will allow you to search within data structures for items matching a query. Something like:
from jsonpath_rw import parse
matches = parse('extended_tweet.full_text').find(data)
if matches:
print(matches[0].value)
This is going to be a lot slower that what you are doing, and for just a few fields is overkill, but if you are doing a lot of this kind of work it could be a handy tool in the box. JSONPath can also express much more complicated paths, or very deeply nested paths where the get method might not work, or would be unweildy.
Parse the JSON first!
The last thing I would mention is to make sure you parse your JSON before you do your test for "retweeted_status". If the text appears anywhere (say inside the text of a tweet) this test will trigger.
JSON parsing with a competent library is usually extremely fast too, so unless you are having real speed problems it's not necessarily worth worrying about.
I need to pass data in a python back-end to a front end through an api call, using a json format. In the python back end, the data is in a dictionary structure, which I can easily and directly convert to a json. But should I?
My front-end developer believes the answer is no, for reasons related to best practice.
But I challenge that:
Is the best to structure a json as it is in python, or should it rather be converted to some other form, such as several arrays (as would be necessary in my example case below)?
Or, differently put:
What should be the governing principles related to collections/dicts/maps/arrays for interfacing information through jsons?
I've done some googling for an answer, but I've not come across much that addresses this directly. Links would be appreciated.
(Note about the example below: of course if the data is written to a database, it would probably make most sense for the front-end to access the database directly, but let's assume this is not the case)
Example:
In the back end there is a collection of objects called pets:
each item in the collection has a unique pet_id, some non-optional properties, e.g. name and date_of_birth, some optional properties registration_certificate_nr, adopted_from_kennel, some lists like siblings and children and some objects like medication.
Assuming that the front end needs all of this info at some point, it could be
{
"pets": {
"17-01-24-01": {
"name": "Buster",
"date_of_birth": "04/01/2017",
"registration_certificate_nr": "AAD-1123-1432"
},
"17-03-04-01": {
"name": "Hooch",
"date_of_birth": "05/02/2015",
"adopted_from_kennel": "Pretoria Shire",
"children": [
"17-05-01-01",
"17-05-01-02",
"17-05-01-03"
]
},
"17-05-01-01": {
"name": "Snappy",
"date_of_birth": "17-05-01",
"siblings": [
"17-05-01-02",
"17-05-01-03"
]
},
"17-05-01-02": {
"name": "Gizmo",
"date_of_birth": "17-05-01",
"siblings": [
"17-05-01-01",
"17-05-01-03"
]
},
"17-05-01-03": {
"name": "Toothless",
"date_of_birth": "17-05-01",
"siblings": [
"17-05-01-01",
"17-05-01-03"
],
"medication": [
{
"name": "anti-worm",
"code": "aw445",
"dosage": "1 pill per day"
},
{
"name": "disinfectant",
"code": "pdi-2",
"dosage": "as required"
}
]
}
}
}
JSON formatting is a somewhat subjective matter, and related disagreements are usually best settled between colleagues.
That being said, there are some potentially valid criticisms to be made against the JSON format in the question, especially if we are trying to create a consistent, RESTful API.
The 2 pain points that stand out:
A map collection is represented in JSON, which isn't really JSON standard compliant, or particularly RESTful.
None of the pet objects have an id defined. There is a pet_id mentioned in the question, but it seems to be maintained separately from the pet object itself. If a value is accessed in the pets map in the question, a user of the API would have to manually add the pet_id to the provided pet object in order to have the id available further down the line, when the full JSON may no longer be available.
The closest things we have to guiding standards in this situation is the REST architectural style and the JSON standard.
We can start by looking at the JSON standard. Here is a quote from the JSON wiki:
JavaScript syntax defines several native data types that are not included in the JSON standard: Map, Set, Date, Error, Regular Expression, Function, Promise, and undefined.
The key takeaway here is that JSON is not meant to represent the map data type. Python dictionaries are a map implementation, so directly serializing a dictionary to JSON with the intent to represent a map-like collection goes against the intended use of JSON.
For an individual object like a pet, the JSON object is appropriate, but for collections there is one option: the JSON array. There is a usage example with the JSON array further down in this answer.
There may be edge cases where deviating from the standard makes sense, but I don't see a reason in this scenario.
There are also some shortcomings in the JSON format from a RESTful design perspective. RESTful API design is nice because it encourages one to keep things simple and consistent. It also happens to be a de facto industry standard.
In a RESTful HTTP API, this is how fetching a single pet resource should look:
Request: GET /api/pets/17-01-24-01
Response: 200 {
"id": "17-01-24-01",
"name": "Buster",
"date_of_birth": "04/01/2017",
"registration_certificate_nr": "AAD-1123-1432"
}
The response is a completely defined resource with an explicitly defined id. It is also the simplest complete JSON representation of a pet.
Next, we define what fetching multiple pet resources looks like, assuming only 2 pets are defined:
Request: GET /api/pets
Response: 200 [
{
"id": "17-01-24-01",
"name": "Buster",
"date_of_birth": "04/01/2017",
"registration_certificate_nr": "AAD-1123-1432"
},
{
"id": "17-03-04-01",
"name": "Hooch",
"date_of_birth": "05/02/2015",
"adopted_from_kennel": "Pretoria Shire",
"children": [
"17-05-01-01",
"17-05-01-02",
"17-05-01-03"
]
}
]
The above response format is the most straight forward way to pluralize the single resource response format, thus keeping the API as simple and consistent as possible. (for the sake of brevity, I only used 2 of the sample resources from the question). Once again, the ids are explicitly defined, and belong to their respective pet objects.
Nothing is gained from adding map keys to the above format.
Proponents of the JSON format in the question may suggest to just add the id field into each pet object in order to work around pain point 2, but that would raise the question of repeating data within the response. Why does the id need to be both inside and outside the object? Surely it should only be on the inside? After eliminating the redundant data, the result will look like the response above.
That is the REST argument. There are use cases where REST doesn't really work, but this is far from that.
PS. Front ends should never access databases directly. The API is responsible for writing to and reading from whatever data persistence infrastructure is used. In a lot of bigger real world systems, there is even an additional BFF layer between the front end and the API(s), separating the front end and the DB even further.
I'm coding an API library in Python, I always chose json before as the response format but this API provides text and json formats, I'm not asking which one is easier or better, I want to know the advantages and the disadvantages of using both as I only worked with json before.
I thought about using text format, it's very easy to parse but the only thing I thought about was the nested elements, but after checking the example they're separated with underscores _ for example:
name=VALUE
lastname=VALUE
age=VALUE
contact_email=VALUE
contact_phone=VALUE
contact_mobile=VALUE
same json response:
{
"name": "VALUE",
"lastname": "VALUE",
"age": "VALUE",
"contact": {
"email": "VALUE",
"phone": "VALUE",
"mobile": "VALUE"
}
}
So is there any advantages or disadvantages of using text over json or the other way around ?
IMHO the main advantages of json over text would be:
DRY - there are third party libs for json processing which are quite performant
The parsed json ends up being a dict, which you cam refer by keys easily
For the text variant, there is the .ini format, which could offer something aling the lines of structure, although the format you described above is not really designed for nested fields and structures.
I'd also look at who's consuming your API. E.g. if it's a web app, then json is the accepted format. if your consumer is something else which is more comfortable with the text format...
JSON can be easier as you can use the json library.
The return you've shown there can be put into a dictionary with the line:
import json
json.loads(return_string)
Much easier to parse than the text!