We Keep Coding

Finding if a path exists within a json in python

So I am trying to work with an api currently which returns json data such as:
{
"emp1" : {
"EmpName" : "Chris Jackman",
"EmpAge" : "34",
"EmpGender" : "Male",
"EmpDept" : "IT",
"EmpDesg" : "Project Manager"
"EmpSal": {
"2019": 20000
"2020": 23000
},
"emp2" : {
"EmpName" : "Mary Jane",
"EmpAge" : "27",
"EmpGender" : "Female",
"EmpDept" : "IT"
}
}
Occasionally when we query the API one of these fields will be missing. When we try to access it
my_var = my_json["emp2"]["EmpDesg"]
Which will return a key error:
data_dict["EmpDesg"] = my_json["emp2"]["EmpDesg"]
KeyError: 'EmpDesg'
Is there a way to neatly deal with json paths missing rather than wrapping each field in a try catch:
try:
data_dict["EmpName"] = my_json["emp1"]["EmpName"]
except KeyError as e:
data_dict["EmpName"] = ""
try:
data_dict["EmpAge"] = my_json["emp1"]["EmpAge"]
except KeyError as e:
data_dict["EmpAge"] = ""
try:
data_dict["EmpGender"] = my_json["emp1"]["EmpGender"]
except KeyError as e:
data_dict["EmpGender"] = ""
try:
data_dict["salary"] = my_json["emp1"]["EmpSal"]["2020"]
except KeyError as e:
data_dict["salary"] = ""

If you want to check if a key exists, use "key" in data, not a try, except KeyError
Otherwise, if all you want is a default when a key doesn't exist, use .get() with the default parameter
emp1 = my_json["emp1"]
data_dict = {
"EmpName": emp1.get("EmpName", '')
"EmpAge": emp1.get("EmpAge", '')
}
Or
keys = ['EmpName', 'EmpAge']
data_dict = {k: emp1.get(k, '') for k in keys}

To check if a path of arbitrary length exists in a given dict:
def path_exists(d, path) -> bool:
"""Return True if path exists in given dict."""
next = d
while path:
k = path.pop(0)
if k in next:
next = next[k]
else:
return False
return True
For example:
In [4]: path_exists(d, ['emp1', 'EmpName'])
Out[4]: True
In [5]: path_exists(d, ['emp1', 'EmpSal', '2019'])
Out[5]: True
In [6]: path_exists(d, ['emp1', 'EmpSal', '2018'])
Out[6]: False
So you can check whether the path exists (just once) before confidently traversing the dict.

Dictionary in Python not printing updated key/value pair

I have a Python dictionary in my Selenium script in which one of the keys ("data") is updated afterwards because it's not a fixed value, so I get it with Selenium:
fields_personal = {
"data": "",
"nome": "João da Silva",
"documento": "CPF",
"numero": "123.456.789-10",
"telefone": "(01)23456-7890",
"email": "teste#email.com"
}
for data, value in fields_personal.items():
element = driver.find_element_by_id(data)
element.send_keys(value)
if data == "data":
for item in element.find_elements_by_tag_name("option"):
option = item.text.strip()
elem = driver.find_element_by_xpath("//*[#id='data']/option[1]").click()
fields_personal["data"] = option
break
print("{}: {}".format(data, value))
No problem with that, it works fine. But when I try to print the dictionary with print("{}: {}".format(data, value)), or simply print(data, value) it prints "data" key with empty value, as if it hasn't been updated:
data:
nome: João da Silva
documento: CPF
numero: 123.456.789-10
telefone: (01)23456-7890
email: teste#email.com
The only way I can print the updated dictionary is calling pprint.pprint(fields_personal), since I want it structured line by line, but this outputs the result formatted with quotes and curly braces, which I don't want:
{'data': '28/06/2021',
'documento': 'CPF',
'email': 'teste#email.com',
'nome': 'João da Silva',
'numero': '123.456.789-10',
'telefone': '(01)23456-7890'}
So, how can I print it line by line, without quotes and curly braces and updated?

In your if block, you updates fields_personal["data"]; however, when you print it as data, this is the original value read at the start of for loop.
You can move your print inside the block, adding an else to the if to not print two times the same value.
for data, value in fields_personal.items():
element = driver.find_element_by_id(data)
element.send_keys(value)
if data == "data":
for item in element.find_elements_by_tag_name("option"):
option = item.text.strip()
elem = driver.find_element_by_xpath("//*[#id='data']/option[1]").click()
fields_personal["data"] = option
break
print("{}: {}".format("data", fields_personal["data"])
else:
print("{}: {}".format(data, value))
However, if you can just wait after the for block, it would be better.
for data, value in fields_personal.items():
element = driver.find_element_by_id(data)
element.send_keys(value)
if data == "data":
for item in element.find_elements_by_tag_name("option"):
option = item.text.strip()
elem = driver.find_element_by_xpath("//*[#id='data']/option[1]").click()
fields_personal["data"] = option
break
for key, value in fields_personal.items():
print(f"{key}: {value}")

Creating a json output in python

I am trying to return a response for a function in json form. The output is a list with each element being a dictionary. I don't see any mistake when I print the output. The problem arises when I iterate through the output. I get all the characters in the output one by one. See the sample code and sample output for proper understanding.
code:
import requests
import json
import sys
from bs4 import BeautifulSoup
from collections import OrderedDict
class Cricbuzz():
url = "http://synd.cricbuzz.com/j2me/1.0/livematches.xml"
def __init__(self):
pass
def getxml(self,url):
try:
r = requests.get(url)
except requests.exceptions.RequestException as e:
print e
sys.exit(1)
soup = BeautifulSoup(r.text,"html.parser")
return soup
def matchinfo(self,match):
d = OrderedDict()
d['id'] = match['id']
d['srs'] = match['srs']
d['mchdesc'] = match['mchdesc']
d['mnum'] = match['mnum']
d['type'] = match['type']
d['mchstate'] = match.state['mchstate']
d['status'] = match.state['status']
return d
def matches(self):
xml = self.getxml(self.url)
matches = xml.find_all('match')
info = []
for match in matches:
info.append(self.matchinfo(match))
data = json.dumps(info)
return data
c = Cricbuzz()
matches = c.matches()
print matches #print matches - output1
for match in matches:
print match #print match - output2
"print matches" i.e output1 in above code gives me following output:
[
{
"status": "Coming up on Dec 24 at 01:10 GMT",
"mchstate": "nextlive",
"mchdesc": "AKL vs WEL",
"srs": "McDonalds Super Smash, 2016-17",
"mnum": "18TH MATCH",
"type": "ODI",
"id": "0"
},
{
"status": "Ind U19 won by 34 runs",
"mchstate": "Result",
"mchdesc": "INDU19 vs SLU19",
"srs": "Under 19 Asia Cup, 2016",
"mnum": "Final",
"type": "ODI",
"id": "17727"
},
{
"status": "PRS won by 48 runs",
"mchstate": "Result",
"mchdesc": "PRS vs ADS",
"srs": "Big Bash League, 2016-17",
"mnum": "5th Match",
"type": "T20",
"id": "16729"
}
]
But "print match" i.e output2 in above code inside the for loop gives this output:
[
{
"
i
d
"
:
"
0
"
,
"
s
r
s
"
:
"
M
c
D
o
n
a
l
d
s
S
u
p
e
r
S
m
a
s
h
,
2
0
1
6
-
1
7
"
,
"
m
c
h
d
e
s
As you can see,a character gets printed on each line from matches. I would like to get the dictionary object when printing the match.

def matches(self):
xml = self.getxml(self.url)
matches = xml.find_all('match')
info = []
for match in matches:
info.append(self.matchinfo(match))
data = json.dumps(info) # This is a string
return data # This is a string
c = Cricbuzz()
matches = c.matches() # This is a string
print matches
for match in matches: # Looping over all characters of a string
print match
I think you just want return info, which is a list. You can json.dumps() outside of that function at a later point when you actually do need JSON.
Or if you do want that function to return a JSON string, then you have to parse it back into a list.
for match in json.loads(matches):

If you call json.dumps like you do on info before returning data, the value is converted to a json string. If you want to iterate over the iterable the json string represents, you have to load the data back out of the json.
Consider:
import json
info = [ { "a": 1}, { "b": 2} ]
data = json.dumps(info,indent=2)
print data
for i in data:
print i
for i in json.loads(data):
print i
$ python t.py
[
{
"a": 1
},
{
"b": 2
}
]
[
{
"
a
"
:
1
}
,
{
"
b
"
:
2
}
]
{u'a': 1}
{u'b': 2}

matches is a JSON string, not a dictionary, so for match in matches: iterates over the characters in the string.
If you want the dictionary, the function should return info rather than json.dumps(info). Or you could do:
for match in json.loads(matches):
to parse the JSON back into a dictionary.
Normally you should move data around in the program as structured types like dictionaries and lists, and only convert them to/from JSON when you're sending over a network or storing into a file.

Json.dumps returns a string.
If you expect to have each dict from list during iteration process you may wrap your response into:
matches = json.loads(matches)
Btw, it's nice to dumps it's previously as a simple JSON validation, because it makes a valid JSON from invalid: first of all replaces single quotes with double quotes, etc. That's why I suggest don't skip json.dumps as you're trying to do.

How to decode dataTables Editor form in python flask?

I have a flask application which is receiving a request from dataTables Editor. Upon receipt at the server, request.form looks like (e.g.)
ImmutableMultiDict([('data[59282][gender]', u'M'), ('data[59282][hometown]', u''),
('data[59282][disposition]', u''), ('data[59282][id]', u'59282'),
('data[59282][resultname]', u'Joe Doe'), ('data[59282][confirm]', 'true'),
('data[59282][age]', u'27'), ('data[59282][place]', u'3'), ('action', u'remove'),
('data[59282][runnerid]', u''), ('data[59282][time]', u'29:49'),
('data[59282][club]', u'')])
I am thinking to use something similar to this really ugly code to decode it. Is there a better way?
from collections import defaultdict
# request.form comes in multidict [('data[id][field]',value), ...]
# so we need to exec this string to turn into python data structure
data = defaultdict(lambda: {}) # default is empty dict
# need to define text for each field to be received in data[id][field]
age = 'age'
club = 'club'
confirm = 'confirm'
disposition = 'disposition'
gender = 'gender'
hometown = 'hometown'
id = 'id'
place = 'place'
resultname = 'resultname'
runnerid = 'runnerid'
time = 'time'
# fill in data[id][field] = value
for formkey in request.form.keys():
exec '{} = {}'.format(d,repr(request.form[formkey]))

This question has an accepted answer and is a bit old but since the DataTable module seems being pretty popular among jQuery community still, I believe this approach may be useful for someone else. I've just wrote a simple parsing function based on regular expression and dpath module, though it appears not to be quite reliable module. The snippet may be not very straightforward due to an exception-relied fragment, but it was only one way to prevent dpath from trying to resolve strings as integer indices I found.
import re, dpath.util
rxsKey = r'(?P<key>[^\W\[\]]+)'
rxsEntry = r'(?P<primaryKey>[^\W]+)(?P<secondaryKeys>(\[' \
+ rxsKey \
+ r'\])*)\W*'
rxKey = re.compile(rxsKey)
rxEntry = re.compile(rxsEntry)
def form2dict( frmDct ):
res = {}
for k, v in frmDct.iteritems():
m = rxEntry.match( k )
if not m: continue
mdct = m.groupdict()
if not 'secondaryKeys' in mdct.keys():
res[mdct['primaryKey']] = v
else:
fullPath = [mdct['primaryKey']]
for sk in re.finditer( rxKey, mdct['secondaryKeys'] ):
k = sk.groupdict()['key']
try:
dpath.util.get(res, fullPath)
except KeyError:
dpath.util.new(res, fullPath, [] if k.isdigit() else {})
fullPath.append(int(k) if k.isdigit() else k)
dpath.util.new(res, fullPath, v)
return res
The practical usage is based on native flask request.form.to_dict() method:
# ... somewhere in a view code
pars = form2dict(request.form.to_dict())
The output structure includes both, dictionary and lists, as one could expect. E.g.:
# A little test:
rs = jQDT_form2dict( {
'columns[2][search][regex]' : False,
'columns[2][search][value]' : None,
'columns[2][search][regex]' : False,
} )
generates:
{
"columns": [
null,
null,
{
"search": {
"regex": false,
"value": null
}
}
]
}
Update: to handle lists as dictionaries (in more efficient way) one may simplify this snippet with following block at else part of if clause:
# ...
else:
fullPathStr = mdct['primaryKey']
for sk in re.finditer( rxKey, mdct['secondaryKeys'] ):
fullPathStr += '/' + sk.groupdict()['key']
dpath.util.new(res, fullPathStr, v)

I decided on a way that is more secure than using exec:
from collections import defaultdict
def get_request_data(form):
'''
return dict list with data from request.form
:param form: MultiDict from `request.form`
:rtype: {id1: {field1:val1, ...}, ...} [fieldn and valn are strings]
'''
# request.form comes in multidict [('data[id][field]',value), ...]
# fill in id field automatically
data = defaultdict(lambda: {})
# fill in data[id][field] = value
for formkey in form.keys():
if formkey == 'action': continue
datapart,idpart,fieldpart = formkey.split('[')
if datapart != 'data': raise ParameterError, "invalid input in request: {}".format(formkey)
idvalue = int(idpart[0:-1])
fieldname = fieldpart[0:-1]
data[idvalue][fieldname] = form[formkey]
# return decoded result
return data

Decoding JSON with Python and storing nested object

I am trying to decode the following JSON file with Python:
{"node":[
{
"id":"12387",
"ip":"172.20.0.1",
"hid":"213",
"coord":{"dist":"12","lat":"-9.8257","lon":"65.0880"},
"status":{"speed":"90","direction":"N"},
"ts":"12387"}
]
}
By using:
json_data=open('sampleJSON')
jdata = json.load(json_data)
for key, value in jdata.iteritems():
print "Key:"
print key
print "Value:"
print value
and i have as output:
Key:
node
Value:
[{u'status': {u'direction': u'N', u'speed': u'90'}, u'ip': u'172.20.0.1', u'ts': u'12387', u'coord': {u'lat': u'-9.8257', u'lon': u'65.0880', u'dist': u'12'}, u'hid': u'213', u'id': u'12387'}]
And i want to be able to print the key's and values of the nested objects status, coord, and also que key/values of node, "hid", "id", "ip" and "ts".
How can i interate throughout all the nested values?
Thank you in advance!

You can use a recursive function to print it all out. This could be improved, but here is the idea:
import json
json_data = open('data.json')
jdata = json.load(json_data)
def printKeyVals(data, indent=0):
if isinstance(data, list):
print
for item in data:
printKeyVals(item, indent+1)
elif isinstance(data, dict):
print
for k, v in data.iteritems():
print " " * indent, k + ":",
printKeyVals(v, indent + 1)
else:
print data
OUTPUT
node:
status:
direction: N
speed: 90
ip: 172.20.0.1
ts: 12387
coord:
lat: -9.8257
lon: 65.0880
dist: 12
hid: 213
id: 12387
Otherwise, you could just use:
import pprint
pprint.pprint(jdata)

Without knowing more about your specific use case, it's hard to give a general answer, but..
If you have an arbitrarily nested structure, this is a good case for recursion.
simple sample code:
def print_object(object, indent=0):
if type(object)==dict:
for key, value in object.iteritems():
print " "*4*indent, "Key: ", key
print " "*4*indent, "Value: "
print_object(value, indent+1)
elif type(object)==list:
for value in object:
print_object(value,indent+1)
else:
print " "*4*indent, object
you really don't want to do strict type checking, but it works for quick-and-dirty sample code.
output:
Key: node
Value:
Key: status
Value:
Key: direction
Value:
N
Key: speed
Value:
90
Key: ip
Value:
172.20.0.1
Key: ts
Value:
12387
Key: coord
Value:
Key: lat
Value:
-9.8257
Key: lon
Value:
65.0880
Key: dist
Value:
12
Key: hid
Value:
213
Key: id
Value:
12387

It looks like the top level element of your JSON is a dictionary containing a list of dictionaries. If you want to keep this structure, try the code below.
from pprint import pprint
json_data=open('sampleJSON')
jdata = json.load(json_data)
node = jdata['node'][0]
for key, value in node.iteritems():
pprint("Key:")
pprint(key)
pprint("Value:")
pprint(value)