We Keep Coding

Get the Parent key and the nested value in nested json

I have a nested json for a JSON schema like this:
{
"config": {
"x-permission": true
},
"deposit_schema": {
"additionalProperties": false,
"$schema": "http://json-schema.org/draft-04/schema#",
"type": "object",
"properties": {
"control_number": {
"type": "string",
"x-cap-permission": {
"users": [
"test#test.com"
]
}
},
"initial": {
"properties": {
"status": {
"x-permission": {
"users": [
"test3#test.com"
]
},
"title": "Status",
"type": "object",
"properties": {
"main_status": {
"type": "string",
"title": "Stage"
}
}
},
"gitlab_repo": {
"description": "Add your repository",
"items": {
"properties": {
"directory": {
"title": "Subdirectory",
"type": "string",
"x-permission": {
"users": [
"test1#test.com",
"test2#test.com"
]
}
},
"gitlab": {
"title": "Gitlab",
"type": "string"
}
},
"type": "object"
},
"title": "Gitlab Repository",
"type": "array"
},
"title": "Initial Input",
"type": "object"
}
},
"title": "Test Analysis"
}
}
The JSON is nested and I want to have the dict of x-permission fields with their parent_key like this:
{
"control_number": {"users": ["test#test.com"]},
"initial.properties.status": {"users": ["test3#test.com"]},
"initial.properties.gitlab_repo.items.properties.directory": {"users": [
"test1#test.com",
"test2#test.com"
]}
}
I am trying to do implement recursive logic for every key in JSON like this:
def extract(obj, parent_key):
"""Recursively search for values of key in JSON tree."""
for k, v in obj.items():
key = parent_key + '.' + k
if isinstance(v, dict):
if v.get('x-permission'):
return key, v.get('x-permission')
elif v.get('properties'):
return extract(v.get('properties'), key)
return None, None
def collect_permission_info(object_):
# _schema = _schema.deposit_schema.get('properties')
_schema = object_ # above json
x_cap_fields = {}
for k in _schema:
parent_key, permission_info = extract(_schema.get(k), k)
if parent_key and permission_info:
x_cap_fields.update({parent_key: permission_info})
return x_cap_fields
I am getting empty dict now, what I am missing here?

You could use this generator of key/value tuples:
def collect_permission_info(schema):
for key, child in schema.items():
if isinstance(child, dict):
if "x-permission" in child:
yield key, child["x-permission"]
if "properties" in child:
for rest, value in collect_permission_info(child["properties"]):
yield key + "." + rest, value
Then call it like this:
result = dict(collect_permission_info(schema))

A few issues I can spot:
You use the parent_key directly in the recursive function. In a case when multiple properties exist in an object ("_experiment" has 2 properties), the path will be incorrect (e.g. _experiment.type.x-permission is constructed in second loop call). Use a new variable so that each subsequent for loop call uses the initial parent_key value
The elif branch is never executed as the first branch has priority. It is a duplicate.
The return value from the recursive execute(...) call is ignored. Anything you might find on deeper levels is therefore ignored
Judging by your example json schema and the desired result, a recursive call on the "initial": {...} object should return multiple results. You would have to modify the extract(...) function to allow for multiple results instead of a single one
You only check if an object contains a x-permission or a properties attribute. This ignores the desired result in the provided "initial" schema branch which contains x-permission nested inside a status and main_status branch. The easiest solution is to invoke a recursive call every time isinstance(v, dict) == true

After reading through the comments and the answers. I got this solution working for my use case.
def parse_schema_permission_info(schema):
x_fields = {}
def extract_permission_field(field, parent_field):
for field, value in field.items():
if field == 'x-permission':
x_fields.update({parent_field: value})
if isinstance(value, dict):
key = parent_field + '.' + field
if value.get('x-permission'):
x_fields.update(
{key: value.get('x-permission')}
)
extract_permission_field(value, key)
for field in schema:
extract_permission_field(schema.get(field), field)
return x_fields

creating dictionary inside a dictionary for converting it into Json data

My workflow is, I'll be having some dictionaries in different files, I'll be calling them into a file, let's say demo.py. I have a variable temp_dict in demo.py, in which the dictionary from different files will get appended one by one.
Example ::::
{
"Action": None,
"Parameter": "abc",
"ParameterDescription": "def",
"ExpectedValue": "ghi",
{
"Extensions":"jkl",
"MappedData": "no",
"Parameters": "pqr",
"Labels": "Stu",
}
{
"Recorder": "abc",
"Diagnostics": "efg",
"AdditionalRemarks": ""
}
}
I want this type of structure, I need to append dictionaries inside a dictionary, how can I do that.
I will also provide the python code
# function to add data to JSON
def write_json(new_data, filename='report.JSON'):
# new_data is the dictioanries coming from other files, it will be converted into json and dump it into a file.
with open(filename, 'w') as f:
json_string=json.dumps(new_data)
f.write(json_string)
Thanks in advance

The data you've provided is not a valid python dictionary, nor valid JSON.
Dictionaries and JSON are key: value pairs. The value might be a nested dict/JSON, however in your example the nested dictionaries do not have a key.
However, something like this would work:
{
"Action": None,
"Parameter": "abc",
"ParameterDescription": "def",
"ExpectedValue": "ghi",
"YOU NEED SOME NAME HERE": {
"Extensions":"jkl",
"MappedData": "no",
"Parameters": "pqr",
"Labels": "Stu",
},
…
}
You might have been thinking of json objects/dicts inside arrays. There the dictionaries don't have to be named, but that's because they implicitly have a name - their index (position) in the ordered array
[
{
"name": "Faboor",
"type": "user"
},
{
"name": "prithvi"
"reputation": 19
}
]

How to get and append output data from json response and store the duplicate keys if any in file with RobotFramework and python?

I have a json response with nested values. I have read the required keys and values from json. What issue I am facing is I am not able to append the particular keys and values inside a json file where i am storing. So, Can anyone help me in this by giving any solutions.
JSON RESPONSE having tags mulitple keys and values LIKE:
${API_Output}= {
"data": {
"resources": {
"edges": [
{
"node": {
"tags": [],
}
},
{
"node": {
"tags": [
{
"name": "app",
"value": "e2e"
},
{
"name": "Cost",
"value": "qwerty"
}
}
},
{
"node": {
"tags": [
{
"name": "app",
"value": "e2e"
},
{
"name": "Cost",
"value": "qwerty"
},
{
"name": "test",
"value": "qwerty"
}
}
}
]
}
}
}
Robot Framework code:
${dict1}= Set Variable ${API_Output}
${cnt}= get length ${dict1['data']['resources']['edges']}
${edge}= set variable ${dict1['data']['resources']['edges']}
run keyword if ${cnt}==0 set test message The resources count is Zero(0)
log to console ${cnt}-count
: FOR ${item} IN RANGE 0 ${cnt}
\ ${readName}= Set Variable ${edge[${item}]['node']['configuration']}
\ ${readvalue2}= Set Variable ${edge[${item}]['node']['tags']}
\ ${tag_Count}= get length ${edge[${item}]['node']['tags']}
\ ${tag_variable}= set variable ${edge[${item}]['node']['tags']}
\ forkeyword ${tag_Count} ${tag_variable} ${readName}
${req_json} Json.Dumps ${dict}
Create File results.json ${req_json}
forkeyword
[Arguments] ${tag_Count} ${tag_variable} ${readName}
#{z}= create list
: FOR ${item} IN RANGE 0 ${tag_Count}
\ ${resourceName}= run keyword if ${tag_Count} > 0 set variable ${readName['name']}
\ log to console ${resourceName}-forloop
\ ${readkey}= set variable ${tag_variable[${item}]['name']}
\ ${readvalue}= set variable ${tag_variable[${item}]['value']}
\ set to dictionary ${dict} resourceName ${resourceName}
\ set to dictionary ${dict} ${readkey} ${readvalue}
set suite variable ${dict}
I expected all values from all tags but only the last tags key and value is getting printed. Can anyone please guide me on the code.
Thanks in advance for any help!!

Why do you have duplicate keys in first place?
Duplicate keys in JSON aren't covered by the spec and can lead to undefined behavior. If you read the JSON into a Python dict, the information will be lost, since Python dict keys must be unique. Your best bet is to change the JSON; duplicate keys are a bad idea
Also if you change the json to have no duplicates it gives out the correct result
as python keys must be unique.
Since all the other keys are duplicate your simplified version would be:
"data": {
"resources": {
"edges": [
{
"node": {
"tags": [
{
"name": "app",
"value": "e2e"
},
{
"name": "Cost",
"value": "qwert"
},
{
"name": "test",
"value": "qwerty"
}
]
}
}
]
}
}
So if your robot framework gives out the result as
{
"name": "app",
"value": "e2e"
},
{
"name": "Cost",
"value": "qwert"
},
{
"name": "test",
"value": "qwerty"
}
It is correct.

Extracting data from JSON depending on other parameters

What are the options for extracting value from JSON depending on other parameters (using python)? For example, JSON:
"list": [
{
"name": "value",
"id": "123456789"
},
{
"name": "needed-value",
"id": "987654321"
}
]
When using json_name["list"][0]["id"] it obviously returns 123456789. Is there a way to indicate "name" value "needed-value" so i could get 987654321 in return?

For example:
import json as j
s = '''
{
"list": [
{
"name": "value",
"id": "123456789"
},
{
"name": "needed-value",
"id": "987654321"
}
]
}
'''
js = j.loads(s)
print [x["id"] for x in js["list"] if x["name"] == "needed-value"]

The best way to handle this is to refactor the json as a single dictionary. Since "name" and "id" are redundant you can make the dictionary with the value from "name" as the key and the value from "id" as the value.
import json
j = '''{
"list":[
{
"name": "value",
"id": "123456789"
},{
"name": "needed-value",
"id": "987654321"
}
]
}'''
jlist = json.loads(j)['list']
d = {jd['name']: jd['id'] for jd in jlist}
print(d) ##{'value': '123456789', 'needed-value': '987654321'}
Now you can iterate the items like you normally would from a dictionary.
for k, v in d.items():
print(k, v)
# value 123456789
# needed-value 987654321
And since the names are now hashed, you can check membership more efficiently than continually querying the list.
assert 'needed-value' in d

jsn = {
"list": [
{
"name": "value",
"id": "123456789"
},
{
"name": "needed-value",
"id": "987654321"
}
]
}
def get_id(list, name):
for el in list:
if el['name'] == name:
yield el['id']
print(list(get_id(jsn['list'], 'needed-value')))

Python innately treats JSON as a list of dictionaries. With this in mind, you can call the index of the list you need to be returned since you know it's location in the list (and child dictionary).
In your case, I would use list[1]["id"]
If, however, you don't know where the position of your needed value is within the list, the you can run an old fashioned for loop this way:
for user in list:
if user["name"] == "needed_value":
return user["id"]
This is assuming you only have one unique needed_value in your list.

Python for loop json.loads error string indices must be integers

I'm have been struggling with understanding the reason for the following Json parsing issue, I have tried many combinations to access the 'val' item value but I have hit a brick wall.
I have used the code below successfully on 'similar' Json style data, but I dont have the knowledge to craft this approach to the data below.
All advice gratefully accepted.
result = xmltodict.parse(my_read)
result = result['REPORT']['REPORT_BODY']
result =json.dumps(result, indent=1)
print(result)
{
"PAGE": [
{
"D-ROW": [
{
"#num": "1",
"type": "wew",
"val": ".000"
},
{
"#num": "2",
"type": "wew",
"val": ".000"
}
]
},
{
"D-ROW": [
{
"#num": "26",
"type": "wew",
"val": ".000"
},
{
"#num": "27",
"type": "wew",
"val": ".000"
},
{
"#num": "28",
"type": "wew",
"val": ".000"
}
]
}
]
}
for item in json.loads(json_data):
print(item['PAGE']['D-ROW']['val']
error string indices must be integers

item['PAGE'] contains a list, so you cannot index it with 'D-ROW'. If your json-loaded data is in a variable data you could use:
for page in data['PAGE']:
for drow in page['D-ROW']:
print drow['val']

The first thing should notice, based on your JSON structure is that it's a dict {"PAGE": [...], ...}, so when you use json.loads() on it, you'll get a dict too
In this for loop, your item iterator actually refers to the key from the dict
for item in json.loads(json_data):
print(item['PAGE']['D-ROW']['val']
Here's a simpler example easier to follow
>>> for key in json.loads('{"a": "a-value", "b": "b-value"}'):
... print(key)
...
a
b
error string indices must be integers
So you can guess that in your loop item would refer to the key "PAGE", and you can't index that string with ['D-ROW'] ("PAGE"['D-ROW'] doesn't make sense, hence your error)
Key/values in the for loop
To get items if you use the loop below, item becomes a tuple of (key, value)
for item in json.loads(json_data).items():
print(item)
You can also expand the key, value like this
>>> for key, value in json.loads('{"a": "a-value", "b": "b-value"}').items():
... print("key is {} value is {}".format(key, value))
...
key is a value is a-value
key is b value is b-value

Your JSON should not include quotes around values with numbers. For example, change
"D-ROW": [
{
"#num": "1",
"type": "wew",
"val": ".000"
},
to
"D-ROW": [
{
"#num": 1, // Key requires quotes, Value omits quotes if number
"type": "wew",
"val": 0.000
},

"D-ROW": [
{
"#num": "26",
"type": "wew",
"val": ".000"
},
{
"#num": "27",
"type": "wew",
"val": ".000"
},
{
"#num": "28",
"type": "wew",
"val": ".000"
}
D-ROW key contains a list, not a dict.
You should change
print(item['PAGE']['D-ROW']['val']
to
print([_item['val'] for _item in item['PAGE']['D-ROW']])
to iterate over the list which contains you dicts.