We Keep Coding

i want to convert sample JSON data into nested JSON using specific key-value in python

I have below sample data in JSON format :
project_cost_details is my database result set after querying.
{
"1": {
"amount": 0,
"breakdown": [
{
"amount": 169857,
"id": 4,
"name": "SampleData",
"parent_id": "1"
}
],
"id": 1,
"name": "ABC PR"
}
}
Here is full json : https://jsoneditoronline.org/?id=2ce7ab19af6f420397b07b939674f49c
Expected output :https://jsoneditoronline.org/?id=56a47e6f8e424fe8ac58c5e0732168d7
I have this sample JSON which i created using loops in code. But i am stuck at how to convert this to expected JSON format. I am getting sequential changes, need to convert to tree like or nested JSON format.
Trying in Python :
project_cost = {}
for cost in project_cost_details:
if cost.get('Parent_Cost_Type_ID'):
project_id = str(cost.get('Project_ID'))
parent_cost_type_id = str(cost.get('Parent_Cost_Type_ID'))
if project_id not in project_cost:
project_cost[project_id] = {}
if "breakdown" not in project_cost[project_id]:
project_cost[project_id]["breakdown"] = []
if 'amount' not in project_cost[project_id]:
project_cost[project_id]['amount'] = 0
project_cost[project_id]['name'] = cost.get('Title')
project_cost[project_id]['id'] = cost.get('Project_ID')
if parent_cost_type_id == cost.get('Cost_Type_ID'):
project_cost[project_id]['amount'] += int(cost.get('Amount'))
#if parent_cost_type_id is None:
project_cost[project_id]["breakdown"].append(
{
'amount': int(cost.get('Amount')),
'name': cost.get('Name'),
'parent_id': parent_cost_type_id,
'id' : cost.get('Cost_Type_ID')
}
)
from this i am getting sample JSON. It will be good if get in this code only desired format.
Also tried this solution mention here : https://adiyatmubarak.wordpress.com/2015/10/05/group-list-of-dictionary-data-by-particular-key-in-python/

I got approach to convert sample JSON to expected JSON :
data = [
{ "name" : "ABC", "parent":"DEF", },
{ "name" : "DEF", "parent":"null" },
{ "name" : "new_name", "parent":"ABC" },
{ "name" : "new_name2", "parent":"ABC" },
{ "name" : "Foo", "parent":"DEF"},
{ "name" : "Bar", "parent":"null"},
{ "name" : "Chandani", "parent":"new_name", "relation": "rel", "depth": 3 },
{ "name" : "Chandani333", "parent":"new_name", "relation": "rel", "depth": 3 }
]
result = {x.get("name"):x for x in data}
#print(result)
tree = [];
for a in data:
#print(a)
if a.get("parent") in result:
parent = result[a.get("parent")]
else:
parent = ""
if parent:
if "children" not in parent:
parent["children"] = []
parent["children"].append(a)
else:
tree.append(a)
Reference help : http://jsfiddle.net/9FqKS/ this is a JavaScript solution i converted to Python

It seems that you want to get a list of values from a dictionary.
result = [value for key, value in project_cost_details.items()]

Get item location in python dictionary by value

I have a dictionary python with keys and values nested.
How do I find the object's index number by providing a value.
As for now, I can get values of the keys in specific object when I know the index of the object.
I mean, if I know the object's index number in the dictionary, I can get the key's and values in this specific object.
my_dict = {
"ftml": {
"people": {
"person": [
{
"#id": "Terach",
"#sex": "male",
"Death": {
"#year": ""
},
"Midrash": {
"#midrah": ""
},
"Old": {
"#age": ""
},
"Role": {
"#role": ""
},
"birth": {
"#year": ""
},
"father": {
"#id": "Nachor"
},
"mother": {
"#id": ""
},
"spouse": ""
},
{
"#id": "Avraham",
"#sex": "male",
"Death": {
"#year": "2123"
},
"Grandson": {
"#son1": "Esav",
"#son2": "Yaakov"
},
"Midrash": {
"#midrah": ""
},
"Old": {
"#age": "175"
},
"Role": {
"#role": ""
},
"birth": {
"#year": "1948"
},
"father": {
"#id": "Terach"
},
"mother": {
"#id": ""
},
"spouse": {
"#wife1": "Sara"
}
},
{
"#husband": "Avraham",
"#id": "Sara",
"#sex": "female"
},
{
"#id": "Nachor",
"#sex": "male",
"Death": {
"#year": ""
},
"Midrash": {
"#midrah": ""
},
"Old": {
"#age": ""
},
"Role": {
"#role": ""
},
"birth": {
"#year": ""
},
"father": {
"#id": "Terach"
},
"mother": {
"#id": ""
},
"spouse": ""
},
]
}
}
}
x = int(input("Type the chronological person number. (i.e 1 is Avraham): "))
print("First Name: ",my_dict['ftml']['people']['person'][x]["#id"]) #1 = avraham
I expect to ask the user for the #id and return the object's index number.
For example, if the user sends the program "Avraham" the program will return 1.
If the user is looking for Nachor the program will return 0.

I don't think revising the dict is a good idea.
Here is my solution:
First get the "position" of your list, i.e. what to find from.
list_to_find = my_dict['ftml']['people']['person']
The list_to_find is a list of dict (people), like
[{"#id": "Terach", '#sex': ...}, {"#id": 'Avraham', ...} ...]
Then what you want to do is to search in all the #id, so you can get all the #id by:
ids = [person['#id'] for person in list_to_find]
And then use index to get the index:
index = ids.index('Avraham')

In here I used dict comprehensions with enumerate() python Built-in Function. It's little bit confusing you. But you know the data structure about Dictionaries. For this example I didn't attach your my_dict dictionary cause it's too large.
>>> obj = {y["#id"]:x for x,y in list(enumerate(my_dict["ftml"]["people"]["person"]))}
>>> obj
{'Terach': 0, 'Avraham': 1, 'Sara': 2, 'Nachor': 3}
This output of the obj looks like the summary of the list of my_dict["ftml"]["people"]["person"]. Isn't it? For your question this obj is simply enough without extracting such a long dictionary and this is fast. If you confusing with dict comprehensions, hopefully this will understand for you.
>>> obj = {}
>>> for x,y in list(enumerate(my_dict["ftml"]["people"]["person"])):
... obj[y["#id"]] = x
...
>>> obj
{'Terach': 0, 'Avraham': 1, 'Sara': 2, 'Nachor': 3}
If you didn't understand what enumerate() does in here, check this small example which I directly get it from original documentation.
Return an enumerate object. iterable must be a sequence, an iterator, or some other object which supports iteration. The __next__() method of the iterator returned by enumerate() returns a tuple containing a count (from start which defaults to 0) and the values obtained from iterating over iterable.
>>> seasons = ['Spring', 'Summer', 'Fall', 'Winter']
>>> list(enumerate(seasons))
[(0, 'Spring'), (1, 'Summer'), (2, 'Fall'), (3, 'Winter')]
>>> list(enumerate(seasons, start=1))
[(1, 'Spring'), (2, 'Summer'), (3, 'Fall'), (4, 'Winter')]
According to above the example we are numbering seasons final output. I saw you comment in #Joery's answer. Now you want to insert a name and get an index of if it.
>>> x = input("Type the chronological person number. (i.e 1 is Avraham): ")
Type the chronological person id. (i.e 1 is Avraham): Avraham
>>> print(obj.get(x, None)) # if not match anything, will return None
1
So this 1 mean the 2nd element of my_dict["ftml"]["people"]["person"] list. Now you can easily access any of it. This is what get() function does.
Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.
In you obj dictionary there are 4 keys available, Terach, Avraham, Sara and Nachor. When you enter one of these as your input, you'll get 0, 1, 2 or 3 as an output. According to the default value of get() function, it is None. When you enter value as an input which is not in obj dictionary mean you'll get the default value which is None.

just reverse the dictionary like so:
reversed_dict = {}
for i in range(0,len(my_dict['ftml']['people']['person'])):
reversed_dict[my_dict['ftml']['people']['person'][i]['#id']] = {'index':i}
print(reversed_dict)
print(reversed_dict['Avraham']['index'])
this should give you the outout of
{'Terach': {'index': 0}, 'Avraham': {'index': 1}, 'Sara': {'index': 2}, 'Nachor': {'index': 3}}
1

The most simple thing would probably be:
my_dict = {} # Defined in the question
x = int(input("Type the chronological person number. (i.e 1 is Avraham): "))
persons = my_dict['ftml']['people']['person']
for i, v in enumerate(persons):
if v['#id'] == x:
break
# i now has the most recent index
print(i)
Your intention: "For example, if the user sends the program "Avraham" the program will return 1. If the user is looking for Nachor the program will return 0." is implemented by this. However, above would work in in reverse... as the iteration will go from top to bottom in this representation...
reversed(persons)... :)

Dictionary length is equal to 3 but when trying to access an index receiving KeyError

I am attempting to parse a json response that looks like this:
{
"links": {
"next": "http://www.neowsapp.com/rest/v1/feed?start_date=2015-09-08&end_date=2015-09-09&detailed=false&api_key=xxx",
"prev": "http://www.neowsapp.com/rest/v1/feed?start_date=2015-09-06&end_date=2015-09-07&detailed=false&api_key=xxx",
"self": "http://www.neowsapp.com/rest/v1/feed?start_date=2015-09-07&end_date=2015-09-08&detailed=false&api_key=xxx"
},
"element_count": 22,
"near_earth_objects": {
"2015-09-08": [
{
"links": {
"self": "http://www.neowsapp.com/rest/v1/neo/3726710?api_key=xxx"
},
"id": "3726710",
"neo_reference_id": "3726710",
"name": "(2015 RC)",
"nasa_jpl_url": "http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3726710",
"absolute_magnitude_h": 24.3,
"estimated_diameter": {
"kilometers": {
"estimated_diameter_min": 0.0366906138,
"estimated_diameter_max": 0.0820427065
},
"meters": {
"estimated_diameter_min": 36.6906137531,
"estimated_diameter_max": 82.0427064882
},
"miles": {
"estimated_diameter_min": 0.0227984834,
"estimated_diameter_max": 0.0509789586
},
"feet": {
"estimated_diameter_min": 120.3760332259,
"estimated_diameter_max": 269.1689931548
}
},
"is_potentially_hazardous_asteroid": false,
"close_approach_data": [
{
"close_approach_date": "2015-09-08",
"close_approach_date_full": "2015-Sep-08 09:45",
"epoch_date_close_approach": 1441705500000,
"relative_velocity": {
"kilometers_per_second": "19.4850295284",
"kilometers_per_hour": "70146.106302123",
"miles_per_hour": "43586.0625520053"
},
"miss_distance": {
"astronomical": "0.0269230459",
"lunar": "10.4730648551",
"kilometers": "4027630.320552233",
"miles": "2502653.4316094954"
},
"orbiting_body": "Earth"
}
],
"is_sentry_object": false
},
}
I am trying to figure out how to parse through to get "miss_distance" dictionary values ? I am unable to wrap my head around it.
Here is what I have been able to do so far:
After I get a Response object from request.get()
response = request.get(url
I convert the response object to json object
data = response.json() #this returns dictionary object
I try to parse the first level of the dictionary:
for i in data:
if i == "near_earth_objects":
dataset1 = data["near_earth_objects"]["2015-09-08"]
#this returns the next object which is of type list
Please someone can explain me :
1. How to decipher this response in the first place.
2. How can I move forward in parsing the response object and get to miss_distance dictionary ?
Please any pointers/help is appreciated.
Thank you

Your data will will have multiple dictionaries for the each date, near earth object, and close approach:
near_earth_objects = data['near_earth_objects']
for date in near_earth_objects:
objects = near_earth_objects[date]
for object in objects:
close_approach_data = object['close_approach_data']
for close_approach in close_approach_data:
print(close_approach['miss_distance'])

The code below gives you a table of date, miss_distances for every object for every date
import json
raw_json = '''
{
"near_earth_objects": {
"2015-09-08": [
{
"close_approach_data": [
{
"miss_distance": {
"astronomical": "0.0269230459",
"lunar": "10.4730648551",
"kilometers": "4027630.320552233",
"miles": "2502653.4316094954"
},
"orbiting_body": "Earth"
}
]
}
]
}
}
'''
if __name__ == "__main__":
parsed = json.loads(raw_json)
# assuming this json includes more than one near_earch_object spread across dates
near_objects = []
for date, near_objs in parsed['near_earth_objects'].items():
for obj in near_objs:
for appr in obj['close_approach_data']:
o = {
'date': date,
'miss_distances': appr['miss_distance']
}
near_objects.append(o)
print(near_objects)
output:
[
{'date': '2015-09-08',
'miss_distances': {
'astronomical': '0.0269230459',
'lunar': '10.4730648551',
'kilometers': '4027630.320552233',
'miles': '2502653.4316094954'
}
}
]

Generating a dynamic nested JSON object and array - python

As the question explains the problem, I've been trying to generate nested JSON object. In this case I have for loops getting the data out of dictionary dic. Below is the code:
f = open("test_json.txt", 'w')
flag = False
temp = ""
start = "{\n\t\"filename\"" + " : \"" +initial_filename+"\",\n\t\"data\"" +" : " +" [\n"
end = "\n\t]" +"\n}"
f.write(start)
for i, (key,value) in enumerate(dic.iteritems()):
f.write("{\n\t\"keyword\":"+"\""+str(key)+"\""+",\n")
f.write("\"term_freq\":"+str(len(value))+",\n")
f.write("\"lists\":[\n\t")
for item in value:
f.write("{\n")
f.write("\t\t\"occurance\" :"+str(item)+"\n")
#Check last object
if value.index(item)+1 == len(value):
f.write("}\n"
f.write("]\n")
else:
f.write("},") # close occurrence object
# Check last item in dic
if i == len(dic)-1:
flag = True
if(flag):
f.write("}")
else:
f.write("},") #close lists object
flag = False
#check for flag
f.write("]") #close lists array
f.write("}")
Expected output is:
{
"filename": "abc.pdf",
"data": [{
"keyword": "irritation",
"term_freq": 5,
"lists": [{
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 2
}]
}, {
"keyword": "bomber",
"lists": [{
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 2
}],
"term_freq": 5
}]
}
But currently I'm getting an output like below:
{
"filename": "abc.pdf",
"data": [{
"keyword": "irritation",
"term_freq": 5,
"lists": [{
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 2
},] // Here lies the problem "," before array(last element)
}, {
"keyword": "bomber",
"lists": [{
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 1
}, {
"occurance": 2
},], // Here lies the problem "," before array(last element)
"term_freq": 5
}]
}
Please help, I've trying to solve it, but failed. Please don't mark it duplicate since I have already checked other answers and didn't help at all.
Edit 1:
Input is basically taken from a dictionary dic whose mapping type is <String, List>
for example: "irritation" => [1,3,5,7,8]
where irritation is the key, and mapped to a list of page numbers.
This is basically read in the outer for loop where key is the keyword and value is a list of pages of occurrence of that keyword.
Edit 2:
dic = collections.defaultdict(list) # declaring the variable dictionary
dic[key].append(value) # inserting the values - useless to tell here
for key in dic:
# Here dic[x] represents list - each value of x
print key,":",dic[x],"\n" #prints the data in dictionary

What #andrea-f looks good to me, here another solution:
Feel free to pick in both :)
import json
dic = {
"bomber": [1, 2, 3, 4, 5],
"irritation": [1, 3, 5, 7, 8]
}
filename = "abc.pdf"
json_dict = {}
data = []
for k, v in dic.iteritems():
tmp_dict = {}
tmp_dict["keyword"] = k
tmp_dict["term_freq"] = len(v)
tmp_dict["lists"] = [{"occurrance": i} for i in v]
data.append(tmp_dict)
json_dict["filename"] = filename
json_dict["data"] = data
with open("abc.json", "w") as outfile:
json.dump(json_dict, outfile, indent=4, sort_keys=True)
It's the same idea, I first create a big json_dict to be saved directly in json. I use the with statement to save the json avoiding the catch of exception
Also, you should have a look to the doc of json.dumps() if you need future improve in your json output.
EDIT
And just for fun, if you don't like tmp var, you can do all the data for loop in a one-liner :)
json_dict["data"] = [{"keyword": k, "term_freq": len(v), "lists": [{"occurrance": i} for i in v]} for k, v in dic.iteritems()]
It could gave for final solution something not totally readable like this:
import json
json_dict = {
"filename": "abc.pdf",
"data": [{
"keyword": k,
"term_freq": len(v),
"lists": [{"occurrance": i} for i in v]
} for k, v in dic.iteritems()]
}
with open("abc.json", "w") as outfile:
json.dump(json_dict, outfile, indent=4, sort_keys=True)
EDIT 2
It looks like you don't want to save your json as the desired output, but be abble to read it.
In fact, you can also use json.dumps() in order to print your json.
with open('abc.json', 'r') as handle:
new_json_dict = json.load(handle)
print json.dumps(json_dict, indent=4, sort_keys=True)
There is still one problem here though, "filename": is printed at the end of the list because the d of data comes before the f.
To force the order, you will have to use an OrderedDict in the generation of the dict. Be careful the syntax is ugly (imo) with python 2.X
Here is the new complete solution ;)
import json
from collections import OrderedDict
dic = {
'bomber': [1, 2, 3, 4, 5],
'irritation': [1, 3, 5, 7, 8]
}
json_dict = OrderedDict([
('filename', 'abc.pdf'),
('data', [ OrderedDict([
('keyword', k),
('term_freq', len(v)),
('lists', [{'occurrance': i} for i in v])
]) for k, v in dic.iteritems()])
])
with open('abc.json', 'w') as outfile:
json.dump(json_dict, outfile)
# Now to read the orderer json file
with open('abc.json', 'r') as handle:
new_json_dict = json.load(handle, object_pairs_hook=OrderedDict)
print json.dumps(json_dict, indent=4)
Will output:
{
"filename": "abc.pdf",
"data": [
{
"keyword": "bomber",
"term_freq": 5,
"lists": [
{
"occurrance": 1
},
{
"occurrance": 2
},
{
"occurrance": 3
},
{
"occurrance": 4
},
{
"occurrance": 5
}
]
},
{
"keyword": "irritation",
"term_freq": 5,
"lists": [
{
"occurrance": 1
},
{
"occurrance": 3
},
{
"occurrance": 5
},
{
"occurrance": 7
},
{
"occurrance": 8
}
]
}
]
}
But be carefull, most of the time, it is better to save a regular .json file in order to be cross languages.

Your current code is not working because the loop iterates through the before-last item adding the }, then when the loop runs again it sets the flag to false, but the last time it ran it added a , since it thought that there will be another element.
If this is your dict: a = {"bomber":[1,2,3,4,5]} then you can do:
import json
file_name = "a_file.json"
file_name_input = "abc.pdf"
new_output = {}
new_output["filename"] = file_name_input
new_data = []
i = 0
for key, val in a.iteritems():
new_data.append({"keyword":key, "lists":[], "term_freq":len(val)})
for p in val:
new_data[i]["lists"].append({"occurrance":p})
i += 1
new_output['data'] = new_data
Then save the data by:
f = open(file_name, 'w+')
f.write(json.dumps(new_output, indent=4, sort_keys=True, default=unicode))
f.close()

build dynamical JSON/LIST obj in Python

I'm a real Python newbee and I'm having problems creating a JSON/LIST obj.
What I want to end up with is the following JSON to send o an API
{
"request": {
"slice": [
{
"origin": "AMS",
"destination": "SYD",
"date": "2015-06-23"
}
],
"passengers": {
"adultCount": 1,
"infantInLapCount": 0,
"infantInSeatCount": 0,
"childCount": 0,
"seniorCount": 0
},
"solutions": 20,
"refundable": false
}
}
I figured to make a list and then to convert to JSON with the dumps() function. This works. The thing is, I need to change the date field with an iterator to add a day, but I'm stuck on changing this field.
Any advice?
thx!

As your question is a bit vague i can only guess that you're trying to modify the JSON version of your data directly, while you should modify the Python object before converting it into JSON... something like this:
d = {
"request": {
"slice": [
{
"origin": "AMS",
"destination": "SYD",
"date": "2015-06-23"
}
],
"passengers": {
"adultCount": 1,
"infantInLapCount": 0,
"infantInSeatCount": 0,
"childCount": 0,
"seniorCount": 0
},
"solutions": 20,
"refundable": False # note how this is python False, not js false!
}
}
# then you can do:
d["request"]["slice"][0]["date"] = "2015-05-23"
# and finally convert to json:
j = json.dumps(d)
If it happens that you get JSON as a string, you should first convert it into a python object so you can work on it:
# if j is your json string, convert it into a python object
d = json.loads(j)
# then do your modifications as above:
d["request"]["slice"][0]["date"] = "2015-05-23"
# and finally convert back to json:
j = json.dumps(d)