We Keep Coding

Append to a json file using python

Trying to append to a nested json file
My goal is to append some values to a JSON file.
Here is my original JSON file
{
"web": {
"all": {
"side": {
"tags": [
"admin"
],
"summary": "Generates",
"operationId": "Key",
"consumes": [],
"produces": [
"application/json"
],
"responses": {
"200": {
"description": "YES",
"schema": {
"type": "string"
}
}
},
"Honor": [
{
"presidential": []
}
]
}
}
}
}
It is my intention to add two additional lines inside the key "Honor", with the values "Required" : "YES" and "Prepay" : "NO". As a result of appending the two values, I will have the following JSON file.
{
"web": {
"all": {
"side": {
"tags": [
"admin"
],
"summary": "Generates",
"operationId": "Key",
"consumes": [],
"produces": [
"application/json"
],
"responses": {
"200": {
"description": "YES",
"schema": {
"type": "string"
}
}
},
"Honor": [
{
"presidential": [],
"Required" : "YES",
"Prepay" : "NO"
}
]
}
}
}
}
Below is the Python code that I have written
import json
def write_json(data,filename ="exmpleData.json"):
with open(filename,"w") as f:
json.dump(data,f,indent=2)
with open ("exmpleData.json") as json_files:
data= json.load(json_files)
temp = data["Honor"]
y = {"required": "YES","type": "String"}
temp.append(y)
write_json(data)
I am receiving the following error message:
** temp = data["Honor"] KeyError: 'Honor'
**
I would appreciate any guidance that you can provide to help me achieve my goal. I am running Python 3.7

'Honor' is deeply nested in other dictionaries, and its value is a 1-element list containing a dictionary. Here's how to access:
import json
def write_json(data, filename='exmpleData.json'):
with open(filename, 'w') as f:
json.dump(data, f, indent=2)
with open('exmpleData.json') as json_files:
data = json.load(json_files)
# 'Honor' is deeply nested in other dictionaries
honor = data['web']['all']['side']['Honor']
# Its value is a 1-element list containing another dictionary.
honor[0]['Required'] = 'YES'
honor[0]['Prepay'] = 'NO'
write_json(data)

I'd recommend that you practice your fundamentals a bit more since you're making many mistakes in your data structure handling. The good news is, your JSON load/dump is fine.
The cause of your error message is that data doesn't have an "Honor" property. Data only has a "web" property, which contains "all" which contains "side" which contains "Honor", which contains an array with a dictionary that holds the properties you are trying to add to. So you want to set temp with temp = data['web']['all']['side']['Honor'][0]
You also cannot use append on python dictionaries. Instead, check out dict.update().

Populating JSON data from API in Python pandas DataFrame - TypeError and IndexError

I am trying to populate a pandas DataFrame with select information from JSON output fetched from an API.
candidate_list = []
for candidate in candidate_response['data']:
if 'error' not in candidate_response:
candidate_list.append([candidate['id'], candidate['attributes']['first_name'], candidate['attributes']
['last_name'], candidate['relationships']['educations']['data']['id']])
The DataFrame populates fine until I add candidate['relationships']['educations']['data']['id'], which throws TypeError: list indices must be integers or slices, not str.
When trying to get the values of the indexes for ['id'] by using candidate['relationships']['educations']['data'][0]['id'] instead, I get IndexError: list index out of range.
The JSON output looks something like:
"data": [
{
"attributes": {
"first_name": "Tester",
"last_name": "Testman",
"other stuff": "stuff",
},
"id": "732887",
"relationships": {
"educations": {
"data": [
{
"id": "605372",
"type": "educations"
},
{
"id": "605371",
"type": "educations"
},
{
"id": "605370",
"type": "educations"
}
]
}
},
How would I go about successfully filling a column in the DataFrame with the 'id's under 'relationships'>'educations'>'data'?

Please note then when using candidate['relationships']['educations']['data']['id'] you get that error because at data there is a list, and not a dictionary. And you cannot access dictionary by name.
Assuming, what you are trying to achieve is one entry per data.attributes.relationships.educations.data entry. Complete code that works and does what you are trying is:
import json
json_string = """{
"data": [
{
"attributes": {
"first_name": "Tester",
"last_name": "Testman",
"other stuff": "stuff"
},
"id": "732887",
"relationships": {
"educations": {
"data": [
{
"id": "605372",
"type": "educations"
},
{
"id": "605371",
"type": "educations"
},
{
"id": "605370",
"type": "educations"
}
]
}
}
}
]
}"""
candidate_response = json.loads(json_string)
candidate_list = []
for candidate in candidate_response['data']:
if 'error' not in candidate_response:
for data in candidate['relationships']['educations']['data']:
candidate_list.append(
[
candidate['id'],
candidate['attributes']['first_name'],
candidate['attributes']['last_name'],
data['id']
]
)
print(candidate_list)
Code run available at ideone.

I have analyzed your code and also ran it on Jupyter notebook all looks good, I am getting the output,
The error you got list indices must be integers or slices, not str, that is because you were not using the index, this required because the value which you are looking for that is in the list.
and about this error: IndexError: list index out of range. Maybe some code typo mistake is done from your side otherwise the code is fine.
here is the output of your following code:
candidate_list = []
for candidate in candidate_response['data']:
if 'error' not in candidate_response:
candidate_list.append([candidate['id'], candidate['attributes']['first_name'], candidate['attributes']['last_name'],candidate['relationships']['educations']['data'][0]['id']])
Output

probably for any candidate candidate['relationships']['educations']['data'] is an empty list

Create nested json file

I want to create a json file that can be used as a configuration file. I have different files from multiple companies that report the same information with different column names.
I want to use the information provided in the json file to run a python script to consolidate all the information from all files and companies in one master file.
The structure looks like follows:
{"companies":
{"company1": [
{"path": "C:/USER/Path/Company1/",
"files": [
{
{"_CO": {"ID": "ID", "Report Number": "Report_Number"}},
{"_TR": {"ID": "Trade_Ident", "Report Number": "Number of Report"}},
},
],
},
],
},
{"company2": [
{"path": "C:/USER/Path/Company2/",
"files": [
{
{"_CO": {"ID": "Identification", "Report Number": "Report-Number"}},
{"_TR": {"ID": "Ident", "Report Number": "NumberReport"}},
},
],
},
],
},
},
However, I receive the following error when trying to read the .json in python.
json.decoder.JSONDecodeError: Expecting property name enclosed in
double quotes: line 6 column 5 (char 90)
To read the file I use:
import json
path = "/user_folder/USER/Desktop/Data/"
file = "ConfigFile.json"
with open(path+file) as f:
my_test = json.load(f)
Any help appreciated, as I can't figure out my mistake in the file structure.

You're getting error because your json file is incorrectly formatted and thus calling json.load() will raise an JSONDecodeError.
Your json structure should look like,
{
"companies": {
"company1": [
{
"path": "C:/USER/Path/Company1/",
"files": [
{
"_CO": {
"ID": "ID",
"Report Number": "Report_Number"
}
},
{
"_TR": {
"ID": "Trade_Ident",
"Report Number": "Number of Report"
}
}
]
}
],
"company2": [
{
"path": "C:/USER/Path/Company2/",
"files": [
{
"_CO": {
"ID": "Identification",
"Report Number": "Report-Number"
}
},
{
"_TR": {
"ID": "Ident",
"Report Number": "NumberReport"
}
}
]
}
]
}
}
Hope it helps you!

You have some object (the ones with curly braces) without keys, for example in
{
{"_CO": {"ID": "ID", "Report Number": "Report_Number"}}, ...
Objects in JSON are key-value pairs. Just remove the external set of braces and it should be ok.
You can use some online JSON formatter/validator just like this one, and it will easily point out the problem. Otherwise, you can use some JSON linter for your editor. It just does the work for you and also improves indentation :)

Compose nested JSON with multi columns in Python

I have a csv file and trying to compose JSON from it. There are mulitple records in a file but I am just giving one set of sample records here.This structure is driven on the claimID. There is nesting on the claimLineDetail and claimSpecDiag.I guess I have to create some sort of list to handle this then the problem is how am I going to append it in the required structure. I really need some guidance here to achieve the desired result. Is it possible to break out different sections and append it later, I am not sure just assuming, as there are multiple columns.
Code :
import csv,json
data = []
with open('JsonRequestPricingMedical.csv','r') as f:
reader = csv.DictReader(f)
for row in reader:
print row
csv file :
claimId,subscriberId,claimType,claimSubType,providerId,totalChargeAmt,claimLineNo,pos_code,procedureCode,subdiagnosisCode,svcLineFromDt,svcLineToDt,chargedAmt,clmLineUnits,presentOnAdmit,diagnosisCode
18A000730400,101924200,M,M,002664514003,585,1,11,92014,H43393,2017-06-19,2017-06-19,160,1,U,H43393
18A000730400,101924200,M,M,002664514003,585,2,12,92015,H43395,2017-06-19,2017-06-19,160,2,U,H43394
Desired JSON
[
{
"claimsHeader":" {
"claimId": "18A000730400",
"subscriberId": "101924200",
"claimType":{
"code": "M"
},
"claimSubType": {
"code": "M"
},
"providerId" :"002664514003",
"totalChargeAmt": "585",
"claimLineDetail" :[
{
"claimLineNo": "1",
"placeOfService": {
"code": "11"
},
"procedureCode": {
"code": "92014"
},
"subDiagnosisCd": {
"code": "H43393"
},
"svcLineFromDt": "2017-06-19",
"svcLineToDt": "2017-06-19",
"chargedAmt": "160",
"clmLineUnits": "1",
},
{
"claimLineNo": "2",
"placeOfService": {
"code": "12"
},
"procedureCode": {
"code": "92015"
},
"subDiagnosisCd": {
"code": "H433945
},
"svcLineFromDt": "2017-06-19",
"svcLineToDt": "2017-06-19",
"chargedAmt": "160",
"clmLineUnits": "2",
}
],
{
"claimSpecDiag": [
"presentOnAdmit": "",
"diagnosisCode": "H43393",
},
{
"presentOnAdmit": "",
"diagnosisCode": "H43394",
}
]
}
]

When you read a csv, each line represents variables separated by a special char, in your case, comas: ",".
You can get each variable separated by doing line_variables = row.split(',')
Just pass the first line, and for all the other, do something like:
result = {
"claimsHeader":" {
"claimId": line_variables[0],
"subscriberId": line_variables[1],
"claimType":{
"code": line_variables[2]
}
...
Finaly, just add the result to a list (created just before your for loop) with your_list.append(result).

python querying a json objectpath

I've a nested json structure, I'm using objectpath (python API version), but I don't understand how to select and filter some information (more precisely the nested information in the structure).
EG.
I want to select the "description" of the action "reading" for the user "John".
JSON:
{
"user":
{
"actions":
[
{
"name": "reading",
"description": "blablabla"
}
]
"name": "John"
}
}
CODE:
$.user[#.name is 'John' and #.actions.name is 'reading'].actions.description
but it doesn't work (empty set but in my JSON it isn't so).
Any suggestion?

Is this what you are trying to do?
import objectpath
data = {
"user": {
"actions": {
"name": "reading",
"description": "blablabla"
},
"name": "John"
}
}
tree = objectpath.Tree(data)
result = tree.execute("$.user[#.name is 'John'].actions[#.name is 'reading'].description")
for entry in result:
print entry
Output
blablabla
I had to fix your JSON. Also, tree.execute returns a generator. You could replace the for loop with print result.next(), but the for loop seemed more clear.

import objectpath import *
your_json = {"name": "felix", "last_name": "diaz"}
# This json path will bring all the key-values of your json
your_json_path='$.*'
my_key_values = Tree(your_json).execute(your_json_path)
# If you want to retrieve the name node...then specify it.
my_name= Tree(your_json).execute('$.name')
# If you want to retrieve a the last_name node...then specify it.
last_name= Tree(your_json).execute('$.last_name')

I believe you're just missing a comma in JSON:
{
"user":
{
"actions": [
{
"name": "reading",
"description": "blablabla"
}
],
"name": "John"
}
}
Assuming there is only one "John", with only one "reading" activity, the following query works:
$.user[#.name is 'John'].actions[0][#.name is 'reading'][0].description
If there could be multiple "John"s, with multiple "reading" activities, the following query will almost work:
$.user.*[#.name is 'John'].actions..*[#.name is 'reading'].description
I say almost because the use of .. will be problematic if there are other nested dictionaries with "name" and "description" entries, such as
{
"user": {
"actions": [
{
"name": "reading",
"description": "blablabla",
"nested": {
"name": "reading",
"description": "broken"
}
}
],
"name": "John"
}
}
To get a correct query, there is an open issue to correctly implement queries into arrays: https://github.com/adriank/ObjectPath/issues/60