We Keep Coding

Need to cut off some unnecessary information from a JSON file and preserve the JSON structure

I have a JSON file
[
{
"api_key": "123123112313121321",
"collaborators_count": 1,
"created_at": "",
"custom_event_fields_used": 0,
"discarded_app_versions": [],
"discarded_errors": [],
"errors_url": "https://api.bugsnag.com/projects/1231231231312/errors",
"events_url": "https://api.bugsnag.com/projects/1231231231213/events",
"global_grouping": [],
"html_url": "https://app.bugsnag.com/lol/kek/",
"id": "34234243224224",
"ignore_old_browsers": true,
"ignored_browser_versions": {},
"is_full_view": true,
"language": "javascript",
"location_grouping": [],
"name": "asdasdaasd",
"open_error_count": 3,
"release_stages": [
"production"
],
"resolve_on_deploy": false,
"slug": "wqeqweqwwqweq",
"type": "js",
"updated_at": "2020-04-06T15:22:10.480Z",
"url": "https://api.bugsnag.com/projects/12312312213123",
"url_whitelist": null
}
]
What I need is to remove all lines apart from "id:" and "name:" and preserve the JSON structure. Can anybody advise a Python or bash script to handle this?

With jq:
$ jq 'map({id: .id, name: .name})' input.json
[
{
"id": "34234243224224",
"name": "asdasdaasd"
}
]

Using python, you could first deserialize the JSON file(JSON array of objects) with json.load, then filter out the keys you want with a list comprehension:
from json import load
keys = ["name", "id"]
with open("test.json") as json_file:
data = load(json_file)
filtered_json = [{k: obj.get(k) for k in keys} for obj in data]
print(filtered_json)
Output:
[{'name': 'asdasdaasd', 'id': '34234243224224'}]
If we want to serialize this python list to another output file, we can use json.dump:
from json import load
from json import dump
keys = ["name", "id"]
with open("test.json") as json_file, open("output.json", mode="w") as json_output:
data = load(json_file)
filtered_json = [{k: obj.get(k) for k in keys} for obj in data]
dump(filtered_json, json_output, indent=4, sort_keys=True)
output.json
[
{
"id": "34234243224224",
"name": "asdasdaasd"
}
]

You can try this:
import json
with open('<input filename>', 'r') as f:
data = json.load(f)
new_data = []
for item in data:
new_item = {key: value for key, value in item.items() if key == "id" or key =="name"}
new_data.append(new_item)
with open('<output filename>', 'w') as f:
json.dump(new_data, f)

Covert your JSON into Pandas Dataframe
{
import pandas as pd
df=pd.read_json('your json variable')
res=df.drop(['url_whitelis','api_key'],axis=1)
pd.to_json(res) }

How to save multiple JSON strings in a single file?

I have the following multiple JSON strings in a file:
{
"col1": "e1",
"col2": "e5"
}
{
"col1": "a1",
"col2": "a4",
"col3": "e8"
}
This is how I read this file:
import json
data = []
for line in open('test.json', 'r', encoding='cp850'):
data.append(json.loads(line))
Is there any way to say this file back in the following format (i.e. all JSON strings would be wrapped inside [..] and there should be commas separating them):
[
{
"col1": "e1",
"col2": "e5"
},
{
"col1": "a1",
"col2": "a4",
"col3": "e8"
}
]
For those who need some examples as a proof of whatever, I tried this:
with open('output.json', 'w') as f:
json.dump(data, f)
It does not write the content of data into JSON file.
And a dirty solution that neither works:
data = ""
with open('test.json', 'r', encoding='cp850') as myfile:
data = str(json.loads(line)) + "," + data
data = data[:len(data)-1] + "]" + data[len(data):]
data = "[" + data
with open('output.json', 'w') as outfile:
json.dump(data, outfile)

It's not a good idea to have multiple JSONs per file without some kind of delimiter.
I recommend newline delimited (meaning, each JSON object is confined to 1 line in the file):
{"name": "json1"}
{"name": "json2"}
{"name": "json3"}
Then you can simply iterate the file and do something like this:
objs = []
with open("jsons.txt") as f:
for line in f:
objs.append(json.loads(line))
with open("jsons.json", "w") as f:
json.dump(obj, f);

Try this:
data = \
"""{
"col1": "e1",
"col2": "e5"
}
{
"col1": "a1",
"col2": "a4",
"col3": "e8"
}
"""
new_json = []
bracket_lvl = 0
curr_json = ''
for c in data:
curr_json += c
if c in (' ', '\t', '\n'):
continue
if c == '{':
bracket_lvl += 1
if c == '}':
bracket_lvl -= 1
if bracket_lvl == 0:
new_json.append(curr_json)
curr_json = ''
output = "[\n" + ','.join(new_json) + "\n]"
print(output)
Output:
[
{
"col1": "e1",
"col2": "e5"
},
{
"col1": "a1",
"col2": "a4",
"col3": "e8"
}
]
Edit:
Please note, this only works if the json strings are surrounded by curly braces - if you need code to handle square braces, I'll need to slightly edit this...
Edit 2:
If you want to save it to a file, use this at the bottom of the program:
with open('output.json', 'w') as f:
f.write(output)

Convert a complex layered JSON to CSV

I am trying to parse through JSON code and write the results into a csv file. The "name" values are supposed to be the column headers and the 'value' values are what need to be stored.This is my code. the CSV file writer does not separate the strings with commas: eventIdlistingsvenueperformer and when I try to do something like: header = col['name']+',' I get: eventId","listings","venue","performer And it isn't read as a csv file so...My questions are: am I going about this right? and how could I separate the strings by commas?
"results": [
{
"columns": [
{
"name": "eventId",
"value": "XXXX",
"defaultHidden": false
},
{
"name": "listings",
"value": "8",
"defaultHidden": false
},
{
"name": "venue",
"value": "Nationwide Arena",
"defaultHidden": false
}]
this is my code:
json_decode=json.loads(data)
report_result = json_decode['results']
with open('testReport2.csv','w') as result_data:
csvwriter = csv.writer(result_data,delimiter=',')
count = 0
for res in report_result:
deeper = res['columns']
for col in deeper:
if count == 0:
header = col['name']
csvwriter.writerow([header,])
count += 1
for written in report_result:
deeper =res['columns']
for col in deeper:
csvwriter.writerow([trouble,])
result_data.close()

try below code:
json_decode=json.loads(data)
report_result = json_decode['results']
new_dict = {}
for result in report_result:
columns = result["columns"]
for value in columns:
new_dict[value['name']] = value['value']
with open('testReport2.csv','w') as result_data:
csvwriter = csv.DictWriter(result_data,delimiter=',',fieldnames=new_dict.keys())
csvwriter.writeheader()
csvwriter.writerow(new_dict)

Try this:
json_decode=json.loads(data)
report_result = json_decode['results']
with open('testReport2.csv','w') as result_data:
csvwriter = csv.writer(result_data,delimiter=',')
header = list(report_result[0]['columns'][0].keys())
csvwriter.writerow(header)
for written in report_result:
for row in written['columns']:
deeper =row.values()
csvwriter.writerow(deeper)
result_data.close()

Write different type of Json Resposne into CSV format by use Python Script

I have different Json Response like below sample :
In My case keys is in form of column Name and data is in rows Node.
{
"count": 2,
"name": "Report",
"columnNames": [
"Name",
"Address",
"Account",
"Completed"
],
"rows": [
[
"'ABC'",
Xyz,
"'Admin'",
"'Yes'"
],
[
"'ABC1'",
Xyz,
"'Admin'",
"'Yes'"
],
[
"'ABC2'",
Xyz,
"'Admin'",
"'Yes'"
]
]
}
and then i want to convert these json into csv format like this
Name,Address,Account, Completed
"'ABC'",Xyz,"'Admin'","'Yes'"
"'ABC1'",Xyz,"'Admin'","'Yes'"
"'ABC2'",Xyz,"'Admin'","'Yes'"

You can use the csv module and some formatting, say you have a json object
import csv
csv_arr = [json["columnNames"]]
for row in json["rows"]:
row_csv_arr = []
csv_arr.append(row)
Finally, you can write to a file
with open("output.csv",'wb') as csv_file:
wr = csv.writer(csv_file)
wr.writerows(csv_arr)
Or get it as a string
# Join the arrays
for i, val in enumerate(csv_arr):
csv_arr[i] = ", ".join(val)
csv_string = "\n".join(csv_arr)

Splitting values and assigning keys

I have a json array with long and lat values combined, and fields are empty. I am trying to figure out a python script to loop through the array and separate them into long and lat keys.
[
{
"Shape": "(47.630880207000075, -122.37391463199998)"
},
{
"Shape": ""
},
{
"Shape": "(47.70118823100006, -122.38447021399998)"
}
]
output desired:
[
{
"Long": "47.630880207000075",
"Lat": "-122.37391463199998"
},
{
"Long": "",
"Lat": ""
},
{
"Long": "47.70118823100006",
"Lat": "-122.38447021399998"
}
]
So far I have converted the CSV into JSON and in the process of figuring out the regex to loop through it.. code so far:
import csv
import json
csvfile = open('geoloc.csv', 'r')
jsonfile = open('file.json', 'w')
# fieldnames = ("S")
reader = csv.DictReader( csvfile)
out = json.dumps( [ row for row in reader ], sort_keys=True, indent=4 )
jsonfile.write(out)

It's probably easy to use ast.literal_eval to turn this into a tuple vs. re, e.g.:
In [1]:
import ast
def tuple_convert(t):
return ast.literal_eval(t) if t else ('', '')
keys = ['Long', 'Lat']
[dict(zip(keys, tuple_convert(d['Shape']))) for d in data]
Out[1]:
[{'Lat': -122.37391463199998, 'Long': 47.630880207000075},
{'Lat': '', 'Long': ''},
{'Lat': -122.38447021399998, 'Long': 47.70118823100006}]

Assuming your json is loaded to d
>>> l = [{"Long": c['Shape'][1:-1].split(',')[0].strip() if len(c['Shape']) > 0 else '',
"Lat": c['Shape'][1:-1].split(',')[1].strip() if len(c['Shape']) > 0 else ''} for c in d]
>>> l
[{'Lat': '-122.37391463199998', 'Long': '47.630880207000075'},
{'Lat': '', 'Long': ''},
{'Lat': '-122.38447021399998', 'Long': '47.70118823100006'}]
EDIT:
To make it more readable:
l = [ dict(zip(['Long', 'Lat'],c['Shape'][1:-1].split(',') if len(c['Shape'])>0 else ['',''] )) for c in d]