We Keep Coding

Delete from JSON nested dictionary in list value by iteration

I have JSON file 'json_HW.json' in which I have this format JSON:
{
"news": [
{
"content": "Prices on gasoline have soared on 40%",
"city": "Minsk",
"news_date_and_time": "21/03/2022"
},
{
"content": "European shares fall on weak earnings",
"city": "Minsk",
"news_date_and_time": "19/03/2022"
}
],
"ad": [
{
"content": "Rent a flat in the center of Brest for a month",
"city": "Brest",
"days": 15,
"ad_start_date": "15/03/2022"
},
{
"content": "Sell a bookshelf",
"city": "Mogilev",
"days": 7,
"ad_start_date": "20/03/2022"
}
],
"coupon": [
{
"content": "BIG sales up to 50%!",
"city": "Grodno",
"days": 5,
"shop": "Marko",
"coupon_start_date": "17/03/2022"
}
]
}
I need to delete field_name and field_value with their keys when I reach them until the whole information in the file is deleted. When there is no information in the file, I need to delete the file itself
The code I have
data = json.load(open('json_HW.json'))
for category, posts in data.items():
for post in posts:
for field_name, field_value in post.items():
del field_name, field_value
print(data)
But the variable data doesn't change when I delete and delete doesn't work. If it worked I could rewrite my JSON

You are deleting the key and the value, after extracting them from the dictionary,
that doesn't affect the dictionary. What you should do is delete the dictionary entry:
import json
import os
file_name = 'json_HW.json'
data = json.load(open(file_name))
for category in list(data.keys()):
posts = data[category]
elem_indices = []
for idx, post in enumerate(posts):
for field_name in list(post.keys()):
del post[field_name]
if not post:
elem_indices.insert(0, idx) # so you get reverse order
for idx in elem_indices:
del posts[idx]
if not posts:
del data[category]
print(data)
if not data:
print('deleting', file_name)
os.unlink(file_name)
which gives:
{}
deleting json_HW.json
Note that the list() is necessary, post.keys() is a generator and
you cannot change the dict while you are iterating over its keys (or items or values).

if you want to delete key-value from dictionary, you can use del post[key].
but i don't think it works for iteration, cause dictionary size keeps changing.
https://www.geeksforgeeks.org/python-ways-to-remove-a-key-from-dictionary/

Python: Identify the duplicate JSON values and generate an output file with sets of duplicate input rows, one row per set of duplicates

I am trying to find duplicated JSON objects in a 30GB jsonlines file.
Given a JSON object A that look like this:
{
"data": {
"cert_index": 691749790,
"cert_link": "http://ct.googleapis.com/icarus/ct/v1/get-entries?start=691749790&end=691749790",
"chain": [{...},{...}],
"leaf_cert": {
"all_domains": [...],
"as_der": "MIIFcjCCBFqgAwIBAgISBDD2+d1gP36/+9uUveS...",
"extensions": {...},
"fingerprint": "0C:E4:AF:24:F1:AE:B1:09:B0:42:67:CB:F8:FC:B6:AF:1C:07:D6:5B",
"not_after": 1573738488,
"not_before": 1565962488,
"serial_number": "430F6F9DD603F7EBFFBDB94BDE4BBA4EC9A",
"subject": {...}
},
"seen": 1565966152.750253,
"source": {
"name": "Google 'Icarus' log",
"url": "ct.googleapis.com/icarus/"
},
"update_type": "PrecertLogEntry"
},
"message_type": "certificate_update"
}
How can I generate an output file where each row looks like this:
{"fingerprint":"0C:E4:AF:24:F1:AE:B1:09:B0:42:67:CB:F8:FC:B6:AF:1C:07:D6:5B", "certificates":[A, B, C]}
Here A, B, and C are the full JSON object for each of the duplicates.

You need to use an array with your information. And before adding a new JSON, check if the fingerprint is already in the array. For example:
currentFingerprint = myJson['data']['leaf_cert']['fingerprint']
for elem in arrayOfFingerprints:
if elem['fingerprint'] == currentFingerprint:
elem['certificates'].append(myJson)
break
else:
arrayOfFingerprints.append({'fingerprint': currentFingerprint, 'certificates': [myJson]}

I'm going to assume that you have already read the file and created a list of dicts.
from collections import defaultdict
import json
d = defaultdict(list)
for jobj in file:
d[jobj['data']['leaf_cert']['fingerprint']].append(jobj)
with open('file.txt', 'w') as out:
for k,v in d:
json.dump({"fingerprint":k, "certificates":v})

How to extract elements from JSON File with python?

I have the JSON file below and i am trying to extract the value dob_year from each item if the name element is jim.
This is my try:
import json
with open('j.json') as json_file:
data = json.load(json_file)
if data['name'] == 'jim'
print(data['dob_year'])
Error:
File "extractjson.py", line 6 if data['name'] == 'jim' ^ SyntaxError: invalid syntax
this is my file.
[
{
"name": "jim",
"age": "10",
"sex": "male",
"dob_year": "2007",
"ssn": "123-23-1234"
},
{
"name": "jill",
"age": "6",
"sex": "female",
"dob_year": "2011",
"ssn": "123-23-1235"
}
]

You need to iterate over the list in the JSON file
data = json.load(json_file)
for item in data:
if item['name'] == 'jim':
print(item['dob_year'])

I suggest to use list comprehension:
import json
with open('j.json') as json_file:
data = json.load(json_file)
print([item["dob_year"] for item in a if item["name"] == "jim"])

json.load()
returns a list of entries, the way you have saved it. You have to iterate through all list items, then search for your field in the list item.
Also, if it is a repetitive task, make a function out of it. You can pass different files, fields to be extracted, etc. this way, and it can make your job easier.
Example:
def extract_info(search_field, extract_field, name, filename):
import json
with open(filename) as json_file:
data = json.load(json_file)
for item in data:
if item[search_field] == name:
print(item[extract_field])
extract_info('name','dob_year','jim','j.json')

data is a list of dictionaries! you cannot directly access the key value pairs.
Try encapsulating it in a loop to check every dict in the list:
import json
with open('j.json') as json_file:
data = json.load(json_file)
for set in data:
if set['name'] == 'jim':
print(set['dob_year'])

Append to an array inside a JSON object based on key in python

I have some JSON I'm looping through in the following format. I need to create an object for each unique primary key found in the source data and append to an array. I'm not sure how I would create the object on first encounter of the key and append to it on the next encounter. My initial attempt just creates a new object for each object in the source. Wasn't able to find an example in python only js.
Source data format:
[
...
{
"Id": "NOT NEEDED DATA",
"Client": {
"Id": "KEY",
"Name": "NOT NEEDED DATA"
},
"Name": "DESIRED DATAPOINT"
},
...
]
Desired format:
[
...
{
"client_id": "KEY",
"locations": ["DATA", "DATA"]
}
...
]
pseudocode
for i in sourcedata:
client_id = i['Client']['Id']
location_name = i['Name']
obj = {
"client_id": client_id,
"locations": [location_name]
}
new_array.append(obj)

You can first iterate and build a dictionary for then creating a list of dictionaries as specified in your output format.
from collections import defaultdict
# create and populate the dictionary
d = defaultdict(list)
for i in sourcedata:
client_id = i['Client']['Id']
location_name = i['Name']
d[client_id].append(location_name)
# build the result
res = [{"client_id": k, "locations": v} for k,v in d.items()]

Sorting a JSON Object

I'm new to JSON and Python, and I'm attempting to load a json file that I created from disk to manipulate and output to an xml file. I've gotten most of it figured out, except, I want to 'sort' the JSON file after I load it by a certain value.
Example of json file:
{
"0" : {
"name": "John Doe",
"finished": "4",
"logo": "http://someurl.com/icon.png"
},
"1" : {
"name": "Jane Doe",
"finished": "10",
"logo": "http://anotherurl.com/icon.png"
},
"2" : {
"name": "Jacob Smith",
"finished": "3",
"logo": "http://example.com/icon.png"
}
}
What I want to do is sort 'tree' by the 'finished' key.
JSONFILE = "file.json"
with open(CHANS) as json_file:
tree = json.load(json_file)

Depends on how do you "consume" the tree dictionary. are you using tree.keys(), tree.values() or tree.items()?
tree.keys()
ordered_keys = sorted(tree.keys(), key=lambda k: int(tree[k]['finished']))
tree.values()
ordered_keys = sorted(tree.values(), key=lambda v: int(v['finished']))
tree.items()
ordered_keys = sorted(tree.items(), key=lambda t: int(t[1]['finished']))
You only keep in mind that JSON is what's inside the actual file, the result of json.load() is just a Python value/object, so just work with them.
If you are walking over the sorted dictionary once, the above snippets will work just fine. However if you need to access it multiple times, then I would follow ~Jean-François suggestion and use OrderedDict, with something like:
from collections import OrderedDict
tree = OrderedDict(sorted(tree.items(), key=lambda t: int(t[1]['finished'])))
That way the sorting operation (arguably the most expensive) is done just once.