We Keep Coding

how to retrieve data from json file using python

I'm doing api requests to get json file to be parsed and converted into data frames. Json file sometimes may have empty fields, I am posting 2 possible cases where 1st json fill have the field I am looking for and the 2nd json file has that field empty.
1st json file:
print(resp2)
{
"entityId": "proc_1234",
"displayName": "oracle12",
"firstSeenTms": 1639034760000,
"lastSeenTms": 1650386100000,
"properties": {
"detectedName": "oracle.sysman.gcagent.tmmain.TMMain",
"bitness": "64",
"jvmVendor": "IBM",
"metadata": [
{
"key": "COMMAND_LINE_ARGS",
"value": "/usr/local/oracle/oem/agent12c/agent_13.3.0.0.0"
},
{
"key": "EXE_NAME",
"value": "java"
},
{
"key": "EXE_PATH",
"value": "/usr/local/oracle/oem/agent*c/agent_*/oracle_common/jdk/bin/java"
},
{
"key": "JAVA_MAIN_CLASS",
"value": "oracle.sysman.gcagent.tmmain.TMMain"
},
{
"key": "EXE_PATH",
"value": "/usr/local/oracle/oem/agent12c/agent_13.3.0.0.0/oracle_common/jdk/bin/java"
}
]
}
}
2nd Json file:
print(resp2)
{
"entityId": "PROCESS_GROUP_INSTANCE-FB8C65551916D57D",
"displayName": "Windows System",
"firstSeenTms": 1619147697131,
"lastSeenTms": 1653404640000,
"properties": {
"detectedName": "Windows System",
"bitness": "32",
"metadata": [],
"awsNameTag": "Windows System",
"softwareTechnologies": [
{
"type": "WINDOWS_SYSTEM"
}
],
"processType": "WINDOWS_SYSTEM"
}
}
as you can see metadata": [] empty.
I need to extract entityId, detectedName and if metada has data, I need to get EXE_NAME and EXE_PATH. if metada section is empty, I still need to get the entityId and detectedName from this json file and form a data frame.
so, I have done this:
#retrieve the detecteName value from the json
det_name = list(resp2.get('properties','detectedName').values())[0]
#retrieve EXE_NAME, EXE_PATH and entityId from the json. This part works when metata section has data
Procdf=(pd.json_normalize(resp2, record_path=['properties', 'metadata'], meta=['entityId']).drop_duplicates(subset=['key']).query("key in ['EXE_NAME','EXE_PATH']").assign(detectedName=det_name).pivot('entityId', 'key', 'value').reset_index())
#Add detectedName to the Procdf data frame
Procdf["detectedName"] = det_name
this above code snippet works when metadata has data, if it has no data [], I still need to create a data frame with entityId, detectedName and EXE_NAME and EXE_PATH being empty.
how can I do this? Right now when metadat[], I get this error name 'key' is not defined and skipps that json.

Why not create a new dict based on whether there's value for metadata or not?
Here's an example (this should work with both response types):
import pandas as pd
def find_value(response: dict, key: str) -> str:
result = []
try:
for x in response['properties']['metadata']:
if x['key'] == key:
result.append(x['value'])
except KeyError:
return ""
return result[0] if result else ""
def get_values(response: dict) -> dict:
return {
"entityId": response['entityId'],
"displayName": response['displayName'],
"EXE_NAME": find_value(response, 'EXE_NAME'),
"EXE_PATH": find_value(response, 'EXE_PATH'),
}
sample_response = {
"entityId": "PROCESS_GROUP_INSTANCE-FB8C65551916D57D",
"displayName": "Windows System",
"firstSeenTms": 1619147697131,
"lastSeenTms": 1653404640000,
"properties": {
"detectedName": "Windows System",
"bitness": "32",
"awsNameTag": "Windows System",
"metadata": [],
"softwareTechnologies": [
{
"type": "WINDOWS_SYSTEM"
}
],
"processType": "WINDOWS_SYSTEM"
}
}
print(pd.json_normalize(get_values(sample_response)))
Sample output for metadata being empty:
entityId displayName EXE_NAME EXE_PATH
0 PROCESS_GROUP_INSTANCE-FB8C65551916D57D Windows System
And one when metadata carries, well, data:
entityId ... EXE_PATH
0 proc_1234 ... /usr/local/oracle/oem/agent*c/agent_*/oracle_c...

Convert json data to dictionary

I am trying to convert my json data into a dictionary with key the id of the json data. for example lets say i have the following json:
{
"id": "1",
"name": "John",
"surname": "Smith"
},
{
"id": "2",
"name": "Steve",
"surname": "Ger"
}
And i want to construct a new dictionary which includes the id as a key and save it into a file so i wrote the following code:
json_dict = []
request = requests.get('http://example.com/...')
with open("data.json", "w") as out:
loaded_data = json.loads(request.text)
for list_item in loaded_data:
json_dict.append({"id": list_item["id"], "data": list_item })
out.write(json.dumps(json_dict))
In the file i get the following output:
[{"data": {"id":"1",
"name":"John",
"Surname":"Smith"
}
},
{"data": {"id":"2",
"name":"Steve",
"Surname":"Ger"
}
},
]
Why the id is not included in my dict before data ?

I'm pretty sure you're looking at a ghost here. You probably tested wrong. It will go away when you try to create a minimal, complete, and verifiable example for us (i. e. with a fixed string as input instead of a request call, with a print instead of an out.write, etc).
This is my test in which I could not reproduce the problem:
entries = [
{
"id": "1",
"name": "John",
"surname": "Smith"
},
{
"id": "2",
"name": "Steve",
"surname": "Ger"
}
]
json_dict = []
for i in entries:
json_dict.append({"id":i["id"], "data": i})
json.dumps(json_dict, indent=2)
This prints as expected:
[
{
"id": "1",
"data": {
"id": "1",
"surname": "Smith",
"name": "John"
}
},
{
"id": "2",
"data": {
"id": "2",
"surname": "Ger",
"name": "Steve"
}
}
]

You could try this instead:
json_dict = []
with open('data.json', 'w) as f:
loaded_data = json.loads(request.text)
for list_item in loaded_data:
json_dict.append({list_item['id'] : list_item})

Accessing nested json objects using python

I am trying to interact with an API and running into issues accessing nested objects. Below is sample json output that I am working with.
{
"results": [
{
"task_id": "22774853-2b2c-49f4-b044-2d053141b635",
"params": {
"type": "host",
"target": "54.243.80.16",
"source": "malware_analysis"
},
"v": "2.0.2",
"status": "success",
"time": 227,
"data": {
"details": {
"as_owner": "Amazon.com, Inc.",
"asn": "14618",
"country": "US",
"detected_urls": [],
"resolutions": [
{
"hostname": "bumbleride.com",
"last_resolved": "2016-09-15 00:00:00"
},
{
"hostname": "chilitechnology.com",
"last_resolved": "2016-09-16 00:00:00"
}
],
"response_code": 1,
"verbose_msg": "IP address in dataset"
},
"match": true
}
}
]
}
The deepest I am able to access is the data portion which returns too much.... ideally I am just trying access as_owner,asn,country,detected_urls,resolutions
When I try to access details / response code ... etc I will get a KeyError. My nested json goes deeper then other Q's mentioned and I have tried that logic.
Below is my current code snippet and any help is appreciated!
import requests
import json
headers = {
'Content-Type': 'application/json',
}
params = (
('wait', 'true'),
)
data = '{"target":{"one":{"type": "ip","target": "54.243.80.16", "sources": ["xxx","xxxxx"]}}}'
r=requests.post('https://fakewebsite:8000/api/services/intel/lookup/jobs', headers=headers, params=params, data=data, auth=('apikey', ''))
parsed_json = json.loads(r.text)
#results = parsed_json["results"]
for item in parsed_json["results"]:
print(item['data'])

You just need to index correctly into the converted JSON. Then you can easily loop over a list of the keys you want to fetch, since they are all in the "details" dictionary.
import json
raw = '''\
{
"results": [
{
"task_id": "22774853-2b2c-49f4-b044-2d053141b635",
"params": {
"type": "host",
"target": "54.243.80.16",
"source": "malware_analysis"
},
"v": "2.0.2",
"status": "success",
"time": 227,
"data": {
"details": {
"as_owner": "Amazon.com, Inc.",
"asn": "14618",
"country": "US",
"detected_urls": [],
"resolutions": [
{
"hostname": "bumbleride.com",
"last_resolved": "2016-09-15 00:00:00"
},
{
"hostname": "chilitechnology.com",
"last_resolved": "2016-09-16 00:00:00"
}
],
"response_code": 1,
"verbose_msg": "IP address in dataset"
},
"match": true
}
}
]
}
'''
parsed_json = json.loads(raw)
wanted = ['as_owner', 'asn', 'country', 'detected_urls', 'resolutions']
for item in parsed_json["results"]:
details = item['data']['details']
for key in wanted:
print(key, ':', json.dumps(details[key], indent=4))
# Put a blank line at the end of the details for each item
print()
output
as_owner : "Amazon.com, Inc."
asn : "14618"
country : "US"
detected_urls : []
resolutions : [
{
"hostname": "bumbleride.com",
"last_resolved": "2016-09-15 00:00:00"
},
{
"hostname": "chilitechnology.com",
"last_resolved": "2016-09-16 00:00:00"
}
]
BTW, when you fetch JSON data using requests there's no need to use json.loads: you can access the converted JSON using the .json method of the returned request object instead of using its .text attribute.
Here's a more robust version of the main loop of the above code. It simply ignores any missing keys. I didn't post this code earlier because the extra if tests make it slightly less efficient, and I didn't know that keys could be missing.
for item in parsed_json["results"]:
if not 'data' in item:
continue
data = item['data']
if not 'details' in data:
continue
details = data['details']
for key in wanted:
if key in details:
print(key, ':', json.dumps(details[key], indent=4))
# Put a blank line at the end of the details for each item
print()

python parsing json response

I'm a newb with JSON and am trying to understand how to parse a JSON response. In the example below, I'd like to know how to retrieve the value of 'issueId':'executions':'id'? in the example below it is '8195'.....
r = requests.get(baseURL + getExecutionsForIssueId + id, auth=('user','pass'))
data = r.json()
JSON Response:
{
"status": {
"1": {
"id": 1,
"color": "#75B000",
"description": "Test was executed and passed successfully.",
"name": "PASS"
},
"2": {
"id": 2,
"color": "#CC3300",
"description": "Test was executed and failed.",
"name": "FAIL"
},
"3": {
.
.
.
}
},
"issueId": 15825,
"executions": [
{
"id": 8195,
"orderId": 7635,
"executionStatus": "-1",
"comment": "",
"htmlComment": "",
.
.
.

Your JSON object is just a dictionary in Python. Access the values you need like so:
data['executions'] yields an array of similar dictionary objects, assuming your JSON response is typed as you intended.
executions = data['executions']
order_id = executions[0]['orderId']
If you wish to loop over them to find the correct object with an id of 8195:
executions = data['executions'] # [{'id':8195,'orderId':7635,...}, {...}, ...]
for e in executions:
if e['id'] == 8195: # e is the dict you want
order_id = e['orderId']

Parsing JSON in python for second object

I have a sample JSON in this format:
JSON FILE:
{
"Name": "ABC",
"Phone":"123",
"Address":[{"City":"City-1"},{"Country":"Country-1"}]
}
{
"Name": "ABC-1",
"Phone":"123-1",
"Address":[{"City":"City-2"},{"Country":"Country-2"}]
}
Is there any approach to parse the JSON and loop through the file and print each key-value pair.
The approach I used was through using
json_open = open(json_file)
json_data = json.load(json_open)
print(json_data[Name]) ##should give ABC
print(json_data[Name]) ##should give ABC-1 - unsure about the syntax and format
But I'm currently able to print only the first object values - i.e. name=ABC and not name=ABC-1

There is error in your json file. I modified your json and written code for traverse each element in it.
Error:
Error: Parse error on line 9:
... "Country-1" }]}{ "Name": "ABC-1",
-------------------^
Expecting 'EOF', '}', ',', ']', got '{'
sample.json
{
"data": [
{
"Name": "ABC",
"Phone": "123",
"Address": [
{
"City": "City-1"
},
{
"Country": "Country-1"
}
]
},
{
"Name": "ABC-1",
"Phone": "123-1",
"Address": [
{
"City": "City-2"
},
{
"Country": "Country-2"
}
]
}
]
}
sample.py
import json
json_file='sample.json'
with open(json_file, 'r') as json_data:
data = json.load(json_data)
jin=data['data']
for emp in jin:
print ("Name :"+emp["Name"])
print ("Phone :"+emp["Phone"])
print ("City :"+emp["Address"][0]["City"])
print ("Country :"+emp["Address"][1]["Country"])

Each record on json file is separated by ','. So, your file should look like:
[{
"Name": "ABC",
"Phone": "123",
"Address": [{
"City": "City-1"
}, {
"Country": "Country-1"
}]
},
{
"Name": "ABC-1",
"Phone": "123-1",
"Address": [{
"City": "City-2"
}, {
"Country": "Country-2"
}]
}
]
You can read the file and output as follows :
import json
my_file='test.json'
with open(my_file, 'r') as my_data:
data = json.load(my_data)
print data
for elm in data:
print elm['Phone']
print elm['Name']
print elm['Address'][0]['City']
print elm['Address'][1]['Country']

First of all this is not valid json format,
You can check here you json format
You should use this try.json file
{
"data":[{
"Name": "ABC",
"Phone":"123",
"Address":[{"City":"City-1"},{"Country":"Country-1"}]
}, {
"Name": "ABC-1",
"Phone":"123-1",
"Address":[{"City":"City-2"},{"Country":"Country-2"}]
}]
}
here is try.py
import json
json_open = open("try.json")
json_data = json.load(json_open)
for i in range(len(json_data['data'])):
print(json_data['data'][i]["Name"])
Note : python code is written in python3
output is look line this
ABC
ABC-1