We Keep Coding

Parsing Json extracting key value python

Hi guys I am trying to extract the same key but with different values over a long JSON response, but i keep getting :
KeyError: 'id'
Not sure what i am doing wrong, but i am accessing it using REST API:
This is what i have as a script :
from requests.auth import HTTPBasicAuth
import requests
import json
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def countries():
data = requests.get("https://10.24.21.4:8543/api/netim/v1/countries/", verify=False, auth=HTTPBasicAuth("admin", "admin"))
rep = data.json()
for cid in rep:
cid = rep["id"]
print(cid)
countries()
The response is rather long, but it is like this, you will see "id", and i need the respective values :
{
"items": [
{
"name": "Afghanistan",
"displayName": "Afghanistan",
"meta": {
"type": "COUNTRY"
},
"id": "AF",
"links": {
"self": {
"path": "/api/netim/v1/countries/AF"
}
}
},
{
"name": "Albania",
"displayName": "Albania",
"meta": {
"type": "COUNTRY"
},
"id": "AL",
"links": {
"self": {
"path": "/api/netim/v1/countries/AL"
}
}
},
{
"name": "Algeria",
"displayName": "Algeria",
"meta": {
"type": "COUNTRY"
},
"id": "DZ",
"links": {
"self": {
"path": "/api/netim/v1/countries/DZ"
}
}
},
{
"name": "American Samoa",
"displayName": "American Samoa",
"meta": {
"type

I rewrote your functions a little, You should now be able to get all teh IDs from the JSON response. I suggest you look into teh basics of Dictionaries and Lists in Python
from requests.auth import HTTPBasicAuth
import requests
import json
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
def countries():
data = requests.get("https://10.24.21.4:8543/api/netim/v1/countries/", verify=False, auth=HTTPBasicAuth("admin", "admin"))
rep = data.json()
return [elem.get("id","") for elem in rep['items']]
countries()
Update:
If you wish to extract the value of the "path" key and simultaneously the value of the "id" key, You would need a list of dictionaries where every dictionary corresponds to a single record from the json.
the modified function is as follows:
def countries():
data = requests.get("https://10.24.21.4:8543/api/netim/v1/countries/", verify=False, auth=HTTPBasicAuth("admin", "admin"))
rep = data.json()
return [{"id":elem.get("id",""),"path":elem["links"]["self"]["path"]} for elem in rep['items']]
the get() returns a default value in case the key is absent in the dictionary. The function, new as well as the previous one, would not fail in case the values were not returned in the JSON response for the id and path keys
If you are sure that the value of links will always be available you can use the above function directly else you will have to write a custom function that would parse the key links and return an empty string if it is empty in the json

The response is not an array, it's a dictionary.
You want the "items" element of that dictionary:
for cid in rep['items']:

flask-sqlalchemy dynamically construct query

I have an input json like the following:
{
"page": 2,
"limit": 10,
"order": [
{
"field": "id",
"type": "asc"
},
{
"field": "email",
"type": "desc"
},
...
{
"field": "fieldN",
"type": "desc"
}
],
"filter": [
{
"field": "company_id",
"type": "=",
"value": 1
},
...
{
"field": "counter",
"type": ">",
"value": 5
}
]
}
How do I dynamically construct sqlalchemy query based on my input json if I don't know fields count?
Something like this:
User.query.filter(filter.field, filter.type, filter.value).filter(filter.field1, filter.type1, filter.value1)...filter(filter.fieldN, filter.typeN, filter.valueN).order_by("id", "ask").order_by("email", "desc").order_by("x1", "y1")....order_by("fieldN"...."desc").all()

Convert the json into a dictionary and retrieve the value.
If your json is in a file (say, data.json), the json library will satisfy your needs:
import json
f = open("data.json")
data = json.load(f)
f.close()
User.query.filter(company_id=1).order_by(data["id"], data["ask"]).order_by(data["email"], data["desc"]).all()
If your json is a string (say, json_data):
import json
data = json.loads(json_data)
User.query.filter(company_id=1).order_by(data["id"], data["ask"]).order_by(data["email"], data["desc"]).all()
If your json is a request from the python requests library i.e. res = requests.get(...), then res.json() will return a dictionary:
data = res.json()
User.query.filter(company_id=1).order_by(data["id"], data["ask"]).order_by(data["email"], data["desc"]).all()

Accessing nested json objects using python

I am trying to interact with an API and running into issues accessing nested objects. Below is sample json output that I am working with.
{
"results": [
{
"task_id": "22774853-2b2c-49f4-b044-2d053141b635",
"params": {
"type": "host",
"target": "54.243.80.16",
"source": "malware_analysis"
},
"v": "2.0.2",
"status": "success",
"time": 227,
"data": {
"details": {
"as_owner": "Amazon.com, Inc.",
"asn": "14618",
"country": "US",
"detected_urls": [],
"resolutions": [
{
"hostname": "bumbleride.com",
"last_resolved": "2016-09-15 00:00:00"
},
{
"hostname": "chilitechnology.com",
"last_resolved": "2016-09-16 00:00:00"
}
],
"response_code": 1,
"verbose_msg": "IP address in dataset"
},
"match": true
}
}
]
}
The deepest I am able to access is the data portion which returns too much.... ideally I am just trying access as_owner,asn,country,detected_urls,resolutions
When I try to access details / response code ... etc I will get a KeyError. My nested json goes deeper then other Q's mentioned and I have tried that logic.
Below is my current code snippet and any help is appreciated!
import requests
import json
headers = {
'Content-Type': 'application/json',
}
params = (
('wait', 'true'),
)
data = '{"target":{"one":{"type": "ip","target": "54.243.80.16", "sources": ["xxx","xxxxx"]}}}'
r=requests.post('https://fakewebsite:8000/api/services/intel/lookup/jobs', headers=headers, params=params, data=data, auth=('apikey', ''))
parsed_json = json.loads(r.text)
#results = parsed_json["results"]
for item in parsed_json["results"]:
print(item['data'])

You just need to index correctly into the converted JSON. Then you can easily loop over a list of the keys you want to fetch, since they are all in the "details" dictionary.
import json
raw = '''\
{
"results": [
{
"task_id": "22774853-2b2c-49f4-b044-2d053141b635",
"params": {
"type": "host",
"target": "54.243.80.16",
"source": "malware_analysis"
},
"v": "2.0.2",
"status": "success",
"time": 227,
"data": {
"details": {
"as_owner": "Amazon.com, Inc.",
"asn": "14618",
"country": "US",
"detected_urls": [],
"resolutions": [
{
"hostname": "bumbleride.com",
"last_resolved": "2016-09-15 00:00:00"
},
{
"hostname": "chilitechnology.com",
"last_resolved": "2016-09-16 00:00:00"
}
],
"response_code": 1,
"verbose_msg": "IP address in dataset"
},
"match": true
}
}
]
}
'''
parsed_json = json.loads(raw)
wanted = ['as_owner', 'asn', 'country', 'detected_urls', 'resolutions']
for item in parsed_json["results"]:
details = item['data']['details']
for key in wanted:
print(key, ':', json.dumps(details[key], indent=4))
# Put a blank line at the end of the details for each item
print()
output
as_owner : "Amazon.com, Inc."
asn : "14618"
country : "US"
detected_urls : []
resolutions : [
{
"hostname": "bumbleride.com",
"last_resolved": "2016-09-15 00:00:00"
},
{
"hostname": "chilitechnology.com",
"last_resolved": "2016-09-16 00:00:00"
}
]
BTW, when you fetch JSON data using requests there's no need to use json.loads: you can access the converted JSON using the .json method of the returned request object instead of using its .text attribute.
Here's a more robust version of the main loop of the above code. It simply ignores any missing keys. I didn't post this code earlier because the extra if tests make it slightly less efficient, and I didn't know that keys could be missing.
for item in parsed_json["results"]:
if not 'data' in item:
continue
data = item['data']
if not 'details' in data:
continue
details = data['details']
for key in wanted:
if key in details:
print(key, ':', json.dumps(details[key], indent=4))
# Put a blank line at the end of the details for each item
print()

Why is the json returned by the Advanced REST Client different than that returned by the Requests module in Python?

ARC:
https://chrome.google.com/webstore/detail/advanced-rest-client/hgmloofddffdnphfgcellkdfbfbjeloo?hl=en-US
I saved the returned json in a .json file and transformed it into a pandas dataframe using:
temp_json = pd.read_json('TempJson.json', orient='columns')
This works great.
But then I used the requests module in Python 2.7.13, specifically:
myResponse = requests.post(url, json= payload, headers = headers)
jData = json.loads(myResponse.content)
And 1) the json structure is much different than temp_json and 2) it completely wrecks my code. Any idea why?
Snippet from temp_json:
{
"expand": "schema,names",
"startAt": 0,
"maxResults": 250,
"total": 3,
"issues": [
{
"expand": "operations,editmeta,changelog,transitions,renderedFields",
"id": "1954523",
"key": "SPGC-14075",
"fields": {"summary": "QA: Build concentration support into CDC automation",
"issuetype": {
"self": "https://itec-jira.fmr.com/rest/api/2/issuetype/20",
"id": "20",
"description": "Default sub-task",
"iconUrl": "https://itec-
jira.fmr.com/images/icons/issuetypes/subtask_alternate.png",
"name": "Sub task",
"subtask": true
Sample from python json:
{
"issues": [
{
"key": "SPGC-25646",
"fields": {
"status": {
"statusCategory": {
"name": "To Do",
"self": "https://itec-jira.fmr.com/rest/api/2/statuscategory/2",
"id": 2,
"key": "new",
"colorName": "blue-gray"
},.....

json.loads will create a python dict which is hashed so the contents will be in scrambled order. Check that json.loads returns the same dict for both the request and the tempfile. If they are different then the data is different. You can use the pretty print library to help you debug deep nested json.

Python for loop posting json to url?

The following code works but only uploads the one result at the index as per the below, see ['result'][2]["text"] - which uploads the 'text' result at index 2 perfectly:
with open("test.json") as json_file:
data = json.load(json_file)
connection.request('POST', '/1/batch', json.dumps({
"requests": [{
"method": "POST",
"path": "/1/classes/test",
"body": {
"playerName": data['results'][2]["text"],
"Url": data['results'][2]["Url"],
"Amount": data['results'][2]["Amount"]
}
}]
})
How can I loop through all of the json results here ([0] to eof) rather than having to change ['results'][0]["text"], press enter, ['results'][1]["text"] press enter, ['results'][2]["text"] press enter etc?

A simple for will solve it:
with open("test.json") as json_file:
data = json.load(json_file)
for entry in data['results']:
connection.request('POST', '/1/batch', json.dumps({
"requests": [{
"method": "POST",
"path": "/1/classes/test",
"body": {
"playerName": entry["text"],
"Url": entry["Url"],
"Amount": entry["Amount"]
}
}]
})
If you really need the index number, you can use the enumerate() function:
for index, entry in enumerate(data["results"]):
# ...
Edit:
If by '/1/batch' you mean '/2/batch' (eg. typo), then you will need to use enumerate(). Eg:
with open("test.json") as json_file:
data = json.load(json_file)
for index, entry in enumerate(data['results']):
connection.request('POST', '/%d/batch' % (index), json.dumps({
"requests": [{
"method": "POST",
"path": "/%d/classes/test" % (index),
"body": {
"playerName": entry["text"],
"Url": entry["Url"],
"Amount": entry["Amount"]
}
}]
})