We Keep Coding

How to print out the exact field/string of the JSON output?

I'm trying to filter all the result that I got in the GET Request.
The Output that I want is just to get the summary: , key: and self:.
But I'm getting a lot of Json data.
I've tried googling on how to do this and I'm going to nowhere.
Here is my code:
The commented lines are the codes that I have tried.
import requests
import json
import re
import sys
url ="--------"
auth='i.g--t----------', 'X4------'
r = requests.get(url, auth=(auth))
data = r.json()
#print( json.dumps(data, indent=2) )
#res1 = " ".join(re.split("summary", data))
#print ("first string result: ", str(res1))
#json_str = json.dumps(data)
#resp = json.loads(json_str)
#print (resp['id'])
#resp_dict = json.loads(resp_str)
#resp_dict.get('name')
#print('dasdasd', json_str["summary"])
Example of the Get Api Output that I'm getting using this code. print( json.dumps(data, indent=2) )
{
"id": "65621",
"self": "https://bboxxltd.atlassian.net/rest/api/2/issue/65621",
"key": "CMS-5901",
"fields": {
"summary": "new starter: Edoardo Bologna",
"customfield_10700": [
{
"id": "2",
"name": "BBOXX Rwanda HQ",
"_links": {
"self": "https://bboxxltd.atlassian.net/rest/servicedeskapi/organization/2"
}
}
},
"inwardIssue": {
"id": "65862",
"key": "BMT-2890",
"self": "https://bboxxltd.atlassian.net/rest/api/2/issue/65862",
"fields": {
"summary": "ERP Databases access with Read Only",
"status": {
"self": "https://bboxxltd.atlassian.net/rest/api/2/status/10000",
"description": "",
"iconUrl": "https://bboxxltd.atlassian.net/",
"name": "To Do",
"id": "10000",
"statusCategory": {
"self": "https://bboxxltd.atlassian.net/rest/api/2/statuscategory/2",
"id": 2,
"key": "new",
"colorName": "blue-gray",
"name": "To Do"
}
},
"priority": {
"self": "https://bboxxltd.atlassian.net/rest/api/2/priority/4",
"iconUrl": "https://bboxxltd.atlassian.net/images/icons/priorities/low.svg",
"name": "Low",
My error is:
Traceback (most recent call last):
File "c:/Users/IanJayloG/Desktop/Python Files/Ex_Files_Learning_Python/Exercise Files/Test/Untitled-1.py", line 17, in <module>
print('dasdasd', data["summary"])
KeyError: 'summary'
PS C:\Users\IanJayloG\Desktop\Python Files\Ex_Files_Learning_Python\Exercise Files> & C:/Users/IanJayloG/AppData/Local/Programs/Python/Python37-32/python.exe "c:/Users/IanJayloG/Desktop/Python Files/Ex_Files_Learning_Python/Exercise Files/Test/Untitled-1.py"
Traceback (most recent call last):
File "c:/Users/IanJayloG/Desktop/Python Files/Ex_Files_Learning_Python/Exercise Files/Test/Untitled-1.py", line 17, in <module>
print('dasdasd', json_str["summary"])
TypeError: string indices must be integers

The problem about your error message
print('dasdasd', json_str["summary"])
TypeError: string indices must be integers
is that you try to access the named field summary on a string (variable json_str), which does not work because strings don't have fields to access by name. If you use the indexing [] operator on a string, you can only provide integers or ranges to extract single characters or sequences from that string. This is obviously not what you're intending.
The keys self and key are on top level of your JSON document, whereas summary is under fields. This should do it, without any extra transformation applied:
import requests
r = requests.get(url, auth=(auth))
data = r.json()
data_summary = data['fields']['summary']
data_self = data['self']
data_key = data['key']

Python - How to print out all json into a list

So im trying to make a script that makes requests on a website and after that it compares if there is new item added into the requests meaning if there is one - alert me, if there isnt any. requests the same page in few minutes.
So what im haivng issues now with is to actually print out to a list where it tells me:
TypeError: string indices must be integers
but whenever I print the element it just gives me a print of "id" which I can't understand the reason -
The code look:
resp = s.get(url)
list = []
i = 0
for element in resp.json()['threads']['product']:
print(element) #<--- this gives me a print of "id"
list.append(element['fullTitle'])
#ERROR - list.append(element['fullTitle'])
#TypeError: string indices must be integers
i+=1
print(list)
while True:
try:
new_list = []
url = 'https://hellowebsite.com'
resp = s.get(url)
for element in resp.json()['threads']['product']:
new_list.append(element['fullTitle'])
print(new_list)
for link in new_list:
if link not in list:
print('New item found! - ' + link)
else:
print("No new itemfound!")
time.sleep(10)
except:
randomtime = random.randint(6, 12)
Json code: (its not fully completed but it looks like this)
{
"country": "GB",
"locale": "Thrill",
"threads": [{
"id": "f44291998750ce50301f36607c7996b5db5a9c3b",
"interestId": "57d4d929-694c-4fdd-9277-f0df465cd7c7",
"createdDate": "2018-06-26T08:05:24.000",
"product": {
"id": "b0bfdaa1-140c-54de-b040-66854ec62d1b",
"interestId": "a53256c1-983b-43d5-895c-c5b7c3eedc85",
"colorCode": "100",
"fullTitle": "Item Number 1",
}
},
{
"id": "0c132053034f79c08dd474d6371875fe421e8694",
"interestId": "ceb1c5f5-4ff2-43f4-8330-d537d1a4f5f9",
"createdDate": "2018-06-26T08:24:42.000",
"product": {
"id": "2b7830e3-2e36-54cd-a290-29c38493b290",
"interestId": "8aaeb398-91d6-45f1-b0d0-f4e8561e62fb",
"colorCode": "800",
"fullTitle": "Item Number 2",
}
},
{
"id": "985ff4b1bb969dd7a95ea589aff4d5f4710ab69e",
"interestId": "aa73be8b-793e-4d78-b512-e80e2a019599",
"createdDate": "2018-06-25T10:20:47.000",
"product": {
"colorCode": "001",
"fullTitle": "Item number 3",
}.....
What would be the suggestion to be able to print out all the fullTitle into a list and then continue to do the compare?
EDIT FULLTRACE
resp = s.get(url)
for element in resp.json()['threads']['product']['fullTitle']:
print(element)
ERROR
Process Process-1:
Traceback (most recent call last):
File "C:\Users\Username\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 249, in _bootstrap
self.run()
File "C:\Users\Usersname\AppData\Local\Programs\Python\Python36\lib\multiprocessing\process.py", line 93, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Test.py", line 84, in script
for element in resp.json()['threads']['product']['fullTitle']:
TypeError: list indices must be integers or slices, not str

You're iterating through the wrong thing. The list here is 'threads', each element of which has a 'product' dictionary which contains a 'fullTitle' element. So you iterate through that:
for thread in resp.json()['threads']
print(thread['product']['fullTitle'])

Unable to parse JSON file, keep getting ValueError: Extra Data

So, leading on from my prior issue [found here][1], I'm attempting to parse a JSON file that I've managed to download with #SiHa's help. The JSON is structured like so:
{"properties": [{"property": "name", "value": "A random company name"}, {"property": "companyId", "value": 123456789}]}{"properties": [{"property": "name", "value": "Another random company name"}, {"property": "companyId", "value": 31415999}]}{"properties": [{"property": "name", "value": "Yet another random company"}, {"property": "companyId", "value": 10101010}]}
I've been able to get this by slightly modifiying #SiHa's code:
def get_companies():
create_get_recent_companies_call = "https://api.hubapi.com/companies/v2/companies/?hapikey={hapikey}".format(hapikey=wta_hubspot_api_key)
headers = {'content-type': 'application/json'}
create_get_recent_companies_response = requests.get(create_get_recent_companies_call, headers=headers)
if create_get_recent_companies_response.status_code == 200:
while True:
for i in create_get_recent_companies_response.json()[u'companies']:
all_the_companies = { "properties": [
{ "property": "name", "value": i[u'properties'][u'name'][u'value'] },
{ "property": "companyId", "value": i[u'companyId'] }
]
}
with open("all_the_companies.json", "a") as myfile:
myfile.write(json.dumps(all_the_companies))
#print(companyProperties)
offset = create_get_recent_companies_response.json()[u'offset']
hasMore = create_get_recent_companies_response.json()[u'has-more']
if not hasMore:
break
else:
create_get_recent_companies_call = "https://api.hubapi.com/companies/v2/companies/?hapikey={hapikey}&offset={offset}".format(hapikey=wta_hubspot_api_key, offset=offset)
create_get_recent_companies_response = requests.get(create_get_recent_companies_call, headers=headers)
else:
print("Something went wrong, check the supplied field values.\n")
print(json.dumps(create_get_recent_companies_response.json(), sort_keys=True, indent=4))
So that was part one. Now I'm trying to use the code below to extract two things: 1) the name and 2) the companyId.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import sys
import os.path
import requests
import json
import csv
import glob2
import shutil
import time
import time as howLong
from time import sleep
from time import gmtime, strftime
# Local Testing Version
findCSV = glob2.glob('*contact*.csv')
theDate = time=strftime("%Y-%m-%d", gmtime())
theTime = time=strftime("%H:%M:%S", gmtime())
# Exception handling
try:
testData = findCSV[0]
except IndexError:
print ("\nSyncronisation attempted on {date} at {time}: There are no \"contact\" CSVs, please upload one and try again.\n").format(date=theDate, time=theTime)
print("====================================================================================================================\n")
sys.exit()
for theCSV in findCSV:
def process_companies():
with open('all_the_companies.json') as data_file:
data = json.load(data_file)
for i in data:
company_name = data[i][u'name']
#print(company_name)
if row[0].lower() == company_name.lower():
contact_company_id = data[i][u'companyId']
#print(contact_company_id)
return contact_company_id
else:
print("Something went wrong, check the \"get_companies()\" function.\n")
print(json.dumps(create_get_recent_companies_response.json(), sort_keys=True, indent=4))
if __name__ == "__main__":
start_time = howLong.time()
process_companies()
print("This operation took %s seconds.\n" % (howLong.time() - start_time))
sys.exit()
Unfortunately, its not working - I'm getting the following traceback:
Traceback (most recent call last):
File "wta_parse_json.py", line 62, in <module>
process_companies()
File "wta_parse_json.py", line 47, in process_companies
data = json.load(data_file)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 369, in decode
raise ValueError(errmsg("Extra data", s, end, len(s)))
ValueError: Extra data: line 1 column 130 - line 1 column 1455831 (char 129 - 1455830)
I've made sure that i'm using json.dumps not json.dump to open the file, but still its not working. :(
I've now given up on JSON, and am trying to export a simple CSV with the code below:
def get_companies():
create_get_recent_companies_call = "https://api.hubapi.com/companies/v2/companies/?hapikey={hapikey}".format(hapikey=wta_hubspot_api_key)
headers = {'content-type': 'application/json'}
create_get_recent_companies_response = requests.get(create_get_recent_companies_call, headers=headers)
if create_get_recent_companies_response.status_code == 200:
while True:
for i in create_get_recent_companies_response.json()[u'companies']:
all_the_companies = "{name},{id}\n".format(name=i[u'properties'][u'name'][u'value'], id=i[u'companyId'])
all_the_companies.encode('utf-8')
with open("all_the_companies.csv", "a") as myfile:
myfile.write(all_the_companies)
#print(companyProperties)
offset = create_get_recent_companies_response.json()[u'offset']
hasMore = create_get_recent_companies_response.json()[u'has-more']
if not hasMore:
break
else:
create_get_recent_companies_call = "https://api.hubapi.com/companies/v2/companies/?hapikey={hapikey}&offset={offset}".format(hapikey=wta_hubspot_api_key, offset=offset)
create_get_recent_companies_response = requests.get(create_get_recent_companies_call, headers=headers)
[1]: http://stackoverflow.com/questions/36148346/unable-to-loop-through-paged-api-responses-with-python
But it looks like this isn't right either - even though i've read up on the formatting issues, and have added the .encode('utf-8') additions. I still end up getting the following traceback:
Traceback (most recent call last):
File "wta_get_companies.py", line 78, in <module>
get_companies()
File "wta_get_companies.py", line 57, in get_companies
all_the_companies = "{name},{id}\n".format(name=i[u'properties'][u'name'][u'value'], id=i[u'companyId'])
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 3: ordinal not in range(128)

The JSON data has three Objects one after the other; simplified:
{ .. }{ .. }{ .. }
That's not something that's supported by the JSON standard. How is Python supposed to parse that? Automatically wrap it in an array? Assign it to three different variables? Just use the first one?
You probably want to wrap it in an array, simplified:
[{ .. },{ .. },{ .. }]
Or full:
[{"properties": [{"property": "name", "value": "A random company name"}, {"property": "companyId", "value": 123456789}]},{"properties": [{"property": "name", "value": "Another random company name"}, {"property": "companyId", "value": 31415999}]},{"properties": [{"property": "name", "value": "Yet another random company"}, {"property": "companyId", "value": 10101010}]}]

How to extract parrticular value from nested json values.?

I am new to python.
I have small requirement (i.e) want to extract only one value from the JSON format.
Please do correct me if i am wrong.
JSON input is:
{
"meta": {
"limit": 1,
"next": "/api/v1/ips/?username=sic1&api_key=689db0740ed73c2bf6402a7de0fcf2d7b57111ca&limit=1&objects=&offset=1",
"offset": 0,
"previous": null,
"total_count": 56714
},
"objects": [
{
"_id": "556f4c81dcddec0c41463529",
"bucket_list": [],
"campaign": [
{
"analyst": "prabhu",
"confidence": "medium",
"date": "2015-06-03 14:50:41.440000",
"name": "Combine"
}
],
"created": "2015-06-03 14:50:41.436000",
"ip": "85.26.162.70",
"locations": [],
"modified": "2015-06-18 09:50:51.612000",
"objects": [],
"relationships": [
{
"analyst": "prabhu",
"date": "2015-06-18 09:50:51.369000",
"rel_confidence": "unknown",
"rel_reason": "N/A",
"relationship": "Related_To",
"relationship_date": "2015-06-18 09:50:51.369000",
"type": "Indicator",
"value": "556f4c81dcddec0c4146353a"
}
],
"releasability": [],
"schema_version": 3,
"screenshots": [],
"sectors": [],
"source": [
{
"instances": [
{
"analyst": "prabhu",
"date": "2015-06-03 14:50:41.438000",
"method": "trawl",
"reference": "http://www.openbl.org/lists/base_30days.txt"
}
],
"name": "www.openbl.org"
}
],
"status": "New",
"tickets": [],
"type": "Address - ipv4-addr"
}
]
}
The code i used for getting value only IP's from objects
import requests
from pprint import pprint
import json
url = 'http://127.0.0.1:8080/api/v1/ips/'
params = {'api_key':'xxxxxx','username': 'abcd'}
r = requests.get(url, params=params, verify=False)
parsed = json.loads(r)
print (parsed['objects']['ip'])
The error i am receiving is:
Traceback (most recent call last):
File "testapi.py", line 9, in <module>
parsed = json.loads(r)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
I just want to get IP's from that JSON input.
Thanks.

You are passing a requests object instead of a str object to json.loads(). You need to change
parsed = json.loads(r)
to
parsed = json.loads(r.text)
Also, parsed['objects'] is a list, you need to access its first element & then get the key ip:
>>> print(parsed['objects'][0]['ip'])

The problem is in this line: parsed = json.loads(r)
You're reciving the json response but insted of feeding json elements to json.loads you're instead feeding it <Response [200]>
>>> r = requests.get('http://www.google.com')
>>> r
<Response [200]>
>>> type(r)
<class 'requests.models.Response'>
(Look closely at the error message. Expected string or buffer Which means you're providing it something that is NOT string or buffer(an object in this case))
This is the reason why str(r) didn't work. Because it just converted <Response 200> to '<Response 200>' which obviously is not json.
change this line to parsed = json.loads(r.text).
>>> type(r.text)
<type 'unicode'>
and then parsed['objects'][0]['ip'] should give you the IP address :)
You can find more about the requests module here

How to parse complex json in python 2.7.5?

I trying to list the names of my puppet classes from a Puppet Enterprise 3.7 puppet master, using Puppet's REST API.
Here is my script:
#!/usr/bin/env python
import requests
import json
url='https://ppt-001.example.com:4433/classifier-api/v1/groups'
headers = {"Content-Type": "application/json"}
data={}
cacert='/etc/puppetlabs/puppet/ssl/certs/ca.pem'
key='/etc/puppetlabs/puppet/ssl/private_keys/ppt-001.example.com.pem'
cert='/etc/puppetlabs/puppet/ssl/certs/ppt-001.example.com.pem'
result = requests.get(url,
data=data, #no data needed for this request
headers=headers, #dict {"Content-Type":"application/json"}
cert=(cert,key), #key/cert pair
verify=cacert
)
print json.dumps( result.json(), sort_keys=True, indent=4, separators=(',', ': '))
for i in result.json:
print i
Here is the error message I get when I execute the script:
Traceback (most recent call last):
File "./add-group.py", line 42, in <module>
for i in result.json:
TypeError: 'instancemethod' object is not iterable
Here is a sample of the data I get back from the REST API:
[
{
"classes": {},
"environment": "production",
"environment_trumps": false,
"id": "00000000-0000-4000-8000-000000000000",
"name": "default",
"parent": "00000000-0000-4000-8000-000000000000",
"rule": [
"and",
[
"~",
"name",
".*"
]
],
"variables": {}
},
{
"classes": {
"puppet_enterprise": {
"certificate_authority_host": "ppt-001.example.com",
"console_host": "ppt-001.example.com",
"console_port": "443",
"database_host": "ppt-001.example.com",
"database_port": "5432",
"database_ssl": true,
"mcollective_middleware_hosts": [
"ppt-001.example.com"
],
"puppet_master_host": "ppt-001.example.com",
"puppetdb_database_name": "pe-puppetdb",
"puppetdb_database_user": "pe-puppetdb",
"puppetdb_host": "ppt-001.example.com",
"puppetdb_port": "8081"
}
},
"environment": "production",
"environment_trumps": false,
"id": "52c479fe-3278-4197-91ea-9127ba12474e",
"name": "PE Infrastructure",
"parent": "00000000-0000-4000-8000-000000000000",
"variables": {}
},
.
.
.
How should I go about access the name key and getting the values like default and PE Infrastructure?
I have read the other answers here on SO saying that one should use json.loads() and I have tried using parsed_json = json.loads(result.json()) but results in this error message:
Traceback (most recent call last):
File "./add-group.py", line 38, in <module>
parsed_json = json.loads(result.json())
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

print json.dumps( result.json(), sort_keys=True, indent=4, separators=(',', ': '))
the first parameter of json.dumps must be a string or buffer, as stated by the TypeError your getting (TypeError: expected string or buffer).
Your variable result is an instance of Response, and the method .json() will return a dictionary. Since you're passing the result of .json() to json.dumps(), you're getting an error. You could either just use result.json() which is already a dictionary corresponding to your response, or change your json.dumps line to print json.dumps( result.text, sort_keys=True, indent=4, separators=(',', ': ')) where result.text is your JSON result as a string/unicode.
After the change, to access something like the name attribute, you could do something like:
for item in r.json():
try:
print item['name']
expect KeyError:
print "There is no 'name' attribute"