I'm having a bit of an issue with parsing json with python using the json library.
Here is the format of the json I'm trying to parse:
{'entry':[
{
JSON Data 1
},
JSON Data 2
}
]}
And here is my Python:
for entry in response['entry'][0]:
video['video_url'] = entry['id']['$t']
video['published'] = entry['published']['$t']
I don't seem to be able to iterate over the two blocks of JSON with the above code, I only get the first block outputted for some reason.
Anybody have any ideas?? Thanks in advance.
If:
response = {'entry':[
{
JSON Data 1
},
{
JSON Data 2
}
]}
And:
response['entry'][0] == { JSON Data 1 }
Then:
for entry in response['entry']:
video['video_url'] = entry['id']['$t']
video['published'] = entry['published']['$t']
Or:
video = dict(zip(['video_url', 'published'], [entry['id']['$t'], entry['published']['$t']]) for entry in response['entry']
That list contains 2 separate dicts. Iterate over the list directly.
Related
How can I find the "property" and "appliances" key's data in Python?
{
"apiCache": "{\"VariantQuery{\\\"zpid\\\":2070457223,\\\"altId\\\":null}\":{\"property\":{\"zpid\":2070457223,\"streetAddress\":\"0 E 1990th Rd\",\"zipcode\":\"74759\",\"city\":\"Soper\",\"state\":\"OK\",\"APPS_IOS_GOOGLE_TRANSITION\":\"ON_SIGNED_URLS\",\"SXP_PHOTO_CAROUSEL\":\"CONTROL\",\"ADT_COOKIE_CONSENT\":\"ONETRUST\",\"SEOTEST__CUSTOM_REGION_FAQ_FOR_SALE_V1\":\"CONTROL\",\"ADS_FiOSAd\":\"ON\",\"HDP_PSL_PHOTOS\":\"ON\",\"HDP_ZO_NFS_UPSELL_ONSITE_MESSAGING\":\"CONTROL\",\"AndroidWowMapDotsV0\":\"ON_WOW_DOTS_WITH_HRS\",\"REV_Replace_PALS_CC_Form_With_CSI_Functionality\":\"CONTROL\",\"SRP_DESKTOP_ADS_VARIANTS\":\"VARIANT_B\",\"TR_ORGANIC_HIGHLIGHTS\":\"CONTROL\",\"VL_BDP_NYDOS\":\"ON\",\"HDP_FULL_SSR_NFS\":\"ON\",\"FOUNDATION_COLORS_TYPOGRAPHY\":\"CONTROL\",\"ADS_EPA_SHOW_OTHER_LEADS\":\"ON\",\"HDP_VT_HIGHER\":\"ON\",\"SEO__SXP_REMOVE_ANCHOR_TEXT\":\"CONTROL\",\"SEOTEST__RE_SRPTitleTags_CityStateHomesForRent\":\"CONTROL\",\"SavedSearchInstantSMS\":\"CONTROL\",\"REALTIME_PERS_SORT_PRE\":\"REALTIME_RECS\",\"NOTIFICATION_SERVICE\":\"CACHED\",\"VELOCITY_TimeIsMoney\":\"CONTROL\",\"MOBILE_ZO_VALUEPROP\":\"VALUEPROP_ON\",\"ACT_AutocompleteHW\":\"CONTROL\",\"ARCS_REGION_PHONE\":\"ON\",\"IOS_HDP_TOP_ACTIONS\":\"CONTROL_OFF\",\"ACQ_TOP_DISMISSLOGIC_Relaunch_2wksVS24hrs\":\"Variant_Relaunch_24hrs\",\"Traffic_Mobile_Menu_Button\":\"APP_STYLE\",\"ARCS_LISTEDBY\":\"CONTROL\",\"IOS_CURRENT_LOCATION\":\"CONTROL_ON\",\"HDP_NO_CARDS\":\"ON\",\"RR_TourSchedulingOnOffSwitch\":\"CONTROL\",\"SELLiveViewsUI\":\"CONTROL\",\"PERS_HOME_INSIGHTS_POSITION\":\"INSIGHTS_MID\",\"IOS_SHARE_SHEET\":\"ON_2\",\"SPT_PSL_PRICE_HISTORY\":\"ON\",\"KeystoneEventHDP\":\"SEND_EVENT\",\"HDP_HarrietZestimate\":\"TOOLS_MODULE\",\"IOS_HOMERECOMMENDATIONS_V1\":\"ON\",\"IOS_HOMERECOMMENDATIONS_V2\":\"CONTROL_OFF\",\"NOTIFICATION_ADMIN_PAGE_ONLY\":\"ADMIN_PAGE_ONLY\",\"SXP_ACT_REMOVE_SEARCHBOX_GLEAM\":\"NO_GLEAM\",\"appliances\":[\"Cooktop\",\"Electric Water Heater\",\"Microwave\",\"Oven\",\"Refrigerator\",\"Electric Range\"],\"ARCS_TOUR_MESSAGE\":\"CONTROL\",\"ARCS_TOUR_TIME\":\"TIME_SHOWN\",\"ARCS_TOUR_REWRITE\":\"ON\",\"ARCS_WOW_LISTING_AGENT_ATTRIBUTION\":\"ON\",\"ARCS_CONTACTABLE_LISTING_AGENT\":\"CONTROL\",\"ARCS_DESKTOP_PHONE\":\"CONTROL\",\"contactFormRenderData\":null}}",
"zpid": 2070457223
}
You can use the json library in python to read a json string into a dictionary, which you can then use to access the keys you want, like so:
import json
data_json = '{ "apiCache": "{"VariantQuery{\"zpid\":2070457223,\"altId\":null}":{"property":{"zpid":2070457223,"streetAddress":"0 E 1990th Rd","zipcode":"74759","city":"Soper","state":"OK","APPS_IOS_GOOGLE_TRANSITION":"ON_SIGNED_URLS","SXP_PHOTO_CAROUSEL":"CONTROL","ADT_COOKIE_CONSENT":"ONETRUST","SEOTEST__CUSTOM_REGION_FAQ_FOR_SALE_V1":"CONTROL","ADS_FiOSAd":"ON","HDP_PSL_PHOTOS":"ON","HDP_ZO_NFS_UPSELL_ONSITE_MESSAGING":"CONTROL","AndroidWowMapDotsV0":"ON_WOW_DOTS_WITH_HRS","REV_Replace_PALS_CC_Form_With_CSI_Functionality":"CONTROL","SRP_DESKTOP_ADS_VARIANTS":"VARIANT_B","TR_ORGANIC_HIGHLIGHTS":"CONTROL","VL_BDP_NYDOS":"ON","HDP_FULL_SSR_NFS":"ON","FOUNDATION_COLORS_TYPOGRAPHY":"CONTROL","ADS_EPA_SHOW_OTHER_LEADS":"ON","HDP_VT_HIGHER":"ON","SEO__SXP_REMOVE_ANCHOR_TEXT":"CONTROL","SEOTEST__RE_SRPTitleTags_CityStateHomesForRent":"CONTROL","SavedSearchInstantSMS":"CONTROL","REALTIME_PERS_SORT_PRE":"REALTIME_RECS","NOTIFICATION_SERVICE":"CACHED","VELOCITY_TimeIsMoney":"CONTROL","MOBILE_ZO_VALUEPROP":"VALUEPROP_ON","ACT_AutocompleteHW":"CONTROL","ARCS_REGION_PHONE":"ON","IOS_HDP_TOP_ACTIONS":"CONTROL_OFF","ACQ_TOP_DISMISSLOGIC_Relaunch_2wksVS24hrs":"Variant_Relaunch_24hrs","Traffic_Mobile_Menu_Button":"APP_STYLE","ARCS_LISTEDBY":"CONTROL","IOS_CURRENT_LOCATION":"CONTROL_ON","HDP_NO_CARDS":"ON","RR_TourSchedulingOnOffSwitch":"CONTROL","SELLiveViewsUI":"CONTROL","PERS_HOME_INSIGHTS_POSITION":"INSIGHTS_MID","IOS_SHARE_SHEET":"ON_2","SPT_PSL_PRICE_HISTORY":"ON","KeystoneEventHDP":"SEND_EVENT","HDP_HarrietZestimate":"TOOLS_MODULE","IOS_HOMERECOMMENDATIONS_V1":"ON","IOS_HOMERECOMMENDATIONS_V2":"CONTROL_OFF","NOTIFICATION_ADMIN_PAGE_ONLY":"ADMIN_PAGE_ONLY","SXP_ACT_REMOVE_SEARCHBOX_GLEAM":"NO_GLEAM","appliances":["Cooktop","Electric Water Heater","Microwave","Oven","Refrigerator","Electric Range"],"ARCS_TOUR_MESSAGE":"CONTROL","ARCS_TOUR_TIME":"TIME_SHOWN","ARCS_TOUR_REWRITE":"ON","ARCS_WOW_LISTING_AGENT_ATTRIBUTION":"ON","ARCS_CONTACTABLE_LISTING_AGENT":"CONTROL","ARCS_DESKTOP_PHONE":"CONTROL","contactFormRenderData":null}}", "zpid": 2070457223 }'
data_dict = json.loads(data_json)
appliances = data_dict['property']['appliances']
print(appliances)
Note: this code doesn't quite run as the json listed in the question is invalid.
I have the following data frame
lat long batchItems
0 64.427482 -161.457786 {"query": "?query=64.42748213233087,-161.45778...
1 31.271877 130.331371 {"query": "?query=31.271877,130.331371"}
which I need in the following format
{
"batchItems": [
{"query": "?query=64.42748213233087,-161.45778592219781"},
{"query": "?query=31.271877,130.331371"}
]
}
using the inbuilt to_json method in pandas gives me unwanted backslashes ("{") which get rejected by the API i'm trying to post this to. I don't need the lat and long fields.
Your batchItems is str type, so if you go to JSON, it's normal that quotes are escaped because they are not delimiting some string, they are inside the content itself.
You may change the type to dict using json.loads, then get the column as a list
import json
result = {"batchItems": df['batchItems'].apply(json.loads).tolist()}
print(result) # {'batchItems': [{'query': '?query=64,-161'}, {'query': '?query=31,130'}]}
df[['batchItems']].to_dict(orient='list') will return a dictionary in the required format
{
"batchItems": [
{"query": "?query=64.42748213233087,-161.45778592219781"},
{"query": "?query=31.271877,130.331371"}
]
}
You have to send this dictionary to the function that sends the data to the external API.
I have some JSON data like:
{
"status": "200",
"msg": "",
"data": {
"time": "1515580011",
"video_info": [
{
"announcement": "{\"announcement_id\":\"6\",\"name\":\"INS\\u8d26\\u53f7\",\"icon\":\"http:\\\/\\\/liveme.cms.ksmobile.net\\\/live\\\/announcement\\\/2017-08-18_19:44:54\\\/ins.png\",\"icon_new\":\"http:\\\/\\\/liveme.cms.ksmobile.net\\\/live\\\/announcement\\\/2017-10-20_22:24:38\\\/4.png\",\"videoid\":\"15154610218328614178\",\"content\":\"FOLLOW ME PLEASE\",\"x_coordinate\":\"0.22\",\"y_coordinate\":\"0.23\"}",
"announcement_shop": "",
etc.
How do I grab the content "FOLLOW ME PLEASE"? I tried using
replay_data = raw_replay_data['data']['video_info'][0]
announcement = replay_data['announcement']
But now announcement is a string representing more JSON data. I can't continue indexing announcement['content'] results in TypeError: string indices must be integers.
How can I get the desired string in the "right" way, i.e. respecting the actual structure of the data?
In a single line -
>>> json.loads(data['data']['video_info'][0]['announcement'])['content']
'FOLLOW ME PLEASE'
To help you understand how to access data (so you don't have to ask again), you'll need to stare at your data.
First, let's lay out your data nicely. You can either use json.dumps(data, indent=4), or you can use an online tool like JSONLint.com.
{
'data': {
'time': '1515580011',
'video_info': [{
'announcement': ( # ***
"""{
"announcement_id": "6",
"name": "INS\\u8d26\\u53f7",
"icon": "http:\\\\/\\\\/liveme.cms.ksmobile.net\\\\/live\\\\/announcement\\\\/2017-08-18_19:44:54\\\\/ins.png",
"icon_new": "http:\\\\/\\\\/liveme.cms.ksmobile.net\\\\/live\\\\/announcement\\\\/2017-10-20_22:24:38\\\\/4.png",
"videoid": "15154610218328614178",
"content": "FOLLOW ME PLEASE",
"x_coordinate": "0.22",
"y_coordinate": "0.23"
}"""),
'announcement_shop': ''
}]
},
'msg': '',
'status': '200'
}
*** Note that the data in the announcement key is actually more json data, which I've laid out on separate lines.
First, find out where your data resides. You're looking for the data in the content key, which is accessed by the announcement key, which is part of a dictionary inside a list of dicts, which can be accessed by the video_info key, which is in turn accessed by data.
So, in summary, "descend" the ladder that is "data" using the following "rungs" -
data, a dictionary
video_info, a list of dicts
announcement, a dict in the first dict of the list of dicts
content residing as part of json data.
First,
i = data['data']
Next,
j = i['video_info']
Next,
k = j[0] # since this is a list
If you only want the first element, this suffices. Otherwise, you'd need to iterate:
for k in j:
...
Next,
l = k['announcement']
Now, l is JSON data. Load it -
import json
m = json.loads(l)
Lastly,
content = m['content']
print(content)
'FOLLOW ME PLEASE'
This should hopefully serve as a guide should you have future queries of this nature.
You have nested JSON data; the string associated with the 'annoucement' key is itself another, separate, embedded JSON document.
You'll have to decode that string first:
import json
replay_data = raw_replay_data['data']['video_info'][0]
announcement = json.loads(replay_data['announcement'])
print(announcement['content'])
then handle the resulting dictionary from there.
The content of "announcement" is another JSON string. Decode it and then access its contents as you were doing with the outer objects.
I have developed a post request with requests module and I retrieve a JSON, this is code
headers_ = {'Authorization': acc_token}
params_ = (('start_time', '2016-08-01T00:00:00+00:00'),('limit', '2'))
r = requests.get('https://server.com/API/v1', headers=headers_, params=params_)
data_ = r.json()
json_result = (json.dumps(data_, indent=4, sort_keys=False))
and this is json result
{
"loc":[
{
"origin":{
"lat":39.72417,
"lng":-104.99984,
"eta_seconds":null,
"address":""
},
"destination":{
"lat":39.77446,
"lng":-104.9379,
"eta_seconds":null,
"address":null
}
},
{
"origin":{
"lat":39.77481,
"lng":-104.93618,
"eta_seconds":null,
"address":"10 Albion Street"
},
"destination":{
"lat":39.6984,
"lng":-104.9652,
"eta_seconds":null,
"address":null
}
}
]
}
now I'm trying to append into array the lat and lng value of each json items array and I have try with this code
for i in json_result:
print(i['loc']['origin']['lat'])
dict.append(i['loc']['origin']['lat'])
but I have this error
TypeError: string indices must be integers, not str
How can I retrieve the value lat and lng for each element of the array and append into dict?
Thanks
json_result is a string, because it is the result of json.dumps().
You don't need that at all. Remove the whole line, and operate directly on data_.
(As a style point, please stop adding those unnecessary trailing underscores.)
So the problem is this for i in json_result, which will return you each of the keys in your json. You can simply do:
for entry in json_result.get('loc'):
do stuff here....
Just a different solution , if you are still getting error then you can also try regex to fetch only lat , lng part :
import re
convert=str(json_result)
pattern=r"(l([ng]|[at]).'):\s[-]?\d{0,15}.\d{0,15}"
match=re.finditer(pattern,convert)
print([i.group() for i in match])
I'm trying to work with json file stored locally. That is formatted as below:
{
"all":{
"variables":{
"items":{
"item1":{
"one":{
"size":"1"
},
"two":{
"size":"2"
}
}
}
}
}
}
I'm trying to get the value of the size key using the following code.
with open('path/to/file.json','r') as file:
data = json.load(file)
itemParse(data["all"]["variables"]["items"]["item1"])
def itemParse(data):
for i in data:
# also tried for i in data.iterkeys():
# data has type dict while i has type unicode
print i.get('size')
# also tried print i['size']
got different errors and nothing seems to work. any suggestions?
also, tried using json.loads got error expect string or buffer
When you iterate over data you are getting the key only. There is 2 ways to solve it.
def itemParse(data):
for i, j in data.iteritems():
print j.get('size')
or
def itemParse(data):
for i in data:
print data[i].get('size')
First, use json.loads().
data = json.loads(open('path/to/file.json','r').read())
Second, your for loop should be changed to this
for k,v in data.iteritems():
print data[k]['size']
Regarding the error expect string or buffer, do you have permissions to read the json file?