I have the following data frame
lat long batchItems
0 64.427482 -161.457786 {"query": "?query=64.42748213233087,-161.45778...
1 31.271877 130.331371 {"query": "?query=31.271877,130.331371"}
which I need in the following format
{
"batchItems": [
{"query": "?query=64.42748213233087,-161.45778592219781"},
{"query": "?query=31.271877,130.331371"}
]
}
using the inbuilt to_json method in pandas gives me unwanted backslashes ("{") which get rejected by the API i'm trying to post this to. I don't need the lat and long fields.
Your batchItems is str type, so if you go to JSON, it's normal that quotes are escaped because they are not delimiting some string, they are inside the content itself.
You may change the type to dict using json.loads, then get the column as a list
import json
result = {"batchItems": df['batchItems'].apply(json.loads).tolist()}
print(result) # {'batchItems': [{'query': '?query=64,-161'}, {'query': '?query=31,130'}]}
df[['batchItems']].to_dict(orient='list') will return a dictionary in the required format
{
"batchItems": [
{"query": "?query=64.42748213233087,-161.45778592219781"},
{"query": "?query=31.271877,130.331371"}
]
}
You have to send this dictionary to the function that sends the data to the external API.
Related
I have some JSON data like:
{
"status": "200",
"msg": "",
"data": {
"time": "1515580011",
"video_info": [
{
"announcement": "{\"announcement_id\":\"6\",\"name\":\"INS\\u8d26\\u53f7\",\"icon\":\"http:\\\/\\\/liveme.cms.ksmobile.net\\\/live\\\/announcement\\\/2017-08-18_19:44:54\\\/ins.png\",\"icon_new\":\"http:\\\/\\\/liveme.cms.ksmobile.net\\\/live\\\/announcement\\\/2017-10-20_22:24:38\\\/4.png\",\"videoid\":\"15154610218328614178\",\"content\":\"FOLLOW ME PLEASE\",\"x_coordinate\":\"0.22\",\"y_coordinate\":\"0.23\"}",
"announcement_shop": "",
etc.
How do I grab the content "FOLLOW ME PLEASE"? I tried using
replay_data = raw_replay_data['data']['video_info'][0]
announcement = replay_data['announcement']
But now announcement is a string representing more JSON data. I can't continue indexing announcement['content'] results in TypeError: string indices must be integers.
How can I get the desired string in the "right" way, i.e. respecting the actual structure of the data?
In a single line -
>>> json.loads(data['data']['video_info'][0]['announcement'])['content']
'FOLLOW ME PLEASE'
To help you understand how to access data (so you don't have to ask again), you'll need to stare at your data.
First, let's lay out your data nicely. You can either use json.dumps(data, indent=4), or you can use an online tool like JSONLint.com.
{
'data': {
'time': '1515580011',
'video_info': [{
'announcement': ( # ***
"""{
"announcement_id": "6",
"name": "INS\\u8d26\\u53f7",
"icon": "http:\\\\/\\\\/liveme.cms.ksmobile.net\\\\/live\\\\/announcement\\\\/2017-08-18_19:44:54\\\\/ins.png",
"icon_new": "http:\\\\/\\\\/liveme.cms.ksmobile.net\\\\/live\\\\/announcement\\\\/2017-10-20_22:24:38\\\\/4.png",
"videoid": "15154610218328614178",
"content": "FOLLOW ME PLEASE",
"x_coordinate": "0.22",
"y_coordinate": "0.23"
}"""),
'announcement_shop': ''
}]
},
'msg': '',
'status': '200'
}
*** Note that the data in the announcement key is actually more json data, which I've laid out on separate lines.
First, find out where your data resides. You're looking for the data in the content key, which is accessed by the announcement key, which is part of a dictionary inside a list of dicts, which can be accessed by the video_info key, which is in turn accessed by data.
So, in summary, "descend" the ladder that is "data" using the following "rungs" -
data, a dictionary
video_info, a list of dicts
announcement, a dict in the first dict of the list of dicts
content residing as part of json data.
First,
i = data['data']
Next,
j = i['video_info']
Next,
k = j[0] # since this is a list
If you only want the first element, this suffices. Otherwise, you'd need to iterate:
for k in j:
...
Next,
l = k['announcement']
Now, l is JSON data. Load it -
import json
m = json.loads(l)
Lastly,
content = m['content']
print(content)
'FOLLOW ME PLEASE'
This should hopefully serve as a guide should you have future queries of this nature.
You have nested JSON data; the string associated with the 'annoucement' key is itself another, separate, embedded JSON document.
You'll have to decode that string first:
import json
replay_data = raw_replay_data['data']['video_info'][0]
announcement = json.loads(replay_data['announcement'])
print(announcement['content'])
then handle the resulting dictionary from there.
The content of "announcement" is another JSON string. Decode it and then access its contents as you were doing with the outer objects.
From api i recive x-www-form-urlencoded data.
data = 'leads%5Bstatus%5D%5B0%5D%5Bid%5D=29078079'
when i try to convert it with urllib.parse.parse_qs or urllib.parse.unquote
i got a dict like
{
"leads[status][0][id]": [
"29078079"
]
}
how can i convert it to normal json or dict ?
like this
{
"leads": [
"id": "29078079"
]
}
The string parsed correctly. So the question is not so much how you can get a 'normal' dict, rather how you can alter the dict key:
dictionary = { "leads[status][0][id]": [ "29078079" ] }
dictionary['leads'] = dictionary.pop(list(dictionary)[0])
Alternatively you could create the dict manually:
dictionary = {data[:5]:[data.split('=')[-1]]}
I have some JSON data like:
{
"status": "200",
"msg": "",
"data": {
"time": "1515580011",
"video_info": [
{
"announcement": "{\"announcement_id\":\"6\",\"name\":\"INS\\u8d26\\u53f7\",\"icon\":\"http:\\\/\\\/liveme.cms.ksmobile.net\\\/live\\\/announcement\\\/2017-08-18_19:44:54\\\/ins.png\",\"icon_new\":\"http:\\\/\\\/liveme.cms.ksmobile.net\\\/live\\\/announcement\\\/2017-10-20_22:24:38\\\/4.png\",\"videoid\":\"15154610218328614178\",\"content\":\"FOLLOW ME PLEASE\",\"x_coordinate\":\"0.22\",\"y_coordinate\":\"0.23\"}",
"announcement_shop": "",
etc.
How do I grab the content "FOLLOW ME PLEASE"? I tried using
replay_data = raw_replay_data['data']['video_info'][0]
announcement = replay_data['announcement']
But now announcement is a string representing more JSON data. I can't continue indexing announcement['content'] results in TypeError: string indices must be integers.
How can I get the desired string in the "right" way, i.e. respecting the actual structure of the data?
In a single line -
>>> json.loads(data['data']['video_info'][0]['announcement'])['content']
'FOLLOW ME PLEASE'
To help you understand how to access data (so you don't have to ask again), you'll need to stare at your data.
First, let's lay out your data nicely. You can either use json.dumps(data, indent=4), or you can use an online tool like JSONLint.com.
{
'data': {
'time': '1515580011',
'video_info': [{
'announcement': ( # ***
"""{
"announcement_id": "6",
"name": "INS\\u8d26\\u53f7",
"icon": "http:\\\\/\\\\/liveme.cms.ksmobile.net\\\\/live\\\\/announcement\\\\/2017-08-18_19:44:54\\\\/ins.png",
"icon_new": "http:\\\\/\\\\/liveme.cms.ksmobile.net\\\\/live\\\\/announcement\\\\/2017-10-20_22:24:38\\\\/4.png",
"videoid": "15154610218328614178",
"content": "FOLLOW ME PLEASE",
"x_coordinate": "0.22",
"y_coordinate": "0.23"
}"""),
'announcement_shop': ''
}]
},
'msg': '',
'status': '200'
}
*** Note that the data in the announcement key is actually more json data, which I've laid out on separate lines.
First, find out where your data resides. You're looking for the data in the content key, which is accessed by the announcement key, which is part of a dictionary inside a list of dicts, which can be accessed by the video_info key, which is in turn accessed by data.
So, in summary, "descend" the ladder that is "data" using the following "rungs" -
data, a dictionary
video_info, a list of dicts
announcement, a dict in the first dict of the list of dicts
content residing as part of json data.
First,
i = data['data']
Next,
j = i['video_info']
Next,
k = j[0] # since this is a list
If you only want the first element, this suffices. Otherwise, you'd need to iterate:
for k in j:
...
Next,
l = k['announcement']
Now, l is JSON data. Load it -
import json
m = json.loads(l)
Lastly,
content = m['content']
print(content)
'FOLLOW ME PLEASE'
This should hopefully serve as a guide should you have future queries of this nature.
You have nested JSON data; the string associated with the 'annoucement' key is itself another, separate, embedded JSON document.
You'll have to decode that string first:
import json
replay_data = raw_replay_data['data']['video_info'][0]
announcement = json.loads(replay_data['announcement'])
print(announcement['content'])
then handle the resulting dictionary from there.
The content of "announcement" is another JSON string. Decode it and then access its contents as you were doing with the outer objects.
I have developed a post request with requests module and I retrieve a JSON, this is code
headers_ = {'Authorization': acc_token}
params_ = (('start_time', '2016-08-01T00:00:00+00:00'),('limit', '2'))
r = requests.get('https://server.com/API/v1', headers=headers_, params=params_)
data_ = r.json()
json_result = (json.dumps(data_, indent=4, sort_keys=False))
and this is json result
{
"loc":[
{
"origin":{
"lat":39.72417,
"lng":-104.99984,
"eta_seconds":null,
"address":""
},
"destination":{
"lat":39.77446,
"lng":-104.9379,
"eta_seconds":null,
"address":null
}
},
{
"origin":{
"lat":39.77481,
"lng":-104.93618,
"eta_seconds":null,
"address":"10 Albion Street"
},
"destination":{
"lat":39.6984,
"lng":-104.9652,
"eta_seconds":null,
"address":null
}
}
]
}
now I'm trying to append into array the lat and lng value of each json items array and I have try with this code
for i in json_result:
print(i['loc']['origin']['lat'])
dict.append(i['loc']['origin']['lat'])
but I have this error
TypeError: string indices must be integers, not str
How can I retrieve the value lat and lng for each element of the array and append into dict?
Thanks
json_result is a string, because it is the result of json.dumps().
You don't need that at all. Remove the whole line, and operate directly on data_.
(As a style point, please stop adding those unnecessary trailing underscores.)
So the problem is this for i in json_result, which will return you each of the keys in your json. You can simply do:
for entry in json_result.get('loc'):
do stuff here....
Just a different solution , if you are still getting error then you can also try regex to fetch only lat , lng part :
import re
convert=str(json_result)
pattern=r"(l([ng]|[at]).'):\s[-]?\d{0,15}.\d{0,15}"
match=re.finditer(pattern,convert)
print([i.group() for i in match])
I'm having a bit of an issue with parsing json with python using the json library.
Here is the format of the json I'm trying to parse:
{'entry':[
{
JSON Data 1
},
JSON Data 2
}
]}
And here is my Python:
for entry in response['entry'][0]:
video['video_url'] = entry['id']['$t']
video['published'] = entry['published']['$t']
I don't seem to be able to iterate over the two blocks of JSON with the above code, I only get the first block outputted for some reason.
Anybody have any ideas?? Thanks in advance.
If:
response = {'entry':[
{
JSON Data 1
},
{
JSON Data 2
}
]}
And:
response['entry'][0] == { JSON Data 1 }
Then:
for entry in response['entry']:
video['video_url'] = entry['id']['$t']
video['published'] = entry['published']['$t']
Or:
video = dict(zip(['video_url', 'published'], [entry['id']['$t'], entry['published']['$t']]) for entry in response['entry']
That list contains 2 separate dicts. Iterate over the list directly.