We Keep Coding

How to filter out data from a print out message

I have this one problem, where I print out a message response from a website(JSON response), and the response I get is this.
Here is my model with fake data:
{"token": "MTAxOTAwNjM4NjEyMzg0OTkwMQ.8hkyLV.n0ir2UA4qFE5pXen9YnPtFzgn4xP8tHmVmmkrl", "user_settings": {"locale": "en-US", "theme": "dark"}, "user_id": "101900638614857883"}
And, if I only want the value of "token" data which are this (MTAxOTAwNjM4NjEyMzg0OTkwMQ.8hkyLV.n0ir2UA4qFE5pXen9YnPtFzgn4xP8tHmVmmkrl) and I want to store it into a txt file, is there any good way to do it?
Thank you, guys!
I tried print(r.text('token')) but it did not work, since it only works on printing the category of the data's (like : Category : {"token" : 'daefafa', "user-id" : 'er121231231', more})

In python, JSON is treated as a dictionary.
To filter it use dictionary comprehension
tokenData = {key: val for key,val in data_json.items() if key == 'token'}
Full Code Snippet :
from urllib.request import urlopen
import json
url = "enter-your-url"
response = urlopen(url)
data_json = json.loads(response.read())
print(type(data_json)) # <class 'dict'>
#use dict comprehension
jsonToken = {key: val for key,val in data_json.items() if key == 'result'}
strToken = json.dumps(jsonToken)
# Only string json can be written to files
with open('data.txt','w') as file:
file.write(strToken)
file.close()

You need to parse the JSON into a dictionary using json.loads(). Like this:
import json
# ...
# request-getting code
# ...
data = json.loads(r.text)
print(data['token'])

Reading JSON data in Python using Pagination, max records 100

I am trying to extract data from a REST API using python and put it into one neat JSON file, and having difficulty. The date is rather lengthy, with a total of nearly 4,000 records, but the max record allowed by the API is 100.
I've tried using some other examples to get through the code, and so far this is what I'm using (censoring the API URL and auth key, for the sake of confidentiality):
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while new_results:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("results", [])
vendors.extend(centiblock)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
For the life of me, I cannot figure out why it isn't working. The output keeps coming out as just:
[
"records"
]
If I play around with the print statement at the end, I can get it to print centiblock (so named for being a block of 100 records at a time) just fine - it gives me 100 records in un-formated text. However, if I try printing vendors at the end, the output is:
['records']
...which leads me to guess that somehow, the vendors array is not getting filled with the data. I suspect that I need to modify the get request where I define new_results, but I'm not sure how.
For reference, this is a censored look at how the json data begins, when I format and print out one centiblock:
{
"records": [
{
"id": "XXX",
"createdTime": "2018-10-15T19:23:59.000Z",
"fields": {
"Vendor Name": "XXX",
"Main Phone": "XXX",
"Street": "XXX",
Can anyone see where I'm going wrong?
Thanks in advance!

When you are extending vendors with centiblock, your are giving a dict to the extend function. extend is expecting an Iterable, so that works, but when you iterate over a python dict, you only iterate over the keys of the dict. In this case, ['records'].
Note as well, that your loop condition becomes False after the first iteration, because centiblock.get("results", []) returns [], since "results" is not a key of the output of the API. and [] has a truthiness value of False.
Hence to correct those errors you need to get the correct field from the API into new_results, and extend vendors with new_results, which is itself an array. Note that on the last iteration, new_results will be the empty list, which means vendors won't be extended with any null value, and will contain exactly what you need:
This should look like:
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while len(new_results) > 0:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("records", [])
vendors.extend(new_results)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
Note that I replaced the while new_results with a while len(new_results)>0 which is equivalent in this case, but more readable, and better practice in general.

Update dictionary value for a particular key with value from iterator of outer loop

I have a list of lists in this format:
nrows = ["['Shock pain', 'attack sharp pain']",
"['bruises', 'lump']",
"['fever', 'cold', 'Anxiety']",
"['neck pain', 'headache']"]
I want to call an API by passing everytime 1 list at a time from the nrows list. Each list should be passed 1 by 1 to the data dict with key sList and the response should be saved.
The API function is as follows:
newres = []
for i in nrows:
url = "https://testabcd.io"
data= {
"sList": i,
"kList" : ["age"],
}
headers = {
'Content-Type': 'application/json',
'Auth': '0644427814339900'
}
response = requests.request("GET", url, headers=headers, json=data)
newres.append(response.text)
print(response.text)
print(newres)
Inside the data dict, in sList, at each iteration, I want to pass 1 sublist at a time and get the response appended in a list.
The current code has all the response same as I think I am unable to iterate and change the value of the data dict with key sList which is what is expected.
The csv looks like this:
['Shock pain', 'attack sharp pain']
['bruises', 'lump']
['fever', 'cold', 'Anxiety']

Assuming the file you work with, really looks like this:
['Shock pain', 'attack sharp pain']
['bruises', 'lump']
['fever', 'cold', 'Anxiety']
you can use ast.literal_eval in order to convert the line from the file into list
import requests
from ast import literal_eval
newres = []
url = "https://testabcd.io"
headers = {'Content-Type':'application/json',
'Auth':'0644427814339900'}
with open("yourfile") as f:
for line in f:
data= {"sList": literal_eval(line),
"kList" : ["age"]}
response = requests.get(url, headers=headers, json=data)
newres.append(response.text)
print(response.text)
print(newres)
Note:
This is not tested.
Given that your data comes from non-standard file, where they were exported in bad manner from a dataframe, created unknown how - I would suggest that you seriously reconsider all your workflow and how you manipulate your data.

Extract JSON attributes using Python and filter out some data

JSON File: http://media1.clubpenguin.com/play/en/web_service/game_configs/paper_items.json
I'm using Python 2.
I am trying to extract all the 'paper_item_id' 's from the JSON file (specified above), using a loop and storing the 'paper_item_id' in a 'item_id' variable, so this variable will change each time the loop iterates to the 'paper_item_id', but also I want to have an if statement which checks if the 'paper_item_id' in the JSON file 'is_bait' is 'true' if it is true the the 'item_id' variable will not store the 'paper_item_id' which has an 'is_bait' of true and go on to the next one.
Step 1) Get JSON Data.
Step 2) Filter out 'paper_item_id' 's with the 'is_bait' to true.
Step 3) Run a loop which assigns a 'item_id' variable to the 'paper_item_id' received.
Step 4) The loop should run so all filtered 'paper_item_id' (item_id) has been passed to 'myFunction'
Sample English Like Code:
foreach ('don't know what to have for the loop cond') {
item_id = 'paper_item_id'
if (item_id[is_bait]) == true {
code which will go to the end of the loop
}
else
{
myFunction(item_id)
}
I know this has a Javascript kind-of syntax but I want it in python.
What I have now:
import json
import urllib2
url = 'http://media1.clubpenguin.com/play/en/web_service/game_configs/paper_items.json'
result = json.loads(url)
request = urllib2.Request(url)
response = urllib2.urlopen(request)
json_obj = json.load(response)
What do I do know?

I've used requests.get, and also checked for a valid HTTP response.
Here's my sample code:
import json
import requests
def myFunction(item):
# do something here
print item['paper_item_id']
pass
json_obj = requests.get('http://media1.clubpenguin.com/play/en/web_service/game_configs/paper_items.json').json()
for item in json_obj:
if 'is_bait' in item and item['is_bait'] == "1":
# item['is_bait'] == "1", in case you ever get is_bait = "0" in your json response.
print item['paper_item_id']
continue
else:
myFunction(item)

Here is the translation of what you give as a pseudo code:
import json
import urllib2
url = 'http://media1.clubpenguin.com/play/en/web_service/game_configs/paper_items.json'
result = json.loads(url)
request = urllib2.Request(url)
response = urllib2.urlopen(request)
json_obj = json.load(response)
for item in json_obj:
if "is_bait" in item and item['is_bait']:
continue
else:
# do stuff
The continue can be skipped if you reverse the condition though.

json.load will read your data into a list of dictionaries, as appropriate. After that, you can filter on whatever it is your heart desires using standard python object manipulation:
import json
import urllib2
url = 'http://media1.clubpenguin.com/play/en/web_service/game_configs/paper_items.json'
result = json.load(url)
request = urllib2.Request(url)
response = urllib2.urlopen(request)
json_obj = json.load(response)
Your issue is that you're reading the URL object as a string and that's a non-starter. I fixed your code above and it now reads in the whole thing (snippet from output below:
{u'cost': 0,
u'is_bait': u'1',
u'is_member': Terue,
u'label': u"Stompin' Bob Fowhawk Hair",
u'layer': 6000,
u'paper_item_id': 1274,
u'prompt': u"Stompin' Bob Fowhawk Hair",
u'type': 2},
{u'cost': 0,
u'is_bait': u'1',
u'is_member': True,
u'label': u'G Billy Hair and Bandana',
u'layer': 6000,
u'paper_item_id': 1275,
u'prompt': u'G Billy Hair and Bandana',
u'type': 2},
... continues for a long time...

python + json: parse to list

I'm somewhat new to parsing JSON data with python (using python 2.7). There is a service that I have to send API calls to, and the JSON response is something like what I have below. the amount of items in 'row' can vary. What I need to do is take only the 'content' from the second line IF there is a second line, and put it into a list. Essentially, it is a list of only the 'campaign confirmation numbers' and nothing else. the number will also always be only 9 numeric numbers if that helps anything. Any advice would be very much appreciated.
{"response":
{"result":
{"Potentials":
{"row":
[
{"no":"1","FL":
{"content":"523836000004148171","val":"POTENTIALID"}
},
{"no":"2","FL":
{"content":"523836000004924051","val":"POTENTIALID"}
},
{"no":"3","FL":
[
{"content":"523836000005318448","val":"POTENTIALID"},
{"content":"694275295","val":"Campaign Confirmation Number"}
]
},
{"no":"4","FL":
[
{"content":"523836000005318662","val":"POTENTIALID"},
{"content":"729545274","val":"Campaign Confirmation Number"}
]
},
{"no":"5","FL":
[
{"content":"523836000005318663","val":"POTENTIALID"},
{"content":"903187021","val":"Campaign Confirmation Number"}
]
},
{"no":"6","FL":
{"content":"523836000005322387","val":"POTENTIALID"}
},
{"no":"7","FL":
[
{"content":"523836000005332558","val":"POTENTIALID"},
{"content":"729416761","val":"Campaign Confirmation Number"}
]
}
]
}
},
"uri":"/crm/private/json/Potentials/getSearchRecords"}
}
EDIT: an example of the output for this example would be:
confs = [694275295, 729545274, 903187021, 729416761]
or
confs = ['694275295', '729545274', '903187021', '729416761']
it really doesn't matter if they're stored as strings or ints
EDIT 2: here's my code snip:
import urllib
import urllib2
import datetime
import json
key = '[removed]'
params = {
'[removed]'
}
final_URL = 'https://[removed]'
data = urllib.urlencode(params)
request = urllib2.Request(final_URL,data)
response = urllib2.urlopen(request)
content = response.read()
j = json.load(content)
confs = []
for no in j["response"]["result"]["Potentials"]["row"]:
data = no["FL"]
if isinstance(data, list) and len(data) > 1:
confs.append(int(data[1]["content"]))
print confs

Assuming j is your JSON object which the above structure has been parsed into:
>>> results = []
>>> for no in j["response"]["result"]["Potentials"]["row"]:
... data = no["FL"]
... if isinstance(data, list) and len(data) > 1:
... results.append(int(data[1]["content"]))
...
>>> results
[694275295, 729545274, 903187021, 729416761]

Assuming that 'response' holds the json string:
import json
data = json.loads(response)
rows = data['response']['result']['Potentials']['rows']
output = []
for row in rows:
contents = row['FL']
if len(contents) > 1:
output.append(contents[1]['content'])
That should do it.
EDIT:
I finally got some time to test this "one liner". It's fun to use Python's functional features:
import json
#initialize response to your string
data = json.loads(response)
rows = data['response']['result']['Potentials']['row']
output = [x['FL'][1]['content'] for x in rows if isinstance(x['FL'], list) and len(x['FL']) > 1]
print output
['694275295', '729545274', '903187021', '729416761']