We Keep Coding

Reading JSON data in Python using Pagination, max records 100

I am trying to extract data from a REST API using python and put it into one neat JSON file, and having difficulty. The date is rather lengthy, with a total of nearly 4,000 records, but the max record allowed by the API is 100.
I've tried using some other examples to get through the code, and so far this is what I'm using (censoring the API URL and auth key, for the sake of confidentiality):
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while new_results:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("results", [])
vendors.extend(centiblock)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
For the life of me, I cannot figure out why it isn't working. The output keeps coming out as just:
[
"records"
]
If I play around with the print statement at the end, I can get it to print centiblock (so named for being a block of 100 records at a time) just fine - it gives me 100 records in un-formated text. However, if I try printing vendors at the end, the output is:
['records']
...which leads me to guess that somehow, the vendors array is not getting filled with the data. I suspect that I need to modify the get request where I define new_results, but I'm not sure how.
For reference, this is a censored look at how the json data begins, when I format and print out one centiblock:
{
"records": [
{
"id": "XXX",
"createdTime": "2018-10-15T19:23:59.000Z",
"fields": {
"Vendor Name": "XXX",
"Main Phone": "XXX",
"Street": "XXX",
Can anyone see where I'm going wrong?
Thanks in advance!

When you are extending vendors with centiblock, your are giving a dict to the extend function. extend is expecting an Iterable, so that works, but when you iterate over a python dict, you only iterate over the keys of the dict. In this case, ['records'].
Note as well, that your loop condition becomes False after the first iteration, because centiblock.get("results", []) returns [], since "results" is not a key of the output of the API. and [] has a truthiness value of False.
Hence to correct those errors you need to get the correct field from the API into new_results, and extend vendors with new_results, which is itself an array. Note that on the last iteration, new_results will be the empty list, which means vendors won't be extended with any null value, and will contain exactly what you need:
This should look like:
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while len(new_results) > 0:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("records", [])
vendors.extend(new_results)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
Note that I replaced the while new_results with a while len(new_results)>0 which is equivalent in this case, but more readable, and better practice in general.

Clockify API, unexpected data returned?

I'm requesting some time entries for users with the Clockify API(). For some reason, I am receiving some responses which include entries without an end-time. I noticed that, the unexpectedly returned entries belong to currently running time entires... However, I did not specify/use the 'in-progress' parameter... What is happening here?
Here is my code:
def fetch_users_time_entries(users):
API_URL = "https://api.clockify.me/api/v1"
for user in users:
url = "{}/workspaces/{}/user/{}/time-entries?hydrated=true&page-size=1000&start=2019-08-05T00:00:01Z".format(API_URL, WORKSPACE_ID, user['clockify_id'])
time_entries = requests.get(url, headers=HEADER)
for time_entry in time_entries.json():
Here is a sample of an unexpected "end" value:
{
'id':'SECRET',
'description':'',
'tags':[
{
'id':'SECRET',
'name':'CERTI',
'workspaceId':'SECRET'
}
],
'user':None,
'billable':True,
'task':{
'id':'SECRET',
'name':'Etapa: Execução e Controle',
'projectId':'SECRET',
'assigneeId':'',
'estimate':'PT0S',
'status':'ACTIVE'
},
'project':{
'id':'SECRET',
'name':'C105',
'hourlyRate':{
'amount':0,
'currency':'USD'
},
'clientId':'SECRET',
'workspaceId':'SECRET',
'billable':True,
'memberships':[
{
'userId':'SECRET',
'hourlyRate':None,
'targetId':'SECRET',
'membershipType':'PROJECT',
'membershipStatus':'ACTIVE'
}
],
'color':'#8bc34a',
'estimate':{
'estimate':'PT0S',
'type':'AUTO'
},
'archived':False,
'duration':'PT25H20M12S',
'clientName':'NEO',
'public':True
},
'timeInterval':{
'start':'2019-08-22T18:55:55Z',
'end':None,
'duration':None
},
'workspaceId':'SECRET',
'totalBillable':None,
'hourlyRate':None,
'isLocked':False,
'userId':'SECRET',
'projectId':'SECRET'
}
I was only expecting time entries that were completed. Any suggestions?

UPDATE (10/16/19):
Another follow-up. They just send me an e-mail saying they fixed the problem. Putting the parameter "in-progress" to false will return only completed time entries. #matthew-e-miller it would be nice to add this to the answer. – Lukas Belck 5 hours ago
Okay, so I finally had a chance to reproduce the problem and it seems... There is not an end-time filter. They have misleadingly provided a start and end parameter, but these both filter on start-time.
The start and end parameters work like this:
The in-progress works as described in the doc, but it doesn't work for your application.
Answer:
I think your best bet is to request all the time entries, place them into a dict/list, and then use your python script to remove elements with "'end: 'None'".
import requests
import json
headers = {"content-type": "application/json", "X-Api-Key": "your api key""}
workspaceId = "your workspace id"
userId = "your user id"
params = {'start': '2019-08-28T11:10:32.998Z', 'end': '2019-08-29T02:05:02Z', 'in-progress': 'true'}
API_URL = "https://api.clockify.me/api/v1/workspaces/{workspaceId}/user/{userId}/time-entries"
print(API_URL)
result_one = requests.get(API_URL, headers=headers, params=params)
print(result_one)
List = json.loads(result_one.text)
for entry in List:
if entry.get("timeInterval")['end'] == None:
List.remove(entry)
print(List)
Output:
List containing only entries which do not have timeInterval.end == 'None'.
Here is time spent on this answer-edit:

Printing dictionary from inside a list puts one character on each line

Yes, yet another. I can't figure out what the issue is. I'm trying to iterate over a list that is a subsection of JSON output from an API call.
This is the section of JSON that I'm working with:
[
{
"created_at": "2017-02-22 17:20:29 UTC",
"description": "",
"id": 1,
"label": "FOO",
"name": "FOO",
"title": "FOO",
"updated_at": "2018-12-04 16:37:09 UTC"
}
]
The code that I'm running that retrieves this and displays it:
#!/usr/bin/python
import json
import sys
try:
import requests
except ImportError:
print "Please install the python-requests module."
sys.exit(-1)
SAT_API = 'https://satellite6.example.com/api/v2/'
USERNAME = "admin"
PASSWORD = "password"
SSL_VERIFY = False # Ignore SSL for now
def get_json(url):
# Performs a GET using the passed URL location
r = requests.get(url, auth=(USERNAME, PASSWORD), verify=SSL_VERIFY)
return r.json()
def get_results(url):
jsn = get_json(url)
if jsn.get('error'):
print "Error: " + jsn['error']['message']
else:
if jsn.get('results'):
return jsn['results']
elif 'results' not in jsn:
return jsn
else:
print "No results found"
return None
def display_all_results(url):
results = get_results(url)
if results:
return json.dumps(results, indent=4, sort_keys=True)
def main():
orgs = display_all_results(KATELLO_API + "organizations/")
for org in orgs:
print org
if __name__ == "__main__":
main()
I appear to be missing a concept because when I print org I get each character per line such as
[
{
"
c
r
e
a
t
e
d
_
a
t
"
It does this through to the final ]
I've also tried to print org['name'] which throws the TypeError: list indices must be integers, not str Python error. This makes me think that org is being seen as a list rather than a dictionary which I thought it would be due to the [{...}] format.
What concept am I missing?
EDIT: An explanation for why I'm not getting this: I'm working with a script in the Red Hat Satellite API Guide which I'm using to base another script on. I'm basically learning as I go.

display_all_results is returning a string since you are doing json.dumps in json.dumps(results, indent=4, sort_keys=True), which converts the dictionary to a string (you are getting that dictionary from r.json() in get_json function)
You then end up iterating over the characters of that string in main, and you see one character per line
Instead just return results from display_all_results and the code will work as intended
def display_all_results(url):
#results is already a dictionary, just return it
results = get_results(url)
if results:
return results

Orgs is a result of json.dump which produces a string. So instead of this code:
for org in orgs:
print(org)
replace it with simply:
#for org in orgs:
print(orgs)

Parsing json file to collect data and store in a list/array

I am trying to build an IOT setup. I am thinking of using a json file to store states of the sensors and lights of the setup.
I have created a function to test out my concept. Here is what I wrote so far for the data side of things.
{
"sensor_data": [
{
"sensor_id": "302CEM/lion/light1",
"sensor_state": "on"
},
{
"sensor_id": "302CEM/lion/light2",
"sensor_state": "off"
}
]
}
def read_from_db():
with open('datajson.json') as f:
data = json.load(f)
for sensors in data['sensor_data']:
name = sensors['sensor_id']
read_from_db()
What I want to do is to parse the sensor_id into an array so that I can access them by saying for example sensor_name[0]. I am not sure how to go about it. I tried array.array but it doesn't save any values, have also tried .append but not the result I expected. Any suggestions?

If I understood correctly, all you have to do is assign all those sensors to names using a for loop and then return the result:
import json
def read_from_db():
with open('sensor_data.json') as f:
data = json.load(f)
names = [sensors['sensor_id'] for sensors in data['sensor_data']]
return names
sensor_names = read_from_db()
for i in range(len(sensor_names)):
print(sensor_names[i])
This will print:
302CEM/lion/light1
302CEM/lion/light2

KeyError In Python With json.dumps

I'm trying to work with dictionaries inside a list in a JSON file. The data imports fine and reads fine. For the life of me I can't figure out how to printout the "member_id" keys. I just want to print the list of "member_id" numbers. I was initially using json.loads, then switched to json.dumps. Any help would really be appreciated.
import urllib2
import json
nyt_api_key = '72c9a68bbc504e91a3919efda17ae621%3A7%3A70586819'
url= 'http://api.nytimes.com/svc/politics/v3/us/legislative/congress/113'
json_obj = urllib2.urlopen(url)
data = json.load(json_obj)
data2 = json.dumps(data, sort_keys=True, indent=True, skipkeys = True)
print data2
Output from print data2: (The list goes on and on so it is truncated. There is a closing bracket at the bottom of the list. So it's dictionaries within a list.)
"positions": [
{
"dw_nominate": "0.466",
"member_id": "A000055",
"vote_position": "Yes"
},
{
"dw_nominate": "0.995",
"member_id": "A000367",
"vote_position": "Yes"
},
{
"dw_nominate": "0.666",
"member_id": "A000369",
"vote_position": "Yes"
},
Output from print data2['member_id'], output is the same if using 'positions', 'vote_position', etc.:
Traceback (most recent call last):
File "/Users/Owner/PycharmProjects/untitled2/1", line 9, in <module>
print data2["positions"]
TypeError: string indices must be integers, not str
Output from print data:
u'positions': [{u'dw_nominate': u'0.466', u'vote_position': u'Yes', u'member_id': u'A000055'}, {u'dw_nominate': u'0.995', u'vote_position': u'Yes', u'member_id': u'A000367'}, {u'dw_nominate': u'0.666', u'vote_position': u'Yes', u'member_id': u'A000369'}
Output from print data['positions']:
print data["positions"]
KeyError: 'positions'
Output from print.data(keys):
[u'status', u'results', u'copyright']
Process finished with exit code 0

I just want to print the list of "member_id" numbers.
So you need to loop over positions and access the member_id in each dict:
data ={"positions": [
{
"dw_nominate": "0.466",
"member_id": "A000055",
"vote_position": "Yes"
},
{
"dw_nominate": "0.995",
"member_id": "A000367",
"vote_position": "Yes"
},
{
"dw_nominate": "0.666",
"member_id": "A000369",
"vote_position": "Yes"
}]}
print([d["member_id"] for d in data["results"]["positions"]])
['A000055', 'A000367', 'A000369']
If you look at the API documentation there are examples of each json response.

data2 is a string value, it doesn't have keys. I think what you want to print is data["positions"]
That's a weird output from data, you don't even have the braces. Try printing the type(data), it should be dict

So I should change the heading of this to Scrapping JSON for XML in Python. I'm sure not everyone else would have the same issues I did with JSON but after many frustrating hours I decided to go down path #2... the xml version. The xml version was much easier to work with right out of the gate. In about 1/10 the time I got what I was looking for.
from urllib2 import urlopen
from xml.dom import minidom
feed = urlopen("http://api.nytimes.com/svc/politics/v3/us/legislative.xml?
doc = minidom.parse(feed)
id_element = doc.getElementsByTagName("member_id")
id_number0 = id_element[0].childNodes[0].nodeValue #just a sample
id_number1 = id_element[1].childNodes[0].nodeValue #just a sample
id_number2 = id_element[2].childNodes[0].nodeValue #just a sample
print len(id_element) #to see how many items were in the variable
count = 0
for item in id_element:
print id_element[count].childNodes[0].nodeValue
count = count + 1
if count == 434:
break
This is definitely not the cleanest loop. I'm still working on that. But the code solves the problem that I had originally posted. The API key is not the actual one, formatting in the answer window was throwing it off so I just erased a bunch of it. You can find the API at the NYT developer website.
Thanks to everyone who posted.