We Keep Coding

Reading JSON data in Python using Pagination, max records 100

I am trying to extract data from a REST API using python and put it into one neat JSON file, and having difficulty. The date is rather lengthy, with a total of nearly 4,000 records, but the max record allowed by the API is 100.
I've tried using some other examples to get through the code, and so far this is what I'm using (censoring the API URL and auth key, for the sake of confidentiality):
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while new_results:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("results", [])
vendors.extend(centiblock)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
For the life of me, I cannot figure out why it isn't working. The output keeps coming out as just:
[
"records"
]
If I play around with the print statement at the end, I can get it to print centiblock (so named for being a block of 100 records at a time) just fine - it gives me 100 records in un-formated text. However, if I try printing vendors at the end, the output is:
['records']
...which leads me to guess that somehow, the vendors array is not getting filled with the data. I suspect that I need to modify the get request where I define new_results, but I'm not sure how.
For reference, this is a censored look at how the json data begins, when I format and print out one centiblock:
{
"records": [
{
"id": "XXX",
"createdTime": "2018-10-15T19:23:59.000Z",
"fields": {
"Vendor Name": "XXX",
"Main Phone": "XXX",
"Street": "XXX",
Can anyone see where I'm going wrong?
Thanks in advance!

When you are extending vendors with centiblock, your are giving a dict to the extend function. extend is expecting an Iterable, so that works, but when you iterate over a python dict, you only iterate over the keys of the dict. In this case, ['records'].
Note as well, that your loop condition becomes False after the first iteration, because centiblock.get("results", []) returns [], since "results" is not a key of the output of the API. and [] has a truthiness value of False.
Hence to correct those errors you need to get the correct field from the API into new_results, and extend vendors with new_results, which is itself an array. Note that on the last iteration, new_results will be the empty list, which means vendors won't be extended with any null value, and will contain exactly what you need:
This should look like:
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while len(new_results) > 0:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("records", [])
vendors.extend(new_results)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
Note that I replaced the while new_results with a while len(new_results)>0 which is equivalent in this case, but more readable, and better practice in general.

missing minute data from coinApi

I'm not sure if this is a problem with my coding or maybe I should rather ask CoinAPI this question directly. If I want to get some minute data from a coin, some data appears to be simply missing. In the picture with the output of my code you can see, that the data for the minute at 2018-05-31T23:42 is missing. Do you know a better site with historical crypto minute data?
This is my code:
import requests
symbol_id = 'BINANCE_SPOT_IOTA_USDT'
period_id = '1MIN'
limit = '5'
time_start='2018-05-31T23:40:00'
headers = {'X-CoinAPI-Key' : 'My CoinAPI-Key'}
response = requests.get(
f'https://rest.coinapi.io/v1/ohlcv/{symbol_id}/history?period_id={period_id}&time_start={time_start}&limit={limit}',
headers=headers)
print(response.text)
Thank you!

CoinAPI provides an additional parameter called period_id which accepts the units second/minute/hour/day/month/year. Data can be requested by the period.
period_id
parameter
Second
1SEC, 2SEC, 3SEC, 4SEC, 5SEC, 6SEC, 10SEC, 15SEC, 20SEC, 30SEC
Minute
1MIN, 2MIN, 3MIN, 4MIN, 5MIN, 6MIN, 10MIN, 15MIN, 20MIN, 30MIN
Hour
1HRS, 2HRS, 3HRS, 4HRS, 6HRS, 8HRS, 12HRS
Day
1DAY, 2DAY, 3DAY, 5DAY, 7DAY, 10DAY
Month
1MTH, 2MTH, 3MTH, 4MTH, 6MTH
Year
1YRS, 2YRS, 3YRS, 4YRS, 5YRS
import requests
url = 'https://rest.coinapi.io/v1/ohlcv/BTC/USD/history?period_id=1MIN&time_start=2016-01-01T00:00:00&period_id=1MIN'
headers = {'X-CoinAPI-Key' : '01E867A9-BB46-4A45-A1B4-BE140767040E'}
response = requests.get(url, headers=headers)
print(response.text)
{
"time_period_start": "2016-01-01T00:00:00.0000000Z",
"time_period_end": "2016-01-01T00:01:00.0000000Z",
"time_open": "2016-01-01T00:00:16.0000000Z",
"time_close": "2016-01-01T00:00:16.0000000Z",
"price_open": 430.350000000,
"price_high": 430.390000000,
"price_low": 430.350000000,
"price_close": 430.390000000,
"volume_traded": 0.072700000,
"trades_count": 4
},
{
"time_period_start": "2016-01-01T00:01:00.0000000Z",
"time_period_end": "2016-01-01T00:02:00.0000000Z",
"time_open": "2016-01-01T00:01:01.1500000Z",
"time_close": "2016-01-01T00:01:46.0000000Z",
"price_open": 430.890000000,
"price_high": 430.890000000,
"price_low": 430.380000000,
"price_close": 430.400000000,
"volume_traded": 1.028431010,
"trades_count": 7
},

While loop to make API calls until a condition is met

I want to make API calls until a condition is met. I figured I might use a while loop.
I have a JSON response from the server that is paginated.
{
"services": [
{
"id": "ABC12",
"name": "Networks",
"description": null,
"status": "active",
"teams": [
{
"id": "XYZ12",
"type": "team_reference",
"summary": "Network Systems ",
}
],
"acknowledgement_timeout": null,
"auto_resolve_timeout": null,
"alert_grouping": "intelligent",
"alert_grouping_timeout": null,
"integrations": [],
"response_play": null,
"type": "service",
"summary": "All Events",
}
],
"limit": 25,
"offset": 0,
"total": null,
"more": true
}
limit - max I can set is 100.
offset - If specified, shows results from that point.
more - If TRUE, there are more results. If FALSE, that is the end.
for more info on this pagination - https://v2.developer.pagerduty.com/docs/pagination
I need to match the name "Networks" and get its corresponding id "ABC12". The problem is, I have to paginate make multiple calls to the API.
I have written this so far.
import requests
import json
import urllib3
# Supress SSL warnings
urllib3.disable_warnings()
# API key
API_KEY = '12345asdfg'
def list_services():
x = 25
y = 0
results = []
url = f'https://api.pagerduty.com/services/?limit={x}&offset={y}'
headers = {
'Accept': 'application/vnd.pagerduty+json;version=2',
'Authorization': 'Token token={token}'.format(token=API_KEY)
}
current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
results.append(current_page)
while current_page['more'] == 'True':
y = y + 1
current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
results.append(current_page)
print(results) # Does not print anything
print(results) # Prints only the first call results, the while loop
# doesn't seem to work.
if __name__ == '__main__':
list_services()
the print(results) outside the while loop prints only the first API call results. The while loop doesn't seem to work. But the code compiles without any errors.
how do I set the value of x to 25 and make API calls and append the results to results until more is false?
OR
how do I make multiple API calls until I find the match. If I found a match, then stop making the call.
Or is there a better cleaner way to do this?

This does not work because you never actually reassign the url variable once y is changed. Also you are checking against 'True' which is a string, not a boolean value. In addition I believe the offset should increase by the amount of results everytime; not just one. For example if on your first call you get results 1-25. Then if you increase y by one, the second call will yield 2-26. Instead you should increase it by the limit. This way on the second call you get results 25-50. Here is how I would do this:
def list_services():
x = 25
y = 0
results = []
serv_id = None
flag = False
url = f'https://api.pagerduty.com/services/?limit={x}&offset={y}'
headers = {
'Accept': 'application/vnd.pagerduty+json;version=2',
'Authorization': 'Token token={token}'.format(token=API_KEY)
}
current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
results.append(current_page)
for serv_set in current_page['services']:
if serv_set['name'] == 'Networks':
serv_id = serv_set['id']
flag = True
while current_page['more'] == True and not flag:
for serv_set in current_page['services']:
if serv_set['name'] == 'Networks':
serv_id = serv_set['id']
break
y += x
url = f'https://api.pagerduty.com/services/?limit={x}&offset={y}'
current_page = json.loads(requests.get(url, verify=False, headers=headers).content.decode('UTF-8'))
results.append(current_page)
print(results)
print(results, serv_id)
You could further clean this up to avoid some redundancy but this should work. You should also check the status of the API call to ensure that you have a valid response.
Edit:
I edited in the issue dealing with obtaining the id attribute when the name == 'Networks'. Once again you could reduce the redundancy in this a lot but this will get you on the right track. Now serv_id = the id of the service with the name of Networks. If no match is found at the end of the iterations then serv_id will be None.

while current_page['more'] == 'True':
You are checking for a string called 'True' instead of a boolean of True, as is defined in your json file. This could be why your while loop is never executing, and you are not receiving your print statement.
Also, generally for API calls that have more than 1 page of data, you need to specify which page you are getting. Which means you need to reinitialize your payload in your while loop.
For example, if an API has a parameter called "page" that you can pass in, in your while loop you would have to pass in page = 1, page = 2, etc. as a payload.

Axios response is a string, with carriage returns in it, should be an array

I'm working on a small project using python and Vue.js. Within this project I make an axios request which should be an array of array's but is returning a string. The string looks like this from my response:
But first, I did see this response: Axios Get request data comes back with "data: ↵ ↵ ↵ ↵" and was like that's my issue. However, as I will state later in this question, I just finished doing a similar project and axios worked fine! The code was almost identical. I pasted that code into this project and it worked!
"[↵ [↵ "United States", ↵ 86.19↵ ], ↵ [↵ …, ↵ 0↵ ], ↵ [↵ "US-France", ↵ 0↵ ]↵]↵"
It should not be this. It should be an array of array. Here is what my axios request looks like:
const path = 'http://localhost:5000/worldMapData';
axios.post(path, varietyObject)
.then((res) => {
console.log(res)
console.log(typeof res.data)
commit('setWineData', res.data)
})
.catch((error) => {
console.log(error);
});
This is what the python route look like:
#app.route('/worldMapData', methods=['GET', 'POST'])
def route_seven():
map = Map()
if request.method == 'POST':
post_data = request.get_json()
variety = post_data.get('variety')
wineData = map.get_wine_data(variety)
print(wineData)
return jsonify(wineData)
Now the strangest thing, as mentioned earlier, is that I just finished a previous project doing a similar thing with no issues. I even took that code, pasted it into my current project, and it worked fine! I'm really curious as to where the carriage returns are coming from in my response. This is some more of the axios response:
headers: {content-type: "application/json"}
request: XMLHttpRequest {onreadystatechange: ƒ, readyState: 4, timeout: 0, withCredentials: false, upload: XMLHttpRequestUpload, …}
status: 200
statusText: "OK"
If you need to see the python code where the request is being made to its here:
def get_wine_data(self, variety):
# print(self.data.dtypes)
#This list will hold all of the country and wine scores
wine_data = []
#Getting all distinct countries
countries = self.data.country.unique()
for country in countries:
#resetting the data for each loop
data = self.data
#This list will hold the data for a single country
single_country = []
data = data[(data.country == country) & (data.variety == variety)]
#Getting the mean score.
mean = data["points"].mean()
#changing the format of the mean
mean = float(format(mean, '.2f'))
if math.isnan(mean):
mean = 0
if country == 'US':
country = 'United States'
single_country.append(country)
single_country.append(mean)
wine_data.append(single_country)
return wine_data
By the way, this is the response, part of it, from the original project:
data: Array(141), status: 200, statusText: "OK"
This is what I want to see on my current project: data: Array(141) Finally, right before I leave Python and Jsonify the information before I send it over its data type is a list. Any help with this matter will be appreciated.

okay, I believe that the axios code was okay. It was something with the CSV file that I was using. JSONP was not needed.

Error retrieving JSON data from Webhose API in Python

I am a beginner in Python and am trying to use the webhose.io API to collect data from the web. The problem is that this crawler retrieves 100 objects from one JSON at a time, i.e., to retrieve 500 data, it is necessary to make 5 requests. When I use the API, I am not able to collect all the data at once. I was able to collect the first 100 results, but when going to the next request, an error occurs, the first post is repeated. Follow the code:
import webhoseio
webhoseio.config(token="Xxxxx")
query_params = {
"q": "trump:english",
"ts": "1498538579353",
"sort": "crawled"
}
output = webhoseio.query("filterWebContent", query_params)
x = 0
for var in output['posts']:
print output['posts'][x]['text']
print output['posts'][x]['published']
if output['posts'] is None:
output = webhoseio.get_next()
x = 0
Thanks.

Use the following:
while output['posts']:
for var in output['posts']:
print output['posts'][0]['text']
print output['posts'][0]['published']
output = webhoseio.get_next()