We Keep Coding

Reading JSON data in Python using Pagination, max records 100

I am trying to extract data from a REST API using python and put it into one neat JSON file, and having difficulty. The date is rather lengthy, with a total of nearly 4,000 records, but the max record allowed by the API is 100.
I've tried using some other examples to get through the code, and so far this is what I'm using (censoring the API URL and auth key, for the sake of confidentiality):
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while new_results:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("results", [])
vendors.extend(centiblock)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
For the life of me, I cannot figure out why it isn't working. The output keeps coming out as just:
[
"records"
]
If I play around with the print statement at the end, I can get it to print centiblock (so named for being a block of 100 records at a time) just fine - it gives me 100 records in un-formated text. However, if I try printing vendors at the end, the output is:
['records']
...which leads me to guess that somehow, the vendors array is not getting filled with the data. I suspect that I need to modify the get request where I define new_results, but I'm not sure how.
For reference, this is a censored look at how the json data begins, when I format and print out one centiblock:
{
"records": [
{
"id": "XXX",
"createdTime": "2018-10-15T19:23:59.000Z",
"fields": {
"Vendor Name": "XXX",
"Main Phone": "XXX",
"Street": "XXX",
Can anyone see where I'm going wrong?
Thanks in advance!

When you are extending vendors with centiblock, your are giving a dict to the extend function. extend is expecting an Iterable, so that works, but when you iterate over a python dict, you only iterate over the keys of the dict. In this case, ['records'].
Note as well, that your loop condition becomes False after the first iteration, because centiblock.get("results", []) returns [], since "results" is not a key of the output of the API. and [] has a truthiness value of False.
Hence to correct those errors you need to get the correct field from the API into new_results, and extend vendors with new_results, which is itself an array. Note that on the last iteration, new_results will be the empty list, which means vendors won't be extended with any null value, and will contain exactly what you need:
This should look like:
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while len(new_results) > 0:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("records", [])
vendors.extend(new_results)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
Note that I replaced the while new_results with a while len(new_results)>0 which is equivalent in this case, but more readable, and better practice in general.

assign export.json() to new variable

In python im using requests library.
payload = {'Code':Code}
headers = {'Content-type': 'application/json'}
export = requests.post('http://service-one.vms.svc.cluster.local/api/done/v1/import/company', data=json.dumps(payload), headers=headers)
print(export.json())
output is:
{'id': 'dcc515e0-8c43-4e2a-871b-99fcc4fd4403'}
How can I assign only "value" part from this key:value to new variable so I can use it later?

The export.json() gives you a dictionary. To save only the value to a variable you can do
exported_value = export.json()["id"]
get() will give you the same result too
exported_value = export.json().get("id")
I suggest to check dictionaries in Python

Update dictionary value for a particular key with value from iterator of outer loop

I have a list of lists in this format:
nrows = ["['Shock pain', 'attack sharp pain']",
"['bruises', 'lump']",
"['fever', 'cold', 'Anxiety']",
"['neck pain', 'headache']"]
I want to call an API by passing everytime 1 list at a time from the nrows list. Each list should be passed 1 by 1 to the data dict with key sList and the response should be saved.
The API function is as follows:
newres = []
for i in nrows:
url = "https://testabcd.io"
data= {
"sList": i,
"kList" : ["age"],
}
headers = {
'Content-Type': 'application/json',
'Auth': '0644427814339900'
}
response = requests.request("GET", url, headers=headers, json=data)
newres.append(response.text)
print(response.text)
print(newres)
Inside the data dict, in sList, at each iteration, I want to pass 1 sublist at a time and get the response appended in a list.
The current code has all the response same as I think I am unable to iterate and change the value of the data dict with key sList which is what is expected.
The csv looks like this:
['Shock pain', 'attack sharp pain']
['bruises', 'lump']
['fever', 'cold', 'Anxiety']

Assuming the file you work with, really looks like this:
['Shock pain', 'attack sharp pain']
['bruises', 'lump']
['fever', 'cold', 'Anxiety']
you can use ast.literal_eval in order to convert the line from the file into list
import requests
from ast import literal_eval
newres = []
url = "https://testabcd.io"
headers = {'Content-Type':'application/json',
'Auth':'0644427814339900'}
with open("yourfile") as f:
for line in f:
data= {"sList": literal_eval(line),
"kList" : ["age"]}
response = requests.get(url, headers=headers, json=data)
newres.append(response.text)
print(response.text)
print(newres)
Note:
This is not tested.
Given that your data comes from non-standard file, where they were exported in bad manner from a dataframe, created unknown how - I would suggest that you seriously reconsider all your workflow and how you manipulate your data.

TypeError: byte indices must be integers

I want to get the top artists from a specific country from the last fm API in JSON and save the name and url in the name and url variables. But it always appears "TypeError: byte indices must be integers". Do you know where is the issue?
Working example:
import requests
api_key = "xxx"
for i in range(2,5):
artists = requests.get('http://ws.audioscrobbler.com/2.0/?method=geo.gettopartists&country=spain&format=json&page='+str(i)+'&api_key='+api_key)
for artist in artists:
print(artist)
#name = artist['topartists']['artist']['name']
#url = artist['topartists']['artist']['url']

You want:
response = requests.get(...)
data = response.json()
for artist in data["topartists"]["artist"]:
name = artist["name"]
# etc
Explanation: requests.get() returns a response object. Iterating over the response object is actually iterating over the raw textual response content, line by line. Since this content is actually json, you want to first decode it to Python (response.json() is mainly a shortcut for json.loads(response.content)). You then get a python dict with, in this case, a single key "topartists" which points to a list of "artist" dicts.
A couple hints:
First you may want to learn to use string formatting instead of string concatenation. This :
'http://ws.audioscrobbler.com/2.0/?method=geo.gettopartists&country=spain&format=json&page='+str(i)+'&api_key='+api_key
is ugly and hardly readable. Using string formatting:
urltemplate = "http://ws.audioscrobbler.com/2.0/?method=geo.gettopartists&country=spain&format=json&page={page}&api_key={api_key}"
url = urltemplate.format(page=i, api_key=api_key)
but actually requests knows how to build a querystring from a dict, so you should really use this instead:
query = {
"method": "geo.gettopartists",
"country":"spain",
"format":"json",
"api_key": api_key
}
url = "http://ws.audioscrobbler.com/2.0/"
for pagenum in range(x, y):
query["page"] = pagenum
response = requests.get(url, params=query)
# etc
Then, you may also want to handle errors - there are quite a few things that can go wrong doing an HTTP request.

pass a string in python requests

I have a function which require to pass some ids as comma seperated to the url string.
My code:
def shipment(self, orderItemIds):
url = "https://api.flipkart.net/sellers/orders/shipments"
payload = {'orderItemsIds':orderItemIds}
return self.session.get(url, data=payload)
I need to pass id1, id2, id3 so that i get a link as:
https://api.flipkart.net/sellers/orders/shipments?orderItemsIds={id1,id2...}
I tried to pass it as a string. and as a list too. But it didn't worked.
oid = 'id1,id2,id3' # or
oid = ['id1',id2'','id3']
How to do it?
I passed an id as oiids = '230501592'.
On running response.url, it gives me:
https://api.flipkart.net/sellers/orders/shipments?orderItemsIds
It shows that parameter values are not passing in the url string.

I believe what you want is this (where orderItemIds is a list of strings):
def shipment(self, orderItemIds):
url = "https://api.flipkart.net/sellers/orders/shipments"
params = {'orderItemsIds': ','.join(orderItemIds)}
return self.session.get(url, params=params)
The differences from the version you have in the question are that orderItemIds is assumed to be a list of strings and is being joined with commas, and the data keyword is replaced with params, which is the right choice for a GET request.

You can easily send get parameters in python requests library-
data = {
'orderItemsIds': ['id1', 'id2', 'id3']
}
response = requests.get(url, data = data)