We Keep Coding

How do i loop through this API to get more results?

I am trying to call all the results from this API and into one big list. Right now the default value is set to 50. By having the paginated section in the code I am still getting 50 results. How will I be able to get more results?
nextRecordKey allows to paginate through the api
import os
import requests
import json
import requests
from http import HTTPStatus
client_id = ""
client_secret = ""
os.environ["DX_GATEWAY"] = "http://api.com"
os.environ["DX_CLIENT_ID"] = client_id
os.environ["DX_CLIENT_SECRET"] = client_secret
dx_request = requests.Request()
path = "/path/to/api"
params = {
"Type": "abc",
"Id": "def",
"limit": 999,
"Category": "abc"
}
params_str = "&".join([f"{k}={v}" for k, v in params.items()])
url = "?".join([path, params_str])
vulns = requests.get(
url=url,
version=1,
)
if vulns.status_code != int(HTTPStatus.OK):
raise RuntimeError("API call did not return expected response: " + str(vulns))
## loop through paginated API ##.
response_data = vulns.json()
while vulns["nextRecordKey"]:
vulns=requests.get(vulns["nextRecordKey"]).json()
print(json.dumps(response_data))```

Pandas Data frame returning only the first Row of JSON Data?

I'm scraping data using the Twitter API, when I use the print command I can see all the data that i want, specifically as many rows of tweets and dates that I input.
However when I format the data into a pandas data frame/csv it only displays the first row of results. I'm really confused what to do and appreciate all help a lot. thanks :)
#importing key term and date of tweets from twitter archive
client_key = 'code'
client_secret = 'code'
import base64
key_secret = '{}:{}'.format(client_key, client_secret).encode('ascii')
b64_encoded_key = base64.b64encode(key_secret)
b64_encoded_key = b64_encoded_key.decode('ascii')
import requests
base_url = 'https://api.twitter.com/'
auth_url = '{}oauth2/token'.format(base_url)
auth_headers = {
'Authorization': 'Basic {}'.format(b64_encoded_key),
'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
}
auth_data = {
'grant_type': 'client_credentials'
}
auth_resp = requests.post(auth_url, headers=auth_headers, data=auth_data)
auth_resp.status_code
auth_resp.json().keys()
access_token = auth_resp.json()['access_token']
search_headers = {
'Authorization': 'Bearer {}'.format(access_token)
}
search_params = {
'q': 'Key Term',
'count': 5,
'start_time' : '2019-1-1',
'end_time' : '2019-2-1',
'place.fields' : 'USA',
'lang' : 'en'
}
search_url = '{}1.1/search/tweets.json'.format(base_url)
search_resp = requests.get(search_url, headers=search_headers, params=search_params)
tweet_data = search_resp.json()
import numpy as np
import pandas as pd
for x in tweet_data['statuses']:
data = {'Date':[(x['created_at'])],'Text':[(x['text'])]}
df = pd.DataFrame(data)
df.to_csv("Tweet_data.csv")
print(df)

Hey before your loop define data=[], then inside your loop do data.append({…}).
What you have at the minute is a loop that at every iteration, creates a dictionary and assigns it to a variable called “data”. Overwriting the previous “data” assignment.
Then you are writing a csv with only one “data” row.
Hope that’s helpful!

gpsread Batch Update Multiple Sheets

Is there a way to Batch Update multiple sheets at once? I am trying to protect every worksheet in a workbook and I am doing:
def protectSheet(sheet):
body = {
"requests": [
{
"addProtectedRange": {
"protectedRange": {
"range": {
"sheetId": sheet._properties['sheetId'],
},
"warningOnly": True
}
}
}
]
}
try:
workbook.batch_update(body)
except gspread.exceptions.APIError:
logger.warning("%s is already protected" % sheet._properties['title'])
for worksheet in workbook.worksheets():
protectSheet(worksheet)
But is there a way to make a single call to protect every sheet in a workbook?
Edit: Alternatively is there a way to tell if a worksheet is protected?

I believe your goal as follows.
In your question, there are following 2 questions.
is there a way to make a single call to protect every sheet in a workbook?
is there a way to tell if a worksheet is protected?.
You want to achieve above using gspread with python.
Answer for question 1:
The following sample script protects all sheets in the Spreadsheet by one API call. But for example, a sheet in the Spreadsheet has already been protected, an error occurs. So please be careful this.
Sample script:
spreadsheetId = "###" # Please set the Spreadsheet ID.
client = gspread.authorize(credentials)
ss = client.open_by_key(spreadsheetId)
worksheets = ss.worksheets()
requests = [
{
"addProtectedRange": {
"protectedRange": {
"range": {
"sheetId": e.id,
},
"warningOnly": True
}
}
} for e in worksheets]
request_body = {"requests": requests}
res = ss.batch_update(request_body)
print(res)
Answer for question 2:
The following sample script retrieves all protected sheets in the Spreadsheet by one API call. In order to retrieve all protected sheets and ranges by one API call, it is required to use the method of "spreadsheets.get". Unfortunately, it seems that there is no method of "spreadsheets.get" in gspread. So I used the access token from credentials for gspread.
When you run the script, the sheets IDs of protected sheets are retrieved.
Sample script:
import requests # This is used.
spreadsheetId = "###" # Please set the Spreadsheet ID.
# client = gspread.authorize(credentials)
access_token = credentials.access_token
endpoint = "https://sheets.googleapis.com/v4/spreadsheets/" + spreadsheetId + "?fields=sheets.protectedRanges.range"
headers = {'Authorization': 'Bearer ' + access_token}
res = requests.get(endpoint, headers=headers)
obj = res.json()
sheetIds = []
for e in obj['sheets']:
if 'protectedRanges' in e:
for f in e.get('protectedRanges'):
if 'range' in f and 'startRowIndex' not in f.get('range') and 'endRowIndex' not in f.get('range') and 'startColumnIndex' not in f.get('range') and 'endColumnIndex' not in f.get('range'):
if 'sheetId' not in f.get('range'):
sheetIds.append(0)
else:
sheetIds.append(f.get('range')['sheetId'])
print(sheetIds)
Other pattern:
As other pattern, the following sample script retrieves all unprotected sheets in the Spreadsheet.
Sample script:
import requests # This is used.
spreadsheetId = "###" # Please set the Spreadsheet ID.
# client = gspread.authorize(credentials)
access_token = credentials.access_token
endpoint = "https://sheets.googleapis.com/v4/spreadsheets/" + spreadsheetId + "?fields=sheets.properties%2Csheets.protectedRanges.range"
headers = {'Authorization': 'Bearer ' + access_token}
res = requests.get(endpoint, headers=headers)
obj = res.json()
sheetIds = []
for e in obj['sheets']:
if 'protectedRanges' in e:
for f in e.get('protectedRanges'):
if 'range' not in f or ('range' in f and (
'startRowIndex' in f.get('range') or
'endRowIndex' in f.get('range') or
'startColumnIndex' in f.get('range') or
'endColumnIndex' in f.get('range'))):
sheetIds.append(f.get('range')['sheetId'])
else:
sheetIds.append(e['properties']['sheetId'])
print(sheetIds)
Reference:
spreadsheets.get

How to print json info with python?

I have a json (url = http://open.data.amsterdam.nl/ivv/parkeren/locaties.json) and I want to print all 'title', 'adres', 'postcode'. How can I do that?
I want to print it like this:
title.
adres.
postcode.
title.
adres.
postcode.
so among themselves
I hope you can help me with this
import urllib, json
url = "http://open.data.amsterdam.nl/ivv/parkeren/locaties.json"
import requests
search = requests.get(url).json()
print(search['title'])
print(search['adres'])
print(search['postcode'])

Using print(json.dumps(r, indent=4)) you can see that the structure is
{
"parkeerlocaties": [
{
"parkeerlocatie": {
"title": "Fietsenstalling Tolhuisplein",
"Locatie": "{\"type\":\"Point\",\"coordinates\":[4.9032801,52.3824545]}",
...
}
},
{
"parkeerlocatie": {
"title": "Fietsenstalling Paradiso",
"Locatie": "{\"type\":\"Point\",\"coordinates\":[4.8833735,52.3621851]}",
...
}
},
So to access the inner properties, you need to follow the JSON path
import requests
url = ' http://open.data.amsterdam.nl/ivv/parkeren/locaties.json'
search = requests.get(url).json()
for parkeerlocatie in search["parkeerlocaties"]:
content = parkeerlocatie['parkeerlocatie']
print(content['title'])
print(content['adres'])
print(content['postcode'])
print()

parsing and getting list from response of get request

I'm trying to parse a website with the requests module:
import requests
some_data = {'a':'',
'b':''}
with requests.Session() as s:
result = s.post('http://website.com',data=some_data)
print(result.text)
The page is responding as below:
{
"arrangetype":"U",
"list": [
{
"product_no":43,
"display_order":4,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
},
{
"product_no":44,
"display_order":6,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
}
],
"length":2
}
I found that instead of parsing full HTML, it would be better to deal with the response as all the data I want is in that response.
What I want to get is a list of the values of product_no, so the expected result is:
[43,44]
How do I do this?

Convert your JSON response to a dictionary with json.loads(), and collect your results in a list comprehension.
Demo:
from json import loads
data = """{
"arrangetype":"U",
"list": [
{
"product_no":43,
"display_order":4,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
},
{
"product_no":44,
"display_order":6,
"is_selling":"T",
"product_empty":"F",
"fix_position":null,
"is_auto_sort":false
}
],
"length":2
}"""
json_dict = loads(data)
print([x['product_no'] for x in json_dict['list']])
# [43, 44]
Full Code:
import requests
from json import loads
some_data = {'a':'',
'b':''}
with requests.Session() as s:
result = s.post('http://website.com',data=some_data)
json_dict = loads(result.text)
print([x["product_no"] for x in json_dict["list"]])