We Keep Coding

TDAmeritrade API wont return 4:00PM time for price history

I am working with the TDAmeritrade API and i am making something that returns the price history when requested. Currently, i am unable to get 4:00PM when the aftermarket data is "false". I can get 4:00PM when 'needExtendedHoursData' is true, but i dont want the extra data that comes with it.
def get_priceHistory(start_date, freq, symbol):
payload = {
'apikey':client_id,
'periodType':'day',
'frequencyType':'minute',
'frequency':60,
'startDate':start_date,
'needExtendedHoursData':'false'
}
endpoint = r'https://api.tdameritrade.com/v1/marketdata/{}/pricehistory'.format(symbol)
content = requests.get(url=endpoint, params=payload)
data = content.json()

Reading JSON data in Python using Pagination, max records 100

I am trying to extract data from a REST API using python and put it into one neat JSON file, and having difficulty. The date is rather lengthy, with a total of nearly 4,000 records, but the max record allowed by the API is 100.
I've tried using some other examples to get through the code, and so far this is what I'm using (censoring the API URL and auth key, for the sake of confidentiality):
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while new_results:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("results", [])
vendors.extend(centiblock)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
For the life of me, I cannot figure out why it isn't working. The output keeps coming out as just:
[
"records"
]
If I play around with the print statement at the end, I can get it to print centiblock (so named for being a block of 100 records at a time) just fine - it gives me 100 records in un-formated text. However, if I try printing vendors at the end, the output is:
['records']
...which leads me to guess that somehow, the vendors array is not getting filled with the data. I suspect that I need to modify the get request where I define new_results, but I'm not sure how.
For reference, this is a censored look at how the json data begins, when I format and print out one centiblock:
{
"records": [
{
"id": "XXX",
"createdTime": "2018-10-15T19:23:59.000Z",
"fields": {
"Vendor Name": "XXX",
"Main Phone": "XXX",
"Street": "XXX",
Can anyone see where I'm going wrong?
Thanks in advance!

When you are extending vendors with centiblock, your are giving a dict to the extend function. extend is expecting an Iterable, so that works, but when you iterate over a python dict, you only iterate over the keys of the dict. In this case, ['records'].
Note as well, that your loop condition becomes False after the first iteration, because centiblock.get("results", []) returns [], since "results" is not a key of the output of the API. and [] has a truthiness value of False.
Hence to correct those errors you need to get the correct field from the API into new_results, and extend vendors with new_results, which is itself an array. Note that on the last iteration, new_results will be the empty list, which means vendors won't be extended with any null value, and will contain exactly what you need:
This should look like:
import requests
import json
from requests.structures import CaseInsensitiveDict
url = "https://api.airtable.com/v0/CENSORED/Vendors?maxRecords=100"
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer CENSORED"
resp = requests.get(url, headers=headers)
resp.content.decode("utf-8")
vendors = []
new_results = True
page = 1
while len(new_results) > 0:
centiblock = requests.get(url + f"&page={page}", headers=headers).json()
new_results = centiblock.get("records", [])
vendors.extend(new_results)
page += 1
full_directory = json.dumps(vendors, indent=4)
print(full_directory)
Note that I replaced the while new_results with a while len(new_results)>0 which is equivalent in this case, but more readable, and better practice in general.

Clockify API, unexpected data returned?

I'm requesting some time entries for users with the Clockify API(). For some reason, I am receiving some responses which include entries without an end-time. I noticed that, the unexpectedly returned entries belong to currently running time entires... However, I did not specify/use the 'in-progress' parameter... What is happening here?
Here is my code:
def fetch_users_time_entries(users):
API_URL = "https://api.clockify.me/api/v1"
for user in users:
url = "{}/workspaces/{}/user/{}/time-entries?hydrated=true&page-size=1000&start=2019-08-05T00:00:01Z".format(API_URL, WORKSPACE_ID, user['clockify_id'])
time_entries = requests.get(url, headers=HEADER)
for time_entry in time_entries.json():
Here is a sample of an unexpected "end" value:
{
'id':'SECRET',
'description':'',
'tags':[
{
'id':'SECRET',
'name':'CERTI',
'workspaceId':'SECRET'
}
],
'user':None,
'billable':True,
'task':{
'id':'SECRET',
'name':'Etapa: Execução e Controle',
'projectId':'SECRET',
'assigneeId':'',
'estimate':'PT0S',
'status':'ACTIVE'
},
'project':{
'id':'SECRET',
'name':'C105',
'hourlyRate':{
'amount':0,
'currency':'USD'
},
'clientId':'SECRET',
'workspaceId':'SECRET',
'billable':True,
'memberships':[
{
'userId':'SECRET',
'hourlyRate':None,
'targetId':'SECRET',
'membershipType':'PROJECT',
'membershipStatus':'ACTIVE'
}
],
'color':'#8bc34a',
'estimate':{
'estimate':'PT0S',
'type':'AUTO'
},
'archived':False,
'duration':'PT25H20M12S',
'clientName':'NEO',
'public':True
},
'timeInterval':{
'start':'2019-08-22T18:55:55Z',
'end':None,
'duration':None
},
'workspaceId':'SECRET',
'totalBillable':None,
'hourlyRate':None,
'isLocked':False,
'userId':'SECRET',
'projectId':'SECRET'
}
I was only expecting time entries that were completed. Any suggestions?

UPDATE (10/16/19):
Another follow-up. They just send me an e-mail saying they fixed the problem. Putting the parameter "in-progress" to false will return only completed time entries. #matthew-e-miller it would be nice to add this to the answer. – Lukas Belck 5 hours ago
Okay, so I finally had a chance to reproduce the problem and it seems... There is not an end-time filter. They have misleadingly provided a start and end parameter, but these both filter on start-time.
The start and end parameters work like this:
The in-progress works as described in the doc, but it doesn't work for your application.
Answer:
I think your best bet is to request all the time entries, place them into a dict/list, and then use your python script to remove elements with "'end: 'None'".
import requests
import json
headers = {"content-type": "application/json", "X-Api-Key": "your api key""}
workspaceId = "your workspace id"
userId = "your user id"
params = {'start': '2019-08-28T11:10:32.998Z', 'end': '2019-08-29T02:05:02Z', 'in-progress': 'true'}
API_URL = "https://api.clockify.me/api/v1/workspaces/{workspaceId}/user/{userId}/time-entries"
print(API_URL)
result_one = requests.get(API_URL, headers=headers, params=params)
print(result_one)
List = json.loads(result_one.text)
for entry in List:
if entry.get("timeInterval")['end'] == None:
List.remove(entry)
print(List)
Output:
List containing only entries which do not have timeInterval.end == 'None'.
Here is time spent on this answer-edit:

Limiting the lenght of response with python requests

I'm setting up a Views that save data from an API into my database on a button click and I'm having a hard time trying to figure out how to limit the size of the requests response for product descriptions in the following way:
If the lenght of a description is higher than 2000, delete some of the letters at the end of it until it reaches the limit of 2000, but don't remove it completely from the request.
As of now, what I've been able to achieve is to completely remove the product infos if the lenght is higher than 2000 as you can see below.
My django views function:
def api_data(request):
if request.GET.get('mybtn'): # to improve, == 'something':
resp_1 = requests.get(
"https://www.test-headout.com/api/public/v1/product/listing/list-by/city?language=fr&cityCode=PARIS&limit=5000&currencyCode=CAD",
headers={
"Headout-Auth": HEADOUT_TEST_API_KEY
})
resp_1_data = resp_1.json()
base_url_2 = "https://www.test-headout.com/api/public/v1/product/get/"
for item in resp_1_data['items']:
# concat ID to the URL string
url = '{}{}'.format(base_url_2, item['id'] + '?language=fr')
# make the HTTP request
resp_2 = requests.get(
url,
headers={
"Headout-Auth": HEADOUT_TEST_API_KEY
})
resp_2_data = resp_2.json()
if len(resp_2_data['contentListHtml'][0]['html']) < 2000: #represent the description of a product
Product.objects.get_or_create(
title=item['name'],
destination=item['city']['name'],
description=resp_2_data['contentListHtml'][0]['html'],
link=item['canonicalUrl'],
image=item['image']['url']
)
return render(request, "form.html")
But I remove way to much rows by doing this so I would like to know how can I fix this?
Please help.

Instead of using a conditional statement, you can use the slice operator to specify the character limit for the description. Using this approach you can refactor the relevant part of your code:
resp_2_data = resp_2.json()
Product.objects.get_or_create(title=item['name'],
destination=item['city']['name'],
description=resp_2_data['contentListHtml'][0]['html'][0:2000],
....
)

Determine the rate limit for requests

I have a question about rate limits.
I take a data from the CSV and enter it into the query and the output is stored in a list.
I get an error because I make too many requests at once.
(I can only make 20 requests per second). How can I determine the rate limit?
import requests
import pandas as pd
df = pd.read_csv("Data_1000.csv")
list = []
def requestSummonerData(summonerName, APIKey):
URL = "https://euw1.api.riotgames.com/lol/summoner/v3/summoners/by-name/" + summonerName + "?api_key=" + APIKey
response = requests.get(URL)
return response.json()
def main():
APIKey = (str)(input('Copy and paste your API Key here: '))
for index, row in df.iterrows():
summonerName = row['Player_Name']
responseJSON = requestSummonerData(summonerName, APIKey)
ID = responseJSON ['accountId']
ID = int(ID)
list.insert(index,ID)
df["accountId"]= list

If you already know you can only make 20 requests per second, you just need to work out how long to wait between each request:
Divide 1 second by 20, which should give you 0.05. So you just need to sleep for 0.05 of a second between each request and you shouldn't hit the limit (maybe increase it a bit if you want to be safe).
import time at the top of your file and then time.sleep(0.05) inside of your for loop (you could also just do time.sleep(1/20))