I have a list of airports I would like to get flight information for from an API. The API only allows users to search one airport as a time. I tried to create a loop that would iterate over the list of airports but it was unsuccessful. I have 5,000 airports I need to pass to the API. Below I give a sample list of airports for example purposes.
apiKey = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
apiUrl = 'https://xxxxxxxxxxxxxxxxxxx.com/xxxxxxx/'
auth_header = {'x-apikey':apiKey}
airports = [['HLZN'], ['HLLQ'],['HLLB'],['HLGT'],['HLMS'],['HLLS'],['HLTQ'],['HLLT'],['HLLM']]
payload = {'max_pages': 1}
#Create an empty list to hold responses
json_responses = []
#Iterate through list
for airport in airports:
response = requests.get(apiUrl + f"airports/{airports}/flights",params=payload,
headers=auth_header)
if response.status_code == 200:
print(response.json())
else:
print("Error executing request")
results = response.json
#This presumes the JSON response is a dict, if the response is a list, use extend instead of
append
json_responses.append(response.json())
#Create a DataFrame from a list containing all of the gathered responses.
all_acct_df = pd.DataFrame(json_responses)
The error I get is : "Error Parsing Arguments Parsing_ERROR Invalid argument ID supplied 400"
I tried passing one airport ID through this looping code and it goes through but it's not iterating over lists.
I'm new to looping and API's so any help would be greatly appreciated. Thank you.
If "The API only allows users to search one airport as a time" then you may have to call it multiple times without hacks.
But you may want to consider a ProcessPoolExecutor or task queue manager like Celery.
responses = []
def get_response(airport):
# call the API and storge the data
res = requests.get(url)
responses.appnd(res.json())
for airport in airports:
ap = airport[0] # as your data
with concurrent.futures.ProcessPoolExecutor(max_work=50) as executor:
executor.submit(call, ap)
Related
So I've been trying with many different methods but I can't get around it. Basically this happens:
API function call returns a Dict inside of a list.
I have a list of arguments that need to be passed to the function above one by one.
I don't care about order.
Last step is to append that list to a Pandas.DataFrame which will remove duplicates and order and etc.
Examples (btw, the API is Python-Binance):
symbols = ['ADAUSDT', 'ETHUSDT', 'BTCUSDT']
orders = pd.DataFrame()
for s in symbols:
orders = orders.append(client.get_all_orders(symbol=s)) # This returns the Dict
I tried using Queue() and Thread(), both with Lock(). I tried ThreadPoolExecutor() as well but I cannot make it work. The furthest I reached was with the last method but the amount of lines where different after each execution:
orders = pd.DataFrame()
temp = []
with ThreadPoolExecutor() as executor:
executor.map(get_orders, symbols)
for x in temp:
orders = orders.append([x])
Any ideas?
Thanks
On the sidelines, used alone orders.append([x])
this can help you
from binance import Client
client = Client(api_key, api_secret)
symbols = ['ADAUSDT', 'ETHUSDT', 'BTCUSDT']
orders_symbol =[]
orders = {}
for s in symbols:
orders.setdefault(s,{})
orders_symbol = client.get_all_orders(symbol=s)
for i in orders_symbol:
orders[s][i['orderId']] = i
print (s,i['orderId'],orders[s][i['orderId']])
print ()
I am trying to get a list of all my saved songs using current_user_saved_tracks(), but the limit is 20 tracks. Is there any way to access all 1000+ songs I have on m account?
The signature is as follows:
def current_user_saved_tracks(self, limit=20, offset=0)
The official Spotify API reference (beta) says that the maximum is limit=50. So, in a loop, call current_user_saved_tracks, but increment the offset by limit each time:
def get_all_saved_tracks(user, limit_step=50):
tracks = []
for offset in range(0, 10000000, limit_step):
response = user.current_user_saved_tracks(
limit=limit_step,
offset=offset,
)
print(response)
if len(response) == 0:
break
tracks.extend(response)
return tracks
Loop until you get an empty response or an exception. I'm not sure which one.
If you don't have to worry about the user deciding to add a saved track while you are retrieving them, this should work.
Yes, the default argument is limit=20. You could set a higher limit with the following code:
current_user_saved_tracks(limit=50)
Or you could set an offset to get the 20 next tracks:
current_user_saved_tracks(offset=20)
Source: https://spotipy.readthedocs.io/en/2.14.0/?highlight=current_user_saved#spotipy.client.Spotify.current_user_saved_tracks
I'm having a bit of trouble with a function I'm trying to write. What it is supposed to do is 1) go to a particular URL and get a list of financial sectors stored in a particular div; 2) visit each sector's respective page and get 3 particular pieces of information from there; 3) put the gathered collection into a dictionary; and 4) append that dictionary to another dictionary.
The desired output is a dictionary containing a list of dictionaries for all the sectors.
Here is my function:
def fsr():
fidelity_sector_report = dict()
url = "https://eresearch.fidelity.com/eresearch/goto/markets_sectors/landing.jhtml"
import requests
from bs4 import BeautifulSoup
# scrape the url page and locate links for each sector
try:
response = requests.get(url)
if not response.status_code == 200:
return 'Main page error'
page = BeautifulSoup(response.content, "lxml")
sectors = page.find_all('a',class_="heading1")
for sector in sectors:
link = 'https://eresearch.fidelity.com/' + sector['href']
name = sector.text
sect = dict()
lst = []
# scrape target pages for required information
try:
details = requests.get(link)
if not details.status_code == 200:
return 'Details page error'
details_soup = BeautifulSoup(details.content,'lxml')
fundamentals = details_soup.find('div',class_='sec-fundamentals')
values = dict()
#locate required values by addressing <tr> text and put them in a dictionary
values['Enterprise Value'] = fundamentals.select_one('th:contains("Enterprise Value") + td').text.strip()
values['Return on Equity (TTM)'] = fundamentals.select_one('th:contains("Return on Equity (TTM)") + td').text.strip()
values['Dividend Yield'] = fundamentals.select_one('th:contains("Dividend Yield") + td').text.strip()
#add values to the sector dictionary
sect[name] = values
# add the dictionary to the list
lst.append(dict(sect))
# for a dictionary using the list
fidelity_sector_report['results'] = lst
except:
return 'Something is wrong with details request'
return fidelity_sector_report
except:
return "Something is horribly wrong"
AS far as I can tell, it performs the main taks wonderfully, and the problem appears at the stage of appending a formed dictionary to a list - instead of adding new piece, it gets overwritten completely. I figured that out by putting print(lst) right after the fidelity_sector_report['results'] = lst line.
What should I change so that list (and, correspondingly, dictionary) gets formed as planned?
You should move the lst=[] outside of the sectors loop.
Your problem appears since for each sector, you reset lst and you append the current sector data to an empty list.
The following code causes the value of fidelity_sector_report['results'] to be replaced with lst.
fidelity_sector_report['results'] = lst
I presume you would want to access the respective values using a key, you can add the following line below fidelity_sector_report = dict() to initialize a dictionary:
fidelity_sector_report['results'] = {}
Then, create a key for each sector using the sector name and set the value with your values dictionary by replacing fidelity_sector_report['results'] = lst with:
fidelity_sector_report['results'][name] = dict(values)
You can access the data by using the relevant keys, i.e fidelity_sector_report['results']['Financials']['Dividend Yield'] for the dividend yield of the financials sector.
I am using Python 3 / Tweepy to create a list that contains the user names associated with various Twitter handles.
My code creates an empty dictionary, loops through the handles in the list to get the user name, saves this info in a dictionary and then appends the dictionary to a new list.
I am getting IndexError: list index out of range when I run the code. When I remove the 4th line of the for loop I do not get errors. Any thoughts on how I can resolve the issue? Why is this line of code causing errors? Thanks!
Here is my code:
def analyzer():
handles = ['#Nasdaq', '#Apple', '#Microsoft', '#amazon', '#Google', '#facebook', '#GileadSciences', '#intel']
data = []
# Grab twitter handles and append the name to data
for handle in handles:
data_dict = {}
tweets = api.user_timeline(handle)
data_dict['Handle'] = handle
data_dict['Name'] = tweets[0]['user']['name']
data.append(data_dict)
i guess main issue in below code
tweets = api.user_timeline(handle)
api.user_timeline() may returns you empty list and you are trying to access
first element from this empty list.
tweets[0]
that's why you are getting 'index out of range' issue.
you can modify your code somthing like this -
for handle in handles:
data_dict = {}
tweets = api.user_timeline(handle)
data_dict['Handle'] = handle
if tweets:
data_dict['Name'] = tweets[0]['user']['name']
data.append(data_dict)
the error is occuring beacause of the empty list which you are trying to access with index 0. you can control this by checking if list is empty or not:
def analyzer():
handles = ['#Nasdaq', '#Apple', '#Microsoft', '#amazon', '#Google', '#facebook', '#GileadSciences', '#intel']
data = []
# Grab twitter handles and append the name to data
for handle in handles:
data_dict = {}
tweets = []
tweets = api.user_timeline(handle)
if tweets:
data_dict['Handle'] = handle
data_dict['Name'] = tweets[0]['user']['name']
data.append(data_dict)
I'm trying to run a search using the New York Times' Article Search API v 2. The way the API works, for queries with more than 10 results, the results are separated into pages, and you must request each page separately. So for instance, to get results 1-10, include "page=0" in the request. For results 1-11, it's "page=1", and so on.
I want to grab all the results. (On the query I'm testing, there are 147.) The following code works only on EVEN-NUMBERED pages. So it returns results just fine when the count is 0, 2, 4. On odd-numbered requests, it seems to successfully query the API -- it returns a "Status": "OK" value -- but the list of "docs", where results are generally stored, is empty.
I can't for the life of me figure out why. Any ideas?
from nytimesarticle import articleAPI
import time
#initialize API
api = articleAPI(API_key)
#conduct search
def custom_search(page_number):
articles = api.search( q = query, page = page_number)
return articles
count = 0
while True:
#get first set of articles
articles = custom_search(count)
#print all the headlines
for article in articles['response']['docs']:
print "%s\nDate: %s\t%s\n" % (article['headline']['main'],
article['pub_date'], count)
#iterate the page number
count += 1
#pause to avoid rate limits
time.sleep(2)