We Keep Coding

API call with for loop doesn't loop through a list in Python

I'm trying to make an API call to coinmarketcap to get the prices for a crypto portfolio and append the prices to a list.
The portfolio is on an excel file. I'm reading the excel with pandas, then looping through the "project" column of the dataframe and appending the names to a list called "project_list".
Then I'm making the API call, and looping through the previously created "project_list", and trying to get the price for each project, and finally appending those prices to an empty list called "price_list".
But when I run the code, it gives and IndexError, IndexError: list index out of range.
But the curious thing is, if I populate the project list manually, the code works just fine.
I thought this might be because I'm appending to the list dynamically and it's not ready by the time the API call is made. But I tried doing other things with the list right after appending, and they all work fine..
Code is below and any help would be greatly appreciated!
Step 1: Read the Excel File
crypto_df = pd.read_excel('D:\Trading\CRYPTO\Crypto Portfolio Tracker.xlsx')
Step 2: Create project_list and append the names from the dataframe
project_list = []
for name in crypto_df['Project'].values:
project_list.append(name)
Step 3: Making the API call
class CMC:
def __init__(self, token):
self.apiurl = 'https://pro-api.coinmarketcap.com'
self.headers = {'Accepts': 'application/json', 'X-CMC_PRO_API_KEY': token, }
self.session = Session() # instead of requesting url through requests, we are requesting through sessions.
self.session.headers.update(self.headers)
def get_all_coins(self):
url = self.apiurl + '/v1/cryptocurrency/map'
r = self.session.get(url)
data = r.json()['data']
return data
def get_price(self, symbol):
url = self.apiurl + '/v2/cryptocurrency/quotes/latest'
parameters = {'symbol': symbol}
r = self.session.get(url, params=parameters)
data = r.json()['data']
return data
Step 4: Appending the prices to price_list
price_list = []
cmc = CMC(secrets_.API_KEY)
for crypto in project_list:
price = cmc.get_price(crypto)
price_list.append(price[crypto][0]['quote']['USD']['price'])
print(price_list)
And this gives the Index Error.
If I manually populate the project_list, for example project_list = ['ETH', 'BTC', 'ADA', 'LINK', etc..] it works just fine. What am I doing wrong?

Thanks for responding guys. I figured it out. There's nothing wrong with my code. Coinmarketcap has a limit on requests/min for basic plan(free) API. So if my list is too big, it's throwing an error. When I reduced my list down to only a few currencies, it worked. I still can't be sure exactly how many I can add to the list, it varies with the amount of requests they are getting I guess.

How to store multiple python dictionaries or merge in a loop and return?

I am trying to get the pagination results of two pages but return is exiting loop and displays only one result from a page.
Is there a way to store or merge them?
def incidents():
m = True
limit = 50
offset = 0
while m == True:
url = f"{URL}/incidents"
params = {
"statuses[]": "resolved",
"include[]" : 'channel',
"limit" : limit,
"offset" : offset,
"total" : "true",
}
r = requests.get(url, headers=headers, params=params)
data = r.json()
offset += 50
print(offset, (r.text))
more = False # Set deliberately for easier understanding
return data
The offset, (r.text) output looks like -
50 {"incidents":[{"incident_number":1,"title":"crit server is on fire" ....
100 {"incidents":[{"incident_number":54,"title":"ghdg","description":"ghdg",....
Return only displays below and not the other one. There should be a way like use a generator for example? So we can merge them both and store in data variable so data can be returned?
100 {"incidents":[{"incident_number":54,....

I believe you could store the results in your own list:
incidents = []
and then
data = r.json()
for element in data['incidents']:
incidents.append(element)
Edited for clarity - that way you're gathering all incidents in a single object.

I'm not sure because you just gave the very start of r.text (is there more than 'incidents' within the result?), but i expect the previous answer to be a bit short; i'd suggest something like
results = []
(before the while) and at the end
data = r.json()
results += data['incidents']
return results
(btw: in your original post, each run through while just set the var "data", so no wonder the return can only deal with the very last part retrieved. But i guess that is just an artifact of you simplification, like the "more=False" would even prevent getting a second page)

Python geocoder limit

I have data of post number in excel. I input in python as a list.
I use geocoder library to get the latitude and longitude by the post number so i can put on map later on.
g = geocoder.google('1700002')
g.latlng
g.latlng brings me a list with [latitude,longitude] in it.
Since is take string only. I changed the values from float to int to get rid of point 0 (133.0 = 130). then make it to string to read it.
yubin_frame = yubin['yubin'] #post data
#1st put it to ing to get rid of float
yubin_list_int = map(int, yubin_list)
#then make it to string to in put all to string
yubin_list_str = map(str, yubin_list_int)
I made this for-loop to make list of both latitude and longitude like this.
#create a new list that include all data in Yubin_zahyou list
Yubin_zahyou = []
for i in range(len(yubin_list_str)):
Yubin_zahyou.append(geocoder.google(yubin_list_str[i]).latlng)
My problem is that i have nearly 30000 data and geocoder brings only nearly 2500 input!. Does this mean geocoder has a limit or I made a mistake somehow?

Yes, it has rate limit as written here in Providers.
https://github.com/DenisCarriere/geocoder
https://developers.google.com/maps/documentation/geocoding/usage-limits
as for google it nearly only gives 2500 limit per day.

elasticsearch-dsl aggregations returns only 10 results. How to change this

I am using elasticsearch-dsl python library to connect to elasticsearch and do aggregations.
I am following code
search.aggs.bucket('per_date', 'terms', field='date')\
.bucket('response_time_percentile', 'percentiles', field='total_time',
percents=percentiles, hdr={"number_of_significant_value_digits": 1})
response = search.execute()
This works fine but returns only 10 results in response.aggregations.per_ts.buckets
I want all the results
I have tried one solution with size=0 as mentioned in this question
search.aggs.bucket('per_ts', 'terms', field='ts', size=0)\
.bucket('response_time_percentile', 'percentiles', field='total_time',
percents=percentiles, hdr={"number_of_significant_value_digits": 1})
response = search.execute()
But this results in error
TransportError(400, u'parsing_exception', u'[terms] failed to parse field [size]')

I had the same issue. I finally found this solution:
s = Search(using=client, index="jokes").query("match", jks_content=keywords).extra(size=0)
a = A('terms', field='jks_title.keyword', size=999999)
s.aggs.bucket('by_title', a)
response = s.execute()
After 2.x, size=0 for all bucket results won't work anymore, please refer to this thread. Here in my example I just set the size equal 999999. You can pick a large number according to your case.
It is recommended to explicitly set reasonable value for size a number
between 1 to 2147483647.
Hope this helps.

This is a bit older but I ran into the same issue. What I wanted was basically an iterator that i could use to go through all aggregations that i got back (i also have a lot of unique results).
The best thing i found is to create a python generator like this
def scan_aggregation_results():
i=0
partitions=20
while i < partitions:
s = Search(using=elastic, index='my_index').extra(size=0)
agg = A('terms', field='my_field.keyword', size=999999,
include={"partition": i, "num_partitions": partitions})
s.aggs.bucket('my_agg', agg)
result = s.execute()
for item in result.aggregations.my_agg.buckets:
yield my_field.key
i = i + 1
# in other parts of the code just do
for item in scan_aggregation_results():
print(item) # or do whatever you want with it
The magic here is that elastic will automatically partition the number of results by 20, ie the number of partitions i define. I just have to set the size to something large enough to hold a single partition, in this case the result can be up to 20 million items large (or 20*999999). If you have much less items, like me, to return (like 20000) then you will just have 1000 results per query in your bucket, regardless that you defined a much larger size.
Using the generator construct as outlined above you can then even get rid of that and create your own scanner so to speak, iterating over all results individually, just what i wanted.

You should read the documentation.
So in your case, this should be like this :
search.aggs.bucket('per_date', 'terms', field='date')\
.bucket('response_time_percentile', 'percentiles', field='total_time',
percents=percentiles, hdr={"number_of_significant_value_digits": 1})[0:50]
response = search.execute()

Google's radarsearch API results

I'm trying to geolocate all the businesses related to a keyword in my city using, first, the radarsearch API in order to retrieve the Place ID and later using the Places API to get more information of each Place ID (such as the name, or the formatted address).
In my first approach I splitted my city in 9 circumferences, each one with radius 22km and avoiding rural zones, where there's no supposed to be a business. This way I obtained (once removing duplicated results, due to the circumferences overlapping) approximately 150 businesses. This result is not reliable because the official webpage of the company asserts there are 245.
In order to retrieve ALL the businesses, I split my city in circumferences of radius 10km. Therefore with approx 50 pairs of coordinates I fill the city, including now all zones, both rural and non-rural. Now, surprisingly I obtain only 81 businesses! How can this be possible?
I'm storing all the information in separated dictionaries and I noticed the amount of data of each dictionary increases with the increasing of the radius and is always the same (for a fixed radius).
Now, apart from the previous question, is there any way to limit the amount of results each request yields?
The code I'm using is the following:
dict1 = {}
radius=20000
keyword='keyworkd'
key=YOUR_API_KEY
url_base="https://maps.googleapis.com/maps/api/place/radarsearch/json?"
list_dicts = []
for i,(lo, la) in enumerate(zip(lon_txt,lat_txt)):
url=url_base+'location='+str(lo)+','+str(la)+'&radius='+str(radius)+'&keyword='+keyword+'&key='+key
response = urllib2.urlopen(url)
table = json.load(response)
if table['status']=='OK':
for j,line in enumerate(table['results']):
temp = {j : line['place_id']}
dict1.update(temp)
list_dicts.append(dict1)
else:
pass

Finally I managed to solve this problem.
The thing was the dict initialization must be done in each loop iteration. Now it stores all the information and I retrieve what I wanted from the beginning.
dict1 = {}
radius=20000
keyword='keyworkd'
key=YOUR_API_KEY
url_base="https://maps.googleapis.com/maps/api/place/radarsearch/json?"
list_dicts = []
for i,(lo, la) in enumerate(zip(lon_txt,lat_txt)):
url=url_base+'location='+str(lo)+','+str(la)+'&radius='+str(radius)+'&keyword='+keyword+'&key='+key
response = urllib2.urlopen(url)
table = json.load(response)
if table['status']=='OK':
for j,line in enumerate(table['results']):
temp = {j : line['place_id']}
dict1.update(temp)
list_dicts.append(dict1)
dict1 = {}
else:
pass