We Keep Coding

Search query keeps returning 0 for google maps API query

I'm trying to find the number of museums in each city in the UK by using the Google maps API. I keep getting a 0 search result with the following code. I thought it might be because I didn't enable billing on my Google Maps projects but I enabled billing and it still didn't work. Then I created a new API key and that didn't work either. Here is my code:
import requests
import json
api_key = ''
query = 'museums'
location = '51.509865,0.1276' # lat,lng of London
radius = 10000 # search radius in meters
url = f'https://maps.googleapis.com/maps/api/place/textsearch/json?query={query}&location={location}&radius={radius}&key={api_key}'
#url = f'https://maps.googleapis.com/maps/api/place/textsearch/json?query={query}&key={api_key}'
response = requests.get(url)
data = json.loads(response.text)
# retrieve the number of results
num_results = len(data['results'])
print(f'Number of results for "{query}" in "{location}": {num_results}')
I'm also open to trying a different method or package if that works.
And what it returns:
Number of results for "museum" in "51.509865,0.1276": 0

KeyError: 'name' in Spotify Web API

I'm using spotify's web API to get song information for a discord bot im making. I'm hosting the bot on heroku. im using the tracks option to get the track name and artist name from a songs ID. When the bot is on heroku, it throws the following error:
https://pastebin.com/smqqqDfY
however, when i host the same code on my laptop, it gives no such error. I even separated the spotify code to see if the JSON file has a key named 'name' and it works!
the code is:
#pulls the name and artist name from the API and link
def spotifypull(uri):
r = requests.get(spotify_base.format(id=uri), headers=headers)
r = r.json()
return (r['name']+" "+r['artists'][0]['name'])
#checks if the link is a spotify link(this is from the "request" function)
if query.find("spotify") !=-1:
uri = query[31:53]
name = spotifypull(uri)
this same code gives the proper output if separated locally
import requests
query = "https://open.spotify.com/track/6WkrFOo6SGAjhGMrjIwAD4?si=VDwYLniGQLGmqzUK3RdBow"
uri = query[31:53]
SPOTIFY_ID = "<id>"
SPOTIFY_SECRET = "<secret>"
AUTH_URL = 'https://accounts.spotify.com/api/token'
ytbase = "https://www.youtube.com/watch?v="
auth_response = requests.post(AUTH_URL, {
'grant_type': 'client_credentials',
'client_id': SPOTIFY_ID,
'client_secret': SPOTIFY_SECRET,
})
auth_response_data = auth_response.json()
access_token = auth_response_data['access_token']
headers = {
'Authorization': 'Bearer {token}'.format(token=access_token)
}
spotify_base = 'https://api.spotify.com/v1/tracks/{id}'
r = requests.get(spotify_base.format(id=uri), headers=headers)
r = r.json()
name = r['name']+" "+r['artists'][0]['name']
print(name)
output of above:
Wasn't Enough CrySpy
Any help would be massively appreciated! the full code is here if needed.
edit:
when ran locally,
r.text is https://pastebin.com/sjrW3exW
r.get_status is 200

Okay i got the issue, the spotify token was expiring after an hour. Thanks to #ygrorg for the idea.
Edit: Since some max brain mods want me to provide more clarity, Spotify tokens are valid for only an hour, I solved it by calling for a new token everytime a song is played. Alternatively, its only possible that you put the token regeneration commands in an infinite loop and put a delay of 30-45 min at the end of the loop, so you have a fresh token every time.

Python post request returning unexpected result all of the sudden

I ran my inventory-watching webscraper overnight, and all of the sudden, I started getting errors and I checked what the inventory post request was returning and it's this:
{"sec-cp-challenge": "true","provider":"crypto","branding_url_content":"/_sec/cp_challenge/crypto_message-2-9.htm","chlg_duration":7}
It's supposed to be a bunch of inventory data, but all of a sudden I'm getting this result. I'm not sure if it's because they've changed something, or if they've banned me or something, but when I go to the inventory source in the F12 menu, everything looks all good.
Here's the store page: https://www.basspro.com/shop/en/herters-hunting-rifle-ammo
and here's the inventory stock page: https://www.basspro.com/shop/BPSGetOnlineInventoryStatusByIDView
If anybody knows why I'm getting that weird crypto-related snippet, I would be indebted to you if you helped me out! I have a suspicion that they added a little waiting page that I'm getting rather than the inventory page that I want.
here's my code:
headers = {
'User-Agent': 'My User Agent 1.0',
'From': 'uadpjones#gmail.com',
}
store_url = 'https://www.basspro.com/shop/en/herters-target-handgun-ammo'
stock_url = 'https://www.basspro.com/shop/BPSGetOnlineInventoryStatusByIDView'
productId = 3074457345619161625
storeId = 715838534
catalogId = 3074457345616676768
while True:
try:
stock_page = requests.post(stock_url, headers=headers,
data={'productId': productId, 'storeId': storeId, 'catalogId': catalogId}).text
ammo_status = re.search(r'\n\t\t{\n\t\t\t"status": "((In|Out of) Stock)"',
stock_page, flags=re.S).group(1)
print('(9mm) {}'.format(ammo_status))
break
except:
print('Error: exception triggered (9mm)')
print(stock_page)
time.sleep(10)
pass

It is akamai's bot_chanllege_html.
The branding_url_content is the url of bot_chanllege_html.
This html while send sensor_data to the server.Then update cookies.So the server while besure you are not a bot.

Ran into the same issue, try updating the headers to include
referrer = URL
In your case this would be store_url

Keep getting Connection Reset Error 10054 when scraping Amazon jobs results

Obviously I'm still new to Python by looking at my code but failing my way through it.
I am scraping Amazon jobs search results but keep getting a connection reset error 10054 after about 50 requests to the url. I added a Crawlera proxy network to prevent getting banned but still not working. I know the url is long but it seems to work without having to add too many other separate parts to the url. The results page has about 12,000 jobs total with 10 jobs per page, so I don't even know if scraping that much data is the problem to begin with. Amazon shows each page in the url as 'result_limit=10', so I've been going through each page by 10s instead of 1 page per request. Not sure if that's right. Also, the last page stops at 9,990.
The code works but not sure how to get passed the connection error. As you can see, I've added things like a user agent but not sure if it even does anything. Any help would be appreciated as I've been stuck on this for countless days and hours. Thanks!
def get_all_jobs(pages):
requests = 0
start_time = time()
total_runtime = datetime.now()
for page in pages:
try:
ua = UserAgent()
header = {
'User-Agent': ua.random
}
response = get('https://www.amazon.jobs/en/search.json?base_query=&city=&country=USA&county=&'
'facets%5B%5D=location&facets%5B%5D=business_category&facets%5B%5D=category&'
'facets%5B%5D=schedule_type_id&facets%5B%5D=employee_class&facets%5B%5D=normalized_location'
'&facets%5B%5D=job_function_id&job_function_id%5B%5D=job_function_corporate_80rdb4&'
'latitude=&loc_group_id=&loc_query=USA&longitude=&'
'normalized_location%5B%5D=Seattle%2C+Washington%2C+USA&'
'normalized_location%5B%5D=San+Francisco'
'%2C+California%2C+USA&normalized_location%5B%5D=Sunnyvale%2C+California%2C+USA&'
'normalized_location%5B%5D=Bellevue%2C+Washington%2C+USA&'
'normalized_location%5B%5D=East+Palo+Alto%2C+California%2C+USA&'
'normalized_location%5B%5D=Santa+Monica%2C+California%2C+USA&offset={}&query_options=&'
'radius=24km&region=&result_limit=10&schedule_type_id%5B%5D=Full-Time&'
'sort=relevant'.format(page),
headers=header,
proxies={
"http": "http://1ea01axxxxxxxxxxxxxxxxxxx:#proxy.crawlera.com:8010/"
}
)
# Monitor the frequency of requests
requests += 1
# Pauses the loop between 8 and 15 seconds
sleep(randint(8, 15))
current_time = time()
elapsed_time = current_time - start_time
print("Amazon Request:{}; Frequency: {} request/s; Total Run Time: {}".format(requests,
requests / elapsed_time, datetime.now() - total_runtime))
clear_output(wait=True)
# Throw a warning for non-200 status codes
if response.status_code != 200:
warn("Request: {}; Status code: {}".format(requests, response.status_code))
# Break the loop if number of requests is greater than expected
if requests > 999:
warn("Number of requests was greater than expected.")
break
yield from get_job_infos(response)
except AttributeError as e:
print(e)
continue
def get_job_infos(response):
amazon_jobs = json.loads(response.text)
for website in amazon_jobs['jobs']:
site = website['company_name']
title = website['title']
location = website['normalized_location']
job_link = 'https://www.amazon.jobs' + website['job_path']
yield site, title, location, job_link
def main():
# Page range starts from 0 and the middle value increases by 10 each page.
pages = [str(i) for i in range(0, 9990, 10)]
with open('amazon_jobs.csv', "w", newline='', encoding="utf-8") as f:
writer = csv.writer(f)
writer.writerow(["Website", "Title", "Location", "Job URL"])
writer.writerows(get_all_jobs(pages))
if __name__ == "__main__":
main()

i'm not expert on amazon anti bot policies, but if they have flagged you once, your ip could be flagged for a while, they might have a limit to how many similar requests you can do in a certain time frame.
google for a patch to urllib so you can see the request headers in real time, other than ip/domain per certain time frame, amazon will look at your request headers to determine if you're not human. compare what you're sending with a regular browser request headers
just standard practice, keep cookies for a normal amount of time, use proper referers and a popular user agent
all this can be done with requests library, pip install requests, see session object
it looks like you're sending a request to an internal amazon url without a referer header..... that doesnt happen in a normal browser
another example, keeping cookies from one user agent and then switching to another is also not what browser does

Refresh page when new price is available in Bottle (Python)

I am making a Bitcoin/Ethereum price ticker webpage in Bottle for Python for my company and I want to refresh the page when new prices are available. I am pulling the price data from an API endpoint available through the company. I have hidden the URL to this for security purposes.
templates.py
from bottle import run, get, route, template
import requests
main_api = #url to company's api
def isDataValid(json_data):
if "status" in json_data:
return True
else:
return False
def returnPrices(coin, curr):
url = main_api + coin + curr
json_data = requests.get(url).json()
prices = {}
if isDataValid(json_data) == True:
buy_price = str(json_data["data"]["buy_price"])
sell_price = str(json_data["data"]["sell_price"])
prices = [buy_price, sell_price]
else:
prices = ["Error"]
return prices
#route('/')
def index():
pricesBTC = returnPrices('BTC','USD')
pricesETH = returnPrices('ETH','USD')
btc_buy_price = pricesBTC[0]
btc_sell_price = pricesBTC[1]
eth_buy_price = pricesETH[0]
eth_sell_price = pricesETH[1]
return template('index', btc_buy_price = btc_buy_price, btc_sell_price = btc_sell_price, eth_buy_price = eth_buy_price, eth_sell_price = eth_sell_price)
run(reLoader = True, debug = True)
So how do I refresh the page everytime prices change? I think the prices for ETH and BTC don't change at the same time, so I might have to refresh whenever either of them change. Thank you.

It is not possible without some browser-side JavaScript.
For example, you create additional Bottle endpoint that provides updated data as JSON, and an in-browser script polls that data via AJAX and update the respective html page elements.
As far as JS concerned, there are too many ways to implement this functionality, from simple JQuery to JS frameworks like Angular and Vue.js.