I recently started learning Python 3 and am trying to write my first program. The essence of the program is the auto-display of items on the trading floor. I use the API https://market.csgo.com/docs-v2. Everything would be fine, if not the errors that appear while the script is running. I know to use "TRY and EXECPT", but how to do it right? My code:
while True:
try:
ip = {'18992549780':'10000', '18992548863':'20000','18992547710':'30000','18992546824':'40000', '18992545927':'50000', '18992544515':'60000', '18992543504':'70000', '18992542365':'80000', '18992541028':'90000', '18992540218':'100000'}
for key,value in ip.items():
url3 = ('https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id={id}&price={price}&cur=RUB')
addtosale = url3.format(id = key, price = value)
onsale = requests.get(addtosale)
onsale.raise_for_status()
r = onsale.json()
print(addtosale)
print(onsale.raise_for_status)
print(r)
time.sleep(5)
except requests.HTTPError as exception:
print(exception)
My task is to run this piece of code between TRY and EXCEPT again on any error (5xx for example)
Traceback (most recent call last):
File "D:\Python\tmsolve1.py", line 30, in <module>
onsale.raise_for_status()
File "C:\Users\���������\AppData\Local\Programs\Python\Python38-32\lib\site-packages\requests\models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id=18992545927&price=50000&cur=RUB
502 Server Error: Bad Gateway for url: https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id=18992549780&price=10000&cur=RUB
Error handling can be done in multiple ways. You have 10 API calls. You can either stop the code on first error, retry the request or continue with additional call.
The example below will continue running through all requests.
Also except requests.HTTPError as exception may not be needed. This error is thrown by response.raise_for_status(). You can preform logging before calling .raise_for_status(). The try/catch only allows the code to continue in the loop.
import requests
import time
import json
# while True: # This will make the code loop continuously
try:
ip = {'18992549780':'10000', '18992548863':'20000','18992547710':'30000','18992546824':'40000', '18992545927':'50000', '18992544515':'60000', '18992543504':'70000', '18992542365':'80000', '18992541028':'90000', '18992540218':'100000'}
for key,value in ip.items():
url= 'https://market.csgo.com/api/v2/add-to-sale'
payload = {'key': 'MYAPIKEY', 'id': id, 'price': value, 'cur': 'RUB'}
response = requests.get(url, params=payload)
print(f'Status code: { response.status_code}')
print(f'Response text: { response.text}') # This will contain an error message or json results.
response.raise_for_status() # This will only error if status code is 4xx or 5xx
results = response.json()
if results.get('error'): # "results" can contains {"error":"Bad KEY","success":false}
raise Exception('Error in response json')
print(json.dumps(results))
time.sleep(5)
except requests.HTTPError as exception: # Captures response.raise_for_status() - 4xx or 5xx status code. If you remove this, then code will use generic handle
print(exception)
except Exception as exception: # Generic error handler for raise Exception('Error in response json') and "Max retries exceeded."
print(exception)
Related
I am pulling data down from an API that has a limit of 250 records per call. There are a total of 100,000 records I need to pull down doing it 250 a time. I run my application leveraging the get_stats function below. It works fine for awhile but when my wifi drops and I am in the middle of the get request the request will hang and I won't get an exception back causing the rest of the application to hang as well.
I have tested turning off my wifi when the function is NOT in the middle of the get request and it does return back the ConnectionError exception.
How do I go about handling the situation where my app is in the middle of the get request and my wifi drops? I am thinking I need to do a timeout to give my wifi time to reconnect and then retry but how do I go about doing that? Or is there another way?
def get_stats(url, version):
headers = {
"API_version": version,
"API_token": "token"
}
try:
r = requests.get(url, headers=headers)
print(f"Status code: 200")
return json.loads(r.text)
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
print("Error here in timeout")
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
print("Redirect errors here")
except requests.exceptions.ConnectionError as r:
print("Connection error")
r = "Connection Error"
return r
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
print("System errors here")
raise SystemExit(e)
To set a timeout on the request, call requests.get like this
r = requests.get(url, headers=headers, timeout=10)
The end goal is to get the data, so just make the call again with a possible sleep after failing
edit: I would say that the timeout is the sleep
Background and Code
I have the below function to handle rate limiting in Twitter's V2 API based on the HTTP status codes.
from datetime import datetime
from osometweet.utils import pause_until
def manage_rate_limits(response):
"""Manage Twitter V2 Rate Limits
This method takes in a `requests` response object after querying
Twitter and uses the headers["x-rate-limit-remaining"] and
headers["x-rate-limit-reset"] headers objects to manage Twitter's
most common, time-dependent HTTP errors.
Wiki Reference: https://github.com/osome-iu/osometweet/wiki/Info:-HTTP-Status-Codes-and-Errors
Twitter Reference: https://developer.twitter.com/en/support/twitter-api/error-troubleshooting
"""
while True:
# The x-rate-limit-remaining parameter is not always present.
# If it is, we want to use it.
try:
# Get number of requests left with our tokens
remaining_requests = int(response.headers["x-rate-limit-remaining"])
# If that number is one, we get the reset-time
# and wait until then, plus 15 seconds (your welcome Twitter).
# The regular 429 exception is caught below as well,
# however, we want to program defensively, where possible.
if remaining_requests == 1:
buffer_wait_time = 15
resume_time = datetime.fromtimestamp( int(response.headers["x-rate-limit-reset"]) + buffer_wait_time )
print(f"One request from being rate limited. Waiting on Twitter.\n\tResume Time: {resume_time}")
pause_until(resume_time)
except Exception as e:
print("An x-rate-limit-* parameter is likely missing...")
print(e)
# Explicitly checking for time dependent errors.
# Most of these errors can be solved simply by waiting
# a little while and pinging Twitter again - so that's what we do.
if response.status_code != 200:
# Too many requests error
if response.status_code == 429:
buffer_wait_time = 15
resume_time = datetime.fromtimestamp( int(response.headers["x-rate-limit-reset"]) + buffer_wait_time )
print(f"Too many requests. Waiting on Twitter.\n\tResume Time: {resume_time}")
pause_until(resume_time)
# Twitter internal server error
elif response.status_code == 500:
# Twitter needs a break, so we wait 30 seconds
resume_time = datetime.now().timestamp() + 30
print(f"Internal server error # Twitter. Giving Twitter a break...\n\tResume Time: {resume_time}")
pause_until(resume_time)
# Twitter service unavailable error
elif response.status_code == 503:
# Twitter needs a break, so we wait 30 seconds
resume_time = datetime.now().timestamp() + 30
print(f"Twitter service unavailable. Giving Twitter a break...\n\tResume Time: {resume_time}")
pause_until(resume_time)
# If we get this far, we've done something wrong and should exit
raise Exception(
"Request returned an error: {} {}".format(
response.status_code, response.text
)
)
# Each time we get a 200 response, exit the function and return the response object
if response.ok:
return response
This function is fed a response object from a requests call like the below
response = requests.get(
url,
headers=self._header,
params=payload
)
response = manage_rate_limits(response)
In the above response call the parameters are the following:
where
url = Twitter's base endpoint URL (in this case it is the full archive academic search)
params/payload = a combination of endpoint search operators (these should be irrelevant but I can include if necessary)
headers/self._bearer_token is a user bearer_token in the below proper header form
self._header = {"Authorization": f"Bearer {MY_BEARER_TOKEN}"}
Question & Error:
Using the above code, I get a long-running script that returns the below error from the rate_limit_manager function.
Traceback (most recent call last):
File "/scratch/mdeverna/Superspreaders/src/get_rts_of_user.py", line 218, in get_rts_of_user
full_archive_search = True
File "/nfs/nfs5/home/scratch/mdeverna/osometweet/osometweet/api.py", line 248, in search
response = self._oauth.make_request(url, payload)
File "/nfs/nfs5/home/scratch/mdeverna/osometweet/osometweet/oauth.py", line 181, in make_request
response = manage_rate_limits(response)
File "/nfs/nfs5/home/scratch/mdeverna/osometweet/osometweet/rate_limit_manager.py", line 67, in manage_rate_limits
response.status_code, response.text
Exception: Request returned an error: 429 {"title":"Too Many Requests","type":"about:blank","status":429,"detail":"Too Many Requests"}
What I don't understand is that the line that prints this exception is...
# If we get this far, we've done something wrong and should exit
raise Exception(
"Request returned an error: {} {}".format(
response.status_code, response.text
)
... and this illustrates the response.status_code prints (equals) 429, however, the conditional earlier in this function checks for exactly this status code but seems to miss it. It seems like the condition which checks if the status code = 429 is being skipped, only to print down below that the status code is 429?
What is going on here?
Even if the status code is 429 or 500 or 503, you're going to flow off the bottom of the if/elif/elif sequence and right into the raise. Did you intend to return at the end of each? Or did you mean for the raise to be in an else: clause?
When I run code locally and try to fetch data from URL and then parse it to text everything work properly.
When I run exactly the same code on the remote server and try to fetch data from URL error HTTP Error 403: Forbidden occur
Answers from questions:
HTTP error 403 in Python 3 Web Scraping,
urllib2.HTTPError: HTTP Error 403: Forbidden helped me when I tried to run it locally and everything work fine.
Do you know what can be different in fetching data from remote server while code is the same(locally and on the server) and way of running code is the same but result is absolutely different?
URL that I want to fetch:
url=https://bithumb.cafe/notice
Code that I was trying to use to fetch data(once it work, second not)
try:
request = urllib.request.Request(url)
request.add_header('User-Agent', 'cheese')
logger.info("request: {}".format(request))
content = urllib.request.urlopen(request).read()
logger.info('content: {}'.format(content))
decoded = content.decode('utf-8')
logger.info('content_decoded: {}'.format(decoded))
return decoded
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''`
second way of fetching data(also work locally but on the remote server not):
class AppURLopener(urllib.request.FancyURLopener):
version = "Mozilla/5.0"
method:
try:
opener = AppURLopener()
response = opener.open(url)
logger.info("request response: {}. response type: {}. response_dict: {}"
.format(response, type(response), response.__dict__))
html_response = response.read()
logger.info("html_Response".format(html_response))
encoding = response.headers.get_content_charset('utf-8')
decoded_html = html_response.decode(encoding)
logger.info('content_decoded: {}'.format(decoded_html))
return decoded_html
except Exception as e:
logger.error('failed with error message: {}'.format(e))
return ''
Making a request to the server, as in code below, I've got status code 500, which was not caught as an exception. The output was "500", but I need for all 500 codes to result in sys.exit(). Does requests.exceptions.RequestException not treat 500 as an exception or is it something else? The requests module docs http://docs.python-requests.org/en/latest/user/quickstart/#errors-and-exceptions are not very clear on what falls under this class. How do I make sure that all 500 codes result in sys.exit()?
import requests
import json
import sys
url = http://www.XXXXXXXX.com
headers = {'user':'me'}
try:
r = requests.post(url, headers=headers)
status = r.status_code
response = json.dumps(r.json(), sort_keys=True, separators=(',', ': '))
print status
except requests.exceptions.RequestException as e:
print "- ERROR - Web service exception, msg = {}".format(e)
if r.status_code < 500:
print r.status_code
else:
sys.exit(-1)
A status code 500 is not an exception. There was a server error when processing the request and the server returned a 500; more of a problem with the server than the request.
You can therefore do away with the try-except:
r = requests.post(url, headers=headers)
status = r.status_code
response = json.dumps(r.json(), sort_keys=True, separators=(',', ': '))
if str(status).startswith('5'):
...
From the Requests documentation:
If we made a bad request (a 4XX client error or 5XX server error
response), we can raise it with Response.raise_for_status():
>>> bad_r = requests.get('http://httpbin.org/status/404')
>>> bad_r.status_code
404
>>> bad_r.raise_for_status()
Traceback (most recent call last):
File "requests/models.py", line 832, in raise_for_status
raise http_error
requests.exceptions.HTTPError: 404 Client Error
So, use
r = requests.post(url, headers=headers)
try:
r.raise_for_status()
except requests.exceptions.HTTPError:
# Gave a 500 or 404
else:
# Move on with your life! Yay!
If you want a successful request, but "non-OK" response to raise an error, call response.raise_for_status(). You can then catch that error and handle it appropriately. It will raise a requests.exceptions.HTTPError that has the response object hung onto the error.
In python, when a http request is invalid, response is None, in this case, how to get the response code from the response? The invalid request in my code are caused by two reasons, one is a invalid token, I expect to get 401 in this case, another reason is invalid parameter, I expect to get 400 in this case, but under both cases, response is always None and I'm not able to get the response code by calling response.getcode(), how to solve this?
req = urllib2.Request(url)
response = None
try: response = urllib2.urlopen(req)
except urllib2.URLError as e:
res_code = response.getcode() #AttributeError: 'NoneType' object has no attribute 'getcode'
You can't get the status code when URLError is raised. Because when it is raised (ex: DNS couldn't resolve domain name), it means request hasn't been sent to server yet so there is no HTTP response generated.
In your scenario, (for 4xx HTTP status code), urllib2 throws HTTPError so you can derive the status code from it.
The documentation says:
code
An HTTP status code as defined in RFC 2616. This numeric value corresponds to a value found in the dictionary of codes as found in BaseHTTPServer.BaseHTTPRequestHandler.responses.
import urllib2
request = urllib2.Request(url)
try:
response = urllib2.urlopen(request)
res_code = response.code
except urllib2.HTTPError as e:
res_code = e.code
Hope this helps.