We Keep Coding

Why am I not getting a connection error when my API request fails on dropped wifi?

I am pulling data down from an API that has a limit of 250 records per call. There are a total of 100,000 records I need to pull down doing it 250 a time. I run my application leveraging the get_stats function below. It works fine for awhile but when my wifi drops and I am in the middle of the get request the request will hang and I won't get an exception back causing the rest of the application to hang as well.
I have tested turning off my wifi when the function is NOT in the middle of the get request and it does return back the ConnectionError exception.
How do I go about handling the situation where my app is in the middle of the get request and my wifi drops? I am thinking I need to do a timeout to give my wifi time to reconnect and then retry but how do I go about doing that? Or is there another way?
def get_stats(url, version):
headers = {
"API_version": version,
"API_token": "token"
}
try:
r = requests.get(url, headers=headers)
print(f"Status code: 200")
return json.loads(r.text)
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
print("Error here in timeout")
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
print("Redirect errors here")
except requests.exceptions.ConnectionError as r:
print("Connection error")
r = "Connection Error"
return r
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
print("System errors here")
raise SystemExit(e)

To set a timeout on the request, call requests.get like this
r = requests.get(url, headers=headers, timeout=10)
The end goal is to get the data, so just make the call again with a possible sleep after failing
edit: I would say that the timeout is the sleep

Getting HTTPError on API request

I recently started learning Python 3 and am trying to write my first program. The essence of the program is the auto-display of items on the trading floor. I use the API https://market.csgo.com/docs-v2. Everything would be fine, if not the errors that appear while the script is running. I know to use "TRY and EXECPT", but how to do it right? My code:
while True:
try:
ip = {'18992549780':'10000', '18992548863':'20000','18992547710':'30000','18992546824':'40000', '18992545927':'50000', '18992544515':'60000', '18992543504':'70000', '18992542365':'80000', '18992541028':'90000', '18992540218':'100000'}
for key,value in ip.items():
url3 = ('https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id={id}&price={price}&cur=RUB')
addtosale = url3.format(id = key, price = value)
onsale = requests.get(addtosale)
onsale.raise_for_status()
r = onsale.json()
print(addtosale)
print(onsale.raise_for_status)
print(r)
time.sleep(5)
except requests.HTTPError as exception:
print(exception)
My task is to run this piece of code between TRY and EXCEPT again on any error (5xx for example)
Traceback (most recent call last):
File "D:\Python\tmsolve1.py", line 30, in <module>
onsale.raise_for_status()
File "C:\Users\���������\AppData\Local\Programs\Python\Python38-32\lib\site-packages\requests\models.py", line 941, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id=18992545927&price=50000&cur=RUB
502 Server Error: Bad Gateway for url: https://market.csgo.com/api/v2/add-to-sale?key=MYAPIKEY&id=18992549780&price=10000&cur=RUB

Error handling can be done in multiple ways. You have 10 API calls. You can either stop the code on first error, retry the request or continue with additional call.
The example below will continue running through all requests.
Also except requests.HTTPError as exception may not be needed. This error is thrown by response.raise_for_status(). You can preform logging before calling .raise_for_status(). The try/catch only allows the code to continue in the loop.
import requests
import time
import json
# while True: # This will make the code loop continuously
try:
ip = {'18992549780':'10000', '18992548863':'20000','18992547710':'30000','18992546824':'40000', '18992545927':'50000', '18992544515':'60000', '18992543504':'70000', '18992542365':'80000', '18992541028':'90000', '18992540218':'100000'}
for key,value in ip.items():
url= 'https://market.csgo.com/api/v2/add-to-sale'
payload = {'key': 'MYAPIKEY', 'id': id, 'price': value, 'cur': 'RUB'}
response = requests.get(url, params=payload)
print(f'Status code: { response.status_code}')
print(f'Response text: { response.text}') # This will contain an error message or json results.
response.raise_for_status() # This will only error if status code is 4xx or 5xx
results = response.json()
if results.get('error'): # "results" can contains {"error":"Bad KEY","success":false}
raise Exception('Error in response json')
print(json.dumps(results))
time.sleep(5)
except requests.HTTPError as exception: # Captures response.raise_for_status() - 4xx or 5xx status code. If you remove this, then code will use generic handle
print(exception)
except Exception as exception: # Generic error handler for raise Exception('Error in response json') and "Max retries exceeded."
print(exception)

Capturing Response Body for a HTTP Error in python

Need to capture the response body for a HTTP error in python. Currently using the python request module's raise_for_status(). This method only returns the Status Code and description. Need a way to capture the response body for a detailed error log.
Please suggest alternatives to python requests module if similar required feature is present in some different module. If not then please suggest what changes can be done to existing code to capture the said response body.
Current implementation contains just the following:
resp.raise_for_status()

I guess I'll write this up quickly. This is working fine for me:
try:
r = requests.get('https://www.google.com/404')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err.request.url)
print(err)
print(err.response.text)

you can do something like below, which returns the content of the response, in unicode.
response.text
or
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
sys.exit(1)
# 404 Client Error: Not Found for url: http://www.google.com/nothere
here you'll get the full explanation on how to handle the exception. please check out Correct way to try/except using Python requests module?

You can log resp.text if resp.status_code >= 400.

There are some tools you may pick up such as Fiddler, Charles, wireshark.
However, those tools can just display the body of the response without including the reason or error stack why the error raises.

Python HTTP Error 429 with urllib2

I am using the following code to resolve redirects to return a links final url
def resolve_redirects(url):
return urllib2.urlopen(url).geturl()
Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
return urllib2.urlopen(url).geturl()
Also, what would happen if there is an exception in the except block?

It would be better to make sure the HTTP code is actually 429 before re-trying.
That can be done like this:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError, e:
if e.code == 429:
time.sleep(5);
return resolve_redirects(url)
raise
This will also allow arbitrary numbers of retries (which may or may not be desired).
https://docs.python.org/2/howto/urllib2.html#httperror

This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.
To recover from an exception within an exception, you can simply embed another try/catch block:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
return "Failed twice :S"
Edit: as #jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.

Adding User-Agent to request header solved my issue:
from urllib import request
from urllib.request import urlopen
url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)

Python requests exception handling

How to handle exceptions with python library requests?
For example how to check is PC connected to internet?
When I try
try:
requests.get('http://www.google.com')
except ConnectionError:
# handle the exception
it gives me error name ConnectionError is not defined

Assuming you did import requests, you want requests.ConnectionError. ConnectionError is an exception defined by requests. See the API documentation here.
Thus the code should be:
try:
requests.get('http://www.google.com')
except requests.ConnectionError:
# handle the exception
The original link to the Python v2 API documentation from the original answer no longer works.

As per the documentation, I have added the below points:
In the event of a network problem (refused connection e.g internet issue), Requests will raise a ConnectionError exception.
try:
requests.get('http://www.google.com')
except requests.ConnectionError:
# handle ConnectionError the exception
In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.
Response.raise_for_status() will raise an HTTPError if the HTTP request returned an unsuccessful status code.
try:
r = requests.get('http://www.google.com/nowhere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
#handle the HTTPError request here
In the event of times out of request, a Timeout exception is raised.
You can tell Requests to stop waiting for a response after a given number of seconds, with a timeout arg.
requests.get('https://github.com/', timeout=0.001)
# timeout is not a time limit on the entire response download; rather,
# an exception is raised if the server has not issued a response for
# timeout seconds
All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException. So a base handler can look like,
try:
r = requests.get(url)
except requests.exceptions.RequestException as e:
# handle all the errors here
The original link to the Python v2 documentation no longer works, and now points to the new documentation.

Actually, there are much more exceptions that requests.get() can generate than just ConnectionError. Here are some I've seen in production:
from requests import ReadTimeout, ConnectTimeout, HTTPError, Timeout, ConnectionError
try:
r = requests.get(url, timeout=6.0)
except (ConnectTimeout, HTTPError, ReadTimeout, Timeout, ConnectionError):
continue

Include the requests module using import requests .
It is always good to implement exception handling. It does not only help to avoid unexpected exit of script but can also help to log errors and info notification. When using Python requests I prefer to catch exceptions like this:
try:
res = requests.get(adress,timeout=30)
except requests.ConnectionError as e:
print("OOPS!! Connection Error. Make sure you are connected to Internet. Technical Details given below.\n")
print(str(e))
continue
except requests.Timeout as e:
print("OOPS!! Timeout Error")
print(str(e))
continue
except requests.RequestException as e:
print("OOPS!! General Error")
print(str(e))
continue
except KeyboardInterrupt:
print("Someone closed the program")

for clarity, that is
except requests.ConnectionError:
NOT
import requests.ConnectionError
You can also catch a general exception (although this isn't recommended) with
except Exception: