We Keep Coding

Python requests.post fails in 421

I have the following code in the implementation (uses requests lib):
def call_service(product_ids: Set[str])
response = requests.post(
json={"product_ids": product_ids},
url="http://whatever.com",
)
if response.status_code == 421:
raise MisRedirectedRequest()
I'm using HTTPretty's to simulate this in a test:
#httpretty.activate
def test_http_status_code_421():
httpretty.register_uri(
method=httpretty.POST,
uri="http://whatever.com",
body="{}",
status=421,
)
with pytest.raises(MisRedirectedRequest):
call_service({"123", "543"})
However, MisRedirectedRequest is never raised. The test doesn't pass.
By debugging, I get this in the implementation:
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
The weird thing is that I tried with other HTTP error codes and it works fine (e.g. 420, 500).

After hours of debugging, I discovered that HTTPretty did not support 421.
I created a PR but meanwhile, this is the workaround:
from httpretty.http import STATUSES
STATUSES[421] = "Misdirected Request"
Update: PR is merged.

Capturing Response Body for a HTTP Error in python

Need to capture the response body for a HTTP error in python. Currently using the python request module's raise_for_status(). This method only returns the Status Code and description. Need a way to capture the response body for a detailed error log.
Please suggest alternatives to python requests module if similar required feature is present in some different module. If not then please suggest what changes can be done to existing code to capture the said response body.
Current implementation contains just the following:
resp.raise_for_status()

I guess I'll write this up quickly. This is working fine for me:
try:
r = requests.get('https://www.google.com/404')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err.request.url)
print(err)
print(err.response.text)

you can do something like below, which returns the content of the response, in unicode.
response.text
or
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
print(err)
sys.exit(1)
# 404 Client Error: Not Found for url: http://www.google.com/nothere
here you'll get the full explanation on how to handle the exception. please check out Correct way to try/except using Python requests module?

You can log resp.text if resp.status_code >= 400.

There are some tools you may pick up such as Fiddler, Charles, wireshark.
However, those tools can just display the body of the response without including the reason or error stack why the error raises.

Django SuspiciousOperation returns as HTTP 500 on Google App Engine, not HTTP 400 as expected

Using Python 2.7, Django on Google App Engine. I'm trying to do some simple URL checking, including checking a JSON data payload, and return a meaningful error to the user. What I have coded is basically this:
from django.core.exceptions import SuspiciousOperation
...
def check(self, request):
json_data = json.loads(request.body)
if not json_data:
raise SuspiciousOperation('Required JSON data not found in the POST request.')
...
But, when I test this in debug mode (DEBUG = True in settings.py) by omitting the JSON data, instead of returning a HTTP 400 as I expect from SuspiciousOperation, I get an HTTP 500 that contains my error message "Required JSON data not found in the POST request." The same thing occurs if I check for a valiud URL with URLValidator(): I can correctly test for a good or bad URL with the URLValidator(), but if I try to raise a custom message on a bad URL with SuspiciousOperation I get HTTP 500 instead of 400.
How can I return a meaningful error to my caller without the server error obfuscating everything when Debug is turned back off and crashing the process in the process? Is SuspiciousOperation not supported by GAE?

There was an issue raised about this on Django's bug tracker and it looks like it was fixed in 1.6 but not backported. Indeed, SuspiciousOperation is handled by a catch-all in 1.5.11 (django/django/core/handlers/base.py line 173):
except: # Handle everything else, including SuspiciousOperation, etc.
# Get the exception info now, in case another exception is thrown later.
signals.got_request_exception.send(sender=self.__class__, request=request)
response = self.handle_uncaught_exception(request, resolver, sys.exc_info())

Python HTTP Error 429 with urllib2

I am using the following code to resolve redirects to return a links final url
def resolve_redirects(url):
return urllib2.urlopen(url).geturl()
Unfortunately I sometimes get HTTPError: HTTP Error 429: Too Many Requests. What is a good way to combat this? Is the following good or is there a better way.
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
return urllib2.urlopen(url).geturl()
Also, what would happen if there is an exception in the except block?

It would be better to make sure the HTTP code is actually 429 before re-trying.
That can be done like this:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError, e:
if e.code == 429:
time.sleep(5);
return resolve_redirects(url)
raise
This will also allow arbitrary numbers of retries (which may or may not be desired).
https://docs.python.org/2/howto/urllib2.html#httperror

This is a fine way to handle the exception, though you should check to make sure you are always sleeping for the appropriate amount of time between requests for the given website (for example twitter limits the amount of requests per minute and has this amount clearly shown in their api documentation). So just make sure you're always sleeping long enough.
To recover from an exception within an exception, you can simply embed another try/catch block:
def resolve_redirects(url):
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
time.sleep(5)
try:
return urllib2.urlopen(url).geturl()
except HTTPError:
return "Failed twice :S"
Edit: as #jesse-w-at-z points out, you should be returning an URL in the second error case, the code I posted is just a reference example of how to write a nested try/catch.

Adding User-Agent to request header solved my issue:
from urllib import request
from urllib.request import urlopen
url = 'https://www.example.com/abc.json'
req = request.Request(url)
req.add_header('User-Agent', 'abc-bot')
response = request.urlopen(req)

Why do I get two different status code from conn.getresponse().status in python?

so I want to check if a URL is reachable from python, and I got this code from googling:
def checkUrl(url):
p = urlparse(url)
conn = http.client.HTTPConnection(p.netloc)
conn.request('HEAD', p.path)
resp = conn.getresponse()
return resp.status < 400
Here is my URL: https://eurotableau.nomisonline.com.
It works fine if I just pass that in to the function. The resp.status is 302. However, if I add a port 443 at the end of it, https://eurotableau.nomisonline.com:443, it returns false. The resp.status is 400. I tried both URL in google Chrome, both of them work. So my question is why is this happening? Anyway I can include the port value and still get valid resp.status value (< 400)? Thanks.

Use http.client.HTTPSConnection instead. The plain old HTTPConnection ignores the protocol that is part of the URL.

If you do not require the HEAD method but just wish to check if host is available then why not do:
from urllib2 import urlopen
try:
u = urlopen("https://eurotableau.nomisonline.com")
u.close()
print "Everything fine!"
except Exception, e:
if hasattr(e, "code"):
print "Server is there but something is wrong with rest of URL"
else: print "Server is on vacations or was never there!"
print e
This will establish a connection with server but it won't download any data unless you read it. It'll only read few KB to get the header (like when using HEAD method) and wait for you to request more. But you will close it there.
So, you can catch an exception and see what the problem is, or if there is no exception, just close the connection.
urllib2 will handle HTTPS and protocol://user#URL:PORT for you neatly.
No worries about anything.