I have some code which invokes a HTTP request and I would like to unit test a negative case where it should raise a specific exception for a 404 response. However I am trying to figure out how to mock the parameter so it can raise the HTTPError as a side-effect in the calling function, the mock object seems to create an invokable function which isn't the parameter that it accepts, it is only a scalar value.
def scrape(variant_url):
try:
with urlopen(variant_url) as response:
doc = response.read()
sizes = scrape_sizes(doc)
price = scrape_price(doc)
return VariantInfo([], sizes, [], price)
except HTTPError as e:
if e.code == 404:
raise LookupError('Variant not found!')
raise e
def test_scrape_negative(self):
with self.assertRaises(LookupError):
scrape('foo')
Mock the urlopen() to raise an exception; you can do this by setting the side_effect attribute of the mock:
with mock.patch('urlopen') as urlopen_mock:
urlopen_mock.side_effect = HTTPError('url', 404, 'msg', None, None)
with self.assertRaises(LookupError):
scrape('foo')
Related
I have the below flask code :
from flask import Flask,request,jsonify
import requests
from werkzeug.exceptions import InternalServerError, NotFound
import sys
import json
app = Flask(__name__)
app.config['SECRET_KEY'] = "Secret!"
class InvalidUsage(Exception):
status_code = 400
def __init__(self, message, status_code=None, payload=None):
Exception.__init__(self)
self.message = message
if status_code is not None:
self.status_code = status_code
self.payload = payload
def to_dict(self):
rv = dict(self.payload or ())
rv['message'] = self.message
rv['status_code'] = self.status_code
return rv
#app.errorhandler(InvalidUsage)
def handle_invalid_usage(error):
response = jsonify(error.to_dict())
response.status_code = error.status_code
return response
#app.route('/test',methods=["GET","POST"])
def test():
url = "https://httpbin.org/status/404"
try:
response = requests.get(url)
if response.status_code != 200:
try:
response.raise_for_status()
except requests.exceptions.HTTPError:
status = response.status_code
print status
raise InvalidUsage("An HTTP exception has been raised",status_code=status)
except requests.exceptions.RequestException as e:
print e
if __name__ == "__main__":
app.run(debug=True)
My question is how do i get the exception string(message) and other relevant params from the requests.exceptions.RequestException object e ?
Also what is the best way to log such exceptions . In case of an HTTPError exceptions i have the status code to refer to.
But requests.exceptions.RequestException catches all request exceptions . So how do i differentiate between them and also what is the best way to log them apart from using print statements.
Thanks a lot in advance for any answers.
RequestException is a base class for HTTPError, ConnectionError, Timeout, URLRequired, TooManyRedirects and others (the whole list is available at the GitHub page of requests module). Seems that the best way of dealing with each error and printing the corresponding information is by handling them starting from more specific and finishing with the most general one (the base class). This has been elaborated widely in the comments in this StackOverflow topic. For your test() method this could be:
#app.route('/test',methods=["GET","POST"])
def test():
url = "https://httpbin.org/status/404"
try:
# some code...
except requests.exceptions.ConnectionError as ece:
print("Connection Error:", ece)
except requests.exceptions.Timeout as et:
print("Timeout Error:", et)
except requests.exceptions.RequestException as e:
print("Some Ambiguous Exception:", e)
This way you can firstly catch the errors that inherit from the RequestException class and which are more specific.
And considering an alternative for printing statements - I'm not sure if that's exactly what you meant, but you can log into console or to a file with standard Python logging in Flask or with the logging module itself (here for Python 3).
This is actually not a question about using the requests library as much as it is a general Python question about how to extract the error string from an exception instance. The answer is relatively straightforward: you convert it to a string by calling str() on the exception instance. Any properly written exception handler (in requests or otherwise) would have implemented an __str__() method to allow an str() call on an instance. Example below:
import requests
rsp = requests.get('https://httpbin.org/status/404')
try:
if rsp.status_code >= 400:
rsp.raise_for_status()
except requests.exceptions.RequestException as e:
error_str = str(e)
# log 'error_str' to disk, a database, etc.
print('The error was:', error_str)
Yes, in this example, we print it, but once you have the string you have additional options. Anyway, saving this to test.py results in the following output given your test URL:
$ python3 test.py
The error was: 404 Client Error: NOT FOUND for url: https://httpbin.org/status/404
At the service level of my application, I am raising an exception and I want it to be printed as JSON to the browser.
I implemented it as stated in the documentation:
raise falcon.HTTPError(
'12345 - My Custom Error',
'some text'
).to_json()
And the output from the console:
TypeError: exceptions must derive from BaseException
Anybody had this issue before and could help me with this one?
You're trying to raise a string. The correct way to do that is with set_error_serializer().
The example from the documentation seems like exactly what you need (plus YAML support).
def my_serializer(req, resp, exception):
representation = None
preferred = req.client_prefers(('application/x-yaml',
'application/json'))
if preferred is not None:
if preferred == 'application/json':
representation = exception.to_json()
else:
representation = yaml.dump(exception.to_dict(),
encoding=None)
resp.body = representation
resp.content_type = preferred
resp.append_header('Vary', 'Accept')
app = falcon.API()
app.set_error_serializer(my_serializer)
Create custom exception class explained in falcon docs, search for add_error_handler
class RaiseUnauthorizedException(Exception):
def handle(ex, req, resp, params):
resp.status = falcon.HTTP_401
response = json.loads(json.dumps(ast.literal_eval(str(ex))))
resp.body = json.dumps(response)
Add custom exception class to falcon API object
api = falcon.API()
api.add_error_handler(RaiseUnauthorizedException)
import Custom exception class and pass your message
message = {"status": "error", "message" : "Not authorized"}
RaiseUnauthorizedException(message)
I have class which called requests method using getattr like this:
import requests
class CustomRequests(object):
def __init__(self):
pass
def _do_requests(self, method='GET', url='', expected_status=200):
make_request = getattr(requests, method.lower())
url = url if url else 'http://example.com'
try:
response = make_request(method, url=url)
except response.exceptions.RequestException as exception:
raise exception
if response.status_code != expected_status:
raise ValueError
def get(self, *args, **kwargs):
self._do_requests(method='GET', *args, **kwargs)
I am trying to test the api using mock and responses lib like this:
import responses
#responses.activate
def test_get_method(self):
responses.add('GET', url='http://test_this_api.com', status=200)
custom_request = CustomRequest()
response_data = custom_request.get(method='GET')
AssertIsNotNone(response_data)
Is there any better or right way to test this method.
Getting this error:
message = message.format(**values)
KeyError: 'method'
There's no need to use getattr. requests.get, requests.post, etc. are just convenience methods for requests.request, which lets you pass the HTTP method as a parameter:
requests.request('GET', url) # equivalent to requests.get(url)
Also:
Your try/except is pointless, since all you do is re-raise the exception.
It doesn't make sense to raise a ValueError when the response status doesn't match what you expected. ValueError is for "when a built-in operation or function receives an argument that has the right type but an inappropriate value." Create your own exception class, e.g. UnexpectedHTTPStatusError.
Since the whole point of your CustomRequests class seems to be to raise an exception when the status code of the response doesn't match what the user expected, your tests should assert that an exception was actually raised with assertRaises().
In some python code, I am using a library to wrap requests to a web service. The behaviour I intend is that any HTTPErrors have their content output with a logging.error along with the status code, and the error passed along:
def my_request_thing(api, other_stuff):
request = make_request_from(api, other_stuff)
try:
with closing(urllib2.urlopen(request)) as fd:
return fd.read()
except HTTPError as e:
logging.error("Error from server: %s\n%s", e.code, e.read())
raise
This code will log, and pass the error along, with one problem, the exceptions content is exhausted in e.read. This code is intended to be used to most clients to the API substituting things like root paths and http headers...
I may then have another function for more domain specific stuff using this:
function get_my_thing(thing_id, conditions):
try:
return json.loads(my_request_thing(<thing_id + conditions into api and stuff...>))
except HTTPError as e:
if e.code == 404 and "my thing does not exist" in e.read():
return False
else:
raise e
Note here that this also tries to get data with e.read - which is now empty, and may still reraise the error. This will fail to work - there is not data in e.read here.
Is there a good way to reraise this exception such that the content is not exhausted, but so I can sniff out particular exception types and log them all on the way?
As per Karel Kubat comment, why don't you inject the results from e.read() into the exception as a data member upon seeing it for the first time?
For example, derive your own error class from HTTPError with an empty self.content. When catching an exception for the first time, fill self.content from self.read(). Next handlers can inspect e.content.
In a script I am creating I am posting a lot of data to a REST API.
The script is quite modularized and at the top level somewhere I am catching a URLError. I need to know what is in the body of the response, because there will be an error message in there.
Is there a method on URLError that I can use?
try:
(calling some function that throws URLError)
except urllib2.URLError, e:
print "Error: " + str(e.body_or_something)
Yes there is. You have an access to the response via e.readlines():
try:
(calling some function that throws URLError)
except urllib2.URLError, e:
print e.readlines()
See the documenet: https://docs.python.org/2/library/urllib2.html#urllib2.URLError
exception urllib2.URLError The handlers raise this exception (or
derived exceptions) when they run into a problem. It is a subclass of
IOError.
reason The reason for this error. It can be a message string or
another exception instance (socket.error for remote URLs, OSError for
local URLs).
exception urllib2.HTTPError Though being an exception (a subclass of
URLError), an HTTPError can also function as a non-exceptional
file-like return value (the same thing that urlopen() returns). This
is useful when handling exotic HTTP errors, such as requests for
authentication.
code An HTTP status code as defined in RFC 2616. This numeric value
corresponds to a value found in the dictionary of codes as found in
BaseHTTPServer.BaseHTTPRequestHandler.responses.
reason The reason for this error. It can be a message string or
another exception instance.
So you can access the response body when the request raise urllib2.HTTPError.
Try this:
try:
(calling some function that throws URLError)
except urllib2.HTTPError as e:
body = e.readlines()
print e.code, e.reason, body
except urllib2.URLError as e:
print e.reason
except:
sys.excepthook(*sys.exc_info())