I am using the python package for the minio server. I have the following piece of code that is used for a login:
from minio.error import [...], SignatureDoesNotMatch, [...]
def login(self):
try:
self.user = Minio(MINIO_CONFIG['MINIO_ENDPOINT'],
access_key=self.username,
secret_key=self.password,
secure=MINIO_CONFIG['MINIO_SECURE'])
return {"msg":"User is now logged in", "status": "OK"}
except SignatureDoesNotMatch as err:
return {"msg": err.message, "status":"F"}
except ResponseError as err:
return {'msg': err.message, 'status': "F"}
except InvalidAccessKeyId as err:
return {"msg": err.message, "status":"F"}
except InvalidArgument as err:
return {"msg": err.message, "status":"F"}
except InvalidArgumentError as err:
return {"msg": err.message, "status":"F"}
The issue I am facing is that even though I do have in the try-except the SignatureDoesNotMatch in case the credentials are not correct, it does not return me the msg it should but it throws an minio.error.SignatureDoesNotMatch instead. Why does that happen?
The error I get:
minio.error.SignatureDoesNotMatch: SignatureDoesNotMatch: message: The request signature we calculated does not match the signature you provided.
This seems fine, looking at the code, this will never run into an error on it's own, regardless of the credentials provided. It will only run into an error when it makes an API call, or when you invoke methods like list_buckets, list_objects etc using this self.user instance, from outside this block.
I think what you're trying to do is-- invoking methods like list_buckets etc from outside this encapsulation-- somewhere else not this part of the code, and then they produce this error and propagate them to the console. You cannot encapsulate the MinIO instance within try-catch and catch errors while you make use of stuff like self.user.list_buckets() from outside this try-catch block.
When I'm using nova.keypairs.create() and I pass it an invalid public key, I get the following:
BadRequest: Keypair data is invalid: failed to generate fingerprint (HTTP 400) (Request-ID: req-12bc6440-f042-4687-9ee9-d89e7edc260d)
I tried doing the following and for obvious reasons (it's a unique exception to OpenStack) it didn't work:
try:
nova.keypairs.create(name=keyname, public_key=key)
except BadRequest:
raise cherrypy.HTTPError(400, "Invalid public key")
How can I use OpenStack specific exceptions such as BadRequest within my own try and except statements?
You will need to import the exceptions for nova package. Going through github for the package, it looks like you will need to do:
from nova.exception import *
Note that the exception you are seeing is actually InvalidKeypair exception, which itself subclasses from exception class Invalid, the BadRequest message is just the template text for it.
So, your complete code would look something like:
from nova.exception import *
# You can import specific ones if you are confident about them
try:
nova.keypairs.create(name=keyname, public_key=key)
except InvalidKeypair:
raise cherrypy.HTTPError(400, "Invalid public key")
I'm trying to make a custom error page in Flask, and I'd like to give the error handler access to the request that generated the API call that caused the error so that the error page it returns can change depend on the circumstances. For instance, say there are two endpoints:
(1) #app.route('/get_item')
(2) #app.route('/submit_item')
If an error occurs during a call to get_item, I want to display a certain error page ("Sorry, an error occurred...") however, if an error occurs during a call to submit_item, I want it to say something more informative, like:
"An error occured! Please contact us.
Your user id: request.json['userid']
Your submission id: request.json['submission']"
Is it possible to allow the error handler to have access to this, or do I just have to wrap the whole of submit_item in try/except statements?
You can use the request context in the error handler function,
something along those lines:
from flask import request
def exception_handler(*args):
if request.url.endswith('submit_item'):
return "MyMoreDescriptiveErrorMessage", 500
else:
return "Something wrong happened", 500
I would probably create a custom exception and specify an error handler for it similar to this example.
class CustomException(Exception):
def __init__(self, message=None, status_code=None, payload=None):
Exception.__init__(self)
if message is None:
message = "Sorry, an error occurred..."
self.message = message
if status_code is not None:
self.status_code = status_code
self.payload = payload
#app.errorhandler(CustomException)
def handle_custom(error):
response = render_template('error.html', message=error.message)
response.status_code = error.status_code
return response
#app.route('/submit_item')
def submit_item():
message = "An error occured! Userid: %(userid)d, submission: %(submission_id)d"
message = message % (request.json)
raise CustomException(message)
I would like to improve my coding style with a more robust grasp of try, except and raise in designing API, and less verbose code.
I have nested functions, and when one catches an execption, I am passing the exception to the other one and so on.
But like this, I could propagate multiple checks of a same error.
I am referring to:
[Using try vs if in python
for considering cost of try operation.
How would you handle an error only once across nested functions ?
E.g.
I have a function f(key) doing some operations on key; result is
passed to other functions g(), h()
if result comply with
expected data structure, g() .. h() will manipulate and return
updated result
a decorator will return final result or return the
first error that was met, that is pointing out in which method it was raised (f(),g() or h()).
I am doing something like this:
def f(key):
try:
#do something
return {'data' : 'data_structure'}
except:
return {'error': 'there is an error'}
#application.route('/')
def api_f(key):
data = f(k)
try:
# do something on data
return jsonify(data)
except:
return jsonify({'error':'error in key'})
IMO try/except is the best way to go for this use case. Whenever you want to handle an exceptional case, put in a try/except. If you can’t (or don’t want to) handle the exception in some sane way, let it bubble up to be handled further up the stack. Of course there are various reasons to take different approaches (e.g. you don’t really care about an error and can return something else without disrupting normal operation; you expect “exceptional” cases to happen more often than not; etc.), but here try/except seems to make the most sense:
In your example, it’d be best to leave the try/except out of f() unless you want to…
raise a different error (be careful with this, as this will reset your stack trace):
try:
### Do some stuff
except:
raise CustomError('Bad things')
do some error handling (e.g. logging; cleanup; etc.):
try:
### Do some stuff
except:
logger.exception('Bad things')
cleanup()
### Re-raise the same error
raise
Otherwise, just let the error bubble up.
Subsequent functions (e.g. g(); h()) would operate the same way. In your case, you’d probably want to have some jsonify helper function that jsonifies when possible but also handles non-json data:
def handle_json(data):
try:
return json.dumps(data)
except TypeError, e:
logger.exception('Could not decode json from %s: %s', data, e)
# Could also re-raise the same error
raise CustomJSONError('Bad things')
Then, you would have handler(s) further up the stack to handle either the original error or the custom error, ending with a global handler that can handle any error. In my Flask application, I created custom error classes that my global handler is able to parse and do something with. Of course, the global handler is configured to handle unexpected errors as well.
For instance, I might have a base class for all http errors…
### Not to be raised directly; raise sub-class instances instead
class BaseHTTPError(Exception):
def __init__(self, message=None, payload=None):
Exception.__init__(self)
if message is not None:
self.message = message
else:
self.message = self.default_message
self.payload = payload
def to_dict(self):
"""
Call this in the the error handler to serialize the
error for the json-encoded http response body.
"""
payload = dict(self.payload or ())
payload['message'] = self.message
payload['code'] = self.code
return payload
…which is extended for various http errors:
class NotFoundError(BaseHTTPError):
code = 404
default_message = 'Resource not found'
class BadRequestError(BaseHTTPError):
code = 400
default_message = 'Bad Request'
class NotFoundError(BaseHTTPError):
code = 500
default_message = 'Internal Server Error'
### Whatever other http errors you want
And my global handler looks like this (I am using flask_restful, so this gets defined as a method on my extended flask_restful.Api class):
class RestAPI(flask_restful.Api):
def handle_error(self, e):
code = getattr(e, 'code', 500)
message = getattr(e, 'message', 'Internal Server Error')
to_dict = getattr(e, 'to_dict', None)
if code == 500:
logger.exception(e)
if to_dict:
data = to_dict()
else:
data = {'code': code, 'message': message}
return self.make_response(data, code)
With flask_restful, you may also just define your error classes and pass them as a dictionary to the flask_restful.Api constructor, but I prefer the flexibility of defining my own handler that can add payload data dynamically. flask_restful automatically passes any unhandled errors to handle_error. As such, this is the only place I’ve needed to convert the error to json data because that is what flask_restful needs in order to return an https status and payload to the client. Notice that even if the error type is unknown (e.g. to_dict not defined), I can return a sane http status and payload to the client without having had to convert errors lower down the stack.
Again, there are reasons to convert errors to some useful return value at other places in your app, but for the above, try/except works well.
try:
r = requests.get(url, params={'s': thing})
except requests.ConnectionError, e:
print(e)
Is this correct? Is there a better way to structure this? Will this cover all my bases?
Have a look at the Requests exception docs. In short:
In the event of a network problem (e.g. DNS failure, refused connection, etc), Requests will raise a ConnectionError exception.
In the event of the rare invalid HTTP response, Requests will raise an HTTPError exception.
If a request times out, a Timeout exception is raised.
If a request exceeds the configured number of maximum redirections, a TooManyRedirects exception is raised.
All exceptions that Requests explicitly raises inherit from requests.exceptions.RequestException.
To answer your question, what you show will not cover all of your bases. You'll only catch connection-related errors, not ones that time out.
What to do when you catch the exception is really up to the design of your script/program. Is it acceptable to exit? Can you go on and try again? If the error is catastrophic and you can't go on, then yes, you may abort your program by raising SystemExit (a nice way to both print an error and call sys.exit).
You can either catch the base-class exception, which will handle all cases:
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.RequestException as e: # This is the correct syntax
raise SystemExit(e)
Or you can catch them separately and do different things.
try:
r = requests.get(url, params={'s': thing})
except requests.exceptions.Timeout:
# Maybe set up for a retry, or continue in a retry loop
except requests.exceptions.TooManyRedirects:
# Tell the user their URL was bad and try a different one
except requests.exceptions.RequestException as e:
# catastrophic error. bail.
raise SystemExit(e)
As Christian pointed out:
If you want http errors (e.g. 401 Unauthorized) to raise exceptions, you can call Response.raise_for_status. That will raise an HTTPError, if the response was an http error.
An example:
try:
r = requests.get('http://www.google.com/nothere')
r.raise_for_status()
except requests.exceptions.HTTPError as err:
raise SystemExit(err)
Will print:
404 Client Error: Not Found for url: http://www.google.com/nothere
One additional suggestion to be explicit. It seems best to go from specific to general down the stack of errors to get the desired error to be caught, so the specific ones don't get masked by the general one.
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
Http Error: 404 Client Error: Not Found for url: http://www.google.com/blahblah
vs
url='http://www.google.com/blahblah'
try:
r = requests.get(url,timeout=3)
r.raise_for_status()
except requests.exceptions.RequestException as err:
print ("OOps: Something Else",err)
except requests.exceptions.HTTPError as errh:
print ("Http Error:",errh)
except requests.exceptions.ConnectionError as errc:
print ("Error Connecting:",errc)
except requests.exceptions.Timeout as errt:
print ("Timeout Error:",errt)
OOps: Something Else 404 Client Error: Not Found for url: http://www.google.com/blahblah
Exception object also contains original response e.response, that could be useful if need to see error body in response from the server. For example:
try:
r = requests.post('somerestapi.com/post-here', data={'birthday': '9/9/3999'})
r.raise_for_status()
except requests.exceptions.HTTPError as e:
print (e.response.text)
Here's a generic way to do things which at least means that you don't have to surround each and every requests call with try ... except:
Basic version
# see the docs: if you set no timeout the call never times out! A tuple means "max
# connect time" and "max read time"
DEFAULT_REQUESTS_TIMEOUT = (5, 15) # for example
def log_exception(e, verb, url, kwargs):
# the reason for making this a separate function will become apparent
raw_tb = traceback.extract_stack()
if 'data' in kwargs and len(kwargs['data']) > 500: # anticipate giant data string
kwargs['data'] = f'{kwargs["data"][:500]}...'
msg = f'BaseException raised: {e.__class__.__module__}.{e.__class__.__qualname__}: {e}\n' \
+ f'verb {verb}, url {url}, kwargs {kwargs}\n\n' \
+ 'Stack trace:\n' + ''.join(traceback.format_list(raw_tb[:-2]))
logger.error(msg)
def requests_call(verb, url, **kwargs):
response = None
exception = None
try:
if 'timeout' not in kwargs:
kwargs['timeout'] = DEFAULT_REQUESTS_TIMEOUT
response = requests.request(verb, url, **kwargs)
except BaseException as e:
log_exception(e, verb, url, kwargs)
exception = e
return (response, exception)
NB
Be aware of ConnectionError which is a builtin, nothing to do with the class requests.ConnectionError*. I assume the latter is more common in this context but have no real idea...
When examining a non-None returned exception, requests.RequestException, the superclass of all the requests exceptions (including requests.ConnectionError), is not "requests.exceptions.RequestException" according to the docs. Maybe it has changed since the accepted answer.**
Obviously this assumes a logger has been configured. Calling logger.exception in the except block might seem a good idea but that would only give the stack within this method! Instead, get the trace leading up to the call to this method. Then log (with details of the exception, and of the call which caused the problem)
*I looked at the source code: requests.ConnectionError subclasses the single class requests.RequestException, which subclasses the single class IOError (builtin)
**However at the bottom of this page you find "requests.exceptions.RequestException" at the time of writing (2022-02)... but it links to the above page: confusing.
Usage is very simple:
search_response, exception = utilities.requests_call('get',
f'http://localhost:9200/my_index/_search?q={search_string}')
First you check the response: if it's None something funny has happened and you will have an exception which has to be acted on in some way depending on context (and on the exception). In Gui applications (PyQt5) I usually implement a "visual log" to give some output to the user (and also log simultaneously to the log file), but messages added there should be non-technical. So something like this might typically follow:
if search_response == None:
# you might check here for (e.g.) a requests.Timeout, tailoring the message
# accordingly, as the kind of error anyone might be expected to understand
msg = f'No response searching on |{search_string}|. See log'
MainWindow.the().visual_log(msg, log_level=logging.ERROR)
return
response_json = search_response.json()
if search_response.status_code != 200: # NB 201 ("created") may be acceptable sometimes...
msg = f'Bad response searching on |{search_string}|. See log'
MainWindow.the().visual_log(msg, log_level=logging.ERROR)
# usually response_json will give full details about the problem
log_msg = f'search on |{search_string}| bad response\n{json.dumps(response_json, indent=4)}'
logger.error(log_msg)
return
# now examine the keys and values in response_json: these may of course
# indicate an error of some kind even though the response returned OK (status 200)...
Given that the stack trace is logged automatically you often need no more than that...
Advanced version when json object returned
(... potentially sparing a great deal of boilerplate!)
To cross the Ts, when a json object is expected to be returned:
If, as above, an exception gives your non-technical user a message "No response", and a non-200 status "Bad response", I suggest that
a missing expected key in the response's JSON structure should give rise to a message "Anomalous response"
an out-of-range or strange value to a message "Unexpected response"
and the presence of a key such as "error" or "errors", with value True or whatever, to a message "Error response"
These may or may not prevent the code from continuing.
... and in fact to my mind it is worth making the process even more generic. These next functions, for me, typically cut down 20 lines of code using the above requests_call to about 3, and make most of your handling and your log messages standardised. More than a handful of requests calls in your project and the code gets a lot nicer and less bloated:
def log_response_error(response_type, call_name, deliverable, verb, url, **kwargs):
# NB this function can also be used independently
if response_type == 'No': # exception was raised (and logged)
if isinstance(deliverable, requests.Timeout):
MainWindow.the().visual_log(f'Time out of {call_name} before response received!', logging.ERROR)
return
else:
if isinstance(deliverable, BaseException):
# NB if response.json() raises an exception we end up here
log_exception(deliverable, verb, url, kwargs)
else:
# if we get here no exception has been raised, so no stack trace has yet been logged.
# a response has been returned, but is either "Bad" or "Anomalous"
response_json = deliverable.json()
raw_tb = traceback.extract_stack()
if 'data' in kwargs and len(kwargs['data']) > 500: # anticipate giant data string
kwargs['data'] = f'{kwargs["data"][:500]}...'
added_message = ''
if hasattr(deliverable, 'added_message'):
added_message = deliverable.added_message + '\n'
del deliverable.added_message
call_and_response_details = f'{response_type} response\n{added_message}' \
+ f'verb {verb}, url {url}, kwargs {kwargs}\nresponse:\n{json.dumps(response_json, indent=4)}'
logger.error(f'{call_and_response_details}\nStack trace: {"".join(traceback.format_list(raw_tb[:-1]))}')
MainWindow.the().visual_log(f'{response_type} response {call_name}. See log.', logging.ERROR)
def check_keys(req_dict_structure, response_dict_structure, response):
# so this function is about checking the keys in the returned json object...
# NB both structures MUST be dicts
if not isinstance(req_dict_structure, dict):
response.added_message = f'req_dict_structure not dict: {type(req_dict_structure)}\n'
return False
if not isinstance(response_dict_structure, dict):
response.added_message = f'response_dict_structure not dict: {type(response_dict_structure)}\n'
return False
for dict_key in req_dict_structure.keys():
if dict_key not in response_dict_structure:
response.added_message = f'key |{dict_key}| missing\n'
return False
req_value = req_dict_structure[dict_key]
response_value = response_dict_structure[dict_key]
if isinstance(req_value, dict):
# if the response at this point is a list apply the req_value dict to each element:
# failure in just one such element leads to "Anomalous response"...
if isinstance(response_value, list):
for resp_list_element in response_value:
if not check_keys(req_value, resp_list_element, response):
return False
elif not check_keys(req_value, response_value, response): # any other response value must be a dict (tested in next level of recursion)
return False
elif isinstance(req_value, list):
if not isinstance(response_value, list): # if the req_value is a list the reponse must be one
response.added_message = f'key |{dict_key}| not list: {type(response_value)}\n'
return False
# it is OK for the value to be a list, but these must be strings (keys) or dicts
for req_list_element, resp_list_element in zip(req_value, response_value):
if isinstance(req_list_element, dict):
if not check_keys(req_list_element, resp_list_element, response):
return False
if not isinstance(req_list_element, str):
response.added_message = f'req_list_element not string: {type(req_list_element)}\n'
return False
if req_list_element not in response_value:
response.added_message = f'key |{req_list_element}| missing from response list\n'
return False
# put None as a dummy value (otherwise something like {'my_key'} will be seen as a set, not a dict
elif req_value != None:
response.added_message = f'required value of key |{dict_key}| must be None (dummy), dict or list: {type(req_value)}\n'
return False
return True
def process_json_requests_call(verb, url, **kwargs):
# "call_name" is a mandatory kwarg
if 'call_name' not in kwargs:
raise Exception('kwarg "call_name" not supplied!')
call_name = kwargs['call_name']
del kwargs['call_name']
required_keys = {}
if 'required_keys' in kwargs:
required_keys = kwargs['required_keys']
del kwargs['required_keys']
acceptable_statuses = [200]
if 'acceptable_statuses' in kwargs:
acceptable_statuses = kwargs['acceptable_statuses']
del kwargs['acceptable_statuses']
exception_handler = log_response_error
if 'exception_handler' in kwargs:
exception_handler = kwargs['exception_handler']
del kwargs['exception_handler']
response, exception = requests_call(verb, url, **kwargs)
if response == None:
exception_handler('No', call_name, exception, verb, url, **kwargs)
return (False, exception)
try:
response_json = response.json()
except BaseException as e:
logger.error(f'response.status_code {response.status_code} but calling json() raised exception')
# an exception raised at this point can't truthfully lead to a "No response" message... so say "bad"
exception_handler('Bad', call_name, e, verb, url, **kwargs)
return (False, response)
status_ok = response.status_code in acceptable_statuses
if not status_ok:
response.added_message = f'status code was {response.status_code}'
log_response_error('Bad', call_name, response, verb, url, **kwargs)
return (False, response)
check_result = check_keys(required_keys, response_json, response)
if not check_result:
log_response_error('Anomalous', call_name, response, verb, url, **kwargs)
return (check_result, response)
Example call (NB with this version, the "deliverable" is either an exception or a response which delivers a json structure):
success, deliverable = utilities.process_json_requests_call('get',
f'{ES_URL}{INDEX_NAME}/_doc/1',
call_name=f'checking index {INDEX_NAME}',
required_keys={'_source':{'status_text': None}})
if not success: return False
# here, we know the deliverable is a response, not an exception
# we also don't need to check for the keys being present:
# the generic code has checked that all expected keys are present
index_status = deliverable.json()['_source']['status_text']
if index_status != 'successfully completed':
# ... i.e. an example of a 200 response, but an error nonetheless
msg = f'Error response: ES index {INDEX_NAME} does not seem to have been built OK: cannot search'
MainWindow.the().visual_log(msg)
logger.error(f'index |{INDEX_NAME}|: deliverable.json() {json.dumps(deliverable.json(), indent=4)}')
return False
So the "visual log" message seen by the user in the case of missing key "status_text", for example, would be "Anomalous response checking index XYZ. See log." (and the log would give a more detailed technical message, constructed automatically, including the stack trace but also details of the missing key in question).
NB
mandatory kwarg: call_name; optional kwargs: required_keys, acceptable_statuses, exception_handler.
the required_keys dict can be nested to any depth
finer-grained exception-handling can be accomplished by including a function exception_handler in kwargs (though don't forget that requests_call will have logged the call details, the exception type and __str__, and the stack trace).
in the above I also implement a check on key "data" in any kwargs which may be logged. This is because a bulk operation (e.g. to populate an index in the case of Elasticsearch) can consist of enormous strings. So curtail to the first 500 characters, for example.
PS Yes, I do know about the elasticsearch Python module (a "thin wrapper" around requests). All the above is for illustration purposes.