We Keep Coding

python swagger + flask abort returned empty error message

so, i'm trying to use custom abort message while using SwaggerClient. when using flask restful abort function, i can see the abort message on the SwaggerUI. but when i'm trying to see the error message while using the python module, im getting bravado.exception error code.
i tried to use #namespace.response decorator, i tried to use "also_return_response": True, "disable_fallback_results": True and still, no hope.
i would love to get some help/hints etc..
some of my code:
_api = Api(_app, version='1.0', title=self.name, description=self.name, errors = errors, )
NAMESPACE = "test"
namespace = _api.namespace(NAMESPACE, description=NAMESPACE)
#namespace.route("/error")
class SystemStatus(Resource):
#namespace.doc("status")
# #namespace.response(409, "Missing Parameter")
def get(self):
_api.abort(409, "testError")
and the usage of the SwaggerCLient:
def testing2(self):
print("In testing2")
return self.swagger_client.test.status().response()
Error for example: bravado.exception.HTTPConflict: 409 CONFLICT: Response specification matching http status_code 409 not found for operation Operation(status). Either add a response specification for the status_code or use a 'default' response.
and the UI: SwaggerUIResponse

RESTful api design: handle exceptions through nested functions (python, flask)

I would like to improve my coding style with a more robust grasp of try, except and raise in designing API, and less verbose code.
I have nested functions, and when one catches an execption, I am passing the exception to the other one and so on.
But like this, I could propagate multiple checks of a same error.
I am referring to:
[Using try vs if in python
for considering cost of try operation.
How would you handle an error only once across nested functions ?
E.g.
I have a function f(key) doing some operations on key; result is
passed to other functions g(), h()
if result comply with
expected data structure, g() .. h() will manipulate and return
updated result
a decorator will return final result or return the
first error that was met, that is pointing out in which method it was raised (f(),g() or h()).
I am doing something like this:
def f(key):
try:
#do something
return {'data' : 'data_structure'}
except:
return {'error': 'there is an error'}
#application.route('/')
def api_f(key):
data = f(k)
try:
# do something on data
return jsonify(data)
except:
return jsonify({'error':'error in key'})

IMO try/except is the best way to go for this use case. Whenever you want to handle an exceptional case, put in a try/except. If you can’t (or don’t want to) handle the exception in some sane way, let it bubble up to be handled further up the stack. Of course there are various reasons to take different approaches (e.g. you don’t really care about an error and can return something else without disrupting normal operation; you expect “exceptional” cases to happen more often than not; etc.), but here try/except seems to make the most sense:
In your example, it’d be best to leave the try/except out of f() unless you want to…
raise a different error (be careful with this, as this will reset your stack trace):
try:
### Do some stuff
except:
raise CustomError('Bad things')
do some error handling (e.g. logging; cleanup; etc.):
try:
### Do some stuff
except:
logger.exception('Bad things')
cleanup()
### Re-raise the same error
raise
Otherwise, just let the error bubble up.
Subsequent functions (e.g. g(); h()) would operate the same way. In your case, you’d probably want to have some jsonify helper function that jsonifies when possible but also handles non-json data:
def handle_json(data):
try:
return json.dumps(data)
except TypeError, e:
logger.exception('Could not decode json from %s: %s', data, e)
# Could also re-raise the same error
raise CustomJSONError('Bad things')
Then, you would have handler(s) further up the stack to handle either the original error or the custom error, ending with a global handler that can handle any error. In my Flask application, I created custom error classes that my global handler is able to parse and do something with. Of course, the global handler is configured to handle unexpected errors as well.
For instance, I might have a base class for all http errors…
### Not to be raised directly; raise sub-class instances instead
class BaseHTTPError(Exception):
def __init__(self, message=None, payload=None):
Exception.__init__(self)
if message is not None:
self.message = message
else:
self.message = self.default_message
self.payload = payload
def to_dict(self):
"""
Call this in the the error handler to serialize the
error for the json-encoded http response body.
"""
payload = dict(self.payload or ())
payload['message'] = self.message
payload['code'] = self.code
return payload
…which is extended for various http errors:
class NotFoundError(BaseHTTPError):
code = 404
default_message = 'Resource not found'
class BadRequestError(BaseHTTPError):
code = 400
default_message = 'Bad Request'
class NotFoundError(BaseHTTPError):
code = 500
default_message = 'Internal Server Error'
### Whatever other http errors you want
And my global handler looks like this (I am using flask_restful, so this gets defined as a method on my extended flask_restful.Api class):
class RestAPI(flask_restful.Api):
def handle_error(self, e):
code = getattr(e, 'code', 500)
message = getattr(e, 'message', 'Internal Server Error')
to_dict = getattr(e, 'to_dict', None)
if code == 500:
logger.exception(e)
if to_dict:
data = to_dict()
else:
data = {'code': code, 'message': message}
return self.make_response(data, code)
With flask_restful, you may also just define your error classes and pass them as a dictionary to the flask_restful.Api constructor, but I prefer the flexibility of defining my own handler that can add payload data dynamically. flask_restful automatically passes any unhandled errors to handle_error. As such, this is the only place I’ve needed to convert the error to json data because that is what flask_restful needs in order to return an https status and payload to the client. Notice that even if the error type is unknown (e.g. to_dict not defined), I can return a sane http status and payload to the client without having had to convert errors lower down the stack.
Again, there are reasons to convert errors to some useful return value at other places in your app, but for the above, try/except works well.

Post json to django but get 500 error

I'm writing an app to post json to Django, but get a 500 error in terminal like
[27/Jan/2015 20:50:38] "POST /datasave/ds/ HTTP/1.1" 500 10414
This is my jQuery code:
$(function() {
$('#upload').click(function() {
var json_obj = {
username: $('#username').val(),
password: $('#password').val(),
game_id1: '123',
csrfmiddlewaretoken: '{{ csrf_token}}'
};
$.post("http://192.168.0.109:8000/datasave/ds/", JSON.stringify(json_obj),
function(data) {
alert("OK");
},
"json");
})
})
And Django views code:
#csrf_exempt
def ds(request):
dicty = {}
if request.is_ajax:
if request.method == 'POST':
req = json.loads(request.body.decode("utf-8"))
obj, created =
Gamer.objects.update_or_create(
username=req.get(u'username', None),
password=req.get(u'password', None),
game_id1=req.get(u'game_id1', None))
print obj, created
dicty['username'] = req.get(u'username', None)
dicty['password'] = req.get(u'password', None)
dicty['create_at'] = str(timezone.now())
return JsonResponse(dicty)

As you are getting a 500 Internal server error one could assume that there is an issue in your view handler code. It's hard to say what's causing it really, so you should try to figure why it's happening. From looking at your code I can see two things that might cause this error.
First you are using dict as a variable name, which is probably a bad idea since dict is a built-in type in python.
The other possible cause could be the way your are accessing the req dict. You are accessing the keys username, password and game_id1. If any of these should be missing in the dict it will throw a KeyError exception. I like to access dicts using req.get('username', None) instead (replace None with another default value if you prefer. Another way to tackle that issue would be try/catch exception handling.
Also, depending on your Gamer model, trying to create a using an existing username (assuming you have unique=True) would probably throw an exception as well, so you should handle that as well (I think get_or_create could be handy here).
Generally when dealing with problems of this kind, use the inspector in your browser. It lets you see all the data sent and received during the request, which means (if you're running django in DEBUG mode) you'll also get to see the default stack trace page in the inspector. It should hold some valuable indication to what's causing the issue. Another option would be to write a small middleware or enable extended error logging to log errors to strout/stderr.

How to process JSON in Flask?

I have asked a few questions about this before, but still haven't solved my problem.
I am trying to allow Salesforce to remotely send commands to a Raspberry Pi via JSON (REST API). The Raspberry Pi controls the power of some RF Plugs via an RF Transmitter called a TellStick. This is all setup, and I can use Python to send these commands. All I need to do now is make the Pi accept JSON, then work out how to send the commands from Salesforce.
Someone kindly forked my repo on GitHub, and provided me with some code which should make it work. But unfortunately it still isn't working.
Here is the previous question: How to accept a JSON POST?
And here is the forked repo: https://github.com/bfagundez/RemotePiControl/blob/master/power.py
What do I need to do? I have sent test JSON messages n the Postman extension and in cURL but keep getting errors.
I just want to be able to send various variables, and let the script work the rest out.
I can currently post to a .py script I have with some URL variables, so /python.py?power=on&device=1&time=10&pass=whatever and it figures it out. Surely there's a simple way to send this in JSON?
Here is the power.py code:
# add flask here
from flask import Flask
app = Flask(__name__)
app.debug = True
# keep your code
import time
import cgi
from tellcore.telldus import TelldusCore
core = TelldusCore()
devices = core.devices()
# define a "power ON api endpoint"
#app.route("/API/v1.0/power-on/<deviceId>",methods=['POST'])
def powerOnDevice(deviceId):
payload = {}
#get the device by id somehow
device = devices[deviceId]
# get some extra parameters
# let's say how long to stay on
params = request.get_json()
try:
device.turn_on()
payload['success'] = True
return payload
except:
payload['success'] = False
# add an exception description here
return payload
# define a "power OFF api endpoint"
#app.route("/API/v1.0/power-off/<deviceId>",methods=['POST'])
def powerOffDevice(deviceId):
payload = {}
#get the device by id somehow
device = devices[deviceId]
try:
device.turn_off()
payload['success'] = True
return payload
except:
payload['success'] = False
# add an exception description here
return payload
app.run()

Your deviceID variable is a string, not an integer; it contains a '1' digit, but that's not yet an integer.
You can either convert it explicitly:
device = devices[int(deviceId)]
or tell Flask you wanted an integer parameter in the route:
#app.route("/API/v1.0/power-on/<int:deviceId>", methods=['POST'])
def powerOnDevice(deviceId):
where the int: part is a URL route converter.
Your views should return a response object, a string or a tuple instead of a dictionary (as you do now), see About Responses. If you wanted to return JSON, use the flask.json.jsonify() function:
# define a "power ON api endpoint"
#app.route("/API/v1.0/power-on/<int:deviceId>", methods=['POST'])
def powerOnDevice(deviceId):
device = devices[deviceId]
# get some extra parameters
# let's say how long to stay on
params = request.get_json()
try:
device.turn_on()
return jsonify(success=True)
except SomeSpecificException as exc:
return jsonify(success=False, exception=str(exc))
where I also altered the exception handler to handle a specific exception only; try to avoid Pokemon exception handling; do not try to catch them all!

To retrieve the Json Post values you must use request.json
if request.json and 'email' in request.json:
request.json['email']

How do you use cookies and HTTP Basic Authentication in CherryPy?

I have a CherryPy web application that requires authentication. I have working HTTP Basic Authentication with a configuration that looks like this:
app_config = {
'/' : {
'tools.sessions.on': True,
'tools.sessions.name': 'zknsrv',
'tools.auth_basic.on': True,
'tools.auth_basic.realm': 'zknsrv',
'tools.auth_basic.checkpassword': checkpassword,
}
}
HTTP auth works great at this point. For example, this will give me the successful login message that I defined inside AuthTest:
curl http://realuser:realpass#localhost/AuthTest/
Since sessions are on, I can save cookies and examine the one that CherryPy sets:
curl --cookie-jar cookie.jar http://realuser:realpass#localhost/AuthTest/
The cookie.jar file will end up looking like this:
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
localhost FALSE / FALSE 1348640978 zknsrv 821aaad0ba34fd51f77b2452c7ae3c182237deb3
However, I'll get an HTTP 401 Not Authorized failure if I provide this session ID without the username and password, like this:
curl --cookie 'zknsrv=821aaad0ba34fd51f77b2452c7ae3c182237deb3' http://localhost/AuthTest
What am I missing?
Thanks very much for any help.

So, the short answer is you can do this, but you have to write your own CherryPy tool (a before_handler), and you must not enable Basic Authentication in the CherryPy config (that is, you shouldn't do anything like tools.auth.on or tools.auth.basic... etc) - you have to handle HTTP Basic Authentication yourself. The reason for this is that the built-in Basic Authentication stuff is apparently pretty primitive. If you protect something by enabling Basic Authentication like I did above, it will do that authentication check before it checks the session, and your cookies will do nothing.
My solution, in prose
Fortunately, even though CherryPy doesn't have a way to do both built-in, you can still use its built-in session code. You still have to write your own code for handling the Basic Authentication part, but in total this is not so bad, and using the session code is a big win because writing a custom session manager is a good way to introduce security bugs into your webapp.
I ended up being able to take a lot of things from a page on the CherryPy wiki called Simple authentication and access restrictions helpers. That code uses CP sessions, but rather than Basic Auth it uses a special page with a login form that submits ?username=USERNAME&password=PASSWORD. What I did is basically nothing more than changing the provided check_auth function from using the special login page to using the HTTP auth headers.
In general, you need a function you can add as a CherryPy tool - specifically a before_handler. (In the original code, this function was called check_auth(), but I renamed it to protect().) This function first tries to see if the cookies contain a (valid) session ID, and if that fails, it tries to see if there is HTTP auth information in the headers.
You then need a way to require authentication for a given page; I do this with require(), plus some conditions, which are just callables that return True. In my case, those conditions are zkn_admin(), and user_is() functions; if you have more complex needs, you might want to also look at member_of(), any_of(), and all_of() from the original code.
If you do it like that, you already have a way to log in - you just submit a valid session cookie or HTTPBA credentials to any URL you protect with the #require() decorator. All you need now is a way to log out.
(The original code instead has an AuthController class which contains login() and logout(), and you can use the whole AuthController object in your HTTP document tree by just putting auth = AuthController() inside your CherryPy root class, and get to it with a URL of e.g. http://example.com/auth/login and http://example.com/auth/logout. My code doesn't use an authcontroller object, just a few functions.)
Some notes about my code
Caveat: Because I wrote my own parser for HTTP auth headers, it only parses what I told it about, which means just HTTP Basic Auth - not, for example, Digest Auth or anything else. For my application that's fine; for yours, it may not be.
It assumes a few functions defined elsewhere in my code: user_verify() and user_is_admin()
I also use a debugprint() function which only prints output when a DEBUG variable is set, and I've left these calls in for clarity.
You can call it cherrypy.tools.WHATEVER (see the last line); I called it zkauth based on the name of my app. Take care NOT to call it auth, or the name of any other built-in tool, though .
You then have to enable cherrypy.tools.WHATEVER in your CherryPy configuration.
As you can see by all the TODO: messages, this code is still in a state of flux and not 100% tested against edge cases - sorry about that! It will still give you enough of an idea to go on, though, I hope.
My solution, in code
import base64
import re
import cherrypy
SESSION_KEY = '_zkn_username'
def protect(*args, **kwargs):
debugprint("Inside protect()...")
authenticated = False
conditions = cherrypy.request.config.get('auth.require', None)
debugprint("conditions: {}".format(conditions))
if conditions is not None:
# A condition is just a callable that returns true or false
try:
# TODO: I'm not sure if this is actually checking for a valid session?
# or if just any data here would work?
this_session = cherrypy.session[SESSION_KEY]
# check if there is an active session
# sessions are turned on so we just have to know if there is
# something inside of cherrypy.session[SESSION_KEY]:
cherrypy.session.regenerate()
# I can't actually tell if I need to do this myself or what
email = cherrypy.request.login = cherrypy.session[SESSION_KEY]
authenticated = True
debugprint("Authenticated with session: {}, for user: {}".format(
this_session, email))
except KeyError:
# If the session isn't set, it either wasn't present or wasn't valid.
# Now check if the request includes HTTPBA?
# FFR The auth header looks like: "AUTHORIZATION: Basic <base64shit>"
# TODO: cherrypy has got to handle this for me, right?
authheader = cherrypy.request.headers.get('AUTHORIZATION')
debugprint("Authheader: {}".format(authheader))
if authheader:
#b64data = re.sub("Basic ", "", cherrypy.request.headers.get('AUTHORIZATION'))
# TODO: what happens if you get an auth header that doesn't use basic auth?
b64data = re.sub("Basic ", "", authheader)
decodeddata = base64.b64decode(b64data.encode("ASCII"))
# TODO: test how this handles ':' characters in username/passphrase.
email,passphrase = decodeddata.decode().split(":", 1)
if user_verify(email, passphrase):
cherrypy.session.regenerate()
# This line of code is discussed in doc/sessions-and-auth.markdown
cherrypy.session[SESSION_KEY] = cherrypy.request.login = email
authenticated = True
else:
debugprint ("Attempted to log in with HTTBA username {} but failed.".format(
email))
else:
debugprint ("Auth header was not present.")
except:
debugprint ("Client has no valid session and did not provide HTTPBA credentials.")
debugprint ("TODO: ensure that if I have a failure inside the 'except KeyError'"
+ " section above, it doesn't get to this section... I'd want to"
+ " show a different error message if that happened.")
if authenticated:
for condition in conditions:
if not condition():
debugprint ("Authentication succeeded but authorization failed.")
raise cherrypy.HTTPError("403 Forbidden")
else:
raise cherrypy.HTTPError("401 Unauthorized")
cherrypy.tools.zkauth = cherrypy.Tool('before_handler', protect)
def require(*conditions):
"""A decorator that appends conditions to the auth.require config
variable."""
def decorate(f):
if not hasattr(f, '_cp_config'):
f._cp_config = dict()
if 'auth.require' not in f._cp_config:
f._cp_config['auth.require'] = []
f._cp_config['auth.require'].extend(conditions)
return f
return decorate
#### CONDITIONS
#
# Conditions are callables that return True
# if the user fulfills the conditions they define, False otherwise
#
# They can access the current user as cherrypy.request.login
# TODO: test this function with cookies, I want to make sure that cherrypy.request.login is
# set properly so that this function can use it.
def zkn_admin():
return lambda: user_is_admin(cherrypy.request.login)
def user_is(reqd_email):
return lambda: reqd_email == cherrypy.request.login
#### END CONDITIONS
def logout():
email = cherrypy.session.get(SESSION_KEY, None)
cherrypy.session[SESSION_KEY] = cherrypy.request.login = None
return "Logout successful"
Now all you have to do is enable both builtin sessions and your own cherrypy.tools.WHATEVER in your CherryPy configuration. Again, take care not to enable cherrypy.tools.auth. My configuration ended up looking like this:
config_root = {
'/' : {
'tools.zkauth.on': True,
'tools.sessions.on': True,
'tools.sessions.name': 'zknsrv',
}
}