I am using Flask to build a ReST/JSON API. I want all of my responses to be JSON, even when they're error responses.
I have already made my app return uniform JSON responses for successful operations and most errors. Here's the relevant portion of my code:
from flask import Flask, jsonify
from werkzeug import exceptions
app = Flask(__name__)
def make_json_response(status, message, content):
"""Function to generate uniform JSON responses across the API"""
response = {
"api_version": __version__,
"content": content, # The data the API is supposed to return
"message": message, # Content description, error message, etc.
"request_id": g.get("request_id", None),
"status": status, # HTTP Status
}
return jsonify(response), status
def json_error_handler(error=exceptions.InternalServerError, message=None):
"""Makes JSON error messages. By default, Flask will return HTML error
messages, but we want this API to return valid JSON even on errors."""
return make_json_response(
status=error.code, message=message or error.description, content=None)
# Registering JSON error handlers for each HTTP error that we might reasonably throw.
# Covers 400, 401, 403, 404, 405, 406, 408, 409, 410, 411, 412, 413, 414, 415, 416, 417, 418, 422, 428, 429, 431, 500, 501, 502, 503, 504, 505
for error_class in exceptions.default_exceptions.values():
app.register_error_handler(error_class, json_error_handler)
So that's all working. However, 300-3XX responses are not addressed. From what I can tell, redirect errors are not raised in the same way that the other errors are. For example, a 500 may strike at any time when there's a bug in my code, but 300s are a routing decision, and happen before my code is reached.
Flask very helpfully generates automatic redirect responses. For example, if a user tries to GET /jobs, but the real path is /jobs/, Flask will reply with a redirect with no additional work on my end:
$ curl -i https://api.example.com/jobs
HTTP/1.1 301 MOVED PERMANENTLY
Server: gunicorn/19.6.0
Date: Thu, 01 Jun 2017 14:40:18 GMT
X-Cnection: close
Content-Type: text/html; charset=utf-8
Location: http://api.example.com/jobs/
Content-Length: 295
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: http://api.example.com/jobs/. If not click the link.
This is desirable behavior. The only issue is that the generated response does not conform to the JSON response pattern used by the rest of the API's responses. This could cause clients that are expecting JSON responses to explode. (Now, any client brittle enough to have that issue should really know the correct URIs to begin with, but still, I would like my API responses to be uniform and helpful.)
From what I can tell, behind the scenes, Flask uses werkzeug.utils.redirect to generate the redirect, but I don't see an obvious way to monkey-patch my own redirect function into my Flask app.
Is there a way to change the automatically-generated Flask redirect responses?
Related
I'm using Python 3.7 with urllib.
All work fine but it seems not to athomatically redirect when it gets an http redirect request (307).
This is the error i get:
ERROR 2020-06-15 10:25:06,968 HTTP Error 307: Temporary Redirect
I've to handle it with a try-except and manually send another request to the new Location: it works fine but i don't like it.
These is the piece of code i use to perform the request:
req = urllib.request.Request(url)
req.add_header('Authorization', auth)
req.add_header('Content-Type','application/json; charset=utf-8')
req.data=jdati
self.logger.debug(req.headers)
self.logger.info(req.data)
resp = urllib.request.urlopen(req)
url is an https resource and i set an header with some Authhorization info and content-type.
req.data is a JSON
From urllib documentation i've understood that the redirects are authomatically performed by the the library itself, but it doesn't work for me. It always raises an http 307 error and doesn't follow the redirect URL.
I've also tried to use an opener specifiyng the default redirect handler, but with the same result
opener = urllib.request.build_opener(urllib.request.HTTPRedirectHandler)
req = urllib.request.Request(url)
req.add_header('Authorization', auth)
req.add_header('Content-Type','application/json; charset=utf-8')
req.data=jdati
resp = opener.open(req)
What could be the problem?
The reason why the redirect isn't done automatically has been correctly identified by yours truly in the discussion in the comments section. Specifically, RFC 2616, Section 10.3.8 states that:
If the 307 status code is received in response to a request other
than GET or HEAD, the user agent MUST NOT automatically redirect the
request unless it can be confirmed by the user, since this might
change the conditions under which the request was issued.
Back to the question - given that data has been assigned, this automatically results in get_method returning POST (as per how this method was implemented), and since that the request method is POST, and the response code is 307, an HTTPError is raised instead as per the above specification. In the context of Python's urllib, this specific section of the urllib.request module raises the exception.
For an experiment, try the following code:
import urllib.request
import urllib.parse
url = 'http://httpbin.org/status/307'
req = urllib.request.Request(url)
req.data = b'hello' # comment out to not trigger manual redirect handling
try:
resp = urllib.request.urlopen(req)
except urllib.error.HTTPError as e:
if e.status != 307:
raise # not a status code that can be handled here
redirected_url = urllib.parse.urljoin(url, e.headers['Location'])
resp = urllib.request.urlopen(redirected_url)
print('Redirected -> %s' % redirected_url) # the original redirected url
print('Response URL -> %s ' % resp.url) # the final url
Running the code as is may produce the following
Redirected -> http://httpbin.org/redirect/1
Response URL -> http://httpbin.org/get
Note the subsequent redirect to get was done automatically, as the subsequent request was a GET request. Commenting out req.data assignment line will result in the lack of the "Redirected" output line.
Other notable things to note in the exception handling block, e.read() may be done to retrieve the response body produced by the server as part of the HTTP 307 response (since data was posted, there might be a short entity in the response that may be processed?), and that urljoin is needed as the Location header may be a relative URL (or simply has the host missing) to the subsequent resource.
Also, as a matter of interest (and for linkage purposes), this specific question has been asked multiple times before and I am rather surprised that they never got any answers, which follows:
How to handle 307 redirection using urllib2 from http to https
HTTP Error 307: Temporary Redirect in Python3 - INTRANET
HTTP Error 307 - Temporary redirect in python script
I'm implementing swagger validation middleware for falcon framework. However, there's a problem with unsuccessful requests.
If a falcon HTTP error is raised in responder or even before coming to a resource (for example in case of 404 Not Found), response still has status code 200 and empty body when it comes to process_response middleware method.
I can't undertsand if it's a bug or a feature.
When I remove validation middleware, everything works as expected. When there's a swagger validation middleware, all responses in process_response have status code 200, including unsuccessful.
I want to skip swagger validation/serialization of response in case of HTTP 404, 401, 400 etc. When a response comes to process_response, how do I find out that an error was raised if the status code is 200? How do I process only successful responses in middleware?
Also, at which point does falcon set the proper response status code and body?
python 3.5, falcon==1.0.0
Any help is much appreciated.
UPD: minimal working example:
app.py:
import falcon
class Validator:
def process_resource(self, req, resp, resource, params):
raise falcon.HTTPBadRequest("Too bad", "All requests are bad")
def process_response(self, req, resp, resource):
print(resp.status, resp.body)
class Resource:
def on_get(self, req, resp):
pass
application = falcon.API(middleware=[Validator()])
application.add_route('/test', Resource())
$ http http http://127.0.0.1:8000/test returns:
HTTP/1.1 400 Bad Request
Connection: close
Date: Tue, 28 Jun 2016 15:09:04 GMT
Server: gunicorn/19.6.0
content-length: 69
content-type: application/json; charset=UTF-8
vary: Accept
{
"description": "All requests are bad",
"title": "Too bad"
}
while web app prints 200 OK None.
Here's my understanding of the desired sequence of operations when uploading a file to an entity's dropbox submission:
Create feedback for a user (in my case, I am populating a rubric assessment automatically)
Upload a file, receive unique file key
Attach that file to the user's feedback, using the key received in (2)
I can upload a file for the user using the web interface to d2L, but obviously would like to streamline this if I am preparing the feedback through automation anyway. My current status is:
Create feedback for a user (in my case, I am populating a rubric assessment automatically) (200 - Success)
Upload a file, receive unique file key (200 - Success)
Attach that file to the user's feedback, using the key received in (2) (404 - Not Found)
I've been struggling with how to debug uploading and attaching a file to a dropbox feedback. I followed the resumable upload instructions at http://docs.valence.desire2learn.com/basic/fileupload.html#resumable-uploads, and (generally) that went OK, but I'm getting a final 404 error when attaching that uploaded file to the dropbox folder for this entity.
Below is my debug output that demonstrates what I understand as the state of the system for me.
Note: I replaced COURSEID, DROPBOXID, ENTITYID, and a few Keys using search and replace in an external editor, to preserve the illusion of security.
POST /d2l/api/le/1.4/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID
Posts the feedback to the dropbox for this user
{}
Confirms that the post was successful
filepath=092215-0952/Testing Download Sep 21, 2015 1114 AM_upload/ENTITYID-DROPBOXID - Student Name-Due 18, YYYY TIME AM - netid_assignmentname.zip_upload.zip
length=76630
https://d2l.domain.tld/d2l/api/le/1.4/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/upload?x_b=_________USERID___________&x_c=_________KEY_I_DONT_UNDERSTAND___________&x_a=_________APP_ID___________&x_d=_________ANOTHER_KEY_I_DONT_UNDERSTAND___________&x_t=1443053113
{'Content-Length': '76630', 'User-Agent': 'python-requests/2.7.0 CPython/3.3.6 Darwin/14.5.0', 'Accept-Encoding': 'gzip, deflate', 'Connection': 'keep-alive', 'Accept': '*/*', 'X-Upload-Content-Type': 'application/zip', 'X-Upload-File-Name': 'temp.zip', 'X-Upload-Content-Length': '76630'}
Received new URL to which to upload this file: revising post for this fileKey
result=qafSoMUYAm
https://d2l.domain.tld/d2l/upload/qafSoMUYAm
result of revised post is 200
With this, it seems that I initiated, and completed, the resumable upload. I will say, this part was made easier by the instructions from valence, but there was a lot of muttering on my part trying to debug it until I realized that some of the built-in post commands in d2lservice would always fail since I had to change the URL after the first post. :) By the way, I can make this return a 416 if I intentionally mess up the byte count for the file length. I did this to make sure that I wasn't getting 200 for the wrong reason.
fileKey=qafSoMUYAm, attaching to the dropbox now
GET /d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID
Feedback exists for org_unit_id=COURSEID,folder_id=DROPBOXID,entity_id=ENTITYID
Here, I am making sure that the feedback truly exists for this user, so I can compare the URL to the one that fails below. NOTE: I am using version 1.4 for almost everything, but in the example below I was hardcoding the version to be 1.1 in case there was a version difference in the APIs; I tried 1.1, 1.4, but nothing else.
I am also force-marking the feedback to be feedback['IsGraded']=False, in case for some reason it's not possible to add a file to feedback if the feedback has been published. I then re-post the feedback (after marking it as ungraded), and attempt to attach the uploaded file to this feedback. An additional variable is that I have also tried (and removed) the use of a fileName param for the post (it is listed as optional in the APIs).
POST /d2l/api/le/1.4/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID
I am turning on debug now...
ready for POST=/d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/attach
INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): d2l.domain.tld
send: b'POST /d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/attach?fileKey=qafSoMUYAm&x_b=_________USERID___________&x_a=_________APP_ID___________&x_d=sw6PXGCvBOsBj07VJCPu4RJ6hN8AN7OIId2ONOj-_CA&x_t=1443053116&x_c=LIrzfSOnb8rz3wkN-ZTbfnckwvrfheUU9kkarOhGpok HTTP/1.1\r\nHost: d2l.domain.tld\r\nConnection: keep-alive\r\nUser-Agent: python-requests/2.7.0 CPython/3.3.6 Darwin/14.5.0\r\nContent-Length: 0\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
DEBUG:requests.packages.urllib3.connectionpool:"POST /d2l/api/le/1.1/COURSEID/dropbox/folders/DROPBOXID/feedback/user/ENTITYID/attach?fileKey=qafSoMUYAm&x_b=_________USERID___________&x_a=_________APP_ID___________&x_d=sw6PXGCvBOsBj07VJCPu4RJ6hN8AN7OIId2ONOj-_CA&x_t=1443053116&x_c=LIrzfSOnb8rz3wkN-ZTbfnckwvrfheUU9kkarOhGpok HTTP/1.1" 404 0
So, I get a 404 when trying to make this post, and a bunch of errors below from Python that don't really help (but I include them anyway).
header: Server header: X-XSS-Protection header: X-UA-Compatible header: X-Powered-By header: Date header: Content-Length header: Set-Cookie <Response [404]>
Traceback (most recent call last):
File "../valence/sprinkle/production/postgradenoui.py", line 239, in <module>
print(postGrades_handler(dropboxid, gradeddirname,gradeddir_uploadname))
File "../valence/sprinkle/production/postgradenoui.py", line 230, in postGrades_handler
result = sprinkleutil.postRubricFeedback(request, _ac, dropboxid, gradeddirname, _CFG_COURSE['courseorgunit'], gradeddir_uploadname)
File "/Users/sprinkle/work/teaching/ece275-2015F/grading/valence/sprinkle/production/sprinkleutil.py", line 1279, in postRubricFeedback
upload_feedback_files_for_dropbox_feedback(uc,orgunitNeeded,dropboxNeeded,dir,dbfFile,'1.4')
File "/Users/sprinkle/work/teaching/ece275-2015F/grading/valence/sprinkle/production/sprinkleutil.py", line 791, in upload_feedback_files_for_dropbox_feedback
result = attach_uploaded_file(uc,org_unit_id,folder_id,entity_id,fileKey,'temp.zip')
File "/Users/sprinkle/work/teaching/ece275-2015F/grading/valence/sprinkle/production/sprinkleutil.py", line 523, in attach_uploaded_file
return d2lservice._fetch_content(r)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/d2lvalence_util/service.py", line 46, in _fetch_content
r.raise_for_status()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/requests/models.py", line 851, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found
I appreciate any pointers from anyway who has successfully carried out this sequence of operations. My d2l admins don't have bandwidth to support me, and I'm not sure if I can access the logs at all, so I don't know how to improve debugging on my side. Thanks in advance!
For your attach stage, it looks like you're sending in the fileKey as a query parameter. It needs to be a POST form parameter, right?
So the content-type for the POST would be application/x-www-form-urlencoded, and the POST body itself would be
fileKey=qafSoMUYAm&fileName=SomeFileName.zip
For example, see How to mimic an HTML form submission in a POST request.
I'm using Flask-uploads to upload files to my Flask server. The max size allowed is set by using flaskext.uploads.patch_request_class(app, 16 * 1024 * 1024).
My client application (A unit test) uses requests to post a file that is to large.
I can see that my server returnes a HTTP response with status 413: Request Entity Too Large. But the client raises an exception in the requests code
ConnectionError: HTTPConnectionPool(host='api.example.se', port=80): Max retries exceeded with url: /images (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
My guess is that the server disconnect the receving socket and sends the reponse back to the client. But when the client gets a broken sending socket, it raises an exception and skips the response.
Questions:
Are my guess about Flask-Uploads and requests correct?
Does Flask-Uploads and request handle the 413 error correct?
Should I expect that my client code gets back some html when the post are to large?
Update
Here is a simple example reproducing my problem.
server.py
from flask import Flask, request
app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 1024
#app.route('/post', methods=('POST',))
def view_post():
return request.data
app.run(debug=True)
client.py
from tempfile import NamedTemporaryFile
import requests
def post(size):
print "Post with size %s" % size,
f = NamedTemporaryFile(delete=False, suffix=".jpg")
for i in range(0, size):
f.write("CoDe")
f.close()
# Post
files = {'file': ("tempfile.jpg", open(f.name, 'rb'))}
r = requests.post("http://127.0.0.1:5000/post", files=files)
print "gives status code = %s" % r.status_code
post(16)
post(40845)
post(40846)
result from client
Post with size 16 gives status code = 200
Post with size 40845 gives status code = 413
Post with size 40846
Traceback (most recent call last):
File "client.py", line 18, in <module>
post(40846)
File "client.py", line 13, in post
r = requests.post("http://127.0.0.1:5000/post", files=files)
File "/opt/python_env/renter/lib/python2.7/site-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/opt/python_env/renter/lib/python2.7/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python_env/renter/lib/python2.7/site-packages/requests/sessions.py", line 357, in request
resp = self.send(prep, **send_kwargs)
File "/opt/python_env/renter/lib/python2.7/site-packages/requests/sessions.py", line 460, in send
r = adapter.send(request, **kwargs)
File "/opt/python_env/renter/lib/python2.7/site-packages/requests/adapters.py", line 354, in send
raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=5000): Max retries exceeded with url: /post (Caused by <class 'socket.error'>: [Errno 32] Broken pipe)
my versions
$ pip freeze
Flask==0.10.1
Flask-Mail==0.9.0
Flask-SQLAlchemy==1.0
Flask-Uploads==0.1.3
Jinja2==2.7.1
MarkupSafe==0.18
MySQL-python==1.2.4
Pillow==2.1.0
SQLAlchemy==0.8.2
Werkzeug==0.9.4
blinker==1.3
itsdangerous==0.23
passlib==1.6.1
python-dateutil==2.1
requests==2.0.0
simplejson==3.3.0
six==1.4.1
virtualenv==1.10.1
voluptuous==0.8.1
wsgiref==0.1.2
Flask is closing the connection, you can set an error handler for the 413 error:
#app.errorhandler(413)
def request_entity_too_large(error):
return 'File Too Large', 413
Now the client should get a 413 error, note that I didn't test this code.
Update:
I tried recreating the 413 error, and I didn't get a ConnectionError exception.
Here's a quick example:
from flask import Flask, request
app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 1024
#app.route('/post', methods=('POST',))
def view_post():
return request.data
app.run(debug=True)
After running the file, I used the terminal to test requests and sending large data:
>>> import requests
>>> r = requests.post('http://127.0.0.1:5000/post', data={'foo': 'a'})
>>> r
<Response [200]>
>>> r = requests.post('http://127.0.0.1:5000/post', data={'foo': 'a'*10000})
>>> r
<Response [413]>
>>> r.status_code
413
>>> r.content
'<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">\n<title>413 Request Entity Too Large</title
>\n<h1>Request Entity Too Large</h1>\n<p>The data value transmitted exceeds the capacity limit.</p>\n'
As you can see, we got a response from flask 413 error and requests didn't raise an exception.
By the way I'm using:
Flask: 0.10.1
Requests: 2.0.0
RFC 2616, the specification for HTTP 1.1, says:
10.4.14 413 Request Entity Too Large
The server is refusing to process a request because the request
entity is larger than the server is willing or able to process. The
server MAY close the connection to prevent the client from continuing
the request.
If the condition is temporary, the server SHOULD include a Retry-
After header field to indicate that it is temporary and after what
time the client MAY try again.
This is what's happening here: flask is closing the connection to prevent the client from continuing the upload, which is giving you the Broken pipe error.
Based on this github issue answers (https://github.com/benoitc/gunicorn/issues/1733#issuecomment-377000612)
#app.before_request
def handle_chunking():
"""
Sets the "wsgi.input_terminated" environment flag, thus enabling
Werkzeug to pass chunked requests as streams. The gunicorn server
should set this, but it's not yet been implemented.
"""
transfer_encoding = request.headers.get("Transfer-Encoding", None)
if transfer_encoding == u"chunked":
request.environ["wsgi.input_terminated"] = True
I am using urllib2 to post data to a form. The problem is that the form replies with a 302 redirect. According to Python HTTPRedirectHandler the redirect handler will take the request and convert it from POST to GET and follow the 301 or 302. I would like to preserve the POST method and the data passed to the opener. I made an unsuccessful attempt at a custom HTTPRedirectHandler by simply adding data=req.get_data() to the new Request.
I am sure this has been done before so I thought I would make a post.
Note: this is similar to this post and this one but I don't want to prevent the redirect I just want to keep the POST data.
Here is my HTTPRedirectHandler that does not work
class MyHTTPRedirectHandler(urllib2.HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
"""Return a Request or None in response to a redirect.
This is called by the http_error_30x methods when a
redirection response is received. If a redirection should
take place, return a new Request to allow http_error_30x to
perform the redirect. Otherwise, raise HTTPError if no-one
else should try to handle this url. Return None if you can't
but another Handler might.
"""
m = req.get_method()
if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
or code in (301, 302, 303) and m == "POST"):
# Strictly (according to RFC 2616), 301 or 302 in response
# to a POST MUST NOT cause a redirection without confirmation
# from the user (of urllib2, in this case). In practice,
# essentially all clients do redirect in this case, so we
# do the same.
# be conciliant with URIs containing a space
newurl = newurl.replace(' ', '%20')
return Request(newurl,
headers=req.headers,
data=req.get_data(),
origin_req_host=req.get_origin_req_host(),
unverifiable=True)
else:
raise HTTPError(req.get_full_url(), code, msg, headers, fp)
This is actually a really bad thing to do the more I thought about it. For instance, if I submit a form to
http://example.com/add (with post data to add a item)
and the response is a 302 redirect to http://example.com/add and I post the same data that I posted the first time I will end up in an infinite loop. Not sure why I didn't think of this before. I'll leave the question here just as a warning to anyone else thinking about doing this.