I'm porting an app from web.py to Flask, mainly because web.py support for Python 3 is spotty and there seems to be less and less interest in web.py.
But what I can't find in Flask/Werkzeug is a way to use the router to do dispatching of internal requests within my application. The app is structured such that there will be a lot of intra-application calls, and in web.py I handle these more or less as follows:
app = web.application(....)
def callUrl(url, method, env, data):
parserUrl = url lib.parse.urlparse(url)
if parsedUrl.scheme == '' and parsedUrl.netloc == '':
# local call
res = app.request(url, method=method, data=data, env=env)
...
else:
assert env == {}
res = requests.request(url, method=method, data=data)
....
I am trying to find a way to do something similar with Flask, but apparently I am looking in the wrong places. Can someone point me in the right direction?
Ok, answering my own question. The solution I chose was to basically re-implement app.request from web.py by filling an environ dictionary with all the required WSGI variables (REQUEST_METHOD, PATH_INFO, etc), including wsgi.input as an io.BytesIO() object that feeds the correct data into the WSGI app.
Then I created a suitable start_response() method to save the headers, and called
resultData = app.wsgi_app(environ, start_response)
The flask app goes through all the motions of pushing requests and environments and does all the routing, and I get the returned data back in resultData (and the headers with any errors and such have been passed to my start_response method).
Related
When using Angular 5 with Flask and ng build --aot --watch, I am having major issues getting Angular's routing and refreshing the page to work.
I have come across many attempts to fix this, but none of them when applied to my project work (and the example projects provided when I try it there also do not seem to work).
What happens (depending on the thing I try) are these:
It can't find the path at all and I get a Jinja2 error
It can't find the document for localhost:5015/ (the port I'm using) and I get a 404 and a white screen
It loads index.html, but upon refresh it gets a 404 (implying that the browser is attempting to fetch the page from the server, not using the Angular routing system)
I get the same issue as this one: Syntax Error in Angular App: Unexpected token <
I am under the impression this is a normal issue with all SPAs and that the solution is to always redirect the server to index.html. We have done this (AFAIK), and attempted multiple ways to do this:
Our structure is roughly the following:
- ProjectDir
- reverseProxy_Calls
- scripts
- swagger_yaml_definitions
- templates
- e2e
- dist
- src
- app
And our server.py contains this (among other stuff)
Swagger(app)
app.template_folder = "./templates/dist/"
app.static_folder = "./templates/dist/"
app.static_path = "./templates/dist/"
app.static_url_path = "./templates/dist/"
app.instance_path = "./templates/dist/"
config = settings.config()
config.read("settings.cfg")
#app.route('/', defaults={'path': ''}, methods=['GET', 'POST'])
#app.route('/<path:path>', methods=['GET', 'POST'])
def get_resource(path): # pragma: no cover
# If we are devmode, proxy through flask.
if config.getKey("devmode") == True:
url = "http://localhost:%s/%s" % (config.getKey('ng_port'), path)
j = requests.get(url, stream=True, params=request.args)
return make_response(j.text)
# return the results
# return render_template('dist/index.html')
# return app.send_static_file("index.html")
return send_from_directory('dist/', path)
if __name__ == '__main__':
app.run(debug=True, port=int(config.getKey("port"))) # , host='0.0.0.0')
^ A variety of ways here show various attempts to handle this.
I have also looked at and tried projects that use variations of the following:
Flask Cli
Flask-Script
Flask-Bootstrap
None of them have solved this problem when I have tried them.
I have been told that using the hashRoute location strategy in Angular rather than the History PushState strategy will also work, however when I attempted to do that it had the same behavior of losing the page upon refresh (additionally we wish to use server side rendering in the future, and hashRoute location strategy prevents this possibility, among many other downsides).
Possibly related is that Flask seems to reload or re-render the page when it does find the index.html and routes initially, wiping anything printed to the browser dev console. I have seen this in some of the flask examples, but not all, and I'm uncertain what causes this.
When running via ng serve --aot --prod, things work exactly correctly and as expected. It's only when serving via Flask that things break down.
Apologies for the long question and thank you for your time in advance.
I am under the impression this is a normal issue with all SPAs and
that the solution is to always redirect the server to index.html. We
have done this (AFAIK), and attempted multiple ways to do this
You also need to redirect all HTTP errors to index.html.
I have a python server that is currently keeping track of the location of all the buses in my university and generating predictions of arrivals to specific locations.
Now, I wanted to attach a lightweight REST API to this server but I have been running intro problems.
I tried using flask with the following code:
from flask import Flask, jsonify
from PredictionWrapper import *
import threading
class RequestHandler():
def __init__(self,predictionWrapper):
self.app = Flask(__name__)
self.predictor = predictionWrapper
self.app.debug = False
self.app.add_url_rule('/<route>/<int:busStop>','getSinglePrediction',self.getSinglePrediction)
t = threading.Thread(target=self.app.run, kwargs={'host':'0.0.0.0', 'port':80, 'threaded':True})
t.start()
def getSinglePrediction(self, route, busStop):
# TODO Get the actual prediction with given parameters
prediction = self.predictor.getPredictionForStop(route, busStop)
return jsonify({'busStop': busStop, 'prediction': prediction})
def getStopPrediction(self, busStop):
# TODO Get the actual prediction with given parameters
return jsonify({'busStop': busStop, 'prediction': 2})
def run(self):
self.app.run(host='0.0.0.0', port=80, threaded=True)
The problem is that I have been encountering the error below after about half a day of running the server. Note that no requests were made to the server around the time it failed with the following error:
ERROR:werkzeug: - - [01/May/2016 09:55:55] code 400, message Bad request syntax ('\x02\xfd\xb1\xc5!')
After investigating I believe I need to deploy to a WSGI production server. But I have no clue what it means in this specific approach given that 1)the flask server is being threaded in order to run the rest of the prediction application, and 2)I am using classes which none of the documentation uses.
Any help on how to setup the wsgi file with apache, gunicorn, or the technology of your choice would be appreciated. Also, any comments on a better approach on making a non-blocking REST API would be helpful.
Let me know if you need any further clarification!
Not sure if this can actually solve your problem but you can use the coroutine based web server gevent. They have a WSGI server that you can use if that's what you meant by saying that you need to deploy a WSGI production server.
If you want to implement the server to your flask application just do the following:
from gevent.pywsgi import WSGIServer
app = Flask(__name__)
http_server = WSGIServer(('', 5000), app)
http_server.serve_forever()
Gevent in general is a very powerful tool and by issuing context switches as necessary it can handle multiple clients very easily. Also, gevent fully supports flask.
First thing to do would be to put exception handling to deal with bad JSON request data (which maybe is what's happening) something like
def getSinglePrediction(self, route, busStop):
try:
prediction = self.predictor.getPredictionForStop(route, busStop)
return jsonify({'busStop': busStop, 'prediction': prediction})
except:
return jsonify({'busStop': 'error', 'prediction': 'error'})
I'm trying to write a middleware that rewrites POST requests to a different method when it finds a "_method" parameter in the body. Someone on the Internet wrote this piece of code:
from werkzeug import Request
class MethodRewriteMiddleware(object):
def __init__(self, app, input_name='_method'):
self.app = app
self.input_name = input_name
def __call__(self, environ, start_response):
request = Request(environ)
if self.input_name in request.form:
method = request.form[self.input_name].upper()
if method in ['GET', 'POST', 'PUT', 'DELETE']:
environ['REQUEST_METHOD'] = method
return self.app(environ, start_response)
As I understand the code, it parses the form, retrieves a possible "_method" parameter and if it's found and whitelisted it will overwrite the current method. It works fine for DELETE requests and it rewrites the method with no problems. However, when I try to send a regular, non-rewritten POST this middleware makes the whole app hang. My best guess is that since I accessed the body in the middleware, the body is no longer available for the application so it hangs forever. However, this doesn't seem to impact rewritten requests, so the code deepest code path (checking the whitelist) works correctly but the other code path is somehow destroying/blocking the request.
I don't think it's relevant, but I'm mounting this middleware on top of a Flask app.
EDIT: I think trying to access the request from a handler in Flask is blocking. Does Flask use mutexes or something like that internally?
I'm not even sure how to debug this.
environ is a dictionary with a wsgi.input key that is a stream object. The manual way to do what you're looking for is to read the data, make any changes and then create a new string to replace the old stream in the environ dictionary. The example below (without any error handling or other important things) allows you to work with the request body:
def __call__(self, environ, start_response):
body = environ['wsgi.input'].read()
... do stuff to body ...
new_stream = io.ByteIO(modified_body)
environ['wsgi.input'] = new_stream
return self.app(environ, start_response)
Does anyone know of a way to set (mock) the User-Agent of the request object provided by FLask (Werkzeug) during unit testing?
As it currently stands, when I attempt to obtain details such as the request.headers['User-Agent'] a KeyError is raised as the Flask test_client() doesn't set these up. (See partial stack trace below)
When attempting to get the User-Agent from the request object in a Flask project during unit testing, a KeyError is raised.
File "/Users/me/app/rest/app.py", line 515, in login
if request.headers['User-Agent']:
File "/Users/me/.virtualenvs/app/lib/python2.7/site-packages/werkzeug/datastructures.py", line 1229, in __getitem__
return self.environ['HTTP_' + key]
KeyError: 'HTTP_USER_AGENT'
-- UPDATE --
Along with the (accepted) solution below, the environ_base hint lead me to this other SO solution. The premise of this solution is to create a wrapper class for the Flask app and override the call method to automatically set the environment variables. This way, the variables are set for all calls. So, the solution I ended up implementing is creating this proxy class:
class FlaskTestClientProxy(object):
def __init__(self, app):
self.app = app
def __call__(self, environ, start_response):
environ['REMOTE_ADDR'] = environ.get('REMOTE_ADDR', '127.0.0.1')
environ['HTTP_USER_AGENT'] = environ.get('HTTP_USER_AGENT', 'Chrome')
return self.app(environ, start_response)
And then wrapping the WSGI container with that proxy:
app.wsgi_app = FlaskTestClientProxy(app.wsgi_app)
test_client = app.test_client()
You need to pass in environ_base when you call get() or post(). E.g.,
client = app.test_client()
response = client.get('/your/url/',
environ_base={'HTTP_USER_AGENT': 'Chrome, etc'})
Then your request.user_agent should be whatever you pass in, and you can access it via request.headers['User-Agent'].
See http://werkzeug.pocoo.org/docs/test/#testing-api for more info.
Even though what #chris-mckinnel wrote works, I wouldn't opt to use environ_base in this case.
You can easily do something as shown below:
with app.test_request_context('url', headers={'User-Agent': 'Your value'}):
pass
This should do the job and at the same time it keeps your code explicit - explicit is better than implicit.
If you want to know what you can pass to the test_request_context reference the EnvironBuilder definition; which can be found here.
I am writing a web application with Pyramid and would like to restrict the maximum length for POST requests, so that people can't post huge amount of data and exhaust all the memory on the server. However I looked pretty much everywhere I could think of (Pyramid, WebOb, Paster) and couldn't find any option to accomplish this. I've seen that Paster has limits for the number of HTTP headers, length each header, etc., but I didn't see anything for the size of the request body.
The server will be accepting POST requests only for JSON-RPC, so I don't need to allow huge request body sizes. Is there a way in the Pyramid stack of accomplishing this?
Just in case this is not obvious from the rest, a solution which has to accept and load the whole request body into memory before checking the length and returning a 4xx error code defeats the purpose of what's I'm trying to do, and is not what I'm looking for.
Not really a direct answer to your question. As far as I know, you can create a wsgi app that will load the request if the body is below the configuration setting you can pass it to the next WSGI layer. If it goes above you can stop reading and return an error directly.
But to be honest, I really don't see the point to do it in pyramid. For example, if you run pyramid behind a reverse proxy with nginx or apache or something else.. you can always limit the size of the request with the frontend server.
unless you want to run pyramid with Waitress or Paster directly without any proxy, you should handle body size in the front end server that should be more efficient than python.
Edit
I did some research, it isn't a complete answer but here is something that can be used I guess. You have to read environ['wsgi_input'] as far as I can tell. This is a file like object that receives chunk of data from nginx or apache for example.
What you really have to do is read that file until max lenght is reached. If it is reached raise an Error if it isn't continue the request.
You might want to have a look at this answer
You can do it in a variety of ways here's a couple of examples. one using wsgi middleware based on webob(installed when you install pyramid among other things). and one that uses pyramids event mechanism
"""
restricting execution based on request body size
"""
from pyramid.config import Configurator
from pyramid.view import view_config
from pyramid.events import NewRequest, subscriber
from webob import Response, Request
from webob.exc import HTTPBadRequest
import unittest
def restrict_body_middleware(app, max_size=0):
"""
this is straight wsgi middleware and in this case only depends on
webob. this can be used with any wsgi compliant web
framework(which is pretty much all of them)
"""
def m(environ, start_response):
r = Request(environ)
if r.content_length <= max_size:
return r.get_response(app)(environ, start_response)
else:
err_body = """
request content_length(%s) exceeds
the configured maximum content_length allowed(%s)
""" % (r.content_length, max_size)
res = HTTPBadRequest(err_body)
return res(environ, start_response)
return m
def new_request_restrict(event):
"""
pyramid event handler called whenever there is a new request
recieved
http://docs.pylonsproject.org/projects/pyramid/en/1.2-branch/narr/events.html
"""
request = event.request
if request.content_length >= 0:
raise HTTPBadRequest("too big")
#view_config()
def index(request):
return Response("HI THERE")
def make_application():
"""
make appplication with one view
"""
config = Configurator()
config.scan()
return config.make_wsgi_app()
def make_application_with_event():
"""
make application with one view and one event subsriber subscribed
to NewRequest
"""
config = Configurator()
config.add_subscriber(new_request_restrict, NewRequest)
return config.make_wsgi_app()
def make_application_with_middleware():
"""
make application with one view wrapped in wsgi middleware
"""
return restrict_body_middleware(make_application())
class TestWSGIApplication(unittest.TestCase):
def testNoRestriction(self):
app = make_application()
request = Request.blank("/", body="i am a request with a body")
self.assert_(request.content_length > 0, "content_length should be > 0")
response = request.get_response(app)
self.assert_(response.status_int == 200, "expected status code 200 got %s" % response.status_int)
def testRestrictedByMiddleware(self):
app = make_application_with_middleware()
request = Request.blank("/", body="i am a request with a body")
self.assert_(request.content_length > 0, "content_length should be > 0")
response = request.get_response(app)
self.assert_(response.status_int == 400, "expected status code 400 got %s" % response.status_int)
def testRestrictedByEvent(self):
app = make_application_with_event()
request = Request.blank("/", body="i am a request with a body")
self.assert_(request.content_length > 0, "content_length should be > 0")
response = request.get_response(app)
self.assert_(response.status_int == 400, "expected status code 400 got %s" % response.status_int)
if __name__ == "__main__":
unittest.main()