I have some experience with different web frameworks (Django, web.py, Pyramid and CherryPy), and I'm wondering in which one will it be easier and hopefully cleaner to implement a route dispatcher to a different "view/handler" based on the "Accept" header and the HTTP method e.g.:
Accept: application/json
POST /post/
is handled different than:
Accept: text/html
POST /post/
So the request gets routed to the particular view of the corresponding handler of the MIME "application/json" and the HTTP method "POST".
I do know how to implement something like that in CherryPy, but I lose the use of the CherryPy tools for the internal redirection of the request because I'm calling the specific method directly instead of automagically from the dispatcher. Another option is to implement a full new dispatcher from scratch, but that's the last option.
I'm aware of the alternative to use extensions in the url like /post.json or /post/.json, but I'm looking to keep the same url?
If all you are looking for is one framework that can do this easily, then use pyramid.
Pyramid view definitions are made with predicates, not just routes, and a view only matches if all predicates match. One such predicate is the accept predicate, which does exactly what you want; make view switching depending on the Accept header easy and simple:
from pyramid.view import view_config
#view_config(route_name='some_api_name', request_method='POST', accept='application/json')
def handle_someapi_json(request):
# return JSON
#view_config(route_name='some_api_name', request_method='POST', accept='text/html')
def handle_someapi_html(request):
# return HTML
I needed to do this in Django, and so I wrote a piece of middleware to make it possible: http://baltaks.com/2013/01/route-requests-based-on-the-http-accept-header-in-django
Here is the code:
# A simple middleware component that lets you use a single Django
# instance to serve multiple versions of your app, chosen by the client
# using the HTTP Accept header.
# In your settings.py, map a value you're looking for in the Accept header
# to a urls.py file.
# HTTP_HEADER_ROUTING_MIDDLEWARE_URLCONF_MAP = {
# u'application/vnd.api-name.v1': 'app.urls_v1'
# }
from django.conf import settings
class HTTPHeaderRoutingMiddleware:
def process_request(self, request):
try:
for content_type in settings.HTTP_HEADER_ROUTING_MIDDLEWARE_URLCONF_MAP:
if (request.META['HTTP_ACCEPT'].find(content_type) != -1):
request.urlconf = settings.HTTP_HEADER_ROUTING_MIDDLEWARE_URLCONF_MAP[content_type]
except KeyError:
pass # use default urlconf (settings.ROOT_URLCONF)
def process_response(self, request, response):
return response
I'm not suite sure what you mean by "internal redirection", but if you look at the code you can see that tools.accept is a really thin wrapper around lib.cptools.accept, which you can call from your own code easily. Hand it a list of Content-Types your server can send, and it will tell you which one the client prefers, or raise 406 if the types you emit and the types the client accepts don't overlap.
Related
I have encountered an issue, as I have to create a cookie in the backend, which I will later use to send a request from the frontend. Both apps are on the same domain. This is the general idea behind it: https://levelup.gitconnected.com/secure-frontend-authorization-67ae11953723.
Frontend - Sending GET request to Backend
#app.get('/')
async def homepage(request: Request, response_class=HTMLResponse):
keycloak_code = 'sksdkssdk'
data = {'code': keycloak_code}
url_post = 'http://127.0.0.1:8002/keycloak_code'
post_token=requests.get(url=url_post, json = data )
return 'Sent'
if __name__ == '__main__':
uvicorn.run(app, host='local.me.me', port=7999,debug=True)
Backend
#app.get("/keycloak_code")
def get_tokens(response: Response, data: dict):
code = data['code']
print(code)
....
requests.get(url='http://local.me.me:8002/set')
return True
#app.get("/set")
async def createcookie(response: Response):
r=response.set_cookie(key='tokic3', value='helloworld', httponly=True)
return True
if __name__ == '__main__':
uvicorn.run(app, host='local.me.me', port=8002, log_level="debug")
When I open the browser and access http://local.me.me:8002/set, I can see that the cookie is created.
But when I make a GET request from my frontend to backend to the same URL, the request is received—as I can see in the terminal—but the backend does not create the cookie. Does anyone know what I might be doing wrong?
I have tried different implementations from FastAPI docs, but none has similar use cases.
127.0.0.1 and localhost (or local.me.me in your case) are two different domains (and origins). Hence, when making a request you need to use the same domain you used for creating the cookie. For example, if the cookie was created for local.me.me domain, then you should use that domain when sending the request. See related posts here, as well as here and here.
You also seem to have a second FastAPI app (listenning on a different port) acting as your frontend (as you say). If that's what you are trying to do, you would need to use Session Objects in Python requests module, or preferably, use a Client instance from httpx library, in order to persist cookies across requests. The advantage of httpx is that it offers an asynchronous API as well, using the httpx.AsyncClient(). You can find more details and examples in this answer, as well as here and here.
I have a Django project that contains a second, third-party Django app. In one of my views, I need to query a view from the other app (within the same Django project).
Currently, it works something like this:
import requests
def my_view(request):
data = requests.get('http://localhost/foo/bar').json()
# mangle data
return some_response
Is there a clean way to send the request without going all the way through DNS and the webserver and just go directly to the other app's view (ideally going through the middleware as well of course)
A Django view function accepts a request and returns a response object. There is no reason that a view cannot invoke another view by constructing a request (or cloning or passing its own) and interpreting the response. (c.f. the testing framework).
Of course, if the other view has undesirable side-effects, then the controlling view will have to unwind them. Working within a transaction should allow it to delve in the results of the view it invoked, and then abort the transaction and perform its own.
You can use urllib as shown below
import urllib, urllib3, urllib.request
url = "http://abcd.com" # API URL
postdata = urllib.parse.urlencode(values).encode("utf-8")
req = urllib.request.Request(url)
with urllib.request.urlopen(req, data=postdata) as response:
resp = response.read()
print(resp)
I have a thorny problem that I can't seem to get to grips with. I am
currently writing unit tests for a django custom auth-backend. On our
system we actually have two backends: one the built-in django backend
and the custom backend that sends out requests to a Java based API
that returns user info in the form of XML. Now, I am writing unit
tests so I don't want to be sending requests outside the system like
that, I'm not trying to test the Java API, so my question is how can I
get around this and mock the side-effects in the most robust way.
The function I am testing is something like this, where the url
settings value is just the base url for the Java server that
authenticates the username and password data and returns the xml, and the service value is
just some magic for building the url query, its unimportant for
us:
#staticmethod
def get_info_from_api_with_un_pw(username, password, service=12345):
url = settings.AUTHENTICATE_URL_VIA_PASSWORD
if AUTH_FIELD == "username":
params = {"nick": username, "password": password}
elif AUTH_FIELD == "email":
params = {"email": username, "password": password}
params["service"] = service
encoded_params = urlencode([(k, smart_str(v, "latin1")) for k, v in params.items()])
try:
# get the user's data from the api
xml = urlopen(url + encoded_params).read()
userinfo = dict((e.tag, smart_unicode(e.text, strings_only=True))
for e in ET.fromstring(xml).getchildren())
if "nil" in userinfo:
return userinfo
else:
return None
So, we get the xml, parse it into a dict and if the key nil is present
then we can return the dict and carry on happy and authenticated.
Clearly, one solution is just to find a way to somehow override or
monkeypatch the logic in the xml variable, I found this answer:
How can one mock/stub python module like urllib
I tried to implement something like that, but the details there are
very sketchy and I couldn't seem to get that working.
I also captured the xml response and put it in a local file in the
test folder with the intention of finding a way to use that as a mock
response that is passed into the url parameter of the test function,
something like this will override the url:
#override_settings(AUTHENTICATE_URL_VIA_PASSWORD=(os.path.join(os.path.dirname(__file__), "{0}".format("response.xml"))))
def test_get_user_info_username(self):
self.backend = RemoteAuthBackend()
self.backend.get_info_from_api_with_un_pw("user", "pass")
But that also needs to take account of the url building logic that the
function defines, (i.e. "url + encoded_params"). Again, I could rename
the response file to be the same as the concatenated url but this is becoming
less like a good unit-test for the function and more of a "cheat", the whole
thing is just getting more and more brittle all the time with these solutions, and its really just a fixture anyway, which is also something I want to avoid if
at all possible.
I also wondered if there might be a way to serve the xml on the django development server and then point the function at that? It seems like a saner solution, but much googling gave me no clues if such a thing would be possible or advisable and even then I don't think that would be a test to run outside of the development environment.
So, ideally, I need to be able to somehow mock a "server" to
take the place of the Java API in the function call, or somehow serve
up some xml payload that the function can open as its url, or
monkeypatch the function from the test itself, or...
Does the mock library have the appropriate tools to do such things?
http://www.voidspace.org.uk/python/mock
So, there are two points to this question 1) I would like to solve my
particular problem in a clean way, and more importantly 2) what are
the best practices for cleanly writing Django unit-tests when you are
dependent on data, cookies, etc. for user authentication from a remote
API that is outside of your domain?
The mock library should work if used properly. I prefer the minimock library and I wrote a small base unit testcase (minimocktest) that helps with this.
If you want to integrate this testcase with Django to test urllib you can do it as follows:
from minimocktest import MockTestCase
from django.test import TestCase
from django.test.client import Client
class DjangoTestCase(TestCase, MockTestCase):
'''
A TestCase class that combines minimocktest and django.test.TestCase
'''
def _pre_setup(self):
MockTestCase.setUp(self)
TestCase._pre_setup(self)
# optional: shortcut client handle for quick testing
self.client = Client()
def _post_teardown(self):
TestCase._post_teardown(self)
MockTestCase.tearDown(self)
Now you can use this testcase instead of using the Django test case directly:
class MySimpleTestCase(DjangoTestCase):
def setUp(self):
self.file = StringIO.StringIO('MiniMockTest')
self.file.close = self.Mock('file_close_function')
def test_urldump_dumpsContentProperly(self):
self.mock('urllib2.urlopen', returns=self.file)
self.assertEquals(urldump('http://pykler.github.com'), 'MiniMockTest')
self.assertSameTrace('\n'.join([
"Called urllib2.urlopen('http://pykler.github.com')",
"Called file_close_function()",
]))
urllib2.urlopen('anything')
self.mock('urllib2.urlopen', returns=self.file, tracker=None)
urllib2.urlopen('this is not tracked')
self.assertTrace("Called urllib2.urlopen('anything')")
self.assertTrace("Called urllib2.urlopen('this is mocked but not tracked')", includes=False)
self.assertSameTrace('\n'.join([
"Called urllib2.urlopen('http://pykler.github.com')",
"Called file_close_function()",
"Called urllib2.urlopen('anything')",
]))
Here's the basics of the solution that I ended up with for the record. I used the Mock library itself rather than Mockito in the end, but the idea is the same:
from mock import patch
#override_settings(AUTHENTICATE_LOGIN_FIELD="username")
#patch("mymodule.auth_backend.urlopen")
def test_get_user_info_username(self, urlopen_override):
response = "file://" + os.path.join(os.path.dirname(__file__), "{0}".format("response.xml"))
# mock patch replaces API call
urlopen_override.return_value = urlopen(response)
# call the patched object
userinfo = RemoteAuthBackend.get_info_from_api_with_un_pw("user", "pass")
assert_equal(type(userinfo), dict)
assert_equal(userinfo["nick"], "user")
assert_equal(userinfo["pass"], "pass")
The problem is simple, we would like CherryPy to not log access log for a particular exposed method/API that gets called.
Basically when this API gets called, there are some parameters in the query string of the URL which are very sensitive and if leaked, would expose potential security. Naturally this is a /GET request and unfortunately it is the only way the parameters could be passed, since its a redirect(302) from an external service to this web server.
If it would not log the URL, that would serve the purpose as well.
So, is there a way that we can filter logging messages in access log by API's, URL's etc?
Thanks in advance for the help.
cherrypy uses Python's standard logging module by default, so you can just add a custom filter. This example will ignore any GET request with /foo as the path prefix:
import logging
class IgnoreURLFilter(logging.Filter):
# simple example of log message filtering
def __init__(self, ignore):
self.ignore = 'GET /' + ignore
def filter(self, record):
return self.ignore not in record.getMessage()
app = cherrypy.tree.mount( YourApplication() )
app.log.access_log.addFilter( IgnoreURLFilter('foo') )
cherrypy.engine.start()
I want to proxy requests made to my Flask app to another web service running locally on the machine. I'd rather use Flask for this than our higher-level nginx instance so that we can reuse our existing authentication system built into our app. The more we can keep this "single sign on" the better.
Is there an existing module or other code to do this? Trying to bridge the Flask app through to something like httplib or urllib is proving to be a pain.
I spent a good deal of time working on this same thing and eventually found a solution using the requests library that seems to work well. It even handles setting multiple cookies in one response, which took a bit of investigation to figure out. Here's the flask view function:
from dotenv import load_dotenv # pip package python-dotenv
import os
#
from flask import request, Response
import requests # pip package requests
load_dotenv()
API_HOST = os.environ.get('API_HOST'); assert API_HOST, 'Envvar API_HOST is required'
#api.route('/', defaults={'path': ''}) # ref. https://medium.com/#zwork101/making-a-flask-proxy-server-online-in-10-lines-of-code-44b8721bca6
#api.route('/<path>')
def redirect_to_API_HOST(path): #NOTE var :path will be unused as all path we need will be read from :request ie from flask import request
res = requests.request( # ref. https://stackoverflow.com/a/36601467/248616
method = request.method,
url = request.url.replace(request.host_url, f'{API_HOST}/'),
headers = {k:v for k,v in request.headers if k.lower() == 'host'},
data = request.get_data(),
cookies = request.cookies,
allow_redirects = False,
)
#region exlcude some keys in :res response
excluded_headers = ['content-encoding', 'content-length', 'transfer-encoding', 'connection'] #NOTE we here exclude all "hop-by-hop headers" defined by RFC 2616 section 13.5.1 ref. https://www.rfc-editor.org/rfc/rfc2616#section-13.5.1
headers = [
(k,v) for k,v in res.raw.headers.items()
if k.lower() not in excluded_headers
]
#endregion exlcude some keys in :res response
response = Response(res.content, res.status_code, headers)
return response
Update April 2021: excluded_headers should probably include all "hop-by-hop headers" defined by RFC 2616 section 13.5.1.
I have an implementation of a proxy using httplib in a Werkzeug-based app (as in your case, I needed to use the webapp's authentication and authorization).
Although the Flask docs don't state how to access the HTTP headers, you can use request.headers (see Werkzeug documentation). If you don't need to modify the response, and the headers used by the proxied app are predictable, proxying is staightforward.
Note that if you don't need to modify the response, you should use the werkzeug.wsgi.wrap_file to wrap httplib's response stream. That allows passing of the open OS-level file descriptor to the HTTP server for optimal performance.
My original plan was for the public-facing URL to be something like http://www.example.com/admin/myapp proxying to http://myapp.internal.example.com/. Down that path leads madness.
Most webapps, particularly self-hosted ones, assume that they're going to be running at the root of a HTTP server and do things like reference other files by absolute path. To work around this, you have to rewrite URLs all over the place: Location headers and HTML, JavaScript, and CSS files.
I did write a Flask proxy blueprint which did this, and while it worked well enough for the one webapp I really wanted to proxy, it was not sustainable. It was a big mess of regular expressions.
In the end, I set up a new virtual host in nginx and used its own proxying. Since both were at the root of the host, URL rewriting was mostly unnecessary. (And what little was necessary, nginx's proxy module handled.) The webapp being proxied to does its own authentication which is good enough for now.