We Keep Coding

Flask URL encoding mis [duplicate]

My application frequently takes URL encoded strings as a URL parameter. Often these strings look like paths with a leading slash. IE /file/foo. In flask, I have an endpoint that takes a path parameter that I send a URL encoded path to. So I have something that looks like:
import flask
app = flask.Flask("Hello World")
#app.route("/blah/<path:argument>", methods=["GET"])
def foo(argument):
return "GOT: %s" % argument
if __name__ == "__main__":
app.run(debug=True)
This works great if I visit this URL:
http://localhost:5000/blah/cats%2F
returns:
GOT: cats/
But a leading slash with %2F fails with 404 in the case of GET and 405 in the case of POST. In other words, this 404s:
http://localhost:5000/blah/%2Fcats
In my research on this problem, I was lead to believe here that URL encoding was sufficient to sole the problem. However that doesn't appear to be the case.

This is because of how Werkzeug parses urls. It decodes the encoded slashes before parsing the route, so they still appear as leading slashes. There are bug reports about this:
https://github.com/mitsuhiko/flask/issues/900
https://github.com/mitsuhiko/werkzeug/pull/478
The second link provides a patch to perform this decoding after routing, but it is not merged.
It looks like the best solution at this point is to follow Martijn's answer here.

One way to get around this without defining your own PathConverter is having two route filters:
import flask
app = flask.Flask("Hello World")
#app.route("/blah/<path:argument>", methods=["GET"])
#app.route("/blah//<path:argument>", methods=["GET"])
def foo(argument):
return "GOT: %s" % argument
if __name__ == "__main__":
app.run(debug=True)
Hitting this with:
http://localhost:5000/blah/%2Fcats
Gives me:
GOT: cats
And with:
http://localhost:5000/blah//cats
Gives me:
GOT: cats
But a better (cleaner) solution is probably the one described in this SO answer: Flask route using path with leading slash

i can't take the "?" character in the URL of django url

I try to code to django a redirect fonction. I provide the url, and i want to redirect to the provided URL.
in my urls.py:
urlpatterns = patterns('',
url(r'^redirect/(?P<name>.*)$', redirect),
# ...
)
when i test the function with a standard link (ex: google.com), it works perfectly.
when i test the function with a link that containt a "?" character, only the part before the "?" is taken into account.
Example :
"GET /redirect/http://www.polyvore.com/lords_liverpool_hey_jude_tee/thing?id=53713291 HTTP/1.1" 302 0
name = http://www.polyvore.com/lords_liverpool_hey_jude_tee/thing
the ?id=53713291 is not taken into account....
i though that .* means all kind character, wrong?
Do you know what happens ? and how to take the entiere url what ever the character it contains?
thank you very much for your help.

You seems to don't understand how URL works. Everything after the ? is parsed as arguments for your current view. If you print data in your request.GET dict, you'll find something like:
{'id': 53713291}
The only way to fix that is to urlencode your URL in the argument and decode it before the redirection.
>>> import urllib
>>> urllib.quote_plus("http://www.polyvore.com/lords_liverpool_hey_jude_tee/thig?id=53713291")
'http%3A%2F%2Fwww.polyvore.com%2Flords_liverpool_hey_jude_tee%2Fthing%3Fid%3D5313291'
# You should make your URL with this value, for example:
# /redirect/http%3A%2F%2Fwww.polyvore.com%2Flords_liverpool_hey_jude_tee%2Fthing%3Fid%3D5313291
# And in your view, use unquote_plus before the redirection:
>>> urllib.unquote_plus('http%3A%2F%2Fwww.polyvore.com%2Flords_liverpool_hey_jude_tee%2Fthing%3Fid%3D5313291')
'http://www.polyvore.com/lords_liverpool_hey_jude_tee/thing?id=5313291'
More information about Query String on Wikipedia.

You're passing in a regex, so you need to escape the special characters (ie, \?)
But if you're trying to pass in querystring parameters you'll want to handle that differently: https://stackoverflow.com/a/3711911

How to ensure URLs generated by Pyramid's route_url and route_path are valid?

Using URL Dispatch in Pyramid and pattern matching inside the route definition, I would like to ensure that URLs generated by route_path are valid and match the given route - what's the best way to do that?
This is my route definition - I want /numbers followed by a numeric string that's equivalent to 1 or greater...
config.add_route('numbers', 'numbers/{n:[1-9]\d*}/')
Then when I generate a route in a view for example, this is fine:
request.route_url('numbers', n=123)
It returns /numbers/123/ which is fine...
However, if I do...
request.route_url('numbers', n='chicken')
Pyramid returns /numbers/chicken/ which clearly doesn't match the pattern and will give a 404 when passed back to the app.
Is there a way that I can force Pyramid to check that the n matches the given pattern when it's generating the route? This would mean that I could be certain that route_url and route_path would give valid URLs that would work in the app.

You can use a pregenerator to add custom logic when generating urls. The regex patterns in the urls have only ever been used for matching purposes.
def check_n(request, elements, kw):
if not is_valid(kw['n']):
raise ValueError('invalid url parameter')
return elements, kw
config.add_route('numbers', 'numbers/{n:[1-9]\d*}/', pregenerator=check_n)
If you want to write something that validates the generated urls, it's not easy. The pregenerator doesn't know what route is being used and the route object matches paths, not kwargs. Your best bet is to wrap the route generation and validate the resulting url, instead of validating the kwargs themselves.
def validated_route_path(request, *args, **kwargs):
route_name = args[0]
path = request.route_path(*args, **kwargs)
route_intr = request.registry.introspector.get('routes', route_name)
route = route_intr['object']
if route.match(path) is None:
raise ValueError('invalid url generated')
return path
config.add_request_method(validated_route_path)
request.validated_route_path('numbers', n='chicken') # -> ValueError

Webapp2 strict_slash returns KeyError: 'Missing Argument' for url with trailing slash when method has 2 or more args... Bug?

I've built my urls as such:
#url = /index/test/argument/second
# Maps to the Index handler's test method and passes in the optional arguments 'argument' and 'second'
# So the handler function looks like this:
def test(argument=None,second=None):
print 'test'
I'm using strict_slash from webapp2 so the handlers with a trailing slash get redirected to handlers without a trailing slash.
#url = /index/ redirects perfectly to /index
#url = /index/test/ # KEYERROR!!
So even though index/test is routed before index/test/second, webapp2 is ignoring the redirect for trailing slashes and returning an error because it's looking (too hard) for the second argument. I think it should recognize there is no second argument so follow the strict_slash redirect route.
This works in all cases except with argument passing. Any insight, anyone?

To solve this problem you just need to set unique name argument for routes.

Best way to make subapps with Traversal

Ok so I have my apps, that takes requests from root / Almost everything is using traversal.
But i'd like to make on top of that site a rest api.
So I'm off with two choices. I either separate the that in two different apps and put that rest application to : rest.site.com, Or I can move it to site.com/rest/*traversal
If I'm doing "/rest/*traversal", I guess I'll have to add a route called rest_traversal where the traversal path will be *traversal with the route /rest/*traversal. I did that once for an admin page.
I was wondering if there was a cleanest way to do that. I tried to use virtual_root, but as I understand virtual_root is actually getting added to the path for traversal.
like having virtual_root = /cms and requesting /fun will create the following path /cms/fun
I on the other hand wish to have /cms/fun turned into /fun

I know this has been answered already, but in case someone arrives here looking for another possible way to make "subapps" and using them in pyramid, I wanted to point out that some interesting things can be done with pyramid.wsgi
"""
example of wsgiapp decorator usage
http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/api/wsgi.html
"""
from pyramid.wsgi import wsgiapp2, wsgiapp
from pyramid.config import Configurator
from webob import Request, Response
import pprint
# define some apps
def wsgi_echo(environ, start_response):
"""pretty print out the environ"""
response = Response(body=pprint.pformat({k: v for k, v in environ.items()
if k not in ["wsgi.errors",
"wsgi.input",
"SCRIPT_NAME"]}))
return response(environ, start_response)
print Request.blank("/someurl").send(wsgi_echo).body
# convert wsgi app to a pyramid view callable
pyramid_echo = wsgiapp(wsgi_echo)
pyramid_echo_2 = wsgiapp2(wsgi_echo)
# wire up a pyramid application
config = Configurator()
config.add_view(pyramid_echo, name="foo") # /foo
config.add_view(pyramid_echo, name="bar") # /bar
config.add_view(pyramid_echo_2, name="foo_2") # /foo
config.add_view(pyramid_echo_2, name="bar_2") # /bar
pyramid_app = config.make_wsgi_app()
#call some urls
foo_body = Request.blank("/foo").send(pyramid_app).body
bar_body = Request.blank("/bar").send(pyramid_app).body
foo_body_2 = Request.blank("/foo_2").send(pyramid_app).body
bar_body_2 = Request.blank("/bar_2").send(pyramid_app).body
# both should be different because we arrived at 2 different urls
assert foo_body != bar_body, "bodies should not be equal"
# should be equal because wsgiapp2 fixes stuff before calling
# application in fact there's an additional SCRIPT_NAME in the
# environment that we are filtering out
assert foo_body_2 == bar_body_2, "bodies should be equal"
# so how to pass the path along? like /foo/fuuuu should come back
# /fuuuu does it
foo_body = Request.blank("/foo_2/fuuuu").send(pyramid_app).body
assert "'/fuuuu'," in foo_body, "path didn't get passed along"
# tldr: a wsgi app that is decorated with wsgiapp2 will recieve data
# as if it was mounted at "/", any url generation it has to do should
# take into account the SCRIPT_NAME variable that may arrive in the
# environ when it is called

If you're using traversal already, why not just use it to return your "rest API root" object when Pyramid traverses to /rest/? From there, everything will work naturally.
class ApplicationRoot(object):
def __getitem__(self, name):
if name == "rest":
return RestAPIRoot(parent=self, name=name)
...
If your "application tree" and "API tree" have the same children and you want to have different views registered for them depending on which branch of the tree the child is located in, you can use containment view predicate to register your API views, so they will only match when the child is inside the "API branch":
containment
This value should be a reference to a Python class or interface that a
parent object in the context resource’s lineage must provide in order
for this view to be found and called. The resources in your resource
tree must be “location-aware” to use this feature.
If containment is not supplied, the interfaces and classes in the
lineage are not considered when deciding whether or not to invoke the
view callable.
Another approach would be not to build a separate "API tree" but to use your "main" application's "URI-space" as RESTful API. The only problem with this is that GET and possibly POST request methods are already "taken" on your resources and mapped to your "normal" views which return HTML or consume HTTP form POSTs. There are numerous ways to work around this:
register the API views with a separate name, so, say GET /users/123 would return HTML and GET /users/123/json would return a JSON object. Similarly, POST /users/123 would expect HTTP form to be posted and POST /users/123/json would expect JSON. A nice thing about this approach is that you can easily add, say, an XML serializer at GET /users/123/xml.
use custom view predicates so GET /users/123 and GET /users/123?format=json are routed to different views. Actually, there's a built-in request_param predicate for that since Pyramid 1.2
use xhr predicate to differentiate requests based on HTTP_X_REQUESTED_WITH header or accept predicate to differentiate on HTTP_ACCEPT header