I'm currently writing a basic dispatch model server based on the Python Eventlet library (http://eventlet.net/doc/). Having looked at the WSGI docs on Eventlet (http://eventlet.net/doc/modules/wsgi.html), I can see that the eventlet.wsgi.server function logs the x-forwarded-for header in addition to the client IP address.
However, the way to obtain this is to attach a file-like object (the default which is sys.stderr) and then have the server pipe that to that object.
I would like to be able to obtain the client IP from within the application itself (i.e. the function that has start_response and environ as parameters). Indeed, an environ key would be perfect for this. Is there a way to obtain the IP address simply (i.e. through the environ dictionary or similar), without having to resort to redirecting the log object somehow?
What you want is in the wsgi environ, specifically environ['REMOTE_ADDR'].
However, if there is a proxy involved, then REMOTE_ADDR will be the address of the proxy, and the client address will be included (most likely) in HTTP_X_FORWARDED_FOR.
Here's a function that should do what you want, for most cases (all credit to Sævar):
def get_client_address(environ):
try:
return environ['HTTP_X_FORWARDED_FOR'].split(',')[-1].strip()
except KeyError:
return environ['REMOTE_ADDR']
You can easily see what is included in the wsgi environ by writing a simple wsgi app and pointing a browser at it, for example:
from eventlet import wsgi
import eventlet
from pprint import pformat
def show_env(env, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
return ['%s\r\n' % pformat(env)]
wsgi.server(eventlet.listen(('', 8090)), show_env)
And combining the two ...
from eventlet import wsgi
import eventlet
from pprint import pformat
def get_client_address(environ):
try:
return environ['HTTP_X_FORWARDED_FOR'].split(',')[-1].strip()
except KeyError:
return environ['REMOTE_ADDR']
def show_env(env, start_response):
start_response('200 OK', [('Content-Type', 'text/plain')])
return ['%s\r\n\r\nClient Address: %s\r\n' % (pformat(env), get_client_address(env))]
wsgi.server(eventlet.listen(('', 8090)), show_env)
Related
From the uwsgi documentation:
def application(env, start_response):
start_response('200 OK', [('Content-Type','text/html')])
return [b"Hello World"]
Is it possible to respond to http request(close http connection) and continue execution flow(without any usage of threads/queues/external services etc)?
like this:
def application(env, start_response):
start_response('200 OK', [('Content-Type','text/html')])
end_response(b"Hello World")
#HTTP connection is closed
#continue execution..
TL;DR - if you're using Django, you can skip to the end of the answer.
Yes, there is a way to do this. You can hook on to a .close() method of the object returned by your application callable (or alternatively, wsgi.file_wrapper returned through the env.
Check PEP333, specifically the server side spec part: https://peps.python.org/pep-0333/#the-server-gateway-side:
result = application(environ, start_response)
try:
for data in result:
if data: # don't send headers until body appears
write(data)
if not headers_sent:
write('') # send headers now if body was empty
finally:
if hasattr(result, 'close'):
result.close()
As you can see, result.close() will be called at the very end, by which point the data is already sent.
This is, of course, just some reference code, and it does not deal with upstream and terminating the connection before continuing execution. But well-behaved wsgi servers, such as uwsgi and (presumably) gunicorn, do.
They will signal to the upstream that the request has finished sending by closing the socket (or whatever the upstream protocol requires), and then call .close().
If you are using Django, you are already set, because it has request_finished signal. Here's how it works, it hooks to .close() as described above:
https://github.com/django/django/blob/57c7220280db19dc9dda0910b90cf1ceac50c66f/django/http/response.py#L323
Unfortunately, there is no way to continue the code execution after you have returned the response. It would be much easier if you use multithreading but if not you can workaround it in Flask by adding an AJAX call to your HTML response which will send a POST request to one of the server extra route whose handler function will be the execution code you want after returning the response. Here's one of the possible approach using Flask:
myflaskapp.py
from flask import Flask, render_template_string
import time
app = Flask(__name__)
#app.route('/run', methods=['POST'])
def run():
# this is where you put your "continue execution..." code
# below code is used to test if it runs after HTTP connection close
time.sleep(8)
print('Do something')
return ''
#app.route('/')
def index():
return render_template_string('''
Hello World!
<script src="//ajax.googleapis.com/ajax/libs/jquery/1.9.1/jquery.min.js"></script>
<script>
$(function() {
$.ajax({
type: "POST",
url: "{{ url_for('run') }}"
});
})
</script>
''')
if __name__ == "__main__":
app.run(host='0.0.0.0')
You can run the server on port 9091 with the command:
uwsgi --http 127.0.0.1:9091 --wsgi-file myflaskapp.py --callable app
To test if it is working or not, you can go to the address localhost:9091. If everything works well, you should see that the page is loaded immediately while the terminal will only print out Do something after 8 seconds have passed, indicating the function run executes after the HTTP connection is closed.
I am trying to alter the Pyramid hello world example so that any request to the Pyramid server serves the same page. i.e. all routes point to the same view. This is what iv got so far:
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
def hello_world(request):
return Response('Hello %(name)s!' % request.matchdict)
if __name__ == '__main__':
config = Configurator()
config.add_route('hello', '/*')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()
server = make_server('0.0.0.0', 8080, app)
server.serve_forever()
All iv done is change the line (from the hello world example):
config.add_route('hello', '/hello/{name}')
To:
config.add_route('hello', '/*')
So I want the route to be a 'catch-all'. Iv tried various variations and cant get it to work. Does anyone have any ideas?
Thanks in advance
The syntax for the catchall route (which is called "traversal subpath" in Pyramid) is *subpath instead of *. There's also *traverse which is used in hybrid routing which combines route dispatch and traversal. You can read about it here: Using *subpath in a Route Pattern
In your view function you'll then be able to access the subpath via request.subpath, which is a tuple of path segments caught by the catchall route. So, your application would look like this:
from wsgiref.simple_server import make_server
from pyramid.config import Configurator
from pyramid.response import Response
def hello_world(request):
if request.subpath:
name = request.subpath[0]
else:
name = 'Incognito'
return Response('Hello %s!' % name)
if __name__ == '__main__':
with Configurator() as config:
config.add_route('hello', '/*subpath')
config.add_view(hello_world, route_name='hello')
app = config.make_wsgi_app()
server = make_server('0.0.0.0', 8081, app)
server.serve_forever()
Don't do it via custom 404 handler, it smells of PHP :)
You could create a custom error handler (don't remember off the top of my head but it was in the Pyramid docs) and capture HTTP 404 errors, and redirect/render your catch-all route from there.
The link I'm thinking of: http://docs.pylonsproject.org/projects/pyramid//en/latest/narr/hooks.html
I've done something like this:
from pyramid.view import (
view_config,
forbidden_view_config,
notfound_view_config
)
from pyramid.httpexceptions import (
HTTPFound,
HTTPNotFound,
HTTPForbidden,
HTTPBadRequest,
HTTPInternalServerError
)
import transaction
import traceback
import logging
log = logging.getLogger(__name__)
#region Custom HTTP Errors and Exceptions
#view_config(context=HTTPNotFound, renderer='HTTPNotFound.mako')
def notfound(request):
if not 'favicon' in str(request.url):
log.error('404 not found: {0}'.format(str(request.url)))
request.response.status_int = 404
return {}
I think you should be able to redirect to a view from within there.
I am building a wsgi application. The production environment is Apache mod_wsgi and configured appropriately. For development, I use wsgiref.simple_server to serve the wsgi app locally. However, I would like my dev server to serve static content also with minimal overhead. For purposes of answering the question, assume I want to serve static file "/favicon.ico".
I have come across the package "static": http://lukearno.com/projects/static/ but found the documentation a bit lacking, and haven't been able to configure it to serve both static content and my application.
Here is a sample wsgi application.
from cgi import parse_qs
def application(environ, start_response):
qs = parse_qs(environ['QUERY_STRING'])
response_body, content_type = ('Hello World', ('Content-type', 'text/plain'))
content_length = 0
for s in response_body:
content_length += len(s)
status = '200 OK'
response_headers = [content_type,
('Content-Length', str(content_length))]
start_response(status, response_headers)
return response_body
if __name__ == '__main__':
from wsgiref.simple_server import make_server
# Instantiate the WSGI web server.
httpd = make_server('192.168.1.1', # The host name.
8080, # A port number where to wait for the request.
application # Our application object name, in this case a function.
)
httpd.serve_forever()
Note: This may be a duplicate of unanswered question Serving static content with python wsgiref? from December. I wanted to add more than just a comment and did not want to dig up a dead thread.
Let me propose the werkzeug package. It comes with a SharedDataMiddleware that solves precisely this task.
So i've looked around at a few things involving writting an HTTP Proxy using python and the Twisted framework.
Essentially, like some other questions, I'd like to be able to modify the data that will be sent back to the browser. That is, the browser requests a resource and the proxy will fetch it. Before the resource is returned to the browser, i'd like to be able to modify ANY (HTTP headers AND content) content.
This ( Need help writing a twisted proxy ) was what I initially found. I tried it out, but it didn't work for me. I also found this ( Python Twisted proxy - how to intercept packets ) which i thought would work, however I can only see the HTTP requests from the browser.
I am looking for any advice. Some thoughts I have are to use the ProxyClient and ProxyRequest classes and override the functions, but I read that the Proxy class itself is a combination of the both.
For those who may ask to see some code, it should be noted that I have worked with only the above two examples. Any help is great.
Thanks.
To create ProxyFactory that can modify server response headers, content you could override ProxyClient.handle*() methods:
from twisted.python import log
from twisted.web import http, proxy
class ProxyClient(proxy.ProxyClient):
"""Mangle returned header, content here.
Use `self.father` methods to modify request directly.
"""
def handleHeader(self, key, value):
# change response header here
log.msg("Header: %s: %s" % (key, value))
proxy.ProxyClient.handleHeader(self, key, value)
def handleResponsePart(self, buffer):
# change response part here
log.msg("Content: %s" % (buffer[:50],))
# make all content upper case
proxy.ProxyClient.handleResponsePart(self, buffer.upper())
class ProxyClientFactory(proxy.ProxyClientFactory):
protocol = ProxyClient
class ProxyRequest(proxy.ProxyRequest):
protocols = dict(http=ProxyClientFactory)
class Proxy(proxy.Proxy):
requestFactory = ProxyRequest
class ProxyFactory(http.HTTPFactory):
protocol = Proxy
I've got this solution by looking at the source of twisted.web.proxy. I don't know how idiomatic it is.
To run it as a script or via twistd, add at the end:
portstr = "tcp:8080:interface=localhost" # serve on localhost:8080
if __name__ == '__main__': # $ python proxy_modify_request.py
import sys
from twisted.internet import endpoints, reactor
def shutdown(reason, reactor, stopping=[]):
"""Stop the reactor."""
if stopping: return
stopping.append(True)
if reason:
log.msg(reason.value)
reactor.callWhenRunning(reactor.stop)
log.startLogging(sys.stdout)
endpoint = endpoints.serverFromString(reactor, portstr)
d = endpoint.listen(ProxyFactory())
d.addErrback(shutdown, reactor)
reactor.run()
else: # $ twistd -ny proxy_modify_request.py
from twisted.application import service, strports
application = service.Application("proxy_modify_request")
strports.service(portstr, ProxyFactory()).setServiceParent(application)
Usage
$ twistd -ny proxy_modify_request.py
In another terminal:
$ curl -x localhost:8080 http://example.com
For two-way proxy using twisted see the article:
http://sujitpal.blogspot.com/2010/03/http-debug-proxy-with-twisted.html
I am trying to make a search engine, running on the web. So I used mod_wsgi to get the query from the webpage by python. Then, I used zeromq to send the query to C++ searching program.
But the problem is, it seems zeromq and mod_wsgi does not work together.
It is definitely true that python can import zmq (I tested it) but when it runs on the web, it shows error message that (actually an error log from apache)
File "D:/wsgi_app/wsgi_app.py", line 2, in <module>, referer: http://localhost/
import zmq, referer: http://localhost/
File "D:\\util\\Python27\\lib\\site-packages\\zmq\\__init__.py", line 35, in <module>, referer: http://localhost/
from zmq.utils import initthreads # initialize threads, referer: http://localhost/
ImportError: DLL load failed: \xc1\xf6\xc1\xa4\xb5\xc8 \xb8\xf0\xb5\xe2\xc0\xbb \xc3\xa3\xc0\xbb \xbc\xf6 \xbe\xf8\xbd\xc0\xb4\xcf\xb4\xd9., referer: http://localhost/
I have no idea why wsgi cannot import zmq. By the way the source below is complete python code
from cgi import parse_qs, escape
import zmq
def application( # It accepts two arguments:
# environ points to a dictionary containing CGI like environment variables
# which is filled by the server for each received request from the client
environ,
# start_response is a callback function supplied by the server
# which will be used to send the HTTP status and headers to the server
start_response):
# get a query from the webpage :)
data = parse_qs(environ['QUERY_STRING'])
query = data.get('query', [''])[0]
query = escape(query) #prevent script injection
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect ("tcp://localhost:5555") #connect to C++ search server database
socket.send (query)
# build the response body possibly using the environ dictionary
response_body = 'The request method was %s' % environ['REQUEST_METHOD']
# HTTP response code and message
status = '200 OK'
# These are HTTP headers expected by the client.
# They must be wrapped as a list of tupled pairs:
# [(Header name, Header value)].
response_headers = [('Content-Type', 'text/plain'),
('Content-Length', str(len(response_body)))]
# Send them to the server using the supplied function
start_response(status, response_headers)
# Return the response body.
# Notice it is wrapped in a list although it could be any iterable.
return [response_body]
In your wsgi script, explicitly add the location of your dependencies:
import site
site.addsitedir(path_to_zeromq)