We Keep Coding

Twisted Producer was never unregistered

I am writing a custom SSL proxy with Twisted. I keep running in to an issue that happens every so often and I cant figure out what the problem is.
When I try to connect the client transport to the server's transport through the registerProducer function exactly as twisted.protocols.portforward functions I keep getting this error.
File "/opt/Memory/Mobile/Proxy/forwarder.py", line 40, in connectionMade
self.peer.transport.registerProducer(self.transport, True)
File "/usr/lib/python3.9/site-packages/twisted/protocols/tls.py", line 602, in registerProducer
self.transport.registerProducer(producer, True)
File "/usr/lib/python3.9/site-packages/twisted/internet/_newtls.py", line 233, in registerProducer
FileDescriptor.registerProducer(self, producer, streaming)
File "/usr/lib/python3.9/site-packages/twisted/internet/abstract.py", line 104, in registerProducer
raise RuntimeError(
builtins.RuntimeError: Cannot register producer <twisted.protocols.tls._ProducerMembrane object at 0x7fb5799f0910>, because producer <twisted.protocols.tls._ProducerMembrane object at 0x7fb579b474c0> was never unregistered.
Here are my Inherited Classes from twisted?
from twisted.internet import reactor
from twisted.internet import ssl
from twisted.protocols import portforward
from twisted.internet import protocol
from twisted.python import log
import sys
##SSLProxy base class that will be inherited
class SSLProxy(protocol.Protocol):
noisy = True
peer = None
def setPeer(self, peer):
#log.msg("SSLProxy.setPeer")
self.peer = peer
def connectionLost(self, reason):
#log.msg("SSLProxy.connectionLost")
if self.peer is not None:
self.peer.transport.loseConnection()
self.peer = None
elif self.noisy:
log.msg("Unable to connect to peer: {}".format(reason))
def dataReceived(self, data):
#log.msg("SSLProxy.dataReceived")
if self.peer is not None:
self.peer.transport.write(data)
##Foward data from Proxy to => Remote Server
class SSLProxyClient(SSLProxy):
def connectionMade(self):
#log.msg("SSLProxyClient.connectionMade")
self.peer.setPeer(self)
self.transport.registerProducer(self.peer.transport, True)
self.peer.transport.registerProducer(self.transport, True)
# We're connected, everybody can read to their hearts content.
self.peer.transport.resumeProducing()
class SSLProxyClientFactory(protocol.ClientFactory):
protocol = SSLProxyClient
def setServer(self, server):
#log.msg("SSLProxyClientFactory.setServer")
self.server = server
def buildProtocol(self, *args, **kw):
#log.msg("SSLProxyClientFactory.buildProtocol")
prot = protocol.ClientFactory.buildProtocol(self, *args, **kw)
prot.setPeer(self.server)
return prot
def clientConnectionFailed(self, connector, reason):
#log.msg("SSLProxyClientFactory.clientConnectionFailed")
self.server.transport.loseConnection()
class SSLProxyServer(SSLProxy):
clientProtocolFactory = SSLProxyClientFactory
reactor = None
def connectionMade(self):
log.msg("SSLProxyServer.connectionMade")
#Get Current SSL Context
ssl_context = self.transport._tlsConnection.get_context()
#Hack to get SNI to do two functions in diffrent classes
ssl_context._finishSNI = self.SNICallback
def SNICallback(self, connection):
#log.msg("SSLProxyServer.SNICallback: {}".format(connection))
#print(connection.get_context().new_host)
self.transport.pauseProducing()
#self.transport.transport.pauseProducing()
#print(dir())
self.dst_host, self.dst_port = connection.get_context().new_host
#Setup Clients
self.client = self.clientProtocolFactory()
self.client.setServer(self)
#Start stuff
log.msg('Redirecting to {}:{}'.format(self.dst_host, self.dst_port))
if self.reactor is None:
self.reactor = reactor
log.msg("Making Connection to Dest Server: {}:{}".format(self.dst_host, self.dst_port))
self.reactor.connectSSL(self.dst_host, self.dst_port, self.client, ssl.ClientContextFactory())
#self.transport.resumeProducing()
#Client -> Proxy
def dataReceived(self, data):
log.msg("SSLProxyServer.dataReceived: {}".format(data))
#Call Inherited Function
super().dataReceived(data)
class SSLProxyFactory(protocol.Factory):
"""Factory for port forwarder."""
protocol = SSLProxyServer
def __init__(self):
super().__init__()
#log.msg("SSLProxyFactory.__init__")
def sslToSSL(localport, remotehost, remoteport, serverContextFactory):
log.msg("SSL on localhost:{} forwarding to SSL {}:{}".format(localport, remotehost, remoteport))
return reactor.listenSSL(localport, SSLProxyFactory(), serverContextFactory)
Any guidance would be appreciated.

I found that if you just unregister the current Producer you can register the new one.
##Foward data from Proxy to => Remote Server
class SSLProxyClient(SSLProxy):
def connectionMade(self):
#log.msg("SSLProxyClient.connectionMade")
self.peer.setPeer(self)
self.transport.registerProducer(self.peer.transport, True)
self.peer.transport.unregisterProducer()
self.peer.transport.registerProducer(self.transport, True)
# We're connected, everybody can read to their hearts content.
self.peer.transport.resumeProducing()

How can I pass parameters to a RequestHandler?

The Python documentation includes an example of creating an HTTP server:
def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
server_address = ('', 8000)
httpd = server_class(server_address, handler_class)
httpd.serve_forever()
A RequestHandler class is provided to the Server, which then takes care of instantiating the handler automatically.
Let's say I want to pass in custom parameters to the request handler when it's created. How can and should I do that?
More specifically, I want to pass in parameters from the command line, and having to access sys.argv inside the request handler class seems unnecessarily clunky.
It seems like this should be possible by overriding parts of the Server class, but I feel like I'm overlooking a simpler and better solution.

I solved this in my code using "partial application".
Example is written using Python 3, but partial application works the same way in Python 2:
from functools import partial
from http.server import HTTPServer, BaseHTTPRequestHandler
class ExampleHandler(BaseHTTPRequestHandler):
def __init__(self, foo, bar, qux, *args, **kwargs):
self.foo = foo
self.bar = bar
self.qux = qux
# BaseHTTPRequestHandler calls do_GET **inside** __init__ !!!
# So we have to call super().__init__ after setting attributes.
super().__init__(*args, **kwargs)
def do_HEAD(self):
self.send_response(200)
self.send_header('Content-type', 'text/plain')
self.end_headers()
def do_GET(self):
self.do_HEAD()
self.wfile.write('{!r} {!r} {!r}\n'
.format(self.foo, self.bar, self.qux)
.encode('utf8'))
# We "partially apply" the first three arguments to the ExampleHandler
handler = partial(ExampleHandler, sys.argv[1], sys.argv[2], sys.argv[3])
# .. then pass it to HTTPHandler as normal:
server = HTTPServer(('', 8000), handler)
server.serve_forever()
This is very similar to a class factory, but in my opinion it has a couple of subtle advantages:
partial objects are much easier to introspect for what's inside them than nested classes defined and returned by factory functions.
partial objects can be serialized with pickle in modern Python, whereas nested class definitions inside factory functions cannot (at least not without going out of your way to code a __reduce__ method on the class to make it possible).
In my limited experience explicit "pre-attaching" of arguments with partial to an otherwise Pythonic and normal class definition is easier (less cognitive load) to read, understand, and verify for correctness than a nested class definition with the parameters of the wrapping function buried somewhere inside it.
The only real disadvantage is that many people are unfamiliar with partial - but in my experience it is better for everyone to become familiar with partial anyway, because partial has a way of popping up as an easy and composable solution in many places, sometimes unexpectedly, like here.

Use a class factory:
def MakeHandlerClassFromArgv(init_args):
class CustomHandler(BaseHTTPRequestHandler):
def __init__(self, *args, **kwargs):
super(CustomHandler, self).__init__(*args, **kwargs)
do_stuff_with(self, init_args)
return CustomHandler
if __name__ == "__main__":
server_address = ('', 8000)
HandlerClass = MakeHandlerClassFromArgv(sys.argv)
httpd = HTTPServer(server_address, HandlerClass)
httpd.serve_forever()

At the time of this writing all the answers here essentially stick to the (very awkward) intention that the author of the socketserver module seemed to have that the handler passed in be a class (i.e. constructor). Really the only thing that's required of the handler is that it's callable, so we can work around the socketserver API by making instances of our handler class callable and having them run the superclass's __init__ code when called. In Python 3:
class MyHandler(http.server.BaseHTTPRequestHandler):
def __init__(self, message):
self.message = message
def __call__(self, *args, **kwargs):
"""Handle a request."""
super().__init__(*args, **kwargs)
def do_GET(self):
self.send_response(200)
self.end_headers()
self.wfile.write(self.message.encode("utf-8"))
This keeps the superclass "constructor" call out of __init__ which eliminates the possibility of dispatching a request (from the superclass's constructor) before the subclass's constructor is finished. Note that the __init__ override must be present to divert execution even if it's not needed for initialization; an empty implementation using pass would work.
With this design the weird interface is hidden and using the API looks more natural:
handler = MyHandler("Hello world")
server = http.server.HTTPServer(("localhost", 8000), handler)
server.serve_forever()

I would just comment on Thomas Orozco's answer but since I can't..
Perhaps this will help others who also run into this problem. Before Python3, Python has "old-style" classes, and BaseHTTPRequestHandler seems to be one of them. So, the factory should look like
def MakeHandlerClassFromArgv(init_args):
class CustomHandler(BaseHTTPRequestHandler, object):
def __init__(self, *args, **kwargs):
do_stuff_with(self, init_args)
super(CustomHandler, self).__init__(*args, **kwargs)
return CustomHandler
to avoid errors like TypeError: must be type, not classobj.

Why not just subclass the RequestHandler ?
class RequestHandler(BaseHTTPRequestHandler):
a_variable = None
class Server(HTTPServer):
def serve_forever(self, variable):
self.RequestHandlerClass.a_variable = variable
HTTPServer.serve_forever(self)
def run(server_class=Server, handler_class=RequestHandler):
server_address = ('', 8000)
httpd = server_class(server_address, handler_class)
variable = sys.argv
httpd.serve_forever(variable)

Ref Subclassing the HTTPServer is another option. Variables on the server are accessible in the Request Handler methods via self.server.context. It basically works like this:
class MyHTTPServer(HTTPServer):
def __init__(self, *args, **kwargs):
HTTPServer.__init__(self, *args, **kwargs)
self.context = SomeContextObject()
class MyHandler(BaseHTTPRequestHandler):
def do_GET(self):
context = self.server.context
...
# Drawback, Notice that you cannot actually pass the context parameter during constructor creation, but can do it within the __init__ of the MyHTTPServer
server = MyHTTPServer(('', port), MyHandler)
server.serve_forever()

If you do not need instance properties, but only class properties you could use this approach:
def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
server_address = ('', 8000)
httpd = server_class(server_address, handler_class)
httpd.RequestHandlerClass.my_custom_variable = "hello!"
httpd.serve_forever()
or maybe you could:
def run(server_class=HTTPServer, handler_class=BaseHTTPRequestHandler):
server_address = ('', 8000)
httpd = server_class(server_address, handler_class)
httpd.my_custom_variable = "hello!"
httpd.serve_forever()
and retrieve in your RequestHandler with:
self.server.my_custom_variable

Using a lambda is a pretty simple way to create a new function that takes the request handler args and creates your custom class.
Here I want to pass a variable that will be used in do_POST(), and set the directory used by SimpleHTTPRequestHandler, so setup calls
HTTPServer(('', 8001), lambda *_: _RequestHandler("[1, 2]", *_, directory=sys.path[0]))
Full program:
from http.server import HTTPServer, SimpleHTTPRequestHandler
import sys
class _RequestHandler(SimpleHTTPRequestHandler):
def __init__(self, x, *args, **kwargs):
self.x = x # NEEDS TO HAPPEN BEFORE super().__init__()
super().__init__(*args, **kwargs)
def _set_headers(self):
self.send_response(200)
self.send_header('Content-type', 'application/json')
self.end_headers()
def do_POST(self):
print("POST")
length = int(self.headers.get('content-length'))
message = self.rfile.read(length).decode('utf-8')
print(message)
self._set_headers()
self.wfile.write(self.x.encode('utf-8'))
def run_server():
server_address = ('', 8001)
httpd = HTTPServer(server_address, lambda *_: _RequestHandler("[1, 2]", *_, directory=sys.path[0]))
print('serving http://localhost:8001')
httpd.serve_forever()
if __name__ == '__main__':
run_server()

Never do it with a global. Use the factory described in other answers.
CONFIG = None
class MyHandler(BaseHTTPRequestHandler):
def __init__(self, ...
self.config = CONFIG # CONFIG is now 'stuff'
if __name__ == "__main__":
global CONFIG
CONFIG = 'stuff'
server_address = ('', 8000)
httpd = HTTPServer(server_address, MyHandler)
httpd.serve_forever()
(except maybe in the privacy of your own home)

Passing a Queue to ThreadedHTTPServer

I would like to pass a Queue object to a base ThreadedHTTPServer implementation. My existing code works just fine, but I would like a safe way to send calls to and from my HTTP requests. Normally this would probably be handled by a web framework, but this is a HW limited environment.
My main confusion comes on how to pass the Queue (or any) object to allow access to other modules in my environment.
The base code template I have currently running:
import base64,threading,urlparse,urllib2,os,re,cgi,sys,time
import Queue
class DemoHttpHandler(BaseHTTPRequestHandler):
def __init__(self, request, client_address, server,qu):
BaseHTTPRequestHandler.__init__(self, request, client_address, server)
def do_GET(self):
...
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
"""Handle requests in a separate thread."""
def main():
listen_interface = "localhost"
listen_port = 2323
server = startLocalServer.ThreadedHTTPServer((listen_interface, listen_port), startLocalServer.DemoHttpHandler)
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
print 'started httpserver thread...'

Your code does not run but I modified it so that it will run:
import base64,threading,urlparse,urllib2,os,re,cgi,sys,time
import Queue
class DemoHttpHandler(BaseHTTPRequestHandler):
def __init__(self, request, client_address, server):
BaseHTTPRequestHandler.__init__(self, request, client_address, server)
self.qu = server.qu # save the queue here.
def do_GET(self):
...
self.qu # access the queue self.server.qu
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
"""Handle requests in a separate thread."""
def main():
listen_interface = "localhost"
listen_port = 2323
qu = Queue.Queue()
server = startLocalServer.ThreadedHTTPServer((listen_interface, listen_port), startLocalServer.DemoHttpHandler)
server.qu = qu # store the queue in the server
server_thread = threading.Thread(target=server.serve_forever)
server_thread.daemon = True
server_thread.start()
print 'started httpserver thread...'

Threaded BaseHTTPServer, one thread per request

I'm trying to create an multi-threaded web server using BaseHttpServer and ThreadingMixIn (as seen on various examples). Pseudo code would be something like:
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
pass
def do_POST(self):
pass
class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
"""Handle requests in a separate thread."""
if __name__ == '__main__':
server = ThreadedHTTPServer(('localhost', 9999), Handler)
print 'Starting server, use <Ctrl-C> to stop'
server.serve_forever()
This works as expected, but my problem is that not every request gets a thread, but threading is done per URL. I've tested it like this: I have an URL bound to execute the following method:
import time
import datetime
def request_with_pause(self):
print datetime.datetime.now().strftime("%H:%M:%S.%f"), 'REQUEST RECEIVED'
time.sleep(10)
print datetime.datetime.now().strftime("%H:%M:%S.%f"), 'SENT RESPONSE'
It works fine, except when I call the url twice with a 5 second pause (click the URL, wait 5 seconds and click it another time) - both "responses" arrive after 10 seconds (response of first click).

In Python 2.7:
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
from threading import Thread
class ThreadedHTTPServer(HTTPServer):
def process_request(self, request, client_address):
thread = Thread(target=self.__new_request, args=(self.RequestHandlerClass, request, client_address, self))
thread.start()
def __new_request(self, handlerClass, request, address, server):
handlerClass(request, address, server)
self.shutdown_request(request)
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
self.wfile.write("hello world")
server = ThreadedHTTPServer(('', 80), Handler)
#server.serve_forever()
You can find the main source code of the HTTPServer class in SocketServer.py which you can find in the Lib folder in the Python directory. (HTTPServer is inherited from TCPServer, TCPServer is inherited from BaseServer.)
The important line is 315:
def process_request(self, request, client_address):
self.finish_request(request, client_address)
self.shutdown_request(request)
def finish_request(self, request, client_address):
self.RequestHandlerClass(request, client_address, self)
In this point the server create new request object with your Handler class. The BaseRequestHandler constructor automatically call the self.setup(), self.handle() and the self.finish() method.
So what I did was to override the process_request method to move this stuff in a new thread.

How to silent/quiet HTTPServer and BasicHTTPRequestHandler's stderr output?

I am writing a simple http server as part of my project. Below is a skeleton of my script:
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class MyHanlder(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
self.wfile.write('<html><body><p>OK</p></body></html>')
httpd = HTTPServer(('', 8001), MyHanlder)
httpd.serve_forever()
My question: how do I suppress the stderr log output my script produces every time a client connects to my server?
I have looked at the HTTPServer class up to its parent, but was unable to find any flag or function call to achieve this. I also looked at the BaseHTTPRequestHandler class, but could not find a clue. I am sure there must be a way. If you do, please share with me and others; I appreciate your effort.

This will probably do it:
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
class MyHandler(BaseHTTPRequestHandler):
def do_GET(self):
self.send_response(200)
self.send_header('Content-type', 'text/html')
self.end_headers()
self.wfile.write('<html><body><p>OK</p></body></html>')
def log_message(self, format, *args):
return
httpd = HTTPServer(('', 8001), MyHandler)
httpd.serve_forever()