Recently I have been playing around with the HTTP Proxy in twisted. After much trial and error I think I finally I have something working. What I want to know though, is how, if it is possible, do I expand this proxy to also be able to handle HTTPS pages? Here is what I've got so far:
from twisted.internet import reactor
from twisted.web import http
from twisted.web.proxy import Proxy, ProxyRequest, ProxyClientFactory, ProxyClient
class HTTPProxyClient(ProxyClient):
def handleHeader(self, key, value):
print "%s : %s" % (key, value)
ProxyClient.handleHeader(self, key, value)
def handleResponsePart(self, buffer):
print buffer
ProxyClient.handleResponsePart(self, buffer)
class HTTPProxyFactory(ProxyClientFactory):
protocol = HTTPProxyClient
class HTTPProxyRequest(ProxyRequest):
protocols = {'http' : HTTPProxyFactory}
def process(self):
print self.method
for k,v in self.requestHeaders.getAllRawHeaders():
print "%s : %s" % (k,v)
print "\n \n"
ProxyRequest.process(self)
class HTTPProxy(Proxy):
requestFactory = HTTPProxyRequest
factory = http.HTTPFactory()
factory.protocol = HTTPProxy
reactor.listenSSL(8001, factory)
reactor.run()
As this code demonstrates, for the sake of example for now I am just printing out whatever is going through the connection. Is it possible to handle HTTPS with the same classes? If not, how should I go about implementing such a thing?
If you want to connect to an HTTPS website via an HTTP proxy, you need to use the CONNECT HTTP verb (because that's how a proxy works for HTTPS). In this case, the proxy server simply connects to the target server and relays whatever is sent by the server back to the client's socket (and vice versa). There's no caching involved in this case (but you might be able to log the hosts you're connecting to).
The exchange will look like this (client to proxy):
C->P: CONNECT target.host:443 HTTP/1.0
C->P:
P->C: 200 OK
P->C:
After this, the proxy simply opens a plain socket to the target server (no HTTP or SSL/TLS yet) and relays everything between the initial client and the target server (including the TLS handshake that the client initiates). The client upgrades the existing socket it has to the proxy to use TLS/SSL (by starting the SSL/TLS handshake). Once the client has read the '200' status line, as far as the client is concerned, it's as if it had made the connection to the target server directly.
I'm not sure about twisted, but I want to warn you that if you implement a HTTPS proxy, a web browser will expect the server's SSL certificate to match the domain name in the URL (address bar). The web browser will issue security warnings otherwise.
There are ways around this, such as generating certificates on the fly, but you'd need the root certificate to be trusted on the browser.
Related
I'm working on a project where we want to assign a whitelist packet filters for incoming traffic on a firewall and we are using python script with requests library to make some https requests to some servers outside of that network. For now the script is using ephemeral ports to connect to the servers, but we would like to make these https requests through specific ports. This would allow us to create strict whitelist for these ports.
How can I specify the port to the requests library through which the request should be sent? Script is currently using the following type of code to send the necessary requests.
response = requests.post(data[0], data=query, headers=headers, timeout=10)
This works, but I would now need to specify the port through which the http post request should be sent to allow for more strict packet filtering on the network. How could this port declaration be achieved? I have searched for solution to this from several sources already and came up with absolutely nothing.
requests is built on urllib3, which offers the ability to set a source address for connections; when you set the source address to ('', port_number) you tell it to use the default host name but pick a specific port.
You can set these options on the pool manager, and you tell requests to use a different pool manager by creating a new transport adapter:
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.poolmanager import PoolManager
class SourcePortAdapter(HTTPAdapter):
""""Transport adapter" that allows us to set the source port."""
def __init__(self, port, *args, **kwargs):
self._source_port = port
super(SourcePortAdapter, self).__init__(*args, **kwargs)
def init_poolmanager(self, connections, maxsize, block=False):
self.poolmanager = PoolManager(
num_pools=connections, maxsize=maxsize,
block=block, source_address=('', self._source_port))
Use this adapter in a session object, the following mounts the adapter for all HTTP and HTTPS connections, using 54321 as the source port:
s = requests.Session()
s.mount('http://', SourcePortAdapter(54321))
s.mount('https://', SourcePortAdapter(54321))
You can only set the one source port, limiting you to one active connection at a time. If you need to rotate between ports, register multiple adapters (one per URL) or re-register the catch-all mounts each time.
See the create_connection() utility function documentation for the details on the source_address option:
If source_address is set it must be a tuple of (host, port) for the socket to bind as a source address before making the connection. An host of '' or port 0 tells the OS to use the default.
As some of you may have seen recently I was struggling to get a simple Client/Server TCP connection up and running, where I could switch to TLS on issue of a command, as per the Twisted example at https://twistedmatrix.com/documents/14.0.0/core/howto/ssl.html#starttls-server, but modified to use certificate authentication. Well, the good news is that now works! Thanks to #Jean-PaulCalderone and #Glyph for their help.
However I now want to extend / transform my server side code into an ESMTP server - ie, let the client connect, server then advertises STARTTLS as part of its EHLO response, and if the client then responds with STARTTLS, negotiate and switch to a secure connection. I believe I need to modify the "TLSServer" class in my server code below to be inherited from twisted.mail.smtp.ESMTP instead of twisted.protocols.basic.LineReceiver - but not 100% sure here. Moreover, the API docs for t.m.s.ESMTP as found at http://twistedmatrix.com/documents/12.1.0/api/twisted.mail.smtp.ESMTP.html are a little thin on the ground (lots of undocumented methods).
Can anyone a) tell me if I am right regarding modifing the inheritance of my existing server code, and b) offer up some explanation of methods such as do_EHLO and ext_STARTTLS?
Ultimately I am looking for an SMTP server that will accept (secured) client connections, receive a mail from the client, and dump the mail to file.
My existing server code:
from twisted.internet import ssl, protocol, defer, task, endpoints
from twisted.protocols.basic import LineReceiver
from twisted.python.modules import getModule
from OpenSSL.crypto import load_privatekey, load_certificate, FILETYPE_PEM
class TLSServer(LineReceiver):
def lineReceived(self, line):
print("received: " + line)
if line == "STARTTLS":
print("-- Switching to TLS")
self.sendLine('READY')
self.transport.startTLS(self.factory.options)
def main(reactor):
caCertFile = open("/some/path/to/cacert.pem","r")
certFile = open("/some/path/to/server.crt","r")
keyFile = open("/some/path/to/server.key","r")
caCertData = caCertFile.read()
pKeyData = keyFile.read()
certData = certFile.read()
caCert = ssl.Certificate.loadPEM(caCertData)
cert = load_certificate(FILETYPE_PEM, certData)
pKey = load_privatekey(FILETYPE_PEM, pKeyData)
factory = protocol.Factory.forProtocol(TLSServer)
factory.options = ssl.CertificateOptions(privateKey=pKey, certificate=cert, trustRoot=caCert)
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
endpoint.listen(factory)
return defer.Deferred()
if __name__ == '__main__':
import starttls_server
task.react(starttls_server.main)
Thanks in advance!
You should not have your own TLSServer class. Instead, just use ESMTP directly as your server class. You've already identified the ext_STARTTLS method as interesting - basically, it already implements the logic that your TLSServer has in its lineReceived method.
Rather than doing factory.options = ..., you will want to pass ESMTP a contextFactory argument upon construction. At the simplest, you could do something like this:
sslCtxFactory = ssl.CertificateOptions(...)
factory = Factory.forProtocol(lambda: ESMTP(contextFactory=sslCtxFactory))
I believe that ESMTP advertises STARTTLS as an extension by default - at least, that's what a quick reading of the implementation of ESMTP.extensions() method says to me.
I am creating a proxy server in python, which is based on BaseHTTPServer.
What it does is create a connection to a squid proxy, identifies the browser request(GET, CONNECT, POST etc) and adds a proxy-authorization header to it, and then forwards this request to the squid proxy.
Problem is, as I understand, when I send a connect request, I should relay all the corresponding traffic to the squid proxy. But, as I can see in wireshark, the squid proxy doesn't reply to the 'Client Hello' part of the handshake, which I think is due to squid proxy not understanding binary data of SSL that I am just forwarding to it.
How do I process HTTPS requests in this case?
The code is more or less similar to TinyHTTPProxy : http://www.oki-osk.jp/esc/python/proxy/
RFC 2817 defines the CONNECT method. It is different from other HTTP methods in that the receiving proxy (your Python proxy) is directed to establish a raw TCP tunnel directly to the destination host (called the authority in the RFC).
A proxy can make no assumptions about the data that will be sent over that tunnel; it will not necessarily be HTTP – the client can use the tunnel to speak any protocol it likes. Indeed, SSL ≠ HTTP.
You have two options:
Open a TCP connection directly to the requested destination host.
Make a CONNECT request to your upstream proxy (Squid). This is within spec:
It may be the case that the proxy itself can only reach the
requested origin server through another proxy. In this case, the
first proxy SHOULD make a CONNECT request of that next proxy,
requesting a tunnel to the authority. A proxy MUST NOT respond
with any 2xx status code unless it has either a direct or tunnel
connection established to the authority.
Make sure that your request includes the required Host header.
CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
Proxy-Authorization: ...
I am using SSL tunneling with a proxy server to connect to a target server. I use http to connect to the proxy server and HTTPS to connect to the target server. The SSL tunneling works as it should and I can exchange HTTPS messages with the remote server, but there is a problem. The proxy server returns a header in its reply to urllib2's request to establish the SSL tunnel that I need to see, but I don't see a way to get access to it using urllib2 (Python 2.7.3).
I suppose I could theoretically implement the SSL tunneling handshake myself, but that would get me way deeper into the protocol than I want to be (or with which I feel comfortable).
Is there a way to get access to the reply using urllib2 when establishing the SSL tunnel?
UPDATE:
Here is the code that uses the proxy server to connect to the target server (the proxy server and the target server's URLs are not the actual ones):
proxy_handler = urllib2.ProxyHandler({'https': 'http://proxy.com'})
url_opener = urllib2.build_opener (proxy_handler)
request = urllib2.Request ('https://target_server.com/')
response = url_opener.open (request)
print response.headers.dict
I used WireShark to look at the message traffic. WireShark won't show me the bodies of the messages exchanged with the target server because they are encrypted, but I can see the body of the SSL Tunnel handshake. I can see the header that I'm interested coming back from the proxy server.
How are you calling the https page.
are you using
resp = urllib2.urlopen('https')
resp.info().headers
I am currently trying to pull together a basic SSL server in twisted. I pulled the following example right off their website:
from twisted.internet import ssl, reactor
from twisted.internet.protocol import Factory, Protocol
class Echo(Protocol):
def dataReceived(self, data):
"""As soon as any data is received, write it back."""
print "dataReceived: %s" % data
self.transport.write(data)
if __name__ == '__main__':
factory = Factory()
factory.protocol = Echo
print "running reactor"
reactor.listenSSL(8080, factory,
ssl.DefaultOpenSSLContextFactory(
"./test/privatekey.pem", "./test/cacert.pem"))
reactor.run()
I then tried to hit this server using firefox by setting the url to https://localhost:8080 yet I receive no response. I do, however, see the data arriving at the server. Any ideas why I'm not getting a response?
You're not sending an http header back to the browser, and you're not closing the connection
You've implemented an SSL echo server here, not an HTTPS server. Use the openssl s_client command to test it interactively, not firefox (or any other HTTP client, for that matter).