We Keep Coding

Client certificates and mutual authentication in Python [duplicate]

I want to make a little update script for a software that runs on a Raspberry Pi and works like a local server. That should connect to a master server in the web to get software updates and also to verify the license of the software.
For that I set up two python scripts. I want these to connect via a TLS socket. Then the client checks the server certificate and the server checks if it's one of the authorized clients. I found a solution for this using twisted on this page.
Now there is a problem left. I want to know which client (depending on the certificate) is establishing the connection. Is there a way to do this in Python 3 with twisted?
I'm happy with every answer.

In a word: yes, this is quite possible, and all the necessary stuff is
ported to python 3 - I tested all the following under Python 3.4 on my Mac and it seems to
work fine.
The short answer is
"use twisted.internet.ssl.Certificate.peerFromTransport"
but given that a lot of set-up is required to get to the point where that is
possible, I've constructed a fully working example that you should be able to
try out and build upon.
For posterity, you'll first need to generate a few client certificates all
signed by the same CA. You've probably already done this, but so others can
understand the answer and try it out on their own (and so I could test my
answer myself ;-)), they'll need some code like this:
# newcert.py
from twisted.python.filepath import FilePath
from twisted.internet.ssl import PrivateCertificate, KeyPair, DN
def getCAPrivateCert():
privatePath = FilePath(b"ca-private-cert.pem")
if privatePath.exists():
return PrivateCertificate.loadPEM(privatePath.getContent())
else:
caKey = KeyPair.generate(size=4096)
caCert = caKey.selfSignedCert(1, CN="the-authority")
privatePath.setContent(caCert.dumpPEM())
return caCert
def clientCertFor(name):
signingCert = getCAPrivateCert()
clientKey = KeyPair.generate(size=4096)
csr = clientKey.requestObject(DN(CN=name), "sha1")
clientCert = signingCert.signRequestObject(
csr, serialNumber=1, digestAlgorithm="sha1")
return PrivateCertificate.fromCertificateAndKeyPair(clientCert, clientKey)
if __name__ == '__main__':
import sys
name = sys.argv[1]
pem = clientCertFor(name.encode("utf-8")).dumpPEM()
FilePath(name.encode("utf-8") + b".client.private.pem").setContent(pem)
With this program, you can create a few certificates like so:
$ python newcert.py a
$ python newcert.py b
Now you should have a few files you can use:
$ ls -1 *.pem
a.client.private.pem
b.client.private.pem
ca-private-cert.pem
Then you'll want a client which uses one of these certificates, and sends some
data:
# tlsclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ClientEndpoint
from twisted.internet.ssl import (
PrivateCertificate, Certificate, optionsForClientTLS)
from twisted.internet.defer import Deferred, inlineCallbacks
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class SendAnyData(Protocol):
def connectionMade(self):
self.deferred = Deferred()
self.transport.write(b"HELLO\r\n")
def connectionLost(self, reason):
self.deferred.callback(None)
#inlineCallbacks
def main(reactor, name):
pem = FilePath(name.encode("utf-8") + b".client.private.pem").getContent()
caPem = FilePath(b"ca-private-cert.pem").getContent()
clientEndpoint = SSL4ClientEndpoint(
reactor, u"localhost", 4321,
optionsForClientTLS(u"the-authority", Certificate.loadPEM(caPem),
PrivateCertificate.loadPEM(pem)),
)
proto = yield clientEndpoint.connect(Factory.forProtocol(SendAnyData))
yield proto.deferred
import sys
react(main, sys.argv[1:])
And finally, a server which can distinguish between them:
# whichclient.py
from twisted.python.filepath import FilePath
from twisted.internet.endpoints import SSL4ServerEndpoint
from twisted.internet.ssl import PrivateCertificate, Certificate
from twisted.internet.defer import Deferred
from twisted.internet.task import react
from twisted.internet.protocol import Protocol, Factory
class ReportWhichClient(Protocol):
def dataReceived(self, data):
peerCertificate = Certificate.peerFromTransport(self.transport)
print(peerCertificate.getSubject().commonName.decode('utf-8'))
self.transport.loseConnection()
def main(reactor):
pemBytes = FilePath(b"ca-private-cert.pem").getContent()
certificateAuthority = Certificate.loadPEM(pemBytes)
myCertificate = PrivateCertificate.loadPEM(pemBytes)
serverEndpoint = SSL4ServerEndpoint(
reactor, 4321, myCertificate.options(certificateAuthority)
)
serverEndpoint.listen(Factory.forProtocol(ReportWhichClient))
return Deferred()
react(main, [])
For simplicity's sake we'll just re-use the CA's own certificate for the
server, but in a more realistic scenario you'd obviously want a more
appropriate certificate.
You can now run whichclient.py in one window, then python tlsclient.py a;
python tlsclient.py b in another window, and see whichclient.py print out
a and then b respectively, identifying the clients by the commonName
field in their certificate's subject.
The one caveat here is that you might initially want to put that call to
Certificate.peerFromTransport into a connectionMade method; that won't
work.
Twisted does not presently have a callback for "TLS handshake complete";
hopefully it will eventually, but until it does, you have to wait until you've
received some authenticated data from the peer to be sure the handshake has
completed. For almost all applications, this is fine, since by the time you
have received instructions to do anything (download updates, in your case) the
peer must already have sent the certificate.

DNS server using Python and Twisted - issues with asynchronous operation

I'm attempting to create a database-driven DNS server (specifically to handle MX records only, and pass everything else upstream) in Twisted using Python 2.7. The code below works (in terms of getting a result), but is not operating asynchronously. Instead, any DNS requests coming in block the entire program from taking any other requests until the first one has been replied to. We need this to scale, and at the moment we can't figure out where we've gone wrong. If anyone has a working example to share, or sees the issue, we'd be eternally grateful.
import settings
import db
from twisted.names import dns, server, client, cache
from twisted.application import service, internet
from twisted.internet import defer
class DNSResolver(client.Resolver):
def __init__(self, servers):
client.Resolver.__init__(self, servers=servers)
#defer.inlineCallbacks
def _lookup_mx_records(self, hostname, timeout):
# Check the DB to see if we handle this domain.
mx_results = yield db.get_domain_mx_record_list(hostname)
if mx_results:
defer.returnValue(
[([dns.RRHeader(hostname, dns.MX, dns.IN, settings.DNS_TTL,
dns.Record_MX(priority, forward, settings.DNS_TTL))
for forward, priority in mx_results]),
(), ()])
# If the hostname isn't in the DB, we forward
# to our upstream DNS provider (8.8.8.8).
else:
i = yield self._lookup(hostname, dns.IN, dns.MX, timeout)
defer.returnValue(i)
def lookupMailExchange(self, name, timeout=None):
"""
The twisted function which is called when an MX record lookup is requested.
:param name: The domain name being queried for (e.g. example.org).
:param timeout: Time in seconds to wait for the query response. (optional, default: None)
:return: A DNS response for the record query.
"""
return self._lookup_mx_records(name, timeout)
# App name, UID, GID to run as. (root/root for port 53 bind)
application = service.Application('db_driven_dns', 1, 1)
# Set the secondary resolver
db_dns_resolver = DNSResolver(settings.DNS_NAMESERVERS)
# Create the protocol handlers
f = server.DNSServerFactory(caches=[cache.CacheResolver()], clients=[db_dns_resolver])
p = dns.DNSDatagramProtocol(f)
f.noisy = p.noisy = False
# Register as a tcp and udp service
ret = service.MultiService()
PORT=53
for (klass, arg) in [(internet.TCPServer, f), (internet.UDPServer, p)]:
s = klass(PORT, arg)
s.setServiceParent(ret)
# Run all of the above as a twistd application
ret.setServiceParent(service.IServiceCollection(application))
EDIT #1
blakev suggested that I might not be using the generator correctly (which is certainly possible). But if I simplify this down a little bit to not even use the DB, I still cannot process more than one DNS request at a time. To test this, I have stripped the class down. What follows is my entire, runnable, test file. Even in this highly stripped-down version of my server, Twisted does not accept any more requests until the first one has come in.
import sys
import logging
from twisted.names import dns, server, client, cache
from twisted.application import service, internet
from twisted.internet import defer
class DNSResolver(client.Resolver):
def __init__(self, servers):
client.Resolver.__init__(self, servers=servers)
def lookupMailExchange(self, name, timeout=None):
"""
The twisted function which is called when an MX record lookup is requested.
:param name: The domain name being queried for (e.g. example.org).
:param timeout: Time in seconds to wait for the query response. (optional, default: None)
:return: A DNS response for the record query.
"""
logging.critical("Query for " + name)
return defer.succeed([
(dns.RRHeader(name, dns.MX, dns.IN, 600,
dns.Record_MX(1, "10.0.0.9", 600)),), (), ()
])
# App name, UID, GID to run as. (root/root for port 53 bind)
application = service.Application('db_driven_dns', 1, 1)
# Set the secondary resolver
db_dns_resolver = DNSResolver( [("8.8.8.8", 53), ("8.8.4.4", 53)] )
# Create the protocol handlers
f = server.DNSServerFactory(caches=[cache.CacheResolver()], clients=[db_dns_resolver])
p = dns.DNSDatagramProtocol(f)
f.noisy = p.noisy = False
# Register as a tcp and udp service
ret = service.MultiService()
PORT=53
for (klass, arg) in [(internet.TCPServer, f), (internet.UDPServer, p)]:
s = klass(PORT, arg)
s.setServiceParent(ret)
# Run all of the above as a twistd application
ret.setServiceParent(service.IServiceCollection(application))
# If called directly, instruct the user to run it through twistd
if __name__ == '__main__':
print "Usage: sudo twistd -y %s (background) OR sudo twistd -noy %s (foreground)" % (sys.argv[0], sys.argv[0])

Matt,
I tried your latest example and it works fine. I think you may be testing it wrong.
In your later comments you talk about using time.sleep(5) in the lookup method to simulate a slow response.
You can't do that. It will block the reactor. If you want to simulate a delay, use reactor.callLater to fire the deferred
https://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTime.html#callLater
eg
def lookupMailExchange(self, name, timeout=None):
d = defer.Deferred()
self._reactor.callLater(
5, d.callback,
[(dns.RRHeader(name, dns.MX, dns.IN, 600,
dns.Record_MX(1, "mail.example.com", 600)),), (), ()]
)
return d
Here's how I tested:
time bash -c 'for n in "google.com" "yahoo.com"; do dig -p 10053 #127.0.0.1 "$n" MX +short +tries=1 +notcp +time=10 & done; wait'
And the output shows that both responses came back after 5 seconds
1 10.0.0.9.
1 10.0.0.9.
real 0m5.019s
user 0m0.015s
sys 0m0.013s
Similarly, you need to make sure that calls to your database don't block:
https://twistedmatrix.com/documents/current/core/howto/rdbms.html
Some other points:
MX records should contain hostnames not IP addresses - https://www.rfc-editor.org/rfc/rfc1035#section-3.3.9
Don't use nslookup -
https://jdebp.eu./FGA/nslookup-flaws.html and http://cr.yp.to/djbdns/nslookup.html
Be careful that stdlib logging doesn't block -
https://twistedmatrix.com/documents/current/api/twisted.python.log.PythonLoggingObserver.html
Configure your test client to only issue one UDP query, otherwise retries and followup TCP queries may confuse your tests.

Python - Twisted, Proxy and modifying content

So i've looked around at a few things involving writting an HTTP Proxy using python and the Twisted framework.
Essentially, like some other questions, I'd like to be able to modify the data that will be sent back to the browser. That is, the browser requests a resource and the proxy will fetch it. Before the resource is returned to the browser, i'd like to be able to modify ANY (HTTP headers AND content) content.
This ( Need help writing a twisted proxy ) was what I initially found. I tried it out, but it didn't work for me. I also found this ( Python Twisted proxy - how to intercept packets ) which i thought would work, however I can only see the HTTP requests from the browser.
I am looking for any advice. Some thoughts I have are to use the ProxyClient and ProxyRequest classes and override the functions, but I read that the Proxy class itself is a combination of the both.
For those who may ask to see some code, it should be noted that I have worked with only the above two examples. Any help is great.
Thanks.

To create ProxyFactory that can modify server response headers, content you could override ProxyClient.handle*() methods:
from twisted.python import log
from twisted.web import http, proxy
class ProxyClient(proxy.ProxyClient):
"""Mangle returned header, content here.
Use `self.father` methods to modify request directly.
"""
def handleHeader(self, key, value):
# change response header here
log.msg("Header: %s: %s" % (key, value))
proxy.ProxyClient.handleHeader(self, key, value)
def handleResponsePart(self, buffer):
# change response part here
log.msg("Content: %s" % (buffer[:50],))
# make all content upper case
proxy.ProxyClient.handleResponsePart(self, buffer.upper())
class ProxyClientFactory(proxy.ProxyClientFactory):
protocol = ProxyClient
class ProxyRequest(proxy.ProxyRequest):
protocols = dict(http=ProxyClientFactory)
class Proxy(proxy.Proxy):
requestFactory = ProxyRequest
class ProxyFactory(http.HTTPFactory):
protocol = Proxy
I've got this solution by looking at the source of twisted.web.proxy. I don't know how idiomatic it is.
To run it as a script or via twistd, add at the end:
portstr = "tcp:8080:interface=localhost" # serve on localhost:8080
if __name__ == '__main__': # $ python proxy_modify_request.py
import sys
from twisted.internet import endpoints, reactor
def shutdown(reason, reactor, stopping=[]):
"""Stop the reactor."""
if stopping: return
stopping.append(True)
if reason:
log.msg(reason.value)
reactor.callWhenRunning(reactor.stop)
log.startLogging(sys.stdout)
endpoint = endpoints.serverFromString(reactor, portstr)
d = endpoint.listen(ProxyFactory())
d.addErrback(shutdown, reactor)
reactor.run()
else: # $ twistd -ny proxy_modify_request.py
from twisted.application import service, strports
application = service.Application("proxy_modify_request")
strports.service(portstr, ProxyFactory()).setServiceParent(application)
Usage
$ twistd -ny proxy_modify_request.py
In another terminal:
$ curl -x localhost:8080 http://example.com

For two-way proxy using twisted see the article:
http://sujitpal.blogspot.com/2010/03/http-debug-proxy-with-twisted.html

how to call a remote fabric method in django

I need to call a fabric method on a remote machine through django. I mean when user send a given request get the hostname of a remote machine.
Something like this:
def get_hostname(request):
hostname = os.system('fab remote_server hostname')
return hostname

For greater control and flexibility you should use fabric as a library. see: http://docs.fabfile.org/en/1.3.3/usage/library.html
import fabric.api as fab
from fabric.network import disconnect_all
from contextlib import contextmanager
#context_manager
def ssh(settings):
with settings:
try:
yield
finally:
disconnect_all()
def hostname(request, host='somehost', user='someuser', pw='secret'):
with ssh(fab.settings(host_string=host, user=user, password=pw)):
return fab.run('hostname')

If you server has the needed parts for fabric you should be able to just import you fabfile on call the function directly.
(This is just some code I dreamt up YMMW)
import fabfile as f #Your fabfile must be somewhere it can be imported
def get_hostname(request):
hostname = f.remote_server(hostname)
return hostname
You can also import and use fabric directly from django

Check fabric.tasks.execute() after version 1.3.

How do you you run a Twisted application via Python (instead of via Twisted)?

I am working my way through learning Twisted, and have stumbled across something I'm not sure I'm terribly fond of - the "Twisted Command Prompt". I am fiddling around with Twisted on my Windows machine, and tried running the "Chat" example:
from twisted.protocols import basic
class MyChat(basic.LineReceiver):
def connectionMade(self):
print "Got new client!"
self.factory.clients.append(self)
def connectionLost(self, reason):
print "Lost a client!"
self.factory.clients.remove(self)
def lineReceived(self, line):
print "received", repr(line)
for c in self.factory.clients:
c.message(line)
def message(self, message):
self.transport.write(message + '\n')
from twisted.internet import protocol
from twisted.application import service, internet
factory = protocol.ServerFactory()
factory.protocol = MyChat
factory.clients = []
application = service.Application("chatserver")
internet.TCPServer(1025, factory).setServiceParent(application)
However, to run this application as a Twisted server, I have to run it via the "Twisted Command Prompt", with the command:
twistd -y chatserver.py
Is there any way to change the code (set Twisted configuration settings, etc) so that I can simply run it via:
python chatserver.py
I've Googled, but the search terms seem to be too vague to return any meaningful responses.
Thanks.

I don't know if it's the best way to do this but what I do is instead of:
application = service.Application("chatserver")
internet.TCPServer(1025, factory).setServiceParent(application)
you can do:
from twisted.internet import reactor
reactor.listenTCP(1025, factory)
reactor.run()
Sumarized if you want to have the two options (twistd and python):
if __name__ == '__main__':
from twisted.internet import reactor
reactor.listenTCP(1025, factory)
reactor.run()
else:
application = service.Application("chatserver")
internet.TCPServer(1025, factory).setServiceParent(application)
Hope it helps!

Don't confuse "Twisted" with "twistd". When you use "twistd", you are running the program with Python. "twistd" is a Python program that, among other things, can load an application from a .tac file (as you're doing here).
The "Twisted Command Prompt" is a Twisted installer-provided convenience to help out people on Windows. All it is doing is setting %PATH% to include the directory containing the "twistd" program. You could run twistd from a normal command prompt if you set your %PATH% properly or invoke it with the full path.
If you're not satisfied with this, perhaps you can expand your question to include a description of the problems you're having when using "twistd".

On windows you can create .bat file with your command in it, use full paths, then just click on it to start up.
For example I use:
runfileserver.bat:
C:\program_files\python26\Scripts\twistd.py -y C:\source\python\twisted\fileserver.tac

Maybe one of run or runApp in twisted.scripts.twistd modules will work for you. Please let me know if it does, it will be nice to know!

I haven't used twisted myself. However, you may try seeing if the twistd is a python file itself. I would take a guess that it is simply managing loading the appropriate twisted libraries from the correct path.

I am successfully using the simple Twisted Web server on Windows for Flask web sites.
Are others also successfully using Twisted on Windows, to validate that configuration?
new_app.py
if __name__ == "__main__":
reactor_args = {}
def run_twisted_wsgi():
from twisted.internet import reactor
from twisted.web.server import Site
from twisted.web.wsgi import WSGIResource
resource = WSGIResource(reactor, reactor.getThreadPool(), app)
site = Site(resource)
reactor.listenTCP(5000, site)
reactor.run(**reactor_args)
if app.debug:
# Disable twisted signal handlers in development only.
reactor_args['installSignalHandlers'] = 0
# Turn on auto reload.
import werkzeug.serving
run_twisted_wsgi = werkzeug.serving.run_with_reloader(run_twisted_wsgi)
run_twisted_wsgi()
old_app.py
if __name__ == "__main__":
app.run()