We Keep Coding

Python Twisted: Is it possible to reload certificates on the fly?

I have a simple web server which serves content over HTTPS:
sslContext = ssl.DefaultOpenSSLContextFactory(
'/home/user/certs/letsencrypt-privkey.pem',
'/home/user/certs/letsencrypt-fullchain.pem',
)
reactor.listenSSL(
port=https_server_port,
factory=website_factory,
contextFactory=sslContext,
interface=https_server_interface
)
do_print(bcolors.YELLOW + 'server.py | running https server on ' + https_server_interface + ':' + str(https_server_port) + bcolors.END)
Is it possible to reload the certificates on the fly (for example by calling a path like https://example.com/server/reload-certificates and having it execute some code) or what do I need to do in order to get it done?
I want to avoid restarting the Python process.

It is possible in several ways. Daniel F's answer is pretty good and shows a good, general technique for reconfiguring your server on the fly.
Here are a couple more techniques that are more specific to TLS support in Twisted.
First, you could reload the OpenSSL "context" object from the DefaultOpenSSLContextFactory instance. When it comes time to reload the certificates, run:
sslContext._context = None
sslContext.cacheContext()
The cacheContext call will create a new OpenSSL context, re-reading the certificate files in the process. This does have the downside of relying on a private interface (_context) and its interaction with a not-really-that-public interface (cacheContext).
You could also implement your own version of DefaultOpenSSLContextFactory so that you don't have to rely on these things. DefaultOpenSSLContextFactory doesn't really do much. Here's a copy/paste/edit that removes the caching behavior entirely:
class DefaultOpenSSLContextFactory(ContextFactory):
"""
L{DefaultOpenSSLContextFactory} is a factory for server-side SSL context
objects. These objects define certain parameters related to SSL
handshakes and the subsequent connection.
"""
_context = None
def __init__(self, privateKeyFileName, certificateFileName,
sslmethod=SSL.SSLv23_METHOD, _contextFactory=SSL.Context):
"""
#param privateKeyFileName: Name of a file containing a private key
#param certificateFileName: Name of a file containing a certificate
#param sslmethod: The SSL method to use
"""
self.privateKeyFileName = privateKeyFileName
self.certificateFileName = certificateFileName
self.sslmethod = sslmethod
self._contextFactory = _contextFactory
def getContext(self):
"""
Return an SSL context.
"""
ctx = self._contextFactory(self.sslmethod)
# Disallow SSLv2! It's insecure! SSLv3 has been around since
# 1996. It's time to move on.
ctx.set_options(SSL.OP_NO_SSLv2)
ctx.use_certificate_file(self.certificateFileName)
ctx.use_privatekey_file(self.privateKeyFileName)
Of course, this reloads the certificate files for every single connection which may be undesirable. You could add your own caching logic back in, with a control interface that fits into your certificate refresh system. This also has the downside that DefaultOpenSSLContextFactory is not really a very good SSL context factory to begin with. It doesn't follow current best practices for TLS configuration.
So you probably really want to use twisted.internet.ssl.CertificateOptions instead. This has a similar _context cache that you could clear out:
sslContext = CertificateOptions(...) # Or PrivateCertificate(...).options(...)
...
sslContext._context = None
It will regenerate the context automatically when it finds that it is None so at least you don't have to call cacheContext this way. But again you're relying on a private interface.
Another technique that's more similar to Daniel F's suggestion is to provide a new factory for the already listening socket. This avoids the brief interruption in service that comes between stopListening and listenSSL. This would be something like:
from twisted.protocols.tls import TLSMemoryBIOFactory
# DefaultOpenSSLContextFactory or CertificateOptions or whatever
newContextFactory = ...
tlsWebsiteFactory = TLSMemoryBIOFactory(
newContextFactory,
isClient=False,
websiteFactory,
)
listeningPortFileno = sslPort.fileno()
websiteFactory.sslPort.stopReading()
websiteFactory.sslPort = reactor.adoptStreamPort(
listeningPortFileno,
AF_INET,
tlsWebsiteFactory,
)
This basically just has the reactor stop servicing the old sslPort with its outdated configuration and tells it to start servicing events for that port's underlying socket on a new factory. In this approach, you have to drop down to the slightly lower level TLS interface since you can't adopt a "TLS port" since there is no such thing. Instead, you adopt the TCP port and apply the necessary TLS wrapping yourself (this is just what listenSSL is doing for you under the hood).
Note this approach is a little more limited than the others since not all reactors provide the fileno or adoptStreamPort methods. You can test for the interfaces the various objects provide if you want to use this where it's supported and degrade gracefully elsewhere.
Also note that since TLSMemoryBIOFactory is how it always works under the hood anyway, you could also twiddle its private interface, if you have a reference to it:
tlsMemoryBIOFactory._connectionCreator = IOpenSSLServerConnectionCreator(
newContextFactory,
)
and it will begin using that for new connections. But, again, private...

It is possible.
reactor.listenSSL returns a twisted.internet.tcp.Port instance which you can store somewhere accessible like in the website resource of your server, so that you can later access it:
website_resource = Website()
website_factory = server.Site(website_resource)
website_resource.sslPort = reactor.listenSSL( # <---
port=https_server_port,
factory=website_factory,
contextFactory=sslContext,
interface=https_server_interface
)
then later in your http handler (render function) you can execute the following:
if request.path == b'/server/reload-certificates':
request.setHeader("connection", "close")
self.sslPort.connectionLost(reason=None)
self.sslPort.stopListening()
self.sslListen()
return b'ok'
where self.sslListen is the initial setup code:
website_resource = Website()
website_factory = server.Site(website_resource)
def sslListen():
sslContext = ssl.DefaultOpenSSLContextFactory(
'/home/user/certs/letsencrypt-privkey.pem',
'/home/user/certs/letsencrypt-fullchain.pem',
)
website_resource.sslPort = reactor.listenSSL(
port=https_server_port,
factory=website_factory,
contextFactory=sslContext,
interface=https_server_interface
)
website_resource.sslListen = sslListen # <---
sslListen() # invoke once initially
# ...
reactor.run()
Notice that request.setHeader("connection", "close") is optional. It indicates the browser that it should close the connection and not reuse it for the next fetch to the server (HTTP/1.1 connections usually are kept open for at least 30 seconds in order to be reused).
If the connection: close header is not sent, then everything will still work, the connection will still be active and usable, but it will still be using the old certificate, which should be no problem if you're just reloading the certificates to refresh them after certbot updated them. New connections from other browsers will start using the new certificates immediately.

How to know if I used http while https was available - Python

I am trying to write a script that checks if HTTPS is available when I used http.
My Idea was to collect all of the HTTP links and use urllib2 in order to open a connection to the server using HTTPS as follows (please ignore syntax problems if there are. I have tried to simplify the code so it will be easier to understand the problem itself):
count=0
for packet in trafficPackets:
if packet["http.host"] != None:
if https_supported(packet["ip.dest"]):
count+=1
where https_supported is the following function:
def https_supported(ip):
try:
if len(urlopen("https://"+ip))>0
return True
except:
return False
return False
I have tried to run the code on a little traffic file which contains an HTTP connection to a site that supports https but the result was unexpected- it was always returned zero.
Where did I go wrong ? Does anyone have an idea of how can I do it?
Thank you!

Using the exact same code with the http.host field instead of the IP seems to work.

gevent TCP server on Windows

I've been trying to create a TCP server with gevent without (any major) success so far. I think that the problem lies within Windows ( I've had some issues with sockets under Windows before ). I'm using Python2.7, gevent0.13 under Windows7. Here's my code:
from gevent import socket
from gevent.server import StreamServer
def handle_echo(sock, address):
try:
fp = sock.makefile()
while True:
# Just echos whatever it receives
try:
line = fp.readline()
except Exception:
break
if line:
try:
fp.write(line)
fp.flush()
except Exception:
break
else:
break
finally:
sock.shutdown(socket.SHUT_WR)
sock.close()
server = StreamServer(("", 2345), handle_echo)
server.server_forever()
This implementation is similar to the one you can find here:
http://blog.pythonisito.com/2012/08/building-tcp-servers-with-gevent.html
Now there are no errors and the server seems to work correctly, however it is not reading ( and thus sending ) anything. Is it possible that sock.makefile() does not work correctly under Windows7? Or maybe the problem lies somewhere else?
I've tried to replace sock.makefile() with simple
while True:
line = sock.recv(2048)
but this operation obviously blocks.
I've also tried to mix gevent's spawn with sock.setblocking(0). Now this was better and it worked, however it would not handle more then ~300 connections at a time.
I'm going to do some tests on Linux and see if it makes difference. In the meantime if you have any ideas, then feel free to share them with me. Cheers!
UPDATE Original code does the same thing under Ubuntu 12.04. So how should I implement gevent TCP server??

What did you send to the server? Make sure it's terminated by newline.. otherwise readline() won't work.
You could also use tcpdump or wireshark to see what's happening at TCP layer if you think you're doing correct things in your code.

Can select() be used with files in Python under Windows?

I am trying to run the following python server under windows:
"""
An echo server that uses select to handle multiple clients at a time.
Entering any line of input at the terminal will exit the server.
"""
import select
import socket
import sys
host = ''
port = 50000
backlog = 5
size = 1024
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((host,port))
server.listen(backlog)
input = [server,sys.stdin]
running = 1
while running:
inputready,outputready,exceptready = select.select(input,[],[])
for s in inputready:
if s == server:
# handle the server socket
client, address = server.accept()
input.append(client)
elif s == sys.stdin:
# handle standard input
junk = sys.stdin.readline()
running = 0
else:
# handle all other sockets
data = s.recv(size)
if data:
s.send(data)
else:
s.close()
input.remove(s)
server.close()
I get the error message (10038, 'An operation was attempted on something that is not a socket'). This probably relates back to the remark in the python documentation that "File objects on Windows are not acceptable, but sockets are. On Windows, the underlying select() function is provided by the WinSock library, and does not handle file descriptors that don’t originate from WinSock.". On internet there are quite some posts on this topic, but they are either too technical for me or simply not clear. So my question is: is there any way the select() statement in python can be used under windows? Please add a little example or modify my code above. Thanks!

Look like it does not like sys.stdin
If you change input to this
input = [server]
the exception will go away.
This is from the doc
Note:
File objects on Windows are not acceptable, but sockets are. On Windows, the
underlying select() function is provided by the WinSock library, and does not
handle file descriptors that don’t originate from WinSock.

I don't know if your code has other problems, but the error you're getting is because of passing input to select.select(), the problem is that it contains sys.stdin which is not a socket. Under Windows, select only works with sockets.
As a side note, input is a python function, it's not a good idea to use it as a variable.

Of course and the answers given are right...
you just have to remove the sys.stdin from the input but still use it in the iteration:
for s in inputready+[sys.stdin]:

python socketserver occasionally stops sending (and receiving?) messages

I've been experiencing a problem with a socketserver I wrote where the socketserver will seem to stop sending and receiving data on one of the ports it uses (while the other port continues to handle data just fine). Interestingly, after waiting a minute (or up to an hour or so), the socketserver will start sending and receiving messages again without any observable intervention.
I am using the Eventlet socketing framework, python 2.7, everything running on an ubuntu aws instance with external apps opening persistent connections to the socketserver.
From some reading I've been doing, it looks like I may not be implementing my socket server correctly.
According to http://docs.python.org/howto/sockets.html:
fundamental truth of sockets: messages must either be fixed length (yuck), or be delimited > > (shrug), or indicate how long they are (much better), or end by shutting down the connection.
I am not entirely sure that I am using a fix length message here (or am I?)
This is how I am receiving my data:
def socket_handler(sock, socket_type):
logg(1,"socket_handler:initializing")
while True:
recv = sock.recv(1024)
if not recv:
logg(1,"didn't recieve anything")
break
if len(recv) > 5:
logg(1,"socket handler: %s" % recv )
plug_id, phone_sid, recv_json = parse_json(recv)
send = 1
if "success" in recv_json and recv_json["success"] == "true" and socket_type == "plug":
send = 0
if send == 1:
send_wrapper(sock, message_relayer(recv, socket_type))
else:
logg(2, 'socket_handler:Ignoring received input: ' + str(recv) )
logg(1, 'Closing socket handle: [%s]' % str(sock))
sock.shutdown(socket.SHUT_RDWR)
sock.close()
"sock" is a socket object returned by the listener.accept() function.
The socket_handler function is called like so:
new_connection, address = listener.accept()
...<code omitted>...
pool.spawn_n(socket_handler, new_connection, socket_type)
Does my implementation look incorrect to anyone? Am I basically implementing a fixed length conversation protocol? What can I do to help investigate the issue or make my code more robust?
Thanks in advance,
T

You might be having buffering related problems if you're requesting to receive more bytes at the server (1024) than you're actually sending from the client.
To fix the problem, what's is usually done is encode the length of the message first and then the message itself. This way, the receiver can get the length field (which is of known size) and then read the rest of the message based on the decoded length.
Note: The length field is usually as many bytes long as you need in your protocol. Some protocols are 4-byte aligned and use a 32 bit field for this, but if you find that you've got enough with 1 or 2 bytes, then you can use that. The point here is that both client and server know the size of this field.