We Keep Coding

Shutdown for socketserver based Python 3 server hangs

I am working on a "simple" server using a threaded SocketServer in Python 3.
I am going through a lot of trouble implementing shutdown for this. The code below I found on the internet and shutdown works initially but stops working after sending a few commands from the client via telnet. Some investigation tells me it hangs in threading._shutdown... threading._wait_for_tstate_lock but so far this does not ring a bell.
My research tells me that there are ~42 different solutions, frameworks, etc. on how to do this in different python versions. So far I could not find a working approach for python3. E.g. I love telnetsrv
(https://pypi.python.org/pypi/telnetsrv/0.4) for python 2.7 (it uses greenlets from gevent) but this one does not work for python 3. So if there is a more pythonic, std lib approach or something that works reliably I would love to hear about it!
My bet currently is with socketserver but I could not figure out yet how to deal with the hanging server. I removed all the log statements and most functionality so I can post this minimal server which exposes the issue:
# -*- coding: utf-8 -*-
import socketserver
import threading
SERVER = None
def shutdown_cmd(request):
global SERVER
request.send(bytes('server shutdown requested\n', 'utf-8'))
request.close()
SERVER.shutdown()
print('after shutdown!!')
#SERVER.server_close()
class service(socketserver.BaseRequestHandler):
def handle(self):
while True:
try:
msg = str(self.request.recv(1024).strip(), 'utf-8')
if msg == 'shutdown':
shutdown_cmd(msg, self.request)
else:
self.request.send(bytes("You said '{}'\n".format(msg), "utf-8"))
except Exception as e:
pass
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def run():
global SERVER
SERVER = ThreadedTCPServer(('', 1520), service)
server_thread = threading.Thread(target=SERVER.serve_forever)
server_thread.daemon = True
server_thread.start()
input("Press enter to shutdown")
SERVER.shutdown()
if __name__ == '__main__':
run()
It would be great being able to stop the server from the handler, too (see shutdown_cmd)

shutdown() works as expected, the server has stopped accepting new connections, but python still waiting for alive threads to terminate.
By default, socketserver.ThreadingMixIn will create new threads to handle incoming connection and by default, those are non-daemon threads, so python will wait for all alive non-daemon threads to terminate.
Of course, you could make the server spawn daemon threads, then python will not waiting:
The ThreadingMixIn class defines an attribute daemon_threads, which indicates whether or not the server should wait for thread termination. You should set the flag explicitly if you would like threads to behave autonomously; the default is False, meaning that Python will not exit until all threads created by ThreadingMixIn have exited.
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
daemon_threads = True
But that is not the ideal solution, you should check why threads never terminate, usually, the server should stop processing connection when no new data available or client shutdown connection:
import socketserver
import threading
shutdown_evt = threading.Event()
class service(socketserver.BaseRequestHandler):
def handle(self):
self.request.setblocking(False)
while True:
try:
msg = self.request.recv(1024)
if msg == b'shutdown':
shutdown_evt.set()
break
elif msg:
self.request.send(b'you said: ' + msg)
if shutdown_evt.wait(0.1):
break
except Exception as e:
break
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def run():
SERVER = ThreadedTCPServer(('127.0.0.1', 10000), service)
server_thread = threading.Thread(target=SERVER.serve_forever)
server_thread.daemon = True
server_thread.start()
input("Press enter to shutdown")
shutdown_evt.set()
SERVER.shutdown()
if __name__ == '__main__':
run()

I tried two solutions to implement a tcp server which runs on Python 3 on both Linux and Windows (I tried Windows 7):
using socketserver (my question) - shutdown is not working
using asyncio (posted an answer for that) - does not work on Windows
Both solutions have been based upon search results on the web. In the end I had to give up on the idea of finding a proven solution because I could not find one. Consequently I implemented my own solution (based on gevent). I post it here because I hope it will be helpful for others to avoid stuggeling the way I did.
# -*- coding: utf-8 -*-
from gevent.server import StreamServer
from gevent.pool import Pool
class EchoServer(StreamServer):
def __init__(self, listener, handle=None, spawn='default'):
StreamServer.__init__(self, listener, handle=handle, spawn=spawn)
def handle(self, socket, address):
print('New connection from %s:%s' % address[:2])
socket.sendall(b'Welcome to the echo server! Type quit to exit.\r\n')
# using a makefile because we want to use readline()
rfileobj = socket.makefile(mode='rb')
while True:
line = rfileobj.readline()
if not line:
print("client disconnected")
break
if line.strip().lower() == b'quit':
print("client quit")
break
if line.strip().lower() == b'shutdown':
print("client initiated server shutdown")
self.stop()
break
socket.sendall(line)
print("echoed %r" % line.decode().strip())
rfileobj.close()
srv = EchoServer(('', 1520), spawn=Pool(20))
srv.serve_forever()

after more research I found a sample that works using asyncio:
# -*- coding: utf-8 -*-
import asyncio
# after further research I found this relevant europython talk:
# https://www.youtube.com/watch?v=pi49aiLBas8
# * protocols and transport are useful if you do not have tons of socket based code
# * event loop pushes data in
# * transport used to push data back to the client
# found decent sample in book by wrox "professional python"
class ServerProtocol(asyncio.Protocol):
def connection_made(self, transport):
self.transport = transport
self.write('Welcome')
def connection_lost(self, exc):
self.transport = None
def data_received(self, data):
if not data or data == '':
return
message = data.decode('ascii')
command = message.strip().split(' ')[0].lower()
args = message.strip().split(' ')[1:]
#sanity check
if not hasattr(self, 'command_%s' % command):
self.write('Invalid command: %s' % command)
return
# run command
try:
return getattr(self, 'command_%s' % command)(*args)
except Exception as ex:
self.write('Error: %s' % str(ex))
def write(self, msg):
self.transport.write((msg + '\n').encode('ascii', 'ignore'))
def command_shutdown(self):
self.write('Okay. shutting down')
raise KeyboardInterrupt
def command_bye(self):
self.write('bye then!')
self.transport.close()
self.transport = None
if __name__ == '__main__':
loop = asyncio.get_event_loop()
coro = loop.create_server(ServerProtocol, '127.0.0.1', 8023)
asyncio.async(coro)
try:
loop.run_forever()
except KeyboardInterrupt:
pass
I understand that this is the most useful way to do this kind of network programming. If necessary the performance could be improved using the same code with uvloop (https://magic.io/blog/uvloop-blazing-fast-python-networking/).

Another way to shut down the server is by creating a process/thread for the serve_forever call.
After server_forever is started, simply wait for a custom flag to trigger and use server_close on the server, and terminate the process.
streaming_server = StreamingServer(('', 8000), StreamingHandler)
FLAG_KEEP_ALIVE.value = True
process_serve_forever = Process(target=streaming_server.serve_forever)
process_serve_forever.start()
while FLAG_KEEP_ALIVE.value:
pass
streaming_server.server_close()
process_serve_forever.terminate()

Twisted close web server connection

Im writing a simple web server application using twisted. The application will get a string and return the reverse of the string it received.
It all works fine. Now I need to close the socket connection if there is an inactivity for 5 mins.
Here is my server code:-
from twisted.internet import reactor, protocol
class Echo(protocol.Protocol):
"""This is just about the simplest possible protocol"""
def dataReceived(self, data):
"As soon as any data is received, write it back."
self.transport.write(data[::-1])
def main():
"""This runs the protocol on port 8000"""
factory = protocol.ServerFactory()
factory.protocol = Echo
reactor.listenTCP(8000,factory)
reactor.run()
# this only runs if the module was *not* imported
if __name__ == '__main__':
main()
~

Add these methods to your class:
def connectionMade(self):
def terminate():
self.terminateLater = None
self.transport.abortConnection()
self.terminateLater = reactor.callLater(60 * 5, terminate)
def connectionLost(self, reason):
delayedCall = self.terminateLater
self.terminateLater = None
if delayedCall is not None:
delayedCall.cancel()
This makes it so that when a connection is established, your protocol will scheduled a timed call in 5 minutes to close the connection. If the connection is closed otherwise, it will cancel the timeout.

Pymodbus/Twisted Asynchronous Client Reconnecting

I have written a test code which reads some coils/registers from a PLC's modbus server. When I call one request the code works. I unplugged the cable then Twisted calls clientConnectionLost function so my client will reconnected, when I plugged back the cable. If I do multiple requests, like in the code below, the handling breaks, nothing happens. I don't know what causes the problem.
#!/usr/bin/env python
from PyQt4 import QtCore, QtGui
from twisted.internet import reactor, protocol,defer
from pymodbus.constants import Defaults
from pymodbus.client.async import ModbusClientProtocol
from time import sleep
def logger():
import logging
logging.basicConfig()
log = logging.getLogger()
log.setLevel(logging.DEBUG)
logger()
class MyModbusClientProtocol(ModbusClientProtocol):
def connectionMade(self):
ModbusClientProtocol.connectionMade(self)
print 'Connected'
self.read()
def read(self):
deferred = self.read_coils(0,1999)
deferred.addCallbacks(self.requestFetched,self.requestNotFetched)
deferred = self.read_holding_registers(0,124)
deferred.addCallbacks(self.requestFetched,self.requestNotFetched)
def requestNotFetched(self,error):
print error
sleep(0.5)
def requestFetched(self,response):
try:
print ("Fetched %d" % response.getRegister(1))
except:
print ("Fetched %d" % response.getBit(1))
self.factory.counter += 1
if self.factory.counter == 2:
self.factory.counter = 0
reactor.callLater(0,self.read)
class MyModbusClientFactory(protocol.ClientFactory):
"""A factory.
A new protocol instance will be created each time we connect to the server.
"""
def __init__(self):
self.counter = 0
def buildProtocol(self, addr):
p = MyModbusClientProtocol()
p.factory = self
return p
def clientConnectionLost(self, connector, reason):
print "connection lost:", reason
connector.connect()
def clientConnectionFailed(self, connector, reason):
print "connection failed:", reason
connector.connect()
if __name__ == "__main__":
factoryinstance = MyModbusClientFactory()
reactor.connectTCP("192.168.2.69", 502, factoryinstance)
reactor.run()

I have tested your code and believe that you have seen a timing related red herring when your code was seen to work after commenting out one of your requests. The behavior you are seeing where clientConnectionLost is not called is covered in the twisted FAQ: Why isn't my connectionLost method called?
What you need to do is create your own protocol specific timeout as you can't always rely on TCP's timeouts to work in your favor. A simple way to fix your code would be to add this to the end of your read method:
self.timeout = reactor.callLater(5, self.transport.abortConnection)
Which will abort the connection after 5 seconds wait. You also need to cancel this timeout when your requests have completed successfully with:
self.timeout.cancel()
in your requestFetched method before you call your read again.

How to manage connections and Clients in Twisted?

I started working with Twisted Framework, I wrote a TCP server and I connect to it throw Telnet, it works fine. Now I want to manage connections and connected clients( sending data, cutting connections, etc etc) using an GUI like PyUI or GTK..
this is my code
import sys
import os
from twisted.internet import reactor, protocol
from twisted.python import log
class Server(protocol.Protocol):
def dataReceived(self, data):
log.msg ("data received: %s"%data)
self.transport.write("you sent: %s"%data)
def connectionMade(self):
self.client_host = self.transport.getPeer().host
self.client_port = self.transport.getPeer().port
if len(self.factory.clients) >= self.factory.clients_max:
log.msg("Too many connections !!")
self.transport.write("Too many connections, sorry\n")
self.transport.loseConnection()
else:
self.factory.clients.append((self.client_host,self.client_port))
log.msg("connection from %s:%s\n"%(self.client_host,str(self.client_port)))
self.transport.write(
"Welcome %s:%s\n" %(self.client_host,str(self.client_port)))
def connectionLost(self, reason):
log.msg('Connection lost from %s:%s. Reason: %s\n' % (self.client_host,str(self.client_port),reason.getErrorMessage()))
if (self.client_host,self.client_port) in self.factory.clients:
self.factory.clients.remove((self.client_host,self.client_port))
class MyFactory(protocol.ServerFactory):
protocol = Server
def __init__(self, clients_max=10):
self.clients_max = clients_max
self.clients = []
def main():
"""This runs the protocol on port 8000"""
log.startLogging(sys.stdout)
reactor.listenTCP(8000,MyFactory)
reactor.run()
if __name__ == '__main__':
main()
Thanks.

If you want to write a single Python program (process) that runs both your UI and your networking, you will first need to choose an appropriate Twisted reactor that integrates with the UI toolkit's event loop. See here.
Next, you might start with something simple, like have a button that when pressed will send a text message to all currently connected clients.
Another thing: what clients will connect? Browsers (also)? If so, you might contemplate about using WebSocket instead of raw TCP.

Twisted client for a send only protocol that is tolerant of disconnects

I've decided to dip my toe into the world of asynchronous python with the help of twisted. I've implemented some of the examples from the documentation, but I'm having a difficult time finding an example of the, very simple, client I'm trying to write.
In short I'd like a client which establishes a tcp connection with a server and then sends simple "\n" terminated string messages off of a queue object to the server. The server doesn't ever respond with any messages so my client is fully unidirectional. I /think/ that what I want is some combination of this example and the twisted.internet.protocols.basic.LineReceiver convenience protocol. This feels like it should be just about the simplest thing one could do in twisted, but none of the documentation or examples I've seen online seem to fit quite right.

What I have done is not used a Queue but I am illustrating the code that sends a line, once a connection is made. There are bunch of print stuff that will help you understand on what is going on.
Usual import stuff:
from twisted.web import proxy
from twisted.internet import reactor
from twisted.internet import protocol
from twisted.internet.protocol import ReconnectingClientFactory
from twisted.protocols import basic
from twisted.python import log
import sys
log.startLogging(sys.stdout)
You create a protocol derived from line receiver, set the delimiter.
In this case, I simply write a string "www" once the connection is made.
The key thing is to look at protocol interface at twisted.internet.interface.py and understand the various methods of protocol and what they do and when they are called.
class MyProtocol(basic.LineReceiver):
#def makeConnection(self, transport):
# print transport
def connectionLost(self, reason):
print reason
self.sendData = False
def connectionMade(self):
print "connection made"
self.delimiter = "\n"
self.sendData = True
print self.transport
self.sendFromQueue()
def sendFromQueue(self):
while self.sendData:
msg = dataQueue.get()
self.sendLine(msg)
# you need to handle empty queue
# Have another function to resume
Finally, A protocol factory that will create a protocol instance for every connection.
Look at method : buildProtcol.
class myProtocolFactory():
protocol = MyProtocol
def doStart(self):
pass
def startedConnecting(self, connectorInstance):
print connectorInstance
def buildProtocol(self, address):
print address
return self.protocol()
def clientConnectionLost(self, connection, reason):
print reason
print connection
def clientConnectionFailed(self, connection, reason):
print connection
print reason
def doStop(self):
pass
Now you use a connector to make a connection:
reactor.connectTCP('localhost', 50000, myProtocolFactory())
reactor.run()
I ran this and connected it to an server that simply prints what it receives and hence send no ack back. Here is the output:
1286906080.08 82 INFO 140735087148064 __main__ conn_made: client_address=127.0.0.1:50277
1286906080.08 83 DEBUG 140735087148064 __main__ created handler; waiting for loop
1286906080.08 83 DEBUG 140735087148064 __main__ handle_read
1286906080.08 83 DEBUG 140735087148064 __main__ after recv
'www\n'
Recieved: 4
The above example if not fault tolerant. To reconnect , when a connection is lost, you can derive your protocol factory from an existing twisted class - ReconnectingClientFactory.
Twisted has almost all the tools that you would need :)
class myProtocolFactory(ReconnectingClientFactory):
protocol = MyProtocol
def buildProtocol(self, address):
print address
return self.protocol()
For further reference
I suggest that you read : http://krondo.com/?page_id=1327
[Edited: As per comment below]