We Keep Coding

How to forward serial port data to a multiclient socket server in Python?

I would like to forward data captured on one serial port to a multiclient TCP Server. In short, I need a serial to TCPIP bridge.
import sys
import socket
from threading import Thread
import serial
import serial.threaded
class SerialToNet(serial.threaded.Protocol):
"""serial->socket"""
def __init__(self):
self.sockets: list[socket.socket] = []
def __call__(self):
return self
def data_received(self, data):
"""Forward data from Serial to IP client Sockets"""
for socket in self.sockets:
socket.sendall(data)
class NetToSerial(Thread):
"""socket->serial"""
serial_worker: serial.threaded.ReaderThread
def __init__(self, client_socket):
Thread.__init__(self)
self._socket = client_socket
def run(self):
try:
while True:
data = self._socket.recv(1024)
serial_worker.write(data)
except (ConnectionAbortedError, ConnectionResetError):
print("NetToSerial client disconnection")
return
if __name__ == "__main__":
# Serial connection
SERIAL_COM_PORT = 'COM9'
try:
ser = serial.Serial(SERIAL_COM_PORT, 115200, timeout=2)
except serial.SerialException:
sys.exit(f"Serial port {SERIAL_COM_PORT} it not available")
serial_to_net = SerialToNet()
serial_worker = serial.threaded.ReaderThread(ser, serial_to_net)
serial_worker.start()
# TCP Server
# :todo Use socketserver.TCPServer
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind(('', 3490))
mythreads = []
try:
# Wait new IP clients
while True:
server_socket.listen()
print("Server: waiting TCP client connection")
(client_socket, _) = server_socket.accept()
# New client
net_to_serial_thread = NetToSerial(client_socket)
net_to_serial_thread.serial_worker = serial_worker
serial_to_net.sockets.append(client_socket)
net_to_serial_thread.start()
mythreads.append(net_to_serial_thread)
except KeyboardInterrupt:
pass
for t in mythreads:
t.join()
This implementation is quite working but I don't known how to update sockets in SerialToNet class when a TCP client disconnect.

You need to implement some logic for when a network client disconnects.
You know a client has disconnected because you receive an empty response (b'') from the socket. You're receiving data from network clients in NetToSerial, here:
def run(self):
try:
while True:
data = self._socket.recv(1024)
serial_worker.write(data)
except (ConnectionAbortedError, ConnectionResetError):
print("NetToSerial client disconnection")
return
You need to check the value of data, and if it's empty implement your disconnect logic:
Close the associated socket.
Exit the thread.
That might look like:
class NetToSerial(Thread):
"""socket->serial"""
serial_worker: serial.threaded.ReaderThread
def __init__(self, client_socket):
Thread.__init__(self)
self._socket = client_socket
def run(self):
try:
while True:
data = self._socket.recv(1024)
if not data:
break
serial_worker.write(data)
except (ConnectionAbortedError, ConnectionResetError):
print("NetToSerial client disconnection")
return
finally:
self._socket.close()
But that's only half the solution, because you're writing to this socket in your SerialToNet class. You need to remove the socket from SerialToNet sockets array. You can have the class remove the socket in response to an exception when writing, like this:
class SerialToNet(serial.threaded.Protocol):
"""serial->socket"""
def __init__(self):
self.sockets: list[socket.socket] = []
def __call__(self):
return self
def data_received(self, data):
"""Forward data from Serial to IP client Sockets"""
for socket in self.sockets[:]:
try:
socket.sendall(data)
except OSError:
self.sockets.remove(socket)
Note that because it's not possible to remove an item from a list over which you're currently iterating, we are iterating over a copy of self.sockets in the above code. This means we're free to remove sockets from self.sockets from inside the loop.
With the above changes I believe your code will operate as you intend.
Not directly related to your question, but I'd like to make a comment about your code: as written, it allows multiple network clients to write to the serial port at the same time. That seems like a recipe for disaster and I cannot think of any situation in which that would make sense. You may want to reconsider that aspect of your code.

sockets with threadpool server python

I have a simple multithreading server, But it creates a new thread for each socket, I don't want to create a lot of threads. My idea is to receive the messages in other way: when the user send a message, it will add the message to a queue of messages and with a threadpool the server will handle these requests.
The simple multithreaded server:
import socket
import threading
class ThreadedServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((self.host, self.port))
def listen(self):
self.sock.listen(5)
while True:
client, address = self.sock.accept()
client.settimeout(60)
threading.Thread(target = self.listenToClient,args = (client,address)).start()
def listenToClient(self, client, address):
size = 1024
while True:
try:
data = client.recv(size)
if data:
# Set the response to echo back the recieved data
response = data
client.send(response)
else:
raise error('Client disconnected')
except:
client.close()
return False
if __name__ == "__main__":
port_num = input("Port? ")
ThreadedServer('',port_num).listen()
How can I implement my idea or is there better way to do it?

The question seems to be pretty old but i also stumble upon the same issue while working on the socket server, so here is the below code which you can use to make threaded socket server which doesnt spawn new threads on arrival.
Just to give gist ThreadingMixIn classes is overided with threaded pool.
class ThreadPoolMixIn(socketserver.ThreadingMixIn):
'''
use a thread pool instead of a new thread on every request
'''
# numThreads = 50
allow_reuse_address = True # seems to fix socket.error on server restart
def serve_forever(self):
'''
Handle one request at a time until doomsday.
'''
print('[X] Server is Running with No of Threads :- {}'.format(self.numThreads))
# set up the threadpool
self.requests = Queue(self.numThreads)
for x in range(self.numThreads):
t = threading.Thread(target = self.process_request_thread)
t.setDaemon(1)
t.start()
# server main loop
while True:
self.handle_request()
self.server_close()
def process_request_thread(self):
'''
obtain request from queue instead of directly from server socket
'''
while True:
socketserver.ThreadingMixIn.process_request_thread(self, *self.requests.get())
def handle_request(self):
'''
simply collect requests and put them on the queue for the workers.
'''
try:
request, client_address = self.get_request()
except socket.error:
return
if self.verify_request(request, client_address):
self.requests.put((request, client_address))
And then it is called in ThreadedTCPRequest Handler and override the numThreads parameter :
class ThreadedTCPServer(ThreadPoolMixIn, socketserver.TCPServer):
#Extend base class and overide the thread paramter to control the number of threads.
def __init__(self, no_of_threads, server_address, ThreadedTCPRequestHandler):
self.numThreads = no_of_threads
super().__init__(server_address, ThreadedTCPRequestHandler)
Ultimately creating the server which serves forever :
def create_multi_threaded_socket(CONFIG, HandlerClass = ThreadedTCPRequestHandler,
ServerClass = ThreadedTCPServer,
protocol="HTTP/1.0"):
server_address = ('', CONFIG.port)
HandlerClass.protocol_version = protocol
# httpd = ServerClass(server_address, HandlerClass)
server = ThreadedTCPServer(CONFIG.no_of_threads, server_address, ThreadedTCPRequestHandler)
sa = server.socket.getsockname()
print("Serving HTTP on {} port : {}".format(sa[0], sa[1]))
server.serve_forever()
I got the sample code from :
http://code.activestate.com/recipes/574454-thread-pool-mixin-class-for-use-with-socketservert/
Modified bit according to my need.
Hope this helps :) .

How to coreclty solve ZMQ socket error "zmq.error.ZMQError: Address in use" in python

I am using a windows 7 machine with Python 2.7 to make a simple Client/Server ZMQ proof of concept. I ran into this scenario where the listening socket(Server side of the app) is already in use and this throws "zmq.error.ZMQError: Address in use" error. How do you think is the best way to avoid this error? I was thinking of, when binding the socket catch this error and if error is thrown restart the context and the socket. This is not working, is still throws and error when binding. Server code:
class ZMQServer:
context = None
socket = None
def __init__(self, port):
self.context = zmq.Context()
self.socket = self.context.socket(zmq.REP)
try:
self.socket.bind("tcp://*:"+str(port))
except zmq.error.ZMQError:
print ("socket already in use, restarting")
self.socket.close()
self.context.destroy()
self.context = zmq.Context()
self.socket = self.context.socket(zmq.REP)
self.socket.bind("tcp://*:"+str(port))

I tried another approach, instead of checking the availability of the socket, I am trying to manage it correclty, to do this:
Created the listener async, on a separate thread.
Created a method that will close the socket at exit
In the socket binding I only catch the ZMQError and display it, I didn;t find a way to fix the blocked socked there.
The recv method that receives the messages is NON-blocking
So, the code now looks like this:
"
import time
import zmq
import threading
class ZMQServer:
context = None
socket = None
alive = False
ZMQthread = None
def __init__(self, port):
self.context = zmq.Context()
self.socket = self.context.socket(zmq.REP)
try:
self.socket.bind("tcp://*:"+str(port))
except zmq.error.ZMQError:
print ("socket already in use, try restarting it")
self.alive = False
self.alive = True
def StartAsync(self):
self.alive = True
ZMQthread = threading.Thread(target=self.Start)
ZMQthread.start()
def Start(self):
while self.alive == True:
# Wait for next request from client
print("Wait for next request from client")
try:
message = self.socket.recv(zmq.NOBLOCK)
print("Received request: %s" % message)
# Do some 'work'
time.sleep(1)
# Send reply back to client
self.socket.send(b"World")
except:
print "no message received in time. trying again"
def CloseServer(self):
print("Stoping the server")
self.alive = False
if(self.socket.closed == False):
self.socket.close()
if self.ZMQthread and self.ZMQthread.is_alive() == True:
self.ZMQthread.join()
if __name__ == '__main__':
zmqServer = ZMQServer(5555)
zmqServer.StartAsync()
time.sleep(20)
zmqServer.CloseServer()

Get threading.Thread target class object when defined as a variable

I'm trying to understand the basics of networking and I believe the right way to start learning basic server handling is through sockets.
I am aware there are better libraries such as twisted or socketServer. I want to do it specifically in socket module.
I am aware using socket module to build servers is unreliable.
I also am aware I need to implement some kind of protocol for future use.
I tried building P2pClient with threading.Thread inheritance:
import socket
import threading
class P2pServer(threading.Thread):
def __init__(self, host, port):
threading.Thread.__init__(self)
print "Info: %s:%s" % (host, port)
self.server = socket.socket()
self.server.bind((host, port))
print "Created the server"
self.server.listen(1) # p2p chat
def run(self):
while True:
c, addr = self.server.accept()
print "Connected from %s (%s)" % (addr, c)
class P2pClient(threading.Thread):
def __init__(self, host, port):
threading.Thread.__init__(self, host, port)
self.client = socket.socket()
self.client.connect((host, port))
def send_msg(self, msg):
try:
self.client.send(msg)
return "Sent: %s" % msg
except socket.error as e:
return "Could not send because of %s" % e
def run(self):
while True:
recv = self.client.recv(1024)
if len(recv) > 0:
print recv
server = P2pServer("localhost", 44444) # This is our server
server.start() # Run it
client = None
print "Ready to take input"
while True:
print "Your command: ",
cmd = raw_input()
if cmd.startswith("connect "):
cmd = cmd.split()
client = P2pClient(cmd[1], cmd[2])
client.start()
else:
client.send_msg(cmd)
But ended up getting the following error when the input is "connect localhost :
Traceback (most recent call last):
File "C:/Users/edız/Desktop/Ediz/Python/Playground/servers test/p2p.py", line 49, in <module>
client = P2pClient(cmd[1], cmd[2])
File "C:/Users/edız/Desktop/Ediz/Python/Playground/servers test/p2p.py", line 22, in __init__
threading.Thread.__init__(self, host, port)
File "C:\Python27\lib\threading.py", line 670, in __init__
assert group is None, "group argument must be None for now"
AssertionError: group argument must be None for now
Solution on the site suggested making it so that I use the threading.Thread() function directly instead of doing it the former way. However, the threading documenation doesn't give me an idea about how to reach the target function. How can I build the thread that will allow me to access P2pClient in the client variable?
Purpose of the following code is so that it will be run in two different port in same host, making a P2P connection between two servers.
import socket
import threading
class P2pServer(threading.Thread):
def __init__(self, host, port):
threading.Thread.__init__(self)
print "Info: %s:%s" % (host, port)
self.server = socket.socket()
self.server.bind((host, port))
print "Created the server"
self.server.listen(1) # p2p chat
def run(self):
while True:
c, addr = self.server.accept()
print "Connected from %s (%s)" % (addr, c)
class P2pClient(object):
def __init__(self, host, port):
self.client = socket.socket()
self.client.connect((host, int(port)))
def send_msg(self, msg):
try:
self.client.send(msg)
return "Sent: %s" % msg
except socket.error as e:
return "Could not send because of %s" % e
def run(self):
while True:
recv = self.client.recv(1024)
if len(recv) > 0:
print recv
server = P2pServer("localhost", 44444) # This is our server
server.start() # Run it
client = None
print "Ready to take input"
while True:
print "Your command: ",
cmd = raw_input()
if cmd.startswith("connect "):
cmd = cmd.split()
client = threading.Thread(target=P2pClient, args=(cmd[1], cmd[2]))
client.start()
else:
pass
# Here I just want something like this:
# client.send_msg("Hello other server")

Your first example is close. The problem is that you passed the extra parameters you need for the child P2pClient to the parent Thread class. thread takes a different set of parameters. Just leave those parameters out when initializing the parent.
class P2pClient(threading.Thread):
def __init__(self, host, port):
threading.Thread.__init__(self)
self.client = socket.socket()
self.client.connect((host, port))

Python Socket Listening

All of the below mentioned is on windows machines using python 2.7
Hello,
I am currently attempting to listen on a socket for data send by a remote program. This data is then printed to the screen and user input is requested that is then returned to remote program. In testing I have been able to have the remote program send me a menu of command line programs (cmd, ipconfig, whoami, ftp) and then my program returns with a number as a selection of the menu option.
The remote program receives my response and sends the output of the selected command. ipconfig and whoami work perfectly, but cmd and ftp only returns the output of the terminal once. (I.E. I can enter one command into the FTP program and send that too the remote program before I never hear back)
The part of my code that fails is that
if ready[0]: never becomes ready a second time after the first conversation.
I know the remote program is functioning correctly as I can use netcat to act in lieu of my code and operate the cmd terminal indefinitely.
How do I go about properly implementing a python socket listener that can account for this type of connection?
My "program" in its entirety:
import socket, sys, struct, time, select
host = ''
port = 50000
connectionSevered=0
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port '+str(port)+'.'
s.bind((host,port))
s.listen(5)
def recvall(the_socket,timeout=2):
global connectionSevered
data=''; # Data found by recv
total_data=[]; # Finally list of everything
s.setblocking(0) #make socket non blocking
begin=time.time() #beginning time
while 1:
ready = select.select([client], [], [], .2)
if time.time()-begin > timeout:
print 'Timeout reached'
#Leave loop, timer has reached its threshold
break
if ready[0]:
print 'In ready loop!'
try:
data = client.recv(4096) #attempt to fetch data
if data:
begin=time.time() #reset timeout timer
total_data.append(data)
data='';
except socket.error:
print '[+] Lost connection to client. Printing buffer...'
connectionSevered=1 # Let main loop know connection has errored
pass
time.sleep(1)
#join all parts to make final string
return ''.join(total_data)
client, address = s.accept()
print '[+] Client connected!'
while (connectionSevered==0): # While connection hasn't errored
print "connectionSevered="+str(connectionSevered) # DEBUG
recvall(s)
response = raw_input() #take user input
client.sendto(response) #send input
client.close(0)
Please let me know if you need more information, any help would be greatly appreciated, I am very new to this and eager to learn.

Playing around with this for a while finally got it working nice with a telnet session locally using python 2.7.
What it does is it sets up a thread that runs when the client connects listening for client stuff.
When the client sends a return ("\r\n" might have to change that if your interacting with a Linux system?) the message gets printed to the server, while this is happening if there is a raw input at the server side this will get sent to the client:
import socket
import threading
host = ''
port = 50000
connectionSevered=0
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port: {0}'.format(port)
conn, address = s.accept()
c = client(conn)
c.start()
print '[+] Client connected: {0}'.format(address[0])
c.send_msg(u"\r\n")
print "connectionSevered:{0}".format(connectionSevered)
while (connectionSevered==0):
try:
response = raw_input()
c.send_msg(response + u"\r\n")
except:
c.close()
The above answer will not work for more than a single connection. I have updated it by adding another thread for taking connections. It it now possible to have more than a single user connect.
import socket
import threading
import sys
host = ''
port = 50000
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
class connectionThread(threading.Thread):
def __init__(self, host, port):
super(connectionThread, self).__init__()
try:
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.bind((host,port))
self.s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
self.clients = []
def run(self):
while True:
conn, address = self.s.accept()
c = client(conn)
c.start()
c.send_msg(u"\r\n")
self.clients.append(c)
print '[+] Client connected: {0}'.format(address[0])
def main():
get_conns = connectionThread(host, port)
get_conns.start()
while True:
try:
response = raw_input()
for c in get_conns.clients:
c.send_msg(response + u"\r\n")
except KeyboardInterrupt:
sys.exit()
if __name__ == '__main__':
main()
Clients are not able to see what other clients say, messages from the server will be sent to all clients. I will leave that as an exercise for the reader.

If you're in Python 3 by now and still wondering about sockets, here's a basic way of using them:
server.py
import time
import socket
# creating a socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
# get local Host machine name
host = socket.gethostname() # or just use (host == '')
port = 9999
# bind to pot
s.bind((host, port))
# Que up to 5 requests
s.listen(5)
while True:
# establish connection
clientSocket, addr = s.accept()
print("got a connection from %s" % str(addr))
currentTime = time.ctime(time.time()) + "\r\n"
clientSocket.send(currentTime.encode('ascii'))
clientSocket.close()
client.py
import socket
# creates socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
host = socket.gethostname() # or just use (host = '')
port = 9999
s.connect((host, port))
tm = s.recv(1024) # msg can only be 1024 bytes long
s.close()
print("the time we got from the server is %s" % tm.decode('ascii'))
Run server.py first, then run client.py.
This is just send and receive the currentTime.
What's new in Python 3.4 sockets?
A major difference between python 2.7 sockets and python 3.4 sockets is the sending messages. you have to .encode() (usually using 'ascii' or blank as parameters/arguments)
and then using .decode()
For example use .encode() to send, and use .decode() to receive.
Extra info: client/server socket tutorial