We Keep Coding

broken pipe error with python multiprocessing and socketserver

Essentially Im using the socketserver python library to try and handle communications from a central server to multiple raspberry pi4 and esp32 peripherals. Currently i have the socketserver running serve_forever, then the request handler calls a method from a processmanager class which starts a process that should handle the actual communication with the client.
It works fine if i use .join() on the process such that the processmanager method doesnt exit, but thats not how i would like it to run. Without .join() i get a broken pipe error as soon as the client communication process tries to send a message back to the client.
This is the process manager class, it gets defined in the main file and buildprocess is called through the request handler of the socketserver class:
import multiprocessing as mp
mp.allow_connection_pickling()
import queuemanager as qm
import hostmain as hmain
import camproc
import keyproc
import controlproc
# method that gets called into a process so that class and socket share memory
def callprocess(periclass, peritype, clientsocket, inqueue, genqueue):
periclass.startup(clientsocket)
class ProcessManager(qm.QueueManager):
def wipeproc(self, target):
# TODO make wipeproc integrate with the queue manager rather than directly to the class
for macid in list(self.procdict.keys()):
if target == macid:
# calls proc kill for the class
try:
self.procdict[macid]["class"].prockill()
except Exception as e:
print("exception:", e, "in wipeproc")
# waits for process to exit naturally (class threads to close)
self.procdict[macid]["process"].join()
# remove dict entry for this macid
self.procdict.pop(macid)
# called externally to create the new process and append to procdict
def buildprocess(self, peritype, macid, clientsocket):
# TODO put some logic here to handle the differences of the controller process
# generates queue object
inqueue = mp.Queue()
# creates periclass instance based on type
if peritype == hmain.cam:
periclass = camproc.CamMain(self, inqueue, self.genqueue)
elif peritype == hmain.keypad:
print("to be added to")
elif peritype == hmain.motion:
print("to be added to")
elif peritype == hmain.controller:
print("to be added to")
# init and start call for the new process
self.procdict[macid] = {"type": peritype, "inqueue": inqueue, "class": periclass, "process": None}
self.procdict[macid]["process"] = mp.Process(target=callprocess,
args=(self.procdict[macid]["class"], self.procdict[macid]["type"], clientsocket, self.procdict[macid]["inqueue"], self.genqueue))
self.procdict[macid]["process"].start()
# updating the process dictionary before class obj gets appended
# if macid in list(self.procdict.keys()):
# self.wipeproc(macid)
print(self.procdict)
print("client added")
to my eye, all the pertinent objects should be stored in the procdict dictionary but as i mentioned it just gets a broken pipe error unless i join the process with self.procdict[macid]["process"].join() before the end of the buildprocess method
I would like it to exit the method but leave the communication process running as is, ive tried a few different things with restructuring what gets defined within the process and without, but to no avail. Thus far i havent been able to find any pertinent solutions online but of course i may have missed something too.
Thankyou for reading this far if you did! Ive been stuck on this for a couple days so any help would be appreciated, this is my first project with multiprocessing and sockets on any sort of scale.
#################
Edit to include pastebin with all the code:
https://pastebin.com/u/kadytoast/1/PPWfyCFT

Without .join() i get a broken pipe error as soon as the client communication process tries to send a message back to the client.
That's because at the time when the request handler handle() returns, the socketserver does shutdown the connection. That socketserver simplifies the task of writing network servers means it does certain things automatically which are usually done in the course of network request handling. Your code is not quite making the intended use of socketserver. Especially, for handling requests asynchronously, Asynchronous Mixins are intended. With the ForkingMixIn the server will spawn a new process for each request, in contrast to your current code which does this by itself with mp.Process. So, I think you have basically two options:
code less of the request handling yourself and use the provided socketserver methods
stay with your own handling and don't use socketserver at all, so it won't get in the way.

Threading and tcp/ip connections problems

I'm new in python and threading so please be indifferent. I'm trying to do 2-players game in python. Data are send through tcp/ip protocol (client-server architecture). On server I have three threads. One comunicate with one user, second with second and in third thread I'm getting data which was send by client form two others threads. This data are used to check if game is over. And it's all working good. Problems start now. When the game is over I want to send another data to client. So Thread 3 need to send data to client, but two others threads are still working and they still have connections with clients. Generally I do not know how to do this. I tried to send information through the Queue from third thread to others that they should close theirs connections. It's thread class code:
class myThread(threading.Thread):
def __init__(self, threadID, name, conn, conn2, kto, wartosc,
wybor,kolejkaZadan,gracz1,gracz2):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.conn = conn
self.conn2 = conn2
self.kto = kto
self.wartosc = wartosc
self.wybor = wybor
self.kolejkaZadan = kolejkaZadan
self.gracz1 = gracz1
self.gracz2 = gracz2
def run(self):
if self.wybor == None:
toClient(self.conn,self.conn2,self.kto,self.wartosc,self.gracz1)
else:
while True:
data,kolejkaZwrotna = self.kolejkaZadan.get() // I receive data from two others threads
time.sleep(10)
dataKolejne,kolejkaZwrotna = self.kolejkaZadan.get() // I receive data from two others threads
if data is dataKolejne: // if end
tworzenieXmla(self.gracz1, self.gracz2)
odczytywanieXmla('itemGracza1',gracz1Otrzymane)
plik = open('Marcin.xml', 'rb')
czyZamknacConnection = True
kolejkaZwrotna.put(czyZamknacConnection) // send data to two others threads
while True:
czescXmla = plik.read(10000)
#self.conn2.send(czescXmla)
And It's my send/receiv function which is executed by two other threads:
def toClient(conn, conn2, kto, wartosc,gracz):
wordsBackup = None
kolejkaZwrotna = queue.Queue()
while True:
data = conn.recv(BUFFER_SIZE)
if not data:
break
if kolejkaZwrotna.get() is True://receive form thread 3
conn2.close()
print('closed')
break
if len(data)>7:
print('WARNING', data)
words = str(data.decode()).split()
#print(words[0], words[1])
if kto==1:
conn2.send(data)
if kto==2:
conn2.send(data)
kolejkaZadan.put(words[2],kolejkaZwrotna) // send to thread 3
xmlTablicaDoZapisu(str(int(words[0])),str(int(words[1])),str(int(words[2])),gracz)
Generally there is no error and we can play but there is only one player on each computer so I think server don't send data. I would appreciate any help.

A fix for your current situation would be to change all those connection variables into an array of connections which you could iterate over. You might want to build some container classes which define their behavior since not all clients are the same ( server client, and player clients ). That way you aren't limited by the amount of variables you've declared, and threads available.
Then once a new client connects you simply add it to the array and your iterator will take care of the rest.
This is a common problem with TCP/IP though, in that you always have to have open connections to n clients, which not only takes up resources but since TCP/IP is a queued protocol it could also set the entire game back if any client has a slower connection. In practice your game will always be as laggy as the player with the worst connection.
You have a couple of options.
You can have one thread always open which handles connections. Your supervisor thread. It holds an array of open connections' data and dispenses actions to the other threads. This isn't the best option since you'll quickly encounter Race Conditions such as two threads trying to use the same data.
You can switch over to UDP which will leave your threads wide open since there's no persistent connection. You'd then need to send states to each client, and once they ACK them you can get rid of the data. The majority of games implement UDP now'a'days, even turn based ones.
Beej's guide is probably the most extensive on the internet about UDP/TCP and socket control theory.
http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html
And there's also Gaffer on Games which is a fantastic resource as well.
http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/

Python Twisted: read from file and send as TCP server

I am completely new to Twisted, but it looks very promising for my project.
I would like to write a Python Twisted application which reads a record from text file every x seconds and contemporary listen on a TCP port (acting as TCP server). If no clients are connected to the TCP server the records are just discarded. If one or more clients are connected to the TCP server, the records are sent to all clients (all clients will receive the same line of the text file)
Can Twisted make this possible with a reasonable amount of LOCs?
Could anybody suggest an example to start with?
Thanks
C

Twisted's documentation includes information about how to run a TCP server. It also includes an information about how to perform work based on the passage of time. This should cover most of what you need to know.

Jean-Paul,
thanks for your answer.
Below is what i put together. The program is sending strings with time stamps to one or more clients connected to the server. Read synchronously from file in this scenario is very simple so i just use a fixed string with the time stamp.
My next step is to substitute the datetime.datetime.now() function call with a call to web service. Basically what i would like to create is kind of proxy that is
client versus a web service and invoke it every x seconds to get the data
TCP server versus a set of clients to stream data continuously, or better to say once a new data chunk is available (as is doing the example below)
The questions are:
Can you point me to an example of a similar system?
How can I combine the runEverySecond() method call with an asynchronous call to the web service using TCPClient capability of Twisted?
Thanks
C
from twisted.internet import protocol, reactor
from twisted.internet import task
import datetime
class Stream(protocol.Protocol):
def __init__(self, f):
self.factory = f
def connectionMade(self):
self.start = True
def forward(self, data):
if self.start:
self.transport.write(data)
class StreamFactory(protocol.Factory):
def __init__(self):
self.connections = []
def buildProtocol(self, addr):
s = Stream(self)
self.connections.append( s )
return s
def runEverySecond(self):
for c in self.connections:
c.forward( str(datetime.datetime.now()) )
f = StreamFactory()
l = task.LoopingCall(f.runEverySecond)
l.start(1.0) # call every second
reactor.listenTCP(8000, f)
reactor.run()

Python Twisted dynamic protocols for sending packets at runtime

I've recently started using Python Twisted, and while its very complex I'm really liking it! I've tried searching for the answer to this but I keep coming up dry so I was hoping someone here is a twisted guru:
I have a large/complex distributed system setup in a hierarchical format with masters, slaves, subslaves, etc..
At several points in my code depending on the packet received, I have a need to send a packet of data to another node. The node the data needs to be sent to is not known before calling reactor.run() so I feel like the answer might be different. I would like the connection to be TCP for reliability, but it only needs to send one packet. Sometimes I need an ACK back and sometimes I don't, but after that the connection can always die. The current way I've been handling this is by keeping a reference to the reactor in my class that is required to send the packet and calling:
tmpConn = MyClientFactory(dataToSend)
self.reactor.connectTCP(ADDR, PORT, tmpConn)
I feel that this might present a few issues however:
What happens to garbage collection if I don't keep the reference to the tmpConn.
If I do keep a reference to it in my class it ends up being garbage anyway because it only needed to send one packet.
As I said there are many different Factories all doing things like this at the same time so I wonder if this is the best way to handle this situation. Any pointers are greatly appreciated.
Here is a code snippet so the question is more clear.
from twisted.internet import reactor
from twisted.internet.protocol import Protocol, Factory, ClientFactory
class OneShotProtocol(Protocol):
def __init__(self, addr, data):
self.myaddr = addr
self.mydata = data
def connectionMade(self):
# We know we have a connection here so send the data
self.transport.write(self.mydata)
# Now we can kill the connection
self.transport.loseConnection()
class OneShotFactory(ClientFactory):
def __init__(self, data):
self.mydata = data
def buildProtocol(self, addr):
return OneShotProtocol(addr, self.mydata)
class ListenProtocol(Protocol):
def __init__(self, addr, factory):
self.myaddr = addr
#NOTE: I only save this because I've read multiple reactors are possible
self.factory = factory
def dataReceived(self, data):
if(data == 'stuff'):
#Alert the other node!
tmpConn = OneShotFactory('The British are coming')
self.factory.reactor.connectTCP(ADDR, PORT, tmpConn)
# Moving on...
class ListenFactory(Factory):
def __init__(self, reactor):
self.reactor = reactor
def buildProtocol(self, addr):
return OneShotProtocol(addr, self)
l = ListenFactory(reactor)
reactor.listenTCP(PORT, l)
reactor.run()

This sounds like a great way to implement the behavior you want.
You don't have to worry very much about garbage collection of your factory. The reactor will keep a reference to it (you passed it to connectTCP, after all) for as long as it needs to and then forget about it. If you also forget about it then Python's garbage collector will clean it up for you before too long.
The only adjustment you might want to make is to use the cool new "endpoint" APIs instead of using connectTCP directory. This doesn't change the basic idea of the solution, it just gives you a little bit more flexibility that you might someday benefit from.

Do I have a threading issue here?

I have been experimenting with GNU Radio and came across the tunnel.py program. This program allows you to tunnel IP traffic over a wireless radio link using Linux TUN/TAP devices. For the most part it is working however one part of the code is confusing me.
There is a class which implements a 'basic MAC layer'. This class has a callback function which writes a new packet to the TUN device. This function (phy_rx_callback) is called from a separate thread.
The function main_loop does a carrier sense before transmitting a new packet. The thing I don't understand is why it is sensing a receive channel before transmitting on a separate non-overlapping transmit channel.
Both the RX and TX channels are separate frequencies, and our hardware allows full-duplex communication.
SO, my question is with main_loop executing, what are the implications of another thread asynchronously calling the phy_rx_callback function? The problem is I am trying to understand the purpose of the carrier sense loop, I found that commenting that code severely decreases performance. It doesn't make sense to me that you would monitor a receive channel before using a transmit channel, essentially turning it into half-duplex. Then I don't see the purpose of using two frequencies, one for transmit and one for receive. I began to wonder if there was a strange threading issue at work here.
A single instance of the cs_mac class is created initially. A 'pointer' to the rx_callback function is passed down a few levels to the thread class which actually calls it. Here is the cs_mac class:
class cs_mac(object):
def __init__(self, tun_fd, verbose=False):
self.tun_fd = tun_fd # file descriptor for TUN/TAP interface
self.verbose = verbose
self.tb = None # top block (access to PHY)
def set_top_block(self, tb):
self.tb = tb
def phy_rx_callback(self, ok, payload):
if self.verbose:
print "Rx: ok = %r len(payload) = %4d" % (ok, len(payload))
if ok:
os.write(self.tun_fd, payload)
def main_loop(self):
min_delay = 0.001 # seconds
while 1:
payload = os.read(self.tun_fd, 10*1024)
if not payload:
self.tb.send_pkt(eof=True)
break
if self.verbose:
print "Tx: len(payload) = %4d" % (len(payload),)
delay = min_delay
while self.tb.carrier_sensed():
sys.stderr.write('B')
time.sleep(delay)
if delay < 0.050:
delay = delay * 2 # exponential back-off
self.tb.send_pkt(payload)
Ok, so using ctypes.CDLL('libc.so.6').syscall(186)), which calls gettid I discovered that the thread calling the rx_callback function has the same PID, but a different TID.
The question becomes, what are the implications of having a separate thread call a function from an object in the main thread (while that thread is constantly looping)?

The function main_loop does a carrier sense before transmitting a new packet. The thing I don't understand is why it is sensing a receive channel before transmitting on a separate non-overlapping transmit channel.
The CSMA/CA is intended to be used with half-duplex systems, where all nodes use the same frequency to TX and RX. So you are right, there is no point in sensing the RX channel if you are transmitting in a different one.
carrier_sensed() is called in the receive_path.py file so it should run in the RX thread. In my code I comment out the lines sys.stderr.write('B') and time.sleep(delay) and this does not seem to affect performance. It might be different in my case since I use an XCVR daughter board which is half-duplex.