I have a Twisted application that's listening for Int32StringReceiver messages and then re-sending them to another app. Basically, it's a router, but it has some intelligence, introspecting where the data is going.
My problem is with the outbound side, getting lot of error messages, etc.
Inbound is a class Receiver(Int32StringReceiver):
def doActualForwarding(self, data):
self.stats.recvBits += 8 * (4 + len(data))
self.stats.recvMsgs += 1
dlen = len(data)
if dlen > 1024*256:
self.logger.info("router.Receiver.doActualForwarding(): data len: %s" % (dlen))
self.router.forward(data)
def stringReceived(self, data):
d = threads.deferToThread(self.doActualForwarding, data)
d.addCallback(self.forwardingDoneOkay)
d.addErrback(self.forwardingDoneError)
The self.router is instantiated object that needs to send these messages out via socket comms in the same format. So, it just turns around and does this in the Router class:
def connect(self):
if self.sock:
try:
self.sock.close()
except:
pass
try:
self.stats.connectAttempts += 1
self.sock = socket.socket()
self.sock.settimeout(self.CONNECT_TIMEOUT)
self.sock.connect(self.destination)
self.sock.settimeout(self.SEND_TIMEOUT)
self.set_keepalive_linux(self.sock)
self.connected = True
self.log.info("connected to %s" % (self.destination,))
self.stats.reconnects += 1
self.stats.connectCompletes += 1
return True
except Exception, e:
self.connected = False
if not self.drop_ok:
self.log.error("connect %s: %s" % (self.destination, e))
return False
def send(self, msg):
trynum = 0
while trynum < self.MAX_SEND_ATTEMPTS:
self.logSent()
if not self.connected:
if not self.connect():
self.stats.badSends += 1
time.sleep(self.DELAY_BEFORE_RECONNECT)
continue
try:
if ((time.time() - self.lastReconnectTime) > self.RECONNECT_EVERY):
self.lastReconnectTime = time.time()
assert False, "Reconnecting with destination to redistribute load."
self.sock.sendall(msg)
#self.closeSocket()
self.stats.events += 1
return True
except Exception, e:
whichKind = None
if 'Broken pipe' in str(e):
self.stats.brokenPipe += 1
elif 'Resource temporarily unavilable' in str(e):
self.stats.resourceTempUnavail += 1
elif 'Bad file descriptor' in str(e):
self.stats.badFileDescriptor += 1
self.log.error("send: %s %s" % (str(self.destination), str(e)))
try:
self.sock.close()
except:
pass
self.connected = False
self.stats.badSends += 1
trynum += 1
if trynum == 1:
self.stats.eventsWithRetry += 1
if trynum > 1:
self.log.warning("recon_sender.send(): Trynum non-singular, was: %s" % (trynum))
return False
def __del__(self):
try:
self.sock.close()
except:
pass
QUESTIONS:
Is Python's Socket library threadsafe? That is, functionally, two or more threads have a pointer to the object Router. Both threads are calling self.sock.sendall(msg) and I'm concerned they'll step on each other.
One symptom is that it might be that successive messages are appended to each other. I'm not sure about this, but it looks that way.
I'm seeing a lot of resource temp. unavail (meaning destination is busy), about the same number of broken pipes, and a small number of bad file descriptor.
[Errno 9] Bad file descriptor
[Errno 11] Resource temporarily unavailable
[Errno 32] Broken pipe
These messages correspond to maybe 0.5% (.005) of the number of messages going through this thing.
I tried to have each send do a connect/sendall/shutdown/close, but that resulted in a ton of messages about 'connection reset by peer'.
Everyone seems to be intent on code that handles multi-threaded receiving on sockets, but not so many comment on multi-threaded SENDING on sockets.
I also tried to use (possibly incorrectly):
import threading
self.lock = threading.Lock()
with self.lock:
sock.sendall(msg)
but this resulted in error messages about timing out (yuck).
Can someone point me in the direction of some good examples (Or PROVIDE SOME?!?!?!?) that demonstrate multithreaded socket sendall()?
I would say that if the processes do not have to communicate with eachother, your best solution will be to spawn a new process to handle each incoming connection. This way you don't have to worry about locking as each connection will be handled separately.
Simple implementation would be:
import socket
import multiprocessing
import pdb
import random
from pycurl import Curl
import os
import time
import re
class query(object):
pid, addr, conn, url, ua, ref = [None for i in range(6)]
compression = True
def __init__(self, conn, addr):
self.pid = addr[1]
self.addr = addr
self.conn = conn
self.process()
def process(self):
#do your socket stuff here
class ProxyServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
def start(self):
logging.info("Server started on %s:%i" % (self.host, self.port))
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.bind((self.host, self.port))
self.sock.listen(0)
while True:
conn, addr = self.sock.accept()
logging.info('Connection made from %s' % conn)
proc = multiprocessing.Process(target=query, args=(conn, addr))
proc.daemon = True
proc.start()
logging.info('Started processing query %r for %s' % (proc, addr))
if __name__ == "__main__":
serv = ProxyServer(host, port)
try:
serv.start()
except:
finally:
for proc in multiprocessing.active_children():
proc.terminate()
proc.join()
Keep in mind that this is an example that I cut from old proof-of-concept code, you will have to tweak it a bit before it's ready for production.
Related
I have implemented a server in Python 3.8.6 using the socket library (running on a Raspberry Pi 4 w/ 8G RAM & Raspberry Pi OS: Buster) which, by design, only handles a single connection at a time. When a connection is gained, the server spawns two threads, one for sending data, and one for receiving it. When the connection is lost (in theory, it will never be intentionally closed), these threads are terminated and a new thread is spawned to accept a new connection. The problem is, the receiving thread never actually terminates, instead getting stuck infinitely on socket.recv(). This persists even after calling socket.shutdown(SHUT_RDWR) and socket.close(). The code is essentially as follows (I've paired it down to only the pertinent bits and left the heartbeat management code, as I've found this is the only way I can consistently tell that the connection has dropped, as the errors that are supposed to be thrown don't always):
import dataclasses
import logging
import pickle
import queue
import socket
import threading
import time
import traceback
log = logging.getLogger()
class Server(threading.Thread):
#dataclasses.dataclass
class Heartbeat:
failures: int = 0
reciprocate: bool = True
time: float = 0
__CLIENT_IP = '10.1.1.58'
__EOF = "\r\nEOF\r\n".encode()
__PORT = 33368
def __init__(self, kill_flag:threading.Event) -> None:
self.__kill_flag: threading.Event = kill_flag
self.__server = socket.create_server(("", self.__PORT), backlog=0, reuse_port=True)
self.__server.settimeout(5)
self.__client = None
self.__queue_out = queue.Queue(-1)
self.__queue_in = queue.Queue(-1)
self._connected = False
self.__connecting_lock = threading.Lock()
self.__receive_thread = None
self.__transmit_thread = None
threading.Thread(target=self.__connect).start()
def __connect(self) -> None:
with self.__connecting_lock:
if self._connected:
return
addr = ("", 0)
if self.__client:
try:
self.__client.shutdown(socket.SHUT_RDWR)
except OSError:
pass
finally:
self.__client.close()
self.__client = None
while (self.__receive_thread and self.__receive_thread.is_alive()) or (self.__transmit_thread and self.__transmit_thread.is_alive()):
log.debug("connected: %s, recv alive: %s, trans alive: %s", self._connected, self.__receive_thread.is_alive(), self.__transmit_thread.is_alive())
time.sleep(1)
log.info("Waiting for connection...")
while addr[0] != self.__CLIENT_IP and not self.__kill_flag.is_set():
try:
self.__client, addr = self.__server.accept()
if addr[0] != self.__CLIENT_IP:
self.__client.close()
log.warning("Connection from %s:%s - Rejected", addr[0], addr[1])
time.sleep(1)
except socket.timeout:
continue
if self.__kill_flag.is_set():
self.__del__()
else:
log.info("Connection from %s:%s - Accepted", addr[0], addr[1])
self.heartbeat = self.Heartbeat()
self._connected = True
self.__receive_thread = threading.Thread(target=self.__receive)
self.__transmit_thread = threading.Thread(target=self.__transmit)
self.__queue_in.queue.clear()
self.__receive_thread.start()
self.__transmit_thread.start()
def get_package(self) -> tuple:
return self.__queue_out.get(True)
def queue_message(self, content:tuple = ()) -> None:
self.__queue_in.put_nowait(content)
def __receive(self) -> None:
buffer = bytearray()
while self._connected and not self.__kill_flag.is_set():
try:
if self.__EOF in buffer:
payload = pickle.loads(buffer.split(self.__EOF, maxsplit=1)[0])
self.__queue_out.put_nowait(payload)
buffer = bytearray(self.__EOF.join(buffer.split(self.__EOF)[1:]))
else:
try:
log.debug("looping into recv")
data = self.__client.recv(1024)
log.debug("looping past recv")
if len(data) == 0:
raise ConnectionResetError
else:
buffer += data
except OSError as error:
log.error(f"Receive error: {error}")
if self._connected:
self._connected = False
threading.Thread(target=self.__connect).start()
except AttributeError:
break
except socket.timeout:
continue
except Exception as error:
log.error(f"Receive error: {error}")
traceback.print_tb(error.__traceback__)
if self._connected:
self._connected = False
threading.Thread(target=self.__connect).start()
def __transmit(self) -> None:
while self._connected and not self.__kill_flag.is_set():
try:
payload = self.__queue_in.get(True, 5)
except queue.Empty:
if self.heartbeat.reciprocate:
payload = (time.time(),)
self.heartbeat.reciprocate = False
elif time.time() - 7 >= self.heartbeat.time:
self.heartbeat.failures += 1
log.debug("Heartbeat failure")
if self.heartbeat.failures >= 3:
log.warning("Lost connection to client: No heartbeat detected")
if self._connected:
self._connected = False
threading.Thread(target=self.__connect).start()
self.heartbeat.reciprocate = True
continue
else:
continue
self.heartbeat.time = time.time()
try:
self.__client.sendall(pickle.dumps(payload))
log.debug("Package sent: %s", payload)
except BrokenPipeError:
log.warning("Lost connection to client: Broken pipe")
self.__queue_in.put_nowait(payload)
if self._connected:
self._connected = False
threading.Thread(target=self.__connect).start()
except AttributeError:
break
def __del__(self) -> None:
if self.__client:
self.__client.close()
self.__server.close()
After the connection drops, the __receive method/thread hangs specifically on this line: data = self.__client.recv(1024).
Perhaps the strangest part is that the client, whose __transmit and __receive methods are virtually identical, does not have this problem.
Any insight anyone can offer into as why this is happening, and how to remedy it, would be greatly appreciated.
Many thanks!
I'm trying to run a heartbeat message on a socket connection (every 30 seconds it sends a message), so I'm running it on it's own thread, and have tried it as daemon or not. I'm on a virtual machine -- Windows x86.
The program runs correctly, and transmits the message, but on the 12th iteration of the heartbeat I begin getting 'Threading Error!' (and Socket disconnects) -- I've tried turning off firewalls, machine related stuff like that, but don't know what's going on with the connection/why it's dropping. I've also tried this recursively, without threading -- I receive WinError 10053, established connection aborted by software in host machine.
def every(delay, task):
next_time = time.time() + delay
while True:
time.sleep(max(0, next_time - time.time()))
try:
task()
except Exception:
print('Error threading!')
next_time += (time.time() - next_time) // delay * delay + delay
def test():
print("Threadtest", time.time())
class HB_Socket():
def __init__(self,IP='127.0.0.1',port=4000,hb = 30):
self.IP = IP
self.port = port
self.HB = hb
self.socket = socket() #AF_INET,SOCK_DGRAM)
self.connected = False
def cnct(self):
s = self.socket
#s.connect((self.IP,self.port)) #MOVED THIS
try:
s.connect((self.IP,self.port))
print('Socket connected to {} on port {}'.format(self.IP,self.port))
on = self.logOn()
self.send(on)
print('LogOn message sent')
self.connected = True
return 1
except:
print('Socket creation failed: {}')
return 0
#Function being called in every()
def sendHB(self):
hbsocket = self.socket
print('PyHeartBeat client sending to IP {}'.format(self.IP,self.port))
#while 1:
hbsocket.sendto(HB_MSG,('127.0.0.1',self.port))
if HB_SIM:
sleep(self.HB)
if datetime.datetime.now().time().hour >= 16:
self.disconnect()
return
self.sendHB()
if __name__ == '__main__':
HB = HB_Socket()
HB.cnct()
HB_SIM = False
threading.Thread(target=every,args=(30,HB.sendHB)).start()
HB_SIM = True #Recursive solution.
#HB.sendHB()
#Daemon / Recursive
dmn = threading.Thread(target=HB.sendHB(), daemon=True)
dmn.start()
Here's an image of the return with errors
self.handlers.append(ConnHandler(sock, self.handlers))I'm new to python and I tried to write a simple socket server to test stuff out and get to know the language better.
import asyncore
import socket
import json
class ConnHandler(asyncore.dispatcher_with_send):
def __init__(self, conn, handlerlist):
asyncore.dispatcher_with_send.__init__(self, conn)
self.handlers = handlerlist
def handle_close(self):
self.close()
print 'Socket closed'
if(self.handlers.count(self) > 0):
self.handlers.remove(self);
def handle_read(self):
data = ''
more = True
while more:
try:
data += self.recv(1024)
except socket.error, e:
more = False
if data == '':
return
try:
message = json.loads(data)
except ValueError:
self.send('Invalid JSON\n')
return
print message
class TestServer(asyncore.dispatcher):
def __init__(self, host, port):
asyncore.dispatcher.__init__(self)
self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
self.set_reuse_addr()
self.bind((host, port))
self.listen(5)
self.handlers = []
def handle_accept(self):
pair = self.accept()
if pair is not None:
sock, addr = pair
print 'Incoming connection from %s' % repr(addr)
self.handlers.append(ConnHandler(sock, self.handlers))
def sendToAll(self, string):
for h in self.handlers:
h.send(string + '\n')
server = TestServer('localhost', 55555)
asyncore.loop()
My problem is the following.
If I connect to the server with telnet and then quit telnet to close the connection, python just spams 'Socket closed' in the terminal. How can I fix this? Are there obvious beginner mistakes in the above code?
handle_close is not meant to detect if peer is disconnected, you get this information in handle_read if .recv call returns an empty string. Then you can close the socket, and then handle_close is called.
Here is a modified version of your code:
def handle_close(self):
print 'Socket closed'
if(self.handlers.count(self) > 0):
self.handlers.remove(self);
def handle_read(self):
data = ''
more = True
while more:
try:
new_data = self.recv(1024)
if not new_data:
self.close()
return
else:
data += new_data
except socket.error, e:
more = False
if data == '':
return
try:
message = json.loads(data)
except ValueError:
self.send('Invalid JSON\n')
return
print message
I have been running a script that contains three threads;
The main thread handles IRC messages, the second is a thread that containing a socket server that sends strings from IRC to a flash display when it connects as a client and the third is a socket server that sends the policy file which is a flash requirement when the client connects.
The python file runs fine my on "localhost", but when I host it on my linode it recieves the the first string only and sends the string response that is displayed on flash before crashing with the following error message.
The following error :
Traceback (most recent call last):
File "/opt/python3.3/lib/python3.3/threading.py", line 616, in _bootstrap
self._bootstrap_inner()
File "/opt/python3.3/lib/python3.3/threading.py", line 682, in _bootstrap_inner
self._stop()
TypeError: 'Event' object is not callable
Here is my socket server code:
class FlashSOCKET (threading.Thread):
def __init__(self, threadID, name, counter):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.counter = counter
self._stop = threading.Event()
self.stopWaitLoop = False
self.conn = None
self.addr = None
self.HOST = None
self.PORT = 7001
self.s = None
def run(self):
global running, toSend, message
print ("Starting " + self.name)
while running:
self.startListening()
while running:
if (messageDeque):
try:
print(self.conn.recv(1024))
self.conn.send(messageDeque.popleft().encode('utf8'))
print (messageDeque)
break
except:
pass # Nothing to do keep listening
if (self.conn != None):
self.close_connection()
if (self.conn != None):
self.close_connection()
print ("Exiting " + self.name)
def startListening(self):
global running, message, toSend
print ("starting to listen")
for res in socket.getaddrinfo(self.HOST, self.PORT, socket.AF_UNSPEC,
socket.SOCK_STREAM, 0, socket.AI_PASSIVE):
af, socktype, proto, canonname, sa = res
try:
self.s = socket.socket(af, socktype, proto)
self.s.settimeout(0.5)
except socket.error as msg:
self.s = None
continue
try:
self.s.bind(sa)
self.s.listen(1)
except socket.error as msg:
print ("socket error")
self.s.close()
self.s = None
continue
break
if self.s is None:
print ('could not open socket')
sys.exit(1)
while(running):
try:
self.conn, self.addr = self.s.accept()
time.sleep(1)
break
except:
continue
print ('Connected by', self.addr)
def close_connection(self):
self.conn.close()
print ("connection closed")
Running is a global bool flag variable that let me stop the program when I need to.
The messageDeque is a deque buffer that gets filled with strings in the main thread when people talk in IRC.
I think, toSend and message are bits of redundant code.
try to avoid global variables. In normal programming they're sketchy, but in multithreaded programming they're more misleading than useful.
Event object: do if self._stop.is_set() vs checking a global
I'm having problems detecting a broken socket when a broken pipe exception occurs. See the below code for an example:
The Server:
import errno, select, socket, time, SocketServer
class MetaServer(object):
def __init__(self):
self.server = Server(None, Handler, bind_and_activate=False)
def run(self, sock, addr):
rfile = sock.makefile('rb', 1)
self.server.process_request(sock, addr)
while 1:
r, _, _ = select.select([rfile], [], [], 1.0)
if r:
print 'Got %s' % rfile.readline()
else:
print 'nothing to read'
class Server(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
allow_reuse_address = True
daemon_threads = True
class Handler(SocketServer.StreamRequestHandler):
def handle(self):
print 'connected!'
try:
while 1:
self.wfile.write('testing...')
time.sleep(1)
except socket.error as e:
if e.errno == errno.EPIPE:
print 'Broken pipe!'
self.finish()
self.request.close()
if __name__ == '__main__':
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('127.0.0.1', 8081))
s.listen(5)
ms = MetaServer()
while 1:
client, address = s.accept()
ms.run(client, address)
The Client:
import select, socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 8081))
while 1:
r, _, _ = select.select([s], [], [], 1.0)
if not r:
continue
msg = s.recv(1024)
print 'Got %s' % (msg,)
Now, if I run the server and client, all is well, and I get a "nothing is read" message every second. As soon as I CTRL-C out of the client, the server goes crazy and starts to "read" from what should be a busted socket, dumping a lot of "Got " messages.
Is there some way to detect this broken socket in the MetaServer.run() function to avoid the above said behavior?
Yes, that's something which is not really in the documentation but old Un*x behavior: You need to abort when you get an empty string.