I'm trying to run a heartbeat message on a socket connection (every 30 seconds it sends a message), so I'm running it on it's own thread, and have tried it as daemon or not. I'm on a virtual machine -- Windows x86.
The program runs correctly, and transmits the message, but on the 12th iteration of the heartbeat I begin getting 'Threading Error!' (and Socket disconnects) -- I've tried turning off firewalls, machine related stuff like that, but don't know what's going on with the connection/why it's dropping. I've also tried this recursively, without threading -- I receive WinError 10053, established connection aborted by software in host machine.
def every(delay, task):
next_time = time.time() + delay
while True:
time.sleep(max(0, next_time - time.time()))
try:
task()
except Exception:
print('Error threading!')
next_time += (time.time() - next_time) // delay * delay + delay
def test():
print("Threadtest", time.time())
class HB_Socket():
def __init__(self,IP='127.0.0.1',port=4000,hb = 30):
self.IP = IP
self.port = port
self.HB = hb
self.socket = socket() #AF_INET,SOCK_DGRAM)
self.connected = False
def cnct(self):
s = self.socket
#s.connect((self.IP,self.port)) #MOVED THIS
try:
s.connect((self.IP,self.port))
print('Socket connected to {} on port {}'.format(self.IP,self.port))
on = self.logOn()
self.send(on)
print('LogOn message sent')
self.connected = True
return 1
except:
print('Socket creation failed: {}')
return 0
#Function being called in every()
def sendHB(self):
hbsocket = self.socket
print('PyHeartBeat client sending to IP {}'.format(self.IP,self.port))
#while 1:
hbsocket.sendto(HB_MSG,('127.0.0.1',self.port))
if HB_SIM:
sleep(self.HB)
if datetime.datetime.now().time().hour >= 16:
self.disconnect()
return
self.sendHB()
if __name__ == '__main__':
HB = HB_Socket()
HB.cnct()
HB_SIM = False
threading.Thread(target=every,args=(30,HB.sendHB)).start()
HB_SIM = True #Recursive solution.
#HB.sendHB()
#Daemon / Recursive
dmn = threading.Thread(target=HB.sendHB(), daemon=True)
dmn.start()
Here's an image of the return with errors
Related
I am writing a simple threaded server that will send a message to all clients. I have an object that is reset after posting the change message, however I am having a hard time figuring out how to reset that object only after all threads have posted the change message.
To add some context to the problem. I am building a multi user Tkinter python app which connects to a remote database to retrieve information and the application needs to know when data changes so that when a user updates data, all other running instances of the app will get the update. From what I understand, MySQL does not support asynchronous application updates. Instead of running a query every 5 seconds on the database to see if there is a change, I am putting this code server side so that it will send a message to a socket on the client that a change has occurred on the database.
The main loop is just a dummy that will simulate a change
Here is my code:
import socket, threading, time, select, os
class dbMonitor:
isDBAltered = False
def postChange(self):
self.isDBAltered = True
def __str__(self):
return str(self.isDBAltered)
class ThreadedServer(object):
def __init__(self, port,dbMon):
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.setblocking(0)
self.sock.bind((socket.gethostname(), self.port))
self.dbMon = dbMon
def listen(self):
self.sock.listen(100)
read_list = [self.sock]
while True:
read,write,error = select.select(read_list,[],[],1)
for s in read:
if s is self.sock:
client, address = self.sock.accept()
client.settimeout(60)
threading.Thread(target = self.listenToClient, args = (client,address)).start()
def listenToClient(self, client, address):
read_list = [client]
size = 1024
while True:
response = b'Ack'
if self.dbMon.isDBAltered:
response = b'CHANGE'
try:
client.send(response)
except:
client.close()
return False
self.dbMon.isDBAltered = False
read,write,error = select.select(read_list,[],[],1)
for s in read:
if s is client:
try:
data = client.recv(size)
print(data)
if data:
client.send(response)
else:
raise error('Client disconnected')
except:
client.close()
return False
def mainLoop():
while True:
time.sleep(15)
print(dbMon)
dbMon.postChange()
dbMon = dbMonitor()
server = ThreadedServer(5005,dbMon)
threading.Thread(target = mainLoop, args=()).start()
threading.Thread(target = server.listen(), args=()).start()
How do I get self.dbMon.isDBAltered = False to execute only after all threads have executed:
response = b'CHANGE'
try:
client.send(response)
You're trying to synchronize something that's asynchronous... This is massively more complicated than it should be. Your dbmon is only storing a boolean flag... why not just asynchronously modify the "database" instead? For example, if the "database" was a thread-safe buffer, you could just append to that buffer or modify that buffer without synchronizing each thread individually, pull the information written to that buffer and write it to the client socket they belong to in another event loop (this is pretty much what asyncore does)
That said, I have some (probably nonworking, but I hope you get the idea) reference modified code for you to go off of if you want to continue pursing this avenue.
Basically, dbmon will keep a mapping of thread ids to [creation time, modified flag]
Our predicate returns true iff all threads created before a certain threshold have ALL set the modified flag. We set the modified flag when we send the response in the data = client.recv(size) portion of your code. And then we wait on that condition in the server send. We keep notifying all waiting threads on each client receive so that when the condition is finally met, our waiting server threads will all unblock and send the subsequent response.
import socket, threading, time, select, os
import collections
class dbMonitor:
def __init__(self):
self.isDBAltered = {}
self.lock = threading.Lock()
def newThread(self, tid):
self.lock.acquire()
# time of creation, boolean whether that thread has sent response
self.isDBAltered[tid] = [time.time(), False]
self.lock.release()
def threadDone(self, tid):
self.lock.acquire()
self.isDBAltered.pop(tid, None)
self.lock.release()
def altered(self, tid):
self.lock.acquire()
self.isDBAltered[tid][1] = True
self.lock.release()
def reset(self, tid):
self.lock.acquire()
self.isDBAltered(tid)[1] = False
self.lock.release()
def __str__(self):
return str(self.isDBAltered)
class ThreadedServer(object):
def __init__(self, port,dbMon):
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.setblocking(0)
self.sock.bind((socket.gethostname(), self.port))
self.dbMon = dbMon
self.lock = threading.lock()
self.cv = threading.Condition()
self.thresh = 2000
def condition_pred(self):
# unblock if threads currently running for longer than self.thresh have all set their flags
return all([timecreate[1] if time.time() - timecreate[0] > self.thresh else True for tid,timecreate in self.dbMon.isDBAltered])
def listen(self):
self.sock.listen(100)
read_list = [self.sock]
while True:
read,write,error = select.select(read_list,[],[],1)
for s in read:
if s is self.sock:
self.lock.acquire()
client, address = self.sock.accept()
client.settimeout(60)
T = threading.Thread(target = self.listenToClient, args = (client,address)).start()
self.dbmon.newThread(T.ident)
self.lock.release()
def listenToClient(self, client, address):
read_list = [client]
size = 1024
while True:
response = b'Ack'
with self.cv:
self.cv.wait_for(self.condition_pred)
self.dbMon.reset(threading.get_ident())
response = b'CHANGE'
try:
client.send(response)
except:
client.close()
self.dbmon.threadDone(threading.get_ident())
return False
read,write,error = select.select(read_list,[],[],1)
for s in read:
if s is client:
with self.cv:
try:
data = client.recv(size)
print(data)
if data:
client.send(response)
self.dbMon.altered(threading.get_ident())
self.cv.notifyAll()
else:
raise error('Client disconnected')
except:
client.close()
self.dbmon.threadDone(threading.get_ident())
return False
I tried to create multithreaded echo server:
echomain.py:
#!/usr/bin/python
from echoserver import echoserver
server = echoserver()
print server.isRunning()
print server.port()
server.start()
print "Main program continues..."\\This part is not displayed(((
echoserver.py:
#!/usr/bin/python
import threading
import socket
class connection(threading.Thread):
def __init__(self, sock, addr):
self.sock = sock
self.addr = addr
threading.Thread.__init__(self)
def run (self):
while True:
buffer = self.sock.recv(1024)
if buffer == "disconnect\r\n":
self.sock.send("bye")
break
elif buffer:
self.sock.send(buffer)
self.sock.close()
class echoserver(object):
def __init__(self, port=12119):
self.running = False
self._port = port
self._socket = None
def isRunning(self):
return self.running
def port(self):
return self._port
def start(self):
self.running = True
self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._socket.bind(("0.0.0.0", self.port()))
self._socket.listen(5)
while True:
conn, addr = self._socket.accept()
connection(conn, addr).start()
def stop(self):
self._socket.close()
print "Server is closed..."
Could somebody help me in how I can launch echoserver class as a thread so it run simultaneously with main program so I could stop it with stop() method in echomain.py part?
Change your runner program to run the server as a thread:
echomain.py:
#!/usr/bin/python
from echoserver import echoserver
from threading import Thread
import time
server = echoserver()
print server.isRunning()
print server.port()
# server.start()
# run server in a different thread
serverThread = Thread(target=server.start)
serverThread.start()
print "main - server started"
# wait ten seconds before stopping
time.sleep(10)
server.stop()
print "main - server stopped"
print "Main program continues..."
This example simply stops the server after 10 seconds.
The simplest way is to have your echoserver itself be a Thread as proposed by Reut Sharabani, but IMHO, you should also implement a correct stop() method, ensuring that all children have ended.
Here is my implementation of your script :
#!/usr/bin/python
import threading
import socket
class connection(threading.Thread):
def __init__(self, sock, addr, parent):
self.sock = sock
self.addr = addr
self.parent = parent
threading.Thread.__init__(self)
self.sock.settimeout(None)
self.closed = False # will be set to True on thread end
def run (self):
while not self.parent._stopped:
buffer = self.sock.recv(1024)
if buffer == "disconnected\r\n":
self.sock.send("bye")
break
elif buffer:
self.sock.send(buffer)
self.sock.close()
self.closed = True
class echoserver(threading.Thread):
def __init__(self, port=12119):
threading.Thread.__init__(self)
self.running = False
self._port = port
self._socket = None
self._stopped = False
self._conns = [] # list of active connections
def isRunning(self):
return self.running
def port(self):
return self._port
def run(self):
self.running = True
self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._socket.bind(("0.0.0.0", self.port()))
self._socket.listen(5)
self._socket.settimeout(5) # use a timeout to respond to stop()
while not self._stopped:
try:
conn, addr = self._socket.accept()
c = connection(conn, addr, self)
self._conns.append(c) # add child the the list
c.start()
except Exception as e:
# print e # in debug
pass
self._conns = self.child_list() # remove closed child from list
self._socket.close()
print "Server is closing..."
for connect in self._conns: # join active children
connect.join()
print "Server is closed"
def stop(self):
self._stopped = True
def child_list(self):
l = []
for conn in self._conns:
if conn.closed:
conn.join()
else:
l.append(conn)
return l
Remarks :
you simply use it that way :
serv=echoserver()
serv.start()
... # sleep of do anything you want
serv.stop()
if no connection is active when you call stop() all stops at the end of the accept timeout and you get :
Server is closing...
Server is closed
if at least one connection is active when you call stop(), you get only Server is closing... at the end of the accept timeout. Then for each connection, it will end as soon as it receives a packet, and will be joined by echoserver. Then when all connection are over, you will get Server is closed and echoserver thread will terminate
that means that in you main thread you have only to do
serv.stop()
serv.join()
to be sure that all other threads are correctly terminated, and that all sockets are closed
I have a Twisted application that's listening for Int32StringReceiver messages and then re-sending them to another app. Basically, it's a router, but it has some intelligence, introspecting where the data is going.
My problem is with the outbound side, getting lot of error messages, etc.
Inbound is a class Receiver(Int32StringReceiver):
def doActualForwarding(self, data):
self.stats.recvBits += 8 * (4 + len(data))
self.stats.recvMsgs += 1
dlen = len(data)
if dlen > 1024*256:
self.logger.info("router.Receiver.doActualForwarding(): data len: %s" % (dlen))
self.router.forward(data)
def stringReceived(self, data):
d = threads.deferToThread(self.doActualForwarding, data)
d.addCallback(self.forwardingDoneOkay)
d.addErrback(self.forwardingDoneError)
The self.router is instantiated object that needs to send these messages out via socket comms in the same format. So, it just turns around and does this in the Router class:
def connect(self):
if self.sock:
try:
self.sock.close()
except:
pass
try:
self.stats.connectAttempts += 1
self.sock = socket.socket()
self.sock.settimeout(self.CONNECT_TIMEOUT)
self.sock.connect(self.destination)
self.sock.settimeout(self.SEND_TIMEOUT)
self.set_keepalive_linux(self.sock)
self.connected = True
self.log.info("connected to %s" % (self.destination,))
self.stats.reconnects += 1
self.stats.connectCompletes += 1
return True
except Exception, e:
self.connected = False
if not self.drop_ok:
self.log.error("connect %s: %s" % (self.destination, e))
return False
def send(self, msg):
trynum = 0
while trynum < self.MAX_SEND_ATTEMPTS:
self.logSent()
if not self.connected:
if not self.connect():
self.stats.badSends += 1
time.sleep(self.DELAY_BEFORE_RECONNECT)
continue
try:
if ((time.time() - self.lastReconnectTime) > self.RECONNECT_EVERY):
self.lastReconnectTime = time.time()
assert False, "Reconnecting with destination to redistribute load."
self.sock.sendall(msg)
#self.closeSocket()
self.stats.events += 1
return True
except Exception, e:
whichKind = None
if 'Broken pipe' in str(e):
self.stats.brokenPipe += 1
elif 'Resource temporarily unavilable' in str(e):
self.stats.resourceTempUnavail += 1
elif 'Bad file descriptor' in str(e):
self.stats.badFileDescriptor += 1
self.log.error("send: %s %s" % (str(self.destination), str(e)))
try:
self.sock.close()
except:
pass
self.connected = False
self.stats.badSends += 1
trynum += 1
if trynum == 1:
self.stats.eventsWithRetry += 1
if trynum > 1:
self.log.warning("recon_sender.send(): Trynum non-singular, was: %s" % (trynum))
return False
def __del__(self):
try:
self.sock.close()
except:
pass
QUESTIONS:
Is Python's Socket library threadsafe? That is, functionally, two or more threads have a pointer to the object Router. Both threads are calling self.sock.sendall(msg) and I'm concerned they'll step on each other.
One symptom is that it might be that successive messages are appended to each other. I'm not sure about this, but it looks that way.
I'm seeing a lot of resource temp. unavail (meaning destination is busy), about the same number of broken pipes, and a small number of bad file descriptor.
[Errno 9] Bad file descriptor
[Errno 11] Resource temporarily unavailable
[Errno 32] Broken pipe
These messages correspond to maybe 0.5% (.005) of the number of messages going through this thing.
I tried to have each send do a connect/sendall/shutdown/close, but that resulted in a ton of messages about 'connection reset by peer'.
Everyone seems to be intent on code that handles multi-threaded receiving on sockets, but not so many comment on multi-threaded SENDING on sockets.
I also tried to use (possibly incorrectly):
import threading
self.lock = threading.Lock()
with self.lock:
sock.sendall(msg)
but this resulted in error messages about timing out (yuck).
Can someone point me in the direction of some good examples (Or PROVIDE SOME?!?!?!?) that demonstrate multithreaded socket sendall()?
I would say that if the processes do not have to communicate with eachother, your best solution will be to spawn a new process to handle each incoming connection. This way you don't have to worry about locking as each connection will be handled separately.
Simple implementation would be:
import socket
import multiprocessing
import pdb
import random
from pycurl import Curl
import os
import time
import re
class query(object):
pid, addr, conn, url, ua, ref = [None for i in range(6)]
compression = True
def __init__(self, conn, addr):
self.pid = addr[1]
self.addr = addr
self.conn = conn
self.process()
def process(self):
#do your socket stuff here
class ProxyServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
def start(self):
logging.info("Server started on %s:%i" % (self.host, self.port))
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.bind((self.host, self.port))
self.sock.listen(0)
while True:
conn, addr = self.sock.accept()
logging.info('Connection made from %s' % conn)
proc = multiprocessing.Process(target=query, args=(conn, addr))
proc.daemon = True
proc.start()
logging.info('Started processing query %r for %s' % (proc, addr))
if __name__ == "__main__":
serv = ProxyServer(host, port)
try:
serv.start()
except:
finally:
for proc in multiprocessing.active_children():
proc.terminate()
proc.join()
Keep in mind that this is an example that I cut from old proof-of-concept code, you will have to tweak it a bit before it's ready for production.
I have the following python script (a bit inefficient, I know) for a P2P chat program using sockets:
#!usr/bin/env python
import socket import threading import select import time import datetime
def main():
class Chat_Server(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.running = 1
self.conn = None
self.addr = None
def run(self):
HOST = ''
PORT = 23647
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST,PORT))
s.listen(1)
self.conn, self.addr = s.accept()
# Select loop for listen
while self.running == True:
inputready,outputready,exceptready \
= select.select ([self.conn],[self.conn],[])
for input_item in inputready:
# Handle sockets
message = self.conn.recv(1024)
if message:
print "Daniel: " + message + ' (' + datetime.datetime.now().strftime('%H:%M:%S') + ')'
else:
break
time.sleep(0)
def kill(self):
self.running = 0
class Chat_Client(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.host = None
self.sock = None
self.running = 1
def run(self):
PORT = 23647
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((self.host, PORT))
# Select loop for listen
while self.running == True:
inputready,outputready,exceptready \
= select.select ([self.sock],[self.sock],[])
for input_item in inputready:
# Handle sockets
message = self.sock.recv(1024)
if message:
print "Daniel: " + message + ' (' + datetime.datetime.now().strftime('%H:%M:%S') + ')'
else:
break
time.sleep(0)
def kill(self):
self.running = 0
class Text_Input(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.running = 1
def run(self):
while self.running == True:
text = raw_input('')
try:
chat_client.sock.sendall(text)
except:
Exception
try:
chat_server.conn.sendall(text)
except:
Exception
time.sleep(0)
def kill(self):
self.running = 0
# Prompt, object instantiation, and threads start here.
ip_addr = raw_input('Type IP address or press enter: ')
if ip_addr == '':
chat_server = Chat_Server()
chat_client = Chat_Client()
chat_server.start()
text_input = Text_Input()
text_input.start()
else:
chat_server = Chat_Server()
chat_client = Chat_Client()
chat_client.host = ip_addr
text_input = Text_Input()
chat_client.start()
text_input.start()
if __name__ == "__main__":
main()
This script works absolutely fine across a home network, with 192.168... internal IP addresses.
On a school network, with 172... IP addresses, it doesn't seem to work. There is no connection error, but messages are not sent or received, with the exception of if there are two instances of the application being run on the same computer with the same internal IP address, in which case the program works flawlessly.
I am very new to sockets and this sort of networking, so I am wondering if anyone can point out to me why this is the case. Might it be the difference in IP addresses, for example?
Thank you in advance.
I see you're connecting on port 23647 - you may not have access to this on your school network. Check if this port has traffic enabled. see: Port Forwarding for details.
Something like: this site/tool may allow you to check quickly.
In Python 3.3.3， i create a thread to listen some connection to the socket.It likes this:
import threading
import socket
import time
Host = ''
Port = 50000
flag = False
class ServerThread(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self._sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def run(self):
try:
self._sock.bind((Host, Port))
self._sock.listen(5)
while True:
conn, addr = self._sock.accept()
print('Connected by', addr)
except socket.error as msg:
print(msg)
except Exception as e:
print(str(e))
finally:
self._sock.close()
def exit(self):
self._sock.close()
def TargetFunc(vlock):
vlock.acquire()
flag = True
vlock.release()
def main():
sthread = ServerThread()
sthread.start()
vlock = threading.Lock()
time.sleep(10)
vthread = threading.Thread(target = TargetFunc, args = (vlock, ))
vthread.start()
while True:
vlock.acquire()
if flag:
sthread.exit()
vlock.release()
break
vlock.release()
sthread.join()
vthread.join()
if __name__ == '__main__':
main()
There are two threads, one is listening socket, the other is to set a flag. When the flag is True, close the socket, then raise a socket error and catch it, so the listening socket terminates.But why it does not work this.
Thanks!
self._sock.accept() is blocking. So it will wait until somebody connects. You should use a nonblocking variant (or blocking but with a time-out). So that you can check the exit conditions.
Alternatively you could force an exception in the ServerThread.