CPU reaching 100% usage with select in python - python

I have a problem with a select in python, I have a piece of code which allow a client to receive data from a server and also send it by reading on stdin and writing on the server socket:
readfds = [s, sys.stdin]
writefds = [s, sys.stdout]
my_level = get_my_level(s)
is_co = True
cmd = ""
while (is_co):
read, write, exception = select.select(readfds, writefds, [], 1)
if (not (read or write or exception)):
print "Timeout"
else:
for sock in read:
if (sock == s):
cmd = readline(s)
print cmd
elif (sock == sys.stdin):
cmd = sys.stdin.readline()
s.sendall(cmd)
if (cmd == "mort"):
is_co = False
I think it's because of the select is non-blocking but when I make it block it's the same thing. Can you explain wath is wrong in my code ?
Thanks

Presumably you can always write to sys.stdout so select.select should be returning immediately to let you know you can write to something. This code then processes the readable list, and then re-enters the loop. But nothing has changed with respect to sys.stdout so it continues to be writable.
This will execute in a tight loop and burn CPU.

I had a similar problem, so I continuously poll the socket (UDP in my case) and use sleep() timeout. In this code the CPU usage went from 100% to about 4%
sUDP = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sUDP.bind((host, portIn))
sUDP.setblocking(0)
sUDP.settimeout(0.5)
while True:
try:
data, addr = sUDP.recvfrom(4096)
... #some data processing code here
except:
time.sleep(0.02)

Related

Python while loop is pausing

I am trying to do multiple while loops but somehow they don't work. I already searched the internet but none of the problems I found has the same issue.
So here is the code containing only the necessary information. I am basically opening a socket, giving an in input (i\n) and receiving the output in the first step. I want to continue receiving the output until I have some specific characters xxx in the output. Then I want to go to the elif statement in the next loop.
def netcat(h, p):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((h,p))
i = 0
a = True
while a == True:
socket_list=[sys.stdin, s]
r,w,x = select.select(socket_list, [], [])
if i==0:
time.sleep(1)
message = s.recv(1024)
print(message)
s.send("i\n")
sys.stdout.flush()
while "xxx" not in message:
message = s.recv(1024)
print(message)
i+=1
elif i==1:
print("+++++++++++++++++++++++++++")
i+=1
print("hello")
server.close()
What I would expect the code to do is to print the message from the if statement, then print hello, then the message from the elif statement and then hello over and over again because the while loop is still active. So in summary this is the expected output:
message
hello
+++++++++++++++++++++++++++
hello
hello
hello
hello...
What it really prints is
message
hello
and then it finishes.
What I found out is that if I comment out the following lines:
while "xxx" not in message:
message = s.recv(1024)
print(message)
it works as expected. The hello at the end of the code gets printed to the screen over and over again. I just don't get it why this second while loop has anything to do with it. I would really appreciate help here.
Since the working code was requested, here is also the full code. The hostname and port are from a CTF which is still working so you will be interacting with the CTF-server:
#!/usr/bin/env python
import socket
import time
import select
import sys
base64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ ="
hostname = "18.188.70.152"
port = 36150
def netcat(h, p):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((h,p))
i = 0
a = True
b = True
while a == True:
socket_list=[sys.stdin, s]
r,w,x = select.select(socket_list, [], [])
if i==0:
time.sleep(1)
message = s.recv(1024)
print(message)
s.send("i\n")
sys.stdout.flush()
while "flag" not in message:
message = s.recv(1024)
print(message)
txtfile = message[9:38]
print(txtfile)
i+=1
elif i==1:
print("+++++++++++++++++++++++++++")
i+=1
print("hello")
server.close()
netcat(hostname, port)
You're mixing event-based code (select.select()) with blocking synchronous code (your small while loop with the s.recv()).
If you want your code not to block, every recv() needs to be paired up with a preceding select().
Not only that, but you must also check the returned values from select(). Only s.recv() if s was in the first returned list. If you s.recv() in any other case, the code will also block on the receive call.
Update:
Try something along the lines of:
not_done = True
while not_done:
read_sockets, _, _ = select.select([sys.stdin, s], [], [])
if s in read_sockets:
message = s.recv(1024)
print(message)
... more code ...
if 'flag' in message:
... react on flag ...
if 'quit' in message:
not_done = False
... processing of other sockets or file descriptors ...
The important point being that there is only this one s.recv() in the if branch that checks for whether select detected something was received.
The outer while will just come back to the same if branch later when additional data was received.
Note that processing stdin alongside socket code is tricky and will likely also block at some point. You will likely have to put the terminal into raw mode or something first and then be ready to process partial lines yourself as well as maybe also echoing the input back to the user.
Update:
If you want to do something while no message was received, you can give a timeout to select() and then do other processing if there was nothing received on the socket. Something like this:
say_hello_from_now_on = False
not_done = True
while not_done:
read_sockets, _, _ = select.select([s], [], [], 1)
if s in read_sockets:
message = s.recv(1024)
print(message)
say_hello_from_now_on = True
elif say_hello_from_now_on:
print("hello")
I'd check your indentation, try installing and running autopep8 on your code and see if that fixes any of your issues.
[edit] user has updated their question and it's clear that this isn't the answer.

IRC client application, how to use select() in python

Hi am working on network application that reads from standard input and sends messages to server both in infinite loops. For every loop I have thread, that calls function with what should be done. I need to be able to catch SIGINT and close both threads properly. My program is stucked on both reading functions. I think I should use select() function, but I dont know how.
def writing_mode():
while 1:
var = raw_input()
# some options with what to do with the var
def listening_mode():
readbuffer = ""
while 1:
readbuffer = readbuffer + s.recv(1024)
temp = string.split(readbuffer, "\r\n")
readbuffer = temp.pop()
for line in temp:
line = string.strip(line)
prefix, command, rest, trailing = parse(line)
# do something with the options u have
s=socket.socket()
try:
s.connect((args.h, args.p))
except Exception, e:
print("Something is wrong with %s:%d, Exception type is:\n%s" % (args.h, args.p, e))
sys.exit()
s.send("NICK AAA\r\n")
s.send("USER AAA AAA AAA AAA\r\n")
thread.start_new_thread(writing_mode)
thread.start_new_thread(listenning_mode)
#main program waiting for signals
I've solved this with non-blocking reading through select().
while condition:
# s is socket to read from
# for stdin use sys.stdin instead
ready = select.select([s], [], [], 0)
if ready[0]:
#u can start read
The select function, goes on the reading function only if there is anything to read. Otherwise it goes to the while loop so the condition can stop it at any time.

Using Processes as Threads with Networking in Python

Basically, my idea was to write some sort of basic server where I could connect to my computer and then run a command remotely. This didn't seem to be much of a problem; but then I had the bright idea that the next step would logically be to add some sort of threading so I could spawn multiple connections.
I read that, because of the GIL, multiprocessing.Process would be the best to try to do this. I don't completely understand threading and it's hard to find good documentation on it; so I'm kind of just throwing stuff and trying to figure out how it works.
Well, it seems like I might be close to doing this right; but I have a feeling I'm just as likely to be no where near doing this correctly. My program now does allow multiple connections, which it didn't when I first started working with threading; but once a connection is established, and then another is established, the first connection is no longer able to send a command to the server. I would appreciate it if someone could give me any help, or point me in the right direction on what I need to learn and understand.
Here's my code:
class server:
def __init__(self):
self.s = socket.socket()
try:
self.s.bind(("",69696))
self.s.listen(1)
except socket.error,(value,message):
if self.s:
self.s.close()
def connection(self):
while True:
client , address = self.s.accept()
data = client.recv(5)
password = 'hello'
while 1:
if data == password:
subprocess.call('firefox')
client.close()
else:
client.send('wrong password')
data = client.recv(5)
p = Process(target=x.connection())
p.start()
x = server()
if __name__ == '__main':
main()
Well, this answer only applies if you're on a unix or unix-like operating system(windows does not have os.fork() which we use).
One of the most common approaches for doing these things on unix platforms is to fork a new process to handle the client connection while the master process continues to listen for requests.
Below is code for a simple echo server that can handle multiple simultaneous connections. You just need to modify handle_client_connection() to fit your needs
import socket
import os
class ForkingServer:
def serve_forever(self):
self.s = socket.socket()
try:
self.s.bind(("", 9000))
self.s.listen(1)
except socket.error, (value,message):
print "error:", message
if self.s:
self.s.close()
return
while True:
client,address = self.s.accept()
pid = os.fork()
# You should read the documentation for how fork() works if you don't
# know it already
# The short version is that at this point in the code, there are 2 processes
# completely identical to each other which are simulatenously executing
# The only difference is that the parent process gets the pid of the child
# returned from fork() and the child process gets a value of 0 returned
if pid == 0:
# only the newly spawned process will execute this
self.handle_client_connection(client, address)
break
# In the meantime the parent process will continue on to here
# thus it will go back to the beginning of the loop and accept a new connection
def handle_client_connection(self, client,address):
#simple echo server
print "Got a connection from:", address
while True:
data = client.recv(5)
if not data:
# client closed the connection
break
client.send(data)
print "Connection from", address, "closed"
server = ForkingServer()
server.serve_forever()

Python Paramiko timeout with long execution, need full output

There's lots of topics touching on part of the title, but nothing that quite satisfies the whole thing. I'm pushing a command on a remote server and need the full output after a long execution time, say 5 minutes or so. Using channel I was able to set a timeout, but when I read back stdout I got only a small portion of output. The solution seemed to be to wait for channel.exit_status_ready(). This worked on a successful call, but a failed call would never trigger the channel timeout. Having reviewed the docs, I theorize that's because the timeout only works on a read operation, and waiting for exit status doesn't qualify. Here's that attempt:
channel = ssh.get_transport().open_session()
channel.settimeout(timeout)
channel.exec_command(cmd) # return on this is not reliable
while True:
try:
if channel.exit_status_ready():
if channel.recv_ready(): # so use recv instead...
output = channel.recv(1048576)
break
if channel.recv_stderr_ready(): # then check error
error = channel.recv_stderr(1048576)
break
except socket.timeout:
print("SSH channel timeout exceeded.")
break
except Exception:
traceback.print_exc()
break
Pretty, ain't it? Wish it worked.
My first attempt at a solution was to use time.time() to get a start, then check start - time.time() > timeout. This seems straightforward, but in my present version, I output start - time.time() with a fixed timeout that should trigger a break...and see differences that double and triple the timeout with no break occurring. To save space, I'll mention my third attempt, which I've rolled up with this one. I read on here about using select.select to wait for output, and noted in the documentation that there's a timeout there as well. As you'll see from the code below, I've mixed all three methods -- channel timeout, time.time timeout, and select timeout -- yet still have to kill the process. Here's the frankencode:
channel = ssh.get_transport().open_session()
channel.settimeout(timeout)
channel.exec_command(cmd) # return on this is not reliable
print("{0}".format(cmd))
start = time.time()
while True:
try:
rlist, wlist, elist = select([channel], [], [],
float(timeout))
print("{0}, {1}, {2}".format(rlist, wlist, elist))
if rlist is not None and len(rlist) > 0:
if channel.exit_status_ready():
if channel.recv_ready(): # so use recv instead...
output = channel.recv(1048576)
break
elif elist is not None and len(elist) > 0:
if channel.recv_stderr_ready(): # then check error
error = channel.recv_stderr(1048576)
break
print("{0} - {1} = {2}".format(
time.time(), start, time.time() - start))
if time.time() - start > timeout:
break
except socket.timeout:
print("SSH channel timeout exceeded.")
break
except Exception:
traceback.print_exc()
break
Here's some typical output:
[<paramiko.Channel 3 (open) window=515488 -> <paramiko.Transport at 0x888414cL (cipher aes128-ctr, 128 bits) (active; 1 open channel(s))>>], [], []
1352494558.42 - 1352494554.69 = 3.73274183273
The top line is [rlist, wlist, elist] from select, the bottom line is time.time() - start = (time.time() - start). I got this run to break by counting the iterations and breaking at the bottom of the try after looping 1000 times. timeout was set to 3 on the sample run. Which proves that we get through the try, but obviously, none of the three ways that should be timing out works.
Feel free to rip into the code if I've fundamentally misunderstood something. I'd like for this to be uber-Pythonic and am still learning.
Here's something that might help, though I'm still in the midst of testing. After struggling with timeouts of various types including a catch-all timeout for Python, and realizing that the real problem is that the server can't be trusted to terminate the process, I did this:
chan = ssh.get_transport().open_session()
cmd = "timeout {0} {1}\n".format(timeouttime, cmd)
chan.exec_command(cmd)
The server times out after timeouttime if cmd doesn't exit sooner, exactly as I'd wish, and the terminated command kills the channel. The only catch is that GNU coreutils must exist on the server. Failing that there are alternatives.
I'm having the same kind of issue. I think we can handle it with signalling. http://docs.python.org/2/library/signal.html
Here is a plain dumb example to show how it works.
import signal, time
def handler(signum, frame):
pass
# Set the signal handler and a 2-second alarm
signal.signal(signal.SIGALRM, handler)
signal.alarm(2)
# This is where your operation that might hang goes
time.sleep(10)
# Disable the alarm
signal.alarm(0)
So here, the alarm is set to 2 seconds. Time.sleep is called with 10 seconds. Of course, the alarm will be triggered before the sleep finishes. If you put some output after the time.sleep, you'll see that program execution resumes there.
If you want the control to continue somewhere else, wrap your hanging call in a try/except and have your handler function raise an exception.
Although I'm pretty sure it would work, I haven't tested it yet over paramiko calls.
I had a lot of problem calling the exec_command from the channel, instead I use directly the exec_command from the ssh connection and call the channel of the std output, the code that works for me is like myexec:
#!/usr/bin/env python
import paramiko
import select
def myexec(ssh, cmd, timeout):
stdin, stdout, stderr = ssh.exec_command(cmd)
channel = stdout.channel
stdin.close() #As I don't need stdin
channel.shutdown_write() #As I will not write to this channel
stdout_chunks = []
stdout_chunks.append(stdout.channel.recv(len(stdout.channel.in_buffer)))
# chunked read to prevent stalls
while not channel.closed or channel.recv_ready()
or channel.recv_stderr_ready():
# stop if channel was closed prematurely,
# and there is no data in the buffers.
got_chunk = False
readq, _, _ = select.select([stdout.channel], [], [], timeout)
for c in readq:
if c.recv_ready():
stdout_chunks.append(stdout.channel.recv(len(c.in_buffer)))
got_chunk = True
if c.recv_stderr_ready():
# make sure to read stderr to prevent stall
stderr.channel.recv_stderr(len(c.in_stderr_buffer))
got_chunk = True
if not got_chunk \
and stdout.channel.exit_status_ready() \
and not stderr.channel.recv_stderr_ready() \
and not stdout.channel.recv_ready():
# indicate that we're not going to read from this channel anymore
stdout.channel.shutdown_read() # close the channel
stdout.channel.close()
break # exit as remote side is finished and our bufferes are empty
# close all the pseudofiles
stdout.close()
stderr.close()
return (''.join(stdout_chunks), stdout.channel.recv_exit_status())
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('remotehost', username='remoteuser', password='remotepassword')
rtrval = myexec(ssh, 'remotecomand', 5*60)
ssh.close()
print rtrval
I use Debian 8 and Python 2.7.13, good luck.

Thread synchronization in Python

I am currently working on a school project where the assignment, among other things, is to set up a threaded server/client system. Each client in the system is supposed to be assigned its own thread on the server when connecting to it. In addition i would like the server to run other threads, one concerning input from the command line and another concerning broadcasting messages to all clients. However, I can't get this to run as i want to. It seems like the threads are blocking each other. I would like my program to take inputs from the command line, at the "same time" as the server listens to connected clients, and so on.
I am new to python programming and multithreading, and allthough I think my idea is good, I'm not suprised my code doesn't work. Thing is I'm not exactly sure how I'm going to implement the message passing between the different threads. Nor am I sure exactly how to implement the resource lock commands properly. I'm going to post the code for my server file and my client file here, and I hope someone could help me with this. I think this actually should be two relative simple scripts. I have tried to comment on my code as good as possible to some extend.
import select
import socket
import sys
import threading
import client
class Server:
#initializing server socket
def __init__(self, event):
self.host = 'localhost'
self.port = 50000
self.backlog = 5
self.size = 1024
self.server = None
self.server_running = False
self.listen_threads = []
self.local_threads = []
self.clients = []
self.serverSocketLock = None
self.cmdLock = None
#here i have also declared some events for the command line input
#and the receive function respectively, not sure if correct
self.cmd_event = event
self.socket_event = event
def openSocket(self):
#binding server to port
try:
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind((self.host, self.port))
self.server.listen(5)
print "Listening to port " + str(self.port) + "..."
except socket.error, (value,message):
if self.server:
self.server.close()
print "Could not open socket: " + message
sys.exit(1)
def run(self):
self.openSocket()
#making Rlocks for the socket and for the command line input
self.serverSocketLock = threading.RLock()
self.cmdLock = threading.RLock()
#set blocking to non-blocking
self.server.setblocking(0)
#making two threads always running on the server,
#one for the command line input, and one for broadcasting (sending)
cmd_thread = threading.Thread(target=self.server_cmd)
broadcast_thread = threading.Thread(target=self.broadcast,args=[self.clients])
cmd_thread.daemon = True
broadcast_thread.daemon = True
#append the threads to thread list
self.local_threads.append(cmd_thread)
self.local_threads.append(broadcast_thread)
cmd_thread.start()
broadcast_thread.start()
self.server_running = True
while self.server_running:
#connecting to "knocking" clients
try:
c = client.Client(self.server.accept())
self.clients.append(c)
print "Client " + str(c.address) + " connected"
#making a thread for each clientn and appending it to client list
listen_thread = threading.Thread(target=self.listenToClient,args=[c])
self.listen_threads.append(listen_thread)
listen_thread.daemon = True
listen_thread.start()
#setting event "client has connected"
self.socket_event.set()
except socket.error, (value, message):
continue
#close threads
self.server.close()
print "Closing client threads"
for c in self.listen_threads:
c.join()
def listenToClient(self, c):
while self.server_running:
#the idea here is to wait until the thread gets the message "client
#has connected"
self.socket_event.wait()
#then clear the event immidiately...
self.socket_event.clear()
#and aquire the socket resource
self.serverSocketLock.acquire()
#the below is the receive thingy
try:
recvd_data = c.client.recv(self.size)
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
self.socket_event.clear()
self.serverSocketLock.release()
return
print recvd_data
#I put these here to avoid locking the resource if no message
#has been received
self.socket_event.clear()
self.serverSocketLock.release()
except socket.error, (value, message):
continue
def server_cmd(self):
#this is a simple command line utility
while self.server_running:
#got to have a smart way to make this work
self.cmd_event.wait()
self.cmd_event.clear()
self.cmdLock.acquire()
cmd = sys.stdin.readline()
if cmd == "":
continue
if cmd == "close\n":
print "Server shutting down..."
self.server_running = False
self.cmdLock.release()
def broadcast(self, clients):
while self.server_running:
#this function will broadcast a message received from one
#client, to all other clients, but i guess any thread
#aspects applied to the above, will work here also
try:
send_data = sys.stdin.readline()
if send_data == "":
continue
else:
for c in clients:
c.client.send(send_data)
self.serverSocketLock.release()
self.cmdLock.release()
except socket.error, (value, message):
continue
if __name__ == "__main__":
e = threading.Event()
s = Server(e)
s.run()
And then the client file
import select
import socket
import sys
import server
import threading
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#making a lock for the socket resource
self.clientSocketLock = threading.Lock()
self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure wat daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append(listen)
self.running_threads.append(speak)
listen.start()
speak.start()
#this while-loop is just for avoiding the script terminating
while self.client_running:
dummy = 1
#closing the threads if the client goes down
print "Client operating on its own"
self.client.close()
#close threads
for t in self.running_threads:
t.join()
return
#defining "listen"-function
def listenToServer(self):
while self.client_running:
#here i acquire the socket to this function, but i realize I also
#should have a message passing wait()-function or something
#somewhere
self.clientSocketLock.acquire()
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
#releasing the socket resource
self.clientSocketLock.release()
#defining "speak"-function, doing much the same as for the above function
def speakToServer(self):
while self.client_running:
self.clientSocketLock.acquire()
try:
send_data = sys.stdin.readline()
if send_data == "close\n":
print "Disconnecting..."
self.client_running = False
else:
self.client.send(send_data)
except socket.error, (value,message):
continue
self.clientSocketLock.release()
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
I realize this is quite a few code lines for you to read through, but as I said, I think the concept and the script in it self should be quite simple to understand. It would be very much appriciated if someone could help me synchronize my threads in a proper way =)
Thanks in advance
---Edit---
OK. So I now have simplified my code to just containing send and receive functions in both the server and the client modules. The clients connecting to the server gets their own threads, and the send and receive functions in both modules operetes in their own separate threads. This works like a charm, with the broadcast function in the server module echoing strings it gets from one client to all clients. So far so good!
The next thing i want my script to do, is taking specific commands, i.e. "close", in the client module to shut down the client, and join all running threads in the thread list. Im using an event flag to notify the listenToServer and the main thread that the speakToServer thread has read the input "close". It seems like the main thread jumps out of its while loop and starts the for loop that is supposed to join the other threads. But here it hangs. It seems like the while loop in the listenToServer thread never stops even though server_running should be set to False when the event flag is set.
I'm posting only the client module here, because I guess an answer to get these two threads to synchronize will relate to synchronizing more threads in both the client and the server module also.
import select
import socket
import sys
import server_bygg0203
import threading
from time import sleep
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
self.disconnected = threading.Event()
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure what daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append((listen,"listen"))
self.running_threads.append((speak, "speak"))
listen.start()
speak.start()
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
#closing the threads if the client goes down
print "Client operating on its own"
self.client.shutdown(1)
self.client.close()
#close threads
#the script hangs at the for-loop below, and
#refuses to close the listen-thread (and possibly
#also the speak thread, but it never gets that far)
for t in self.running_threads:
print "Waiting for " + t[1] + " to close..."
t[0].join()
self.disconnected.clear()
return
#defining "speak"-function
def speakToServer(self):
#sends strings to server
while self.client_running:
try:
send_data = sys.stdin.readline()
self.client.send(send_data)
#I want the "close" command
#to set an event flag, which is being read by all other threads,
#and, at the same time set the while statement to false
if send_data == "close\n":
print "Disconnecting..."
self.disconnected.set()
self.client_running = False
except socket.error, (value,message):
continue
return
#defining "listen"-function
def listenToServer(self):
#receives strings from server
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
return
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
Later on, when I get this server/client system up and running, I will use this system on some elevator models we have here on the lab, with each client receiving floor orders or "up" and "down" calls. The server will be running an distribution algorithm and updating the elevator queues on the clients that are most appropriate for the requested order. I realize it's a long way to go, but I guess one should just take one step at the time =)
Hope someone has the time to look into this. Thanks in advance.
The biggest problem I see with this code is that you have far too much going on right away to easily debug your problem. Threading can get extremely complicated because of how non-linear the logic becomes. Especially when you have to worry about synchronizing with locks.
The reason you are seeing clients blocking on each other is because of the way you are using your serverSocketLock in your listenToClient() loop in the server. To be honest this isn't exactly your problem right now with your code, but it became the problem when I started to debug it and turned the sockets into blocking sockets. If you are putting each connection into its own thread and reading from them, then there is no reason to use a global server lock here. They can all read from their own sockets at the same time, which is the purpose of the thread.
Here is my recommendation to you:
Get rid of all the locks and extra threads that you don't need, and start from the beginning
Have the clients connect as you do, and put them in their thread as you do. And simply have them send data every second. Verify that you can get more than one client connecting and sending, and that your server is looping and receiving. Once you have this part working, you can move on to the next part.
Right now you have your sockets set to non-blocking. This is causing them all to spin really fast over their loops when data is not ready. Since you are threading, you should set them to block. Then the reader threads will simply sit and wait for data and respond immediately.
Locks are used when threads will be accessing shared resources. You obviously need to for any time a thread will try and modify a server attribute like a list or a value. But not when they are working on their own private sockets.
The event you are using to trigger your readers doesn't seem necessary here. You have received the client, and you start the thread afterwards. So it is ready to go.
In a nutshell...simplify and test one bit at a time. When its working, add more. There are too many threads and locks right now.
Here is a simplified example of your listenToClient method:
def listenToClient(self, c):
while self.server_running:
try:
recvd_data = c.client.recv(self.size)
print "received:", c, recvd_data
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
return
print recvd_data
except socket.error, (value, message):
if value == 35:
continue
else:
print "Error:", value, message
Backup your work, then toss it - partially.
You need to implement your program in pieces, and test each piece as you go. First, tackle the input part of your program. Don't worry about how to broadcast the input you received. Instead worry that you are able to successfully and repeatedly receive input over your socket. So far - so good.
Now, I assume you would like to react to this input by broadcasting to the other attached clients. Well too bad, you can't do that yet! Because, I left one minor detail out of the paragraph above. You have to design a PROTOCOL.
What is a protocol? It's a set of rules for communication. How does your server know when the client had finished sending it's data? Is it terminated by some special character? Or perhaps you encode the size of the message to be sent as the first byte or two of the message.
This is turning out to be a lot of work, isn't it? :-)
What's a simple protocol. A line-oriented protocol is simple. Read 1 character at a time until you get to the end of record terminator - '\n'. So, clients would send records like this to your server --
HELO\n
MSG DAVE Where Are Your Kids?\n
So, assuming you have this simple protocol designed, implement it. For now, DON'T WORRY ABOUT THE MULTITHREADING STUFF! Just worry about making it work.
Your current protocol is to read 1024 bytes. Which may not be bad, just make sure you send 1024 byte messages from the client.
Once you have the protocol stuff setup, move on to reacting to the input. But for now you need something that will read input. Once that is done, we can worry about doing something with it.
jdi is right, you have too much program to work with. Pieces are easier to fix.

Categories