Python - User input to CGI via. Threading and reading file - python

Look at the bottom of this post, for final working code.
It's a working Python/CGI script which can get user-input to a CGI-script by calling another script which then sends it's commands through a local socket.
Original post:
As far as I know, there isn't any way to send user input directly to a Python/CGI script which has allready sent it's header. Like, warning the user under specific circumstances and waiting for a confirmation.
Neither have I been able to find any published solutions to this.
If I'm wrong, please correct me.
I currently have a Python script which can connect to servers, upload firmware, reboot, re-connect, change a few configuration files and such.
Sometimes, it would help alot of the user could send input to script, without having to re-launch the script and execute it from the beginning. Re-connecting over a 2G network takes too long.
I'm thinking that it must be possible to send user input to another script, which then posts it to a file, which the first/main script is watching, until it recieves the input.
It would also be nice, if the was able to stop the execution of the script, with a stop/kill input command.
As for the stop/kill command, the main script would need to have 2 threads. If it did not, it would know it should stop the script, if a process such as a large file upload is being executed, before the upload is completed.
At the same time, I think multipe users should be able to use the script at the same time. Therefore, a unique ID must be generated every time the main script launches.
Here's how I think it could be made:
Main script gets called
Global variable with a unique session ID is generated and sent to client.
Thread 1
pexpect spawns a "tail -F /var/www/cgi/tmp_cmd.log"
Thread 2
Thread status "Busy"
Connects to network element
Does its usual stuff until it reaches a point where the user needs to interact.
Prints the message to user and waits for Thread 1 with a timeout of x seconds.
Thread status "Ready"
Second script gets called by the user through AJAX with 2 headers (session ID & input)
Second script
Session ID and user input is saved to "/var/www/cgi/tmp_cmd.log"
Execution of the input script ends
Main script
Thread 1
User input recieved.
Wait for Thread 2 status to become "Ready" or ignore status if command is equals to "kill" ect.
Send user input (single line) and start Thread 1 from the beginning
Thread 2
Thread 2 status "Busy"
Input recieved and process stops/continues.
Thread 2 status "Ready"
I have made a script allready for connecting, uploading files, and running commands.
However, it cannot recieve user-input.
I could really use some good help, or someone to tell me how to approach this.
Of course, whenever the script has been completed, I will post it here or on pastebin and link to it, for other people to use. :)
Final code
With help from the post below, I have finally have the working code.
It could use Threads, but stopping/cancelling processes appeared to be way easier for me to figure out.
Client - cgi_send.py
#!/usr/bin/python
import sys, cgi, cgitb, socket
cgitb.enable()
TASKS_DIR = "/var/www/cgi-bin/tmp"
def main():
global TASKS_DIR
url = cgi.FieldStorage()
cmd = str(url.getvalue('cmd'))
sessionId = str(url.getvalue('session'))
socketLocation = TASKS_DIR + '/%s.socket' % sessionId
print 'End script Cancel task'
print '<form action=""><input type="hidden" name="session" id="session" value="'+sessionId+'" /><input type="text" name="cmd" id="cmd" value="" /><input type="submit" value="Fun!" />'
try:
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.connect(socketLocation)
sock.send(cmd)
sock.close()
print '<br />Command sent: '+ cmd;
except IOError:
print '<br /><b>Operation failed.</b><br /> Could not write to socket: '+ socketLocation
pass
sock.close()
sys.exit();
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
Server
#!/usr/bin/python
import sys, os, socket, uuid, time, multiprocessing
# Options
TASKS_DIR = "/var/www/cgi-bin/tmp/"
def main():
sessionId = str(uuid.uuid4())
print 'Session ID: '+ sessionId
sys.stdout.write ('<br />Send test command')
sys.stdout.flush()
address = os.path.join(TASKS_DIR, '%s.socket' % sessionId)
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.settimeout(.1)
sock.bind(address)
sock.listen(1)
taskList = [foo_task, foo_task, foo_task]
try:
for task in taskList:
print "<br />Starting new task"
runningTask = multiprocessing.Process(target=task)
runningTask.daemon = True # Needed to make KeyboardInterrupt possible when testing in shell
runningTask.start()
while runningTask.is_alive():
conn = None
try:
conn, addr = sock.accept()
data = conn.recv(100).strip()
except socket.timeout:
# nothing ready from a client
continue
except socket.error, e:
print "<br />Connection Error from client"
else:
print "<br />"+ data
sys.stdout.flush()
conn.close()
if data == "CANCEL":
# temp way to cancel our task
print "<br />Cancelling current task."
runningTask.terminate()
elif data == "QUIT":
print "<br />Quitting entire process."
runningTask.terminate()
taskList[:] = []
finally:
if conn:
conn.close()
except (KeyboardInterrupt, SystemExit):
print '\nReceived keyboard interrupt, quitting threads.'
finally:
sock.close()
os.remove(address)
def foo_task():
i = 1
while 10 >= i:
print "<br />Wating for work... "+ str(i)
sys.stdout.flush()
i = i + 1
time.sleep(1)
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()

A CGI script is a pretty primitive operation. It works basically the same as any normal script you run from your command shell. An http request is made to the web server. The server starts a new process and passes the arguments in via stdin to the script. At this point, it's like a normal script.
A script can't get any more input unless it's looking for input by some means, so you are correct in assuming that once the headers are sent, the web client can no longer directly send more input, because the request is already in progress, and the response is already in progress as well.
A thread watching a file is one way to introduce a control loop to the script. Another is to open a UNIX socket to a path based on your unique ID for each instance. Then have the thread sitting on the socket for input. What you would then have to do is pass the ID back to the web client. And the client could make a call to the second script with the ID, which would then know the proper UNIX socket path to send control commands to: ie.
/tmp/script-foo/control/<id>.socket
You actually might only need 1 thread. You main thread could simply loop over checking for information on the socket, and monitoring the current operation being run in a thread or subprocess. It might be like this in pseudocode:
uid = generate_unique_id()
sock = socket.socket(AF_UNIX)
sock.bind('/tmp/script-foo/control/%s.socket' % uid)
# and set other sock options like timeout
taskList = [a,b,c]
for task in taskList:
runningTask = start task in thread/process
while runningTask is running:
if new data on socket, with timeout N ms
if command == restart:
kill runningTask
taskList = [a,b,c]
break
else:
process command
When the web client sends a command via ajax to your second script, it might look like this in pseudocode:
jobid = request.get('id')
cmd = request.get('cmd')
sock = socket.socket(socket.AF_UNIX)
sock.connect('/tmp/script-foo/control/%s.socket' % jobid)
sock.sendall(cmd)
sock.close()
Update
Based on your code update, here is a working example of what I was suggesting:
import sys
import os
import socket
import uuid
import time
# Options
TASKS_DIR = "."
def main():
sessionId = str(uuid.uuid4())
print 'Session ID: '+ sessionId
sys.stdout.write ('<br />Send test command')
sys.stdout.flush()
address = os.path.join(TASKS_DIR, '%s.socket' % sessionId)
sock = socket.socket(socket.AF_UNIX)
sock.setblocking(0)
sock.settimeout(.1)
sock.bind(address)
sock.listen(1)
fakeTasks = [foo_task, foo_task, foo_task]
try:
for task in fakeTasks:
# pretend we started a task
runningTask = task()
# runningTask = Thread(target=task)
# runningTask.start()
# while runningTask.is_alive():
while runningTask:
conn = None
try:
conn, addr = sock.accept()
data = conn.recv(100).strip()
except socket.timeout:
# nothing ready from a client
continue
except socket.error, e:
print "<br />Connection Error from client"
else:
print "<br />"+ data
sys.stdout.flush()
conn.close()
# for the thread version, you will need some
# approach to kill or interrupt it.
# This is just simulating.
if data == "CANCEL":
# temp way to cancel our task
print "<br />Cancelling current task."
runningTask = False
elif data == "QUIT":
print "<br />Quitting entire process."
runningTask = False
fakeTasks[:] = []
finally:
if conn:
conn.close()
finally:
sock.close()
os.remove(address)
def foo_task():
print 'foo task'
return True
if __name__ == '__main__':
sys.stdout.write("Content-type:text/html;charset=utf-8\r\n\r\n")
sys.stdout.write('<!DOCTYPE html>\n<html><head><title>Test</title></head><body>')
main()
print '</body></html>'
sys.exit()
Instead of using a 10 second global timeout, you set it to something small like 100ms. It loops over each task and starts it (eventually in a thread), and then tries to loop over waiting for a socket connection. If there is no connection within 100ms, it will timeout and continue to loop, while checking if the task is done. At any point, a client can connect and issue either a "CANCEL" or "QUIT" command. The socket will accept the connection, read it, and react.
You can see how you do not need multiple threads here for the solution. The only threading or subprocess you need is to run the task.

Related

Python: How to interrupt raw_input() in other thread

I am writing a simple client-server program in python. In the client program, I am creating two threads (using Python's threading module), one for receiving, one for sending. The receiving thread continuously receives strings from the server side; while the sending thread continuously listens to the user input (using raw_input()) and send it to the server side. The two threads communicate using a Queue (which is natively synchronized, LIKE!).
The basic logic is like following:
Receiving thread:
global queue = Queue.Queue(0)
def run(self):
while 1:
receive a string from the server side
if the string is QUIT signal:
sys.exit()
else:
put it into the global queue
Sending thread:
def run(self):
while 1:
str = raw_input()
send str to the server side
fetch an element from the global queue
deal with the element
As you can see, in the receiving thread, I have a if condition to test whether the server has sent a "QUIT signal" to the client. If it has, then I want the whole program to stop.
The problem here is that for most of its time, the sending thread is blocked by "raw_input()" and waiting for the user input. When it is blocked, calling "sys.exit()" from the other thread (receiving thread) will not terminate the sending thread immediately. The sending thread has to wait for the user to type something and hit the enter button.
Could anybody inspire me how to get around with this? I do not mind using alternatives of "raw_input()". Actually I do not even mind changing the whole structure.
-------------EDIT-------------
I am running this on a linux machine, and my Python version is 2.7.5
You could just make the sending thread daemonic:
send_thread = SendThread() # Assuming this inherits from threading.Thread
send_thread.daemon = True # This must be called before you call start()
The Python interpreter won't be blocked from exiting if the only threads left running are daemons. So, if the only thread left is send_thread, your program will exit, even if you're blocked on raw_input.
Note that this will terminate the sending thread abruptly, no matter what its doing. This could be dangerous if it accesses external resources that need to be cleaned up properly or shouldn't be interrupted (like writing to a file, for example). If you're doing anything like that, protect it with a threading.Lock, and only call sys.exit() from the receiving thread if you can acquire that same Lock.
The short answer is you can't. input() like a lot of such input commands is blocking and it's blocking whether everything about the thread has been killed. You can sometimes call sys.exit() and get it to work depending on the OS, but it's not going to be consistent. Sometimes you can kill the program by deferring out to the local OS. But, then you're not going to be widely cross platform.
What you might want to consider if you have this is to funnel the functionality through the sockets. Because unlike input() we can do timeouts, and threads and kill things rather easily. It also gives you the ability to do multiple connections and maybe accept connections more broadly.
import socket
import time
from threading import Thread
def process(command, connection):
print("Command Entered: %s" % command)
# Any responses are written to connection.
connection.send(bytes('>', 'utf-8'))
class ConsoleSocket:
def __init__(self):
self.keep_running_the_listening_thread = True
self.data_buffer = ''
Thread(target=self.tcp_listen_handle).start()
def stop(self):
self.keep_running_the_listening_thread = False
def handle_tcp_connection_in_another_thread(self, connection, addr):
def handle():
while self.keep_running_the_listening_thread:
try:
data_from_socket = connection.recv(1024)
if len(data_from_socket) != 0:
self.data_buffer += data_from_socket.decode('utf-8')
else:
break
while '\n' in self.data_buffer:
pos = self.data_buffer.find('\n')
command = self.data_buffer[0:pos].strip('\r')
self.data_buffer = self.data_buffer[pos + 1:]
process(command, connection)
except socket.timeout:
continue
except socket.error:
if connection is not None:
connection.close()
break
Thread(target=handle).start()
connection.send(bytes('>', 'utf-8'))
def tcp_listen_handle(self, port=23, connects=5, timeout=2):
"""This is running in its own thread."""
sock = socket.socket()
sock.settimeout(timeout)
sock.bind(('', port))
sock.listen(connects) # We accept more than one connection.
while self.keep_running_the_listening_thread:
connection = None
try:
connection, addr = sock.accept()
address, port = addr
if address != '127.0.0.1': # Only permit localhost.
connection.close()
continue
# makes a thread deals with that stuff. We only do listening.
connection.settimeout(timeout)
self.handle_tcp_connection_in_another_thread(connection, addr)
except socket.timeout:
pass
except OSError:
# Some other error.
if connection is not None:
connection.close()
sock.close()
c = ConsoleSocket()
def killsocket():
time.sleep(20)
c.stop()
Thread(target=killsocket).start()
This launches a listener thread for the connections set on port 23 (telnet), and you connect and it passes that connection off to another thread. And it starts a killsocket thread that disables the various threads and lets them die peacefully (for demonstration purposes). You cannot however connect localhost within this code, because you'd need input() to know what to send to the server, which recreates the problem.

PyZMQ one-way communication

I took a look at the various ZMQ messaging patterns and I'm not sure which one would do for my project. All I want to do is be able to connect to a server and send a command (the client never receive anything). On the server side, I want to be able to check if there is a message, if there is one, process it, else continue to do other stuff without blocking. That way, the server could continue to work even if there is no client connected.
#client.py
while(True):
select = raw_input()
if select == "1":
socket.send(msg1)
elif select == "2":
socket.send(msg2)
...
#server.py
while(True):
msg = socket.recv() #should not block
if msg == ...
#do stuff
#do other stuff
So which pattern should I use with ZMQ to do that? Example code would be appreciated.
First, since you want one-way communication with only one socket receiving messages, that generally means PUSH-PULL. Here is a version of the client:
import zmq
ctx = zmq.Context.instance()
s = ctx.socket(zmq.PUSH)
url = 'tcp://127.0.0.1:5555'
s.connect(url)
while True:
msg = raw_input("msg > ")
s.send(msg)
if msg == 'quit':
break
so a PUSH socket sends the messages we get from raw_input. It should be clear how to change that logic to generate the messages you want. A bit of bonus is that if you type 'quit', both the client and the server will quit.
There are a variety of ways to do the non-blocking server, depending on the complexity of your application. I'll show a few examples, from the most basic one to the most powerful / extensible.
All of these server examples assume this at the top, setting up the server's PULL socket:
import time
import zmq
ctx = zmq.Context.instance()
s = ctx.socket(zmq.PULL)
url = 'tcp://127.0.0.1:5555'
s.bind(url)
The first example is simple non-blocking recv, which raises a zmq.Again Exception if there are no messages ready to be received:
# server0.py
while True:
try:
msg = s.recv(zmq.NOBLOCK) # note NOBLOCK here
except zmq.Again:
# no message to recv, do other things
time.sleep(1)
else:
print("received %r" % msg)
if msg == 'quit':
break
But that pattern is pretty hard to extend beyond very simple cases. The second example uses a Poller, to check for events on the socket:
# server1.py
poller = zmq.Poller()
poller.register(s)
while True:
events = dict(poller.poll(0))
if s in events:
msg = s.recv()
print("received %r" % msg)
if msg == 'quit':
break
else:
# no message to recv, do other things
time.sleep(1)
In this toy example, this is very similar to the first. But, unlike the first, it is easy to extend to many sockets or events with further calls to poller.register, or passing a timeout other than zero to poller.poll.
The last example uses an eventloop, and actually registers a callback for when messages arrive. You can build very complex applications with this sort of pattern, and it is a fairly straightforward way to write code that only does work when there is work to be done.
# server2.py
from zmq.eventloop import ioloop
from zmq.eventloop.zmqstream import ZMQStream
def print_msg(msg):
print("received %r" % ' '.join(msg))
if msg[0] == 'quit':
ioloop.IOLoop.instance().stop()
# register the print_msg callback to be fired
# whenever there is a message on our socket
stream = ZMQStream(s)
stream.on_recv(print_msg)
# do other things in the meantime
tic = time.time()
def do_other_things():
print("%.3f" % (time.time() - tic))
pc = ioloop.PeriodicCallback(do_other_things, 1000)
pc.start()
# start the eventloop
ioloop.IOLoop.instance().start()
So that's a few basic ways to deal with zmq messages without blocking. You can grab these examples together as a gist.

Have a function time out if a certain condition is not fulfilled in time

The issue I have is that my chat client is supposed to recieve and print data from server when the server sends it, and then allow the client to reply.
This works fine, except that the entire process stops when the client is prompted to reply. So messages pile up until you type something, and after you do that, then it prints all the recieved messages.
Not sure how to fix this, so I decided why not have the client's time to type a reply timeout after 5 seconds, so that the replies can come through regardless. It's pretty flawed, because the input will reset itself, but it works better anyways.
Here's the function that needs to have a timeout:
# now for outgoing data
def outgoing():
global out_buffer
while 1:
user_input=input("your message: ")+"\n"
if user_input:
out_buffer += [user_input.encode()]
# for i in wlist:
s.send(out_buffer[0])
out_buffer = []
How should I go about using a timeout? I was thinking of using time.sleep, but that just pauses the entire operation.
I tried looking for documentation. But I didn't find anything that would help me make the program count up to a set limit, then continue.
Any idea's about how to solve this? (Doesn't need to use a timeout, just needs to stop the message pileup before the clients reply can be sent) (Thanks to all who helped me get this far)
For Ionut Hulub:
from socket import *
import threading
import json
import select
import signal # for trying to create timeout
print("client")
HOST = input("connect to: ")
PORT = int(input("on port: "))
# create the socket
s = socket(AF_INET, SOCK_STREAM)
s.connect((HOST, PORT))
print("connected to:", HOST)
#--------- need 2 threads for handling incoming and outgoing messages--
# 1: create out_buffer:
out_buffer = []
# for incoming data
def incoming():
rlist,wlist,xlist = select.select([s], out_buffer, [])
while 1:
for i in rlist:
data = i.recv(1024)
if data:
print("\nreceived:", data.decode())
# now for outgoing data
def outgoing():
global out_buffer
while 1:
user_input=input("your message: ")+"\n"
if user_input:
out_buffer += [user_input.encode()]
# for i in wlist:
s.send(out_buffer[0])
out_buffer = []
thread_in = threading.Thread(target=incoming, args=())
thread_out = threading.Thread(target=outgoing, args=())
thread_in.start() # this causes the thread to run
thread_out.start()
thread_in.join() # this waits until the thread has completed
thread_out.join()
We can use signals for the same. I think the below example will be useful for you.
import signal
def timeout(signum, frame):
raise Exception
#this is an infinite loop, never ending under normal circumstances
def main():
print 'Starting Main ',
while 1:
print 'in main ',
#SIGALRM is only usable on a unix platform
signal.signal(signal.SIGALRM, timeout)
#change 5 to however many seconds you need
signal.alarm(5)
try:
main()
except:
print "whoops"

Using Processes as Threads with Networking in Python

Basically, my idea was to write some sort of basic server where I could connect to my computer and then run a command remotely. This didn't seem to be much of a problem; but then I had the bright idea that the next step would logically be to add some sort of threading so I could spawn multiple connections.
I read that, because of the GIL, multiprocessing.Process would be the best to try to do this. I don't completely understand threading and it's hard to find good documentation on it; so I'm kind of just throwing stuff and trying to figure out how it works.
Well, it seems like I might be close to doing this right; but I have a feeling I'm just as likely to be no where near doing this correctly. My program now does allow multiple connections, which it didn't when I first started working with threading; but once a connection is established, and then another is established, the first connection is no longer able to send a command to the server. I would appreciate it if someone could give me any help, or point me in the right direction on what I need to learn and understand.
Here's my code:
class server:
def __init__(self):
self.s = socket.socket()
try:
self.s.bind(("",69696))
self.s.listen(1)
except socket.error,(value,message):
if self.s:
self.s.close()
def connection(self):
while True:
client , address = self.s.accept()
data = client.recv(5)
password = 'hello'
while 1:
if data == password:
subprocess.call('firefox')
client.close()
else:
client.send('wrong password')
data = client.recv(5)
p = Process(target=x.connection())
p.start()
x = server()
if __name__ == '__main':
main()
Well, this answer only applies if you're on a unix or unix-like operating system(windows does not have os.fork() which we use).
One of the most common approaches for doing these things on unix platforms is to fork a new process to handle the client connection while the master process continues to listen for requests.
Below is code for a simple echo server that can handle multiple simultaneous connections. You just need to modify handle_client_connection() to fit your needs
import socket
import os
class ForkingServer:
def serve_forever(self):
self.s = socket.socket()
try:
self.s.bind(("", 9000))
self.s.listen(1)
except socket.error, (value,message):
print "error:", message
if self.s:
self.s.close()
return
while True:
client,address = self.s.accept()
pid = os.fork()
# You should read the documentation for how fork() works if you don't
# know it already
# The short version is that at this point in the code, there are 2 processes
# completely identical to each other which are simulatenously executing
# The only difference is that the parent process gets the pid of the child
# returned from fork() and the child process gets a value of 0 returned
if pid == 0:
# only the newly spawned process will execute this
self.handle_client_connection(client, address)
break
# In the meantime the parent process will continue on to here
# thus it will go back to the beginning of the loop and accept a new connection
def handle_client_connection(self, client,address):
#simple echo server
print "Got a connection from:", address
while True:
data = client.recv(5)
if not data:
# client closed the connection
break
client.send(data)
print "Connection from", address, "closed"
server = ForkingServer()
server.serve_forever()

Thread synchronization in Python

I am currently working on a school project where the assignment, among other things, is to set up a threaded server/client system. Each client in the system is supposed to be assigned its own thread on the server when connecting to it. In addition i would like the server to run other threads, one concerning input from the command line and another concerning broadcasting messages to all clients. However, I can't get this to run as i want to. It seems like the threads are blocking each other. I would like my program to take inputs from the command line, at the "same time" as the server listens to connected clients, and so on.
I am new to python programming and multithreading, and allthough I think my idea is good, I'm not suprised my code doesn't work. Thing is I'm not exactly sure how I'm going to implement the message passing between the different threads. Nor am I sure exactly how to implement the resource lock commands properly. I'm going to post the code for my server file and my client file here, and I hope someone could help me with this. I think this actually should be two relative simple scripts. I have tried to comment on my code as good as possible to some extend.
import select
import socket
import sys
import threading
import client
class Server:
#initializing server socket
def __init__(self, event):
self.host = 'localhost'
self.port = 50000
self.backlog = 5
self.size = 1024
self.server = None
self.server_running = False
self.listen_threads = []
self.local_threads = []
self.clients = []
self.serverSocketLock = None
self.cmdLock = None
#here i have also declared some events for the command line input
#and the receive function respectively, not sure if correct
self.cmd_event = event
self.socket_event = event
def openSocket(self):
#binding server to port
try:
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind((self.host, self.port))
self.server.listen(5)
print "Listening to port " + str(self.port) + "..."
except socket.error, (value,message):
if self.server:
self.server.close()
print "Could not open socket: " + message
sys.exit(1)
def run(self):
self.openSocket()
#making Rlocks for the socket and for the command line input
self.serverSocketLock = threading.RLock()
self.cmdLock = threading.RLock()
#set blocking to non-blocking
self.server.setblocking(0)
#making two threads always running on the server,
#one for the command line input, and one for broadcasting (sending)
cmd_thread = threading.Thread(target=self.server_cmd)
broadcast_thread = threading.Thread(target=self.broadcast,args=[self.clients])
cmd_thread.daemon = True
broadcast_thread.daemon = True
#append the threads to thread list
self.local_threads.append(cmd_thread)
self.local_threads.append(broadcast_thread)
cmd_thread.start()
broadcast_thread.start()
self.server_running = True
while self.server_running:
#connecting to "knocking" clients
try:
c = client.Client(self.server.accept())
self.clients.append(c)
print "Client " + str(c.address) + " connected"
#making a thread for each clientn and appending it to client list
listen_thread = threading.Thread(target=self.listenToClient,args=[c])
self.listen_threads.append(listen_thread)
listen_thread.daemon = True
listen_thread.start()
#setting event "client has connected"
self.socket_event.set()
except socket.error, (value, message):
continue
#close threads
self.server.close()
print "Closing client threads"
for c in self.listen_threads:
c.join()
def listenToClient(self, c):
while self.server_running:
#the idea here is to wait until the thread gets the message "client
#has connected"
self.socket_event.wait()
#then clear the event immidiately...
self.socket_event.clear()
#and aquire the socket resource
self.serverSocketLock.acquire()
#the below is the receive thingy
try:
recvd_data = c.client.recv(self.size)
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
self.socket_event.clear()
self.serverSocketLock.release()
return
print recvd_data
#I put these here to avoid locking the resource if no message
#has been received
self.socket_event.clear()
self.serverSocketLock.release()
except socket.error, (value, message):
continue
def server_cmd(self):
#this is a simple command line utility
while self.server_running:
#got to have a smart way to make this work
self.cmd_event.wait()
self.cmd_event.clear()
self.cmdLock.acquire()
cmd = sys.stdin.readline()
if cmd == "":
continue
if cmd == "close\n":
print "Server shutting down..."
self.server_running = False
self.cmdLock.release()
def broadcast(self, clients):
while self.server_running:
#this function will broadcast a message received from one
#client, to all other clients, but i guess any thread
#aspects applied to the above, will work here also
try:
send_data = sys.stdin.readline()
if send_data == "":
continue
else:
for c in clients:
c.client.send(send_data)
self.serverSocketLock.release()
self.cmdLock.release()
except socket.error, (value, message):
continue
if __name__ == "__main__":
e = threading.Event()
s = Server(e)
s.run()
And then the client file
import select
import socket
import sys
import server
import threading
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#making a lock for the socket resource
self.clientSocketLock = threading.Lock()
self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure wat daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append(listen)
self.running_threads.append(speak)
listen.start()
speak.start()
#this while-loop is just for avoiding the script terminating
while self.client_running:
dummy = 1
#closing the threads if the client goes down
print "Client operating on its own"
self.client.close()
#close threads
for t in self.running_threads:
t.join()
return
#defining "listen"-function
def listenToServer(self):
while self.client_running:
#here i acquire the socket to this function, but i realize I also
#should have a message passing wait()-function or something
#somewhere
self.clientSocketLock.acquire()
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
#releasing the socket resource
self.clientSocketLock.release()
#defining "speak"-function, doing much the same as for the above function
def speakToServer(self):
while self.client_running:
self.clientSocketLock.acquire()
try:
send_data = sys.stdin.readline()
if send_data == "close\n":
print "Disconnecting..."
self.client_running = False
else:
self.client.send(send_data)
except socket.error, (value,message):
continue
self.clientSocketLock.release()
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
I realize this is quite a few code lines for you to read through, but as I said, I think the concept and the script in it self should be quite simple to understand. It would be very much appriciated if someone could help me synchronize my threads in a proper way =)
Thanks in advance
---Edit---
OK. So I now have simplified my code to just containing send and receive functions in both the server and the client modules. The clients connecting to the server gets their own threads, and the send and receive functions in both modules operetes in their own separate threads. This works like a charm, with the broadcast function in the server module echoing strings it gets from one client to all clients. So far so good!
The next thing i want my script to do, is taking specific commands, i.e. "close", in the client module to shut down the client, and join all running threads in the thread list. Im using an event flag to notify the listenToServer and the main thread that the speakToServer thread has read the input "close". It seems like the main thread jumps out of its while loop and starts the for loop that is supposed to join the other threads. But here it hangs. It seems like the while loop in the listenToServer thread never stops even though server_running should be set to False when the event flag is set.
I'm posting only the client module here, because I guess an answer to get these two threads to synchronize will relate to synchronizing more threads in both the client and the server module also.
import select
import socket
import sys
import server_bygg0203
import threading
from time import sleep
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
self.disconnected = threading.Event()
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure what daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append((listen,"listen"))
self.running_threads.append((speak, "speak"))
listen.start()
speak.start()
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
#closing the threads if the client goes down
print "Client operating on its own"
self.client.shutdown(1)
self.client.close()
#close threads
#the script hangs at the for-loop below, and
#refuses to close the listen-thread (and possibly
#also the speak thread, but it never gets that far)
for t in self.running_threads:
print "Waiting for " + t[1] + " to close..."
t[0].join()
self.disconnected.clear()
return
#defining "speak"-function
def speakToServer(self):
#sends strings to server
while self.client_running:
try:
send_data = sys.stdin.readline()
self.client.send(send_data)
#I want the "close" command
#to set an event flag, which is being read by all other threads,
#and, at the same time set the while statement to false
if send_data == "close\n":
print "Disconnecting..."
self.disconnected.set()
self.client_running = False
except socket.error, (value,message):
continue
return
#defining "listen"-function
def listenToServer(self):
#receives strings from server
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
return
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
Later on, when I get this server/client system up and running, I will use this system on some elevator models we have here on the lab, with each client receiving floor orders or "up" and "down" calls. The server will be running an distribution algorithm and updating the elevator queues on the clients that are most appropriate for the requested order. I realize it's a long way to go, but I guess one should just take one step at the time =)
Hope someone has the time to look into this. Thanks in advance.
The biggest problem I see with this code is that you have far too much going on right away to easily debug your problem. Threading can get extremely complicated because of how non-linear the logic becomes. Especially when you have to worry about synchronizing with locks.
The reason you are seeing clients blocking on each other is because of the way you are using your serverSocketLock in your listenToClient() loop in the server. To be honest this isn't exactly your problem right now with your code, but it became the problem when I started to debug it and turned the sockets into blocking sockets. If you are putting each connection into its own thread and reading from them, then there is no reason to use a global server lock here. They can all read from their own sockets at the same time, which is the purpose of the thread.
Here is my recommendation to you:
Get rid of all the locks and extra threads that you don't need, and start from the beginning
Have the clients connect as you do, and put them in their thread as you do. And simply have them send data every second. Verify that you can get more than one client connecting and sending, and that your server is looping and receiving. Once you have this part working, you can move on to the next part.
Right now you have your sockets set to non-blocking. This is causing them all to spin really fast over their loops when data is not ready. Since you are threading, you should set them to block. Then the reader threads will simply sit and wait for data and respond immediately.
Locks are used when threads will be accessing shared resources. You obviously need to for any time a thread will try and modify a server attribute like a list or a value. But not when they are working on their own private sockets.
The event you are using to trigger your readers doesn't seem necessary here. You have received the client, and you start the thread afterwards. So it is ready to go.
In a nutshell...simplify and test one bit at a time. When its working, add more. There are too many threads and locks right now.
Here is a simplified example of your listenToClient method:
def listenToClient(self, c):
while self.server_running:
try:
recvd_data = c.client.recv(self.size)
print "received:", c, recvd_data
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
return
print recvd_data
except socket.error, (value, message):
if value == 35:
continue
else:
print "Error:", value, message
Backup your work, then toss it - partially.
You need to implement your program in pieces, and test each piece as you go. First, tackle the input part of your program. Don't worry about how to broadcast the input you received. Instead worry that you are able to successfully and repeatedly receive input over your socket. So far - so good.
Now, I assume you would like to react to this input by broadcasting to the other attached clients. Well too bad, you can't do that yet! Because, I left one minor detail out of the paragraph above. You have to design a PROTOCOL.
What is a protocol? It's a set of rules for communication. How does your server know when the client had finished sending it's data? Is it terminated by some special character? Or perhaps you encode the size of the message to be sent as the first byte or two of the message.
This is turning out to be a lot of work, isn't it? :-)
What's a simple protocol. A line-oriented protocol is simple. Read 1 character at a time until you get to the end of record terminator - '\n'. So, clients would send records like this to your server --
HELO\n
MSG DAVE Where Are Your Kids?\n
So, assuming you have this simple protocol designed, implement it. For now, DON'T WORRY ABOUT THE MULTITHREADING STUFF! Just worry about making it work.
Your current protocol is to read 1024 bytes. Which may not be bad, just make sure you send 1024 byte messages from the client.
Once you have the protocol stuff setup, move on to reacting to the input. But for now you need something that will read input. Once that is done, we can worry about doing something with it.
jdi is right, you have too much program to work with. Pieces are easier to fix.

Categories