We Keep Coding

Python ZMQ responder not receiving message

I am trying a simple zmq script but somehow the responder is not getting the first message.
The Responder looks like this:
def main():
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.connect("tcp://localhost:{}".format(5560))
print("connected ...")
while True:
# Wait for next request from client
message = socket.recv_pyobj()
#print (message)
print(message)
if __name__ == '__main__':
main()
I am sending the request with the following code from another process:
def main():
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:{}".format(5560))
print("sending object")
socket.send_pyobj("ok")
print("done")
if __name__ == '__main__':
main()
Does anybody have an idea why it does not arrive?

You must add .bind() an IP address in your REP code snippet instead of .connect(). In REP/REQ pattern there are the request and response factor, so you can throw feedback in the responder code. Thus your code will be as follows:
Responder:
import zmq
def main():
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://127.0.0.1:5560")
while True:
message = socket.recv_pyobj()
print(message)
socket.send_string('Your message received.')
if __name__ == '__main__':
main()
Request code snippet:
import zmq
def main():
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5560")
print("sending object")
socket.send_pyobj("ok")
message = socket.recv()
print(message)
if __name__ == '__main__':
main()
Out (req):
sending object
b'Your message received.'
Out (rep):
ok
[NOTE]:
If you want to send a simple string without any response, using the PUSH/PULL or SUB/PUB pattern is more suitable instead of using REP/REQ and also you could use the socket.send_string('ok') instead of socket.send_pyobj('ok') in request section and socket.recv() instead of socket.recv_pyobj() in responder.
Note that in .bind() case, you shouldn't use the localhost string Relevant Post.

Q : Does anybody have an idea why it does not arrive?
Oh sure I have.
There are a few principal points that govern how the ZeroMQ-based distributed-computing systems may and do work.
If you are new to using ZeroMQ or other its derivatives ( nanomsg et al ), be sure not to miss Pieter Hintjen's must read book "Code Connected. Volume 1".
There are two places, where missed (or principally undeliverable) messages may come:
Not yet ready process, which is assumed to receive a message, being the first
Not available resource (port) for a successful .bind(), being the second
Harsh network transport conditions related problems are not the case for localhost-only ( vmci://-virtualised of internal port-abstracted network ) experimentations
Curable:
def main():
context = zmq.Context()
socket = context.socket( zmq.REQ )
socket.setsockopt( zmq.LINGER, 0 ) # ALWAYS PREVENT BLOCKING
socket.setsockopt( zmq.IMMEDIATE, 1 ) # BLOCK UNTIL CONN-READY
#ocket.setsockpt( zmq.ZMQ_HANDSHAKE_IVL, ... ) # IF TWEAKING NETWORK-WIDE
# OR:
# a stone-age wait for the other part get started in a one-computer demo:
# sleep( 20 )
# :o)
socket.connect( "tcp://localhost:{}".format( 5560 ) )
print( "Will try to dispatch an object to Context() instance" )
socket.send_pyobj( "ok" )
print( ".send() method has returned from a blocking-call mode" )
...
#--------------------------------------------------------# ALWAYS
socket.close() # ALWAYS RELEASE RESOURCES
context.term() # ALWAYS RELEASE RESOURCES
# # ALWAYS (not all versions
# # have "friendly"
# # defeaults to rely
# # on others,
# # so be explicit)
#--------------------------------------------------------# ALWAYS
One side, obviously not necessarily the REP, yet here it fits better, due to while, must .bind(), the other(s) just .connect() to a known connection target:
def main():
context = zmq.Context()
socket = context.socket( zmq.REP )
socket.setsockopt( zmq.LINGER, 0 ) # ALWAYS PREVENT BLOCKING
socket.setsockopt( zmq.IMMEDIATE, 1 ) # BLOCK UNTIL CONN-READY
#ocket.setsockpt( zmq.ZMQ_HANDSHAKE_IVL, ... ) # IF TWEAKING NETWORK-WIDE
socket.bind( "tcp://localhost:{}".format( 5560 ) )
print( ".bind() ...")
try:
while True: # Wait for next request from client:
message = socket.recv_pyobj()
print( message )
except:
print( "EXC'd" )
finally:
#----------------------------------------------------# ALWAYS
socket.unbind( "tcp://localhost:{}".format( 5560 ) ) # ALWAYS RELEASE PORT
socket.close() # ALWAYS RELEASE RESOURCES
context.term() # ALWAYS RELEASE RESOURCES
# # ALWAYS (not all versions
# # have "friendly"
# # defeaults to rely
# # on others,
# # so be explicit)
#----------------------------------------------------# ALWAYS
Last but not least, this will start to work, yet it will hang in an infinite waiting, due to a perhaps missed principle of the REQ/REP-behaviour archetype. One must ASK ( REQ.send()-s ), the other one, who is to REPLY has to listen-up the question REP.recv(), however it also has to ANSWER... REP.send("something") before we may move forwards in two-step-tango for two and the ASKER has also to get the answer listened to REQ.recv().
Then and only then the ASKER can send another question by another REQ.send().
So, both your sending REQ-parts, yet mainly the receiving REP-part, inside infinite while True:{...} loop has to get revised, in order to receive any second and further messages, even for cases when REQ-s die after a single shot and never listen to any answer from REP.

ZMQError: Address already in use while using sockets in loops

I am trying to "simulate message passing between 6 nodes in a distributed environment" using Zero MQ in Python, in particular with the classic client/server architecture with REQ and REP. My idea is, while using a TCP/IP connection between these nodes, in the first iteration node-1 must be the server and the clients are the other nodes. In the next one, node-2 will be server and the rest (including node-1) should be clients and so on. At every iteration, server tells that it has established itself and clients send requests to the server to which an acknowledgement is sent back. Once the ACK has been received, the clients send their "MESSAGE" to the server (which is of course viewed as output) and we move to the next iteration.
The problem now is that I am facing the well known ZMQError: Address already in use
I'm not sure if it's due to the socket binding. I have added a socket.close() and context.term() into both client and server functions, but in vain.
And when I somehow try to run the code the VM goes into deadlock and I'm unable to recover unless I perform a hard reboot. Here is a snippet of my code -
#staticmethod
def server(node_id):
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:%s" % port_s)
print "Running server node %s on port: %s and value of server = __temp__" % (node_id, port_s)
message = socket.recv()
print "Received request : %s from c_node %s and value (temp): __value__" % (message, c_node)
socket.send("Acknowledged - from %s" % port_s)
time.sleep(1)
socket.close()
context.term()
#staticmethod
def client(c_node):
context = zmq.Context()
# print "Server node __num__ with port %s" % port_s
socket = context.socket(zmq.REQ)
#for port in ports:
socket.connect ("tcp://localhost:%s" % port_c)
#for request in range(20):
print "c_node %s Sending request to server node __num__" % c_node
socket.send ("Hello")
message = socket.recv()
print "Received ack from server %s and message %s" % (node_id, message)
time.sleep (1)
socket.close()
context.term()
def node(self, node_id):
#global node_id
# global key
# ser_p = Process(target=self.server, args=(node_id,))
print 'Memory content of node %d\n' % node_id
for key in nodes_memory[node_id]:
print 'Neighbor={%s}, Temp={%s}\n' % (key, nodes_memory[node_id][key])
#return key
global c_node
#key1 = key
# cli_p = Process(target=self.client, args=(c_node,))
with open("Book5.csv","r+b") as input:
has_header = csv.Sniffer().has_header(input.read(1024))
input.seek(0) # rewind
incsv = csv.reader(input)
if has_header:
next(incsv) # skip header
csv_dict = csv.DictReader(input, skipinitialspace=True, delimiter=",")
node_id = 0
for row in csv_dict:
for i in row:
#print(row[i])
if type(row[i]) is str:
g.add_edge(node_id, int(i), conn_prob=(float(row[i])))
max_wg_ngs = sorted(g[node_id].items(), key=lambda e: e[1]["conn_prob"], reverse=True)[:2]
#maxim = max_wg_ngs.values.tolist()
#sarr = [str(a) for a in max_wg_ngs]
print "\nNeighbours of Node %d are:" % node_id
#print(max_wg_ngs)
ser_p = multiprocessing.Process(target=self.server, args=(node_id,))
ser_p.start()
for c_node, data in max_wg_ngs:
for key in nodes_memory[node_id]: #print ''.join(str(item))[1:-1]
#if type(key1) == node_id:
cli_p = multiprocessing.Process(target=self.client, args=(c_node,))
cli_p.start()
print('Node {a} with Connection Rate = {w}'.format(a=c_node, w=data['conn_prob']))
print('Temperature of Node {a} = {b}'.format(a=c_node, b=nodes_memory[node_id][key]))
node_id += 1
pos=nx.spring_layout(g, scale=100.)
nx.draw_networkx_nodes(g, pos)
nx.draw_networkx_edges(g,pos)
nx.draw_networkx_labels(g,pos)
#plt.axis('off')
#plt.show()
The "message" is "temperature" (file not shown in the code snippet, but not needed at the moment) and for reference the values of Book5.csv are -
0,1,2,3,4,5
0,0.257905291,0.775104118,0.239086843,0.002313744,0.416936603
0.346100279,0,0.438892758,0.598885794,0.002263231,0.406685237
0.753358102,0.222349243,0,0.407830809,0.001714776,0.507573592
0.185342928,0.571302688,0.51784403,0,0.003231018,0.295197533
0,0,0,0,0,0
0.478164621,0.418192795,0.646810223,0.410746629,0.002414973,0
ser_p and cli_p are objects for the server and client functions which are called in the node function, i.e ser_p is called in the loop for row in csv_dict and cli_p is called even further in for c_node, data in max_wg_ngs. I'm using Networkx Python library here as well (Only to find 2 nearest neighbours out of the clients using the connection probability values from Book5.csv).
Does anyone know where I might be going wrong? Why is it that it shows address is already in use even though the socket is closed at every iteration?
Thanks a lot in advance :) (Using Ubuntu 14.04 32-bit VM)

StackOverflow has agreed to use an MCVE-based question asking:
Would you mind to reach that level and complete the missing part of the MCVE -- neither port_c, nor port_s are lexically correct ( never defined, elsewhere ).
If the code presented here works with a file, always kindly prepare such Minimum-version of the file, that will ensure the MCVE-code work as your expect it to work with the file. Statements like: "file not shown in the code snippet, but not needed at the moment" are not compatible with StackOverflow MCVE-rules.
Next, analyse the logic.
If multiple processes try to .bind() onto a same port# set via port_s, they simply will ( and have to ) collide and fall into an exception with ZMQError: Address already in use. First reboot the O/S, next pre-scan the already used IP:port#-s, next setup the non-colliding server-side .bind() ( there could be still a hanging .context() with non-terminated .socket() instance(s) ( typically from manual prototyping or from unhandled exceptions ), that keep the IP:port# without making it free. So the reboot + port scan is a way ahead.
Use any deterministic, principally non-colliding server-2-<transport-class://address:port> mapping ( .bind()-s using wildcards, locking on all IP-adresses, is a bit dangerous habit ) & your code will work smooth.
Always use <socket>.setsockopt( zmq.LINGER, 0 ) so as to prevent infinite deadlocks.
Always use try: {...} except: {...} finally: {...} formal expressions, so as to avoid any unhandled exceptions to orphan any .context() instance outside of your control, perhaps without a graceful .term()-ination and release ( even if new API tells you, that this is not necessary - it is professional to explicitly handle these situations and remain in control, so no exceptions, no excuse ).

Use of pyzmq's logging handler in python

I want to introduce a zmq based logging into a Python program. As I was facing ZMQError: Address in use errors I decided to boil it down to a simple proof of concept. I was able to run the boiled down version, but not to receive any log entries. This is the code I used:
Log Publisher:
import time
import logging
from zmq.log import handlers as zmqHandler
logger = logging.getLogger('myapp')
logger.setLevel(logging.ERROR)
zmqH=zmqHandler.PUBHandler('tcp://127.0.0.1:12344')
logger.addHandler(zmqH)
for i in range(50):
logger.error('error test...')
print "Send error #%s" % (str(i))
time.sleep(1)
Result
Send error #0
Send error #1
Send error #2
Send error #3
Send error #4
...
Log Subscriber:
import time
import zmq
def sub_client():
port = "12344"
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://127.0.0.1:%s" % port)
# Generate 30 entries
for i in range (30):
print "Listening to publishers..."
message = socket.recv()
print "Received error #%s: %s" % (str(i), message)
time.sleep(1)
sub_client()
Result
Listening to publishers...
So the subscriber is locked at the call of socket.recv(). I started publisher and subscriber in different consoles. Both processes appear when I use netstat:
C:\>netstat -a -n -o | findstr 12344
TCP 127.0.0.1:12344 0.0.0.0:0 LISTEN 1336
TCP 127.0.0.1:12344 127.0.0.1:51937 ESTABLISHED 1336
TCP 127.0.0.1:51937 127.0.0.1:12344 ESTABLISHED 8624
I fail to see my mistake here, any ideas?
In addition to the problem at hand, how do I use this zmq listener in general.
Do I have to create one instance of the PUBHandler per process and then add it to all instances of logger (logging.getLogger('myapp') creates a own logger instance, right?) or do I have to create an own PUBHandler for all of the different classes I use? As the PUBHandlerclass has a createLock() I assume that it is not thread save...
For completness I want to mention the doc of the PUBHandler class
I am using a python(x,y) distribution at Win7 with python 2.7.10 and pyzmq 14.7.0-14
[update]
I ruled out the the windows firewall as the source of the missing packages

The problem is on the subscriber side. Initially a subscriber filters out all messages, until a filter is set. Use socket.setsockopt(opt, value) function to archive this.
The pyZMQ description is not very clear about the use of this function:
getsockopt(opt) get default socket options for new sockets created by
this Context
But the documentation of the zmq_setsockopt function is quite clear (see here):
int zmq_setsockopt (void *socket, int option_name, const void *option_value, size_t option_len)
...
ZMQ_SUBSCRIBE: Establish message filter The ZMQ_SUBSCRIBE option shall establish a new message filter on a ZMQ_SUB socket.
Newly created ZMQ_SUB sockets shall filter out all incoming
messages, therefore you should call this option to establish an
initial message filter.
So the solution is to set a filter with socket.setsockopt(zmq.SUBSCRIBE,filter), where filter is the string you want to filter for. Use filter='' to show all messages. A filter like filter='ERROR' will only display the error messages and suppress all other types like WARNING,INFO or DEBUG.
With this the sub_client() function looks like this:
import time
import zmq
def sub_client():
port = "12344"
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://127.0.0.1:%s" % port)
socket.setsockopt(zmq.SUBSCRIBE,'')
# Process 30 updates
print "Listening to publishers..."
for i in range (30):
print "Listening to publishers..."
message = socket.recv()
print "Received error #%s: %s" % (str(i), message)
time.sleep(1)
sub_client()

I know it is an older post, but if someone lands here this is what the subscriber looks like
def sub_client():
port = "12345"
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.connect("tcp://localhost:%s" % port)
socket.subscribe("")
# Process 30 updates
for i in range (30):
print("Listening to publishers...")
message = socket.recv()
print("Received error #%s: %s",str(i), message)
time.sleep(1)
sub_client()
And Publisher looks like
import zmq
import logging
import time
from zmq.log.handlers import PUBHandler
port = "12345"
context = zmq.Context()
pub = context.socket(zmq.PUB)
pub.bind("tcp://*:%s" % port)
handler = PUBHandler(pub)
logger = logging.getLogger()
logger.setLevel(logging.ERROR)
logger.addHandler(handler)
for i in range(50):
logger.error('error test...')
print("publish error",str(i))
time.sleep(1)

I guess you missed to set PUB socket in server.
something like this should do,
publisher = context.socket(zmq.PUB)
publisher.bind('tcp://127.0.0.1:12344')
zmqh = PUBHandler(publisher)
logger = logging.getLogger('myapp')
logger.setLevel(logging.ERROR)
logger.addHandler(zmqh)
I hope this helps.

PyZMQ one-way communication

I took a look at the various ZMQ messaging patterns and I'm not sure which one would do for my project. All I want to do is be able to connect to a server and send a command (the client never receive anything). On the server side, I want to be able to check if there is a message, if there is one, process it, else continue to do other stuff without blocking. That way, the server could continue to work even if there is no client connected.
#client.py
while(True):
select = raw_input()
if select == "1":
socket.send(msg1)
elif select == "2":
socket.send(msg2)
...
#server.py
while(True):
msg = socket.recv() #should not block
if msg == ...
#do stuff
#do other stuff
So which pattern should I use with ZMQ to do that? Example code would be appreciated.

First, since you want one-way communication with only one socket receiving messages, that generally means PUSH-PULL. Here is a version of the client:
import zmq
ctx = zmq.Context.instance()
s = ctx.socket(zmq.PUSH)
url = 'tcp://127.0.0.1:5555'
s.connect(url)
while True:
msg = raw_input("msg > ")
s.send(msg)
if msg == 'quit':
break
so a PUSH socket sends the messages we get from raw_input. It should be clear how to change that logic to generate the messages you want. A bit of bonus is that if you type 'quit', both the client and the server will quit.
There are a variety of ways to do the non-blocking server, depending on the complexity of your application. I'll show a few examples, from the most basic one to the most powerful / extensible.
All of these server examples assume this at the top, setting up the server's PULL socket:
import time
import zmq
ctx = zmq.Context.instance()
s = ctx.socket(zmq.PULL)
url = 'tcp://127.0.0.1:5555'
s.bind(url)
The first example is simple non-blocking recv, which raises a zmq.Again Exception if there are no messages ready to be received:
# server0.py
while True:
try:
msg = s.recv(zmq.NOBLOCK) # note NOBLOCK here
except zmq.Again:
# no message to recv, do other things
time.sleep(1)
else:
print("received %r" % msg)
if msg == 'quit':
break
But that pattern is pretty hard to extend beyond very simple cases. The second example uses a Poller, to check for events on the socket:
# server1.py
poller = zmq.Poller()
poller.register(s)
while True:
events = dict(poller.poll(0))
if s in events:
msg = s.recv()
print("received %r" % msg)
if msg == 'quit':
break
else:
# no message to recv, do other things
time.sleep(1)
In this toy example, this is very similar to the first. But, unlike the first, it is easy to extend to many sockets or events with further calls to poller.register, or passing a timeout other than zero to poller.poll.
The last example uses an eventloop, and actually registers a callback for when messages arrive. You can build very complex applications with this sort of pattern, and it is a fairly straightforward way to write code that only does work when there is work to be done.
# server2.py
from zmq.eventloop import ioloop
from zmq.eventloop.zmqstream import ZMQStream
def print_msg(msg):
print("received %r" % ' '.join(msg))
if msg[0] == 'quit':
ioloop.IOLoop.instance().stop()
# register the print_msg callback to be fired
# whenever there is a message on our socket
stream = ZMQStream(s)
stream.on_recv(print_msg)
# do other things in the meantime
tic = time.time()
def do_other_things():
print("%.3f" % (time.time() - tic))
pc = ioloop.PeriodicCallback(do_other_things, 1000)
pc.start()
# start the eventloop
ioloop.IOLoop.instance().start()
So that's a few basic ways to deal with zmq messages without blocking. You can grab these examples together as a gist.

Thread synchronization in Python

I am currently working on a school project where the assignment, among other things, is to set up a threaded server/client system. Each client in the system is supposed to be assigned its own thread on the server when connecting to it. In addition i would like the server to run other threads, one concerning input from the command line and another concerning broadcasting messages to all clients. However, I can't get this to run as i want to. It seems like the threads are blocking each other. I would like my program to take inputs from the command line, at the "same time" as the server listens to connected clients, and so on.
I am new to python programming and multithreading, and allthough I think my idea is good, I'm not suprised my code doesn't work. Thing is I'm not exactly sure how I'm going to implement the message passing between the different threads. Nor am I sure exactly how to implement the resource lock commands properly. I'm going to post the code for my server file and my client file here, and I hope someone could help me with this. I think this actually should be two relative simple scripts. I have tried to comment on my code as good as possible to some extend.
import select
import socket
import sys
import threading
import client
class Server:
#initializing server socket
def __init__(self, event):
self.host = 'localhost'
self.port = 50000
self.backlog = 5
self.size = 1024
self.server = None
self.server_running = False
self.listen_threads = []
self.local_threads = []
self.clients = []
self.serverSocketLock = None
self.cmdLock = None
#here i have also declared some events for the command line input
#and the receive function respectively, not sure if correct
self.cmd_event = event
self.socket_event = event
def openSocket(self):
#binding server to port
try:
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind((self.host, self.port))
self.server.listen(5)
print "Listening to port " + str(self.port) + "..."
except socket.error, (value,message):
if self.server:
self.server.close()
print "Could not open socket: " + message
sys.exit(1)
def run(self):
self.openSocket()
#making Rlocks for the socket and for the command line input
self.serverSocketLock = threading.RLock()
self.cmdLock = threading.RLock()
#set blocking to non-blocking
self.server.setblocking(0)
#making two threads always running on the server,
#one for the command line input, and one for broadcasting (sending)
cmd_thread = threading.Thread(target=self.server_cmd)
broadcast_thread = threading.Thread(target=self.broadcast,args=[self.clients])
cmd_thread.daemon = True
broadcast_thread.daemon = True
#append the threads to thread list
self.local_threads.append(cmd_thread)
self.local_threads.append(broadcast_thread)
cmd_thread.start()
broadcast_thread.start()
self.server_running = True
while self.server_running:
#connecting to "knocking" clients
try:
c = client.Client(self.server.accept())
self.clients.append(c)
print "Client " + str(c.address) + " connected"
#making a thread for each clientn and appending it to client list
listen_thread = threading.Thread(target=self.listenToClient,args=[c])
self.listen_threads.append(listen_thread)
listen_thread.daemon = True
listen_thread.start()
#setting event "client has connected"
self.socket_event.set()
except socket.error, (value, message):
continue
#close threads
self.server.close()
print "Closing client threads"
for c in self.listen_threads:
c.join()
def listenToClient(self, c):
while self.server_running:
#the idea here is to wait until the thread gets the message "client
#has connected"
self.socket_event.wait()
#then clear the event immidiately...
self.socket_event.clear()
#and aquire the socket resource
self.serverSocketLock.acquire()
#the below is the receive thingy
try:
recvd_data = c.client.recv(self.size)
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
self.socket_event.clear()
self.serverSocketLock.release()
return
print recvd_data
#I put these here to avoid locking the resource if no message
#has been received
self.socket_event.clear()
self.serverSocketLock.release()
except socket.error, (value, message):
continue
def server_cmd(self):
#this is a simple command line utility
while self.server_running:
#got to have a smart way to make this work
self.cmd_event.wait()
self.cmd_event.clear()
self.cmdLock.acquire()
cmd = sys.stdin.readline()
if cmd == "":
continue
if cmd == "close\n":
print "Server shutting down..."
self.server_running = False
self.cmdLock.release()
def broadcast(self, clients):
while self.server_running:
#this function will broadcast a message received from one
#client, to all other clients, but i guess any thread
#aspects applied to the above, will work here also
try:
send_data = sys.stdin.readline()
if send_data == "":
continue
else:
for c in clients:
c.client.send(send_data)
self.serverSocketLock.release()
self.cmdLock.release()
except socket.error, (value, message):
continue
if __name__ == "__main__":
e = threading.Event()
s = Server(e)
s.run()
And then the client file
import select
import socket
import sys
import server
import threading
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#making a lock for the socket resource
self.clientSocketLock = threading.Lock()
self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure wat daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append(listen)
self.running_threads.append(speak)
listen.start()
speak.start()
#this while-loop is just for avoiding the script terminating
while self.client_running:
dummy = 1
#closing the threads if the client goes down
print "Client operating on its own"
self.client.close()
#close threads
for t in self.running_threads:
t.join()
return
#defining "listen"-function
def listenToServer(self):
while self.client_running:
#here i acquire the socket to this function, but i realize I also
#should have a message passing wait()-function or something
#somewhere
self.clientSocketLock.acquire()
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
#releasing the socket resource
self.clientSocketLock.release()
#defining "speak"-function, doing much the same as for the above function
def speakToServer(self):
while self.client_running:
self.clientSocketLock.acquire()
try:
send_data = sys.stdin.readline()
if send_data == "close\n":
print "Disconnecting..."
self.client_running = False
else:
self.client.send(send_data)
except socket.error, (value,message):
continue
self.clientSocketLock.release()
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
I realize this is quite a few code lines for you to read through, but as I said, I think the concept and the script in it self should be quite simple to understand. It would be very much appriciated if someone could help me synchronize my threads in a proper way =)
Thanks in advance
---Edit---
OK. So I now have simplified my code to just containing send and receive functions in both the server and the client modules. The clients connecting to the server gets their own threads, and the send and receive functions in both modules operetes in their own separate threads. This works like a charm, with the broadcast function in the server module echoing strings it gets from one client to all clients. So far so good!
The next thing i want my script to do, is taking specific commands, i.e. "close", in the client module to shut down the client, and join all running threads in the thread list. Im using an event flag to notify the listenToServer and the main thread that the speakToServer thread has read the input "close". It seems like the main thread jumps out of its while loop and starts the for loop that is supposed to join the other threads. But here it hangs. It seems like the while loop in the listenToServer thread never stops even though server_running should be set to False when the event flag is set.
I'm posting only the client module here, because I guess an answer to get these two threads to synchronize will relate to synchronizing more threads in both the client and the server module also.
import select
import socket
import sys
import server_bygg0203
import threading
from time import sleep
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
self.disconnected = threading.Event()
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure what daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append((listen,"listen"))
self.running_threads.append((speak, "speak"))
listen.start()
speak.start()
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
#closing the threads if the client goes down
print "Client operating on its own"
self.client.shutdown(1)
self.client.close()
#close threads
#the script hangs at the for-loop below, and
#refuses to close the listen-thread (and possibly
#also the speak thread, but it never gets that far)
for t in self.running_threads:
print "Waiting for " + t[1] + " to close..."
t[0].join()
self.disconnected.clear()
return
#defining "speak"-function
def speakToServer(self):
#sends strings to server
while self.client_running:
try:
send_data = sys.stdin.readline()
self.client.send(send_data)
#I want the "close" command
#to set an event flag, which is being read by all other threads,
#and, at the same time set the while statement to false
if send_data == "close\n":
print "Disconnecting..."
self.disconnected.set()
self.client_running = False
except socket.error, (value,message):
continue
return
#defining "listen"-function
def listenToServer(self):
#receives strings from server
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
return
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
Later on, when I get this server/client system up and running, I will use this system on some elevator models we have here on the lab, with each client receiving floor orders or "up" and "down" calls. The server will be running an distribution algorithm and updating the elevator queues on the clients that are most appropriate for the requested order. I realize it's a long way to go, but I guess one should just take one step at the time =)
Hope someone has the time to look into this. Thanks in advance.

The biggest problem I see with this code is that you have far too much going on right away to easily debug your problem. Threading can get extremely complicated because of how non-linear the logic becomes. Especially when you have to worry about synchronizing with locks.
The reason you are seeing clients blocking on each other is because of the way you are using your serverSocketLock in your listenToClient() loop in the server. To be honest this isn't exactly your problem right now with your code, but it became the problem when I started to debug it and turned the sockets into blocking sockets. If you are putting each connection into its own thread and reading from them, then there is no reason to use a global server lock here. They can all read from their own sockets at the same time, which is the purpose of the thread.
Here is my recommendation to you:
Get rid of all the locks and extra threads that you don't need, and start from the beginning
Have the clients connect as you do, and put them in their thread as you do. And simply have them send data every second. Verify that you can get more than one client connecting and sending, and that your server is looping and receiving. Once you have this part working, you can move on to the next part.
Right now you have your sockets set to non-blocking. This is causing them all to spin really fast over their loops when data is not ready. Since you are threading, you should set them to block. Then the reader threads will simply sit and wait for data and respond immediately.
Locks are used when threads will be accessing shared resources. You obviously need to for any time a thread will try and modify a server attribute like a list or a value. But not when they are working on their own private sockets.
The event you are using to trigger your readers doesn't seem necessary here. You have received the client, and you start the thread afterwards. So it is ready to go.
In a nutshell...simplify and test one bit at a time. When its working, add more. There are too many threads and locks right now.
Here is a simplified example of your listenToClient method:
def listenToClient(self, c):
while self.server_running:
try:
recvd_data = c.client.recv(self.size)
print "received:", c, recvd_data
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
return
print recvd_data
except socket.error, (value, message):
if value == 35:
continue
else:
print "Error:", value, message

Backup your work, then toss it - partially.
You need to implement your program in pieces, and test each piece as you go. First, tackle the input part of your program. Don't worry about how to broadcast the input you received. Instead worry that you are able to successfully and repeatedly receive input over your socket. So far - so good.
Now, I assume you would like to react to this input by broadcasting to the other attached clients. Well too bad, you can't do that yet! Because, I left one minor detail out of the paragraph above. You have to design a PROTOCOL.
What is a protocol? It's a set of rules for communication. How does your server know when the client had finished sending it's data? Is it terminated by some special character? Or perhaps you encode the size of the message to be sent as the first byte or two of the message.
This is turning out to be a lot of work, isn't it? :-)
What's a simple protocol. A line-oriented protocol is simple. Read 1 character at a time until you get to the end of record terminator - '\n'. So, clients would send records like this to your server --
HELO\n
MSG DAVE Where Are Your Kids?\n
So, assuming you have this simple protocol designed, implement it. For now, DON'T WORRY ABOUT THE MULTITHREADING STUFF! Just worry about making it work.
Your current protocol is to read 1024 bytes. Which may not be bad, just make sure you send 1024 byte messages from the client.
Once you have the protocol stuff setup, move on to reacting to the input. But for now you need something that will read input. Once that is done, we can worry about doing something with it.
jdi is right, you have too much program to work with. Pieces are easier to fix.