We Keep Coding

Can pyzmq publishers be operated from class instances?

I have following code for a Publisher, which instantiates a few class instances and publishes some messages.
However, I don't receive anything at Subscriber side.
Publisher
import zmq
import time
from multiprocessing import Process
class SendData:
def __init__(self, msg, port):
self.msg = msg
self.port = port
ctx = zmq.Context()
self.sock = ctx.socket(zmq.PUB)
self.sock.bind('tcp://127.0.0.1:'+str(self.port))
time.sleep(1)
def sender(self):
self.sock.send_json(self.msg)
def main():
for device, port in zip(['2.2.2.2', '5.5.5.5'],[5001, 5002]):
msg = {device:'Some random message'}
instance = SendData(device, port)
Process(target=instance.sender).start()
if __name__ == "__main__":
main()
Subscriber
import zmq
ctx = zmq.Context()
recv_sock1 = ctx.socket(zmq.SUB)
recv_sock1.connect('tcp://127.0.0.1:5001')
recv_sock1.setsockopt(zmq.SUBSCRIBE, '')
recv_sock2 = ctx.socket(zmq.SUB)
recv_sock2.connect('tcp://127.0.0.1:5002')
recv_sock2.setsockopt(zmq.SUBSCRIBE, '')
while True:
if recv_sock1.poll(10):
msg = recv_sock1.recv_json()
print msg
if recv_sock2.poll(10):
msg = recv_sock2.recv_json()
print msg
I had subscribers started before publishers could publish anything. Also, I can see TCP connections are in Established so connections are made.
pyzmq version 16.0.0
python version 2.7
Q1: Are 0mq publishers supported from class instances?
Q2: What am I missing?

As was said before, trying to share the ZeroMQ context between processes is the problem here and the solution by user3666197 will work.
However, I would suggest subclassing multiprocessing.Process in this case. That way, it is much clearer what part of the code is executed in which process. It also makes your code more readable and reusable.
The following code creates one sender process per device. The sender processes can be reused during the runtime of your program to send more data:
import multiprocessing
import queue
import zmq
import time
class Sender(multiprocessing.Process):
def __init__(self, port):
super(Sender, self).__init__()
self._port = port
self._messages = multiprocessing.Queue()
self._do_stop = multiprocesing.Event()
def run(self):
"""
This code is executed in a new process.
"""
ctx = zmq.Context()
sock = ctx.socket(zmq.PUB)
sock.bind("tcp://127.0.0.1:" + str(self._port))
while not self._do_stop.is_set():
try:
msg = self._message.get_nowait()
sock.send_json(msg)
except queue.Empty:
time.sleep(0.01)
def stop(self):
self._do_stop.set()
def send_message(self, msg):
self._messages.put(msg)
def main():
data = zip(['2.2.2.2', '5.5.5.5'], [5001, 5002])
# create senders
senders = {device: Sender(port) for device, port in data}
# start senders
for device in senders:
senders[device].start()
# send messages
for device, port in zip(['2.2.2.2', '5.5.5.5'],[5001, 5002]):
msg = {device: 'Some random message'}
senders[device].send_message(msg)
# do more stuff here....
# ... e.g. send more messages
# ...
# once you are finished, stop the subprocesses
for device in senders:
senders[device].stop()
I hope this helps solving your problem.

A1: Yes, they are.
A2: Conflicts of scope-of-use v/s Zero-sharing, one of ZeroMQ maxims
Once your original Publisher code is being executed in main(), the class instantiation process creates ( i.e. inside the main()-process scope-of-use ), via the .__init__() constructor-method, it's own Context() -instance, which thus belongs ( incl. all of it's derived child-objects ( sockets et al ) ) to this main()-process.
Next, the call to the Process(...) initiates another few processes, that receive the class-instances ( the pity is that these have already created ZeroMQ non-shareable toys ) from the main()-scope-of-use.
Solution?
A possible dirty quick hack could be to defer the ZeroMQ Context() instantiation -- yes, just move it from .__init__() to some .aDeferredSETUP() that will be executed specifically under different scope-of-use inside each of the spinned-of Process()-process, different from main()-process and you ought be done, as Zero-sharing is safely obeyed.
class SendData:
def __init__(self, msg, port):
self.msg = msg
self.port = port
self.NotSETUP = True
self.ctx = None
self.sock = None
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ L8R
# ctx = zmq.Context()
# self.sock = ctx.socket( zmq.PUB )
# self.sock.bind( 'tcp://127.0.0.1:' + str( self.port ) )
# time.sleep( 1 )
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ L8R
def sender( self ):
if self.NotSETUP:
self.aDeferredSETUP()
self.sock.send_json( self.msg )
def aDeferredSETUP( self ): # create I/O-threads in Process(), not main()
self.ctx = zmq.Context()
self.sock = self.ctx.socket( zmq.PUB )
self.sock.bind( 'tcp://127.0.0.1:' + str( self.port ) )
time.sleep( 1 )

Python socket programming with multiple threads

I'm trying to use sockets in python. For now, I'm trying to get it such that if any client sends any message it is received at all clients. However I'm getting very weird results. I think it's because I'm using multiple threads. The output of the program changes every time I run it. Is this a threading problem or is it something else?
import socket
import sys
from thread import *
from server import Server
from client import Client
s = Server()
start_new_thread(s.acceptConnection,())
m = Client("m")
k = Client("k")
start_new_thread(m.recieveData,())
start_new_thread(k.recieveData,())
k.sendData("Hey!")
print "*"*100
print repr(k.data()), repr(m.data())
print "*"*100
m.sendData("okay okay")
print "*"*100
print repr(k.data()), repr(m.data())
print "*"*100
m.client.close()
k.client.close()
s.s.close()
Server Class:
import socket
import sys
from thread import *
class Server(object):
def __init__(self,port = 5555):
self.host = 'localhost' # '' means connect to all hosts
self.port = port
self.text = ""
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
try:
self.s.bind((self.host, self.port))
except socket.error as e:
print(str(e))
self.s.listen(2)
print "Waiting for a connection.\n"
self.connections = []
def threaded_client(self,conn):
# conn.send("Connected to server\n")
while True:
try:
data = conn.recv(2048)
except:
data = ""
if(not data):
break
# conn.sendall(reply)
for c,a in self.connections:
try:
c.sendall(data + "\n")
except:
print "connection lost\n"
self.connections.remove((c,a))
conn.close()
def acceptConnection(self):
while True:
conn, addr = self.s.accept()
self.connections += [(conn,addr)]
start_new_thread(self.threaded_client,(conn,))
Client class:
import socket
import sys
from thread import *
class Client(object):
def __init__(self,name):
self.host = 'localhost'
self.port = 5555
self.name = name
self.client= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.client.connect((self.host,self.port))
self.text = ""
def sendData(self,data):
self.client.send(data)
def recieveData(self):
while True:
try:
data = self.client.recv(2048)
except:
break
if data:
self.text = data
self.client.close()
def data(self):
return self.text
def closeClient(self):
self.client.close()

Anyway you have no warrants about the data was already come back to the clients when you try to print it. You should introduce some Conditions and use wait() and notifyAll() to make sure the data was arrive.... To check if my guess is correct put some sleep() in your test:
import time
k.sendData("Hey!")
print "*"*100
time.sleep(200)
print repr(k.data()), repr(m.data())
print "*"*100
m.sendData("okay okay")
print "*"*100
time.sleep(200)
print repr(k.data()), repr(m.data())
print "*"*100
If it works you should use conditions and notify to do your tests.
Moreover you must protect data access by a Lock().
def recieveData(self):
while True:
try:
data = self.client.recv(2048)
except:
break
if data:
self.l.acquire()
self.text = data
self.l.release()
self.client.close()
def data(self):
self.l.acquire()
ret = self.text
self.l.release()
return ret
Where attribute l of clients are defined in __init__ by
self.l=threading.Lock()

Python P2P Sockets Chat Script working only on home network (not on school network)

I have the following python script (a bit inefficient, I know) for a P2P chat program using sockets:
#!usr/bin/env python
import socket import threading import select import time import datetime
def main():
class Chat_Server(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.running = 1
self.conn = None
self.addr = None
def run(self):
HOST = ''
PORT = 23647
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST,PORT))
s.listen(1)
self.conn, self.addr = s.accept()
# Select loop for listen
while self.running == True:
inputready,outputready,exceptready \
= select.select ([self.conn],[self.conn],[])
for input_item in inputready:
# Handle sockets
message = self.conn.recv(1024)
if message:
print "Daniel: " + message + ' (' + datetime.datetime.now().strftime('%H:%M:%S') + ')'
else:
break
time.sleep(0)
def kill(self):
self.running = 0
class Chat_Client(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.host = None
self.sock = None
self.running = 1
def run(self):
PORT = 23647
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect((self.host, PORT))
# Select loop for listen
while self.running == True:
inputready,outputready,exceptready \
= select.select ([self.sock],[self.sock],[])
for input_item in inputready:
# Handle sockets
message = self.sock.recv(1024)
if message:
print "Daniel: " + message + ' (' + datetime.datetime.now().strftime('%H:%M:%S') + ')'
else:
break
time.sleep(0)
def kill(self):
self.running = 0
class Text_Input(threading.Thread):
def __init__(self):
threading.Thread.__init__(self)
self.running = 1
def run(self):
while self.running == True:
text = raw_input('')
try:
chat_client.sock.sendall(text)
except:
Exception
try:
chat_server.conn.sendall(text)
except:
Exception
time.sleep(0)
def kill(self):
self.running = 0
# Prompt, object instantiation, and threads start here.
ip_addr = raw_input('Type IP address or press enter: ')
if ip_addr == '':
chat_server = Chat_Server()
chat_client = Chat_Client()
chat_server.start()
text_input = Text_Input()
text_input.start()
else:
chat_server = Chat_Server()
chat_client = Chat_Client()
chat_client.host = ip_addr
text_input = Text_Input()
chat_client.start()
text_input.start()
if __name__ == "__main__":
main()
This script works absolutely fine across a home network, with 192.168... internal IP addresses.
On a school network, with 172... IP addresses, it doesn't seem to work. There is no connection error, but messages are not sent or received, with the exception of if there are two instances of the application being run on the same computer with the same internal IP address, in which case the program works flawlessly.
I am very new to sockets and this sort of networking, so I am wondering if anyone can point out to me why this is the case. Might it be the difference in IP addresses, for example?
Thank you in advance.

I see you're connecting on port 23647 - you may not have access to this on your school network. Check if this port has traffic enabled. see: Port Forwarding for details.
Something like: this site/tool may allow you to check quickly.

zeromq: why dealer sends one message and receives from one client?

I'm trying to understand my code's behavior.
I'm using zeromq to create a server that sends a "ping" and waits for "pong" responses.
What i'm seeing is that when I send a ping, only one client receives it.
when I run this code and send "ping" for the first time i receive:
pong: A
and when i run it again, i get
pong: B
why is that? I want to send one "ping" and receive two pongs.
here's the code:
from threading import Thread
import zmq
class zmqdealer(object):
def __init__(self, port):
context = zmq.Context()
self.sock = context.socket(zmq.DEALER)
#self.sock.setsockopt(zmq.RCVTIMEO, 1000)
self.sock.bind("tcp://*:%s" % port)
thread = Thread(target=lambda: self.poll())
thread.daemon = True
thread.start()
def poll(self):
while True:
reply = self.sock.recv()
if reply != "":
print(reply)
def ping(self):
self.sock.send_multipart(['', 'ping'])
class zmqrep(object):
def __init__(self, ident,host, port):
context = zmq.Context()
self.sock = context.socket(zmq.REP)
self.sock.connect("tcp://%s:%s" % (host, port))
self.ident = ident
thread = Thread(target=lambda: self.pong())
thread.daemon = True
thread.start()
def pong(self):
while True:
request = self.sock.recv()
if request == "ping":
msg = "pong: %s" % self.ident
self.sock.send(msg)
if __name__ == "__main__":
port = 11112
host = "localhost"
server = zmqdealer(port)
client1 = zmqrep('A',host,port)
client2 = zmqrep('B',host,port)
answer = raw_input('press <ENTER> to exit or type \'ping\' to get a pong\n')
while True:
if answer == "":
break
if answer == "ping":
server.ping()
answer = raw_input()
EDIT
I found a way to make this work. I really hope there is another way because i genuinely hate this one! so it looks like dealer sends to the clients in a round robin fashion. so to make my ping work i had to send it to all the clients. how? i subscribed to the monitor socket and added every connected client to a list. every time i ping, i ping to every client. look:
import threading
import zmq
from zmq.utils import monitor
def threadify(func, daemon=True):
thread = threading.Thread(target=func)
thread.daemon = daemon
thread.start()
class zmqdealer(object):
def __init__(self, port):
context = zmq.Context()
self.sock = context.socket(zmq.DEALER)
self.monitor_sock = self.sock.get_monitor_socket()
self.sock.bind("tcp://*:%s" % port)
self.connected_clients = {}
threadify(func=self.poll)
threadify(func=self.monitor)
def poll(self):
while True:
reply = self.sock.recv()
if reply != "":
print reply
def add_client(self, event):
endpoint = event['endpoint']
value = event['value']
if endpoint in self.connected_clients:
self.connected_clients[endpoint].append(value)
else:
self.connected_clients[endpoint] = [value]
def remove_client(self, event):
endpoint = event['endpoint']
value = event['value']
if endpoint in self.connected_clients \
and value in self.connected_clients[endpoint]:
self.connected_clients[endpoint].remove(value)
def monitor(self):
options = {zmq.EVENT_ACCEPTED: lambda e: self.add_client(e),
zmq.EVENT_DISCONNECTED: lambda e: self.remove_client(e)}
while True:
event = monitor.recv_monitor_message(self.monitor_sock)
event_type = event['event']
if event_type in options:
options[event_type](event)
event['event'] = event_types[event_type]
print event
def ping(self):
connected_clients_amount = sum([len(clients) for clients in self.connected_clients.values()])
for i in xrange(connected_clients_amount):
self.sock.send_multipart(['', 'ping'])
if connected_clients_amount <= 0:
print "there are no connected clients!"
class zmqrep(object):
def __init__(self, ident, host, port):
context = zmq.Context()
self.sock = context.socket(zmq.REP)
self.sock.connect("tcp://%s:%s" % (host, port))
self.identity = ident
self.stopped = threading.Event()
threadify(self.pong)
def pong(self):
while not self.stopped.isSet():
request = self.sock.recv()
if request == "ping":
msg = "pong: %s" % self.identity
self.sock.send(msg)
self.sock.close()
def stop(self):
self.stopped.set()
if __name__ == "__main__":
port = 11112
host = "localhost"
num = 5
server = zmqdealer(port)
clients = [zmqrep(i.__str__(), host, port) for i in xrange(num)]
answer = raw_input('press <ENTER> to exit or type \'ping\' to get a pong\n')
while True:
if answer == "":
break
if answer == "ping":
server.ping()
if answer == "kill":
if len(clients) > 0:
die = clients[0]
clients.remove(die)
die.stop()
else:
print "there are no connected clients!\n"
answer = raw_input()

Router/Dealer sockets are best used for distributing tasks. Say you have 10 tasks and 2 workers, you do not care who does what. Dealer/Router will distribute in a round robin fashion.
Maybe Pub/Sub or Push/Pull sockets would fit your usecase better? They are both broadcast sockets.
Here's an example of Push/Pull used in a similar fashion as what you're doing.
You often end up doing pairs of sockets, one to transmit and one other to receive results. You could for example do a PUSH with a ping message + random identifier, and ask clients to answer on PUB/SUB where you subscribe to this random identifier. This way you can match requests and responses.

Python multiprocessing RemoteManager under a multiprocessing.Process

I'm trying to start a data queue server under a managing process (so that it can later be turned into a service), and while the data queue server function works fine in the main process, it does not work in a process created using multiprocessing.Process.
The dataQueueServer and dataQueueClient code is based on the code from the multiprocessing module documentation here.
When run on its own, dataQueueServer works well. However, when run using a multiprocessing.Process's start() in mpquueue, it doesn't work (when tested with the client). I am using the dataQueueClient without changes to test both cases.
The code does reach the serve_forever in both cases, so I think the server is working, but something is blocking it from communicating back to the client in the mpqueue case.
I have placed the loop that runs the serve_forever() part under a thread, so that it can be stoppable.
Here is the code:
mpqueue # this is the "manager" process trying to spawn the server in a child process
import time
import multiprocessing
import threading
import dataQueueServer
class Printer():
def __init__(self):
self.lock = threading.Lock()
def tsprint(self, text):
with self.lock:
print text
class QueueServer(multiprocessing.Process):
def __init__(self, name = '', printer = None):
multiprocessing.Process.__init__(self)
self.name = name
self.printer = printer
self.ml = dataQueueServer.MainLoop(name = 'ml', printer = self.printer)
def run(self):
self.printer.tsprint(self.ml)
self.ml.start()
def stop(self):
self.ml.stop()
if __name__ == '__main__':
printer = Printer()
qs = QueueServer(name = 'QueueServer', printer = printer)
printer.tsprint(qs)
printer.tsprint('starting')
qs.start()
printer.tsprint('started.')
printer.tsprint('Press Ctrl-C to quit')
try:
while True:
time.sleep(60)
except KeyboardInterrupt:
printer.tsprint('\nTrying to exit cleanly...')
qs.stop()
printer.tsprint('stopped')
dataQueueServer
import time
import threading
from multiprocessing.managers import BaseManager
from multiprocessing import Queue
HOST = ''
PORT = 50010
AUTHKEY = 'authkey'
## Define some helper functions for use by the main process loop
class Printer():
def __init__(self):
self.lock = threading.Lock()
def tsprint(self, text):
with self.lock:
print text
class QueueManager(BaseManager):
pass
class MainLoop(threading.Thread):
"""A thread based loop manager, allowing termination signals to be sent
to the thread"""
def __init__(self, name = '', printer = None):
threading.Thread.__init__(self)
self._stopEvent = threading.Event()
self.daemon = True
self.name = name
if printer is None:
self.printer = Printer()
else:
self.printer = printer
## create the queue
self.queue = Queue()
## Add a function to the handler to return the queue to clients
self.QM = QueueManager
self.QM.register('get_queue', callable=lambda:self.queue)
self.queue_manager = self.QM(address=(HOST, PORT), authkey=AUTHKEY)
self.queue_server = self.queue_manager.get_server()
def __del__(self):
self.printer.tsprint( 'closing...')
def run(self):
self.printer.tsprint( '{}: started serving'.format(self.name))
self.queue_server.serve_forever()
def stop(self):
self.printer.tsprint ('{}: stopping'.format(self.name))
self._stopEvent.set()
def stopped(self):
return self._stopEvent.isSet()
def start():
printer = Printer()
ml = MainLoop(name = 'ml', printer = printer)
ml.start()
return ml
def stop(ml):
ml.stop()
if __name__ == '__main__':
ml = start()
raw_input("\nhit return to stop")
stop(ml)
And a client:
dataQueueClient
import datetime
from multiprocessing.managers import BaseManager
n = 0
N = 10**n
HOST = ''
PORT = 50010
AUTHKEY = 'authkey'
def now():
return datetime.datetime.now()
def gen(n, func, *args, **kwargs):
k = 0
while k < n:
yield func(*args, **kwargs)
k += 1
class QueueManager(BaseManager):
pass
QueueManager.register('get_queue')
m = QueueManager(address=(HOST, PORT), authkey=AUTHKEY)
m.connect()
queue = m.get_queue()
def load(msg, q):
return q.put(msg)
def get(q):
return q.get()
lgen = gen(N, load, msg = 'hello', q = queue)
t0 = now()
while True:
try:
lgen.next()
except StopIteration:
break
t1 = now()
print 'loaded %d items in ' % N, t1-t0
t0 = now()
while queue.qsize() > 0:
queue.get()
t1 = now()
print 'got %d items in ' % N, t1-t0

So it seems like the solution is simple enough: Don't use serve_forever(), and use manager.start() instead.
According to Eli Bendersky, the BaseManager (and it's extended version SyncManager) already spawns the server in a new process (and looking at the multiprocessing.managers code confirms this). The problem I have been experiencing stems from the form used in the example, in which the server is started under the main process.
I still don't understand why the current example doesn't work when run under a child process, but that's no longer an issue.
Here's the working (and much simplified from OP) code to manage multiple queue servers:
Server:
from multiprocessing import Queue
from multiprocessing.managers import SyncManager
HOST = ''
PORT0 = 5011
PORT1 = 5012
PORT2 = 5013
AUTHKEY = 'authkey'
name0 = 'qm0'
name1 = 'qm1'
name2 = 'qm2'
description = 'Queue Server'
def CreateQueueServer(HOST, PORT, AUTHKEY, name = None, description = None):
name = name
description = description
q = Queue()
class QueueManager(SyncManager):
pass
QueueManager.register('get_queue', callable = lambda: q)
QueueManager.register('get_name', callable = name)
QueueManager.register('get_description', callable = description)
manager = QueueManager(address = (HOST, PORT), authkey = AUTHKEY)
manager.start() # This actually starts the server
return manager
# Start three queue servers
qm0 = CreateQueueServer(HOST, PORT0, AUTHKEY, name0, description)
qm1 = CreateQueueServer(HOST, PORT1, AUTHKEY, name1, description)
qm2 = CreateQueueServer(HOST, PORT2, AUTHKEY, name2, description)
raw_input("return to end")
Client:
from multiprocessing.managers import SyncManager
HOST = ''
PORT0 = 5011
PORT1 = 5012
PORT2 = 5013
AUTHKEY = 'authkey'
def QueueServerClient(HOST, PORT, AUTHKEY):
class QueueManager(SyncManager):
pass
QueueManager.register('get_queue')
QueueManager.register('get_name')
QueueManager.register('get_description')
manager = QueueManager(address = (HOST, PORT), authkey = AUTHKEY)
manager.connect() # This starts the connected client
return manager
# create three connected managers
qc0 = QueueServerClient(HOST, PORT0, AUTHKEY)
qc1 = QueueServerClient(HOST, PORT1, AUTHKEY)
qc2 = QueueServerClient(HOST, PORT2, AUTHKEY)
# Get the queue objects from the clients
q0 = qc0.get_queue()
q1 = qc1.get_queue()
q2 = qc2.get_queue()
# put stuff in the queues
q0.put('some stuff')
q1.put('other stuff')
q2.put({1:123, 2:'abc'})
# check their sizes
print 'q0 size', q0.qsize()
print 'q1 size', q1.qsize()
print 'q2 size', q2.qsize()
# pull some stuff and print it
print q0.get()
print q1.get()
print q2.get()
Adding an additional server to share a dictionary with the information of the running queue servers so that consumers can easily tell what's available where is easy enough using that model. One thing to note, though, is that the shared dictionary requires slightly different syntax than a normal dictionary: dictionary[0] = something will not work. You need to use dictionary.update([(key, value), (otherkey, othervalue)]) and dictionary.get(key) syntax, which propagates across to all other clients connected to this dictionary..