I've got some sort of distributed control system in which I send a heart beat every second. On the receiving end I need to take action if no message has been received for more than 2 seconds. The problem is that when zeroMQ is waiting for an answer I can't do anything else, like checking how much time has passed since the last message was received.
The code I have now is below. Does anybody know how I could take action if no message has been received for more than 2 seconds? All tips are welcome!
[EDIT] With the tip of Pieter Hintjes I added polling to the code, but I still doesn't work. Any other ideas?
from datetime import datetime
import zmq
context = zmq.Context()
# Set up subscriber connection to receive message from broker
subscriber = context.socket(zmq.SUB)
subscriber.connect('tcp://localhost:8888')
subscriber.setsockopt(zmq.SUBSCRIBE, 'beat')
# Initialise poll set
poller = zmq.Poller()
poller.register(subscriber, zmq.POLLIN)
while True:
socks = dict(poller.poll(2000))
if subscriber in socks and socks[subscriber] == zmq.POLLIN:
message = subscriber.recv()
print(message)
print('do other stuff')
Use poll on the ZMQ socket instead of a blocking recv. There are lots of examples of this in the ZeroMQ Guide.
Related
I'm trying to do UDP socket programming in Python, and I want both the client and the server to be able to send messages without the need to wait for the other party to send a message first.
Here is my server code:
import socket
sock=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.bind(('127.0.0.1',12345))
while True:
data,addr=sock.recvfrom(4096) #byte size
print("Client says: ")
print(str(data))
message = bytes(input("Enter message here: ").encode('utf_8'))
sock.sendto(message,addr)
and here is my client code:
import socket
client_socket=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
while True:
msg=input("Enter your message here: ")
client_socket.sendto(msg.encode('utf_8'),('127.0.0.1',12345))
data,addr=client_socket.recvfrom(4096) #byte size
print("Server says")
print(data)
What should I edit in my code to make this work?
You need to create two functions to handle listening and sending.
Then you start them as threads so they run in parallel.
Each function has its own loop. The receive thread waits for messages and the send thread waits for user input
def client_receive():
while True:
data,addr=client_socket.recvfrom(4096) #byte size
print("Server says")
print(data)
def send_chat_message():
while True:
msg=input("Enter your message here: ")
client_socket.sendto(msg.encode('utf_8'),('127.0.0.1',12345))
receive_thread = threading.Thread(target=client_receive)
receive_thread.start()
send_thread = threading.Thread(target=send_chat_message)
send_thread.start()
The above code will not really work at this point. But there are good tutorials out there. Search for python socket chat threaded tutorial
Often they are not based on a server handling inputs, but multiple clients connecting to a server. If you understand the concept, you'll have no problem adjusting the server to allow inputs there aswell.
I am trying to get message from activemq using stomp.py and then doing some processing on it. But there is a case when that processing fails for certain messages and that message is lost.
How can I prevent the deletion of message untill the message is fully processed?
For example in my code when there is new entry in queue the on_message function will be called and the processing starts but if it is interrupted in between the message is lost. How do I stop it?
Here is my code:
conn = stomp.Connection([(host, 61613)])
conn.set_listener('ML', MyListener())
conn.start()
conn.connect('admin', 'admin', wait=True)
conn.subscribe(destination=/queue/someque, id=1, ack='auto')
print "running"
while 1:
print 'waiting'
time.sleep(2.5)
Here is my Listener class:
class MyListener(stomp.ConnectionListener):
def on_message(self, headers, message):
print headers
print message
do_something()
Thanks in advance.
The issue appears to be that you are using the 'auto' ack mode so the message will be acknowledged before delivery to the client by the broker meaning that even if you fail to process it, it's to late as it is already forgotten on the broker side. You'd need to use either 'client' ack or 'client-individual' ack mode as described in the STOMP specification. Using one of the client ack modes you control when a message or messages are actually acknowledged and dropped by the broker.
I started using zeromq with python with the Publisher/Subscriber reference. However, I don't find any documentation about how to treat messages in the queue. I want to treat the last received message different as the rest of the elements of the queue.
Example
publisher.py
import zmq
import random
import time
port = "5556"
topic = "1"
context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:%s" % port)
while True:
messagedata = random.randrange(1,215)
print "%s %d" % (topic, messagedata)
socket.send("%s %d" % (topic, messagedata))
time.sleep(.2)
subscriber.py
import zmq
port = "5556"
topic = "1"
context = zmq.Context()
socket = context.socket(zmq.SUB)
print "Connecting..."
socket.connect ("tcp://localhost:%s" % port)
socket.setsockopt(zmq.SUBSCRIBE,topic)
while True:
if isLastMessage(): # probably based on socket.recv()
analysis_function() # time consuming function
else:
simple_function() # something simple like print and save in memory
I just want to know how to create the isLastMessage() function described in the subscriber.py file. If there's something directly in zeromq or a workaround.
Welcome to the world of non-blocking messaging / signalling
this is a cardinal feature for any serious distributed-system design.
If you assume a "last" message via a not having another one in the pipe, then a Poller() instance may help your main event-loops, where you may control the amount of time to "wait"-a-bit before considering the pipe "empty", not to devastate your IO-resources with zero-wait spinning-loops.
Explicit signalling is always better ( if you can design the remote end behaviour )
There is Zero-knowledge on the receiver-side, what is the context of the "last"-message received ( and explicit signalling is advised to be rather broadcast from the message sender-side ), however there is a reversed feature to this -- that instructs ZeroMQ archetypes to "internally"-throw away all such messages, that are not the "last"-message, thus reducing the receiver-side processing to right the "last"-message available.
aQuoteStreamMESSAGE.setsockopt( zmq.CONFLATE, 1 )
If you may like to read more on ZeroMQ patterns and anti-patterns, do not miss Pieter HINTJENS' fabulous book "Code Connected, Volume 1" ( also in pdf ) and may like a broader view on distributed-computing using principally a non-blocking ZeroMQ approach
If isLastMessage() is meant to identify the last message within the stream of messages produced by publisher.py, than this is impossible since there is no last message. publisher.py produces an infinite amount of messages!
However, if publisher.py knows its last "real" message, i.e. no while True:, it could send a "I am done" message afterwards. Identifying that in subscriber.py is trivial.
Sorry, I will keep the question for reference. I just found the answer, in the documentation there is a NOBLOCK flag that you can add to the receiver. With this the recv command doesn't block. A simple workaround, extracted from a part of an answer, is the following:
while True:
try:
#check for a message, this will not block
message = socket.recv(flags=zmq.NOBLOCK)
#a message has been received
print "Message received:", message
except zmq.Again as e:
print "No message received yet"
As for the real implementation, one is not sure that it is the last call you use the flag NOBLOCK and once you have entered the exception block. Wich translates to something like the following:
msg = subscribe(in_socket)
is_last = False
while True:
if is_last:
msg = subscribe(in_socket)
is_last = False
else:
try:
old_msg = msg
msg = subscribe(in_socket,flags=zmq.NOBLOCK)
# if new message was received, then process the old message
process_not_last(old_msg)
except zmq.Again as e:
process_last(msg)
is_last = True # it is probably the last message
I created a zmq_forwarder.py that's run separately and it passes messages from the app to a sockJS connection, and i'm currently working on right now on how a flask app could receive a message from sockJS via zmq. i'm pasting the contents of my zmq_forwarder.py. im new to ZMQ and i dont know why everytime i run it, it uses 100% CPU load.
import zmq
# Prepare our context and sockets
context = zmq.Context()
receiver_from_server = context.socket(zmq.PULL)
receiver_from_server.bind("tcp://*:5561")
forwarder_to_server = context.socket(zmq.PUSH)
forwarder_to_server.bind("tcp://*:5562")
receiver_from_websocket = context.socket(zmq.PULL)
receiver_from_websocket.bind("tcp://*:5563")
forwarder_to_websocket = context.socket(zmq.PUSH)
forwarder_to_websocket.bind("tcp://*:5564")
# Process messages from both sockets
# We prioritize traffic from the server
while True:
# forward messages from the server
while True:
try:
message = receiver_from_server.recv(zmq.DONTWAIT)
except zmq.Again:
break
print "Received from server: ", message
forwarder_to_websocket.send_string(message)
# forward messages from the websocket
while True:
try:
message = receiver_from_websocket.recv(zmq.DONTWAIT)
except zmq.Again:
break
print "Received from websocket: ", message
forwarder_to_server.send_string(message)
as you can see, i've setup 4 sockets. the app connects to port 5561 to push data to zmq, and port 5562 to receive from zmq (although im still figuring out how to actually set it up to listen for messages sent by zmq). on the other hand, sockjs receives data from zmq on port 5564 and sends data to it on port 5563.
i've read the zmq.DONTWAIT makes receiving of message asynchronous and non-blocking so i added it.
is there a way to improve the code so that i dont overload the CPU? the goal is to be able to pass messages between the flask app and the websocket using zmq.
You are polling your two receiver sockets in a tight loop, without any blocking (zmq.DONTWAIT), which will inevitably max out the CPU.
Note that there is some support in ZMQ for polling multiple sockets in a single thread - see this answer. I think you can adjust the timeout in poller.poll(millis) so that your code only uses lots of CPU if there are lots of incoming messages, and idles otherwise.
Your other option is to use the ZMQ event loop to respond to incoming messages asynchronously, using callbacks. See the PyZMQ documentation on this topic, from which the following "echo" example is adapted:
# set up the socket, and a stream wrapped around the socket
s = ctx.socket(zmq.REP)
s.bind('tcp://localhost:12345')
stream = ZMQStream(s)
# Define a callback to handle incoming messages
def echo(msg):
# in this case, just echo the message back again
stream.send_multipart(msg)
# register the callback
stream.on_recv(echo)
# start the ioloop to start waiting for messages
ioloop.IOLoop.instance().start()
I am trying to write a server/client script with a server that vents the tasks, and multiple workers that execute it.
The problem is that my ventilator has so many tasks that it would fill up the memory in a heartbeat.
I tried to set the HWM before it binds, but with no success. It just keeps on sending messages as soon as a worker connects, completely disregarding the HWM that was set. I also have a sink that keeps record of the tasks that were done.
server.py
import zmq
def ventilate():
context = zmq.Context()
# Socket to send messages on
sender = context.socket(zmq.PUSH)
sender.setsockopt(zmq.SNDHWM, 30) #Big messages, so I don't want to keep too many in queue
sender.bind("tcp://*:5557")
# Socket with direct access to the sink: used to syncronize start of batch
sink = context.socket(zmq.PUSH)
sink.connect("tcp://localhost:5558")
print "Sending tasks to workers…"
# The first message is "0" and signals start of batch
sink.send('0')
print "Sent starting signal"
while True:
sender.send("Message")
if __name__=="__main__":
ventilate()
worker.py
import zmq
from multiprocessing import Process
def work():
context = zmq.Context()
# Socket to receive messages on
receiver = context.socket(zmq.PULL)
receiver.connect("tcp://localhost:5557")
# Socket to send messages to
sender = context.socket(zmq.PUSH)
sender.connect("tcp://localhost:5558")
# Process t asks forever
while True:
msg = receiver.recv_msg()
print "Doing sth with msg %s"%(msg)
sender.send("Message %s done"%(msg))
if __name__ == "__main__":
for worker in range(10):
Process(target=work).start()
sink.py
import zmq
def sink():
context = zmq.Context()
# Socket to receive messages on
receiver = context.socket(zmq.PULL)
receiver.bind("tcp://*:5558")
# Wait for start of batch
s = receiver.recv()
print "Received start signal"
while True:
msg = receiver.recv_msg()
print msg
if __name__=="__main__":
sink()
Ok, I had a play around, I don't think the issue is with the PUSH HWM, but rather that you can't set a HWM for PULL. If you look at this documentation, you can see there it says N/A for action on HWM.
The PULL sockets seem to be taking hundreds of messages each (and I did try setting a HWM just in case it did anything on the PULL socket. It didn't.). I evidenced this by changing the ventilator to send messages with an incrementing integer, and changing each worker in the pool to wait 2 seconds between calls to recv(). The workers print out that they are processing messages with vastly different integers. For instance, one worker will be working on message 10, while the next is working on message 400. As time goes on, you see the worker who was processing message 10, is now processing message 11, 12, 13, etc. while the other is processing 401, 402, etc.
This indicates to me that the ZMQ_PULL socket is buffering the messages somewhere. So while the ZMQ_PUSH socket does have a HWM, the PULL socket is requesting messages quickly, despite them not actually being accessed by a call to recv(). So that results in the PUSH HWM effectively being ignored if a PULL socket is connected. As far as I can see, you can't control the length of the buffer of the PULL socket (I would expect the RCVHWM socket option to control this but it doesn't appear to).
This behaviour of course begs the question what is the point of the ZMQ_PULL HWM option, which only makes sense to have if you can also control the receiving sockets HWM.
At this point, I'd start asking the 0MQ people whether you are missing something obvious, or if this is considered a bug.
Sorry I couldn't be more help!
ZeroMQ has buffers on both sending and receiving ends of a socket, hence you need to set high water marks on both the PUSH and the PULL socket in your code (and indeed before a bind() or connect()).
In the Python bindings this is now conveniently done via socket.hwm = 1 which will set both ZMQ_SNDHWM and ZMQ_RCVHWM in one go.