We Keep Coding

How to keep connectivity status between node.js and python?

I have web application based on Node.js and python based hardware. I want to keep status of connectivity between web server and hardware. If Hardware gets disconnected from web application then web application should get event or notification so based on that i can send notification to user. I have used mqtt for data communication, but to keep connection status I can't use MQTT coz it is connected with broker. I don't want to increase more load on server.
Which tools/technology/protocol/method should i use to keep connection status that device is offline or online?. That I also want use when user tries to send data to hardware using web application and if device is not connected with server then user should get notification that device is offline based on connection status.

The following code demonstrates the process I was hinting at in the comments.
The LWT feature tells the broker to publish a message marking the client as offline when it fails to respond in 1.5 times the keepalive period. If the client cleanly disconnects then it needs to mark it's self as offline. The client marks it's self as online when it connects to the broker.
All the status messages have the retained bit set so they will always be delivered when a client subscribes to the status topic.
import paho.mqtt.client as mqtt
# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, rc):
print("Connected with result code "+str(rc))
# Subscribing in on_connect() means that if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("some/application/topic")
# set status message to online
client.publish("status/client1", payload="online", retain=True)
# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
if str(msg.payload) == "shutdown":
# update status to offline as this will be a clean dissconect
client.publish("status/client1", payload="offline", retain=True)
client.disconnect()
client = mqtt.Client(client_id="client1")
client.on_connect = on_connect
client.on_message = on_message
client.will_set("status/client1", payload="offline", retain=True)
client.connect("mqtt.eclipse.org", 1883, 60)
# Blocking call that processes network traffic, dispatches callbacks and
# handles reconnecting.
# Other loop*() functions are available that give a threaded interface and a
# manual interface.
client.loop_forever()
It will be up to the OP to implement the notification about sending messages to offline clients requested at the end of the question.

creating producer and consumer application in python

I am trying to write a producer and consumer code in python using pika for rabbitmq. However for my specific case, I need to run producer on a different host and consumer on other.
I have already written a producer code as:
import pika
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters('ip add of another host', 5672, '/', credentials)
connection = pika.BlockingConnection()
channel = connection.channel()
channel.queue_declare(queue='test')
channel.basic_publish(exchange='', routing_key='test', body='hello all!')
print (" [x] sent 'Hello all!")
connection.close()
The above producer code is running without any error. I also created a new user and gave administrator credentials to it on rabbitmq-server. However when I run the consumer code on another host running rabbitmq-server, I do not see any output:
import pika
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters('localhost', 5672, '/', credentials)
connection = pika.BlockingConnection()
channel = connection.channel()
channel.queue_declare(queue='test')
def callback(ch, method, properties, body):
print(" [x] Recieved %r" % body)
channel.basic_consume(
queue='test', on_message_callback=callback, auto_ack=True)
print (' [x] waiting for messages. To exit press ctrl+c')
channel.start_consume()
So, here i had two hosts on the same network which had rabbitmq installed. However one has 3.7.10 and other had 3.7.16 version of rabbitmq.
The producer is able to send the text without error, but the consumer on another host is not receiving any text.
I do not get any problem when both run on same machine, as i just replace connection settings with localhost. Since user guest is only allowed to connect on localhost by default, i created a new user on consumer host running rabbitmq-server.
Please look if anyone can help me out here...

I have a couple of questions when I see your problem:
Are you 100% sure that on your RabbitMQ management monitoring
you see 2 connections? One from your local host and another from the another host? This will help to debug
Second, Did you check that your ongoing port 5672 on the server that host RabbitMQ is open? Because maybe your producer does not manage to connect What is your cloud provider?
If you don't want to manage those kinds of issues, you should use a service like https://zenaton.com. They host everything for you, and you have integrated monitoring, error handling etc.

Your consumer and producer applications must connect to the same RabbitMQ server. If you have two instances of RabbitMQ running they are independent. Messages do not move from one instance of RabbitMQ to another unless you configure Shovel or Federation.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.

You don't seem to be passing the parameters to the BlockingConnection instance.
import pika
rmq_server = "ip_address_of_rmq_server"
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters(rmq_server, 5672, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
Also, your consumer is attaching to the localhost hostname. Make sure this actually resolves and that your RabbitMQ service is listening on the localhost address (127.0.0.1) It may not be bound to that address. I believe that RMQ will bind to all interfaces (and thus all addresses) by default but I'm not sure.

Sending message from Kafka to socket.io in python

I have an end-to-end pipeline of an web application like below in Python3.6
Socket(connection from client to server) -> Flask Server -> Kafka Producer ->Kafka Consumer ->NLPService
Now when I get some result back from the NLPService, I need to send it back to the client. I am thinking below steps
NLP service writes the result to a different topic on Kafka producer (done)
Kafka consumer retrieves the result from Kafka broker (done)
Kafka consumer needs to write the result to the flask server
Then flask server will send the result back to the socket
Socket writes to client
I have already done steps 1-2. But stuck at step 3, 4. How do I write from Kafka to the flask server? If I just call a function at my server.py, then logically it seems like I have to create a socket within at function at server.py which will do the job of sending to client through socket. But syntax wise it looks weird. What am I missing?
at consumer.py
#receiving reply
topicReply = 'Reply'
consumerReply = KafkaConsumer(topicReply, value_deserializer=lambda m: json.loads(m.decode('ascii')))
for message in consumerReply:
#send reply back to Server
fromConsumer(message.value)
at server.py
socketio = SocketIO(app)
def fromConsumer(msg):
#socketio.on('reply')
def replyMessage(msg):
send(msg)
The above construct in server.py doesn't make sense to me. Please suggest.

Detect when Websocket is disconnected, with Python Bottle / gevent-websocket

I'm using the gevent-websocket module with Bottle Python framework.
When a client closes the browser, this code
$(window).on('beforeunload', function() { ws.close(); });
helps to close the websocket connection properly.
But if the client's network connection is interrupted, no "close" information can be sent to the server.
Then, often, even 1 minute later, the server still believes the client is connected, and the websocket is still open on the server.
Question: How to detect properly that a websocket is closed because the client is disconnected from network?
Is there a websocket KeepAlive feature available in Python/Bottle/gevent-websocket?
One answer from Web Socket: cannot detect client connection on internet disconnect suggests to use a heartbeat/ping packet every x seconds to tell the server "I'm still alive". The other answer suggests using a setKeepAlive(true). feature. Would this feature be available in gevent-websocket?
Example server code, taken from here:
from bottle import get, template, run
from bottle.ext.websocket import GeventWebSocketServer
from bottle.ext.websocket import websocket
users = set()
#get('/')
def index():
return template('index')
#get('/websocket', apply=[websocket])
def chat(ws):
users.add(ws)
while True:
msg = ws.receive()
if msg is not None:
for u in users:
u.send(msg)
else:
break
users.remove(ws)
run(host='127.0.0.1', port=8080, server=GeventWebSocketServer)

First you need to add a timeout to the receive() method.
with gevent.Timeout(1.0, False):
msg = ws.receive()
Then the loop will not block, if you send even an empty packet and the client doesn't respond, WebsocketError will be thrown and you can close the socket.

Python pubsub / message queue over HTTP?

I have a python script that will run on a local machine that needs to access a message queue (RabbitMQ) or receive subscribed events over HTTP. I've researched several solutions, but none seem natively designed to allow desktop clients to access them over HTTP. I'm thinking that using Twisted as a proxy is an option as well. Any guidance or recommendations would be greatly appreciated. Thanks in advance.

I've read this tutorial on RabbitMQ site, and they provide name of some libraries that could solve receiving messages.
Sender: send.py
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
channel.basic_publish(exchange='',
routing_key='hello',
body='Hello World!')
print " [x] Sent 'Hello World!'"
connection.close()
Receiver: receive.py
#!/usr/bin/env python
import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(
host='localhost'))
channel = connection.channel()
channel.queue_declare(queue='hello')
print ' [*] Waiting for messages. To exit press CTRL+C'
def callback(ch, method, properties, body):
print " [x] Received %r" % (body,)
channel.basic_consume(callback,
queue='hello',
no_ack=True)
channel.start_consuming()
Now we can try out our programs in a terminal. First, let's send a message using our send.py program:
$ python send.py
[x] Sent 'Hello World!'
The producer program send.py will stop after every run. Let's receive it:
$ python receive.py
[*] Waiting for messages. To exit press CTRL+C
[x] Received 'Hello World!'
Hurray! We were able to send our first message through RabbitMQ. As you might have noticed, the receive.py program doesn't exit. It will stay ready to receive further messages, and may be interrupted with Ctrl-C.
Try to run send.py again in a new terminal.
We've learned how to send and receive a message from a named queue. It's time to move on to part 2 and build a simple work queue.

I've decided to use wamp http://wamp.ws/. Still experimenting with it, but it's working quite well at the moment.

Choice #1
You may be interested in this RabbitHub
Choice #2
If you want it to be on port#80, cant you do port forwarding using a proxy? It could be challenging, but
Choice #3
If your script is not tightly coupled with RMQ message format , you can try celery ( which uses RMQ underneath), then u can try celery Http gateway or celery web hooks if u want any other application to be triggered directly
It might be time consuming to get it up. However, Celery opens up loads of flexibility
Choice #4
For one of my projects, I developed an intermediate web service (Flask Service) to use RMQ
Not ideal, but it served the purpose at that time.