We Keep Coding

Paho MQTT client failing for special payloads (forward slashes) when using OpenVPN (Windows)

I have been struggeling with a strange case of random client MQTT publish failing for certain payloads. It happends randomly when trying to publish some large amount of BASE64 data.
I've finally managed to narrow it down to payloads containing a lot of consequtive forwards slashes (/). I've searched the net to find a good reason why this happends, but havent found anythong. Is it a MQTT feature, a Paho client feature or a broker feature, or just some bug...
Setup:
Python 3.8.8 (Windows 10)
paho-mqtt 1.5.0
mosquitto 1.6.9-1 amd64
On my setup, it fails when I send a payload of 255 '/' to a 1 character topic 'a'. Larger topic length, reduces the possible number of forward slashes.
Code to reproduce error:
import paho.mqtt.client as mqtt_client
import time
address = 'some.server.com'
port = 1883
connected = False
def on_connect(client, userdata, flags, rc):
global connected
connected = True
print("Connected!")
client = mqtt_client.Client()
client.on_connect = on_connect
client.connect(host=address, port=port, keepalive=60)
client.loop_start()
while not connected:
time.sleep(1)
payload = '/'*205
print('Payload: {}'.format(payload))
client.publish(topic='a', payload=payload)
time.sleep(2)
client.loop_stop()
client.disconnect()
print('Done!')
This generates this output:
Connected!
Payload: /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
Connected!
Done!
This produces the following error in /var/log/mosquitto/mosquitto.log for the mosquitto broker:
1616605010: New connection from a.b.c.d on port 1883.
1616605010: New client connected from a.b.c.d as auto-CEF15129-E74C-F00A-A6FA-5B5FDA0CEF1D (p2, c1, k60).
1616605011: Socket error on client auto-CEF15129-E74C-F00A-A6FA-5B5FDA0CEF1D, disconnecting.
1616605012: New connection from a.b.c.d on port 1883.
1616605012: New client connected from a.b.c.d as auto-0149B6DB-5997-9E08-366A-304F21FDF2E1 (p2, c1, k60).
1616605013: Client auto-0149B6DB-5997-9E08-366A-304F21FDF2E1 disconnected.
I observe that the client() connects twice, but do not know why, but this is probably caused by a disconnect...
Any Ideas?
Update 1: I've tested this on Linux Ubunit running Python 3.7.3, and same paho-mqtt version, and this does not produce the same error... Seems like some problem in Windows then.
Update 2:
I also tried running mosquitto_pub and experienced the same error, so this has to be Windows-related (or system related) in some way. Possibly firewall? I will close question if I find manage to solve this.
"C:\Program Files\mosquitto\mosquitto_pub.exe" -h some.server.com -t a -m '/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////'
Update 3 The issue is related to OpenVPN. I closed my open VPN connection and MQTT messages were passing! I'm running OpenVPN Client (Windows version 3.2.2 (1455). I have no idea what causes this conlflict...

How to connect to mq queue with python and bindings file?

I'm trying to connect with a remote MQ queue/serie and I only have a .bindings file to do it. I'm trying the python library "pymqi" but I can't connect using binding mode. Does someone knows what should I do or where should I place the file in order to use it with the library? Is there some other python solution to connect to the MQ queue?
this is a glimpse of my .bindings file:
JMSC/ClassName=com.ibm.mq.jms.MQQueueConnectionFactory
JMSC/FactoryName=com.ibm.mq.jms.MQQueueConnectionFactoryFactory
JMSC/RefAddr/0/Type=VER
JMSC/RefAddr/0/Encoding=String
JMSC/RefAddr/0/Content=7
JMSC/RefAddr/1/Type=TRAN
JMSC/RefAddr/1/Encoding=String
JMSC/RefAddr/1/Content=1
JMSC/RefAddr/2/Type=QMGR
JMSC/RefAddr/2/Encoding=String
JMSC/RefAddr/2/Content=MQFEND00
JMSC/RefAddr/3/Type=HOST
JMSC/RefAddr/3/Encoding=String
JMSC/RefAddr/3/Content=somehost
JMSC/RefAddr/4/Type=PORT
JMSC/RefAddr/4/Encoding=String
JMSC/RefAddr/4/Content=1414
JMSC/RefAddr/5/Type=CHAN
JMSC/RefAddr/5/Encoding=String
JMSC/RefAddr/5/Content=PORTALS.MQFEND00
It has like 100 params, that are the first 6,
Thanks
update 22/05/2019:
I will add more information about what I try.
I tried to connect with bindings mode as I saw on the pymqi documentation:
qmgr = pymqi.connect('MQFEND00')
And I get this error:
MQI Error. Comp: 2, Reason 2058: FAILED: MQRC_Q_MGR_NAME_ERROR
I'm not sure if it's the queue_manager, someone knows how can I get the queue_manager from bindings file?
I've also tried to connect with host, channel and port:
qmgr = pymqi.connect(queue_manager, channel, conn_info)
And I get an error of not authorized, I think it's because this second way is to connect with the client and I would need user and password which I haven't.

If you want to use binding mode, you should setup pymqi with server or binding parameters. You can not use binding and client mode simultaneously:
#From pymqi folder
cd ./code
./setup.py build server
I am not sure, that you can use .bindings file with pymqi without parsing it himself.

Probably I'm very late to this discussion, but:
import pymqi
queue_manager = 'MQFEND00'
channel = 'PORTALS.MQFEND00'
host = 'somehost'
port = '1414'
conn_info = '%s(%s)' % (host, port)
qmgr = pymqi.connect(queue_manager, channel, conn_info)
# other operations, see https://dsuch.github.io/pymqi/examples.html for more.
qmgr.disconnect()
You must have a MQ Client installed in the same machine that you run pymqi;
Are you using conn_info like the code snnipet?

mqtt connection establishment between different systems using paho-mqtt-python :

I used the following code and I get this error :
Time out error : [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host failed to respond.
I also turned off the firewall in the systems i am using , still i get that error.
#!/usr/bin/env python3
import paho.mqtt.client as mqtt
# This is the Publisher
client = mqtt.Client()
client.connect("10.12.114.103",1883,60)
client.publish("topic/test", "Hello world!");
client.disconnect();

The problem probably isn't with your python code as I tried a slightly modified version with hivemq's public test server and it worked fine. Here is the version I tried:
#!/usr/bin/env python3
import paho.mqtt.client as mqtt
# This is the Publisher
def on_log(client, userdata, level, buf):
print(level, buf)
client = mqtt.Client()
client.on_log = on_log
client.connect("broker.hivemq.com",1883,60)
client.publish("topic/test", "Hello world!");
client.disconnect();
Note that I modified it to return the log output which would be useful in debugging this. The script returned:
16 Sending PUBLISH (dFalse, q0, r0, m1, 'topic/test', ... (12 bytes)
Which is what I would expect from a successful connection. I also checked your code with the mosquitto test server and it worked fine.
It looks like your broker is not accepting your connection attempt. If you tried a public test server and it still didn't work then it suggests that something is interfering with your traffic on port 1883.
I think this is a broker issue, however, I did notice that you are not using any of the network loop functions (like client.loop_start) which, according to the paho documentation, may cause unpredictable behavior. You could try adding a loop like this to see if that helps
client = mqtt.Client()
client.on_log = on_log
client.connect("broker.hivemq.com",1883,60)
client.loop_start
client.publish("topic/test", "Hello world!");
client.disconnect();
client.loop_stop

AWS IoT and Raspberry Pi with paho-mqtt don't connect

I have installed the last version of raspbian on my raspberry pi, and I have opened an account AWS IoT on Amazon, then in the IoT web interface I have created a thing, with "RaspberryPi_2" name and create certificate and connect the certificate to the thing, I have followed this guide:
http://blog.getflint.io/blog/get-started-with-aws-iot-and-raspberry-pi
I have then created the script in the guide, to connect and subscribed the raspberry pi, this is my code:
#!/usr/bin/python3
#required libraries
import sys
import ssl
import paho.mqtt.client as mqtt
#called while client tries to establish connection with the server
def on_connect(mqttc, obj, flags, rc):
if rc==0:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: successful")
elif rc==1:
print ("Subscriber Connection status code: "+str(rc)+" | Connection status: Connection refused")
#called when a topic is successfully subscribed to
def on_subscribe(mqttc, obj, mid, granted_qos):
print("Subscribed: "+str(mid)+" "+str(granted_qos)+"data"+str(obj))
#called when a message is received by a topic
def on_message(mqttc, obj, msg):
print("Received message from topic: "+msg.topic+" | QoS: "+str(msg.qos)+" | Data Received: "+str(msg.payload))
#creating a client with client-id=mqtt-test
mqttc = mqtt.Client(client_id="mqtt-test")
mqttc.on_connect = on_connect
mqttc.on_subscribe = on_subscribe
mqttc.on_message = on_message
#Configure network encryption and authentication options. Enables SSL/TLS support.
#adding client-side certificates and enabling tlsv1.2 support as required by aws-iot service
mqttc.tls_set("/home/pi/aws_iot/things/raspberryPi_2/certs/aws-iot-rootCA.crt",
certfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-certificate.pem.crt",
keyfile="/home/pi/aws_iot/things/raspberryPi_2/certs/0ea2cd7eb6-private.pem.key",
tls_version=ssl.PROTOCOL_TLSv1_2,
ciphers=None)
#connecting to aws-account-specific-iot-endpoint
mqttc.connect("A2GF7W5U5A46J1.iot.us-west-2.amazonaws.com", port=8883) #AWS IoT service hostname and portno
#the topic to publish to
mqttc.subscribe("$aws/things/RaspberryPi_2/shadow/update/#", qos=1) #The names of these topics start with $aws/things/thingName/shadow."
#automatically handles reconnecting
mqttc.loop_forever()
but when I do this command:
python3 mqtt_test.py
or this command:
python mqtt_test.py
and press enter, and the cursor flash and doesn't print anything and stay stuck there, someone can help me?
I haven't also understand if the client-id name should be the same of the things name, and the meaning of the subscribe path, for example in a tutorial I have found this:
mqttc.publish("temperature", tempreading, qos=1)
why there isn't the complete path?
or this:
$aws/things/RaspberryPi_2/shadow/update/delta
so I can put everything I want in the path?
thanks

The code is subscribing to a topic but there is no one publishing to it. So, the code also has a on_connect function that would be triggered after a success connection. Is the message "Subscriber Connection status code: ..." being printed? If it is, the message from on_subscribe should also appear. Is it?
If it is not you have a problem before connect to the AWS server. Use netstat command to see where your Raspberry Pi is connected or not and post more debug information in this case.
If the connect and subscribe messages are shown and nothing happens after it, this is normal because you are only subscribing to a topic but not publishing anything.
Regarding topics, think of them as a directory structure. There is no strict rule for topic hierarchy. A "temperature" topic would be temperature topic on the server and a "temperature/livingroom" would be temperature/livingroom, you can subscribe to one, another or both on the same server. The path you choose for your things shall be meaningful to your application. A house, for instance, might be represented as:
house/kitchen/env/temperature
house/kitchen/env/humidity
house/kitchen/lamp/sinklamp
house/kitchen/lamp/mainlap
house/masterbed/env/temperature
house/masterbed/env/humidity
house/masterbed/lamp/readinglampleft
house/masterbed/lamp/readinglampright
house/masterbed/lamp/mainlamp
house/masterbed/lamp/mirrorlamp
And so on.
Let´s say you have a thermostat at master bedroom. It is interested only in temperature but not humidity. It is also interested only in master bedroom temperature. This thermostat shall subscribe to house/masterbed/env/temperature. Opposite to this, a room wide panel that shows state of every thing in the room, would subscribe to house/masterbed/#, meaning "everything after house/masterbed". Read more on wildcards here
The topic you subscribed for: $aws/things/RaspberryPi_2/shadow/update/# means, "every thing after $aws/things/RaspberryPi_2/shadow/update/". Notice that his is a special topic, it starts with $aws, specially, it starts with $ character. In the AWS context this means:
Any topics beginning with $ are considered reserved and are not
supported for publishing and subscribing except when working with the
Thing Shadows service. For more information, see Thing Shadows.
So you need to understand what thing shadow is. This is a AWS specific (and very util) concept. Please read docs on this subject.
Finally, I would you suggest you install a local broker (mosquitto is available on respbian) and try with it before got to AWS. This way you can master mqtt concept without connectivity issues. Later you put AWS to the mix.

Easiest way to push RabbitMQ events to browser using WebSockets in Python?

I have an existing Python system that receives messages using Rabbit MQ. What is the absolute easiest way to get these events pushed to a browser using WebSockets using Python? Bonus if the solution works in all major browsers too.
Thanks,
Virgil

Here https://github.com/Gsantomaggio/rabbitmqexample I wrote an complete example that uses tornado and RabbitMQ.
You can find all the instruction from the site:
anyway ..you need:
pip install pika
pip install tornado
First you register your rabbitmq:
def threaded_rmq():
channel.queue_declare(queue="my_queue")
logging.info('consumer ready, on my_queue')
channel.basic_consume(consumer_callback, queue="my_queue", no_ack=True)
channel.start_consuming()
then you register your web-sockets clients:
class SocketHandler(tornado.websocket.WebSocketHandler):
def open(self):
logging.info('WebSocket opened')
clients.append(self)
def on_close(self):
logging.info('WebSocket closed')
clients.remove(self)
When you get a message, you can redirect it to the web-socket page.
def consumer_callback(ch, method, properties, body):
logging.info("[x] Received %r" % (body,))
# The messagge is brodcast to the connected clients
for itm in clients:
itm.write_message(body)

You could use Twisted, txAMQP and Autobahn|Python on the server to write a bridge in probably 50 lines of code, and Autobahn|JS on the browser side. Autobahn implements WebSocket, and WAMP on top, which provides you with Publish & Subscribe (as well as Remote Procedure Calls) over WebSocket.
When using raw WebSocket you would have to invent your own Publish & Subscribe over WebSocket - since I guess that is what you are after: extending the AMQP PubSub to the Web. Or you could check out STOMP.
Disclaimer: I am original author of WAMP and Autobahn.