We Keep Coding

Not able to connect to AWS Active MQ broker

We are trying to connect to AWS Amazon MQ using python but facing an issue.
Please find below code and error.
code:
import stomp
#Establish a connection
con = stomp.Connection([('stomp+ssl://xxxxxxxxxxxxxxxxxx.mq.xxxxxxxxxxxxxxxx.amazonaws.com',61616)])
#listener class to be instantiated.
class Listener(stomp.ConnectionListener):
def on_error(self, headers, message):
print('received an error "%s"' % message)
def on_message(self, headers, message):
print('received a message "%s"' % message)
con.set_listener('', Listener())
#wait will ensure it waits till connection is established and acknowledged.
# con.start()
con.connect('xxxxxxxx', 'xxxxxxxxx', wait=True)
#subscribe to a particular topic or queue by giving the path and headers if required by the server.
con.subscribe('#', headers={})
Error:
Could not connect to host stomp+ssl://xxxxxxxxxxxxxxxxxxxx.mq.xxxxxxx.amazonaws.com, port 61616
Could not connect to host stomp+ssl://xxxxxxxxxxxxxxxxxxxx.mq.xxxxxxx.amazonaws.com, port 61616
Could not connect to host stomp+ssl://xxxxxxxxxxxxxxxxxxxx.mq.xxxxxxx.amazonaws.com, port 61616
Traceback (most recent call last):
File "stomp_mqtt_subscribe.py", line 34, in <module>
con.connect('xxxxxxxx', 'xxxxxxxx', wait=True)
File "/home/ubuntu/.local/lib/python3.8/site-packages/stomp/connect.py", line 150, in connect
self.transport.start()
File "/home/ubuntu/.local/lib/python3.8/site-packages/stomp/transport.py", line 130, in start
self.attempt_connection()
File "/home/ubuntu/.local/lib/python3.8/site-packages/stomp/transport.py", line 834, in attempt_connection
raise exception.ConnectFailedException()
stomp.exception.ConnectFailedException
Any help appreciated.

This is what I use with stand-alone ActiveMQ Artemis, not AWS. I also filter message subscription on a selector field called "key". Remove that selector from the headers if you want. I'm using 30 second heartbeats here.
import stomp
def connect_and_subscribe(conn):
conn.connect('username', 'p#$$w0rd', wait=True)
conn.subscribe(destination='my-topic', id=12345, ack='auto', headers={'subscription-type': 'MULTICAST', 'selector': "key like '"+key+"'"})
class MyListener(stomp.ConnectionListener):
def __init__(self):
self.counter=0
def on_message(self, frame):
self.counter+=1
print('counter', self.counter)
print('got:', frame)
def on_disconnected(self):
print('disconnected')
connect_and_subscribe(self.conn)
key='test'
hosts = [('mqserver', 61616)]
conn = stomp.Connection(host_and_ports=hosts, heartbeats=(30000, 30000))
conn.set_listener('', MyListener())
connect_and_subscribe(conn)
while True:
pass
conn.disconnect()

There's a handful of problems (and potential problems) with your code.
First, you're using this as the hostname:
stomp+ssl://xxxxxxxxxxxxxxxxxx.mq.xxxxxxxxxxxxxxxx.amazonaws.com
The stomp+ssl:// should not be included in the hostname.
Second, you're using this as the port:
61616
By default the broker will not accept STOMP connection on port 61616. You should likely use 61613 instead.
Third, you're subscribing to # which looks like something you might do with an MQTT client (although even that is discouraged with MQTT). The destination # has no special meaning in STOMP as the STOMP specification defines no wildcards. Your STOMP client will subscribe to a destination literally named # which is likely not what you want.

Rabbitmq error: [Errno 10054] An existing connection was forcibly closed by the remote host

I am using Kombu in Python to consume a durable RabbitMQ queue.
There is only one consumer consuming the queue in Windows. This consumer produces the below error:
Traceback (most recent call last):
File ".\consumer_windows.py", line 66, in <module>
message.ack()
File "C:\Users\Administrator\Anaconda2\lib\site-packages\kombu\message.py", line 88, in ack
self.channel.basic_ack(self.delivery_tag)
File "C:\Users\Administrator\Anaconda2\lib\site-packages\amqp\channel.py", line 1584, in basic_ack
self._send_method((60, 80), args)
File "C:\Users\Administrator\Anaconda2\lib\site-packages\amqp\abstract_channel.py", line 56, in _send_method
self.channel_id, method_sig, args, content,
File "C:\Users\Administrator\Anaconda2\lib\site-packages\amqp\method_framing.py", line 221, in write_method
write_frame(1, channel, payload)
File "C:\Users\Administrator\Anaconda2\lib\site-packages\amqp\transport.py", line 182, in write_frame
frame_type, channel, size, payload, 0xce,
File "C:\Users\Administrator\Anaconda2\lib\socket.py", line 228, in meth
return getattr(self._sock,name)(*args)
error: [Errno 10054] An existing connection was forcibly closed by the remote host
There are at most 500 messages in the queue at any one time. Each message is small in size however it is a task and takes up to 10 minutes to complete (although it usually takes less then 5 mins per message).
I have tried restarting the consumer, RabbitMQ server and deleting the queue however the error still persists.
I've seen this question however the answer is from 2010 and my rabbitmq.log has different entries:
=ERROR REPORT==== 24-Apr-2016::08:26:20 ===
closing AMQP connection <0.6716.384> (192.168.X.X:59602 -> 192.168.Y.X:5672):
{writer,send_failed,{error,timeout}}
There were no recent events in the rabbitmq-sasl.log.
Why is this error happening and how can I prevent it from occurring?

I'm still looking for an answer. In the meantime I restart the connection to my rabbit server:
while True:
try:
​
connection = pika.BlockingConnection(params)
channel = connection.channel() # start a channel
channel.queue_declare(queue=amqp_q, durable=True) # Declare a queue
...
​
except pika.exceptions.ConnectionClosed:
print('connection closed... and restarted')

I had the same issue with MySQL server which was hosted...
I came to understand that it happened if we open the connection for a long time or unmodified for a long time..
If your program opens the DB or anything until the whole program runs make it in a such a way that it opens the DB writes everything and closes and repeats
I don't know what exactly rabbitmq is but I think the error you wrote as title may be for this reason

I had the same error (using pure PIKA library) and trying to connect to a Rabbitmq broker through Amazon MQ.
The problem resolved when setting up correctly the ssl configuration.
Please check full blog post here: https://docs.aws.amazon.com/amazon-mq/latest/developer-guide/amazon-mq-rabbitmq-pika.html
Core snippets that I used:
Define Pika Client:
import ssl
import pika
class BasicPikaClient:
def __init__(self, rabbitmq_broker_id, rabbitmq_user, rabbitmq_password, region):
# SSL Context for TLS configuration of Amazon MQ for RabbitMQ
ssl_context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2)
ssl_context.set_ciphers('ECDHE+AESGCM:!ECDSA')
url = f"amqps://{rabbitmq_user}:{rabbitmq_password}#{rabbitmq_broker_id}.mq.{region}.amazonaws.com:5671"
parameters = pika.URLParameters(url)
parameters.ssl_options = pika.SSLOptions(context=ssl_context)
self.connection = pika.BlockingConnection(parameters)
self.channel = self.connection.channel()
Producer:
from basicClient import BasicPikaClient
class BasicMessageSender(BasicPikaClient):
def declare_queue(self, queue_name, durable):
print(f"Trying to declare queue({queue_name})...")
self.channel.queue_declare(queue=queue_name, durable=durable)
def send_message(self, exchange, routing_key, body):
channel = self.connection.channel()
channel.basic_publish(exchange=exchange,
routing_key=routing_key,
body=body)
print(f"Sent message. Exchange: {exchange}, Routing Key: {routing_key}, Body: {body}")
def close(self):
self.channel.close()
self.connection.close()
Calling Producer:
# Initialize Basic Message Sender which creates a connection
# and channel for sending messages.
basic_message_sender = BasicMessageSender(
credentials["broker_id"],
credentials["username"],
credentials['password'],
credentials['region']
)
# Declare a queue
basic_message_sender.declare_queue("q_name", durable=True)
# Send a message to the queue.
basic_message_sender.send_message(exchange="", routing_key="q_name", body=b'Hello World 2!')
# Close connections.
basic_message_sender.close()
Define Consumer:
class BasicMessageReceiver(BasicPikaClient):
def get_message(self, queue):
method_frame, header_frame, body = self.channel.basic_get(queue)
if method_frame:
print(method_frame, header_frame, body)
self.channel.basic_ack(method_frame.delivery_tag)
return method_frame, header_frame, body
else:
print('No message returned')
def close(self):
self.channel.close()
self.connection.close()
Calling Consumer:
# Create Basic Message Receiver which creates a connection
# and channel for consuming messages.
basic_message_receiver = BasicMessageReceiver(
credentials["broker_id"],
credentials["username"],
credentials['password'],
credentials['region']
)
# Consume the message that was sent.
basic_message_receiver.get_message("q_name")
# Close connections.
basic_message_receiver.close()
I hope the above helps.
Thanks

Create linux daemon which uses Multiprocessing and Multiprocessing.Queues

I have a task to listen UDP datagrams, decode them(datagrams have binary information), decoded information put in dictionary, dump dictionary to json string and then send json string to remote server(ActiveMQ).
Both decoding and sending to remote could be time consuming. In order to make program more scalable we create two processes (Multiprocessing.Process):
Listner(listen datagrams, analize, create json and put it in Multiprocessing.Queue)
Sender(constantly tries to get a json string from the queue to array, if length of array becomes above threshold - send all collected strings to remote server)
Now I need to make from it a proper linux daemon (which could be start, stop and restart via service command).
The question: How to make a daemon from python multiprocessing program. I have found no guide about this. Does anybody know how to do this, or have working example.
The following text is my attemts to acheve this:
I found small example of python daemon: http://www.gavinj.net/2012/06/building-python-daemon-process.html
so I rewrited my code (sorry for big code):
import socket
import time
import os
from select import select
import multiprocessing
from multiprocessing import Process, Queue, Value
import stomp
import json
import logging
logger = logging.getLogger("DaemonLog")
logger.setLevel(logging.INFO)
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler = logging.FileHandler("/var/log/testdaemon/testdaemon.log")
handler.setFormatter(formatter)
logger.addHandler(handler)
log = logger
#Config listner
domain = 'example.host.ru'
port = int(9930)
#Config remote queue access
queue_cfg = {
'host': 'queue.test.ru',
'port': 61113,
'user': 'user',
'password': 'pass',
'queue': '/topic/test.queue'
}
class UDPListener():
def __init__(self, domain, port, queue_cfg):
# If I initialize socket during init I see strange error:
# on the line: data, addr = sock_inst.recvfrom(int(10000))
# error: [Errno 88] Socket operation on non-socket
# So I put initialization to runListner function
#self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#self.sock.bind((domain, port))
self.domain = domain
self.port = port
self.remote_queue_cfg = queue_cfg
self.queue = Queue()
self.isWorking = Value('b', True)
self.decoder = Decoder()
self.reactor = ParallelQueueReactor(self.queue)
self.stdin_path = '/dev/null'
self.stdout_path = '/dev/tty'
self.stderr_path = '/dev/tty'
self.pidfile_path = '/var/run/testdaemon/testdaemon.pid'
self.pidfile_timeout = 5
def __assignData(self, addr, data):
receive_time = time.time()
messages = self.decoder.decode(receive_time, addr, data)
for msg in messages:
self.reactor.addMessage(msg)
def runListner(self):
self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.sock.bind((domain, port))
while self.isWorking.value:
inputready, outputready, exceptready = select([self.sock], [], [])
for sock_inst in inputready:
if sock_inst == self.sock:
data, addr = sock_inst.recvfrom(int(10000))
if data:
self.__assignData(addr[0], data)
self.sock.close()
def runQueueDispatcher(self):
while self.isWorking.value:
connected = False
while not connected:
try:
conn = stomp.Connection(host_and_ports=[(self.remote_queue_cfg['host'], self.remote_queue_cfg['port'])])
conn.start()
conn.connect(self.remote_queue_cfg['user'], self.remote_queue_cfg['password'], wait=True)
connected = True
except socket.error:
log.error('Could not connect to activemq server.')
time.sleep(20)
if connected == True:
while self.isWorking.value:
msg = None
if not self.queue.empty():
#Now error appear hear even when not self.queue.empty()
msg = self.queue.get()
else:
time.sleep(1)
if msg is not None:
try:
data = json.dumps(msg)
conn.send(body=data, destination=self.remote_queue_cfg['queue'])
count += 1
except:
log.error('Failed to send message to queue.')
time.sleep(1)
def stop(self):
self.isWorking.value = False
def run(self):
log.error('StartProcesses')
dispatcher_process = Process(target=self.runQueueDispatcher, name='Dispatcher')
listner_process = Process(target=self.runListner, name='Listner')
dispatcher_process.start()
listner_process.start()
dispatcher_process.join()
listner_process.join()
log.info('Finished')
#------------------------------------------------------------------
def main():
from daemon import runner
app = UDPListener(domain, port, queue_cfg)
daemon_runner = runner.DaemonRunner(app)
daemon_runner.daemon_context.files_preserve=[handler.stream]
daemon_runner.do_action()
if __name__ == "__main__":
main()
Now I see error on msg = self.queue.get()
Traceback (most recent call last): File "/usr/lib64/python2.6/multiprocessing/process.py", line 232, in
_bootstrap
self.run() File "/usr/lib64/python2.6/multiprocessing/process.py", line 88, in run
self._target(*self._args, **self._kwargs) File "/root/ipelevan/dream/src/parallel_main.py", line 116, in runQueueDispatcher
msg = self.queue.get() File "/usr/lib64/python2.6/multiprocessing/queues.py", line 91, in get
res = self._recv() EOFError
I did not see this errors when run UDPListner.run() manually. But with daemon runner it looks like new instances of UDPListner is created underneath and in different processes we have different Queue(and different self.socket too, when it initialized in init).

First of all: It was a bad idea to keep links to shared objects(Queue, Value) as members of class for the purpose of using by processes. It was working somehow without demonization. But when the same code was run in DaemonContext the os.fork() happened and somehow messed up links to objects. I am not quite sure if Multiprocessing module was designed to work 100% correctly inside a method of an object.
Second: DaemonContext helps to detach process from shell, redirect streams and do several other things related to daemon processes but I have not found any good way to check if such a daemon is already running. So I just used
if os.path.isfile(pidfile_path):
print 'pidfile %s exists. Already running?' % pidfile_path
sys.exit(1)

Send facebook messages via SleekXMPP

I am trying to send a message on facebook chat with sleekXMPP, using the answer from here as a boilerplate: Send a Facebook Message with XMPP using Access Tokens in Python
My code is
import sleekxmpp
class SendMsgBot(sleekxmpp.ClientXMPP):
def init(self, jid, recipient, message):
print "..."
sleekxmpp.ClientXMPP.__init__(self, jid, 'ignore')
self.recipient = recipient
self.msg = message
self.add_event_handler("session_start", self.start, threaded=True)
def start(self, event):
self.send_presence()
self.get_roster()
self.send_message(mto=self.recipient, mbody=self.msg, mtype='chat')
self.disconnect(wait=True)
if __name__ == "__main__":
xmpp = SendMsgBot(from_id, to_id, unicode(message))
xmpp.credentials['apikey'] = api_key
xmpp.credentials['accesstoken'] = o_auth_token
if xmpp.connect(('chat.facebook.com', 5222)):
xmpp.process(block=True)
print("Done")
else:
print("Unable to connect")
However, when I run the script I get this error message:
Traceback (most recent call last):
File "sendMessagesScript.py", line 33, in <module>
xmpp = SendMsgBot(from_id, to_id, unicode(message))
File "/Library/Python/2.7/site-packages/sleekxmpp/clientxmpp.py", line 112, in __init__
self.register_plugin('feature_starttls')
File "/Library/Python/2.7/site-packages/sleekxmpp/basexmpp.py", line 264, in register_plugin
pconfig = self.plugin_config.get(plugin, {})
AttributeError: 'unicode' object has no attribute 'get'
Any ideas would be appreciated!

In the class SendMsgBot(sleekxmpp.ClientXMPP):, you need to change
def init(self, jid, recipient, message) to def __init__(self, jid, recipient, message)
I hope it will work.

Additionally, it seems that some important dashes have been ommitted from the original code.
I also had to change
xmpp.credentials['apikey'] = api_key
xmpp.credentials['accesstoken'] = o_auth_token
to
xmpp.credentials['api_key'] = api_key
xmpp.credentials['access_token'] = o_auth_token
These are apparently the parameter names that Facebook expects, as you can see in Facebook's PHP example

Twisted - SleekXMPP compatibility

I am making a XMPP middleware in python which listens on a an address(host,port) and when it receives some connection on that port, it sends a XMPP message to a jid(XMPP user) on a server.
A quick review of my setup
For networking part I am using twisted
For XMPP - SleekXMPP
XMPP server - Openfire
Now when I tried using sleekXMPP without importing anything from twistedm it was working fine.
However I i try to mix sleekXMPP and twisted in one program (by importing them) I get the follwing error.
Traceback (most recent call last):
File "sleekXMPP/prog_3.py", line 124, in <module>
main()
File "sleekXMPP/prog_3.py", line 111, in main
xmppThread = ClientThread(dir_q)
File "sleekXMPP/prog_3.py", line 41, in __init__
self.xmpp = xmppClient(self.jid, self.password)
File "sleekXMPP/prog_3.py", line 17, in __init__
sleekxmpp.ClientXMPP.__init__(self, jid, password)
File "build\bdist.win32\egg\sleekxmpp\clientxmpp.py", line 65, in __init__
File "build\bdist.win32\egg\sleekxmpp\basexmpp.py", line 72, in __init__
File "build\bdist.win32\egg\sleekxmpp\jid.py", line 461, in __init__
File "build\bdist.win32\egg\sleekxmpp\jid.py", line 150, in _parse_jid
File "build\bdist.win32\egg\sleekxmpp\jid.py", line 202, in _validate_domain
File "C:\Python27\lib\site-packages\twisted-12.2.0-py2.7-win32.egg\twisted\python \compat.py", line 22, in inet_pton
raise ValueError("Illegal characters: %r" % (''.join(x),))
ValueError: Illegal characters: u't'
The code is as follows:
import sleekxmpp
import ssl
import Queue
import threading
import time
import logging
import traceback
import sys
from twisted.internet import reactor, protocol , endpoints
class xmppClient(sleekxmpp.ClientXMPP):
"""
This class defines the xmppClient object used to interact with the XMPP server
"""
def __init__(self, jid, password):
# the constructor
sleekxmpp.ClientXMPP.__init__(self, jid, password)
self.add_event_handler('session_start', self.start)
def start(self, event):
self.send_presence()
self.get_roster()
def send_note(self):
self.mssg = r"Hello from XMPP Service"
self.recipient = r"testuser2#ghost"
self.send_message(mto=self.recipient,
mbody=self.mssg,
mtype='chat')
print "Message sent"
class ClientThread(threading.Thread):
def __init__(self, dir_q):
super(ClientThread, self).__init__()
self.dir_q = dir_q
self.stoprequest = threading.Event()
self.jid = 'testuser1#ghost'
self.password = 'password'
self.xmpp = xmppClient(self.jid, self.password)
self.xmpp.register_plugin('xep_0030') # Service Discovery
self.xmpp.register_plugin('xep_0004') # Data Forms
self.xmpp.register_plugin('xep_0060') # PubSub
self.xmpp.register_plugin('xep_0199') # XMPP Ping
self.xmpp.ssl_version = ssl.PROTOCOL_SSLv3
if self.xmpp.connect():
print("Connected")
self.xmpp.process(block=False)
else:
print("Unable to connect.")
def run(self):
while not self.stoprequest.isSet():
try:
req = self.dir_q.get(True, 0.05)
if req == 1:
self.xmpp.send_note()
except Queue.Empty:
continue
def join(self, timeout=None):
self.stoprequest.set()
super(ClientThread, self).join(timeout)
class reqSimulator(threading.Thread):
def __init__(self, dir_q):
super(reqSimulator, self).__init__()
self.dir_q = dir_q
def run(self):
while(1):
self.dir_q.put(1)
time.sleep(0.5)
"""class sendProtocol(protocol.Protocol):
def connectionMade(self):
r = reqSimulator(self.factory.dir_q)
r.run()
def connectionLost(self, reason):
pass
def dataReceived(self, data):
self.transport.loseConnection()"""
def main():
logging.basicConfig()
dir_q = Queue.Queue()
xmppThread = ClientThread(dir_q)
xmppThread.start()
sim = reqSimulator(dir_q)
"""factory = protocol.ServerFactory()
factory.dir_q = dir_q
factory.protocol = sendProtocol
endpoints.serverFromString(reactor, "tcp:8001").listen(factory)
reactor.run()
"""
sim.start()
if __name__ == "__main__":
main()
Note that in this code, the actual networking code is commented and I am using a reqSimulator class to simulate the oncoming requests.
I tried to google for any issued but got no result. Does anybody have an idea what is wrong here?

This is due to the fact that Twisted implements inet_pton for Windows, but uses different exceptions for failures than the stdlib version.
I've fixed Sleek on github for this (master and develop branches), and there will be a new point release for it shortly.