I've a Python script which reads stuff from a stream, and when a new string gets readed, it pushes its content (a string) to a RabbitMQ queue.
The thing is that the stream might not send messages in 1, 2 or 9h or so, so I would like to have the RabbitMQ connection always open.
The problem is that when I create the conection and the channel:
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=self.host, credentials=self.credentials))
channel = self.connection.channel()
channel.exchange_declare(exchange=self.exchange_name, exchange_type='fanout')
... and if after an hour a message arrives, I get this error:
File "/usr/local/lib/python3.7/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/var/opt/rabbitmq-agent.py", line 34, in push_to_queue
raise Exception("Error sending the message to the queue: " + format(e))
Exception: Error sending the message to the queue: Send message to publisher error: Channel allocation requires an open connection: <SelectConnection CLOSED socket=None params=<ConnectionParameters host=x port=xvirtual_host=/ ssl=False>>
Which I suppose is that the connection has been closed between the rabbitmq server and client.
How can I avoid this? I would like to have a "please, keep the connection alive always". Maybe setting a super-big heartbeat in the connection parameters of Pika? Something like this:
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=self.host, credentials=self.credentials, heartbeat=6000))
Any other cooler solutions would be highly appreciated.
Thanks in advance
I would suggest you check connection every time before sending message and if the connection is closed then simply reconnect.
if not self.connection or self.connection.is_closed:
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host=self.host, credentials=self.credentials))
channel = self.connection.channel()
channel.exchange_declare(exchange=self.exchange_name, exchange_type='fanout')
You could try adding heartbeat to your ConnectionParameters. This will create light traffic by sending heartbeats every specified seconds. This will exercise the connections. Some firewalls or proxies tend to scrape idle connections. Even RabbitMQ has a timeout on connections that are idle.
import pika
# Set the connection parameters to connect to rabbit-server1 on port 5672
# on the / virtual host using the username "guest" and password "guest"
credentials = pika.PlainCredentials('guest', 'guest')
parameters = pika.ConnectionParameters('rabbit-server1',
5672,
'/',
heartbeat=60,
credentials)
See here for pika documentation.
Additionally you should have code in place that mitigates network disconnection. This can always happen and will. So appart from the heartbeat have some exception handling ready to re-open closed connections in a graceful way.
Related
Client side:
data = b'\xff' * 1000000
ssock = socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0)
#context is created by ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
ssock = context.wrap_socket(ssock, server_hostname='xd1337sv')
ssock.connect((SERVERADDR, SERVERPORT))
ssock.sendall(data)
#time.sleep(3)
ssock.close()
If I just use regular non-SSL socket, everything works correctly with the server receiving exact amount of data. If I use TLS socket, the behavior then depends on the version.
If I run either the server or client on Python 3.6 and therefore the TLSv1.2 will be used, there's no problem.
Problem arises only when TLSv1.3 is used and depends on the size of data and how soon client ssocket.close() line is executed.
If I put a right amount of time.sleep before ssocket.close() depending on the size of data, then I get no error. Otherwise, the server will get ConnectionResetError [WinError 10054] An existing connection was forcibly closed by the remote host and receive only part of the data, or throw ConnectionAbortedError [WinError 10053] An established connection was aborted by the software in your host machine and receive no data.
I'm testing both the server and client on my local machine with local address 192.168.1.2.
The difference is caused by TLS 1.3 sending a session ticket after the TLS handshake while with previous TLS versions the session ticket is send inside the TLS handshake. Thus, with TLS 1.3 data from the server (the session ticket) will arrive after the ssock.connect(...) is done. Since your application does not read any data after the connect it closes the socket while unread data are still inside the socket buffer of the underlying TCP socket. This will cause RST send to the server and cause there the connection reset error.
This is a known problems with applications which never attempt to read from the server. If the application would expect a response from the server and use recv to get it this would implicitly also read the session ticket.
To fix this situation when you don't expect the server to return any application data do a proper SSL shutdown of the socket before closing it. Since this will read the servers SSL shutdown message it will also implicitly read the session ticket send before by the server.
try:
ssock = ssock.unwrap()
except:
True
ssock.close()
For more information see also this issue and this documentation.
I was getting a similar problem when the application was running through gunicorn with certificates. The jsondecodeerror problem randomly came to the client, i.e. the response was empty. The only thing that TLS 1.2 was used.
The solution was simple, I deployed the application on uwsgi and the problem went away
I am trying to write a producer and consumer code in python using pika for rabbitmq. However for my specific case, I need to run producer on a different host and consumer on other.
I have already written a producer code as:
import pika
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters('ip add of another host', 5672, '/', credentials)
connection = pika.BlockingConnection()
channel = connection.channel()
channel.queue_declare(queue='test')
channel.basic_publish(exchange='', routing_key='test', body='hello all!')
print (" [x] sent 'Hello all!")
connection.close()
The above producer code is running without any error. I also created a new user and gave administrator credentials to it on rabbitmq-server. However when I run the consumer code on another host running rabbitmq-server, I do not see any output:
import pika
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters('localhost', 5672, '/', credentials)
connection = pika.BlockingConnection()
channel = connection.channel()
channel.queue_declare(queue='test')
def callback(ch, method, properties, body):
print(" [x] Recieved %r" % body)
channel.basic_consume(
queue='test', on_message_callback=callback, auto_ack=True)
print (' [x] waiting for messages. To exit press ctrl+c')
channel.start_consume()
So, here i had two hosts on the same network which had rabbitmq installed. However one has 3.7.10 and other had 3.7.16 version of rabbitmq.
The producer is able to send the text without error, but the consumer on another host is not receiving any text.
I do not get any problem when both run on same machine, as i just replace connection settings with localhost. Since user guest is only allowed to connect on localhost by default, i created a new user on consumer host running rabbitmq-server.
Please look if anyone can help me out here...
I have a couple of questions when I see your problem:
Are you 100% sure that on your RabbitMQ management monitoring
you see 2 connections? One from your local host and another from the another host? This will help to debug
Second, Did you check that your ongoing port 5672 on the server that host RabbitMQ is open? Because maybe your producer does not manage to connect What is your cloud provider?
If you don't want to manage those kinds of issues, you should use a service like https://zenaton.com. They host everything for you, and you have integrated monitoring, error handling etc.
Your consumer and producer applications must connect to the same RabbitMQ server. If you have two instances of RabbitMQ running they are independent. Messages do not move from one instance of RabbitMQ to another unless you configure Shovel or Federation.
NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.
You don't seem to be passing the parameters to the BlockingConnection instance.
import pika
rmq_server = "ip_address_of_rmq_server"
credentials = pika.PlainCredentials('username', 'password')
parameters = pika.ConnectionParameters(rmq_server, 5672, '/', credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
Also, your consumer is attaching to the localhost hostname. Make sure this actually resolves and that your RabbitMQ service is listening on the localhost address (127.0.0.1) It may not be bound to that address. I believe that RMQ will bind to all interfaces (and thus all addresses) by default but I'm not sure.
So I have a publisher which is using schedule python package to read data from a file and every 5-10 mins and publish each line to a queue.
On the other side I have consumer using something like:
self.connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
self.channel = self.connection.channel()
while True:
method, properties, body = self.channel.basic_get(queue=conf.UNIVERSAL_MESSAGE_QUEUE, no_ack=False)
if body is not None:
self.assign_task(body=body)
self.channel.basic_ack(delivery_tag=method.delivery_tag)
else:
self.logger.info('channel empty')
self.move_to_done()
time.sleep(5)
Assign task function looks like:
def assign_task(body=body):
<do something with the message body>
For some reason after a while it throws the following error:
2017-08-03 15:27:43,756: ERROR: base_connection.py: _handle_error: 335: Socket Error: 10054
2017-08-03 15:27:43,756: WARNING: base_connection.py: _check_state_on_disconnect: 180: Socket closed when connection was open
2017-08-03 15:27:43,756: WARNING: connection.py: _on_disconnect: 1360: Disconnected from RabbitMQ at localhost:5672 (0): Not specified
Essentially both publisher and consumer are 2 different python programs intended to run on a single machine with Windows Server 2012. Can community help understand what might be going wrong here.
The same code runs absolutely fine locally on my windows machine
Following is the output from my log file.
=ERROR REPORT==== 3-Aug-2017::15:06:48 ===
closing AMQP connection <0.617.0> ([::1]:53485 -> [::1]:5672):
missed heartbeats from client, timeout: 60s
Simple answer to this was to create a durable queue and set heartbeat_interval to 0.
I am trying to connect to my remote rabbitmq using pika but I am getting Connectionclosed() error. I have made the required changes in rabbit.config for guest user to allow all connections and also the same connection works from my Java code. I even tried creating a new user with all the permission and connecting it, but it still doesn't work. The same code works fine on my localhost though. Can anyone please let me know what might I be doing wrong here?
def queue_message(message, queue):
credentials = pika.PlainCredentials('xxxx', 'xxxx')
parameters = pika.ConnectionParameters('remote-server',
5672,
'/',
credentials)
connection = pika.BlockingConnection(parameters)
channel = connection.channel()
channel.queue_declare(queue='python_update_queue')
channel.basic_publish(exchange='update.fanout',
body=message)
logger.info("Sent message: {} to queue: {}".format(message, queue))
print 'message sent'
connection.close()
Below is the error I get:
app/project/rabbitmq.py" in queue_message
connection = pika.BlockingConnection(parameters)
env/lib/python2.7/site-packages/pika/adapters/blocking_connection.py" in __init__
self._process_io_for_connection_setup()
env/lib/python2.7/site-packages/pika/adapters/blocking_connection.py" in ss_io_for_connection_setup
self._open_error_result.is_ready)
env/lib/python2.7/site-packages/pika/adapters/blocking_connection.py" in _flush_output
raise exceptions.ConnectionClosed
add a connection timeout to your connection parameters - you're probably running into a timeout issue where the connection isn't happening fast enough, across the network.
also, your code is explicitly calling connection.close() ... so that may be why your connection is closing
It was indeed a timeout issue. After increasing the timeout in the connection parameters, the connection was established properly.
parameters = pika.ConnectionParameters('remote-server',
5672,
'/',
socket_timeout=2)
If you connect to remote rabbitmq server, check this:
remote server port open with firewall
remote server have public ip and rabbitmq user have access to that server
rabbitmq server is activately running
add your user admin in administrator tag;
rabbitmqctl set_user_tags admin administrator
add enough permissions to the user admin
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
I have program which has servers interacting with each other using Twisted's remote procedure calls, and I run in problems with closing connections when they are not needed anymore. Connections should be able to close itself in both sides.
Case 1: How do I close connection in connecting part?
factory = pb.PBClientFactory()
reactor.connectTCP(ip, port, factory)
deferred = factory.login(credentials.UsernamePassword(username, password), client=self)
deferred.addCallbacks(self.connectedToServer, self.errorConnectingToServer)
def connectedToServer(self, server):
self.server = server
# Closing connection comes here
Case 2: How do I close connection in server part?
class MyPerspective(pb.Avatar):
def connected(self, server):
self.client = server
# Closing connection comes here
At the moment I use raising pb.Error() to close connection, but I don't think that's the proper way to do it.
Another option is reference.broker.transport.loseConnection().
RemoteReference instances which are created over a PB connection are given a broker attribute. The broker attribute refers to the protocol instance that created them. As usual for a protocol, the broker has a transport attribute, and the transport has a loseConnection method.