I have 2 rabbitmq in cluster load balanced by an Azure Internal Load Balancer. The client connect to the LB with BlockingConnection.
When there a are message exchanged by the clients, everything works fine. But when there is no activity, it seems my clients are disconnected and and not able to receive messages anymore.
I am wondering if there is a solution to this problem ? I assume the load balancer or rabbitmq is closing the connection due to inactivity. I wanted to let pika trigger some heartbeat to rabbitmq (so that the load balancer keep the connection opened) but didn't find any good solution. Could you advice ?
EDIT 1
It seems pika BlockingConnections does not support heartbeat. Heart beat disables blocking connection
Thank you.
According to the Pika doc http://pika.readthedocs.org/en/0.10.0/modules/parameters.html, it seems Pika BlockingConnentions with URLParameters specified heart_interval(such as amqps://www-data:rabbit_pwd#rabbit1/web_messages?heartbeat_interval=30) that can keep connection opened, but the value of heart_interval can not larger than the value of rabbit server suggested.
Heartbeat Timeout Interval
The heartbeat timeout value defines after what period of time the peer TCP connection should be considered dead by RabbitMQ and client libraries. This value is negotiated between the client and RabbitMQ server at the time of connection. The client must be configured to request heartbeats. In RabbitMQ versions 3.0 and higher, the broker will attempt to negotiate heartbeats by default (although the client can still veto them). The timeout is in seconds, and default value is 60 (580 prior to release 3.5.5).
Heartbeat frames are sent about every timeout / 2 seconds. After two missed heartbeats, the peer is considered to be unreachable. Different clients manifest this differently but the TCP connection will be closed. When a client detects that RabbitMQ node is unreachable due to a heartbeat, it needs to re-connect.
Heartbeats can be disabled by setting the timeout interval to 0.
Sample code from Pika doc:
import pika
parameters = pika.URLParameters('amqps://www-data:rabbit_pwd#rabbit1/web_messages?heartbeat_interval=30')
connection = pika.BlockingConnection(parameters)
Related
I have a program that opens an HTTPS connection with keep-alive, performs an HTTP request, does something else and then tries to perform a second HTTP request using the same connection.
If more than 5 minutes have elapsed, the server closes the connection by sending TLS Alert and the connection moves to CLOSE_WAIT state. The HTTPS client detects that the connection is bad (it uses poll to check) and opens a new connection instead.
On a few different computers, this works every time.
In two environments it fails every time. After the server has forgotten about the connection, the client still thinks it is in ESTABLISHED state and the client sends the second HTTP request using the same socket and waits for the ACK. The ACK never comes, the client re-transmits and eventually the client throws a timeout exception. The client never received any indication from the server that the connection is closed. I've checked using Wireshark.
The client is written in python (using a session object from the requests library) and runs on Linux in a cloud VM and the server is AWS API Gateway, but I don't know if it matters.
What could make the server not send anything when closing the connection, or alternatively what could make the client not receive the message from the server about the connection closing?
I have a basic implementation of a TCP client using python sockets, all the client does is connect to a server and send heartbeats every X seconds. The problem is that I don't want to send the server a heartbeat if the connection is closed, but I'm not sure how to detect this situation without actually sending a heartbeat and catch an exception. When I turn off the server, in the traffic capture I see FIN/ACK arriving and the client sends an ACK back, this is when I want my code to do something (or at least change some internal state of the connection). Currently, what happens is after the server went down and X seconds passed since last heartbeat the client will try to send another heartbeat, only then I see RST packet in the traffic capture and get an exception of broken pipe (errno 32). Clearly python socket handles the transport layer and the heartbeats are part of application layer, the problem I want to solve is not to send the redundant heartbeat after FIN/ACK arrived from server, any simple way to know the connection state with python socket?
I'm working in a NetHack clone that is supposed to be playing through Telnet, like many NetHack servers. As I've said, this is a clone, so it's being written from scratch, on Python.
I've set up my socket server reusing code from a SMTP server I wrote a while ago and all of suddenly my attention jumped to this particular line of code:
s.listen(15)
My server was designed to be able to connect to 15 simultaneous clients just in case the data exchange with any took too long, but ideally listen(1) or listen(2) would be enough. But this case is different.
As it happens with Alt.org when you telnet their NetHack servers, people connected to my server should be able to play my roguelike remotely, through a single telnet session, so I guess this connection should not be interrupted. Yet, I've read here that
[...] if you are really holding more than 128 queued connect requests you are
a) taking too long to process them or b) need a heavy-weight
distributed server or c) suffering a DDoS attack.
What is the better practice to carry out here? Should I keep every connection open until the connected user disconnects or is there any other way? Should I go for listen(128) (or whatever my system's socket.SOMAXCONN is) or is that a bad practice?
number in listen(number) request limits number of pending connect requests.
Connect request is pending from initial SYN request received by OS until you called accept socket method. So number does not limits open (established) connection number but it limits number of connections in SYN_RECV state.
It is bad idea not to answer on incoming connection because:
Client will retransmit SYN requests until answer SYN is received
Client can not distinguish situation when your server is not available and it just in queue.
Better idea is to answer on connection but send some message to client with rejection reason and then close connection.
Currently developing something like "smart home" and I have few different devices in my home. All of them connected to OpenHab via MQTT. I'm using Paho MQTT library (Python) for my purposes.
Generally, MQTT has "keepalive" property. This property describes how much time my client will be connected (AFAIK it sends the ping to the server) to MQTT server when there are no updates on the subscribed topic.
But here I have a huge problem. Needed topic could be updated once per hour or even once per few days/months. Let's say that this is indoor alarm.
How can I avoid that keepalive timeout or ignore that field? Could it be unlimited?
You have miss understood what the keepalive value represents.
MQTT clients can stay connected indefinitely even if they do not publish or receive any messages. But the broker needs to keep track of which clients are still connected so it knows when to send the Last Will and Testament (LWT) message for the client. In order to do this it uses the keepalive time.
Every time a message is sent or received by the client, the broker resets a timer, if this timer exceeds 1.5 times the value of the keepalive time then the broker marks the client as disconnected and processes the LWT. To prevent clients with very low messages rates from being disconnected, such a client can send a PINGREQ packet at any time (most likely on timeout of the keepalive value) to the server/broker. The server receives the PINGREQ, answers with a PINGRESP packet and it will reset the keepalive timer to zero and leave the client in the connected state.
See Keep Alive section of the MQTT standard: (http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc385349238)
The Client can send PINGREQ at any time, irrespective of the Keep Alive value, and use the PINGRESP to determine that the network and the Server are working. If the Keep Alive value is non-zero and the Server does not receive a Control Packet from the Client within one and a half times the Keep Alive time period, it MUST disconnect the Network Connection to the Client as if the network had failed
When sending the initial MQTT CONNECT message from a client, you can supply an optional "keep-alive" value. This value is a time interval, measured in seconds, during which the broker expects a client to send a message, such as a PUBLISH message. If no message is sent from the client to the broker during the interval, the broker automatically closes the connection. Note that the keep-alive value you specify is multiplied by 1.5, so setting a 10-minute keep-alive actually results in a 15 minute interval.
Have a look at the Keep Alive section of the MQTT specification:
A Keep Alive value of 0 has the effect of turning off the Keep Alive mechanism. If Keep Alive is 0 the Client is not obliged to send MQTT Control Packets on any particular schedule. v5 spec source
Therefore, set the keep alive to 0, and then the client doesn't have to send a keep alive signal as often. The server should respect that this connection with a client (e.g. from last year) should still be connected, but it won't be guaranteed (The client might be disconnected when the server is shut down).
I'm writing a script which receives HTTP requests (using Tornado), parses them, and sends them to a RabbitMQ broker using pika.
The code looks like this:
def main():
conn_params = pika.ConnectionParameters(
host=BROKER_NAME,
port=BROKER_PORT,
ssl=True,
virtual_host=VIRTUAL_HOST,
credentials=pika.PlainCredentials(BROKER_USER, BROKER_PASS),
heartbeat_interval=HEARTBEAT_INTERVAL
)
conn = pika.BlockingConnection(conn_params)
channel = conn.channel()
# Create the web server which handles application requests.
application = tornado.web.Application([
(URL_BILLING, SomeHandler, dict(channel=channel))
])
# Start the server
application.listen(LISTENING_PORT)
tornado.ioloop.IOLoop.instance().start()
As you can see, I open a single connection and channel, and pass the channel to any instance of the handler which is created, the idea being to save traffic and avoid opening a new connection/channel for every request.
The issue I'm experiencing is that the connection is closed after 3 heartbeats. I used Wireshark in order to figure out what the problem is, but all I can see is that the server sends a PSH (I'm assuming this is the heartbeat) and my scripts replies with an ACK. This happens 3 times with HEARTBEAT_INTERVAL in between them, and then the server just sends a FIN and the connection dies.
Any idea why that happens? Also, should I keep the connection open or is it better to create a new one for every message I need to send?
Thanks for the help.
UPDATE: I looked in the RabbitMQ log, and it says:
Missed heartbeats from client, timeout: 10s
I thought the server was meant to send heartbeats to the client, to make sure it answers, and this agrees with what I observed using Wireshark, but from this log it seems it is the client which is meant to report to the server, not the other way around, and the client, evidently, doesn't report. Am I getting this right?
UPDATE: Figured it out, sort of. A blocking connection (which is what I used) is unable to send heartbeats because it's, well, blocking. As mentioned in this issue, the heartbeat_interval parameters is only used to negotiate the connection with the server, but the client doesn't actually send heartbeats. Since this is the case, what is the best way to keep a long-running connection with pika? Even if I don't specify heartbeat_interval, the server defaults to a heartbeat every 10 minutes, so the connection will die after 30 minutes...
For future visitors:
Pika has an async example which uses heartbeat:
http://pika.readthedocs.org/en/0.10.0/examples/asynchronous_publisher_example.html
For Tornado specific, this example shows how to use Tornado's IOLoop in pika's async model:
http://pika.readthedocs.org/en/0.10.0/examples/tornado_consumer.html