I created a session using the following script:
import dolphindb as ddb
s = ddb.session()
s.connect("localhost", 8848)
The session often got disconnected when interacting with the DolphinDB server. How can I keep the session alive for a specified period of time?
In usual circumstances, sessions created with ddb.session will not get disconnected.
DolphinDB Python API provides the following parameters to keep sessions alive in case of network error:
(1) keepAliveTime and reconnect of the connect function
keepAliveTime - specifies the duration between two keepalive transmissions to detect the TCP connection status. Set the parameter to release half-open TCP connections timely when the network is unstable. The default value is 30 (seconds). For example, when you set keepAliveTime to 50, the session would not got disconnected if the network was down for 30 seconds.
reconnect - When set to true, Python API tries to reconnect the session to the server when a connection error is detected.
(2) ddb.session.setTimeout()
Only available on Linux
The static method setTimeout of the Session object specifies the TCP connection timeout period (in seconds) before the peer acknowledges (ACK) receiving the data packets. After the timeout period expires, the TCP connection is closed. It is equivalent to the TCP_USER_TIMEOUT option. The default value is 30. If the value is specified as 0, TCP will use the system default.
Use ddb.session.setTimeout() in the following data transmission scenarios:
Your network bandwidth is full and the amount of data you want to transmit is large
One-time transmission of a large amount of data. Transmitting a large amount of data takes a lot of time and, as a result, ACK may not be received in time. After the timeout period expires, the system closes the TCP connection (the session).
Related
My setup:
I am using an IP and port provided by portmap.io to allow me to perform port forwarding.
I have OpenVPN installed (as required by portmap.io), and I run a ready-made config file when I want to operate my project.
My main effort involves sending messages between a client and a server using sockets in Python.
I have installed a software called tcping, which basically allows me to ping an IP:port over a tcp connection.
This figure basically sums it up:
Results I'm getting:
When I try to "ping" said IP, the average RTT ends up being around 30ms consistently.
I try to use the same IP to program sockets in Python, where I have a server script on my machine running, and a client script on any other machine but binding to this IP. I try sending a small message like "Hello" over the socket, and I am finding that the message is taking a significantly greater amount of time to travel across, and an inconsistent one for that matter. Sometimes it ends up taking 1 second, sometimes 400ms...
What is the reason for this discrepancy?
What is the reason for this discrepancy?
tcpping just measures the time needed to establish the TCP connection. The connection establishment is usually completely done in the OS kernel, so there is not even a switch to user space involved.
Even some small data exchange at the application is significantly more expensive. First, the initial TCP handshake must be done. Usually only once the TCP handshake is done the client starts sending the payload, which then needs to be delivered to the other side, put into the sockets read buffer, schedule the user space application to run, read the data from the buffer in the application and process, create and deliver the response to the peers OS kernel, let the kernel deliver the response to the local system and lots of stuff here too until the local app finally gets the response and ends the timing of how long this takes.
Given that the time for the last one is that much off from the pure RTT I would assume though that the server system has either low performance or high load or that the application is written badly.
I have a basic implementation of a TCP client using python sockets, all the client does is connect to a server and send heartbeats every X seconds. The problem is that I don't want to send the server a heartbeat if the connection is closed, but I'm not sure how to detect this situation without actually sending a heartbeat and catch an exception. When I turn off the server, in the traffic capture I see FIN/ACK arriving and the client sends an ACK back, this is when I want my code to do something (or at least change some internal state of the connection). Currently, what happens is after the server went down and X seconds passed since last heartbeat the client will try to send another heartbeat, only then I see RST packet in the traffic capture and get an exception of broken pipe (errno 32). Clearly python socket handles the transport layer and the heartbeats are part of application layer, the problem I want to solve is not to send the redundant heartbeat after FIN/ACK arrived from server, any simple way to know the connection state with python socket?
Currently developing something like "smart home" and I have few different devices in my home. All of them connected to OpenHab via MQTT. I'm using Paho MQTT library (Python) for my purposes.
Generally, MQTT has "keepalive" property. This property describes how much time my client will be connected (AFAIK it sends the ping to the server) to MQTT server when there are no updates on the subscribed topic.
But here I have a huge problem. Needed topic could be updated once per hour or even once per few days/months. Let's say that this is indoor alarm.
How can I avoid that keepalive timeout or ignore that field? Could it be unlimited?
You have miss understood what the keepalive value represents.
MQTT clients can stay connected indefinitely even if they do not publish or receive any messages. But the broker needs to keep track of which clients are still connected so it knows when to send the Last Will and Testament (LWT) message for the client. In order to do this it uses the keepalive time.
Every time a message is sent or received by the client, the broker resets a timer, if this timer exceeds 1.5 times the value of the keepalive time then the broker marks the client as disconnected and processes the LWT. To prevent clients with very low messages rates from being disconnected, such a client can send a PINGREQ packet at any time (most likely on timeout of the keepalive value) to the server/broker. The server receives the PINGREQ, answers with a PINGRESP packet and it will reset the keepalive timer to zero and leave the client in the connected state.
See Keep Alive section of the MQTT standard: (http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc385349238)
The Client can send PINGREQ at any time, irrespective of the Keep Alive value, and use the PINGRESP to determine that the network and the Server are working. If the Keep Alive value is non-zero and the Server does not receive a Control Packet from the Client within one and a half times the Keep Alive time period, it MUST disconnect the Network Connection to the Client as if the network had failed
When sending the initial MQTT CONNECT message from a client, you can supply an optional "keep-alive" value. This value is a time interval, measured in seconds, during which the broker expects a client to send a message, such as a PUBLISH message. If no message is sent from the client to the broker during the interval, the broker automatically closes the connection. Note that the keep-alive value you specify is multiplied by 1.5, so setting a 10-minute keep-alive actually results in a 15 minute interval.
Have a look at the Keep Alive section of the MQTT specification:
A Keep Alive value of 0 has the effect of turning off the Keep Alive mechanism. If Keep Alive is 0 the Client is not obliged to send MQTT Control Packets on any particular schedule. v5 spec source
Therefore, set the keep alive to 0, and then the client doesn't have to send a keep alive signal as often. The server should respect that this connection with a client (e.g. from last year) should still be connected, but it won't be guaranteed (The client might be disconnected when the server is shut down).
The Python elasticsearch client in my applicaiton is having connectivity issues (refused connections) because idle TCP connections timeout due to a firewall (I have no way to prevent this).
The easiest way for me to fix this would be if I could prevent the connection from going idle by sending some data over it periodically, the sniffing options in the elasticsearch client seem ideal for this, however they're not very well documented:
sniff_on_start – flag indicating whether to obtain a list of nodes
from the cluser at startup time
sniffer_timeout – number of seconds
between automatic sniffs
sniff_on_connection_fail – flag controlling
if connection failure triggers a sniff
sniff_timeout – timeout used for the sniff request - it should be a fast api call and we are talking potentially to more nodes so we want to fail quickly. Not used during initial sniffing (if sniff_on_start is on) when the connection still isn’t initialized.
What I would like is for the client to sniff every (say) 5 minutes, should I be using the sniff_timeout or sniffer_timeout option? Also, should the sniff_on_start parameter be set to True?
I used the suggestion from #val and found that these settings solved my problem:
sniff_on_start=True
sniffer_timeout=60
sniff_on_connection_fail=True
The sniffing puts enough traffic on the TCP connections so that they are never idle for long enough for our firewall to kill the conneciton.
I have 2 rabbitmq in cluster load balanced by an Azure Internal Load Balancer. The client connect to the LB with BlockingConnection.
When there a are message exchanged by the clients, everything works fine. But when there is no activity, it seems my clients are disconnected and and not able to receive messages anymore.
I am wondering if there is a solution to this problem ? I assume the load balancer or rabbitmq is closing the connection due to inactivity. I wanted to let pika trigger some heartbeat to rabbitmq (so that the load balancer keep the connection opened) but didn't find any good solution. Could you advice ?
EDIT 1
It seems pika BlockingConnections does not support heartbeat. Heart beat disables blocking connection
Thank you.
According to the Pika doc http://pika.readthedocs.org/en/0.10.0/modules/parameters.html, it seems Pika BlockingConnentions with URLParameters specified heart_interval(such as amqps://www-data:rabbit_pwd#rabbit1/web_messages?heartbeat_interval=30) that can keep connection opened, but the value of heart_interval can not larger than the value of rabbit server suggested.
Heartbeat Timeout Interval
The heartbeat timeout value defines after what period of time the peer TCP connection should be considered dead by RabbitMQ and client libraries. This value is negotiated between the client and RabbitMQ server at the time of connection. The client must be configured to request heartbeats. In RabbitMQ versions 3.0 and higher, the broker will attempt to negotiate heartbeats by default (although the client can still veto them). The timeout is in seconds, and default value is 60 (580 prior to release 3.5.5).
Heartbeat frames are sent about every timeout / 2 seconds. After two missed heartbeats, the peer is considered to be unreachable. Different clients manifest this differently but the TCP connection will be closed. When a client detects that RabbitMQ node is unreachable due to a heartbeat, it needs to re-connect.
Heartbeats can be disabled by setting the timeout interval to 0.
Sample code from Pika doc:
import pika
parameters = pika.URLParameters('amqps://www-data:rabbit_pwd#rabbit1/web_messages?heartbeat_interval=30')
connection = pika.BlockingConnection(parameters)