We Keep Coding

Python websockets - message send with success after 'Broken pipe' error

I have server and client written in python. My server is implemented using asyncio and library called 'websockets'. This is asynchronous architecture. Client from the other hand is implemented in library called 'websocket-client'. They are 2 different code bases and repositories.
In server repository i am calling serve method to start websocket server that is accepting connection from clients and allows them to send messages to sever. It looks like this:
async with serve(
self.messages_loop, host, port, create_protocol=CentralRouterServerProtocol
) as ws_server:
...
Client is using websocket-client library and it is connecting to websocket by calling 'create_connection' method. Later it is calling 'send' method to send message to server. Code:
client = create_connection(f'ws://{central_router.public_ip}', timeout=24*60*60, header=cls.HEADERS)
cls.get_client().send(json.dumps(message_dict)) // Sends message later loop. After user will type something from input.
Main requirement is that client can only send messages. It cant read it.After that server is sending ping every X seconds to confirm that connection is alive. Server waits another Y seconds for client to reply to him. Client cant reply to server, because it is running on synchronous block of code. The server is closing the connection but client doesnt know about it. Client is not reading from websocket (so he cant get information about closed websocket -> is that true?). After that sobody is typing something into input, and client is sending message to server. AND NOW -> the websocket-client send method is not raising any exception (that connection is closed), but messages will never get to the server. If user will type message one more time, it will get finnaly exception
[Errno 32] Broken pipe
but the first message after connection close will never raise and error/exception.
Why is that? What is going on? My first solution was to set ping_timeout to None on server side. It will make server not to wait this Y seconds for response, and it will never close a connection. However this is wrong solution, because it can cause zombie connections on server side.
Do anyone know, why client can sand one more message with success, after the pipe was broken?

Notification for FIN/ACK using python socket

I have a basic implementation of a TCP client using python sockets, all the client does is connect to a server and send heartbeats every X seconds. The problem is that I don't want to send the server a heartbeat if the connection is closed, but I'm not sure how to detect this situation without actually sending a heartbeat and catch an exception. When I turn off the server, in the traffic capture I see FIN/ACK arriving and the client sends an ACK back, this is when I want my code to do something (or at least change some internal state of the connection). Currently, what happens is after the server went down and X seconds passed since last heartbeat the client will try to send another heartbeat, only then I see RST packet in the traffic capture and get an exception of broken pipe (errno 32). Clearly python socket handles the transport layer and the heartbeats are part of application layer, the problem I want to solve is not to send the redundant heartbeat after FIN/ACK arrived from server, any simple way to know the connection state with python socket?

Pika connection closed after 3 heartbeats

I'm writing a script which receives HTTP requests (using Tornado), parses them, and sends them to a RabbitMQ broker using pika.
The code looks like this:
def main():
conn_params = pika.ConnectionParameters(
host=BROKER_NAME,
port=BROKER_PORT,
ssl=True,
virtual_host=VIRTUAL_HOST,
credentials=pika.PlainCredentials(BROKER_USER, BROKER_PASS),
heartbeat_interval=HEARTBEAT_INTERVAL
)
conn = pika.BlockingConnection(conn_params)
channel = conn.channel()
# Create the web server which handles application requests.
application = tornado.web.Application([
(URL_BILLING, SomeHandler, dict(channel=channel))
])
# Start the server
application.listen(LISTENING_PORT)
tornado.ioloop.IOLoop.instance().start()
As you can see, I open a single connection and channel, and pass the channel to any instance of the handler which is created, the idea being to save traffic and avoid opening a new connection/channel for every request.
The issue I'm experiencing is that the connection is closed after 3 heartbeats. I used Wireshark in order to figure out what the problem is, but all I can see is that the server sends a PSH (I'm assuming this is the heartbeat) and my scripts replies with an ACK. This happens 3 times with HEARTBEAT_INTERVAL in between them, and then the server just sends a FIN and the connection dies.
Any idea why that happens? Also, should I keep the connection open or is it better to create a new one for every message I need to send?
Thanks for the help.
UPDATE: I looked in the RabbitMQ log, and it says:
Missed heartbeats from client, timeout: 10s
I thought the server was meant to send heartbeats to the client, to make sure it answers, and this agrees with what I observed using Wireshark, but from this log it seems it is the client which is meant to report to the server, not the other way around, and the client, evidently, doesn't report. Am I getting this right?
UPDATE: Figured it out, sort of. A blocking connection (which is what I used) is unable to send heartbeats because it's, well, blocking. As mentioned in this issue, the heartbeat_interval parameters is only used to negotiate the connection with the server, but the client doesn't actually send heartbeats. Since this is the case, what is the best way to keep a long-running connection with pika? Even if I don't specify heartbeat_interval, the server defaults to a heartbeat every 10 minutes, so the connection will die after 30 minutes...

For future visitors:
Pika has an async example which uses heartbeat:
http://pika.readthedocs.org/en/0.10.0/examples/asynchronous_publisher_example.html
For Tornado specific, this example shows how to use Tornado's IOLoop in pika's async model:
http://pika.readthedocs.org/en/0.10.0/examples/tornado_consumer.html

Sleep after ZMQ connect?

In a ROUTER-ROUTER setup, after I connect one ROUTER socket to another, if I don't sleep (for say 0.1s or so) after I connect() to the other ROUTER socket, the send() usually doesn't go through (although it sometimes does, by chance).
Is there a way to make sure I am connected before I send?
Why aren't the send()s queued and properly executed until the connection is made?
Also, this is not about whether the server on the other end is alive but rather that I send() too soon after I connect() and somehow it fails. I am not sure why.

Is there a way to make sure I am connected before I send?
Not directly. The recommended approach is to use something like the Freelanch Protocol and keep pinging until you receive a response. If you stop receiving responses to your pings you should consider yourself disconnected.
Why aren't the send()s queued and properly executed until the connection is made?
A router cannot send a message to a peer until both sides have completed an internal ZeroMQ handshake. That's just the way it works, since the ROUTER requires the ID of its peer in order to "route". Apparently sleeping for .1sec is the right amount of time on your dev system. If you need the ability to connect and then send without sleeping or retrying, then you need to use a different pattern.
For example, with DEALER-ROUTER, a DEALER client can connect and immediately send and ZeroMQ will queue the message until it is delivered. The reason the works is that the DEALER does not require the ID of the peer - since it does not "route". When the ROUTER server receives the message, that handshake is already complete so it can respond right away without sleeping.

Does a pop3 connection timeout?

I am using python poplib.
Does a POP3 connection timeout after a period of inactivity? Is there a way to set this timeout? Would such a timeout cause emails flagged for deletion to be deleted as if quit() had been called?
The only timeout I see mentioned refers to the connection-attempt timeout
class poplib.POP3(host[, port[, timeout]])
My POP3.quit() call is throwing a poplib.error_proto exception with the message "timed out" and I am wondering if the connection will ever timeout on its own.

Does a POP3 connection timeout after a period of inactivity?
This mostly depends on the server software on the other end. Some socket libraries may let you specify a local timeout. The connection will timeout at whatever time is the shortest between the local and the remote timeout.
Is there a way to set this timeout?
Unless you control the server software settings (e.g. you run your own server), you can't change the remote timeout.
Would such a timeout cause emails flagged for deletion to be deleted as if quit() had been called?
Timeout behavior is usually the same as manually closing the socket (with or without shutdown). If you don't send the quit command, messages won't be deleted from the server.

POP3 is sort of transactional. So if you download and delete a message from the server, it will not be effective until you close the connection gracefully. The POP3.quit() method has an insightful description about it. The magic word is commit.
The connection timeout is usually a server setting and, of course, when the server times out the connection it is considered a rollback.

Yes a POP3 socket will eventually time out. Looking here you pass the timeout in the constructor.
http://docs.python.org/library/poplib.html
class poplib.POP3(host[, port[, timeout]])