We Keep Coding

python socket send data after receiving fin (stuck in LAST_ACK)

I have a python program that is supposed to receive some arbitrary bytes and send them back after receiving a fin. I already was able to implement this as you can see below.
The problem I am having is that the connection is never properly closed. Using ss -tan I can see that the connection keeps being stuck in LAST_ACK state. This is although the connection seems to be closed correctly looking at the Wireshark packet trace. I have attached an Image of the Wireshark packet trace that is the result of first sending "AAAAA" then an out of order "BBB" and then filling the hole with "CCC".
Looking at the Wireshark packets I think that all packets should be correctly acknowledged and the connection shoud terminate normally without being stuck in LAST_ACK state and without doing the retransmissions at the bottom. But I Still would guess that there is a problem with the connection closing packets.
# Echo server program
import socket
HOST = '' # Symbolic name meaning all available interfaces
PORT = 6000 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print ('Connected by', addr)
data_acc = b''
while 1:
data = conn.recv(1024)
if not data: break
data_acc += data
print(data)
print("send data back")
conn.sendall(data_acc)
conn.close()

I found the answer to the problem. The sequence number of ACK on line 56 in the screenshot needs to be increased by one. By doing so Wireshark also stops interpreting it as a keep alive packet.

how to handle variable length message and close the socket automatically?

I have the following case:
SERVER
sock = socket.socket()
sock.bind((hostaddr, port))
sock.listen(backlog)
print(f'Server listenning on {hostaddr}:{port}')
while True:
client_sock, client_address = self.sock.accept()
print(f'Incoming connection from {client_address[0]}:{client_address[1]}')
while True:
data = client_socket.recv(buffer_size)
if not data:
break
print(f'Received "{data.decode()}" from {client_address[0]}:{client_address[1]}')
reply = f'Server: I got the message "{data.decode()}"'.encode()
client_socket.sendall(reply)
client_socket.close()
CLIENT
sock = socket.socket()
sock.connect(server_address)
sock.sendall('Lorem Ipsum'.encode())
while True:
data = sock.recv(buffer_size)
if not data:
break
print(data.decode())
sock.close()
I first start the server, then I start the client, and I get the following logs:
SERVER
Server listening on 172.16.0.110:8081
Incoming connection from 172.16.0.110:62388
Received "Lorem Ipsum" from 172.16.0.110:62388
CLIENT
Server reply: I got the message "Lorem Ipsum"
I wanted to get the server reply and then the client should finish, but both server and client enter an infinite loop and keep running forever. Why and how to fix that? I'm using Python 3.6.0 on Windows 10 x64 in an IPv4 network.

You must define a protocol, which is just the rules for how messages are exchanged and formatted, and how message boundaries are communicated. It appears you simply want the client to send some data and read the server response. You can achieve this by closing the write half of the client connection, in your case by calling sock.shutdown(socket.SHUT_WR), after the sock.sendall(...).
On the server side this is the read half of the same connection, and the servers detects this as EOF, causing socket.recv() to return a zero-length bytes object.
For more complicated protocols for which you want to send multiple messages on the same connection, a different strategy must be used. One simple example for a binary protocol would be to send 4 bytes representing the length in bytes of the message, then send that many bytes for the subsequent message itself.

One way is to set a timeout for the socket so it doesn't block forever when waiting for reply using socket.settimeout() as the following:
sock = socket.socket()
sock.connect(server_address)
sock.sendall('Lorem Ipsum'.encode())
sock.settimeout(5.0) # sets timeout to 5 seconds
while True:
data = sock.recv(buffer_size)
if not data:
break
print(data.decode())
sock.close()

Python Bidirectional TCP Socket Hanging on socket.recv

Referencing this example (and the docs): https://pymotw.com/2/socket/tcp.html I am trying to achieve bidirectional communication with blocking sockets between a client and a server using TCP.
I can get one-way communication to work from client->server or server->client, but the socket remains blocked or "hangs" when trying to receive messages on both the server and client. I am using a simple algorithm(recvall), which uses recv, to consolidate the packets into the full message.
I understand the sockets remain blocked by design until all the data is sent or read(right?), but isn't that what sendall and recvall take care of? How come disabling recv on either the client or server "unblocks" it and causes it to work? And ultimately what am I doing wrong that is causing the socket to stay blocked?
Here is my code, the only fundamental difference really being the messages that are sent:
recvall(socket)(shared between client and server):
def recvall(socket):
data = ''
while True:
packet = socket.recv(16)
if not packet: break
data += packet
return data
server.py (run first):
import socket
host = 'localhost'
port = 8080
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
s.listen(5)
while True:
(client, address) = s.accept()
print 'client connected'
try:
print recvall(client)
client.sendall('hello client')
finally:
client.close()
client.py:
import socket
s = socket.create_connection((args.ip, args.port))
try:
s.sendall('hello server')
print recvall(s)
finally:
s.close()

From my understanding (epiphany here), the main problem is that recv inside recvall is only concerned with retrieving the stream (in the same way send is only concerned with sending the stream), it has no concept of a "message" and therefore cannot know when to finish reading. It read all the bytes and did not return any additional bytes, but that is NOT a signal that the message is finished sending, there could be more bytes waiting to be sent and it would not be safe to assume otherwise.
This requires us to have an explicit indicator for when to stop reading. recv and send are only concerned with managing the stream and therefore have no concept of a message (our "unit"). This article has some great solutions to this problem. Since I am sending fixed-length messages, I opted to check that the length is as expected before finishing recv. Here is the updated version of recvall, note MSG_LENGTH must be defined and enforced in order for recvall to not block the socket.
def recvall(socket):
data = ''
while len(data) < MSG_LENGTH:
packet = socket.recv(BUFFER_SIZE)
if not packet: break
data += packet
return data
Bidirectional communication now works, the only catch being the client and server must know the length of the message they will receive, again this is not an issue in my case. This is all new to me so someone please correct me on terminology and concepts.

Python socket receiving corrupted information

I'm trying to understand how send and receive are working.
I was trying to send continuously data to a server and i noticed that the server would receive mixed bytes because i was sending to much data at a time. See my code:
Server:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind(("",1996))
server.listen(0)
c,d = server.accept()
while True:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
Client:
import socket, struct
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.connect(("192.168.1.4",1996))
while True:
data = 1
server.send( struct.pack("i", data) )
Then i change the while loops to this:
Server:
data = c.recv(1024)
print( struct.unpack("i", data)[0] )
c.send( str.encode("Server received your message. You now can continue
sending more data") )
Client:
data = 1
server.send( struct.pack("i", data) )
#Wait to secure the send.
server.recv(1024)
This is working. I'm making sure that the client won't send data before the
server already receive the previous send.
But what if i want to do the same for the server too? How can i make sure that the server will send bytes to the client in a safe way?
I already tried this and i notice that i created an infinity loop because(I used multi-threading in order to send and receive at the same time on the server):
client was sending some data and then waiting to get a signal from the server
that he can send again.
the server was getting some data then sending the signal and after that waiting for a signal from the user that he can send again.
But because the client was actually sending data again, the whole thing was going on again and this caused me an infinity talk-reply loop.
So what can i do to make a continuously conversation between two sockets without mixing the bytes together?

Your problem is caused by Nagle algorithm which works by combining a number of small outgoing messages, and sending them all at once as TCP is a stream protocol. You can enable TCP_NODELAY socket option by calling sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) to sent data as soon as possible, even if there is only a small amount of data. And on the receiver side, it isn't going to get one packet at a time either, you must implement message boundaries itself if you want "continuous conversation between two sockets without mixing the bytes together".

Retrieve all the response in socket at one time

I am writing a client-sever program based on Python socket.
The client sends a command to the server and the server responds.
But now, some client can broadcast a message to other clients, so the client can receive more than one response at the same time.
data = s.recv(1024)
the line of code above will retrieve only one response from the server.
but if I use a while loop like this
while True:
data = s.recv(1024)
if not data: break
actually, data=s.recv(1024) will block the program when there is no data left.
I don't want to block the program and want to retrieve all the responses available in the connection at one time. Can anyone find a solution? Thank you.

You can use the select module to wait until the socket is readable or until a timeout has elapsed; you can then perform other processing. For example:
while True:
# If data can be received without blocking (timeout=0), read it now
ready = select.select([s], [], [], 0)
if s in ready[0]:
data = s.recv(1024)
# Process data
else:
# No data is available, perform other tasks

You could make the socket (s) non-blocking. This way, it will retrieve all the received responses and when there is none, it will return back. Of course, with non-blocking, you will have to periodically retry.
You could make the socket (s) non-blocking using the setblocking() method:
s.setblocking(0)
The other option is to use another thread to handle the receive part. This way, your main thread can continue doing its main task and act upon the message only if it receives one.

You can use socket.setblocking or socket.settimeout:
import socket
import sys
HOST = 'www.google.com'
PORT = 80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.setblocking(0)
s.sendall('Hello, world')
try:
data = s.recv(1024)
except:
print 'Oh noes! %s' % sys.exc_info()[0]
s.close()
socket.recv takes two parameters, the second is a set of flags. If you're on a Linux system, you can do man recv for a list of flags you can supply, and their corresponding errors.
Lastly, in general, you can't really know that the other side is done with sending you data (unless you're controlling both sides), even if you're both following a protocol. I believe the right way to go about it is to use timeouts, and quit after sending a reset (how you do this will depend upon what protocol you're using).