I have a udp client function written in python running on Windows 10. I'm aware this is not production quality code but I'm only trying to grasp the fundamentals here
client = socket(AF_INET, SOCK_DGRAM)
client.bind(('192.168.0.107', CLIENT_PORT))
client.setblocking(False)
while True:
try:
data = client.recv( 1024 )
except:
continue
if data is not None:
print(data.decode('utf-8'))
I have a server running on an embedded device sending out small udp packets periodically (udp payload size of 22). This client gets about 10 of these packets give or take a few then the script stops receiving the udp packets. The only exception raised in the try/catch block is that there is no data to receive. If I change to blocking the behaviour is the same.
[WinError 10035] A non-blocking socket operation could not be completed immediately
The server is still sending the packets, I can see them in Wireshark with the expected IP addresses, ports, and verified checksums.
The thing is if I add a send after the receive stops the problem goes away entirely and I can continue receiving the udp payloads
client = socket(AF_INET, SOCK_DGRAM)
client.bind(('192.168.0.107', CLIENT_PORT))
client.setblocking(False)
while True:
try:
data = client.recv( 1024 )
except:
continue
if data is not None:
print(data.decode('utf-8'))
client.sendto("a_udp_payload".encode('utf-8'), ('192.168.0.108' , SERVER_PORT))
Is there something fundamental that I am missing here? Why does the stop working on the first snippet but not the second? Is there some buffer flushing issue?
Many thanks
Try to set timeout for the socket client.settimeout(2)
I see a lot of examples online regarding socket recv() that follow this format:
Example from here.
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost', 50000))
s.listen(1)
conn, addr = s.accept()
while 1:
data = conn.recv(1024)
if not data:
break
The code here does not set the socket as non-blocking; nor does it set a timeout. My understanding is that recv() is blocking by default. If so, how does the code get to the break since, if there is no data then it stays on the recv()?
EDIT: what is the proper way to handle 'no more data'? Is it to set the socket to have a timeout or be non-blocking; or is there another way?
According to this, recv could return an empty string (therefore falsey) if the peer shuts down or disconnects. In other words, keep receiving data using the blocking recv function until the peer disconnects, then leave the loop.
In response to your edit: generally, if no more data is to be sent, the machine that is no longer sending data would send one last message to tell the server there's no more data to receive, or otherwise just end the connection and you would get your empty string.
I created socket for two PC, one is Raspberry Pi and the other one is my laptop. I just connected two then I send string to test the connection. If I send a character "q" from the RPi, my PC should break out of the loop and close the connection but it does not. The part print("Listening") is still running. Why? See code below.
import socket
import time
# IP address of this PC.
TCP_IP = '192.168.137.1'
# Port.
TCP_PORT = 5005
# Size of buffer.
BUFFER_SIZE = 1024
# Create a socket, connect and listen to it.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print('Connection address:', addr)
while 1:
print("Listening")
data = conn.recv(BUFFER_SIZE)
data = data.decode()
if data=='q':
break
if data:
print ("Received data:", data)
# Echo back.
conn.send(data.encode())
time.sleep(1)
print("It breaks.")
conn.close()
s.close()
TCP is a stream oriented protocol. So data transmitted is a stream not a sequence of messages. So when you expect data to be q it actually is some_data_sent_before_q_and_finally_q.
The simplest way to repair the code is to use if data.endswith('q') instead of if data=='q'. May work and may not depending on how you actually use the connection. For example, this approach may fail with some_data_sent_before_q pretty long pause more_data_and_q and with some_data_sent_before_q_and_finally_q_plus_something_else_why_not.
Little bit more advanced way to solve the problem is to divide the stream into messages with separators - message_1<separator>message_2<separator>q<separator>. This method will allow you to treat every message separately.
Referencing this example (and the docs): https://pymotw.com/2/socket/tcp.html I am trying to achieve bidirectional communication with blocking sockets between a client and a server using TCP.
I can get one-way communication to work from client->server or server->client, but the socket remains blocked or "hangs" when trying to receive messages on both the server and client. I am using a simple algorithm(recvall), which uses recv, to consolidate the packets into the full message.
I understand the sockets remain blocked by design until all the data is sent or read(right?), but isn't that what sendall and recvall take care of? How come disabling recv on either the client or server "unblocks" it and causes it to work? And ultimately what am I doing wrong that is causing the socket to stay blocked?
Here is my code, the only fundamental difference really being the messages that are sent:
recvall(socket)(shared between client and server):
def recvall(socket):
data = ''
while True:
packet = socket.recv(16)
if not packet: break
data += packet
return data
server.py (run first):
import socket
host = 'localhost'
port = 8080
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
s.listen(5)
while True:
(client, address) = s.accept()
print 'client connected'
try:
print recvall(client)
client.sendall('hello client')
finally:
client.close()
client.py:
import socket
s = socket.create_connection((args.ip, args.port))
try:
s.sendall('hello server')
print recvall(s)
finally:
s.close()
From my understanding (epiphany here), the main problem is that recv inside recvall is only concerned with retrieving the stream (in the same way send is only concerned with sending the stream), it has no concept of a "message" and therefore cannot know when to finish reading. It read all the bytes and did not return any additional bytes, but that is NOT a signal that the message is finished sending, there could be more bytes waiting to be sent and it would not be safe to assume otherwise.
This requires us to have an explicit indicator for when to stop reading. recv and send are only concerned with managing the stream and therefore have no concept of a message (our "unit"). This article has some great solutions to this problem. Since I am sending fixed-length messages, I opted to check that the length is as expected before finishing recv. Here is the updated version of recvall, note MSG_LENGTH must be defined and enforced in order for recvall to not block the socket.
def recvall(socket):
data = ''
while len(data) < MSG_LENGTH:
packet = socket.recv(BUFFER_SIZE)
if not packet: break
data += packet
return data
Bidirectional communication now works, the only catch being the client and server must know the length of the message they will receive, again this is not an issue in my case. This is all new to me so someone please correct me on terminology and concepts.
Possibly related questions that seem close but don't describe my issue as I understand it:
Reading all the data from a UDP socket
Python UDP socket semi-randomly failing to receive
Python raw socket listening for UDP packets; only half of the packets received
problem
Long file sent line by line doesn't go all the way through UDP over loopback.
long story
I have a long file consisting of lines and breaks that is identical to what I will get from another program over UDP locally. Let me emphasize that the program sending the packets will do so over UDP (there is no choice here), and cannot be feasibly modified to process ACK requests etc. while sending.
It looks like this (this is the tail):
StimulusTime 56398
Signal(0,2) -79.5457
Signal(0,4) -81.7426
Signal(0,6) -83.9978
Signal(0,9) -63.3755
Signal(0,11) -15.6045
Signal(0,13) 31.1299
Signal(0,16) 75.7539
Signal(0,18) 98.301
Signal(0,20) 98.301
Signal(0,22) 48.4546
Signal(0,25) 3.73159
Signal(0,27) -49.9798
Signal(0,29) -77.8449
Signal(1,0) -22.0332
Signal(1,2) -60.6384
Signal(1,4) -98.0858
Signal(1,6) -86.4579
Signal(1,9) -68.9173
Signal(1,11) -31.5552
Signal(1,13) 35.2906
Signal(1,16) 77.0686
Signal(1,18) 96.3836
Signal(1,20) 95.7246
Signal(1,23) 25.6074
Signal(1,25) -20.2101
Signal(1,27) -60.2933
Signal(1,29) -83.8169
Signal(2,0) -31.8826
Signal(2,2) -53.5045
Signal(2,4) -89.9895
Signal(2,7) -84.4503
Signal(2,9) -59.7016
Signal(2,11) -12.8569
Signal(2,13) 28.857
Signal(2,15) 58.0577
Signal(2,18) 96.4222
Signal(2,20) 79.783
Signal(2,22) 58.6463
Signal(2,25) -3.24883
Signal(2,27) -45.5
Signal(2,29) -88.8937
Signal(3,0) -18.6625
Signal(3,2) -53.3978
Signal(1,16) 58.784
Signal(1,17) 44.7782
Signal(1,18) 6.247
Signal(1,19) -12.0855
Signal(1,20) -33.7644
Signal(1,21) -49.4406
Signal(1,22) -67.5791
Signal(1,23) -92.0336
Signal(1,24) -93.9841
END
I wrote code that takes this file and sends it a line at a time over UDP locally, and then code that receives it and parses it based on the data type.
Sender:
import socket
import sys
# Sends udp test data piped in from STDIN to the listener.
# ex: cat sampleoutput.txt | python testsender.py
UDP_IP = "127.0.0.1"
UDP_PORT = 5000
print "UDP target IP:", UDP_IP
print "UDP target port:", UDP_PORT
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
# Send from stdin
if len(sys.argv) < 2:
while True:
line = sys.stdin.readline()
if line:
sock.sendto(line, (UDP_IP, UDP_PORT))
else:
break
# get from file arg
else:
myfile = open(str(sys.argv[1]), "r")
while True:
line = myfile.readline()
if line:
sock.sendto(line, (UDP_IP, UDP_PORT))
else:
break
sock.close()
Listener:
import socket
from array import array
UDP_IP = "127.0.0.1"
UDP_PORT = 5000
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size arg
print data
# write to file for later testing
# file = open("testdummy.txt", "a")
# file.write(data)
if data == "END\n":
break
I was able to use the above listener to produce the test file from the original program, so it should work. Unfortunately, it fails around 500 lines of payload, as tested by tail -n 500 testdummy.txt | python testsender.py, although it's somewhat random. Specifically, the listener does not receive all of the sent lines before the sender exits, leaving it hanging, waiting for the "END\n" string.
As I understand it, the socket is already in blocking mode--how can I prevent this from occurring?
My first advice to you, Don't use UDP if you want to transfer files with the sequence of lines preserved, Use TCP if you don't want to code alot. The reasons are;
UDP is an unreliable Protocol, in the sense a Packet sent is not guaranteed to be received by the recipient.
UDP doesn't guarantee the sequence of packets being received, This is because UDP packets may go to recipient via several routes (Hops between computers). So latter sent packets can take a short route and reach the recipient before former sent packets. ("End\n" packet can come before other packets)
TCP on the other hand is reliable and sequence of packets received is guaranteed. Ideal for file transfer.
But don't worry File sharing applications like Bear Share, Bit Torrents make use of UDP but there are some additional coding you have to do.
You need to implement an Acknowledgement protocol, as in you need to have a unique id for each packet you send to the recipient and when the packet is received recipient should send an Acknowledgement packet back to the sender with that id saying the packet was received.
If in case the packet got lost and didn't reach the recipient (No Acknowledgement from recipient) you must resend the packet again (and again) until you get the Acknowledgement.
You can control the order by not sending the next packet until you get the Acknowledgement for the previous one.