I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()
You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.
Related
I am on working a project with Socket TCP/IP ( Server-C# and Client-Python).
Streaming video after sometimes,the data of Recv Socket is splitted.
My data is buff = 22000 bytes,if it is splitted it will become :
buff = 1460
buff = 20600
I don't know why,i have researched some methods with MTU,Fragmentation,Windows Size,....but not have result
Specially,if i setsocketopt the process will appear less.
self.sk.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1048576)
enter image description here --Image about the data is splitted
This is my Recv Socket.
buff = self.sk.recv(1048576)
print("BUFF RECEIVE ::: ::::: ---->>>>> ",len(buff))
if buff == b'' :
self.sk=None
buff = None
return buff
Suggestions: This just only happened to Chrome Browser(It mean,can't Streaming Video if loss data ).But at Firefox,it is not.It seem to blink a moment if loss data but It can continue stream after.
enter image description here - Chrome and FireFox
That is just the way TCP works. It is a streaming protocol with no built-in message framing, so it is free to place bytes into packets in any amounts it chooses — all it guarantees is that the bytes will still be in the correct order when they are read() by the receiving socket.
Therefore, if your program requires a certain number of bytes before it can proceed, it is up to your program to do the necessary buffering to assemble those bytes together.
As for why TCP might behave the way you observed —it is likely reacting to network conditions (dropped packets, feedback from the receiving host’s TCP stack, etc), and trying to make transmission as efficient as possible given its current environment. It’s 100% up to the TCP stack how it wants to transmit the data, and different TCP stacks may behave differently, which is fine as long as they follow the rules of the TCP specification.
After a long time,I have found the answer for my issue.
Solution for TCP/IP client socket message boundary problem
**1/**When you send a package from Server to Client with Send(Write).At Client side,the Receive will not get full data in sometimes.It not mean,Send/write at Server not send enough data.Just because this is TCP/IP protocol,Receive is not graduatee and the package will be fragmentation at Client Side ( your code ).
**2/**You can solve this issue by add more pattern at send/write Server Side. For example, send(data) --> send ( Pattern + data) and at Client side,you can use patern to check data.
**3/**Limitations of this method,the package after fragmentation,it can "combine together" or sometime it can't not.For example,your send data = 4000 and at Client side,your receive = 1460 + 2540.
This is what I understood with my issue.
I am reading a networking book and from what I have read about the TCP protocol, it makes sure the data will be sent. I want to write some code to do a file transfer. Before getting to that, I also read in the Python documents this passage:
"Applications are responsible for checking that all data has been
sent; if only some of the data was transmitted, the application needs
to attempt delivery of the remaining data"
This seems to contradict what I read in the networking book. The passage above says applications are responsible for the lost data.
I may be misunderstanding so I want to ask some questions:
1-If I have to check that the data is sent, then why use TCP?
2-I read in the networking book that TCP does the math to make sure that the data is there. Then why isn't using TCP a waste of time ?
3- The python docs didn't specify a buffer size. what is the maximum size of buffer to send at a time?
4-I read in the networking book that the server can increase the amount of
data that it can send if it knows the client can receive it. can this change
the size of the buffer more than the maximum number?
Here is my code attempt so far:
Server code:
import socket
s = socket.socket()
host = socket.gethostname()
port = 3000
s.bind((host,port))
s.listen(1)
c,addr = s.accept()
with open("Filetosend","rb") as File:
data= File.read(1024)
while data:
c.send(data)
data = File.read(1024)
s.close()
Client code:
import socket
s= socket.socket()
host = socket.gethostname()
port = 3000
s.connect((host,port))
with open("Filetowrite","wb") as File:
data = s.recv(1024)
while data:
File.write(data)
data = s.recv(1024)
s.close()
TCP tries to guarantee that if the data is delivered, it's correct and in order. It uses checksums to ensure data isn't corrupted, and sequence numbers to ensure that data is delivered in order and with no gaps. And it uses acknowledgements so the sender will know that data has been received.
But suppose there's a network failure in the middle of a transmission. If it happens after the data segment is received, but before the acknowledgement is sent back, the sender will not know that the data was received. The sender will keep trying to resend the data, and will eventually time out and report an error to the application.
Most TCP APIs don't allow the application to find out precisely where in the communication the error happened. If you sent a megabyte, and get an error, it could have happened at the beginning, when hardly anything was sent, or at the end when most of the data was sent. It could even have happened after all the data was sent -- maybe just the last ACK was lost.
Furthermore, the write() system call generally just puts the data in a kernel buffer. It doesn't wait for the data to be sent to the network, and doesn't wait for the receiver to acknowledge it.
Even if you successfully close the connection, you can't be totally sure. When you close the connection, the sender sends a message to the recipient saying they're done sending data. But closing the connection just queues this in the network stack, it doesn't wait for the other system to acknowledge it.
This is why application protocols have their own level of acknowledgements, on top of the basic TCP protocol. For instance, in the SMTP protocol, the client sends the message contents, followed by a line with a . to indicate the end, then waits for the server to send back a response code that indicates that the message was received successfully and is being delivered or queued. TCP's checking ensures that if you receive this response, the message contents were sent intact.
Regarding the general ability of any protocol to guarantee perfect delivery of all messages, you should read about the Two Generals' Problem. No matter what you do, there's no way to verify delivery of all messages in any communication, because the only way to confirm that the last message was delivered is by sending another message in reply, and now that reply is the last message, and needs confirmation.
I am having a tcp server and a client written in python. The aim of the programs is that the server will be sending some numbers one after the other to the client and the client should process each one of the number in a separate thread.
The server loops over the number list and sends each one of them to client.
as:
for num in nums:
client_sock.send(str(num))
and the client loop as:
while True:
data = tcpClientSock.recv(BUFSIZE)
thread.start_new_thread( startFunction, (data, ) )
The problem is even though the server sends the program in separate send() call the client receives it all at once.
How can I avoid it? Should I use UDP instead of TCP in this situation?
you'll have to flush the socket on the sending end - add a CR/NL to do so (since you're sending a string)
TCP is a stream based protocol and not message based. This means there are no message boundaries for each time the server calls send(). In fact, each time send() is called, the bytes of data are just added to the stream.
On the receiving end, you'll receive bytes of the stream as they arrive. Since there are no message boundaries, you may receive part of a message or whole messages or whole + part of the next message.
In order to send message over a TCP stream, your protocol needs to establish message boundaries. This allows the receiver to interpret whether it has received a partial, full, or multiple messages.
In your example, the server is sending strings. The string termination servers as the message boundary. On the receiving side, you should be parsing out the strings and have handling for receiving partial strings
I am writing a program to send/receive data to a device using an IPv6 UDP port. My python code is running on a Win7 platform and the device is a piece of custom hardware. The PC and device are directly connected and do not get routed through a LAN.
The program work... sort of. When I initially attempt to send a message to the device, the send fails. To be more specific, as viewed by wireshark, I never see the message get sent to the device. When I ping the device using ping <ipv6_addr> -t on the command line, the first 1 or 2 attempts fail, after which, the pings start being received. I kill the ping and attempt to send the message just as before, but now it succeeds. Wireshark now shows all the messages. If I keep sending messages over a relatively short duration (somewhere between 10 sec to a 2 minute), the program works as expected. If I wait longer, my message will once again not be sent. If I keep the ping program continuously pinging, my messages also get sent regardless of the time between message sending activity. Here is my code:
import socket
def connect(port):
if socket.has_ipv6:
client = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
client.bind(('', port))
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client.settimeout(0.25)
def send_data(addr, port, data):
client.sendto(data, (addr, port))
def main_app():
connect(10000)
bin_data = binascii.a2b_hex('deadbeef')
# Do some processing
send_data(<ipv6_address>, 10000, bin_data)
Some googling did not yield any good leads on whether the problem is Python or something else, so I figured I would start with the Python as see where it takes me.
Any assistance would be greatly appreciated!
After some digging around I believe I found my problem. I was not including the IP address of the PC running the python in my bind statement. So changing my connect() function above to
def connect(self):
# ...
client.bind(('fe80::', port))
# ...
allowed my messages to be sent consistently. Im pretty new to socket programming, so can anyone provide an explanation why this fix works?
Thanks
Couple of hints:
Call client.setsockopt() before you call client.bind().
You need to call socket.close() when you are done with the socket.
What i suspect to be the reason behind the weird issues with ping and timing issues could be the combination of a low timeout value and the NDP caching.
The timeout of 0.25 is probably for some reason too low for NDP resolution + data transmission to complete (maybe your embedded device stack is slow). When you do the ping, it does NDP for you, and creates a cached info that you can use from your app, without breaking the timeout limit.
I would suggest trying to up the limit: client.settimeout(2)
So I'm making a basic "ping" application using UDP for an assignment, and everything is working except the implementation of socket.settimeout().
I can't seem to figure out why, but it has to do with the bound socket. It might just be a quirk in Python but I'd like to confirm it before I document it.
I'm not looking for a functional code answer (that'd be cheating), but rather why what I have is broken. (e.g: some undocumented reason that Python doesn't like client/server on same machine etc)
Python Socket Timeout Details: http://docs.python.org/2/library/socket.html#socket.socket.settimeout
In the code represented below, the communication with server running on same machine is successful, but only when the client does not bind to the socket. But if it does not bind to the socket, the timeout system fails (this being tested by turning off the server, in which all ten timeouts get printed immediately and at once).
Note: Code is not ideal, but this is a networking theory class and not a programming class. It just has to work in the end. I could hand it in right now and get an A, but I want to understand why the timeout function does not work.
EDIT: To clarify an issue, the use of Bind in the client was after seeing the server code had it before I realized UDP doesn't need it, but it happened to make the timeout function work properly, but breaks the normal operation.
Could the socket.settimeout() declaration only work for TCP maybe?
Client Code (which has the timeout):
import socket
import time
import select
data = "Re-verify our range to target... one ping only. \n"
addrDest = ("127.0.0.1",8675)
addrLocal = ("127.0.0.1",12345)
totalTime = 0
averageTime = 0
totalPings = 0
#timeout_seconds = 1.0
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind(addrLocal)
# adding bind here even though it's UDP makes timeout work,
# but breaks normal functionality
UDPSock.settimeout(1)
while (totalPings < 10):
totalPings = (totalPings + 1)
start = time.clock()
str_list = []
str_list.append(str(totalPings))
str_list.append(" ")
str_list.append(str(start))
str_list.append(" ")
str_list.append(data)
dataOut = ''.join(str_list)
UDPSock.sendto(dataOut,addrDest)
try:
dataIn,addrIn = UDPSock.recvfrom(1024)
print dataIn.strip(),"\n",addrIn
elapsed = ((time.clock() - start) * 1000)
print elapsed, " ms round trip"
except socket.error:
print "Connection timed out for Packet #", totalPings
Server Code:
import socket
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
# (to all IP addresses on this system)
listen_addr = ("",8675)
UDPSock.bind(listen_addr)
# Report on all data packets received and
# where they came from in each case (as this is
# UDP, each may be from a different source and it's
# up to the server to sort this out!)
while True:
data,addr = UDPSock.recvfrom(1024)
print data.strip(),addr
UDPSock.sendto(data,addr)
Why do you need to bind to local address of the client? Will the client act as a server too at any point? If not there is no need to bind the client at all. You need a specific port only if you need your client to act as a server, if you don't call bind it will create a random port no ranging from (0 - 1023 are reserved) 1024 - 65535 (if I remember correctly) and that will be Source Port in the UDP Packet, Source Address is the Address where client runs.
According to Berkley Sockets
bind() assigns a socket to an address. When a socket is created using socket(),
it is only given a protocol family, but not assigned an address. This association with an address must be performed with the bind() system call before the socket can accept connections to other hosts
If this is a Networking class project and you are trying to implement Client-Server architecture then you should never call bind from within your client code because Client should never act as a Server and Client should connect to a listening Server not Server connecting to Client.
Update:
Bind may be required to be called from a TCP Client-Server design but not from a UDP Client-Server model because UDP is a send and forget design and doesn't have low level packet send success acknowledgement. A UDP packet will have Source Address and Port within itself.
I found the cause of the problem by removing the exception handling. There is a socket error when the server is turned off, specifically "socket.error: [Errno 10054] An existing connection was forcibly closed by the remote host" when it tries to read from the socket.
This apparently ignores the timeout function when the socket is not bound in Python (which is why the timeout worked when I bound it).
If I run the server, but just have it not send any data (by commenting the last line), the program times out correctly when it does not receive its packet back.
I am also going to use a more specific exception handler
In the end it's just a quirk in Python and the fact that UDP is connection-less.
Also, someone mentioned the use of "select" to solve this problem. I looked into it, and I would end up with an if...else statement block which kinda works, but the native exceptions are preferred.
Thanks all.
-Jimmy