I am reading a networking book and from what I have read about the TCP protocol, it makes sure the data will be sent. I want to write some code to do a file transfer. Before getting to that, I also read in the Python documents this passage:
"Applications are responsible for checking that all data has been
sent; if only some of the data was transmitted, the application needs
to attempt delivery of the remaining data"
This seems to contradict what I read in the networking book. The passage above says applications are responsible for the lost data.
I may be misunderstanding so I want to ask some questions:
1-If I have to check that the data is sent, then why use TCP?
2-I read in the networking book that TCP does the math to make sure that the data is there. Then why isn't using TCP a waste of time ?
3- The python docs didn't specify a buffer size. what is the maximum size of buffer to send at a time?
4-I read in the networking book that the server can increase the amount of
data that it can send if it knows the client can receive it. can this change
the size of the buffer more than the maximum number?
Here is my code attempt so far:
Server code:
import socket
s = socket.socket()
host = socket.gethostname()
port = 3000
s.bind((host,port))
s.listen(1)
c,addr = s.accept()
with open("Filetosend","rb") as File:
data= File.read(1024)
while data:
c.send(data)
data = File.read(1024)
s.close()
Client code:
import socket
s= socket.socket()
host = socket.gethostname()
port = 3000
s.connect((host,port))
with open("Filetowrite","wb") as File:
data = s.recv(1024)
while data:
File.write(data)
data = s.recv(1024)
s.close()
TCP tries to guarantee that if the data is delivered, it's correct and in order. It uses checksums to ensure data isn't corrupted, and sequence numbers to ensure that data is delivered in order and with no gaps. And it uses acknowledgements so the sender will know that data has been received.
But suppose there's a network failure in the middle of a transmission. If it happens after the data segment is received, but before the acknowledgement is sent back, the sender will not know that the data was received. The sender will keep trying to resend the data, and will eventually time out and report an error to the application.
Most TCP APIs don't allow the application to find out precisely where in the communication the error happened. If you sent a megabyte, and get an error, it could have happened at the beginning, when hardly anything was sent, or at the end when most of the data was sent. It could even have happened after all the data was sent -- maybe just the last ACK was lost.
Furthermore, the write() system call generally just puts the data in a kernel buffer. It doesn't wait for the data to be sent to the network, and doesn't wait for the receiver to acknowledge it.
Even if you successfully close the connection, you can't be totally sure. When you close the connection, the sender sends a message to the recipient saying they're done sending data. But closing the connection just queues this in the network stack, it doesn't wait for the other system to acknowledge it.
This is why application protocols have their own level of acknowledgements, on top of the basic TCP protocol. For instance, in the SMTP protocol, the client sends the message contents, followed by a line with a . to indicate the end, then waits for the server to send back a response code that indicates that the message was received successfully and is being delivered or queued. TCP's checking ensures that if you receive this response, the message contents were sent intact.
Regarding the general ability of any protocol to guarantee perfect delivery of all messages, you should read about the Two Generals' Problem. No matter what you do, there's no way to verify delivery of all messages in any communication, because the only way to confirm that the last message was delivered is by sending another message in reply, and now that reply is the last message, and needs confirmation.
Related
I have this simple python script
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.sendto(b'ok',('123.123.123.123',1234))
recv,addr = sock.recvfrom(128) # This line is receiving packet?
How do I handle packet while it's in down streaming?
I mean I want ignore packet if certain bytes detected for example b'\xfc'
I can achieve it AFTER the buffer/data completely received with doing this:
if b'\xfc' in recv: sock.close()
But this weakness is I need to wait data/buffer completely received, what if destination server reply with big packet size? This will not efficient, I need to fully received destination server reply before close.
If my question is hard to understand, just understand this analogy. Suppose you want talk to me, and I'm starting listening you. While you are talking, I'm processing every your character, if there's certain character for example letter 's' you told to me, I will close my ear suddenly exactly after you send me such character.
I am on working a project with Socket TCP/IP ( Server-C# and Client-Python).
Streaming video after sometimes,the data of Recv Socket is splitted.
My data is buff = 22000 bytes,if it is splitted it will become :
buff = 1460
buff = 20600
I don't know why,i have researched some methods with MTU,Fragmentation,Windows Size,....but not have result
Specially,if i setsocketopt the process will appear less.
self.sk.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1048576)
enter image description here --Image about the data is splitted
This is my Recv Socket.
buff = self.sk.recv(1048576)
print("BUFF RECEIVE ::: ::::: ---->>>>> ",len(buff))
if buff == b'' :
self.sk=None
buff = None
return buff
Suggestions: This just only happened to Chrome Browser(It mean,can't Streaming Video if loss data ).But at Firefox,it is not.It seem to blink a moment if loss data but It can continue stream after.
enter image description here - Chrome and FireFox
That is just the way TCP works. It is a streaming protocol with no built-in message framing, so it is free to place bytes into packets in any amounts it chooses — all it guarantees is that the bytes will still be in the correct order when they are read() by the receiving socket.
Therefore, if your program requires a certain number of bytes before it can proceed, it is up to your program to do the necessary buffering to assemble those bytes together.
As for why TCP might behave the way you observed —it is likely reacting to network conditions (dropped packets, feedback from the receiving host’s TCP stack, etc), and trying to make transmission as efficient as possible given its current environment. It’s 100% up to the TCP stack how it wants to transmit the data, and different TCP stacks may behave differently, which is fine as long as they follow the rules of the TCP specification.
After a long time,I have found the answer for my issue.
Solution for TCP/IP client socket message boundary problem
**1/**When you send a package from Server to Client with Send(Write).At Client side,the Receive will not get full data in sometimes.It not mean,Send/write at Server not send enough data.Just because this is TCP/IP protocol,Receive is not graduatee and the package will be fragmentation at Client Side ( your code ).
**2/**You can solve this issue by add more pattern at send/write Server Side. For example, send(data) --> send ( Pattern + data) and at Client side,you can use patern to check data.
**3/**Limitations of this method,the package after fragmentation,it can "combine together" or sometime it can't not.For example,your send data = 4000 and at Client side,your receive = 1460 + 2540.
This is what I understood with my issue.
I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()
You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.
I have a client written using python-twisted (http://pastebin.com/X7UYYLWJ) which sends a UDP packet to a UDP Server written in C using libuv. When the client sends a packet to the server, it is successfully received by the server and it sends a response back to the python client. But the client not receiving any response, what could be the reason ?
Unfortunately for you, there are many possibilities.
Your code uses connect to set up a "connected UDP" socket. Connected UDP sockets filter the packets they receive. If packets are received from any address other than the one to which the socket is connected, they are dropped. It may be that the server sends its responses from a different address than you've connected to (perhaps it uses another port or perhaps it is multi-homed and uses a different IP).
Another possibility is that a NAT device is blocking the return packets. UDP NAT hole punching has come a long way but it's still not perfect. It could be that the server's response arrives at the NAT device and gets discarded or misrouted.
Related to this is the possibility that an intentionally configured firewall is blocking the return packets.
Another possibility is that the packets are simply lost. UDP is not a reliable protocol. A congested router, faulty networking gear, or various other esoteric (often transient) concerns might be resulting in the packet getting dropped at some point, instead of forwarded to the next hop.
Your first step in debugging this should be to make your application as permissive as possible. Get rid of the use of connected UDP so that all packets that make it to your process get delivered to your application code.
If that doesn't help, use tcpdump or wireshark or a similar tool to determine if the packets make it to your computer at all. If they do but your application isn't seeing them, look for a local firewall configuration that might reject them.
If they're not making it to your computer, see if they make it to your router. Use whatever diagnostic tools are available (along the lines of tcpdump) on your router to see whether packets make it that far or not. Or if there are no such tools, remove the router from the equation. If you see packets making it to your router but no further, look for firewall or NAT configuration issues there.
If packets don't make it as far as your router, move to the next hop you have access to. This is where things might get difficult since you may not have access to the next hop or the next hop might be the server (with many intervening hops - which you have to just hope are all working).
Does the server actually generate a reply? What addressing information is on that reply? Does it match the client's expectations? Does it get dropped at the server's outgoing interface because of congestion or a firewall?
Hopefully you'll discover something interesting at one of these steps and be able to fix the problem.
I had a similar problem. The problem was windows firewall. In firewall allowed programs settings, allowing the communication for pythonw/python did solve the problem. My python program was:
from socket import *
import time
address = ( '192.168.1.104', 42) #Defind who you are talking to (must match arduino IP and port)
client_socket = socket(AF_INET, SOCK_DGRAM) #Set Up the Socket
client_socket.bind(('', 45)) # arduino sending to port 45
client_socket.settimeout(1) #only wait 1 second for a response
data = "xyz"
client_socket.sendto(data, address)
try:
rec_data, addr = client_socket.recvfrom(2048) #Read response from arduino
print rec_data #Print the response from Arduino
except:
pass
while(1):
pass
I'm trying to write an IRC bot but I'm not exactly sure how the receiving of data works. What I currently have:
while True:
data = socket.recv(1024)
#process data
Let's say that for whatever reason it takes it more time to process the data, what would happen if something is sent at that time? Will it get skipped or get added to some sort of a queue and processed after the current one is done?
Depending upon the protocol type the behavior will be different.
TCP:
The TCP RFC clearly states:
TCP provides a means for the receiver to govern the amount of data
sent by the sender. This is achieved by returning a "window" with
every ACK indicating a range of acceptable sequence numbers beyond
the last segment successfully received. The window indicates an
allowed number of octets that the sender may transmit before
receiving further permission.
Also from wikipedia the information is similar:
TCP uses an end-to-end flow control protocol to avoid having the
sender send data too fast for the TCP receiver to receive and process
it reliably. For example, if a PC sends data to a smartphone that is
slowly processing received data, the smartphone must regulate the data
flow so as not to be overwhelmed. TCP uses a sliding window flow
control protocol. In each TCP segment, the receiver specifies in the
receive window field the amount of additionally received data (in
bytes) that it is willing to buffer for the connection. The sending
host can send only up to that amount of data before it must wait for
an acknowledgment and window update from the receiving host.
UDP:
UDP doesn't have any flow control mechanism as TCP. However there is an other implementation of UDP such as RUDP that have some of the features of TCP like flow control.
Here is an other interesting link for the differences between TCP & UDP.