I have this simple python script
sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
sock.sendto(b'ok',('123.123.123.123',1234))
recv,addr = sock.recvfrom(128) # This line is receiving packet?
How do I handle packet while it's in down streaming?
I mean I want ignore packet if certain bytes detected for example b'\xfc'
I can achieve it AFTER the buffer/data completely received with doing this:
if b'\xfc' in recv: sock.close()
But this weakness is I need to wait data/buffer completely received, what if destination server reply with big packet size? This will not efficient, I need to fully received destination server reply before close.
If my question is hard to understand, just understand this analogy. Suppose you want talk to me, and I'm starting listening you. While you are talking, I'm processing every your character, if there's certain character for example letter 's' you told to me, I will close my ear suddenly exactly after you send me such character.
Related
I apologize if this question is stupid or if this is actually a thing. Here is a scenario:
You're trying to exchange TCP packets while dealing with low-bandwidth communication between a desktop and RPi. Say that halfway during transmission of a particular packet, the connection drops. So the receiver of that packet only receives half of it.
Is there a way that the sender of the packet to know that only a portion of the packet is received, and thus send the remaining lost portion of the packet so that the receiving end is able to fully create the original packet?
Again, please take it easy on me haha this was just a really curious question.
Edit: I believe I can also realize that this would require the additional sending of packets to convey that only part of the packet has been received, so please point out that from a optimization standpoint if that is pointless.
I am on working a project with Socket TCP/IP ( Server-C# and Client-Python).
Streaming video after sometimes,the data of Recv Socket is splitted.
My data is buff = 22000 bytes,if it is splitted it will become :
buff = 1460
buff = 20600
I don't know why,i have researched some methods with MTU,Fragmentation,Windows Size,....but not have result
Specially,if i setsocketopt the process will appear less.
self.sk.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 1048576)
enter image description here --Image about the data is splitted
This is my Recv Socket.
buff = self.sk.recv(1048576)
print("BUFF RECEIVE ::: ::::: ---->>>>> ",len(buff))
if buff == b'' :
self.sk=None
buff = None
return buff
Suggestions: This just only happened to Chrome Browser(It mean,can't Streaming Video if loss data ).But at Firefox,it is not.It seem to blink a moment if loss data but It can continue stream after.
enter image description here - Chrome and FireFox
That is just the way TCP works. It is a streaming protocol with no built-in message framing, so it is free to place bytes into packets in any amounts it chooses — all it guarantees is that the bytes will still be in the correct order when they are read() by the receiving socket.
Therefore, if your program requires a certain number of bytes before it can proceed, it is up to your program to do the necessary buffering to assemble those bytes together.
As for why TCP might behave the way you observed —it is likely reacting to network conditions (dropped packets, feedback from the receiving host’s TCP stack, etc), and trying to make transmission as efficient as possible given its current environment. It’s 100% up to the TCP stack how it wants to transmit the data, and different TCP stacks may behave differently, which is fine as long as they follow the rules of the TCP specification.
After a long time,I have found the answer for my issue.
Solution for TCP/IP client socket message boundary problem
**1/**When you send a package from Server to Client with Send(Write).At Client side,the Receive will not get full data in sometimes.It not mean,Send/write at Server not send enough data.Just because this is TCP/IP protocol,Receive is not graduatee and the package will be fragmentation at Client Side ( your code ).
**2/**You can solve this issue by add more pattern at send/write Server Side. For example, send(data) --> send ( Pattern + data) and at Client side,you can use patern to check data.
**3/**Limitations of this method,the package after fragmentation,it can "combine together" or sometime it can't not.For example,your send data = 4000 and at Client side,your receive = 1460 + 2540.
This is what I understood with my issue.
I have an application that sends data over a TCP connection to a production server. I need to sniff the contents of that TCP connection and resend it to a debug server.
I've gotten quite close with this:
from scapy.all import *
packets = 0
def dup_pkt(pkt):
global packets
read = raw(pkt[TCP].payload)
print(str(packets))
s.sendall(read)
print("connecting")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("x.x.x.x", 12345))
print("connected")
print("sniffing")
pkts = sniff(prn=dup_pkt, filter="tcp dst port 12345 and not host x.x.x.x") # don't sniff the packets we're resending
The problem is that the packets appear to be missing data in the first two headers. I have set the debug server to save each received packet to a file, and set the application to connect directly to the debug server to compare the known good packet data with what the sniffer is sending. In the first packet, the first 1546/2079 bytes are good, but after that, each byte is zero instead of full of the correct data. In the second packet, the first 11 bytes are missing, but the rest is good.
Interestingly, after those initial two big setup packets, the remainder of the TCP packets seem to be sniffed properly - perhaps because they are usually far simpler and less than 40 bytes each.
Is there a better way to read packet data? Am I missing something? Unfortunately I don't have access to the source of the application, so I can't tell if it's doing anything special with those two big packets I'm having trouble with.
The issue with the first packet could indicate a problem in your operating system's TCP stack w.r.t. fragment reassembly.
In any case, try using another tool like tcpdump or wireshark to capture the packets. If they have the same problem, the problem lies with your operating system. If not, it could be a bug or configuration issue with scapy.
It could also mean that your IP packets are fragmented. Scapy does not automatically defragment packets, but fragments them.
you need to use the defrag function, or defragment (have a look at help(defrag) ), on the received packet list.
Maybe the packet you are checking is a fragment
I am reading a networking book and from what I have read about the TCP protocol, it makes sure the data will be sent. I want to write some code to do a file transfer. Before getting to that, I also read in the Python documents this passage:
"Applications are responsible for checking that all data has been
sent; if only some of the data was transmitted, the application needs
to attempt delivery of the remaining data"
This seems to contradict what I read in the networking book. The passage above says applications are responsible for the lost data.
I may be misunderstanding so I want to ask some questions:
1-If I have to check that the data is sent, then why use TCP?
2-I read in the networking book that TCP does the math to make sure that the data is there. Then why isn't using TCP a waste of time ?
3- The python docs didn't specify a buffer size. what is the maximum size of buffer to send at a time?
4-I read in the networking book that the server can increase the amount of
data that it can send if it knows the client can receive it. can this change
the size of the buffer more than the maximum number?
Here is my code attempt so far:
Server code:
import socket
s = socket.socket()
host = socket.gethostname()
port = 3000
s.bind((host,port))
s.listen(1)
c,addr = s.accept()
with open("Filetosend","rb") as File:
data= File.read(1024)
while data:
c.send(data)
data = File.read(1024)
s.close()
Client code:
import socket
s= socket.socket()
host = socket.gethostname()
port = 3000
s.connect((host,port))
with open("Filetowrite","wb") as File:
data = s.recv(1024)
while data:
File.write(data)
data = s.recv(1024)
s.close()
TCP tries to guarantee that if the data is delivered, it's correct and in order. It uses checksums to ensure data isn't corrupted, and sequence numbers to ensure that data is delivered in order and with no gaps. And it uses acknowledgements so the sender will know that data has been received.
But suppose there's a network failure in the middle of a transmission. If it happens after the data segment is received, but before the acknowledgement is sent back, the sender will not know that the data was received. The sender will keep trying to resend the data, and will eventually time out and report an error to the application.
Most TCP APIs don't allow the application to find out precisely where in the communication the error happened. If you sent a megabyte, and get an error, it could have happened at the beginning, when hardly anything was sent, or at the end when most of the data was sent. It could even have happened after all the data was sent -- maybe just the last ACK was lost.
Furthermore, the write() system call generally just puts the data in a kernel buffer. It doesn't wait for the data to be sent to the network, and doesn't wait for the receiver to acknowledge it.
Even if you successfully close the connection, you can't be totally sure. When you close the connection, the sender sends a message to the recipient saying they're done sending data. But closing the connection just queues this in the network stack, it doesn't wait for the other system to acknowledge it.
This is why application protocols have their own level of acknowledgements, on top of the basic TCP protocol. For instance, in the SMTP protocol, the client sends the message contents, followed by a line with a . to indicate the end, then waits for the server to send back a response code that indicates that the message was received successfully and is being delivered or queued. TCP's checking ensures that if you receive this response, the message contents were sent intact.
Regarding the general ability of any protocol to guarantee perfect delivery of all messages, you should read about the Two Generals' Problem. No matter what you do, there's no way to verify delivery of all messages in any communication, because the only way to confirm that the last message was delivered is by sending another message in reply, and now that reply is the last message, and needs confirmation.
I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()
You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.