We Keep Coding

scapy - Missing data in sniffed TCP packets

I have an application that sends data over a TCP connection to a production server. I need to sniff the contents of that TCP connection and resend it to a debug server.
I've gotten quite close with this:
from scapy.all import *
packets = 0
def dup_pkt(pkt):
global packets
read = raw(pkt[TCP].payload)
print(str(packets))
s.sendall(read)
print("connecting")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("x.x.x.x", 12345))
print("connected")
print("sniffing")
pkts = sniff(prn=dup_pkt, filter="tcp dst port 12345 and not host x.x.x.x") # don't sniff the packets we're resending
The problem is that the packets appear to be missing data in the first two headers. I have set the debug server to save each received packet to a file, and set the application to connect directly to the debug server to compare the known good packet data with what the sniffer is sending. In the first packet, the first 1546/2079 bytes are good, but after that, each byte is zero instead of full of the correct data. In the second packet, the first 11 bytes are missing, but the rest is good.
Interestingly, after those initial two big setup packets, the remainder of the TCP packets seem to be sniffed properly - perhaps because they are usually far simpler and less than 40 bytes each.
Is there a better way to read packet data? Am I missing something? Unfortunately I don't have access to the source of the application, so I can't tell if it's doing anything special with those two big packets I'm having trouble with.

The issue with the first packet could indicate a problem in your operating system's TCP stack w.r.t. fragment reassembly.
In any case, try using another tool like tcpdump or wireshark to capture the packets. If they have the same problem, the problem lies with your operating system. If not, it could be a bug or configuration issue with scapy.

It could also mean that your IP packets are fragmented. Scapy does not automatically defragment packets, but fragments them.
you need to use the defrag function, or defragment (have a look at help(defrag) ), on the received packet list.
Maybe the packet you are checking is a fragment

Python socket recv() doesn't get every message if send too fast

I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()

You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.

UDP fragmentation in Python standard library

Am I right that UDP fragmentation should not be handled by developer himself in Python?
Am I right that I'll get my UDP packet with all originally sent data and not just one fragment of it?
while True:
data, addr = s.recvfrom(65535)
# Process packet

You will get the whole datagram that was sent through another UDP socket in one send or write operation. UDP is not stream oriented like TCP.

Can't receive UDP packet in Python

I'm having trouble receiving a UDP packet sent from an FPGA in a python program. I've checked similar questions and did the following:
Checked that Wireshark can the see UDP packet
Disabled windows firewall in PC
Used sock.bind() since it's UDP packets
Manually set the destination MAC address on Ethernet frame since FPGA does not support ARP
Set dest IP to broadcast 10.10.255.255 for testing, no packets received
Set the UDP checksum of the packet from the sender to 0x0000
Here's the python receiver code:
import socket
import sys
UDP_IP = "10.10.10.87"
UDP_PORT = 4660
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((UDP_IP, UDP_PORT))
print("Socket: "+str(sock.getsockname()))
while True:
data, addr = sock.recvfrom(1024) # buffer size is 1024 bytes
print(data)
print(addr)
sys.stdout.flush()
When tested against another python script that sends to 10.10.10.87:4660
(from another PC in the 10.10.10 network) the receiver script works fine. I've even tried to recreate the UDP pcket byte-by-byte in the FPGA from packets that I know are received OK (Differences are source IP, port & MAC, checksums (disabled), identification).
Here's the output for both packets from Wireshark:
Wireshark UDP packet (Python UDP packet that gets received OK on the left, Xilinx FPGA packet that is not received by python on the right)
I'm not sure what else to try. Any help would be appreciated.

Apparently the IPv4 header checksum from an FPGA calculation was wrong. It can get confusing to follow since the TTL (Time to Live) changes on a router hop, and the new TTL will also change the IPv4 header, forcing a new checksum per hop until it gets to Wireshark on the receiver end. By default Wireshark has IPv4 checksum validation disabled (as can be seen in the question screenshot), the answer is easier to spot with validation on.
I set the IPv4 checksum during packet construction to zero (x0000). It gets recalculated correctly at the router, and with the correct checksum Python can receive the packet.
I also tested a direct connection (without router) from the FPGA to the host PC. The IPv4 header also gets recalculated correctly (I'm not sure where, probably the PC's NIC?)
Hope this is useful to someone with similar problems.

Python client server how UDP is supposed to work?

I have a client-server "snake" game working really well with TCP connections, and I would like to try it the UDP way.
I wonder how it is supposed to be used ? I know how UDP works, how to make a simple ECHO example, but I wonder how to do the following :
For instance with TCP, every TICK (1/15 second) server sends to the client the new Snake head position.
With UDP, am I supposed to do something like this :
Client SIDE :
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
serverAddr = (('localhost', PORT))
while 1:
client.sendto('askForNewHead', serverAddr)
msg, addrServer = client.recvfrom(1024)
game.addPosition(msg)
Server SIDE :
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((HOST, PORT))
while 1:
data, addr = server.recvfrom(1024)
server.sendto(headPosition, addr)
So here Client has to ask server to get the new head position, and then server sends the answer. I managed to make it work this way, but I can't figure out if it is a good way of doing.
Seems weird that client has to ask udp for an update while with my TCP connection, client has just to wait untill he receives a message.

There are differences between TCP and UDP but not the way you describe. Like with TCP the client can recvfrom to get messages from the server without asking each time for new data. The differences are:
With TCP the initial connect includes a packet exchange between client and server. Unless the client socket was already bound to an IP and port it will be bound to the clients IP and a free port will be allocated. Because of the handshake between client and server the server knows where to contact the client and thus can send data to the packet without getting data from the client before.
With UDP there is no initial handshake. Unless already bound, the socket will be bound to clients IP and a free port when sending the first packet to the server. Only when receiving this packet the server knows the IP and port of the client and can send data back.
Which means, that you don't need to 'askForNewHead' all the time. Instead the client has to send only a single packet to the server so that the server knows where to send all future packets.
But there are other important differences between TCP and UDP:
With UDP packets may be lost or could arrive in a different order. With TCP you have a guaranteed delivery.
With UDP there is no real connection, only an exchange of packets between two peers. With TCP you have the start and end of a connection. This is relevant for packet filters in firewalls or router, which often need to maintain the state of a connection. Because UDP has no end-of-connection the packet filters will just use a simple timeout, often as low as 30 seconds. Thus, if the client is inside a home network and waits passively for data from server, it might wait forever if the packet filter closed the state because of the timeout. To work around this data have to be transmitted in regular intervals so that the state does not time out.
One often finds the argument, that UDP is faster then TCP. This is plain wrong. But you might see latency problems if packets get lost because TCP will notice packet loss and send the packet again and also reduce wire speed to loose less packets. With UDP instead you have to deal with the packet loss and other congestion problems yourself. There are situations like real time audio, where it is ok to loose some packets but low latency is important. These are situations where UDP is good, but in most other situations TCP is better.

UDP is different to TCP, and I believe with python the client does have to ask for an update from the server.
Although it is fun to learn and use a different way of communicating over the internet, for python I would really recommend sticking with TCP.

You don't have to ask the server for a update. But since UDP is connection-less the server can send head-positions without being asked. But the client should send i'm-alive-packets to the server, but this could happen every 10 seconds or so.