We Keep Coding

scapy - Missing data in sniffed TCP packets

I have an application that sends data over a TCP connection to a production server. I need to sniff the contents of that TCP connection and resend it to a debug server.
I've gotten quite close with this:
from scapy.all import *
packets = 0
def dup_pkt(pkt):
global packets
read = raw(pkt[TCP].payload)
print(str(packets))
s.sendall(read)
print("connecting")
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("x.x.x.x", 12345))
print("connected")
print("sniffing")
pkts = sniff(prn=dup_pkt, filter="tcp dst port 12345 and not host x.x.x.x") # don't sniff the packets we're resending
The problem is that the packets appear to be missing data in the first two headers. I have set the debug server to save each received packet to a file, and set the application to connect directly to the debug server to compare the known good packet data with what the sniffer is sending. In the first packet, the first 1546/2079 bytes are good, but after that, each byte is zero instead of full of the correct data. In the second packet, the first 11 bytes are missing, but the rest is good.
Interestingly, after those initial two big setup packets, the remainder of the TCP packets seem to be sniffed properly - perhaps because they are usually far simpler and less than 40 bytes each.
Is there a better way to read packet data? Am I missing something? Unfortunately I don't have access to the source of the application, so I can't tell if it's doing anything special with those two big packets I'm having trouble with.

The issue with the first packet could indicate a problem in your operating system's TCP stack w.r.t. fragment reassembly.
In any case, try using another tool like tcpdump or wireshark to capture the packets. If they have the same problem, the problem lies with your operating system. If not, it could be a bug or configuration issue with scapy.

It could also mean that your IP packets are fragmented. Scapy does not automatically defragment packets, but fragments them.
you need to use the defrag function, or defragment (have a look at help(defrag) ), on the received packet list.
Maybe the packet you are checking is a fragment

Python socket recv() doesn't get every message if send too fast

I send mouse coordinates from python server to python client via socket. Mouse coordinates are send every time when mouse movement event is catch on the server which means quite often (dozen or so per second).
Problem is when I use python server and python client on different hosts. Then only part of messages are delivered to the client.
e.g. 3 first messages are delivered, 4 messages aren't delivered, 4 messages are delivered etc...
Everything is fine when server and client are on the same host (localhost).
Everything is fine when server and client are on different hosts but instead of python client I use standard windows Telnet client to read messages from the server.
I noticed that when I use time.sleep(0.4) break between each message that is send then all messages are delivered. Problem is I need that information in real time not with such delay. Is it possible to achieve that in Python using sockets?
Below python client code that I use:
import pickle
import socket
import sys
host = '192.168.1.222'
port = 8888
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error, msg:
print "Faile. Error:" + str(msg[0]), "Error message : " + msg[1]
sys.exit()
mySocket = socket.socket()
mySocket.connect((host,port))
while 1:
data = mySocket.recv(1024)
if not data: break
load_data = pickle.loads(data)
print 'parametr x: ' + str(load_data[0])
print 'parametr y : ' + str(load_data[1])
mySocket.close()

You are using TCP (SOCK_STREAM) which is a reliable protocol which (contrary to UDP) does not loose any messages, even if the recipient is not reading the data fast enough. Instead TCP will reduce the sending speed.
This means that the problem must be somewhere in your application code.
One possibility is that the problem is in your sender, i.e. that you use socket.send and do not check if all the bytes you've intended to send are really got send. But this check needs to be done since socket.send might only send part of the data if the socket buffer of the OS is full which can happen if the client does not read the data fast enough.
Another possibility is that your socket.recv call receives more data than your pickle.loads needs and that the rest of the data gets discarded (not sure if pickle.loads will throw an exception if too much data are provided). Note that TCP is not a message but a stream protocol so it might be that you have more that socket.recv will return a buffer which contains more than one pickled object but you only read the first. The chance that this will happen on a network is higher than on localhost because by default the TCP layer will try to concatenate multiple send buffers into a single TCP packet for better use of the connection (i.e. less overhead). And the chance is high that these will then be received within the same recv call. By putting a sleep(0.4) on the sender side you've effectively switch off this optimization of TCP, see NAGLE algorithm for details.
Thus the correct way to implement what you want would be:
Make sure that all data are delivered at the server, i.e. check the return of socket.send.
Make sure that you unpack all messages you receive. To do this you probable need to add some message layer on top of the TCP stream to find out where the message boundary is.

scapy sr function problems

I'm having trouble receiving all related packets to a request when using scapy's sr function.
ans, unans = sr(IP(dst="172.xxx.xxx.xxx")/TCP(dport=80,flags="S"))
returns:
Received 2 packets, got 1 answers, remaining 0 packets
What happens is that I first get an ICMP redirect. Afterwards I get the answer from my local service with SA. Sometimes it tells me it received two packets, which makes sense, but when I look at it in the summary it prints the following:
<bound method SndRcvList.summary of <Results: TCP:0 UDP:0 ICMP:1 Other:0>>
and in summary() I find this:
IP / TCP 172.xxx.xxx.xxx:ftp_data > 172.zzz.zzz.zzz:http S ==> IP / ICMP 172.yyy.yyy.yyy > 172.xxx.xxx.xxx redirect host-redirect / IPerror / TCPerror
For one, I wonder where my SA flagged TCP packet is. When I look at the network dump, I definitely see it, right after the ICMP packet. I've made sure to run scapy with and without running tcpdump, just in case it would interfere, which it shouldn't.
I've also tried to set and increased timeout, just in case it didn't wait long enough to receive the TCP packet. Didn't work.
I've also tried it out on loopback interface, the local lan and systems that are located in the internet. Same result everywhere.
Any ideas on where the error could be located?

Scapy considers this ICMP packet to be response on initial TCP SYN request (which it actually is). Try using sr(..., multi = True) to get multiple response packets.

Python Scapy - Loading HTTP from a file

I'm using this extension for scapy to detect and analyze HTTP packets. It works great, but when I save the HTTP packets to a pcap file with wrpcap and then load it with rdpcap it doesn't give me the same packet, it only detects its HTTP packet but not HTTP Requests, it also occurs when I do this -
from scapy.all import *
from scapy_http.http import *
packets = sniff(count=10, lfilter=lambda p: HTTPRequest in p)
wrpcap('file.pcap', packets)
restored = rdpcap('file.pcap')
print len([x for x in restored if HTTPRequest in p]) # prints 0
Why this is happening? how can I recover the packets?

I am very new to Python in general, Scapy in particular but is this what you are looking for?
from scapy.all import *
def http_request(pkt):
if pkt.haslayer('HTTPRequest'): ##Use HTTPResponse for response packets
pkt.show()
exit(0) ##Omit to show more then the first packet
pkts = rdpcap('/root/Desktop/example_network_traffic.pcap')
for p in pkts:
http_request(p)
##For sniffing packets
##sniff(prn=http_request)
I think the problem may be the way Scapy exorts packets. When I run your code and inspect the packet in Wireshark, the protocol is listed as TCP. When I use Wireshark to capture the same type of packet, it lists the protocol as HTTP. If I export the packet from Wireshark and read it using rdpcap, you get the results you are looking for, ie the HTTPRequest/HTTPResponse layers. I don't know this for fact, but I checked the Berkeley Packet Filter syntax, and they don't list HTTP as a protocol. If Scapy is based on the BPF syntax, and they don't use the HTTP protocol, maybe it exports the packet with a protocol of TCP and Scapy-Http just parses the Raw load during sniff(). Just a guess.

Python Twisted client not able to receive response from server

I have a client written using python-twisted (http://pastebin.com/X7UYYLWJ) which sends a UDP packet to a UDP Server written in C using libuv. When the client sends a packet to the server, it is successfully received by the server and it sends a response back to the python client. But the client not receiving any response, what could be the reason ?

Unfortunately for you, there are many possibilities.
Your code uses connect to set up a "connected UDP" socket. Connected UDP sockets filter the packets they receive. If packets are received from any address other than the one to which the socket is connected, they are dropped. It may be that the server sends its responses from a different address than you've connected to (perhaps it uses another port or perhaps it is multi-homed and uses a different IP).
Another possibility is that a NAT device is blocking the return packets. UDP NAT hole punching has come a long way but it's still not perfect. It could be that the server's response arrives at the NAT device and gets discarded or misrouted.
Related to this is the possibility that an intentionally configured firewall is blocking the return packets.
Another possibility is that the packets are simply lost. UDP is not a reliable protocol. A congested router, faulty networking gear, or various other esoteric (often transient) concerns might be resulting in the packet getting dropped at some point, instead of forwarded to the next hop.
Your first step in debugging this should be to make your application as permissive as possible. Get rid of the use of connected UDP so that all packets that make it to your process get delivered to your application code.
If that doesn't help, use tcpdump or wireshark or a similar tool to determine if the packets make it to your computer at all. If they do but your application isn't seeing them, look for a local firewall configuration that might reject them.
If they're not making it to your computer, see if they make it to your router. Use whatever diagnostic tools are available (along the lines of tcpdump) on your router to see whether packets make it that far or not. Or if there are no such tools, remove the router from the equation. If you see packets making it to your router but no further, look for firewall or NAT configuration issues there.
If packets don't make it as far as your router, move to the next hop you have access to. This is where things might get difficult since you may not have access to the next hop or the next hop might be the server (with many intervening hops - which you have to just hope are all working).
Does the server actually generate a reply? What addressing information is on that reply? Does it match the client's expectations? Does it get dropped at the server's outgoing interface because of congestion or a firewall?
Hopefully you'll discover something interesting at one of these steps and be able to fix the problem.

I had a similar problem. The problem was windows firewall. In firewall allowed programs settings, allowing the communication for pythonw/python did solve the problem. My python program was:
from socket import *
import time
address = ( '192.168.1.104', 42) #Defind who you are talking to (must match arduino IP and port)
client_socket = socket(AF_INET, SOCK_DGRAM) #Set Up the Socket
client_socket.bind(('', 45)) # arduino sending to port 45
client_socket.settimeout(1) #only wait 1 second for a response
data = "xyz"
client_socket.sendto(data, address)
try:
rec_data, addr = client_socket.recvfrom(2048) #Read response from arduino
print rec_data #Print the response from Arduino
except:
pass
while(1):
pass