I am reading USBPcap files created in Wireshark with large, isochronous packets up to 494439 bytes. I am then trying to parse the packets in python with scapy. Unfortunately, scapy 2.4.3 defaults to ignoring bytes past the 65535th byte for packets greater than 65535 bytes.
Is there a way to read packets larger than 65535 bytes?
In order to instruct scapy to read packets larger than 65535 bytes, you must read packets one at a time off the many pcap reading generators found in scapy.utils API. Specifically, the following classes within scapy.utils support reading large packets from a pcap or pcapng file:
class scapy.utils.PcapReader(filename)
class scapy.utils.PcapNgReader(filename)
class scapy.utils.RawPcapReader(filename)
class scapy.utils.RawPcapNgReader(filename)
Note that scapy.utils.rdpcap(filename, count=- 1) does not support this functionality.
My solution:
from scapy.all import *
packet_reader = RawPcapNgReader('my_pcapng_file.pcapng')
while True:
try:
# created my own usb_packet class to parse packet header on __init__
p = usb_packet(packet_reader.read_packet(size=500000)) # read packet up to 500 kB large
if p.filter(): # filter packets based on usb_packet custom filter method
print(len(p.hex_str), len(p.data_packets)) # do stuff to the filtered packets
except EOFError:
break
For capturing packets via sniff or similar, one might set the buffer size via class scapy.config.Conf.bufsize which defaults to 65536 to allow a 65535 byte packet to be captured (see docs). For pragmatic programming, one may set this configuration when reading pcaps as well rather than maintaining the size argument wherever read_packet() is called on a generator.
An interesting note: This default appears to be due to IPv4 packet structure in which the packet length is represented by a 16 bit number. In other words, the largest packet this header value can depict is 1111111111111111 bytes, or 2^16=65536. Counting from zero, we have the largest IPv4 packet being 65535 bytes. For USB, as in my case, larger packets are possible.
Related
I've searched in several places, but I didn't find a simple answer to this question -
I have a .pcap file, generated using Wireshark, with several packets in it, and I wish to extract from each packet it's TCP-Timestamp (TSval).
I've managed to open each packet using scapy
packets = rdpcap('pcap_file.pcap')
for packet in packets:
print(packet.payload.id)
but I can't find the TSval of the packet (even though I can see the TSval field in the packet when I open it with Wireshark as shown in the picture below).
Packets can be accessed like dictionaries whose keys are protocols and values are payloads. For instance you can print the TCP payload in a packet like this:
if TCP in packet:
packet[TCP].show()
Now to get the TSval of the payload you have to look in TCP options. Each TCP option is encoded by scapy as a couple (option name, option value). For the timestamp option, the option value is itself a couple (TSval, TSecr). So you can basically get what you want doing the following:
from scapy.all import TCP, rdpcap
packets = rdpcap('packets.pcapng')
for packet in packets:
if TCP in packet: # ignore packets without TCP payload
for opt, val in packet[TCP].options: # consider all TCP options
if opt == 'Timestamp':
TSval, TSecr = val # decode the value of the option
print('TSval =', TSval)
I like to read a pcap file through scapy and use a filter of source address,destination address and length of the packet above or equal to 400,
After matching those packets, i would like to remove the first 16 bytes and then extract the remaining bytes sequentially.
file=rdpcap(pcap)
for pkt in file:
if pkt[0].src=='198.18.32.1' and pkt[0].dst=='198.18.50.97':
This is more than normal. pkt[0] does not really mean anything...
When doing pkt.src, you’ll get the Ethernet address (of the first layer), which is no IP address. You would need pkt[IP].src to get the IP address
I have a pcap file which contains traffic trace from an experiment in a binary format. What I'm trying to do is to find out how much data different hosts are exchanging with each other, but I'm pretty new to working with pcap and I've been searching and trying different things without success.
Can tcpdump be useful here? I've processed the raw file with it and got something like this:
2009-12-17 17:26:04.398500 IP 41.177.117.184.1618 > 41.177.3.224.51332: Flags [P.], seq 354231048:354231386, ack 3814681859, win 65535, length 338
2009-12-17 17:26:04.398601 IP 90.218.72.95.10749 > 244.3.160.239.80: Flags [P.], seq 1479609190:1479610159, ack 3766710729, win 17520, length 969
2009-12-17 17:26:04.398810 IP 244.3.160.239.80 > 90.218.72.95.10749: Flags [.], ack 969, win 24820, length 0
2009-12-17 17:26:04.398879 IP 41.177.3.224.51332 > 41.177.117.184.1618: Flags [P.], seq 1:611, ack 338, win 65535, length 610
Are the "length" values at the end of each line good indicators of how much data two hosts have transferred to each other?
The problem is that if I look at the raw file with Wireshark it seems like this length is actually the TCP header length, however the data/payload size is specified separately in Wireshark (38 bytes for the first of these four packets) which is confusing me.
So to sum up - Wireshark says (about the first packet):1) "396 bytes on wire", 2) "96 bytes captured", 3) "len: 338", 4) "Data (38 bytes)".
Tcpdump says: "length 338"
How do I find payload size? I'm willing to use Python if possible as I'll be working with a huge capture file.
Can tcpdump be useful here?
Yes.
Are the "length" values at the end of each line good indicators of how much data two hosts have transferred to each other?
Yes. That's the amount of bytes transferred sans headers.
How do I find payload size? I'm willing to use Python if possible as I'll > be working with a huge capture file.
You didn't specify a protocol so let's assume by "payload size" you mean "everything after the IP header". This is easy to do with Python and dpkt. As per Jon's tutorial, assuming IP packets with no options, some code that does probably what you want looks like this:
#!/usr/bin/env python
import dpkt
from socket import inet_ntoa
with open("sample.pcap") as f:
pcap = dpkt.pcap.Reader(f)
for ts, buf in pcap:
ip = dpkt.ethernet.Ethernet(buf).data
print "{} --> {} {}".format(inet_ntoa(ip.src), inet_ntoa(ip.dst), ip.len)
The following code is for a python server that can receive a string.
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 8001
BUFFER_SIZE = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connection address:', addr
while 1:
length = conn.recv(1027)
data = conn.recv(int(length))
import StringIO
buff = StringIO.StringIO()
buff.write(data)
if not data: break
print "received data:", data
conn.send('Thanks') # echo
get_result(buff)
conn.close()
Can anyone help me to edit this code or create a similar one to be able to receive images instead of string?
First, your code actually can't receive a string. Sockets are byte streams, not message streams.
This line:
length = conn.recv(1027)
… will receive anywhere from 1 to 1027 bytes.
You need to loop around each recv and accumulate a buffer, like this:
def recvall(conn, length):
buf = b''
while len(buf) < length:
data = conn.recv(length - len(buf))
if not data:
return data
buf += data
return buf
Now you can make it work like this:
while True:
length = recvall(conn, 1027)
if not length: break
data = recvall(conn, int(length))
if not data: break
print "received data:", data
conn.send('Thanks') # echo
You can use StringIO or other techniques instead of concatenation for performance reasons, but I left that out because it's simpler and more concise this way, and understanding the code is more important than performance.
Meanwhile, it's worth pointing out that 1027 bytes is a ridiculous huge amount of space to use for a length prefix. Also, your sending code has to make sure to actually send 1027 bytes, no matter what. And your responses have to always be exactly 6 bytes long for this to work.
def send_string(conn, msg):
conn.sendall(str(len(msg)).ljust(1027))
conn.sendall(msg)
response = recvall(conn, 6)
return response
But at least now it is workable.
So, why did you think it worked?
TCP is a stream of bytes, not a stream of messages. There's no guarantee that a single send from one side will match up with the next recv on the other side. However, when you're running both sides on the same computer, sending relatively small buffers, and aren't loading the computer down too badly, they will often happen to match up 1-to-1. After all, each time you call recv, the other side has probably only had time to send one message, which is sitting in the OS's buffers all by itself, so the OS just gives you the whole thing. So, your code will appear to work in initial testing.
But if you send the message through a router to another computer, or if you wait long enough for the other side to make multiple send calls, or if your message is too big to fit into a single buffer, or if you just get unlucky, there could be 2-1/2 messages waiting in the buffer, and the OS will give you the whole 2-1/2 messages. And then your next recv will get the leftover 1/2 message.
So, how do you make this work for images? Well, it depends on what you mean by that.
You can read an image file into memory as a sequence of bytes, and call send_string on that sequence, and it will work fine. Then the other side can save that file, or interpret it as an image file and display it, or whatever it wants.
Alternatively, you can use something like PIL to parse and decompress an image file into a bitmap. Then, you encode the header data (width, height, pixel format, etc.) in some way (e.g., pickle it), send_string the header, then send_string the bitmap.
If the header has a fixed size (e.g., it's a simple structure that you can serialize with struct.pack), and contains enough information for the other side to figure out the length of the bitmap in bytes, you don't need to send_string each one; just use conn.sendall(serialized_header) then conn.sendall(bitmap).
I am trying to sniff UDP packets using scapy sniff function, I send the packets in the Looback interface, the sending code is simple as follows:
from socket import *
IPv4 = "127.0.0.1"
Port = 45943
ClientSock = socket(AF_INET, SOCK_DGRAM)
while True:
MESSAGE = raw_input()
ClientSock.sendto(MESSAGE, (IPv4, Port))
However when i run (in another terminal tab after importing scapy):
a = sniff(iface="lo0", count = 5)
I get the following result:
>>> a.nsummary()
0000 Raw
0001 Raw
0002 Raw
0003 Raw
0004 Raw
whereas i am supposed to get UDP packets!, can any one point out to anything that i am missing here.
thanks
Unable to guess datalink type (interface=lo0 linktype=0)
That message translates as "Scapy doesn't understand the DLT_NULL link-layer header type, as used on the loopback device in *BSD and OS X, so it doesn't support the loopback device on *BSD and OS X".
So you're out of luck if you want to use Scapy on OS X to capture on the loopback device, unless and until Scapy is enhanced to handle DLT_NULL. (DLT_NULL is not that hard to handle, so presumably the only reason it's not handled is that most of the people using it on a loopback device are doing so on Linux, where the link-layer header type value on the loopback device is DLT_EN10MB, i.e. Ethernet, so nobody's bothered to fix it. I'll see if I can get it working and, if so, send them a patch.)
Some suggestions.
Instead of a.nsummary(), you can print out more information on individual packets using something like
a[1].show()
a[1].show2()
hexdump(a[1])
to examine the first packet.
2) You can force the protocol decoding to a particular type of packet format. For instance, a RAW_IP packet capture (link layer header type = 101) can be forced to be IPv6 using
conf.l2types.register(101, IPv6)
If you want to add a new layer on top of UDP, you can add a new dissector based on the port used.