We Keep Coding

Usage of scapy tcp_reassembly classmethod when using TCPSession?

im currently working on a new protocol layer for scapy. unfortunately the tcp_reassembly gives me a headache.
the protocol i dissect uses a bundle_len field that specifies how long a bundle is.
if the packet is exactly as long as the bundle len everything works.
if it is to large i split off one packet and return the rest to the queue (i think)
if the packet is less then the bundle_len im sure i would have to wait for additional packets or prepend the rest of the previous (to long) packet.
i know what the problem is but can't figure out how to use the tcp_reassembly function correctly.
here some example code:
def tcp_reassemble(cls, data, metadata):
"""[called by sniff(session=TCPSession),
reassembles the tcp stream if packet spans over multiple TCP packets]
Args:
data ([Packet]): [a raw packed strippt by the TCP Layer]
metadata ([dict]): [stores partial streams]
Returns:
[Packet]: [reassembled Packet]
"""
#pylint: disable=unused-argument
length = struct.unpack("<I", data[24:28])[0] # get bundle_len, not readable if offset wrong or not a bundle
if length > 10000: # desaster containment, not good.
length = 64
if len(data) > length: # got to much
# return OWN_PROTOCOL bundle up to bundle_len
pkt = OWN_PROTOCOL(data[:length])
if hasattr(pkt.payload, "tcp_reassemble"): # not sure if needed or what it does
if pkt.payload.tcp_reassemble(data[length:], metadata):
return pkt
else:
return pkt
elif len(data) < length: # got less, not working
print(
f"### Got LESS actual len: {len(data)} proposed bundle_len: {length} ###")
return None # push rest back to queue
else:
return OWN_PROTOCOL(data) # got exactly one bundle in one packet

How to extract an SSL/TLS message using scapy and python?

I'm trying to read a TLS message. Specifically, the one with the certificate details (handshake_type = 11). What I'm doing is first checking that the message contains Raw. If so, I'm extracting the payload like so: b = bytes(pkt[Raw].load). Next, I'm checking that the first byte is 0x16 and the following two bytes need to be a proper TLS version.
The problem is that this message doesn't pass these conditions. WireShark is showing me that \x16\x03\x03 are the bytes at position 0000 (picture is attached), but I guess it is done for convenience.
So is my assumption that the payload MUST start with 0x16 wrong?
P.S
I don't want to use scapy-ssl_tls library.
EDIT
This is the code:
def handle_tls_packet(pkt):
if pkt.haslayer(Raw):
b = bytes(pkt[Raw].load)
if b[0] == 0x16:
version = int.from_bytes(b[1:3], 'big')
message_len = int.from_bytes(b[3:5], 'big')
handshake_type = b[5]
handshake_length = int.from_bytes(b[6:9], 'big')
print("v = ", version, " len = ", message_len, " htype =", handshake_type
, "hlen =", handshake_length)
if handshake_type == 11:
# never happens - Why?
certs_len = int.from_bytes(b[7:11], 'big')
EDIT2:
As suggested by Cukic0d, I used load_layer("ssl").
Apparently, pkt[TLS].msg is a list (to hold multiple TLS messages in one packet?). Anyways, I printed the type of every such message like so:
def handle_tls_packet(pkt):
for tls_msg in pkt[TLS].msg:
print(type(tls_msg))
I expected to see a TLSCertificate object, yet such object is never seen.
Why?
EDIT3:
I'm willing to use scapy-ssl_tls if that would make life easier.

If you want to play with TLS handshake, enable TLS on scapy using load_layer("tls").
That enables the TLS module, which supports handshake (requires scapy >= 2.4.0). Scapy will then correctly dissect TLS handshake/key... packets
You should first try
load_layer("tls")
packets = sniff(prn=lambda x:x.summary(), lfilter=lambda x: TLS in x)
And if you're using Scapy 2.4.4+, for better consistency you can even use
sniff([...], session=TLSSession)
Have a look on how the packets are built:
Example:
There is also a quite fancy guide here: https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook2_tls_protected.ipynb
So summarize:
You will find each packet when using load_layer("tls").
Note that there are a lot of packets and that TLSCertificate will only appear once. msg is a list because many informations can be contained in a single TLS packet

Why can my UDP socket handle at most 12 packets before dropping them

So I've been working on a UDP server-client program in Python (using Ubuntu, in the event that it's relevant). Right now, I've got an implementation to mitigate packet loss, by essentially having the client acknowledge received packets before the server sends more. In my attempt to speed up the performance of this transfer, I began sending ACKs less frequently (I was originally ACKing every packet). In my testing, I've found that 12 packets is, for whatever reason, the maximum amount of packets I can guarantee to be delivered. I'm curious as to why 12 is the "magic number" for my socket.
I'm sending packets of 14000 bytes on my local 127.0.0.1 network.
And to test if this was a matter of how many bytes the socket could receive, I attempted to lower the number of bytes/packet, and then increase the number of packets to ACK. So, if 14000*12 = 168000, I tried various combinations of packet size and number of packets that all totaled 168000 bytes. In any case, 12 was still the magic number.
I've tested this on other computers as well, to be sure that it wasn't an issue with the socket buffers, MTU, or other system dependent variable.
EDIT:
I've even tried a similar process in a different program. I've been working on rewriting this program, but still keeping the central idea (less frequent ACKs) the same, and 12 still seems to be the "lucky" number.
My code is below:
Client:
try:
while(dataChunk):
if(numRecv >= 12):
#print "Stuff"
sock.settimeout(.1)
tcp_sock.send("I got 12 packets")
count +=1
numRecv = 0
else:
actualNumChunks += 1
numRecv +=1
f.write(dataChunk)
sock.settimeout(.1)
dataChunk, addr = sock.recvfrom(buf)
packID, junkAddr = sock.recvfrom(buf)
packIDString = str(packID)
#tcp_sock.send(packIDString)
else:
print "Data no longer being recv"
except socket.timeout:
f.close()
sock.close()
Server:
for chunk in read_in_chunks(f):
if(s.sendto(chunk,addr) and s.sendto(str(local_ID),addr)):
numSent += 1
'''
Store the packet being sent in the appropriate spot
Indices are arranged by packet localID, and each spot contains the packets load (as a string)
'''
packetQueue.insert(local_ID,chunk)
'''
Wait on ACK packets from the client
We will receive these packets over TCP (expecting the unique Packet ID/sequence number)
We won't send another packet until we've received the ACK from the client
If, after x seconds, we haven't received an ACK, we will assume packet was lost, and queue it for retransmission
'''
if(numSent >= 12):
conn.settimeout(.05)
try:
ACK = conn.recv(tcp_buf)
if ACK:
numSent = 0
else:
print "ACK is NONE!"
except socket.timeout:
ACK = None
print "Socket timed out (after "+str(count) +" loops) because we didn't get an ACK, let's send him some more packets"
#conn.settimeout(5)
while (ACK is not "I got 12 packets"):
"""
Continue with the main loop, and send more packets,
until we get something back (i.e. the client got another 12 packets)
"""
bigDataChunk = packetQueue[local_ID] + packetQueue[local_ID] + packetQueue[local_ID]
s.sendto(bigDataChunk, addr)
try:
ACK = conn.recv(tcp_buf)
print "He just ACKed the big ass data chunk: ",ACK
except socket.timeout:
print "Socket just timed out again..."
if ACK:
print "This is the ACK: ", ACK
break
else:
print "Need to send more packets yet"
else:
print "We got an ACK: " + ACK
'''
Compare this ACK (likely the packets received) against what we know we sent.
There is probably some discrepancy that will need to be addressed. That is to say, he is probably missing some
packets, and we're gonna have to resend them now, or make a list to send them later
'''
count += 1
numSent = 0
local_ID += 1
Just as a note to whoever might have an answer to this: I'm really asking out of curiosity. In practice, I'm going to have to tolerate a certain level of packet loss, and then retransmit lost datagrams. However, there comes a point at which only having 12 packets delivered is way too few, even if I did allow for a certain amount of data loss. This program is designed for large file transfers, and I'm going to need to get more than 12 packets consistently (try several thousands), as I can't constantly be asking for packets to be resent.
Thank you

Specifying packet length with scapy

I'm trying to send a specific packet size (100 bytes) with scapy but cant seem to get it.
I'm using this to start.
sr(IP(dst="192.168.1.1")/TCP(dport=443))
Looking at the docs / help I cant tell if I can use PacketLenField to specify the length of the packet. I can do it with NMAP & NSE but would like to do it outside of NMAP.
Any ideas on this one?
Thanks!

You can just add on the required number of bytes as a String when crafting the packet e.g.:
payload = 'ZZZZZZZZZZZZZZZZZZZZZ'
pkt = Ether() / IP() / TCP() / payload
will work. You just need to adjust the length of the payload as you require.

Scapy's Raw() function populates the payload of the packet. If you know your header size, you only need to fill in the remaining bytes with random data.
You can use RandString() to generate random padding. The following command sends a packet of length 100 (and listens for a response):
sr(IP(dst="192.168.1.1")/TCP(dport=443)/Raw(RandString(size=72))

You may use inet.Padding() from scapy library:
packet = IP(dst="192.168.1.1")/TCP(dport=443)
if len(packet)<100:
#"\x00" is a single zero byte
myString = "\x00"*(100 - len(packet))
packet = packet/inet.Padding(myString)

How do I dump the TCP client's buffer in order to accept more data?

I've got a simple TCP server and client. The client receives data:
received = sock.recv(1024)
It seems trivial, but I can't figure out how to recieve data larger than the buffer. I tried chunking my data and sending it multiple times from the server (worked for UDP), but it just told me that my pipe was broken.
Suggestions?

If you have no idea how much data is going to pour over the socket, and you simply want to read everything until the socket closes, then you need to put socket.recv() in a loop:
# Assumes a blocking socket.
while True:
data = sock.recv(4096)
if not data:
break
# Do something with `data` here.

Mike's answer is the one you're looking for, but that's not a situation you want to find yourself in. You should develop an over-the-wire protocol that uses a fixed-length field that describes how much data is going to be sent. It's a Type-Length-Value protocol, which you'll find again and again and again in network protocols. It future-proofs your protocol against unforeseen requirements and helps isolate network transmission problems from programmatic ones.
The sending side becomes something like:
socket.write(struct.pack("B", type) #send a one-byte msg type
socket.write(struct.pack("H", len(data)) #send a two-byte size field
socket.write(data)
And the receiving side something like:
type = socket.read(1) # get the type of msg
dataToRead = struct.unpack("H", socket.read(2))[0] # get the len of the msg
data = socket.read(dataToRead) # read the msg
if TYPE_FOO == type:
handleFoo(data)
elif TYPE_BAR == type:
handleBar(data)
else:
raise UnknownTypeException(type)
You end up with an over-the-wire message format that looks like:
struct {
unsigned char type;
unsigned short length;
void *data;
}

Keep in mind that:
Your operating system has it's own idea of what it's TCP/IP socket buffer size is.
TCP/IP packet maximum size (generally is 1500 bytes)
pydoc for socket suggests that 4096 is a good buffer size
With that said, it'd really be helpful to see the code around that one line. There are a few things that could play into this, if you're using select or just polling, is the socket non-blocking, etc.
It also matters how you're sending the data, if your remote end disconnects. More details.