I am trying to capture https traffic with scapy. So far I can see the TLS handshakes, but I am having trouble decoding the application data using a NSS keyfile. I'm using scapy 2.5.0rc2.dev39.
The structure of my script is: (1) set up the conf to point to the NSS file that will be created; (2) kick off a thread to get the sniff going; and then (3) do a https request on the main thread that generates the traffic that gets sniffed on the other thread.
I suspect my problem is a chicken-and-egg problem: I'm giving scapy the name of a file that doesn't exist yet and asking sniff to use it to decode traffic. But to generate the file, I need to generate traffic. But then it is too late to set up the sniff. And so on.
I'm basing the setup on the conf file on the code example in https://github.com/secdev/scapy/pull/3374. Is there a way to get the NSS file parsed and applied after the packets are sniffed? (Note: I have looked at https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb, but that example applies the NSS file to a captured pcap file. I don't see how to apply that to my scenario. Unless I export the sniffed packets to a pcap file and then apply the now-existent NSS file to it?)
Here's my code:
from scapy.all import *
from time import sleep
from IPython import embed
myiface = 'lo'
mycount = 30
response_time_delta = 0.0
NSS_SECRETS_LOG = "secrets.log"
SERVER_HOST = "127.0.0.1"
SERVER_PORT = 8443
def analyze_https_sniffed_traffic():
# sniff traffic for mycount packets
myfilter = ""
myprn = lambda x:x.summary()
sniff_out = sniff(filter=myfilter,prn=myprn,count=mycount,iface=myiface)
# iterate through the sniffed packets to report on contents
for idx,s in enumerate(sniff_out):
print("===\npacket %d\n%s" % (idx,s.summary()))
# if we can convert to a TLS packet, print out TLS summary
if s.haslayer('IP') and hasattr(s,'load'):
tls_r = TLS(s.load)
print(tls_r.summary())
# if this is TLS application data, do a complete show
if tls_r.type == 23:
print(tls_r.show())
#embed()
return
def send_https_request_and_analyze():
import http.client, ssl
# start another thread that sniffs traffic and analyzes their contents
t = threading.Thread(target=analyze_https_sniffed_traffic)
t.start()
# use python requests to make a HTTPS query to a local server
# put SSLKEYLOGFILE into the environment so I can decode captured TLS traffic
import os; os.environ["SSLKEYLOGFILE"] = NSS_SECRETS_LOG
time.sleep(3)
# unverified context: using self signed cert, make requests happy
conn = http.client.HTTPSConnection(SERVER_HOST, SERVER_PORT, context=ssl._create_unverified_context())
conn.request('GET', '/')
r = conn.getresponse()
print("response: %s" % r.read())
# wait for the sniff thread to finish up
t.join()
load_layer("tls")
# conf commands from https://github.com/secdev/scapy/pull/3374
conf.tls_session_enable = True
conf.tls_nss_filename = NSS_SECRETS_LOG
print("scapy version: %s" % scapy.__version__)
print("conf contents:\n%s" % conf)
send_https_request_and_analyze()
Here's what I get back for the first application data packet:
===
packet 22
Ether / IP / TCP 127.0.0.1:46066 > 127.0.0.1:8443 PA / Raw
TLS None:None > None:None / TLS / TLS Application Data
###[ TLS ]###
type = application_data
version = TLS 1.2
len = 91 [deciphered_len= 91]
iv = b''
\msg \
|###[ TLS Application Data ]###
| data = '\\xce\\xd2:\\x87\\xd0\\ h\x7f\\x81C\\xbd\x1af\\xd6y\\x91\x07wnn\\xca!ji3\\xb2\\xbbT\t\\xfc\\x80F#\\x88\x13<\\x83\\xa4\x08p\\x96\\xfb\\xf7\\x875u\\xfa\\xb9\x11\\x97M\\xf9\\xf5\\xb0\\x8fR\\x8c\ue650sI/ɇop\x1d\\xe2\n\\x91"\\x80\\x91la"d\\xe5\\xa0yk\\xc2\\xfa\\xe2A\\x8d\\x8dKB'
mac = b''
pad = b''
padlen = None
None
Any ideas on how I can make this work?
PS: Shout out to the devs. Scapy is amazing.
EDIT: I tried to get a write to pcap, set the NSS file in the conf, read the pcap approach, but ran into the same problem. I think I'm running into https://github.com/secdev/scapy/issues/3722.
EDIT #2: I went through the same steps decoding the raw data in https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb in the section "Decrypt a PCAP file" and it worked fine. When I compared the wireshark output for the notebook's pcap file and the one I wrote with wrpcap, I see duplicate packets. It's my PCAP file that is breaking the decrypt, I think. I'm out of time today, but I will try something along the lines of Scapy: Processing partial TLS segments and report back.
I see I didn't update this with my eventual solution. In one thread I kicked off the HTTPS request and generated the NSS key files; in another thread I sniffed the traffic, then afterwards I iterated through the packetlist, updating the session with the NSS keys and mirroring the session as needed, as described in the "Decrypting Manually" section of the Scapy TLS documentation here: https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook3_tls_compromised.ipynb.
Example code here: https://github.com/PollyP/scapy_tls_example
Related
I am trying to use scapy to send and receive packets to discover the network path. I would like to modify the packet details when it reaches a certain hop due to tunnels. For some reason, sniff is not matching on ICMP packets received. TCPDUMP on the clone session shows the ICMP packet was received on the sniff interface.
while lasthop:
outer_ttl = outer_ttl + 1
inner = IP(src=sw3, dst=sw3, flags="DF", ttl=inner_ttl, id=id) / UDP(sport=37631, dport=33435)
outer = IP(src=sw1, dst=sw2, flags="DF", ttl=outer_ttl, id=id) / GRE() / inner
if sniff(iface="ens1", count=1, filter="icmp and icmp[0]=11 and src host 10.1.1.1", prn=send(outer), timeout=1):
inner_ttl = 2
send(outer)
lasthop = 0
I tried sniff from the scapy console. Its the same problem. But interestingly it matches if i run traceroute for a clone session.
I have an extremely simple tcp server in python the code for which is below:
#!/usr/bin/env python
import socket
sock = socket.socket()
sock.bind(('',3912))
sock.listen(100)
num_cons = 10
cons = []
for i in range(num_cons):
con, addr = sock.accept()
cons.append(con)
while True:
for con in cons:
msg = "a"* 1000
num_sent = con.send(msg.encode())
print("sent: {} bytes of msg:{}".format(str(num_sent), msg))
The corresponding client code is
#!/usr/bin/env python
import socket
sock = socket.socket()
sock.connect(('',3912)) # in reality here I use the IP of the host where
# I run the server since I launch the clients on a different host
while True:
data = sock.recv(1000)
print("received data: {} ".format(str(data)))
Now, if I start the server with
./server.py
and 10 clients in parallel from a different host:
for i in `seq 1 10`; do ./client.py 2>/dev/null 1>/dev/null & done
And I send kill -SIGSTOP %1 to the first client, I expect the server to successfully keep trying to send data because it cannot know that the client has been stopped. Instead, the server blocks when it tries to send the data to client 1. I can understand the behaviour if the clients were on the same host as the server: we tried to write data, but the kernel buffers are full, so we block in the server, but the client never reads, so the buffer is never freed. However, if the clients are on a different machine, the kernel buffers of the server host should only be full temporarily and then the kernel should send the data over the network card and free them. So why is my server blocking on the send call? I have not verified if the same behaviour is seen when using a different language (C for example)
It is weird because 1000 characters is a small size for TCP. I have no available Linux machine but on a FreeBSD box, I could successfully send 130000 bytes on a TCP connection where the peer was stopped before the sender blocks. And more that 1000000 on Windows.
But as TCP is a connected protocol, a send call will block if it cannot queue its data because the internal TCP stack queue is full.
The gist of your problem seems to be that you're creating a SOCK_STREAM socket (i.e. TCP), and then abruptly terminating the client. As discussed in the Python Socket Programming HOWTO, a hang is expected in this situation.
TCP is a reliable protocol, meaning that every transmitted packet has to be acked. If the receiving side is dead, the sender will block waiting for that acknowledgement. Try setting a timeout and see if your send raises a socket.timeout after the expected time.
I am trying to use client side certificates for authentication in Python, but I just can't figure it out.
I am using this for my testing:
https://blog.cloudflare.com/introducing-tls-client-auth/
(Skip to "TLS Client Authentication On The Edge" for the pem file and example curl call).
It works as expected in curl, but when I try it in Python I get a 403 back from the server.
Here is what I have tried in Python:
import ssl
import socket
s = socket.socket()
addr_info = socket.getaddrinfo('auth.pizza',443)
addr = addr_info[0][-1]
ss = ssl.wrap_socket(s, certfile='pizza.pem')
ss.connect(addr)
ss.write(b'GET / HTTP/1.0\r\n\r\n')
print(ss.read(4096))
I'm at a loss for how to even debug this further.
I used SCAPY to write a program deployed in the WEB server and would like to send TCP RST using SCAPY to block some specific HTTP client access.
After running the program, the client uses Telnet to connect to the server WEB listening port can be SCAPY program interrupt, but the use of browser access can not interrupt. Why is it so?
The code is as follows:
# coding: utf-8
# web server : 10.28.16.20 ;
# http client : 10.28.1.70;
from scapy.all import *
def pkgs(pkg):
if pkg.getlayer(TCP) and pkg[IP].dst=="10.28.16.20" and "10.28.1.70" in pkg[IP].src:
resp=IP(dst=pkg[IP].src,src=pkg[IP].dst)/TCP(dport=pkg[TCP].sport,sport=pkg[TCP].dport,flags="RA",seq=pkg[TCP].ack,ack=pkg[TCP].seq+(len(pkg[TCP].payload) if pkg.getlayer(Raw) else 1))
send(resp,count=2,verbose=0)
if __name__=="__main__":
conf.L3socket=L3RawSocket
sniff(filter="tcp",prn=pkgs,store=0)
Program code screenshot
I think the best way is to get into "Scapy"
then paste that code and enter:
#!/usr/bin/python
from scapy.all import *
ip = IP(src="10.28.16.20", dst="10.28.1.70")
tcp = TCP(sport=80, dport=####, flags=####, seq=####, ack=####)
pkt = ip/tcp
ls(pkt)
send(pkt,verbose=0)
You need to change dport to the source port from the PC, flags to the action you want to do, seq&ack follow the packets you started from.
I have solved it.import threading.
I'm using Python 2.4.4 and OpenSSL 0.9.8k (not by choice)
I've referred to the documentation: https://docs.python.org/release/2.4.4/lib/module-socket.html
and to pretty much every mention of "openSSL" and "python" on the internet, and I haven't found a solution to my problem.
I'm simply writing a test program to initiate an SSL connection. Here is the code:
server
#!/usr/bin/python
import socket
import _ssl
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 4433))
s.listen(5)
while True:
client, address = s.accept()
ssl_client = socket.ssl(client,
keyfile='keyfile',
certfile='certfile')
print "Connection: ", address
data = ssl_client.read(1024)
if data:
print "received data: ", data
ssl_client.write(data + " Hello, World!")
del ssl_client
client.close()
client
#!/usr/bin/python
import socket
import _ssl
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('host', 4433))
ssl_s = socket.ssl(s,
keyfile='keyfile',
certfile='certfile')
print 'writing ', ssl_s.write("Hello, World!"), ' bytes to ssl stream'
data = ssl_s.read(1024)
del ssl_s
s.close()
print "received data: ", data
Some notes about this code - keyfile and certfile are paths to my actual key and cert file. Those arguments are not the issue. The hostnames are also not the issue. I'm aware that the port used is 4433 - in our requirements, we're meant to use a generic port, not 443. I was unaware that it was possible to use SSL over a different port, but regardless, even when I use 443 I get the exact same error.
I can run the server fine, and then when I run the client, I get the following error on the wrap_socket lines for both client and server:
error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
I've read it's due to using a non-443 port, but again, using 443 didn't fix things. I've read it could be a protocol mismatch, but the client and the server are both defaulting to SSL2.3. We're meant to use TLS1.2 as per our requirements, but the docs don't seem to have any information on how to set the SSL protocol version. I'm unsure if that's related to my issue. Please keep in mind I'm not here to open a dialogue regarding to use of outdated SSL and Python versions.
socket.ssl is only able to initiate a SSL connection and the given optional cert and key are for use of client certificates. socket.ssl is not able to be used on the server side and it looks like python 2.4.4 does not offer this feature in any of the core modules at all. In later versions of python you can use the ssl module for this but 2.4.4 does not seem to have this.