I have a capture of some TCP packets in pcapng format and I'd like to open it in python to inspect the TCP payloads with address 192.168.1.198. I've only found this library: https://python-pcapng.readthedocs.io/en/latest/api/blocks.html but it does not support inspecting TCP payloads.
Is there an easy way?
You can use python-pcapng package. First install python-pcapng package by following command.
pip install python-pcapng
Then use following sample code.
from pcapng import FileScanner
with open(r'C:\Users\zahangir\Downloads\MDS19 Wireshark Log 08072021.pcapng', 'rb') as fp:
scanner = FileScanner(fp)
for block in scanner:
print(block)
print(block._raw) #byte type raw data
Above code worked for me.
Reference: https://pypi.org/project/python-pcapng/
Related
I am reading a pcap file I have acquired with tcpdump. The pcap file is ~500MB. I read the file with FileCapture() and then I want to loop through each packet to extract the TLS payload. When I create the FileCapture object I also use override_prefs={'tls.keylog_file': os.path.abspath('tlsKey') where tlsKey is the file with the master keys to decrypt the file. The decryption works just fine, I can extract all the information from each single packet. However, if I want to loop through each packet and extract some information, the loop stops working at the packet for which packet.number = 258. My file contains more than 258 packets. What is going on?
My code
import pyshark
import os
cap = pyshark.FileCapture('traffic.pcap')
for packet in cap:
print(packet.number)
if "IP" in packet:
print(packet)
print('Finished')
the last output I get is here. As you can see, the layer TLS does not get printed. Why?
Expected behavior
I would expect my script to print Finished at the end, but it doesn't. The for loop looks stuck. Since the pcap file is large I cannot attach it. Any explanation of what's happening?
Versions (please complete the following information):
OS: MacOS 13.1
pyshark version: 0.5.3
tshark version: TShark (Wireshark) 4.0.2 (v4.0.2-0-g415456d13370)
I have downloaded a pickle file:
foo.pickle.gz.pickle
The page from where I downloaded this file describes decompressing it to .pickle. I searched about python pickle, there are many pages that describe how to use in python, but not system wide. How can I decompress or unzip it? I am using ubuntu 16.04
Thanks in advance!
Pickle is the name of Python object serialisation module. So, you have to 'unpickle' it with a python script. Basic synthax is:
import pickle
with open('filename', 'rb') as pickled_one:
data = pickle.load(pickled_one)
More details are available here, on official Python documentation.
I do have to warn you about this, from that same page:
The pickle module is not secure against erroneous or maliciously
constructed data. Never unpickle data received from an untrusted or
unauthenticated source.
Pickle object can only be deserialized in python. You can't use non-python environments to deserialize the object. Please see the official page
If there are multiple pickled objects, as the answers above only unpickle 1 object.Use
pickle_list =[]
pickle_file = open(file_name, 'rb')
while True:
try:
pickle_list.append(pickle.load(pickle_file))
except EOFError:
break
pickle_file.close()
Not able to indent the code properly, but try and except are inside the while loop
I have about 10GB pcap data with IPv6 traffic to analyze infos stored in IPv6 header and other extension header. To do this I decided to use Scapy framework. I tried rdpcap function , but for such big files it is not recommended. It tries to load all file into memory and get stuck in my case.
I found in the Net that in such situation sniff is recommended, my code look like:
def main():
sniff(offline='traffic.pcap', prn=my_method,store=0)
def my_method(packet):
packet.show()
In function called my_method I receive each packet separately and I can parse them, but....
When I call show function with is in-build framework method I got sth like this:
When opened in wireshark I got properly looking packet:
Could you tell me how to parse this packets in scapy to get proper results?
EDIT:
According to the discussion in comments I found a way to parse PCAP file with Python. In my opinion the easies way is to use pyshark framework:
import pyshark
pcap = pyshark.FileCapture(pcap_path) ### for reading PCAP file
It is possible to easily iterate read file with for loop
for pkt in pcap:
#do what you want
For parsing IPv6 header following methods may be useful:
pkt['ipv6'].tclass #Traffic class field
pkt['ipv6'].tclass_dscp #Traffic class DSCP field
pkt['ipv6'].tclass_ecn #Traffic class ECN field
pkt['ipv6'].flow #Flow label field
pkt['ipv6'].plen #Payload length field
pkt['ipv6'].nxt #Next header field
pkt['ipv6'].hlim #Hop limit field
Update
The latest scapy versions now support ipv6 parsing.
So to parse an ipv6 ".pcap" file with scapy now it can be done like so:
from scapy.all import *
scapy_cap = rdpcap('file.pcap')
for packet in scapy_cap:
print packet[IPv6].src
Now as I had commented back when this question was originally asked, for older
scapy versions (that don't support ipv6 parsing):
pyshark can be used instead (pyshark is a tshark wrapper) like so:
import pyshark
shark_cap = pyshark.FileCapture('file.pcap')
for packet in shark_cap:
print packet.ipv6.src
or even of course tshark (kind of the terminal version of wireshark):
$ tshark -r file.pcap -q -Tfields -e ipv6.src
If you want to keep using scapy and read the file Iteratively I'd recommend you to give it a shot to PcapReader()
It would do the same you tried to do with pyshark but in Scapy
from scapy.all import *
for packet in PcapReader('file.pcap')
try:
print(packet[IPv6].src)
except:
pass
I'd recommend wrapping this around just as a failsafe if you have any packet that does not have an IPv6 address.
Following the links (Eg: https://pypi.python.org/packages/source/p/py-web-search/py-web-search-0.2.1.tar.gz#md5=4e2f7363acdc1e7c08629edfa7996a5a ) provided on pypi from a browser will allow us to download the source code.
Is there a way to do this from a script?
So far I have this:
import requests
s = requests.get('https://pypi.python.org/packages/source/p/py-web-search/py-web-search-0.2.1.tar.gz#md5=4e2f7363acdc1e7c08629edfa7996a5a')
with open('pws.tar.gz', 'w') as fp:
fp.write(s.text)
Note: Opening the file in binary mode causes this error TypeError: 'str' does not support the buffer interface
When I open the tar file using the archive manager it tells that an error occurred while loading the archive.
I tried printing s.text and then redirecting the output to pws.tar.gz but it makes no difference.
It's optional(if you want to download a very large file then you can turn it on)stream=True
import requests
s = requests.get('https://pypi.python.org/packages/source/p/py-web-search/py-web-search-0.2.1.tar.gz#md5=4e2f7363acdc1e7c08629edfa7996a5a',stream=True)
with open('pws.tar.gz', 'wb') as fp:
for chunk in s.iter_content():
if chunk:
fp.write(chunk)
fp.flush()
This post seems to think it would work with opening it in binary mode and using write(bytes(s.text, 'UTF-8')) would work.
Using urllib2 in Python 2.7.4, I can readily download an Excel file:
output_file = 'excel.xls'
url = 'http://www.nbmg.unr.edu/geothermal/GEOTHERM-30Jun11.xls'
file(output_file, 'wb').write(urllib2.urlopen(url).read())
This results in the expected file that I can use as I wish.
However, trying to download just an HTML file gives me an empty file:
output_file = 'webpage.html'
url = 'http://www.nbmg.unr.edu/geothermal/mapfiles/nvgeowel.html'
file(output_file, 'wb').write(urllib2.urlopen(url).read())
I had the same results using urllib. There must be something simple I'm missing or don't understand. How do I download an HTML file from a URL? Why doesn't my code work?
If you want to download files or simply save a webpage you can use urlretrieve(from urllib library)instead of use read and write.
import urllib
urllib.urlretrieve("http://www.nbmg.unr.edu/geothermal/mapfiles/nvgeowel.html","doc.html")
#urllib.urlretrieve("url","save as..")
If you need to set a timeout you have to put it at the start of your file:
import socket
socket.setdefaulttimeout(25)
#seconds
It also Python 2.7.4 in my OS X 10.9, and the codes work well on it.
So I think there maybe other problems prevent its working. Can you open "http://www.nbmg.unr.edu/geothermal/GEOTHERM-30Jun11.xls" in your browser?
This may not directly answer the question, but if you're working with HTTP and have sufficient privileges to install python packages, I'd really recommend doing this with 'requests'. There's a related answered here - https://stackoverflow.com/a/13137873/45698