I have one pcap file (~90M), and i want to replay that file. I came across scapy and it provides the way to read the pcap file and replay it. I tried following two ways to replay the packets
sendp(rdpcap(<filename>)
and
pkts = PcapReader(<filename>);
for pkt in pkts:
sendp(pkt)
First one game me memory error, memory consumption of the python process went up to 3 gig and finally it died. But second option worked fine for me because it did not read the whole file into memory. I have following three question
Is 90M pcap file is too big for scapy to replay?
Whenever we use tcpdump/wireshark, every packet has its timestamp associated with it. Assume packet 1 came at time T and packet 2 came at time T+10, will scapy replay the packets in similar manner, first packet at time T and second at T+10? or it will just keep sending them in loop, i think later is the case with PcapReader.
If the answer is no for above question ( its just replay in loop, without considering the packet inter arrival time), do we have any other python library which can do this job for me? Even python is not the constraint for me.
To answer your first question, well it sounds like you answered it yourself! Try running the first option again on another pcap file that's 40-50 MB instead and see if that errors out. That way you can at least check it the file is to big for your system in combination with Scapy to handle (not enough RAM in your system to handle how Scapy runs its algorithms as it was built to handle a few packets at a time, not a 90MB pcap file) or if it's just something in the code.
To answer your second question, based off of reading's I've been doing on Scapy over the past few weeks I strongly believe that this is a yes. However, I don't know of any sources off the top of my head to back up this verification.
Ninja edit - I saw this on another StackOverflow question - Specify timestamp on each packet in Scapy?
While that is for a single packet - if every packet is timestamped within Scapy then I imagine that it would be the same for every packet in a large pcap file that you read in. In this way when you replay the packets it should go in the same order.
A lot of educated guessing going on in this answer, hope it helps you though!
No. It shouldn't take up 3GBs of memory. I frequently open up larger pcap files on machines with only 2GBs of memory. Just try doing pkts = rdpcap(<filename>) to see how much memory that takes, then go from there. If the problem persists, you may wish to try different versions of scapy.
No, sendp() does not do this by default. You could try the realtime parameter (type help(sendp) on the console). But overall, based on my experience, scapy isn't so good at keeping accurate timing.
tcpreplay (linux CLI tool) is what I use. It has many options including various time keeping mechanisms.
Related
I apologize if my question is incorrect.
I want to write a python program that will sniff the interface, look for unique SIP sessions and RTP packets, and then write them to a separate PCAP file. those. 1xSIP call (SIP+RTP) == 1xPCAP file.
I don't quite understand how this can be done. I understand that the solution should be simple, but I just can't figure out how to make it work. Mb has anyone experienced this or can tell me how to do it?
Tried using pyshark. But it was not possible to select individual SIPs in live capture and send them to different PCAPs. I select SIP Calls by Call-ID. There may also be several simultaneous SIP calls, which also causes difficulties. Is it possible to do without multithreadings?.. Or maybe there are other solutions much easier?
Is there a way to capture and write very fast serial data to a file?
I'm using a 32kSPS external ADC and a baud rate of 2000000 while printing in the following format: adc_value (32bits) \t millis()
This results in ~15 prints every 1 ms. Unfortunately every single soulution I have tried fails to capture and store real time data to a file. This includes: Processing sketches, TeraTerm, Serial Port Monitor, puTTY and some Python scripts. All of them are unable to log the data in real time.
Arduino Serial Monitor on the other hand is able to display real time serial data, but it's unable to log it in a file, as it lacks this function.
Here's a printscreen of the serial monitor in Arduino with the incoming data:
One problematic thing is probably that you try to do a write each time you receive a new record. That will waste a lot of time writing data.
Instead try to collect the data into buffers, and as a buffer is about to overflow write the whole buffer in a single and as low-level as possible write call.
And to not stop the receiving of the data to much, you could use threads and double-buffering: Receive data in one thread, write to a buffer. When the buffer is about to overflow signal a second thread and switch to a second buffer. The other thread takes the full buffer and writes it to disk, and waits for the next buffer to become full.
After trying more than 10 possible solutions for this problem including dedicated serial capture software, python scripts, Matlab scripts, and some C projects alternatives, the only one that kinda worked for me proved to be MegunoLink Pro.
It does not achieve the full 32kSPS potential of the ADC, rather around 12-15kSPS, but it is still much better than anything I've tried.
Not achieving the full 32kSPS might also be limited by the Serial.print() method that I'm using for printing values to the serial console. By the way, the platform I've been using is ESP32.
Later edit: don't forget to edit MegunoLinkPro.exe.config file in the MegunoLink Pro install directory in order to add further baud rates, like 1000000 or 2000000. By default it is limited to 500000.
I am using Scapy to capture Wi-Fi client request frames. I am only interested in the MAC and requested SSID address of the clients. I do something like the following.
sniff(iface="mon0", prn=Handler)
def Handler(pkt):
if pkt.hasLayer(Dot11):
if pkt.type == 0 and pkt.subtype == 4:
print pkt.addr2 + " " + pkt.info
My issue is that I am doing this on a embedded device with limited processing power. When I run my script, my processor utilization rises to nearly 100%. I assume this is because of the sheer volume of frames that Scapy is sniffing and passing to my Python code. I also assume that if I could employ the right filter in my sniff command, I could eliminate many of the frames that are not being used and thus reduce the processor load.
Is there a filter statement that could be used to do this?
With Scapy it is possible to sniff with an BPF Filter applied to the capture. This will filter out packets at a much lower level then your Handler function is doing, and should significantly improve performance.
There is a simple example in the Scapy documentation.
# Could also add a subtype to this.
sniff(iface="mon0", prn=Handler, filter="type mgt")
Filter pieced together from here specifically.
Unfortunately I can't test this right now, but this information should provide you with a stepping stone to your ultimate solution, or someone else to post exactly what you need. I believe you will also need to set the interface to monitor mode.
You may also find this question of interest - Accessing 802.11 Wireless Management Frames from Python.
Scapy is extremely slow due to the way it decodes the data. You may
Use a BPF-filter on the input to only get the frames you are looking for before handing them to scapy. See this module for example. The module uses libpcap to get the data from the air or from a file first and passes it through a dynamically updated BPF-filter to keep unwanted traffic out
Write your own parser for wifi in c (which is not too hard, given the limited amount of information you need, there are things like prismhead though)
Use tshark from wireshark as a subprocess and collect data from there
I highly recommend the third approach although wiresharek comes with a >120mb library that your embedded device might not be able to handle.
I have to filter and modify network traffic using Linux kernel libnetfilter_queue (precisely the python binding) and dpkt, and i'm trying to implement delayed packet forward.
Normal filtering works really well, but if i try to delay packets with function like this
def setVerdict(pkt, nf_payload):
nf_payload.set_verdict_modified(nfqueue.NF_ACCEPT, str(pkt), len(pkt))
t = threading.Timer(10, setVerdict, [pkt, nf_payload])
t.start()
It crashs throwing no exception (surely is a low level crash). Can i implement delay using directly libnetfilter like this or I must copy pkt, drop it and send the copy using standard socket.socket.send()?
Thank you
Sorry for the late reply, but I needed to do something like this, although slightly more complicated. I used the C-version of the library and I copied packets to a buffer inside my program, and then issued a DROP verdict. After a timeout relating to your delay, I reinject the packet using a raw socket. This works fine, and seems quite efficient.
I think the reason for your crash was due to the fact that you didnt issue a verdict fast enough.
I can't answer your question, but why not use the "netem" traffic-queue module on the outgoing interface to delay the packet?
It is possible to configure tc queues to apply different policies to packets which are "marked" in some way; the normal way to mark such packets is with a netfilter module (e.g. iptables or nfqueue).
I am looking to create a client/server application that I can use to slit network packets in half, tunnel each half of the packet over a separate udp connection (because each udp connection will be going over a different wifi link) and then reassemble the split packets on the other end. In addition to splitting the packets each half packet will also have to have an ID and sequence number so that they can be reassembled properly.
Basically I am trying to do something similar to MLPPP
I am looking to do this using python and the TUN/TAP network driver.
I have found the following python code samples and modules that I think might be helpful for this project.
Python tun/tap
http://www.secdev.org/projects/tuntap_udp/files/tunproxy.py
http://twistedmatrix.com/trac/browser/trunk/twisted/pair/tuntap.py
http://pastebin.com/gMB8zUfj
Python raw packet manipulation
http://libdnet.sourceforge.net/
http://pypi.python.org/pypi/pyip/
http://code.google.com/p/python-packet/
My question is can the necessary packet modification be done using python and what would be a possible way to approach this? Can I use the modules above to do this or is there a better solution? I am looking for some input that will steer me in the right direction as I am not an experienced programmer. Any code samples or additional links are welcome.
We are doing something like this in production and it works quite well. We don't split individual packets though. We set fractional weights for each connection (unlimited) and send the packets out. We have some code in place to deal with different latencies on each line. On the other end we buffer them and reorder. Performance is pretty good - we have sites with 5+ ADSL lines and get good speeds, 40+ Mbps on the download.
Splitting packets (eg 1500/2 = 750) would introduce unnecessary overhead... keep your packets as big as possible.
We have developed our own protocol (header format) for the UDP packets. We have done loopback testing on the tun/tap up to 200 Mbps, so definitely the kernel to user space interaction works well. Previously we used NFQUEUE but that had reliability issues.
All of the above was written in Python.
It looks perfectly possible to me.
The tun/tap modules you've discovered look like they would do the job. Twisted will be high performance and the cost of hurting your head working it all out.
As for splitting the packets, you don't need to interpret the data in any way, just treat it as a blob of binary data, split it in two and add a header - I wouldn't use any 3rd party modules for that, just normal python string handling.
Or you could use netstrings if you want an easy to use packet encapsulation format.
I don't suppose it would go like a rocket, but I'm sure you would learn lots doing it!