We Keep Coding

Counting the number of SYN and ACK packets using Python and Scapy

I am analysing a pcap file using Python and Scapy.
Currently, I have it counting the number of packets
I would like to count the number of SYN and ACK packets, is there a way to do this?
My main piece of code thus far is
for (pkt_data, pkt_metadata,) in RawPcapReader(file_name):
count+=1

the code is the folowing:
import scapy.all as scapy
from scapy.layers.inet import TCP
pkt_count = 0
pkt_tcp_ack_count = 0
pkt_tcp_syn_count = 0
for pkt in scapy.PcapReader(file_name):
pkt_count += 1
if TCP in pkt:
if "A" in pkt[TCP].flags:
pkt_tcp_ack_count += 1
if "S" in pkt[TCP].flags:
pkt_tcp_syn_count += 1
print("pkt_count: %d" % pkt_count)
print("pkt_tcp_ack_count: %d" % pkt_tcp_ack_count)
print("pkt_tcp_syn_count: %d" % pkt_tcp_syn_count)
now, a bit of context.
Scapy is building all the layers, so you can simply query for their presence in the packet.
for a given packet you can run:
pkt.show()
which show you how the packet has been decoded by scapy

unable to unpack information between custom Preamble in Python and telnetlib

I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)

Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.

I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.

Python sockets stealing each other's packets

I'm trying to write a program to test data transfer speeds for various-sized packets in parallel. I noticed something odd, though, that the size of the packet seemed to have no effect on transfer time according to my program, whereas the Unix ping binary would time out on some of the packet sizes I'm using. I was sending 4 packets containing the string 'testquest' and one that was just 2000 bytes set to 0. However, when I printed the results, they all contained 'testquest' (and were far shorter than 2000 bytes). The only thing I can conclude is that these sockets are somehow all receiving the same packet, which would explain how they all had the same rtt.
I made this MCVE to illustrate the issue (you can ignore the 'checksum' function, it's included for completeness but I know from experience that it works):
#!/usr/bin/env python3
import socket
import struct
import time
from multiprocessing.pool import ThreadPool as Pool
from sys import argv, byteorder
def calculate_checksum(pkt):
"""
Implementation of the "Internet Checksum" specified in RFC 1071 (https://tools.ieft.org/html/rfc1071)
Ideally this would act on the string as a series of 16-bit ints (host
packed), but this works.
Network data is big-endian, hosts are typically little-endian,
which makes this much more tedious than it needs to be.
"""
countTo = len(pkt) // 2 * 2
total, count = 0, 0
# Handle bytes in pairs (decoding as short ints)
loByte, hiByte = 0, 0
while count < countTo:
if (byteorder == "little"):
loByte = pkt[count]
hiByte = pkt[count + 1]
else:
loByte = pkt[count + 1]
hiByte = pkt[count]
total += hiByte * 256 + loByte
count += 2
# Handle last byte if applicable (odd-number of bytes)
# Endianness should be irrelevant in this case
if countTo < len(pkt): # Check for odd length
total += pkt[len(pkt) - 1]
total &= 0xffffffff # Truncate sum to 32 bits (a variance from ping.c, which
# uses signed ints, but overflow is unlikely in ping)
total = (total >> 16) + (total & 0xffff) # Add high 16 bits to low 16 bits
total += (total >> 16) # Add carry from above (if any)
return socket.htons((~total) & 0xffff)
def ping(args):
sock, payload = args[0], args[1]
header = struct.pack("!BBH", 8, 0, 0)
checksum = calculate_checksum(header+payload)
header = struct.pack("!BBH", 8, 0, checksum)
timestamp = time.time()
sock.send(header+payload)
try:
response = sock.recv(20+len(payload))
except socket.timeout:
return 0
return (len(response), (time.time() - timestamp) * 1000)
host = argv[1] # A host that doesn't respond to ping packets > 1500B
# 1 is ICMP protocol number
sockets = [socket.socket(socket.AF_INET, socket.SOCK_RAW, proto=1) for i in range(12)]
for i, sock in enumerate(sockets):
sock.settimeout(0.1)
sock.bind(("0.0.0.0", i))
sock.connect((host, 1)) # Port number should never matter for ICMP
args = [(sockets[i], bytes(2**i)) for i in range(12)]
for arg in args:
print(ping(arg))
arg[0].close()
This actually shows me something more troubling - it seems that the rtt is actually decreasing with increasing packet size! Calling this program (as root, to get socket permissions) outputs:
0
0
(24, 15.784025192260742)
(28, 0.04601478576660156)
(28, 0.025033950805664062)
(28, 0.033855438232421875)
(28, 0.03528594970703125)
(28, 0.04887580871582031)
(28, 0.05316734313964844)
(28, 0.03790855407714844)
(28, 0.0209808349609375)
(28, 0.024080276489257812)
but now notice what happens when I try to send a packet of size 2048 using ping:
user#mycomputer ~/src/connvitals $ time ping -c1 -s2048 $box
PING <hostname redacted> (<IP address redacted>): 2048 data bytes
--- <hostname redacted> ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss
real 0m11.018s
user 0m0.005s
sys 0m0.008s
Not only is the packet dropped, but it takes 11 seconds to do so! So why - if my timeout is set to 100ms - is this packet getting a "successful" response from my python script in only ~0.04ms??
Thank you in advance for any help you can provide.
Update:
I just checked again, and it seems that it's multiple sockets that are the problem, and the threading seems to have nothing to do with it. I get the same issue when I ping with each socket - then immediately close it - sequentially.

All your sockets are identical, and all bound to the same host. There simply isn't any information in the packet for the kernel to know which socket to go to, and raw(7) seems to imply all sockets will receive them.
You're probably getting all the responses in all the threads, meaning you're getting 12 times as many responses per thread as you're expecting.

Scapy TCP Checksum Recalculation Odd behaviour

I'm trying to do a TCP ACK Spoofing. I sniff one ACK packet from a pcap file and send it in a loop incrementing its ACK number as well as another option field.
Sniffing Part: (Prespoofing)
from scapy.all import *
from struct import unpack, pack
pkt = sniff(offline="mptcpdemo.pcap", filter="tcp", count=15)
i=6
while True:
ack_pkt = pkt[i]
if ack_pkt.sprintf('%TCP.flags%') == 'A':
break
i+=1
del ack_pkt.chksum
del ack_pkt[TCP].chksum
print ack_pkt.chksum, ack_pkt[TCP].chksum
hex2pkt = ack_pkt.__class__
Spoofing Part: (Non Optimized)
count=1
while count<5:
ack_pkt[TCP].ack += 1
pkt_hex = str(ack_pkt)
rest = pkt_hex[:-4]
last_4_bit = unpack('!I',pkt_hex[-4:])[0]
new_hex_pkt = rest + pack('>I',(last_4_bit+1))
new_pkt=hex2pkt(new_hex_pkt)
#sendp(new_pkt, verbose=0)
print new_pkt.chksum, new_pkt[TCP].chksum
count+=1
The output comes like this: (Which is changing)
None None
27441 60323
27441 58895
27441 57467
27441 56039
After sending, The average time gap between two packets is around 15 ms. (For 1000 Packets)
When I check it with Wireshark, it shows "checksum is correct" for the 1st packet and "incorrect" for others.
Spoofing Part: (Little bit Optimized)
pkt_hex=str(ack_pkt)
rest1=pkt_hex[:42]
tcp_ack=unpack('!I',pkt_hex[42:46])[0]
rest2=pkt_hex[46:-4]
last_4_bit = unpack('!I',pkt_hex[-4:])[0]
count=1
while count<5:
new_hex_pkt = rest1 + pack('>I',(tcp_ack+1)) + rest2 + pack('>I',(last_4_bit+1))
new_pkt = hex2pkt(new_hex_pkt)
#sendp(new_pkt, verbose=0)
print new_pkt.chksum, new_pkt[TCP].chksum
count+=1
The output comes like this: (Which is not changing)
None None
27441 61751
27441 61751
27441 61751
27441 61751
After sending, The average time gap between two packets is around 10 ms. (For 1000 Packets)
The Checksum is not changing for the 2nd case. The process is quite same. Then what is the problem in the 2nd optimized case? And why the TCP checksum calculated in a loop are wrong for subsequent packets?
Note:
last_4_bit is not the checksum field.
I'm able to see the ack number of the packets being incremented in tcpdump.

After a extended testing, I saw that, del ack_pkt[TCP].checksum deletes the checksum. But while converting to hex string with str(ack_pkt), I guess, it recalculates the checksum. After trying:
ack_pkt = sniff(offline="mptcpdemo.pcap", filter="tcp", count=15)[14]
del ack_pkt[TCP].chksum
print ack_pkt[TCP].chksum
print str(ack_pkt)
It 1st prints the checksum as None. But while printing the hex string, I'm able to see that the checksum field is non zero and contains the actual recalculated checksum.
In the non-optimized code, inside the loop, the packet is converted to hex-string and hence it's re-calculating the checksum each time. But in the optimized version, conversion is outside the loop and hence it carries one value only.

Python: ABP bit doesn't flip despite multiple attempted methods, how to fix?

I have revised my question in response to being put on hold. Hopefully this will better match SO standards.
The purpose of this program is to build and send UDP packets which use Alternating Bit Protocol as a simple resending mechanism. I have confirmed already that the packets can be sent and received correctly. The issue is with the ABP bit and its flipping.
The problem facing me now is that despite trying multiple different methods, I cannot flip the ABP bit used for confirming that a packet or ack received is the correct numbered one. I start out by sending a packet with ABP bit=0, and in response, the receiving process should see this and send back an ack with ABP bit=0. Upon receiving that, the sender program flips its ABP bit to 1 and sends a new packet with this ABP bit value. The receiver will get that, send a matching ack with ABP bit=1, and the sender will receive, flip its bit back to 0, and continue the cycle until the program has finished sending information.
Code below, sorry for length, but it is complete and ready to run. The program takes four command line arguments, here is the command I have been using:
% python ftpc.py 164.107.112.71 4000 8000 manygettysburgs.txt
where ftpc.py is the name of the sender program, 164.107.112.71 is an IP address, 4000 and 8000 are port numbers, and manygettysburgs.txt is a text file I have been sending. It should not make a difference if a different .txt is used, but for full accuracy use a file with a length of between 8000 and 9000 characters.
import sys
import socket
import struct
import os
import select
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
foo = 1 - val
return foo
def buildPacketHeader(IP, Port, Flag, ABP):
#pack IP for transport
#split into four 1-byte values
SplitIP = IP.split('.')
#create a 4-byte struct to pack IP, and pack it in remoteIP
GB = struct.Struct("4B")
remoteIP = GB.pack(int(SplitIP[0]),int(SplitIP[1]),int(SplitIP[2]),int(SplitIP[3]))
#remoteIP is now a 4-byte string of packed IP values
#pack Port for transport
#create a 2-byte struct to pack port, and pack it in remotePort
GBC = struct.Struct("H")
remotePort = GBC.pack(int(Port)) #needs another byte
#remotePort is now a 2-byte string
#add flag
flag = bytearray(1)
flag = str(Flag)
#add ABP
ABP = flipBit(ABP)
abp = str(ABP)
#create header and join the four parts together
Header = ''.join(remoteIP)
Header += remotePort
Header += flag
Header += abp
return Header
#assign arguments to local values
IP = sys.argv[1]
PORT = sys.argv[2]
TPORT = sys.argv[3]
localfile = sys.argv[4]
#declare the socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
#create destination information arrays
remoteIP = bytearray(4)
remotePort = bytearray(2)
#create flag array
flag = bytearray(1)
#create ABP bit
bit = 1
print bit
#send file size packet
#get size of file from os command
filesize = bytearray(4)
#save as 4-byte string
filesize = str(os.stat(localfile).st_size)
#package the filesize string
filesizestr = buildPacketHeader(IP,PORT,1,bit) #build header
print bit
filesizestr += filesize #complete the packet
s.sendto(filesizestr, ('127.0.0.1', int(TPORT))) #send packet
# end of send file name packet
#begin listening for responses
read, write, err = select.select([s], [], [], 1) #timeout set to 1 seconds
if len(read) > 0:
#data received
data = read[0].recv(1200)
if data[7] != bit:
print "failed ack"
#resend packet
else:
print "Timeout."
#resend packet
#send next data packet
#get filename as string (from arg4 diredtly)
filename = bytearray(20)
#save as 20-byte string
filename = localfile
#package the filename string
filenamestr = buildPacketHeader(IP,PORT,2,bit) #build header
print bit
filenamestr += filename #complete the packet
s.sendto(filenamestr, ('127.0.0.1', int(TPORT))) #send packet
#end of send file name packet
#send file content packet
#reading while loop goes here
with open(localfile, 'rb', 0) as f: #open the file
while True:
fstr = f.read(1000)
if not fstr:
print "NOTHING"
break
#put together the main packet base
filecontentstr = buildPacketHeader(IP,PORT,3,bit)
print bit
filecontentbytearray = bytearray(1000) #create ytear array
filecontentbytearray = fstr #assign fstr to byte array
filecontentsend = ''.join(filecontentstr) #copy filecontentstr to new string since we will be using filecontentstr again in the future for other packets
filecontentsend += filecontentbytearray #append read data to be sent
s.sendto(filecontentsend, ('127.0.0.1', int(TPORT))) #send the file content packet
#end of send file content packet
s.close()
In this code, every time that buildPacketHeader is called, it performs flipBit as part of its operations. flipBit is supposed to flip the bit's value for ABP. I have prints set up to print out the new value of bit after all calls to buildPacketHeader, so as to track the value. Whenever I run the program, however, I always see the same value for the ABP bit.
I've tried several methods, including changing to a bool. Here are some changes to flipBit I have tried:
def flipBit(val): #flip ABP bit from 0 to 1 and vice versa
if val == 0:
val = 1
else:
val = 0
return val
and some with bools instead:
def flipBit(val):
val = not val
return val
def flipBit(val):
val = (True, False)[val]
return val
I figure that many of these methods are in fact working options due to past experience. That said, I am completely baffled as to why it is not working as expected in this program. I would assume that it is my inexperience with python that is at fault, for despite having now used it for a decent amount of time, there still are peculiarities which escape me. Any help is greatly appreciated.

I don't understand what your objection is to Python ints, but the ctypes module provides a world of low-level mutable objects; e.g.,
>>> import ctypes
>>> i = ctypes.c_ushort(12) # 2-byte unsigned integer, initial value 12
>>> i
c_ushort(12)
>>> i.value += 0xffff - 12
>>> hex(i.value)
'0xffff'
>>> i.value += 1
>>> i.value # silently overflowed to 0
0

This is my first time ever answering my own SO question, so hopefully this turns out right. If anyone has additions or further answers, then feel free to answer as well, or comment on this one.
I ended up solving the problem by adding another return value to the return statement of buildPacketHeader, so that in addition to returning a string I also return the new value of bit as well. I confirmed that it was working by setting up the following prints inside of buildPacketHeader:
#add ABP
print "before:",ABP #test line for flipBit
ABP = flipBit(ABP)
abp = str(ABP)
print "after:",ABP #test line for flipBit
The output of which is shown here (I ended it early but the proof of functionality is still visible)
% python ftpc.py 164.107.112.70 4000 8000 manygettysburgs.txt
before: 1
after: 0
Timeout, resending packet...
before: 0
after: 1
Timeout, resending packet...
before: 1
after: 0
As can be seen, the before of the second packet is the after of the first packet, and the before of the third packet is the after of the second packet. Through this, you can see that the program is now flipping bits correctly.
The change made to buildPacketHeader is shown below:
return Header
becomes
return Header, ABP
and calls to buildPacketHeader:
filesizestr = buildPacketHeader(IP,PORT,1,bit)
become
filesizestr, bit = buildPacketHeader(IP,PORT,1,bit)
Very simple for such a bother. Make sure you return values if you want to make them change.