We Keep Coding

Is the output of scapy.sprintf a raw string?why the length is wrong?

I want to analyze TCP packets by scapy. And I use pkt.sprintf('%Raw.load%') to extract tcp data. But the output string has something wrong with length. but the '\' is deemed to be a str instead of a Escaped character.so '\x11' is considered as 4 different strings instead of a ASCII character.
Here are my codes:
from scapy.all import *
def findTCPdata(pkt):
raw = pkt.sprintf("%Raw.load%")
print raw
print 'length of TCP data: '+ str(len(raw))
def main():
pkts = rdpcap('XXX.pcap')
for pkt in pkts:
findTCPdata(pkt)
if __name__ == '__main__':
main()
enter image description here
The length of each tcp data should be 17 instead of the value in screen(53,52,46,52).
4 tcp data are:
'U\x11\x04\x92\x02\x03\x1e\x03#\x03\xf8q=e\xcb\x15\r'
'U\x11\x04\x92\x02\x03.\x03#\x03\xf8q=e\xcb\xb8\x05'
'U\x11\x04\x92\x02\x03X\x03#\x03\xf8q=e\xcbiO'
'U\x11\x04\x92\x02\x03n\x03#\x03\xf8q=e\xcb\xdb\xe3'
Please help me solve the problem.Thank you!

i see. i need a function to transform rawstring to string.
so i add codes after line 3(raw = pkt.sprintf("%Raw.load%")) like:
raw = raw.replace('\'','')
string = raw.decode('string_escape')
then the output is right

Python (pySerial): cannot trim/remove CR, LF or unexpected characters from string

I am using an Arduino to output Temp & Hum data from a sensor, this is being read by a PC running Python using pySerial. The data is reading in correctly but I would like to remove the CR/LF and unexpected characters. One idea I found on this site was to use lstrip or lreplace but they do not seem to work correctly. They will remove one instance of the character but even repeating the line or making a small loop has no effect.
This is what the program Prints (Bottom line is string after I've tried to cut out the unnecessary characters):
[b'\n', b'\r\n', b'Read sensor: OK\r\n', b'Hum40.00\r\n', b'TempC18.00\r\n']
[" b'Hum40.00\r\n'", " b'TempC18.00\r\n']"]
I am aiming for it to read:
[Hum40.00, TempC18.00]
I can hopefully fine tune the message later.
This is the code:
import serial as ser
import time
count = 0
msgArray = []
saveMsg = []
ser = ser.Serial('COM16', 9600, timeout=1, parity='N', stopbits=1, bytesize=8, xonxoff=0) # Setting up and opening COM port
ser.close()
ser.open()
def readSerial(): #reads a whole line from COM port
serLine = ser.readline()
return serLine
def sveMsgCut(): #saves the buffer as a message then cuts message
cutMsg = saveMsg
words = cutMsg.split(',')
return words
while True: #main program
dataSerial = readSerial()
if count < 5: #reads COM port 5 times and passes along to buffer msgArray
msgArray.append(dataSerial)
count = count+1
else:
print(msgArray) #~display msgArray
saveMsg = str(msgArray) #convert to string
splitMsg = saveMsg.split(',') #splits string (csv)
phrase = splitMsg[3:5] #cuts out excess either side of Temp & Hum/
phraseString = str(phrase)
phraseNew = phraseString.lstrip("/n") #an attempts ot remove CR
print(phraseNew) #~print adjusted string
saveMsg = msgArray
count = 0 #resets msgArray and counter
msgArray = []
time.sleep(5)
I am fairly new to programming, especially Python so it may be something simple that I've missed but have tried several different ideas and cannot remove the extra characters.

not sure why rstrip/lstrip do not work for you.
this code runs as expected on my machine:
s = '\r\nHum40.00\r\n'
print (s.rstrip().lstrip())
The only difference I see is the "/n" parameter so instead try:
phraseNew = phraseString.lstrip()

Decided to go about this another way. Placed the raw serial data into a list and extracted the temp and humidity readings then joined them together as a string:
else:
print(msgArray) #Raw serial data
msgString = str(msgArray) #convert serial data to str
character = list(msgString) #str to list
# the next two lines extract the temp & hum readings then converts
them to their respective strings
humidity = ''.join(character[46:51])
temperature = ''.join(character[65:70])
print('H:' + (humidity) + ' T:' + (temperature))
The output now looks like this (Raw then processed data):
[b'\n', b'\r\n', b'Read sensor: OK\r\n', b'Hum40.00\r\n', b'TempC21.00\r\n']
H:40.00 T:21.00
Now onto the next stage.

One thought:
So what you have now is:
"b'Hum40.00\r\n'"
Now what I see here is that the ' symbol is now a part of the string. This means that ' is the last symbol instead of \r\n. I have found success with starting on the outside are working inwards. Let's first start by removing the b. To do this try:
x = str(ser.readline())
x = x.lstrip("b")
Now we should see this:
"'Hum40.00\r\n'"
Next remove the '
x = x.strip("'")
Now we see this:
"Hum40.00\r\n"
Now the tricky part here is deleting the \r\n. There is a post here:Can't delete "\r\n" from a string
that explains why:
x = x.rstrip("\r\n")
will not work.
Instead type:
x = x.rstrip("\\r\\n")
Final code:
x = str(ser.readline())
x = x.lstrip("b")
x = x.strip("'")
x = x.rstrip("\\r\\n")
print(x)
Should yeild:
"Hum40.00"

Backporting bytearray to Python 2.5

I'm trying to convert this websocket example for use in Python 2.5, but am running into errors with the use of the bytearray type.
The code stops working for Python 2.5 here (in the send_text method of websocket_server/websocket_server.py):
FIN = 0x80
OPCODE = 0x0f
def send_text(self, message):
header = bytearray();
payload = encode_to_UTF8(message)
payload_length = len(payload)
header.append(FIN | OPCODE_TEXT)
header.append(payload_length)
self.request.send(header + payload)
The message variable stores the string input that is sent to clients.
It attempts to create an array of bytes and send that using the self.request.send method. How would I change this to make it work in Python 2.5 which doesn't have the bytes type or bytearray?

Using struct MIGHT work, I haven't tested this.
What I would do, as a workaround, would be to use struct.pack to pack byte by byte.
mensaje = "saludo"
FIN = 0x80
OPCODE = 0x0f
payload = ''
for c in mensaje:
payload += struct.pack("H", ord(c))
msj = struct.pack("H",FIN | OPCODE )
msj+= struct.pack("H",len(payload))
print msj + payload
I'm using "H" as the 'fmt' parameter in the struct.pack function, but you better check how is your package sent and how many bytes per 'character' (since I'm guessing you're using unicode, I'm using 'H', unsigned short = 2 bytes).
More info: https://docs.python.org/2/library/struct.html, section 7.3.2.1 and 7.3.2.2.
EDIT:
I'll answer here, what do I mean by using 'chr()' instead of 'struct.pack()':
mensaje = "saludo"
FIN = 0x80
OPCODE = 0x0f
payload = mensaje
msj = chr( FIN | OPCODE )
msj+= chr(len(payload))
print msj + payload
if you print the message, then you should see the same output when using struct.pack("B", ord(something)) than when using ord(something), I just used struct.pack() because I thought your message was two bytes per char (as unicode).

Read Wireshark dump file for packet times

My instructions were to read the wireshark.bin data file dumped from the Wireshark program and pick out the packet times. I have no idea how to skip the header and find the first time.
"""
reads the wireshark.bin data file dumped from the wireshark program
"""
from datetime import datetime
import struct
import datetime
#file = r"V:\workspace\Python3_Homework08\src\wireshark.bin"
file = open("wireshark.bin", "rb")
idList = [ ]
with open("wireshark.bin", "rb") as f:
while True:
bytes_read = file.read(struct.calcsize("=l"))
if not bytes_read:
break
else:
if len(bytes_read) > 3:
idList.append(struct.unpack("=l", bytes_read)[0])
o = struct.unpack("=l111", bytes_read)[0]
print( datetime.date.fromtimestamp(o))

Try reading the entire file at once, and then accessing it as a list:
data = open("wireshark.bin", "rb").read() # let Python automatically close file
magic = data[:4] # magic wireshark number (also reveals byte order)
gmt_correction = data[8:12] # GMT offset
data = data[24:] # actual packets
Now you can loop through data in (16?) byte size chunks, looking at the appropriate offset in each chunk for the timestamp.
The magic number is 0xa1b2c3d4, which takes four bytes, or two words. We can determine the order (big-endian or little-endian) by examining those first four bytes by using the struct module:
magic = struct.unpack('>L', data[0:4])[0] # before the data = data[24:] line above
if magic == 0xa1b2c3d4:
order = '>'
elif magic == 0xd4c3b2a1:
order = '<'
else:
raise NotWireSharkFile()
Now that we have the order (and know it's a wireshark file), we can loop through the packets:
field0, field1, field2, field3 = \
struct.unpack('%sllll' % order, data[:16])
payload = data[16 : 16 + field?]
data = data[16 + field?]
I left the names vague, since this is homework, but those field? names represent the information stored in the packet header which includes the timestamp and the length of the following packet data.
This code is incomplete, but hopefully will be enough to get you going.

Decoding tcp packets using python

I am trying to decode data received over a tcp connection. The packets are small, no more than 100 bytes. However when there is a lot of them I receive some of the the packets joined together. Is there a way to prevent this. I am using python
I have tried to separate the packets, my source is below. The packets start with STX byte and end with ETX bytes, the byte following the STX is the packet length, (packet lengths less than 5 are invalid) the checksum is the last bytes before the ETX
def decode(data):
while True:
start = data.find(STX)
if start == -1: #no stx in message
pkt = ''
data = ''
break
#stx found , next byte is the length
pktlen = ord(data[1])
#check message ends in ETX (pktken -1) or checksum invalid
if pktlen < 5 or data[pktlen-1] != ETX or checksum_valid(data[start:pktlen]) == False:
print "Invalid Pkt"
data = data[start+1:]
continue
else:
pkt = data[start:pktlen]
data = data[pktlen:]
break
return data , pkt
I use it like this
#process reports
try:
data = sock.recv(256)
except: continue
else:
while data:
data, pkt = decode(data)
if pkt:
process(pkt)
Also if there are multiple packets in the data stream, is it best to return the packets as a collection of lists or just return the first packet
I am not that familiar with python, only C, is this method OK. Any advice would be most appreciated. Thanks in advance
Thanks

I would create a class that is responsible for decoding the packets from a stream, like this:
class PacketDecoder(object):
STX = ...
ETX = ...
def __init__(self):
self._stream = ''
def feed(self, buffer):
self._stream += buffer
def decode(self):
'''
Yields packets from the current stream.
'''
while len(self._stream) > 2:
end = self._stream.find(self.ETX)
if end == -1:
break
packet_len = ord(self._stream[1])
packet = self._stream[:end]
if packet_len >= 5 and check_sum_valid(packet):
yield packet
self._stream = self._stream[end+1:]
And then use like this:
decoder = PacketDecoder()
while True:
data = sock.recv(256)
if not data:
# handle lost connection...
decoder.feed(data)
for packet in decoder.decode():
process(packet)

TCP provides a data stream, not individual packets, at the interface level. If you want discrete packets, you can use UDP (and handle lost or out of order packets on your own), or put some data separator inline. It sounds like you are doing that already, with STX/ETX as your separators. However, as you note, you get multiple messages in one data chunk from your TCP stack.
Note that unless you are doing some other processing, data in the code you show does not necessarily contain an integral number of messages. That is, it is likely that the last STX will not have a matching ETX. The ETX will be in the next data chunk without an STX.
You should probably read individual messages from the TCP data stream and return them as they occur.

Try scapy, a powerful interactive packet manipulation program.

Where does the data come from ? Instead of trying to decode it by hand, why not use the excellent Impacket package:
http://oss.coresecurity.com/projects/impacket.html

Nice and simple... :)
The trick is in the file object.
f=sock.makefile()
while True:
STX = f.read(1)
pktlen = f.read(1)
wholePacket = STX + pktlen + f.read(ord(pktlen)-2)
doSomethingWithPacket(wholePacket)
And that's it! (There is also no need to check checksums when using TCP.)
And here is a more "robust"(?) version (it uses STX and checksum):
f=sock.makefile()
while True:
while f.read(1)!=STX:
continue
pktlen = f.read(1)
wholePacket = STX + pktlen + f.read(ord(pktlen)-2)
if checksum_valid(wholePacket):
doSomethingWithPacket(wholePacket)