im currently working on a new protocol layer for scapy. unfortunately the tcp_reassembly gives me a headache.
the protocol i dissect uses a bundle_len field that specifies how long a bundle is.
if the packet is exactly as long as the bundle len everything works.
if it is to large i split off one packet and return the rest to the queue (i think)
if the packet is less then the bundle_len im sure i would have to wait for additional packets or prepend the rest of the previous (to long) packet.
i know what the problem is but can't figure out how to use the tcp_reassembly function correctly.
here some example code:
def tcp_reassemble(cls, data, metadata):
"""[called by sniff(session=TCPSession),
reassembles the tcp stream if packet spans over multiple TCP packets]
Args:
data ([Packet]): [a raw packed strippt by the TCP Layer]
metadata ([dict]): [stores partial streams]
Returns:
[Packet]: [reassembled Packet]
"""
#pylint: disable=unused-argument
length = struct.unpack("<I", data[24:28])[0] # get bundle_len, not readable if offset wrong or not a bundle
if length > 10000: # desaster containment, not good.
length = 64
if len(data) > length: # got to much
# return OWN_PROTOCOL bundle up to bundle_len
pkt = OWN_PROTOCOL(data[:length])
if hasattr(pkt.payload, "tcp_reassemble"): # not sure if needed or what it does
if pkt.payload.tcp_reassemble(data[length:], metadata):
return pkt
else:
return pkt
elif len(data) < length: # got less, not working
print(
f"### Got LESS actual len: {len(data)} proposed bundle_len: {length} ###")
return None # push rest back to queue
else:
return OWN_PROTOCOL(data) # got exactly one bundle in one packet
Related
I am trying to send large byte arrays of a Protobuf class from a Java client to the Python server. However, they have a variable length, because sometimes I send the bytes of an object from ClassA and sometimes from ClassB.
I have a Python socket server with the following code inside the function that listens to the socket:
byte_array = bytearray()
# receive the data in small chunks and print it
while True:
data = connection.recv(64)
if data:
# output received data
logger.debug("Data: %s" % data)
byte_array.extend(data)
else:
# no more data -- quit the loop
logger.debug("no more data.")
break
logger.info("Generating response...")
connection.send(generate_response(byte_array))
logger.info("Sent response.")
I am assembling the large byte array that I receive by putting together the 64 bytes as I get them.
However, when the byte array is fully transmitted and there is nothing left to send, the server hangs on the connection.recv line.
I read that this is because recv blocks until either it receives something or the connection is closed. However, I do not want to close the connection because I want to send my response back to the client after processing the whole byte array.
I want to know when the byte array I am receiving has been fully transmitted, so that I can avoid this blocking.
I can think of three options:
Set a predefined "end" byte, delimiting the end of the byte array.
Send the size of the byte array beforehand and then instead of while True I have a while bytes_read < expected_bytes cycle.
Set a timeout on the connection and I assume that when a timeout occurs it means everything has already been sent.
I am inclined for the first option, however I do not know what character I should use to end the byte array nor how to read it in my Python code.
Any suggestions?
Thanks.
I would personally go for the second option (combined with a reasonable timeout to cater for evil clients that send only half of the file and hang there forever). Delimiting character is good if you can absolutely guarantee it is unique in your stream (but you still need the timeout).
If you cannot guarantee your delimiter to be unique, sending the size the client needs to expect solves the problem. If your metadata is padded to a fixed length, you do not need to worry about delimiters and detecting them.
Option 1 :
So for the first option you could set end byte which won't occur anywhere in your actual message.
You can create a string for eg."END" and convert it into byte array and send it through your java program. After recieving you could use decode() to convert it to string and compare it. :
Note : The end byte which you will send must be less than or equal to the size of chunk to decode and get the exact end byte.
byte_array = bytearray()
# receive the data in small chunks and print it
while True:
data = connection.recv(64)
command = data.decode()
if command != "END":
# output received data
logger.debug("Data: %s" % data)
byte_array.extend(data)
else:
# no more data -- quit the loop
logger.debug("no more data.")
break
logger.info("Generating response...")
connection.send(generate_response(byte_array))
logger.info("Sent response.")
Option 2 :
For the second option you will need to modify the while loop to run according to metadata. I have considered the metadata will consist of first chunk which will be the number of chunks that will be sent.It could go something like :
byte_array = bytearray()
# receive the data in small chunks and print it
loop_count = 0
count = 1
meta = 1
while loop_count >= count:
data = connection.recv(64)
if(meta):
count = int(data.decode()) # first chunk is the number of chunks that will be sent
meta = 0
logger.debug("Data: %s" % data)
byte_array.extend(data)
loop_count = loop_count + 1
else:
# no more data
logger.debug("no more data.")
logger.info("Generating response...")
connection.send(generate_response(byte_array))
logger.info("Sent response.")
Option 3 :
It will also work fine provided you are sure there will be no network delay and only issue will be your java program will have to wait for the response from the python server untill the timeout takes place
Option 4 :
You could use a non blocking socket which will run untill it dosen't recieve for a pre determined period of time. Although i don't recommend it for your situation you can read about it and see if it suits your needs.
I'm trying to read a TLS message. Specifically, the one with the certificate details (handshake_type = 11). What I'm doing is first checking that the message contains Raw. If so, I'm extracting the payload like so: b = bytes(pkt[Raw].load). Next, I'm checking that the first byte is 0x16 and the following two bytes need to be a proper TLS version.
The problem is that this message doesn't pass these conditions. WireShark is showing me that \x16\x03\x03 are the bytes at position 0000 (picture is attached), but I guess it is done for convenience.
So is my assumption that the payload MUST start with 0x16 wrong?
P.S
I don't want to use scapy-ssl_tls library.
EDIT
This is the code:
def handle_tls_packet(pkt):
if pkt.haslayer(Raw):
b = bytes(pkt[Raw].load)
if b[0] == 0x16:
version = int.from_bytes(b[1:3], 'big')
message_len = int.from_bytes(b[3:5], 'big')
handshake_type = b[5]
handshake_length = int.from_bytes(b[6:9], 'big')
print("v = ", version, " len = ", message_len, " htype =", handshake_type
, "hlen =", handshake_length)
if handshake_type == 11:
# never happens - Why?
certs_len = int.from_bytes(b[7:11], 'big')
EDIT2:
As suggested by Cukic0d, I used load_layer("ssl").
Apparently, pkt[TLS].msg is a list (to hold multiple TLS messages in one packet?). Anyways, I printed the type of every such message like so:
def handle_tls_packet(pkt):
for tls_msg in pkt[TLS].msg:
print(type(tls_msg))
I expected to see a TLSCertificate object, yet such object is never seen.
Why?
EDIT3:
I'm willing to use scapy-ssl_tls if that would make life easier.
If you want to play with TLS handshake, enable TLS on scapy using load_layer("tls").
That enables the TLS module, which supports handshake (requires scapy >= 2.4.0). Scapy will then correctly dissect TLS handshake/key... packets
You should first try
load_layer("tls")
packets = sniff(prn=lambda x:x.summary(), lfilter=lambda x: TLS in x)
And if you're using Scapy 2.4.4+, for better consistency you can even use
sniff([...], session=TLSSession)
Have a look on how the packets are built:
Example:
There is also a quite fancy guide here: https://github.com/secdev/scapy/blob/master/doc/notebooks/tls/notebook2_tls_protected.ipynb
So summarize:
You will find each packet when using load_layer("tls").
Note that there are a lot of packets and that TLSCertificate will only appear once. msg is a list because many informations can be contained in a single TLS packet
I'm trying to send a specific packet size (100 bytes) with scapy but cant seem to get it.
I'm using this to start.
sr(IP(dst="192.168.1.1")/TCP(dport=443))
Looking at the docs / help I cant tell if I can use PacketLenField to specify the length of the packet. I can do it with NMAP & NSE but would like to do it outside of NMAP.
Any ideas on this one?
Thanks!
You can just add on the required number of bytes as a String when crafting the packet e.g.:
payload = 'ZZZZZZZZZZZZZZZZZZZZZ'
pkt = Ether() / IP() / TCP() / payload
will work. You just need to adjust the length of the payload as you require.
Scapy's Raw() function populates the payload of the packet. If you know your header size, you only need to fill in the remaining bytes with random data.
You can use RandString() to generate random padding. The following command sends a packet of length 100 (and listens for a response):
sr(IP(dst="192.168.1.1")/TCP(dport=443)/Raw(RandString(size=72))
You may use inet.Padding() from scapy library:
packet = IP(dst="192.168.1.1")/TCP(dport=443)
if len(packet)<100:
#"\x00" is a single zero byte
myString = "\x00"*(100 - len(packet))
packet = packet/inet.Padding(myString)
Hello kind folks of StackOverflow.
I am trying to make a sort of 'bot' which can connect to a Minecraft Classic server, post messages and possibly build.
Anyway, I'm having some trouble understanding how to send packets in python, and how to correctly encode them.
Here are the packets I need to send, I want to send the 'Player Identification' one: http://www.minecraftwiki.net/wiki/Classic_server_protocol#Client_.E2.86.92_Server_packets
I know I need to be using sockets, and I need to be using struct.pack, but how exactly can I send it?
An example piece code that sends a login packet would be marvellous.
Thanks.
I'll get the ball rolling:
import socket
import struct
username = "username_value"
verification_key = "verification_key"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # boilerplate
s.connect(("example.com", 1234)) # adjust accordingly
# now for the packet
# note that the String type is specified as having a length of 64, we'll pad that
packet = ""
packet += struct.pack("B", 1) # packet type
packet += struct.pack("B", 7) # protocol version
packet += "%-64s" % username # magic!
packet += "%-64s" % verification_key
packet += struct.pack("B", 0) # that unused byte, assuming a NULL byte here
# send what we've crafted
s.send(packet)
The "%-20s" may be weird for you if you've never used format strings. Essentially..
print "%s" % 5
.. would print out 5 ..
print "%10s" % 5
.. would pad the output to be exactly 10 characters in width. But it pads them on the right side, we want that padding on the left -- hence the - ..
print "%-10s" % s, "<this will be 9 spaces away from the '5'>"
.. play around with it.
If anything is unclear, let me know. I like what you're doing, reminds me of an old project of mine. Except that I didn't have a neat protocol specification like you did, lucky bastard. ;)
I've got a simple TCP server and client. The client receives data:
received = sock.recv(1024)
It seems trivial, but I can't figure out how to recieve data larger than the buffer. I tried chunking my data and sending it multiple times from the server (worked for UDP), but it just told me that my pipe was broken.
Suggestions?
If you have no idea how much data is going to pour over the socket, and you simply want to read everything until the socket closes, then you need to put socket.recv() in a loop:
# Assumes a blocking socket.
while True:
data = sock.recv(4096)
if not data:
break
# Do something with `data` here.
Mike's answer is the one you're looking for, but that's not a situation you want to find yourself in. You should develop an over-the-wire protocol that uses a fixed-length field that describes how much data is going to be sent. It's a Type-Length-Value protocol, which you'll find again and again and again in network protocols. It future-proofs your protocol against unforeseen requirements and helps isolate network transmission problems from programmatic ones.
The sending side becomes something like:
socket.write(struct.pack("B", type) #send a one-byte msg type
socket.write(struct.pack("H", len(data)) #send a two-byte size field
socket.write(data)
And the receiving side something like:
type = socket.read(1) # get the type of msg
dataToRead = struct.unpack("H", socket.read(2))[0] # get the len of the msg
data = socket.read(dataToRead) # read the msg
if TYPE_FOO == type:
handleFoo(data)
elif TYPE_BAR == type:
handleBar(data)
else:
raise UnknownTypeException(type)
You end up with an over-the-wire message format that looks like:
struct {
unsigned char type;
unsigned short length;
void *data;
}
Keep in mind that:
Your operating system has it's own idea of what it's TCP/IP socket buffer size is.
TCP/IP packet maximum size (generally is 1500 bytes)
pydoc for socket suggests that 4096 is a good buffer size
With that said, it'd really be helpful to see the code around that one line. There are a few things that could play into this, if you're using select or just polling, is the socket non-blocking, etc.
It also matters how you're sending the data, if your remote end disconnects. More details.