In my project I inherited twisted protocol from twisted.internet.protocol import Protocol, As we know, I should do something packet manipulation in the function dataRecieved, However, the doc said:
data: a string of indeterminate length. Please keep in mind that you will probably need to buffer some data, as partial (or multiple) protocol messages may be received! I recommend that unit tests for protocols call through to this method with differing chunk sizes, down to one byte at a time.
So my data format is that:
packet len | payload
2 bytes | variable bytes
But things become complected if I want to encrypt my data, How should I do then?
Should I encrypt both packet length and payload? then how to judge if the packet is end?
Or should I encrypt only the payload, then alter the packet length ? what if the encrypted payload length is larger than the max value of 2bytes?
Plus: If I use raw socket instead of twisted, can I omit the 2 bytes packet len prefix?
Thanks!
Related
I'm trying to reverse engineer the Quasar RAT protobuf protocol structure.
Quasar is a Remote Administration Tool written in C# which is open source and can be found online here.
https://github.com/quasar/QuasarRAT
I've managed to reverse most of it and I can now connect to the Quasar server client from a python script. How ever one question remains open, it appears that every byte stream that is being sent from the client to the server begins with a 3 byte field which is not registered within the protobuf class within Quasar. This field seems to provide the length of the message not including the prefixed bytes. As can be seen within this block for an example a prefixed byte stream generated for an array of size 0x2d2, these are the prefixed bytes being appended to the message.
0x0A, 0xCF, 0x05
If how ever I decide to change the message fields before serializing the message, this byte stream would change except from the first 0x0A byte. It seems that if I keep appending bytes to the message fields the second byte grows and if I overflow the second byte(make it reach above 0xff) - it would increment the third byte and reset the second byte to 0x80. But the math wont make sense to me at all as this field should return the size of the array but doesn't under any sensible formula that I could compute. I know that protobuf-net can generate PreLengthPrefix bytes to prefix the message with the length of it but this is not the case here.
Any help would be appreciated.
The encoding rules are here: https://developers.google.com/protocol-buffers/docs/encoding
Basically, each field is a encoded as a field-header (aka "tag"), followed by a payload. The field-header is a "varint" (see the encoding guide), the value of which is an integer composed of a field number and the wire-type. The wire-type is the 3 least significant bits, and the field number is the rest (shifted by 3 bits). In the case of 0x0A (binary 1010), the wire type is 2 (binary 010), and the field number is 1.
How you treat the payload depends on the wire type. For wire type 2 (length prefixed), you should expect next:
a varint that is the length of the payload in bytes, then
that many bytes of the actual payload
Unfortunately protobuf is ambiguous without a schema, so knowing that you have length prefixed data doesn't tell you what the data is; a length prefixed payload could be:
a UTF-8 string
a raw BLOB (bytes)
a sub-message
a "packed" array of some primitive type (integers/floating point numbers/etc) - remembering that the length prefix is the number of bytes, not the number of elements; the elements are not even necessarily fixed size (they could themselves be varints
In many ways, the purpose of the wire type isn't to tell you how to interpret the data; it is to tell you how to skip (or just store verbatim) the field if it isn't one you know about. For example, somebody else is using V3 of the API and you have only updated your schema to V2; they send a V3 message to your V2 API; V3 has extra fields you don't care about - the deserializer needs to not break when it hits them, so the wire type tells it how to ignore the field (i.e. what the rules are for finding the next field). Otherwise, we could just use the schema information and not store the wire type in the payload at all (although it is also used as an optimization on repeated primitive data, via "packed" arrays - it is up to the serializer whether it encodes such as length-prefixed vs lots of field header/value pairs).
So I've got a socket like the following:
sock = socket.socket(socket.AF_INET6, socket.SOCK_RAW, socket.getprotobyname("icmp"))
and when i send out a properly constructed ICMP6 ECHO REQUEST per RFC4443 with type 128 and code 0 (also validated checksum) the packet is dropped by the destination's stack because the packet is malformed, obviously, since the 'next header' byte in the IPv6 header is set to 1 per RFC (# for ICMP).
Two things:
1. I know socket.getprotobyname("icmp") makes the socket ICMP compatible with IPv4 (right?)...
2. Wireshark reads the packet as IPv6 but the protocol as ICMP not ICMPv6...
EITHER set the socket to use protocolbyname("icmpv6") (which is invalid, apparently. unless someone knows the proper string... I've tried "icmp6" "icmpv6" but there's probably some tries with an underscore I could make).
OR change the 'next header' byte before I send the packet to 58.
LAST RESORT construct the packet by hand.
Anyone have an idea? I'm not the most experienced in Python, obviously.
This should do it:
socket.getprotobyname('ipv6-icmp')
For anyone that comes across this,
socket.getprotobyname('<proto>')
returns an integer corresponding to a protocol listed in a flat file in /etc/protocols (Unix) and /c/windows/system32/drivers/etc/protocols (Windows).
Check them out! That's the only byte in an IP header that is specific the the data within its payload. The list on a Unix machine contains ALL the protocols, well, at least on a mac, and Windows contains just the most common. You can edit this file too.
In Scapy (or even just Python, for that sake), how do I get the size in bytes of a given packet?
I'm tempted to use the function len but I'm not sure what exactly it returns in the case of packets.
>>> len(IP(dst="www.google.com"))
20
>>> len(IP(dst="www.google.com")/TCP(dport=80))
40
>>> len(IP(dst="www.google.com"))
20
There are 20 bytes in a minimal IP header.
>>> len(IP(dst="www.google.com")/TCP(dport=80))
40
There are another 20 bytes in a minimal TCP header (20+20==40).
So it seems that len is returning the packet length.
What I have been observing is that Len(packet[Layer]) will actually perform the action of the LenField type. It will return the number of bytes in the packet, starting with the specified layer, all the way to the end of the packet. So while this method will work for determining the overall packet size, just beware that it will not work to determine the length of an individual layer.
Here is how I grab the packet size/length when sniffing packets with scapy.
pkt.sprintf("%IP.len%")
Full example:
from scapy.all import *
# callback function - called for every packet
def traffic_monitor_callbak(pkt):
if IP in pkt:
print pkt.sprintf("%IP.len%")
# capture traffic for 10 seconds
sniff(iface="eth1", prn=traffic_monitor_callbak, store=0, timeout=10)
I've only used scapy for sniffing packets, so I'm not sure if the above makes sense when using scapy for other things like creating packets.
I'm currently working on a project where I need to communicate to a microhard cellular modem IPn3G. I have the modem set up to send messages to my computer through TCP and I can pick up the message in a socket.
The message looks like this though:
���������DKReadyCANRogersWirelessInc. Home354626030393530302720391029547
Now, I can recognize a few of these fields like the Status or the Carrierinfo as well as the imei and imsi in the end.
My problem is, how do I parse the funny looking things? I have tried struct, but it didn't seem to help me out very much.
In the documentation of the modem I only found this:
Modem_event message structure:
fixed header (fixed size 20 bytes)
Modem ID (uint64_t (8 bytes))
Message type mask (uint8_t(1 byte))
reserved
packet length (uint16_t(2 bytes))
Note: packet length = length of fixed header + length of message payload.
Carrier info:
Content length 2 BYTES (UINT16_T)
RSSI 1 BYTE (UINT8_T)
RF Band 2 BYTES (UINT16_T)
Service type STRING (1-30 Bytes)
Channel number STRING (1-30 Bytes)
SIM card number STRING (1-30 Bytes)
Phone number STRING (1-30 Bytes)
To me it seems like the message doesn't even line up with what it's supposed to be. I would be very glad if anyone had advice on how to tackle this problem.
Thank you
Python has great struct module, which allows you to pack and unpack binary data.
I see, that you've already tried to use it. I don't have a documentation, for your device (go and check it!), but I can suppose, that strings are null-terminated (they don't have their size given anywhere before).
Get the size of the string part of the message (from size of the other fields and packet length) and read all strings into one Python string using <number>s and split them finding the null characters.
having a bit of an issue here.
I am working on a multiplayer game with flash, and using python for the server side of things. I have the socket connection working... sorta, and quite a bit of the python work done, but I am running into a weird problem.
Let's say, upon logging in, I send some data to the client containing some of their info.
After which, I send some data to assign them to a room.
This data doesn't seem to be read in AS3 as two different things, instead, after readUTFBytes, it is all in the same string.
var str:String = event.currentTarget.readUTFBytes(event.currentTarget.bytesAvailable);
In python, I have defined methods for sending data, which just sends data via transport.write (Twisted), and I am receieving via a progress socket data event in action script. Any idea what could be wrong here? Here's a bit of code...
if ( ! event.currentTarget.bytesAvailable > 0) {
return;
}
var str:String = event.currentTarget.readUTFBytes(event.currentTarget.bytesAvailable);
var Char1:String = str.charAt(0);
var Char2:String = str.charAt(1);
str = str.replace(Char1, "");
str = str.replace(Char2, "");
// Various messages
if (Char1 == "\x03") {
if (Char2 == "\x03") {
trace("Got ping thread");
}
else {
trace("x03 but no secondary prefix handled");
}
return;
}
Quite sloppy I know, but I'm trying to just determine an issue.
All data comes with two prefixes, something like \x02 and \x09 for me to determine what to do, then most data in the string is split on \x01 to get values.
The problem essentially is, where I should get an /x08 /x08 data, I get /x08 /x08 data /x05 /x03 data, when it should be two separate things.
TCP connections are reliable, ordered, stream oriented transports. A stream is a sequence of bytes with no inherent message boundaries. If you want to split up your bytes into separate messages, the bytes themselves must tell you how to do this splitting (or you need some external rule that always applies, like "a message is 5 bytes long").
This applies to all TCP connections, regardless of what language you use them from, or what weird library-specific API gets dropped on top of them (like readUTFBytes).
There are many options for protocols which can help you frame your messages. For example, you could use a length prefix. Then your messages would look like:
\x07 \x08 \x08 h e l l o \x05 \x05 \x03 m a n
\x07 gives the length of the first message, 7 bytes: \x08 \x 08 h e l l o. The next byte after that message, \x05, gives the length of the second message: \x05 \x03 m a n.
You can use multibyte length prefixes if your messages need to be longer, or netstrings which use a decimal representation and a : delimiter to support arbitrary sized prefixes. There are also more sophisticated protocols which offer more features than just separating your bytes into messages. For example, there's AMP which gives you a form of RPC with arguments and responses.