We Keep Coding

How to get raw hex values from pcap file?

I've been playing around with scapy and want to read through and analyse every hex byte. So far I've been using scapy simply because I don't know another way currently. Before just writing tools myself to go through the pcap files I was wondering if there was an easy way to do it. Here's what I've done so far.
packets = rdpcap('file.pcap')
tcpPackets = []
for packet in packets:
if packet.haslayer(TCP):
tcpPackets.append(packet)
When I run type(tcpPackets[0]) the type I get is:
<class 'scapy.layers.l2.Ether'>
Then when I try to covert the Ether object into a string it gives me a mix of hex and ascii (as noted by the random parenthesis and brackets).
str(tcpPackets[0])
"b'$\\xa2\\xe1\\xe6\\xee\\x9b(\\xcf\\xe9!\\x14\\x8f\\x08\\x00E\\x00\\x00[:\\xc6#\\x00#\\x06\\x0f\\xb9\\n\\x00\\x01\\x04\\xc6)\\x1e\\xf1\\xc0\\xaf\\x07[\\xc1\\xe1\\xff0y<\\x11\\xe3\\x80\\x18 1(\\xb8\\x00\\x00\\x01\\x01\\x08\\n8!\\xd1\\x888\\xac\\xc2\\x9c\\x10%\\x00\\x06MQIsdp\\x03\\x02\\x00\\x05\\x00\\x17paho/34AAE54A75D839566E'"
I have also tried using hexdump but I can't find a way to parse through it.

I can't find the proper dupe now, but this is just a miss-use/miss-understanding of str(). The original data is in a bytes format, for instance x = b'moo'.
When str() retrieves your bytes string, it will do so by calling the __str__ function of the bytes class/object. That will return a representation of itself. The representation will keep b at the beginning because it's believed to distinguish and make it easier for humans to understand that it's a bytes object, as well as avoid encoding issues I guess (alltho that's speculations).
Same as if you tried accessing tcpPackets[0] from a terminal, it would call __repr__ and show you something like <class 'scapy.layers.l2.Ether'> most likely.
As an example code you can experiment with, try this out:
class YourEther(bytes):
def __str__(self):
return '<Made Up Representation>'
print(YourEther())
Obviously scapy's returns another representation, not just a static string that says "made up representation". But you probably get the idea.
So in the case of <class 'scapy.layers.l2.Ether'> it's __repr__ or __str__ function probably returns b'$\\xa2\\....... instead of just it's default class representation (some correction here might be in place tho as I don't remember/know all the technical namification of the behaviors).
As a workaround, this might fix your issue:
hexlify(str(tcpPackets[0]))
All tho you probably have to account for the prepended b' as well as trailing ' and remove those accordingly. (Note: " won't be added in the beginning or end, those are just a second representation in your console when printing. They're not actually there in terms of data)
Scapy is probably more intended to use tcpPackets[0].dst rather than grabing the raw data. But I've got very little experience with Scapy, but it's an abstraction layer for a reason and it's probably hiding the raw data or it's in the core docs some where which I can't find right now.
More info on the __str__ description: Does python `str()` function call `__str__()` function of a class?
Last note, and that is if you actually want to access the raw data, it seams like you can access it with the Raw class: Raw load found, how to access?

You can put all the bytes of a packet into a numpy array as follows:
for p in tcpPackets:
raw_pack_data = np.frombuffer(p.load, dtype = np.uint8)
# Manipulate the bytes stored in raw_pack_data as you like.
This is fast. In my case, rdpcap takes ~20 times longer than putting all the packets into a big array in a similar for loop for a 1.5GB file.

Scapy documentation uses confusing code for a function

I've been having fun with Scapy. While reading the documentation, it explains how to show the important information of a port-scan reply with "a simple loop", which goes like this:
ans.summary( lambda(s,r): r.sprintf("%TCP.sport% \t %TCP.flags%") )
I can sort of understand what lambda does, however what trips me up is everything after lambda. I assume (s,r) is mapping s and r to the "send" and "received" packets of the tuple returned by ans.summary. I've been looking around for sprintf, and can't find anything pertaining to python about it. I also assume that %TCP.sport% is a scapy specific format string.
What does it all mean and how does it work?

ans.summary() accepts a function as a parameter that will be used to represent each (sent packed, received response) tuple.
If no function is specified, something equivalent to lambda s, r: "%s ==> %s" % (s.summary(), r.summary()) is used.
The function must accept to arguments (the sent packet and the corresponding received answer), and return a string.
In your case, the function is lambda(s, r): r.sprintf("%TCP.sport% \t %TCP.flags%"). It only uses the second parameter (the answer), and calls .sprintf(), which is a method defined of Scapy Packet objects (that's not a Python standard, but something specific to these objects). The formatting elements in .sprintf() are related to layers and field values. Here, the call will return a string containing the TCP source port (from the answer packet) followed by "space tab space" and the TCP flags.
If you apply this to sr() result for a TCP scan, this will hence show each scanned port with "SA" if the port is open and "RA" if it is closed.

snmpset in python

I am using the python to send an snmpset message to an snmp agent. I have the correct OID as I can use it to get data with snmpget. However, it will not set the data. I am sending octets in hex format (two hex values) separated by a colon. I might have to put apostrophes around it, right? Here is the example of the line I am sending
foo = os.popen("snmpset -v 2c -c private 192.1.55.222
.1.2.6.5.4.1.24022.4.27.1.6.4.4 x 00:00:04:cf:00:00:00:00:00:00")
as you can see, I am sending an Octet string with type x.
Can anyone hazard a guess as to how I should pass in the set value of "00:00:04:cf:00:00:00:00:00:00". I know the setvalue is of the right length and type because I have used it in a MIB browser.

A couple of things:
1) there is a native python interface you could use instead of calling a system command. There are in fact multiple choices, including pysnmp (done in python) and Net-SNMP's python module (done in C).
2) The Net-SNMP snmpset command expects straight hex code without any :s in it. So if you remove the :s you may find it'll work.

Uncompress Zlib string in using ByteArrays

I have a web application developed in Adobe Flex 3 and Python 2.5 (deployed on Google App Engine). A RESTful web service has been created in Python and its results are currently in an XML format which is being read by Flex using the HttpService object.
Now the main objective is to compress the XML so that there is as less a time between the HttpService send() method and result events. I looked up Python docs and managed to use zlib.compress() to compress the XML result.
Then I set the HttpService result type from "xml" to "text" and tried using ByteArrays to uncompress the string back to XML. Here's where I failed. I am doing something like this:
var byteArray:ByteArray = new ByteArray();
byteArray.writeUTF( event.result.toString() );
byteArray.uncompress();
var xmlResult:XML = byteArray.readUTF();
Its throwing an exception at byteArray.uncompress() and says unable to uncompress the byteArray. Also when I trace the length of the byteArray it gets 0.
Unable to figure out what I'm doing wrong. All help is appreciated.
-- Edit --
The code:
# compressing the xml result in Python
print zlib.compress(xmlResult)
# decompresisng it in AS3
var byteArray:ByteArray = new ByteArray();
byteArray.writeUTF( event.result.toString() );
byteArray.uncompress()
Event is of type ResultEvent.
The error:
Error: Error #2058: There was an error decompressing the data.
The error could be because the value of byteArray.bytesAvailable = 0 which means the raw bytes python generated hasn't been written into byteArray properly..
-- Sri

What is byteArray.writeUTF( event.result.toString() ); supposed to do? The result of zlib.compress() is neither unicode nor "UTF" (meaningless without a number after it!?); it is binary aka raw bytes; you should neither decode it nor encode it nor apply any other transformation to it. The receiver should decompress immediately the raw bytes that it receives, in order to recover the data that was passed to zlib.compress().
Update What documentation do you have to support the notion that byteArray.uncompress() is expecting a true zlib stream and not a deflate stream (i.e. a zlib stream after you've snipped the first 2 bytes and the last 4)?
The Flex 3 documentation of ByteArray gives this example:
bytes.uncompress(CompressionAlgorithm.DEFLATE);
but unhelpfully doesn't say what the default (if any) is. If there is a default, it's not documented anywhere obvious, so it would be a very good idea for you to use
bytes.uncompress(CompressionAlgorithm.ZLIB);
to make it obvious what you intend.
AND the docs talk about a writeUTFBytes method, not a writeUTF method. Are you sure that you copy/pasted the exact receiver code in your question?
Update 2
Thanks for the URL. Looks like I got hold of the "help", not the real docs :=(. A couple of points:
(1) Yes, there is an explicit inflate() method. However uncompress DOES have an algorithm arg; it can be either CompressionAlgorithm.ZLIB (the default) or CompressionAlgorithm.DEFLATE ... interestingly the latter is however only available in Adobe Air, not in Flash Player. At least we know the uncompress() call appears OK, and we can get back to the problem of getting the raw bytes onto the wire and off again into a ByteArray instance.
(2) More importantly, there are both writeUTF (Writes a UTF-8 string to the byte stream. The length of the UTF-8 string in bytes is written first, as a 16-bit integer, followed by the bytes representing the characters of the string) and writeUTFBytes (Writes a UTF-8 string to the byte stream. Similar to the writeUTF() method, but writeUTFBytes() does not prefix the string with a 16-bit length word).
Whatever the merits of supplying UTF8-encoded bytes (nil, IMHO), you don't want a 2-byte length prefix there; using writeUTF() is guaranteed to cause uncompress() to bork.
Getting it on to the wire: using Python print on binary data doesn't seem like a good idea (unless sys.stdout has been nobbled to run in raw mode, which you didn't show in your code).
Likewise doing event.result.toString() getting a string (similar to a Python unicode object, yes/no?) -- with what and then encoding it in UTF-8 seem rather unlikely to work.
Given I didn't know that flex existed until today, I really can't help you effectively. Here are some further suggestions towards self-sufficiency in case nobody who knows more flex comes along soon:
(1) Do some debugging. Start off with a minimal XML document. Show repr(xml_doc). Show repr(zlib_compress_output). In (a cut-down version of) your flex script, use the closest function/method to repr() that you can find to show: event.result, event.result.toString() and the result of writeUTF*(). Make sure you understand the effects of everything that can happen after zlib.compress(). Reading the docs carefully may help.
(2) Look at how you can get raw bytes out of event.result.
HTH,
John

Type of object from udp buffer in python using metaclasses/reflection

Is it possible to extract type of object or class name from message received on a udp socket in python using metaclasses/reflection ?
The scenario is like this:
Receive udp buffer on a socket.
The UDP buffer is a serialized binary string(a message). But the type of message is not known at this time. So can't de-serialize into appropriate message.
Now, my ques is Can I know the classname of the seraialized binary string(recvd as UDP buffer) so that I can de-serialize into appropriate message and process further.
Thanks in Advance.

What you receive from the udp socket is a byte string -- that's all the "type of object or class name" that's actually there. If the byte string was built as a serialized object (e.g. via pickle, or maybe marshal etc) then you can deserialize it back to an object (using e.g. pickle.loads) and then introspect to your heart's content. But most byte strings were built otherwise and will raise exceptions when you try to loads from them;-).
Edit: the OP's edit mentions the string is "a serialized object" but still doesn't say what serialization approach produced it, and that makes all the difference. pickle (and for a much narrower range of type marshal) place enough information on the strings they produce (via the .dumps functions of the modules) that their respective loads functions can deserialize back to the appropriate type; but other approaches (e.g., struct.pack) do not place such metadata in the strings they produce, so it's not feasible to deserialize without other, "out of bands" so to speak, indications about the format in use. So, o O.P., how was that serialized string of bytes produced in the first place...?

You need to use a serialization module. pickle and marshal are both options. They provide functions to turn objects into bytestreams, and back again.

Updated answer after updated question:
"But the type of message is not known at this time. So can't de-serialize into appropriate message."
What you get is a sequence of bytes. How that sequence of types should be interpreted is a question of how the protocol looks. Only you know what protocol you use. So if you don't know the type of message, then there is nothing you can do about it. If you are to receive a stream of data an interpret it, you must know what that data means, otherwise you can't interpret it.
It's as simple as that.
"Now, my ques is Can I know the classname of the seraialized binary string"
Yes. The classname is "str", as all strings. (Unless you use Python 3, in which case you would not get a str but a binary). The data inside that str has no classname. It's just binary data. It means whatever the sender wants it to mean.
Again, I need to stress that you should not try to make this into a generic question. Explain exactly what you are trying to do, not generically, but specifically.