I'm currently working on using scapy for sending data packets, and I've run into a weird issue. When I create a packet as such:
pack = IP(dst="127.0.0.1", id=local_ID)/UDP()/chunk
and then convert that packet to a string (so I can send it via a socket)
sendPack = str(pack)
the result of sendPack is wrong.
For instance, in my test file, I have the numbers 1-8000 ordered as such
1
2
3
...
then, when I print("SEND_PACK: "+sendPack)
it produces the following:
E
2
3
...
Everything else is perfect except for the E
I can't understand where that E is coming from, or what it means.
It's also worth noting that I have verified that pack contains the correct data, and that regardless of what the first line of the test file is, the first line of the output is always an E
Thanks!
To those interested, I fixed the issue by doing the following:
As pointed out above, the E was a result of me printing the packet, not it's contents. In order to access the contents I wanted, I had to do the following:
sendPack = pack[UDP].load #Get the packet's load at the UDP layer
id = pack[IP].ID #Get the ID at the IP layer
The documentation for Scapy is sparse, so I didn't realize that I could access the individual fields of each packet this way.
Here's where I found this fix
First, you are printing a packet, not the content of your UDP datagram. The first two fields of an IPv4 packet are version and IHL which values are respectively 4 (for IP version 4) and 5 (5 words of 32 bits) by default in Scapy. This gives 45 if you put it on one byte in hexadecimal which is letter "E" in ascii. This is why you are always seeing an E as the first byte of your string.
If you want to check what is in your packet, you should use Scapy's show() function: sendPack.show().
Second, you could use Scapy's send(), function to send your packet without bothering about sockets yourself: send(sendPack)
Related
I have a LabVIEW application that is flattening a cluster (array) of Doubles to a string, before transmitting over TCP/IP to my python application. It does this because TCP/IP will only transmit strings.
The problem is that python reads the string as a load of nonsense ASCII characters, and I can't seem to unscramble them back to the original array of doubles.
How do I interpret the string data that LabVIEW sends after flattening a data strings. My only hint of useful information after hours of google was a PyPI entry called pyLFDS, however it has since been taken down.
The LabVIEW flattened data format is described in some detail here. That document doesn't explicitly describe how double-precision floats (DBL type) are represented, but a little more searching found this which clarifies that they are stored in IEEE 754 format.
However it would probably be simpler and more future proof to send your data in a standard text format such as XML or JSON, both of which are supported by built-in functions in LabVIEW and standard library modules in Python.
A further reason not to use LabVIEW flattened data for exchange with other programs, if you have the choice, is that the flattened string doesn't include the type descriptor you need to convert it back into the original data type - you need to know what type the data was in order to decode it.
I wanted to document the problem and solution so others can hopefully avoid the hours I have wasted looking for a solution on google.
When LabVIEW flattens data, in this case a cluster of doubles, it sends them simply as a concatonated string with each double represented by 8 bytes. This is interpreted by python as 8 ASCII characters per double, which appears as nonsense in your console.
To get back to the transmitted doubles, you need to take each 8-byte section in turn and convert the ASCII characters to their ASCII codes, in Python's case using ord().
This will give you an 8 bytes of decimal codes (e.g. 4.8 = [64 19 51 51 51 51 51 51])
It turns out that LabVIEW does most things, including TCP/IP transmissions, Big Endian. Unless you are working Big Endian, you will probably need to change it around. For example the example above will become [51 51 51 51 51 51 19 64]. I put each of my doubles into a list, so was able to use the list(reversed()) functions to change the endienness.
You can then convert this back to a double. Example python code:
import struct
b = bytearray([51,51,51,51,51,51,19,64]) #this is the number 4.8
value = struct.unpack('d', b)
print(value) #4.8
This is probably obvious to more experienced programmers, however it had me flummuxed for days. I apologise for using stackoverflow as the platform to share this by answering my own question, but hopefully this post helps the next person who is struggling.
EDIT: Note if you are using an earlier version than Python 2.7.5 then you might find struct.unpack() will fail. Using the example code above substituting the following code worked for me:
b = bytes(bytearray([51,51,51,51,51,51,19,64]))
This code works for me. UDP server accept flattened dbl array x, return x+1 to port 6503. Modify LabView UDP client to your needs.
import struct
import socket
import numpy as np
def get_ip():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
# doesn't even have to be reachable
s.connect(('10.255.255.255', 1))
IP = s.getsockname()[0]
except:
IP = '127.0.0.1'
finally:
s.close()
return IP
#bind_ip = get_ip()
print("\n\n[*] Current ip is %s" % (get_ip()))
bind_ip = ''
bind_port = 6502
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((bind_ip,bind_port))
print("[*] Ready to receive UDP on %s:%d" % (bind_ip,bind_port))
while True:
data, address = server.recvfrom(1024)
print('[*] Received %s bytes from %s:' % (len(data), address))
arrLen = struct.unpack('>i', data[:4])[0]
print('[*] Received array of %d doubles:' % (arrLen,))
x = []
elt = struct.iter_unpack('>d', data[4:])
while True:
try:
x.append(next(elt)[0])
print(x[-1])
except StopIteration:
break
x = np.array(x)
y = x+1 # np.sin(x)
msg = data[:4]
for item in y:
msg += struct.pack('>d', item)
print(msg)
A = (address[0], 6503)
server.sendto(msg, A)
break
server.close()
print('[*] Server closed')
print('[*] Done')
LabView UDP client:
I understand that this does not solve your problem as you mentioned you didn't have the ability to modify the LabVIEW code. But, I was hoping to add some clarity on common ways string data is transmitted over TCP in LabVIEW.
The Endianness of the data string sent through the Write TCP can be controlled. I recommend using the Flatten To String Function as it gives you the ability to select which byte order you want to use when you flatten your data; big-endian (default if unwired), native (use the byte-order of the host machine), or little-endian.
Another common technique I've seen is using the Type Cast Function. Doing this will convert the numeric to a big-endian string. This of course can be confusing when you read it on the other end of the network as most everything else is little-endian, so you'll need to do some byte-swapping.
In general, if you're not sure what the code looks like, assume that it will be big-endian if its coming from LabVIEW code.
The answer from nekomatic is good one. Using a standard text format when available is always a good option.
So I've got a socket like the following:
sock = socket.socket(socket.AF_INET6, socket.SOCK_RAW, socket.getprotobyname("icmp"))
and when i send out a properly constructed ICMP6 ECHO REQUEST per RFC4443 with type 128 and code 0 (also validated checksum) the packet is dropped by the destination's stack because the packet is malformed, obviously, since the 'next header' byte in the IPv6 header is set to 1 per RFC (# for ICMP).
Two things:
1. I know socket.getprotobyname("icmp") makes the socket ICMP compatible with IPv4 (right?)...
2. Wireshark reads the packet as IPv6 but the protocol as ICMP not ICMPv6...
EITHER set the socket to use protocolbyname("icmpv6") (which is invalid, apparently. unless someone knows the proper string... I've tried "icmp6" "icmpv6" but there's probably some tries with an underscore I could make).
OR change the 'next header' byte before I send the packet to 58.
LAST RESORT construct the packet by hand.
Anyone have an idea? I'm not the most experienced in Python, obviously.
This should do it:
socket.getprotobyname('ipv6-icmp')
For anyone that comes across this,
socket.getprotobyname('<proto>')
returns an integer corresponding to a protocol listed in a flat file in /etc/protocols (Unix) and /c/windows/system32/drivers/etc/protocols (Windows).
Check them out! That's the only byte in an IP header that is specific the the data within its payload. The list on a Unix machine contains ALL the protocols, well, at least on a mac, and Windows contains just the most common. You can edit this file too.
I've been stuck on this one for a while now. I need to send a serial command to a device in python. Here's an example of the format:
ser = serial.Serial('/dev/ttyO1',baudrate=115200)
ser.write('\xAA\x04\x05\x00\x00\x00')
I'm able to communicate just fine if I communicate using the format .write(~hex bytes~)
However, when I need to receive larger chunks of data, the device's communication protocol splits things into packets and I have to confirm receiving each packet. To avoid getting carpal tunnel syndrome while typing out individual ser.write() commands for each packet I want to write a loop that does the packet counting/confirmation for me. here's what I have:
for n in range(0,num_packets,1):
x = '\\x'"%0.2x"%(n)
print('Picture Ready Host Ack')
ser.write('\xAA\x0E\x00\x00' + x + '\x00')
time.sleep(.1)
response = ser.read(ser.inWaiting())
image_write(response,base_path, filename)
However, the double slashes ('\x'..) gives me two slashes:
'\xaa\x0e\x00\x00\\x05\x00'
While a single slash ('\x'...) returns a problem when I define the variable x:
ValueError: invalid \x escape
...Help?
If instead of defining x as '\\x'"%0.2x"%(n), you do x = chr(n), everything should work.
\xij just expands to the character chr(int(ij, 16))
Since you are anyway sending bytes, I would say in a loop, send your first chunk. Do not add anything to it, then send your delimiter by chr(int(n),16) and subsequently the next chunks. This should solve your problem.
In Scapy (or even just Python, for that sake), how do I get the size in bytes of a given packet?
I'm tempted to use the function len but I'm not sure what exactly it returns in the case of packets.
>>> len(IP(dst="www.google.com"))
20
>>> len(IP(dst="www.google.com")/TCP(dport=80))
40
>>> len(IP(dst="www.google.com"))
20
There are 20 bytes in a minimal IP header.
>>> len(IP(dst="www.google.com")/TCP(dport=80))
40
There are another 20 bytes in a minimal TCP header (20+20==40).
So it seems that len is returning the packet length.
What I have been observing is that Len(packet[Layer]) will actually perform the action of the LenField type. It will return the number of bytes in the packet, starting with the specified layer, all the way to the end of the packet. So while this method will work for determining the overall packet size, just beware that it will not work to determine the length of an individual layer.
Here is how I grab the packet size/length when sniffing packets with scapy.
pkt.sprintf("%IP.len%")
Full example:
from scapy.all import *
# callback function - called for every packet
def traffic_monitor_callbak(pkt):
if IP in pkt:
print pkt.sprintf("%IP.len%")
# capture traffic for 10 seconds
sniff(iface="eth1", prn=traffic_monitor_callbak, store=0, timeout=10)
I've only used scapy for sniffing packets, so I'm not sure if the above makes sense when using scapy for other things like creating packets.
having a bit of an issue here.
I am working on a multiplayer game with flash, and using python for the server side of things. I have the socket connection working... sorta, and quite a bit of the python work done, but I am running into a weird problem.
Let's say, upon logging in, I send some data to the client containing some of their info.
After which, I send some data to assign them to a room.
This data doesn't seem to be read in AS3 as two different things, instead, after readUTFBytes, it is all in the same string.
var str:String = event.currentTarget.readUTFBytes(event.currentTarget.bytesAvailable);
In python, I have defined methods for sending data, which just sends data via transport.write (Twisted), and I am receieving via a progress socket data event in action script. Any idea what could be wrong here? Here's a bit of code...
if ( ! event.currentTarget.bytesAvailable > 0) {
return;
}
var str:String = event.currentTarget.readUTFBytes(event.currentTarget.bytesAvailable);
var Char1:String = str.charAt(0);
var Char2:String = str.charAt(1);
str = str.replace(Char1, "");
str = str.replace(Char2, "");
// Various messages
if (Char1 == "\x03") {
if (Char2 == "\x03") {
trace("Got ping thread");
}
else {
trace("x03 but no secondary prefix handled");
}
return;
}
Quite sloppy I know, but I'm trying to just determine an issue.
All data comes with two prefixes, something like \x02 and \x09 for me to determine what to do, then most data in the string is split on \x01 to get values.
The problem essentially is, where I should get an /x08 /x08 data, I get /x08 /x08 data /x05 /x03 data, when it should be two separate things.
TCP connections are reliable, ordered, stream oriented transports. A stream is a sequence of bytes with no inherent message boundaries. If you want to split up your bytes into separate messages, the bytes themselves must tell you how to do this splitting (or you need some external rule that always applies, like "a message is 5 bytes long").
This applies to all TCP connections, regardless of what language you use them from, or what weird library-specific API gets dropped on top of them (like readUTFBytes).
There are many options for protocols which can help you frame your messages. For example, you could use a length prefix. Then your messages would look like:
\x07 \x08 \x08 h e l l o \x05 \x05 \x03 m a n
\x07 gives the length of the first message, 7 bytes: \x08 \x 08 h e l l o. The next byte after that message, \x05, gives the length of the second message: \x05 \x03 m a n.
You can use multibyte length prefixes if your messages need to be longer, or netstrings which use a decimal representation and a : delimiter to support arbitrary sized prefixes. There are also more sophisticated protocols which offer more features than just separating your bytes into messages. For example, there's AMP which gives you a form of RPC with arguments and responses.