I have a LabVIEW application that is flattening a cluster (array) of Doubles to a string, before transmitting over TCP/IP to my python application. It does this because TCP/IP will only transmit strings.
The problem is that python reads the string as a load of nonsense ASCII characters, and I can't seem to unscramble them back to the original array of doubles.
How do I interpret the string data that LabVIEW sends after flattening a data strings. My only hint of useful information after hours of google was a PyPI entry called pyLFDS, however it has since been taken down.
The LabVIEW flattened data format is described in some detail here. That document doesn't explicitly describe how double-precision floats (DBL type) are represented, but a little more searching found this which clarifies that they are stored in IEEE 754 format.
However it would probably be simpler and more future proof to send your data in a standard text format such as XML or JSON, both of which are supported by built-in functions in LabVIEW and standard library modules in Python.
A further reason not to use LabVIEW flattened data for exchange with other programs, if you have the choice, is that the flattened string doesn't include the type descriptor you need to convert it back into the original data type - you need to know what type the data was in order to decode it.
I wanted to document the problem and solution so others can hopefully avoid the hours I have wasted looking for a solution on google.
When LabVIEW flattens data, in this case a cluster of doubles, it sends them simply as a concatonated string with each double represented by 8 bytes. This is interpreted by python as 8 ASCII characters per double, which appears as nonsense in your console.
To get back to the transmitted doubles, you need to take each 8-byte section in turn and convert the ASCII characters to their ASCII codes, in Python's case using ord().
This will give you an 8 bytes of decimal codes (e.g. 4.8 = [64 19 51 51 51 51 51 51])
It turns out that LabVIEW does most things, including TCP/IP transmissions, Big Endian. Unless you are working Big Endian, you will probably need to change it around. For example the example above will become [51 51 51 51 51 51 19 64]. I put each of my doubles into a list, so was able to use the list(reversed()) functions to change the endienness.
You can then convert this back to a double. Example python code:
import struct
b = bytearray([51,51,51,51,51,51,19,64]) #this is the number 4.8
value = struct.unpack('d', b)
print(value) #4.8
This is probably obvious to more experienced programmers, however it had me flummuxed for days. I apologise for using stackoverflow as the platform to share this by answering my own question, but hopefully this post helps the next person who is struggling.
EDIT: Note if you are using an earlier version than Python 2.7.5 then you might find struct.unpack() will fail. Using the example code above substituting the following code worked for me:
b = bytes(bytearray([51,51,51,51,51,51,19,64]))
This code works for me. UDP server accept flattened dbl array x, return x+1 to port 6503. Modify LabView UDP client to your needs.
import struct
import socket
import numpy as np
def get_ip():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
# doesn't even have to be reachable
s.connect(('10.255.255.255', 1))
IP = s.getsockname()[0]
except:
IP = '127.0.0.1'
finally:
s.close()
return IP
#bind_ip = get_ip()
print("\n\n[*] Current ip is %s" % (get_ip()))
bind_ip = ''
bind_port = 6502
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((bind_ip,bind_port))
print("[*] Ready to receive UDP on %s:%d" % (bind_ip,bind_port))
while True:
data, address = server.recvfrom(1024)
print('[*] Received %s bytes from %s:' % (len(data), address))
arrLen = struct.unpack('>i', data[:4])[0]
print('[*] Received array of %d doubles:' % (arrLen,))
x = []
elt = struct.iter_unpack('>d', data[4:])
while True:
try:
x.append(next(elt)[0])
print(x[-1])
except StopIteration:
break
x = np.array(x)
y = x+1 # np.sin(x)
msg = data[:4]
for item in y:
msg += struct.pack('>d', item)
print(msg)
A = (address[0], 6503)
server.sendto(msg, A)
break
server.close()
print('[*] Server closed')
print('[*] Done')
LabView UDP client:
I understand that this does not solve your problem as you mentioned you didn't have the ability to modify the LabVIEW code. But, I was hoping to add some clarity on common ways string data is transmitted over TCP in LabVIEW.
The Endianness of the data string sent through the Write TCP can be controlled. I recommend using the Flatten To String Function as it gives you the ability to select which byte order you want to use when you flatten your data; big-endian (default if unwired), native (use the byte-order of the host machine), or little-endian.
Another common technique I've seen is using the Type Cast Function. Doing this will convert the numeric to a big-endian string. This of course can be confusing when you read it on the other end of the network as most everything else is little-endian, so you'll need to do some byte-swapping.
In general, if you're not sure what the code looks like, assume that it will be big-endian if its coming from LabVIEW code.
The answer from nekomatic is good one. Using a standard text format when available is always a good option.
Related
I have an issue with the pymodbus decoder with strings. For example, when I try to read 'abcdefg' pymodbus gives me 'badcfehg'. The byteorder and the wordorder don't change the result.
Here is my code:
result=client.read_holding_registers(25000,4)
decoder = BinaryPayloadDecoder.fromRegisters(result.registers,byteorder=Endian.Little,wordorder=Endian.Big)
decoder.decode_string(8)
Can someone explain why the order does not change the result? I try with the builder and it's the same problem. However, I don't have this problem with 32 bits floats for example.
I also tried with an older version of pymodbus and it works:
decoder = BinaryPayloadDecoder.fromRegisters(registers,endian=Endian.Little)
Note: I already read the following topic: pymodbus: Issue reading String & multiple type of data from Modbus device but I don't have any access to the modbus server.
The problem is that Modbus specs does not define in what order the two bytes for char strings are sent or even in what order 16-bit words are sent for 32-bit types.
Then some Modbus devices send bytes or words in an order and others do the opposite.
If you are writing a Modbus client then you should add the option in the configuration to be able to invert the order of both bytes and 16-bit words in 32-bit data types.
https://en.wikipedia.org/wiki/Endianness
I've encountered this same issue with byteorder not working (wordorder seems to be fine). The solution I came up with is to use Struct:
import struct
Then:
count = 4 #Read 4 16bit registers
result = client.read_holding_registers(25000,count)
for i in range(count):
result.registers[i] = struct.unpack("<H", struct.pack(">H", result.registers[i]))[0]
decoder = BinaryPayloadDecoder.fromRegisters(result.registers)
print(decoder.decode_string(7)) #Since string is 7 characters long
This uses Struct to unpack and pack as an unsigned short integer. The endianness does not matter since all you're doing is swapping bytes. The result overwrites the registers so you can then use the BinaryPayloadDecoder as you normally would.
I would have preferred to iterate through the responses instead of using range(count), but couldn't find a way to do it and wanted to post this workaround. If I figure it out, I will let you know.
I'm creating a networking protocol in application space on top of UDP in python for homework. I need to represent the source port and destination port as 16-bit numbers. All attempts have failed.
The way I'm testing this is by creating a udp socket and looking at the return value of sendto(). Here's your typical socket code:
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
addr = ("127.0.0.1", 1234)
Ports range from 0 to 65535. Let's say I choose a port of 65000. I want sendto() to return 2 (2 bytes = 16 bits sent). Here's what I've tried:
I call the following and get:
>>>mySock.sendto(655000, addr)
TypeError: must be string or buffer, not int
Ok, let's try using bytes()
>>>mySock.sendto(bytes(65000), addr)
5
Hm, that's not what I want. That is making each number into a character that is a single byte.
What if I bitwise or it with 0x0000?
>>>mySock.sendto(bytes(65000 | 0x0000), addr)
5
Well, darn it! The closest thing I've come to is messing around with hex() and bytearray(). See below.
>>>hex(65000)
'0xfde8'
>>>mySock.sendto('\xfde8', addr)
3
Shouldn't that say 2 bytes? I'm not sure how this works. Also, when the number is less than 16384 I want to preserve the preceding 0's. So, for example, if the port number is 255 (0b0000000011111111) I want it to remain as a 2 byte data structure (0x00FF) rather than truncating down to 0xFF or 0b11111111.
If you want to send binary data, please use module struct. That will help you to encode the string and to make sure that you are using the proper endianness. For example:
>>> import struct
>>> struct.pack('!H', 65000)
'\xfd\xe8'
That's 65000 as an unsigned short, in network order (big endian)
code:
import socket, binascii, struct
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_UDP)
while True:
print s.recv(2048)
output:
Ek�9##�F5��W�jq��� stackexchangecom� electronics �
h h
stackexchangecomDa�scifi ET##�<���� stackoverflowcom���meta
,��� stackoverflowcom�A���meta
,��� stackexchangecomG��security Ee##�+���� stackexchangecom���scifi
as you can see some of the data has been decoded/interpreted but the rest isn't not sure as to why
Can anyone help?
You're printing raw UDP packets, which contain arbitrary binary data. Some of those bytes are in the printable range, but those that aren't in that range get converted into �.
You can get a better look at that data by printing its representation, which shows the printable bytes as normal and shows the unprintable ones as hexadecimal escape codes. To do that, change your print statement to:
print repr(s.recv(2048))
I suspect you'd like to actually decode those packets. That's quite possible, but it's a bit technical, and you should probably study the topic a bit first. :) This article by Silver Moon, Code a network packet sniffer in python for Linux, looks quite helpful.
I'm here to ask for a remote bufferoverflow tip. I'm currently solving a wargame and it's my first time trying these some new types of coding.
Currently I am trying to make a code like this:
Pretend I have an address variable which has a range of 0xbfffff00 and 0xbfffffff. I want to brute-force this return address, which I'll socket.send through some socket programming. The problem is, the os I am using to learn from is in little-endian format, so I would like to send my return address backwards(by byte).
Example) addr = 0xb1234567
#I'd want to send 674523b1.
I've searched for a function like that and I found that print ('something'[::-1]).encode("hex") does the work, but this seems to work on strings, not when it is the number itself. Is there a way in python to send the address backwards?
Thanks.
struct.pack will convert a numeric value into a sequence of bytes, suitable for sending over the network. The sequence of bytes will be either little-endian or big-endian according to the format string:
In [7]: addr=0x30313233
In [8]: struct.pack("#I", addr) # native
Out[8]: '3210'
In [9]: struct.pack("<I", addr) # little-endian
Out[9]: '3210'
In [10]: struct.pack(">I", addr) # big-endian
Out[10]: '0123'
But note that in the ordinary course of network programming, you'll never use the above formats. Instead, you'll always send in "network byte order":
In [11]: struct.pack("!I", addr) # network byte order
Out[11]: '0123'
I was looking at some code in Python (I know nothing about Python) and I came across this portion:
def do_req(body):
global host, req
data = ""
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, 80))
s.sendall(req % (len(body), body))
tmpdata = s.recv(8192)
while len(tmpdata) > 0:
data += tmpdata
tmpdata = s.recv(8192)
s.close()
return data
This is then called later on with body of huge size, as in over 500,000 bytes. This is sent to an Apache server that has the max request size on the default 8190 bytes.
My question is what is happening at the "s.sendall()" part? Obviously the entire body cannot be sent at once and I'm guessing it is reduced by way of the modulus operator. I don't know how it works in Python, though. Can anyone explain? Thanks.
It is not really the modulus operator (technically it is since strings simply implement __mod__) but the python2-style string formatting operator.
Given format % values (where format is a string or Unicode object), % conversion specifications in format are replaced with zero or more elements of values. The effect is similar to the using sprintf() in the C language.
Obviously the entire body cannot be sent at once
While it indeed doesn't fit into a single packet that's a low-level thing which is handled internally (most likely not even by python but by the underlying syscall that writes to a socket)
No, body is not reduced here, because % is format operator, when operates on strings.
http://docs.python.org/release/2.5.2/lib/typesseq-strings.html
All the data is sent by sendall method by parts.
socket.sendall works like that:
do {
n = sendsegmented(s->sock_fd, buf, len, flags);
len -= n;
} while (len > 0);
where sendsegmented sends data and returns len or SEGMENT_SIZE