We Keep Coding

Reading mixed ASCII/binary serial data stream -- how to find delimiter?

I am trying to read serial data from a device that outputs in a mix of ASCII and binary, using Python 3.
The message format is: "$PASHR,msg type,binary payload,checksum,\r\n" (minus the quotes)
To make it more interesting, there are several different message types, and they have different payload lengths, so I can't just read X bytes (I can infer the payload length based on the message type). The sequence of a variable number of messages of each type (around 15 in total) is sent every 20 seconds, at 115200 baud.
I haven't been able to read this with serial.readline(), probably because of newlines embedded in the payload.
I think that if I could set the line-end character to the sequence "$PASHR" that would give me a way to frame the messages -- ie, everything between one $PASHR and the next is one message, and the likelihood of seeing the sequence in the binary payload is nil. But I have not been able to make it work either using serial.newline = b'$PASHR' or readuntil(b'$PASHR') -- I still get variable length reads. I suspect that the serial.timeout setting enters into the solution, but I am not sure how.
Here's the last version I ended up with last night:
delimiter = b'$PASHR'
while True:
if self.serial.in_waiting:
message = self.serial.read_until(delimiter)
Every method I've tried gives a variable response length, when each record of a give should be the same length. Is there a way to set the newline to that multi-character string, or is there a different/better approach I should use?
Thanks!

I think I figured it out. This seems to work with a timeout of 1 second:
self.serial.timeout = 1
delimiter = b'$PASHR'
if self.serial.in_waiting:
message = self.serial.read_until(delimiter)
if len(message) > 6: # ignore carryover PASHR'
***process message***
When I looked carefully at the variable length returns, I discovered that the message sequence after the first incomplete cycle was:
b'$PASHR' (6 bytes)
b',MPC,*data**checksum*\r\n$PASHR' (108 bytes)
...
b',MPC,*data**checksum*\r\n' (102 bytes)
(delay)
b'$PASHR' (6 bytes)
The delimiter from the last message in the sequence is chopped off and output at the beginning of the next sequence. I can deal with that.
I will try Deepstop's idea of the inter_byte timeout as it is probably more elegant than using the raw timeout, and might get around the partial message problem.

Converting a scapy packet to string produces an E?

I'm currently working on using scapy for sending data packets, and I've run into a weird issue. When I create a packet as such:
pack = IP(dst="127.0.0.1", id=local_ID)/UDP()/chunk
and then convert that packet to a string (so I can send it via a socket)
sendPack = str(pack)
the result of sendPack is wrong.
For instance, in my test file, I have the numbers 1-8000 ordered as such
1
2
3
...
then, when I print("SEND_PACK: "+sendPack)
it produces the following:
E
2
3
...
Everything else is perfect except for the E
I can't understand where that E is coming from, or what it means.
It's also worth noting that I have verified that pack contains the correct data, and that regardless of what the first line of the test file is, the first line of the output is always an E
Thanks!

To those interested, I fixed the issue by doing the following:
As pointed out above, the E was a result of me printing the packet, not it's contents. In order to access the contents I wanted, I had to do the following:
sendPack = pack[UDP].load #Get the packet's load at the UDP layer
id = pack[IP].ID #Get the ID at the IP layer
The documentation for Scapy is sparse, so I didn't realize that I could access the individual fields of each packet this way.
Here's where I found this fix

First, you are printing a packet, not the content of your UDP datagram. The first two fields of an IPv4 packet are version and IHL which values are respectively 4 (for IP version 4) and 5 (5 words of 32 bits) by default in Scapy. This gives 45 if you put it on one byte in hexadecimal which is letter "E" in ascii. This is why you are always seeing an E as the first byte of your string.
If you want to check what is in your packet, you should use Scapy's show() function: sendPack.show().
Second, you could use Scapy's send(), function to send your packet without bothering about sockets yourself: send(sendPack)

Reading LabVIEW TCP data (Flattened String / Data Cluster) in Python

I have a LabVIEW application that is flattening a cluster (array) of Doubles to a string, before transmitting over TCP/IP to my python application. It does this because TCP/IP will only transmit strings.
The problem is that python reads the string as a load of nonsense ASCII characters, and I can't seem to unscramble them back to the original array of doubles.
How do I interpret the string data that LabVIEW sends after flattening a data strings. My only hint of useful information after hours of google was a PyPI entry called pyLFDS, however it has since been taken down.

The LabVIEW flattened data format is described in some detail here. That document doesn't explicitly describe how double-precision floats (DBL type) are represented, but a little more searching found this which clarifies that they are stored in IEEE 754 format.
However it would probably be simpler and more future proof to send your data in a standard text format such as XML or JSON, both of which are supported by built-in functions in LabVIEW and standard library modules in Python.
A further reason not to use LabVIEW flattened data for exchange with other programs, if you have the choice, is that the flattened string doesn't include the type descriptor you need to convert it back into the original data type - you need to know what type the data was in order to decode it.

I wanted to document the problem and solution so others can hopefully avoid the hours I have wasted looking for a solution on google.
When LabVIEW flattens data, in this case a cluster of doubles, it sends them simply as a concatonated string with each double represented by 8 bytes. This is interpreted by python as 8 ASCII characters per double, which appears as nonsense in your console.
To get back to the transmitted doubles, you need to take each 8-byte section in turn and convert the ASCII characters to their ASCII codes, in Python's case using ord().
This will give you an 8 bytes of decimal codes (e.g. 4.8 = [64 19 51 51 51 51 51 51])
It turns out that LabVIEW does most things, including TCP/IP transmissions, Big Endian. Unless you are working Big Endian, you will probably need to change it around. For example the example above will become [51 51 51 51 51 51 19 64]. I put each of my doubles into a list, so was able to use the list(reversed()) functions to change the endienness.
You can then convert this back to a double. Example python code:
import struct
b = bytearray([51,51,51,51,51,51,19,64]) #this is the number 4.8
value = struct.unpack('d', b)
print(value) #4.8
This is probably obvious to more experienced programmers, however it had me flummuxed for days. I apologise for using stackoverflow as the platform to share this by answering my own question, but hopefully this post helps the next person who is struggling.
EDIT: Note if you are using an earlier version than Python 2.7.5 then you might find struct.unpack() will fail. Using the example code above substituting the following code worked for me:
b = bytes(bytearray([51,51,51,51,51,51,19,64]))

This code works for me. UDP server accept flattened dbl array x, return x+1 to port 6503. Modify LabView UDP client to your needs.
import struct
import socket
import numpy as np
def get_ip():
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
try:
# doesn't even have to be reachable
s.connect(('10.255.255.255', 1))
IP = s.getsockname()[0]
except:
IP = '127.0.0.1'
finally:
s.close()
return IP
#bind_ip = get_ip()
print("\n\n[*] Current ip is %s" % (get_ip()))
bind_ip = ''
bind_port = 6502
server = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server.bind((bind_ip,bind_port))
print("[*] Ready to receive UDP on %s:%d" % (bind_ip,bind_port))
while True:
data, address = server.recvfrom(1024)
print('[*] Received %s bytes from %s:' % (len(data), address))
arrLen = struct.unpack('>i', data[:4])[0]
print('[*] Received array of %d doubles:' % (arrLen,))
x = []
elt = struct.iter_unpack('>d', data[4:])
while True:
try:
x.append(next(elt)[0])
print(x[-1])
except StopIteration:
break
x = np.array(x)
y = x+1 # np.sin(x)
msg = data[:4]
for item in y:
msg += struct.pack('>d', item)
print(msg)
A = (address[0], 6503)
server.sendto(msg, A)
break
server.close()
print('[*] Server closed')
print('[*] Done')
LabView UDP client:

I understand that this does not solve your problem as you mentioned you didn't have the ability to modify the LabVIEW code. But, I was hoping to add some clarity on common ways string data is transmitted over TCP in LabVIEW.
The Endianness of the data string sent through the Write TCP can be controlled. I recommend using the Flatten To String Function as it gives you the ability to select which byte order you want to use when you flatten your data; big-endian (default if unwired), native (use the byte-order of the host machine), or little-endian.
Another common technique I've seen is using the Type Cast Function. Doing this will convert the numeric to a big-endian string. This of course can be confusing when you read it on the other end of the network as most everything else is little-endian, so you'll need to do some byte-swapping.
In general, if you're not sure what the code looks like, assume that it will be big-endian if its coming from LabVIEW code.
The answer from nekomatic is good one. Using a standard text format when available is always a good option.

Raspberrypi Python bus.read_byte

Is there a Python function that will respond like the Wire.available function in arduino to get all the data on the wire rather than having to specify how many bytes to grab?
This is what I have now, and it works fine, but I have to know how much data is coming down the wire, or it will provide unexpected results.
for i in range(0, 13):
data += chr(bus.read_byte(address));
Thanks!

Not a perfect solution, but I found a way around knowing exactly how many bytes are on the way.
On the Arduino, I specified the max size of the buffer, (128), add my data, then zero out the rest, and then send the whole thing. On the Pi, I receive the whole buffer, and then the first thing that happens is to filter the \x00 characters. It's not perfect, but it works for now.
for i in range(0, 128):
data += chr(bus.read_byte(address))
print repr(data)
#prints the whole string as it is received
data = filter(lambda a: a != '\x00')
print repr(data)
#prints the string without any '\x00' characters.

I use the PIGPIO library for i2c commands on the raspberrypi, it has functions much closer to wire.
http://abyz.co.uk/rpi/pigpio/python.html#i2c_read_device
I think that's the function you're looking for.

AS3 server socket, progress event, readutfbytes

having a bit of an issue here.
I am working on a multiplayer game with flash, and using python for the server side of things. I have the socket connection working... sorta, and quite a bit of the python work done, but I am running into a weird problem.
Let's say, upon logging in, I send some data to the client containing some of their info.
After which, I send some data to assign them to a room.
This data doesn't seem to be read in AS3 as two different things, instead, after readUTFBytes, it is all in the same string.
var str:String = event.currentTarget.readUTFBytes(event.currentTarget.bytesAvailable);
In python, I have defined methods for sending data, which just sends data via transport.write (Twisted), and I am receieving via a progress socket data event in action script. Any idea what could be wrong here? Here's a bit of code...
if ( ! event.currentTarget.bytesAvailable > 0) {
return;
}
var str:String = event.currentTarget.readUTFBytes(event.currentTarget.bytesAvailable);
var Char1:String = str.charAt(0);
var Char2:String = str.charAt(1);
str = str.replace(Char1, "");
str = str.replace(Char2, "");
// Various messages
if (Char1 == "\x03") {
if (Char2 == "\x03") {
trace("Got ping thread");
}
else {
trace("x03 but no secondary prefix handled");
}
return;
}
Quite sloppy I know, but I'm trying to just determine an issue.
All data comes with two prefixes, something like \x02 and \x09 for me to determine what to do, then most data in the string is split on \x01 to get values.
The problem essentially is, where I should get an /x08 /x08 data, I get /x08 /x08 data /x05 /x03 data, when it should be two separate things.

TCP connections are reliable, ordered, stream oriented transports. A stream is a sequence of bytes with no inherent message boundaries. If you want to split up your bytes into separate messages, the bytes themselves must tell you how to do this splitting (or you need some external rule that always applies, like "a message is 5 bytes long").
This applies to all TCP connections, regardless of what language you use them from, or what weird library-specific API gets dropped on top of them (like readUTFBytes).
There are many options for protocols which can help you frame your messages. For example, you could use a length prefix. Then your messages would look like:
\x07 \x08 \x08 h e l l o \x05 \x05 \x03 m a n
\x07 gives the length of the first message, 7 bytes: \x08 \x 08 h e l l o. The next byte after that message, \x05, gives the length of the second message: \x05 \x03 m a n.
You can use multibyte length prefixes if your messages need to be longer, or netstrings which use a decimal representation and a : delimiter to support arbitrary sized prefixes. There are also more sophisticated protocols which offer more features than just separating your bytes into messages. For example, there's AMP which gives you a form of RPC with arguments and responses.