I am developing an application wherein I am receiving serial byte packets of varying length which are marked by framing flags into a buffer and then utilizing the collected bytes to access a dictionary to identify values. Currently, my serial read function watches the serial line until an END_FLAG byte is present or a timeout occurs as such...
def receive_command(self):
self.rxBuff = list(self.ser.read(1))
#Keep receiving bytes until END_FLAG or timeout
while self.rxBuff[-1] != chr(flagDict["END_FLAG"]):
tBuff = self.ser.read(1)
if len(tBuff) != 1:
print "Timeout"
else:
self.rxBuff.append(tBuff)
This ultimately reads the hex bytes into the buffer wherein they are represented with the escape character "\x##" or as their ASCII value. Within the code I have a dictionary which allows me to identify the microcontoller model based on a four byte hex value as such:
STM32_TYPE = {
0x410: "STM32F103RB",
0x415: "STM32L152RG",
0x417: "STM32L053R8",
0x421: "STM32F446RE",
0x431: "STM32F411RE",
0x433: "STM32F401RE",
0x437: "STM32L152RE",
0x439: "STM32F302R8",
0x438: "STM32F334R8",
0x440: "STM32F030R8",
0x442: "STM32F091RC",
0x446: "STM32F303RE",
0x447: "STM32L073RZ",
0x448: "STM32F070RB/STM32F072RB",
0x458: "STM32F410RB",
}
The problem I am having is that in order to access the values in this dictionary I need to combine two separate bytes(bytes 3 & 4 from the following packet).
['\x12','\x03','F','\x04','\x13']
My goal is to reassemble bytes 3 and 4 into the value 0x0446 to use to access the previously mentioned dictionary. However I am at a loss as to how to accomplish this as all attempts to do so have yielded either simply combined strings (which makes sense given that these are string representations), or integer values which are not held to be equivalent when I attempt to access the dictionary. My question is, is there a way to combine these hex values (the 'F' and '\x04') so as to use them to access the dictionary value with key 0x0446? OR, is there a better way to receive the hex bytes during my serial read function so that they are stored in a manner more conducive to combining them later on?
struct
>>> struct.unpack('<H', b'F\x04')
(1094,)
Related
I need to transmit data through UART from different sensors.
I need to form packet with data from sensors on microcontroller. Then transmit it through uart. and then decode it on laptop.
For 1 sensor i have a code:
while (1)
{
get_Temperature();
HAL_Delay(2000);
char buffer[100];
seg = Temp[0]; //here our temperature
sprintf(buffer, "%d",seg); //transform temperautre to string
HAL_UART_Transmit(&huart2, (uint8_t*)buffer, strlen(buffer), 1000); //string transmit
/* USER CODE END WHILE */
/* USER CODE BEGIN 3 */
}
For decode it on laptop i have a code(python with pyserial):
cons = wox.read_all().decode("utf-8")
*in loop
And my output is a value of temperature in string format:
24
24
25
24
depends on temperature of the sensor.
I can use the second sensor with another value Temp[1]
But i need to form packet with that data and decode it on laptop. How can i do it?
You might use delimiters to separate and carry the data in a single message and you should also use a delimiter for the end (or start, or both) of the data rather than relying on the delay to separate "messages". You might later want the data at a much higher rate or streaming, and "24,3224,32" would be ambiguous. For example
sprintf(buffer, "%d,%d\n", Temp[0], Temp[1] ) ;
Then the output would be clearly delimited:
24,32<newline>
24,32<newline>
...
Then at the receiver (in Python - not my area of expertise), you can use read line semantics (message = wox.readline().decode("utf-8")?) to get the entire message rather than relying on any specific timing.
A more flexible alternative, that would be extensible to more sensors of different types, outputting at different intervals in multiple threads, would be to tag the data with its sensor ID. For example, you might independently have:
sprintf( buffer, "T0:%d\n", Temp[0] ) ;
and
sprintf( buffer, "T1:%d\n", Temp[0] ) ;
Or even an accelerometer with:
sprintf( buffer, "AC:%d,%d,%d\n", x, y, z ) ;
So the output stream might look like:
AC:24,2,33<newline>
AC:24,2,33<newline>
AC:24,2,33<newline>
AC:24,2,33<newline>
T1:33<newline>
AC:24,2,33<newline>
AC:24,2,33<newline>
AC:24,2,33<newline>
AC:24,2,33<newline>
T0:24<newline>
...
Where the messages are free to arrive in any order, and at any frequency. Of course if multi-threaded, you'd probably want a single thread actually outputting to the UART with sensor threads placing messages in a queue, or otherwise ensure the independent messages were correctly serialised and not "interleaved".
You might split out sensor type from sensor instance so you could extend to any number for sensors of a particular type, so rather then say T0:, T1: and AC: you might have T#0:, T#1: and AC#0: for example followed by comma delimited parameters then <newline>.
If synchronisation/timing is critical and you have a source of time (relative or absolute), you might even add a timestamp to the message; e.g.:
sprintf( buffer, "$%lu,T0:%d\n", tick_ms(), Temp[0] ) ;
to the output message might look like:
$1256,T1:33<newline>
$1256,T0:55<newline>
$3256,T1:34<newline>
$3256,T0:54<newline>
The specific delimiters are arbitrary - your design.
If you want a great deal of flexibility and compatibility you could even output the data as an XML or JSON schema. A bit heavyweight for the embedded end perhaps, but there are no end of parsers available for Python to handle that, and the schema would be publishable to allow others to easily parse your data.
I'm trying to serialize a Record in Delphi by using MessagePack and then using ZeroMQ TCP protocol I send it to a Python server.
b'\xa8DataType\x01\xa4data\xbf{"major":1,"minor":0,"build":2}\x00'
I'm having trouble deserializing it on the server side. Any ideas why this is happening? Is it some kind of encoding issues? Thanks!
Update #1:
I use the messagepack library "QMsgPack" recommended on www.msgpack.org Here's some Delphi code. My user defined Records and an enum:
Version = Record
major : Integer;
minor : Integer;
build : Integer;
end;
TDataType = (dtUnset, dtVersion, dtEntityDescription, dtEntityDescriptionVector, dtEntityState, dtEntityStateVector, dtCommandContinue, dtCommandComplete);
TPacket = Record
DataType : TDataType;
data : string;
end;
And the code to serialize the object:
begin
dmVersion.major := 1;
dmVersion.minor := 1;
dmVersion.build := 1;
lvMsg := TQMsgPack.Create;
lvMsg.FromRecord(dmVersion);
lvMsgString := lvMsg.ToString();
packet.DataType := dtVersion;
packet.data := lvMsgString;
lvMsg.Clear;
lvMsg.FromRecord(packet);
lvbytes:=lvMsg.Encode;
ZeroMQ.zSendByteArray(skt, lvbytes);
I then try to deserialize the received byte array in the python server which looks like this:
b'\xa8DataType\x01\xa4data\xbf{"major":1,"minor":0,"build":2}\x00'
by using umsgpack.unpack() and then print out the result in the result like this:
packet_packed = command.recv()
# Unpack the packet
umsgpack.compatibility = True
packet = umsgpack.unpackb( packet_packed )
print (packet)
for item in packet:
print (item)
and this is what I get printed out on the screen:
b'DataType'
68
97
116
97
84
121
112
101
I hope this helps! Thanks!
Update #2
Here is some server code on the python side. The VDS_PACKET_VERSION is a constant int set to 1.
# Make sure its a version packet
if VDS_PACKET_VERSION == packet[0]:
# Unpack the data portion of the packet
version = umsgpack.unpackb( packet[1] )
roster = []
if ( VDS_VERSION_MAJOR == version[0] ) and ( VDS_VERSION_MINOR == version[1] ) and ( VDS_VERSION_BUILD == version[2] ):
dostuff()
With the current serialized object
b'\x82\xa8DataType\x01\xa4data\xbf{"major":1,"minor":1,"build":1}'
I get
KeyError: 0 on packet[0]
Why is that?
The packed data appears to be invalid.
>>> packet = { "DataType": 1, "data": "{\"major\":1,\"minor\":0,\"build\":2}"}
>>> umsgpack.packb(packet)
b'\x82\xa4data\xbf{"major":1,"minor":0,"build":2}\xa8DataType\x01'
The first byte is \x82 which, as can be seen in the specification, is a two entry fixmap.
Your packed data is missing that information, and launches straight in to a fixstr. So, yes, there could be a mismatch between your Delphi based packer and the Python based unpacker. However, when I take your Delphi code, using the latest qmsgpack from the repo, it produces the following bytes:
82A8446174615479706501A464617461
BF7B226D616A6F72223A312C226D696E
6F72223A312C226275696C64223A317D
Let's convert that into a Python bytes object. It looks like this:
b'\x82\xa8DataType\x01\xa4data\xbf{"major":1,"minor":1,"build":1}'
Now, that's quite different from what you report. And umsgpack can unpack it just fine. Note that the first byte is \x82, a two entry fixmap, just as expected. Yes, the entries are in a different order, but that's just fine. Order is not significant for a map.
So, I've been able to encode using qmsgpack in Delphi, and decode using umsgpack in Python. Which then suggests that this issue is really in the transmission. It looks to me as though there has been an off-by-one error. Instead of transmitting bytes 0 to N-1, bytes 1 to N have been transmitted. Note the spurious trailing zero in your received data.
In the comments you obverse that the data field is being coded as JSON and passed as a string. But you'd rather have that data encoded using MessagePack. So here's what to do:
In the Delphi code change the data field's type from string to TBytes. That's because we are going to put a byte array in there.
Populate the data field using Encode, like this: packet.data := lvMsg.Encode.
On the Python side, when you unpack data you'll find that it is an array of integers. Convert that to bytes and then unpack: umsgpack.unpackb(bytes(data)).
Simply put, I have a dictionary (dictData[name]=namedTuple) of namedTuples (records) on a server.
Objective: I want to send the entire thing (dictData) or a single instance (dictData[key]) to the client via a SOCKET connection so it can be printed (shown on screen).
To send a single record I have tried to do the following:
response = dictData["John"]
print (response) #ensure it is the correct record
s.send(response)
However this generates the following error:
"TypeError: 'record' does not support the buffer interface"
I have tried to encode it and convert it but nothing I do seems to work. I am even open to converting it to a STRING and sending the string but I can't seem to find out how to convert a namedTuple to a string either.
And then, no clue where to start to send the entire dictionary to the client so they can print the entire set?
Any help would be much appreciated.
Sockets can only send and receive bytes, so you need to serialise your named tuples and dictionary to something else.
You could use JSON to produce a string representation of your tuples and dictionary, for example. The json library produces a (unicode) string when encoding, you'll need to encode that to UTF-8 (or similar) to produce bytes:
import json
# sending one tuple
response = json.dumps(dictData["John"])
s.send(response.encode('utf8'))
# sending all of the dictionary
response = json.dumps(dictData)
s.send(response.encode('utf8'))
This will not preserve the named tuple attribute names; the values are sent over as a JSON array instead (so an ordered list).
Another option is to use the pickle module; this would require the listener on the other side to also be coded in Python and to have the same record named tuple type importable from the exact same location, however.
When Pickle loads the data, the name of the full qualifying name of the namedtuple type is included, and you must be able to import that type on both ends of the socket. If you have a line in a module at the global level:
record = namedtuple('record', 'field1 field2 field3')
then from yourmodule import record is possible, but the exact same import should work on the other side too.
pickle.dumps() produces a bytes object which can be written to the socket without encoding.
I need to send an array of namedtuples by a socket.
To create the array of namedtuples I use de following:
listaPeers=[]
for i in range(200):
ipPuerto=collections.namedtuple('ipPuerto', 'ip, puerto')
ipPuerto.ip="121.231.334.22"
ipPuerto.puerto="8988"
listaPeers.append(ipPuerto)
Now that is filled, i need to pack "listaPeers[200]"
How can i do it?
Something like?:
packedData = struct.pack('XXXX',listaPeers)
First of all you are using namedtuple incorrectly. It should look something like this:
# ipPuerto is a type
ipPuerto=collections.namedtuple('ipPuerto', 'ip, puerto')
# theTuple is a tuple object
theTuple = ipPuerto("121.231.334.22", "8988")
As for packing, it depends what you want to use on the other end. If the data will be read by Python, you can just use Pickle module.
import cPickle as Pickle
pickledTuple = Pickle.dumps(theTuple)
You can pickle whole array of them at once.
It is not that simple - yes, for integers and simple numbers, it s possible to pack straight from named tuples to data provided by the struct package.
However, you are holding your data as strings, not as numbers - it is a simple thing to convert to int in the case of the port - as it is a simple integer, but requires some juggling when it comes to the IP.
def ipv4_from_str(ip_str):
parts = ip_str.split(".")
result = 0
for part in parts:
result <<= 8
result += int(part)
return result
def ip_puerto_gen(list_of_ips):
for ip_puerto in list_of_ips:
yield(ipv4_from_str(ip_puerto.ip))
yield(int(ip_puerto.puerto))
def pack(list_of_ips):
return struct.pack(">" + "II" * len(list_of_ips),
*ip_puerto_gen(list_of_ips)
)
And you then use the "pack" function from here to pack your structure as you seem to want.
But first, attempt to the fact that you are creating your "listaPiers" incorrectly (your example code simply will fail with an IndexError) - use an empty list, and the append method on it to insert new named tuples with ip/port pairs as each element:
listaPiers = []
ipPuerto=collections.namedtuple('ipPuerto', 'ip, puerto')
for x in range(200):
new_element = ipPuerto("123.123.123.123", "8192")
listaPiers.append(new_element)
data = pack(listaPiers)
ISTR that pickle is considered insecure in server processes, if the server process is receiving pickled data from untrusted clients.
You might want to come up with some sort of separator character(s) for the records and fields (perhaps \0 and \001 or \376 and \377). Then putting together a message is kind of like a text file broken up into records and fields separated by spaces and newlines. Or for that matter, you could use spaces and newlines, if your normal data doesn't include these.
I find this module very valuable for framing data in socket-based protocols:
http://stromberg.dnsalias.org/~strombrg/bufsock.html
It lets you do things like "read up until the next null byte" or "read the next 10 characters" - without needing to worry about the complexities of IP aggregating or splitting packets.
I have a file that was written with the following Delphi declaration ...
Type
Tfulldata = Record
dpoints, dloops : integer;
dtime, bT, sT, hI, LI : real;
tm : real;
data : array[1..armax] Of Real;
End;
...
Var:
fh: File Of Tfulldata;
I want to analyse the data in the files (many MB in size) using Python if possible - is there an easy way to read in the data and cast the data into Python objects similar in form to the Delphi records? Does anyone know of a library perhaps that does this?
This is compiled on Delphi 7 with the following options which may (or may not) be pertinent,
Record Field Alignment: 8
Pentium Safe FDIV: False
Stack Frames: False
Optimization: True
Here is the full solutions thanks to hints from KillianDS and Ritsaert Hornstra
import struct
fh = open('my_file.dat', 'rb')
s = fh.read(40256)
vals = struct.unpack('iidddddd5025d', s)
dpoints, dloops, dtime, bT, sT, hI, LI, tm = vals[:8]
data = vals[8:]
I do not know how Delphi internally stores data, but if it is as simple byte-wise data (so not serialized and mangled), use struct. This way you can treat a string from a python file as binary data. Also, open files as binary file(open,'rb').
Please note that when you define a record in Delphi (like struct in C) the fields are layed out in order and in binary given the current alignment (eg Bytes are aligned on 1 byte boundaries, Words on 2 byte, Integers on 4 byte etc, but it may vary given the compiler settings.
When serialized to a file, you probably mean that this record is written in binary to the file and the next record is written after the first one starting at position sizeof( structure) etc etc. Delphi does not specify how thing should be serialized to/from file, So the information you give leaves us guessing.
If you want to make sure it is always the same without interference of any compiler setings, use packed record.
Real can have multiple meanings (it is an 48 bit float type for older Delphi versions and later on a 64 bit float (IEEE double)).
If you cannot access the Delphi code or compile it yourself, just ty to check the data with a HEX editor, you should see the boundaries of the records clearly since they start with Integers and only floats follow.