We Keep Coding

Converting Bytes to float format

I have the following data from a datagram message received by UDP socket:
b'/13/raw\x00,ffffffffffffff\x00=\xca\x00\x00=[\x00\x00\xbf\x82H\x00B\x14\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00B;\x81\n\xbf\xbat4C-\xfd\xb0B\x87\xac\xd9\x00\x00\x00\x00E\xcd\xc0\x00I\xcc\xc1\xe0'
This is my code:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((IP, p))
print("Listening for UDP packets on {0}:{1} ...".format(IP, p))
while 1:
rawdata, _ = sock.recvfrom(1024)
print(rawdata)
How can I convert rawdata to floats?
This is the data type that I am expecting:
Both are different samples!
Thank you

each float is 4 bytes of data (typically... not always... again you need to consult the docs for whatever p[ayload you need)
but assuming that /13/raw is easy enough for you and the rest is what you need to decode
if we assume that \x00 is some sort of delimiter we are left with the bytestring as follows
msgBytes = b',ffffffffffffff\x00=\xca\x00\x00=[\x00\x00\xbf\x82H\x00B\x14\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00B;\x81\n\xbf\xbat4C-\xfd\xb0B\x87\xac\xd9\x00\x00\x00\x00E\xcd\xc0\x00I\xcc\xc1\xe0'
we can see there are 72 characters which means 18 floats (18*4 === 72)
however your expected output only seems to have 14 floats so that means 16 bytes are some sort of metadata (maybe a checksum, or some payload information (how many floats to expect))
we can just struct unpack it as floats now ... but i dont get your "expected value"
so some of those may be doubles or something other than 4 byte floats
floats = struct.unpack('18f',msgBytes)
now you have a list of 18 floats ... but again not matching your expected output, it may be related to endianness or some other method that you need to do additional processing on (eg maybe it should be ints and they become floats by dividing by 100 or something?)
[edit] investigating further I suspect ffffffffff\x00 is some sort of header or metadata
so if we start with
myBytes = b'=\xca\x00\x00=[\x00\x00\xbf\x82H\x00B\x14\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00B;\x81\n\xbf\xbat4C-\xfd\xb0B\x87\xac\xd9\x00\x00\x00\x00E\xcd\xc0\x00I\xcc\xc1\xe0'
and unpack it with struct.unpack('14f',myBytes) we get 14 floats as expected ... some are zero like in your example ... but the rest dont match
(7.254942539348875e-41, 3.273012823123475e-41, 6.659058584274783e-39, 1.1762210642058864e-38, 0.0, 0.0, 0.0, 1.244453845292514e-32, 2.2792208653754642e-07, -1.84210369180704e-09, -6070301691478016.0, 0.0, 1.7706052095336117e-38, -1.1171693447134314e+20)
so I think we still dont have enough info to actually answer this question, need to know more about the datasource and other stuff

python integer format conversion

I need an application that will do the following:
read UDP message, get data from it - the data is textual, comma separated.
build a new message based on this values
send a new UDP message to other destination.
For the first and the last I'm fine. for the second I have a challenge:
Lets say that the data I've received is Year = 2020. The message on the Wireshark (step 3), I shall see it as 2 bytes valued 07 e4
How can I do it? I've tried couple of ways, none of them provided me with the desired format.
Sample of the code:
data1 = '\xab\xcd\xef\xgh'
...
data, addr = sock.recvfrom (200)
elements = data.decode().split(",")
Date=elements[15].split("/")
Year = int(Date(2))
year = <do something with Year to convert to right format>
newMsg = data1 + year
...
newSock.sendto(newMsg.encode('charmap'), (IP, int(port)))
Python version 3.5

there seem to be a couple of issues here. first, if your data1 is really supposed to be raw bytes then you're better off declaring it as such by prefixing it with a b, something like:
data1 = b'\xab\xcd\xef\xff'
if it comes from somewhere else, then encode it to bytes appropriately.
for answering your main question, the struct module has useful tools for encoding numbers as bytes in the way you want. ctypes can also be useful, but struct tends to be easier to use for these sorts of things:
data, addr = sock.recvfrom (200)
elements = data.decode().split(",")
date = elements[15].split("/")
year = int(date(2))
newMsg = data1 + struct.pack('!h', year)
and then you can send it with:
newSock.sendto(newMsg, (IP, int(port)))

Enconding temperature for exchange over bluetooth

I'm trying to understand how to encode some data for transfer over BLE (bluetooth low energy).
Specifically, I'm interested in this particular line:
https://github.com/micropython/micropython/blob/05f95682e7ddfb08c317e83826df9a1d636676f3/ports/nrf/examples/ubluepy_temp.py#L68
Which comes from the snippet:
temp = Temp.read()
temp = temp * 100
char_temp.write(bytearray([temp & 0xFF, temp >> 8]))
Before we even come to the why part, I need to understand the how. In this snippet, the temperature is read from a sensor as a float, in Celsius. Let's say "20.00" for now. We then multiply it by 100, and then comes the encoding part:
2000 & 0xFF -> 208
2000 >> 8 -> 7
So we're basically sending:
>>> bytearray([208, 7])
bytearray(b'\xd0\x07')
Is this correct? I would say so, I checked it with my own device and this seems to be the data that is being sent, and it also works, I can read the temperature sent from my BLE device.
What I don't understand is why all these bit manipulations are required. For example, I tried to just send bytearray([hex(20)]) but it doesn't work (when trying to read the temperature from my phone, the data couldn't be parsed/converted).
Could you explain the format of the data sent please?

As shown in the tests, the straight forward way to do the conversion in Python is with to_bytes and from_bytes functionality
Using your data it would be for the bytes to an int direction:
>>> int.from_bytes([208, 7], byteorder='little', signed=True)
2000
From int to bytes:
>>> int(2000).to_bytes(2, byteorder='little', signed=True)
b'\xd0\x07'
Or from the reading to bytes:
>>> int(20.00*100).to_bytes(2, byteorder='little', signed=True)
b'\xd0\x07'
Bluetooth data should be a list or bytearray in little endian format. Each integer in the list needs to represent an octet.
From the source you linked to I can see the characteristic UUID is 0x2A6E:
uuid_temp = UUID("0x2A6E") # Temperature characteristic
This is an official UUID so is described in "16-bit UUID Numbers Document" at https://www.bluetooth.com/specifications/assigned-numbers/
There is more detailed explanation in the GATT Specification Supplement document on the Bluetooth SIG website in section: "2: Values and represented values". In that document it also explains how to represent Temperature this way:

unable to unpack information between custom Preamble in Python and telnetlib

I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)

Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.

I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.

Python struct as networking data packets (uknown byte sequence)

I am working on a server engine in Python, for my game made in GameMaker Studio 2. I'm currently having some issues with making and sending a packet.
I've successfully managed to establish a connection and send the first packet, but I can't find a solution for sending data in a sequence of which if the first byte in the packed struct is equal to a value, then unpack other data into a given sequence.
Example:
types = 'hhh' #(message_id, x, y) example
message_id = 0
x = 200
y = 200
buffer = pack(types, 0,x, y)
On the server side:
data = conn.recv(BUFFER_SIZE)
mid = unpack('h', data)[0]
if not data: break
if mid == 0:
sequnce = 'hhh'
x = unpack(sequnce, data)[1]
y = unpack(sequnce, data)[2]

It looks like your subsequent decoding is going to vary based on the message ID?
If so, you will likely want to use unpack_from which allows you to pull only the first member from the data (as written now, your initial unpack call will generate an exception because the buffer you're handing it is not the right size). You can then have code that varies the unpacking format string based on the message ID. That code could look something like this:
from struct import pack, unpack, unpack_from
while True:
data = conn.recv(BUFFER_SIZE)
# End of file, bail out of loop
if not data: break
mid = unpack_from('!h', data)[0]
if mid == 0:
# Message ID 0
types = '!hhh'
_, x, y = unpack(types, data)
# Process message type 0
...
elif mid == 1:
types = '!hIIq'
_, v, w, z = unpack(types, data)
# Process message type 1
...
elif mid == 2:
...
Note that we're unpacking the message ID again in each case along with the ID-specific parameters. You could avoid that if you like by using the optional offset argument to unpack_from:
x, y = unpack_from('!hh', data, offset=2)
One other note of explanation: If you are sending messages between two different machines, you should consider the "endianness" (byte ordering). Not all machines are "little-endian" like x86. Accordingly it's conventional to send integers and other structured numerics in a certain defined byte order - traditionally that has been "network byte order" (which is big-endian) but either is okay as long as you're consistent. You can easily do that by prepending each format string with '!' or '<' as shown above (you'll need to do that for every format string on both sides).
Finally, the above code probably works fine for a simple "toy" application but as your program increases in scope and complexity, you should be aware that there is no guarantee that your single recv call actually receives all the bytes that were sent and no other bytes (such as bytes from a subsequently sent buffer). In other words, it's often necessary to add a buffering layer, or otherwise ensure that you have received and are operating on exactly the number of bytes you intended.

Could you unpack whole data to list, and then check its elements in the loop? What is the reason to unpack it 3 times? I guess, you could unpack it once, and then work with that list - check its length first, if not empty -> check first element -> if equal to special one, continue on list parsing. Did you try like that?