I'm trying to understand how to encode some data for transfer over BLE (bluetooth low energy).
Specifically, I'm interested in this particular line:
https://github.com/micropython/micropython/blob/05f95682e7ddfb08c317e83826df9a1d636676f3/ports/nrf/examples/ubluepy_temp.py#L68
Which comes from the snippet:
temp = Temp.read()
temp = temp * 100
char_temp.write(bytearray([temp & 0xFF, temp >> 8]))
Before we even come to the why part, I need to understand the how. In this snippet, the temperature is read from a sensor as a float, in Celsius. Let's say "20.00" for now. We then multiply it by 100, and then comes the encoding part:
2000 & 0xFF -> 208
2000 >> 8 -> 7
So we're basically sending:
>>> bytearray([208, 7])
bytearray(b'\xd0\x07')
Is this correct? I would say so, I checked it with my own device and this seems to be the data that is being sent, and it also works, I can read the temperature sent from my BLE device.
What I don't understand is why all these bit manipulations are required. For example, I tried to just send bytearray([hex(20)]) but it doesn't work (when trying to read the temperature from my phone, the data couldn't be parsed/converted).
Could you explain the format of the data sent please?
As shown in the tests, the straight forward way to do the conversion in Python is with to_bytes and from_bytes functionality
Using your data it would be for the bytes to an int direction:
>>> int.from_bytes([208, 7], byteorder='little', signed=True)
2000
From int to bytes:
>>> int(2000).to_bytes(2, byteorder='little', signed=True)
b'\xd0\x07'
Or from the reading to bytes:
>>> int(20.00*100).to_bytes(2, byteorder='little', signed=True)
b'\xd0\x07'
Bluetooth data should be a list or bytearray in little endian format. Each integer in the list needs to represent an octet.
From the source you linked to I can see the characteristic UUID is 0x2A6E:
uuid_temp = UUID("0x2A6E") # Temperature characteristic
This is an official UUID so is described in "16-bit UUID Numbers Document" at https://www.bluetooth.com/specifications/assigned-numbers/
There is more detailed explanation in the GATT Specification Supplement document on the Bluetooth SIG website in section: "2: Values and represented values". In that document it also explains how to represent Temperature this way:
Related
I am trying to read a binary message from an ESP32 using a broker; i wrote a phyton script where I subscribe the topic. the message that i actually receive is:
b'\x00\x00\x00?'
this is a float binary little endian message but I don't the key to decode it. Is there a way to find the decode key based on this data?
This is my python code:
import paho.mqtt.client as mqtt
def on_connect1(client1, userdata1, flags1, rc1):
client1.subscribe("ESP32DevKit123/mytopic")
def on_message1(client1, userdata1, msg1):
print(msg1.topic+" "+ "TESTENZA: "+str(msg1.payload))
client1 = mqtt.Client()
client1.username_pw_set(username="myuser",password="mypassword")
client1.on_connect = on_connect1
client1.on_message = on_message1
client1.connect("linkclient", portnumber, 60)
def twosComplement_hex(hexval):
bits = 16 # Number of bits in a hexadecimal number format
on_message1 = int(hexval, bits)
if on_message1 & (1 << (bits-1)):
on_message1 -= 1 << bits
return on_message1
client1.loop_forever()
It also gives me an error in the line on_message1 -= 1 << bits; the error says: Expected intended block pylance. Any solutions?
The data you provided is b'\x00\x00\x00?' - I'm going to assume that this is 0000003f (please output hex with msg1.payload.hex()).
I'll also assume that by "float binary little endian" you mean a big endian floating point (IEE754) - note that this does not match up with the algorithm you are using (twos compliment). Plugging this input into an online tool indicates that the expected result ("Float - Big Endian (ABCD)") is 8.82818e-44 (it's worth checking with this tool; sometimes the encoding may not be what you think it is!).
Lets unpack this using python (see the struct docs for more information):
>>> from struct import unpack
>>> unpack('>f', b'\x00\x00\x00\x3f')[0]
8.828180325246348e-44
Notes:
The [0] is there because unpack returns an array (you can unpack more than one item from the input)
>f - the > means big-endian and the f float (standard size = 4 bytes)
The reason your original code gives the error "Expected intended block" is due to the lack of indentation in the line on_message1 -= 1 << bits (as it follows an if it needs to be indented). The algorithm does not appear relevant to your task (but there may be details I'm missing).
I'm a heavy LabVIEW user who is just starting to learn Python. I work with industrial and aerospace equipment a lot and something I need to do very often is process some data, then export it over some communications protocol in binary. For example, let's say I have a packet that contains a struct/cluster/other-complex-data-element that has the following underlying data elements:
sync - unsigned 32-bit integer
time - 64-bit double
payload ID - 16-bit signed integer
source - 16-but signed integer
destination - 16-bit signed integer
payload length - 32 bit signed integer
data 1 - 8-bit unsigned integer
data 2 - 8-bit unsigned integer
data 3 - 8-bit unsigned integer
data 4 - 8-bit unsigned integer
data 4 - 64-bit double
data 5 - 32-bit single
data 6 - 16 bit unsigned integer
crc - 32-bit unsigned integer
(This frame should be 42 bytes long)
I call this a frame, where there is some header information, a payload, then a crc, I think that's a common term for what I'm creating. The data types, and their location in the byte stream is critical. Any extraneous or missing bytes breaks the data transfer protocol and the data cannot be tolerated.
My question is this:
How do you achieve this easily in Python? In LabVIEW (and probably other languages), there are good, built in functions and methods to clearly define the data types, then flatten them to a string of bytes that is very efficient. It seems that with picking, there are things going on that I don't understand.
In my example code, I have a simple function to get some memory information, then serialize it. I would expect the integer version to have 88 bytes and the float version to have 172 bytes, but I get 87 and 115 respectively. Here is the code, thanks for your help!
import psutil
import time
import pickle
def getMemoryInfo():
while True:
virtual_memory = psutil.virtual_memory()
swap_memory = psutil.swap_memory()
memoryInfo = list(virtual_memory+swap_memory)
# memoryInfo = [float(x) for x in memoryInfo]
time.sleep(1.000)
print(memoryInfo)
string = pickle.dumps(memoryInfo)
print(string)
print(len(memoryInfo))
print(len(string))
getMemoryInfo()
The struct module worked for me. It was a little more tedious than I would have hoped, but it worked just fine. Here is the code I ended up using:
def build_frame(payload, payload_class, payload_id, source, destination):
# form frame header and payload
frame = {"sync": int(0x64617665),
"absolute_time": time.time(),
"relative_time": time.monotonic(),
"source": int(source),
"destination": int(destination),
"counter": 0,
"payload_class": int(payload_class),
"payload_id": int(payload_id),
"payload_length": int(len(payload)),
"payload": payload}
# form bytearray to crc
sync = (struct.pack('<I', frame['sync']))
absolute_time = (struct.pack('<d', frame['absolute_time']))
relative_time = (struct.pack('<d', frame['relative_time']))
source = (struct.pack('i', frame['source']))
destination = (struct.pack('i', frame['destination']))
counter = (struct.pack('I', frame['counter']))
payload_class = (struct.pack('i', frame['payload_class']))
payload_id = (struct.pack('i', frame['payload_id']))
payload_length = (struct.pack('i', frame['payload_length']))
payload_bytes = frame['payload']
crc_bytes = sync + absolute_time + relative_time + source + destination + counter + payload_class + payload_id + payload_length + payload_bytes
# crc bytes and add to frame
frame['crc'] = binascii.crc32(crc_bytes)
return frame
I have an industrial sensor which provides me information via telnet over port 10001.
It has a Data Format as follows:
Also the manual:
All the measuring values are transmitted int32 or uint32 or float depending on the sensors
Code
import telnetlib
import struct
import time
# IP Address, Port, timeout for Telnet
tn = telnetlib.Telnet("169.254.168.150", 10001, 10)
while True:
op = tn.read_eager() # currently read information limit this till preamble
print(op[::-1]) # make little-endian
if not len(op[::-1]) == 0: # initially an empty bit starts (b'')
data = struct.unpack('!4c', op[::-1]) # unpacking `MEAS`
time.sleep(0.1)
my initial attempt:
Connect to the sensor
read data
make it to little-endian
OUTPUT
b''
b'MEAS\x85\x8c\x8c\x07\xa7\x9d\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x86\x8c\x8c\x07\xa7\x9e\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x85\x8c\x8c\x07\xa7\x9f\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa0\x01\x0c'
b'\xa7\xa2\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xa1\x01\x0c'
b'\x8c\x07\xa7\xa3\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07'
b'\x88\x8c\x8c\x07\xa7\xa4\x01\x0c\x15\x04\xf6MEAS\x88\x8c'
b'MEAS\x8b\x8c\x8c\x07\xa7\xa5\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xa6\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xa7\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa8\x01\x0c'
b'\x01\x0c\x15\x04\xf6MEAS\x88\x8c\x8c\x07\xa7\xa9\x01\x0c'
b'\x8c\x07\xa7\xab\x01\x0c\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xaa'
b'\x8c\x8c\x07\xa7\xac\x01\x0c\x15\x04\xf6MEAS\x8c\x8c'
b'AS\x89\x8c\x8c\x07\xa7\xad\x01\x0c\x15\x04\xf6MEAS\x8a'
b'MEAS\x88\x8c\x8c\x07\xa7\xae\x01\x0c\x15\x04\xf6ME'
b'\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xaf\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb0\x01\x0c'
b'\x0c\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb1\x01\x0c'
b'\x07\xa7\xb3\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c\x07\xa7\xb2\x01'
b'\x8c\x8c\x07\xa7\xb4\x01\x0c\x15\x04\xf6MEAS\x89\x8c\x8c'
b'\x85\x8c\x8c\x07\xa7\xb5\x01\x0c\x15\x04\xf6MEAS\x84'
b'MEAS\x87\x8c\x8c\x07\xa7\xb6\x01\x0c\x15\x04\xf6MEAS'
b'\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb7\x01\x0c\x15\x04\xf6'
b'\x15\x04\xf6MEAS\x8b\x8c\x8c\x07\xa7\xb8\x01\x0c\x15'
b'\x15\x04\xf6MEAS\x8a\x8c\x8c\x07\xa7\xb9\x01\x0c'
b'\xa7\xbb\x01\x0c\x15\x04\xf6MEAS\x87\x8c\x8c\x07\xa7\xba\x01\x0c'
try to unpack the preamble !?
How do I read information like Article number, Serial number, Channel, Status, Measuring Value between the preamble?
The payload size seems to be fixed here for 22 Bytes (via Wireshark)
Parsing the reversed buffer is just weird; please use struct's support for endianess. Using big-endian '!' in a little-endian context is also odd.
The first four bytes are a text constant. Ok, fine perhaps you'll need to reverse those. But just those, please.
After that, use struct.unpack to parse out 'IIQI'. So far, that was kind of working OK with your approach, since all fields consume 4 bytes or a pair of 4 bytes. But finding frame M's length is the fly in the ointment since it is just 2 bytes, so parse it with 'H', giving you a combined 'IIQIH'. After that, you'll need to advance by only that many bytes, and then expect another 'MEAS' text constant once you've exhausted that set of measurements.
I managed to avoid TelnetLib altogether and created a tcp client using python3. I had the payload size already from my wireshark dump (22 Bytes) hence I keep receiving 22 bytes of Information. Apparently the module sends two distinct 22 Bytes payload
First (frame) payload has the preamble, serial, article, channel information
Second (frame) payload has the information like bytes per frame, measuring value counter, measuring value Channel 1, measuring value Channel 2, measuring value Channel 3
The information is in int32 and thus needs a formula to be converted to real readings (mentioned in the instruction manual)
(as mentioned by #J_H the unpacking was as He mentioned in his answer with small changes)
Code
import socket
import time
import struct
DRANGEMIN = 3261
DRANGEMAX = 15853
MEASRANGE = 50
OFFSET = 35
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('169.254.168.150', 10001)
print('connecting to %s port %s' % server_address)
sock.connect(server_address)
def value_mm(raw_val):
return (((raw_val - DRANGEMIN) * MEASRANGE) / (DRANGEMAX - DRANGEMIN) + OFFSET)
if __name__ == '__main__':
while True:
Laser_Value = 0
data = sock.recv(22)
preamble, article, serial, x1, x2 = struct.unpack('<4sIIQH', data)
if not preamble == b'SAEM':
status, bpf, mValCounter, CH1, CH2, CH3 = struct.unpack('<hIIIII',data)
#print(CH1, CH2, CH3)
Laser_Value = CH3
print(str(value_mm(Laser_Value)) + " mm")
#print('RAW: ' + str(len(data)))
print('\n')
#time.sleep(0.1)
Sure enough, this provides me the information that is needed and I compared the information via the propreitary software which the company provides.
I'm currently working on a project that involves some very remote data gathering. At the end of each day, a very short summary is sent back to the server via a satellite connection.
Because sending data over the satellite is very expensive, I want the data to be as compact as possible. Additionally, the service I'm using only allows for sending data in the following two formats: ASCII, hexadecimal. Most of the data I will be sending consists of floats, where the precision should be as high as possible without taking up too much space.
Below I have a working version of what I'm currently using, but there should be a more efficient way to store the data. Any help would be much appreciated.
import ctypes
#------------------------------------------------------------------
#This part is known to the client as well as the server
class my_struct(ctypes.Structure):
_pack_ = 1
_fields_ = [('someText', ctypes.c_char * 12),
('underRange', ctypes.c_float),
('overRange', ctypes.c_float),
('TXPDO', ctypes.c_float)]
def print_struct(filled_struct):
d = filled_struct
for name,typ in d._fields_:
value = getattr(d, name)
print('{:20} {:20} {}'.format(name, str(value), str(typ)))
#------------------------------------------------------------------
# This part of the code is performed on the client side
#Filling the struct with some random data, real data will come from sensors
s = my_struct()
s.someText = 'Hello World!'.encode()
s.underRange = 4.01234
s.overRange = 4.012345
s.TXPDO = 1.23456789
#Rounding errors occurred when converting to ctypes.c_float
print('Data sent:')
print_struct(s)
data = bytes(s) #Total length is 24 bytes (12 for the string + 3x4 for the floats)
data_hex = data.hex() #Total length is 48 bytes in hexadecimal format
#Now the data is sent over a satellite connection, it should be as small as possible
print('\nLength of sent data: ',len(data_hex),'bytes\n')
#------------------------------------------------------------------
# This part of the code is performed on the server side
def move_bytes_to_struct(struct_to_fill,bytes_to_move):
adr = ctypes.addressof(struct_to_fill)
struct_size = ctypes.sizeof(struct_to_fill)
bytes_to_move = bytes_to_move[0:struct_size]
ctypes.memmove(adr, bytes_to_move, struct_size)
return struct_to_fill
#Data received can be assumed to be the same as data sent
data_hex_received = data_hex
data_bytes = bytes.fromhex(data_hex_received)
data_received = move_bytes_to_struct(my_struct(), data_bytes)
print('Data received:')
print_struct(data_received)
I don't know if you are overcomplicating things a bit.
The struct module will let you do pretty much what you are doing, but simpler:
struct.pack("fffs12", 4.01234, 4.012345, 1.23456789, 'Hello World!'.encode())
This of course depends on you knowing how long your string is, but you could also not care about that:
struct.pack("fff", 4.01234, 4.012345, 1.23456789) + 'Hello World!'.encode()
But, about saving things more efficient:
The more you know about your data, the more shortcuts you can take.
Is the string only ascii? You could squeeze each char into maybe 7 bits, or even 6.
That could bring your 12 bytes string to 9.
If you know the range of your floats, you could perhaps trim that as well.
If you can send larger batches, compression might help as well.
I've been able to find packets of interest using code based on this example:
How can I filter a pcap file by specific protocol using python?
The next child from the TCP packet is the actual data:
if isinstance(child1, TCP):
if child1.get_th_dport() == 80:
x = child1.child()
print x
This prints out the packet data like wire shark and shows hex and ascii versions. However I have been unable so far to find a way to simply get the hex contents. I know I can manipulate the printable output but I figured there must be a way to get the data in the hex form...
I've looked through the samples but none seem to do this. Anybody know the right way?
You can use packet.get_data_as_string() to get the raw bytes, and then display it however you like. I've replicated the "hex column" output produced by print child. Should be easy to tweak to produce ASCII columns as well:
def display_hex(pkt, cols=8):
size = cols * 4
data = ''.join('%02x' % ord(b) for b in pkt.get_data_as_string())
for i in range(0, len(data), size):
for j in range(0, size, 4):
print data[i+j:i+j+4],
print
if isinstance(child, TCP):
display_hex(child)
Output:
1703 0103 b0b1 9387 be4e fe00 9230 6192
e3bb 217e c1cb 8511 556f f986 4f31 542d
15c6 f42e f3bb 93d5 cf33 f126 c174 dbc4
... snip ...
8b1d 8707 96d6 7a18 2aab fd0b 48ee c4eb
b7d8 a67f 8bc0 597d 1044 a076 1a9e 24ba
959b fda3 1adb 2384 669c e6c8 c3b5 bef4
1189 eda8 3e