We Keep Coding

Issue packing / unpacking message for UART between ESP-01S and Raspberry Pi Pico

My end goal is to set up a Raspberry Pi Pico with an ESP-01S to enable wifi. The Pico will periodically check in with the server and put the ESP to sleep when not in use. Communication between the two is over UART. I want to support both a GET and POST HTTP request from the ESP.
This is the message structure I am using between the two devices.
| Always start with MSG | Request Type | Next message Size
| | Next message size | | | URL
V V V V V
|-----|-------------------|-----|-------------------|-----------|
[M|S|G|\x00\|x00\|x00\|x03|G|E|T|\x00|\x00|\x00|\x1f|h|t|t|p|...]
[M|S|G|\x00\|x00\|x00\|x04|P|O|S|T|URL SIZE...|URL...|\x00|\x00|\x006|POST Data...]
|-----|-------------------|-------|-----------|------|---------------|------------|
^ ^
| | Post Data
| Post data size
For testing purposes I am generating the strings in python, printing them and pasting them directly in the .cpp file I'm flashing onto the ESP.
Here is the snippet of code I'm using on my pc to generate the message.
import struct
import json
url = "http://192.168.X.X:8090/korok"
size_of_url = struct.pack('!I', len(url))
data = json.dumps({
"serial": "12345",
"sensor_data": {"0": 75, "1": 67}
})
size_of_data = struct.pack('!I', len(data))
print(f"{struct.pack('!I', len('GET'))}GET{size_of_url}{url}")
print(f"{struct.pack('!I', len('POST'))}POST{size_of_url}{url}{size_of_data}{data}")
>>>> ...
b'\x00\x00\x00\x03'GETb'\x00\x00\x00\x1f'http://192.168.X.X:8090/korok
b'\x00\x00\x00\x03'GETb'\x00\x00\x00\x1f'http://192.168.X.X:8090/korokb'\x00\x00\x006'{"serial": "12345", "sensor_data": {"0": 75, "1": 67}}
And here is the code running on the ESP. I left comments on some of the behaviors I'm experiencing.
#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#define GPIO_STATUS 2
#define BUFFER_SIZE 256
#define DATA_SIZE 4
char buf[BUFFER_SIZE];
void setup() {...}
...
// Struct is used so I can also get the size to use as an offset
// and read out part of the message in the buffer.
struct Message
{
char *value;
unsigned int size;
unsigned int totalMessageSize;
};
Message readMessage(char *data)
{
struct Message message;
message.size = ntohl(*(unsigned int *)(data)); // Unable to read shorthand hex
Serial.println(message.size);
message.totalMessageSize = message.size + DATA_SIZE; // Shorthand is only 3 char
message.value = (char *)malloc(message.size);
Serial.println(message.size);
int idx = DATA_SIZE;
int jdx = 0;
while (idx < message.size + DATA_SIZE && idx < BUFFER_SIZE)
{
message.value[jdx++] = data[idx++];
}
return message;
}
void loop()
{
delay(3000);
char * msg = "\x00\x00\x00\x03GET\x00\x00\x00\x1fhttp://192.168.X.X:8090/korok";
// char *msg = read_message();
if (msg)
{
Serial.print("\n");
int bufferIdx = 0;
struct Message request = readMessage(msg);
bufferIdx = request.totalMessageSize;
// This is odd and doing it due to odd behavior with ntohl and a variable
// offset. See below.
memcpy(msg, msg + bufferIdx, BUFFER_SIZE - bufferIdx);
struct Message url = readMessage(msg);
struct Message data;
if (memcmp(request.value, "GET", 3) == 0)
{
}
else if (memcmp(request.value, "POST", 4) == 0)
{
bufferIdx = url.totalMessageSize;
memcpy(msg, msg + bufferIdx, BUFFER_SIZE - bufferIdx);
struct Message data = readMessage(msg);
Serial.println(data.value);
}
free(request.value);
free(url.value);
if (memcmp(request.value, "POST", 4) == 0)
{
free(data.value);
}
}
}
Here is one of the two issues, though I found a workaround by doing a memcpy of my original char* offsetting the beginning index. The first line below works but the second throws LoadStoreAlignmentCause exception.
Ideally I'd like to understand what's going on here and to get this working without the memcpy.
ntohl(*(unsigned int *)(msg + 7)); // Works
int offset = 7;
ntohl(*(unsigned int *)(msg + offset)); // Throws Exception (9) LoadStoreAlignmentCause
The main issue I'm experiencing is when I'm packing the size in python some hex values are being shorthanded. e.g struct.pack('!I', 54) == \x00\x00\x006
When this happens ntohl() seems to read an address it shouldn't and outputs 635. Also since I'm expecting four char the rest of the message is off by one index.
A few questions regarding this issue. What is the name of this shorthand hex syntax? Is there anyway to get python to not output this short hand? Or are there any suggestions on how to get this working on the ESP?

How to create a list of messages in a message with protobuf in python

I have a protobuf structure defined in the following way:
syntax = "proto3";
message Register {
string name = 1;
Access access = 2;
uint64 deafult_value = 3;
uint64 value = 4;
uint64 offset = 5;
int32 index = 6;
string description = 7;
int32 register_size = 8;
repeated Field fields_list = 9;
}
message Field {
string name = 1;
string regName = 2;
Access access = 3;
int32 offset = 4;
int32 length = 5;
string description = 6;
uint64 value = 7;
}
message Access {
bool read = 1;
bool write = 2;
}
now I want to create a new Register with a list of fields in python and i've tried the following thing:
proto_reg = DataStructs.Register()
proto_field = DataStructs.Field()
proto_reg.name ="test"
proto_reg.fields_list.extend(proto_field)
but im getting an exception:
TypeError: Value must be iterable
what am i doing wrong?

Well the answer was that i need to wrap proto_reg.fields_list.extend(proto_field) with []
so this change works:
proto_reg.fields_list.extend([proto_field])

Byte string deserialization taking into account zero values ​in the fields

I collect the message according to my protofile (listed below). Then I serialize it into a byte string using SerializeToString() method. Then i get the byte string message and deserialization into proto object using ParseFromString() method.
But if i fill in the some fields zero values and execute the above algorithm like this:
def test():
fdm = device_pb2.FromDeviceMessage()
fdm.deveui = bytes.fromhex('1122334455667788')
fdm.fcntup = 0
fdm.battery = 3.5999999046325684
fdm.mode = 0
fdm.event = 1
port = fdm.data.add()
port.port = 1 #device_pb2.PortData.Name(0)
port.value = 0
c = fdm.SerializeToString()
return c
def parse_test(data):
print(data)
res = device_pb2.FromDeviceMessage()
res.ParseFromString(data)
return res
print(parse_test(test()))
, then python console will show me:
deveui: "\021\"3DUfw\210"
battery: 3.5999999046325684
event: PERIOD_EVENT
data {
port: VIBR2
}
without fields values are zero.
But i want to see:
deveui: "\021\"3DUfw\210"
fcntup: 0
battery: 3.5999999046325684
mode: BOUNDARY
event: PERIOD_EVENT
data {
port: VIBR2
value: 0
}
Why is it happening, and if it's fixed how can i fix it?
=============Proto_File================
message FromDeviceMessage{
bytes deveui = 1;
uint32 ts = 2;
int32 fcntup = 3;
float battery = 4;
int32 period = 5;
Mode mode = 6;
Event event = 7;
repeated PortData data = 8;
}
message PortData{
DevicePort port = 1;
int32 value = 2;
}
enum Mode{
BOUNDARY = 0;
PERIOD = 1;
BOUNDARY_PERIOD = 2;
}
enum Event{
BOUNDARY_EVENT = 0;
PERIOD_EVENT = 1;
}
enum DevicePort{
VIBR1 = 0;
VIBR2 = 1;
TEMPL2 = 3;
}

So, i think i guessed the reason. In case enum type ( DevicePort, Event, Mode): the default value is the first defined enum value, which must be 0. So i will set up 1 value to see required fields. In other cases: the fields with zero values are not displayed to reduce memory size of package. But if i turn to the required field using this way: res.data[0].value (in def parse_test(data)) it will show me 0, if i set value 0 in field value, for example. And this rule works in all cases.

Difficulty with sending sensor data

I try to send data over an arduino Ethernet shield to client (python on PC)
the problem that i had is when i read like example the pin A0 in arduino i get 1023 but when i send this value to python i get 49152...
arduino code
#include <SPI.h>
#include <Ethernet.h>
#include <EthernetUdp.h>
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; // Enter a MAC address for your controller below.
IPAddress ip(192,168,0,101);
IPAddress gateway(192,168,0,254);
IPAddress subnet(255,255,255,0);
unsigned int UDPport = 5000;// local port to listen for UDP packets
IPAddress UDPServer(192,168,0,100); // destination device server
const int UDP_PACKET_SIZE= 48;
byte packetBuffer[ UDP_PACKET_SIZE]; //buffer to hold incoming and outgoing packets
unsigned int noChange = 0;
int UDPCount = 0;
EthernetUDP Udp;
unsigned long currentTime;
unsigned long secondTime;
unsigned long msPerSecond = 100UL;
float temperature;
float vitesse;
float charge;
float current;
void setup()
{
Serial.begin(9600);
Ethernet.begin(mac,ip,gateway,gateway,subnet);
Udp.begin(UDPport);
delay(1500);
currentTime=millis();
secondTime = currentTime;
}
void loop()
{
currentTime = millis();
getUDPpacket();
if(currentTime - secondTime > msPerSecond) {
temperature = analogRead(0); //read analog input on pin A0
vitesse = analogRead(1); //read analog input on pin A1
charge = analogRead(2); //read analog input on pin A2
current = analogRead(3); //read analog input on pin A3
Serial.println(temperature);
sendUDPpacket(UDPServer); // send an NTP packet to a time server
secondTime += msPerSecond;
}
}
unsigned int udpCount = 0;
unsigned long sendUDPpacket(IPAddress& address)
{
udpCount++;
memset(packetBuffer, 0, UDP_PACKET_SIZE); sprintf((char*)packetBuffer,"%u,%u,%u,%u",temperature,vitesse,charge,current);
Udp.beginPacket(address, UDPport);
Udp.write(packetBuffer,UDP_PACKET_SIZE);
Udp.endPacket();
}
void getUDPpacket() {
if ( Udp.parsePacket() ) {
if(Udp.remoteIP() == UDPServer) {
Serial.print(F("UDP IP OK "));
}
else {
Serial.println(F("UDP IP Bad"));
return;
}
if(Udp.remotePort() == UDPport) {
Serial.println(F("Port OK"));
}
else {
Serial.println(F("Port Bad"));
return;
}
Udp.read(packetBuffer,UDP_PACKET_SIZE); // read the packet into the buffer
Serial.print(F("Received: "));
Serial.println((char*)packetBuffer);
}
}
the python code
import socket
import time
UDP_IP = "192.168.0.100"
UDP_PORT = 5000
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(48)
print data
time.sleep(5)
i think the problem is in this line
sprintf((char*)packetBuffer,"%u,%u,%u,%u",temperature,vitesse,charge,current);
but i dont know what to do

You should enable all warnings (eg in gcc use -Wall) when you compile your C code. That way, it would warn you when you try to print floats with the %u unsigned integer format specifier, as that leads to undefined behaviour.
You should either a) convert temperature, vitesse, charge, and currentto unsigned integer before passing them to sprintf(), or b) change the format specifier to something like %f so that sprintf() knows to expect floating-point data.
Also you should include <stdio.h> so that your program has a prototype for sprintf(); I assume that those headers you have included contain prototypes & constants relevant to the Ethernet & Arduino port IO functions.

Bitcoin google protocol buffer error using python django? missing serialized_payment_details

Here is my problem guys,
I initialise the protofile as shown in the bitcoin for developers wiki shown here:
package payments;
option java_package = "org.bitcoin.protocols.payments";
option java_outer_classname = "Protos";
message Output {
optional uint64 amount = 1 [default = 0];
required bytes script = 2;
}
message PaymentDetails {
optional string network = 1 [default = "test"];
repeated Output outputs = 2;
required uint64 time = 3;
optional uint64 expires = 4;
optional string memo = 5;
optional string payment_url = 6;
optional bytes merchant_data = 7;
}
message PaymentRequest {
optional uint32 payment_details_version = 1 [default = 1];
optional string pki_type = 2 [default = "none"];
optional bytes pki_data = 3;
required bytes serialized_payment_details = 4;
optional bytes signature = 5;
}
message X509Certificates {
repeated bytes certificate = 1;
}
message Payment {
optional bytes merchant_data = 1;
repeated bytes transactions = 2;
repeated Output refund_to = 3;
optional string memo = 4;
}
message PaymentACK {
required Payment payment = 1;
optional string memo = 2;
}
throw this view into django which fetches the public key associated with a newly created address, hashes it into the correct format for a script, serializes the 'serialized_payment_details' field and returns a response object.
def paymentobject(request):
def addr_160(pub):
h3 = hashlib.sha256(unhexlify(pub))
return hashlib.new('ripemd160', h3.digest())
x = payments_pb2
btc_address = bc.getnewaddress()
pubkey_hash = bc.validateaddress(btc_address).pubkey
pubkey_hash160 = addr_160(pubkey_hash).hexdigest()
hex_script = "76" + "a9" + "14" + pubkey_hash160 + "88" + "ac"
serialized_script = hex_script.decode("hex")
xpd = x.PaymentDetails()
xpd.time = int(time())
xpd.outputs.add(amount = 0, script = serialized_script)
xpr = x.PaymentRequest()
xpr.serialized_payment_details = xpd.SerializeToString()
return HttpResponse(xpr.SerializeToString(), content_type="application/octet-stream")
When I point my bitcoin v0.9 client at URI
bitcoin:?r=http://127.0.0.1:8000/paymentobject
I am met with an error
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse message of type "payments.PaymentRequest" because it is missing required fields: serialized_payment_details
But it isn't missing the details field is it?
Any help much appreciated, thanks :)

The answer was that (at the time of writing) you cannot specify zero as the Output.amount. The bitcoin-qt 0.9 client considers it dust and does not allow the transaction to proceed.
More info here.