I am working on an mqtt project where I subscribe to a broker that returns data to me in bytes string. I have to convert that data into protobuf then in dict. Here is an example of the data I receive.
b'\n\x108cf9572000023509\x10\x03\x1a\x06uplink \xd4\xc9\xc6\xea\x9a/*\x10b827ebfffebce2d30\xbe\xff\xff\xff\xff\xff\xff\xff\xff\x01=\x00\x00\x18AE4\x13YDH\x05P\x01Z\x01C`\x8c\x06h\x08z \x02\x05e!\x08\x01\x00f\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xd4\x0f\x00\x82\x01"\n\x10b827ebfffebce2d3\x10\xbe\xff\xff\xff\xff\xff\xff\xff\xff\x01\x1d\x00\x00\x18A'
Structure of my .proto file
message RXInfoSimplified {
string ID = 1;
int32 RSSI = 2;
float SNR = 3;
}
message DeviceUplink {
string DevEUI = 1;
int64 ApplicationID = 2;
string MsgType = 3;
int64 Timestamp = 4;
string GatewayID = 5;
int32 RSSI = 6;
float SNR = 7;
float Frequency = 8;
int32 DataRate = 9;
bool ADR = 10;
string Class = 11;
uint32 FCnt = 12;
int32 FPort = 13;
bool Confirm = 14;
bytes Data = 15;
repeated RXInfoSimplified Gateways = 16;
}
I tried this in my callback:
m = sub_message.DeviceUplink()
def on_message(client, userdata, msg):
m.ParseFromString(msg.payload)
print(m)
I got :
DevEUI: "8cf9572000023509"
ApplicationID: 3
MsgType: "uplink"
Timestamp: 1622117284775
GatewayID: "b827ebfffebce2d3"
RSSI: -61
SNR: 10.25
Frequency: 868.1
DataRate: 5
ADR: true
Class: "C"
FCnt: 796
FPort: 8
Data: "\002\005\220!\023\004\000p\001\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\000\335\016\000"
Gateways {
ID: "b827ebfffebce2d3"
RSSI: -61
SNR: 10.25
}
how can i convert Data value in string ?
Since Data has a bytes type in the proto definition, it has a bytes type in Python 3+. There isn't any further information in the .proto file on how to interpret the field, as far as the Protocol Buffers library is concerned it's a raw byte string and that's it.
It's up to you to know what to do with the field value. What's supposed to be in the Data field? Perhaps a serialized message of some type that you know? In that case, just use ParseFromString again.
Python gRPC message does not serialize first field. I updated protos and cleaned everything many times but it is not fixed. You can see logs, settings_dict has stop field but after passing these fields to AgentSetting it is not seen in logs and server side. Also I tried to pass stop field manually but it is not seen as well. Annoying thing is that gRPC does not throw any exception it is accepting stop field but it is not sending to server also it is not seen in AgentSetting while printing but stop field is reachable like agent_setting.stop.
This is my proto file:
syntax = 'proto3';
import "base.proto";
message ConnectionCheckRequest {
string host_name = 1;
}
message RecordChunkRequest {
string host_name = 1;
string name = 2;
bytes chunk = 3;
uint32 chunk_index = 4;
uint32 chunk_number = 5;
}
message AgentSetting {
bool stop = 1;
bool connection = 2;
bool registered = 3;
string image_mode = 4;
uint32 sending_fail_limit = 5;
uint64 limit_of_old_records = 6;
string offline_records_exceed_policy = 7;
}
message AgentSettingRequest {
AgentSetting agent_setting = 1;
string host_name = 2;
}
message AgentSettingResponse {
AgentSetting agent_setting = 1;
}
message AgentRegistryRequest {
AgentSetting agent_setting = 1;
string host_name = 2;
}
service Chief {
rpc agent_update_setting(AgentSettingRequest) returns (AgentSettingResponse) {};
rpc agent_registry(AgentRegistryRequest) returns (Response) {};
rpc send (stream RecordChunkRequest) returns (Response) {};
rpc connection_check (ConnectionCheckRequest) returns (Response) {};
}
This is my code snippet:
def register():
try:
with insecure_channel(settings.server_address) as channel:
setting_dict = settings.dict()
logger.info(f'\nSetting Dict: {setting_dict}')
agent_setting = AgentSetting(**setting_dict)
logger.info(f'\nAgent Setting Instance: \n{agent_setting}')
response = ChiefStub(channel).agent_registry(
AgentRegistryRequest(
agent_setting=agent_setting,
host_name=settings.host_name
)
)
return response
except Exception as e:
logger.exception(f'Register Error: {str(e)}')
return Response(success=False, message="failure")
Logs:
|2020-05-05T18:33:56.931140+0300| |5480| |INFO| |reqs:register:28|
Setting Dict: {'stop': False, 'connection': True, 'registered': False, 'image_mode': 'RGB', 'sending_fail_limit': 3, 'limit_of_old_records': 5368709120, 'offline_records_exceed_policy': 'OVERWRITE'}
|2020-05-05T18:33:56.932137+0300| |5480| |INFO| |reqs:register:32|
Agent Setting Instance:
connection: true
image_mode: "RGB"
sending_fail_limit: 3
limit_of_old_records: 5368709120
offline_records_exceed_policy: "OVERWRITE"
In proto3, an unset value and a default value are considered equivalent.
So stop: false is considered equivalent to omitting stop entirely.
See Language Guide (proto3)
Note that for scalar message fields, once a message is parsed there's no way of telling whether a field was explicitly set to the default value (for example whether a boolean was set to false) or just not set at all: you should bear this in mind when defining your message types. For example, don't have a boolean that switches on some behaviour when set to false if you don't want that behaviour to also happen by default. Also note that if a scalar message field is set to its default, the value will not be serialized on the wire.
Note that this is different from proto2.
I have a protobuf structure defined in the following way:
syntax = "proto3";
message Register {
string name = 1;
Access access = 2;
uint64 deafult_value = 3;
uint64 value = 4;
uint64 offset = 5;
int32 index = 6;
string description = 7;
int32 register_size = 8;
repeated Field fields_list = 9;
}
message Field {
string name = 1;
string regName = 2;
Access access = 3;
int32 offset = 4;
int32 length = 5;
string description = 6;
uint64 value = 7;
}
message Access {
bool read = 1;
bool write = 2;
}
now I want to create a new Register with a list of fields in python and i've tried the following thing:
proto_reg = DataStructs.Register()
proto_field = DataStructs.Field()
proto_reg.name ="test"
proto_reg.fields_list.extend(proto_field)
but im getting an exception:
TypeError: Value must be iterable
what am i doing wrong?
Well the answer was that i need to wrap proto_reg.fields_list.extend(proto_field) with []
so this change works:
proto_reg.fields_list.extend([proto_field])
I was trying to convert a hash algorithm which is written on Python to node.js
The python code looks something as
import uuid
import hashlib
import struct
CLIENT_ID = uuid.UUID('c5f92e0d-e762-32cd-98cb-8c546c410dbe')
SECRET = uuid.UUID('2cf26ff5-bd06-3245-becf-4d5a3baa704f')
data = CLIENT_ID.bytes_le + SECRET.bytes_le + struct.pack("I", 2017) + struct.pack("I", 9) + struct.pack("I", 2)
token = str(uuid.UUID(bytes_le=hashlib.sha256(data).digest()[0:16]))
The token generated is 32d86f00-eb49-2739-e957-91513d2b9969
Here the date values struct.pack values are generated using datetime but for convenient I have hard coded here.
I tried to convert the same by looking at the python doc for the respective libraries and did so far as
let CLIENT_ID = new Buffer('c5f92e0d-e762-32cd-98cb-8c546c410dbe');
let SECRET = new Buffer('2cf26ff5-bd06-3245-becf-4d5a3baa704f');
let d = new Buffer(2);
let m = new Buffer(9);
let y = new Buffer(2017);
let data = CLIENT_ID+SECRET+y+m+d;
const uuidv4 = require('uuid/v4');
const hash = crypto.createHash('sha256');
let token = uuidv4({random: hash.update(data, 'utf8').digest().slice(0, 16)}, 0);
And the hash it generates is b7b82474-eab4-4295-8318-cc258577ff9b
So, basically I am miserably missing something for the nodejs part.
Could you please guide me on where what went wrong. Thanks for the help
There's a lot of missed parts actually it tuned out.
###node parts:
new Buffer('c5')
does not represent <Buffer c5>, but <Buffer 63 35>.
To write c5 you would need to use Buffer.from([0xc5]) or Buffer.from([197]) (dec).
new Buffer(2)
does not represent <Buffer 02>, it just allocates 2 bytes.
CLIENT_ID+SECRET+y+m+d
concatenation of buffers does not work that way.
Use array of buffers and Buffer.concat([buffers]) to concatenate buffers.
###uuid parts:
it turned out that uuid operates modified version of buffers (bytes_le part in python code)
#the most interesting part:
in the python version of uuid, if no version argument is passed to uuid.UUID(...), uuid would generate an ID without fixing bits
According to the RFC-4122 4.4 uuid should fix that bits.
uuid.py skips RFC-4122 4.4
node-uuid/v4.js fixes required bits
that way even with the same results for sha256 hashing, the results between python and node implementation still would differ
python: 32d86f00-eb49-2739-e957-91513d2b9969
node: 32d86f00-eb49-4739-a957-91513d2b9969
^ ^
So, I see here 2 options
to pass version to python uuid (only for the last uuid call uuid.UUID(bytes_le=..., version=4)), that way python would return 32d86f00-eb49-4739-a957-91513d2b9969
if there's no way to change source code in python project, I guess there's an option to fork uuid and remove two lines of code in node-uuid/v4.js?
##See node version of your code below:
const uuidv4 = require('uuid/v4');
const crypto = require('crypto');
const hash = crypto.createHash('sha256');
const client_id_hex_str = "c5f92e0d-e762-32cd-98cb-8c546c410dbe".replace(/-/g, "");
const secret_hex_str = "2cf26ff5-bd06-3245-becf-4d5a3baa704f".replace(/-/g, "");
let CLIENT_ID = Buffer.from(to_bytes_le(to_bytes(client_id_hex_str, null, 16, 'big')));
let SECRET = Buffer.from(to_bytes_le(to_bytes(secret_hex_str, null, 16, 'big')));
let d = Buffer.from(to_bytes(null, 2, 4));
let m = Buffer.from(to_bytes(null, 9, 4));
let y = Buffer.from(to_bytes(null, 2017, 4));
let data = Buffer.concat([CLIENT_ID, SECRET, y, m, d]);
let hashBytes = hash.update(data, 'utf8').digest().slice(0, 16);
hashBytes = [].slice.call(hashBytes, 0);
hashBytes = Buffer.from(to_bytes_le(hashBytes));
let token = uuidv4({random: hashBytes});
console.log(token);
// https://stackoverflow.com/questions/16022556/has-python-3-to-bytes-been-back-ported-to-python-2-7
function to_bytes(hexString, number, length, endianess) {
if (hexString == null && number == null) {
throw new Error("Missing hex string or number.");
}
if (!length || isNaN(length)) {
throw new Error("Missing or invalid bytes array length number.");
}
if (hexString && typeof hexString != "string") {
throw new Error("Invalid format for hex value.");
}
if (hexString == null) {
if (isNaN(number)) {
throw new Error("Invalid number.");
}
hexString = number.toString(16);
}
let byteArray = [];
if (hexString.length % 2 !== 0) {
hexString = '0' + hexString;
}
const bitsLength = length * 2
hexString = ("0".repeat(bitsLength) + hexString).slice(-1 * bitsLength);
for (let i = 0; i < hexString.length; i += 2) {
const byte = hexString[i] + hexString[i + 1];
byteArray.push(parseInt(byte, 16));
}
if (endianess !== "big") {
byteArray = byteArray.reverse();
}
return byteArray;
}
// https://github.com/python/cpython/blob/master/Lib/uuid.py#L258
function to_bytes_le(bytes) {
const p1 = bytes.slice(0, 4).reverse();
const p2 = bytes.slice(4, 6).reverse();
const p3 = bytes.slice(6, 8).reverse();
const p4 = bytes.slice(8);
const bytes_le = [].concat.apply([], [p1, p2, p3, p4]);
return bytes_le;
}
Do you want the hashing of the data to be the same as the Python code above?
If not, you can take a look at the the Sha256 module below in NodeJS
https://www.npmjs.com/package/sha256
Here is my problem guys,
I initialise the protofile as shown in the bitcoin for developers wiki shown here:
package payments;
option java_package = "org.bitcoin.protocols.payments";
option java_outer_classname = "Protos";
message Output {
optional uint64 amount = 1 [default = 0];
required bytes script = 2;
}
message PaymentDetails {
optional string network = 1 [default = "test"];
repeated Output outputs = 2;
required uint64 time = 3;
optional uint64 expires = 4;
optional string memo = 5;
optional string payment_url = 6;
optional bytes merchant_data = 7;
}
message PaymentRequest {
optional uint32 payment_details_version = 1 [default = 1];
optional string pki_type = 2 [default = "none"];
optional bytes pki_data = 3;
required bytes serialized_payment_details = 4;
optional bytes signature = 5;
}
message X509Certificates {
repeated bytes certificate = 1;
}
message Payment {
optional bytes merchant_data = 1;
repeated bytes transactions = 2;
repeated Output refund_to = 3;
optional string memo = 4;
}
message PaymentACK {
required Payment payment = 1;
optional string memo = 2;
}
throw this view into django which fetches the public key associated with a newly created address, hashes it into the correct format for a script, serializes the 'serialized_payment_details' field and returns a response object.
def paymentobject(request):
def addr_160(pub):
h3 = hashlib.sha256(unhexlify(pub))
return hashlib.new('ripemd160', h3.digest())
x = payments_pb2
btc_address = bc.getnewaddress()
pubkey_hash = bc.validateaddress(btc_address).pubkey
pubkey_hash160 = addr_160(pubkey_hash).hexdigest()
hex_script = "76" + "a9" + "14" + pubkey_hash160 + "88" + "ac"
serialized_script = hex_script.decode("hex")
xpd = x.PaymentDetails()
xpd.time = int(time())
xpd.outputs.add(amount = 0, script = serialized_script)
xpr = x.PaymentRequest()
xpr.serialized_payment_details = xpd.SerializeToString()
return HttpResponse(xpr.SerializeToString(), content_type="application/octet-stream")
When I point my bitcoin v0.9 client at URI
bitcoin:?r=http://127.0.0.1:8000/paymentobject
I am met with an error
[libprotobuf ERROR google/protobuf/message_lite.cc:123] Can't parse message of type "payments.PaymentRequest" because it is missing required fields: serialized_payment_details
But it isn't missing the details field is it?
Any help much appreciated, thanks :)
The answer was that (at the time of writing) you cannot specify zero as the Output.amount. The bitcoin-qt 0.9 client considers it dust and does not allow the transaction to proceed.
More info here.