We Keep Coding

Errors using python's ECDSA lib to sign AWS messages

I'm signing my messages using my code below:
def sign_msg_hash(self, msg_hash: HexBytes):
signature = self._kms_client.sign(
KeyId=self._key_id,
Message=msg_hash,
MessageType="DIGEST",
SigningAlgorithm="ECDSA_SHA_256",
)
act_signature = signature["Signature"]
return (act_signature)
However, when trying to do the following:
sign = kms.sign_msg_hash(transaction)
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
)
I get the following error:
ecdsa.util.MalformedSignature: Invalid length of signature, expected 64 bytes long, provided string is 72 bytes long
Now, when trying to use sign[:64] instead, it sometimes works but at other times gives me the following error:
raise SquareRootError("%d has no square root modulo %d" % (a, p))
ecdsa.numbertheory.SquareRootError: 6631794589973073742270549970789085262483305971731159368608959895351281521222 has no square root modulo 115792089237316195423570985008687907853269984665640564039457584007908834671663
I'm not sure if this has anything to do with the encoding of the signature from the KMS or not.

So after some digging, it turned it out it was a matter of encodings, as the kms sign function returns the signature in DER format.
Hence, what I did was the following:
I took the signature and split it, and got the r and s values as follows:
decoding=ecdsa.util.sigdecode_der(sign,int("fffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141",16))
I then concatenated the result (note that r and s should be in big-endian order):
new_sign = decoding[0].to_bytes(32, byteorder= 'big') + decoding[1].to_bytes(32, byteorder= 'big')
I then used the new sign:
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
new_sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
)
Another thing I encountered was with kms_client.get_public_key()
First call the method:
kms_pub_key_bytes = kms_client.get_public_key(KeyId=key_id)["PublicKey"]
Then get the last 128 chars which are essentially the r and s values
kms_pre_sha3 = kms_pub_key_bytes.hex()[-128:]
Then do what you want with the public key.
Related blog post on Medium

What is the appropriate way to flatten or serialize data in Python so it only contains the data bytes byes?

I'm a heavy LabVIEW user who is just starting to learn Python. I work with industrial and aerospace equipment a lot and something I need to do very often is process some data, then export it over some communications protocol in binary. For example, let's say I have a packet that contains a struct/cluster/other-complex-data-element that has the following underlying data elements:
sync - unsigned 32-bit integer
time - 64-bit double
payload ID - 16-bit signed integer
source - 16-but signed integer
destination - 16-bit signed integer
payload length - 32 bit signed integer
data 1 - 8-bit unsigned integer
data 2 - 8-bit unsigned integer
data 3 - 8-bit unsigned integer
data 4 - 8-bit unsigned integer
data 4 - 64-bit double
data 5 - 32-bit single
data 6 - 16 bit unsigned integer
crc - 32-bit unsigned integer
(This frame should be 42 bytes long)
I call this a frame, where there is some header information, a payload, then a crc, I think that's a common term for what I'm creating. The data types, and their location in the byte stream is critical. Any extraneous or missing bytes breaks the data transfer protocol and the data cannot be tolerated.
My question is this:
How do you achieve this easily in Python? In LabVIEW (and probably other languages), there are good, built in functions and methods to clearly define the data types, then flatten them to a string of bytes that is very efficient. It seems that with picking, there are things going on that I don't understand.
In my example code, I have a simple function to get some memory information, then serialize it. I would expect the integer version to have 88 bytes and the float version to have 172 bytes, but I get 87 and 115 respectively. Here is the code, thanks for your help!
import psutil
import time
import pickle
def getMemoryInfo():
while True:
virtual_memory = psutil.virtual_memory()
swap_memory = psutil.swap_memory()
memoryInfo = list(virtual_memory+swap_memory)
# memoryInfo = [float(x) for x in memoryInfo]
time.sleep(1.000)
print(memoryInfo)
string = pickle.dumps(memoryInfo)
print(string)
print(len(memoryInfo))
print(len(string))
getMemoryInfo()

The struct module worked for me. It was a little more tedious than I would have hoped, but it worked just fine. Here is the code I ended up using:
def build_frame(payload, payload_class, payload_id, source, destination):
# form frame header and payload
frame = {"sync": int(0x64617665),
"absolute_time": time.time(),
"relative_time": time.monotonic(),
"source": int(source),
"destination": int(destination),
"counter": 0,
"payload_class": int(payload_class),
"payload_id": int(payload_id),
"payload_length": int(len(payload)),
"payload": payload}
# form bytearray to crc
sync = (struct.pack('<I', frame['sync']))
absolute_time = (struct.pack('<d', frame['absolute_time']))
relative_time = (struct.pack('<d', frame['relative_time']))
source = (struct.pack('i', frame['source']))
destination = (struct.pack('i', frame['destination']))
counter = (struct.pack('I', frame['counter']))
payload_class = (struct.pack('i', frame['payload_class']))
payload_id = (struct.pack('i', frame['payload_id']))
payload_length = (struct.pack('i', frame['payload_length']))
payload_bytes = frame['payload']
crc_bytes = sync + absolute_time + relative_time + source + destination + counter + payload_class + payload_id + payload_length + payload_bytes
# crc bytes and add to frame
frame['crc'] = binascii.crc32(crc_bytes)
return frame

Enconding temperature for exchange over bluetooth

I'm trying to understand how to encode some data for transfer over BLE (bluetooth low energy).
Specifically, I'm interested in this particular line:
https://github.com/micropython/micropython/blob/05f95682e7ddfb08c317e83826df9a1d636676f3/ports/nrf/examples/ubluepy_temp.py#L68
Which comes from the snippet:
temp = Temp.read()
temp = temp * 100
char_temp.write(bytearray([temp & 0xFF, temp >> 8]))
Before we even come to the why part, I need to understand the how. In this snippet, the temperature is read from a sensor as a float, in Celsius. Let's say "20.00" for now. We then multiply it by 100, and then comes the encoding part:
2000 & 0xFF -> 208
2000 >> 8 -> 7
So we're basically sending:
>>> bytearray([208, 7])
bytearray(b'\xd0\x07')
Is this correct? I would say so, I checked it with my own device and this seems to be the data that is being sent, and it also works, I can read the temperature sent from my BLE device.
What I don't understand is why all these bit manipulations are required. For example, I tried to just send bytearray([hex(20)]) but it doesn't work (when trying to read the temperature from my phone, the data couldn't be parsed/converted).
Could you explain the format of the data sent please?

As shown in the tests, the straight forward way to do the conversion in Python is with to_bytes and from_bytes functionality
Using your data it would be for the bytes to an int direction:
>>> int.from_bytes([208, 7], byteorder='little', signed=True)
2000
From int to bytes:
>>> int(2000).to_bytes(2, byteorder='little', signed=True)
b'\xd0\x07'
Or from the reading to bytes:
>>> int(20.00*100).to_bytes(2, byteorder='little', signed=True)
b'\xd0\x07'
Bluetooth data should be a list or bytearray in little endian format. Each integer in the list needs to represent an octet.
From the source you linked to I can see the characteristic UUID is 0x2A6E:
uuid_temp = UUID("0x2A6E") # Temperature characteristic
This is an official UUID so is described in "16-bit UUID Numbers Document" at https://www.bluetooth.com/specifications/assigned-numbers/
There is more detailed explanation in the GATT Specification Supplement document on the Bluetooth SIG website in section: "2: Values and represented values". In that document it also explains how to represent Temperature this way:

TLS MAC message verification

I'm developing a SSL de-cipher in python but I'm having some problems on HMAC verification:
I've extracted all keyring related material (client IV, MAC, Key and Server IV, MAC, key).
When I receive the first Application_Data message (0x17), I am able to decrypt it, but unable to verify message integrity.
On RFC 2246 (https://www.ietf.org/rfc/rfc2246.txt), tells:
The MAC is generated as:
HMAC_hash(MAC_write_secret, seq_num + TLSCompressed.type +
TLSCompressed.version + TLSCompressed.length +
TLSCompressed.fragment));
where "+" denotes concatenation.
seq_num
The sequence number for this record.
hash
The hashing algorithm specified by
SecurityParameters.mac_algorithm.
Taking this as an example:
Chosen cipher_suite is TLS_RSA_WITH_AES_256_CBC_SHA256
client_mac = "some random stuff"
message_type = 0x17
message_version = 0x0303
encrypted_message_length = 1184 (IV|Message|MAC|Offset)
decrypted_message_length = 1122 (removing IV, MAC and offset)
message = "some message of length 1122"
client_mac is extracted from keyring_material
message_type is 0x17, because as an Application_data message type, the correct value should be 0x17
message version is 0x0303 as it's TLS 1.2
message length is 1122, removing preceding IV, offset and MAC verification, message, gets a final length of 1122
seq_number is 1 as it's the first message
HMAC_SHA256 calculation, in python, is as follows:
import hashlib
import hmac
hmac.new(<client_mac>,label+message,hashlib.sha256).digest()
My question is, how do I calculate label?
As RFC mentions, "+" denotes concatenation, but concatenation of what
HEX values converted to string
"1" + "17" + "0303" + "462"
INT values converted to strings
"1" + "23" + "771" + "1122"
And other thing to mention, TLSCompressed.version means:
0x0303
771
"1.2"
"12"
"TLS 1.2"
In this maillist (http://www.ietf.org/mail-archive/web/tls/current/msg14357.html) I found a supposed clarification of MAC values,
MAC(MAC_write_key, seq_num +
TLSCipherText.type +
TLSCipherText.version +
length of ENC(content + padding + padding_length) +
IV +
ENC(content + padding + padding_length));
where the length is encoded as two bytes in the usual way.
but it makes no sense to me, because it's useless to re-encode decrypted values to check to compute MAC. And from last line "where length is encoded as two bytes in the usual way", does it means that I should use
struct.pack("!H",length)
Then remove "\x" and use this value? or should I encode this value in HEX and then concatenate it?
I'm a bit lost, because RFC are not clear about how values should be used.
I've been trying several combinations (even brute forcing), but none of them worked, I hope you can light my way.

Well, after diggin' a bit I've managed to solve the issue.
RFC 5246, in section 6.2.3.1 (https://www.rfc-editor.org/rfc/rfc5246#section-6.2.3.1)
The MAC is generated as:
MAC(MAC_write_key, seq_num +
TLSCompressed.type +
TLSCompressed.version +
TLSCompressed.length +
TLSCompressed.fragment);
where "+" denotes concatenation.
But it does not points the data size, either representation format (hex, string...).
The way every field must be represented is as follows:
seq_num:
Description: A int counter, starting in 0, which will be incremented every frame received or sended. For a TCP Session, two seq_numbers must be used, one for the server and other for the client, incrementing everytime each of them sends a frame.
Representation: This value must be represented as Unsigned Long Long with 8 bytes
Representation example:
struct.pack("!Q",seq_num)
TLSCompressed.type
Description: This field is extracted from TLS Record layer (the encrypted payload). For example, if it's an Application Data frame, we must use 0x17.
Representation: This value must be represented as Signed Char, with 2 bytes.
Representation example:
struct.pack("!b",TLSCompressed.type)
TLSCompressed.version
Description: This field is also extracted from TLS Record layer (the encrypted payload). For example, if the frame is transferred using TLS 1.2, we must use it's hex representation 0x0303.
Representation: This value must be represented as Unsigned Short, with 2 bytes.
Representation example:
struct.pack("!H",TLSCompressed.version)
TLSCompressed.length
Description: This field represents the actual length of the decrypted payload.
Representation: This value must be represented as Unsigned Short, with 2 bytes.
Representation example:
struct.pack("!H",TLSCompressed.length)
TLSCompressed.fragment
Description: This field **is the actual decrypted payload.
Representation: This value must be represented as a string
As a python example, the HMAC hashing will be as follows for our previous example:
hmac_digest = hmac.new(mac_secret,'',digestmod=hashlib.sha256)
hmac_digest.update(struct.pack('!QbHH',seq_num,TLSCompressed.type,TLSCompressed.version, len(decrypted)))
hmac_digest.update(decrypted)
hmac_digest.digest()

Base64 and non standard

I try to create a python client for bacula, but I have some problem with the authentication.
The algorithm is :
import hmac
import base64
import re
...
challenge = re.search("auth cram-md5 ()", data)
#exemple ''
passwd = 'b489c90f3ee5b3ca86365e1bae27186e'
hm = hmac.new(passwd, challenge).digest()
rep = base64.b64encode(hm).strp().rstrip('=')
#result with python : 9zKE3VzYQ1oIDTpBuMMowQ
#result with bacula client : 9z+E3V/YQ1oIDTpBu8MowB'
There's a way more simple than port the bacula's implemenation of base 64?
int
bin_to_base64(char *buf, int buflen, char *bin, int binlen, int compatible)
{
uint32_t reg, save, mask;
int rem, i;
int j = 0;
reg = 0;
rem = 0;
buflen--; /* allow for storing EOS */
for (i=0; i >= (rem - 6);
if (j

To verify your CRAM-MD5 implementation, it is best to use some simple test vectors and check combinations of (challenge, password, username) inputs against the expected output.
Here's one example (from http://blog.susam.in/2009/02/auth-cram-md5.html):
import hmac
username = 'foo#susam.in'
passwd = 'drowssap'
encoded_challenge = 'PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg=='
challenge = encoded_challenge.decode('base64')
digest = hmac.new(passwd, challenge).hexdigest()
response = username + ' ' + digest
encoded_response = response.encode('base64')
print encoded_response
# Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy
That said, I've certainly found examples on the net where the response generated by the above code differs from the expected response stated on the relevant site, so I'm still not entirely clear as to what is happening in those cases.

I HAVE CRACKED THIS.
I ran into exactly the same problem you did, and have just spent about 4 hours identifying the problem, and reimplementing it.
The problem is the Bacula's base64 is BROKEN, AND WRONG!
There are two problems with it:
The first is that the incoming bytes are treated as signed, not unsigned. The effect of this is that, if a byte has the highest bit set (>127), then it is treated as a negative number; when it is combined with the "left over" bits from previous bytes are all set to (binary 1).
The second is that, after b64 has processed all the full 6-bit output blocks, there may be 0, 2 or 4 bits left over (depending on input block modulus 3). The standard Base64 way to handle this is to multiply the remaining bits, so they are the HIGHEST bits in the last 6-bit block, and process them - Bacula leaves them as the LOWEST bits.
Note that some versions of Bacula may accept both the "Bacula broken base64 encoding" and the standard ones, for incoming authentication; they seem to use the broken one for their authentication.
def bacula_broken_base64(binarystring):
b64_chars="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
remaining_bit_count=0
remaining_bits=0
output=""
for inputbyte in binarystring:
inputbyte=ord(inputbyte)
if inputbyte>127:
# REPRODUCING A BUG! set all the "remaining bits" to 1.
remaining_bits=(1 << remaining_bit_count) - 1
remaining_bits=(remaining_bits<<8)+inputbyte
remaining_bit_count+=8
while remaining_bit_count>=6:
# clean up:
remaining_bit_count-=6
new64=(remaining_bits>>remaining_bit_count) & 63 # 6 highest bits
output+=b64_chars[new64]
remaining_bits&=(1 << remaining_bit_count) - 1
if remaining_bit_count>0:
output+=b64_chars[remaining_bits]
return output
I realize it's been 6 years since you asked, but perhaps someone else will find this useful.