We Keep Coding

Errors using python's ECDSA lib to sign AWS messages

I'm signing my messages using my code below:
def sign_msg_hash(self, msg_hash: HexBytes):
signature = self._kms_client.sign(
KeyId=self._key_id,
Message=msg_hash,
MessageType="DIGEST",
SigningAlgorithm="ECDSA_SHA_256",
)
act_signature = signature["Signature"]
return (act_signature)
However, when trying to do the following:
sign = kms.sign_msg_hash(transaction)
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
)
I get the following error:
ecdsa.util.MalformedSignature: Invalid length of signature, expected 64 bytes long, provided string is 72 bytes long
Now, when trying to use sign[:64] instead, it sometimes works but at other times gives me the following error:
raise SquareRootError("%d has no square root modulo %d" % (a, p))
ecdsa.numbertheory.SquareRootError: 6631794589973073742270549970789085262483305971731159368608959895351281521222 has no square root modulo 115792089237316195423570985008687907853269984665640564039457584007908834671663
I'm not sure if this has anything to do with the encoding of the signature from the KMS or not.

So after some digging, it turned it out it was a matter of encodings, as the kms sign function returns the signature in DER format.
Hence, what I did was the following:
I took the signature and split it, and got the r and s values as follows:
decoding=ecdsa.util.sigdecode_der(sign,int("fffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141",16))
I then concatenated the result (note that r and s should be in big-endian order):
new_sign = decoding[0].to_bytes(32, byteorder= 'big') + decoding[1].to_bytes(32, byteorder= 'big')
I then used the new sign:
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
new_sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
)
Another thing I encountered was with kms_client.get_public_key()
First call the method:
kms_pub_key_bytes = kms_client.get_public_key(KeyId=key_id)["PublicKey"]
Then get the last 128 chars which are essentially the r and s values
kms_pre_sha3 = kms_pub_key_bytes.hex()[-128:]
Then do what you want with the public key.
Related blog post on Medium

How can I find the big endian key in a message?

I am trying to read a binary message from an ESP32 using a broker; i wrote a phyton script where I subscribe the topic. the message that i actually receive is:
b'\x00\x00\x00?'
this is a float binary little endian message but I don't the key to decode it. Is there a way to find the decode key based on this data?
This is my python code:
import paho.mqtt.client as mqtt
def on_connect1(client1, userdata1, flags1, rc1):
client1.subscribe("ESP32DevKit123/mytopic")
def on_message1(client1, userdata1, msg1):
print(msg1.topic+" "+ "TESTENZA: "+str(msg1.payload))
client1 = mqtt.Client()
client1.username_pw_set(username="myuser",password="mypassword")
client1.on_connect = on_connect1
client1.on_message = on_message1
client1.connect("linkclient", portnumber, 60)
def twosComplement_hex(hexval):
bits = 16 # Number of bits in a hexadecimal number format
on_message1 = int(hexval, bits)
if on_message1 & (1 << (bits-1)):
on_message1 -= 1 << bits
return on_message1
client1.loop_forever()
It also gives me an error in the line on_message1 -= 1 << bits; the error says: Expected intended block pylance. Any solutions?

The data you provided is b'\x00\x00\x00?' - I'm going to assume that this is 0000003f (please output hex with msg1.payload.hex()).
I'll also assume that by "float binary little endian" you mean a big endian floating point (IEE754) - note that this does not match up with the algorithm you are using (twos compliment). Plugging this input into an online tool indicates that the expected result ("Float - Big Endian (ABCD)") is 8.82818e-44 (it's worth checking with this tool; sometimes the encoding may not be what you think it is!).
Lets unpack this using python (see the struct docs for more information):
>>> from struct import unpack
>>> unpack('>f', b'\x00\x00\x00\x3f')[0]
8.828180325246348e-44
Notes:
The [0] is there because unpack returns an array (you can unpack more than one item from the input)
>f - the > means big-endian and the f float (standard size = 4 bytes)
The reason your original code gives the error "Expected intended block" is due to the lack of indentation in the line on_message1 -= 1 << bits (as it follows an if it needs to be indented). The algorithm does not appear relevant to your task (but there may be details I'm missing).

Python C# datatypes for Sockets

I am creating a socket server to connect and speak with a C# program over TCP. Currently I am trying to create a way to convert the hex sent over the TCP socket to specific variables (the variable type will be in the packet header, and yes I do know tcp is a stream not technically sending packets but I am designing it like this). Currently I have all of the C# integral numeric types converting to and from bytesarray/integers correctly via the code below (All of the different types are the same with a couple edits to fit the c# type)
## SBYTE Type Class definition
## C#/Unity "sbyte" can be from -128 to 127
##
## Usage:
##
## Constructor
## variable = sbyte(integer)
## variable = sbyte(bytearray)
##
## Variables
## sbyte.integer (Returns integer representation)
## sbyte.bytes (Returns bytearray representation)
class sbyte:
def __init__(self, input):
if type(input) == type(int()):
self.integer = input
self.bytes = self.__toBytes(input)
elif type(input) == type(bytearray()):
self.bytes = input
self.integer = self.__toInt(input)
else:
raise TypeError(f"sbyte constructor can take integer or bytearray type not {type(input)}")
## Return Integer from Bytes Array
def __toInt(self, byteArray):
## Check that there is only 1 byte
if len(byteArray) != 1:
raise OverflowError(f"sbyte.__toInt length can only be 1 byte not {len(byteArray)} bytes")
## Return signed integer
return int.from_bytes(byteArray, byteorder='little', signed=True)
## Return Bytes Array from Integer
def __toBytes(self, integer):
## Check that the passed integer is not larger than 128 and is not smaller than -128
if integer > 127 or integer < -128:
raise ValueError(f"sbyte.__toBytes can only take an integer less than or equal to 127, and greater than or equal to -128, not \"{integer}\"")
## Convert the passed integer to Bytes
return integer.to_bytes(1, byteorder='little', signed=True)
This is working for all the types I currently implemented, but I do wonder if there is a better way to handle this? Such as using ctype's or some other python library. Since this will be a socket server with potentially many connections handling this as fast as possible is best. Or if there is anything else you see that I can improve I would love to know.

If all you want is an integer value from a byte array, simply index the byte array:
>>> b = bytearray.fromhex('1E')
>>> b[0]
30

After testing the differenced between from_bytes, struct.unpack, and numpy.frombuffer with the following code:
setup1 = """\
byteArray = bytearray.fromhex('1E')
"""
setup2 = """\
import struct
byteArray = bytearray.fromhex('1E')
"""
setup3 = """\
import numpy as np
type = np.dtype(np.byte)
byteArray = bytearray.fromhex('1E')
"""
stmt1 = "int.from_bytes(byteArray, byteorder='little', signed=True)"
stmt2 = "struct.unpack('b', byteArray)"
stmt3 = "np.frombuffer(byteArray, type)"
print(f"First statement execution time = {timeit.timeit(stmt=stmt1, setup=setup1, number=10**8)}")
print(f"Second statement execution time = {timeit.timeit(stmt=stmt2, setup=setup2, number=10**8)}")
print(f"Third statement execution time = {timeit.timeit(stmt=stmt3, setup=setup3, number=10**8)}")
Results:
First statement execution time = 14.456886599999999
Second statement execution time = 6.671141799999999
Third statement execution time = 21.8327342
From the initial results it looks like struct is the fastest way to accomplish this. Unless there are other libraries I am missing.
EDIT:
Per AKX's suggestion I added the following test for signed byte:
stmt4 = """\
if byteArray[0] <=127:
byteArray[0]
else:
byteArray[0]-127
"""
and got the following execution time:
Fourth statement execution time = 4.581732600000002
Going this path is the fastest although slightly over just using structs. I will have to test with each type for the fastest way to cast the bytes and vice versa but this question gave me 4 different ways to test each one now. Thanks!

Python short url generator

I'm using this code to generate short url. http://code.activestate.com/recipes/576918/
The idea is to encode an integer id using base62 and function enbase just works fine.
class UrlEncoder(object):
...
def enbase(self, x, min_length=0):
result = self._enbase(x)
padding = self.alphabet[0] * (min_length - len(result))
return '%s%s' % (padding, result)
But I don't quite understand what this code is for:
class UrlEncoder(object):
...
def encode_url(self, n, min_length=0):
return self.enbase(self.encode(n), min_length)
def decode_url(self, n):
return self.decode(self.debase(n))
def encode(self, n):
return (n & ~self.mask) | self._encode(n & self.mask)
Why encode then enbase? What does that bitwise operation do?
Could someone shed some light on me? Thanks.

Looking at the whole code: The net effect of encode() is to apply _encode() to the least-significant self.block_size-many bits of the value. _encode() appears to reverse those bits. It seems to be just a bit of additional scrambling. The documentation below the code explains why they are doing all these extra shuffles.

Base64 and non standard

I try to create a python client for bacula, but I have some problem with the authentication.
The algorithm is :
import hmac
import base64
import re
...
challenge = re.search("auth cram-md5 ()", data)
#exemple ''
passwd = 'b489c90f3ee5b3ca86365e1bae27186e'
hm = hmac.new(passwd, challenge).digest()
rep = base64.b64encode(hm).strp().rstrip('=')
#result with python : 9zKE3VzYQ1oIDTpBuMMowQ
#result with bacula client : 9z+E3V/YQ1oIDTpBu8MowB'
There's a way more simple than port the bacula's implemenation of base 64?
int
bin_to_base64(char *buf, int buflen, char *bin, int binlen, int compatible)
{
uint32_t reg, save, mask;
int rem, i;
int j = 0;
reg = 0;
rem = 0;
buflen--; /* allow for storing EOS */
for (i=0; i >= (rem - 6);
if (j

To verify your CRAM-MD5 implementation, it is best to use some simple test vectors and check combinations of (challenge, password, username) inputs against the expected output.
Here's one example (from http://blog.susam.in/2009/02/auth-cram-md5.html):
import hmac
username = 'foo#susam.in'
passwd = 'drowssap'
encoded_challenge = 'PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg=='
challenge = encoded_challenge.decode('base64')
digest = hmac.new(passwd, challenge).hexdigest()
response = username + ' ' + digest
encoded_response = response.encode('base64')
print encoded_response
# Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy
That said, I've certainly found examples on the net where the response generated by the above code differs from the expected response stated on the relevant site, so I'm still not entirely clear as to what is happening in those cases.

I HAVE CRACKED THIS.
I ran into exactly the same problem you did, and have just spent about 4 hours identifying the problem, and reimplementing it.
The problem is the Bacula's base64 is BROKEN, AND WRONG!
There are two problems with it:
The first is that the incoming bytes are treated as signed, not unsigned. The effect of this is that, if a byte has the highest bit set (>127), then it is treated as a negative number; when it is combined with the "left over" bits from previous bytes are all set to (binary 1).
The second is that, after b64 has processed all the full 6-bit output blocks, there may be 0, 2 or 4 bits left over (depending on input block modulus 3). The standard Base64 way to handle this is to multiply the remaining bits, so they are the HIGHEST bits in the last 6-bit block, and process them - Bacula leaves them as the LOWEST bits.
Note that some versions of Bacula may accept both the "Bacula broken base64 encoding" and the standard ones, for incoming authentication; they seem to use the broken one for their authentication.
def bacula_broken_base64(binarystring):
b64_chars="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
remaining_bit_count=0
remaining_bits=0
output=""
for inputbyte in binarystring:
inputbyte=ord(inputbyte)
if inputbyte>127:
# REPRODUCING A BUG! set all the "remaining bits" to 1.
remaining_bits=(1 << remaining_bit_count) - 1
remaining_bits=(remaining_bits<<8)+inputbyte
remaining_bit_count+=8
while remaining_bit_count>=6:
# clean up:
remaining_bit_count-=6
new64=(remaining_bits>>remaining_bit_count) & 63 # 6 highest bits
output+=b64_chars[new64]
remaining_bits&=(1 << remaining_bit_count) - 1
if remaining_bit_count>0:
output+=b64_chars[remaining_bits]
return output
I realize it's been 6 years since you asked, but perhaps someone else will find this useful.