We Keep Coding

Errors using python's ECDSA lib to sign AWS messages

I'm signing my messages using my code below:
def sign_msg_hash(self, msg_hash: HexBytes):
signature = self._kms_client.sign(
KeyId=self._key_id,
Message=msg_hash,
MessageType="DIGEST",
SigningAlgorithm="ECDSA_SHA_256",
)
act_signature = signature["Signature"]
return (act_signature)
However, when trying to do the following:
sign = kms.sign_msg_hash(transaction)
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
)
I get the following error:
ecdsa.util.MalformedSignature: Invalid length of signature, expected 64 bytes long, provided string is 72 bytes long
Now, when trying to use sign[:64] instead, it sometimes works but at other times gives me the following error:
raise SquareRootError("%d has no square root modulo %d" % (a, p))
ecdsa.numbertheory.SquareRootError: 6631794589973073742270549970789085262483305971731159368608959895351281521222 has no square root modulo 115792089237316195423570985008687907853269984665640564039457584007908834671663
I'm not sure if this has anything to do with the encoding of the signature from the KMS or not.

So after some digging, it turned it out it was a matter of encodings, as the kms sign function returns the signature in DER format.
Hence, what I did was the following:
I took the signature and split it, and got the r and s values as follows:
decoding=ecdsa.util.sigdecode_der(sign,int("fffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141",16))
I then concatenated the result (note that r and s should be in big-endian order):
new_sign = decoding[0].to_bytes(32, byteorder= 'big') + decoding[1].to_bytes(32, byteorder= 'big')
I then used the new sign:
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
new_sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
)
Another thing I encountered was with kms_client.get_public_key()
First call the method:
kms_pub_key_bytes = kms_client.get_public_key(KeyId=key_id)["PublicKey"]
Then get the last 128 chars which are essentially the r and s values
kms_pre_sha3 = kms_pub_key_bytes.hex()[-128:]
Then do what you want with the public key.
Related blog post on Medium

How can I find the big endian key in a message?

I am trying to read a binary message from an ESP32 using a broker; i wrote a phyton script where I subscribe the topic. the message that i actually receive is:
b'\x00\x00\x00?'
this is a float binary little endian message but I don't the key to decode it. Is there a way to find the decode key based on this data?
This is my python code:
import paho.mqtt.client as mqtt
def on_connect1(client1, userdata1, flags1, rc1):
client1.subscribe("ESP32DevKit123/mytopic")
def on_message1(client1, userdata1, msg1):
print(msg1.topic+" "+ "TESTENZA: "+str(msg1.payload))
client1 = mqtt.Client()
client1.username_pw_set(username="myuser",password="mypassword")
client1.on_connect = on_connect1
client1.on_message = on_message1
client1.connect("linkclient", portnumber, 60)
def twosComplement_hex(hexval):
bits = 16 # Number of bits in a hexadecimal number format
on_message1 = int(hexval, bits)
if on_message1 & (1 << (bits-1)):
on_message1 -= 1 << bits
return on_message1
client1.loop_forever()
It also gives me an error in the line on_message1 -= 1 << bits; the error says: Expected intended block pylance. Any solutions?

The data you provided is b'\x00\x00\x00?' - I'm going to assume that this is 0000003f (please output hex with msg1.payload.hex()).
I'll also assume that by "float binary little endian" you mean a big endian floating point (IEE754) - note that this does not match up with the algorithm you are using (twos compliment). Plugging this input into an online tool indicates that the expected result ("Float - Big Endian (ABCD)") is 8.82818e-44 (it's worth checking with this tool; sometimes the encoding may not be what you think it is!).
Lets unpack this using python (see the struct docs for more information):
>>> from struct import unpack
>>> unpack('>f', b'\x00\x00\x00\x3f')[0]
8.828180325246348e-44
Notes:
The [0] is there because unpack returns an array (you can unpack more than one item from the input)
>f - the > means big-endian and the f float (standard size = 4 bytes)
The reason your original code gives the error "Expected intended block" is due to the lack of indentation in the line on_message1 -= 1 << bits (as it follows an if it needs to be indented). The algorithm does not appear relevant to your task (but there may be details I'm missing).

What is the appropriate way to flatten or serialize data in Python so it only contains the data bytes byes?

I'm a heavy LabVIEW user who is just starting to learn Python. I work with industrial and aerospace equipment a lot and something I need to do very often is process some data, then export it over some communications protocol in binary. For example, let's say I have a packet that contains a struct/cluster/other-complex-data-element that has the following underlying data elements:
sync - unsigned 32-bit integer
time - 64-bit double
payload ID - 16-bit signed integer
source - 16-but signed integer
destination - 16-bit signed integer
payload length - 32 bit signed integer
data 1 - 8-bit unsigned integer
data 2 - 8-bit unsigned integer
data 3 - 8-bit unsigned integer
data 4 - 8-bit unsigned integer
data 4 - 64-bit double
data 5 - 32-bit single
data 6 - 16 bit unsigned integer
crc - 32-bit unsigned integer
(This frame should be 42 bytes long)
I call this a frame, where there is some header information, a payload, then a crc, I think that's a common term for what I'm creating. The data types, and their location in the byte stream is critical. Any extraneous or missing bytes breaks the data transfer protocol and the data cannot be tolerated.
My question is this:
How do you achieve this easily in Python? In LabVIEW (and probably other languages), there are good, built in functions and methods to clearly define the data types, then flatten them to a string of bytes that is very efficient. It seems that with picking, there are things going on that I don't understand.
In my example code, I have a simple function to get some memory information, then serialize it. I would expect the integer version to have 88 bytes and the float version to have 172 bytes, but I get 87 and 115 respectively. Here is the code, thanks for your help!
import psutil
import time
import pickle
def getMemoryInfo():
while True:
virtual_memory = psutil.virtual_memory()
swap_memory = psutil.swap_memory()
memoryInfo = list(virtual_memory+swap_memory)
# memoryInfo = [float(x) for x in memoryInfo]
time.sleep(1.000)
print(memoryInfo)
string = pickle.dumps(memoryInfo)
print(string)
print(len(memoryInfo))
print(len(string))
getMemoryInfo()

The struct module worked for me. It was a little more tedious than I would have hoped, but it worked just fine. Here is the code I ended up using:
def build_frame(payload, payload_class, payload_id, source, destination):
# form frame header and payload
frame = {"sync": int(0x64617665),
"absolute_time": time.time(),
"relative_time": time.monotonic(),
"source": int(source),
"destination": int(destination),
"counter": 0,
"payload_class": int(payload_class),
"payload_id": int(payload_id),
"payload_length": int(len(payload)),
"payload": payload}
# form bytearray to crc
sync = (struct.pack('<I', frame['sync']))
absolute_time = (struct.pack('<d', frame['absolute_time']))
relative_time = (struct.pack('<d', frame['relative_time']))
source = (struct.pack('i', frame['source']))
destination = (struct.pack('i', frame['destination']))
counter = (struct.pack('I', frame['counter']))
payload_class = (struct.pack('i', frame['payload_class']))
payload_id = (struct.pack('i', frame['payload_id']))
payload_length = (struct.pack('i', frame['payload_length']))
payload_bytes = frame['payload']
crc_bytes = sync + absolute_time + relative_time + source + destination + counter + payload_class + payload_id + payload_length + payload_bytes
# crc bytes and add to frame
frame['crc'] = binascii.crc32(crc_bytes)
return frame

AES Encryption in Google App Engine (Python) and Decryption on iOS (Objective-C)

I'm trying to encrypt some data from python (Google App Engine) and then decrypt it on iOS.
There are several issues surrounding this based on the fact that there are so many options with AES Encryption and the different formats available in Python and Objective-C.
Because of the limited availability of the PyCrypto libraries on Google App Engine and the AES code on the iOS/Objective-C side requiring PKCS7Padding, I decided to use slowAES on the python side.
I'm also using a 16-bit key, CBC Mode, and PKCS7Padding.
Given that, this is my encrypt function and helper variables/function:
def str2nums(s):
return map(ord, s)
key = "hjt4mndfy234n5fs"
moo = aes.AESModeOfOperation()
iv = [12, 34, 96, 15] * 4
CBC_mode = moo.modeOfOperation['CBC']
nkey = str2nums(key)
def encrypt(plaintext):
funcName = inspect.stack()[0][3]
logging.debug("Enter " + funcName)
logging.debug("Input string: " + plaintext)
m, s, encData = moo.encrypt(plaintext, CBC_mode, nkey, len(nkey), iv)
fmt = len(encData)*'B'
dataAsStr = ""
for j in encData:
dataAsStr = dataAsStr + str(j) + ","
logging.debug("Output encrypted data:[" + dataAsStr + "]")
encoded = base64.b64encode(struct.pack(fmt, *encData))
logging.debug("Output encrypted string: " + encoded)
decoded = struct.unpack(fmt, base64.b64decode(encoded))
decrypted = moo.decrypt(decoded, s, CBC_mode, nkey, len(nkey), iv)
logging.debug("Output decrypted back: " + decrypted)
return encoded
Note that on the Python side, I am encrypting, packing, and then base64 encoding the data. Before returning this encrypted data, I'm also doing a test run at decrypting it just to show that in the logs and it does indeed work.
On the iOS side, I'm using the following AES NSData addition:
- (NSData *)AES128DecryptWithKey:(NSString *)key {
// 'key' should be 16 bytes for AES128, will be null-padded otherwise
char keyPtr[kCCKeySizeAES128+1]; // room for terminator (unused)
bzero(keyPtr, sizeof(keyPtr)); // fill with zeroes (for padding)
// fetch key data
[key getCString:keyPtr maxLength:sizeof(keyPtr) encoding:NSASCIIStringEncoding];
NSUInteger dataLength = [self length];
//See the doc: For block ciphers, the output size will always be less than or
//equal to the input size plus the size of one block.
//That's why we need to add the size of one block here
size_t bufferSize = dataLength + kCCBlockSizeAES128;
void *buffer = malloc(bufferSize);
size_t numBytesDecrypted = 0;
CCCryptorStatus cryptStatus = CCCrypt(kCCDecrypt, kCCAlgorithmAES128, kCCOptionPKCS7Padding,
keyPtr, kCCKeySizeAES128,
NULL /* initialization vector (optional) */,
[self bytes], dataLength, /* input */
buffer, bufferSize, /* output */
&numBytesDecrypted);
if (cryptStatus == kCCSuccess) {
//the returned NSData takes ownership of the buffer and will free it on deallocation
return [NSData dataWithBytesNoCopy:buffer length:numBytesDecrypted];
}
free(buffer); //free the buffer;
return nil;
}
And I'm making use of this when I pull down the base64/packed/encrypted data like so:
NSMutableString * result = [[NSMutableString alloc] initWithData:receivedData encoding:NSUTF8StringEncoding];
if (LOG) { NSLog(#"Base64 encoded / Encrypted string: %#", result); }
NSData* encryptedData = [NSData decodeBase64ForString:result]; // base64 decode
if (LOG) { NSLog(#"Encrypted string: %#", encryptedData); }
NSData* decryptedData = [encryptedData AES128DecryptWithKey:#"hjt4mndfy234n5fs"]; // AES Decrypt
The problem is, I can't seem to get the data to decrypt correctly on the client (iOS) side even though it decrypts just fine on the server (in python).
During the decryption, the cryptStatus always ends up : kCCAlignmentError. Which I don't quite understand.
I've also messed with AES 256 but I need a 32bit key I think and that doesn't seem to be an option for slowAES in CBC mode (at least according to the examples?).
Logging of the Server (Notice the actual unencrypted data is merely an empty set [] . That's a JSON representation of such to return to the client.
2012-01-04 08:48:13.962
Enter encrypt
D 2012-01-04 08:48:13.962
Input string: []
D 2012-01-04 08:48:13.967
Output encrypted data:[4,254,226,26,101,240,22,113,44,54,209,203,233,64,208,255,]
D 2012-01-04 08:48:13.967
Output encrypted string: BP7iGmXwFnEsNtHL6UDQ/w==
D 2012-01-04 08:48:13.971
Output decrypted back: []
Logging of the client (iOS):
2012-01-04 12:45:13.891 Base64 encoded / Encrypted string: BP7iGmXwFnEsNtHL6UDQ/w==
2012-01-04 12:45:13.892 Encrypted string: <04fee21a 65f01671 2c36d1cb e940d0ff>
2012-01-04 12:45:29.126 Decrypted string:
So my questions are:
What does it mean by an "Alignment Error"? Or what am I doing wrong that it doesn't want to decrypt on the client?
Do I need to worry about unpacking the data at all considering it looks like it matches up just fine? And if so, how would I go about an unpack() function in C or Objective-C?
Is there just plain a better way to do AES encryption between Google App Engine and iOS?
EDIT: I should also note that besides the answer of using the same Initialization Vector, I found a discrepancy between the python and iOS code for the padding. The encrypt/decrypt worked fine for small amounts of data but then I ran into failure to decrypt with larger ones. I fixed this by changing the iOS side to NOT use PKCS7Padding and put 0 there instead.

The IV needs to match on both ends.
The IV (initialization vector) is a string of bytes that's sent through the encryptor/decryptor to place its "memory" in a pseudo-random state before the "real" data is sent through. Since the encryption results depend on what's gone through before, this initialization makes it impossible (without knowing the IV) for a malicious 3rd party o know whether a given cleartext and key could have produced a given cypertext.
Ideally the IV is itself somehow variable, based, perhaps, on a serial number or some other text that's sent along with the cyphertext, or based on a counter that's synchronized between ends.
The presence of the IV (even if semi-predictable) significantly increases the difficulty of using a "known cleartext" attack. This is especially important for relatively short, frequent messages.

Base64 and non standard

I try to create a python client for bacula, but I have some problem with the authentication.
The algorithm is :
import hmac
import base64
import re
...
challenge = re.search("auth cram-md5 ()", data)
#exemple ''
passwd = 'b489c90f3ee5b3ca86365e1bae27186e'
hm = hmac.new(passwd, challenge).digest()
rep = base64.b64encode(hm).strp().rstrip('=')
#result with python : 9zKE3VzYQ1oIDTpBuMMowQ
#result with bacula client : 9z+E3V/YQ1oIDTpBu8MowB'
There's a way more simple than port the bacula's implemenation of base 64?
int
bin_to_base64(char *buf, int buflen, char *bin, int binlen, int compatible)
{
uint32_t reg, save, mask;
int rem, i;
int j = 0;
reg = 0;
rem = 0;
buflen--; /* allow for storing EOS */
for (i=0; i >= (rem - 6);
if (j

To verify your CRAM-MD5 implementation, it is best to use some simple test vectors and check combinations of (challenge, password, username) inputs against the expected output.
Here's one example (from http://blog.susam.in/2009/02/auth-cram-md5.html):
import hmac
username = 'foo#susam.in'
passwd = 'drowssap'
encoded_challenge = 'PDc0NTYuMTIzMzU5ODUzM0BzZGNsaW51eDIucmRzaW5kaWEuY29tPg=='
challenge = encoded_challenge.decode('base64')
digest = hmac.new(passwd, challenge).hexdigest()
response = username + ' ' + digest
encoded_response = response.encode('base64')
print encoded_response
# Zm9vQHN1c2FtLmluIDY2N2U5ZmE0NDcwZGZmM2RhOWQ2MjFmZTQwNjc2NzIy
That said, I've certainly found examples on the net where the response generated by the above code differs from the expected response stated on the relevant site, so I'm still not entirely clear as to what is happening in those cases.

I HAVE CRACKED THIS.
I ran into exactly the same problem you did, and have just spent about 4 hours identifying the problem, and reimplementing it.
The problem is the Bacula's base64 is BROKEN, AND WRONG!
There are two problems with it:
The first is that the incoming bytes are treated as signed, not unsigned. The effect of this is that, if a byte has the highest bit set (>127), then it is treated as a negative number; when it is combined with the "left over" bits from previous bytes are all set to (binary 1).
The second is that, after b64 has processed all the full 6-bit output blocks, there may be 0, 2 or 4 bits left over (depending on input block modulus 3). The standard Base64 way to handle this is to multiply the remaining bits, so they are the HIGHEST bits in the last 6-bit block, and process them - Bacula leaves them as the LOWEST bits.
Note that some versions of Bacula may accept both the "Bacula broken base64 encoding" and the standard ones, for incoming authentication; they seem to use the broken one for their authentication.
def bacula_broken_base64(binarystring):
b64_chars="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
remaining_bit_count=0
remaining_bits=0
output=""
for inputbyte in binarystring:
inputbyte=ord(inputbyte)
if inputbyte>127:
# REPRODUCING A BUG! set all the "remaining bits" to 1.
remaining_bits=(1 << remaining_bit_count) - 1
remaining_bits=(remaining_bits<<8)+inputbyte
remaining_bit_count+=8
while remaining_bit_count>=6:
# clean up:
remaining_bit_count-=6
new64=(remaining_bits>>remaining_bit_count) & 63 # 6 highest bits
output+=b64_chars[new64]
remaining_bits&=(1 << remaining_bit_count) - 1
if remaining_bit_count>0:
output+=b64_chars[remaining_bits]
return output
I realize it's been 6 years since you asked, but perhaps someone else will find this useful.