We Keep Coding

Lua AES Encrypting with \xhh encoding

in python I implementing AES Encrypting
that encrypt string or binary string with AES-128 and encode the output to regular hex literals like \xhh I want to implementing this function with lua
class AESCipher(object):
def __init__(self, key):
self.bs = 16
self.key = key
def encrypt(self, raw, use_base64=True):
if Crypto:
raw = self._pad(raw)
cipher = AES.new(self.key, mode=AES.MODE_ECB)
crypted_text = cipher.encrypt(raw)
else:
_ = self._pad(raw)
cipher = pyaes.blockfeeder.Encrypter(
pyaes.AESModeOfOperationECB(self.key)) # no IV, auto pads to 16
crypted_text = cipher.feed(raw)
crypted_text += cipher.feed() # flush final block
if use_base64:
return base64.b64encode(crypted_text)
else:
return crypted_text
def decrypt(self, enc, use_base64=True):
if use_base64:
enc = base64.b64decode(enc)
if Crypto:
cipher = AES.new(self.key, AES.MODE_ECB)
raw = cipher.decrypt(enc)
return self._unpad(raw).decode('utf-8')
else:
cipher = pyaes.blockfeeder.Decrypter(
pyaes.AESModeOfOperationECB(self.key)) # no IV, auto pads to 16
plain_text = cipher.feed(enc)
plain_text += cipher.feed() # flush final block
return plain_text
def _pad(self, s):
padnum = self.bs - len(s) % self.bs
return s + padnum * chr(padnum).encode()
#staticmethod
def _unpad(s):
return s[:-ord(s[len(s)-1:])]
localkey = "30a91c993c721aa9"
localkey = localkey.encode('latin1')
print(localkey)
test_cipher = AESCipher(localkey)
payload = b'{"gwId":"315716865002915c2717","devId":"315716865002915c2717","uid":"315716865002915c2717","t":"1633628358"}'
#payload = payload.encode("utf-8")
encrypted = test_cipher.encrypt(payload,False)
print(encrypted)
The Output is:
b'30a91c993c721aa9'
b'\xe15\xf8\xc3\xc1)eW\x19\x0b/\x8d\xaf\xa8\xe0\xb6\xd4\xb7\xf7\xcf\xe4\x02\xa4\xab\xfft\x00]B\xea\xf5\r\xcbI\xef\xf9\xd8\xd6v\x02\xa6&\x0e\xe7\x19\x14\xde\xa3\xd0\xc4\xfb\xcb\xfa\x8c\xb8\x9fd\xa6v#\xdf\x03\xc7\xb8\xe5fe\xaf_\xdc\x931\xff\xfc\xfc\x85\x0b3\x92w\xe7k\x1c~}\xb2"\xe9\xc1\xe0\x9a\x12\xb3\xcak\x8en<t\xdd\x12n9Q\x8eX!\xaba2\x16\x9c'
How can i implemneting this encrypting function in lua with this encoding
i tried some ways and many libraries but the output encoding is not like above
what can i do?

How to decrypt an AES/CBC encrypted string in Kotlin?

I have this Python method on the server to encrypt a string into bytes (AES/CBC).
class AESCipher(object, key):
def __init__(self, key):
self.bs = AES.block_size
self.key = hashlib.sha256(key.encode()).digest()
def encrypt(self, raw):
raw = self._pad(raw)
iv = Random.new().read(AES.block_size)
cipher = AES.new(self.key, AES.MODE_CBC, iv)
return base64.b64encode(iv + cipher.encrypt(raw.encode()))
def decrypt(self, enc):
enc = base64.b64decode(enc)
iv = enc[:AES.block_size]
cipher = AES.new(self.key, AES.MODE_CBC, iv)
return self._unpad(cipher.decrypt(enc[AES.block_size:])).decode('utf-8')
def _pad(self, s):
return s + (self.bs - len(s) % self.bs) * chr(self.bs - len(s) % self.bs)
The output of encrypt() is in bytes like this: b'PMgMOkBkciIKfWy/DfntVMyAcKtVsM8LwEwnTYE5IXY='
I would like to store this into database, and send it as string via API to Kotlin. And in there I would like to decrypt it via the same shared secret key.
In what format do I save the bytes above into database?
Once arrived in Kotlin client, how do I convert that string into ByteArray?
My theory is that I have to store the bytes as base64 string in the database.
And on the other side I have to decode the string as base64 into bytes. Is this approach correct? Will the encryption/decryption work like this end-to-end with the code below?
fun decrypt(context:Context, dataToDecrypt: ByteArray): ByteArray {
val cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING")
val ivSpec = IvParameterSpec(getSavedInitializationVector(context))
cipher.init(Cipher.DECRYPT_MODE, getSavedSecretKey(context), ivSpec)
val cipherText = cipher.doFinal(dataToDecrypt)
val sb = StringBuilder()
for (b in cipherText) {
sb.append(b.toChar())
}
return cipherText
}
fun getSavedSecretKey(context: Context): SecretKey {
val sharedPref = PreferenceManager.getDefaultSharedPreferences(context)
val strSecretKey = sharedPref.getString("secret_key", "")
val bytes = android.util.Base64.decode(strSecretKey, android.util.Base64.DEFAULT)
val ois = ObjectInputStream(ByteArrayInputStream(bytes))
val secretKey = ois.readObject() as SecretKey
return secretKey
}
fun getSavedInitializationVector(context: Context) : ByteArray {
val sharedPref = PreferenceManager.getDefaultSharedPreferences(context)
val strInitializationVector = sharedPref.getString("initialization_vector", "")
val bytes = android.util.Base64.decode(strInitializationVector, android.util.Base64.DEFAULT)
val ois = ObjectInputStream(ByteArrayInputStream(bytes))
val initializationVector = ois.readObject() as ByteArray
return initializationVector
}
UPDATE
I have tried to remove the Base64 to remove the memory overhead as suggested.
Python:
def encrypt(self, raw):
raw = self._pad(raw)
iv = Random.new().read(AES.block_size)
cipher = AES.new(self.key, AES.MODE_CBC, iv)
return iv + cipher.encrypt(raw.encode())
So this is no longer possible.
enc = AESCipher('abc').encrypt("myLife")
value_to_save_in_db = enc.decode("utf8")
So I need to find a way to store the byte array directly in the database. I think I should be able to do this as blob. But some challenges remain as how to send the bytearray as part of JSON over the API to the android device. I think I have to convert it to Base64 string again. Not sure if I have gained anything in that case...

The following Kotlin code:
val decrypted = decrypt("blEOKMQtUbNOzJbvEkL2gNhjF+qQ/ZK84f2ADu8xyUFme6uBhNYqvEherF/RRO9YRImz5Y04/ll+T07kqv+ExQ==");
println(decrypted);
decrypts a ciphertext of the Python code. Here decrypt() is:
fun decrypt(dataToDecryptB64 : String) : String {
// Base64 decode Python data
val dataToDecrypt = Base64.getDecoder().decode(dataToDecryptB64)
// Separate IV and Ciphertext
val ivBytes = ByteArray(16)
val cipherBytes = ByteArray(dataToDecrypt.size - ivBytes.size)
System.arraycopy(dataToDecrypt, 0, ivBytes, 0, ivBytes.size)
System.arraycopy(dataToDecrypt, ivBytes.size, cipherBytes, 0, cipherBytes.size)
// Derive key
val keyBytes = MessageDigest.getInstance("SHA256").digest("abc".toByteArray(Charsets.UTF_8))
// Decrypt
val cipher = Cipher.getInstance("AES/CBC/PKCS5PADDING")
cipher.init(Cipher.DECRYPT_MODE, SecretKeySpec(keyBytes, "AES"), IvParameterSpec(ivBytes))
val cipherText = cipher.doFinal(cipherBytes)
return String(cipherText, Charsets.ISO_8859_1)
}
For this, the ciphertext was generated using the posted Python class AESCipher as follows:
plaintext = 'The quick brown fox jumps over the lazy dog'
cipher = AESCipher('abc')
ciphertext = cipher.encrypt(plaintext)
print(ciphertext.decode('utf8')) # Base64 string, which can be stored e.g. in a DB
I applied the originally posted Python implementation that derives the key using SHA256. However, if the key is derived from a password, for security reasons not SHA256 but a reliable key derivation function, e.g. Argon2 or PBKDF2, should be used.
The Kotlin code first Base64 decodes the Python data and then separates IV and the actual ciphertext. Then, the key is derived by generating the SHA256 hash of the password. Finally the data is decrypted.
The current Python code Base64 encodes the data so that it can be stored as a string in the DB. Alternatively, the Python code could be modified so that no Base64 encoding is performed, and the raw data can be stored (which requires less memory, Base64 overhead: 33%).
Depending on the solution chosen, the Kotlin code may or may not need to Base64 decode the data.

Implementing AES/ECB/PKCS5 padding in Python

I am trying to implement a python program to encrypt a plain text using AES/ECB/PKCS5 padding. The output I am getting is slightly different from expected.
Python3 program:
import base64
from Crypto.Cipher import AES
def add_to_16(value):
while len(value) % 16 != 0:
value += '\0'
return str.encode (value) # returns bytes
# Encryption method
def encrypt(text):
# Secret key
key='92oifgGh893*cj%7'
# Text to be encrypted
# Initialize encryptor
aes = AES.new(key, AES.MODE_ECB)
# Aes encryption to be
encrypt_aes = aes.encrypt(add_to_16(text))
# Converted into a string with base64
encrypted_text = str(base64.encodebytes (encrypt_aes), encoding = 'utf-8')
print(encrypted_text)
return encrypted_text
if __name__ == '__main__':
text = '{ "Message": "hello this is a plain text" , "user":"john.doe", "Email":"john.doe#example.com}'
entrypted_text = encrypt(text)
The output for above program is:
oo8jwHQNQnBwVUsJ5piShFRM3PFFIfULwcoFOEQhPMTAvexSr6eE9aFLVQTpAKBFkGi8vNbtScvyexSxHBlwVapJ5Szz1JPR9q9cHHJYYMzGocln4TRPFQ6S3e8jjVud
where as when verified with 3rd party tools online, the results is:
oo8jwHQNQnBwVUsJ5piShFRM3PFFIfULwcoFOEQhPMTAvexSr6eE9aFLVQTpAKBFkGi8vNbtScvyexSxHBlwVapJ5Szz1JPR9q9cHHJYYMwnIIuNCUVn/IExpxebqXV1
Can someone please guide me where I am doing wrong?

I have framed the code with below for padding with PKCS5 and is working as expected.
block_size=16
pad = lambda s: s + (block_size - len(s) % block_size) * chr(block_size - len(s) % block_size)
and the encrypt method was re-written as below:
def encrypt(plainText,key):
aes = AES.new(key, AES.MODE_ECB)
encrypt_aes = aes.encrypt(pad(plainText))
encrypted_text = str(base64.encodebytes (encrypt_aes), encoding = 'utf-8')
return encrypted_text

Here is the complete code, in case if anyone is still looking.
tested against:
python3.6
python3.8
** used pycryptodome
encrypt_aes.py
import hashlib
from Crypto.Cipher import AES
import base64
class AES_pkcs5:
def __init__(self,key:str, mode:AES.MODE_ECB=AES.MODE_ECB,block_size:int=16):
self.key = self.setKey(key)
self.mode = mode
self.block_size = block_size
def pad(self,byte_array:bytearray):
"""
pkcs5 padding
"""
pad_len = self.block_size - len(byte_array) % self.block_size
return byte_array + (bytes([pad_len]) * pad_len)
# pkcs5 - unpadding
def unpad(self,byte_array:bytearray):
return byte_array[:-ord(byte_array[-1:])]
def setKey(self,key:str):
# convert to bytes
key = key.encode('utf-8')
# get the sha1 method - for hashing
sha1 = hashlib.sha1
# and use digest and take the last 16 bytes
key = sha1(key).digest()[:16]
# now zero pad - just incase
key = key.zfill(16)
return key
def encrypt(self,message:str)->str:
# convert to bytes
byte_array = message.encode("UTF-8")
# pad the message - with pkcs5 style
padded = self.pad(byte_array)
# new instance of AES with encoded key
cipher = AES.new(self.key, AES.MODE_ECB)
# now encrypt the padded bytes
encrypted = cipher.encrypt(padded)
# base64 encode and convert back to string
return base64.b64encode(encrypted).decode('utf-8')
def decrypt(self,message:str)->str:
# convert the message to bytes
byte_array = message.encode("utf-8")
# base64 decode
message = base64.b64decode(byte_array)
# AES instance with the - setKey()
cipher= AES.new(self.key, AES.MODE_ECB)
# decrypt and decode
decrypted = cipher.decrypt(message).decode('utf-8')
# unpad - with pkcs5 style and return
return self.unpad(decrypted)
if __name__ == '__main__':
# message to encrypt
message = 'hello world'
secret_key = "65715AC165715AC165715AC165715AC1"
AES_pkcs5_obj = AES_pkcs5(secret_key)
encrypted_message = AES_pkcs5_obj.encrypt(message)
print(encrypted_message)
decrypted_message = AES_pkcs5_obj.decrypt(encrypted_message)
print(decrypted_message)
Output:
>>> python encrypt_aes.py
>>> PDhIFEVqLrJiZQC90FPHiQ== # encrypted message
>>> hello world # and the decrypted one
I had tested many already available codes but none of them gave exact encryption as java ones. So, this is combined of all the found blogs and early written code compatible with python2

PKCS 5 (or 7) padding is not adding 0 bytes, but adding a c byteswith valuec(where1 <= c <= 16) if you're c` bytes short of a block length multiple.
So if you already have a multiple of 16, add a full 16 bytes of value 16, and if your last block is 'stop' (4 bytes), we add 12 bytes with value 0xc (12 in hex) to fill up the block. Etc.
This way the receiver (after decryption of the final block) can check the last byte c and check if the value is 1 <= c <= 16 (if not, reject the decryption) and then check that the last c bytes indeed are all that same value, and then remove them from the decryption. This way the receiver does not have to guess how many bytes of the last block were only padding or really part of the plain text. It's unambiguous doing it the PKCS way.
I'll leave the coding up to you.

You can use aes-pkcs5 package. I'm the author, it uses cryptography package instead of outdated pycrypto used in others answers and is compatible with Python 3.7+.
You can install via pip:
pip install aes-pkcs5
The same code that you posted using aes-pkcs5:
from aes_pkcs5.algorithms.aes_ecb_pkcs5_padding import AESECBPKCS5Padding
key = "92oifgGh893*cj%7"
cipher = AESECBPKCS5Padding(key, "b64")
text = '{ "Message": "hello this is a plain text" , "user":"john.doe", "Email":"john.doe#example.com}'
encrypted_text = cipher.encrypt(text)

You can use a random string separated by a null byte for the padding to add a bit of randomness sometimes.
import random
import string
from Crypto.Cipher import AES
NULL_BYTE = '\x00'
def random_string(size: int) -> str:
return ''.join([
random.choice(string.printable) for _ in range(size)
])
def encode_aes(value: str, key: str) -> bytes:
cipher = AES.new(key[:32], AES.MODE_ECB)
mod = len(value) % cipher.block_size
padding = (cipher.block_size - mod) % cipher.block_size
if padding > 0:
value += NULL_BYTE + random_string(padding - 1)
return cipher.encrypt(value)
def decode_aes(value: bytes, key: str) -> str:
cipher = AES.new(key[:32], AES.MODE_ECB)
decrypted = cipher.decrypt(value).decode('utf8')
return decrypted.rsplit(NULL_BYTE, 1)[0]

How is AES implemented in CryptoJS?

I want to verify that CryptoJS's AES is actually implementing AES. However, I am not able to verify it by encrypting a plain using CryptoJS and decrypting the cipher with a python script using PyCrypto.
Source for CryptoJS:
http://ramblings.mcpher.com/Home/excelquirks/gassnips/cryptogs
https://script.google.com/d/1IEkpeS8hsMSVLRdCMprij996zG6ek9UvGwcCJao_hlDMlgbWWvJpONrs/edit?usp=sharing
I gathered that in CryptoJS-AES,
it does 1 iteration of MD5 on the passphrase to generate the symmetric key.
In the cipher, the first 16bytes is the salt in the form of "Salted__[8byte random]".
The salt is used as the iv for the encryption/decryption.
done in CBC-Mode
The cipher is created with the below code in Google App Script.
var encryptedMessage = CryptoJS.AES.encrypt ('this is my message to be encrypted', '0123456789abcdef').toString();
//var decryptedMessage = CryptoJS.AES.decrypt (encryptedMessage, '0123456789abcdef').toString(CryptoJS.enc.Utf8);
I then tried to decrypt the cipher with the below python code. (adapted from some stackoverflow thread)
import base64
import hashlib
from Crypto import Random
from Crypto.Cipher import AES
class AESCipher(object):
def __init__(self, key):
self.bs = 16
self.key = hashlib.md5(key.encode()).digest()
self.key = key
def decrypt(self, enc):
enc = base64.b64decode(enc)
iv = enc[:16]
cipher = AES.new(self.key, AES.MODE_CBC, iv)
return self._unpad(cipher.decrypt(enc[16:])).decode("utf-8")
def _pad(self, s):
return s + (self.bs - len(s) % self.bs) * chr(self.bs - len(s) % self.bs)
#staticmethod
def _unpad(s):
print s
return s[:-ord(s[len(s)-1:])]
key = "0123456789abcdef"
aes = AESCipher(key)
cipher = "U2FsdGVkX1/CuZIB2c+sxY7rA0jw19EOKrL87cU1bzwU9BlIDP6VZvsSzXAQ+6v8S2GQOzf4VpEw7AK0LSFfKw=="
bbb = aes.decrypt(cipher)
print bbb #prints out gibberish
How can i verify that the AES is correct? Or what have i done wrong here?

CryptoGS is a direct port of CryptoJS and you can find the CryptoJS AES implementation at the following link:
https://github.com/brix/crypto-js/blob/develop/src/aes.js
CryptoGS' code is minified but the following gist breaks up some of the encryption algos into separate sections:
https://gist.github.com/DimuDesigns/5666e9c6df531b8d11099dfd291ed0e4
You can try running the AES algo in the gist through a code beautfier and comparing the two.

AES Decryption using Pycrypto Python Exception : 'builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte'

I am using following implementation of AES cipher :-
import hashlib
from Crypto.Cipher import AES
class AESCipher:
def __init__(self, key):
self.BS = 128
try:
self.key = hashlib.sha256(key.encode()).digest()[:self.BS]
except:
self.key = hashlib.sha256(key).digest()[:self.BS]
self.iv = Random.new().read(AES.block_size)
def encrypt(self, raw):
raw = self._pad(raw)
cipher = AES.new(self.key, AES.MODE_CBC, self.iv)
return base64.b64encode(self.iv + cipher.encrypt(raw))
def decrypt(self, enc):
enc = base64.b64decode(enc)
self.iv = enc[:AES.block_size]
cipher = AES.new(self.key, AES.MODE_CBC, self.iv)
return self._unpad(cipher.decrypt(enc[AES.block_size:])).decode()
def _pad(self, s):
return s + (self.BS - len(s) % self.BS) * chr(self.BS - len(s) % self.BS).encode()
#staticmethod
def _unpad(s):
return s[:-ord(s[len(s)-1:])]
Encryption for a binary encoded dictionary object causes no errors but when I try to decrypt the same encrypted object, following exception is raised :-
return self._unpad(cipher.decrypt(enc[AES.block_size:])).decode()
builtins.UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
I tried to use 'ISO' and 'latin' encoding and decoding functions. But after that the socket on other side of the LAN recognizes it as a string and not as a dictionary object.
My question :- What I am doing wrong here ?
Additional information :-
key = 'SHSJDS-DSJBSJDS-DSKNDS' # some thing following this pattern
bin_json_object = pickle.dumps(dict_object)
enc_json_object = AESenc(bin_json_object, key)
def AESenc(self, data, key):
return AESCipher(key).encrypt(data)
def AESdec(self, data, key):
return AESCipher(key).decrypt(data)
For example If I use "ISO-8859-1" encoding in the above code :-
binary encoded representation of dictionary object :-
b'\x80\x03}q\x00(X\x02\x00\x00\x00idq\x01X$\x00\x00\x0096e09f6c-1e80-4cd1-9225-159e35bcacb4q\x02X\x0c\x00\x00\x00request_codeq\x03K\x01X\x0e\x00\x00\x00payload_lengthq\x04K!X\x0b\x00\x00\x00session_keyq\x05Nu.'
encrypted representation of binary encoded dictionary object :-
b'cZi+L4Wi51B5oDGQKlFb9bioxKH3TFRO1piECklafwTe6GYm/VeVjJaCDKiI+o6f6CcUnMvx+2EfEwcHCH/KDDeHTivIUou7WGVrd1P++HxfYNutY/aOn30Y/yiICvwWRHBn/3zU3xXvr/4XrtoVddM2cQEgXupIcC99TIxurrr8CCZd74ZnWj6QB8quCtHD'
But if I now try to decrypt the same on other node on same LAN via socket. I get following decrypted representation :-
}q(XidqX$96e09f6c-1e80-4cd1-9225-159e35bcacb4qX
request_codeqKXpayload_lengthqK!X
session_keyqNu.
which is completely different from original binary representation of the same dictionary object. And produces the following exception :-
data = pickle.loads(data)
builtins.TypeError: 'str' does not support the buffer interface

Finally after hours of debugging I came with a working code, but I am not able to understand, why this is working. Please if someone could explain this in comments.
Modified version AES cipher code :-
class AESCipher:
def __init__(self, key):
self.BS = AES.block_size
try:
self.key = hashlib.sha256(key.encode('ISO-8859-1')).digest()[:self.BS]
except:
self.key = hashlib.sha256(key).digest()[:self.BS]
self.iv = Random.new().read(AES.block_size)
def encrypt(self, raw):
raw = self._pad(raw)
cipher = AES.new(self.key, AES.MODE_CBC, self.iv)
return base64.b64encode(self.iv + cipher.encrypt(raw))
def decrypt(self, enc):
enc = base64.b64decode(enc)
self.iv = enc[:AES.block_size]
cipher = AES.new(self.key, AES.MODE_CBC, self.iv)
return self._unpad(cipher.decrypt(enc[AES.block_size:])).decode('ISO-8859-1')
def _pad(self, s):
return s + (self.BS - len(s) % self.BS) * chr(self.BS - len(s) % self.BS).encode('ISO-8859-1')
#staticmethod
def _unpad(s):
print('returning : ', s[:-ord(s[len(s)-1:])])
return s[:-ord(s[len(s)-1:])]
Now without modifying the AES encryption and decryption functions. I introduced a following variation in the code. Whenever another node receives a binary stream it first decrypts it with the AES decrypt function. But after decryption encoded dictionary object has to be encoded again with 'ISO-8859-1' as shown below :-
dict_object = self.AESdecryption(binary_stream, self.session_key)
dict = pickle.loads(dict_object.encode('ISO-8859-1'))
print(dict)
The above produces correct dictionary object. But what I don't understand is when a dictionary object was encrypted in 'ISO-8859-1' encoding, and and then decrypted on other node in 'ISO-8859-1' encoding, then why before passing it to the pickle.loads() I have to encode it again to get the original dictionary object. Please if someone could explain why it is happening ?