I have data in my database that I need to encrypt. I will then download the database to csv files. I have a python program that can decrypt the specific columns in a csv file. The problem is that I don't get my data out from the python program.
sql function:
CREATE OR REPLACE FUNCTION AESEncrypt (data TEXT,pass TEXT)
RETURNS TEXT AS $crypted$
declare
crypted TEXT;
key BYTEA;
iv BYTEA;
BEGIN
key := digest(convert_to(pass, 'utf-8'), 'sha256');
iv := digest(convert_to(CONCAT(data , 'salt'), 'utf-8'), 'md5');
crypted := encode(encrypt_iv(convert_to(data, 'utf-8'), key, iv, 'aes'), 'base64');
RETURN crypted;
END;
$crypted$ LANGUAGE plpgsql;
python program:
import csv
import time
import base64
from hashlib import sha256, md5
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes
from cryptography.hazmat.backends import default_backend
password = 'Password'
inputFile = 'test.txt'
outputFile = 'out.txt'
delimiter = ';'
columns = [0]
backend = default_backend()
key = sha256(password.encode('utf-8')).digest()
iv = md5((password + 'salt').encode('utf-8')).digest()
cipher = Cipher(algorithms.AES(key), modes.CBC(iv), backend=backend)
def encrypt(input):
input = bytes(input, 'utf-8')
#Padding
length = 16 - (len(input) % 16)
input += bytes([length])*length
#Encrypt
encryptor = cipher.encryptor()
return base64.b64encode(encryptor.update(input) + encryptor.finalize()).decode("utf-8")
def decrypt(input):
input = base64.b64decode(input)
decryptor = cipher.decryptor()
data = decryptor.update(input) + decryptor.finalize()
data = data[:-data[-1]] #Remove padding
print(data)
return data.decode('utf-8')
def main():
start_time = time.time()
with open(inputFile, 'r') as csvfileIn:
with open(outputFile, 'w', newline='') as csvfileOut:
spamreader = csv.reader(csvfileIn, delimiter=delimiter)
spamwriter = csv.writer(csvfileOut, delimiter=delimiter)
firstRow = True
for row in spamreader:
if not firstRow:
for pos in columns:
row[pos] = decrypt(row[pos])
firstRow = False
spamwriter.writerow(row)
print("--- %s seconds ---" % (time.time() - start_time))
main()
If I encrypt the file with the encrypt function written in the python program then I get the correct result if i would decrypt it.
If I would call the sql funcion as AESEncrypt('data', 'Password') then it returns the base64 string Ojq6RKg7NgDx8YFdLzfVhQ==
But after decryption I get the empty string as result and not the string data. If I look at the print statment before the utf-8 decode step in the decryption function it prints out the following on the console b'', so it looks like it could be something wrong with the padding. If I would print before I remove the padding I get b'\x85\x90sz\x0cQS\x9bs\xeefvA\xc63-'. If I will encrypt a long sentence then I will actually see parts of the text in the byte outputs above.
Do anyone know what I have done wrong?
Related
I am encrypting a large (100GB+) file with Python using PyCryptodome using AES-256 in CBC mode.
Rather than read the entire file into memory and encrypt it in one fell swoop, I would like to read the input file a 'chunk' at a time and append to the output file with the results of encrypting each 'chunk.'
Regrettably, the documentation for PyCryptodome is lacking in that I can't find any examples of how to encrypt a long plaintext with multiple calls to encrypt(). All the examples use a short plaintext and encrypt the entire plaintext in a single call to encrypt().
I had assumed that if my input 'chunk' is a multiple of 16 bytes (the block size of AES in CBC mode) I wouldn't need to add padding to any 'chunk' but the last one. However, I wasn't able to get that to work. (I got padding errors while decrypting.)
I'm finding that in order to successfully decrypt the file, I need to add padding to every 'chunk' when encrypting, and decrypt in units of the input chunk size plus 16 bytes. This means the decrypting process needs to know the 'chunk size' used for encryption, which makes me believe that this is probably an incorrect implementation.
While I do have my encryption/decryption working as described, I wonder if this is the 'correct' way to do it. (I suspect it is not.) I've read inconsistent claims on whether or not every such 'chunk' needs padding. If not, I'd like some handholding to get Pycryptodome to encrypt and then decrypt a large plaintext across multiple calls to encrypt() and decrypt().
EDIT: This code throws a ValueError, "Padding is incorrect," when decrpyting the first 'chunk'.
def encrypt_file(infile, outfile, aeskey, iv):
cipher = AES.new(aeskey, AES.MODE_CBC, iv)
with open(infile, "rb") as fin:
with open(outfile, "wb") as fout:
while True:
data = fin.read(16 * 32)
if len(data) ==0:
break
insize = len(data)
if insize == (16 * 32):
padded_data = data
else:
padded_data = pad(data, AES.block_size)
fout.write(cipher.encrypt(padded_data))
def decrypt_file(infile, outfile, aeskey, iv):
cipher = AES.new(aeskey, AES.MODE_CBC, iv)
with open (infile, "rb") as fin:
with open(outfile, "wb") as fout:
while True:
data = fin.read(16 * 32)
if len(data) == 0:
break
fout.write(unpad(cipher.decrypt(data), AES.block_size))
My problem was related to the PAD of the last block. It is necessary to detect which is the last fragment read in bytes in order to add the PAD.
def decrypt_file(
self, filename: str, output_file: str, save_path: str, key, iv
):
cipher_aes = AES.new(key, AES.MODE_CBC, iv)
log.info(f'Decrypting file: {filename} output: {output_file}')
count = 0
previous_data = None
with open(filename, "rb") as f, open(
f"{save_path}/{output_file}", "wb"
) as f2:
while True:
count+=1
data = f.read(self.block_size)
if data == b"":
decrypted = cipher_aes.decrypt(previous_data)
log.info(f'Last block UnPadding Count: {count} BlockSize: {self.block_size}')
decrypted = unpad(decrypted, AES.block_size, style="pkcs7")
f2.write(decrypted)
break
if previous_data:
decrypted = cipher_aes.decrypt(previous_data)
f2.write(decrypted)
previous_data = data
And apply the decrypt:
def decrypt_file(
self, filename: str, output_file: str, save_path: str, key, iv
):
cipher_aes = AES.new(key, AES.MODE_CBC, iv)
log.info(f'Decrypting file: {filename} output: {output_file}')
count = 0
previous_data = None
with open(filename, "rb") as f, open(
f"{save_path}/{output_file}", "wb"
) as f2:
while True:
count+=1
data = f.read(self.block_size)
if data == b"":
decrypted = cipher_aes.decrypt(previous_data)
log.info(f'Last block UnPadding Count: {count} BlockSize: {self.block_size}')
decrypted = unpad(decrypted, AES.block_size, style="pkcs7")
f2.write(decrypted)
break
if previous_data:
decrypted = cipher_aes.decrypt(previous_data)
f2.write(decrypted)
previous_data = data
It looks like the fix is to do similar chunksize/padding comparison in the decrypt function as I used in the encrypt function:
def decrypt_file(infile, outfile, aeskey, iv):
cipher = AES.new(aeskey, AES.MODE_CBC, iv)
with open (infile, "rb") as fin:
with open(outfile, "wb") as fout:
while True:
data = fin.read(16 * 32)
if len(data) == 0:
break
if len(data) == (16 * 32):
decrypted_data = cipher.decrypt(data)
else:
decrypted_data = unpad(cipher.decrypt(data), AES.block_size)
fout.write(decrypted_data)
My input ciphertext from my csv doesnt seem to be decrypting properly. It decrypts as a random string of bytes. I've checked my key and IV and they are exactly the same from encryption, Its just the decryption that doesnt seem to work properly.
I wondered if the way I have put my encrypted data into my csv, or retrieved it is the issue? maybe it alters the bytes etc? if not im stumped. I've been on this issue for days, help!
Program works like this:
User inputs credentials -- encrypt -- generate unique ID and hash values and stores in db -- store ciphertext in csv // user inputs ID -- matches in db and fetches encryption key stored with ID -- Fetches matching ID ciphertext from CSV, puts into pandas dataframe and decrypts ciphertext with key
def decoder():
from Crypto.Cipher import AES
import hashlib
from secrets import token_bytes
cursor.execute(
'''
Select enc_key FROM Login where ID = (?);
''',
(L_ID_entry.get(), ))
row = cursor.fetchone()
if row is not None:
keys = row[0]
#design padding function for encryption
def padded_text(data_in):
while len(data_in)% 16 != 0:
data_in = data_in + b"0"
return data_in
#calling stored key from main file and reverting back to bytes
key_original = keys
print(key_original)
print("Key original above")
mode = AES.MODE_CBC
#model
cipher = AES.new(key_original, mode, IV2.encode('utf8'))
print(IV2)
print("IV2 above")
#padding data
p4 = padded_text(df1.tobytes())
p5 = padded_text(df2.tobytes())
p6 = padded_text(df3.tobytes())
#decrypting data
d_fname = cipher.decrypt(p4)
d_sname = cipher.decrypt(p5)
d_email = cipher.decrypt(p6)
print(d_fname)
print(d_sname)
print(d_email)
#connecting to db
try:
conn = sqlite3.connect('login_details.db')
cursor = conn.cursor()
print("Connected to SQLite")
except sqlite3.Error as error:
print("Failure, error: ", error)
finally:
#downloading txt from dropbox and converting to dataframe to operate on
import New_user
import ast
_, res = client.files_download("/user_details/enc_logins.csv")
with io.BytesIO(res.content) as csvfile:
with open("enc_logins.csv", 'rb'):
df = pd.read_csv(csvfile, names=['ID', 'Fname', 'Sname', 'Email'], encoding='utf-8')
newdf = df[df['ID'] == L_ID_entry.get()]
print(newdf)
df1 = newdf['Fname'].values
df2 = newdf['Sname'].values
df3 = newdf['Email'].values
print(df1)
print(df2)
print(df3)
decoder()
Good afternoon, friends, I just started learning python, I found this code that suits my needs, but on the way out everything is synchronized in one line, help me with this problem.
"
import ecdsa
import hashlib
import base58
with open("my_private_key.txt", "r") as f: #Input file path
for line in f:
#Convert hex private key to bytes
private_key = bytes.fromhex(line)
#Derivation of the private key
signing_key = ecdsa.SigningKey.from_string(private_key, curve=ecdsa.SECP256k1)
verifying_key = signing_key.get_verifying_key()
public_key = bytes.fromhex("04") + verifying_key.to_string()
#Hashes of public key
sha256_1 = hashlib.sha256(public_key)
ripemd160 = hashlib.new("ripemd160")
ripemd160.update(sha256_1.digest())
#Adding prefix to identify Network
hashed_public_key = bytes.fromhex("00") + ripemd160.digest()
#Checksum calculation
checksum_full = hashlib.sha256(hashlib.sha256(hashed_public_key).digest()).digest()
checksum = checksum_full[:4]
#Adding checksum to hashpubkey
bin_addr = hashed_public_key + checksum
#Encoding to address
address = str(base58.b58encode(bin_addr))
final_address = address[2:-1]
print(final_address)
with open("my_addresses.txt", "a") as i:
i.write(final_address)
"
print writes a trailing newline after writing all its arguments. write does not; you have to supply it yourself.
with open("my_addresses.txt", "a") as i:
i.write(final_address + "\n")
Or, you can use print:
with open("my_addresses.txt", "a") as i:
print(final_address, file=i)
Ignoring many of its keyword arguments, print is defined something like
def print(*args, end='\n', sep=' ', file=sys.stdout):
file.write(sep.join(args))
file.write(end)
Also, note that you don't need to repeatedly open your output file. You can open it at the same time as the input and leave it open for the duration of the loop.
with open("my_private_key.txt", "r") as f, \
open("my_addresses.txt", "a") as i:
for line in f:
...
print(final_address, file=i)
I want to make a script to decrypt my filess, but when I try to run my script then show me this message , how can I fix it?
Traceback (most recent call last): File "F:\bug_bounty\decrypt.py",
line 46, in File "F:\bug_bounty\decrypt.py", line 24, in
decrypt File
"C:\Python27\lib\site-packages\Crypto\Cipher\blockalgo.py", line 295,
in decrypt
return self._cipher.decrypt(ciphertext) ValueError: Input strings must be a multiple of 16 in length
from Crypto.Hash import SHA256
from Crypto.Cipher import AES
import os
import random
import sys
def decrypt(key, filename):
outFile = os.path.join(os.path.dirname(filename),
os.path.basename(filename[11:]))
chunksize = 64 * 1024
with open(filename, 'rb') as infile:
filesize = infile.read(16)
IV = infile.read(16)
decryptor = AES.new(key, AES.MODE_CBC, IV)
with open(outFile, 'wb') as outfile:
while True:
chunk = infile.read(chunksize)
if len(chunk) == 0:
break
outfile.write(decryptor.decrypt(chunk))
outfile.truncate(int(filesize))
def allfiles():
allFiles = []
for (root, subfiles, files) in os.walk(os.getcwd()):
for names in files:
allFiles.append(os.path.join(root, names))
return allFiles
password = 'M4st3rRul3zs'
files = allfiles();
for filename in files:
if os.path.basename(filename).startswith("(encrypted)"):
print "%s is already encrypted" %filename
pass
else:
decrypt(SHA256.new(password).digest(), filename)
print "Done decrypting %s" %filename
"""os.remove(filename)"""
Here is the small trick you could use while encrypting the data if your data size is not large.
plaintext = "some text"
encryptor = AES.new(key, AES.MODE_CBC, iv)
ciphertext = encryptor.encrypt(plaintext*16)
This will ensure that your input data is a multiple of 16. And of course, you would like to get the original data back when decrypting.
cipher = AES.new(key, AES.MODE_CBC, iv)
decrypttext = cipher.decrypt(ciphertext)
decrypttext = decrypttext[0:len(plaintext)]
Now, decrpyttext has your original plaintext.
From Crypto++ wiki.
The block size is determined by AES::BLOCKSIZE. For AES, this is
always 16 bytes
AES is a block cipher, it works on 16-byte (128-bit) blocks. It can't work with data smaller or bigger than 16 bytes. Smaller data needs to be padded until they're 16 bytes, and larger data needs to be split into 16-byte blocks.
Also there are algorithms that help you achieve just that (work on data larger than the cipher's block size), they're called block cipher modes of operation.
Have a look at this How to encrypt more than 16 bytes using AES?
ValueError: Input strings must be a multiple of 16 in length
That is because AES works with blocks of 128 bits (16 chars). You can consider adding padding to fix this.
AES works with blocks of 16 chars. This how you can add extra padding
import random
import string
plaintext = "Encrypt me"
encryptor = AES.new(key, AES.MODE_CBC, iv)
while len(bytes(plaintext, encoding='utf-8')) % 16 != 0:
plaintext = plaintext + random.choice(string.ascii_letters)
ciphertext = encryptor.encrypt(plaintext)
I'm making programm for my school project and have one problem above.
Here's my code:
def aes():
#aes
os.system('cls')
print('1. Encrypt')
print('2. Decrypt')
c = input('Your choice:')
if int(c) == 1:
#cipher
os.system('cls')
print("Let's encrypt, alright")
print('Input a text to be encrypted')
text = input()
f = open('plaintext.txt', 'w')
f.write(text)
f.close()
BLOCK_SIZE = 32
PADDING = '{'
pad = lambda s: s + (BLOCK_SIZE - len(s) % BLOCK_SIZE) * PADDING
EncodeAES = lambda c, s: base64.b64encode(c.encrypt(pad(s)))
secret = os.urandom(BLOCK_SIZE)
f = open('aeskey.txt', 'w')
f.write(str(secret))
f.close()
f = open('plaintext.txt', 'r')
privateInfo = f.read()
f.close()
cipher = AES.new(secret)
encoded = EncodeAES(cipher, privateInfo)
f = open('plaintext.txt', 'w')
f.write(str(encoded))
f.close()
print(str(encoded))
if int(c) == 2:
os.system('cls')
print("Let's decrypt, alright")
f = open('plaintext.txt','r')
encryptedString = f.read()
f.close()
PADDING = '{'
DecodeAES = lambda c, e: c.decrypt(base64.b64decode(e)).rstrip(PADDING)
encryption = encryptedString
f = open('aeskey.txt', 'r')
key = f.read()
f.close()
cipher = AES.new(key)
decoded = DecodeAES(cipher, encryption)
f = open('plaintext.txt', 'w')
f.write(decoded)
f.close()
print(decoded)
Full error text:
Traceback (most recent call last): File "C:/Users/vital/Desktop/Prog/Python/Enc_dec/Enc_dec.py", line 341, in aes()
File "C:/Users/vital/Desktop/Prog/Python/Enc_dec/Enc_dec.py", line 180, in aes cipher = AES.new(key)
File "C:\Users\vital\AppData\Local\Programs\Python\Python35-32\lib\site-packages\Crypto\Cipher\AES.py", line 179, in new return AESCipher(key, *args, **kwargs)
File "C:\Users\vital\AppData\Local\Programs\Python\Python35-32\lib\site-packages\Crypto\Cipher\AES.py", line 114, in init blockalgo.BlockAlgo.init(self, _AES, key, *args, **kwargs)
File "C:\Users\vital\AppData\Local\Programs\Python\Python35-32\lib\site-packages\Crypto\Cipher\blockalgo.py", line 401, in init self._cipher = factory.new(key, *args, **kwargs)
ValueError: AES key must be either 16, 24, or 32 bytes long
Process finished with exit code 1
What am I doing wrong?
The error is very clear. The key must be exactly of that size. os.urandom will return you the correct key. However this key is a bytes (binary string value). Furthermore, by using str(secret), the value of repr(secret) is written into the file instead of secret.
What is more confusing is that AES.new allows you to pass the key as Unicode! However, suppose the key was the ASCII bytes 1234123412341234. Now,
f.write(str(secret))
will write b'1234123412341234' to the text file! Instead of 16 bytes, it now contains those 16 bytes + the b, and two ' quote characters; 19 bytes in total.
Or if you take a random binary string from os.urandom,
>>> os.urandom(16)
b'\xd7\x82K^\x7fe[\x9e\x96\xcb9\xbf\xa0\xd9s\xcb'
now, instead of writing 16 bytes D7, 82,.. and so forth, it now writes that string into the file. And the error occurs because the decryption tries to use
"b'\\xd7\\x82K^\\x7fe[\\x9e\\x96\\xcb9\\xbf\\xa0\\xd9s\\xcb'"
as the decryption key, which, when encoded as UTF-8 results in
b"b'\\xd7\\x82K^\\x7fe[\\x9e\\x96\\xcb9\\xbf\\xa0\\xd9s\\xcb'"
which is a 49-bytes long bytes value.
You have 2 good choices. Either you continue to write your key to a text file, but convert it to hex, or write the key into a binary file; then the file should be exactly the key length in bytes. I am going for the latter here:
Thus for storing the key, use
with open('aeskey.bin', 'wb') as keyfile:
keyfile.write(secret)
and
with open('aeskey.bin', 'rb') as keyfile:
key = keyfile.read()
Same naturally applies to the cipher text (that is the encrypted binary), you must write and read it to and from a binary file:
with open('ciphertext.bin', 'wb') as f:
f.write(encoded)
and
with open('ciphertext.bin', 'rb') as f:
encryptedString = f.read()
If you want to base64-encode it, do note that base64.b64encode/decode are bytes-in/bytes-out.
By the way, plaintext is the original, unencrypted text; the encrypted text is called ciphertext. AES is a cipher that can encrypt plaintext to ciphertext and decrypt ciphertext to plaintext using a key.
Despite these being called "-text" neither of them is textual data per se, as understood by Python, but they're binary data, and should be represented as bytes.