We Keep Coding

Python UnicodeDecodeError: 'utf-8' codec can't decode byte

First, I'm sorry, there's so many questions regarding this issue, and I drowned in so many answers, but I still don't get it, because my code works just if I write to the file(?)
So, why this works:
# encrypt and write to file
data_to_encrypt = "mytext"
data = data_to_encrypt.encode('utf-8')
ciphered_bytes = cipher_encrypt.encrypt(data)
ciphered_data = ciphered_bytes
print(ciphered_data)
with open(fileConfigBIN, "wb") as binary_file:
binary_file.write(ciphered_data)
# read the file and decrypt
with open(fileConfigBIN, "rb") as binary_file:
data1 = binary_file.read()
print(data1)
deciphered_bytes = cipher_decrypt.decrypt(data1)
decrypted_data = deciphered_bytes.decode('utf-8')
print(decrypted_data)
Output:
b'\xa0U\xee\xda\xa8R'
b'\xa0U\xee\xda\xa8R'
mytext
But just by commenting the write to file (the file still there with the same information):
#data_to_encrypt = "mytext"
#data = data_to_encrypt.encode('utf-8')
#ciphered_bytes = cipher_encrypt.encrypt(data)
#ciphered_data = ciphered_bytes
#print(ciphered_data)
#with open(fileConfigBIN, "wb") as binary_file:
# binary_file.write(ciphered_data)
with open(fileConfigBIN, "rb") as binary_file:
data1 = binary_file.read()
print(data1)
deciphered_bytes = cipher_decrypt.decrypt(data1)
decrypted_data = deciphered_bytes.decode('utf-8')
print(decrypted_data)
I get this:
b'\xa0U\xee\xda\xa8R'
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Users...................\Python37-32\lib\tkinter__init__.py", line 1705, in call
return self.func(*args)
File "C:\Users\Fabio\source.............", line 141, in AbrirConfig
decrypted_data = deciphered_bytes.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdd in position 1: invalid continuation byte
The read code is using the same file as the commented part save it!
I tried to save and read in 'latin-1', 'ISO-8859-1' and others.. it give no errors, but returns just strange characters
Using Python 3.7
Thank's in advance!
EDIT:
Full working minimal code:
from Crypto.Cipher import AES
fileConfigBIN = "data.dat"
key = b'\x16\x18\xed\x1c^\xaaGN\rl\xc0]\xf0t=\xd0\xdc]t\xaf\xb2\x12,\xe6\xfc\xd6\x11-\x10\xb4\xb1\x0b'
cipher_encrypt = AES.new(key, AES.MODE_CFB)
iv = cipher_encrypt.iv
cipher_decrypt = AES.new(key, AES.MODE_CFB, iv=iv)
def Encrypt():
data_to_encrypt = "MyTextToEncrypt"
data = data_to_encrypt.encode('utf-8')
ciphered_bytes = cipher_encrypt.encrypt(data)
ciphered_data = ciphered_bytes
with open(fileConfigBIN, "wb") as binary_file:
binary_file.write(ciphered_data)
print("CRYPTED DATA SAVED TO FILE: ", str(ciphered_data))
def Decrypt():
with open(fileConfigBIN, "rb") as binary_file:
data1 = binary_file.read()
print("DATA READ FROM FILE: ", data1)
deciphered_bytes = cipher_decrypt.decrypt(data1)
decrypted_data = deciphered_bytes.decode('utf-8')
print("DECRYPTED DATA: ", decrypted_data)
Encrypt() #comment this function to show the problem
print('---------------------')
Decrypt()
I did it guys! Thanks to #Hoenie insight, for every encryption, I store the iv value and use it to decrypt when needed. If happens to need to encrypt again, it save another iv value and so on... Thanks thanks!

Ok, it seems this does have nothing to do with encoding/decoding.
You need to create a new cipher object, because it can only be used once.
def Decrypt():
cipher_decrypt = AES.new(key, AES.MODE_CFB, iv=iv) # <-- add this
with open(fileConfigBIN, "rb") as binary_file:
data1 = binary_file.read()
print("DATA READ FROM FILE: ", data1)
deciphered_bytes = cipher_decrypt.decrypt(data1)
decrypted_data = deciphered_bytes.decode('utf-8')
print("DECRYPTED DATA: ", decrypted_data)
In the source code: GitHub
A cipher object is stateful: once you have decrypted a message
you cannot decrypt (or encrypt) another message with the same
object.
Thanks to: https://stackoverflow.com/a/54082879/7547749

Python encryption unicode error when converting from Python 2 to python 3

I found some code which I want to incorporate into my Python encryption program. It should encrypt the files in the code's same directory, and I want it to target a directory. But, it's written in Python 2 and when I change around some code to fit Python 3, I get the following error:
Traceback (most recent call last):
File "/home/pi/Desktop/Projects/FyleCript/Dev Files/encryption.py", line 77, in <module>
encrypt(SHA256.new(password).digest(), str(Tfiles))
File "/usr/lib/python3/dist-packages/Crypto/Hash/SHA256.py", line 88, in new
return SHA256Hash().new(data)
File "/usr/lib/python3/dist-packages/Crypto/Hash/SHA256.py", line 75, in new
return SHA256Hash(data)
File "/usr/lib/python3/dist-packages/Crypto/Hash/SHA256.py", line 72, in __init__
HashAlgo.__init__(self, hashFactory, data)
File "/usr/lib/python3/dist-packages/Crypto/Hash/hashalgo.py", line 51, in __init__
self.update(data)
File "/usr/lib/python3/dist-packages/Crypto/Hash/hashalgo.py", line 69, in update
return self._hash.update(data)
TypeError: Unicode-objects must be encoded before hashing
But the code works perfectly in Python 2. I have tried looking for similar questions on SO and Googling, but no help.
Code:
def encrypt(key, filename):
chunksize = 64 * 1024
outFile = os.path.join(os.path.dirname(filename), "(encrypted)"+os.path.basename(filename))
filesize = str(os.path.getsize(filename)).zfill(16)
IV = ''
for i in range(16):
IV += chr(random.randint(0, 0xFF))
encryptor = AES.new(key, AES.MODE_CBC, IV)
with open(filename, "rb") as infile:
with open(outFile, "wb") as outfile:
outfile.write(filesize)
outfile.write(IV)
while True:
chunk = infile.read(chunksize)
if len(chunk) == 0:
break
elif len(chunk) % 16 !=0:
chunk += ' ' * (16 - (len(chunk) % 16))
outfile.write(encryptor.encrypt(chunk))
def decrypt(key, filename):
outFile = os.path.join(os.path.dirname(filename), os.path.basename(filename[11:]))
chunksize = 64 * 1024
with open(filename, "rb") as infile:
filesize = infile.read(16)
IV = infile.read(16)
decryptor = AES.new(key, AES.MODE_CBC, IV)
with open(outFile, "wb") as outfile:
while True:
chunk = infile.read(chunksize)
if len(chunk) == 0:
break
outfile.write(decryptor.decrypt(chunk))
outfile.truncate(int(filesize))
def allfiles():
allFiles = []
for root, subfiles, files in os.walk(os.getcwd()):
for names in files:
allFiles.append(os.path.join(root, names))
return allFiles
choice = input("Do you want to (E)ncrypt or (D)ecrypt? ")
password = input("Enter the password: ")
encFiles = allfiles()
if choice == "E" or 'e':
for Tfiles in encFiles:
if os.path.basename(Tfiles).startswith("(encrypted)"):
print("%s is already encrypted" %str(Tfiles))
pass
elif Tfiles == os.path.join(os.getcwd(), sys.argv[0]):
pass
else:
encrypt(SHA256.new(password).digest(), str(Tfiles))
print("Done encrypting %s" %str(Tfiles))
os.remove(Tfiles)
elif choice == "D" or 'd':
filename = input("Enter the filename to decrypt: ")
if not os.path.exists(filename):
print("The file does not exist")
sys.exit()
elif not filename.startswith("(encrypted)"):
print("%s is already not encrypted" %filename)
sys.exit()
else:
decrypt(SHA256.new(password).digest(), filename)
print("Done decrypting %s" %filename)
os.remove(filename)
else:
print("Please choose a valid command.")
sys.exit()
Can anyone help me with this problem? I have used a Python 2 to 3 tool, but it still didn't work.
Also, could you please fix the directory problem? It isn't necessary, but I would like it.
EDIT: I have replaced str with bytes and bytearray but it returns the same error.

Your 'password' variable is a string, but SHA256.new expects bytes (to allow for unicode, for example). That you need bytes is listed in the Crypto.Hash.SHA256 documentation.
The solution is to encode the password to bytes before hashing. This is almost literally what the error message says (if you know that all strings in python 3 are unicode objects):
TypeError: Unicode-objects must be encoded before hashing
The solution is for example to encode with utf8 :
SHA256.new(password.encode('utf8')).digest()

Encoding issues while saving json to file

I am reading logs from mobile device, from system terminal line by line and saving it to string.
Then I cut this string into parts and want to generate .*json file from it.
I have my .*json in form of dictionary and I want to save it to file with usage of this custom made method:
def save_json_to_file(json_dict, folder, file_name, extension):
directory = add_ending_slash(GlobalConfig.OUTPUT_DIR) + add_ending_slash(str(folder))
file_path = clean_path(directory + str(file_name) + "." + extension)
output_file = None
try:
if not os.path.exists(directory):
os.makedirs(directory)
output_file = open(file_path, "w")
absolute_path = os.path.abspath(file_path)
Printer.system_message(TAG, "Created json file '" + absolute_path + "'.")
json.dump(json_dict, output_file, indent=4, ensure_ascii=False)
except Exception as e:
message = "Unable to create file '{}.{}'. Error message: {}"
message = message.format(file_path, extension, str(e))
raise LauncherFlowInterruptedException(TAG, message)
finally:
if output_file is not None and hasattr(output_file, "close"):
output_file.close()
return absolute_path
At some point of json.dump doing it's job I get crash:
File "/Users/F1sherKK/anaconda3/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc0 in position 89: invalid start byte
I tried changing every string I receive from console by doing:
received_terminal_line = received_terminal_line.encode('utf-8')
I know that in incoming strings there can be various different encodings, in example emoji.

UnicodeEncodeError: 'latin-1' codec can't encode character PyPDF2

A simple beginner challenge; a password breaker for a PDF document.
Not a clue as to why this is happening. I would continue to trawl the web to find a solution myself but for the belief that this is likely something I could spend a day on and still not solve. Nevertheless I will add encoding, unicode etc to my list.
>>>
RESTART: C:\Users\Dave\Desktop\2016Coding\AutomateBoring\13-PDF\bruteforce.py
2016-11-23 06:14:51,974 -- DEBUG -- variations: ['AARHUS', 'aarhus', 'Aarhus']
2016-11-23 06:14:52,010 -- DEBUG -- variation: AARHUS
Traceback (most recent call last):
File "C:\Users\Dave\Desktop\2016Coding\AutomateBoring\13-PDF\bruteforce.py", line 64, in <module>
main()
File "C:\Users\Dave\Desktop\2016Coding\AutomateBoring\13-PDF\bruteforce.py", line 48, in main
if pdf_reader.decrypt(variation) == 0:
File "C:\Users\Dave\AppData\Local\Programs\Python\Python35-32\lib\site-packages\PyPDF2\pdf.py", line 1987, in decrypt
return self._decrypt(password)
File "C:\Users\Dave\AppData\Local\Programs\Python\Python35-32\lib\site-packages\PyPDF2\pdf.py", line 2017, in _decrypt
val = utils.RC4_encrypt(new_key, val)
File "C:\Users\Dave\AppData\Local\Programs\Python\Python35-32\lib\site-packages\PyPDF2\utils.py", line 181, in RC4_encrypt
retval += b_(chr(ord_(plaintext[x]) ^ t))
File "C:\Users\Dave\AppData\Local\Programs\Python\Python35-32\lib\site-packages\PyPDF2\utils.py", line 238, in b_
r = s.encode('latin-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0195' in position 0: ordinal not in range(256)
>>>
Script:
#! python3
# Brute force password breaker
# 10th word = ABANDONS
import docx
import logging
import PyPDF2
logging.basicConfig(level=logging.DEBUG, format="%(asctime)s -- " +\
"%(levelname)s -- %(message)s")
def main():
"""Attempts to password break a pdf file using three variations of each
word in a .txt file containing 44,000 total words"""
with open("dictionary.txt", "r") as f:
words = f.read().splitlines()
with open("bruteforce2.pdf", "rb") as f:
pdf_reader = PyPDF2.PdfFileReader(f)
for word in words:
variations = [word.upper(), word.lower(), word.title()]
for variation in variations:
logging.debug("variations: {}".format(variations))
logging.debug("variation: {}".format(variation))
if pdf_reader.decrypt(variation) == 0:
pass
else:
print("\n*****The password is: {!r} *****\n".format(word))
break
if __name__ == "__main__":
main()

UnicodeDecodeError with QFileDialog in PyQt

Hello I am having an issue with my program when it comes to a file dialog function I have.
First here is my code:
def getFileInfo(self):
global logName
logName = QtGui.QFileDialog.getOpenFileName()
return logName
def getFileName(self):
return logName
def compareAction(self):
def process(infile, outfile, keywords):
keys = [[k[0], k[1], 0] for k in keywords]
endk = None
with open(infile, 'rb') as fdin:
with open(outfile, 'ab') as fdout:
fdout.write("<" + words + ">" + "\r\n")
for line in fdin:
if endk is not None:
fdout.write(line)
if line.find(endk) >= 0:
fdout.write("\r\n")
endk = None
else:
for k in keys:
index = line.find(k[0])
if index >= 0:
fdout.write(line[index + len(k[0]):].lstrip())
endk = k[1]
k[2] += 1
if endk is not None:
raise Exception(endk + "Not found before end of file")
return keys
clearOutput = open('test.txt', 'wb')
clearOutput.truncate()
clearOutput.close()
outputText = 'test.txt'
end_token = "[+][+]"
inputFile = logName
start_token = self.serialInputText.toPlainText()
split_start = start_token.split(' ')
for words in split_start:
process(inputFile,outputText,((words + "SHOWALL"),))
fo = open(outputText, "rb")
text = fo.read()
print start_token + '\r\n'
print split_start
print inputFile
Okay, So the general idea of this piece of code is grabbing a some inputted text from a TextEdit in my PyQt GUI. Then, splitting that string into a List that can be used to 'scan' through the file and if there are any matches then print out those matches into another text document.
Steps:
User inputs texts into TextEdit
Texts inside TextEdit gets stored into a QString
That QString has a space as a delimiter so we split each entry into a list. i.e This is a list -> [u'This', u'Is', u'A', u'List'] (The list has a u due to my code using sip)
Now that we have this QStringList we can pass it through my def process function.
We need a file to search through obviously, this is where the def getFileInfo(self) and def GetFileName(Self) function come into play.
So after the user has inputted some text, selected a file to search through, he/she will press a Button, lets call it CompareButton, and it will execute the def compareAction(self) function.
Issue
Currently, my issue is this error that appears after doing all the steps it fails on step number 6. This is my error:
Traceback (most recent call last):
File "RETRACTED.py", line 278, in compareAction
process(inputFile,outputText,((words + "SHOWALL"),))
File "RETRACTED.py", line 260, in process
index = line.find(k[0])
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
I am unsure as to why this error is happening. I have been searching for a similar issue but i believe it has to do with my process function. I am unsure

That specific error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 0: ordinal not in range(128)
looks like a problem with an (unexpected) Byte Order Mark (BOM) in the input file. I suspect the log file is UTF-8 with BOM.
Try changing your file open line to:
open(infile, 'rb', encoding='utf-8-sig')
to have the the BOM marker stripped from the file.