We Keep Coding

Reading variable length binary values from a file in python

I have three text values that I am encrypting and then writing to a file. Later I want to read the values back (in another script) and decrypt them.
I've successfully encrypted the values:
cenc = rsa.encrypt(client_name.encode('utf8'), publicKey)
denc = rsa.encrypt(expiry_date.encode('utf8'), publicKey)
fenc = rsa.encrypt(features.encode('utf8'), publicKey)
and written to a binary file:
licensefh = open("license.sfb", "wb")
licensefh.write(cenc)
licensefh.write(denc)
licensefh.write(fenc)
licensefh.close()
The three values cenc, denc and fenc are all of different lengths so when I read the file back:
licensefh = open("license.sfb", "rb")
encMessage = licensefh.read()
encMessage contains the entire file and I don't know how to get the three values back again.
I've tried using a separator between the values:
SEP = bytes(chr(0x02).encode('utf8'))
...
licensefh.write(cenc)
licensefh.write(SEP)
...
and then using encMessage.partition(SEP) or encMessage.split(SEP) but the data invariably contains the SEP value in it somewhere (I've tried a few different characters) so that didn't work.
I tried getting the length of the bytes objects cenc, denc and fenc, but this returned 256 for each value even though the contents of the variables are all different lengths.
My question is this. How do I write these three variable length values to a binary file and then separate them when I read them back again?
Here's an example of the 3 binary values:
b'tX\x10Fo\x89\x10~\x83Pok\xd1\xfb\xbe\x0e<a\xe5\x11md:\xe6\x84#\xfa\xf8\xe5\xeb\xf8\xdc{\xc0Z\xa0\xc0^\xc1\xd9\x820\xec\xec\xb0R\x99/\xa2l\x88\xa9\xa6g\xa3\x01m\xf9\x7f\x91\xb9\xe1\x80\xccs|\xb7_\xa9Fp\x11yvG\xdc\x02d\x8aK2\x92t\x0e\x1f\xca\x19\xbb&\xaf{\xc0y>\t|\x86\xab\x16.\xa5kZ"\xab6\xaaV\xf4w\x7f\xc5q\x07\xef\xa9\xa5\xa3\xf3 6\xdb\x03\x19S\xbd\x81\xf9\xc8\xc5\x90\x1e\x19\x86\xa4q\xe3?i\xc4\xac\t\xd5=3C\x9b#\xc3IuAN,\xeat\xc6\x96VFL\x1eFWZ\xa4\xd73\x92P#\x1d\xb9\x12\x15\xc9\xd4~\x8aWm^\xb8\x8b\x9d\x88\n)\xeb#\xe3\x93\xb1\\\xd6^\xe0\xce\xa2(\x05\xf5\xe6\x8b\xd1\x15\xd8v\xf0\xae\x90\xd8?\x01\r\x00\xf4\xa5\xadM|%\x98\xa9SR\xc6\xd0K\x9e&\xc3\xe0M\x81\x87\xdea\xcc\xd5\x9c\xcd\xfd1l\x1f\xb9?\xed\xd1\x95\xbc\x11\x85U9'
b'l\xd3S\xcc\x03\x9a\xf2\xfdr\xca\xbbA\x06\xfb\xd8\xbbWi\xdc\xb1\xf6&\x97T\x81Kl\r\x86\x9b\x95?\x94}\x8a\xd3\xa1V\x81\xd3]*B\x1f\x96`\xa3\xd1\xf2|B\x84?\xa0\ns\xb7\xcf\x18Y\x87\xcfR\x87!\x14\x81!\xf7\xf2\xe5x|=O\xe3\xba2\xf2!\x93\x0fT7\x0c~4\xa3\xe5\xb7\xf9wy\xb5\x12FM\x96\xd9\xfd\xedn\x9c\xacw\x1b\xc2\x17+\xb6\x05`\x10\xf8\xe4\x01\xde\xc7\xa2\xa0\x80\xd8\x15\xb1+<s\xc7\x19\x9c\x14\xb0\x1a"\x10\xbb\x0f\xe1\x05\x93\xd2?xX\xd9\x93\x8an\x8d\xcd\xbd!c\xd0,\xa45\xbai\xe3\xccx\x08\xaa,\xd1\xe5\'t\x91\xb8\xf2n$\x0c\xf9-\xb4\xc2\x07\x81\xe1\xe7\x8e\xb3\x98\x11\xf3\xa6\xd9wz\x9a3\xc9\x9c?z\xd8\xaa\x08}\xa2\x9c[\xf2\x9d\xe4\xcdb\xddl\xceV\x7f\xf1\x81\xb3\x88\x1e\x9c5?k\x0f\xc9\x86\x86&\xedV.\xa7\x8d\x13&V\xad\xca\xe5\x93\xfe\xa5\x94\xbc\xf5\xd1{Cl\xc0\x030\x92\x03\xc9'
b'#\xbdd7\xe9\xa0{\t\xb9\x87B\x9e\xf9\x97P^\xf3V\xb6\x93\x1f(J\x0b\xa3\xbf\xd8\x04\x86T\xa4\xca\xf3\xe8%\xddC\x11\xdb5\xff,\xf7\x13\xd7\xd2\xbc\xf3\x893\x83\xdcmJ\xc8p\xdf\x07V\x7fb\xeb\xa9\x8b\x0f\xca\xf9\x05\xfc\xdfS\x94b\x90\xcd\xfcn?/]\x11\xaf\xe606\xfb\\U59\xa0>\xbd\xd8\x1c\xa8\xca\x83\xf4C\x95v7\xc6\xe00\xe4,d_/\x83\xa0\xb9mO\x0e\xc4\x97J\x15\xf0\xca-\xa0\xafT\xe4\x82\x03\n\x14:\xa1\xdcL\x98\x9d,1\xfa\x10\xf4\xfd\xa0\x0b\xc7\x13!\xf7\xdb/\xda\x1a\x9df\x1cQ\xc0\x99H\x08\xa0c\x8f9/4\xc4\x05\xc6\x9eM\x8e\xe5V\xf8D\xc3\xfd\xad4\x94A\xb9[\x80\xb9\xcf\xe6\xd9\xb3M2\xd9N\xfbA\x18\x84/W\x9b\x92\xfe\xbb\xd6C\x85\xa3\xc6\xd2T\xd0\xb2\xb9\xf7R\xb4(s\xda\xbcX,9w\x17\x1c\xfb|\xa0\x87\xba\xca6>y\xba\\L4wc\x94\xe7$Y\x89\x07\x9b\xfe\x9b?{\x85'

#pippo1980 's comment is how I would do it, using struct :
import struct
cenc = b'tX\x10Fo\x89\x10~\x83Pok\xd1\xfb\xbe\x0e<a\xe5\x11md:\xe6\x84#\xfa\xf8\xe5\xeb\xf8\xdc{\xc0Z\xa0\xc0^\xc1\xd9\x820\xec\xec\xb0R\x99/\xa2l\x88\xa9\xa6g\xa3\x01m\xf9\x7f\x91\xb9\xe1\x80\xccs|\xb7_\xa9Fp\x11yvG\xdc\x02d\x8aK2\x92t\x0e\x1f\xca\x19\xbb&\xaf{\xc0y>\t|\x86\xab\x16.\xa5kZ"\xab6\xaaV\xf4w\x7f\xc5q\x07\xef\xa9\xa5\xa3\xf3 6\xdb\x03\x19S\xbd\x81\xf9\xc8\xc5\x90\x1e\x19\x86\xa4q\xe3?i\xc4\xac\t\xd5=3C\x9b#\xc3IuAN,\xeat\xc6\x96VFL\x1eFWZ\xa4\xd73\x92P#\x1d\xb9\x12\x15\xc9\xd4~\x8aWm^\xb8\x8b\x9d\x88\n)\xeb#\xe3\x93\xb1\\\xd6^\xe0\xce\xa2(\x05\xf5\xe6\x8b\xd1\x15\xd8v\xf0\xae\x90\xd8?\x01\r\x00\xf4\xa5\xadM|%\x98\xa9SR\xc6\xd0K\x9e&\xc3\xe0M\x81\x87\xdea\xcc\xd5\x9c\xcd\xfd1l\x1f\xb9?\xed\xd1\x95\xbc\x11\x85U9'
denc = b'l\xd3S\xcc\x03\x9a\xf2\xfdr\xca\xbbA\x06\xfb\xd8\xbbWi\xdc\xb1\xf6&\x97T\x81Kl\r\x86\x9b\x95?\x94}\x8a\xd3\xa1V\x81\xd3]*B\x1f\x96`\xa3\xd1\xf2|B\x84?\xa0\ns\xb7\xcf\x18Y\x87\xcfR\x87!\x14\x81!\xf7\xf2\xe5x|=O\xe3\xba2\xf2!\x93\x0fT7\x0c~4\xa3\xe5\xb7\xf9wy\xb5\x12FM\x96\xd9\xfd\xedn\x9c\xacw\x1b\xc2\x17+\xb6\x05`\x10\xf8\xe4\x01\xde\xc7\xa2\xa0\x80\xd8\x15\xb1+<s\xc7\x19\x9c\x14\xb0\x1a"\x10\xbb\x0f\xe1\x05\x93\xd2?xX\xd9\x93\x8an\x8d\xcd\xbd!c\xd0,\xa45\xbai\xe3\xccx\x08\xaa,\xd1\xe5\'t\x91\xb8\xf2n$\x0c\xf9-\xb4\xc2\x07\x81\xe1\xe7\x8e\xb3\x98\x11\xf3\xa6\xd9wz\x9a3\xc9\x9c?z\xd8\xaa\x08}\xa2\x9c[\xf2\x9d\xe4\xcdb\xddl\xceV\x7f\xf1\x81\xb3\x88\x1e\x9c5?k\x0f\xc9\x86\x86&\xedV.\xa7\x8d\x13&V\xad\xca\xe5\x93\xfe\xa5\x94\xbc\xf5\xd1{Cl\xc0\x030\x92\x03\xc9'
fenc = b'#\xbdd7\xe9\xa0{\t\xb9\x87B\x9e\xf9\x97P^\xf3V\xb6\x93\x1f(J\x0b\xa3\xbf\xd8\x04\x86T\xa4\xca\xf3\xe8%\xddC\x11\xdb5\xff,\xf7\x13\xd7\xd2\xbc\xf3\x893\x83\xdcmJ\xc8p\xdf\x07V\x7fb\xeb\xa9\x8b\x0f\xca\xf9\x05\xfc\xdfS\x94b\x90\xcd\xfcn?/]\x11\xaf\xe606\xfb\\U59\xa0>\xbd\xd8\x1c\xa8\xca\x83\xf4C\x95v7\xc6\xe00\xe4,d_/\x83\xa0\xb9mO\x0e\xc4\x97J\x15\xf0\xca-\xa0\xafT\xe4\x82\x03\n\x14:\xa1\xdcL\x98\x9d,1\xfa\x10\xf4\xfd\xa0\x0b\xc7\x13!\xf7\xdb/\xda\x1a\x9df\x1cQ\xc0\x99H\x08\xa0c\x8f9/4\xc4\x05\xc6\x9eM\x8e\xe5V\xf8D\xc3\xfd\xad4\x94A\xb9[\x80\xb9\xcf\xe6\xd9\xb3M2\xd9N\xfbA\x18\x84/W\x9b\x92\xfe\xbb\xd6C\x85\xa3\xc6\xd2T\xd0\xb2\xb9\xf7R\xb4(s\xda\xbcX,9w\x17\x1c\xfb|\xa0\x87\xba\xca6>y\xba\\L4wc\x94\xe7$Y\x89\x07\x9b\xfe\x9b?{\x85'
packing_format = "<HHH" # little-endian, 3 * (2-byte unsigned short)
with open("license.sfb", "wb") as licensefh:
licensefh.write(struct.pack(packing_format, len(cenc), len(denc), len(fenc)))
licensefh.write(cenc)
licensefh.write(denc)
licensefh.write(fenc)
# close is automatic with a context-manager
with open("license.sfb", "rb") as licensefh2:
header_length = struct.calcsize(packing_format)
cenc2_len, denc2_len, fenc2_len = struct.unpack(packing_format, licensefh2.read(header_length))
cenc2 = licensefh2.read(cenc2_len)
denc2 = licensefh2.read(denc2_len)
fenc2 = licensefh2.read(fenc2_len)
assert len(cenc2) == cenc2_len and len(denc2) == denc2_len and len(fenc2) == fenc2_len # the file was not truncated
unread_bytes = licensefh2.read() # until EOF
assert len(unread_bytes) == 0 # there is nothing else in the file, everything has been read
assert cenc == cenc2
assert denc == denc2
assert fenc == fenc2

Value Error, invalid literal for int() with base 2 (Python Steganography !)

I am working on a simple python steganography project, It can add the text to the image, but I couldn't retrieve the text from the image.
Here's my code
'''
iStegIO is a simple python script/ application which is used to hide text messages or plain text within png images.
It usees LSB Steganography technique to hide the text within the images. Least Significant Bit Steganography method, replaces
the blue bits, with the text bits.
'''
from tkinter import filedialog as f
from pyfiglet import figlet_format
import binascii
from PIL import Image,ImageColor
'''
The below function will take three int arguments and gives hex code for the corresponding color !
params:r,g,b color code (int)
return: hex code for color
'''
def rgb_to_hex(r,g,b):
return '#{:02x}{:02x}{:02x}'.format(r,g,b)
'''
The below function will take a hex code and gives a tuple containing the rgb values(three len)
params:hex code for color !
return: rgb tuple
'''
def hex_to_rgb(hexcode):
#using the getcolor method we got the rgb tuple !
return ImageColor.getcolor(hexcode,"RGB")
'''
We first convert the string message into hexadecimal and then into binary
params: string message
return: binary representation of the message
'''
def string_to_binary(string):
#the output will be 0b01100100010010
#we have to translate the binary to 01100100010010
binary=bin(int(binascii.hexlify(string.encode()),16)).replace('0b','')
return binary
'''
The below funcition will convert the binary format into corresponding byte strem message
params: binary
return : byte stream message !
'''
def binary_to_string(binary):
# we first convert the binary to hex and then into string !
#each hex bit is translated into int base 2 and then to string
#we get the byte stream of the message !
message= binascii.unhexlify('%x' % (int('0b'+ binary ,2)))
return message
'''
If end of the hex code is ether 0 or 1, then return the last part of the hexcode
params: hexcode
return: last hex
'''
def decrypt(hexcode):
if hexcode[-1] in [0,1]:
return hexcode[-1]
else:return None
def encrypt(hexcode,digit):
if hexcode[-1] in ['0','1','2','3','4','5']:
hexcode=hexcode[:-1]+digit
return hexcode
else:return None
'''
We'll use all the helper function above to hide the message string within the photo
'''
def hide(filename,message,out):
image =Image.open(filename)
#convert the message to binary and append 15 ones and one zero !
binary=string_to_binary(message)+'1111111111111110'
if image.mode in ('RGBA'):
image =image.convert('RGBA')
#getting the data from the image !
img_data=image.getdata()
#the text and the image will appended here !
enc_data=[]
#the current bin digit we are working on !
digit = 0
for i in img_data:
if (digit < len(binary)):
#reference made to the rgb_to_hex function to convert the rgb tuple to the hexadecimal and encode to check whether it lies in the 0 to 5 hex range !
new_data=encrypt(rgb_to_hex(i[0],i[1],i[2]),binary[digit])
#if it lies there we get a non None value!
if new_data is None:
enc_data.append(i)
else:
r,g,b=hex_to_rgb(new_data)
#using the rgba format !
enc_data.append((r,g,b,255))
digit+=1
else:
enc_data.append(i)
image.putdata(enc_data)
image.save(out+".png","PNG")
print('Completed !')
return "Invalid Format !!"
def extract(filename):
image =Image.open(filename)
#the binary data, which is to be store, is initlized as null string
binary=''
if image.mode in ('RGBA'):
image=image.convert('RGBA')
img_data=image.getdata()
for i in img_data:
#here we obtain the digit from which it is added to this image !
digit = decrypt(rgb_to_hex(i[0],i[1],i[2]))
if digit is None:
pass
else:
binary=binary+digit
#checking for the delimiter !
if (binary[-16:] =='1111111111111110'):
print('We got it !')
#now just have to convert the binary to string upto the delimiter !
return binary_to_string(binary[:-16])
#else do normal conversion !
return binary_to_string(binary)
return "Invalid Image Mode !"
#-----------------------------End of the Implementation --------------------------------#
if __name__=='__main__':
heading = figlet_format('i S t e g I O')
print(heading)
print('VERSION 1.0')
print(*65*('-'))
while True:
choice = int(input('1)Encrypt\n2)Decrypt\n3)Exit'))
if choice == 1:
image = f.askopenfilename()
mess=input('Enter the message !')
out=input('Output File name')
hide(image,mess,out)
elif choice == 2:
image =f.askopenfilename()
extract(image)
elif choice == 3:
print('Exiting.........')
break
else:print('Invalid !')
It is a simple menu-type program that can accept three inputs namely encrypt, decrypt and exit. When we enter the choice for encrypt, we can add the message(text ) to the image. When we choose the decrypt option we can extract the text from the image. But whenever we try to extract the message from the image, I get the error invalid literal for int() with base 2.
Edit: Traceback:
Traceback (most recent call last): File "C:\Users\USER\Documents\Workspace\Python\iStegIO\iStegIO.py", line 132,
in <module> extract(image) File "C:\Users\USER\Documents\Workspace\Python\iStegIO\iStegIO.py", line 113,
in extract return binary_to_string(binary) File "C:\Users\USER\Documents\Workspace\Python\iStegIO\iStegIO.py", line 44,
in binary_to_string message= binascii.unhexlify('%x' % (int('0b'+ binary ,2))) ValueError: invalid literal for int() with base 2: '0b'

How do you encrypt an image using CTR encryption?

Good afternoon, I am trying to write a program to read a .bmp file and encrypt it using the given initial value using 𝑐𝑡𝑟 and the one-time pad.
The first 36 bytes form the header of the image and are not encrypted, but just copied to the new file
The image data beginning at 0x36 to the end are grouped into four-byte words and each word is encrypted using 𝑐𝑡𝑟.
To avoid changing the size of the image, do not include 𝑐0=𝑐𝑡𝑟 in the encrypted image.
As of now, this is what I have:
from Crypto.Cipher import AES
from Crypto.Util.Padding import pad, unpad
from Crypto.Util import Counter
filename = "Image11.bmp"
filename_out = "Image11Encrypted.bmp"
key = 0xe0984dd3
bkey = key.to_bytes(32, 'big')
cipher = AES.new(bkey, AES.MODE_CTR, initial_value= 0xff128eff)
def encrypt(filename, filename_out, key):
with open(filename, "rb") as f:
clear = f.read()
clear_trimmed = clear[64:-2]
ciphertext = clear_trimmed
ciphertext = cipher.encrypt(pad(clear_trimmed, 16))
ciphertext = clear[0:64] + ciphertext + clear[-2:]
with open(filename_out, "wb") as f:
f.write(ciphertext)
encrypt(filename, filename_out, key)
print("Encrypted using AES in CTR mode and saved to \"" + filename_out + "\"")
However, I keep running into this Error:
.
Any help would be great, not sure where to go from here

The configuration of the CTR mode with PyCryptodome is described here. There are two ways to specify the counter block: By setting a nonce (parameter nonce) and a start value (parameter initial_value). If no nonce is specified, a random nonce of half the block size is generated implicitly.
The other way is to define a counter block object (parameter counter), which can be used to specify the components of the counter block in detail (prefix, counter, suffix).
If only the start value is to be specified, the parameter initial_value must be used instead of counter:
cipher = AES.new(bkey, AES.MODE_CTR, initial_value=0xff128eff)
As mentioned above, this implicitly creates a random nonce with half the block size, which can be determined with cipher.nonce.
Please note: The code lacks the determination of the 16 byte IV, which is needed for decryption. The IV consists of nonce and counter and is usually placed on byte level before the ciphertext.Furthermore, according to the question the first 36 bytes should not be encrypted, a little later it is stated that the data starts at 0x36 (=54) and in the code 64 is used as the beginning of the data. This seems to be inconsistent.

How to decrypt a .txt.gz.enc file with a python program without knowing the key?

For this problem I have been given an encrypted text file and have been asked to find the key and then decrypt the file into a .txt.gz file.
So far, I know that the cipher I should be using is a type of substitution cipher. I was given the code that was used to encrypt the message and I know that I will need an XOR Rotation in order to find the key and decipher the message.
This is code that I had developed when I was given a key
import sys
import gzip
with open("juliaplaintext.txt.gz.enc", "rb") as f:
data = f.read()
k = data.decode("utf-8")
i =0
key = "IbSeMGjyepOr" * 10000
rotated = b""
s = open("juliaplaintext.txt.gz", "wb")
for ch0, ch1 in zip(k, key):
eb = chr(ord(ch0) ^ ord(ch1))
rotated += bytes(ord(eb) >> 7 & 0xff | ord(eb) << 7)
s.write(rotated)
s.close()
I am very new to python and am unsure how to go about creating a decoding program when I am not given the key. Any help is very appreciated.

Decrypting Large files with RSA in pycrypto?

I have been using pycrypto module for encryption and decryption with RSA key pair and algorithm. The problem is when I try encrypting large files (10kB of text file) I take the block size of 32 byte when reading the file and encrypting it
>>> f = open('10kb','rb')
>>> p = open('enc','wb')
>>> while True:
data = f.read(32)
if not data:
break
enc_data = public_key.encrypt(data,32)
p.write(enc_data[0])
p.close()
f.close()
It gives the output:
128
128
.......and the many 128 blocks it is writing
When I try to decrypt the encrypted file, I need to read it with 128 byte block so as to give back 32 byte blocks,
>>> f = open('enc','rb')
>>> p = open('dec','wb')
>>> while True:
data = f.read(128)
if not data:
break
dec_data = private_key.decrypt(data)
p.write(dec_data)
p.close()
f.close()
It is giving the output:
32
32
.....so many 32 byte blocks it is decrypting, then
128
128
128
128
Traceback (most recent call last):
File "<pyshell#251>", line 5, in <module>
enc_data = private_key.decrypt(data)
File "/usr/lib/python3/dist-packages/Crypto/PublicKey/RSA.py", line 174, in decrypt
return pubkey.pubkey.decrypt(self, ciphertext)
File "/usr/lib/python3/dist-packages/Crypto/PublicKey/pubkey.py", line 93, in decrypt
plaintext=self._decrypt(ciphertext)
File "/usr/lib/python3/dist-packages/Crypto/PublicKey/RSA.py", line 237, in _decrypt
cp = self.key._blind(ciphertext, r)
ValueError: Message too large
To the point where it is outputting the block size of 32, it is decrypting right, but where it starts with 128, its messing up. Why it is saying Message size too large ? Is there any better and fast way to decrypt large text files using pycrypto module ?

Partial answer coming along ...
RSA works on numbers. You only get bytes out of it when you serialize those long integers. Since those numbers don't have a fixed size, they are serialized with as much bytes as are necessary, but not more.
An RSA encryption c = me mod n can result in ciphertexts, which are so much smaller than n, that not all the bytes are filled, because leading zeros of the the number don't have to be serialized.
Sometimes (depending on modulus and plaintext) it may happen that you're writing a 127 byte chunk instead of a 128 byte chunk during encryption, but you're always reading a 128 byte chunk during decryption. That means, you're taking away one byte from the next chunk. When the alignment breaks, you can run into various random behaviors such as a chunk being larger than the modulus and therefore not a valid ciphertext.
There are two ways to solve that:
Always write the length of the ciphertext chunk before it.
Encryption:
data = f.read(readsize)
if not data:
break
i += 1
enc_data = public_key.encrypt(data, 32)[0]
p.write(chr(len(enc_data)))
p.write(enc_data)
Decryption:
length = f.read(1)
if not length:
break
data = f.read(ord(length))
print(length, len(data))
j += 1
dec_data = private_key.decrypt(data)
p.write(dec_data[:readsize])
At the end you have to reduce the ciphertext to the original plaintext size, because you're working without PKCS#1 v1.5 padding or OAEP.
Pad the zero bytes that are missing during encryption.
Encryption:
data = f.read(readsize)
if not data:
break
i += 1
enc_data = public_key.encrypt(data, 32)[0]
while len(enc_data) < writesize:
enc_data = "\x00" + enc_data
p.write(enc_data)
Decryption:
data = f.read(writesize)
if not data:
break
j += 1
dec_data = private_key.decrypt(data)
p.write(dec_data[:readsize])
Note that readsize = 127 and writesize = 128. Here are the full source codes for both variants.
Now, this is a partial answer, because this still leads to corrupt files, which are also too short, but at least it fixes the OP's error.