We Keep Coding

cryptography.fernet.InvalidToken error when decrypting existing salt

Preamble:
I did search StackOverflow and I know someone had a question like mine, but now (a couple months after first seeing it), I could not find that answer again. I'm fully aware there is likely a duplicate, but for the life of me, I cannot find it in my searches.
The problem:
I have a cryptography module which generates a hashed password used for encrypting and decrypting secrets stored in a TinyDB database. All the functionality works except for decrypting the secret. The password is verifying correctly, so I know that isn't the issue. I am almost positive my issue is in getting the salt encoded properly in the decryption function.
Encryption code:
pas = use_password(args[0])
salt = urandom(16)
kdf = PBKDF2HMAC(algorithm=hashes.SHA256(),length=32,salt=salt,iterations=100000,)
sec = base64.b64encode(kdf.derive(bytes(pas,'utf-8')))
token = Fernet(sec).encrypt(bytes(key,'utf-8'))
salt = base64.b64encode(salt).decode('utf-8')
return token, salt
Decryption:
pas = use_password(pw)
***salt = base64.b64decode(salt)***
kdf = PBKDF2HMAC(algorithm=hashes.SHA256(),length=32,salt=salt,iterations=100000,)
sec = base64.b64encode(kdf.derive(bytes(pas,'utf-8')))
# BUG: Reported defects -#ctrenthem at 7/26/2021, 8:07:00 PM: cryptography.fernet.InvalidToken error
key = Fernet(sec).decrypt(bytes(token,'utf-8'))
return key # Returns a byte object which may need to be converted to a string.
What I tried:
I've already tried different variations of duplicating the salt encryption, but keep getting errors. Part of the problem is that base64.encode requires two "file objects" for input and output, and does not accept a string variable, which makes it unusable for my needs.
I could resolve this by creating a temp file, but that would be the worst solution because it involves
creating a new file in the filesystem with partially decrypted information, outside of RAM and the database, weakening the security of the whole system
Creates a new temp file, which is just a waste of system resources
Adds a whole lot more code just to implement something that I am positive can be implemented differently with one or two words.
Despite knowing that there is another solution, I cannot figure it out and am lost as to which base64 or bytes function will accomplish the job.

How to limit key length with Passlib 1.7+

A server protocol requires me to derive a password hash with a limited key size. This is the given JavaScript + CryptoJS implementation:
var params = {keySize: size/32, hasher: CryptoJS.algo.SHA512, iterations: 5000}
var output = CryptoJS.PBKDF2(password, salt, params).toString();
I want to re-implement this in Python using Passlib, i.e. something like
from passlib.hash import pkbdf2_sha512
output = pbkdf2_sha512.hash(password, salt=salt, rounds=5000)
The Passlib API does not allow me to specify the key size. How to do it though?

If the derived key it to long just truncate it to the required length. Each byte is just as valid as every other byte, it makes no difference which bytes you use, there is no ordering.

Unique Salt per User using Flask-Security

After reading here a bit about salting passwords, it seems that it's best to use a unique salt for each user. I'm working on implementing Flask-Security atm, and from the documentation it appears you can only set a global salt: ie SECURITY_PASSWORD_SALT = 'thesalt'
Question: How would one go about making a unique salt for each password?
Thanks!
edit: from the docs on Flask-Security, I found this, which seems to again suggest that this module only uses a single salt for all passwords out of the box.
flask_security.utils.get_hmac(password)
Returns a Base64 encoded HMAC+SHA512 of the password signed with the salt
specified by SECURITY_PASSWORD_SALT.

Yes, Flask-Security does use per-user salts by design if using bcrypt (and other schemes such as des_crypt, pbkdf2_sha256, pbkdf2_sha512, sha256_crypt, sha512_crypt).
The config for 'SECURITY_PASSWORD_SALT' is only used for HMAC encryption. If you are using bcrypt as the hashing algorithm Flask-Security uses passlib for hashing and it generates a random salt during hashing. This confustion is noted in issue 268: https://github.com/mattupstate/flask-security/issues/268
It can be verified in the code, walking from encrypt to passlib:
flask_security/utils.py (lines 143-151, 39, and 269)
def encrypt_password(password):
...
return _pwd_context.encrypt(signed)
_pwd_context = LocalProxy(lambda: _security.pwd_context)
flask_security/core.py (269, 244-251, and 18)
pwd_context=_get_pwd_context(app)
def _get_pwd_context(app):
...
return CryptContext(schemes=schemes, default=pw_hash, deprecated=deprecated)
from passlib.context import CryptContext
and finally from: https://pythonhosted.org/passlib/password_hash_api.html#passlib.ifc.PasswordHash.encrypt
note that each call to encrypt() generates a new salt,

Turns out that if you use bcrypt, it takes care of the salting and stores it with the hash. So I'll go that route!
Thanks to this topic which lead me to this discovery:
Do I need to store the salt with bcrypt?

Making RSA keys from a password in python

I want to be able to generate and re-generate the same RSA keys from a password (and salt) alone in python.
Currently I was doing it using pycrypto, however, it does not seem to generate the same exact keys from the password alone. The reason seems to be that when pycrypto generates a RSA key it uses some sort of random number internally.
Currently my code looks as follows:
import DarkCloudCryptoLib as dcCryptoLib #some costume library for crypto
from Crypto.PublicKey import RSA
password = "password"
new_key1 = RSA.generate(1024) #rsaObj
exportedKey1 = new_key1.exportKey('DER', password, pkcs=1)
key1 = RSA.importKey(exportedKey1)
new_key2 = RSA.generate(1024) #rsaObj
exportedKey2 = new_key2.exportKey('DER', password, pkcs=1)
key2 = RSA.importKey(exportedKey2)
print dcCryptoLib.equalRSAKeys(key1, key2) #wish to return True but it doesn't
I don't really care if I have to not use pycrypto, as long as I can generate these RSA keys from passwords and salts alone.
Thanks for the help in advance.
Just for reference, this is how dcCryptoLib.equalRSAKeys(key1, key2) function looks like:
def equalRSAKeys(rsaKey1, rsaKey2):
public_key = rsaKey1.publickey().exportKey("DER")
private_key = rsaKey1.exportKey("DER")
pub_new_key = rsaKey2.publickey().exportKey("DER")
pri_new_key = rsaKey2.exportKey("DER")
boolprivate = (private_key == pri_new_key)
boolpublic = (public_key == pub_new_key)
return (boolprivate and boolpublic)
NOTE: Also, I am only using RSA for authentication. So any solution that provides a way of generating secure asymmetric signatures/verifying generated from passwords are acceptable solutions for my application. Though, generating RSA keys from passwords I feel, is a question that should also be answered as it seems useful if used correctly.

If you're trying to implement an authenticated encryption scheme using a shared password, you don't really need an RSA key: all you need is an AES key for encryption and an HMAC key for authentication.
If you do need to generate an asymmetric signature than can be verified without knowing the password, you're going to have to somehow generate RSA (or DSA, etc.) keys in a deterministic manner based on the password. Based on the documentation, this should be possible by defining a custom randfunc, something like this:
from Crypto.Protocol.KDF import PBKDF2
from Crypto.PublicKey import RSA
password = "swordfish" # for testing
salt = "yourAppName" # replace with random salt if you can store one
master_key = PBKDF2(password, salt, count=10000) # bigger count = better
def my_rand(n):
# kluge: use PBKDF2 with count=1 and incrementing salt as deterministic PRNG
my_rand.counter += 1
return PBKDF2(master_key, "my_rand:%d" % my_rand.counter, dkLen=n, count=1)
my_rand.counter = 0
RSA_key = RSA.generate(2048, randfunc=my_rand)
I've tested this, and it does generate deterministic RSA keys (as long as you remember to reset the counter, at least). However, note that this is not 100% future-proof: the generated keys might change, if the pycrypto RSA key generation algorithm is changed in some way.
In either case, you'll almost certainly want to preprocess your password using a slow key-stretching KDF such as PBKDF2, with an iteration count as high as you can reasonably tolerate. This makes breaking your system by brute-force password guessing considerably less easy. (Of course, you still need to use strong passwords; no amount of key-stretching is going to help if your password is abc123.)

Pass "randfunc" to the RSA.generate, and randfunc should return the output bytes, in order, of a well-known key derivation function that has been configured with enough output bits for RSA to "always complete" without needing more bits.
Argon2, scrypt, PBKDF2 are examples of KDFs designed for this purpose.
It may be possible to use Keccak directly as a KDF by specifying a high number of output bits.
If your generation function follows a well known standard closely, it should work across multiple implementations.

Securely Erasing Password in Memory (Python)

How do you store a password entered by the user in memory and erase it securely after it is no longer need?
To elaborate, currently we have the following code:
username = raw_input('User name: ')
password = getpass.getpass()
mail = imaplib.IMAP4(MAIL_HOST)
mail.login(username, password)
After calling the login method, what do we need to do to fill the area of memory that contains password with garbled characters so that someone cannot recover the password by doing a core dump?
There is a similar question, however it is in Java and the solution uses character arrays:
How does one store password hashes securely in memory, when creating accounts?
Can this be done in Python?

Python doesn't have that low of a level of control over memory. Accept it, and move on. The best you can do is to del password after calling mail.login so that no references to the password string object remain. Any solution that purports to be able to do more than that is only giving you a false sense of security.
Python string objects are immutable; there's no direct way to change the contents of a string after it is created. Even if you were able to somehow overwrite the contents of the string referred to by password (which is technically possible with stupid ctypes tricks), there would still be other copies of the password that have been created in various string operations:
by the getpass module when it strips the trailing newline off of the inputted password
by the imaplib module when it quotes the password and then creates the complete IMAP command before passing it off to the socket
You would somehow have to get references to all of those strings and overwrite their memory as well.

There actually -is- a way to securely erase strings in Python; use the memset C function, as per Mark data as sensitive in python
Edited to add, long after the post was made: here's a deeper dive into string interning. There are some circumstances (primarily involving non-constant strings) where interning does not happen, making cleanup of the string value slightly more explicit, based on CPython reference counting GC. (Though still not a "scrubbing" / "sanitizing" cleanup.)

The correct solution is to use a bytearray() ... which is mutable, and you can safely clear keys and sensitive material from RAM.
However, there are some libraries, notably the python "cryptography" library that prevent "bytearray" from being used. This is problematic... to some extent these cryptographic libraries should ensure that only mutable types be used for key material.
There is SecureString which is a pip module that allows you to fully remove a key from memory...(I refactored it a bit and called it SecureBytes). I wrote some unit tests that demonstrate that the key is fully removed.
But there is a big caveat: if someone's password is "type", then the word "type" will get wiped from all of python... including in function definitions and object attributes.
In other words... mutating immutable types is a terrible idea, and unless you're extremely careful, can immediately crash any running program.
The right solution is: never use immutable types for key material, passwords, etc. Anyone building a cryptographic library or routine like "getpass" should be working with a "bytearray" instead of python strings.

If you don't need the mail object to persist once you are done with it, I think your best bet is to perform the mailing work in a subprocess (see the subprocess module.) That way, when the subprocess dies, so goes your password.

This could be done using numpy chararray:
import numpy as np
username = raw_input('User name: ')
mail = imaplib.IMAP4(MAIL_HOST)
x = np.chararray((20,))
x[:] = list("{:<20}".format(raw_input('Password: ')))
mail.login(username, x.tobytes().strip())
x[:] = ''
You would have to determine the maximum size of password, but this should remove the data when it is overwritten.

EDIT: removed the bad advice...
You can also use arrays like the java example if you like, but just overwriting it should be enough.
http://docs.python.org/library/array.html

Store the password in a list, and if you just set the list to null, the memory of the array stored in the list is automatically freed.