Related
I keep important settings like the hostnames and ports of development and production servers in my version control system. But I know that it's bad practice to keep secrets (like private keys and database passwords) in a VCS repository.
But passwords--like any other setting--seem like they should be versioned. So what is the proper way to keep passwords version controlled?
I imagine it would involve keeping the secrets in their own "secrets settings" file and having that file encrypted and version controlled. But what technologies? And how to do this properly? Is there a better way entirely to go about it?
I ask the question generally, but in my specific instance I would like to store secret keys and passwords for a Django/Python site using git and github.
Also, an ideal solution would do something magical when I push/pull with git--e.g., if the encrypted passwords file changes a script is run which asks for a password and decrypts it into place.
EDIT: For clarity, I am asking about where to store production secrets.
You're exactly right to want to encrypt your sensitive settings file while still maintaining the file in version control. As you mention, the best solution would be one in which Git will transparently encrypt certain sensitive files when you push them so that locally (i.e. on any machine which has your certificate) you can use the settings file, but Git or Dropbox or whoever is storing your files under VC does not have the ability to read the information in plaintext.
Tutorial on Transparent Encryption/Decryption during Push/Pull
This gist https://gist.github.com/873637 shows a tutorial on how to use the Git's smudge/clean filter driver with openssl to transparently encrypt pushed files. You just need to do some initial setup.
Summary of How it Works
You'll basically be creating a .gitencrypt folder containing 3 bash scripts,
clean_filter_openssl
smudge_filter_openssl
diff_filter_openssl
which are used by Git for decryption, encryption, and supporting Git diff. A master passphrase and salt (fixed!) is defined inside these scripts and you MUST ensure that .gitencrypt is never actually pushed.
Example clean_filter_openssl script:
#!/bin/bash
SALT_FIXED=<your-salt> # 24 or less hex characters
PASS_FIXED=<your-passphrase>
openssl enc -base64 -aes-256-ecb -S $SALT_FIXED -k $PASS_FIXED
Similar for smudge_filter_open_ssl and diff_filter_oepnssl. See Gist.
Your repo with sensitive information should have a .gitattribute file (unencrypted and included in repo) which references the .gitencrypt directory (which contains everything Git needs to encrypt/decrypt the project transparently) and which is present on your local machine.
.gitattribute contents:
* filter=openssl diff=openssl
[merge]
renormalize = true
Finally, you will also need to add the following content to your .git/config file
[filter "openssl"]
smudge = ~/.gitencrypt/smudge_filter_openssl
clean = ~/.gitencrypt/clean_filter_openssl
[diff "openssl"]
textconv = ~/.gitencrypt/diff_filter_openssl
Now, when you push the repository containing your sensitive information to a remote repository, the files will be transparently encrypted. When you pull from a local machine which has the .gitencrypt directory (containing your passphrase), the files will be transparently decrypted.
Notes
I should note that this tutorial does not describe a way to only encrypt your sensitive settings file. This will transparently encrypt the entire repository that is pushed to the remote VC host and decrypt the entire repository so it is entirely decrypted locally. To achieve the behavior you want, you could place sensitive files for one or many projects in one sensitive_settings_repo. You could investigate how this transparent encryption technique works with Git submodules http://git-scm.com/book/en/Git-Tools-Submodules if you really need the sensitive files to be in the same repository.
The use of a fixed passphrase could theoretically lead to brute-force vulnerabilities if attackers had access to many encrypted repos/files. IMO, the probability of this is very low. As a note at the bottom of this tutorial mentions, not using a fixed passphrase will result in local versions of a repo on different machines always showing that changes have occurred with 'git status'.
Heroku pushes the use of environment variables for settings and secret keys:
The traditional approach for handling such config vars is to put them under source - in a properties file of some sort. This is an error-prone process, and is especially complicated for open source apps which often have to maintain separate (and private) branches with app-specific configurations.
A better solution is to use environment variables, and keep the keys out of the code. On a traditional host or working locally you can set environment vars in your bashrc. On Heroku, you use config vars.
With Foreman and .env files Heroku provide an enviable toolchain to export, import and synchronise environment variables.
Personally, I believe it's wrong to save secret keys alongside code. It's fundamentally inconsistent with source control, because the keys are for services extrinsic to the the code. The one boon would be that a developer can clone HEAD and run the application without any setup. However, suppose a developer checks out a historic revision of the code. Their copy will include last year's database password, so the application will fail against today's database.
With the Heroku method above, a developer can checkout last year's app, configure it with today's keys, and run it successfully against today's database.
The cleanest way in my opinion is to use environment variables. You won't have to deal with .dist files for example, and the project state on the production environment would be the same as your local machine's.
I recommend reading The Twelve-Factor App's config chapter, the others too if you're interested.
I suggest using configuration files for that and to not version them.
You can however version examples of the files.
I don't see any problem of sharing development settings. By definition it should contain no valuable data.
An option would be to put project-bound credentials into an encrypted container (TrueCrypt or Keepass) and push it.
Update as answer from my comment below:
Interesting question btw. I just found this: github.com/shadowhand/git-encrypt which looks very promising for automatic encryption
Since asking this question I have settled on a solution, which I use when developing small application with a small team of people.
git-crypt
git-crypt uses GPG to transparently encrypt files when their names match certain patterns. For intance, if you add to your .gitattributes file...
*.secret.* filter=git-crypt diff=git-crypt
...then a file like config.secret.json will always be pushed to remote repos with encryption, but remain unencrypted on your local file system.
If I want to add a new GPG key (a person) to your repo which can decrypt the protected files then run git-crypt add-gpg-user <gpg_user_key>. This creates a new commit. The new user will be able to decrypt subsequent commits.
BlackBox was recently released by StackExchange and while I have yet to use it, it seems to exactly address the problems and support the features requested in this question.
From the description on https://github.com/StackExchange/blackbox:
Safely store secrets in a VCS repo (i.e. Git or Mercurial). These
commands make it easy for you to GPG encrypt specific files in a repo
so they are "encrypted at rest" in your repository. However, the
scripts make it easy to decrypt them when you need to view or edit
them, and decrypt them for for use in production.
I ask the question generally, but in my specific instance I would like
to store secret keys and passwords for a Django/Python site using git
and github.
No, just don't, even if it's your private repo and you never intend to share it, don't.
You should create a local_settings.py put it on VCS ignore and in your settings.py do something like
from local_settings import DATABASES, SECRET_KEY
DATABASES = DATABASES
SECRET_KEY = SECRET_KEY
If your secrets settings are that versatile, I am eager to say you're doing something wrong
EDIT: I assume you want to keep track of your previous passwords versions - say, for a script that would prevent password reusing etc.
I think GnuPG is the best way to go - it's already used in one git-related project (git-annex) to encrypt repository contents stored on cloud services. GnuPG (gnu pgp) provides a very strong key-based encryption.
You keep a key on your local machine.
You add 'mypassword' to ignored files.
On pre-commit hook you encrypt the mypassword file into the mypassword.gpg file tracked by git and add it to the commit.
On post-merge hook you just decrypt mypassword.gpg into mypassword.
Now if your 'mypassword' file did not change then encrypting it will result with same ciphertext and it won't be added to the index (no redundancy). Slightest modification of mypassword results in radically different ciphertext and mypassword.gpg in staging area differs a lot from the one in repository, thus will be added to the commit. Even if the attacker gets a hold of your gpg key he still needs to bruteforce the password. If the attacker gets an access to remote repository with ciphertext he can compare a bunch of ciphertexts, but their number won't be sufficient to give him any non-negligible advantage.
Later on you can use .gitattributes to provide an on-the-fly decryption for quit git diff of your password.
Also you can have separate keys for different types of passwords etc.
Usually, i seperate password as a config file. and make them dist.
/yourapp
main.py
default.cfg.dist
And when i run main.py, put the real password in default.cfg that copied.
ps. when you work with git or hg. you can ignore *.cfg files to make .gitignore or .hgignore
Provide a way to override the config
This is the best way to manage a set of sane defaults for the config you checkin without requiring the config be complete, or contain things like hostnames and credentials. There are a few ways to override default configs.
Environment variables (as others have already mentioned) are one way of doing it.
The best way is to look for an external config file that overrides the default config values. This allows you to manage the external configs via a configuration management system like Chef, Puppet or Cfengine. Configuration management is the standard answer for the management of configs separate from the codebase so you don't have to do a release to update the config on a single host or a group of hosts.
FYI: Encrypting creds is not always a best practice, especially in a place with limited resources. It may be the case that encrypting creds will gain you no additional risk mitigation and simply add an unnecessary layer of complexity. Make sure you do the proper analysis before making a decision.
Encrypt the passwords file, using for example GPG. Add the keys on your local machine and on your server. Decrypt the file and put it outside your repo folders.
I use a passwords.conf, located in my homefolder. On every deploy this file gets updated.
No, private keys and passwords do not fall under revision control. There is no reason to burden everyone with read access to your repository with knowing sensitive service credentials used in production, when most likely not all of them should have access to those services.
Starting with Django 1.4, your Django projects now ship with a project.wsgi module that defines the application object and it's a perfect place to start enforcing the use of a project.local settings module that contains site-specific configurations.
This settings module is ignored from revision control, but it's presence is required when running your project instance as a WSGI application, typical for production environments. This is how it should look like:
import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "project.local")
# This application object is used by the development server
# as well as any WSGI server configured to use this file.
from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()
Now you can have a local.py module who's owner and group can be configured so that only authorized personnel and the Django processes can read the file's contents.
If you need VCS for your secrets you should at least keep them in a second repository seperated from you actual code. So you can give your team members access to the source code repository and they won't see your credentials. Furthermore host this repository somewhere else (eg. on your own server with an encrypted filesystem, not on github) and for checking it out to the production system you could use something like git-submodule.
This is what I do:
Keep all secrets as env vars in $HOME/.secrets (go-r perms) that $HOME/.bashrc sources (this way if you open .bashrc in front of someone, they won't see the secrets)
Configuration files are stored in VCS as templates, such as config.properties stored as config.properties.tmpl
The template files contain a placeholder for the secret, such as:
my.password=##MY_PASSWORD##
On application deployment, script is ran that transforms the template file into the target file, replacing placeholders with values of environment variables, such as changing ##MY_PASSWORD## to the value of $MY_PASSWORD.
Another approach could be to completely avoid saving secrets in version control systems and instead use a tool like vault from hashicorp, a secret storage with key rolling and auditing, with an API and embedded encryption.
You could use EncFS if your system provides that. Thus you could keep your encrypted data as a subfolder of your repository, while providing your application a decrypted view to the data mounted aside. As the encryption is transparent, no special operations are needed on pull or push.
It would however need to mount the EncFS folders, which could be done by your application based on an password stored elsewhere outside the versioned folders (eg. environment variables).
Problem:
Customer would like to make sure that the script I've developed in Python, running under CentOS7, can sufficiently obscure the credentials required to access a protected web service (that only supports Basic Auth) such that someone with access to the CentOS login cannot determine those credentials.
Discussion:
I have a Python script that needs to run as a specific CentOS user, say "joe".
That script needs to access a protected Web Service.
In order to attempt to make the credentials external to the code I have put them into a configuration file.
I have hidden the file (name starts with a period "."), and base64 encoded the credentials for obscurity, but the requirement is to only allow the Python script to be able to "see" and open the file, vs anyone with access to the CentOS account.
Even though the file is hidden, "joe" can still do an ls -a and see the file, and then cat the contents.
As a possible solution, is there a way to set the file permissions in CentOS such that they will only allow that Python script to see and open the file, but still have the script run under the context of a CentOS account?
Naive solution
For this use-case I would probably create with a script (sh or zsh or whatever, but I guess u use the default one here) a temporal user iamtemporal-and-wontstayafterifinish. Then creating the config file for being able to read ONLY by specifically this user (and none permission for all the others). Read here for the how: https://www.thegeekdiary.com/understanding-basic-file-permissions-and-ownership-in-linux/
Getting harder
If the problem still raises in case someone would have root-rights (for any such reason), then just simply forget everything above, and start planning for a vacation, cuz' this will be a lot longer then anyone would think.
Is not anymore a simple python problem, but needs a different business logic. The best u could do is to implement (at least this credentials handling part) in a low-level language so could handle memory in a customized way and ask for them runtime only, don't store them...
Or maybe if u could limit the scope of this user accesses towards the protected Web Service as u say.
Bonus
Even tho it wasn't explicitly asked, I would discourage you from storing credentials with using a simple base64...
For this purpose a simple solution could be the following one at least (without the knowledge of the whole armada of cryptography):
encrypt the passw with a asymmetric cryptographic algorithm (probably RSA with a huge key
inject the key for decryption as a env var while you have an open ssh session to the remote terminal
ideally u use this key only while u decrypt and send it, afterwards make sure u delete the references to the variables
Sidenote: it's still filled with 'flaws'. If security is really a problem, I would consider changing technology or using some sort of lib that handles these stuff more securely. I would start probably here: Securely Erasing Password in Memory (Python)
Not to mention memory dumps can be read 'easily' (if u know what u are looking for...): https://cyberarms.wordpress.com/2011/11/04/memory-forensics-how-to-pull-passwords-from-a-memory-dump/
So yeah, having a key-server which sends you the private key to decrypt is not enough, if you read these last two web entries...
I'm searching for best solution to store sensitive data in database.
I know that this is common problem and i have done my homework (at least this is what i think), but i wanted to ask here before i will make a decision.
Assumptions:
Encrypted data needs to be decrypted. We are talking about SMTP credentials like username, password, host, port itp.
I was thinking about 2 concepts:
Encrypt data with help of passlib.totp library. To make those data a bit safer i will keep key in separate file. Then from what i can see i can use this library to decrypt data to plain text using my key.
The other concept was to encrypt and decrypt data during query request with help of postgres:
insert into demo(pw) values ( encrypt( 'data', 'key', 'aes') );
And:
decrypt(pw, 'key', 'aes'), 'utf-8')
Here the key will be stored also in separate file.
So my questions are:
What is better approach to encrypt / decrypt data, in code or in database?
Are there any better (stronger) libraries to use than passlib.totp -> i have no experience with that library (i'm aware that encryption / decryption is not the moste secure way of storing password -> password supposed to be hased but i need it in plain text to use users smtp gate).
2) The other concept was to encrypt and decrypt data during query request with help of postgres: insert into demo(pw) values ( encrypt( 'data', 'key', 'aes') ); and decrypt(pw, 'key', 'aes'), 'utf-8') Here the key will be stored also in separate file.
I wouldn't recommend that, because it's way too easy for the keys to get exposed in pg_stat_activity, the logs, etc. PostgreSQL doesn't have log masking features that would protect against that.
I strongly advise you to use app-side crypto. Use a crypto offload device if security is crucial, so key extraction isn't possible for most attackers. Or require the key to be unlocked by the admin entering a passphrase at app start, so the key is never stored unencrypted on disk - then the attacker has to steal it from memory. But even an unencrypted key file somewhere non-obvious is better than in-db crypto, IMO, since it at least separates the key from the data.
Ultimately, your application needs to be able to recover the plaintext passwords using some sort of key. If your system is compromised, you have to assume that the malicious user will simply find your key (whether in the database or on disk) and perform the exact same decryption that your application performs.
When your system needs to store passwords in a recoverable form, i.e. to authenticate with external systems, at best you can only obfuscate that information. This is what people mean when they refer to "security through obscurity", and it's a bad idea.
When you give the appearance of security, without actually securing something, then that can make things even more dangerous. You, or other people administering the system, may overlook other important security measures because they believe that there is already a layer of protection for the sensitive information. It can create a social situation where sensitive information may be more likely to leak, than it would if it was assumed that the only way to safeguard the info is to safeguard the systems that hold it. It can also cause people to believe that if data is stolen from the database, "maybe it's okay because they can't decrypt it". This is how you end up responsible for leaking credentials to the wide world, because you must assume that if your application can get the plaintext data, so can an attacker that has compromised your application.
There may be a very very small advantage to encrypting a multipart key on multiple systems (i.e. part on the file system, part in the database), so that somebody who gains access to one system doesn't necessarily gain access to the other. It's reasonable to say that this might, in fact, delay an attack, or deter a lazy attacker. Generally speaking however, the place where your application lives has access to both these things anyway so if your application is compromised then you must assume that the data is compromised. (Edit: You mentioned in another comment you have users who a) shouldn't know the passwords stored in the DB, but b) will have access to the DB directly. Believe me, there is a non-zero chance of one of those users getting all of those passwords if you do this. By going down this path, you're putting your faith in a faulty layer of protection.)
tl;dr reversible encryption when storing sensitive data rarely has practical, real security value. If you're trying to tick a compliance check box, and you have no power to overrule someone up the chain who needs you to tick that box, then by all means implement something that "encrypts" the data. But if you're actually trying to secure your system, look elsewhere: here be dragons.
You may check Vault project, a tool for managing secrets:
General Secret Storage
At a bare minimum, Vault can be used for the storage of any secrets.
For example, Vault would be a fantastic way to store sensitive
environment variables, database credentials, API keys, etc.
Compare this with the current way to store these which might be
plaintext in files, configuration management, a database, etc. It
would be much safer to query these using vault read or the API. This
protects the plaintext version of these secrets as well as records
access in the Vault audit log.
Employee Credential Storage
While this overlaps with "General Secret Storage", Vault is a good
mechanism for storing credentials that employees share to access web
services. The audit log mechanism lets you know what secrets an
employee accessed and when an employee leaves, it is easier to roll
keys and understand which keys have and haven't been rolled.
Vault server stores data in encrypted form. Data can be retrieved via command line or REST API. Server must be in unsealed state to return decrypted data - unsealing needs specific number of shards of the master key. Once server is restarted, you need to unseal it again.
Our google app engine app stores a fair amount of personally identifying information (email, ssn, etc) to identify users. I'm looking for advice as to how to secure that data.
My current strategy
Store the sensitive data in two forms:
Hashed - using SHA-2 and a salt
Encrypted - using public/private key RSA
When we need to do look ups:
Do look-ups on the hashed data (hash the PII in a query, compare it to the hashed PII in the datastore).
If we ever need to re-hash the data or otherwise deal with it in a raw form:
Decrypt the encrypted version with our private key. Never store it in raw form, just process it then re-hash & re-encrypt it.
My concerns
Keeping our hash salt secret
If an attacker gets ahold of the data in the datastore, as well as our hash salt, I'm worried they could brute force the sensitive data. Some of it (like SSN, a 9-digit number) does not have a big key space, so even with a modern hash algorithm I believe it could be done if the attacker knew the salt.
My current idea is to keep the salt out of source control and in it's own file. That file gets loaded on to GAE during deployment and the app reads the file when it needs to hash incoming data.
In between deployments the salt file lives on a USB key protected by an angry bear (or a safe deposit box).
With the salt only living in two places
The USB key
Deployed to google apps
and with code download permanently disabled, I can't think of a way for someone to get ahold of the salt without stealing that USB key. Am I missing something?
Keeping our private RSA key secret
Less worried about this. It will be rare that we'll need to decrypt the encrypted version (only if we change the hash algorithm or data format).
The private key never has to touch the GAE server, we can pull down the encrypted data, decrypt it locally, process it, and re-upload the encrypted / hashed versions.
We can keep our RSA private key on a USB stick guarded by a bear AND a tiger, and only bring it out when we need it.
I realize this question isn't exactly google apps specific, but I think GAE makes the situation somewhat unique.
If I had total control, I'd do things like lock down deployment access and access to the datastore viewer with two-factor authentication, but those options aren't available at the moment (Having a GAE specific password is good, but I like having RSA tokens involved).
I'm also neither a GAE expert nor a security expert, so if there's a hole I'm missing or something I'm not thinking of specific to the platform, I would love to hear it.
When deciding on a security architecture, the first thing in your mind should always be threat models. Who are your potential attackers, what are their capabilities, and how can you defend against them? Without a clear idea of your threat model, you've got no way to assess whether or not your proposed security measures are sufficient, or even if they're necessary.
From your text, I'm guessing you're seeking to protect against some subset of the following:
An attacker who compromises your datastore data, but not your application code.
An attacker who obtains access to credentials to access the admin console of your app and can deploy new code.
For the former, encrypting or hashing your datastore data is likely sufficient (but see the caveats later in this answer). Protecting against the latter is tougher, but as long as your admin users can't execute arbitrary code without deploying a new app version, storing your keys in a module that's not checked in to source control, as you suggest, ought to work just fine, since even with admin access, they can't recover the keys, nor can they deploy a new version that reveals the keys to them. Make sure to disable downloading of source, obviously.
You rightly note some concerns about hashing of data with a limited amount of entropy - and you're right to be concerned. To some degree, salts can help with this by preventing precomputation attacks, and key stretching, such as that employed in PBKDF2, scrypt, and bcrypt, can make your attacker's life harder by increasing the amount of work they have to do. However, with something like SSN, your keyspace is simply so small that no amount of key stretching is going to help - if you hash the data, and the attacker gets the hash, they will be able to determine the original SSN.
In such situations, your only viable approach is to encrypt the data with a secret key. Now your attacker is forced to brute-force the key in order to get the data, a challenge that is orders of magnitude harder.
In short, my recommendation would be to encrypt your data using a standard (private key) cipher, with the key stored in a module not in source control. Using hashing instead will only weaken your data, while using public key cryptography doesn't provide appreciable security against any plausible threat model that you don't already have by using a standard cipher.
Of course, the number one way to protect your users' data is to not store it in the first place, if you can. :)
You can increase your hashing algorithm security by using HMAC, a secret key, and a unique salt per entry (I know people will disagree with me on this but it's my belief from my research that it helps avoid certain attacks). You can also use bcrypt or scrypt to hash which will make reversing the hash an extremely time consuming process (but you'll also have to factor this in as time it takes your app to compute the hash).
By disabling code downloads and keeping your secret key protected, I can't imagine how someone can get a hold of it. Just make sure your code is kept protected under similar safe guards or that you remove the secret key from your code during development and only pull it out to deploy. I assume you will keep your secret key in your code (I've heard many people say to keep it in memory to be ultra secure but given the nature of AppEngine and instances, this isn't feasible).
Update:
Be sure to enable 2-factor authentication for all Google accounts that have admin rights to your app. Google offers this so not sure if your restriction for enabling this was imposed by an outside force or not.
Interesting approach to encrypt data on a datastore. After going through this, one question that comes to my mind is how do you query data on your hashes? Are you using comparison of two hashes or more fine grained hashing? Again how do you accomplish operations like greater than value, less than value after hashing and encrypting the data in your table?
Fine grained hashing meaning, do you hash consecutive bytes of a data stream to get the accumulated hash. i.e hash(abcd) = hash(a,b) + hash (b,c) + etc. This type of hashing would tell how similar the underlying data are rather than a match.
For a git repository that is shared with others, is it a vulnerability to expose your database password in the settings.py file? (My initial thought was no, since you still need the ssh password.)
It depends on who has read access to the repository, but it's generally a good idea not to put passwords into version control. It's probably better to put it in a seperate file like password.py with only the password in it, like this:
password = 'asdasd'
and import or execfile this in your settings.py. You can then add the password.py to your .gitignore.
That assumes your database is only accessible from one specific host, and even then, why would you want to give a potential attacker another piece of information? Suppose you deploy this to a shared host and I have an account on there, I could connect to your database just by logging into my account on that box.
Also, depending on who you are writing this for and what kind of auditing they need to go through (PCI, state audits, etc), this might just not be allowed.
I would try to find a way around checking in the password.