I have a flask application and I use a config file with some sensitive information. I was wondering how to deploy my application with the config file without releasing the sensitive information it holds.
TLDR; Create a class to hold your config secrets, store the actual secrets in environment variables on your host machine, and read in the environment variables in your app.
Detailed implementation below.
This is my folder structure:
api
|_cofig
|_config.py
|_app.py
Then inside of my app.py, which actually starts my Flask application, it looks roughly like this (I've excluded everything that doesn't matter).
from config.config import config
def create_app(app_environment=None):
if app_environment is None:
app = Flask(__name__)
app.config.from_object(config[os.getenv('FLASK_ENV', 'dev')])
else:
app = Flask(__name__)
app.config.from_object(config[app_environment])
if __name__ == "__main__":
app = create_app(os.getenv('FLASK_ENV', 'dev'))
app.run()
This allows you to dynamically specify an app environment. For example, you can pass the app environment by setting an environment variable and reading it in before you call create_app(). This is extremely useful if you containerize your Flask app using Docker or some other virtualization tool.
Lastly, my config.py file looks like this. You would change the attributes in each of my environment configs to your secrets.
import os
class ProdConfig:
# Database configuration
API_TOKEN = os.environ.get('PROD_MARKET_STACK_API_KEY_SECRET')
class DevConfig:
# Database configuration
API_TOKEN = os.environ.get('API_KEY_SECRET')
class TestConfig:
# Database configuration
API_TOKEN = os.environ.get('MARKET_STACK_API_KEY')
config = {
'dev': DevConfig,
'test': TestConfig,
'prod': ProdConfig
}
Further, you would access your config secrets throughout any modules in your Flask application via...
from flask import current_app
current_app.config['API_TOKEN']`
I believe the answer to your question may be more related to where your application is being deployed, rather than which web-framework you are using.
As far as I understand, it's a bad practice to store/track sensitive information (passwords and API keys for example) on your source files and you should probably avoid that.
If you have already commited that sensitive data and you want to remove it completely from your git history, I recommend checking this GitHub page.
A couple of high level solutions could be:
Have you config file access environment variables instead of hard coded values.
If you are using a cloud service such as Google Cloud Platform or AWS, you could use a secret manager to store your data and fetch it safely from your app.
Another approach could be storing the information encrypted (maybe with something like KMS), and decrypt it when needed (my least favorite).
I have deployed my flask web app api on azure. I have lot of config files for that I have created a separate directory where I keep all my config files. This is how my project directory looks like
configs
-> app_config.json
-> client_config.json
logs
-> app_debug.log
-> app_error.log
data
-> some other data related files
app.py
app.py is my main python file from which I have imported all the config files and below is how I use it
config_file = os.path.join(os.path.dirname(__file__), 'configs', 'app_config.json')
# Get the config data from config json file
json_data = open(config_file)
config_data = json.load(json_data)
json_data.close()
After this I can easily use config_data anywhere in the code:
mongo_db = connect_mongodb(username=config_data['MongoUsername'], password=config_data['MongoPassword'], url=config_data['MongoDBURL'], port=config_data['Port'], authdbname=config_data['AuthDBName'])
I'm making an app using Google Calendar API, and planning to build it on Heroku.
I have a problem about authentication. Usually I use credential json file for that, but this time I don't want to upload it on Heroku for security reason.
How can I make authentiation on Heroku?
For now, I put my json to an env variable, and use oauth2client's from_json method.
def get_credentials():
credentials_json = os.environ['GOOGLE_APPLICATION_CREDENTIALS']
credentials = GoogleCredentials.from_json(credentials_json)
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
flow.user_agent = APPLICATION_NAME
if flags:
credentials = tools.run_flow(flow, store, flags)
else: # Needed only for compatibility with Python 2.6
credentials = tools.run(flow, store)
print('Storing credentials to ' + credential_path)
return credentials
But this code isn't perfect. If the credentials is invalid, I want the code to write the new credentials to the env variable, not to a new file.
Is there any better way?
I spent an entire day to find the solution because it's tricky.
No matter your language, the solution will be the same.
1 - Declare your env variables from in Heroku dashboard like :
The GOOGLE_CREDENTIALS variable is the content of service account credential JSON file as is.
The GOOGLE_APPLICATION_CREDENTIALS env variable in the string "google-credentials.json"
2 - Once variables are declared, add the builpack from command line :
$ heroku buildpacks:add https://github.com/elishaterada/heroku-google-application-credentials-buildpack
3 - Make a push. Update a tiny thing and push.
4 - The buildpack will automatically generate a google-credentials.json and fill it with the content of the google credentials content.
If you failed at something, it will not work. Check the content of the google-credentials.json with the Heroku bash.
The buildpack mentioned by Maxime Boué is not working anymore because of the Heroku updates(18+). However, below is a similar buildpack which is working. It is actually a fork from the previous one.
Use the below link in the buildpack setting of your Heroku app settings
https://github.com/gerywahyunugraha/heroku-google-application-credentials-buildpack
Define in Config Vars GOOGLE_CREDENTIALS as key and content of your credential file as Value
Define in Config Vars GOOGLE_APPLICATION_CREDENTIALS as Key and google-credentials.json as Value
Redeploy the application, it should work!!
If anyone is still looking for this, I've just managed to get this working for Google Cloud Storage by storing the JSON directly in an env variable (no extra buildpacks).
You'll need to place the json credentials data into your env vars and install google-auth
Then, parse the json and pass google credentials to the storage client:
from google.cloud import storage
from google.oauth2 import service_account
# the json credentials stored as env variable
json_str = os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')
# project name
gcp_project = os.environ.get('GCP_PROJECT')
# generate json - if there are errors here remove newlines in .env
json_data = json.loads(json_str)
# the private_key needs to replace \n parsed as string literal with escaped newlines
json_data['private_key'] = json_data['private_key'].replace('\\n', '\n')
# use service_account to generate credentials object
credentials = service_account.Credentials.from_service_account_info(
json_data)
# pass credentials AND project name to new client object (did not work wihout project name)
storage_client = storage.Client(
project=gcp_project, credentials=credentials)
Hope this helps!
EDIT: Clarified that this was for Google Cloud Storage. These classes will differ for other services, but from the looks of other docs the different Google Client classes should allow the passing of credentials objects.
The recommended buildpack doesn't work anymore. Here's a quick, direct way to do the same:
Set config variables:
heroku config:set GOOGLE_APPLICATION_CREDENTIALS=gcp_key.json
heroku config:set GOOGLE_CREDENTIALS=<CONTENTS OF YOU GCP KEY>
The GOOGLE_CREDENTIALS is easier to set in the Heroku dashboard.
Create a .profile file in your repo with a line to write the json file:
echo ${GOOGLE_CREDENTIALS} > /gcp_key.json
.profile is run every time the container starts.
Obviously, commit the changes to .profile and push to Heroku, which will trigger a rebuild and thus create that gcp_key.json file.
A more official Heroku documentation in this topic: https://elements.heroku.com/buildpacks/buyersight/heroku-google-application-credentials-buildpack
I also used buyersight's buildpack and it was the only one, which worked for me
As state above, the official Heroku documentation works (https://elements.heroku.com/buildpacks/buyersight/heroku-google-application-credentials-buildpack).
For PHP Laravel users though, the config variable GOOGLE_APPLICATION_CREDENTIALS should be set to ../google-credentials.json. Otherwise, PHP will not find the file.
Screenshot from Heroku
I know this is old, but there is another alternative - ie to "split" the json file and store each important field as its own environment variable.
Something like:
PRIVATE_KEY_ID=qsd
PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"
CLIENT_EMAIL=blabla#lalala.iam.gserviceaccount.com
CLIENT_ID=7777
CLIENT_X509_CERT_URL=https://www.googleapis.com/robot/v1/metadata/x509/whatever.iam.gserviceaccount.com
This can be in a .env file locally, and put on heroku using the UI or heroku config:set commands
Then in the python file, you can initialize the ServiceAccount using a dict instead of a JSON
CREDENTIALS = {
"type": "service_account",
"project_id": "iospress",
"private_key_id": os.environ["PRIVATE_KEY_ID"],
"private_key": os.environ["PRIVATE_KEY"],
"client_email": os.environ["CLIENT_EMAIL"],
"client_id": os.environ["CLIENT_ID"],
"auth_uri": "https://accounts.google.com/o/oauth2/auth",
"token_uri": "https://oauth2.googleapis.com/token",
"auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
"client_x509_cert_url": os.environ["CLIENT_X509_CERT_URL"]
}
credentials = ServiceAccountCredentials.from_json_keyfile_dict(CREDENTIALS, SCOPES)
It's a bit more verbose than some of the options presented here, but it works without any buildback or other
In case you do not want to use buildpack
1 - Add env variables in Heroku via dashboard or CLI:
GOOGLE_CREDENTIALS variable is the content of the service account credential JSON file.
GOOGLE_APPLICATION_CREDENTIALS = google-credentials.json
2 - Create a file called .profile on the root of your project with the following content
echo ${GOOGLE_CREDENTIALS} > /app/google-credentials.json
3 - Push your code
4 - During startup, the container starts a bash shell that runs any code in $HOME/.profile before executing the dyno’s command.
Note: For Laravel projects GOOGLE_APPLICATION_CREDENTIALS = ../google-credentials.json
You can use the Heroku Platform API to update Heroku env vars from within your app.
It seems that those buildpacks where you can upload the credentials.json file are not working as expected. I finally managed with Lepinsk's buildpack (https://github.com/lepinsk/heroku-google-cloud-buildpack.git), which requires all keys and values to be set as config vars in Heroku. It does do the job though, so lots of thanks for that!
I have done this like below:
Created two env variable
CREDENTIALS - base64 encoded value of google api credential
CONFIG_FILE_PATH - /app/.gcp/key.json (this file we will create it at run time. in heroku preinstall phase as below)
Create a preinstall script.
"heroku-prebuild": "bash preinstall.sh",
And in preinstall.sh file, decode CREDENTIALS and create a config file and update it there.
if [ "$CREDENTIALS" != "" ]; then
echo "Detected credentials. Adding credentials" >&1
echo "" >&1
# Ensure we have an gcp folder
if [ ! -d ./.gcp ]; then
mkdir -p ./.gcp
chmod 700 ./.gcp
fi
# Load the private key into a file.
echo $GCP_CREDENTIALS | base64 --decode > ./.gcp/key.json
# Change the permissions on the file to
# be read-only for this user.
chmod 400 ./.gcp/key.json
fi
If you're still having an issue running the app after following the buildpack instructions already mentioned in this article, try setting your Heroku environment variable GOOGLE_APPLICATION_CREDENTIALS to the full path instead.
GOOGLE_APPLICATION_CREDENTIALS = /app/google-credentials.json
This worked for me.
Previously, I could see that the google-credentials file was already generated by the buildpack (or .profile) but the Heroku app wasn't finding it, and giving errors as:
Error: Cannot find module './google-credentials.json'
Require stack:
- /app/server/src/config/firebase.js
- /app/server/src/middlewares/authFirebase.js
- /app/server/src/routes/v1/auth.route.js
- /app/server/src/routes/v1/index.js
- /app/server/src/app.js
- /app/server/src/index.js
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
at Module.Hook._require.Module.require (/app/server/node_modules/require-in-the-middle/index.js:61:29)
at require (internal/modules/cjs/helpers.js:93:18)
at Object.<anonymous> (/app/server/src/config/firebase.js:3:24)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
I just have to add this tip, be careful on making GOOGLE_APPLICATION_CREDENTIALS variable in heroku dashborad, it caused me a day, if you have path like that for the credential file: server\credential.json , that will not work because using the backslash , so use slash instead / :
this will work as path (without "):
server/credential.json
The simplest way I've found is to
Save the credentials as a string in a heroku ENV variable
In you app, load them into a Ruby Tempfile
Then have the GoogleDrive::Session.from_service_account_key load them from that temp file
require "tempfile"
...
google_credentials_tempfile = Tempfile.new("credentials.json")
google_credentials_tempfile.write(ENV["GOOGLE_CREDENTIALS_JSON"])
google_credentials_tempfile.rewind
session = GoogleDrive::Session.from_service_account_key(google_credentials_tempfile)
I have this in a heroku app and it works flawlessly.
I am using an Ubuntu 14.04 server instance on AWS and I am having trouble moving a python script that works in the us-east-1 region to the Frankfurt eu-central-1 region. The script is to get the ip-addressess of all of the instances in a particular autoscaling group:
def get_autoscale_instance_ips(region, group):
conn = boto.connect_ec2(region)
as_conn = boto.connect_autoscale()
try:
group = as_conn.get_all_groups(names=[group])[0]
instances_ids = [i.instance_id for i in group.instances]
reservations = conn.get_all_reservations(instances_ids)
instances = [i for r in reservations for i in r.instances]
ip_addresses = [i.private_ip_address for i in instances]
According to the Boto documentation I need to change the autoscale endpoint from the default US region, by putting it in a boto config file either at /etc/boto.cfg for site-wide settings or ~/.boto for user-specific settings.
My config file just contains the following:
[Boto]
autoscale_endpoint = autoscaling.eu-central-1.amazonaws.com
autoscale_region_name = eu-central-1
I have tried both of these options and in both cases the boto config file seems to ignored.
What am I doing wrong, I'm sure I've missed something obvious?
I use Amazon S3 as a part of my webservice. The workflow is the following:
User uploads lots of files to web server. Web server first stores them locally and then uploads to S3 asynchronously
User sends http-request to initiate job (which is some processing of these uploaded files)
Web service asks worker to do the job
Worker does the job and uploads result to S3
User requests the download link from web-server, somedbrecord.result_file.url is returned
User downloads result using this link
To work with files I use QueuedStorage backend. I initiate my FileFields like this:
user_uploaded_file = models.FileField(..., storage=queued_s3storage, ...)
result_file = models.FileField(..., storage=queued_s3storage, ...)
Where queued_s3storage is an object of class derived from ...backends.QueuedStorage and remote field is set to '...backends.s3boto.S3BotoStorage'.
Now I'm planning to deploy the whole system on one machine to run everything locally, I want to replace this '...backends.s3boto.S3BotoStorage' with something based on my local filesystem.
The first workaround was to use FakeS3 which can "emulate" S3 locally. Works, but this is not ideal, just extra unnecessary overhead.
I have Nginx server running and serving static files from particular directories. How do I create my "remote storage" class that actually stores files locally, but provides download links which lead to files served by Nginx? (something like http://myip:80/filedir/file1). Is there a standard library class for that in django?
The default storage backend for media files is local storage.
Your settings.py defines these two environment variables:
MEDIA_ROOT (link to docs) -- this is the absolute path to the local file storage folder
MEDIA_URL (link to docs) -- this is the webserver HTTP path (e.g. '/media/' or '//%s/media' % HOSTNAME
These are used by the default storage backend to save media files. From Django's default/global settings.py:
# Default file storage mechanism that holds media.
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
This configured default storage is used in FileFields for which no storage kwarg is provided. It can also be accessed like so: rom django.core.files.storage import default_storage.
So if you want to vary the storage for local development and production use, you can do something like this:
# file_storages.py
from django.conf import settings
from django.core.files.storage import default_storage
from whatever.backends.s3boto import S3BotoStorage
app_storage = None
if settings.DEBUG == True:
app_storage = default_storage
else:
app_storage = S3BotoStorage()
And in your models:
# models.py
from file_storages import app_storage
# ...
result_file = models.FileField(..., storage=app_storage, ...)
Lastly, you want nginx to serve the files directly from your MEDIA_URL. Just make sure that the nginx URL matches the path in MEDIA_URL.
I'm planning to deploy the whole system on one machine to run everything locally
Stop using QueuedStorage then, because "[QueuedStorage] enables having a local and a remote storage backend" and you've just said you don't want a remote.
Just use FileSystemStorage and configure nginx to serve the location / settings.MEDIA_ROOT
I have a flask app hosted on Heroku that needs to run commands on an AWS EC2 instance (Amazon Linux AMI) using boto.cmdshell. A couple of questions:
Is using a key pair to access the EC2 instance the best practice? Or is using username/password better?
If using a key pair is the preferred method, what's the best practice on managing/storing private keys on Heroku? Obviously putting the private key in git is not an option.
Thanks.
Heroku lets you take advantage of config variables to manage your application. Here is an exmaple of my config.py file that lives inside my flask application:
import os
# flask
PORT = int(os.getenv("PORT", 5000))
basedir = str(os.path.abspath(os.path.dirname(__file__)))
SECRET_KEY = str(os.getenv("APP_SECRET_KEY"))
DEBUG = str(os.getenv("DEBUG"))
ALLOWED_EXTENSIONS = str(os.getenv("ALLOWED_EXTENSIONS"))
TESTING = os.getenv("TESTING", False)
# s3
AWS_ACCESS_KEY_ID = str(os.getenv("AWS_ACCESS_KEY_ID"))
AWS_SECRET_ACCESS_KEY = str(os.getenv("AWS_SECRET_ACCESS_KEY"))
S3_BUCKET = str(os.getenv("S3_BUCKET"))
S3_UPLOAD_DIRECTORY = str(os.getenv("S3_UPLOAD_DIRECTORY"))
Now i can have two different sets of results. It pulls from my Environment variables. One when my application is on my local computer and from Heroku config variables when in production. For example.
DEBUG = str(os.getenv("DEBUG"))
is "TRUE" on my local computer. But False on Heroku. In order to check your Heroku config run.
Heroku config
Also keep in mind that if you ever want to keep some files part of your project locally but not in heroku or on github you can use git ignore. Of course those files won't exist on your production application then.
What I was looking for was guidance on how to deal with private keys. Both #DrewV and #yfeldblum pointed me to the right direction. I ended up turning my private key into a string and storing it in a Heroku config variables.
If anyone is looking to do something similar, here's a sample code snippit using paramiko:
import paramiko, base64
import StringIO
import os
key = paramiko.RSAKey.from_private_key(StringIO.StringIO(str(os.environ.get("AWS_PRIVATE_KEY"))))
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(str(os.environ.get("EC2_PUBLIC_DNS")), username='ec2-user', pkey=key)
stdin, stdout, stderr = ssh.exec_command('ps')
for line in stdout:
print '... ' + line.strip('\n')
ssh.close()
Thanks to #DrewV and #yfeldblum for helping (upvote for both).
You can use config vars to store config items in an application running on Heroku.
You can use a username/password combination. You may make the username something easy; but be sure to generate a strong password, e.g., with something like openssl rand -base64 32.