I just started using Fabric to better control the specific settings for test and deployment environments, and I'm trying to get an idea of the best approach to swapping configurations.
Let's say I have a module in my application that defines a simple database connection and some constants for authentication by default:
host = 'db.host.com'
user = 'someuser'
passw = 'somepass'
db = 'somedb'
class DB():
def __init__(self,host=host,user=user,passw=passw,db=db,cursor='DictCursor'):
#make a database connection here and all that jazz
Before I found fabric, I would use the getfqdn() function from the socket library to check the domain name of the host the system was being pushed to and then conditionalize the authentication credentials.
if getfqdn() == 'test.somedomain.com':
host = 'db.host.com'
user = 'someuser'
passw = 'somepass'
db = 'somedb'
elif getfqdn() == 'test.someotherdomain.com':
host = 'db.other.com'
user = 'otherguy'
passw = 'otherpass'
db = 'somedb'
This, for obvious reasons, is really not that great. What I would like to know is what's the smartest way of adapting something like this in Fabric, so that when the project gets pushed to a certain test/deployment server, these values are changed at post-push.
I can think of a few approaches just from looking through the docs. Should I have a file that just defines the constants that Fabric could output to using the shell commands based off what the deployment was, and then the file defining the database handler could import them? Does it makes sense to run open and write from within the fabfile like this? I assumed I'd also have to .gitignore these kinds of files so they don't get committed into the repo and just rely on Fabric to deploy them.
I plan on adapting whatever approach is the best suggested to all the configuration settings that I currently either swap using getfqdn or adjust manually. Thanks!
You can do all of that off of the env.host and then use something like the contrib template function to render the conf file and push it up. But templates are best in these instances (see: puppet and other config managers as well)
Related
I want to make the setup of my Flask application as easy as possible by providing a custom CLI command for automatically generating SECRET_KEY and saving it to .env. However, it seems that invoking a command, creates an instance of the Flask app, which then needs the not-yet-existing configuration.
Flask documentation says that it's possible to run commands without application context, but it seems like the app is still created, even if the command doesn't have access to it. How can I prevent the web server from running without also preventing the CLI command?
Here's almost-working code. Note, that it requires the python-dotenv package, and it runs with flask run instead of python3 app.py.
import os
import secrets
from flask import Flask, session
app = Flask(__name__)
#app.cli.command(with_appcontext=False)
def create_dotenv():
cwd = os.getcwd()
filename = '.env'
env_path = os.path.join(cwd, filename)
if os.path.isfile(env_path):
print('{} exists, doing nothing.'.format(env_path))
return
with open(env_path, 'w') as f:
secret_key = secrets.token_urlsafe()
f.write('SECRET_KEY={}\n'.format(secret_key))
#app.route('/')
def index():
counter = session.get('counter', 0)
counter += 1
session['counter'] = counter
return 'You have visited this site {} time(s).'.format(counter)
# putting these under if __name__ == '__main__' doesn't work
# because then they are not executed during `flask run`.
secret_key = os.environ.get('SECRET_KEY')
if not secret_key:
raise RuntimeError('No secret key')
app.config['SECRET_KEY'] = secret_key
Some alternative options I considered:
I could allow running the app without a secret key, so the command works fine, but I don't want to make it possible to accidentally run the actual web server without a secret key. That would result in an error 500 when someone tries to use routes that have cookies and it might not be obvious at the time of starting the server.
The app could check the configuration just before the first request comes in as suggested here, but this approach would not be much better than the previous option for the ease of setup.
Flask-Script is also suggested, but Flask-Script itself says it's no longer mainained and points to Flask's built-in CLI tool.
I could use a short delay before killing the application if the configuration is missing, so the CLI command would be able to run, but a missing secret key would be easy to notice when trying to run the server. This would be quite a cursed approach though and who knows maybe even be illegal.
Am I missing something? Is this a bug in Flask or should I do something completely different for automating secret key generation? Is this abusing the Flask CLI system's philosophy? Is it bad practice to generate environment files in the first place?
As a workaround, you can use a separate Python/shell script file for generating SECRET_KEY and the rest of .env.
That will likely be the only script that needs to be able to run without the configuration, so your repository shouldn't get too cluttered from doing so. Just mention the script in README and it probably doesn't result in a noticeably different setup experience either.
I think you are making it more complicated than it should be.
Consider the following code:
import os
app.config['SECRET_KEY'] = os.urandom(24)
It should be sufficient.
(But I prefer to use config files for Flask).
AFAIK the secret key does not have to be permanent, it should simply remain stable for the lifetime of Flask because it will be used for session cookies and maybe some internal stuff but nothing critical to the best of my knowledge.
If your app were interrupted users would lose their session but no big deal.
It depends on your current deployment practices, but if you were using Ansible for example you could automate the creation of the config file and/or the environment variables and also make some sanity checks before starting the service.
The problem with your approach is that as I understand it is that you must give the web server privileges to write to the application directory, which is not optimal from a security POV. The web user should not have the right to modify application files.
So I think it makes sense to take care of deployment separately, either using bash scripts or Ansible, and you can also tighten permissions and automate other stuff.
I agree with precedent answers, anyway you could write something like
secret_key = os.environ.get('SECRET_KEY') or secrets.token_urlsafe(32)
so that you can still use your configured SECRET_KEY variable from environment. If, for any reason, python doesn't find the variable it will be generated by the part after the 'or'.
Hi I'm new to the community and new to Python, experienced but rusty on other high level languages, so my question is simple.
I made a simple script to connect to a private ftp server, and retrieve daily information from it.
from ftplib import FTP
#Open ftp connection
#Connect to server to retrieve inventory
#Open ftp connection
def FTPconnection(file_name):
ftp = FTP('ftp.serveriuse.com')
ftp.login('mylogin', 'password')
#List the files in the current directory
print("Current File List:")
file = ftp.dir()
print(file)
# # #Get the latest csv file from server
# ftp.cwd("/pub")
gfile = open(file_name, "wb")
ftp.retrbinary('RETR '+ file_name, gfile.write)
gfile.close()
ftp.quit()
FTPconnection('test1.csv')
FTPconnection('test2.csv')
That's the whole script, it passes my credentials, and then calls the function FTPconnection on two different files I'm retrieving.
Then my other script that processes them has an import statement, as I tried to call this script as a module, what my import does it's just connect to the FTP server and fetch information.
import ftpconnect as ftpc
This is the on the other Python script, that does the processing.
It works but I want to improve it, so I need some guidance on best practices about how to do this, because in Spyder 4.1.5 I get an 'Module ftpconnect called but unused' warning ... so probably I am missing something here, I'm developing on MacOS using Anaconda and Python 3.8.5.
I'm trying to build an app, to automate some tasks, but I couldn't find anything about modules that guided me to better code, it simply says you have to import whatever .py file name you used and that will be considered a module ...
and my final question is how can you normally protect private information(ftp credentials) from being exposed? This has nothing to do to protect my code but the credentials.
There are a few options for storing passwords and other secrets that a Python program needs to use, particularly a program that needs to run in the background where it can't just ask the user to type in the password.
Problems to avoid:
Checking the password in to source control where other developers or even the public can see it.
Other users on the same server reading the password from a configuration file or source code.
Having the password in a source file where others can see it over your shoulder while you are editing it.
Option 1: SSH
This isn't always an option, but it's probably the best. Your private key is never transmitted over the network, SSH just runs mathematical calculations to prove that you have the right key.
In order to make it work, you need the following:
The database or whatever you are accessing needs to be accessible by SSH. Try searching for "SSH" plus whatever service you are accessing. For example, "ssh postgresql". If this isn't a feature on your database, move on to the next option.
Create an account to run the service that will make calls to the database, and generate an SSH key.
Either add the public key to the service you're going to call, or create a local account on that server, and install the public key there.
Option 2: Environment Variables
This one is the simplest, so it might be a good place to start. It's described well in the Twelve Factor App. The basic idea is that your source code just pulls the password or other secrets from environment variables, and then you configure those environment variables on each system where you run the program. It might also be a nice touch if you use default values that will work for most developers. You have to balance that against making your software "secure by default".
Here's an example that pulls the server, user name, and password from environment variables.
import os
server = os.getenv('MY_APP_DB_SERVER', 'localhost')
user = os.getenv('MY_APP_DB_USER', 'myapp')
password = os.getenv('MY_APP_DB_PASSWORD', '')
db_connect(server, user, password)
Look up how to set environment variables in your operating system, and consider running the service under its own account. That way you don't have sensitive data in environment variables when you run programs in your own account. When you do set up those environment variables, take extra care that other users can't read them. Check file permissions, for example. Of course any users with root permission will be able to read them, but that can't be helped. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.
Option 3: Configuration Files
This is very similar to the environment variables, but you read the secrets from a text file. I still find the environment variables more flexible for things like deployment tools and continuous integration servers. If you decide to use a configuration file, Python supports several formats in the standard library, like JSON, INI, netrc, and XML. You can also find external packages like PyYAML and TOML. Personally, I find JSON and YAML the simplest to use, and YAML allows comments.
Three things to consider with configuration files:
Where is the file? Maybe a default location like ~/.my_app, and a command-line option to use a different location.
Make sure other users can't read the file.
Obviously, don't commit the configuration file to source code. You might want to commit a template that users can copy to their home directory.
Option 4: Python Module
Some projects just put their secrets right into a Python module.
# settings.py
db_server = 'dbhost1'
db_user = 'my_app'
db_password = 'correcthorsebatterystaple'
Then import that module to get the values.
# my_app.py
from settings import db_server, db_user, db_password
db_connect(db_server, db_user, db_password)
One project that uses this technique is Django. Obviously, you shouldn't commit settings.py to source control, although you might want to commit a file called settings_template.py that users can copy and modify.
I see a few problems with this technique:
Developers might accidentally commit the file to source control. Adding it to .gitignore reduces that risk.
Some of your code is not under source control. If you're disciplined and only put strings and numbers in here, that won't be a problem. If you start writing logging filter classes in here, stop!
If your project already uses this technique, it's easy to transition to environment variables. Just move all the setting values to environment variables, and change the Python module to read from those environment variables.
I am using Invoke/Fabric with boto3 to create an AWS instance and hand it over to an Ansible script. In order to do that, a few things have to be prepared on the remote machine before Ansible can take over, notably installing Python, create a user, and copy public SSH keys.
The AWS image comes with a particular user. I would like to use this user only to create my own user, copy public keys, and remove password login afterwards. While using the Fabric CLI the connection object is not created and cannot be modified within tasks.
What would be a good way to switch users (aka recreate a connection object between tasks) and run the following tasks with the user that I just created?
I might not go about it the right way (I am migrating from Fabric 1 where a switch of the env values has been sufficient). Here are a few strategies I am aware of, most of them remove some flexibility we have been relying on.
Create a custom AMI on which all preparations has been done already.
Create a local Connection object within a task for the user setup before falling back to the connection object provided by the Fabric CLI.
Deeper integrate AWS with Ansible (the problem is that we have users that might use Ansible after the instance is alive but don't have AWS privileges).
I guess this list includes also a best practice question.
The AWS image comes with a particular user. I would like to use this user
only to create my own user, copy public keys, and remove password login
afterwards. While using the Fabric CLI the connection object is not created
and cannot be modified within tasks.
I'm not sure this is accurate. I have switched users during the execution of a task just fine. You just have to make sure that all subsequent calls that need the updated env use the execute operation.
e.g.
def create_users():
run('some command')
def some_other_stuff():
run('whoami')
#task
def new_instance():
# provision instance using boto3
env.host = [ ip_address ]
env.user = 'ec2-user'
env.password = 'sesame'
execute(create_users)
env.user = 'some-other-user'
execute(some_other_stuff)
I'm doing the Python Eve tutorials on authentication and authorization. In the settings.py file it specifies this block of code:
# Let's just use the local mongod instance. Edit as needed.
# Please note that MONGO_HOST and MONGO_PORT could very well be
left
# out as they already default to a bare bones local 'mongod'
instance.
MONGO_HOST = 'localhost'
MONGO_PORT = 27017
# Skip these if your db has no auth. But it really should.
MONGO_USERNAME = '<your username>'
MONGO_PASSWORD = '<your password>'
But in the tutorial it sets up a separate 'accounts' collection in Mongo to handle users and accounts. It looks like it wants you to create a separate authentication process just for the python application, and not use Mongo's built in user access features. Is this correct, and is there a reason for this?
It seems to me that hardcoding a Mongo database user + password into a file on a server isn't so secure, and that it'd make more sense to try to use Mongo's own authentication stuff as much as possible. I know pymongo has some tools to work with Mongo users but I wanted to check if there was a good reason for doing it this way before I go diving into that.
I have a python script that I'm running locally which has a password for another application embedded in the os.system call. I obfuscated my password by storing it in a DB that only I have access to and then using windows auth to connect to the DB (Because the script needs to be automated I cant have a prompt for the PW).
With the above said, it occurred to me, couldn't someone just modify my script and print the 'pw' var to obtain my password? I'm working in a shared cloud environment where other developers would have access to my script. Is there any way to abstract it further so someone couldnt just modify my script and get the pw?
import os
import sqlalchemy as sa
import urllib
import pandas as pd
#Specify the databases and servers used for reading and writing data.
read_server = '~~SERVER_HERE~~'
read_database = '~~DATABASE_HERE~~'
#Establish DB Connection
read_params = urllib.quote_plus("DRIVER={SQL Server};SERVER="+read_server+";DATABASE="+read_database+";TRUSTED_CONNECTION=Yes")
read_engine = sa.create_engine("mssql+pyodbc:///?odbc_connect=%s" % read_params)
#Read PW from DB and set to variable
pw_query = """ SELECT DISTINCT PW FROM ~~TABLENAME_HERE~~ """
pw = pd.read_sql_query(pw_query,con=read_engine,index_col=None)
pw = pw.values.tolist()
pw = str(pw[0])
pw = pw.lstrip("['").rstrip("]'")
#Establish connection to server
os.chdir(r"C:\tabcmd\Command Line Utility")
os.system(r'tabcmd login -s https://~~myURL~~ -u tabadmin -p {mypw}'.format(mypw = str(pw)))
#Make sure you update the below workbook, site names, and destination directory.
os.system(r'tabcmd get "~~FILE_Location~~" -f "~~Destination_DIR~~"')
I'm using standard python (Cpython) and MS SQL Server.
There's no real way to protect your password if someone can modify the script.
However, if the shared cloud environment has separate users (i.e logging in via ssh where each person has their own user on the server), then you can change the permissions to restrict access to your code. If not, then I don't think this is possible.
Given you are also hardcoding your database address and access code, nothing prevents others from just connecting to your database for example.
There are ways of obsfuscating your code, but in the end, there is no secure way for storing your password, just ways which require more effort to extract it.
Also see https://crypto.stackexchange.com/questions/19959/is-python-a-secure-programming-language-for-cryptography
TLDR; As long as somebody has access to your program or even source code, the hardcoded password can be extracted - So in your case it would make sense to restrict access to that program.