I have a python program running in a Docker container. My authentication method depends on whether the container is deployed in GCP or not. Ideally I'd have a function like this:
def deployment_environment():
# return 'local' if [some test] else 'cloud'
pass
What's the most idiomatic way of checking this? My instinct is to use env named [APP_NAME]_DEPLOYMENT_ENVIRONMENT which gets set either way -- but making sure this is set correctly has too many moving parts. Is there a GCP package/tool which can check for me?
There are two solutions I've arrived at:
With env
Set an env var when deploying, like so:
gcloud functions deploy [function-name] --set-env-vars ENV_GCP=1
Then, in your code:
import socket
def deployment_environment():
return 'cloud' if ('ENV_GCP' in os.environ) else 'local'
Pros
Cons
intent is clear, both setting and using env
more involved
idiomatic
relies on user setting env correctly
Via Python, with Sockets
import socket
def deployment_environment():
try:
socket.getaddrinfo('metadata.google.internal', 80)
return 'cloud'
except socket.gaierror:
return 'local'
Pros
Cons
more succinct
makes improper use of try/catch
doesn't rely on an extra step of setting env
dependency on socket package & GCP runtime contract
Related
I want to make the setup of my Flask application as easy as possible by providing a custom CLI command for automatically generating SECRET_KEY and saving it to .env. However, it seems that invoking a command, creates an instance of the Flask app, which then needs the not-yet-existing configuration.
Flask documentation says that it's possible to run commands without application context, but it seems like the app is still created, even if the command doesn't have access to it. How can I prevent the web server from running without also preventing the CLI command?
Here's almost-working code. Note, that it requires the python-dotenv package, and it runs with flask run instead of python3 app.py.
import os
import secrets
from flask import Flask, session
app = Flask(__name__)
#app.cli.command(with_appcontext=False)
def create_dotenv():
cwd = os.getcwd()
filename = '.env'
env_path = os.path.join(cwd, filename)
if os.path.isfile(env_path):
print('{} exists, doing nothing.'.format(env_path))
return
with open(env_path, 'w') as f:
secret_key = secrets.token_urlsafe()
f.write('SECRET_KEY={}\n'.format(secret_key))
#app.route('/')
def index():
counter = session.get('counter', 0)
counter += 1
session['counter'] = counter
return 'You have visited this site {} time(s).'.format(counter)
# putting these under if __name__ == '__main__' doesn't work
# because then they are not executed during `flask run`.
secret_key = os.environ.get('SECRET_KEY')
if not secret_key:
raise RuntimeError('No secret key')
app.config['SECRET_KEY'] = secret_key
Some alternative options I considered:
I could allow running the app without a secret key, so the command works fine, but I don't want to make it possible to accidentally run the actual web server without a secret key. That would result in an error 500 when someone tries to use routes that have cookies and it might not be obvious at the time of starting the server.
The app could check the configuration just before the first request comes in as suggested here, but this approach would not be much better than the previous option for the ease of setup.
Flask-Script is also suggested, but Flask-Script itself says it's no longer mainained and points to Flask's built-in CLI tool.
I could use a short delay before killing the application if the configuration is missing, so the CLI command would be able to run, but a missing secret key would be easy to notice when trying to run the server. This would be quite a cursed approach though and who knows maybe even be illegal.
Am I missing something? Is this a bug in Flask or should I do something completely different for automating secret key generation? Is this abusing the Flask CLI system's philosophy? Is it bad practice to generate environment files in the first place?
As a workaround, you can use a separate Python/shell script file for generating SECRET_KEY and the rest of .env.
That will likely be the only script that needs to be able to run without the configuration, so your repository shouldn't get too cluttered from doing so. Just mention the script in README and it probably doesn't result in a noticeably different setup experience either.
I think you are making it more complicated than it should be.
Consider the following code:
import os
app.config['SECRET_KEY'] = os.urandom(24)
It should be sufficient.
(But I prefer to use config files for Flask).
AFAIK the secret key does not have to be permanent, it should simply remain stable for the lifetime of Flask because it will be used for session cookies and maybe some internal stuff but nothing critical to the best of my knowledge.
If your app were interrupted users would lose their session but no big deal.
It depends on your current deployment practices, but if you were using Ansible for example you could automate the creation of the config file and/or the environment variables and also make some sanity checks before starting the service.
The problem with your approach is that as I understand it is that you must give the web server privileges to write to the application directory, which is not optimal from a security POV. The web user should not have the right to modify application files.
So I think it makes sense to take care of deployment separately, either using bash scripts or Ansible, and you can also tighten permissions and automate other stuff.
I agree with precedent answers, anyway you could write something like
secret_key = os.environ.get('SECRET_KEY') or secrets.token_urlsafe(32)
so that you can still use your configured SECRET_KEY variable from environment. If, for any reason, python doesn't find the variable it will be generated by the part after the 'or'.
Trying to find out if there is a programmatic way to know, when running a cloud function, if it's running in dev (locally with functions-framework) or in prod (deployed).
E.g. in Google AppEngine, we could know if it was running in dev or prod by:
if os.getenv('SERVER_SOFTWARE', '').startswith('Google App Engine'):
# Running in prod!
else:
# Running in dev!
Is there anything similar with GCF?
Thanks!
So, I found this blog post: https://rominirani.com/google-cloud-functions-tutorial-using-environment-variables-20b4f0f82aa0
Turns out, there's lots of env vars in the GCF environment when it deploys. Now, having said that, I don't know how many of those are present in the test framework but I suspect not many of them. So you could rely on checking for the presence of one of the env vars that's in the production deployment and isn't present in the dev.
e.g. FUNCTION_REGION is one of the vars that probably isn't set up for the framework since the framework likely doesn't care what region it's in.
It's a bit hacky, but it'll work.
Even hackier, if you wanted, you could when you deploy the GCF, set your own env var that you rely on (that way, in the case of any of those vars changing over time, you're still safe).
I am writing some functions, using paramiko, to execute commands and create files on a remote host. I would like to write some unit tests for them, but I don't know what would be the simplest way to achieve this? This is what I envisage as being an example outline of my code:
import os
import paramiko
import pytest
def my_function(hostname, relpath='.', **kwargs):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname, **kwargs)
sftp = ssh.open_sftp()
sftp.chdir(relpath)
stdin, stdout, stderr = ssh.exec_command("echo hallo > test.txt")
#pytest.fixture("module")
def mock_remote_host():
# start a remote host here with a local test path
try:
yield hostname, testpath, {"username":"bob", "password":"1234"}
finally:
# delete the test path
# close the remote host
def test_my_function(mock_remote_host):
hostname, dirpath, kwargs = mock_remote_host
my_function(hostname, **kwargs)
filepath = os.path.join(dirpath, 'test.txt')
assert os.path.exists(filepath)
I have had a look at the paramiko test modules, but they seem quite complex for my use case and I'm not sure how to go about simplifying them.
I think what you really need to mock is paramiko.SSHClientobject. You are unittesting your function my_function, you can assume paramiko module works correctly and the only thing you need to unit test is if my_function calls methods of this paramiko.SSHClient in correct way.
To mock paramiko.SSH module you can use unittest.mock and decorate your test_my_function function with #mock.patch.object(paramiko.SSHClient, sshclientmock). You have to define sshclientmock as some kind of Mock or MagicMock first.
Also in python 2.7 there is some equivalent of unittest.mock but I dont remember where to find it exactly.
EDIT: As #chepner mentioned in comment. For python 2.7 you can find mock module in pypi and install it using pip install mock
To answer my own question, I have created: https://github.com/chrisjsewell/atomic-hpc/tree/master/atomic_hpc/mockssh.
As the readme discusses; it is based on https://github.com/carletes/mock-ssh-server/tree/master/mockssh with additions made (to implement more sftp functions) based on https://github.com/rspivak/sftpserver
The following changes have also been made:
revised users parameter, such that either a private_path_key or password can be used
added a dirname parameter to the Server context manager, such that the this will be set as the root path for the duration of the context.
patched paramiko.sftp_client.SFTPClient.chdir to fix its use with relative paths.
See test_mockssh.py for example uses.
If you want to test remote connectivity, remote filesystem structure and remote path navigation you have to set-up a mock host server (a VM maybe). In other words if you want to test your actions on the host you have to mock the host.
If you want to test your actions with the data of the host the easiest way seems to proceed as running.t said in the other answer.
I agree with HraBal, because of "Infrastructure as code". You can treat virtual machine as a block of code.
For example:
you can use vagrant or docker to initialize a SSH server and then, and modify your DNS configuration file. target domain 127.0.0.1
put application into the server. and run paramiko to connect target domain and test what you want.
I think it is the benefit that you can do this for all programming languages and not need to reinvent the wheel . In addition, you and your successors will know the detail of the system.
(My English is not very good)
I'm on an app engine project where I'd like to put in a link to a Javascript test runner that I'd like to only exist when running the development server. I've made some experiments on a local shell with configuration loaded using the technique found in NoseGAE versus live on the 'App Engine Console' [1] and it looks to me like a distinction btw real instance and dev server is the presence of the module google.appengine.tools. Which lead me to this utility function:
def is_dev():
"""
Tells us if we're running under the development server or not.
:return:
``True`` if the code is running under the development server.
"""
try:
from google.appengine import tools
return True
except ImportError:
return False
The question (finally!) would be: is this a bad idea? And in that case, can anyone suggest a better approach?
[1] http://con.appspot.com/console/ (try it! very handy indeed)
The standard way to test for the development server is as follows:
DEBUG = os.environ['SERVER_SOFTWARE'].startswith("Dev")
Relying on the existence or nonexistence of a particular module - especially an undocumented one - is probably a bad idea.
I'd recommend doing it this way:
import os
def onDevServer():
return os.environ['SERVER_SOFTWARE'].find('Development') >= 0
This looks at the environment you're running in, and returns true if you're running on the development server. However, its a much cleaner way than checking an import, in my opinion.
I'm not a google app developer, but I wouldn't make this 100% dynamic, but also look at a value from a config file. I'm pretty sure you will be running into the problem, that you want to see this console on the prod system (google servers) or run your local version without the dev code (for testing).
To sum it up: Such a logic is fine for small stuff, like adding a debug link, but provide a way to overwrite it (e.g. by a configuration value)
How can I be certain that my application is running on development server or not? I suppose I could check value of settings.DEBUG and assume if DEBUG is True then it's running on development server, but I'd prefer to know for sure than relying on convention.
I put the following in my settings.py to distinguish between the standard dev server and production:
import sys
RUNNING_DEVSERVER = (len(sys.argv) > 1 and sys.argv[1] == 'runserver')
This also relies on convention, however.
(Amended per Daniel Magnusson's comment)
server = request.META.get('wsgi.file_wrapper', None)
if server is not None and server.__module__ == 'django.core.servers.basehttp':
print('inside dev')
Of course, wsgi.file_wrapper might be set on META, and have a class from a module named django.core.servers.basehttp by extreme coincidence on another server environment, but I hope this will have you covered.
By the way, I discovered this by making a syntatically invalid template while running on the development server, and searched for interesting stuff on the Traceback and the Request information sections, so I'm just editing my answer to corroborate with Nate's ideas.
Usually this works:
import sys
if 'runserver' in sys.argv:
# you use runserver
Typically I set a variable called environment and set it to "DEVELOPMENT", "STAGING" or "PRODUCTION". Within the settings file I can then add basic logic to change which settings are being used, based on environment.
EDIT: Additionally, you can simply use this logic to include different settings.py files that override the base settings. For example:
if environment == "DEBUG":
from debugsettings import *
Relying on settings.DEBUG is the most elegant way AFAICS as it is also used in Django code base on occasion.
I suppose what you really want is a way to set that flag automatically without needing you update it manually everytime you upload the project to production servers.
For that I check the path of settings.py (in settings.py) to determine what server the project is running on:
if __file__ == "path to settings.py in my development machine":
DEBUG = True
elif __file__ in [paths of production servers]:
DEBUG = False
else:
raise WhereTheHellIsThisServedException()
Mind you, you might also prefer doing this check with environment variables as #Soviut suggests. But as someone developing on Windows and serving on Linux checking the file paths was plain easier than going with environment variables.
I came across this problem just now, and ended up writing a solution similar to Aryeh Leib Taurog's. My main difference is that I want to differentiate between a production and dev environments when running the server, but also when running some one-off scripts for my app (which I run like DJANGO_SETTINGS_MODULE=settings python [the script] ). In this case, simply looking at whether argv[1] == runserver isn't enough. So what I came up with is to pass an extra command-line argument when I run the devserver, and also when I run my scripts, and just look for that argument in settings.py. So the code looks like this:
if '--in-development' in sys.argv:
## YES! we're in dev
pass
else:
## Nope, this is prod
pass
then, running the django server becomes
python manage.py runserver [whatever options you want] --in-development
and running my scripts is as easy as
DJANGO_SETTINGS_MODULE=settings python [myscript] --in-development
Just make sure the extra argument you pass along doens't conflict with anything django (in reality I use my app's name as part of the argument).
I think this is pretty decent, as it lets me control exactly when my server and scripts will behave as prod or dev, and I'm not relying on anyone else's conventions, other than my own.
EDIT: manage.py complains if you pass unrecognized options, so you need to change the code in settings.py to be something like
if sys.argv[0] == 'manage.py' or '--in-development' in sys.argv:
# ...
pass
Although this works, I recognize it's not the most elegant of solutions...
If you want to switch your settings files automatically dependent on the runtime environment
you could just use something that differs in environ, e.g.
from os import environ
if environ.get('_', ''):
print "This is dev - not Apache mod_wsgi"
You can determine whether you're running under WSGI (mod_wsgi, gunicorn, waitress, etc.) vs. manage.py (runserver, test, migrate, etc.) or anything else:
import sys
WSGI = 'django.core.wsgi' in sys.modules
settings.DEBUG could be True and running under Apache or some other non-development server. It will still run. As far as I can tell, there is nothing in the run-time environment short of examining the pid and comparing to pids in the OS that will give you this information.
I use:
DEV_SERVERS = [
'mymachine.local',
]
DEVELOPMENT = platform.node() in DEV_SERVERS
which requires paying attention to what is returned by .node() on your machines. It's important that the default be non-development so that you don't accidentally expose sensitive development information.
You could also look into more complicated ways of uniquely identifying computers.
One difference between the development and deployment environment is going to be the server that it’s running on. What exactly is different will depend on your dev and deployment environments.
Knowing your own dev and deploy environments, the HTTP request variables could be used to distinguish between the two. Look at request variables like request.META.HTTP_HOST, request.META.SERVER_NAME and request.META.SERVER_PORT and compare them in the two environments.
I bet you’ll find something quite obvious that’s different and can be used to detect your development environment. Do the test in settings.py and set a variable that you can use elsewhere.
Inspired by Aryeh's answer, the trick I devised for my own use is to just look for the name of my management script in sys.argv[0]:
USING_DEV_SERVER = "pulpdist/manage_site.py" in sys.argv[0]
(My use case is to automatically enable Django native authentication when running the test server - when running under Apache, even on development servers, all authentication for my current project is handled via Kerberos)
You could check request.META["SERVER_SOFTWARE"] value:
dev_servers = ["WSGIServer", "Werkzeug"]
if any(server in request.META["SERVER_SOFTWARE"] for server in dev_servers):
print("is local")
Simple you may check the path you work on server. Something like:
import os
SERVER = True if os.path.exists('/var/www/your_project') else False