I want to make the setup of my Flask application as easy as possible by providing a custom CLI command for automatically generating SECRET_KEY and saving it to .env. However, it seems that invoking a command, creates an instance of the Flask app, which then needs the not-yet-existing configuration.
Flask documentation says that it's possible to run commands without application context, but it seems like the app is still created, even if the command doesn't have access to it. How can I prevent the web server from running without also preventing the CLI command?
Here's almost-working code. Note, that it requires the python-dotenv package, and it runs with flask run instead of python3 app.py.
import os
import secrets
from flask import Flask, session
app = Flask(__name__)
#app.cli.command(with_appcontext=False)
def create_dotenv():
cwd = os.getcwd()
filename = '.env'
env_path = os.path.join(cwd, filename)
if os.path.isfile(env_path):
print('{} exists, doing nothing.'.format(env_path))
return
with open(env_path, 'w') as f:
secret_key = secrets.token_urlsafe()
f.write('SECRET_KEY={}\n'.format(secret_key))
#app.route('/')
def index():
counter = session.get('counter', 0)
counter += 1
session['counter'] = counter
return 'You have visited this site {} time(s).'.format(counter)
# putting these under if __name__ == '__main__' doesn't work
# because then they are not executed during `flask run`.
secret_key = os.environ.get('SECRET_KEY')
if not secret_key:
raise RuntimeError('No secret key')
app.config['SECRET_KEY'] = secret_key
Some alternative options I considered:
I could allow running the app without a secret key, so the command works fine, but I don't want to make it possible to accidentally run the actual web server without a secret key. That would result in an error 500 when someone tries to use routes that have cookies and it might not be obvious at the time of starting the server.
The app could check the configuration just before the first request comes in as suggested here, but this approach would not be much better than the previous option for the ease of setup.
Flask-Script is also suggested, but Flask-Script itself says it's no longer mainained and points to Flask's built-in CLI tool.
I could use a short delay before killing the application if the configuration is missing, so the CLI command would be able to run, but a missing secret key would be easy to notice when trying to run the server. This would be quite a cursed approach though and who knows maybe even be illegal.
Am I missing something? Is this a bug in Flask or should I do something completely different for automating secret key generation? Is this abusing the Flask CLI system's philosophy? Is it bad practice to generate environment files in the first place?
As a workaround, you can use a separate Python/shell script file for generating SECRET_KEY and the rest of .env.
That will likely be the only script that needs to be able to run without the configuration, so your repository shouldn't get too cluttered from doing so. Just mention the script in README and it probably doesn't result in a noticeably different setup experience either.
I think you are making it more complicated than it should be.
Consider the following code:
import os
app.config['SECRET_KEY'] = os.urandom(24)
It should be sufficient.
(But I prefer to use config files for Flask).
AFAIK the secret key does not have to be permanent, it should simply remain stable for the lifetime of Flask because it will be used for session cookies and maybe some internal stuff but nothing critical to the best of my knowledge.
If your app were interrupted users would lose their session but no big deal.
It depends on your current deployment practices, but if you were using Ansible for example you could automate the creation of the config file and/or the environment variables and also make some sanity checks before starting the service.
The problem with your approach is that as I understand it is that you must give the web server privileges to write to the application directory, which is not optimal from a security POV. The web user should not have the right to modify application files.
So I think it makes sense to take care of deployment separately, either using bash scripts or Ansible, and you can also tighten permissions and automate other stuff.
I agree with precedent answers, anyway you could write something like
secret_key = os.environ.get('SECRET_KEY') or secrets.token_urlsafe(32)
so that you can still use your configured SECRET_KEY variable from environment. If, for any reason, python doesn't find the variable it will be generated by the part after the 'or'.
Related
POOL = redis.ConnectionPool(host='localhost', port=6379, db=0)
app = Flask(__name__)
#app.route('/get_cohort_curve/', methods=['GET'])```
def get_cohort_curve():
curve = str(request.args.get('curve'))
cohort = str(request.args.get('cohort'))
key = curve+cohort
return get_from_redis(key)
def get_from_redis(key):
try:
my_server = redis.Redis(connection_pool=POOL)
return json.dumps(my_server.get(key))
except Exception, e:
logging.error(e)
app.run()
I need to write unit-tests for this.
How do I test just the route, i.e. a get request goes to the right place?
Do I need to create and destroy instances of the app in the test for each function?
Do I need to create a mock redis connection?
If you are interested in running something in Flask, you could create a virtual environment and test the whole shebang, but in my opinion that is THE HARDEST way to do it.
When I built my site installing Redis locally, setting the port and then depositing some data inside it with an appropriate key was essential. I did all of my development in iPython (jupyter) notebooks so that I could test the functions and interactions with Redis before adding the confounding layer of Flask.
Then you set up a flawless template, solid HTML around it and CSS to style the site. If it works without data as an html page, then you move on to the python part of Flask.
Solidify the Flask directories. Make sure that you house them in a virtual environment so that you can call them from your browser and it will treat your virtual environment as a server.
You create your app.py application. I created each one of the page functions one at a time. tested to see that it was properly pushing the variables to post on the page and calling the right template. After you get on up and running right, with photos and data, then at the next page's template, using #app.route
Take if very slow, one piece at a time with debugging on so that you can see where when and how you are going wrong. You will only get the site to run with redis server on and your virtual environment running.
Then you will have to shut down the VE to edit and reboot to test. At first it is awful, but over time it becomes rote.
EDIT :
If you really want to test just the route, then make an app.py with just the #app.route definition and return just the page (the template you are calling). You can break testing into pieces as small as you like, but you have to be sure that the quanta you pick are executable as either a python script in a notebook or commandline execution or as a compact functional self-contained website....unless you use the package I mentioned in the comment: Flask Unit Testing Applications
And you need to create REAL Redis connections or you will error out.
I'm having an issue with my application on Heroku where sessions aren't persisting. Specifically, flask's SecureCookieSession object is empty, every time a request is made. Contrast this with running my application on localhost, where the contents of SecureCookieSession persist the way they should.
Also I'm using flask-login + flask-seasurf, but I'm pretty sure the issue happening somewhere between flask / gunicorn / heroku.
Here are three questions that describe a similar issue:
Flask sessions not persisting on heroku
Flask session not persisting
Flask-Login and Heroku issues
Except I'm not dealing with AJAX or multiple workers here (it's a single heroku free dyno, with a single line in the Procfile). I do get the feeling that using server side sessions with redis or switching from Heroku to something like EC2 might solve my problem though.
Also, here's my git repo if it helps https://gitlab.com/collectqt/quirell/tree/develop. And I'm testing session stuff with
def _before_request(self):
LOG.debug('SESSION REQUEST '+str(flask.session))
def _after_request(self, response):
LOG.debug('SESSION RESPONSE '+str(flask.session))
return response
Got the solved with some external help, mainly by changing the secret key to use a random string I came up with, instead of os.urandom(24)
Changing to server side redis sessions helped too, if only by making testing simpler
Just in case someone else comes across this question, check APPLICATION_ROOT configuration variable. I recently deployed a Flask application to a subdirectory under nginx with a reverse-proxy and setting the APPLICATION_ROOT variable broke Flask's session. Cookies aren't being set under the correct path because of that.
We have a hosting setup where we have one top level domain, and we host web applications under subpaths. For instance:
/projects -> Plone
/interal -> Tomcat
etc
In this scenario we need a way to tell the web application at the back end what its base path is, so that it can correctly generate links to its views and static content. For the examples above this is fine.
We have just started using Pyramid, served by Waitress, but so far we've not figure out how to do this. Is there a clean way to configure this base path in Waitress, or is there a more flexible application server that we can use that will support Pyramid?
Everything in WSGI is relative to the current request. You just have to have your environ setup properly (usually by your WSGI server).
For example your web application will know it is mounted at subpath /projects if request.environ['SCRIPT_NAME'] == '/projects'. If you want your application to be agnostic to its mount point, you can simply code it up as if it serves a view at /foo/bar. Then you mount your application on /projects via some middleware which can mutate the environ properly (mod_wsgi and some other servers should be able to do this for you automatically). Now when the incoming URL is /projects/foo/bar the environ['SCRIPT_NAME'] == '/projects' and environ['PATH_INFO'] == '/foo/bar', and your app can focus on the relative path.
In Pyramid this would boil down to an extra step in your ini where you add the prefix middleware to your WSGI stack. The middleware handles mutating the PATH_INFO and SCRIPT_NAME keys in the environ for you.
[app:myapp]
use = egg:myapp
# ...
[filter:proxy-prefix]
use = egg:PasteDeploy#prefix
prefix = /projects
[pipeline:main]
pipeline =
proxy-prefix
myapp
In my pyramid app, in the .ini config files (production and development) I'm doing something like this:
filter-with = urlprefix
[filter:urlprefix]
use = egg:PasteDeploy#prefix
prefix = /mysubfolder
I think it probably accomplishes the same as Michael's answer above; I'm still relatively new to Pyramid as well and am going off of recipes like you. But the end result is that it creates a base URL of /mysubfolder from my root and the rest of the app is relative to that. This is running under pserve locally and I think nginix on my web host.
repoze.vhm should work just fine for your use case.
I think it won't work if you want to use the virtual root feature. I.e a subpath of your proxied web app (https://hidden.tld/root/ should appear as https://example.com/ )
For exposing your app at a subpath of an external domain repoze.vhm works just fine. IMO the best thing about it is, that you don't need to put any subpath config or whatsoever into your web app deployment. This allows you to change the url to whatever you want on the proxy, or even expose the same app instance on multiple domain names and/or subpaths.
I'm on an app engine project where I'd like to put in a link to a Javascript test runner that I'd like to only exist when running the development server. I've made some experiments on a local shell with configuration loaded using the technique found in NoseGAE versus live on the 'App Engine Console' [1] and it looks to me like a distinction btw real instance and dev server is the presence of the module google.appengine.tools. Which lead me to this utility function:
def is_dev():
"""
Tells us if we're running under the development server or not.
:return:
``True`` if the code is running under the development server.
"""
try:
from google.appengine import tools
return True
except ImportError:
return False
The question (finally!) would be: is this a bad idea? And in that case, can anyone suggest a better approach?
[1] http://con.appspot.com/console/ (try it! very handy indeed)
The standard way to test for the development server is as follows:
DEBUG = os.environ['SERVER_SOFTWARE'].startswith("Dev")
Relying on the existence or nonexistence of a particular module - especially an undocumented one - is probably a bad idea.
I'd recommend doing it this way:
import os
def onDevServer():
return os.environ['SERVER_SOFTWARE'].find('Development') >= 0
This looks at the environment you're running in, and returns true if you're running on the development server. However, its a much cleaner way than checking an import, in my opinion.
I'm not a google app developer, but I wouldn't make this 100% dynamic, but also look at a value from a config file. I'm pretty sure you will be running into the problem, that you want to see this console on the prod system (google servers) or run your local version without the dev code (for testing).
To sum it up: Such a logic is fine for small stuff, like adding a debug link, but provide a way to overwrite it (e.g. by a configuration value)
How can I be certain that my application is running on development server or not? I suppose I could check value of settings.DEBUG and assume if DEBUG is True then it's running on development server, but I'd prefer to know for sure than relying on convention.
I put the following in my settings.py to distinguish between the standard dev server and production:
import sys
RUNNING_DEVSERVER = (len(sys.argv) > 1 and sys.argv[1] == 'runserver')
This also relies on convention, however.
(Amended per Daniel Magnusson's comment)
server = request.META.get('wsgi.file_wrapper', None)
if server is not None and server.__module__ == 'django.core.servers.basehttp':
print('inside dev')
Of course, wsgi.file_wrapper might be set on META, and have a class from a module named django.core.servers.basehttp by extreme coincidence on another server environment, but I hope this will have you covered.
By the way, I discovered this by making a syntatically invalid template while running on the development server, and searched for interesting stuff on the Traceback and the Request information sections, so I'm just editing my answer to corroborate with Nate's ideas.
Usually this works:
import sys
if 'runserver' in sys.argv:
# you use runserver
Typically I set a variable called environment and set it to "DEVELOPMENT", "STAGING" or "PRODUCTION". Within the settings file I can then add basic logic to change which settings are being used, based on environment.
EDIT: Additionally, you can simply use this logic to include different settings.py files that override the base settings. For example:
if environment == "DEBUG":
from debugsettings import *
Relying on settings.DEBUG is the most elegant way AFAICS as it is also used in Django code base on occasion.
I suppose what you really want is a way to set that flag automatically without needing you update it manually everytime you upload the project to production servers.
For that I check the path of settings.py (in settings.py) to determine what server the project is running on:
if __file__ == "path to settings.py in my development machine":
DEBUG = True
elif __file__ in [paths of production servers]:
DEBUG = False
else:
raise WhereTheHellIsThisServedException()
Mind you, you might also prefer doing this check with environment variables as #Soviut suggests. But as someone developing on Windows and serving on Linux checking the file paths was plain easier than going with environment variables.
I came across this problem just now, and ended up writing a solution similar to Aryeh Leib Taurog's. My main difference is that I want to differentiate between a production and dev environments when running the server, but also when running some one-off scripts for my app (which I run like DJANGO_SETTINGS_MODULE=settings python [the script] ). In this case, simply looking at whether argv[1] == runserver isn't enough. So what I came up with is to pass an extra command-line argument when I run the devserver, and also when I run my scripts, and just look for that argument in settings.py. So the code looks like this:
if '--in-development' in sys.argv:
## YES! we're in dev
pass
else:
## Nope, this is prod
pass
then, running the django server becomes
python manage.py runserver [whatever options you want] --in-development
and running my scripts is as easy as
DJANGO_SETTINGS_MODULE=settings python [myscript] --in-development
Just make sure the extra argument you pass along doens't conflict with anything django (in reality I use my app's name as part of the argument).
I think this is pretty decent, as it lets me control exactly when my server and scripts will behave as prod or dev, and I'm not relying on anyone else's conventions, other than my own.
EDIT: manage.py complains if you pass unrecognized options, so you need to change the code in settings.py to be something like
if sys.argv[0] == 'manage.py' or '--in-development' in sys.argv:
# ...
pass
Although this works, I recognize it's not the most elegant of solutions...
If you want to switch your settings files automatically dependent on the runtime environment
you could just use something that differs in environ, e.g.
from os import environ
if environ.get('_', ''):
print "This is dev - not Apache mod_wsgi"
You can determine whether you're running under WSGI (mod_wsgi, gunicorn, waitress, etc.) vs. manage.py (runserver, test, migrate, etc.) or anything else:
import sys
WSGI = 'django.core.wsgi' in sys.modules
settings.DEBUG could be True and running under Apache or some other non-development server. It will still run. As far as I can tell, there is nothing in the run-time environment short of examining the pid and comparing to pids in the OS that will give you this information.
I use:
DEV_SERVERS = [
'mymachine.local',
]
DEVELOPMENT = platform.node() in DEV_SERVERS
which requires paying attention to what is returned by .node() on your machines. It's important that the default be non-development so that you don't accidentally expose sensitive development information.
You could also look into more complicated ways of uniquely identifying computers.
One difference between the development and deployment environment is going to be the server that it’s running on. What exactly is different will depend on your dev and deployment environments.
Knowing your own dev and deploy environments, the HTTP request variables could be used to distinguish between the two. Look at request variables like request.META.HTTP_HOST, request.META.SERVER_NAME and request.META.SERVER_PORT and compare them in the two environments.
I bet you’ll find something quite obvious that’s different and can be used to detect your development environment. Do the test in settings.py and set a variable that you can use elsewhere.
Inspired by Aryeh's answer, the trick I devised for my own use is to just look for the name of my management script in sys.argv[0]:
USING_DEV_SERVER = "pulpdist/manage_site.py" in sys.argv[0]
(My use case is to automatically enable Django native authentication when running the test server - when running under Apache, even on development servers, all authentication for my current project is handled via Kerberos)
You could check request.META["SERVER_SOFTWARE"] value:
dev_servers = ["WSGIServer", "Werkzeug"]
if any(server in request.META["SERVER_SOFTWARE"] for server in dev_servers):
print("is local")
Simple you may check the path you work on server. Something like:
import os
SERVER = True if os.path.exists('/var/www/your_project') else False