I am trying to set up Logging for my Flask app which is deployed in ElasticBeanstalk.
I have a basic logging in place that prints stuff in my console.
But on deploying in Beanstalk, I am not able to see any of the application logs.
Is there any specific config that needs to be setup in order to achieve this?
Just sharing a sample code.
user.py
import logging
class User(Resource):
def details(self):
user_info = "Hello"
logging.info(user_info)
You can add a config file to your .ebextensions folder that adds your application's logs to the logs that EB downloads automatically.
Assuming that your application logs to a folder called logs in the Flask app root, you would make a file called .ebextensions/logging.config that looks like this:
files:
"/opt/elasticbeanstalk/tasks/taillogs.d/your_app_name_logs.conf" :
mode: "000644"
owner: root
group: root
content: |
/var/app/current/logs/*.log
(Which is a slight variation on the official documents here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html)
Then, after you deploy, you should be able to use the GUI to pull down logs.
Related
I have written a Python package hwrt (see installation instructions if you want to try it) which serves a website when executed with
$ hwrt serve
2014-12-04 20:27:07,182 INFO * Running on http://127.0.0.1:5000/
2014-12-04 20:27:07,183 INFO * Restarting with reloader
I would like to let it run on http://www.pythonanywhere.com, but when I start it there I get
19:19 ~ $ hwrt serve
2014-12-04 19:19:59,282 INFO * Running on http://127.0.0.1:5000/
Traceback (most recent call last):
File "/home/MartinThoma/.local/bin/hwrt", line 108, in <module>
main(args)
File "/home/MartinThoma/.local/bin/hwrt", line 102, in main
serve.main()
File "/home/MartinThoma/.local/lib/python2.7/site-packages/hwrt/serve.py", line 95, in main
app.run()
File "/usr/local/lib/python2.7/dist-packages/flask/app.py", line 739, in run
run_simple(host, port, self, **options)
File "/usr/local/lib/python2.7/dist-packages/werkzeug/serving.py", line 613, in run_simple
test_socket.bind((hostname, port))
File "/usr/lib/python2.7/socket.py", line 224, in meth
return getattr(self._sock,name)(*args)
socket.error: [Errno 98] Address already in use
I only found this in the documentation:
Flask
never use app.run(), it will break your webapp. Just import the
app into your wsgi file...
By searching for wsgi file, I found mod_wsgi (Apache). However, I don't understand how I can adjust my current minimalistic Flask application to work with that. Currently, the script behind hwrt serve is:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Start a webserver which can record the data and work as a classifier."""
import pkg_resources
from flask import Flask, request, render_template
from flask_bootstrap import Bootstrap
import os
import json
# hwrt modules
import hwrt
import hwrt.utils as utils
def show_results(results, n=10):
"""Show the TOP n results of a classification."""
import nntoolkit
classification = nntoolkit.evaluate.show_results(results, n)
return "<pre>" + classification.replace("\n", "<br/>") + "</pre>"
# configuration
DEBUG = True
template_path = pkg_resources.resource_filename('hwrt', 'templates/')
# create our little application :)
app = Flask(__name__, template_folder=template_path)
Bootstrap(app)
app.config.from_object(__name__)
#app.route('/', methods=['POST', 'GET'])
def show_entries():
heartbeat = request.args.get('heartbeat', '')
return heartbeat
#app.route('/interactive', methods=['POST', 'GET'])
def interactive():
if request.method == 'POST':
raw_data_json = request.form['drawnJSON']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
print(model)
results = utils.evaluate_model_single_recording(model, raw_data_json)
# Show classification page
page = show_results(results, n=10)
page += 'back'
return page
else:
# Page where the user can enter a recording
return render_template('canvas.html')
def get_json_result(results, n=10):
s = []
for res in results[:min(len(results), n)]:
s.append({res['semantics']: res['probability']})
return json.dumps(s)
#app.route('/worker', methods=['POST', 'GET'])
def worker():
# Test with
# wget --post-data 'classify=%5B%5B%7B%22x%22%3A334%2C%22y%22%3A407%2C%22time%22%3A1417704378719%7D%5D%5D' http://127.0.0.1:5000/worker
if request.method == 'POST':
raw_data_json = request.form['classify']
# TODO: Check recording
# TODO: Submit recorded json to database
# Classify
model_path = pkg_resources.resource_filename('hwrt', 'misc/')
model = os.path.join(model_path, "model.tar")
results = utils.evaluate_model_single_recording(model, raw_data_json)
return get_json_result(results, n=10)
else:
# Page where the user can enter a recording
return "Classification Worker (Version %s)" % hwrt.__version__
def get_parser():
"""Return the parser object for this script."""
from argparse import ArgumentParser, ArgumentDefaultsHelpFormatter
parser = ArgumentParser(description=__doc__,
formatter_class=ArgumentDefaultsHelpFormatter)
return parser
def main():
app.run()
if __name__ == '__main__':
main()
Ok, a not so non-sequitur answer to your question is around what mod_wsgi does to interface with your app. A typical flask app would look something like this:
from flask import Flask
app = Flask(__name__)
app.route("/")
def hello():
return "Holy moly that tunnel was bright.. said Bit to NIC"
if __name__ == "__main__":
app.run()
Unfortunately, Apache has no way to know what to do with this (though the app would run happily on its own). In order to get the app and Apache to play nice together we're going to use something called mod_wsgi. What Mod_WSGI does that's important to us, is that it provides a known interface (a file type called wsgi) that's going to wrap our application and initialize it so that we can serve it through Apache.
I'm going to assume you are using a python virtual environment, but if you aren't you can omit the step that deals with this in the instructions below. If you're curious why virtual environments are so great, feel free read about the python ecosystem.
Also - you can include an extra flag (assuming you are running wsgi as a daemon) to automatically reload the daemon whenever you touch or alter your wsgi file. This is quite useful during development and debugging so I'll include is below.
Anyway, let's get started. I'll break this down to steps below.
Configuring Apache for mod_wsgi
Enable mod_wsgi in Apache:
sudo apt-get install libapache2-mod-wsgi
Edit your /etc/apache2/sites-available/<yoursite>.conf.
<VirtualHost interface:port>
WSGIDaemonProcess yourapp user=someUser processes=2 threads=15
WSGIProcessGroup yourapp
# In this case / refers to whatever relative URL path hosts flask
WSGIScriptAlias / /absolute/path/to/yourapp.wsgi
<Directory /path/to/your/main/py/file/ >
# Use good judgement here when server hardening, this assumes dev env
Order allow,deny
Allow from all
Require all granted
#The below enables 'auto-reload' of WSGI
WSGIScriptReloading On
</Directory>
# If you want to serve static files as well and bypass flask in those cases
Alias /relative/url/to/static/content/
<Directory /absolute/path/to/static/root/directory/>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Create your yourapp.wsgi file and put it in the appropriate place: Be wary of file permissions!
#!/usr/bin/python
import sys
import logging
# Activate virtual environment.
# If you are not using venv, skip this.
# But you really should be using it!
activate_this = "/path/to/venv/bin/activate_this.py"
execfile(activate_this, dict(__file__=activate_this))
# Handle logging
logging.basicConfig(stream=sys.stderr)
sys.path.insert(0, "/path/to/your/main/py/file/")
from YourMainPyFileName import app as application
application.secret_key = "your_secret_key"
Reload Apache and troubleshoot problems. I set this up probably every few weeks for a different project or idea I have and... I usually have to fix one thing or another when doing it from scratch. Don't despair though! Flask has great documentation on this.
Once you've done all this you should be at a place where flask runs all on its own. The sample flask app above is the actual code I use to verify everything works whenever I set this up.
This was left here in case it's some use, but is not really directly related to the question...
The answer here is to use x-send-file. This takes advantage of letting Apache do what it's good at (serving static content), while at the same time first letting flask (or other python framework) do it's work first. I do this often to let flask handle my auth layers in single page web apps and have so far been happy with the results.
Doing so requires two things:
First - Enable xsendfile on Apache2 sudo apt-get install libapache2-mod-xsendfile.
Second - Alter your apache2 configuration so allow x-send-file headers:
Alter your conf file in /etc/apache2/sites-available/<yoursite>.conf and add...
XSendFile On
XSendFilePath /path/to/static/directory
This can be entered top level within the <Virtualhost></Virtualhost> tags.
Don't forget to restart Apache sudo service apache2 restart.
Finally - Configure your flask app to use x-send-file in your app.py file:
app.user_x_sendfile = True
Note: Must be done after app initialization. Consequently can also be passed as an initialization parameter.
Flask has documentation on this (excerpt below):
use_x_sendfile
Enable this if you want to use the X-Sendfile feature. Keep in mind that the server has to support this. This only affects files sent with the send_file() method.
New in version 0.2.
This attribute can also be configured from the config with the USE_X_SENDFILE configuration key. Defaults to False.
I ran into a similar issue #moose was having. Getting connection refused and couldnt even telnet localhost 5000.
Turns out theres a ports.conf file i had to add Listen 5000
Happy days.
I would like to write doctests for my pyramid web app, using the webtest module. I tried it like this:
from my_webapp import main
from webtest import TestApp
app = TestApp(main({}))
result = app.get('/')
This raises a KeyError (because some.url is not known) when my code reaches this line:
url = request.registry.settings['some.url']
The value of some.url is specified in the paster ini file of my application. Is there a simple way to use my development.ini when running my test code? I did not yet fully understand how/when the ini file is loaded during pyramid start up, so it's hard to figure out where to load it while testing.
main is invoked with the contents of your ini file. A simple way to load your app from an ini is:
from pyramid.paster import get_app
app = get_app('testing.ini#main')
test_app = TestApp(app)
This expects "testing.ini" to be in the current working directory, so you may need to tweak that. If you'd like it to be relative to a spot in your tree you can use:
import os.path
import some_module
here = os.path.dirname(some_module.__file__)
app = get_app(os.path.join(here, 'testing.ini'))
I've split handlers between 2 python files (main.py and main_cms.py). app.yaml defines the URLs that each python file will handle.
When I look at the Appstats, only the handlers from one of the 2 python files are profiled (the ones from main.py).
The 'magic' of webapp_add_wsgi_middleware(app) always used to work just fine, until the split. How can I make Appstats recording apply to all handlers?
appengine_config.py:
def webapp_add_wsgi_middleware(app):
from google.appengine.ext.appstats import recording
app = recording.appstats_wsgi_middleware(app)
return app
app.yaml:
builtins:
- appstats: on
...
- url: /services/.*
script: main.application
- url: /cms.*
script: main_cms.application
main.py and main_cms.py:
application = webapp2.WSGIApplication(url_mapping, config=config)
Running python2.7 / GAE 1.6.3, the dev_appserver.py shows Appstats correct for all handlers. The described problem is only seen in production.
How do i get a basic web2py server up and running on
PythonAnywhere?
[update - 29/05] We now have a big button on the web tab that will do all this stuff for you. Just click where it says Web2Py, fill in your admin password, and you're good to go.
Here's the old stuff for historical interest...
I'm a PythonAnywhere developer. We're not massive web2py experts (yet?) but I've managed to get web2py up and running like this:
First download and unpack web2py:
wget http://www.web2py.com/examples/static/web2py_src.zip
unzip web2py_src.zip
Go to the PythonAnywhere "Web" panel and edit your wsgi.py. Add these lines:
import os
import sys
path = '/home/my_username/web2py'
if path not in sys.path:
sys.path.append(path)
from wsgihandler import application
replacing my_username with your username.
You will also need to comment out the last two lines in wsgi.py, where we have the default hello world web.py application...
# comment out these two lines if you want to use another framework
#app = web.application(urls, globals())
#application = app.wsgifunc()
Thanks to Juan Martinez for his instructions on this part, which you can view here:
http://web2py.pythonanywhere.com/
then open a Bash console, and cd into the main web2py folder, then run
python web2py.py --port=80
enter admin password
press ctrl-c
(this will generate the parameters_80.py config file)
then go to your Web panel on PythonAnywhere, click reload web app,
and things should work!
You can also simply run this bash script:
http://pastebin.com/zcA5A89k
admin will be disabled because of no HTTPS unless you bypass it as in the previous post. It will create a security vulnerability.
Pastebin was down, I retrieved this from the cache.
cd ~
wget -O web2py_srz.zip http://web2py.com/examples/static/web2py_src.zip
unzip web2py_src.zip
echo "
PATH = '/home/"`whoami`"/web2py'
import os
import sys
sys.stdout = sys.stderr
os.chdir(PATH)
if not './' in sys.path[:1]: sys.path.insert(0,'./')
from gluon.main import wsgibase as application
" > /var/www/wsgi.py
cd web2py
python -c "from gluon.main import save_password; save_password(raw_input('admin password: '),433)"
I have recently summarized my experience with deployment of Web2Py on PythonAnywhere here
Hope it helps
NeoToren
I'll try to add something new to the discussion. The EASIEST way I've found is to go here when you aren't logged in. This makes it so you don't have to mess around with the terminal:
https://www.pythonanywhere.com/try-web2py
Come up with a domain name, then you'll get redirected to a page showing your login information and created dashboard for that domain. From there just create an account so your app isn't erased after 24 hours. When you sign up, your app has a 3 month expiry date (if you're not paying). I believe this is a new policy. Then simply go to https://appname.pythonanywhere.com/admin and then enter the password you were given and then upload your Web2Py file into the dashboard and then visit the page.
I'm not sure how to upload a Web2Py app on PythonAnywhere for an existing account, but that's the easiest method I've found.
I have spent quite a while trying to figure out how to set .env and .flaskenv configuration values in my flask backend in Google Cloud Platform server. I am using apache2, mod_wsgi, Flask, Python 3.6 and SQLAlchemy. My backend works fine locally on my Mac using pure Flask.
Having python-dotenv installed, running the flask command will set environment variables defined in the files .env and .flaskenv. This, however, does not work with wsgi. The request from apache is redirected to execute my run.wsgi-file. There is no mechanism (that I have knowledge about) to set the environment variables defined in .env and .flaskenv.
The minimun requirement is to pass to the application information if test or development environment should be used. From there I could within init.py populate app.config values from an object. However, being somehow able to use config-values from .env and .flaskenv would be far better. I would really appreciate if somebody had any good ideas here - the best practice to set app.config values in wsgi environment.
There are two posts where this same problem has been presented - they really do not have a best practice how to tackle this challenge (and I am sure I am not the only one having a hard time with this):
Why can't Flask can't see my environment variables from Apache (mod_wsgi)?
Apache SetEnv not working as expected with mod_wsgi
My run.wsgi:
import sys
sys.path.append("/var/www/contacts-api/venv/lib/python3.6/site-packages")
sys.path.insert(0,"/var/www/contacts-api/")
from contacts import create_app
app = create_app('settings.py')
app.run()
[3]:Allows you to configure an application using pre-set methods.
from flask_appconfig import AppConfig
def create_app(configfile=None):
app = Flask('myapp')
AppConfig(app, configfile)
return app
The application returned by create_app will, in order:
Load default settings from a module called myapp.default_config, if it exists. (method described in http://flask.pocoo.org/docs/config/#configuring-from-files )
Load settings from a configuration file whose name is given in the environment variable MYAPP_CONFIG (see link from 1.).
Load json or string values directly from environment variables that start with a prefix of MYAPP_, i.e. setting MYAPP_SQLALCHEMY_ECHO=true will cause the setting of SQLALCHEMY_ECHO to be True.
Any of these behaviors can be altered or disabled by passing the appropriate options to the constructor or init_app().
[4]: Using “ENV-only”
If you only want to use the environment-parsing functions of Flask-AppConfig, the appropriate functions are exposed:
from flask_appconfig.heroku import from_heroku_envvars
from flask_appconfig.env import from_envvars
# from environment variables. note that you need to set the prefix, as
# no auto-detection can be done without an app object
from_envvars(app.config, prefix=app.name.upper() + '_')
# also possible: parse heroku configuration values
# any dict-like object will do as the first parameter
from_heroku_envvars(app.config)
After reading more about this and trying many different things. I reached to the conclusion that there is no reasonable way for configuring a Flask-application using .env- and .flaskenv -files. I ended up using a method presented in Configuration Handling which enables managing development/testing/production-environments in a reasonable manner:
app = Flask(__name__)
app.config.from_object('yourapplication.default_settings')
app.config.from_envvar('YOURAPPLICATION_SETTINGS')
My run.wsgi (being used in google cloud platform compute instance):
import sys
import os
from contacts import create_app
sys.path.append("/var/www/myapp/venv/lib/python3.6/site-packages")
sys.path.insert(0,"/var/www/myapp/")
os.environ['SETTINGS_PLATFORM_SPECIFIC'] = "/path/settings_platform_specific.py"
os.environ['CONFIG_ENVIRONMENT'] = 'DevelopmentConfig'
app = create_app(')
app.run()
Locally on my mac I use run.py (for flask run):
import os
from contacts import create_app
os.environ['SETTINGS_PLATFORM_SPECIFIC'] ="/path/settings_platform_specific.py"
os.environ['CONFIG_ENVIRONMENT'] = 'DevelopmentConfig'
if __name__ == '__main__':
app = create_app()
app.run()
For app creation init.py
def create_app():
app = Flask(__name__, template_folder='templates')
app.config.from_object(f'contacts.settings_common.{os.environ.get("CONFIG_ENVIRONMENT")}')
app.config.from_envvar('SETTINGS_PLATFORM_SPECIFIC')
db.init_app(app)
babel.init_app(app)
mail.init_app(app)
bcrypt.init_app(app)
app.register_blueprint(routes)
create_db(app)
return app
At this point it looks like this works out fine for my purposes. The most important thing is that I can easily manage different environments and deploy the backend service to google platform using git.
I was wrestling with the same conundrum, wanting to use the same .env file I was using with docker-compose while developing with flask run. I ended up using python-dotenv, like so:
In .env:
DEBUG=True
APPLICATION_ROOT=${PWD}
In config.py:
import os
from dotenv import load_dotenv
load_dotenv()
class Config(object):
SECRET_KEY = os.getenv('SECRET_KEY') or 'development-secret'
DEBUG = os.getenv("DEBUG") or False
APPLICATION_ROOT = os.getenv("APPLICATION_ROOT") or os.getcwd()
I haven't experimented with it yet, but I may also give flask-env a try in combination with this solution.
The easy to go will be using load_dotenv() and from_mapping
from flask import Flask
from dotenv import load_dotenv , dotenv_values
load_dotenv()
app = Flask(__name__)
config = dotenv_values()
app.config.from_mapping(config)