I am attempting to deploy a Flask app to Heroku. I'm using Peewee as an ORM for a Postgres database. When I follow the standard Heroku steps to deploying Flask, the web process crashes after I enter heroku ps:scale web=1. Here's what the logs say:
Starting process with command `python app.py`
/app/.heroku/venv/lib/python2.7/site-packages/peewee.py:2434: UserWarning: Table for <class 'flask_peewee.auth.User'> ("user") is reserved, please override using Meta.db_table
cls, _meta.db_table,
Traceback (most recent call last):
File "app.py", line 167, in <module>
auth.User.create_table(fail_silently=True)
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 2518, in create_table if fail_silently and cls.table_exists():
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 2514, in table_exists return cls._meta.db_table in cls._meta.database.get_tables()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 507, in get_tables ORDER BY c.relname""")
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 313, in execute cursor = self.get_cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 310, in get_cursor return self.get_conn().cursor()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 306, in get_conn self.connect()
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 296, in connect self.__local.conn = self.adapter.connect(self.
database, **self.connect_kwargs)
File "/app/.heroku/venv/lib/python2.7/site-packages/peewee.py", line 199, in connect return psycopg2.connect(database=database, **kwargs)
File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 179, in connect connection_factory=connection_factory, async=async)
psycopg2.OperationalError: could not connect to server: No such file or directory
Is the server running locally and accepting connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Process exited with status 1
State changed from starting to crashed
I've tried a bunch of different things to get Heroku to allow my app to talk to a Postgres db, but haven't had any luck. Is there an easy way to do this? What do I need to do to configure Flask/Peewee so that I can use a db on Heroku?
According to the Peewee docs, you don't want to use Proxy() unless your local database driver is different than your remote one (i.e. locally, you're using SQLite and remotely you're using Postgres). If, however, you are using Postgres both locally and remotely it's a much simpler change. In this case, you'll want to only change the connection values (database name, username, password, host, port, etc.) at runtime and do not need to use Proxy().
Peewee has a built-in URL parser for database connections. Here's how to use it:
import os
from peewee import *
from playhouse.db_url import connect
db = connect(os.environ.get('DATABASE_URL'))
class BaseModel(Model):
class Meta:
database = db
In this example, peewee's db_url module reads the environment variable DATABASE_URL and parses it to extract the relevant connection variables. It then creates a PostgresqlDatabase object with those values.
Locally, you'll want to set DATABASE_URL as an environment variable. You can do this according to the instructions of whatever shell you're using. Or, if you want to use the Heroku toolchain (launch your local server using heroku local) you can add it to a file called .env at the top level of your project. For the remote setup, you'll want to add your database URL as a remote Heroku environment variable. You can do this with the following command:
heroku config:set DATABASE_URL=postgresql://myurl
You can find that URL by going into Heroku, navigating to your database, and clicking on "database credentials". It's listed under URI.
Are you parsing the DATABASE_URL environment variable? It will look something like this:
postgres://username:password#host:port/database_name
So you will want to pull that in and parse it before you open a connection to your database. Depending on how you've declared your database (in your config or next to your wsgi app) it might look like this:
import os
import urlparse
urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ['DATABASE_URL'])
# for your config
DATABASE = {
'engine': 'peewee.PostgresqlDatabase',
'name': url.path[1:],
'password': url.password,
'host': url.hostname,
'port': url.port,
}
See the notes here: https://devcenter.heroku.com/articles/django
heroku config:set HEROKU=1
import os
import urlparse
import psycopg2
from flask import Flask
from flask_peewee.db import Database
if 'HEROKU' in os.environ:
DEBUG = False
urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ['DATABASE_URL'])
DATABASE = {
'engine': 'peewee.PostgresqlDatabase',
'name': url.path[1:],
'user': url.username,
'password': url.password,
'host': url.hostname,
'port': url.port,
}
else:
DEBUG = True
DATABASE = {
'engine': 'peewee.PostgresqlDatabase',
'name': 'framingappdb',
'user': 'postgres',
'password': 'postgres',
'host': 'localhost',
'port': 5432 ,
'threadlocals': True
}
app = Flask(__name__)
app.config.from_object(__name__)
db = Database(app)
Modified coleifer's answer to answer hasenj's comment.
Please mark one of these as the accepted answer.
I have managed to get my Flask app which uses Peewee working on Heroku using the below code:
# persons.py
import os
from peewee import *
db_proxy = Proxy()
# Define your models here
class Person(Model):
name = CharField(max_length=20, unique=True)
age = IntField()
class Meta:
database = db_proxy
# Import modules based on the environment.
# The HEROKU value first needs to be set on Heroku
# either through the web front-end or through the command
# line (if you have Heroku Toolbelt installed, type the following:
# heroku config:set HEROKU=1).
if 'HEROKU' in os.environ:
import urlparse, psycopg2
urlparse.uses_netloc.append('postgres')
url = urlparse.urlparse(os.environ["DATABASE_URL"])
db = PostgresqlDatabase(database=url.path[1:], user=url.username, password=url.password, host=url.hostname, port=url.port)
db_proxy.initialize(db)
else:
db = SqliteDatabase('persons.db')
db_proxy.initialize(db)
if __name__ == '__main__':
db_proxy.connect()
db_proxy.create_tables([Person], safe=True)
You should already have a Postgres database add-on attached to your app. You can do this via the command line or through the web front-end. Assuming that the database is already attached to your app and you have already deployed your with the above changes, log-in to Heroku and create the table(s):
$ heroku login
$ heroku run bash
$ python persons.py
Check that the table was created:
$ heroku pg:psql
your_app_name::DATABASE=> \dt
You then import this file (persons.py in this example) in another Python script, e.g. a request handler. You need to manage the database connection explicitly:
# server.py
from flask import g
from persons import db_proxy
#app.before_request
def before_request():
g.db = db_proxy
g.db.connect()
#app.after_request
def after_request(response):
g.db.close()
return response
…
References:
https://devcenter.heroku.com/articles/heroku-postgresql
http://peewee.readthedocs.org/en/latest/peewee/database.html#dynamically-defining-a-database
http://peewee.readthedocs.org/en/latest/peewee/example.html#establishing-a-database-connection
https://stackoverflow.com/a/20131277/3104465
https://gist.github.com/fprieur/9561148
Related
I have a very basic Heroku web app where you can click on links (on the main page) to be redirected to other pages. I have done this to test the database page. All the pages work apart from when I click to view the database page. when I try I get an error message:
Internal Server Error
The server encountered an internal error and was unable to complete
your request.
Either the server is overloaded or there is an error in
the application.
I did try to play around with the procfile but in the end nothig worked. My current proctfile looks like this:
web: gunicorn flask-sqlalchemy-test-02.wsgi:application --log-file=-
To be honest I'm not too sure if its the procfile or the sytax in app.py which is causing me problems.
My app.py file:
import os
import psycopg2
from flask import Flask, render_template, g, url_for
app = Flask(__name__)
DATABASE_URL = os.environ.get('postgres://fikwczdiymxhwf:73bf42c2c8a15fa59b77e93654b6383e1cf4f85bdf0156818d1cf39a77815f13#ec2-54-243-47-196.compute-1.amazonaws.com:5432/d3uburco4fea1b'
#app.route('/')
def index():
return render_template("index.html")
#app.route('/page2')
def page_2():
return render_template("random_page_2.html")
#app.route('/hello')
def hello():
return render_template("hello.html")
#app.route('/view_database')
def view_db():
conn = psycopg2.connect(DATABASE_URL)
db = conn.cursor()
data = db.execute("SELECT * FROM example").fetchall()
db.close()
conn.close()
return render_template('view_database.html', data=data)
I expected to view the database in a form of an unordered list but recieved an error message instead:
Internal Server Error
The server encountered an internal error and was unable to complete your request.
Either the server is overloaded or there is an error in the application.
The way you are using os.environ.get() is wrong.
os.environ.get() is used to obtain environment variables that are exported by your OS, so DATABASE_URL returns None, you cannot connect to None URL, so internal server error.
**Correct Way : **
First, export the environment variable, if using Linux :
export DATABASE_URL=postgres://fikwczdiymxhwf:73bf42c2c8a15fa59b77e93654b6383e1cf4f85bdf0156818d1cf39a77815f13#ec2-54-243-47-196.compute-1.amazonaws.com:5432/d3uburco4fea1b
Then, in your code, replace that line as :
DATABASE_URL = os.environ.get('DATABASE_URL', '')
I've read the official docs yet I'm not quite sure I understand how to apply what they tell. Also I've seen this QA, I also use a factory pattern. Just can't see the whole picture.
The connection pool as long as other redis/huey settings may differ depending on the given environment (development, production). How do we wire huey up so we can configure it similar to the Flask application?
As long as I understand to fire a task from a view we need to import tasks moudule and call the specific task (call a function passing the sensitive params). Where shoud we instantiate, keep the huey instance?
Should tasks know about the application dependencies? Should we consider another stripped-down Flask app for this matter?
Can you help a little bit?
Here's how I wired it all up.
First off, here's the contents of my project folder:
Get a stripped-down Flask application to be used by your tasks. As it was suggested in the post I created a secondary application factory:
# global dependencies
db = SQLAlchemy()
def create_app_huey(config_name):
app = Flask(__name__)
# apply configuration
app.config.from_object(config[config_name])
# init extensions
db.init_app(app)
return app
Create tasking package. Two important files here are config.py and tasks.py. This post helped a lot. Let's start with configuration. Note, this is very simple approach.
# config.py (app.tasking.config)
import os
from huey import RedisHuey
settings__development = {
'host': 'localhost'
}
settings__testing = {
'host': 'localhost'
}
settings__production = {
'host': 'production_server'
}
settings = {
'development': settings__development,
'testing': settings__testing,
'production': settings__production,
'default': settings__development
}
huey = RedisHuey(**settings[os.getenv('FLASK_ENV') or 'default'])
Then the tasks.py module will look like this:
import os
from app.tasking.config import huey
from app import create_app_huey
app = create_app_huey(config_name=os.getenv('FLASK_ENV') or 'default')
#huey.task()
def create_thumbnails(document):
pass
Run the consumer. Activate your virtual environment. Then run from cmd (I'm on Windows):
huey_consumer.py app.tasking.config.huey
Where app.tasking.config is a package.package.module path (in my case!) and
huey is the name of available (in the config module) huey instance. Check your huey instance name.
Reading this helped.
The tutorials I have seen use the following code to run the server:
if __name__ == '__main__':
socketio.run(app)
My __init__.py file is:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from sqlalchemy.orm import sessionmaker
from sqlalchemy import *
from flask.ext.socketio import SocketIO, emit
app = Flask(__name__)
socketio = SocketIO(app)
app.debug = True
engine = create_engine('mysql://root:my_pw#localhost/db_name')
DBSession = sessionmaker(bind=engine)
import couponmonk.views
My views.py file contains all the #app.route and #socketio decorators.
My question is, where should I be placing the code:
socketio.run(app)
When I put it in the __init__.py_ file, I receive the errors:
File "/opt/lampp/htdocs/flaskapp/flask.wsgi", line 7, in <module>
from couponmonk import app as application
File "/home/giri/Desktop/couponmonk/venv/couponmonk/__init__.py", line 14, in <module>
socketio.run(app)
File "/home/giri/Desktop/couponmonk/venv/lib/python2.7/site-packages/flask_socketio/__init__.py", line 411, in run
run_with_reloader(run_server)
File "/home/giri/Desktop/couponmonk/venv/lib/python2.7/site-packages/werkzeug/serving.py", line 632, in run_with_reloader
return run_with_reloader(*args, **kwargs)
File "/home/giri/Desktop/couponmonk/venv/lib/python2.7/site-packages/werkzeug/_reloader.py", line 231, in run_with_reloader
sys.exit(reloader.restart_with_reloader())
SystemExit: 2
Author of Flask-SocketIO here.
Unfortunately this extension cannot work with a standard web server, you will not be able to host an app that uses it over apache/mod_wsgi. You need to use a gevent server, and not a generic one, but one that is customized for Socket.IO.
That means that Apache is out (it does not even support WebSocket traffic). Also uWSGI is out (supports gevent, but not possible to use a custom gevent server). As a side note, Python 3 is also currently out, as gevent only runs on Python 2 (though I think there's going to be good news about this soon, I'm working on some ideas to get socketio running on Python 3 right now).
The choices that you have are given in the documentation. Summary:
socketio.run(app), which runs the custom socketio gevent server directly.
Gunicorn with a custom socketio worker (command line shown in docs)
You can put nginx as reverse proxy in front of your server if you like. The configuration is also shown in the docs.
Good luck!
Seems lime you are trying to use Miguel's Flask-socketIO extension, right? It only supports Guinicorn as the WSGI server, and advice you to use NGINX as a proxy pass. I know nothing about xampp but as far as I have read; It's possible to do a proxy pass since one of the latest versions of Apache. Have not tried though.
I am following this example here:
http://docs.mongodb.org/manual/tutorial/write-a-tumblelog-application-with-flask-mongoengine/
My problem lies here:
from flask import Flask
from flask.ext.mongoengine import MongoEngine
app = Flask(__name__)
app.config["MONGODB_DB"] = "my_tumble_log"
app.config["SECRET_KEY"] = "KeepThisS3cr3t"
db = MongoEngine(app)
if __name__ == '__main__':
app.run()
This assumes that the MONGODB_DB is "my_tumble_log" on my local machine and default port. What if my mongo database is on a remote machine with a different port? How would I modify the example to allow this?
I have tried adding a line before the MONGODB_DB app.config:
app.config['MONGODB_CONNSTRING'] = "mongodb://myremotehost:myport"
Though it has no effect whatsoever.
The following configuration settings are available:
MONGODB_DB
MONGODB_USERNAME
MONGODB_PASSWORD
MONGODB_HOST
MONGODB_PORT
So I'm following the getting started guide from heroku with django. However when I run this command:
heroku run python manage.py syncdb
I get this error
psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" and accepting
TCP/IP connections on port 5432?
I assumed this meant that the db wasn't set up yet... so I manually added the shared_db option as well:
heroku addons:add shared-database:5mb
But.. I still get the same error. What gives?
EDITED:
As #mipadi has pointed out here (http://stackoverflow.com/questions/13001031/django-heroku-settings-injection/13092534), it can actually be as simple as this:
import dj_database_url
DATABASES = {'default' : dj_database_url.config() }
This works if you have a DATABASE_URL env variable set. heroku:pg_promote gets your there. Details below
Make sure you have Postgres on your Heroku
heroku addons:add heroku-postgresql:dev
Step 1: figure out your database url
heroku config | grep POSTGRESQL
The output will look something like this:
HEROKU_POSTGRESQL__URL:
postgres://user:password#host:5432/blabla
Step 2: Grab the setting name from the previous step (e.g. HEROKU_POSTGRESQL_ROSE_URL) and put it in your settings file like so
DATABASES = {'default': dj_database_url.config(default=os.environ["HEROKU_POSTGRESQL_ROSE_URL"])}
[UPDATE] As Ted has pointed out, there's a way to promote the color URL to DATABASE_URL variable:
heroku pg:promote HEROKU_POSTGRESQL_ROSE_URL
Your database settings can then use DATABASE_URL as opposed to more exotic colored URLS
DATABASES = {'default': dj_database_url.config(default=os.environ["DATABASE_URL"])}
Bob is your uncle
I got it working by adding the following code to settings.py myself, seems for some reason Heroku didn't add it for me....
Normally it always added the code on Heroku dynamically but I guess after django 1.4 it didnt do it anymore for some reason. Or I was just doing something wrong.
Anyway this is the code just append it to your settings.py and it should work as before.
import sys
import urlparse
import os
# Register database schemes in URLs.
urlparse.uses_netloc.append('postgres')
urlparse.uses_netloc.append('mysql')
try:
# Check to make sure DATABASES is set in settings.py file.
# If not default to {}
if 'DATABASES' not in locals():
DATABASES = {}
if 'DATABASE_URL' in os.environ:
url = urlparse.urlparse(os.environ['DATABASE_URL'])
# Ensure default database exists.
DATABASES['default'] = DATABASES.get('default', {})
# Update with environment configuration.
DATABASES['default'].update({
'NAME': url.path[1:],
'USER': url.username,
'PASSWORD': url.password,
'HOST': url.hostname,
'PORT': url.port,
})
if url.scheme == 'postgres':
DATABASES['default']['ENGINE'] = 'django.db.backends.postgresql_psycopg2'
if url.scheme == 'mysql':
DATABASES['default']['ENGINE'] = 'django.db.backends.mysql'
except Exception:
print 'Unexpected error:', sys.exc_info()
My app structure was off... heroku wants the structure to look like this:
toplevel
requirements.txt
myapp
manage.py
all other django stuff
I had the same problem, this is how I solved it
Step1: Follow Phillip's Step 1 to get database name(color)
Step2:
$ heroku pg:promote HEROKU_POSTGRESQL_<COLOR>
leads to the output
Promoting HEROKU_POSTGRESQL_<COLOR> to DATABASE_URL... done
You need to add this to your requirements.txt:
psycopg2
By default Heroku configures a Postgres database and injects code into your settings.py (https://devcenter.heroku.com/articles/django#postgres_database_config). This reads from the environment variable DATABASE_URL, but does require psycopg2 is installed.