We Keep Coding

Flask DB migrate to support multiple databases using binds

I am using Flask Application with flask-migrate. I have initialized the app with single database but now the requirement is to add multiple databases.
Here are the steps that I followed with the below code :-
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
import os
from flask_migrate import Migrate
basedir = os.path.abspath(os.path.dirname(__file__))
app = Flask(__name__)
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'database1.db')
"""
app.config['SQLALCHEMY_BINDS'] = {
'database2' : 'sqlite:///' + os.path.join(basedir, 'database2.db')
}
"""
db = SQLAlchemy(app)
migrate = Migrate(app, db)
class Table1(db.Model):
id = db.Column(db.Integer, primary_key=True)
"""
class Table2(db.Model):
__bind_key__ = 'database2'
id = db.Column(db.Integer, primary_key=True)
"""
#app.route('/add1')
def index():
item = Table1(id=2)
db.session.add(item)
db.session.commit()
return 'Added value'
"""
#app.route('/add2')
def index2():
item = Table2(id=2)
db.session.add(item)
db.session.commit()
return 'Added table2 value'
"""
if __name__ == '__main__':
app.run(debug=True)
Initialized the code as below
flask db init
flask db migrate
flask db upgrade
Visit http://127.0.0.1:5000/add1 to add to database1
Now uncomment the code so that the database2 can be added. On running flask db init again with --multidb switch, an error is reported.
(venv) bash-4.1$ flask db init --multidb
Error: Directory migrations already exists and is not empty
How can I convert an existing flask project with migrations to support multiple databases? Also, these multiple databases will evolve over the time i.e. I might have to add more databases. So, what is the best way to keep on adding more databases?
EDIT
One solution, I find is to remove the alembic_version table from existing database, remove the migrations folder and then do db init --multidb. I am okay resetting the migrations but want to know if this is the only solution?

`flask db upgrade` runs fine, but subsequent calls to `flask db migrate` drop all tables

At some point I was able to use the flask db CLI to generate my initial data model migration. My application is Dockerized, and when I start it up I call flask db upgrade in entrypoint.sh which echos:
backend_1 | INFO [alembic.runtime.migration] Context impl MySQLImpl.
backend_1 | INFO [alembic.runtime.migration] Will assume non-transactional DDL.
backend_1 | INFO [alembic.runtime.migration] Running upgrade -> 7f973d2726ef, initial data model
So that's great. I have now made updates to the data model and want to create a migration, exactly like I was able to before, with flask db migrate -m 'update games tables', which echos:
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected removed table 'games'
INFO [alembic.autogenerate.compare] Detected removed table 'game_status'
INFO [alembic.autogenerate.compare] Detected removed table 'positions'
INFO [alembic.autogenerate.compare] Detected removed table 'users'
INFO [alembic.autogenerate.compare] Detected removed table 'game_invites'
Generating /home/backend/database/migrations/versions/55f785302c18_update_games_tables.py ... done
My flask db migrate command sees the database tables, but since it isn't reading any model data it thinks that we want to delete them all. I'm really stumped on this one. The most recent post I checked out was this one. Per Miguel's troubleshooting suggestions in the post, I have:
Verified that I am using the same db across the entire application context (see below)
Explicitly imported all data models prior to calling migrate.init_app
Torn down and rebuild my database multiple times, including re-starting the migrations directory with flask db init
My directory structure looks like this, and I'll include what I think are the most important snippets of code below:
/backend/
| |__/__init__.py (empty)
| |__/app_factory.py
| |__/backend-entrypoint.sh
| |__/wsgi.py
| |__/config.py
| |__/.flaskenv
|
|__/api (nothing important here)
|
|__/database/
| |__/migrations/... (all the usual stuff here)
| |__/db.py
| |__/models.py
|
|__/tests (nothing important here)
db.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy() # <- This is the db that's going to get passed around
models.py
from aenum import Enum
from backend.database.db import db
class Users(db.Model):
__tablename__ = "users"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.Text)
email = db.Column(db.Text)
profile_pic = db.Column(db.Text)
username = db.Column(db.Text)
created_at = db.Column(db.DATETIME)
...
# Here's the part where I'm making changes to the schema.
app_factory.py
from flask import Flask
from flask_migrate import Migrate
from database.db import db
from api import routes as routes
def create_app():
app = Flask(__name__)
app.config.from_object('config.Config')
db.init_app(app)
migrate = Migrate()
with app.app_context():
app.register_blueprint(routes.routes)
# It's annoying that flask works this way. migrate and models both become part of the application context here
from backend.database.models import (
Users,
Games,
GameStatus,
GameInvites,
Positions
)
migrate.init_app(app, db, directory="/home/backend/database/migrations")
return app
backend-entrypoint.sh
#!/usr/bin/env bash
until nc -z -v -w30 $DB_HOST 3306
do
echo "Waiting a second until the database is receiving connections..."
sleep 1
done
flask db upgrade # construct the data model
python wsgi.py # start the web application
wsgi.py
from backend.app_factory import create_app
from config import Config
app = create_app()
if __name__ == "__main__":
app.run(host='0.0.0.0', debug=Config.DEBUG_MODE, ssl_context=("cert.pem", "key.pem"))
config.py
...
class Config:
...
SQLALCHEMY_DATABASE_URI = f"mysql+pymysql://{db_user}:{db_password}#{db_host}:{db_port}/{db_name}?charset=utf8"
...
.flaskenv
FLASK_APP=/home/backend/wsgi.py
This worked at some point, but nothing I've tried today, including tearing down the entire db, rebuilding the container, and starting from scratch with flask db init has gotten my anywhere. The problem doesn't seem to be that the changes aren't being detected, but that flask db isn't even connecting to the right database, even though I setup Migrate within the flask app context. I have functional tests that I run against the API server that the flask app runs, and it has no issue communicating with the populated database. The problem seems pretty contained to the flask db CLI. I have tried rolling back to earlier versions to see if I can identify whether a change in my file structure broke something, but this issue is persisting across multiple commits.

Got it sorted out just by cleaning up the layout of create_app.py. Thanks to #Miguel for the pointers on this. The issue seems to have been 1) Inappropriate use of app_context and 2) the import order of models. When I strip out the with app.app_context() line and move my model imports to the bottom I get something that works great:
from flask import Flask
from flask_migrate import Migrate
from backend.database.db import db
from backend.api import routes as routes
from config import Config
def create_app():
app = Flask(__name__)
app.config.from_object('config.Config')
app.register_blueprint(routes.routes)
db.init_app(app)
migrate = Migrate(app, db, directory=Config.MIGRATIONS_DIRECTORY)
migrate.init_app(app, db)
return app
from backend.database import models
I used the __init__ file from Miguel's Flask Mega Tutorial for this. The two main ways that my architecture is different than his is that I create my db object in a separate file (see above) to avoid circular imports when using elsewhere (e.g. in my api routes definitions), and I define a create_app.py file rather than placing all the app setup in __init__

Registering sqlalchemy extension with current application using init_app()

I'm trying to run an existing app in a new environment. The app ran fine in a previous environment, but when I run it locally it refuses to connect with my DB.
AssertionError: The sqlalchemy extension was not registered to the current application. Please make sure to call init_app() first.
Where I'm confused is that this exact code worked in a previous environment. It's asking me to call init_app() but, from my understanding, that's only if there are multiple apps, which there aren't.
In debugging mode, the app does recognize the object i.e.
>>> type(Candidate)
<class 'flask_sqlalchemy.model.DefaultMeta'>
I am unclear how and where to incorporate app_init() into my code. I have tried...
db = SQLAlchemy(app)
db.init_app(app)
but this didn't have any impact.
Folder structure...
app
- static folder
- templates folder
- _init_.py
- config.py
- views.py
- run.py
- Procfile

You could try following code and structure in init.py
db = SQLAlchemy()
def create_app():
app = Flask()
db.init_app(app)
from user_model import User
with app.app_context():
db.create_all()

The problem was that there was legacy code that was supposed to be removed. I was initializing an app twice by accident.

Instead of creating init.py, why don't you import db directly from models
The code below shows my models.py with table students
from flask import Flask,session
from flask_sqlalchemy import SQLAlchemy
#initialize this two objects below app and db parameters so that app.py will have the same db session.
#then in app.py add from models import db
#finally add db.init_app(app) at the bottom of app.py or run.py
app = Flask(__name__)
db = SQLAlchemy(app)
class Students(db.Model):
id = db.Column('student_id', db.Integer, primary_key = True)
name = db.Column(db.String(50))
email = db.Column(db.String(50))
def __repr__(self):
return '<Student {}>'.format(self.email)
Then in your app.py or run.py
You will have to make an import for db
from models import db
Your assertion error for sql-alchemy can be mitigate by moving the two following line of code
immediately after this line of code towards the bottom
if __name__ == '__main__':
hence the code will become like below and your application will run without issue
if __name__ == '__main__':
#create table
db.create_all()
db.init_app(app)
# remember to turn app debug by setting it to false in production
app.run(debug=True)

Do I need to create a sessions table to use Flask-Session SqlAlchemySessionInterface

I am attempting to implement Flask-Session in my python application. I read in the docs that its recommended to use another interface like the SqlAlchemySessionInterface instead of the default NullSessionInterface which is used when nothing is provided to the SESSION_TYPE configuration key.
From the flask_session/init.py file under class Session it reads
By default Flask-Session will use :class:NullSessionInterface, you
really should configurate your app to use a different SessionInterface.
After setting the SESSION_TYPE configuration key to "sqlalchemy" I get an error
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "sessions" does not exist
This indicates that Flask-Session is looking to use a table with the name "sessions" in my database model but I cannot find anywhere in the Flask-Session documentation where it points out that a table should be created and what fields it should have.
Can anyone suggest a solution to this please?

I wanted to use Flask-session, but I was also using Flask-migrate and didn't want to call db.create_all() manually and break the migration path. Fortunately, #Flashspeedlife's suggestion of just importing the Interface and instantiating it worked.
app/__init__.py:
from flask_session import SqlAlchemySessionInterface
from app.extensions import db, sess, migrate # My extensions file
def create_app():
app = Flask(__name__)
with app.app_context():
db.init_app(app)
migrate.init_app(app, db)
sess.init_app(app)
SqlAlchemySessionInterface(app, db, "sessions", "sess_")
Now, flask db migrate generates an alembic script with the new sessions table.

After studying the Flask-Session/init.py code I found that class SqlAlchemySessionInterface under its __init__ contains a Flask-SQLAlchemy model
class Session(self.db.Model).
To cause this table model to be created, in the file where I create my models I imported SqlAlchemySessionInterface from flask_sessionstore and put the line
SqlAlchemySessionInterface(myApp, sqlAlchemyDbObject, "table_name", "prefix_")
and then ran db.create_all().
class SqlAlchemySessionInterface(SessionInterface):
"""Uses the Flask-SQLAlchemy from a flask app as a session backend.
.. versionadded:: 0.2
:param app: A Flask app instance.
:param db: A Flask-SQLAlchemy instance.
:param table: The table name you want to use.
:param key_prefix: A prefix that is added to all store keys.
:param use_signer: Whether to sign the session id cookie or not.
:param permanent: Whether to use permanent session or not.
"""
serializer = pickle
session_class = SqlAlchemySession
def __init__(self, app, db, table, key_prefix, use_signer=False,
permanent=True):
if db is None:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
self.db = db
self.key_prefix = key_prefix
self.use_signer = use_signer
self.permanent = permanent
class Session(self.db.Model):
__tablename__ = table
id = self.db.Column(self.db.Integer, primary_key=True)
session_id = self.db.Column(self.db.String(255), unique=True)
data = self.db.Column(self.db.LargeBinary)
expiry = self.db.Column(self.db.DateTime)
def __init__(self, session_id, data, expiry):
self.session_id = session_id
self.data = data
self.expiry = expiry
def __repr__(self):
return '<Session data %s>' % self.data
# self.db.create_all()
self.sql_session_model = Session
I'am definitely using Django for my next project. Documentation for many Flask Extensions aren't great at all.
EDIT
Changed (imported SqlAlchemySessionInterface from flask_session) to (imported SqlAlchemySessionInterface from flask_sessionstore)

Flask-Migrate not creating tables

I have the following models in file listpull/models.py:
from datetime import datetime
from listpull import db
class Job(db.Model):
id = db.Column(db.Integer, primary_key=True)
list_type_id = db.Column(db.Integer, db.ForeignKey('list_type.id'),
nullable=False)
list_type = db.relationship('ListType',
backref=db.backref('jobs', lazy='dynamic'))
record_count = db.Column(db.Integer, nullable=False)
status = db.Column(db.Integer, nullable=False)
sf_job_id = db.Column(db.Integer, nullable=False)
created_at = db.Column(db.DateTime, nullable=False)
compressed_csv = db.Column(db.LargeBinary)
def __init__(self, list_type, created_at=None):
self.list_type = list_type
if created_at is None:
created_at = datetime.utcnow()
self.created_at = created_at
def __repr__(self):
return '<Job {}>'.format(self.id)
class ListType(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80), unique=True, nullable=False)
def __init__(self, name):
self.name = name
def __repr__(self):
return '<ListType {}>'.format(self.name)
I call ./run.py init then ./run.py migrate then ./run.py upgrade, and I see the migration file generated, but its empty:
"""empty message
Revision ID: 5048d48b21de
Revises: None
Create Date: 2013-10-11 13:25:43.131937
"""
# revision identifiers, used by Alembic.
revision = '5048d48b21de'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
pass
### end Alembic commands ###
def downgrade():
### commands auto generated by Alembic - please adjust! ###
pass
### end Alembic commands ###
run.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from listpull import manager
manager.run()
listpull/__init__.py
# -*- coding: utf-8 -*-
# pylint: disable-msg=C0103
""" listpull module """
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand
from mom.client import SQLClient
from smartfocus.restclient import RESTClient
app = Flask(__name__)
app.config.from_object('config')
db = SQLAlchemy(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)
mom = SQLClient(app.config['MOM_HOST'],
app.config['MOM_USER'],
app.config['MOM_PASSWORD'],
app.config['MOM_DB'])
sf = RESTClient(app.config['SMARTFOCUS_URL'],
app.config['SMARTFOCUS_LOGIN'],
app.config['SMARTFOCUS_PASSWORD'],
app.config['SMARTFOCUS_KEY'])
import listpull.models
import listpull.views
UPDATE
If I run the shell via ./run.py shell and then do from listpull import * and call db.create_all(), I get the schema:
mark.richman#MBP:~/code/nhs-listpull$ sqlite3 app.db
-- Loading resources from /Users/mark.richman/.sqliterc
SQLite version 3.7.12 2012-04-03 19:43:07
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .schema
CREATE TABLE job (
id INTEGER NOT NULL,
list_type_id INTEGER NOT NULL,
record_count INTEGER NOT NULL,
status INTEGER NOT NULL,
sf_job_id INTEGER NOT NULL,
created_at DATETIME NOT NULL,
compressed_csv BLOB,
PRIMARY KEY (id),
FOREIGN KEY(list_type_id) REFERENCES list_type (id)
);
CREATE TABLE list_type (
id INTEGER NOT NULL,
name VARCHAR(80) NOT NULL,
PRIMARY KEY (id),
UNIQUE (name)
);
sqlite>
Unfortunately, the migrations still do not work.

When you call the migrate command Flask-Migrate (or actually Alembic underneath it) will look at your models.py and compare that to what's actually in your database.
The fact that you've got an empty migration script suggests you have updated your database to match your model through another method that is outside of Flask-Migrate's control, maybe by calling Flask-SQLAlchemy's db.create_all().
If you don't have any valuable data in your database, then open a Python shell and call db.drop_all() to empty it, then try the auto migration again.
UPDATE: I installed your project here and confirmed that migrations are working fine for me:
(venv)[miguel#miguel-linux nhs-listpull]$ ./run.py db init
Creating directory /home/miguel/tmp/mark/nhs-listpull/migrations...done
Creating directory /home/miguel/tmp/mark/nhs-listpull/migrations/versions...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/script.py.mako...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/env.pyc...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/env.py...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/README...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/alembic.ini...done
Please edit configuration/connection/logging settings in
'/home/miguel/tmp/mark/nhs-listpull/migrations/alembic.ini' before
proceeding.
(venv)[miguel#miguel-linux nhs-listpull]$ ./run.py db migrate
INFO [alembic.migration] Context impl SQLiteImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate] Detected added table 'list_type'
INFO [alembic.autogenerate] Detected added table 'job'
Generating /home/miguel/tmp/mark/nhs-
listpull/migrations/versions/48ff3456cfd3_.py...done
Try a fresh checkout, I think your setup is correct.

Ensure to import the Models in the manage.py file (or the file with the migrate instance). You have to import the models in the file, even if you are not explicitly using them. Alembic needs these imports to migrate, and to create the tables in the database. For example:
# ... some imports ...
from api.models import User, Bucketlist, BucketlistItem # Import the models
app = create_app('dev')
manager = Manager(app)
migrate = Migrate(app, db)
manager.add_command('db', MigrateCommand)
# ... some more code here ...
if __name__ == "__main__":
manager.run()
db.create_all()

I had the same issue but a different problem caused it.
Flask-migrate workflow consists of two consequent commands:
flask db migrate
which generates the migration and
flask db upgrade
which applies the migration. I forgot to run the last one and tried to start next migration without applying the previous one.

For anyone coming who comes across this, my problem was having
db.create_all()
in my main flask application file
which created the new table without the knowledge of alembic
Simply comment it out or delete it altogether so it doesn't mess with future migrations.
but unlike #Miguel's suggestion, instead of dropping the whole database (i had important information in it), i was able to fix it by deleting the new table created by Flask SQLAlchemy and then running the migration.
and this time alembic detected the new table and created a proper migration script

I just encountered a similar problem. I'd like to share my solution for anyone else encountering this thread. For me, I had my models in a package. For example models/user.py and I tried from app.models import * which did not detect anything on the migrate. However, if I changed the import to from app.models import user this is okay why my project is young, but as I have more models a bulk import would be preferable.

Strange solve for me is: delete database and folder migrations. Then
>>> from app import db
>>> db.create_all()
After flask db init or python app.py db init and then flask db migrate or python app.py db migrate. Wow, It's strange, but it works for me.