We Keep Coding

Python Flask Restful and Sqlalchemy with postgres: Tables not created

I want to connect to a postgres database running in my dev docker container.
I can establish a connection by the connection String:
def create_api():
from .models import db
from .models.userModels import User
# Initialize and Configure the Flask App
app = Flask(__name__)
app.config.from_object(AppConfig)
db = SQLAlchemy(app)
I have a separate module for my routes and the related classes as well as a separate module for my models I will use for postgres model based communication.
In the init file I execute the create_api() method. Here I create the routes and connect them to a Resource class ( userResource.py).
In userReource.py I use the userModels.User for my db stuff.
Structure:
- api
__init__.py
- models
- userModels.py
- resources
- userResource.py
Model:
class User(db.Model):
id = db.Column(db.Integer,primary_key=True)
name= db.Column(db.String(100))
def __init__(self, name):
self.name=name
The docsenter link description here say that I should use the
db.create_all() method to create the table user id if not exists.
I've also read other answsers of that topic which mention to import the model class if it is placed in a separate module before executing this command. As you can see, I've imported the model before, but nothing happens in my database.
Is there something else I have to take care of?

`flask db upgrade` runs fine, but subsequent calls to `flask db migrate` drop all tables

At some point I was able to use the flask db CLI to generate my initial data model migration. My application is Dockerized, and when I start it up I call flask db upgrade in entrypoint.sh which echos:
backend_1 | INFO [alembic.runtime.migration] Context impl MySQLImpl.
backend_1 | INFO [alembic.runtime.migration] Will assume non-transactional DDL.
backend_1 | INFO [alembic.runtime.migration] Running upgrade -> 7f973d2726ef, initial data model
So that's great. I have now made updates to the data model and want to create a migration, exactly like I was able to before, with flask db migrate -m 'update games tables', which echos:
INFO [alembic.runtime.migration] Context impl MySQLImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected removed table 'games'
INFO [alembic.autogenerate.compare] Detected removed table 'game_status'
INFO [alembic.autogenerate.compare] Detected removed table 'positions'
INFO [alembic.autogenerate.compare] Detected removed table 'users'
INFO [alembic.autogenerate.compare] Detected removed table 'game_invites'
Generating /home/backend/database/migrations/versions/55f785302c18_update_games_tables.py ... done
My flask db migrate command sees the database tables, but since it isn't reading any model data it thinks that we want to delete them all. I'm really stumped on this one. The most recent post I checked out was this one. Per Miguel's troubleshooting suggestions in the post, I have:
Verified that I am using the same db across the entire application context (see below)
Explicitly imported all data models prior to calling migrate.init_app
Torn down and rebuild my database multiple times, including re-starting the migrations directory with flask db init
My directory structure looks like this, and I'll include what I think are the most important snippets of code below:
/backend/
| |__/__init__.py (empty)
| |__/app_factory.py
| |__/backend-entrypoint.sh
| |__/wsgi.py
| |__/config.py
| |__/.flaskenv
|
|__/api (nothing important here)
|
|__/database/
| |__/migrations/... (all the usual stuff here)
| |__/db.py
| |__/models.py
|
|__/tests (nothing important here)
db.py
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy() # <- This is the db that's going to get passed around
models.py
from aenum import Enum
from backend.database.db import db
class Users(db.Model):
__tablename__ = "users"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.Text)
email = db.Column(db.Text)
profile_pic = db.Column(db.Text)
username = db.Column(db.Text)
created_at = db.Column(db.DATETIME)
...
# Here's the part where I'm making changes to the schema.
app_factory.py
from flask import Flask
from flask_migrate import Migrate
from database.db import db
from api import routes as routes
def create_app():
app = Flask(__name__)
app.config.from_object('config.Config')
db.init_app(app)
migrate = Migrate()
with app.app_context():
app.register_blueprint(routes.routes)
# It's annoying that flask works this way. migrate and models both become part of the application context here
from backend.database.models import (
Users,
Games,
GameStatus,
GameInvites,
Positions
)
migrate.init_app(app, db, directory="/home/backend/database/migrations")
return app
backend-entrypoint.sh
#!/usr/bin/env bash
until nc -z -v -w30 $DB_HOST 3306
do
echo "Waiting a second until the database is receiving connections..."
sleep 1
done
flask db upgrade # construct the data model
python wsgi.py # start the web application
wsgi.py
from backend.app_factory import create_app
from config import Config
app = create_app()
if __name__ == "__main__":
app.run(host='0.0.0.0', debug=Config.DEBUG_MODE, ssl_context=("cert.pem", "key.pem"))
config.py
...
class Config:
...
SQLALCHEMY_DATABASE_URI = f"mysql+pymysql://{db_user}:{db_password}#{db_host}:{db_port}/{db_name}?charset=utf8"
...
.flaskenv
FLASK_APP=/home/backend/wsgi.py
This worked at some point, but nothing I've tried today, including tearing down the entire db, rebuilding the container, and starting from scratch with flask db init has gotten my anywhere. The problem doesn't seem to be that the changes aren't being detected, but that flask db isn't even connecting to the right database, even though I setup Migrate within the flask app context. I have functional tests that I run against the API server that the flask app runs, and it has no issue communicating with the populated database. The problem seems pretty contained to the flask db CLI. I have tried rolling back to earlier versions to see if I can identify whether a change in my file structure broke something, but this issue is persisting across multiple commits.

Got it sorted out just by cleaning up the layout of create_app.py. Thanks to #Miguel for the pointers on this. The issue seems to have been 1) Inappropriate use of app_context and 2) the import order of models. When I strip out the with app.app_context() line and move my model imports to the bottom I get something that works great:
from flask import Flask
from flask_migrate import Migrate
from backend.database.db import db
from backend.api import routes as routes
from config import Config
def create_app():
app = Flask(__name__)
app.config.from_object('config.Config')
app.register_blueprint(routes.routes)
db.init_app(app)
migrate = Migrate(app, db, directory=Config.MIGRATIONS_DIRECTORY)
migrate.init_app(app, db)
return app
from backend.database import models
I used the __init__ file from Miguel's Flask Mega Tutorial for this. The two main ways that my architecture is different than his is that I create my db object in a separate file (see above) to avoid circular imports when using elsewhere (e.g. in my api routes definitions), and I define a create_app.py file rather than placing all the app setup in __init__

Flask-Migrate No Changes Detected to Schema on first migration

I'm using Flask with Flask-SQLAlchemy and Flask-Migrate to create an application, however when I try to create a migration nothing happens.
I've created two tables in app/models.py:
from flask import current_app
from . import db
class Student(db.Model):
__tablename__ = 'students'
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(64), unique=True, nullable=False)
password_hash = db.Column(db.String(128))
def __init__(self, **kwargs):
super(Student, self).__init__(**kwargs)
def __repr__(self):
return '<Tutor {}>' % self.id
class Tutor(db.Model):
__tablename__ = 'tutors'
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(64), unique=True, index=True)
password_hash = db.Column(db.String(128))
def __init__(self, **kwargs):
super(Tutor, self).__init__(**kwargs)
def __repr__(self):
return '<Student %r>' % self.id
Then I also have app/__init__.py with the following code:
from flask import Flask
from flask_bootstrap import Bootstrap
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
#from .models import User, Task, Project, UserProject
from config import config
bootstrap = Bootstrap()
db = SQLAlchemy()
migrate = Migrate()
def create_app(config_name='default'):
#print config_name.name
app = Flask(__name__)
app.config.from_object(config[config_name])
config[config_name].init_app(app)
bootstrap.init_app(app)
db.init_app(app)
migrate.init_app(app, db)
## Register the main blueprint for main app functionality
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
return app
and app.py:
import os
from app import create_app, db
from app.models import Tutor, Student
app = create_app('default')
#app.shell_context_processor
def make_shell_context():
return dict(db=db, Tutor=Tutor, Student=Student)
I can run flask db init with no problem and it creates the migrations directory and all necessary files with the following output:
Creating directory /Users/Jasmine/projects/flask/flask-tutoring/migrations ... done
Creating directory /Users/Jasmine/projects/flask/flask-tutoring/migrations/versions ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/script.py.mako ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/env.py ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/README ... done
Generating /Users/Jasmine/projects/flask/flask-tutoring/migrations/alembic.ini ... done
Please edit configuration/connection/logging settings in '/Users/Jasmine/projects/flask/flask-tutoring/migrations/alembic.ini' before proceeding.
but when I try and run flask db migrate alembic can't detect that I've got tables in app/models.py. I get the following output:
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.env] No changes in schema detected.
There is no migration script created, its as though models.py doesn't exist.
Apologies if this is a repeated question, but I can't find another example where its the first migration that fails and no migration script at all is created.
I've tried checking if there is already a table created somewhere by running db.drop_all() in the shell but that doesn't seem to be the problem.
UPDATE
I figured out a way to solve this on my own but would like a better understanding of why this worked.
I re-named app.py to flasktutor.py and re-ran export FLASK_APP='flasktutor.py'. Subsequently the migration worked perfectly.
Please could someone explain why when the file was called app.py and I used export FLASK_APP='app.py' the migration did not register changes to the schema.

I encountered this problem and solved it by importing my models at env.py in the migrations folder right after the following comments
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
from app.models import Student, Tutor

Pls make sure that you already import your models (Tutor, Student, ...) before your migrate.init_app(app, db)

I just ran into the same issue, following Miguel's great tutorial, and the hints here were very helpful!
My solution - which doesn't require "hacking" env.py - is to simply add
from app import models
to __init__.py.
If you have a different project layout you might need to adapt the import but it seems like you need to make sure that models gets imported properly for flask db migrate to work.

This is how I solved the problem in my case:
Import the Migrate model: from flask_migrate import Migrate
Initiate Migrate class: migrate = Migrate(app, db)
Comment db.create_all()
Drop your database now => DROP DATABASE db_name;
Create it again => CREATE DATABSE db_name OWNER owner_name;
Export you flask entry file => export FLASK_APP=name_app.py
Run flask db migrate
Note: The 6th step should be used in case you get this error:
Error: Could not locate a Flask application
Hope this will help someone.

Well, I encountered the same problem following Miguel Grinberg tutorial.
Previously I created the tables using the shell calling
db.create_all()
So, I thougth to drop the tables
db.drop_all()
and trying the migrate command again, it worked as expected:
Roberto#MyPC MINGW64 /e/Projects/Flask/flasky ((5c))
$ flask db migrate -m "initial migration - Role Users"
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate.compare] Detected added table 'roles'
INFO [alembic.autogenerate.compare] Detected added table 'users'
INFO [alembic.autogenerate.compare] Detected added index 'ix_users_username' on '['username']'
Generating E:\Projects\Flask\flasky\migrations\versions\4de323c9c089_initial_migration_role_users.py ... done
After that, I used flask-migrate to re-create the tables
$ flask db upgrade
INFO [alembic.runtime.migration] Context impl SQLiteImpl.
INFO [alembic.runtime.migration] Will assume non-transactional DDL.
INFO [alembic.runtime.migration] Running upgrade -> b641ee80a60d, initial migration - Role Users

I encountered the same problem following Miguel Grinberg tutorial.
I solved this my adding
from app.models import User, Post
to migrations/env.py

The flask-migrate works after you change your table schema.
Such as, before:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app)
class Test(db.Model):
content = db.Column(db.String(60))
and then you change your Test schema, like:
class Test(db.Model):
content = db.Column(db.String(60))
add = db.Column(db.String(60))
Now, you can use flask db migrate -m "migrate test" to work.
You will get migrate version information.

This worked for me:
In migrations/env.py, import your models
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
from app.models import * # <= Add this line
from flask import current_app

You can point to empty database only to create migration file, then return back to upgrade your real database.

After some time trying to fix it and reading about it, I just deleted the migrations directory (folder) and the app.db.
Then ran again:
"flask db init"
"flask db migrate"
"flask db upgrade"
This re-generate the directories and all working fine now.
If you want to improve your migration script, this link to an upgrade from Miguel, can be useful: https://www.youtube.com/watch?v=wpRVZFwsD70&feature=emb_logo

Flask-Sqlalchemy + Sqlalchemy-searchable returning empty list

First time on the site, so hi to all and thanks in advance. Longtime lurker and newb.
I'm working on a web app in flask, using Flask-SqlAlchemy and SqlAlchemy-Searchable (docs-> https://sqlalchemy-searchable.readthedocs.org/en/latest/index.html). For a reason I can't figure out, when I try a similar example to the code shown on the docs page:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy, BaseQuery
from sqlalchemy_searchable import SearchQueryMixin
from sqlalchemy_utils.types import TSVectorType
from sqlalchemy_searchable import make_searchable
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'postgresql+psycopg2://usr:admin#localhost/dev'
app.config['SECRET_KEY'] = 'notreallyasecret'
db = SQLAlchemy(app)
make_searchable()
class ArticleQuery(BaseQuery, SearchQueryMixin):
pass
class Article(db.Model):
query_class = ArticleQuery
__tablename__ = 'article'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.Unicode(255))
content = db.Column(db.UnicodeText)
search_vector = db.Column(TSVectorType('name', 'content'))
My search queries don't work properly. I opened a python shell and created the db, and inserted five identical articles
a= Article(name='finland',content='finland')
db.session.add(a)
db.session.commit() #a-e
with 'finland' both as name and content. According to the example:
Article.query.search(u'finland').limit(5).all()
There should be articles returned that have finland somewhere in them. In my case, I get an empty list. I get an object back if I modify the example query to:
Article.query.search(' ').first()
But it's rather useless searching for empty spaces. Any ideas?
Adding a bit more to it: I noticed in the article table, the 'search_vector tsvector' column is completely empty despite data being in the content and name columns; I'm not sure if that has anything to do with it.

I ran into this exact issue once, too, when using Flask-Script to add a manage.py management tool to my application.
The fact that the search_vector column is empty despite you having added the appropriate TSVectorType parameters means that the SQLAlchemy-Searchable trigger isn't present in the postgres DB. You can verify its absence by doing a \df+ in psql command line tool -- you will not see a trigger named article_search_vector_update. SQLAlchemy-Searchable sets up this trigger to update the content of the search_vector column when the columns named in TSVectorType(...) change.
In the case of manage.py, I had to first call:
db.configure_mappers()
Essentially, you have to configure SQLAlchemy's mappers before calling create_all(). Without doing this, SQLAlchemy-Searchable will not be given the opportunity to add its search_vector trigger to populate the TSVectorType column in the model.The SQLAlchemy-Searchable docs have more on this.
In total, a minimal manage.py that properly configures SQLAlchemy-Searchable as you require might look like:
#!/usr/bin/env python
from flask.ext.script import Manager
from app import app, db
manager = Manager(app)
#manager.command
def init_db():
"""
Drops and re-creates the SQL schema
"""
db.drop_all()
db.configure_mappers()
db.create_all()
db.session.commit()

On Collin Allen's answer: actually, the flask-sqlalchemy ''db'' exposes the configure_mappers function.
Replace:
from sqlalchemy.orm.mapper import configure_mappers
...
configure_mappers()
with:
...
db.configure_mappers()

Flask-Migrate not creating tables

I have the following models in file listpull/models.py:
from datetime import datetime
from listpull import db
class Job(db.Model):
id = db.Column(db.Integer, primary_key=True)
list_type_id = db.Column(db.Integer, db.ForeignKey('list_type.id'),
nullable=False)
list_type = db.relationship('ListType',
backref=db.backref('jobs', lazy='dynamic'))
record_count = db.Column(db.Integer, nullable=False)
status = db.Column(db.Integer, nullable=False)
sf_job_id = db.Column(db.Integer, nullable=False)
created_at = db.Column(db.DateTime, nullable=False)
compressed_csv = db.Column(db.LargeBinary)
def __init__(self, list_type, created_at=None):
self.list_type = list_type
if created_at is None:
created_at = datetime.utcnow()
self.created_at = created_at
def __repr__(self):
return '<Job {}>'.format(self.id)
class ListType(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80), unique=True, nullable=False)
def __init__(self, name):
self.name = name
def __repr__(self):
return '<ListType {}>'.format(self.name)
I call ./run.py init then ./run.py migrate then ./run.py upgrade, and I see the migration file generated, but its empty:
"""empty message
Revision ID: 5048d48b21de
Revises: None
Create Date: 2013-10-11 13:25:43.131937
"""
# revision identifiers, used by Alembic.
revision = '5048d48b21de'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
pass
### end Alembic commands ###
def downgrade():
### commands auto generated by Alembic - please adjust! ###
pass
### end Alembic commands ###
run.py
#!/usr/bin/env python
# -*- coding: utf-8 -*-
from listpull import manager
manager.run()
listpull/__init__.py
# -*- coding: utf-8 -*-
# pylint: disable-msg=C0103
""" listpull module """
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from flask.ext.script import Manager
from flask.ext.migrate import Migrate, MigrateCommand
from mom.client import SQLClient
from smartfocus.restclient import RESTClient
app = Flask(__name__)
app.config.from_object('config')
db = SQLAlchemy(app)
migrate = Migrate(app, db)
manager = Manager(app)
manager.add_command('db', MigrateCommand)
mom = SQLClient(app.config['MOM_HOST'],
app.config['MOM_USER'],
app.config['MOM_PASSWORD'],
app.config['MOM_DB'])
sf = RESTClient(app.config['SMARTFOCUS_URL'],
app.config['SMARTFOCUS_LOGIN'],
app.config['SMARTFOCUS_PASSWORD'],
app.config['SMARTFOCUS_KEY'])
import listpull.models
import listpull.views
UPDATE
If I run the shell via ./run.py shell and then do from listpull import * and call db.create_all(), I get the schema:
mark.richman#MBP:~/code/nhs-listpull$ sqlite3 app.db
-- Loading resources from /Users/mark.richman/.sqliterc
SQLite version 3.7.12 2012-04-03 19:43:07
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .schema
CREATE TABLE job (
id INTEGER NOT NULL,
list_type_id INTEGER NOT NULL,
record_count INTEGER NOT NULL,
status INTEGER NOT NULL,
sf_job_id INTEGER NOT NULL,
created_at DATETIME NOT NULL,
compressed_csv BLOB,
PRIMARY KEY (id),
FOREIGN KEY(list_type_id) REFERENCES list_type (id)
);
CREATE TABLE list_type (
id INTEGER NOT NULL,
name VARCHAR(80) NOT NULL,
PRIMARY KEY (id),
UNIQUE (name)
);
sqlite>
Unfortunately, the migrations still do not work.

When you call the migrate command Flask-Migrate (or actually Alembic underneath it) will look at your models.py and compare that to what's actually in your database.
The fact that you've got an empty migration script suggests you have updated your database to match your model through another method that is outside of Flask-Migrate's control, maybe by calling Flask-SQLAlchemy's db.create_all().
If you don't have any valuable data in your database, then open a Python shell and call db.drop_all() to empty it, then try the auto migration again.
UPDATE: I installed your project here and confirmed that migrations are working fine for me:
(venv)[miguel#miguel-linux nhs-listpull]$ ./run.py db init
Creating directory /home/miguel/tmp/mark/nhs-listpull/migrations...done
Creating directory /home/miguel/tmp/mark/nhs-listpull/migrations/versions...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/script.py.mako...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/env.pyc...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/env.py...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/README...done
Generating /home/miguel/tmp/mark/nhs-listpull/migrations/alembic.ini...done
Please edit configuration/connection/logging settings in
'/home/miguel/tmp/mark/nhs-listpull/migrations/alembic.ini' before
proceeding.
(venv)[miguel#miguel-linux nhs-listpull]$ ./run.py db migrate
INFO [alembic.migration] Context impl SQLiteImpl.
INFO [alembic.migration] Will assume non-transactional DDL.
INFO [alembic.autogenerate] Detected added table 'list_type'
INFO [alembic.autogenerate] Detected added table 'job'
Generating /home/miguel/tmp/mark/nhs-
listpull/migrations/versions/48ff3456cfd3_.py...done
Try a fresh checkout, I think your setup is correct.

Ensure to import the Models in the manage.py file (or the file with the migrate instance). You have to import the models in the file, even if you are not explicitly using them. Alembic needs these imports to migrate, and to create the tables in the database. For example:
# ... some imports ...
from api.models import User, Bucketlist, BucketlistItem # Import the models
app = create_app('dev')
manager = Manager(app)
migrate = Migrate(app, db)
manager.add_command('db', MigrateCommand)
# ... some more code here ...
if __name__ == "__main__":
manager.run()
db.create_all()

I had the same issue but a different problem caused it.
Flask-migrate workflow consists of two consequent commands:
flask db migrate
which generates the migration and
flask db upgrade
which applies the migration. I forgot to run the last one and tried to start next migration without applying the previous one.

For anyone coming who comes across this, my problem was having
db.create_all()
in my main flask application file
which created the new table without the knowledge of alembic
Simply comment it out or delete it altogether so it doesn't mess with future migrations.
but unlike #Miguel's suggestion, instead of dropping the whole database (i had important information in it), i was able to fix it by deleting the new table created by Flask SQLAlchemy and then running the migration.
and this time alembic detected the new table and created a proper migration script

I just encountered a similar problem. I'd like to share my solution for anyone else encountering this thread. For me, I had my models in a package. For example models/user.py and I tried from app.models import * which did not detect anything on the migrate. However, if I changed the import to from app.models import user this is okay why my project is young, but as I have more models a bulk import would be preferable.

Strange solve for me is: delete database and folder migrations. Then
>>> from app import db
>>> db.create_all()
After flask db init or python app.py db init and then flask db migrate or python app.py db migrate. Wow, It's strange, but it works for me.