We Keep Coding

create multiple table with different schema using flask-migrate on a single database

I am using flask-migrate to handle the creation and maintenance of the table. I have multiples tables with different schema.
class A(db.model):
__tablename__ = 'A'
__table_args__ = {'schema': 'public'}
# rest of data
class B(db.model):
__tablename__ = 'B'
__table_args__ = {'schema': 'schema_b'}
# rest of data
so when I am running flask db init and flask db migrate, a migration script is created in the migration folder. But when I am running flask db upgrade to add tables in database it is showing me error
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidSchemaName) schema "schema_b.B" does not exist
As I search about the issue, I found this include_schemas and migrate using different schema, in both the part it is mentioned that to use include_schemas=True in configure in migration/env.py. Also, the link mentioned in the solution in the answer has invalid link, so this becoming little programmatical to me.
I have made changes accordingly. then I am running flask db migrate which is detecting all the tables with all schema. but as I am running flask db upgrate
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.InvalidSchemaName) schema "schema_b.B" does not exist
error again appears.
Help me, how to solve this problem using flask migrate. to create a table I have a SQL command which is working fine.

The include_schemas=True option makes Alembic look for tables in your non-default schemas, but it cannot generate new schemas when you add them to a model definition.
In such a case, what you have to do is run flask db migrate to generate the migration, and then open the generated migration script and add the schema creation right before the new table is created. For example, in the example in your question, your migration for your B model would have been generated more or less like this:
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('schema_b.B',
# ...
)
# ### end Alembic commands ###
So you need to add the schema creation statement above the table creation, so that the schema exists already when the table is created:
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.execute('create schema schema_b') # <--- add this
op.create_table('schema_b.B',
# ...
)
# ### end Alembic commands ###
And then for consistency also drop the schema on the downgrade path:
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('schema_b.B')
op.execute('drop schema schema_b') # <--- add this
# ### end Alembic commands ###

Alembic operations with multiple schemas

NOTE: This is a question I have already found the answer for, but wanted to share it so it could help other people facing the same problem.
I was trying to perform some alembic operations in my multi-schema postgresql database such as .add_column or .alter_table (although it will be the same question for .create_table or .drop_table). For example: op.add_column('table_name', 'new_column_name')
However, I was getting the same error saying basically that the table name could not be found. This, as far as I understand it, is caused because alembic is not recognizing the schema and is searching for that table in the public schema. Then, I tried to specify the schema in the table_name as 'schema_name.table_name' but had no luck.
I came across similar questions Perform alembic upgrade in multiple schemas or Alembic support for multiple Postgres schemas, but didn't find a satisfactory answer.

After searching for it into the alembic documentation, I found that there is actually an schema argument for the different operations. For example:
op.add_column('table_name', 'column_name', schema='schema_name')

Alembic will automatically pick up the schema from a table if it is already defined in a declarative SQLAlchemy model.
For example, with the following setup:
# models.py
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class SomeClass(Base):
__tablename__ = 'some_table'
id = Column(Integer, primary_key=True)
name = Column(String(50))
__table_args__ = {"schema": "my_schema"}
# alembic/env.py
from models import Base
target_metadata = Base.metadata
[...]
Running:
alembic revision --autogenerate -m "test"
Would result in a default migration script with a schema specified:
def upgrade_my_db():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('some_table',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=50), nullable=True),
sa.PrimaryKeyConstraint('id'),
schema='my_schema'
)
# ### end Alembic commands ###

Alembic Migrations on Multiple Models

I am attempting to create a revision with --autogenerate using Alembic for two Models, but am receiving a duplicate table keys error. Does, a schema need to be specified? If so, how can it be set? The documentation I've read says to use __table_args__ = {'schema': 'somename'}, but that hasn't helped. Any tips or suggestions are greatly appreciated.
My current setup is:
base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
workspace.py
from sqlalchemy import Column, Integer, String
from base import Base
class WorkspaceModel(Base):
__tablename__ = 'workspaces'
id = Column(Integer, primary_key=True)
name = Column(String)
host.py
from sqlalchemy import Column, Integer, String
from base import Base
class HostModel(Base):
__tablename__ = 'hosts'
id = Column(Integer, primary_key=true)
ip = Column(String)
alembic/env.py
from host import HostModel
from workspace import WorkspaceModel
target_metadata = [HostModel.metadata, WorkspaceModel.metadata]
Error
ValueError: Duplicate table keys across multiple MetaData objects: "hosts", "workspaces"

To make it clear from what #esdotzed and #univerio said, you have to use a single Base.metadata - but still import the individual models.
In the original question, this is how the alembic/env.py should look like:
from base import Base
# This two won't be referenced, but *have* to be imported to populate `Base.metadata`
from host import HostModel
from workspace import WorkspaceModel
target_metadata = Base.metadata
If you didn't import both models, the autogenerated migration would end up deleting your whole database - because Base.metadata doesn't know about any model by itself.

quoting univerio's answer from the comment section:
target_metadata should just be target_metadata = Base.metadata
Using Base.metadata doesn't mean you can remove the imports from host import HostModel and from workspace import WorkspaceModel
It worked for me.

I just want to add to #mgarciaisaia answer, it will work but the thing is when I tried changing, for example, the max length of username field of User model and running alembic revision --autogenerate -m "test migration", alembic output a migration file with empty upgrade() and downgrade() functions!
Note: the following operations will erase your data from the database so please back them up beforehand!
In order to update the changes made to the original User model, I had to
Delete the first migration file
Rerun alembic revision --autogenerate -m "update user model" and alembic upgrade head again for the changes to appear inside upgrade() and downgrade() functions of the migration file.

Hard to upgrade my staging db through flask migrate

I want to apply the changes made on local db to the cloud db.
My local db has three tables, User, email_history, event_monitor. I have deleted my local migrations folder and then ran python manage.py db init, python manage.py db migrate commands.
It creates an revision file like below.
"""empty message
Revision ID: 9bd307a576ce
Revises:
Create Date: 2017-03-01 00:10:32.344698
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import mysql
# revision identifiers, used by Alembic.
revision = '9bd307a576ce'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('user')
op.drop_table('email_history')
op.drop_table('event_monitor')
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('event_monitor',
sa.Column('id', mysql.INTEGER(display_width=11), nullable=False),
sa.Column('event_type', mysql.VARCHAR(length=80), nullable=True),
sa.Column('event_description', mysql.TEXT(), nullable=True),
sa.PrimaryKeyConstraint('id'),
mysql_default_charset=u'latin1',
mysql_engine=u'InnoDB'
)
op.create_table('email_history',
sa.Column('id', mysql.INTEGER(display_width=11), nullable=False),
sa.Column('user_id', mysql.INTEGER(display_width=11), autoincrement=False, nullable=False),
sa.Column('email_type', mysql.ENUM(u'SAMPLE'), nullable=True),
sa.Column('datetime_created', mysql.DATETIME(), nullable=True),
sa.Column('datetime_sent', mysql.DATETIME(), nullable=True),
sa.Column('status', mysql.TINYINT(display_width=1), autoincrement=False, nullable=False),
sa.ForeignKeyConstraint(['user_id'], [u'user.id'], name=u'email_history_ibfk_1', ondelete=u'CASCADE'),
sa.PrimaryKeyConstraint('id'),
mysql_default_charset=u'latin1',
mysql_engine=u'InnoDB'
)
op.create_table('user',
sa.Column('id', mysql.INTEGER(display_width=11), nullable=False),
sa.Column('username', mysql.VARCHAR(length=80), nullable=True),
sa.Column('email', mysql.VARCHAR(length=120), nullable=False),
sa.Column('password_hash', mysql.VARCHAR(length=256), nullable=True),
sa.PrimaryKeyConstraint('id'),
mysql_default_charset=u'latin1',
mysql_engine=u'InnoDB'
)
# ### end Alembic commands ###
If I do
export config=prod && python manage.py db upgrade
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1051, "Unknown table 's2sdevdb.user'") [SQL: u'\nDROP TABLE user'] error. Yep, I already deleted my tables on cloud db.
And my question is, why the migrate command fails to create code for creating tables? How I achieve this task?

Locally, you deleted your migrations, but your data still exists. When you create a migration, Alembic looks at your code and finds some models, looks at the database and finds those tables still present, so does not generate code to create the tables.
Presumably, there are no longer models representing the three tables it generated drop commands for.
To create a migration representing all your models, your database must be empty. Either drop the tables or point to an empty database. Alembic generates an alembic_version table which you may also need to drop.
Remotely, you dropped the tables, then tried to run a migration that drops the tables. It fails for the given reason: the tables don't exist to be dropped.
Since you messed up and performed the migration manually, use manage.py db stamp head to tell Alembic that your database already represents the current migration.
Assuming you really did reset the remote database, then the command you showed works fine to run your new migration.

Creating seed data in a flask-migrate or alembic migration

How can I insert some seed data in my first migration? If the migration is not the best place for this, then what is the best practice?
"""empty message
Revision ID: 384cfaaaa0be
Revises: None
Create Date: 2013-10-11 16:36:34.696069
"""
# revision identifiers, used by Alembic.
revision = '384cfaaaa0be'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.create_table('list_type',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('list_type_id', sa.Integer(), nullable=False),
sa.Column('record_count', sa.Integer(), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.Column('sf_job_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('compressed_csv', sa.LargeBinary(), nullable=True),
sa.ForeignKeyConstraint(['list_type_id'], ['list_type.id'], ),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
# ==> INSERT SEED DATA HERE <==
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table('job')
op.drop_table('list_type')
### end Alembic commands ###

Alembic has, as one of its operation, bulk_insert(). The documentation gives the following example (with some fixes I've included):
from datetime import date
from sqlalchemy.sql import table, column
from sqlalchemy import String, Integer, Date
from alembic import op
# Create an ad-hoc table to use for the insert statement.
accounts_table = table('account',
column('id', Integer),
column('name', String),
column('create_date', Date)
)
op.bulk_insert(accounts_table,
[
{'id':1, 'name':'John Smith',
'create_date':date(2010, 10, 5)},
{'id':2, 'name':'Ed Williams',
'create_date':date(2007, 5, 27)},
{'id':3, 'name':'Wendy Jones',
'create_date':date(2008, 8, 15)},
]
)
Note too that the alembic has an execute() operation, which is just like the normal execute() function in SQLAlchemy: you can run any SQL you wish, as the documentation example shows:
from sqlalchemy.sql import table, column
from sqlalchemy import String
from alembic import op
account = table('account',
column('name', String)
)
op.execute(
account.update().\
where(account.c.name==op.inline_literal('account 1')).\
values({'name':op.inline_literal('account 2')})
)
Notice that the table that is being used to create the metadata that is used in the update statement is defined directly in the schema. This might seem like it breaks DRY (isn't the table already defined in your application), but is actually quite necessary. If you were to try to use the table or model definition that is part of your application, you would break this migration when you make changes to your table/model in your application. Your migration scripts should be set in stone: a change to a future version of your models should not change migrations scripts. Using the application models will mean that the definitions will change depending on what version of the models you have checked out (most likely the latest). Therefore, you need the table definition to be self-contained in the migration script.
Another thing to talk about is whether you should put your seed data into a script that runs as its own command (such as using a Flask-Script command, as shown in the other answer). This can be used, but you should be careful about it. If the data you're loading is test data, then that's one thing. But I've understood "seed data" to mean data that is required for the application to work correctly. For example, if you need to set up records for "admin" and "user" in the "roles" table. This data SHOULD be inserted as part of the migrations. Remember that a script will only work with the latest version of your database, whereas a migration will work with the specific version that you are migrating to or from. If you wanted a script to load the roles info, you could need a script for every version of the database with a different schema for the "roles" table.
Also, by relying on a script, you would make it more difficult for you to run the script between migrations (say migration 3->4 requires that the seed data in the initial migration to be in the database). You now need to modify Alembic's default way of running to run these scripts. And that's still not ignoring the problems with the fact that these scripts would have to change over time, and who knows what version of your application you have checked out from source control.

Migrations should be limited to schema changes only, and not only that, it is important that when a migration up or down is applied that data that existed in the database from before is preserved as much as possible. Inserting seed data as part of a migration may mess up pre-existing data.
As most things with Flask, you can implement this in many ways. Adding a new command to Flask-Script is a good way to do this, in my opinion. For example:
#manager.command
def seed():
"Add seed data to the database."
db.session.add(...)
db.session.commit()
So then you run:
python manager.py seed

MarkHildreth has supplied an excellent explanation of how alembic can handle this. However, the OP was specifically about how to modify a flask-migration migration script. I'm going to post an answer to that below to save people the time of having to look into alembic at all.
Warning
Miguel's answer is accurate with respect to normal database information. That is to say, one should follow his advice and absolutely not use this approach to populate a database with "normal" rows. This approach is specifically for database rows which are required for the application to function, a kind of data which I think of as "seed" data.
OP's script modified to seed data:
"""empty message
Revision ID: 384cfaaaa0be
Revises: None
Create Date: 2013-10-11 16:36:34.696069
"""
# revision identifiers, used by Alembic.
revision = '384cfaaaa0be'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
list_type_table = op.create_table('list_type',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('list_type_id', sa.Integer(), nullable=False),
sa.Column('record_count', sa.Integer(), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.Column('sf_job_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('compressed_csv', sa.LargeBinary(), nullable=True),
sa.ForeignKeyConstraint(['list_type_id'], ['list_type.id'], ),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
op.bulk_insert(
list_type_table,
[
{'name':'best list'},
{'name': 'bester list'}
]
)
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table('job')
op.drop_table('list_type')
### end Alembic commands ###
Context for those new to flask_migrate
Flask migrate generates migration scripts at migrations/versions. These scripts are run in order on a database in order to bring it up to the latest version. The OP includes an example of one of these auto-generated migration scripts. In order to add seed data, one must manually modify the appropriate auto-generated migration file. The code I have posted above is an example of that.
What changed?
Very little. You will note that in the new file I am storing the table returned from create_table for list_type in a variable called list_type_table. We then operate on that table using op.bulk_insert to create a few example rows.

You can also use Python's faker library which may be a bit quicker as you don't need to come up with any data yourself. One way of configuring it would be to put a method in a class that you wanted to generate data for as shown below.
from extensions import bcrypt, db
class User(db.Model):
# this config is used by sqlalchemy to store model data in the database
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(150))
email = db.Column(db.String(100), unique=True)
password = db.Column(db.String(100))
def __init__(self, name, email, password, fav_movie):
self.name = name
self.email = email
self.password = password
#classmethod
def seed(cls, fake):
user = User(
name = fake.name(),
email = fake.email(),
password = cls.encrypt_password(fake.password()),
)
user.save()
#staticmethod
def encrypt_password(password):
return bcrypt.generate_password_hash(password).decode('utf-8')
def save(self):
db.session.add(self)
db.session.commit()
And then implement a method that calls the seed method which could look something like this:
from faker import Faker
from users.models import User
fake = Faker()
for _ in range(100):
User.seed(fake)

If you prefer to have a separate function to seed your data, you could do something like this:
from alembic import op
import sqlalchemy as sa
from models import User
def upgrade():
op.create_table('users',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
# data seed
seed()
def seed():
op.bulk_insert(User.__table__,
[
{'name': 'user1'},
{'name': 'user2'},
...
]
)
This way, you don't need to save the return of create_table into a separate variable to then pass it on to bulk_insert.