We Keep Coding

SQLAlchemy: ForeignKey across schemas

In my postgres server we have a database database with 2 schemas: public and api.
public has several tables, and I need to create a table in api with a foreign key to a table in public called model.
So it's:
-Schemas
--public
---tables
----models
--api
---tables
Using SQLAlchemy I have the following class:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
However upon launch I receive the following error:
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'training.model_id' could not find table 'public.model' with which to generate a foreign key to target column 'id'
It seems it does look for the correct thing but can't find it? I'm very confused as to why this is happening, especially because the error refers to not finding "public", which is created by default by postgres, rather than "api" which I created myself in pgAdmin.
Am I missing some cruicial config?

The error you are getting means that you are trying to create a foreign key referencing a table that SQLAlchemy does not know about. You can tell sqlalchemy about it by creating a Table associated with the same MetaData describing the referenced table. You can also do this using sqlalchemy's reflection capabilities. For example:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
metadata.reflect(schema="public", only=["model"])
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
By default, MetaData.create_all() will check for the existence of tables first, before creating them, but you can also specify the exact tables to create: metadata.create_all(tables=[self.__table])

Alembic Migrations on Multiple Models

I am attempting to create a revision with --autogenerate using Alembic for two Models, but am receiving a duplicate table keys error. Does, a schema need to be specified? If so, how can it be set? The documentation I've read says to use __table_args__ = {'schema': 'somename'}, but that hasn't helped. Any tips or suggestions are greatly appreciated.
My current setup is:
base.py
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
workspace.py
from sqlalchemy import Column, Integer, String
from base import Base
class WorkspaceModel(Base):
__tablename__ = 'workspaces'
id = Column(Integer, primary_key=True)
name = Column(String)
host.py
from sqlalchemy import Column, Integer, String
from base import Base
class HostModel(Base):
__tablename__ = 'hosts'
id = Column(Integer, primary_key=true)
ip = Column(String)
alembic/env.py
from host import HostModel
from workspace import WorkspaceModel
target_metadata = [HostModel.metadata, WorkspaceModel.metadata]
Error
ValueError: Duplicate table keys across multiple MetaData objects: "hosts", "workspaces"

To make it clear from what #esdotzed and #univerio said, you have to use a single Base.metadata - but still import the individual models.
In the original question, this is how the alembic/env.py should look like:
from base import Base
# This two won't be referenced, but *have* to be imported to populate `Base.metadata`
from host import HostModel
from workspace import WorkspaceModel
target_metadata = Base.metadata
If you didn't import both models, the autogenerated migration would end up deleting your whole database - because Base.metadata doesn't know about any model by itself.

quoting univerio's answer from the comment section:
target_metadata should just be target_metadata = Base.metadata
Using Base.metadata doesn't mean you can remove the imports from host import HostModel and from workspace import WorkspaceModel
It worked for me.

I just want to add to #mgarciaisaia answer, it will work but the thing is when I tried changing, for example, the max length of username field of User model and running alembic revision --autogenerate -m "test migration", alembic output a migration file with empty upgrade() and downgrade() functions!
Note: the following operations will erase your data from the database so please back them up beforehand!
In order to update the changes made to the original User model, I had to
Delete the first migration file
Rerun alembic revision --autogenerate -m "update user model" and alembic upgrade head again for the changes to appear inside upgrade() and downgrade() functions of the migration file.

How to use Enum with SQLAlchemy and Alembic?

Here's my Post model:
class Post(Base):
__tablename__ = 'posts'
title = db.Column(db.String(120), nullable=False)
description = db.Column(db.String(2048), nullable=False)
I'd like to add Enum status to it. So, I've created a new Enum:
import enum
class PostStatus(enum.Enum):
DRAFT='draft'
APPROVE='approve'
PUBLISHED='published'
And added a new field to model:
class Post(Base):
...
status = db.Column(db.Enum(PostStatus), nullable=False, default=PostStatus.DRAFT.value, server_default=PostStatus.DRAFT.value)
After doing FLASK_APP=server.py flask db migrate, a such migration was generated:
def upgrade():
op.add_column('posts', sa.Column('status', sa.Enum('DRAFT', 'APPROVE', 'PUBLISHED', name='poststatus'), server_default='draft', nullable=False))
After trying to upgrade DB, I'm getting:
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) type "poststatus" does not exist
LINE 1: ALTER TABLE posts ADD COLUMN status poststatus DEFAULT 'draf...
^
[SQL: "ALTER TABLE posts ADD COLUMN status poststatus DEFAULT 'draft' NOT NULL"]
Why type poststatus was not created on DB-level automatically? In the similar migration it was.
How to specify server_default option properly? I need both ORM-level defaults and DB-level ones, because I'm altering existing rows, so ORM defaults are not applied.
Why real values in DB are 'DRAFT', 'APPROVE', 'PUBLISHED', but not draft, etc? I supposed there should be ENUM values, not names.
Thank you in advance.

Why real values in DB are 'DRAFT', 'APPROVE', 'PUBLISHED', but not draft, etc? I supposed there should be ENUM values, not names.
As Peter Bašista's already mentioned SQLAlchemy uses the enum names (DRAFT, APPROVE, PUBLISHED) in the database. I assume that was done because the enum values ("draft", "approve", ...) can be arbitrary types in Python and they are not guaranteed to be unique (unless #unique is used).
However since SQLAlchemy 1.2.3 the Enum class accepts a parameter values_callable which can be used to store enum values in the database:
status = db.Column(
db.Enum(PostStatus, values_callable=lambda obj: [e.value for e in obj]),
nullable=False,
default=PostStatus.DRAFT.value,
server_default=PostStatus.DRAFT.value
)
Why type poststatus was not created on DB-level automatically? In the similar migration it was.
I think basically you are hitting a limitation of alembic: It won't handle enums on PostgreSQL correctly in some cases. I suspect the main issue in your case is Autogenerate doesn't correctly handle postgresql enums #278.
I noticed that the type is created correctly if I use alembic.op.create_table so my workaround is basically:
enum_type = SQLEnum(PostStatus, values_callable=lambda enum: [e.value for e in enum])
op.create_table(
'_dummy',
sa.Column('id', Integer, primary_key=True),
sa.Column('status', enum_type)
)
op.drop_table('_dummy')
c_status = Column('status', enum_type, nullable=False)
add_column('posts', c_status)

Use the following function example in case you are using PostgreSQL:
from sqlalchemy.dialects import postgresql
from ... import PostStatus
from alembic import op
import sqlalchemy as sa
def upgrade():
post_status = postgresql.ENUM(PostStatus, name="status")
post_status.create(op.get_bind(), checkfirst=True)
op.add_column('posts', sa.Column('status', post_status))
def downgrade():
post_status = postgresql.ENUM(PostStatus, name="status")
post_status.drop(op.get_bind())

This and related StackOverflow threads resort to PostgreSQL dialect-specific typing. However, generic support may be easily achieved in an Alembic migration as follows.
First, import the Python enum, the SQLAlchemy Enum, and your SQLAlchemy declarative base wherever you're going to declare your custom SQLAlchemy Enum column type.
import enum
from sqlalchemy import Enum
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
Let's take OP's original Python enumerated class:
class PostStatus(enum.Enum):
DRAFT='draft'
APPROVE='approve'
PUBLISHED='published'
Now we create a SQLAlchemy Enum instantiation:
PostStatusType: Enum = Enum(
PostStatus,
name="post_status_type",
create_constraint=True,
metadata=Base.metadata,
validate_strings=True,
)
When you run your Alembic alembic revision --autogenerate -m "Revision Notes" and try to apply the revision with alembic upgrade head, you'll likely get an error about the type not existing. For example:
...
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject) type "post_status_type" does not exist
LINE 10: post_status post_status_type NOT NULL,
...
To fix this, import your SQLAlchemy Enum class and add the following to your upgrade() and downgrade() functions in the Alembic autogenerated revision script.
from myproject.database import PostStatusType
...
def upgrade() -> None:
PostStatusType.create(op.get_bind(), checkfirst=True)
... the remainder of the autogen code...
def downgrade() -> None:
...the autogen code...
PostStatusType.drop(op.get_bind(), checkfirst=True)
Finally, be sure to update the auto-generated sa.Column() declaration in the table(s) using the enumerated type to simply reference the SQLAlchemy Enum type instead of using Alembic's attempt to re-declare it. For example in def upgrade() -> None:
op.create_table(
"my_table",
sa.Column(
"post_status",
PostStatusType,
nullable=False,
),
)

I can only answer the third part of your question.
The documentation for the Enum type in SQLAlchemy states that:
Above, the string names of each element, e.g. “one”, “two”, “three”, are persisted to the database; the values of the Python Enum, here indicated as integers, are not used; the value of each enum can therefore be any kind of Python object whether or not it is persistable.
So, it is by SQLAlchemy design that Enum names, not values are persisted into the database.

Alembic --autogenerate tries to recreate every table

I am trying to autogenerate an alembic revision for the first time against a pre-existing database but when I run the following command
alembic revision --autogenerate
It generates a migration which attempts to create every table and index in my database. Similar to this:
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.create_table('table1',
sa.Column('id', sa.SmallInteger(), nullable=False),
sa.Column('name', sa.String(length=100), nullable=True),
sa.Column('desc', sa.Text(), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name'),
schema='schema1'
)
op.create_index(op.f('ix_index1'), 'table1', ['name'], unique=False, schema='schema1')
... all my other tables/indexes ..
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_index1'), table_name='table1', schema='schema1')
op.drop_table('table1', schema='schema1')
... all my other tables/indexes ..
Then if I try and run the migration it fails because the objects already exist:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) relation "table1" already exists
So it looks to me like alembic thinks that my database doesn't contain any tables, but it does.
Any ideas why this might be happening?

Configure alembic to look at your database
Have you set the target_metadata to your Base meta data?
From the documentation.
To use autogenerate, we first need to modify our env.py so that it
gets access to a table metadata object that contains the target.
Suppose our application has a declarative base in myapp.mymodel. This
base contains a MetaData object which contains Table objects defining
our database. We make sure this is loaded in env.py and then passed to
EnvironmentContext.configure() via the target_metadata argument. The
env.py sample script used in the generic template already has a
variable declaration near the top for our convenience, where we
replace None with our MetaData. Starting with:
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = None
we change to:
from myapp.mymodel import Base
target_metadata = Base.metadata

Creating seed data in a flask-migrate or alembic migration

How can I insert some seed data in my first migration? If the migration is not the best place for this, then what is the best practice?
"""empty message
Revision ID: 384cfaaaa0be
Revises: None
Create Date: 2013-10-11 16:36:34.696069
"""
# revision identifiers, used by Alembic.
revision = '384cfaaaa0be'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.create_table('list_type',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('list_type_id', sa.Integer(), nullable=False),
sa.Column('record_count', sa.Integer(), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.Column('sf_job_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('compressed_csv', sa.LargeBinary(), nullable=True),
sa.ForeignKeyConstraint(['list_type_id'], ['list_type.id'], ),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
# ==> INSERT SEED DATA HERE <==
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table('job')
op.drop_table('list_type')
### end Alembic commands ###

Alembic has, as one of its operation, bulk_insert(). The documentation gives the following example (with some fixes I've included):
from datetime import date
from sqlalchemy.sql import table, column
from sqlalchemy import String, Integer, Date
from alembic import op
# Create an ad-hoc table to use for the insert statement.
accounts_table = table('account',
column('id', Integer),
column('name', String),
column('create_date', Date)
)
op.bulk_insert(accounts_table,
[
{'id':1, 'name':'John Smith',
'create_date':date(2010, 10, 5)},
{'id':2, 'name':'Ed Williams',
'create_date':date(2007, 5, 27)},
{'id':3, 'name':'Wendy Jones',
'create_date':date(2008, 8, 15)},
]
)
Note too that the alembic has an execute() operation, which is just like the normal execute() function in SQLAlchemy: you can run any SQL you wish, as the documentation example shows:
from sqlalchemy.sql import table, column
from sqlalchemy import String
from alembic import op
account = table('account',
column('name', String)
)
op.execute(
account.update().\
where(account.c.name==op.inline_literal('account 1')).\
values({'name':op.inline_literal('account 2')})
)
Notice that the table that is being used to create the metadata that is used in the update statement is defined directly in the schema. This might seem like it breaks DRY (isn't the table already defined in your application), but is actually quite necessary. If you were to try to use the table or model definition that is part of your application, you would break this migration when you make changes to your table/model in your application. Your migration scripts should be set in stone: a change to a future version of your models should not change migrations scripts. Using the application models will mean that the definitions will change depending on what version of the models you have checked out (most likely the latest). Therefore, you need the table definition to be self-contained in the migration script.
Another thing to talk about is whether you should put your seed data into a script that runs as its own command (such as using a Flask-Script command, as shown in the other answer). This can be used, but you should be careful about it. If the data you're loading is test data, then that's one thing. But I've understood "seed data" to mean data that is required for the application to work correctly. For example, if you need to set up records for "admin" and "user" in the "roles" table. This data SHOULD be inserted as part of the migrations. Remember that a script will only work with the latest version of your database, whereas a migration will work with the specific version that you are migrating to or from. If you wanted a script to load the roles info, you could need a script for every version of the database with a different schema for the "roles" table.
Also, by relying on a script, you would make it more difficult for you to run the script between migrations (say migration 3->4 requires that the seed data in the initial migration to be in the database). You now need to modify Alembic's default way of running to run these scripts. And that's still not ignoring the problems with the fact that these scripts would have to change over time, and who knows what version of your application you have checked out from source control.

Migrations should be limited to schema changes only, and not only that, it is important that when a migration up or down is applied that data that existed in the database from before is preserved as much as possible. Inserting seed data as part of a migration may mess up pre-existing data.
As most things with Flask, you can implement this in many ways. Adding a new command to Flask-Script is a good way to do this, in my opinion. For example:
#manager.command
def seed():
"Add seed data to the database."
db.session.add(...)
db.session.commit()
So then you run:
python manager.py seed

MarkHildreth has supplied an excellent explanation of how alembic can handle this. However, the OP was specifically about how to modify a flask-migration migration script. I'm going to post an answer to that below to save people the time of having to look into alembic at all.
Warning
Miguel's answer is accurate with respect to normal database information. That is to say, one should follow his advice and absolutely not use this approach to populate a database with "normal" rows. This approach is specifically for database rows which are required for the application to function, a kind of data which I think of as "seed" data.
OP's script modified to seed data:
"""empty message
Revision ID: 384cfaaaa0be
Revises: None
Create Date: 2013-10-11 16:36:34.696069
"""
# revision identifiers, used by Alembic.
revision = '384cfaaaa0be'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
list_type_table = op.create_table('list_type',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('list_type_id', sa.Integer(), nullable=False),
sa.Column('record_count', sa.Integer(), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.Column('sf_job_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('compressed_csv', sa.LargeBinary(), nullable=True),
sa.ForeignKeyConstraint(['list_type_id'], ['list_type.id'], ),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
op.bulk_insert(
list_type_table,
[
{'name':'best list'},
{'name': 'bester list'}
]
)
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table('job')
op.drop_table('list_type')
### end Alembic commands ###
Context for those new to flask_migrate
Flask migrate generates migration scripts at migrations/versions. These scripts are run in order on a database in order to bring it up to the latest version. The OP includes an example of one of these auto-generated migration scripts. In order to add seed data, one must manually modify the appropriate auto-generated migration file. The code I have posted above is an example of that.
What changed?
Very little. You will note that in the new file I am storing the table returned from create_table for list_type in a variable called list_type_table. We then operate on that table using op.bulk_insert to create a few example rows.

You can also use Python's faker library which may be a bit quicker as you don't need to come up with any data yourself. One way of configuring it would be to put a method in a class that you wanted to generate data for as shown below.
from extensions import bcrypt, db
class User(db.Model):
# this config is used by sqlalchemy to store model data in the database
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(150))
email = db.Column(db.String(100), unique=True)
password = db.Column(db.String(100))
def __init__(self, name, email, password, fav_movie):
self.name = name
self.email = email
self.password = password
#classmethod
def seed(cls, fake):
user = User(
name = fake.name(),
email = fake.email(),
password = cls.encrypt_password(fake.password()),
)
user.save()
#staticmethod
def encrypt_password(password):
return bcrypt.generate_password_hash(password).decode('utf-8')
def save(self):
db.session.add(self)
db.session.commit()
And then implement a method that calls the seed method which could look something like this:
from faker import Faker
from users.models import User
fake = Faker()
for _ in range(100):
User.seed(fake)

If you prefer to have a separate function to seed your data, you could do something like this:
from alembic import op
import sqlalchemy as sa
from models import User
def upgrade():
op.create_table('users',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
# data seed
seed()
def seed():
op.bulk_insert(User.__table__,
[
{'name': 'user1'},
{'name': 'user2'},
...
]
)
This way, you don't need to save the return of create_table into a separate variable to then pass it on to bulk_insert.