We Keep Coding

Get alembic autogenerate migrations for all schemas in Postgres database

Expectation
Support Postgres Multitenancy using Schemas with Sqlalchemy and alembic.
Model
class User(Base):
__tablename__ = 'users'
__table_args__ = ({"schema": "test"})
id = Column(Integer, primary_key=True)
name = Column(String(80), unique=True, nullable=False)
def __repr__(self):
return '<User %r>' % self.name
We have two tenants tenant_1, tenant_2. Each tenant will have its own schema created in the same Postgres DB and one more schema to maintain shared tables between all tenants. Each tenant schema will have exact same tables.
So, the use case is any model that we have in models.py should be created in all tenant schemas. So the users table has to be created in both schemas tenant_1, tenant_2.
I am trying to use alembic to auto-generate migrations for all schemas for the same model.
alembic env.py
target_metadata = app.Base.metadata
...
def run_migrations_online() -> None:
""" Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
connectable = engine_from_config(
config.get_section(config.config_ini_section),
prefix="sqlalchemy.",
poolclass=pool.NullPool,
)
with connectable.connect() as connection:
all_my_tenant_names = ["tenant_1", "tenant_2"]
for tenant_schema_name in all_my_tenant_names:
conn = connection.execution_options(schema_translate_map={"test": tenant_schema_name})
print("Migrating tenant schema %s" % tenant_schema_name)
context.configure(
connection=conn, target_metadata=target_metadata, include_schemas=True,
version_table_schema=target_metadata.schema,
)
with context.begin_transaction():
context.execute('SET search_path TO public')
context.run_migrations()
The expectation is migration file will have statements to create tenant_1.users, tenant_2.users tables. But it only has a statement to create test.users table.
def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('users',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name'),
schema='test'
)
op.drop_table('user', schema='tenant_2')
op.drop_table('alembic_version', schema='tenant_2')
op.drop_table('user', schema='tenant_1')
op.drop_table('alembic_version', schema='tenant_1')
# ### end Alembic commands ###
Can someone please help to accomplish this?

Generate database schema using SQLAlchemy [duplicate]

This question already has answers here:
SQLAlchemy printing raw SQL from create()
(6 answers)
Closed 3 years ago.
I have a very simple One-to-Many database schema (a parent can have many children but a child can only have one parent). My SQLAlchemy models looks like so:
from sqlalchemy import Table, Column, Integer, ForeignKey
from sqlalchemy.orm import relationship
from models import Base
class Parent(Base):
__tablename__ = 'Parent'
id = Column(Integer, primary_key=True)
children = relationship('Child', backref='parent')
class Child(Base):
__tablename__ = 'Child'
id = Column(Integer, primary_key=True)
parent_id = Column(Integer, ForeignKey('Parent.id'))
I'm able to create the tables in the database using the following commands:
engine = create_engine('mysql://localhost:3306/testdb')
Base.metadata.create_all(engine)
I'm curious what the raw SQL looks like to create these tables. I'd imagine it would look something like this, but I would like to be sure:
CREATE TABLE Parent (
id INT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (id)
);
CREATE TABLE Child (
id INT NOT NULL AUTO_INCREMENT,
parent_id int,
PRIMARY KEY (id),
CONSTRAINT FK_Parent FOREIGN KEY (parent_id) REFERENCES Parent(id)
);
Is there anyway to generate the database schema in raw sql using SQLAlchemy? I know I can generate a query in raw sql but I'm wondering how to generate the initial database schema.
On top of that, is there anyway to generate the schema depending on the actual database type (e.g. the raw sql would look slightly different for MySQL and PostgreSQL)?

how-can-i-get-the-create-table-drop-table-output-as-a-string
from sqlalchemy import MetaData, Table, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.schema import CreateTable
import sqlalchemy
print(sqlalchemy.__version__)
Base = declarative_base()
class Parent(Base):
__tablename__ = 'Parent'
id = Column(Integer, primary_key=True)
children = Column(String(255))
user = Table('users', MetaData(bind=None),
Column('id', Integer(), primary_key=True, nullable=False),
Column('name', String()),
Column('fullname', String()),
Column('password', String()), schema=None)
print(CreateTable(Parent.__table__))
print(CreateTable(user))
Output:
1.3.0b1
CREATE TABLE "Parent" (
id INTEGER NOT NULL,
children VARCHAR(255),
PRIMARY KEY (id)
)
CREATE TABLE users (
id INTEGER NOT NULL,
name VARCHAR,
fullname VARCHAR,
password VARCHAR,
PRIMARY KEY (id)
)

SQLAlchemy Can't Find Column when adding ForeignKey to another table

I have been fighting this error for a few days now in a variety of configurations.
I tried adding Relationship lines under each of the ForeignKeys with no luck.
I also swapped my queries from using the declarative_base style calls to SQL queries.
The error only appears once I add the second ForeignKey to Task. (I originally had several foreign keys in Task and Project but pared things down until it worked and tried slowly adding things back. Adding the second ForeignKey did it no matter which table it was added to.)
My current Flask method is:
#app.route('/project/<int:project_id>/')
def showProject(project_id):
project = session.query(Project).from_statement(text("SELECT * FROM project WHERE project.id = project_id ORDER BY project.projnum DESC"))
tasks = session.query(Task).filter_by(project_id = project_id).all()
return render_template('project.html', project = project, tasks = tasks)
And I get a "no such column" error:
OperationalError: (sqlite3.OperationalError) no such column: task.project_id [SQL: u'SELECT task.id AS task_id, task.name AS task_name, task.description AS task_description, task.assigned_id AS task_assigned_id, task.project_id AS task_project_id, task.due AS task_due \nFROM task \nWHERE task.project_id = ?'] [parameters: (2,)]
This is the setup file for my database:
import os
import sys
import time
from sqlalchemy import Column, ForeignKey, Integer, String, Float
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship
from sqlalchemy import create_engine
Base = declarative_base()
engine = create_engine('sqlite:///gmnpm.db')
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
email = Column(String(250), nullable=False)
picture = Column(String(250))
class Task(Base):
__tablename__ = 'task'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
description = Column(String(2000))
assigned_id = Column(Integer, ForeignKey('user.id'))
project_id = Column(Integer, ForeignKey('project.id'))
due = Column(Integer, default=int(time.time()))
class Project(Base):
__tablename__ = 'project'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
description = Column(String(1000))
projnum = Column(Integer)
Base.metadata.create_all(engine)
After reading a bunch of documentation and going crazy trying random things on the off chance they work, I'm grateful for any suggestions!

How do I execute inserts and updates in an Alembic upgrade script?

I need to alter data during an Alembic upgrade.
I currently have a 'players' table in a first revision:
def upgrade():
op.create_table('player',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.Unicode(length=200), nullable=False),
sa.Column('position', sa.Unicode(length=200), nullable=True),
sa.Column('team', sa.Unicode(length=100), nullable=True)
sa.PrimaryKeyConstraint('id')
)
I want to introduce a 'teams' table. I've created a second revision:
def upgrade():
op.create_table('teams',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False)
)
op.add_column('players', sa.Column('team_id', sa.Integer(), nullable=False))
I would like the second migration to also add the following data:
Populate teams table:
INSERT INTO teams (name) SELECT DISTINCT team FROM players;
Update players.team_id based on players.team name:
UPDATE players AS p JOIN teams AS t SET p.team_id = t.id WHERE p.team = t.name;
How do I execute inserts and updates inside the upgrade script?

What you are asking for is a data migration, as opposed to the schema migration that is most prevalent in the Alembic docs.
This answer assumes you are using declarative (as opposed to class-Mapper-Table or core) to define your models. It should be relatively straightforward to adapt this to the other forms.
Note that Alembic provides some basic data functions: op.bulk_insert() and op.execute(). If the operations are fairly minimal, use those. If the migration requires relationships or other complex interactions, I prefer to use the full power of models and sessions as described below.
The following is an example migration script that sets up some declarative models that will be used to manipulate data in a session. The key points are:
Define the basic models you need, with the columns you'll need. You don't need every column, just the primary key and the ones you'll be using.
Within the upgrade function, use op.get_bind() to get the current connection, and make a session with it.
Or use bind.execute() to use SQLAlchemy's lower level to write SQL queries directly. This is useful for simple migrations.
Use the models and session as you normally would in your application.
"""create teams table
Revision ID: 169ad57156f0
Revises: 29b4c2bfce6d
Create Date: 2014-06-25 09:00:06.784170
"""
revision = '169ad57156f0'
down_revision = '29b4c2bfce6d'
from alembic import op
import sqlalchemy as sa
from sqlalchemy import orm
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Player(Base):
__tablename__ = 'players'
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String, nullable=False)
team_name = sa.Column('team', sa.String, nullable=False)
team_id = sa.Column(sa.Integer, sa.ForeignKey('teams.id'), nullable=False)
team = orm.relationship('Team', backref='players')
class Team(Base):
__tablename__ = 'teams'
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String, nullable=False, unique=True)
def upgrade():
bind = op.get_bind()
session = orm.Session(bind=bind)
# create the teams table and the players.team_id column
Team.__table__.create(bind)
op.add_column('players', sa.Column('team_id', sa.ForeignKey('teams.id'), nullable=False)
# create teams for each team name
teams = {name: Team(name=name) for name in session.query(Player.team).distinct()}
session.add_all(teams.values())
# set player team based on team name
for player in session.query(Player):
player.team = teams[player.team_name]
session.commit()
# don't need team name now that team relationship is set
op.drop_column('players', 'team')
def downgrade():
bind = op.get_bind()
session = orm.Session(bind=bind)
# re-add the players.team column
op.add_column('players', sa.Column('team', sa.String, nullable=False)
# set players.team based on team relationship
for player in session.query(Player):
player.team_name = player.team.name
session.commit()
op.drop_column('players', 'team_id')
op.drop_table('teams')
The migration defines separate models because the models in your code represent the current state of the database, while the migrations represent steps along the way. Your database might be in any state along that path, so the models might not sync up with the database yet. Unless you're very careful, using the real models directly will cause problems with missing columns, invalid data, etc. It's clearer to explicitly state exactly what columns and models you will use in the migration.

You can also use direct SQL see (Alembic Operation Reference) as in the following example:
from alembic import op
# revision identifiers, used by Alembic.
revision = '1ce7873ac4ced2'
down_revision = '1cea0ac4ced2'
branch_labels = None
depends_on = None
def upgrade():
# ### commands made by andrew ###
op.execute('UPDATE STOCK SET IN_STOCK = -1 WHERE IN_STOCK IS NULL')
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
pass
# ### end Alembic commands ###

I recommend using SQLAlchemy core statements using an ad-hoc table, as detailed in the official documentation, because it allows the use of agnostic SQL and pythonic writing and is also self-contained. SQLAlchemy Core is the best of both worlds for migration scripts.
Here is an example of the concept:
from sqlalchemy.sql import table, column
from sqlalchemy import String
from alembic import op
account = table('account',
column('name', String)
)
op.execute(
account.update().\\
where(account.c.name==op.inline_literal('account 1')).\\
values({'name':op.inline_literal('account 2')})
)
# If insert is required
from sqlalchemy.sql import insert
from sqlalchemy import orm
bind = op.get_bind()
session = orm.Session(bind=bind)
data = {
"name": "John",
}
ret = session.execute(insert(account).values(data))
# for use in other insert calls
account_id = ret.lastrowid

Can only define secondary relationship using association table object but not name

Trying to create an association relationship between the classes Note and Document. The problem I'm facing is that my secondary relationship only works when I use the association table object and not that table name. What I mean is that the relationship:
notes = relationship(u'Note', secondary=t_Documented, backref='documents')
works but the following does NOT work:
notes = relationship(u'Note', secondary='Documented', backref='documents')
When querying, I get the error:
sqlalchemy.exc.InvalidRequestError: When initializing mapper
Mapper|Document|Document, expression 'Documented' failed to locate a
name ("name 'Documented' is not defined"). If this is a class name,
consider adding this relationship() to the
class after both dependent classes have been defined.
I would rather use the name as my model is generated using sqlacodegen.
Moreover, SQLAlchemy docs say I can use the name (http://docs.sqlalchemy.org/en/rel_0_9/orm/relationships.html#many-to-many) with caveta "with the declarative extension in use". I Googled the term which led me to this. Question is how in my case can I augment the Base.
# model.py
# coding: utf-8
from sqlalchemy import Column, Date, DateTime, ForeignKey, ForeignKeyConstraint, Index, Integer, Numeric, String, Table, Text, text
from sqlalchemy.orm import backref, relationship
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
metadata = Base.metadata
t_Documented = Table(
'Documented', metadata,
Column('id', Integer, primary_key=True),
Column('note_id', ForeignKey(u'MySchema.Note.id'), nullable=False),
Column('document_id', ForeignKey(u'MySchema.Document.id'), nullable=False, index=True),
Column('inserted', DateTime, nullable=False, server_default=text("'0000-00-00 00:00:00'")),
Column('updated', DateTime, nullable=False, server_default=text("'0000-00-00 00:00:00'")),
Index('Documented_AK1', 'note_id', 'document_id'),
schema='MySchema'
)
class Note(Base):
__tablename__ = 'Note'
__table_args__ = {u'schema': 'MySchema'}
id = Column(Integer, primary_key=True)
class Document(Note):
__tablename__ = 'Document'
__table_args__ = {u'schema': 'MySchema'}
id = Column(ForeignKey(u'MySchema.Note.id'), primary_key=True)
title = Column(String(100), nullable=False)
author = Column(String(100), nullable=False)
notes = relationship(u'Note', secondary='Documented', backref='documents')
Using SQLAlchemy 0.9.4 and Python 2.6.6. Connector is MySQLDB and I'm using MySQL database.

your table has a schema of "MySchema" so that has to be part of it:
notes = relationship(u'Note', secondary='MySchema.Documented', backref='documents')