We Keep Coding

Alembic trying to delete my pre-existing Postgres table

Hello I am trying extend my pre-existing Postgres database that containing one table using SQLAlchemy. But when I am trying use migrate, I have this from alembic migration:
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.ddl.postgresql] Detected sequence named 'vocabulary_id_seq' as owned by integer column 'vocabulary(id)', assuming SERIAL and omitting
INFO [alembic.autogenerate.compare] Detected removed table 'vocabulary'
INFO [alembic.ddl.postgresql] Detected sequence named 'users_id_seq' as owned by integer column 'users(id)', assuming SERIAL and omitting
Generating /home/alex/prog/projects/FlaskHasherChallenge/migrations/versions/c4f
456d9500d_.py ... done
Here is my revision file:
def upgrade():
op.drop_table('vocabulary')
def downgrade():
op.create_table('vocabulary',
sa.Column('id', sa.INTEGER(), nullable=False),
sa.Column('word', sa.VARCHAR(length=255), autoincrement=False, nullable=True),
sa.PrimaryKeyConstraint('id', name='vocabulary_pkey')
)
My Postgres table:
id | integer | not null default nextval('vocabulary_id_seq'::regclass)
word | character varying(255) |
Part of my __ init__.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
import os
engine = create_engine(os.environ['DATABASE_URL'])
Base = declarative_base(bind=engine)
And part of my models.py:
from . import Base, db
class Vocabulary(Base):
__tablename__ = "vocabulary"
__table_args__ = {'extend_existing': True}
id = db.Column(db.Integer, primary_key=True)
word = db.Column(db.String(255))
hash_word = db.Column(db.String(JSON))
author_id = db.Column(db.Integer, db.ForeignKey("users.id"))
def __init__(self, **kwargs):
super().__init__(**kwargs)
Any ideas why this might be happening?

name 'Text' is not defined in HSTORE Sqlalchemy migration

When I try to add HSTORE to my model and try to upgrade head it throws me NameError: name 'Text' is not defined.
I used PYTHONPATH=. alembic revision --autogenerate -m "Added user_id, admin_id and role to Customer" to create the revision.
Thanks in advance?
Update:
#Daniel Roseman suggested me to add sa.Text(). My question is why it has not been generated automatically during revision
Error:
op.add_column('customers', sa.Column('user_id',
postgresql.HSTORE(text_type=Text()), nullable=True)) NameError: name
'Text' is not defined
Model:
class Customer(Base):
__tablename__ = "customers"
id = Column(Integer, primary_key=True)
name = Column(String, nullable=False)
img = Column(String, nullable=False)
auth_token = Column(String, nullable=True)
notification_config = Column(JSONB, nullable=True)
admin_id = Column(Integer, nullable=True)
user_id = Column(MutableDict.as_mutable(HSTORE))
Generated Migration revision:
"""Added user_id, admin_id and role to Customer
Revision ID: 1ebe3d18442f
Revises: 88b4dccb5c1e
Create Date: 2017-06-21 17:03:21.181933
"""
# revision identifiers, used by Alembic.
revision = '1ebe3d18442f'
down_revision = '88b4dccb5c1e'
branch_labels = None
depends_on = None
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('customers', sa.Column('admin_id', sa.Integer(), nullable=True))
op.add_column('customers', sa.Column('auth_token', sa.String(), nullable=True))
op.add_column('customers', sa.Column('user_id', postgresql.HSTORE(text_type=Text()), nullable=True))
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column('customers', 'user_id')
op.drop_column('customers', 'auth_token')
op.drop_column('customers', 'admin_id')
# ### end Alembic commands ###

Getting Flask-Migrate to Ignore SQL Views that are mapped as Flask-SQLAlchemy Models

I am using Flask-SQLAlchemy to define my models, and then using Flask-Migrate to auto-generate migration scripts for deployment onto a PostgreSQL database. I have defined a number of SQL Views on the database that I use in my application like below.
However, Flask-Migrate now generates a migration file for the view as it thinks it's a table. How do I correctly get Flask-Migrate / Alembic to ignore the view during autogenerate?
SQL View name: vw_SampleView with two columns: id and rowcount.
class ViewSampleView(db.Model):
__tablename__ = 'vw_report_high_level_count'
info = dict(is_view=True)
id = db.Column(db.String(), primary_key=True)
rowcount = db.Column(db.Integer(), nullable=False)
Which means I can now do queries like so:
ViewSampleView.query.all()
I tried following instructions on http://alembic.zzzcomputing.com/en/latest/cookbook.html and added the info = dict(is_view=True) portion to my model and the following bits to my env.py file, but don't know where to go from here.
def include_object(object, name, type_, reflected, compare_to):
"""
Exclude views from Alembic's consideration.
"""
return not object.info.get('is_view', False)
...
context.configure(url=url,include_object = include_object)

I think (though haven't tested) that you can mark your Table as a view with the __table_args__ attribute:
class ViewSampleView(db.Model):
__tablename__ = 'vw_report_high_level_count'
__table_args__ = {'info': dict(is_view=True)}
id = db.Column(db.String(), primary_key=True)
rowcount = db.Column(db.Integer(), nullable=False)

alembic autogenerate get the metadata for a Flask-SQLAlchemy bind

My flask application has a single database(db1) before, now I bind a new database(db2) to it, both has 10 tables.
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://root#localhost:3306/db1'
SQLALCHEMY_BINDS = {
'test': 'mysql+pymysql://root#localhost:3306/db2'
}
db = SQLAlchemy()
class table1(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
.......
class table10(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
class table11(db.Model):
__bind_key__ = 'db2'
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
......
class table20(db.Model):
__bind_key__ = 'db2'
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True)
I want to use alembic autogenerate function to auto detecting and generating migrations for db 1 and db2 separately, but db.metadata will get all tables metadata, but how to just get bind db metadata?
Thanks #davidism help! I can use include_symbol to make it.
def include_symbol(tablename, schema):
return tablename in ('table1', 'table2'.......'table10') # for db1
　# return tablename not in ('table1', 'table2'.......'table10') # for db2
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
include_symbol=include_symbol
)

You can't because that feature doesn't exist yet. Currently, there is one metadata instance for all models across all binds. As long as all the models have unique names, this isn't a huge problem.
When we apply this patch and make a new release, each bind will have its own metadata. You will then be able to access it with db.get_metadata(bind='db2').

How do I execute inserts and updates in an Alembic upgrade script?

I need to alter data during an Alembic upgrade.
I currently have a 'players' table in a first revision:
def upgrade():
op.create_table('player',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.Unicode(length=200), nullable=False),
sa.Column('position', sa.Unicode(length=200), nullable=True),
sa.Column('team', sa.Unicode(length=100), nullable=True)
sa.PrimaryKeyConstraint('id')
)
I want to introduce a 'teams' table. I've created a second revision:
def upgrade():
op.create_table('teams',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False)
)
op.add_column('players', sa.Column('team_id', sa.Integer(), nullable=False))
I would like the second migration to also add the following data:
Populate teams table:
INSERT INTO teams (name) SELECT DISTINCT team FROM players;
Update players.team_id based on players.team name:
UPDATE players AS p JOIN teams AS t SET p.team_id = t.id WHERE p.team = t.name;
How do I execute inserts and updates inside the upgrade script?

What you are asking for is a data migration, as opposed to the schema migration that is most prevalent in the Alembic docs.
This answer assumes you are using declarative (as opposed to class-Mapper-Table or core) to define your models. It should be relatively straightforward to adapt this to the other forms.
Note that Alembic provides some basic data functions: op.bulk_insert() and op.execute(). If the operations are fairly minimal, use those. If the migration requires relationships or other complex interactions, I prefer to use the full power of models and sessions as described below.
The following is an example migration script that sets up some declarative models that will be used to manipulate data in a session. The key points are:
Define the basic models you need, with the columns you'll need. You don't need every column, just the primary key and the ones you'll be using.
Within the upgrade function, use op.get_bind() to get the current connection, and make a session with it.
Or use bind.execute() to use SQLAlchemy's lower level to write SQL queries directly. This is useful for simple migrations.
Use the models and session as you normally would in your application.
"""create teams table
Revision ID: 169ad57156f0
Revises: 29b4c2bfce6d
Create Date: 2014-06-25 09:00:06.784170
"""
revision = '169ad57156f0'
down_revision = '29b4c2bfce6d'
from alembic import op
import sqlalchemy as sa
from sqlalchemy import orm
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Player(Base):
__tablename__ = 'players'
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String, nullable=False)
team_name = sa.Column('team', sa.String, nullable=False)
team_id = sa.Column(sa.Integer, sa.ForeignKey('teams.id'), nullable=False)
team = orm.relationship('Team', backref='players')
class Team(Base):
__tablename__ = 'teams'
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String, nullable=False, unique=True)
def upgrade():
bind = op.get_bind()
session = orm.Session(bind=bind)
# create the teams table and the players.team_id column
Team.__table__.create(bind)
op.add_column('players', sa.Column('team_id', sa.ForeignKey('teams.id'), nullable=False)
# create teams for each team name
teams = {name: Team(name=name) for name in session.query(Player.team).distinct()}
session.add_all(teams.values())
# set player team based on team name
for player in session.query(Player):
player.team = teams[player.team_name]
session.commit()
# don't need team name now that team relationship is set
op.drop_column('players', 'team')
def downgrade():
bind = op.get_bind()
session = orm.Session(bind=bind)
# re-add the players.team column
op.add_column('players', sa.Column('team', sa.String, nullable=False)
# set players.team based on team relationship
for player in session.query(Player):
player.team_name = player.team.name
session.commit()
op.drop_column('players', 'team_id')
op.drop_table('teams')
The migration defines separate models because the models in your code represent the current state of the database, while the migrations represent steps along the way. Your database might be in any state along that path, so the models might not sync up with the database yet. Unless you're very careful, using the real models directly will cause problems with missing columns, invalid data, etc. It's clearer to explicitly state exactly what columns and models you will use in the migration.

You can also use direct SQL see (Alembic Operation Reference) as in the following example:
from alembic import op
# revision identifiers, used by Alembic.
revision = '1ce7873ac4ced2'
down_revision = '1cea0ac4ced2'
branch_labels = None
depends_on = None
def upgrade():
# ### commands made by andrew ###
op.execute('UPDATE STOCK SET IN_STOCK = -1 WHERE IN_STOCK IS NULL')
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
pass
# ### end Alembic commands ###

I recommend using SQLAlchemy core statements using an ad-hoc table, as detailed in the official documentation, because it allows the use of agnostic SQL and pythonic writing and is also self-contained. SQLAlchemy Core is the best of both worlds for migration scripts.
Here is an example of the concept:
from sqlalchemy.sql import table, column
from sqlalchemy import String
from alembic import op
account = table('account',
column('name', String)
)
op.execute(
account.update().\\
where(account.c.name==op.inline_literal('account 1')).\\
values({'name':op.inline_literal('account 2')})
)
# If insert is required
from sqlalchemy.sql import insert
from sqlalchemy import orm
bind = op.get_bind()
session = orm.Session(bind=bind)
data = {
"name": "John",
}
ret = session.execute(insert(account).values(data))
# for use in other insert calls
account_id = ret.lastrowid