How can I insert some seed data in my first migration? If the migration is not the best place for this, then what is the best practice?
"""empty message
Revision ID: 384cfaaaa0be
Revises: None
Create Date: 2013-10-11 16:36:34.696069
"""
# revision identifiers, used by Alembic.
revision = '384cfaaaa0be'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.create_table('list_type',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('list_type_id', sa.Integer(), nullable=False),
sa.Column('record_count', sa.Integer(), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.Column('sf_job_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('compressed_csv', sa.LargeBinary(), nullable=True),
sa.ForeignKeyConstraint(['list_type_id'], ['list_type.id'], ),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
# ==> INSERT SEED DATA HERE <==
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table('job')
op.drop_table('list_type')
### end Alembic commands ###
Alembic has, as one of its operation, bulk_insert(). The documentation gives the following example (with some fixes I've included):
from datetime import date
from sqlalchemy.sql import table, column
from sqlalchemy import String, Integer, Date
from alembic import op
# Create an ad-hoc table to use for the insert statement.
accounts_table = table('account',
column('id', Integer),
column('name', String),
column('create_date', Date)
)
op.bulk_insert(accounts_table,
[
{'id':1, 'name':'John Smith',
'create_date':date(2010, 10, 5)},
{'id':2, 'name':'Ed Williams',
'create_date':date(2007, 5, 27)},
{'id':3, 'name':'Wendy Jones',
'create_date':date(2008, 8, 15)},
]
)
Note too that the alembic has an execute() operation, which is just like the normal execute() function in SQLAlchemy: you can run any SQL you wish, as the documentation example shows:
from sqlalchemy.sql import table, column
from sqlalchemy import String
from alembic import op
account = table('account',
column('name', String)
)
op.execute(
account.update().\
where(account.c.name==op.inline_literal('account 1')).\
values({'name':op.inline_literal('account 2')})
)
Notice that the table that is being used to create the metadata that is used in the update statement is defined directly in the schema. This might seem like it breaks DRY (isn't the table already defined in your application), but is actually quite necessary. If you were to try to use the table or model definition that is part of your application, you would break this migration when you make changes to your table/model in your application. Your migration scripts should be set in stone: a change to a future version of your models should not change migrations scripts. Using the application models will mean that the definitions will change depending on what version of the models you have checked out (most likely the latest). Therefore, you need the table definition to be self-contained in the migration script.
Another thing to talk about is whether you should put your seed data into a script that runs as its own command (such as using a Flask-Script command, as shown in the other answer). This can be used, but you should be careful about it. If the data you're loading is test data, then that's one thing. But I've understood "seed data" to mean data that is required for the application to work correctly. For example, if you need to set up records for "admin" and "user" in the "roles" table. This data SHOULD be inserted as part of the migrations. Remember that a script will only work with the latest version of your database, whereas a migration will work with the specific version that you are migrating to or from. If you wanted a script to load the roles info, you could need a script for every version of the database with a different schema for the "roles" table.
Also, by relying on a script, you would make it more difficult for you to run the script between migrations (say migration 3->4 requires that the seed data in the initial migration to be in the database). You now need to modify Alembic's default way of running to run these scripts. And that's still not ignoring the problems with the fact that these scripts would have to change over time, and who knows what version of your application you have checked out from source control.
Migrations should be limited to schema changes only, and not only that, it is important that when a migration up or down is applied that data that existed in the database from before is preserved as much as possible. Inserting seed data as part of a migration may mess up pre-existing data.
As most things with Flask, you can implement this in many ways. Adding a new command to Flask-Script is a good way to do this, in my opinion. For example:
#manager.command
def seed():
"Add seed data to the database."
db.session.add(...)
db.session.commit()
So then you run:
python manager.py seed
MarkHildreth has supplied an excellent explanation of how alembic can handle this. However, the OP was specifically about how to modify a flask-migration migration script. I'm going to post an answer to that below to save people the time of having to look into alembic at all.
Warning
Miguel's answer is accurate with respect to normal database information. That is to say, one should follow his advice and absolutely not use this approach to populate a database with "normal" rows. This approach is specifically for database rows which are required for the application to function, a kind of data which I think of as "seed" data.
OP's script modified to seed data:
"""empty message
Revision ID: 384cfaaaa0be
Revises: None
Create Date: 2013-10-11 16:36:34.696069
"""
# revision identifiers, used by Alembic.
revision = '384cfaaaa0be'
down_revision = None
from alembic import op
import sqlalchemy as sa
def upgrade():
### commands auto generated by Alembic - please adjust! ###
list_type_table = op.create_table('list_type',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
op.create_table('job',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('list_type_id', sa.Integer(), nullable=False),
sa.Column('record_count', sa.Integer(), nullable=False),
sa.Column('status', sa.Integer(), nullable=False),
sa.Column('sf_job_id', sa.Integer(), nullable=False),
sa.Column('created_at', sa.DateTime(), nullable=False),
sa.Column('compressed_csv', sa.LargeBinary(), nullable=True),
sa.ForeignKeyConstraint(['list_type_id'], ['list_type.id'], ),
sa.PrimaryKeyConstraint('id')
)
### end Alembic commands ###
op.bulk_insert(
list_type_table,
[
{'name':'best list'},
{'name': 'bester list'}
]
)
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_table('job')
op.drop_table('list_type')
### end Alembic commands ###
Context for those new to flask_migrate
Flask migrate generates migration scripts at migrations/versions. These scripts are run in order on a database in order to bring it up to the latest version. The OP includes an example of one of these auto-generated migration scripts. In order to add seed data, one must manually modify the appropriate auto-generated migration file. The code I have posted above is an example of that.
What changed?
Very little. You will note that in the new file I am storing the table returned from create_table for list_type in a variable called list_type_table. We then operate on that table using op.bulk_insert to create a few example rows.
You can also use Python's faker library which may be a bit quicker as you don't need to come up with any data yourself. One way of configuring it would be to put a method in a class that you wanted to generate data for as shown below.
from extensions import bcrypt, db
class User(db.Model):
# this config is used by sqlalchemy to store model data in the database
__tablename__ = 'users'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(150))
email = db.Column(db.String(100), unique=True)
password = db.Column(db.String(100))
def __init__(self, name, email, password, fav_movie):
self.name = name
self.email = email
self.password = password
#classmethod
def seed(cls, fake):
user = User(
name = fake.name(),
email = fake.email(),
password = cls.encrypt_password(fake.password()),
)
user.save()
#staticmethod
def encrypt_password(password):
return bcrypt.generate_password_hash(password).decode('utf-8')
def save(self):
db.session.add(self)
db.session.commit()
And then implement a method that calls the seed method which could look something like this:
from faker import Faker
from users.models import User
fake = Faker()
for _ in range(100):
User.seed(fake)
If you prefer to have a separate function to seed your data, you could do something like this:
from alembic import op
import sqlalchemy as sa
from models import User
def upgrade():
op.create_table('users',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=80), nullable=False),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name')
)
# data seed
seed()
def seed():
op.bulk_insert(User.__table__,
[
{'name': 'user1'},
{'name': 'user2'},
...
]
)
This way, you don't need to save the return of create_table into a separate variable to then pass it on to bulk_insert.
Related
NOTE: This is a question I have already found the answer for, but wanted to share it so it could help other people facing the same problem.
I was trying to perform some alembic operations in my multi-schema postgresql database such as .add_column or .alter_table (although it will be the same question for .create_table or .drop_table). For example: op.add_column('table_name', 'new_column_name')
However, I was getting the same error saying basically that the table name could not be found. This, as far as I understand it, is caused because alembic is not recognizing the schema and is searching for that table in the public schema. Then, I tried to specify the schema in the table_name as 'schema_name.table_name' but had no luck.
I came across similar questions Perform alembic upgrade in multiple schemas or Alembic support for multiple Postgres schemas, but didn't find a satisfactory answer.
After searching for it into the alembic documentation, I found that there is actually an schema argument for the different operations. For example:
op.add_column('table_name', 'column_name', schema='schema_name')
Alembic will automatically pick up the schema from a table if it is already defined in a declarative SQLAlchemy model.
For example, with the following setup:
# models.py
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class SomeClass(Base):
__tablename__ = 'some_table'
id = Column(Integer, primary_key=True)
name = Column(String(50))
__table_args__ = {"schema": "my_schema"}
# alembic/env.py
from models import Base
target_metadata = Base.metadata
[...]
Running:
alembic revision --autogenerate -m "test"
Would result in a default migration script with a schema specified:
def upgrade_my_db():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('some_table',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('name', sa.String(length=50), nullable=True),
sa.PrimaryKeyConstraint('id'),
schema='my_schema'
)
# ### end Alembic commands ###
I want to apply the changes made on local db to the cloud db.
My local db has three tables, User, email_history, event_monitor. I have deleted my local migrations folder and then ran python manage.py db init, python manage.py db migrate commands.
It creates an revision file like below.
"""empty message
Revision ID: 9bd307a576ce
Revises:
Create Date: 2017-03-01 00:10:32.344698
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects import mysql
# revision identifiers, used by Alembic.
revision = '9bd307a576ce'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.drop_table('user')
op.drop_table('email_history')
op.drop_table('event_monitor')
# ### end Alembic commands ###
def downgrade():
# ### commands auto generated by Alembic - please adjust! ###
op.create_table('event_monitor',
sa.Column('id', mysql.INTEGER(display_width=11), nullable=False),
sa.Column('event_type', mysql.VARCHAR(length=80), nullable=True),
sa.Column('event_description', mysql.TEXT(), nullable=True),
sa.PrimaryKeyConstraint('id'),
mysql_default_charset=u'latin1',
mysql_engine=u'InnoDB'
)
op.create_table('email_history',
sa.Column('id', mysql.INTEGER(display_width=11), nullable=False),
sa.Column('user_id', mysql.INTEGER(display_width=11), autoincrement=False, nullable=False),
sa.Column('email_type', mysql.ENUM(u'SAMPLE'), nullable=True),
sa.Column('datetime_created', mysql.DATETIME(), nullable=True),
sa.Column('datetime_sent', mysql.DATETIME(), nullable=True),
sa.Column('status', mysql.TINYINT(display_width=1), autoincrement=False, nullable=False),
sa.ForeignKeyConstraint(['user_id'], [u'user.id'], name=u'email_history_ibfk_1', ondelete=u'CASCADE'),
sa.PrimaryKeyConstraint('id'),
mysql_default_charset=u'latin1',
mysql_engine=u'InnoDB'
)
op.create_table('user',
sa.Column('id', mysql.INTEGER(display_width=11), nullable=False),
sa.Column('username', mysql.VARCHAR(length=80), nullable=True),
sa.Column('email', mysql.VARCHAR(length=120), nullable=False),
sa.Column('password_hash', mysql.VARCHAR(length=256), nullable=True),
sa.PrimaryKeyConstraint('id'),
mysql_default_charset=u'latin1',
mysql_engine=u'InnoDB'
)
# ### end Alembic commands ###
If I do
export config=prod && python manage.py db upgrade
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1051, "Unknown table 's2sdevdb.user'") [SQL: u'\nDROP TABLE user'] error. Yep, I already deleted my tables on cloud db.
And my question is, why the migrate command fails to create code for creating tables? How I achieve this task?
Locally, you deleted your migrations, but your data still exists. When you create a migration, Alembic looks at your code and finds some models, looks at the database and finds those tables still present, so does not generate code to create the tables.
Presumably, there are no longer models representing the three tables it generated drop commands for.
To create a migration representing all your models, your database must be empty. Either drop the tables or point to an empty database. Alembic generates an alembic_version table which you may also need to drop.
Remotely, you dropped the tables, then tried to run a migration that drops the tables. It fails for the given reason: the tables don't exist to be dropped.
Since you messed up and performed the migration manually, use manage.py db stamp head to tell Alembic that your database already represents the current migration.
Assuming you really did reset the remote database, then the command you showed works fine to run your new migration.
I am trying to autogenerate an alembic revision for the first time against a pre-existing database but when I run the following command
alembic revision --autogenerate
It generates a migration which attempts to create every table and index in my database. Similar to this:
def upgrade():
### commands auto generated by Alembic - please adjust! ###
op.create_table('table1',
sa.Column('id', sa.SmallInteger(), nullable=False),
sa.Column('name', sa.String(length=100), nullable=True),
sa.Column('desc', sa.Text(), nullable=True),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name'),
schema='schema1'
)
op.create_index(op.f('ix_index1'), 'table1', ['name'], unique=False, schema='schema1')
... all my other tables/indexes ..
def downgrade():
### commands auto generated by Alembic - please adjust! ###
op.drop_index(op.f('ix_index1'), table_name='table1', schema='schema1')
op.drop_table('table1', schema='schema1')
... all my other tables/indexes ..
Then if I try and run the migration it fails because the objects already exist:
sqlalchemy.exc.ProgrammingError: (ProgrammingError) relation "table1" already exists
So it looks to me like alembic thinks that my database doesn't contain any tables, but it does.
Any ideas why this might be happening?
Configure alembic to look at your database
Have you set the target_metadata to your Base meta data?
From the documentation.
To use autogenerate, we first need to modify our env.py so that it
gets access to a table metadata object that contains the target.
Suppose our application has a declarative base in myapp.mymodel. This
base contains a MetaData object which contains Table objects defining
our database. We make sure this is loaded in env.py and then passed to
EnvironmentContext.configure() via the target_metadata argument. The
env.py sample script used in the generic template already has a
variable declaration near the top for our convenience, where we
replace None with our MetaData. Starting with:
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = None
we change to:
from myapp.mymodel import Base
target_metadata = Base.metadata
Considering Storm, a python ORM, I would like to automatically generate the schema for a (mysql) database. The home page states
"Storm works well with existing database schemas." ( https://storm.canonical.com/FrontPage ),
hence I was hoping to not having to create model classes. However, the 'getting started' tutorial ( https://storm.canonical.com/Tutorial ) suggests that a class, like the one below, needs to be created manually for each table, and each field needs to be manually specified:
class Person(object):
__storm_table__ = "person"
id = Int(primary=True)
name = Unicode()
Alternatively, SQLAlchemy doesn't seem to support a reverse engineering feature either, but does need a schema like this one:
user = Table('user', metadata,
Column('user_id', Integer, primary_key = True),
Column('user_name', String(16), nullable = False),
Column('email_address', String(60)),
Column('password', String(20), nullable = False)
)
Of course, these classes/schemas make sense, since each table will likely represent some 'object of interest' and one can extend them with all kinds of functionality. However, they are tedious to create, and their (initial) content is straight forward when a database already exists.
One ORM that does allow for reverse engineering is:
http://docs.doctrine-project.org/en/2.0.x/reference/tools.html
Are there similar reverse engineering tools for Storm or SQLAlchemy or any python ORM or python database fancyfier?
I am not aware of how Storm manages this process, but you can certainly reflect tables in a database with sqlalchemy. For example, below is a basic example using a SQL Server instance that I have access to at the moment.
AN ENTIRE DATABASE
>>> from sqlalchemy import create_engine, MetaData
>>> engine = create_engine('mssql+pyodbc://<username>:<password>#<host>/<database>') # replace <username> with user name etc.
>>> meta = MetaData()
>>> meta.reflect(bind=engine)
>>> funds_table = meta.tables['funds'] # tables are stored in meta.tables dict
>>> funds_table # now stores database schema object
Table(u'funds', MetaData(bind=None), Column(u'fund_token', INTEGER(), table=<funds>, primary_key=True, nullable=False), Column(u'award_year_token', INTEGER(), ForeignKey(u'award_year_defn.award_year_token'), table=<funds>, nullable=False), ... Column(u'fin_aid_disclosure_category', VARCHAR(length=3, collation=u'SQL_Latin1_General_CP1_CI_AS'), table=<funds>), Column(u'report_as_additional_unsub', BIT(), table=<funds>, server_default=DefaultClause(<sqlalchemy.sql.expression.TextClause object at 0x000000000545B6D8>, for_update=False)), schema=None)
If you merely want to reflect one table at a time, you can use the following code instead.
ONE TABLE AT A TIME (much faster)
>>> from sqlalchemy import Table, create_engine, MetaData
>>> engine = create_engine('mssql+pyodbc://<username>:<password>#<host>/<database>')
>>> meta = MetaData()
>>> funds_table = Table('funds', meta, autoload=True, autoload_with=engine) # indicate table name (here 'funds') with a string passed to Table as the first argument
>>> funds_table # now stores database schema object
Table(u'funds', MetaData(bind=None), Column(u'fund_token', INTEGER(), table=<funds>, primary_key=True, nullable=False), Column(u'award_year_token', INTEGER(), ForeignKey(u'award_year_defn.award_year_token'), table=<funds>, nullable=False), ... Column(u'fin_aid_disclosure_category', VARCHAR(length=3, collation=u'SQL_Latin1_General_CP1_CI_AS'), table=<funds>), Column(u'report_as_additional_unsub', BIT(), table=<funds>, server_default=DefaultClause(<sqlalchemy.sql.expression.TextClause object at 0x000000000545B6D8>, for_update=False)), schema=None)
As you can probably imagine, you can then save the relevant tables' data for accessing the tables more quickly again in the future.
I'm looking at the example at bulk_insert.
# Create an ad-hoc table to use for the insert statement.
accounts_table = table('account',
column('id', Integer),
column('name', String),
column('create_date', Date)
)
I would like to bulk insert to a specific schema called kpi but I can't figure out how to do it. Can someone help me out?
The answer I've found so far is to use the sqlalchemy function instead like below:
accounts_table = sa.Table('accounts_table', sa.MetaData(),
sa.Column('id', sa.Integer),
sa.Column('name', sa.String(100)),
sa.Column('create_date', sa.Date),
schema='phone'
)