Is there SQLAlchemy automigration tool like South for Django?
I looked to sqlalchemy-migrate but it doesn't seem to generate sql update scripts automatically or upgrade downgrade DB
Looks like with sqlalchemy-migrate you need to
a) manually copy your old model to a new file
b) crate new model in application and copy it to a new file
c) write manually create/drop/alter tables in python sqlalchemy extended dialect
d) generate sql alter script
e) run command to execute alter sql script
As for me it doesn't solve the problem and only adds overhead, as I can simply do d) manually and it will be much faster then do a), b), c) manually just to d) that you can do in one step.
Is there any auto migration libraries for SQLAlchemy like South for Django, or many RoR-like migration tools?
What I need is to change SQLAlchemy model in python app, run tool and it will compare current DB schema to new DB schema that new model should use, and create Alter scripts that I can adjust manually and execute.
Is there any solution like this in Python?
You can perform automatic migrations with Alembic. I use it in two large-scale projects I am currently working on. The automatic migration generator can recognize:
Table additions and removals
Column additions and removals
Change of nullable status on columns
Basic changes in indexes, explicitly-named unique constraints, and foreign keys
(see also: https://alembic.sqlalchemy.org/en/latest/autogenerate.html)
Install alembic
pip install alembic
or (depending on the version of Python you are using):
pip3 install alembic
Configure alembic
Execute the following command in your project:
alembic init alembic
This will set up alembic for your project, in a folder called alembic.
You will then need to edit the generated alembic.ini configuration file.
In the file env.py, tell Alembic where to find SQLAlchemy's metadata object in your project.
(see also: https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file)
Generate the migration
Simply execute the following command line:
alembic revision --autogenerate -m "Message for this migration"
Or even (not recommended):
alembic revision --autogenerate
Upgrade the database
After this, I upgrade the database with this simple command from the folder containing the alembic.ini configuration file:
alembic upgrade head
(see also: http://rodic.fr/blog/automatic-migrations-sqlalchemy-alembic/)
There is Alembic which looks very promising, but the problem is (for now) that the support for SQlite databases is very limited.
No there is none at this moment. Alembic require you to write code regarding adding/deleting/altering table structure or creating/dropping table. So nothing like django south exists for sqlalchemy.
Have you looked into using sqlalchemy-migrate?
http://shane.caraveo.com/2010/09/13/database-migrations-for-sqlalchemy-part-duex/
Related
I'm trying to abstract some functionality from my models into a Mixin, and I don't actually want to change any tables — I just want to check to make sure I've done this correctly.
How can I see what changes alembic detects without creating a revisions file? Or is my only option to run alembic revision --autogenerate -m "...", then delete the file?
Alembic v1.9.0 (https://alembic.sqlalchemy.org/en/latest/changelog.html#change-1.9.0) introduced a new check command:
Added new Alembic command alembic check. This performs the widely requested feature of running an “autogenerate” comparison between the current database and the MetaData that’s currently set up for autogenerate, returning an error code if the two do not match, based on current autogenerate settings. Pull request courtesy Nathan Louie.
I have a question about alembic multidb template or using alembic for multiple database in general.
Our project has your usual three levels of database (development, testing, production). Depending on which level of database and whether it runs via Airflow or not, the URL for each database will be different. I'd like to use a .env file to store the URLs and credentials and run the migration using a command like:
alembic -x db=development airflow=False upgrade head
What should I do to achieve this? Thank you in advance for any assitance!
My flask application now has 20+ migrations built with flask-migrate and they all have hashed file names like: 389d9662fec7_.py
I want to double check the settings on the latest migration that I ran, but don't want to open every file to look for the correct one. I could create a new dummy migration and look at what it references as the down_revision but that seems clunky.
I'm using flask-script, flask-migrate, and flask-sqlalchemy
My question is: How can I quickly find the latest migration that I created?
./manage.py db history -r current: will show the migrations in the order they will be applied. -r current: shows only the migrations since the currently applied one.
./manage.py db heads will show the most recent migration for each branch (typically there's only one branch). ./manage.py db upgrade would apply all migrations to get to the head.
Use the -v flag to get verbose output, including the full path to the migration.
You can also check in your database and the current version should be displayed in a table called alembic_version.
I could create tables using the command alembic revision -m 'table_name' and then defining the versions and migrate using alembic upgrade head.
Also, I could create tables in a database by defining a class in models.py (SQLAlchemy).
What is the difference between the two? I'm very confused. Have I messed up the concept?
Also, when I migrate the database using Alembic, why doesn't it form a new class in my models.py? I know the tables have been created because I checked them using a SQLite browser.
I have done all the configurations already. The target for Alembic's database and SQLALCHEMY_DATABASE-URI in config.py are the same .db file.
Yes, you are thinking about it in the wrong way.
Let's say you don't use Alembic or any other migration framework. In that case you create a new database for your application with the following steps:
Write your model classes
Create and configure a brand new database
Run db.create_all(), which looks at your models and creates the corresponding tables in your database.
So now consider the case of an upgrade. For example, let's say you release version 1.0 of your application and now start working on version 2.0, which requires some changes to your database. How can you achieve that? The limitation here is that db.create_all() does not modify tables, it can only create them from scratch. So it goes like this:
Make the necessary changes to your model classes
Now you have two options to transfer those changes to the database:
5.1 Destroy the database so that you can run db.create_all() again to get the updated tables, maybe backing up and restoring the data so that you don't lose it. Unfortunately SQLAlchemy does not help with the data, you'll have to use database tools for that.
5.2 Apply the changes manually, directly to the database. This is error prone, and it would be tedious if the change set is large.
Now consider that you have development and production databases, that means the work needs to be done twice. Also think about how tedious would it be when you have several releases of your application, each with a different database schema and you need to investigate a bug in one of the older releases, for which you need to recreate the database as it was in that release.
See what the problem is when you don't have a migration network?
Using Alembic, you have a little bit of extra work when you start, but it pays off because it simplifies your workflow for your upgrades. The creation phase goes like this:
Write your model classes
Create and configure a brand new database
Generate an initial Alembic migration, either manually or automatically. If you go with automatic migrations, Alembic looks at your models and generates the code that applies those to the database.
Run the upgrade command, which runs the migration script, effectively creating the tables in your database.
Then when you reach the point of doing an upgrade, you do the following:
Make the necessary changes to your model classes
Generate another Alembic migration. If you let Alembic generate this for you, then it compares your models classes against the current schema in your database, and generates the code necessary to make the database match the models.
Run the upgrade command. This applies the changes to the database, without the need to destroy any tables or back up data. You can run this upgrade on all your databases (production, development, etc.).
Important things to consider when using Alembic:
The migration scripts become part of your source code, so they need to be committed to source control along with your own files.
If you use the automatic migration generation, you always have to review the generated migrations. Alembic is not always able to determine the exact changes, so it is possible that the generated script needs some manual fine tuning.
Migration scripts have upgrade and downgrade functions. That means that they not only simplify upgrades, but also downgrades. If you need to sync the database to an old release, the downgrade command does it for you without any additional work on your part!
I can add that for Django there are two commands
makemigrations (which creates migrations files)
migrate (which translates migrations into sql and executes them on database)
I found its great for somebody's understanding to switch between batteries included framework(Django) and other frameworks like Flask/ Falcon.
So using alembic migrations is very convenient, and makes easy to track database changes.
I am using Django to build a website with MySQL. Now as I am learning so I need to change the Model very often so I want that all tables get cleared and new table get created.
But syncdb doesn't touch existing tables. Is there any better way to handle this problem?
If you don't care about data:
Best way would be to drop the database and run syncdb again. Or you can run:
For Django >= 1.5
python manage.py flush
For Django < 1.5
python manage.py reset appname
(you can add --no-input to the end of the command for it to skip the interactive prompt.)
If you do care about data:
From the docs:
syncdb will only create tables for
models which have not yet been
installed. It will never issue ALTER
TABLE statements to match changes made
to a model class after installation.
Changes to model classes and database
schemas often involve some form of
ambiguity and, in those cases, Django
would have to guess at the correct
changes to make. There is a risk that
critical data would be lost in the
process.
If you have made changes to a model
and wish to alter the database tables
to match, use the sql command to
display the new SQL structure and
compare that to your existing table
schema to work out the changes.
https://docs.djangoproject.com/en/dev/ref/django-admin/
Reference: FAQ - https://docs.djangoproject.com/en/dev/faq/models/#if-i-make-changes-to-a-model-how-do-i-update-the-database
People also recommend South ( http://south.aeracode.org/docs/about.html#key-features ), but I haven't tried it.
Using Django Extensions, running:
./manage.py reset_db
Will clear the database tables, then running:
./manage.py syncdb
Will recreate them (south may ask you to migrate things).
I think Django docs explicitly mention that if the intent is to start from an empty DB again (which seems to be OP's intent), then just drop and re-create the database and re-run migrate (instead of using flush):
If you would rather start from an empty database and re-run all
migrations, you should drop and recreate the database and then run
migrate instead.
So for OP's case, we just need to:
Drop the database from MySQL
Recreate the database
Run python manage.py migrate
Quickest (drops and creates all tables including data):
./manage.py reset appname | ./manage.py dbshell
Caution:
Might not work on Windows correctly.
Might keep some old tables in the db
You can use the Django-Truncate library to delete all data of a table without destroying the table structure.
Example:
First, install django-turncate using your terminal/command line:
pip install django-truncate
Add "django_truncate" to your INSTALLED_APPS in the settings.py file:
INSTALLED_APPS = [
...
'django_truncate',
]
Use this command in your terminal to delete all data of the table from the app.
python manage.py truncate --apps app_name --models table_name