Due to some quirks in our database we need to reassign table owners post creation. Currently we leverage Alembic - has anyone got a simple way to create some sort of post hook that runs certain SQL commands after a migration?
It sounds like what you're asking for can be resolved by editing the env.py file of alembic.. From the docs (emphasis mine):
env.py - The env.py script is part of the generated environment so that the way
migrations run is entirely customizable. The exact specifics of how to
connect are here, as well as the specifics of how the migration
environment are invoked. The script can be modified so that multiple
engines can be operated upon, custom arguments can be passed into the
migration environment, application-specific libraries and models can
be loaded in and made available.
I bet you could get the behavior you want by adding a call in run_migrations_online().
(This is assuming you want to run the post_hook after EACH migration. If it was migration specific, you can update the upgrade() function in the generated migration file)
Related
I'm trying to abstract some functionality from my models into a Mixin, and I don't actually want to change any tables — I just want to check to make sure I've done this correctly.
How can I see what changes alembic detects without creating a revisions file? Or is my only option to run alembic revision --autogenerate -m "...", then delete the file?
Alembic v1.9.0 (https://alembic.sqlalchemy.org/en/latest/changelog.html#change-1.9.0) introduced a new check command:
Added new Alembic command alembic check. This performs the widely requested feature of running an “autogenerate” comparison between the current database and the MetaData that’s currently set up for autogenerate, returning an error code if the two do not match, based on current autogenerate settings. Pull request courtesy Nathan Louie.
The Problem (Short Description)
I have a migration file deleted, but it was previously applied in the database. I want to revert the changes it has made.
The Problem (Long Description)
I had forked a python package and had made some changes on one of their models. I used the fork instead of the official package in my project and had my new migration applied in my database.
A couple of days ago, I replaced the fork with the official version of the package and did what I wanted to do in another way without changing the model anymore. This means, that for the same app, now one migration is missing (the migration that I created).
However, the changes that this migration made on the database are still present and the migration name is still listed on the django_migrations table. This has occurred weird problems, since the code and the database are not synchronized.
What I have tried
I tried running python manage.py migrate <appname> <previous_migration> where the appname is the app I have problem with migrations and the previous_migration is the last migration the app had before I added mine.
I tried modifying the database directly by manually reverting the changes that were made by that now deprecated migration.
I tried deleting the table completely and letting the migration re-create it, but apparently I was wrong since Django thinks that the migrations are applied and doesn't re-apply them
What I want to achieve
I want to get rid of that migration, revert the changes it has made, without causing any problem. I do not care about the data that this table has (even in production), so even if the solution includes deleting the affected table, I don't mind.
However, I'm looking for a clean solution that won't complicate things too much. More likely I would love if there was something that I could do in the code and not in the database directly.
I am trying to change some field names of Django models, but for some reasons Django does not want to see my migrations. I made sure the following things are properly set up:
myApp.apps.MyAppConfig is added to INSTALLED_APPS in settings.py
A migration folder with an empty __init__.py is present in my app folder
In myApp folder, I also have an empty __init__.py
The whole procedure works just fine locally with a SQLite database and I can change field names and run python manage.py makemigrations and python manage.py migrate to migrate the database. However in production where I use Docker Compose for orchestration and Postgres as database only the first migration works fine. Afterwards when I change any model field name and try to run docker-compose exec projectname django-admin makemigrations or docker-compose exec projectname python manage.py makemigrations or add the app name behind these commands nothing helps.
Then Postgres can't handle the request, because it doesn't know the new field name:
2022-03-11 14:40:22.136 UTC [81] ERROR: column model_name.new_field_name does not exist at character 357
What am I missing here? How can I get Django to migrate?
I had the same issues when using version control to submit changes to another environment.
What probably happens is that django believes it already made the migration, because of the information that is being passed along when you push those changes. What you need to do is to correct that behavior manually deleting those migrations or, if you don't need to keep information yet, force the migration from the beginning.
You may want to read the docs for future use, and I used the answers in this questions to solve the same issue.
At the end I was able to solve this issue by adding a Docker volume for the migrations folder. I believe a lot of times people put their entire code in a persistent volume, which would've also prevented this issue. However at least everything that needs to be persistent (obviously) needs a persistent volume.
Inside my Django project, I want to fire up a python shell wherein I can make simple changes to my database (saving new records, updating old ones, etc.) which are then automatically reverted back when I exit the shell. Essentially the same functionality as running:
rails console --sandbox
in a Ruby on Rails project. I'm not looking for all the features of a traditional sandbox such as blocking web-service calls, just the basic database rollback. Is this--or something similar--possible in Django?
I want to use this for testing purposes. I know I can use Django's testing infrastructure to do this, but there are some basic instances where this would be very useful (e.g. quickly confirming that a specific model functions the way I expect it to).
This is mostly done in python manage.py shell (or with django_extensions in INSTALLED_APPS better ... shell_plus) which is python command line from which you can manipulate the data and test some features you don't wan't to program before you test. Rollback in django shell is not possible as I know, but you can have separate settings.py with different DB settings so you will have another database where you can experiment in shell without corrupting your data.
Here some sources:
Django Best Practice: Settings file for multiple environments
Django shell (from DJango docs)
I could create tables using the command alembic revision -m 'table_name' and then defining the versions and migrate using alembic upgrade head.
Also, I could create tables in a database by defining a class in models.py (SQLAlchemy).
What is the difference between the two? I'm very confused. Have I messed up the concept?
Also, when I migrate the database using Alembic, why doesn't it form a new class in my models.py? I know the tables have been created because I checked them using a SQLite browser.
I have done all the configurations already. The target for Alembic's database and SQLALCHEMY_DATABASE-URI in config.py are the same .db file.
Yes, you are thinking about it in the wrong way.
Let's say you don't use Alembic or any other migration framework. In that case you create a new database for your application with the following steps:
Write your model classes
Create and configure a brand new database
Run db.create_all(), which looks at your models and creates the corresponding tables in your database.
So now consider the case of an upgrade. For example, let's say you release version 1.0 of your application and now start working on version 2.0, which requires some changes to your database. How can you achieve that? The limitation here is that db.create_all() does not modify tables, it can only create them from scratch. So it goes like this:
Make the necessary changes to your model classes
Now you have two options to transfer those changes to the database:
5.1 Destroy the database so that you can run db.create_all() again to get the updated tables, maybe backing up and restoring the data so that you don't lose it. Unfortunately SQLAlchemy does not help with the data, you'll have to use database tools for that.
5.2 Apply the changes manually, directly to the database. This is error prone, and it would be tedious if the change set is large.
Now consider that you have development and production databases, that means the work needs to be done twice. Also think about how tedious would it be when you have several releases of your application, each with a different database schema and you need to investigate a bug in one of the older releases, for which you need to recreate the database as it was in that release.
See what the problem is when you don't have a migration network?
Using Alembic, you have a little bit of extra work when you start, but it pays off because it simplifies your workflow for your upgrades. The creation phase goes like this:
Write your model classes
Create and configure a brand new database
Generate an initial Alembic migration, either manually or automatically. If you go with automatic migrations, Alembic looks at your models and generates the code that applies those to the database.
Run the upgrade command, which runs the migration script, effectively creating the tables in your database.
Then when you reach the point of doing an upgrade, you do the following:
Make the necessary changes to your model classes
Generate another Alembic migration. If you let Alembic generate this for you, then it compares your models classes against the current schema in your database, and generates the code necessary to make the database match the models.
Run the upgrade command. This applies the changes to the database, without the need to destroy any tables or back up data. You can run this upgrade on all your databases (production, development, etc.).
Important things to consider when using Alembic:
The migration scripts become part of your source code, so they need to be committed to source control along with your own files.
If you use the automatic migration generation, you always have to review the generated migrations. Alembic is not always able to determine the exact changes, so it is possible that the generated script needs some manual fine tuning.
Migration scripts have upgrade and downgrade functions. That means that they not only simplify upgrades, but also downgrades. If you need to sync the database to an old release, the downgrade command does it for you without any additional work on your part!
I can add that for Django there are two commands
makemigrations (which creates migrations files)
migrate (which translates migrations into sql and executes them on database)
I found its great for somebody's understanding to switch between batteries included framework(Django) and other frameworks like Flask/ Falcon.
So using alembic migrations is very convenient, and makes easy to track database changes.