I'm working on a codebase which has a lot of migrations. We also have SQL views which are defined in the migrations and later updated in the migrations (whenever changes to the SQL views were required). We have a challenge where when running unit tests the migrations are not applied in the same appear as they currently appear in the django_migrations table. This causes unit tests to fail.
My thought is, I could probably write a script to the dependencies of all the migrations files to follow the same order of django_migrations thus ensuring that migrations are run in the correct order. Then I can look into squashing migrations.
But this is a lengthy process and I'm not sure if this will actually work.
Has anyone come across this issue before? If so, how did you overcome it?
Related
The Problem (Short Description)
I have a migration file deleted, but it was previously applied in the database. I want to revert the changes it has made.
The Problem (Long Description)
I had forked a python package and had made some changes on one of their models. I used the fork instead of the official package in my project and had my new migration applied in my database.
A couple of days ago, I replaced the fork with the official version of the package and did what I wanted to do in another way without changing the model anymore. This means, that for the same app, now one migration is missing (the migration that I created).
However, the changes that this migration made on the database are still present and the migration name is still listed on the django_migrations table. This has occurred weird problems, since the code and the database are not synchronized.
What I have tried
I tried running python manage.py migrate <appname> <previous_migration> where the appname is the app I have problem with migrations and the previous_migration is the last migration the app had before I added mine.
I tried modifying the database directly by manually reverting the changes that were made by that now deprecated migration.
I tried deleting the table completely and letting the migration re-create it, but apparently I was wrong since Django thinks that the migrations are applied and doesn't re-apply them
What I want to achieve
I want to get rid of that migration, revert the changes it has made, without causing any problem. I do not care about the data that this table has (even in production), so even if the solution includes deleting the affected table, I don't mind.
However, I'm looking for a clean solution that won't complicate things too much. More likely I would love if there was something that I could do in the code and not in the database directly.
I have been dealing with this issue for a long time now. Our company has two django servers one for testing purpose and the other for deployment purpose, each with their own databases and migrations.
My initial solution was to simply maintain two migration folders:
-->migrations
-->migrations(P)
As seen above migrations(P) denotes the production level migrations, this config is used mainly during the testing phase where all the migrations are related to test database and some times it is swapped to the following folder structure when we are in production mode migrations:
-->migrations
-->migrations(T)
In the above case migrations(T) are related to test database and migrations are related to production level server.
This worked well but sometimes when there are multiple commits from other developers and also i work on it myself, due to the folder swapping the migration files get merged and messed up causing in a crash.
Sorry if my question is a little confusing.
Any alternative suggestion or approaches for maintaining production and test level database migrations would be helpful
There should only be one migrations directory for an app. Your release management process needs to be more robust to prevent overwriting migrations. This could be caught in code reviews before code gets merged into master or some other stable-type branch. A reviewer should notice a migration being deleted and raise the issue.
Now regarding how do you manage a test database with multiple developers staging different and potentially conflicting migrations. You need communication as a team and attention to detail. For 95% of cases, the migrations need to be reverse-able. If a migration isn't, that developer needs to communicate that to the other developers that it's going out on staging and is going to control it until A) it's deployed or B) the test database is forked from production or restored from a backup before the migration.
The other part of communication is determining who is taking the test environment and determining if there's applied migrations that aren't in your branch. Before your branch gets deployed, you need to reverse those other migrations. Then you're safe to deploy your code and run your migrations. This is why reversible migrations are extremely important.
A difficult part is when migrations do get merged to master, any development that has migrations that depend on the newly merged migrations applications need to be merged via makemigrations --merge or they can renamed to the count and have their dependencies corrected.
If these things are hard for your organization to do via unwritten rules, then it sounds like you need to implement some more rigid process around your releases. Getting your database out of sync with migrations can be a difficult problem to solve. It's made especially worse when it's on a production database and causes customer facing problems. Your organization needs to manage its risk effectively via release management.
I had my migrations file in version control, and it went out of sync between production and development (I know, that was the mistake, but it's happened now).
As a result, I couldn't run migrations in production. I deleted the migrations folder, and then re-ran makemigrations and migrate. makemigrations logs out creating all the fields. However, migrate simply says "no migrations to apply", and the extra fields do not appear in the database.
All I've changed is adding nullable fields to a model, so it should be a straightforward migration.
I can drop the whole db and start over, but I'd prefer not to because it takes a long time to re-populate.
Is there a way that I can force Django to find the differences between the DB and the models, and build the correct migrations to add the fields?
I attempted adding nonsense models to try and trigger a refresh. But that hasn't changed anything.
I see the same error many people have seen.
django.db.migrations.exceptions.InconsistentMigrationHistory: Migration message.0001_initial is applied before its dependency users.0001_initial on database 'default'.
Seems like the cause is because I'm using CustomUser model (using django-allauth).
I already delete all the migration folders and sqlite3 and tried some ways other people answered. But I cannot avoid this error.
Seems like I was successful in avoiding this error (I have no idea how I did.)
And because the migration files still remains on my gitHub, I saw them and all the files look the same as the ones I created before. I'm wondering if I can ignore this error and waring. If not, what is the way to avoid this error?
what I did
using --fake when doing migrations
hiding django.admin when doing migrations
applying zero
all things failed.
I actually solved the problem.
What I did is applying migration into the app CustomUser model is located, and then applying migration into other apps. And then I did python manage.py migrate.
Other orders doesn't work for me.
So, I committed and pushed all my code, and then deployed my web application successfully. Then, I added a new model to my 'home' app, which (for a reason I now understand, but doesn't matter here), created an IntegrityError (django.db.utils.IntegrityError: insert or update on table "foo" violates foreign key constraint "bar"). I ran python manage.py makemigrations, python manage.py migrate, which causes the the IntegrityError.
However, even if I remove all of my new model code(so that git status comes up with nothing), the IntegrityError still happens. If I connect to my db via a different python instance and download select * from django_migrations;, the latest db migration: 0020 there is eight migrations away from my latest local home/migrations migration file: 0028.
--> My question is: is it safe for me to delete my local 0021-0028 migration files? Will this fix my problem?
If you haven't applied your migrations to db, it is safe to delete them and recreate them.
Possible reasons of why you run into this error are:
You deleted your model code but, when you run migrate it reads your migration files (which has information about your deleted model) and tries to apply migration operations. If you didn't run makemigrations command after you've deleted your model, migration system won't be able to detect your changes and will think that your model is still there.
Even if you've run makemigrations after you've deleted your model there'll be dependency issues in your migrations files, because the new migration files will depend on old ones (with which you had problems)
That's why we can say that it is safe to delete them, if they haven't applied, but at the same time you should be careful with your migration dependencies.
This documentation information maybe useful.
OK, so I crossed my fingers, backed my local 0021-0028 migration files, and then deleted them. It worked. I think they key is that the migration files were not yet in the database yet, but not 100% sure. +1 if anyone can answer further for clarification.