We Keep Coding

How to create a new table using model

So I have this django installation in which there are a bunch of migration scripts. They look like so:
00001_initial.py
00002_blah_blah.py
00003_bleh_bleh.py
Now I know these are "database building" scripts which will take stuff defined in models.py and run them against the db to "create" tables and stuff.
I want to create a new table(so I created its definition in models.py). For this, I have copied another model class and edited its name and fields and it is all fine. Lets call this new model class 'boom'.
My question is now how do I "create" this boom table using the migration script and the boom model?
I am worried that I might accidentally disrupt anything that is already in DB. How do I run the migration to create only boom table? How do I create a migration script specifically for it?
I know that it has something to do with manage.py and running migrate or runmigration (or is it sqlmigrate?...im confused). While creating the boom table, I dont want the database to go boom if you know what I mean :)

First, create a backup of your database. Copy it to your development machine. Try things out on that. That way it doesn't matter if it does go "boom" for some reason.
The first thing to do is
python manage.py showmigrations
This shows all the existing migrations, and it should show that they have been applied with an [X].
Then,
python manage.py makemigrations
Makes a new migration file for your new model (name 00004_...).
Then do
python manage.py migrate
to apply it. To undo it, go back to the state of migrations 00003, with
python manage.py migrate <yourappname> 00003

There are two steps to migrations in Django.
./manage.py makemigrations
will create the migration files that you see - these describe the changes that should be made to the database.
You also need to run
./manage.py migrate
this will apply the migrations and actually run the alter table commands in SQL to change the actual database structure.
Generally adding fields or tables won't affect anything else in the database. Be more careful when altering or deleting existing fields as that can affect your data.
The reason for two steps is so that you can make changes on a dev machine and once happy commit the migration files and release to your production environment. Then you run the migrate command on your production machine to bring the production database to the same state as your dev machine (no need for makemigrations on production assuming that your databases started the same).

My question is now how do I "create" this boom table using the
migration script and the boom model?
./manage.py makemigrations
I am worried that I might accidentally disrupt anything that is
already in DB.
The whole point of migrations, is that it doesn't
I know that it has something to do with manage.py and running migrate
or runmigration
For more information please refer to : https://docs.djangoproject.com/en/1.10/topics/migrations/
And rest assured that your database will not go boom! :-)

I solved it simply, changing the name of the new model to the original name, and then I checked if there is the table in the database, if not, I just create a new table with the old name with just a field like id.
And then clear migrations and create new migrations, migrate and verify table was fixed in DB and has all missing fields.
If it still doesn't work, then change the model name back to a new one.
but when django asks you if you are renaming the model you should say NO to get the old one removed properly and create a new one.
This type of error usually occurs when you delete some table in dB manually, and then the migration history changes in the tables are lost.
But it is not necessary to erase the entire database and start from scratch.

Renaming model breakes migration history

I have two models in two different apps:
# app1 models.py
class App1Model(models.Model):
pass
# app2 models.py
from app1.models import App1Model
class App2Model(App1Model):
pass
And I want to rename App1Model and then recreate App2Model to have explicit OneToOneField instead of magic app1model_ptr. So I create migration which deletes App2Model completely (I don't care about data because whatever reason), it runs successfully. Then I create migration in first app which renames App1Model and it runs perfect too, I check this table with new name and all it's relationships (and to it too), it's fine.
And then weird thing happens: when I run makemigrations or migrate on app2 I get an error
django.db.migrations.exceptions.InvalidBasesError: Cannot resolve bases for [<ModelState: 'app2.App2Model'>]
It fails while creating current project state on very first migration of app2 (0001_initial.py in app2 migrations) where this model was created first time by inheriting from App1Model with its old name.
Is there some way of fixing this? In current state App2Model already deleted, App1Model renamed and I can't do anything with migrations on app2 because of this issue.
P.S. I use Django 1.10.2

Just found solution:
Need to add last migration of app2 where I deleted App2Model as dependency to migration of app1 where I renamed App1Model so project state will be built in correct order. Actually error message itself has something related to it but I failed to catch the point:
This can happen if you are inheriting models from an app with
migrations (e.g. contrib.auth) in an app with no migrations; see
https://docs.djangoproject.com/en/1.10/topics/migrations/#dependencies
for more
I put it here for future me and for those who will suffer from similar thing.

Loading initial data with Django 1.7+ and data migrations

I recently switched from Django 1.6 to 1.7, and I began using migrations (I never used South).
Before 1.7, I used to load initial data with a fixture/initial_data.json file, which was loaded with the python manage.py syncdb command (when creating the database).
Now, I started using migrations, and this behavior is deprecated :
If an application uses migrations, there is no automatic loading of fixtures.
Since migrations will be required for applications in Django 2.0, this behavior is considered deprecated. If you want to load initial data for an app, consider doing it in a data migration.
(https://docs.djangoproject.com/en/1.7/howto/initial-data/#automatically-loading-initial-data-fixtures)
The official documentation does not have a clear example on how to do it, so my question is :
What is the best way to import such initial data using data migrations :
Write Python code with multiple calls to mymodel.create(...),
Use or write a Django function (like calling loaddata) to load data from a JSON fixture file.
I prefer the second option.
I don't want to use South, as Django seems to be able to do it natively now.

Update: See #GwynBleidD's comment below for the problems this solution can cause, and see #Rockallite's answer below for an approach that's more durable to future model changes.
Assuming you have a fixture file in <yourapp>/fixtures/initial_data.json
Create your empty migration:
In Django 1.7:
python manage.py makemigrations --empty <yourapp>
In Django 1.8+, you can provide a name:
python manage.py makemigrations --empty <yourapp> --name load_intial_data
Edit your migration file <yourapp>/migrations/0002_auto_xxx.py
2.1. Custom implementation, inspired by Django' loaddata (initial answer):
import os
from sys import path
from django.core import serializers
fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
fixture_filename = 'initial_data.json'
def load_fixture(apps, schema_editor):
fixture_file = os.path.join(fixture_dir, fixture_filename)
fixture = open(fixture_file, 'rb')
objects = serializers.deserialize('json', fixture, ignorenonexistent=True)
for obj in objects:
obj.save()
fixture.close()
def unload_fixture(apps, schema_editor):
"Brutally deleting all entries for this model..."
MyModel = apps.get_model("yourapp", "ModelName")
MyModel.objects.all().delete()
class Migration(migrations.Migration):
dependencies = [
('yourapp', '0001_initial'),
]
operations = [
migrations.RunPython(load_fixture, reverse_code=unload_fixture),
]
2.2. A simpler solution for load_fixture (per #juliocesar's suggestion):
from django.core.management import call_command
fixture_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), '../fixtures'))
fixture_filename = 'initial_data.json'
def load_fixture(apps, schema_editor):
fixture_file = os.path.join(fixture_dir, fixture_filename)
call_command('loaddata', fixture_file)
Useful if you want to use a custom directory.
2.3. Simplest: calling loaddata with app_label will load fixtures from the <yourapp>'s fixtures dir automatically :
from django.core.management import call_command
fixture = 'initial_data'
def load_fixture(apps, schema_editor):
call_command('loaddata', fixture, app_label='yourapp')
If you don't specify app_label, loaddata will try to load fixture filename from all apps fixtures directories (which you probably don't want).
Run it
python manage.py migrate <yourapp>

Short version
You should NOT use loaddata management command directly in a data migration.
# Bad example for a data migration
from django.db import migrations
from django.core.management import call_command
def load_fixture(apps, schema_editor):
# No, it's wrong. DON'T DO THIS!
call_command('loaddata', 'your_data.json', app_label='yourapp')
class Migration(migrations.Migration):
dependencies = [
# Dependencies to other migrations
]
operations = [
migrations.RunPython(load_fixture),
]
Long version
loaddata utilizes django.core.serializers.python.Deserializer which uses the most up-to-date models to deserialize historical data in a migration. That's incorrect behavior.
For example, supposed that there is a data migration which utilizes loaddata management command to load data from a fixture, and it's already applied on your development environment.
Later, you decide to add a new required field to the corresponding model, so you do it and make a new migration against your updated model (and possibly provide a one-off value to the new field when ./manage.py makemigrations prompts you).
You run the next migration, and all is well.
Finally, you're done developing your Django application, and you deploy it on the production server. Now it's time for you to run the whole migrations from scratch on the production environment.
However, the data migration fails. That's because the deserialized model from loaddata command, which represents the current code, can't be saved with empty data for the new required field you added. The original fixture lacks necessary data for it!
But even if you update the fixture with required data for the new field, the data migration still fails. When the data migration is running, the next migration which adds the corresponding column to the database, is not applied yet. You can't save data to a column which does not exist!
Conclusion: in a data migration, the loaddata command introduces potential inconsistency between the model and the database. You should definitely NOT use it directly in a data migration.
The Solution
loaddata command relies on django.core.serializers.python._get_model function to get the corresponding model from a fixture, which will return the most up-to-date version of a model. We need to monkey-patch it so it gets the historical model.
(The following code works for Django 1.8.x)
# Good example for a data migration
from django.db import migrations
from django.core.serializers import base, python
from django.core.management import call_command
def load_fixture(apps, schema_editor):
# Save the old _get_model() function
old_get_model = python._get_model
# Define new _get_model() function here, which utilizes the apps argument to
# get the historical version of a model. This piece of code is directly stolen
# from django.core.serializers.python._get_model, unchanged. However, here it
# has a different context, specifically, the apps variable.
def _get_model(model_identifier):
try:
return apps.get_model(model_identifier)
except (LookupError, TypeError):
raise base.DeserializationError("Invalid model identifier: '%s'" % model_identifier)
# Replace the _get_model() function on the module, so loaddata can utilize it.
python._get_model = _get_model
try:
# Call loaddata command
call_command('loaddata', 'your_data.json', app_label='yourapp')
finally:
# Restore old _get_model() function
python._get_model = old_get_model
class Migration(migrations.Migration):
dependencies = [
# Dependencies to other migrations
]
operations = [
migrations.RunPython(load_fixture),
]

Inspired by some of the comments (namely n__o's) and the fact that I have a lot of initial_data.* files spread out over multiple apps I decided to create a Django app that would facilitate the creation of these data migrations.
Using django-migration-fixture you can simply run the following management command and it will search through all your INSTALLED_APPS for initial_data.* files and turn them into data migrations.
./manage.py create_initial_data_fixtures
Migrations for 'eggs':
0002_auto_20150107_0817.py:
Migrations for 'sausage':
Ignoring 'initial_data.yaml' - migration already exists.
Migrations for 'foo':
Ignoring 'initial_data.yaml' - not migrated.
See django-migration-fixture for install/usage instructions.

In order to give your database some initial data, write a data migration.
In the data migration, use the RunPython function to load your data.
Don't write any loaddata command as this way is deprecated.
Your data migrations will be run only once. The migrations are an ordered sequence of migrations. When the 003_xxxx.py migrations is run, django migrations writes in the database that this app is migrated until this one (003), and will run the following migrations only.

The solutions presented above didn't work for me unfortunately. I found that every time I change my models I have to update my fixtures. Ideally I would instead write data migrations to modify created data and fixture-loaded data similarly.
To facilitate this I wrote a quick function which will look in the fixtures directory of the current app and load a fixture. Put this function into a migration in the point of the model history that matches the fields in the migration.

On Django 2.1, I wanted to load some models (Like country names for example) with initial data.
But I wanted this to happen automatically right after the execution of initial migrations.
So I thought that it would be great to have a sql/ folder inside each application that required initial data to be loaded.
Then within that sql/ folder I would have .sql files with the required DMLs to load the initial data into the corresponding models, for example:
INSERT INTO appName_modelName(fieldName)
VALUES
("country 1"),
("country 2"),
("country 3"),
("country 4");
To be more descriptive, this is how an app containing a sql/ folder would look:
Also I found some cases where I needed the sql scripts to be executed in a specific order. So I decided to prefix the file names with a consecutive number as seen in the image above.
Then I needed a way to load any SQLs available inside any application folder automatically by doing python manage.py migrate.
So I created another application named initial_data_migrations and then I added this app to the list of INSTALLED_APPS in settings.py file. Then I created a migrations folder inside and added a file called run_sql_scripts.py (Which actually is a custom migration). As seen in the image below:
I created run_sql_scripts.py so that it takes care of running all sql scripts available within each application. This one is then fired when someone runs python manage.py migrate. This custom migration also adds the involved applications as dependencies, that way it attempts to run the sql statements only after the required applications have executed their 0001_initial.py migrations (We don't want to attempt running a SQL statement against a non-existent table).
Here is the source of that script:
import os
import itertools
from django.db import migrations
from YourDjangoProjectName.settings import BASE_DIR, INSTALLED_APPS
SQL_FOLDER = "/sql/"
APP_SQL_FOLDERS = [
(os.path.join(BASE_DIR, app + SQL_FOLDER), app) for app in INSTALLED_APPS
if os.path.isdir(os.path.join(BASE_DIR, app + SQL_FOLDER))
]
SQL_FILES = [
sorted([path + file for file in os.listdir(path) if file.lower().endswith('.sql')])
for path, app in APP_SQL_FOLDERS
]
def load_file(path):
with open(path, 'r') as f:
return f.read()
class Migration(migrations.Migration):
dependencies = [
(app, '__first__') for path, app in APP_SQL_FOLDERS
]
operations = [
migrations.RunSQL(load_file(f)) for f in list(itertools.chain.from_iterable(SQL_FILES))
]
I hope someone finds this helpful, it worked just fine for me!. If you have any questions please let me know.
NOTE: This might not be the best solution since I'm just getting started with Django, however I still wanted to share this "How-to" with you all since I didn't find much information while googling about this.

In my opinion fixtures are a bit bad. If your database changes frequently, keeping them up-to-date will came a nightmare soon. Actually, it's not only my opinion, in the book "Two Scoops of Django" it's explained much better.
Instead I'll write a Python file to provide initial setup. If you need something more I'll suggest you look at Factory boy.
If you need to migrate some data you should use data migrations.
There's also "Burn Your Fixtures, Use Model Factories" about using fixtures.

Although #rockallite's answer is excellent, it does not explain how to handle fixtures that rely on natural keys instead of integer pk values.
Simplified version
First, note that #rockallite's solution can be simplified by using unittest.mock.patch as a context manager, and by patching apps instead of _get_model:
...
from unittest.mock import patch
...
def load_fixture(apps, schema_editor):
with patch('django.core.serializers.python.apps', apps):
call_command('loaddata', 'your_data.json', ...)
...
This works well, as long as your fixtures do not rely on natural keys.
If they do, you're likely to see a DeserializationError: ... value must be an integer....
The problem with natural keys
Under the hood, loaddata uses django.core.serializers.deserialize() to load your fixture objects.
The deserialization of fixtures based on natural keys relies on two things:
the presence of a get_by_natural_key() method on the model's default manager
the presence of a natural_key() method on the model itself
The get_by_natural_key() method is necessary for the deserializer to know how to interpret the natural key, instead of an integer pk value.
Both methods are necessary for the deserializer to get existing objects from the database by natural key, as also explained here.
However, the apps registry which is available in your migrations uses historical models, and these do not have access to custom managers or custom methods such as natural_key().
Possible solution: step 1
The problem of the missing get_by_natural_key() method from our custom model manager is relatively easy to solve:
Just set use_in_migrations=True on your custom manager, as described in the documentation.
This ensures that your historical models can access the current get_by_natural_key() during migrations, and fixture loading should now succeed.
However, your historical models still don't have a natural_key() method. As a result, your fixtures will be treated as new objects, even if they are already present in the database.
This may lead to a variety of errors if the data-migration is ever re-applied, such as:
unique-constraint violations (if your models have unique-constraints)
duplicate fixture objects (if your models do not have unique-constraints)
"get returned multiple objects" errors (due to duplicate fixture objects created previously)
So, effectively, you're still missing out on a kind of get_or_create-like behavior during deserialization.
To experience this, just apply a data-migration as described above (in a test environment), then roll back the same data-migration (without removing the data), then re-apply the data-migration.
Possible solution: step 2
The problem of the missing natural_key() method from the model itself is a bit more difficult to solve.
One solution would be to assign the natural_key() method from the current model to the historical model, for example:
...
from unittest.mock import patch
from django.apps import apps as current_apps
from django.core.management import call_command
...
def load_fixture(apps, schema_editor):
def _get_model_patch(app_label):
""" add natural_key method from current model to historical model """
historical_model = apps.get_model(app_label=app_label)
current_model = current_apps.get_model(app_label=app_label)
historical_model.natural_key = current_model.natural_key
return historical_model
with patch('django.core.serializers.python._get_model', _get_model_patch):
call_command('loaddata', 'your_data.json', ...)
...
Notes:
For clarity, I omitted things like error handling and attribute checking from the example. You should implement those where necessary.
This solution uses the current model's natural_key method, which may still lead to trouble in certain scenarios, but the same goes for Django's use_in_migrations option for model managers.

Django 1.6: Clear data in one table

I've a table name UGC and would like to clear all the data inside that table. I don't want to reset the entire app which would delete all the data in all the other models as well. Is it possible to clear only one single model? I also have South configured with my app, if that would help.

You could use raw SQL :
cursor.execute(“DROP TABLE UGC”)
or you could just use the ORM directly inside a Django shell :
UGCModel.objects.all().delete()
That would erase the data (not the table, though), so you have to be careful :)
Another way (for completeness and to make use of South) would be to comment out the model in your models declaration, migrate and then put it back again (assuming there are no models with a reference to it.)
HTH

In the admin interface, you can go to the list page for that model, then you can select all models and use the Delete selected ... action at the top of the table.
Remember that, in whatever way you delete the data, foreign keys default to ON DELETE CASCADE, so any model with a foreign key to a model you want to delete will be deleted as well. The admin interface will give you a complete overview of models that will be deleted.

Faced such issues today with django 2.0.2 because i created my model with
class Front(models.Model):
pass
and migrated it but when i later updated the model, i couldn't run
python manage.py makemigrations because it was saying
You are trying to add a non-nullable field 'update' to front without a default; we can't do that (the database needs something to populate existing rows).
Please select a fix:
1) Provide a one-off default now (will be set on all existing rows
with a null value for this column)
2) Quit, and let me add a default in models.py
Select an option:
What was my solution?
I choose option 2 which is to quit
I commented out the troublesome model/table which is Front in my case
I ran python manage.py makemigrations which deleted the troublesome table/model
I uncommented my model and ran python manage.py makemigrations again which recreated the table/model and finally
I migrated my changes with python manage.py migrate and everyhing was resolved!
Note: Be careful with the above instruction cause it will delete all references/foreign keys to the table/model with on_delete=models.CASCADE which is the default!

Django 3.1.14
If you're interested in doing it on the command line, and you'll be doing this type of cleaning of your db frequently, I do this:
# project_name/app_name/management/commands/clear_test_models.py
from django.core.management.base import BaseCommand
from django.apps import apps
keep_models = ['KeepModel0', 'KeepModel1']
class Command(BaseCommand):
"""
clear all in app_name app except listed in keep_models list
"""
help = 'clear all models except those listed in keep_models list'
def handle(self, *args, **kwargs):
my_app = apps.get_app_config('app_name')
my_models = my_app.get_models()
for model in my_models:
if model.__name__ not in keep_models:
model.objects.all().delete()
To run on the command line:
DJANGO_SETTINGS_MODULE=app_name.settings.local python manage.py clear_test_models

Django database error: missing table social_auth_usersocialauth when social_auth is not installed

I'm trying to deal with a very puzzling error in a Django app. When DEBUG=False, trying to delete a user (via user.delete()) gives this database error:
DatabaseError: relation "social_auth_usersocialauth" does not exist
LINE 1: ...", "social_auth_usersocialauth"."extra_data" FROM "social_au...
However, I do not have social_auth or anything by a similar name in INSTALLED_APPS, nor are there any such tables in my database, nor does any of my code reference anything of the sort (I ran a text search on 'social' in the entire project folder) - and again, this works fine when DEBUG=True. social_auth is installed on my system and on my PYTHONPATH, but I cannot see where this app is getting the idea it should be having social_auth's tables in its database, let alone why it only thinks so when DEBUG=False.
What possible pathways could my app be getting this table from and how could I convince it it's not supposed to be there?

The problem could be caused by saved generic relations realized by Django content types. Relations in Django are not only static, implemented by models and INSTALLED_APPS but also dynamic implemented by table django_content_type that saves mapping from a numeric id to app_label + model. An example of possible dynamic relationship is a permission or a comment. You can have or have not a permission to any table of any installed application. You can write a comment to everything e.g to an article, to a user to a comment itself without changing any model. This relation is realized by saving numeric id of ContentType related to that model (table) and a primary key of related object (row).
Django does not expect that someone can manipulate the database manually. If you use south for manipulation then if you after uninstalling an application then run syncdb, you are asked by south if you want automatically remove orphant content types. Then can be unused tables removed securely without beeing later referenced.
(Possible hack: delete from django_content_type where app_label='social_auth' but south is unfallible.)
Many parts of the question are still open.
Edit:
Why it was not the right way: All generic relations are from descendants to the parent and all data about the relation are saved in descendant. If the child app is removed from INSTALLED_APPS then django.db code can nevermore try to remove descendants because it can not recognize which columns contain the relation data.

This table is created by django-social-auth application.
Looks like you've added it to your project and haven't launched migrate (or syncdb).