We Keep Coding

Django swappable model foreign keys

Suppose I have a reusable app, that defines a model Person and a model Invite.
A Person has a OneToOne field to AUTH_USER_MODEL and defines some basic fields (such as birthday). It is a swappable model, so that a project which uses this app can easily add other fields (such as gender, etc.)
In my reusable app, I define a setting that provides the swapping model (otherwise, a default one will be used, exactly as django.contrib.auth does it.
The Invite model has a OneToOneField to the swappable Person model and an email field. (I think, it's quite clear what this model is for). The model itself is swappable as well, but I don't think that this makes any difference for the kind of problem I am facing.
reusable app models:
class AbstractPerson(models.Model):
user = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.SET_NULL, null=True, blank=True, related_name='person')
birthdate = models.DateField()
class Meta:
abstract = True
class Person(AbstractPerson):
class Meta(AbstractPerson.Meta):
swappable = 'REUSABLEAPP_PERSON_MODEL'
class AbstractInvite(models.Model):
email = models.EmailField()
person = models.OneToOneField(settings.REUSABLEAPP_PERSON_MODEL, on_delete=models.CASCADE, null=False, related_name='+')
class Meta:
abstract = True
class Invite(AbstractInvite):
class Meta(AbstractInvite.Meta):
swappable = 'REUSABLEAPP_INVITE_MODEL'
If I create the initial migration for my reusable app (using a dummy project and not swapping out my models), I get the following migration for my reusable app:
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='Person',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('birthdate', models.DateField()),
('user', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='person', to=settings.AUTH_USER_MODEL)),
],
options={
'abstract': False,
'swappable': 'REUSABLEAPP_PERSON_MODEL',
},
),
migrations.CreateModel(
name='Invite',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('email', models.EmailField(max_length=254)),
('person', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='+', to=settings.REUSABLEAPP_PERSON_MODEL)),
],
options={
'abstract': False,
'swappable': 'REUSABLEAPP_INVITE_MODEL',
},
),
]
If I then include my reusable app in another project, and swap out the Person and Invite model, I get an error when running makemigrations:
ValueError: The field myreusable_app.Invite.person was declared with a lazy reference to 'tester.myperson', but app 'tester' isn't installed.
(tester is the app that defines the swapped models, obviously)
If I delete the migration from my reusable app, and run makemigrations again, it works. the created migration is almost identical to the one above, with the exception of a new dependency:
migrations.swappable_dependency(settings.REUSABLEAPP_PERSON_MODEL),
The migration created in the tester app looks as follows:
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name='MyPerson',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('birthdate', models.DateField()),
('user', models.OneToOneField(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='person', to=settings.AUTH_USER_MODEL)),
],
options={
'abstract': False,
},
),
migrations.CreateModel(
name='MyInvite',
fields=[
('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('email', models.EmailField(max_length=254)),
('person', models.OneToOneField(on_delete=django.db.models.deletion.CASCADE, related_name='+', to=settings.REUSABLEAPP_PERSON_MODEL)),
],
options={
'abstract': False,
},
),
]
I have looked up what swappable_dependency actually does: It looks only which app name is defined (let's say, the setting is tester.mymodel), and creates a dependency to this apps initial migration.
Now, if I delete the created migration from my tester app, I cannot run makemigrations again, I get the same error as above.
To clarify up until this point: Everything works as desired, if I delete the initial migration (and therefore all future migrations!) from my reusable app.
The problem, as I understand it, is the following:
The reusable app has a dependency to the initial migration of the client application that defines the swapped models. But this migration does not exist yet (heck, I am trying to create it!), so makemigration failes. (running makemigrations tester does not help).
But this almost exact same thing works flawlessly when swapping out the standard User model for a custom one. Furthermore, I do not fully understand why the error message states that the tester app is not installed. It definitly is inside my INSTALLED_APPS and it is picked up by the django-ecosystem.
After a few hours, I came up with a possible (but hacky) workaround:
Remove my reusable app from INSTALLED_APPS
Create MyInvite and MyPerson in the tester app (they both inherit from django.models.Model
Create those models by running makemigrations tester
Add my reusable app to INSTALLED_APPS
Define the swap settings
Change the inheritance of my models to their respective abstract counterparts
Run makemigrations again.
This works, because the initial migration of my reusable app now can fullfill the dependency to the swapped models by looking at the initial migration of tester that defines the models with the same name that is defined in the swap variables.
But I am sure that there must be a better way to do that.
This leaves me with the following questions:
How can I handle foreign key relationships to swapped models?
Why can't I create migrations for one app without looking at the migrations of other apps?

Unfortunately, I have to answer my own question.
This package provides a public interface to the swapping API. I have learned, that looking at closed issues is sometimes more helpful than reading the open ones.
Especially #12, and the open #10 answer my question. To sum it up, what works for the AUTH_USER_MODEL setting, does not work for any other swappable model, because the lazy_reference error is ignored, when the target model is set as AUTH_USER_MODEL within django, here:
# There shouldn't be any operations pending at this point.
from django.core.checks.model_checks import _check_lazy_references
ignore = {make_model_tuple(settings.AUTH_USER_MODEL)} if ignore_swappable else set()
errors = _check_lazy_references(self, ignore=ignore)
if errors:
raise ValueError("\n".join(error.msg for error in errors))
The proposed solution would be a registry, where all swapped models are registered and then looked up, to ignore the lazy reference error. Unfortunately, there seems to be no plans from the django devs to support this, as I have not found any open issues or feature requests for that.

Django trying to add an existing field when making a model managed

How can I stop Django 2.2.4 from trying to create a database column that already exists when making a model managed?
I have 2 models, ticket and message, which were connected to tables in a third-party database so the models were created with managed=False. I'm moving away from the third-party tool. The ticket model was change to managed=True a while ago by somebody else, and now I'm trying to do the same with the message model.
These are the relevant parts of the model:
from django.db import models
class Message(models.Model):
mid = models.BigAutoField(db_column='MID', primary_key=True)
ticket = models.ForeignKey('Ticket', on_delete=models.CASCADE, db_column='TID')
author = models.CharField(db_column='AUTHOR', max_length=32)
date = models.DateTimeField(db_column='DATE')
internal = models.CharField(db_column='INTERNAL', max_length=1)
isoper = models.CharField(db_column='ISOPER', max_length=1)
headers = models.TextField(db_column='HEADERS')
msg = models.TextField(db_column='MSG')
class Meta:
# managed = False
db_table = 'messages'
permissions = (
("can_change_own_worked_time", "Can change own worked time"),
("can_change_own_recently_worked_time", "Can change own recently worked time"),
("can_change_subordinate_worked_time", "Can change subordinate worked time"),
)
This are the migrations that get generated by commenting out managed=False:
# Generated by Django 2.2.4 on 2020-06-18 20:56 (0017_auto_20200618_1656)
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('troubleticket', '0016_auto_20200511_1644'),
]
operations = [
migrations.AlterModelOptions(
name='message',
options={'permissions': (('can_change_own_worked_time', 'Can change own worked time'), ('can_change_own_recently_worked_time', 'Can change own recently worked time'), ('can_change_subordinate_worked_time', 'Can change subordinate worked time'))},
),
]
# Generated by Django 2.2.4 on 2020-06-18 21:14 (0018_message_ticket)
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
('troubleticket', '0017_auto_20200618_1656'),
]
operations = [
migrations.AddField(
model_name='message',
name='ticket',
field=models.ForeignKey(db_column='TID', default=1, on_delete=django.db.models.deletion.CASCADE, to='troubleticket.Ticket'),
preserve_default=False,
),
]
When I try to apply those migrations I get this error:
django.db.utils.OperationalError: (1060, "Duplicate column name 'TID'")
The initial migration didn't include the TID column and neither do any of the subsequent migrations so I understand why Django thinks it's a new column. But it isn't a new column (the model has had it since the first time it was committed to the git repo) so I also understand why MySQL is throwing an error.
This is the initial migration:
# Generated by Django 2.0.8 on 2018-08-20 14:43 (0001_initial)
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
]
operations = [
migrations.CreateModel(
name='Message',
fields=[
('mid', models.BigAutoField(db_column='MID', primary_key=True, serialize=False)),
('author', models.CharField(db_column='AUTHOR', max_length=32)),
('date', models.DateTimeField(db_column='DATE')),
('internal', models.CharField(db_column='INTERNAL', max_length=1)),
('isoper', models.CharField(db_column='ISOPER', max_length=1)),
('headers', models.TextField(db_column='HEADERS')),
('msg', models.TextField(db_column='MSG')),
],
options={
'managed': False,
'db_table': 'messages',
},
),
migrations.CreateModel(
name='Ticket',
fields=[
('id', models.BigIntegerField(db_column='ID', primary_key=True, serialize=False)),
('accesskey', models.CharField(db_column='ACCESSKEY', max_length=64)),
('open', models.DateTimeField(db_column='OPEN')),
('updated', models.DateTimeField(db_column='UPDATED')),
('closed', models.DateTimeField(db_column='CLOSED', null=True)),
('status', models.CharField(db_column='STATUS', max_length=3)),
('oper', models.CharField(db_column='OPER', max_length=32)),
('email', models.CharField(db_column='EMAIL', max_length=128)),
('name', models.CharField(db_column='NAME', max_length=128)),
('subject', models.CharField(db_column='SUBJECT', max_length=255)),
('lname', models.CharField(db_column='LNAME', max_length=50)),
('company', models.CharField(db_column='C0', max_length=255)),
('type', models.CharField(db_column='C1', max_length=255)),
('c2', models.CharField(db_column='C2', max_length=255)),
('c3', models.DecimalField(db_column='C3', decimal_places=2, max_digits=6)),
('c4', models.CharField(db_column='C4', max_length=255)),
('pending', models.CharField(db_column='C5', max_length=255)),
('c6', models.CharField(db_column='C6', max_length=255)),
('c7', models.CharField(db_column='C7', max_length=255)),
('c8', models.CharField(db_column='C8', max_length=255)),
('cc', models.CharField(db_column='C9', max_length=255)),
('grp', models.CharField(db_column='GRP', max_length=10)),
('item', models.CharField(db_column='ITEM', max_length=255)),
],
options={
'managed': False,
'db_table': 'tickets',
},
),
]

2 Seconds after writing my comment I figured out how to solve the problem.
When 'migrating' from another ORM to Django always consider the following aspects.
The following is my recommendation of order, but I'm still learning how to use django migrations, so keep that in mind.
1. Consider which fields of the model are currently known to django
It is important to note, that the actual content of the database does not matter to django when calculating which columns to add or alter. This means that, in the initial migration of the django models, where managed is still =False, the columns are "created" and registered by django. When makemigrations calculates which columns to add, it only takes the columns mentioned in the initial migration as given. Not the actual contents of the db.
Knowing that, we can now go field for field and decide for each one according to point 2.
2. Consider which fields of the model should or should not be created by django
Now, generally, when setting a Model managed=True fields will fall into 3 categories.
2.1 The field is currently in the model definition, the database, and is also in the initial commit.
For this case, no actions need to be taken.
2.2 The field is currently in the model definition, the database, but is not in the initial commit.
For this case, the field definition must be added to the original initial migration script. As such, the migrations.CreateModel call might look like this:
...
('field_already_present', models.FloatField(blank=True, null=True)),
...
2.3 The field is currently in the database, but not in the model definition or the initial commit
If the field is not needed in the django application it can be left out.
If it is needed at some point it will have to be added to the model as well as the first initial migration. That way, django makemigrations will not attempt to create the field.
3. Set managed=True
Now set managed=True and make your migrations. Do so twice! The first time, makemigrations will set the model to managed, then it will add fields not in the initial commit.
After this, the model can be treated same as a normally created and managed-from-the-start django model.

How to use a user as a ForeignKey in a Django model

I've got this model in my Django application:
class ClubSession(models.Model):
location = models.CharField(max_length=200)
coach = models.ForeignKey('auth.User', on_delete=models.CASCADE)
date = models.DateTimeField(default=now)
details = models.TextField()
def __str__(self):
return self.title
I can run python manage.py makemigrations club_sessions without issue but when I thn run python manage.py migrate club_sessions I get ValueError: Field 'id' expected a number but got 'username'. username is a superuser and already exists.
How do I resolve this?
This is the latest migration:
# Generated by Django 3.0.6 on 2020-05-28 15:07
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('club_sessions', '0004_auto_20200528_1450'),
]
operations = [
migrations.AlterField(
model_name='clubsession',
name='coach',
field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL),
),
migrations.AlterField(
model_name='clubsession',
name='location',
field=models.CharField(max_length=200),
),
]

By default Django lets a ForeignKey refer to the primary key of the target model. This also has some advantages to make relations more uniform.
If you really want to save the username in the ForeignKey, you can specify a to_field=… parameter [Django-doc] and let it refer to a column that is unique (the username of the default User model is unique), so we can refer to it with:
from django.conf import settings
class ClubSession(models.Model):
location = models.CharField(max_length=200)
coach = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE,
to_field='username'
)
date = models.DateTimeField(default=now)
details = models.TextField()
def __str__(self):
return self.title
You will need to remove the already existing migration and make a new one in order to migrate the database properly.
Note: It is normally better to make use of the settings.AUTH_USER_MODEL [Django-doc] to refer to the user model, than to use the User model [Django-doc] directly. For more information you can see the referencing the User model section of the documentation.

Django FileField not respecting null=False parameter

I have a Django FileField which is set to not nullable. However it is behaving like it is nullable.
class Thing(models.Model):
document = models.FileField(null=False, blank=False)
thing_id = models.CharField()
# This does not raise but I would like it to
Thing.objects.create(thing_id='123')
edit: the migrations
class Migration(migrations.Migration):
dependencies = [
('data_source', '0002_auto_20190212_1913'),
]
operations = [
migrations.CreateModel(
name='Thing',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('document', models.FileField(upload_to='')),
('thing_id', models.CharField(default=None, max_length=200)),
],
options={
'abstract': False,
},
),
]

The problem is not with your field definition. It's got to do with validation when manually creating a model instance. You need to perform that full validation yourself, since you're not using a ModelForm.
Use this example:
from django.core.exceptions import ValidationError
thing = Thing(thing_id='123')
try:
thing.full_clean()
thing.save()
except ValidationError:
# Handle validation issues.
Read this section in the documentation: https://docs.djangoproject.com/en/2.1/ref/models/instances/#validating-objects
Quoting from there:
Note that full_clean() will not be called automatically when you call
your model’s save() method. You’ll need to call it manually when you
want to run one-step model validation for your own manually created
models

You can add the validation directly in your model overriding save() method:
class Thing(models.Model):
document = models.FileField(null=False, blank=False, default=None)
thing_id = models.CharField(max_length=200)
def save(self, *args, **kwargs):
self.full_clean()
super().save(*args, **kwargs)

By default null and blank are False. Link for every model field. Don't need to add those.
from django.db import models
class Thing(models.Model):
document = models.FileField()
thing_id = models.CharField()

Django migrations with foreign key

I' trying to migrate this model:
class Questionpart_image(models.Model):
questionpart = models.ForeignKey(Questionpart, null=True, blank=True)
image = models.ImageField()
to this:
class Questionpart_image(Questionpart): # notice this base class
image = models.ImageField()
to make advantage of inheritance. Django produces the following migration:
class Migration(migrations.Migration):
dependencies = [
('ochsite', '0016_auto_20150809_1903'),
]
operations = [
migrations.RemoveField(
model_name='questionpart_image',
name='id',
),
migrations.RemoveField(
model_name='questionpart_image',
name='questionpart',
),
migrations.AddField(
model_name='questionpart_image',
name='questionpart_ptr',
field=models.OneToOneField(default='', primary_key=True, to='ochsite.Questionpart', serialize=False, parent_link=True, auto_created=True),
preserve_default=False,
),
]
but this does not set the right foreign key to questionpart_ptr from questionpart field. How can I achieve that?
I've been searching for a long time, but nothing...thanks

Simply don't rely on automatic migrations, make your own or modify that one.
Simplest solution will be by moving AddField in migration to the top of the list and inject between it and RemoveFields RunPython block that will rewrite id's from old field to new (and in other direction in reverse, if needed).