We Keep Coding

Pulling several Django models together into a single list

I have a MySQL database with four related tables: project, unit, unit_equipment, and equipment. A project can have many units; a unit can have many related equipment entries. A single unit can only belong to one project, but there is a many-to-many between equipment and unit (hence the unit_equipment bridge table in the DB). I'm using Django and trying to create a view (or a list?) that shows all 3 models on the same page, together. So it would list all projects, all units, and all equipment. Ideally, the display would be like this:
Project --------- Unit ------------- Equipment
Project 1 first_unit some_equipment1, some_equipment2
Project 1 second_unit more_equipment1, more_equipment2
Project 2 another_unit some_equipment1, more_equipment1
Project 2 and_another_unit some_equipment2, more_equipment2
but at this point I'd also be happy with just having a separate line for each piece of equipment, if comma-separating them is a pain.
Although it seems straightforward to create a form where I can add a new project and add related unit and equipment data (using the TabularInline class), I cannot for the life of me figure out how to bring this data together and just display it. I just want a "master list" of everything in the database, basically.
Here's the code I have so far:
models.py
class Project(models.Model):
name = models.CharField(max_length=255, blank=True, null=True)
class Meta:
managed = False
db_table = 'project'
def __str__(self):
return self.name
class Unit(models.Model):
project = models.ForeignKey(Project, models.DO_NOTHING, blank=True, null=True)
name = models.CharField(max_length=255, blank=True, null=True)
class Meta:
managed = False
db_table = 'unit'
def __str__(self):
return self.name
class UnitEquipment(models.Model):
unit = models.ForeignKey(Unit, models.DO_NOTHING, blank=True, null=True)
equipment = models.ForeignKey(Equipment, models.DO_NOTHING, blank=True, null=True)
class Meta:
managed = False
db_table = 'unit_equipment'
class Equipment(models.Model):
name = models.CharField(max_length=100, blank=True, null=True)
description = models.CharField(max_length=255, blank=True, null=True)
class Meta:
managed = False
db_table = 'equipment'
def __str__(self):
return self.name
views.py
def project_detail_view(request):
obj = Project.objects.all()
context = {'object': obj}
return render(request, "project/project_detail.html", context)
urls.py
urlpatterns = [
path('project/', project_detail_view),
path('', admin.site.urls),
]
admin.py
class UnitTabularInLine(admin.TabularInline):
model = Unit
extra = 0
class ProjectAdmin(admin.ModelAdmin):
inlines = [UnitTabularInLine]
class Meta:
model = Project
# a list of displayed columns name.
list_display = ['name']
# define search columns list, then a search box will be added at the top of list page.
search_fields = ['name']
# define filter columns list, then a filter widget will be shown at right side of list page.
list_filter = ['name']
# define model data list ordering.
ordering = ('name')
I think I need to somehow add more entries to the list_display in the admin file, but every time I try to add unit or equipment it throws an error. I've also tried adding more attributes to Project, but I can't seem to get the syntax right, and I'm never sure which model class I'm supposed to make it.
I've also looked at FormSets, but I cannot get my head around how to alter my current code to get it to work.
How do I get these models together into a unified view?

You don't need to edit the admin view to add your own view: which you may find you are able to do in this case to get your data displayed exactly as you want.
If you do want to show the related object values in the admin list, then you can use lookups and custom columns: however in this case your list would be based upon the Unit.
# You don't need an explicit UnitEquipment model here: you can
# use a simple ManyToManyField
class Unit(models.Model):
project = ...
name = ...
equipment = models.ManyToManyField(Equipment, related_name='units')
def equipment_list(admin, instance):
return ', '.join([x.name for x in instance.equimpent.all()])
class UnitAdmin(admin.ModelAdmin):
class Meta:
model = Unit
list_display = ['project__name', 'name', equipment_list]
def get_queryset(self, request):
return super().get_queryset(request)\
.select_related('project')\
.prefetch_related('equipment')
Note that you need to have the queryset override, otherwise there will be a bunch of extra queries as each unit also requires fetching the project and list of equipment for that unit.
There's also a further improvement you can make to your queries: you could aggregate the related equipment names using a Subquery annotation, and prevent the second query (that fetches all related equipment items for the units in the queryset). This would replace the prefetch_related()

Thanks to #Matthew Schinckel, I was able to find my way to the answer. Here's what my files look like now (only edited the Unit class in models.py):
models.py
class Unit(models.Model):
project = models.ForeignKey(Project, models.DO_NOTHING, blank=True, null=True)
name = models.CharField(max_length=255, blank=True, null=True)
equipment = models.ManyToManyField(Equipment, related_name='units')
class Meta:
managed = False
db_table = 'unit'
def __str__(self):
return self.name
def equipment_list(self):
return ', '.join([x.name for x in self.equipment.all()])
admin.py
class UnitAdmin(admin.ModelAdmin):
class Meta:
model = Unit
# a list of displayed columns name.
list_display = ('project', 'name', 'equipment_list')
# define search columns list, then a search box will be added at the top of list page.
search_fields = ['project']
# define filter columns list, then a filter widget will be shown at right side of list page.
list_filter = ['project', 'name']
# define model data list ordering.
ordering = ('project', 'name')
def get_queryset(self, request):
return super().get_queryset(request)\
.select_related('project')\
.prefetch_related('equipment')
So the changes I made were:
1. Make list_display a tuple instead of a list.
2. Throw def equipment_list(self) into the Unit class (so it's callable as an attribute of Unit) and pass (self) instead of (admin, instance) (I kept getting an error that was looking for the instance argument).

Modifying value on serialization - Django Rest Framework

I have a model which contains sensitive data, let's say a social security number, I would like to transform that data on serialization to display only the last four digits.
I have the full social security number stored: 123-45-6789.
I want my serializer output to contain: ***-**-6789
My model:
class Employee (models.Model):
name = models.CharField(max_length=64,null=True,blank=True)
ssn = models.CharField(max_length=16,null=True,blank=True)
My serializer:
class EmployeeSerializer(serializers.ModelSerializer):
id = serializers.ReadOnlyField()
class Meta:
model = Employee
fields = ('id','ssn')
read_only_fields = ['id']

You can use SerializerMethodField:
class EmployeeSerializer(serializers.ModelSerializer):
id = serializers.ReadOnlyField()
ssn = SerializerMethodField()
class Meta:
model = Employee
fields = ('id','ssn')
read_only_fields = ['id']
def get_ssn(self, obj):
return '***-**-{}'.format(obj.ssn.split('-')[-1]

If you don't need to update the ssn, just shadow the field with a SerializerMethodField and define get_ssn(self, obj) on the serializer.
Otherwise, the most straightforward way is probably to just override .to_representation():
def to_representation(self, obj):
data = super(EmployeeSerializer, self).to_representation(obj)
data['ssn'] = self.mask_ssn(data['ssn'])
return data
Please add special case handling ('ssn' in data) as necessary.

Elaborating on #dhke’s answer, if you want to be able to reuse this logic to modify serialization across multiple serializers, you can write your own field and use that as a field in your serializer, such as:
from rest_framework import serializers
from rest_framework.fields import CharField
from utils import mask_ssn
class SsnField(CharField):
def to_representation(self, obj):
val = super().to_representation(obj)
return mask_ssn(val) if val else val
class EmployeeSerializer(serializers.ModelSerializer):
ssn = SsnField()
class Meta:
model = Employee
fields = ('id', 'ssn')
read_only_fields = ['id']
You can also extend other fields like rest_framework.fields.ImageField to customize how image URLs are serialized (which can be nice if you’re using an image CDN on top of your images that lets you apply transformations to the images).

many-to-many in list display django

class Product(models.Model):
products = models.CharField(max_length=256)
def __unicode__(self):
return self.products
class PurchaseOrder(models.Model):
product = models.ManyToManyField('Product')
vendor = models.ForeignKey('VendorProfile')
dollar_amount = models.FloatField(verbose_name='Price')
I have that code. Unfortunately, the error comes in admin.py with the ManyToManyField
class PurchaseOrderAdmin(admin.ModelAdmin):
fields = ['product', 'dollar_amount']
list_display = ('product', 'vendor')
The error says:
'PurchaseOrderAdmin.list_display[0]', 'product' is a ManyToManyField
which is not supported.
However, it compiles when I take 'product' out of list_display. So how can I display 'product' in list_display without giving it errors?
edit: Maybe a better question would be how do you display a ManyToManyField in list_display?

You may not be able to do it directly. From the documentation of list_display
ManyToManyField fields aren’t supported, because that would entail
executing a separate SQL statement for each row in the table. If you
want to do this nonetheless, give your model a custom method, and add
that method’s name to list_display. (See below for more on custom
methods in list_display.)
You can do something like this:
class PurchaseOrderAdmin(admin.ModelAdmin):
fields = ['product', 'dollar_amount']
list_display = ('get_products', 'vendor')
def get_products(self, obj):
return "\n".join([p.products for p in obj.product.all()])
OR define a model method, and use that
class PurchaseOrder(models.Model):
product = models.ManyToManyField('Product')
vendor = models.ForeignKey('VendorProfile')
dollar_amount = models.FloatField(verbose_name='Price')
def get_products(self):
return "\n".join([p.products for p in self.product.all()])
and in the admin list_display
list_display = ('get_products', 'vendor')

This way you can do it, kindly checkout the following snippet:
class Categories(models.Model):
""" Base category model class """
title = models.CharField(max_length=100)
description = models.TextField()
parent = models.ManyToManyField('self', default=None, blank=True)
when = models.DateTimeField('date created', auto_now_add=True)
def get_parents(self):
return ",".join([str(p) for p in self.parent.all()])
def __unicode__(self):
return "{0}".format(self.title)
And in your admin.py module call method as follows:
class categories(admin.ModelAdmin):
list_display = ('title', 'get_parents', 'when')

If you want to save extra queries, you can use prefetch_related in the get_queryset method like below:
class PurchaseOrderAdmin(admin.ModelAdmin):
fields = ['product', 'dollar_amount']
list_display = ('get_products', 'vendor')
def get_queryset(self, request):
qs = super().get_queryset(request)
return qs.prefetch_related('product')
def get_products(self, obj):
return ",".join([p.products for p in obj.product.all()])
According to the Docs, In this way, there would be just one extra query needed to fetch related Product items of all PurchaseOrder instances instead of needing one query per each PurchaseOrder instance.

list Box Django Forms

I'm using Django forms and need to create a list box.
What would be the equivalent of listbox in Django form fields?
I checked the documentation #
https://docs.djangoproject.com/en/dev/ref/forms/fields/#modelchoicefield
but unable to find it.
Here is my code snippet,
Models.py
class Volunteer(models.Model):
NO_OF_HRS = (('1','1')
('2','2'))
datecreated = models.DateTimeField()
volposition = models.CharField(max_length=300)
roledesc = models.CharField(max_length=300)
Duration = models.CharField(choices=NO_OF_HRS,max_length=1)**
forms.py
class VolunteerForm(forms.ModelForm)
datecreated = forms.DateField(label=u'Creation Date')
volposition = forms.CharField(label=u'Position Name', max_length=300)
roledesc = forms.roledesc(label=u'Role description',max_length=5000)
Duration = forms.CharField(widget=forms.select(choices=NO_OF_HRS),max_length=2)
When I try to run, I get the following error,
NO_OF_HRS is not defined

Your NO_OF_HRS tuple is defined inside the model and not available to the form. It has to be imported in forms.py just like any other Python object. Try moving the tuple outside the model definition and import in your forms.py like this:
models.py
NO_OF_HRS = (('1','1')
('2','2'))
class Volunteer(models.Model):
# ...
duration = models.CharField(choices=NO_OF_HRS, max_length=1)
forms.py
from path.to.models import NO_OF_HRS
class VolunteerForm(forms.Form):
# ...
duration = forms.CharField(widget=forms.Select(choices=NO_OF_HRS), max_length=1)
It also looks like you want to use a ModelForm. In this case you don't need to add any field definitions to your VolunteerForm, simply set your model in the inner Meta class.
forms.py
from path.to.models Volunteer
class VolunteerForm(forms.ModelForm):
class Meta:
model = Volunteer

How does Django Know the Order to Render Form Fields?

If I have a Django form such as:
class ContactForm(forms.Form):
subject = forms.CharField(max_length=100)
message = forms.CharField()
sender = forms.EmailField()
And I call the as_table() method of an instance of this form, Django will render the fields as the same order as specified above.
My question is how does Django know the order that class variables where defined?
(Also how do I override this order, for example when I want to add a field from the classe's init method?)

New to Django 1.9 is Form.field_order and Form.order_fields().
# forms.Form example
class SignupForm(forms.Form):
password = ...
email = ...
username = ...
field_order = ['username', 'email', 'password']
# forms.ModelForm example
class UserAccount(forms.ModelForm):
custom_field = models.CharField(max_length=254)
def Meta:
model = User
fields = ('username', 'email')
field_order = ['username', 'custom_field', 'password']

[NOTE: this answer is now pretty completely outdated - please see the discussion below it, and more recent answers].
If f is a form, its fields are f.fields, which is a django.utils.datastructures.SortedDict (it presents the items in the order they are added). After form construction f.fields has a keyOrder attribute, which is a list containing the field names in the order they should be presented. You can set this to the correct ordering (though you need to exercise care to ensure you don't omit items or add extras).
Here's an example I just created in my current project:
class PrivEdit(ModelForm):
def __init__(self, *args, **kw):
super(ModelForm, self).__init__(*args, **kw)
self.fields.keyOrder = [
'super_user',
'all_districts',
'multi_district',
'all_schools',
'manage_users',
'direct_login',
'student_detail',
'license']
class Meta:
model = Privilege

I went ahead and answered my own question. Here's the answer for future reference:
In Django form.py does some dark magic using the __new__ method to load your class variables ultimately into self.fields in the order defined in the class. self.fields is a Django SortedDict instance (defined in datastructures.py).
So to override this, say in my example you wanted sender to come first but needed to add it in an init method, you would do:
class ContactForm(forms.Form):
subject = forms.CharField(max_length=100)
message = forms.CharField()
def __init__(self,*args,**kwargs):
forms.Form.__init__(self,*args,**kwargs)
#first argument, index is the position of the field you want it to come before
self.fields.insert(0,'sender',forms.EmailField(initial=str(time.time())))

Fields are listed in the order they are defined in ModelClass._meta.fields. But if you want to change order in Form, you can do by using keyOrder function.
For example :
class ContestForm(ModelForm):
class Meta:
model = Contest
exclude=('create_date', 'company')
def __init__(self, *args, **kwargs):
super(ContestForm, self).__init__(*args, **kwargs)
self.fields.keyOrder = [
'name',
'description',
'image',
'video_link',
'category']

With Django >= 1.7 your must modify ContactForm.base_fields as below:
from collections import OrderedDict
...
class ContactForm(forms.Form):
...
ContactForm.base_fields = OrderedDict(
(k, ContactForm.base_fields[k])
for k in ['your', 'field', 'in', 'order']
)
This trick is used in Django Admin PasswordChangeForm: Source on Github

Form fields have an attribute for creation order, called creation_counter. .fields attribute is a dictionary, so simple adding to dictionary and changing creation_counter attributes in all fields to reflect new ordering should suffice (never tried this, though).

Use a counter in the Field class. Sort by that counter:
import operator
import itertools
class Field(object):
_counter = itertools.count()
def __init__(self):
self.count = Field._counter.next()
self.name = ''
def __repr__(self):
return "Field(%r)" % self.name
class MyForm(object):
b = Field()
a = Field()
c = Field()
def __init__(self):
self.fields = []
for field_name in dir(self):
field = getattr(self, field_name)
if isinstance(field, Field):
field.name = field_name
self.fields.append(field)
self.fields.sort(key=operator.attrgetter('count'))
m = MyForm()
print m.fields # in defined order
Output:
[Field('b'), Field('a'), Field('c')]

If either fields = '__all__':
class AuthorForm(ModelForm):
class Meta:
model = Author
fields = '__all__'
or exclude are used:
class PartialAuthorForm(ModelForm):
class Meta:
model = Author
exclude = ['title']
Then Django references the order of fields as defined in the model. This just caught me out, so I thought I'd mention it. It's referenced in the ModelForm docs:
If either of these are used, the order the fields appear in the form will be the order the fields are defined in the model, with ManyToManyField instances appearing last.

As of Django 1.7 forms use OrderedDict which does not support the append operator. So you have to rebuild the dictionary from scratch...
class ChecklistForm(forms.ModelForm):
class Meta:
model = Checklist
fields = ['name', 'email', 'website']
def __init__(self, guide, *args, **kwargs):
self.guide = guide
super(ChecklistForm, self).__init__(*args, **kwargs)
new_fields = OrderedDict()
for tier, tasks in guide.tiers().items():
questions = [(t['task'], t['question']) for t in tasks if 'question' in t]
new_fields[tier.lower()] = forms.MultipleChoiceField(
label=tier,
widget=forms.CheckboxSelectMultiple(),
choices=questions,
help_text='desired set of site features'
)
new_fields['name'] = self.fields['name']
new_fields['email'] = self.fields['email']
new_fields['website'] = self.fields['website']
self.fields = new_fields

For future reference: things have changed a bit since newforms. This is one way of reordering fields from base formclasses you have no control over:
def move_field_before(form, field, before_field):
content = form.base_fields[field]
del(form.base_fields[field])
insert_at = list(form.base_fields).index(before_field)
form.base_fields.insert(insert_at, field, content)
return form
Also, there's a little bit of documentation about the SortedDict that base_fields uses here: http://code.djangoproject.com/wiki/SortedDict

The easiest way to order fields in django 1.9 forms is to use field_order in your form Form.field_order
Here is a small example
class ContactForm(forms.Form):
subject = forms.CharField(max_length=100)
message = forms.CharField()
sender = forms.EmailField()
field_order = ['sender','message','subject']
This will show everything in the order you specified in field_order dict.

Using fields in inner Meta class is what worked for me on Django==1.6.5:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""
Example form declaration with custom field order.
"""
from django import forms
from app.models import AppModel
class ExampleModelForm(forms.ModelForm):
"""
An example model form for ``AppModel``.
"""
field1 = forms.CharField()
field2 = forms.CharField()
class Meta:
model = AppModel
fields = ['field2', 'field1']
As simple as that.

I've used this to move fields about:
def move_field_before(frm, field_name, before_name):
fld = frm.fields.pop(field_name)
pos = frm.fields.keys().index(before_name)
frm.fields.insert(pos, field_name, fld)
This works in 1.5 and I'm reasonably sure it still works in more recent versions.

To add something, you can use this (Django 3+):
class ...(forms.ModelForm):
field = ...
class Meta:
model = Xxxxxx
fields = '__all__'
field_order = ['field', '__all__']
__all__ works

It has to do with the meta class that is used in defining the form class. I think it keeps an internal list of the fields and if you insert into the middle of the list it might work. It has been a while since I looked at that code.

None of these answers worked for me, Actually, you do not have to do anything custom, you can just order the fields in the order you want in your Model class. For eg ... the below code
from django.db import models
class Student(models.Model):
class Meta:
verbose_name_plural = "categories"
id = models.AutoField(primary_key=True)
name = models.CharField(max_length=300)
nick_name = models.CharField(max_length=300)
created_at = models.DateTimeField(auto_now_add=True)
updated_at = models.DateTimeField(auto_now=True)
def __str__(self):
return self.name
Your admin interface for model Will display the fields exactly in the same order in which you have declared in this case it will be (id, name, nick_name )

The order of the fields in the form depends on the order of the enumeration in the View , tested in Django 4.0.5.
class Sec_CreateView(CreateView):
model = Sec
template_name = 'forms/sec_create.html'
fields = ['rto', 'ssid', 'lic', 'IPv4', 'vlans']