We Keep Coding

Passing values to required fields that do not come from inputs in Django ModelForm

I have a ModelForm. There are several fields such as:
['display_from', 'display_to', 'position', 'image']
However, display_from and display_to are not in my template. I want to override the validation and assign those fields some values before any validation errors appear. I have done something like this:
def clean(self):
position = self.cleaned_data.get('position')
image = self.cleaned_data.get('image')
display_from = "value here"
display_to = "value here"
self.cleaned_data['display_from'] = display_from
self.cleaned_data['display_to'] = display_to
return self.cleaned_data
When I print the cleaned_data, I can see the values in the terminal. However, when I display the error messages on the page, it tells me that display_from and display_to are required.
Perhaps this should not be done in the clean() method?
I tried to use empty hidden inputs as well, but it didn't work. Any ideas?
EDIT:
Excluding the fields does not help, it returns an IntegrityError message:
null value in column "display_from" violates not-null constraint

Keep in mind the clean() function is called after the various clean_ functions for each field on the model (docs here).
So those individual field clean functions will be raising ValidationErrors that get pinned to the form.errors dict before your clean() function comes in and adds the default values to the cleaned_data dict.
You could also carry on with the approach you already started with. Just remove the errors from the errors dict.
def clean(self):
del form.errors['display_from']
del form.errors['display_to']
display_from = "value here"
display_to = "value here"
self.cleaned_data['display_from'] = display_from
self.cleaned_data['display_to'] = display_to
return self.cleaned_data
There are a few other approaches.
For very simple cases you can get away with using a hidden field in your form.
You can also override the clean_ method to return the appropriate value. These generated values may be passed to the ModelForm constructor:
form = SomeModelForm(request.POST or None, initial={"option": "10"})

You would need to add those fields to your exclude tuple:
class YourForm(forms.ModelForm):
class Meta:
exclude = ('display_from', 'display_to',)
model = YourModel
def clean(self):
. . .
Then you won't get the required field error and can override the validation as you wish.

Order of Serializer Validation in Django REST Framework

Situation
While working with validation in the Django REST Framework's ModelSerializer, I have noticed that the Meta.model fields are always validated, even when it does not necessarily make sense to do so. Take the following example for a User model's serialization:
I have an endpoint that creates a user. As such, there is a password field and a confirm_password field. If the two fields do not match, the user cannot be created. Likewise, if the requested username already exists, the user cannot be created.
The user POSTs improper values for each of the fields mentioned above
An implementation of validate has been made in the serializer (see below), catching the non-matching password and confirm_password fields
Implementation of validate:
def validate(self, data):
if data['password'] != data.pop('confirm_password'):
raise serializers.ValidationError("Passwords do not match")
return data
Problem
Even when the ValidationError is raised by validate, the ModelSerializer still queries the database to check to see if the username is already in use. This is evident in the error-list that gets returned from the endpoint; both the model and non-field errors are present.
Consequently, I would like to know how to prevent model validation until after non-field validation has finished, saving me a call to my database.
Attempt at solution
I have been trying to go through the DRF's source to figure out where this is happening, but I have been unsuccessful in locating what I need to override in order to get this to work.

Since most likely your username field has unique=True set, Django REST Framework automatically adds a validator that checks to make sure the new username is unique. You can actually confirm this by doing repr(serializer()), which will show you all of the automatically generated fields, which includes the validators.
Validation is run in a specific, undocumented order
Field deserialization called (serializer.to_internal_value and field.run_validators)
serializer.validate_[field] is called for each field
Serializer-level validators are called (serializer.run_validation followed by serializer.run_validators)
serializer.validate is called
So the problem that you are seeing is that the field-level validation is called before your serializer-level validation. While I wouldn't recommend it, you can remove the field-level validator by setting extra_kwargs in your serilalizer's meta.
class Meta:
extra_kwargs = {
"username": {
"validators": [],
},
}
You will need to re-implement the unique check in your own validation though, along with any additional validators that have been automatically generated.

I was also trying to understand how the control flows during serializer validation and after carefully going through the source code of djangorestframework-3.10.3 I came up with below request flow diagram. I have described the flow and what happens in the flow to the best of my understanding without going into too much detail as it can be looked up from source.
Ignore the incomplete method signatures. Only focusing on what methods are called on what classes.
Assuming you have an overridden is_valid method on your serializer class (MySerializer(serializers.Serializer)) when you call my_serializer.is_valid() the following takes place.
MySerializer.is_valid() is executed.
Assuming you are calling the super class (BaseSerializer) is_valid method (like: super(MySerializer, self).is_valid(raise_exception) in your MySerializer.is_valid() method, that will be called.
Now since MySerializer is extending serializers.Serializer, the run_validation() method from serializer.Serializers is called. This is validating only the data dict the first. So we haven't yet started field level validations.
Then the validate_empty_values from fields.Field gets called. This again happens on the entire data and not a single field.
Then the Serializer.to_internal_method is called.
Now we loop over each fields defined on the serializer. And for each field, first we call the field.run_validation() method. If the field has overridden the Field.run_validation() method then that will be called first. In case of a CharField it is overridden and calls the run_validation method of Field base class. Step 6-2 in the figure.
On that field we again call the Field.validate_empty_values()
The to_internal_value of the type of field is called next.
Now there is a call to the Field.run_validators() method. I presume this is where the additional validators that we add on the field by specifying the validators = [] field option get executed one by one
Once all this is done, we are back to the Serializer.to_internal_value() method. Now remember that we are doing the above for each field within that for loop. Now the custom field validators you wrote in your serializer (methods like validate_field_name) are run. If an exception occurred in any of the previous steps, your custom validators wont run.
read_only_defaults()
update validate data with defaults I think
run object level validators. I think the validate() method on your object is run here.

I don't believe the above solutions work any more. In my case, my model has fields 'first_name' and 'last_name', but the API will only receive 'name'.
Setting 'extra_kwargs' and 'validators' in the Meta class seems to have no effect, first_name and last_name are allways deemed required, and validators are always called. I can't overload the first_name/last_name character fields with
anotherrepfor_first_name = serializers.CharField(source=first_name, required=False)
as the names make sense. After many hours of frustration, I found the only way I could override the validators with a ModelSerializer instance was to override the class initializer as follows (forgive the incorrect indentation):
class ContactSerializer(serializers.ModelSerializer):
name = serializers.CharField(required=True)
class Meta:
model = Contact
fields = [ 'name', 'first_name', 'last_name', 'email', 'phone', 'question' ]
def __init__(self, *args, **kwargs):
self.fields['first_name'] = serializers.CharField(required=False, allow_null=True, allow_blank=True)
self.fields['last_name'] = serializers.CharField(required=False, allow_null=True, allow_blank=True)
return super(ContactSerializer, self).__init__(*args, **kwargs)
def create(self, validated_data):
return Contact.objects.create()
def validate(self, data):
"""
Remove name after getting first_name, last_name
"""
missing = []
for k in ['name', 'email', 'question']:
if k not in self.fields:
missing.append(k)
if len(missing):
raise serializers.ValidationError("Ooops! The following fields are required: %s" % ','.join(missing))
from nameparser import HumanName
names = HumanName(data['name'])
names.capitalize()
data['last_name'] = names.last
if re.search(r'\w+', names.middle):
data['first_name'] = ' '.join([names.first, names.middle])
else:
data['first_name'] = names.first
del(data['name'])
return data
Now the doc says that allowing blank and null with character fields is a no no, but this is a serializer, not a model, and as the API gets called by all kinds of cowboys, I need to cover my bases.

Here's the approach that worked for me.
Use a sentinel error type that gets caught in an overridden view function
The sentinel is raised from the custom serializer
The sentinel error type:
from django.core.exceptions import ValidationError
class CustomValidationErrors(ValidationError):
""" custom validation error for the api view to catch the status code """
And in the serializer we override errors, _errors, validated_data, and _validated_data, as well as is_valid:
class CustomSerializer(serializers.ModelSerializer):
# fields that usually run validation before parent serializer validation
child_field1 = Child1Serializer()
child_field2 = Child2Serializer()
# override DRF fields
errors = {}
_errors = None
validated_data = {}
_validated_data = []
def is_valid(self, *args, **kwargs):
# override drf.serializers.Serializer.is_valid
# and raise CustomValidationErrors from parent validate
self.validate(self.initial_data)
return not bool(self.errors)
def validate(self, attrs):
self._errors = {}
if len(attrs.get("child_field1", {}).get("name", "")) > 100:
self._errors.update({"child_field1": {"name": "child 1 name > 100"}})
if len(attrs.get("child_field2", {}).get("description", "")) > 1000:
self._errors.update({"child_field2.description": "child 2 description > 100"})
if len(self._errors):
# set the overriden DRF values
self.errors = self._errors
# raise the sentinel error type
raise CustomValidationErrors(self._errors)
# set the overriden DRF serializer values
self._errors = None
self.validated_data = attrs
self._validated_data = [[k, v] for k, v in attrs.items()]
return attrs
class Meta:
model = CustomModel
And in the view we can override the default method, and catch the sentinel error type:
class CustomSerializerView(ListCreateAPIView):
serializer_class = CustomeSerializer
def post(self, *args, **kwargs):
try:
# if this fails for any exception
# other than CustomValidationErrors
# it will return the default error
return super().post(*args, **kwargs)
except CustomValidationErrors as e:
############
# returns a 400 with the following
# {"child_field1":
# [[{"name": "child 1 name > 100"}]],
# "child_field2.description":
# [["child 2 description > 100"]]
# }
############
return Response(e.error_dict, status=400)
drf version:
djangorestframework==3.11.0

What is the difference between a Model field type and a similar validator in Django?

What is the difference between, say, a EmailField and a validator_email? And is it a bad idea to use both?
Or for those who perfer code
import django.db import models
email = models.EmailField()
vs
import django.db import models
email = models.CharField( max_length=75, validators = validate_email )
From the doc it seems like you could also use validators inside forms but if you already specify a validation restriction inside models.py, then you don't need specify again in the forms, right? So it seems better to me to take care of all of the restriction inside models.py.

I suppose the difference is very little, but then you would be violating the DRY principal, which you probably shouldn't do, unless you have a good reason to do it.
If you go to the code base:
#django.db.fields.__init__.py
class EmailField(CharField):
default_validators = [validators.validate_email]
description = _("E-mail address")
def __init__(self, *args, **kwargs):
kwargs['max_length'] = kwargs.get('max_length', 75)
CharField.__init__(self, *args, **kwargs)
def formfield(self, **kwargs):
# As with CharField, this will cause email validation to be performed
# twice.
defaults = {
'form_class': forms.EmailField,
}
defaults.update(kwargs)
return super(EmailField, self).formfield(**defaults)
As you can see, the model inherits from Charfield, so you lose nothing by using emailfield, where appropriate. Furthermore, the default validator is validate_email. Additionally you get the description variable already defined for you. Lastly, on the backend it is already setting max_length for you at '75'. You could of course override this easily enough by defining a max_length in the same way you would when creating a CharField.
You can see formfields() is returning forms.EmailField from django.forms.
Looking at that, you can see:
#django.forms.fields.py
class EmailField(CharField):
default_error_messages = {
'invalid': _(u'Enter a valid e-mail address.'),
}
default_validators = [validators.validate_email]
def clean(self, value):
value = self.to_python(value).strip()
return super(EmailField, self).clean(value)
However, you would lose any default values that using the EmailField might provide, such as the "correct" error message and the custom clean() method.
In the end, while it looks small, actually a good bit of work has already been done for you. So, in general, you shouldn't violate the DRY principal unless you have a good reason to do so.
Edit:
Regarding the second question, you want the form to validate against whatever criteria you are concerned about, so when you call form.is_valid() it returns True / False when it should and generates the appropriate failure message. Otherwise, is_valid() would validate True, and when you model goes to save, it would fail silently, which would be very hard to track down.

Django update instead of insert a new record

I having a weird issue with Django 1.3. When I try to add a new Activity using the ActivityForm, the Activity is often updated even if I explicitly create a new Activity object with not Id.
In addition, when the init of my ActivityForm class has an explicit trip_id parameter, the UI displays "Select a valid choice. That choice is not one of the available choices." for the field location (even though the selection shown is valid). However, if I make this an optional parameter and I pop it from kwargs, I don't see this issue.
Can someone please take a look at the code below and see what I am doing wrong?
forms.py
class DestinationMultipleChoiceField(ModelMultipleChoiceField):
def label_from_instance(self, obj):
return obj.city_name
class ActivityForm(forms.Form):
description = forms.CharField(max_length=100, required=True)
location = DestinationChoiceField(queryset = Visit.objects.none(), empty_label=None, required=True)
def __init__(self, trip_id = None, *args, **kwargs):
super(ActivityForm, self).__init__(*args, **kwargs)
if trip_id is not None:
self.fields['location'].queryset = Visit.objects.filter(trip=trip_id).all().select_related('destination')
# A new Activity() is created if nothing is provided
def save(self, activity = Activity()):
if not self.is_valid():
raise forms.ValidationError('ActivityForm was not validated first before trying to call save().')
activity.visit = self.cleaned_data['location']
activity.description = self.cleaned_data['description']
activity.added_by = User.objects.get(pk=1)
activity.save()
views.py
def add_activity(request, trip_id = None, template_name = 'edit-form.html'):
if request.POST:
form = ActivityForm(trip_id, request.POST)
if form.is_valid():
form.save()
return HttpResponseRedirect(reverse('views.list_trip'))
else:
form = ActivityForm(trip_id)
return render_to_response(template_name, {
'page_title': 'Add',
'form': form,
}, context_instance=RequestContext(request))

Look at this line:
def save(self, activity = Activity()):
In Python, default parameters are evaluated once; thus, when the save method is called the first time, a new Activity will be created, but subsequent calls will use that Activity.
Change save to something like this:
def save(self, activity=None):
if activity is None:
activity = Activity()
# Rest of the method
Then, a new Activity will be created on every call (if one is not supplied as an argument).

Model instances are mutable. As such they should never be used as default arguments to a method, as you do in save. There's a reason why you've never seen that in any documentation or examples.
As noted in very many other questions on this site, default arguments are evaluated on definition, not execution. So each call to save with no activity argument will use the same Activity that was originally defined.
However, I don't understand why you're not using a ModelForm, which does most of this for you in any case.

Does Django Have a Way to Auto-Sort Model Fields?

So basically, I've got a rather large Django project going. It's a private web portal that allows users to manage various phone-related tasks.
Several pages of the portal provide a listing of Model objects to users, and list all of their attributes in a HTML table (so that users can visually look through a list of these items).
The problem I'm having is: I cannot find a Django-ish or pythonic way to handle the sorting of these Model objects by field name. As an example of what I'm talking about, here is one of my views which lists all Partyline Model objects:
def list_partylines(request):
"""
List all `Partyline`s that we own.
"""
# Figure out which sort term to use.
sort_field = request.REQUEST.get('sortby', 'did').strip()
if sort_field.startswith('-'):
search = sort_field[1:]
sort_toggle = ''
else:
search = sort_field
sort_toggle = '-'
# Check to see if the sort term is valid.
if not (search in Partyline._meta.get_all_field_names()):
sort_field = 'did'
if is_user_type(request.user, ['admin']):
partylines = Partyline.objects.all().order_by(sort_field)
else:
partylines = get_my_partylines(request.user, sort_field)
variables = RequestContext(request, {
'partylines': partylines,
'sort_toggle': sort_toggle
})
return render_to_response('portal/partylines/list.html', variables)
The sorting code basically allows users to specify a /url/?sortby=model_field_name parameter which will then return a sorted listing of objects whenever users click on the HTML table name displayed on the page.
Since I have various views in various apps which all show a listing of Model objects, and require sorting, I'm wondering if there is a generic way to do this sorting so that I don't have to?
I'm sorry if this question is a bit unclear, I'm struggling to find the right way to phrase this question.
Thanks.

The way that I'd look at doing this is through a custom QuerySet. In your model, you can define the class QuerySet and add your sorting there. In order to maintain all the logic in the model object, I'd also move the contents of get_my_partylines into the QuerySet, too.
## This class is used to replicate QuerySet methods into a manager.
## This way: Partyline.objects.for_user(foo) works the same as
## Partyline.objects.filter(date=today).for_user(foo)
class CustomQuerySetManager(models.Manager):
def get_query_set(self):
return self.model.QuerySet(self.model)
def __getattr__(self, attr, *args):
try:
return getattr(self.__class__, attr, *args)
except AttributeError:
return getattr(self.get_query_set(), attr, *args)
class Partyline(models.Model):
## Define fields, blah blah.
objects = CustomQuerySetManager()
class QuerySet(QuerySet):
def sort_for_request(self, request):
sort_field = request.REQUEST.get('sortby', 'did').strip()
reverse_order = False
if sort_field.startswith('-'):
search = sort_field[1:]
else:
search = sort_field
reverse_order = True
# Check to see if the sort term is valid.
if not (search in Partyline._meta.get_all_field_names()):
sort_field = 'did'
partylines = self.all().order_by(sort_field)
if reverse_order:
partylines.reverse()
return partylines
def for_user(self, user):
if is_user_type(request.user, ['admin']):
return self.all()
else:
## Code from get_my_partylines goes here.
return self.all() ## Temporary.
views.py:
def list_partylines(request):
"""
List all `Partyline`s that we own.
"""
partylines = Partylines.objects.for_user(request.user).sort_for_request(request)

There's a great example of how this is done in a generic way in django.contrib.admin.views.main.ChangeList although that does much more than sorting you can browse it's code for some hints and ideas. You may also want to look at django.contrib.admin.options.ModelAdmin the changelist method in particular to get more context.