We Keep Coding

How to automatically fill-in model fields in Django rest_framework serializer?

Let's assume I have a model like this:
class Data(models.Model):
a = models.CharField()
b = models.CharField()
c = models.IntegerField()
I would like to setup a serializer in such a way that it automatically fills in field c and it is not required for a POST. I tried to overwrite the create function of the serializer, but it doesn't work:
class DataSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Data
fields = ('a', 'b')
def create(self, validated_data, **kwargs):
Data.objects.c = 5
return Data.objects.create(**validated_data)
However, if I try this, I end up with an IntegrityError: NOT NULL constraint failed: model_data.c. What is the syntax that I have to use here?

EDIT: Updated formatting.
The reason you're getting the error because field c is not set to null = True - as such an error is raised at the validation stage even before the serializer hits the create method.
Bear in mind that the process goes like this:
Submit serializer data
field-level validation happens - this includes checks for null integrity, min/max length etc and also any custom field validations defined in def validate_<field_name>
object-level validation happens - this calls the def validate method
validated data is passed to the save method, depending on how you designed the serializer - it will save the instance, or route the data to either create or update
All of the info regarding this can be found in Django's and DRF's docs.
A few things to consider:
are you setting a global default for that field? If so, set the default in your models - c = models.IntegerField(default=a_number_or_a_callable_that_returns_an_integer)
do you intend to display the field? If so, include c in your fields and add one more Meta attribute - read_only_fields = ('c',)
If it's neither of the above, you might want to override the validate_c method
Apologies for the poor formatting, typing it on my phone - will update once I get to a computer

In your code Data.objects.c = 5 does nothing.
If you want to set this value yourself use validated_data['c'] = 5 or Data.objects.create(c=5, **validated_data) (just not both at the same time).

Rather than doing this in the serializer, there are hooks in the generic views that allow you to pass values to the serializer. So in your case you might have:
class DataViewSet(ModelViewSet):
# ...
def perform_create(self, serializer):
serializer.save(c=5)
See the "Save and deletion hooks" section here

Validating a Django model field based on another field's value?

I have a Django app with models accessible by both Django REST Framework and a regular form interface. The form interface has some validation checks before saving changes to the model, but not using any special Django framework, just a simple local change in the view.
I'd like to apply the same validation to forms and REST calls, so I want to move my validation into the model. I can see how to do that for simple cases using the validators field of the Field, but in one case I have a name/type/value model where the acceptable values for 'value' change depending on which type is selected. The validator doesn't get sent any information about the model that the field is in, so it doesn't have access to other fields.
How can I perform this validation, without having essentially the same code in a serializer for DRF and my POST view for the form?

I dug around codebase of drf a little bit. You can get values of all fields using following approach. Doing so, you can throw serialization error as
{'my_field':'error message} instead of {'non_field_error':'error message'}.
def validate_myfield(self, value):
data = self.get_initial() # data for all the fields
#do your validation
However, if you wish to do it for ListSerializer, i.e for serializer = serializer_class(many=True), this won't work. You will get list of empty values.
In that scenario, you could write your validations in def validate function and to avoid non_field_errors in your serialization error, you can raise ValidationError with error message as a dictionary instead of string.
def validate(self, data):
# do your validation
raise serializers.ValidationError({"your_field": "error_message"})

The validation per-field doesn't get sent any information about other fields, when it is defined like this:
def validate_myfield(self, value):
...
However, if you have a method defined like this:
def validate(self, data):
...
Then you get all the data in a dict, and you can do cross-field validation.

You can use the required package for your cross-field validation. It allows you to express your validation rules declaratively in python. You would have something like this with DRF:
class MySerializer(serializers.Serializer):
REQUIREMENTS = (
Requires("end_date", "start_date") +
Requires("end_date", R("end_date") > R("start_date")) +
Requires("end_date", R("end_date") < today.date() + one_year) +
Requires("start_date", R("start_date") < today.date() + one_year)
)
start_date = serializers.DateField(required=False, null=True, blank=True)
end_date = serializers.DateField(required=False, null=True, blank=True)
def validate(self, data):
self.REQUIREMENTS.validate(data) # handle validation error
You could put the REQUIREMENTS on your Model and have both your DRF and Django Form validate your data using it.
Here is a blog post explaining more

Passing values to required fields that do not come from inputs in Django ModelForm

I have a ModelForm. There are several fields such as:
['display_from', 'display_to', 'position', 'image']
However, display_from and display_to are not in my template. I want to override the validation and assign those fields some values before any validation errors appear. I have done something like this:
def clean(self):
position = self.cleaned_data.get('position')
image = self.cleaned_data.get('image')
display_from = "value here"
display_to = "value here"
self.cleaned_data['display_from'] = display_from
self.cleaned_data['display_to'] = display_to
return self.cleaned_data
When I print the cleaned_data, I can see the values in the terminal. However, when I display the error messages on the page, it tells me that display_from and display_to are required.
Perhaps this should not be done in the clean() method?
I tried to use empty hidden inputs as well, but it didn't work. Any ideas?
EDIT:
Excluding the fields does not help, it returns an IntegrityError message:
null value in column "display_from" violates not-null constraint

Keep in mind the clean() function is called after the various clean_ functions for each field on the model (docs here).
So those individual field clean functions will be raising ValidationErrors that get pinned to the form.errors dict before your clean() function comes in and adds the default values to the cleaned_data dict.
You could also carry on with the approach you already started with. Just remove the errors from the errors dict.
def clean(self):
del form.errors['display_from']
del form.errors['display_to']
display_from = "value here"
display_to = "value here"
self.cleaned_data['display_from'] = display_from
self.cleaned_data['display_to'] = display_to
return self.cleaned_data
There are a few other approaches.
For very simple cases you can get away with using a hidden field in your form.
You can also override the clean_ method to return the appropriate value. These generated values may be passed to the ModelForm constructor:
form = SomeModelForm(request.POST or None, initial={"option": "10"})

You would need to add those fields to your exclude tuple:
class YourForm(forms.ModelForm):
class Meta:
exclude = ('display_from', 'display_to',)
model = YourModel
def clean(self):
. . .
Then you won't get the required field error and can override the validation as you wish.

Order of Serializer Validation in Django REST Framework

Situation
While working with validation in the Django REST Framework's ModelSerializer, I have noticed that the Meta.model fields are always validated, even when it does not necessarily make sense to do so. Take the following example for a User model's serialization:
I have an endpoint that creates a user. As such, there is a password field and a confirm_password field. If the two fields do not match, the user cannot be created. Likewise, if the requested username already exists, the user cannot be created.
The user POSTs improper values for each of the fields mentioned above
An implementation of validate has been made in the serializer (see below), catching the non-matching password and confirm_password fields
Implementation of validate:
def validate(self, data):
if data['password'] != data.pop('confirm_password'):
raise serializers.ValidationError("Passwords do not match")
return data
Problem
Even when the ValidationError is raised by validate, the ModelSerializer still queries the database to check to see if the username is already in use. This is evident in the error-list that gets returned from the endpoint; both the model and non-field errors are present.
Consequently, I would like to know how to prevent model validation until after non-field validation has finished, saving me a call to my database.
Attempt at solution
I have been trying to go through the DRF's source to figure out where this is happening, but I have been unsuccessful in locating what I need to override in order to get this to work.

Since most likely your username field has unique=True set, Django REST Framework automatically adds a validator that checks to make sure the new username is unique. You can actually confirm this by doing repr(serializer()), which will show you all of the automatically generated fields, which includes the validators.
Validation is run in a specific, undocumented order
Field deserialization called (serializer.to_internal_value and field.run_validators)
serializer.validate_[field] is called for each field
Serializer-level validators are called (serializer.run_validation followed by serializer.run_validators)
serializer.validate is called
So the problem that you are seeing is that the field-level validation is called before your serializer-level validation. While I wouldn't recommend it, you can remove the field-level validator by setting extra_kwargs in your serilalizer's meta.
class Meta:
extra_kwargs = {
"username": {
"validators": [],
},
}
You will need to re-implement the unique check in your own validation though, along with any additional validators that have been automatically generated.

I was also trying to understand how the control flows during serializer validation and after carefully going through the source code of djangorestframework-3.10.3 I came up with below request flow diagram. I have described the flow and what happens in the flow to the best of my understanding without going into too much detail as it can be looked up from source.
Ignore the incomplete method signatures. Only focusing on what methods are called on what classes.
Assuming you have an overridden is_valid method on your serializer class (MySerializer(serializers.Serializer)) when you call my_serializer.is_valid() the following takes place.
MySerializer.is_valid() is executed.
Assuming you are calling the super class (BaseSerializer) is_valid method (like: super(MySerializer, self).is_valid(raise_exception) in your MySerializer.is_valid() method, that will be called.
Now since MySerializer is extending serializers.Serializer, the run_validation() method from serializer.Serializers is called. This is validating only the data dict the first. So we haven't yet started field level validations.
Then the validate_empty_values from fields.Field gets called. This again happens on the entire data and not a single field.
Then the Serializer.to_internal_method is called.
Now we loop over each fields defined on the serializer. And for each field, first we call the field.run_validation() method. If the field has overridden the Field.run_validation() method then that will be called first. In case of a CharField it is overridden and calls the run_validation method of Field base class. Step 6-2 in the figure.
On that field we again call the Field.validate_empty_values()
The to_internal_value of the type of field is called next.
Now there is a call to the Field.run_validators() method. I presume this is where the additional validators that we add on the field by specifying the validators = [] field option get executed one by one
Once all this is done, we are back to the Serializer.to_internal_value() method. Now remember that we are doing the above for each field within that for loop. Now the custom field validators you wrote in your serializer (methods like validate_field_name) are run. If an exception occurred in any of the previous steps, your custom validators wont run.
read_only_defaults()
update validate data with defaults I think
run object level validators. I think the validate() method on your object is run here.

I don't believe the above solutions work any more. In my case, my model has fields 'first_name' and 'last_name', but the API will only receive 'name'.
Setting 'extra_kwargs' and 'validators' in the Meta class seems to have no effect, first_name and last_name are allways deemed required, and validators are always called. I can't overload the first_name/last_name character fields with
anotherrepfor_first_name = serializers.CharField(source=first_name, required=False)
as the names make sense. After many hours of frustration, I found the only way I could override the validators with a ModelSerializer instance was to override the class initializer as follows (forgive the incorrect indentation):
class ContactSerializer(serializers.ModelSerializer):
name = serializers.CharField(required=True)
class Meta:
model = Contact
fields = [ 'name', 'first_name', 'last_name', 'email', 'phone', 'question' ]
def __init__(self, *args, **kwargs):
self.fields['first_name'] = serializers.CharField(required=False, allow_null=True, allow_blank=True)
self.fields['last_name'] = serializers.CharField(required=False, allow_null=True, allow_blank=True)
return super(ContactSerializer, self).__init__(*args, **kwargs)
def create(self, validated_data):
return Contact.objects.create()
def validate(self, data):
"""
Remove name after getting first_name, last_name
"""
missing = []
for k in ['name', 'email', 'question']:
if k not in self.fields:
missing.append(k)
if len(missing):
raise serializers.ValidationError("Ooops! The following fields are required: %s" % ','.join(missing))
from nameparser import HumanName
names = HumanName(data['name'])
names.capitalize()
data['last_name'] = names.last
if re.search(r'\w+', names.middle):
data['first_name'] = ' '.join([names.first, names.middle])
else:
data['first_name'] = names.first
del(data['name'])
return data
Now the doc says that allowing blank and null with character fields is a no no, but this is a serializer, not a model, and as the API gets called by all kinds of cowboys, I need to cover my bases.

Here's the approach that worked for me.
Use a sentinel error type that gets caught in an overridden view function
The sentinel is raised from the custom serializer
The sentinel error type:
from django.core.exceptions import ValidationError
class CustomValidationErrors(ValidationError):
""" custom validation error for the api view to catch the status code """
And in the serializer we override errors, _errors, validated_data, and _validated_data, as well as is_valid:
class CustomSerializer(serializers.ModelSerializer):
# fields that usually run validation before parent serializer validation
child_field1 = Child1Serializer()
child_field2 = Child2Serializer()
# override DRF fields
errors = {}
_errors = None
validated_data = {}
_validated_data = []
def is_valid(self, *args, **kwargs):
# override drf.serializers.Serializer.is_valid
# and raise CustomValidationErrors from parent validate
self.validate(self.initial_data)
return not bool(self.errors)
def validate(self, attrs):
self._errors = {}
if len(attrs.get("child_field1", {}).get("name", "")) > 100:
self._errors.update({"child_field1": {"name": "child 1 name > 100"}})
if len(attrs.get("child_field2", {}).get("description", "")) > 1000:
self._errors.update({"child_field2.description": "child 2 description > 100"})
if len(self._errors):
# set the overriden DRF values
self.errors = self._errors
# raise the sentinel error type
raise CustomValidationErrors(self._errors)
# set the overriden DRF serializer values
self._errors = None
self.validated_data = attrs
self._validated_data = [[k, v] for k, v in attrs.items()]
return attrs
class Meta:
model = CustomModel
And in the view we can override the default method, and catch the sentinel error type:
class CustomSerializerView(ListCreateAPIView):
serializer_class = CustomeSerializer
def post(self, *args, **kwargs):
try:
# if this fails for any exception
# other than CustomValidationErrors
# it will return the default error
return super().post(*args, **kwargs)
except CustomValidationErrors as e:
############
# returns a 400 with the following
# {"child_field1":
# [[{"name": "child 1 name > 100"}]],
# "child_field2.description":
# [["child 2 description > 100"]]
# }
############
return Response(e.error_dict, status=400)
drf version:
djangorestframework==3.11.0

What is the difference between a Model field type and a similar validator in Django?

What is the difference between, say, a EmailField and a validator_email? And is it a bad idea to use both?
Or for those who perfer code
import django.db import models
email = models.EmailField()
vs
import django.db import models
email = models.CharField( max_length=75, validators = validate_email )
From the doc it seems like you could also use validators inside forms but if you already specify a validation restriction inside models.py, then you don't need specify again in the forms, right? So it seems better to me to take care of all of the restriction inside models.py.

I suppose the difference is very little, but then you would be violating the DRY principal, which you probably shouldn't do, unless you have a good reason to do it.
If you go to the code base:
#django.db.fields.__init__.py
class EmailField(CharField):
default_validators = [validators.validate_email]
description = _("E-mail address")
def __init__(self, *args, **kwargs):
kwargs['max_length'] = kwargs.get('max_length', 75)
CharField.__init__(self, *args, **kwargs)
def formfield(self, **kwargs):
# As with CharField, this will cause email validation to be performed
# twice.
defaults = {
'form_class': forms.EmailField,
}
defaults.update(kwargs)
return super(EmailField, self).formfield(**defaults)
As you can see, the model inherits from Charfield, so you lose nothing by using emailfield, where appropriate. Furthermore, the default validator is validate_email. Additionally you get the description variable already defined for you. Lastly, on the backend it is already setting max_length for you at '75'. You could of course override this easily enough by defining a max_length in the same way you would when creating a CharField.
You can see formfields() is returning forms.EmailField from django.forms.
Looking at that, you can see:
#django.forms.fields.py
class EmailField(CharField):
default_error_messages = {
'invalid': _(u'Enter a valid e-mail address.'),
}
default_validators = [validators.validate_email]
def clean(self, value):
value = self.to_python(value).strip()
return super(EmailField, self).clean(value)
However, you would lose any default values that using the EmailField might provide, such as the "correct" error message and the custom clean() method.
In the end, while it looks small, actually a good bit of work has already been done for you. So, in general, you shouldn't violate the DRY principal unless you have a good reason to do so.
Edit:
Regarding the second question, you want the form to validate against whatever criteria you are concerned about, so when you call form.is_valid() it returns True / False when it should and generates the appropriate failure message. Otherwise, is_valid() would validate True, and when you model goes to save, it would fail silently, which would be very hard to track down.