We Keep Coding

Django Rest Framework ListSerializer Partial Update

I'm writing a serializer to provide multiple partial updates to a django model. I'm following the example implementation that appears in the DRF api guide, reproduced below and linked here: https://www.django-rest-framework.org/api-guide/serializers/#customizing-multiple-update.
The following was retrieved from django-rest-framework documentation:
serializer.py
class BookListSerializer(serializers.ListSerializer):
def update(self, instance, validated_data):
# Maps for id->instance and id->data item.
book_mapping = {book.id: book for book in instance}
data_mapping = {item['id']: item for item in validated_data}
# Perform creations and updates.
ret = []
for book_id, data in data_mapping.items():
book = book_mapping.get(book_id, None)
if book is None:
ret.append(self.child.create(data))
else:
ret.append(self.child.update(book, data))
# Perform deletions.
for book_id, book in book_mapping.items():
if book_id not in data_mapping:
book.delete()
return ret
class BookSerializer(serializers.Serializer):
# We need to identify elements in the list using their primary key,
# so use a writable field here, rather than the default which would be read-only.
id = serializers.IntegerField()
...
class Meta:
list_serializer_class = BookListSerializer
In my code I'm getting a NotImplementedError('update() must be implemented.') in my views.py when .save() is called on the serializer that is returned.
My understanding is that the ListsSerializer overrides the .update(), so could anyone help explain why I'm getting the NotImpletmentedError?
views.py
elif request.method == 'PATCH':
data = JSONParser().parse(request)
books = Book.objects.all()
# both partial and many set to True
serializer = BookSerializer(books, data=data, partial=True, many=True)
if serializer.is_valid():
serializer.save()
return JsonResponse(serializer.data)
return JsonResponse(serializer.errors, status=400)

With the help of #luistm, I managed to solve this. Continuing with the DRF example above my implementation of the update() override in the bookSerializer class was as below.
serializer.py
class BookSerializer(serializers.Serializer):
# We need to identify elements in the list using their primary key,
# so use a writable field here, rather than the default which would be read-only.
id = serializers.IntegerField()
...
class Meta:
list_serializer_class = BookListSerializer
def update(self, instance, validated_data):
"""update the page number of the book and save"""
instance.page = validated_data.get('page', instance.page)
instance.save()
return instance

Multiple file upload DRF

I have a requirement which I would like to allow multiple files to be uploaded within the same post request to create an object. I currently have a method of doing this, but after looking at some other examples it doesn't appear to be intended way to do it.
models.py
class Analyzer(models.Model):
name = models.CharField(max_length=100, editable=False, unique=True)
class Atomic(models.Model):
name = models.CharField(max_length=20, unique=True)
class Submission(models.Model):
class Meta:
ordering = ['-updated_at']
issued_at = models.DateTimeField(auto_now_add=True, editable=False)
completed = models.BooleanField(default=False)
analyzers = models.ManyToManyField(Analyzer, related_name='submissions')
atomic = models.ForeignKey(Atomic, verbose_name='Atomic datatype', related_name='submission', on_delete=models.CASCADE)
class BinaryFile(models.Model):
class Meta:
verbose_name = 'Binary file'
verbose_name_plural = 'Binary files'
def __str__(self):
return self.file.name
submission = models.ForeignKey(Submission, on_delete=models.CASCADE, related_name='binary_files')
file = models.FileField(upload_to='uploads/binary/')
serializers.py
class BinaryFileSerializer(serializers.ModelSerializer):
class Meta:
model = models.BinaryFile
fields = '__all__'
class SubmissionCreateSerializer(serializers.ModelSerializer):
class Meta:
model = models.Submission
fields = ['id', 'completed', 'atomic', 'analyzers', 'binary_files']
id = serializers.ReadOnlyField()
completed = serializers.ReadOnlyField()
atomic = serializers.PrimaryKeyRelatedField(many=False, queryset=models.Atomic.objects.all()
analyzers = serializers.PrimaryKeyRelatedField(many=True, queryset=models.Analyzer.objects.all()
binary_files = BinaryFileSerializer(required=True, many=True)
def validate(self, data):
# # I dont really like manually taking invalidated input!!
data['binary_files'] = self.initial_data.getlist('binary_files')
return data
def create(self, validated_data):
submission = models.Submission.objects.create(
atomic=validated_data['atomic']
)
submission.analyzers.set(validated_data['analyzers'])
# # Serialize the files - this seems too late to be doing this!
for file in validated_data['binary_files']:
binary_file = BinaryFileSerializer(
data={'file': file, 'submission': submission.id}
)
if binary_file.is_valid():
binary_file.save()
return submission
Main question: While the above works, the child serializer (BinaryFileSerializer) doesn't get called until I explicitly call it in create(), which is after the validation should have occurred. Why does this never get called?
I also don't like the fact I have to manually do a self.initial_data.getlist('binary_files') and manually add it to data - this should have already been added and validated, no?
My thought is that as I defined binary_files = BinaryFileSerializer, this serializer should be called to validate that particular fields input?
FYI, I'm using the following to test POST uploads:
curl -F "binary_files=#file2.txt" -F "binary_files=#file2.txt" -F "atomic=7" -F "analyzers=12" -H "Accept: application/json; indent=4" http://127.0.0.1:8000/api/submit/
TIA!
Update: The question is now, if a validate() funciton is added to the BinaryFileSerializer, why does it not get called?

Possible duplicate --- Django REST: Uploading and serializing multiple images.
From the DRF Writable Nested Serializer doc,
By default nested serializers are read-only. If you want to support write-operations to a nested serializer field you'll need to create create() and/or update() methods in order to explicitly specify how the child relationships should be saved.
From this, it's clear that the child serializer (BinaryFileSerializer) won't call its own create() method unless explicitly called.
The aim of your HTTP POST request is to create new Submission instance (and BinaryFile instance). The creation process undergoes in the create() method of the SubmissionCreateSerializer serializer, which is you'd overridden. So, it will act/execute as per your code.
UPDATE-1
Things to remember
1. AFAIK, we can't send nested multipart/form-data
2. Here I'm only trying to implementing the least case scenario
3. I'm tested this solution with POSTMAN rest api test tool.
4. This method may be complex (until we found a better one).
5. Assuming your view class is subclass of ModelViewSet class
What I'm going to do?
1. Since we can't send the files/data in a nested fashion, we have to send it flat mode.
image-1
2. Override the __init__() method of the SubmissionSerializer serializer and dynamically add as much FileField() attribute according to the request.FILES data.We could somehow use ListSerializer or ListField here. Unfortunately I couldn't find out a way :(
# init method of "SubmissionSerializer"
def __init__(self, *args, **kwargs):
file_fields = kwargs.pop('file_fields', None)
super().__init__(*args, **kwargs)
if file_fields:
field_update_dict = {field: serializers.FileField(required=False, write_only=True) for field in file_fields}
self.fields.update(**field_update_dict)
So, what id file_fields here? Since the form-data is a key-value pair, every file data must be associated with a key. Here in image-1, you could see file_1 and file_2.
3. Now we need to pass the file_fields values from the view. Since this operation is creating new instance, we need to override the create() method of the API class.
# complete view code
from rest_framework import status
from rest_framework import viewsets
class SubmissionAPI(viewsets.ModelViewSet):
queryset = Submission.objects.all()
serializer_class = SubmissionSerializer
def create(self, request, *args, **kwargs):
# main thing starts
file_fields = list(request.FILES.keys()) # list to be passed to the serializer
serializer = self.get_serializer(data=request.data, file_fields=file_fields)
# main thing ends
serializer.is_valid(raise_exception=True)
self.perform_create(serializer)
headers = self.get_success_headers(serializer.data)
return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
4. Now, all values will be serialized properly. It's time to override the create() method of the SubmissionSerializer() to map the relations
def create(self, validated_data):
from django.core.files.uploadedfile import InMemoryUploadedFile
validated_data_copy = validated_data.copy()
validated_files = []
for key, value in validated_data_copy.items():
if isinstance(value, InMemoryUploadedFile):
validated_files.append(value)
validated_data.pop(key)
submission_instance = super().create(validated_data)
for file in validated_files:
BinaryFile.objects.create(submission=submission_instance, file=file)
return submission_instance
5. That's it!!!
Complete Code Snippet
# serializers.py
from rest_framework import serializers
from django.core.files.uploadedfile import InMemoryUploadedFile
class SubmissionSerializer(serializers.ModelSerializer):
def __init__(self, *args, **kwargs):
file_fields = kwargs.pop('file_fields', None)
super().__init__(*args, **kwargs)
if file_fields:
field_update_dict = {field: serializers.FileField(required=False, write_only=True) for field in file_fields}
self.fields.update(**field_update_dict)
def create(self, validated_data):
validated_data_copy = validated_data.copy()
validated_files = []
for key, value in validated_data_copy.items():
if isinstance(value, InMemoryUploadedFile):
validated_files.append(value)
validated_data.pop(key)
submission_instance = super().create(validated_data)
for file in validated_files:
BinaryFile.objects.create(submission=submission_instance, file=file)
return submission_instance
class Meta:
model = Submission
fields = '__all__'
# views.py
from rest_framework import status
from rest_framework import viewsets
class SubmissionAPI(viewsets.ModelViewSet):
queryset = Submission.objects.all()
serializer_class = SubmissionSerializer
def create(self, request, *args, **kwargs):
# main thing starts
file_fields = list(request.FILES.keys()) # list to be passed to the serializer
serializer = self.get_serializer(data=request.data, file_fields=file_fields)
# main thing ends
serializer.is_valid(raise_exception=True)
self.perform_create(serializer)
headers = self.get_success_headers(serializer.data)
return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
Screenhots and other stuffs
1. POSTMAN console
2. Django Shell
In [2]: Submission.objects.all()
Out[2]: <QuerySet [<Submission: Submission object>]>
In [3]: sub_obj = Submission.objects.all()[0]
In [4]: sub_obj
Out[4]: <Submission: Submission object>
In [5]: sub_obj.__dict__
Out[5]:
{'_state': <django.db.models.base.ModelState at 0x7f529a7ea240>,
'id': 5,
'issued_at': datetime.datetime(2019, 3, 27, 8, 45, 42, 193943, tzinfo=<UTC>),
'completed': False,
'atomic_id': 1}
In [6]: sub_obj.binary_files.all()
Out[6]: <QuerySet [<BinaryFile: uploads/binary/logo-800.png>, <BinaryFile: uploads/binary/Doc.pdf>, <BinaryFile: uploads/binary/invoice_2018_11_29_04_57_53.pdf>, <BinaryFile: uploads/binary/Screenshot_from_2019-02-13_16-22-53.png>]>
In [7]: for _ in sub_obj.binary_files.all():
...: print(_)
...:
uploads/binary/logo-800.png
uploads/binary/Doc.pdf
uploads/binary/invoice_2018_11_29_04_57_53.pdf
uploads/binary/Screenshot_from_2019-02-13_16-22-53.png
3. Django Admin Screenhot

'collections.OrderedDict' object has no attribute 'pk' - django rest framework

I have a model and I want to write an update() method for it in order to update.
The below snippet is my model:
class Klass(models.Model):
title = models.CharField(max_length=50)
description = models.CharField(max_length=500)
university = models.CharField(max_length=50,blank=True, null=True)
teacher = models.ForeignKey(Profile, related_name='teacher', on_delete=models.CASCADE)
and the below snippet is corresponding Serializer:
class KlassSerializer(ModelSerializer):
teacher = ProfileSerializer()
url = HyperlinkedIdentityField(view_name='mainp-api:detail', lookup_field='pk')
klass_settings = KlassSettingsSerializer()
class Meta:
model = Klass
fields = ('url', 'id', 'title', 'description', 'university','teacher')
def update(self, instance, validated_data):
instance.title = validated_data.get('title', instance.title)
instance.description = validated_data.get('description', instance.description)
instance.university = validated_data.get('university', instance.university)
instance.save()
return instance
And for update, I use below snippet:
class KlassAPIView(APIView):
def put(self, request, pk=None):
if pk == None:
return Response({'message': 'You must specify class ID'}, status=HTTP_400_BAD_REQUEST)
klass = Klass.objects.get(pk=pk)
if request.user.profile.type != 't':
raise PermissionDenied(detail={'message': 'You aren't teacher of this class, so you can't edit information.'})
serializer = KlassSerializer(data=request.data, context={'request': request})
serializer.initial_data['teacher'] = request.user.profile.__dict__
if serializer.is_valid():
serializer.update(instance=klass, validated_data=serializer.data) # Retrieve teacher and store
return Response({'data': serializer.data}, status=HTTP_200_OK)
else:
return Response({'data': serializer.errors}, status=HTTP_400_BAD_REQUEST)
but when I send data with PUT method, it returns below error:
AttributeError at /api/class/49/
'collections.OrderedDict' object has no attribute 'pk'
and the error occurs in serializer.update(instance=klass, validated_data=serializer.data) line.

Just ran into the same error.
In my case the problem was I accessed serializer.data before doing serializer.save().
Google dropped me here, so maybe someone else will also find this helpful.
Source: https://github.com/encode/django-rest-framework/issues/2964

i don't know if this helps. I always add the id field in the serializer due to that similar issue:
id = serializers.ModelField(model_field=YourModel._meta.get_field('id'), required=False)
Make sure it's required=False because when you create a new record the id field is not present.

Well in my case, I was doing:
champions_list = []
for champion in champions_serializer.data:
c = {"id": champion.id}
champions_list.append(c)
And the correct way to do it is:
champions_list = []
for champion in champions_serializer.data:
c = {"id": champion["id"]}
champions_list.append(c)
And make sure that you return the id inside the serializer.

Many answers to this question note that serializer.save() must be called before using serializer.data.
In my case, I was definitely calling serializer.save(), however, I was overriding the save method on my serializer and did not set self.instance on the serializer in that method.
So if you are overriding save be sure to do:
class MySerializer(serializers.ModelSerializer):
def save(self, *args, **kwargs):
...
self.instance = instance
return self.instance

Add extract fields to Django model serializer while doing validations

Supposed I had the following model, and modelSerializer:
models.py
class Approve(models.Model):
process = models.IntegerField(verbose_name='Associated Process')
content = models.CharField(max_length=300, verbose_name="Approval Content")
serializers.py
class ApproveSerializer(serializers.ModelSerializer):
class Meta:
model = Approve
fields = ('id', 'process' ,'content')
def validate(self, value):
process = value['process']
try:
something_else = Something.objects.get(process)
except Something.DoesNotExist:
raise serializers.ValidationError('Invalid Process')
return value
The thing is that I want to validate the incoming data in serializers.py, instead of views.py. You can notice that in order to make a validation check, I had query the database for something_else.
The problem is that I want to use this object something_else in the views.py, instead of making another database query. Is there any ways I can pass it with the serializer, without causing a serializer validation error when call serializer.is_valid() method.
Any suggestions will be welcomed, thanks in advance.

Well i am not sure how your views.py might look but essentially, you will need to instantiate the serializer there:
def approve_view(request):
if request.method == "POST":
process = request.POST.get('value', None)
something_else = Something.objects.filter(process=process).first()
serializer = ApproveSerializer(something_else=something_else) # This is where you pass the something_else object.
if serializer.is_valid():
# Your code for successful validation.
else:
# Your code for failed validation
Now that you have passed something_else into the ApproveSerializer, you need to set it up as a property:
class ApproveSerializer(serializers.ModelSerializer):
def __init__(self, something_else=None):
self.something_else = something_else
class Meta:
model = Approve
fields = ('id', 'process' ,'content')
def validate(self, value):
process = value['process']
if not self.something_else:
raise serializers.ValidationError('Invalid Process')
return value

I figure out a way to address the problem.
serializers.py
class ApproveSerializer(serializers.ModelSerializer):
class Meta:
model = Approve
fields = ('id', 'process' ,'content')
def validate(self, value):
process = value['process']
try:
something_else = Something.objects.get(process)
value['something_else'] = something_else
except Something.DoesNotExist:
raise serializers.ValidationError('Invalid Process')
return value
views.py
Before calling the serializer.save() method, you should pop the value that had been added to the serializer.
def post(self, request):
serializer = ApproveSerializer(data=request.data)
if serializer.is_valid():
something_else = serializer.validated_data.pop('something_else')
something_else.property = new_property
something_else.save()
serializer.save()
else:
# Error handling goes here serialzier.errors
Not sure this is a good practice, but at least it works for me right now. Hoping to know a better solution.

How can modify request.data in django REST framework

I am using Django REST Framework
request.data = '{"id": "10", "user": "tom"}'
I want to add extra attribute like "age": "30" before sending it to further like
request.data = new_data
response = super().post(request, *args, **kwargs)
I have two issues
Why request.data is coming as string rather than dict
How can i update the request.data

In case your API is APIView then you should use update function to expand your request data object without losing the data sent from the client-side.
request.data.update({"id": "10", "user": "tom"})

request.data should be an immutable QueryDict, rather than a string. If you need to modify it:
if isinstance(request.data, QueryDict): # optional
request.data._mutable = True
request.data['age'] = "30"
The only reason you might check if it's an instance of QueryDict is so it's easier to unit test with a regular dict.

A good friend just took me to school on a much simpler approach than I illustrate above
class CreateSomething(CreateAPIView):
model = Something
queryset = Something.objects.all()
serializer_class = SomethingSerializer
perform_create(self,serializer):
def perform_create(self,serializer):
ip = self.get_ip()
## magic here: add kwargs for extra fields to write to db
serializer.save(ip_addr=ip)
def get_ip(self):
x_forwarded_for = self.request.META.get('HTTP_X_FORWARDED_FOR',None)
if x_forwarded_for:
ip = x_forwarded_for.split(',')[0]
else:
ip = self.request.META.get('REMOTE_ADDR',None)
return ip
class SomethingSerializer(serializers.ModelSerializer):
email = serializers.EmailField(validators=[UniqueValidator(queryset=Something.objects.all())])
fieldA = serializers.CharField()
fieldB = serializers.CharField()
class Meta:
model = Customer2
fields = ['email','fieldA','fieldB','ip_addr']
read_only_fields = ['ip_addr']

Generally request in drf views is rest_framework.request.Request instance. Following to it's source code (djangorestframework==3.8.2):
#property
def data(self):
if not _hasattr(self, '_full_data'):
self._load_data_and_files()
return self._full_data
You can do:
request._full_data = your_data

It looks like a json string. To convert it to a dict you should do:
import json
data = json.loads(request.data)
then you can add extra attributes:
data['age'] = 30
Then you will have to make a new request because it seem like you cant just change the old one. This assumes that you are posting to /notes/:
from rest_framework.test import APIRequestFactory
factory = APIRequestFactory()
request = factory.post('/notes/', data, format='json')

If you are afraid of altering your request object, then use deep copy to copy the object and then you can easily alter it.
Usage::
from copy import deepcopy
# here is your other code and stuffs
data = deepcopy(request.data)
feel free to alter the data as you want as it is now mutable.
So far, this is my preferred way of altering if not using a generic view.
For any drawbacks of this method, please comment below!

If your endpoint is implemented with a DRF (Django REST Framework) ViewSet the solution can be to implement the corresponding serializer class's to_internal_value method and modify the data there.
class MyModelViewSet(viewsets.ModelViewSet):
authentication_classes = ...
...
serializer_class = MyModelSerializer
class MyModelSerializer(serializers.ModelSerializer):
class Meta:
model = MyModel
fields = ('id', 'user', ...)
def to_internal_value(self, data):
instance = super(MyModelSerializer, self).to_internal_value(data)
if "lastModified" in data:
# instance["id"] = 10 # That's sketchy though
instance["user"] = "tom"
return instance

According to your comment:
"because before posting i need to chnage the field names aqs required by the API"
You should be using the Field's source argument instead.
This will make error messages more consistent otherwise your user will face errors with field names they didn't provide.

I have dealt with this differently. I override the CreateAPIView create method, as follows
class IPAnnotatedObject(CreateAPIView):
model = IPAnnotatedObject
queryset = IPAnnotatedObject.objects.all()
serializer_class = IPAnnotatedObject
def create(self, request, *args, **kwargs):
request.data['ip_addr'] = self.get_ip()
serializer = self.get_serializer(data=request.data)
serializer.is_valid(raise_exception=True)
## perform_create calls serializer.save() which calls the serializer's create() method
self.perform_create(serializer)
headers = self.get_success_headers(serializer.data)
return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
def get_ip(self):
x_forwarded_for = self.request.META.get('HTTP_X_FORWARDED_FOR',None)
if x_forwarded_for:
ip = x_forwarded_for.split(',')[0]
else:
ip = self.request.META.get('REMOTE_ADDR',None)
return ip
The corresponding serializer class looks like follows
class IPAnnotatedObjectSerializer(serializers.ModelSerializer):
email = serializers.EmailField(validators=[UniqueValidator(queryset=IPAnnotatedObject.objects.all())])
password = serializers.CharField(write_only=True)
ip_addr = serializers.IPAddressField(write_only=True)
class Meta:
model = IPAnnotatedObject
fields = ['email','password','created_ip']
def create(self, validated_data):
email, password, created_ip = validated_data['email'], validated_data['password'],validated_data['created_ip']
try:
ipAnnoObject = IPAnnotatedObject.objects.create(email=email,password=make_password(password),ip_addr=ip_addr)
except Exception as e:
# you can think of better error handler
pass
return ipAnnoOjbect

I did the following:
import json
data = json.dumps(request.data)
data = json.loads(data)
data['age'] = 100
now use variable data instead of request.data.

Though other answers are good but i just wanted to add one thing here;
We have now two ways to update a request.data object But before that, check if it is a QueryDict (as already mentioned by #mikebridge);
from django.http.request import QueryDict
if isInstance(request.data, QueryDict):
request.data._mutable = True
After that, to update the request.data, first way is;
request.data.update({'key': 'new_value'})
this will work fine, but say if the request.data['key'] which you want to update is a list, then the value will not get changed completely by the new_value but it will get appended to the older list which may cause trouble and you may not get the desired result.
So to overcome that, i.e, to completely change the value of some key, use the second method;
request.data['key'] = 'new_value'
This will change the value of request.data['key'] completely to new_value.

May be override is_valid in your serializer class will be more readable
class YourSerializer
# Don't forget it
age = serializers.IntegerField()
def is_valid(self, raise_exception: bool = ...) -> bool:
self.initial_data["age"] = 30
return super().is_valid(raise_exception)