We Keep Coding

Flexible Schema - BaseModel of Pydantic

I'm using pydantic 1.9.0 and fastapi 0.71.0 with Python 3.7.8. Here's how I've defined my model:
class PartModel(BaseModel):
_id: str
_key: str
number: str = Field(...)
name: str = Field(...)
price: int = Field(...)
class Config:
schema_extra = {
"example": {
"_key": "12453",
"number": "5F22",
"name": "SHA9-169",
"price": 4,
}
}
I would like to be flexible with the Model because my dictionaries are not consisted when it comes to the length. For example one dictionary might have additional key/value pairs. At the moment when i try to make the request through the FastApi it doesn't allow me to POST in the Database. Is there any way to keep my Model flexible/extendable without defining every key in the class?

In order to work with flexible models, I would recommend to work with Optional fields, as described in https://pydantic-docs.helpmanual.io/usage/models/#required-optional-fields

FastAPI: Having a dependency through Depends() and a schema refer to the same root level key without ending up with multiple body parameters

I have a situation where I want to authorize the active user against one of the values (Organization) in a FastAPI route. When an object of a particular type is being submitted, one of the keys (organization_id) should be present and the user should be verified as having access to the organization.
I've solved this as a dependency in the API signature to avoid having to replicate this code across all routes that needs access to this property:
def get_organization_from_body(organization_id: str = Body(None),
user: User = Depends(get_authenticated_user),
organization_service: OrganizationService = Depends(get_organization_service),
) -> Organization:
if not organization_id:
raise HTTPException(status_code=400, detail='Missing organization_id.')
organization = organization_service.get_organization_for_user(organization_id, user)
if not organization:
raise HTTPException(status_code=403, detail='Organization authorization failed.')
return organization
This works fine, and if the API endpoint expects an organization through an organization_id key in the request, I can get the organization directly populated by introducing get_organization_from_body as a dependency in my route:
#router.post('', response_model=Bundle)
async def create_bundle([...]
organization: Organization = Depends(get_organization_from_body),
) -> Model:
.. and if the user doesn't have access to the organization, an 403 exception is raised.
However, I also want to include my actual object on the root level through a schema model. So my first attempt was to make a JSON request as:
{
'name': generated_name,
'organization_id': created_organization['id_key']
}
And then adding my incoming Pydantic model:
#router.post('', response_model=Bundle)
async def create_bundle(bundle: BundleCreate,
organization: Organization = Depends(get_organization_from_body),
[...]
) -> BundleModel:
[...]
return bundle
The result is the same whether the pydantic model / schema contains organization_id as a field or not:
class BundleBase(BaseModel):
name: str
class Config:
orm_mode = True
class BundleCreate(BundleBase):
organization_id: str
client_id: Optional[str]
.. but when I introduce my get_organization_from_body dependency, FastAPI sees that I have another dependency that refers to a Body field, and the description of the bundle object has to be moved inside a bundle key instead (so instead of "validating" the organization_id field, the JSON layout needs to change - and since I feel that organization_id is part of the bundle description, it should live there .. if possible).
The error message tells me that bundle was expected as a separate field:
{'detail': [{'loc': ['body', 'bundle'], 'msg': 'field required', 'type': 'value_error.missing'}]}
And rightly so, when I move name inside a bundle key instead:
{
'bundle': {
'name': generated_name,
},
'organization_id': created_organization['id_key']
}
.. my test passes and the request is successful.
This might be slightly bike shedding, but if there's a quick fix to work around this limitation in any way I'd be interested to find a way to both achieve validation (either through Depends() or in some alternative way without doing it explicitly in each API route function that requires that functionality) and a flat JSON layout that matches my output format closer.

Prior to FastAPI 0.53.2, dependencies for the body were resolved the way you are trying to do. Such code:
class Item(BaseModel):
val_1: int
val_2: int
def get_val_1(val_1: int = Body(..., embed=True)):
return val_1
#app.post("/item", response_model=Item)
def handler(full_body: Item, val_1: int = Depends(get_val_1)):
return full_body
Expected such body:
{
"val_1": 0,
"val_2": 0
}
But starting from version 0.53.2, different body dependencies are automatically embedded (embed=True) and the code above expects the following body:
{
"full_body": {
"val_1": 0,
"val_2": 0
},
"val_1": 0
}
Now, in order to have access to the model for the whole body and to have access to its elements as a separate dependency, you need to use same dependency for the body model everywhere:
def get_val_1(full_body: Item):
return full_body.val_1
#app.post("/item", response_model=Item)
def handler(full_body: Item, val_1: int = Depends(get_val_1)):
return full_body
Update for shared dependency
You can share one body dependency for multiple models, but in this case, two conditions must be met: the names of the dependencies must be the same and their types must be compatible (through inheritance or not). Example below:
class Base(BaseModel):
val_1: int
class NotBase(BaseModel):
val_1: int
class Item1(Base):
val_2: int
class Item2(Base):
val_3: int
def get_val1_base(full_body: Base):
return full_body.val_1
def get_val1_not_base(full_body: NotBase):
return full_body.val_1
#app.post("/item1", response_model=Item1)
def handler(full_body: Item1, val_1: int = Depends(get_val1_base)):
return full_body
#app.post("/item2", response_model=Item2)
def handler(full_body: Item2, val_1: int = Depends(get_val1_not_base)):
return full_body

How to validate JSON field with name "from"

I want to validate JSON object (it is in Telegram Bot API) which contains from field (which is reserved word in Python) by using pydantic validator. So my model should look like the following:
class Message(BaseModel):
message_id: int
from: Optional[str]
date: int
chat: Any
...
But using from keyword is not allowed in this context.
How could I do this?
Note: this is different than "Why we can't use keywords as attributes" because here we get external JSON we don't control and we anyway should handle JSON with from field.

I believe you can replace from with from_.
You can do it like this:
class Message(BaseModel):
message_id: int
from_: Optional[str]
date: int
chat: Any
class Config:
fields = {
'from_': 'from'
}
...

There might be a way to do this using a class statement, but I didn't see anything in a quick skim of the documentation. What you could do is use dynamic model creation instead.
fields = {
'message_id': (int,),
'from': (Optional[str], ...),
'date': (int, ...),
'chat': (Any, ...)
}
Message = create_model("Message", **fields)

Django Rest Framework: POST and PUT in a single nested request

I'm using Django Rest Framework to create an object. The JSON contains nested objects as well; an array of objects to create and link to the "main object" and an object that should be partially updated.
JSON looks like this:
{
"opcitem_set" : [
{
"comment" : "Test comment",
"grade" : "1",
"name" : "1a"
},
{
"comment" : "Test comment",
"grade" : "2",
"name" : "1b"
},
{
"description" : "Additional test item",
"comment" : "Additional comment",
"grade" : "1",
"name" : "extra_1"
}
],
"is_right_seat_training" : false,
"checked_as" : "FC",
"date" : "2015-10-23",
"check_reason" : "Check ride",
"opc_program" : "2",
"is_pc" : true,
"questionnaire_test_passed" : "Passed",
"pnf_time" : 2,
"other_comments_complete_crew" : "Other comments",
"other_comments_flying_pilot" : "Other comments",
"is_cat_2_training" : false,
"opc_passed" : "Passed",
"pilot" : {
"pc_valid_to" : "2015-10-23",
"id" : 721,
"email" : "jens.nilsson#nextjet.se",
"total_time" : 3120,
"medical_valid_to" : "2015-10-23"
},
"pf_time" : 2,
"aircraft_type" : "S340",
"typeratingexaminer" : 734
}
The "opcitem_set" contains objects of type OpcItem that should be created and have a ForeignKey to the main object. So far so good, I can do this by overriding the create() method on the ModelSerializer as outlined in http://www.django-rest-framework.org/api-guide/serializers/#writable-nested-representations.
Then we have the case of the "pilot" object. This will always contain an ID and some other fields to PATCH the object with that ID.
The "typeratingexaminer" field is just another "Pilot" object, but it shouldn't be PATCHed, just set as a foreign key.
My question is: Can I PATCH (partially update) the "pilot" as well in the create() method, or would that break some sort of design pattern? Since it's really a PATCH and not a POST, should I do it in a separate request after the original request has finished? In that case, can I have a transaction spanning two requests, so that if the second request fail, the first request will be rolled back?
Would love to be able to send only one request from the client instead of splitting it in two requests. Maybe you could separate the JSON already in the ViewSet and send it to different serializers?
Happy to hear your thoughts about this, I'm a bit lost.

If you are not creating a main object but only nested objects you should override .update() method in the serializer and do somethink like this:
def update(self, instance, validated_data):
if 'opcitem_set' in validated_data:
opcitem_set_data = validated_data.pop('opcitem_set')
if 'pilot' in validated_data:
pilot_data = validated_data.pop('pilot')
...
for opcitem_set in opcitem_set_data:
Opcitem.objects.create(main_object=instance,
**opcitem_set)
current_pilot = self.instance.pilot
current_pilot.pc_valid_to = pilot_data.get('name', current_pilot.pc_valid_to)
...
current_pilot.save()
"""
Update instance as well if you need
"""
return instance
If you need to create main object as well then you need to override .create() method. But then PATCHing pilot will be not really a good way to do that.

I would recommend to move away from the serializer create method and build your extensive logic in the view, where you could make good use of simpler, dumb serializers where needed. You could surly do updates in the create method of your serializer but suddenly that's not a serializer anymore, it's more of a controller, thus it would be better placed in the view code, by overwriting the create or post method; this design enables you to have only one request from the client, you can massage the request data in the view code and use simple serializers to instantiate/update objects, with embedded data validation, if needed.
If you have models and serializers you could share, we might be able to comment more to the point.

Excluding primary key in Django dumpdata with natural keys

How do you exclude the primary key from the JSON produced by Django's dumpdata when natural keys are enabled?
I've constructed a record that I'd like to "export" so others can use it as a template, by loading it into a separate databases with the same schema without conflicting with other records in the same model.
As I understand Django's support for natural keys, this seems like what NKs were designed to do. My record has a unique name field, which is also used as the natural key.
So when I run:
from django.core import serializers
from myapp.models import MyModel
obj = MyModel.objects.get(id=123)
serializers.serialize('json', [obj], indent=4, use_natural_keys=True)
I would expect an output something like:
[
{
"model": "myapp.mymodel",
"fields": {
"name": "foo",
"create_date": "2011-09-22 12:00:00",
"create_user": [
"someusername"
]
}
}
]
which I could then load into another database, using loaddata, expecting it to be dynamically assigned a new primary key. Note, that my "create_user" field is a FK to Django's auth.User model, which supports natural keys, and it output as its natural key instead of the integer primary key.
However, what's generated is actually:
[
{
"pk": 123,
"model": "myapp.mymodel",
"fields": {
"name": "foo",
"create_date": "2011-09-22 12:00:00",
"create_user": [
"someusername"
]
}
}
]
which will clearly conflict with and overwrite any existing record with primary key 123.
What's the best way to fix this? I don't want to retroactively change all the auto-generated primary key integer fields to whatever the equivalent natural keys are, since that would cause a performance hit as well as be labor intensive.
Edit: This seems to be a bug that was reported...2 years ago...and has largely been ignored...

Updating the answer for anyone coming across this in 2018 and beyond.
There is a way to omit the primary key through the use of natural keys and unique_together method. Taken from the Django documentation on serialization:
You can use this command to test :
python manage.py dumpdata app.model --pks 1,2,3 --indent 4 --natural-primary --natural-foreign > dumpdata.json ;
Serialization of natural keys
So how do you get Django to emit a natural key when serializing an object? Firstly, you need to add another method – this time to the model itself:
class Person(models.Model):
objects = PersonManager()
first_name = models.CharField(max_length=100)
last_name = models.CharField(max_length=100)
birthdate = models.DateField()
def natural_key(self):
return (self.first_name, self.last_name)
class Meta:
unique_together = (('first_name', 'last_name'),)
That method should always return a natural key tuple – in this example, (first name, last name). Then, when you call serializers.serialize(), you provide use_natural_foreign_keys=True or use_natural_primary_keys=True arguments:
serializers.serialize('json', [book1, book2], indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)
When use_natural_foreign_keys=True is specified, Django will use the natural_key() method to serialize any foreign key reference to objects of the type that defines the method.
When use_natural_primary_keys=True is specified, Django will not provide the primary key in the serialized data of this object since it can be calculated during deserialization:
{
"model": "store.person",
"fields": {
"first_name": "Douglas",
"last_name": "Adams",
"birth_date": "1952-03-11",
}
}

The problem with json is that you can't omit the pk field since it will be required upon loading of the fixture data again. If not existing, json will fail with
$ python manage.py loaddata some_data.json
[...]
File ".../django/core/serializers/python.py", line 85, in Deserializer
data = {Model._meta.pk.attname : Model._meta.pk.to_python(d["pk"])}
KeyError: 'pk'
As pointed out in the answer to this question, you can use yaml or xml if you really want to omit the pk attribute OR just replace the primary key value with null.
import re
from django.core import serializers
some_objects = MyClass.objects.all()
s = serializers.serialize('json', some_objects, use_natural_keys=True)
# Replace id values with null - adjust the regex to your needs
s = re.sub('"pk": [0-9]{1,5}', '"pk": null', s)

Override the Serializer class in a separate module:
from django.core.serializers.json import Serializer as JsonSerializer
class Serializer(JsonSerializer):
def end_object(self, obj):
self.objects.append({
"model" : smart_unicode(obj._meta),
"fields" : self._current,
# Original method adds the pk here
})
self._current = None
Register it in Django:
serializers.register_serializer("json_no_pk", "path.to.module.with.custom.serializer")
Add use it:
serializers.serialize('json_no_pk', [obj], indent=4, use_natural_keys=True)