We Keep Coding

Pydantic from_orm to load Django model with related list field

I have the following Django models:
from django.db import models
class Foo(models.Model):
id: int
name = models.TextField(null=False)
class Bar(models.Model):
id: int
foo = models.ForeignKey(
Foo,
on_delete=models.CASCADE,
null=False,
related_name="bars",
)
And Pydantic models (with orm_mode set to True):
from pydantic import BaseModel
class BarPy(BaseModel):
id: int
foo_id: int
class FooPy(BaseModel):
id: int
name: str
bars: list[BarPy]
Now I want to perform a query on the model Foo and load it into FooPy, so i wrote this query:
foo_db = Foo.objects.prefetch_related("bars").all()
pydantic_model = FooPy.from_orm(foo_db)
But it gives me this error:
pydantic.error_wrappers.ValidationError: 1 validation error for FooPy
bars
value is not a valid list (type=type_error.list)
I am able to do it when explicitly using the FooPy constructor and assigning the values manually but i want to use from_orm.

The bars attribute on your Foo model is a ReverseManyToOneDescriptor that just returns a RelatedManager for the Bar model. As with any manager in Django, to get a queryset of all the instances managed by it, you need to call the all method on it. Typically you would do something like foo.bars.all().
You can add your own custom validator to FooPy and make it pre=True to grab all the related Bar instances and pass a sequence of them along to the default validators:
from django.db.models.manager import BaseManager
from pydantic import BaseModel, validator
...
class FooPy(BaseModel):
id: int
name: str
bars: list[BarPy]
#validator("bars", pre=True)
def get_all_from_manager(cls, v: object) -> object:
if isinstance(v, BaseManager):
return list(v.all())
return v
Note that it is not enough to just do .all() because that will return a queryset, which will not pass the default sequence validator built into Pydantic models. You would get the same error.
You need to give it an actual sequence (e.g. list or tuple). A QuerySet is not a sequence, but an iterable. But you can consume it and turn it into a sequence, by calling for example list on it.
More generalized version
You could make an attempt at generalizing that validator and add it to your own (Pydantic) base model. Something like this should work on any field you annotate as list[Model], with Model being a subclass of pydantic.BaseModel:
from django.db.models.manager import BaseManager
from pydantic import BaseModel, validator
from pydantic.fields import ModelField, SHAPE_LIST
...
class CustomBaseModel(BaseModel):
#validator("*", pre=True)
def get_all_from_manager(cls, v: object, field: ModelField) -> object:
if not (isinstance(field.type_, type) and issubclass(field.type_, BaseModel)):
return v
if field.shape is SHAPE_LIST and isinstance(v, BaseManager):
return list(v.all())
return v
I have not thoroughly tested this, but I think you get the idea.
Side note
It is worth mentioning that prefetch_related has nothing to do with the problem. The problem and its solution are the same, whether you do that or not. The difference is that without prefetch_related, you'll trigger additional database queries, when calling from_orm and thus executing the validator that consumes the queryset of .bars.all().

Return type in a class method based on a class variable in Python

I am working with a codebase where you have couples of classes, always one dataclass and another execution class. The dataclass serves as a data collector (as the name suggests).
To "connect" the dataclass to the other class, I set a class variable in the other class to make clear what the relevant dataclass is. This works fine - I can use this class variable to instantiate the data class as I please. However, it is not clear to me how I can use this to specify for a given method that it will return an instance of the linked data class.
Take this example (executable):
from abc import ABC
from dataclasses import dataclass
from typing import ClassVar
#dataclass
class Name(ABC):
name: str
class RelatedName(ABC):
_INDIVIDAL: ClassVar[Name]
def return_name(self, **properties) -> Name:
# There is a typing issue here too but you can ignore that for now
return self._INDIVIDAL(**properties)
#dataclass
class BiggerName(Name):
other_name: str
class RelatedBiggerName(RelatedName):
_INDIVIDAL: ClassVar[Name] = BiggerName
if __name__ == "__main__":
biggie = RelatedBiggerName()
biggiename = biggie.return_name(name="Alfred", other_name="Biggie").other_name
print(biggiename)
The script works fine, but there is a typing problem. In the last but one line, you'll see the issue that the attribute other_name is undefined for the Name class. This is to be expected, but I am not sure how I can change the output type of return_name so that it will use the class that is defined in _INDIVIDUAL.
I tried def return_name(self, **properties) -> _INDIVIDAL but that naturally leads to name '_INDIVIDAL' is not defined.
Perhaps it is not possible what I am after. Is it at all possible to have typing within a class that depends on class variables? I'm interested in Python 3.8 and higher.

I agree with #cherrywoods that a custom generic base class seems like the way to go here.
I would like to add my own variation that should do what you want:
from abc import ABC
from dataclasses import dataclass
from typing import Any, Generic, Optional, Type, TypeVar, get_args, get_origin
T = TypeVar("T", bound="Name")
#dataclass
class Name(ABC):
name: str
class RelatedName(ABC, Generic[T]):
_INDIVIDUAL: Optional[Type[T]] = None
#classmethod
def __init_subclass__(cls, **kwargs: Any) -> None:
"""Identifies and saves the type argument"""
super().__init_subclass__(**kwargs)
for base in cls.__orig_bases__: # type: ignore[attr-defined]
origin = get_origin(base)
if origin is None or not issubclass(origin, RelatedName):
continue
type_arg = get_args(base)[0]
# Do not set the attribute for GENERIC subclasses!
if not isinstance(type_arg, TypeVar):
cls._INDIVIDUAL = type_arg
return
#classmethod
def get_individual(cls) -> Type[T]:
"""Getter ensuring that we are not dealing with a generic subclass"""
if cls._INDIVIDUAL is None:
raise AttributeError(
f"{cls.__name__} is generic; type argument unspecified"
)
return cls._INDIVIDUAL
def __setattr__(self, name: str, value: Any) -> None:
"""Prevent instances from overwriting `_INDIVIDUAL`"""
if name == "_INDIVIDUAL":
raise AttributeError("Instances cannot modify `_INDIVIDUAL`")
super().__setattr__(name, value)
def return_name(self, **properties: Any) -> T:
return self.get_individual()(**properties)
#dataclass
class BiggerName(Name):
other_name: str
class RelatedBiggerName(RelatedName[BiggerName]):
pass
if __name__ == "__main__":
biggie = RelatedBiggerName()
biggiename = biggie.return_name(name="Alfred", other_name="Biggie").other_name
print(biggiename)
Works without problems or complaints from mypy --strict.
Differences
The _INDIVIDUAL attribute is no longer marked as a ClassVar because that (for no good reason) disallows type variables.
To protect it from being changed by instances, we use a simple customization of the __setattr__ method.
You no longer need to explicitly set _INDIVIDUAL on any specific subclass of RelatedName. This is taken care of automatically during subclassing by __init_subclass__. (If you are interested in details, I explain them in this post.)
Direct access to the _INDIVIDUAL attribute is discouraged. Instead there is the get_individual getter. If the additional parentheses annoy you, I suppose you can play around with discriptors to construct a property-like situation for _INDIVIDUAL. (Note: You can still just use cls._INDIVIDUAL or self._INDIVIDUAL, it's just that there will be the possible None-type issue.)
The base class is obviously a bit more complicated this way, but on the other hand the creation of specific subclasses is much nicer in my opinion.
Hope this helps.

Can you use generics?
from abc import ABC
from dataclasses import dataclass
from typing import ClassVar, TypeVar, Generic, Type
T = TypeVar("T", bound="Name")
#dataclass
class Name(ABC):
name: str
class RelatedName(ABC, Generic[T]):
# This would resolve what juanpa.arrivillaga pointed out, but mypy says:
# ClassVar cannot contain type variables, so I guess your use-case is unsupported
# _INDIVIDAL: ClassVar[Type[T]]
# One option:
# _INDIVIDAL: ClassVar
# Second option to demonstrate Type[T]
_INDIVIDAL: Type[T]
def return_name(self, **properties) -> T:
return self._INDIVIDAL(**properties)
#dataclass
class BiggerName(Name):
other_name: str
class RelatedBiggerName(RelatedName[BiggerName]):
# see above
_INDIVIDAL: Type[BiggerName] = BiggerName
if __name__ == "__main__":
biggie = RelatedBiggerName()
biggiename = biggie.return_name(name="Alfred", other_name="Biggie").other_name
print(biggiename)
mypy reports no errors on this and I think conceptually this is what you want.
I tested on python 3.10.

What is the purpose of overriding from_orm classmethod in pydantic models?

I was writing models for my database tables using pydantic. My code does not override the from_orm() classmethod. However one of my peer had overridden from_orm() classmethod. The output looks the same for me with and without overriding. For instance,
from pydantic import BaseModel
class User(BaseModel):
id: str
name: Optional[str]
class Config:
orm_mode = True
works the same as
class User(BaseModel):
id: str
name: Optional[str]
class Config:
orm_mode = True
#classmethod
def from_orm(cls, obj: User):
return User(id=obj.id, name=obj.name)
The [docs][1] just say
from_orm()
loads data into a model from an arbitrary class; cf
The special constructor from_orm must be used to create the model instance
Could someone please help me understand the purpose of overriding from_orm() method?
[1]: https://pydantic-docs.helpmanual.io/usage/models/

How to make #root_validator in Pydantic return model class object rather than values?

I am currently using a root_validator in my FastAPI project using Pydantic like this:
class User(BaseModel):
id: Optional[int]
name: Optional[str]
#root_validator
def validate(cls, values):
if not values.get("id") and not values.get("name"):
raise ValueError("It's an error")
return values
The issue is, when I query the request body in FastAPI, because of return values, instead of the request body being of type class User, it is just a simple python dictionary. How do I get the same object of type class User?
So initially, my request body would like this when I printed it: id=0 name='string' (which is how I want it) (and when I print the type() of it, it shows: <class 'User'>)
Here is what it looks like with return values: {"id":0, "name"="string"}
I have tried making it as just return cls, but this is what it looks like when I print it: <class 'User'> (and when I print the type() of it, it shows: <class 'pydantic.main.ModelMetaclass'>)
How to get my solution?

I had raised this issue in FastAPI and Pydantic discussions. And found the answer here by a community member: https://github.com/tiangolo/fastapi/discussions/4563
The solution is to rename the def validate func to anything else like def validate_all_fields
And the reason for this is validate is base method for BaseModel in Pydantic!!

How to validate a pydantic object after editing it

Is there any obvious to validate a pydantic model after some changing some attribute?
Say I create a simple Model and object:
from pydantic import BaseModel
class A(BaseModel):
b: int = 0
a=A()
Then edit it, so that it is actually invalid:
a.b = "foobar"
Can I force a re-validation and expect a ValidationError to be raised?
I tried
A.validate(a) # no error
a.copy(update=dict(b='foobar')) # no error
What did work was
A(**dict(a._iter()))
ValidationError: 1 validation error for A
b
value is not a valid integer (type=type_error.integer)
But that is not really straightforward and I need to use the supposedly private method _iter.
Is there a clean alternative?

pydantic can do this for you, you just need validate_assignment:
from pydantic import BaseModel
class A(BaseModel):
b: int = 0
class Config:
validate_assignment = True