We Keep Coding

Pydantic model for array of jsons

I am using FastAPI to write a web service. It is good and fast.
FastAPI is using pydantic models to validate input and output data, everything is good but when I want to declare a nested model for array of jsons like below:
[
{
"name": "name1",
"family": "family1"
},
{
"name": "name2",
"family": "family2"
}
]
I get empty response.
I think there is a problem with my model which is:
class Test(BaseModel):
name: str
family: str
class Config:
orm_mode = True
class Tests(BaseModel):
List[Test]
class Config:
orm_mode = True
So, my question is how should I write a model for array of jsons?

Update (26/09/2020)
In Python 3.9 (not yet released), you can do the same as below but with the built-in list generic type (which is always in scope) rather than needing to import the capitalized List type from typing, e.g.
#app.get("/tests", response_model=list[Test])
The issue here is that you are trying to create a pydantic model where it is not needed. If you want to serialize/deserialize a list of objects, just wrap your singular model in a List[] from python's builtin typing module. There is no need to try to create a plural version of your object with a pydantic BaseModel (and as you can see, it does not work anyway).
With that said, the simplest way to do what you want is to just specify a List[Test] at any point where you need a list of Tests, e.g.
from typing import List
from fastapi import FastAPI
from pydantic import BaseModel
existing_tests = [
{
"name": "name1",
"family": "family1"
},
{
"name": "name2",
"family": "family2"
}
]
class Test(BaseModel):
name: str
family: str
class Config:
orm_mode = True
app = FastAPI()
#app.get("/tests", response_model=List[Test])
async def fetch_tests():
return existing_tests
#app.post("/tests")
async def submit_tests(new_tests: List[Test]):
print(new_tests)
But of course if you find yourself repeatedly (or only) specifying Test as a list, you can of course just assign this to a variable and then use that variable where needed, like so:
Tests = List[Test]
#app.get("/tests", response_model=Tests)
async def fetch_tests():
return existing_tests
#app.post("/tests")
async def submit_tests(new_tests: Tests):
print(new_tests)
I think the first option is probably slightly clearer in your code though, and unless you are specifying List[Test] many times, using a variable for this purpose is probably not worth the extra layer of indirection.

Serializing data with marshmallow if data key is unknown

I'm trying to serialise data where the data key is not known beforehand. The data would look something like this:
{
"unknown key": ["list", "of", "random", "strings"]
}
Is this something that's possible with marshmallow? I can't seem to find anything in the docs about it.

You can implement a method to dynamically generatea Schema class at runtime.
from marshmallow import Schema as BaseSchema
class Schema(BaseSchema):
#classmethod
def from_dict(cls, fields_dict):
attrs = fields_dict.copy()
attrs["Meta"] = type(
"GeneratedMeta", (getattr(cls, "Meta", object),), {"register": False}
)
return type("", (cls,), attrs)
Usage:
from marshmallow import fields
MySchema = Schema.from_dict({
"unknown_key": fields.List(fields.Str())
})
Update: Schema.from_dict is now built-in method in marshmallow 3.0, which will be released soon. See https://stevenloria.com/dynamic-schemas-in-marshmallow/ for usage examples.

Django: Object of type 'datetime' is not JSON serializable

I'm trying to save a date in my sessions. I always receive the error Object of type 'datetime' is not JSON serializable. I found this here in the Django documentation: stored as seconds since epoch since datetimes are not serializable in JSON.
How can I save my expiry_date as seconds instead of datetime?
code = social_ticketing_form.cleaned_data['a']
expiry_date = timezone.now() + timezone.timedelta(days=settings.SOCIAL_TICKETING_ATTRIBUTION_WINDOW)
request.session[social_ticketing_cookie_name(request.event)] = {'code': code, 'expiry_date': expiry_date}

Either write your own session serialiser to allow you to serialise datetime objects directly, or store the datetime value in some other form.
If you want to save it as seconds, then use the datetime.timestamp() method:
request.session[social_ticketing_cookie_name(request.event)] = {
'code': code,
'expiry_date': expiry_date.timestamp()
}
Your own SESSION_SERIALIZER class only needs to provide loads and dumps methods, directly analogous to json.loads() and json.dumps() (which is how the standard JSON serializer is implemented).
If you want to encode datetime objects and be able to transparently turn those back into datetime objects again, I'd use a nested object format to flag such values as special:
from datetime import datetime
class JSONDateTimeSerializer:
#staticmethod
def _default(ob):
if isinstance(ob, datetime):
return {'__datetime__': ob.isoformat()}
raise TypeError(type(ob))
#staticmethod
def _object_hook(d):
if '__datetime__' in d:
return datetime.fromisoformat(d['__datetime__'])
return d
def dumps(self, obj):
return json.dumps(
obj, separators=(',', ':'), default=self._default
).encode('latin-1')
def loads(self, data):
return json.loads(
data.decode('latin-1'), object_hook=self._object_hook
)
and set SESSION_SERIALIZER to the full qualified name of the above module (path.to.module.JSONDateTimeSerializer).
The above uses the datetime.fromisoformat() method, new in Python 3.7.

How to set set JSON encoder in marshmallow?

How do I override the JSON encoder used the marshmallow library so that it can serialize a Decimal field?I think I can do this by overriding json_module in the base Schema or Meta class, but I don't know how:
https://github.com/marshmallow-code/marshmallow/blob/dev/marshmallow/schema.py#L194
I trawled all the docs and read the code, but I'm not a Python native.

If you want to serialize a Decimal field (and keep the value as a number), you can override the default json module used by Marshmallow in its dumps() call to use simplejson instead.
To do this, just add a class Meta definition to your schema, and specify a json_module property for this class.
Example:
import simplejson
class MySchema(Schema):
amount = fields.Decimal()
class Meta:
json_module = simplejson
Then, to serialize:
my_schema = MySchema()
my_schema.dumps(my_object)

I think the solution is to use marshmallow.fields.Decimal with as_string=True:
This field serializes to a decimal.Decimal object by default. If you
need to render your data as JSON, keep in mind that the json module
from the standard library does not encode decimal.Decimal. Therefore,
you must use a JSON library that can handle decimals, such as
simplejson, or serialize to a string by passing as_string=True.

I had the same issue and I endup changing the field on Schema to string. In my case, since I'm only going to return it in json, it really doesn't matter if it is string or decimal.
from marshmallow_sqlalchemy import ModelSchema
from marshmallow import fields
class CurrencyValueSchema(ModelSchema):
class Meta:
model = CurrencyValue
value = fields.String()
My returned json:
{
"currency_values": [
{
"id": 9,
"timestamp": "2016-11-18T23:59:59+00:00",
"value": "0.944304"
},
{
"id": 10,
"timestamp": "2016-11-18T23:59:59+00:00",
"value": "3.392204"
},
}

Excluding primary key in Django dumpdata with natural keys

How do you exclude the primary key from the JSON produced by Django's dumpdata when natural keys are enabled?
I've constructed a record that I'd like to "export" so others can use it as a template, by loading it into a separate databases with the same schema without conflicting with other records in the same model.
As I understand Django's support for natural keys, this seems like what NKs were designed to do. My record has a unique name field, which is also used as the natural key.
So when I run:
from django.core import serializers
from myapp.models import MyModel
obj = MyModel.objects.get(id=123)
serializers.serialize('json', [obj], indent=4, use_natural_keys=True)
I would expect an output something like:
[
{
"model": "myapp.mymodel",
"fields": {
"name": "foo",
"create_date": "2011-09-22 12:00:00",
"create_user": [
"someusername"
]
}
}
]
which I could then load into another database, using loaddata, expecting it to be dynamically assigned a new primary key. Note, that my "create_user" field is a FK to Django's auth.User model, which supports natural keys, and it output as its natural key instead of the integer primary key.
However, what's generated is actually:
[
{
"pk": 123,
"model": "myapp.mymodel",
"fields": {
"name": "foo",
"create_date": "2011-09-22 12:00:00",
"create_user": [
"someusername"
]
}
}
]
which will clearly conflict with and overwrite any existing record with primary key 123.
What's the best way to fix this? I don't want to retroactively change all the auto-generated primary key integer fields to whatever the equivalent natural keys are, since that would cause a performance hit as well as be labor intensive.
Edit: This seems to be a bug that was reported...2 years ago...and has largely been ignored...

Updating the answer for anyone coming across this in 2018 and beyond.
There is a way to omit the primary key through the use of natural keys and unique_together method. Taken from the Django documentation on serialization:
You can use this command to test :
python manage.py dumpdata app.model --pks 1,2,3 --indent 4 --natural-primary --natural-foreign > dumpdata.json ;
Serialization of natural keys
So how do you get Django to emit a natural key when serializing an object? Firstly, you need to add another method – this time to the model itself:
class Person(models.Model):
objects = PersonManager()
first_name = models.CharField(max_length=100)
last_name = models.CharField(max_length=100)
birthdate = models.DateField()
def natural_key(self):
return (self.first_name, self.last_name)
class Meta:
unique_together = (('first_name', 'last_name'),)
That method should always return a natural key tuple – in this example, (first name, last name). Then, when you call serializers.serialize(), you provide use_natural_foreign_keys=True or use_natural_primary_keys=True arguments:
serializers.serialize('json', [book1, book2], indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)
When use_natural_foreign_keys=True is specified, Django will use the natural_key() method to serialize any foreign key reference to objects of the type that defines the method.
When use_natural_primary_keys=True is specified, Django will not provide the primary key in the serialized data of this object since it can be calculated during deserialization:
{
"model": "store.person",
"fields": {
"first_name": "Douglas",
"last_name": "Adams",
"birth_date": "1952-03-11",
}
}

The problem with json is that you can't omit the pk field since it will be required upon loading of the fixture data again. If not existing, json will fail with
$ python manage.py loaddata some_data.json
[...]
File ".../django/core/serializers/python.py", line 85, in Deserializer
data = {Model._meta.pk.attname : Model._meta.pk.to_python(d["pk"])}
KeyError: 'pk'
As pointed out in the answer to this question, you can use yaml or xml if you really want to omit the pk attribute OR just replace the primary key value with null.
import re
from django.core import serializers
some_objects = MyClass.objects.all()
s = serializers.serialize('json', some_objects, use_natural_keys=True)
# Replace id values with null - adjust the regex to your needs
s = re.sub('"pk": [0-9]{1,5}', '"pk": null', s)

Override the Serializer class in a separate module:
from django.core.serializers.json import Serializer as JsonSerializer
class Serializer(JsonSerializer):
def end_object(self, obj):
self.objects.append({
"model" : smart_unicode(obj._meta),
"fields" : self._current,
# Original method adds the pk here
})
self._current = None
Register it in Django:
serializers.register_serializer("json_no_pk", "path.to.module.with.custom.serializer")
Add use it:
serializers.serialize('json_no_pk', [obj], indent=4, use_natural_keys=True)