We Keep Coding

Serializing data with marshmallow if data key is unknown

I'm trying to serialise data where the data key is not known beforehand. The data would look something like this:
{
"unknown key": ["list", "of", "random", "strings"]
}
Is this something that's possible with marshmallow? I can't seem to find anything in the docs about it.

You can implement a method to dynamically generatea Schema class at runtime.
from marshmallow import Schema as BaseSchema
class Schema(BaseSchema):
#classmethod
def from_dict(cls, fields_dict):
attrs = fields_dict.copy()
attrs["Meta"] = type(
"GeneratedMeta", (getattr(cls, "Meta", object),), {"register": False}
)
return type("", (cls,), attrs)
Usage:
from marshmallow import fields
MySchema = Schema.from_dict({
"unknown_key": fields.List(fields.Str())
})
Update: Schema.from_dict is now built-in method in marshmallow 3.0, which will be released soon. See https://stevenloria.com/dynamic-schemas-in-marshmallow/ for usage examples.

How to deserialize a bjson structure to an marshmallow schema

I'm trying to convert bjson structure to a schema in marshmallow library.
Below is the marshmallow schema:
class GeneSchema(Schema):
"""description of class"""
id_entrez = fields.Integer(required = True, error_messages={'required': "The 'id_entrez' field is requeired."})
symbol = fields.String()
#validates('id_entrez')
def validate_id_entrez(self, data):
if data <= 0:
raise ValidationError("The 'id_entrez' field must be greater than zero.")
Below is the bjson will be converted to schema:
[{"symbol": "VAMP4", "_id": {"$oid": "57ae3b175a945932fcbdf41d"}, "id_entrez": 8674}, {"symbol": "CCT5", "_id": {"$oid": "57ae3b175a945932fcbdf41e"}, "id_entrez": 22948}]
Note that the bjson has the "_id" as ObjectId - "$oid". This is because the result of the query using the mongodb.
Please, does anyone know why not be to convert from bjson to marshmallow schema correctly ?
Thank you all!

I don't know if this question is still valid, but I would like to show my solution, how to push an ObjectId to a Marshmallow schema:
I simply use the pre processing method pre_load to convert the ObjectId to a String:
#pre_load
def convert_objectid(self, in_data, **kwargs):
if "_id" in in_data:
in_data["_id"] = str(in_data["_id"])
return in_data

You can still use your schema to parse MongoDB output, just ignore extra "_id" field. If on the other hand you do want to parse that "_id", just add extra non-required field in your schema.

Django Object Serialize And Custom Date Formatting

So I am using this code to serialize Django objects to a JSON format (so I can send to RabbitMQ over Celery):
import django.core.serializers as serializers
import json
def serialize(obj):
d = json.loads(serializers.serialize('json', [obj]).strip('[]'))
filtered_fields = {
'id': d['pk'],
}
for key, value in d['fields'].iteritems():
filtered_fields[underscore_to_camelcase(str(key))] = value
return filtered_fields
This returns something like:
{
"firstName": "Foo",
"lastName": "Bar",
"createdAt": "2013-12-15T20:53:59.615",
"updatedAt": "2013-12-15T20:53:59.615",
"dateOfBirth": "1990-05-17",
"id": "foo#bar.com"
}
Is there a way to tell the Django serialiser to convert date time objects to a Zulu format? So instead of:
2013-12-15T20:53:59.615
I want:
2013-12-15T20:53:59Z

If you use django's simplejson for serializing you could do something like:
simplejson.dumps(data, cls=LazyEncoder)
and have that LazyEncoder take care of special cases like translations and in your case datetimeformat:
class LazyEncoder(simplejson.JSONEncoder):
def default(self, obj):
elif isinstance(obj, datetime.datetime):
return #your formatting goes here
Only problem is that simplejson.dumps returns string. But if you look into it, then perhaps you can use the same lazyencoder for your purposes (https://docs.djangoproject.com/en/dev/topics/serialization/#id2). Just an idea here - i have not had the need to dig deeper into this, so i am not sure if this is useful to you

Serialize Class Django

I'm trying to serialize a class in Django, so that I can get all the available fields on a json file, I would need something like
{"tablename": ["Verbose Name", "ModelType", "RelatedClass", "relatefield"]}
The idea is that most of the objects will only have the verbose name and the model type, but for related fields, it will also have the name of the class that the relation refers to, and a field that I can add manually say on helptext as a default, or I can just leave that out probably and work on it differently.
I need to do this to the class not to the object I've tried with pickle and jsonpickle but they don't seem to work as I was expecting, I'm out of ideas, any input would be greatly appreciated.
Thanks.
Edit: Need to clarify better
class Test(models.Model):
name = models.CharField(verbose_name="Name")
email = models.CharField()
books = models.ForeignKey(Book, verbose_name="Books")
class Book(models.Model):
name = models.CharField()
now I just want to serialize Test, no value has to be in it at all, just the class itself, so you'd have.
"name": {"verbose_name": "Name", "type": "CharField"}
"books": {"verbose_name": "Books", "type": "ForeignKey", "related_class": "Book", "related_class": "Country", "related_field": "name"}
I need a json around them lines to come out, but I wont have to run any queries with data in them just model information.

Django’s serialization framework provides a mechanism for “translating” Django models into other formats. Usually these other formats will be text-based and used for sending Django data over a wire, but it’s possible for a serializer to handle any format (text-based or not).
from django.core import serializers
data = serializers.serialize("xml", SomeModel.objects.all())
xml example:
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">
<object pk="123" model="sessions.session">
<field type="DateTimeField" name="expire_date">2013-01-16T08:16:59.844560+00:00</field>
<!-- ... -->
</object>
</django-objects>
json:
[
{
"pk": "4b678b301dfd8a4e0dad910de3ae245b",
"model": "sessions.session",
"fields": {
"expire_date": "2013-01-16T08:16:59.844Z",
...
}
}
]
Deserializing data is also a fairly simple operation:
for obj in serializers.deserialize("xml", data):
do_something_with(obj)
https://docs.djangoproject.com/en/dev/topics/serialization/

Classes are objects, model fields too, so you can easily inspect them. One solution would be to write your custom json encoder for model classes (not instances). All informations about the model fields, verbose name etc are stored in YourModelClass.__meta.

Excluding primary key in Django dumpdata with natural keys

How do you exclude the primary key from the JSON produced by Django's dumpdata when natural keys are enabled?
I've constructed a record that I'd like to "export" so others can use it as a template, by loading it into a separate databases with the same schema without conflicting with other records in the same model.
As I understand Django's support for natural keys, this seems like what NKs were designed to do. My record has a unique name field, which is also used as the natural key.
So when I run:
from django.core import serializers
from myapp.models import MyModel
obj = MyModel.objects.get(id=123)
serializers.serialize('json', [obj], indent=4, use_natural_keys=True)
I would expect an output something like:
[
{
"model": "myapp.mymodel",
"fields": {
"name": "foo",
"create_date": "2011-09-22 12:00:00",
"create_user": [
"someusername"
]
}
}
]
which I could then load into another database, using loaddata, expecting it to be dynamically assigned a new primary key. Note, that my "create_user" field is a FK to Django's auth.User model, which supports natural keys, and it output as its natural key instead of the integer primary key.
However, what's generated is actually:
[
{
"pk": 123,
"model": "myapp.mymodel",
"fields": {
"name": "foo",
"create_date": "2011-09-22 12:00:00",
"create_user": [
"someusername"
]
}
}
]
which will clearly conflict with and overwrite any existing record with primary key 123.
What's the best way to fix this? I don't want to retroactively change all the auto-generated primary key integer fields to whatever the equivalent natural keys are, since that would cause a performance hit as well as be labor intensive.
Edit: This seems to be a bug that was reported...2 years ago...and has largely been ignored...

Updating the answer for anyone coming across this in 2018 and beyond.
There is a way to omit the primary key through the use of natural keys and unique_together method. Taken from the Django documentation on serialization:
You can use this command to test :
python manage.py dumpdata app.model --pks 1,2,3 --indent 4 --natural-primary --natural-foreign > dumpdata.json ;
Serialization of natural keys
So how do you get Django to emit a natural key when serializing an object? Firstly, you need to add another method – this time to the model itself:
class Person(models.Model):
objects = PersonManager()
first_name = models.CharField(max_length=100)
last_name = models.CharField(max_length=100)
birthdate = models.DateField()
def natural_key(self):
return (self.first_name, self.last_name)
class Meta:
unique_together = (('first_name', 'last_name'),)
That method should always return a natural key tuple – in this example, (first name, last name). Then, when you call serializers.serialize(), you provide use_natural_foreign_keys=True or use_natural_primary_keys=True arguments:
serializers.serialize('json', [book1, book2], indent=2, use_natural_foreign_keys=True, use_natural_primary_keys=True)
When use_natural_foreign_keys=True is specified, Django will use the natural_key() method to serialize any foreign key reference to objects of the type that defines the method.
When use_natural_primary_keys=True is specified, Django will not provide the primary key in the serialized data of this object since it can be calculated during deserialization:
{
"model": "store.person",
"fields": {
"first_name": "Douglas",
"last_name": "Adams",
"birth_date": "1952-03-11",
}
}

The problem with json is that you can't omit the pk field since it will be required upon loading of the fixture data again. If not existing, json will fail with
$ python manage.py loaddata some_data.json
[...]
File ".../django/core/serializers/python.py", line 85, in Deserializer
data = {Model._meta.pk.attname : Model._meta.pk.to_python(d["pk"])}
KeyError: 'pk'
As pointed out in the answer to this question, you can use yaml or xml if you really want to omit the pk attribute OR just replace the primary key value with null.
import re
from django.core import serializers
some_objects = MyClass.objects.all()
s = serializers.serialize('json', some_objects, use_natural_keys=True)
# Replace id values with null - adjust the regex to your needs
s = re.sub('"pk": [0-9]{1,5}', '"pk": null', s)

Override the Serializer class in a separate module:
from django.core.serializers.json import Serializer as JsonSerializer
class Serializer(JsonSerializer):
def end_object(self, obj):
self.objects.append({
"model" : smart_unicode(obj._meta),
"fields" : self._current,
# Original method adds the pk here
})
self._current = None
Register it in Django:
serializers.register_serializer("json_no_pk", "path.to.module.with.custom.serializer")
Add use it:
serializers.serialize('json_no_pk', [obj], indent=4, use_natural_keys=True)