We Keep Coding

Serialize UUID objects in flask-restful

I have a flask-restful project that interfaces with some custom classes, containing uuid (uuid.UUID) types used as ids. There are a couple of api endpoints which return the object associated with the given id, parsed by flask as an UUID. The issue is that, when I return them as a json payload, I get the following exception:
UUID('…') is not JSON serializable
I want to have those uuids represented as strings to the final user, making the process seamless (the user can take the returned uuid and use it for his next api request).

In order to fix this problem, I had to put together suggestions from two different places:
first, I need to create a custom json encoder, which when dealing with uuids, returns their string representation. StackOverflow answer here
import json
from uuid import UUID
class UUIDEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, UUID):
# if the obj is uuid, we simply return the value of uuid
return str(obj) # <- notice I'm not returning obj.hex as the original answer
return json.JSONEncoder.default(self, obj)
second, I need to take this new encoder and set it as the flask-restful encoder used for the responses. GitHub answer here
class MyConfig(object):
RESTFUL_JSON = {'cls': MyCustomEncoder}
app = Flask(__name__)
app.config.from_object(MyConfig)
api = Api(app)
putting it together:
# ?: custom json encoder to be able to fix the UUID('…') is not JSON serializable
class UUIDEncoder(json.JSONEncoder):
def default(self, obj: Any) -> Any: # pylint:disable=arguments-differ
if isinstance(obj, UUID):
return str(obj) # <- notice I'm not returning obj.hex as the original answer
return json.JSONEncoder.default(self, obj)
# ?: api configuration to switch the json encoder
class MyConfig(object):
RESTFUL_JSON = {"cls": UUIDEncoder}
app = Flask(__name__)
app.config.from_object(MyConfig)
api = Api(app)
on a side note, if you are using vanilla flask, the process is simpler, just set your app json encoder directly (app.json_encoder = UUIDEncoder)
I hope it's useful to someone!

Using a custom JSON encoder for SQLAlchemy's PostgreSQL JSONB implementation

I am using SQLAlchemy's core library to access some PostgreSQL database. Consider I have the following table:
create table foo (j jsonb);
And the following python code:
from decimal import *
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey, DateTime
from sqlalchemy.dialects.postgresql import JSONB
metadata = MetaData(schema="public")
foo = Table('foo', metadata,Column('f', JSONB))
d = Decimal(2)
ins = foo.insert().values(j = {'d': d})
# assuming engine is a valid sqlalchemy's connection
engine.execute(ins)
This last sentence fails with the following error:
StatementError("(builtins.TypeError) Decimal('2') is not JSON serializable",)
Which is why I am asking this question: Is there a way to specify a custom encoder for SQLAchemy to use when encoding json data into PostgreSQL dialect?

This is supported via the json_serializer keyword argument to create_engine, as documented under sqlalchemy.dialects.postgresql.JSON:
def _default(val):
if isinstance(val, Decimal):
return str(val)
raise TypeError()
def dumps(d):
return json.dumps(d, default=_default)
engine = create_engine(..., json_serializer=dumps)

If you, like me, are finding a nice way to get this running with Flask-SQLAlchemy, this is what I did. If you import and pass flask.json instead of the standard library json module, you’ll get automatic deserialization of dates, datetimes and uuid.UUID instances.
class HackSQLAlchemy(SQLAlchemy):
""" Ugly way to get SQLAlchemy engine to pass the Flask JSON serializer
to `create_engine`.
See https://github.com/mitsuhiko/flask-sqlalchemy/pull/67/files
"""
def apply_driver_hacks(self, app, info, options):
options.update(json_serializer=json.dumps)
super(HackSQLAlchemy, self).apply_driver_hacks(app, info, options)

If you're using Flask, you already have an extended JSONEncoder defined in flask.json which handles UUID, but not Decimal. It can be mapped into the SqlAlchemy engine with the json_serializer param as in #univerio's answer:
from flask import json
engine = create_engine(
app.config['SQLALCHEMY_DATABASE_URI'],
convert_unicode=True,
json_serializer=json.dumps,
)
You can further extend the Flask JSONEncoder to support decimal.Decimal with the following:
import decimal
from flask import json
class CustomJSONEncoder(json.JSONEncoder):
"""
Override Flask's `JSONEncoder.default`, which is called
when the encoder doesn't handle a type.
"""
def default(self, o):
if isinstance(o, decimal.Decimal):
return str(o)
else:
# raises TypeError: o not JSON serializable
return json.JSONEncoder.default(self, o)
def init_json(app):
"""
Use custom JSON encoder with Flask
"""
app.json_encoder = CustomJSONEncoder

I found anwser here: https://github.com/flask-restful/flask-restful/issues/116#issuecomment-128419699 Summing it up, to run it with Flask-SQLAlchemy:
from flask import Flask, json
from decimal import Decimal
# define encoder
class JSONEncoder(json.JSONEncoder):
def default(self, value):
if isinstance(value, Decimal):
return str(value)
return json.JSONEncoder.default(self, value)
class Config:
RESTFUL_JSON = {}
# make sure RESTful and Flask encoders stay synchronized
#staticmethod
def init_app(app):
app.config['RESTFUL_JSON']['cls'] = app.json_encoder = JSONEncoder
app = Flask(__name__)
app.config.from_object(Config)
Config.init_app(app)

TypeError: Collection(Database(MongoClient("localhost", 27017), u'demo_database'), u'entries) is not JSON serializable

Good afternoon.
I'm trying to combine Python, MongoDB (via pymongo) and Flask to create client-server application. I want to use one of methods to return all the entire collection, like here:
#app.route('/entries', methods = ['GET'])
def get_entries():
client = MongoClient(db_host, db_port)
db_demo = client['demo_database']
entries = db_demo.entries
return JSONEncoder().encode(entries)
I also have an Encoder class, as advised here:
class JSONEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, ObjectId):
return str(o)
return json.JSONEncoder.default(self, o)
Data collection is very simple - actually, only one item with few fields. What am I doing wrong? Perhaps I should develop more sophisticated encoder class?

Use bson.json_util.dumps, which already supports all the MongoDB extended JSON types:
>>> from bson.json_util import dumps
>>> c.test.test.insert_many([{} for _ in range(3)])
<pymongo.results.InsertManyResult object at 0x7f6ed3189550>
>>> dumps(c.test.test.find())
'[{"_id": {"$oid": "554faa99fa5bd8782e1698cf"}}, {"_id": {"$oid": "554faa99fa5bd8782e1698d0"}}, {"_id": {"$oid": "554faa99fa5bd8782e1698d1"}}]'

Using a combination of both approaches, I prefer the solution that I provided here
from flask import Flask
from flask.json import JSONEncoder
from bson import json_util
from . import resources
# define a custom encoder point to the json_util provided by pymongo (or its dependency bson)
class CustomJSONEncoder(JSONEncoder):
def default(self, obj): return json_util.default(obj)
application = Flask(__name__)
application.json_encoder = CustomJSONEncoder
if __name__ == "__main__":
application.run()

Celery: is there a way to write custom JSON Encoder/Decoder?

I have some objects I want to send to celery tasks on my application. Those objects are obviously not json serializable using the default json library. Is there a way to make celery serialize/de-serialize those objects with custom JSON Encoder/Decoder?

A bit late here, but you should be able to define a custom encoder and decoder by registering them in the kombu serializer registry, as in the docs: http://docs.celeryproject.org/en/latest/userguide/calling.html#serializers.
For example, the following is a custom datetime serializer/deserializer (subclassing python's builtin json module) for Django:
myjson.py (put it in the same folder of your settings.py file)
import json
from datetime import datetime
from time import mktime
class MyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime):
return {
'__type__': '__datetime__',
'epoch': int(mktime(obj.timetuple()))
}
else:
return json.JSONEncoder.default(self, obj)
def my_decoder(obj):
if '__type__' in obj:
if obj['__type__'] == '__datetime__':
return datetime.fromtimestamp(obj['epoch'])
return obj
# Encoder function
def my_dumps(obj):
return json.dumps(obj, cls=MyEncoder)
# Decoder function
def my_loads(obj):
return json.loads(obj, object_hook=my_decoder)
settings.py
# Register your new serializer methods into kombu
from kombu.serialization import register
from .myjson import my_dumps, my_loads
register('myjson', my_dumps, my_loads,
content_type='application/x-myjson',
content_encoding='utf-8')
# Tell celery to use your new serializer:
CELERY_ACCEPT_CONTENT = ['myjson']
CELERY_TASK_SERIALIZER = 'myjson'
CELERY_RESULT_SERIALIZER = 'myjson'

Including repeated view function into multiple views?

I have the following function in many views of many of my apps. Like the following:
def json_response(data):
return HttpResponse(
simplejson.dumps(data),
content_type = 'application/json; charset=utf8'
)
How would I include this on all of my apps' views.py? Define it in a single app and just import from it?
from main.global import simplejson
Also, is there a github page of a well organized django project that I can look at?

If you are using django 1.3, a class based view can be used to abstract this function. You would simply extend your view from a base view that would return json of whatever is passed in. You would save a file with this class at some common location (as described in the answer linked in the comment by Ignacio).
In fact, this is one of the example types in the documentation for class based views:
from django import http
from django.utils import simplejson as json
class JSONResponseMixin(object):
def render_to_response(self, context):
"Returns a JSON response containing 'context' as payload"
return self.get_json_response(self.convert_context_to_json(context))
def get_json_response(self, content, **httpresponse_kwargs):
"Construct an `HttpResponse` object."
return http.HttpResponse(content,
content_type='application/json',
**httpresponse_kwargs)
def convert_context_to_json(self, context):
"Convert the context dictionary into a JSON object"
# Note: This is *EXTREMELY* naive; in reality, you'll need
# to do much more complex handling to ensure that arbitrary
# objects -- such as Django model instances or querysets
# -- can be serialized as JSON.
return json.dumps(context)
This is how you would use it (also from the documentation):
class HybridDetailView(JSONResponseMixin,
SingleObjectTemplateResponseMixin, BaseDetailView):
def render_to_response(self, context):
# Look for a 'format=json' GET argument
if self.request.GET.get('format','html') == 'json':
return JSONResponseMixin.render_to_response(self, context)
else:
return SingleObjectTemplateResponseMixin.render_to_response(self, context)

Yes, you can just define it in a single view, or a utils file, or whatever you want, and just import it in all of your views. I frequently do this with ubiquitous functions.