I have some objects I want to send to celery tasks on my application. Those objects are obviously not json serializable using the default json library. Is there a way to make celery serialize/de-serialize those objects with custom JSON Encoder/Decoder?
A bit late here, but you should be able to define a custom encoder and decoder by registering them in the kombu serializer registry, as in the docs: http://docs.celeryproject.org/en/latest/userguide/calling.html#serializers.
For example, the following is a custom datetime serializer/deserializer (subclassing python's builtin json module) for Django:
myjson.py (put it in the same folder of your settings.py file)
import json
from datetime import datetime
from time import mktime
class MyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime):
return {
'__type__': '__datetime__',
'epoch': int(mktime(obj.timetuple()))
}
else:
return json.JSONEncoder.default(self, obj)
def my_decoder(obj):
if '__type__' in obj:
if obj['__type__'] == '__datetime__':
return datetime.fromtimestamp(obj['epoch'])
return obj
# Encoder function
def my_dumps(obj):
return json.dumps(obj, cls=MyEncoder)
# Decoder function
def my_loads(obj):
return json.loads(obj, object_hook=my_decoder)
settings.py
# Register your new serializer methods into kombu
from kombu.serialization import register
from .myjson import my_dumps, my_loads
register('myjson', my_dumps, my_loads,
content_type='application/x-myjson',
content_encoding='utf-8')
# Tell celery to use your new serializer:
CELERY_ACCEPT_CONTENT = ['myjson']
CELERY_TASK_SERIALIZER = 'myjson'
CELERY_RESULT_SERIALIZER = 'myjson'
Related
I have a flask-restful project that interfaces with some custom classes, containing uuid (uuid.UUID) types used as ids. There are a couple of api endpoints which return the object associated with the given id, parsed by flask as an UUID. The issue is that, when I return them as a json payload, I get the following exception:
UUID('…') is not JSON serializable
I want to have those uuids represented as strings to the final user, making the process seamless (the user can take the returned uuid and use it for his next api request).
In order to fix this problem, I had to put together suggestions from two different places:
first, I need to create a custom json encoder, which when dealing with uuids, returns their string representation. StackOverflow answer here
import json
from uuid import UUID
class UUIDEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, UUID):
# if the obj is uuid, we simply return the value of uuid
return str(obj) # <- notice I'm not returning obj.hex as the original answer
return json.JSONEncoder.default(self, obj)
second, I need to take this new encoder and set it as the flask-restful encoder used for the responses. GitHub answer here
class MyConfig(object):
RESTFUL_JSON = {'cls': MyCustomEncoder}
app = Flask(__name__)
app.config.from_object(MyConfig)
api = Api(app)
putting it together:
# ?: custom json encoder to be able to fix the UUID('…') is not JSON serializable
class UUIDEncoder(json.JSONEncoder):
def default(self, obj: Any) -> Any: # pylint:disable=arguments-differ
if isinstance(obj, UUID):
return str(obj) # <- notice I'm not returning obj.hex as the original answer
return json.JSONEncoder.default(self, obj)
# ?: api configuration to switch the json encoder
class MyConfig(object):
RESTFUL_JSON = {"cls": UUIDEncoder}
app = Flask(__name__)
app.config.from_object(MyConfig)
api = Api(app)
on a side note, if you are using vanilla flask, the process is simpler, just set your app json encoder directly (app.json_encoder = UUIDEncoder)
I hope it's useful to someone!
I am building an app using Django Rest Framework (Versions : django 3.1, djangorestframework 3.11.1, python 3.7)
I want to override the encoder used in JsonResponse.
An oversimplification of my problem:
from django.http import JsonResponse
from django.core.serializers.json import DjangoJSONEncoder
class CustomEncoder(DjangoJSONEncoder):
def default(self, o):
return o + 10
response = JsonResponse({"data": 1}, encoder=CustomEncoder)
What I expect from response.getvalue() is '{"data": 11}' but instead I get '{"data": 1}'.
What am I doing wrong ?
because DjangoJSONEncoder use for
JSONEncoder subclass that knows how to encode date/time, decimal types, and
UUIDs.
you must pass value like date/time, decimal types, or
UUIDs for your method override default be called.
Try change to
class CustomEncoder(DjangoJSONEncoder):
def default(self, o):
return int(o) + 10
response = JsonResponse({"data": Decimal(1)}, encoder=CustomEncoder)
it will return {"data": 11}
I have a model class defined as;
class SiteConfig(object):
def __init__(self,command,category,frequency,delay):
self.command=command
self.category=category
self.frequency=frequency
self.delay=delay
im calling this model from another class like;
from flask import Flask,request,jsonify
import httplib
from models.SiteConfig import *
app = Flask(__name__)
#app.route('/')
def index():
return "Endpoint is not configured", 500
#app.route('/SitePollingConfiguration',methods=['GET'])
def SitePollingConfiguration():
response = jsonify(statusCode=httplib.OK, config=BuildSiteConfig())
response.status_code = httplib.OK
return response
def BuildSiteConfig():
a= SiteConfig('20c','1',60,60)
print (a)
return a
if __name__ == '__main__':
app.run()
But im getting SiteConfig is not JSON serializable error.
How can i serialize the class object in python?
EDIT
i try to use existing flask json encoder;
modified my siteconfig class like;
from flask import Flask, jsonify
from flask.json import JSONEncoder
class JsonSerializable(object):
def toJson(self):
return JSONEncoder.default(self.__dict__)
class SiteConfig(JsonSerializable):
def __init__(self,command,category,frequency,delay):
self.command=command
self.category=category
self.frequency=frequency
self.delay=delay
And then calling in my main class like;
def ATGSitePollingConfiguration():
response = jsonify(statusCode=httplib.OK, config=BuildATGSiteConfig())
response.status_code = httplib.OK
return response
def BuildATGSiteConfig():
a= SiteConfig('20c','1',60,60)
print (a)
return a
But still serializable issue. Sorry im new to python;
can anyone provide simple code sample on how to achieve this?
By default, flask uses flask.json.JSONEncoder to encode objects as JSON. In order to jsonify your user-defined classes you will need to create a subclass of JSONEncoder that implements the serialisation of your class. How you implement the serialisation is up to you, but the easiest method is probably to convert your object into something that JSONEncoder already knows about (like a dict).
Try adding this code to your app. (You should use the first version of the SiteConfig class that you posted, not your modified one).
from flask.json import JSONEncoder
# A customized JSON encoder that knows about your SiteConfig class
class CustomJSONEncoder(JSONEncoder):
def default(self, obj):
if isinstance(obj, SiteConfig):
return obj.__dict__
return JSONEncoder.default(self, obj)
# Tell your flask app to use your customised JSON encoder
app.json_encoder = CustomJSONEncoder
With this you should get the response:
{
"config": {
"category": "1",
"command": "20c",
"delay": 60,
"frequency": 60
},
"statusCode": 200
}
See here for a similar example.
You may also consider jsonpickle:
import jsonpickle
frozen = jsonpickle.encode(a)
I am using SQLAlchemy's core library to access some PostgreSQL database. Consider I have the following table:
create table foo (j jsonb);
And the following python code:
from decimal import *
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey, DateTime
from sqlalchemy.dialects.postgresql import JSONB
metadata = MetaData(schema="public")
foo = Table('foo', metadata,Column('f', JSONB))
d = Decimal(2)
ins = foo.insert().values(j = {'d': d})
# assuming engine is a valid sqlalchemy's connection
engine.execute(ins)
This last sentence fails with the following error:
StatementError("(builtins.TypeError) Decimal('2') is not JSON serializable",)
Which is why I am asking this question: Is there a way to specify a custom encoder for SQLAchemy to use when encoding json data into PostgreSQL dialect?
This is supported via the json_serializer keyword argument to create_engine, as documented under sqlalchemy.dialects.postgresql.JSON:
def _default(val):
if isinstance(val, Decimal):
return str(val)
raise TypeError()
def dumps(d):
return json.dumps(d, default=_default)
engine = create_engine(..., json_serializer=dumps)
If you, like me, are finding a nice way to get this running with Flask-SQLAlchemy, this is what I did. If you import and pass flask.json instead of the standard library json module, you’ll get automatic deserialization of dates, datetimes and uuid.UUID instances.
class HackSQLAlchemy(SQLAlchemy):
""" Ugly way to get SQLAlchemy engine to pass the Flask JSON serializer
to `create_engine`.
See https://github.com/mitsuhiko/flask-sqlalchemy/pull/67/files
"""
def apply_driver_hacks(self, app, info, options):
options.update(json_serializer=json.dumps)
super(HackSQLAlchemy, self).apply_driver_hacks(app, info, options)
If you're using Flask, you already have an extended JSONEncoder defined in flask.json which handles UUID, but not Decimal. It can be mapped into the SqlAlchemy engine with the json_serializer param as in #univerio's answer:
from flask import json
engine = create_engine(
app.config['SQLALCHEMY_DATABASE_URI'],
convert_unicode=True,
json_serializer=json.dumps,
)
You can further extend the Flask JSONEncoder to support decimal.Decimal with the following:
import decimal
from flask import json
class CustomJSONEncoder(json.JSONEncoder):
"""
Override Flask's `JSONEncoder.default`, which is called
when the encoder doesn't handle a type.
"""
def default(self, o):
if isinstance(o, decimal.Decimal):
return str(o)
else:
# raises TypeError: o not JSON serializable
return json.JSONEncoder.default(self, o)
def init_json(app):
"""
Use custom JSON encoder with Flask
"""
app.json_encoder = CustomJSONEncoder
I found anwser here: https://github.com/flask-restful/flask-restful/issues/116#issuecomment-128419699 Summing it up, to run it with Flask-SQLAlchemy:
from flask import Flask, json
from decimal import Decimal
# define encoder
class JSONEncoder(json.JSONEncoder):
def default(self, value):
if isinstance(value, Decimal):
return str(value)
return json.JSONEncoder.default(self, value)
class Config:
RESTFUL_JSON = {}
# make sure RESTful and Flask encoders stay synchronized
#staticmethod
def init_app(app):
app.config['RESTFUL_JSON']['cls'] = app.json_encoder = JSONEncoder
app = Flask(__name__)
app.config.from_object(Config)
Config.init_app(app)
Good afternoon.
I'm trying to combine Python, MongoDB (via pymongo) and Flask to create client-server application. I want to use one of methods to return all the entire collection, like here:
#app.route('/entries', methods = ['GET'])
def get_entries():
client = MongoClient(db_host, db_port)
db_demo = client['demo_database']
entries = db_demo.entries
return JSONEncoder().encode(entries)
I also have an Encoder class, as advised here:
class JSONEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, ObjectId):
return str(o)
return json.JSONEncoder.default(self, o)
Data collection is very simple - actually, only one item with few fields. What am I doing wrong? Perhaps I should develop more sophisticated encoder class?
Use bson.json_util.dumps, which already supports all the MongoDB extended JSON types:
>>> from bson.json_util import dumps
>>> c.test.test.insert_many([{} for _ in range(3)])
<pymongo.results.InsertManyResult object at 0x7f6ed3189550>
>>> dumps(c.test.test.find())
'[{"_id": {"$oid": "554faa99fa5bd8782e1698cf"}}, {"_id": {"$oid": "554faa99fa5bd8782e1698d0"}}, {"_id": {"$oid": "554faa99fa5bd8782e1698d1"}}]'
Using a combination of both approaches, I prefer the solution that I provided here
from flask import Flask
from flask.json import JSONEncoder
from bson import json_util
from . import resources
# define a custom encoder point to the json_util provided by pymongo (or its dependency bson)
class CustomJSONEncoder(JSONEncoder):
def default(self, obj): return json_util.default(obj)
application = Flask(__name__)
application.json_encoder = CustomJSONEncoder
if __name__ == "__main__":
application.run()