I have a basic dict as follows:
sample = {}
sample['title'] = "String"
sample['somedate'] = somedatetimehere
When I try to do jsonify(sample) I get:
TypeError: datetime.datetime(2012, 8, 8, 21, 46, 24, 862000) is not JSON serializable
What can I do such that my dictionary sample can overcome the error above?
Note: Though it may not be relevant, the dictionaries are generated from the retrieval of records out of mongodb where when I print out str(sample['somedate']), the output is 2012-08-08 21:46:24.862000.
My quick & dirty JSON dump that eats dates and everything:
json.dumps(my_dictionary, indent=4, sort_keys=True, default=str)
default is a function applied to objects that aren't serializable.
In this case it's str, so it just converts everything it doesn't know to strings. Which is great for serialization but not so great when deserializing (hence the "quick & dirty") as anything might have been string-ified without warning, e.g. a function or numpy array.
Building on other answers, a simple solution based on a specific serializer that just converts datetime.datetime and datetime.date objects to strings.
from datetime import date, datetime
def json_serial(obj):
"""JSON serializer for objects not serializable by default json code"""
if isinstance(obj, (datetime, date)):
return obj.isoformat()
raise TypeError ("Type %s not serializable" % type(obj))
As seen, the code just checks to find out if object is of class datetime.datetime or datetime.date, and then uses .isoformat() to produce a serialized version of it, according to ISO 8601 format, YYYY-MM-DDTHH:MM:SS (which is easily decoded by JavaScript). If more complex serialized representations are sought, other code could be used instead of str() (see other answers to this question for examples). The code ends by raising an exception, to deal with the case it is called with a non-serializable type.
This json_serial function can be used as follows:
from datetime import datetime
from json import dumps
print dumps(datetime.now(), default=json_serial)
The details about how the default parameter to json.dumps works can be found in Section Basic Usage of the json module documentation.
Updated for 2018
The original answer accommodated the way MongoDB "date" fields were represented as:
{"$date": 1506816000000}
If you want a generic Python solution for serializing datetime to json, check out #jjmontes' answer for a quick solution which requires no dependencies.
As you are using mongoengine (per comments) and pymongo is a dependency, pymongo has built-in utilities to help with json serialization:
http://api.mongodb.org/python/1.10.1/api/bson/json_util.html
Example usage (serialization):
from bson import json_util
import json
json.dumps(anObject, default=json_util.default)
Example usage (deserialization):
json.loads(aJsonString, object_hook=json_util.object_hook)
Django
Django provides a native DjangoJSONEncoder serializer that deals with this kind of properly.
See https://docs.djangoproject.com/en/dev/topics/serialization/#djangojsonencoder
from django.core.serializers.json import DjangoJSONEncoder
return json.dumps(
item,
sort_keys=True,
indent=1,
cls=DjangoJSONEncoder
)
One difference I've noticed between DjangoJSONEncoder and using a custom default like this:
import datetime
import json
def default(o):
if isinstance(o, (datetime.date, datetime.datetime)):
return o.isoformat()
return json.dumps(
item,
sort_keys=True,
indent=1,
default=default
)
Is that Django strips a bit of the data:
"last_login": "2018-08-03T10:51:42.990", # DjangoJSONEncoder
"last_login": "2018-08-03T10:51:42.990239", # default
So, you may need to be careful about that in some cases.
I have just encountered this problem and my solution is to subclass json.JSONEncoder:
from datetime import datetime
import json
class DateTimeEncoder(json.JSONEncoder):
def default(self, o):
if isinstance(o, datetime):
return o.isoformat()
return json.JSONEncoder.default(self, o)
In your call do something like: json.dumps(yourobj, cls=DateTimeEncoder) The .isoformat() I got from one of the answers above.
Convert the date to a string
sample['somedate'] = str( datetime.utcnow() )
For others who do not need or want to use the pymongo library for this.. you can achieve datetime JSON conversion easily with this small snippet:
def default(obj):
"""Default JSON serializer."""
import calendar, datetime
if isinstance(obj, datetime.datetime):
if obj.utcoffset() is not None:
obj = obj - obj.utcoffset()
millis = int(
calendar.timegm(obj.timetuple()) * 1000 +
obj.microsecond / 1000
)
return millis
raise TypeError('Not sure how to serialize %s' % (obj,))
Then use it like so:
import datetime, json
print json.dumps(datetime.datetime.now(), default=default)
output:
'1365091796124'
Here is my solution:
import json
class DatetimeEncoder(json.JSONEncoder):
def default(self, obj):
try:
return super().default(obj)
except TypeError:
return str(obj)
Then you can use it like that:
json.dumps(dictionnary, cls=DatetimeEncoder)
if you are using python3.7, then the best solution is using
datetime.isoformat() and
datetime.fromisoformat(); they work with both naive and
aware datetime objects:
#!/usr/bin/env python3.7
from datetime import datetime
from datetime import timezone
from datetime import timedelta
import json
def default(obj):
if isinstance(obj, datetime):
return { '_isoformat': obj.isoformat() }
raise TypeError('...')
def object_hook(obj):
_isoformat = obj.get('_isoformat')
if _isoformat is not None:
return datetime.fromisoformat(_isoformat)
return obj
if __name__ == '__main__':
#d = { 'now': datetime(2000, 1, 1) }
d = { 'now': datetime(2000, 1, 1, tzinfo=timezone(timedelta(hours=-8))) }
s = json.dumps(d, default=default)
print(s)
print(d == json.loads(s, object_hook=object_hook))
output:
{"now": {"_isoformat": "2000-01-01T00:00:00-08:00"}}
True
if you are using python3.6 or below, and you only care about the time value (not
the timezone), then you can use datetime.timestamp() and
datetime.fromtimestamp() instead;
if you are using python3.6 or below, and you do care about the timezone, then
you can get it via datetime.tzinfo, but you have to serialize this field
by yourself; the easiest way to do this is to add another field _tzinfo in the
serialized object;
finally, beware of precisions in all these examples;
You should apply .strftime() method on .datetime.now() method to make it as a serializable method.
Here's an example:
from datetime import datetime
time_dict = {'time': datetime.now().strftime('%Y-%m-%dT%H:%M:%S')}
sample_dict = {'a': 1, 'b': 2}
sample_dict.update(time_dict)
sample_dict
Output:
Out[0]: {'a': 1, 'b': 2, 'time': '2017-10-31T15:16:30'}
[UPDATE]:
In Python3.7 and later, you can simply use the .isoformat() method:
from datetime import datetime
datetime.now().isoformat()
I have an application with a similar issue; my approach was to JSONize the datetime value as a 6-item list (year, month, day, hour, minutes, seconds); you could go to microseconds as a 7-item list, but I had no need to:
class DateTimeEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
encoded_object = list(obj.timetuple())[0:6]
else:
encoded_object =json.JSONEncoder.default(self, obj)
return encoded_object
sample = {}
sample['title'] = "String"
sample['somedate'] = datetime.datetime.now()
print sample
print json.dumps(sample, cls=DateTimeEncoder)
produces:
{'somedate': datetime.datetime(2013, 8, 1, 16, 22, 45, 890000), 'title': 'String'}
{"somedate": [2013, 8, 1, 16, 22, 45], "title": "String"}
The json.dumps method can accept an optional parameter called default which is expected to be a function. Every time JSON tries to convert a value it does not know how to convert it will call the function we passed to it. The function will receive the object in question, and it is expected to return the JSON representation of the object.
def myconverter(o):
if isinstance(o, datetime.datetime):
return o.__str__()
print(json.dumps(d, default = myconverter))
My solution (with less verbosity, I think):
def default(o):
if type(o) is datetime.date or type(o) is datetime.datetime:
return o.isoformat()
def jsondumps(o):
return json.dumps(o, default=default)
Then use jsondumps instead of json.dumps. It will print:
>>> jsondumps({'today': datetime.date.today()})
'{"today": "2013-07-30"}'
I you want, later you can add other special cases to this with a simple twist of the default method. Example:
def default(o):
if type(o) is datetime.date or type(o) is datetime.datetime:
return o.isoformat()
if type(o) is decimal.Decimal:
return float(o)
This Q repeats time and time again - a simple way to patch the json module such that serialization would support datetime.
import json
import datetime
json.JSONEncoder.default = lambda self,obj: (obj.isoformat() if isinstance(obj, datetime.datetime) else None)
Than use json serialization as you always do - this time with datetime being serialized as isoformat.
json.dumps({'created':datetime.datetime.now()})
Resulting in: '{"created": "2015-08-26T14:21:31.853855"}'
See more details and some words of caution at:
StackOverflow: JSON datetime between Python and JavaScript
You have to supply a custom encoder class with the cls parameter of json.dumps. To quote from the docs:
>>> import json
>>> class ComplexEncoder(json.JSONEncoder):
... def default(self, obj):
... if isinstance(obj, complex):
... return [obj.real, obj.imag]
... return json.JSONEncoder.default(self, obj)
...
>>> dumps(2 + 1j, cls=ComplexEncoder)
'[2.0, 1.0]'
>>> ComplexEncoder().encode(2 + 1j)
'[2.0, 1.0]'
>>> list(ComplexEncoder().iterencode(2 + 1j))
['[', '2.0', ', ', '1.0', ']']
This uses complex numbers as the example, but you can just as easily create a class to encode dates (except I think JSON is a little fuzzy about dates)
Here is a simple solution to over come "datetime not JSON serializable"
problem.
enco = lambda obj: (
obj.isoformat()
if isinstance(obj, datetime.datetime)
or isinstance(obj, datetime.date)
else None
)
json.dumps({'date': datetime.datetime.now()}, default=enco)
Output:-> {"date": "2015-12-16T04:48:20.024609"}
The simplest way to do this is to change the part of the dict that is in datetime format to isoformat. That value will effectively be a string in isoformat which json is ok with.
v_dict = version.dict()
v_dict['created_at'] = v_dict['created_at'].isoformat()
Actually it is quite simple.
If you need to often serialize dates, then work with them as strings. You can easily convert them back as datetime objects if needed.
If you need to work mostly as datetime objects, then convert them as strings before serializing.
import json, datetime
date = str(datetime.datetime.now())
print(json.dumps(date))
"2018-12-01 15:44:34.409085"
print(type(date))
<class 'str'>
datetime_obj = datetime.datetime.strptime(date, '%Y-%m-%d %H:%M:%S.%f')
print(datetime_obj)
2018-12-01 15:44:34.409085
print(type(datetime_obj))
<class 'datetime.datetime'>
As you can see, the output is the same in both cases. Only the type is different.
Try this one with an example to parse it:
#!/usr/bin/env python
import datetime
import json
import dateutil.parser # pip install python-dateutil
class JSONEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, datetime.datetime):
return obj.isoformat()
return super(JSONEncoder, self).default(obj)
def test():
dts = [
datetime.datetime.now(),
datetime.datetime.now(datetime.timezone(-datetime.timedelta(hours=4))),
datetime.datetime.utcnow(),
datetime.datetime.now(datetime.timezone.utc),
]
for dt in dts:
dt_isoformat = json.loads(json.dumps(dt, cls=JSONEncoder))
dt_parsed = dateutil.parser.parse(dt_isoformat)
assert dt == dt_parsed
print(f'{dt}, {dt_isoformat}, {dt_parsed}')
# 2018-07-22 02:22:42.910637, 2018-07-22T02:22:42.910637, 2018-07-22 02:22:42.910637
# 2018-07-22 02:22:42.910643-04:00, 2018-07-22T02:22:42.910643-04:00, 2018-07-22 02:22:42.910643-04:00
# 2018-07-22 06:22:42.910645, 2018-07-22T06:22:42.910645, 2018-07-22 06:22:42.910645
# 2018-07-22 06:22:42.910646+00:00, 2018-07-22T06:22:42.910646+00:00, 2018-07-22 06:22:42.910646+00:00
if __name__ == '__main__':
test()
If you are working with django models you can directly pass encoder=DjangoJSONEncoder to the field constructor. It will work like a charm.
from django.core.serializers.json import DjangoJSONEncoder
from django.db import models
from django.utils.timezone import now
class Activity(models.Model):
diff = models.JSONField(null=True, blank=True, encoder=DjangoJSONEncoder)
diff = {
"a": 1,
"b": "BB",
"c": now()
}
Activity.objects.create(diff=diff)
Generally there are several ways to serialize datetimes, like:
ISO string, short and can include timezone info, e.g. #jgbarah's answer
Timestamp (timezone data is lost), e.g. #JayTaylor's answer
Dictionary of properties (including timezone).
If you're okay with the last way, the json_tricks package handles dates, times and datetimes including timezones.
from datetime import datetime
from json_tricks import dumps
foo = {'title': 'String', 'datetime': datetime(2012, 8, 8, 21, 46, 24, 862000)}
dumps(foo)
which gives:
{"title": "String", "datetime": {"__datetime__": null, "year": 2012, "month": 8, "day": 8, "hour": 21, "minute": 46, "second": 24, "microsecond": 862000}}
So all you need to do is
`pip install json_tricks`
and then import from json_tricks instead of json.
The advantage of not storing it as a single string, int or float comes when decoding: if you encounter just a string or especially int or float, you need to know something about the data to know if it's a datetime. As a dict, you can store metadata so it can be decoded automatically, which is what json_tricks does for you. It's also easily editable for humans.
Disclaimer: it's made by me. Because I had the same problem.
I usually use orjson. Not only because of its tremendous performance, but also for its great (RFC-3339 compliant) support of datetime:
import orjson # via pip3 install orjson
from datetime import datetime
data = {"created_at": datetime(2022, 3, 1)}
orjson.dumps(data) # returns b'{"created_at":"2022-03-01T00:00:00"}'
If you would like to use datetime.datetime objects without a tzinfo as UTC you can add the related option:
orjson.dumps(data, option=orjson.OPT_NAIVE_UTC) # returns b'{"created_at":"2022-03-01T00:00:00+00:00"}'
If you are using the result in a view be sure to return a proper response. According to the API, jsonify does the following:
Creates a Response with the JSON representation of the given arguments
with an application/json mimetype.
To mimic this behavior with json.dumps you have to add a few extra lines of code.
response = make_response(dumps(sample, cls=CustomEncoder))
response.headers['Content-Type'] = 'application/json'
response.headers['mimetype'] = 'application/json'
return response
You should also return a dict to fully replicate jsonify's response. So, the entire file will look like this
from flask import make_response
from json import JSONEncoder, dumps
class CustomEncoder(JSONEncoder):
def default(self, obj):
if set(['quantize', 'year']).intersection(dir(obj)):
return str(obj)
elif hasattr(obj, 'next'):
return list(obj)
return JSONEncoder.default(self, obj)
#app.route('/get_reps/', methods=['GET'])
def get_reps():
sample = ['some text', <datetime object>, 123]
response = make_response(dumps({'result': sample}, cls=CustomEncoder))
response.headers['Content-Type'] = 'application/json'
response.headers['mimetype'] = 'application/json'
return response
My solution ...
from datetime import datetime
import json
from pytz import timezone
import pytz
def json_dt_serializer(obj):
"""JSON serializer, by macm.
"""
rsp = dict()
if isinstance(obj, datetime):
rsp['day'] = obj.day
rsp['hour'] = obj.hour
rsp['microsecond'] = obj.microsecond
rsp['minute'] = obj.minute
rsp['month'] = obj.month
rsp['second'] = obj.second
rsp['year'] = obj.year
rsp['tzinfo'] = str(obj.tzinfo)
return rsp
raise TypeError("Type not serializable")
def json_dt_deserialize(obj):
"""JSON deserialize from json_dt_serializer, by macm.
"""
if isinstance(obj, str):
obj = json.loads(obj)
tzone = timezone(obj['tzinfo'])
tmp_dt = datetime(obj['year'],
obj['month'],
obj['day'],
hour=obj['hour'],
minute=obj['minute'],
second=obj['second'],
microsecond=obj['microsecond'])
loc_dt = tzone.localize(tmp_dt)
deserialize = loc_dt.astimezone(tzone)
return deserialize
Ok, now some tests.
# Tests
now = datetime.now(pytz.utc)
# Using this solution
rsp = json_dt_serializer(now)
tmp = json_dt_deserialize(rsp)
assert tmp == now
assert isinstance(tmp, datetime) == True
assert isinstance(now, datetime) == True
# using default from json.dumps
tmp = json.dumps(datetime.now(pytz.utc), default=json_dt_serializer)
rsp = json_dt_deserialize(tmp)
assert isinstance(rsp, datetime) == True
# Lets try another timezone
eastern = timezone('US/Eastern')
now = datetime.now(eastern)
rsp = json_dt_serializer(now)
tmp = json_dt_deserialize(rsp)
print(tmp)
# 2015-10-22 09:18:33.169302-04:00
print(now)
# 2015-10-22 09:18:33.169302-04:00
# Wow, Works!
assert tmp == now
Here is my full solution for converting datetime to JSON and back..
import calendar, datetime, json
def outputJSON(obj):
"""Default JSON serializer."""
if isinstance(obj, datetime.datetime):
if obj.utcoffset() is not None:
obj = obj - obj.utcoffset()
return obj.strftime('%Y-%m-%d %H:%M:%S.%f')
return str(obj)
def inputJSON(obj):
newDic = {}
for key in obj:
try:
if float(key) == int(float(key)):
newKey = int(key)
else:
newKey = float(key)
newDic[newKey] = obj[key]
continue
except ValueError:
pass
try:
newDic[str(key)] = datetime.datetime.strptime(obj[key], '%Y-%m-%d %H:%M:%S.%f')
continue
except TypeError:
pass
newDic[str(key)] = obj[key]
return newDic
x = {'Date': datetime.datetime.utcnow(), 34: 89.9, 12.3: 90, 45: 67, 'Extra': 6}
print x
with open('my_dict.json', 'w') as fp:
json.dump(x, fp, default=outputJSON)
with open('my_dict.json') as f:
my_dict = json.load(f, object_hook=inputJSON)
print my_dict
Output
{'Date': datetime.datetime(2013, 11, 8, 2, 30, 56, 479727), 34: 89.9, 45: 67, 12.3: 90, 'Extra': 6}
{'Date': datetime.datetime(2013, 11, 8, 2, 30, 56, 479727), 34: 89.9, 45: 67, 12.3: 90, 'Extra': 6}
JSON File
{"Date": "2013-11-08 02:30:56.479727", "34": 89.9, "45": 67, "12.3": 90, "Extra": 6}
This has enabled me to import and export strings, ints, floats and datetime objects.
It shouldn't be to hard to extend for other types.
Convert the date to string
date = str(datetime.datetime(somedatetimehere))
As per the #jjmontes answer, I have used the following approach.
For flask and flask-restful users
# get json string
jsonStr = json.dumps(my_dictionary, indent=1, sort_keys=True, default=str)
# then covert json string to json object
return json.loads(jsonStr)
I got the same error message while writing the serialize decorator inside a Class with sqlalchemy. So instead of :
Class Puppy(Base):
...
#property
def serialize(self):
return { 'id':self.id,
'date_birth':self.date_birth,
...
}
I simply borrowed jgbarah's idea of using isoformat() and appended the original value with isoformat(), so that it now looks like:
...
'date_birth':self.date_birth.isoformat(),
...
A quick fix if you want your own formatting
for key,val in sample.items():
if isinstance(val, datetime):
sample[key] = '{:%Y-%m-%d %H:%M:%S}'.format(val) #you can add different formating here
json.dumps(sample)
If you are on both sides of the communication you can use repr() and eval() functions along with json.
import datetime, json
dt = datetime.datetime.now()
print("This is now: {}".format(dt))
dt1 = json.dumps(repr(dt))
print("This is serialised: {}".format(dt1))
dt2 = json.loads(dt1)
print("This is loaded back from json: {}".format(dt2))
dt3 = eval(dt2)
print("This is the same object as we started: {}".format(dt3))
print("Check if they are equal: {}".format(dt == dt3))
You shouldn't import datetime as
from datetime import datetime
since eval will complain. Or you can pass datetime as a parameter to eval. In any case this should work.
I had encountered same problem when externalizing django model object to dump as JSON.
Here is how you can solve it.
def externalize(model_obj):
keys = model_obj._meta.get_all_field_names()
data = {}
for key in keys:
if key == 'date_time':
date_time_obj = getattr(model_obj, key)
data[key] = date_time_obj.strftime("%A %d. %B %Y")
else:
data[key] = getattr(model_obj, key)
return data
Playing with new Google App Engine MapReduce library filters for input_reader I would like to know how can I filter by ndb.Key.
I read this post and I've played with datetime, string, int, float, in filters tuples, but How I can filter by ndb.Key?
When I try to filter by a ndb.Key I get this error:
BadReaderParamsError: Expected Key, got u"Key('Clients', 406)"
Or this error:
TypeError: Key('Clients', 406) is not JSON serializable
I tried to pass a ndb.Key object and string representation of the ndb.Key.
Here are my two filters tuples:
Sample 1:
input_reader': {
'input_reader': 'mapreduce.input_readers.DatastoreInputReader',
'entity_kind': 'model.Sales',
'filters': [("client","=", ndb.Key('Clients', 406))]
}
Sample 2:
input_reader': {
'input_reader': 'mapreduce.input_readers.DatastoreInputReader',
'entity_kind': 'model.Sales',
'filters': [("client","=", "%s" % ndb.Key('Clients', 406))]
}
This is a bit tricky.
If you look at the code on Google Code you can see that mapreduce.model defines a JSON_DEFAULTS dict which determines the classes that get special-case handling in JSON serialization/deserialization: by default, just datetime. So, you can monkey-patch the ndb.Key class into there, and provide it with functions to do that serialization/deserialization - something like:
from mapreduce import model
def _JsonEncodeKey(o):
"""Json encode an ndb.Key object."""
return {'key_string': o.urlsafe()}
def _JsonDecodeKey(d):
"""Json decode a ndb.Key object."""
return ndb.Key(urlsafe=d['key_string'])
model.JSON_DEFAULTS[ndb.Key] = (_JsonEncodeKey, _JsonDecodeKey)
model._TYPE_IDS['Key'] = ndb.Key
You may also need to repeat those last two lines to patch mapreduce.lib.pipeline.util as well.
Also note if you do this, you'll need to ensure that this gets run on any instance that runs any part of a mapreduce: the easiest way to do this is to write a wrapper script that imports the above registration code, as well as mapreduce.main.APP, and override the mapreduce URL in your app.yaml to point to your wrapper.
Make your own input reader based on DatastoreInputReader, which knows how to decode key-based filters:
class DatastoreKeyInputReader(input_readers.DatastoreKeyInputReader):
"""Augment the base input reader to accommodate ReferenceProperty filters"""
def __init__(self, *args, **kwargs):
try:
filters = kwargs['filters']
decoded = []
for f in filters:
value = f[2]
if isinstance(value, list):
value = db.Key.from_path(*value)
decoded.append((f[0], f[1], value))
kwargs['filters'] = decoded
except KeyError:
pass
super(DatastoreKeyInputReader, self).__init__(*args, **kwargs)
Run this function on your filters before passing them in as options:
def encode_filters(filters):
if filters is not None:
encoded = []
for f in filters:
value = f[2]
if isinstance(value, db.Model):
value = value.key()
if isinstance(value, db.Key):
value = value.to_path()
entry = (f[0], f[1], value)
encoded.append(entry)
filters = encoded
return filters
Are you aware of the to_old_key() and from_old_key() methods?
I had the same problem and came up with a workaround with computed properties.
You can add to your Sales model a new ndb.ComputedProperty with the Key id. Ids are just strings, so you wont have any JSON problems.
client_id = ndb.ComputedProperty(lambda self: self.client.id())
And then add that condition to your mapreduce query filters
input_reader': {
'input_reader': 'mapreduce.input_readers.DatastoreInputReader',
'entity_kind': 'model.Sales',
'filters': [("client_id","=", '406']
}
The only drawback is that Computed properties are not indexed and stored until you call the put() parameter, so you will have to traverse all the Sales entities and save them:
for sale in Sales.query().fetch():
sale.put()