I have some Data that I need to write to a JSON string.
I have it working with dict items but want to encompass it all in classes to help ensure the correct data.
The following code is a comparison between the dict items and the class item output. They don't match, and I can't figure out what I am missing.
I get a "bound method Event.encode of Event..." in my JSON string.
from collections import namedtuple
import json
class Event(namedtuple('Event', 'itemName, itemID')):
def encode(self):
obj = {}
obj['itemName'] = str(self.itemName)
obj['itemID'] = int(self.itemID)
return json.dumps(obj)
curEv = Event('MyName', 5)
print 'ClassEv : ', curEv.encode()
curEv2 = {'itemName':'MyName', 'itemID':5}
print 'DictEv : ', json.dumps(curEv2)
class Packet(namedtuple('Packet', 'pID, itemType, itemData')):
def encode(self):
obj = {}
obj['pID'] = int(self.pID)
obj['itemType'] = int(self.itemType)
obj['itemData'] = str(self.itemData.encode)
return json.dumps(obj)
packet = Packet(11, 0, curEv)
print 'ClassPacket: ', packet.encode()
packet2 = {'pID':11, 'itemType':0}
packet2['itemData'] = curEv2
print 'DictPacket : ', json.dumps(packet2)
You are failing to call the itemData.encode() function. Instead you are simply returning a reference to it.
Try:
obj['itemData'] = str(self.itemData.encode())
Note the extra () at the end.
Related
I am writing a api to download json file. Suppose I have 3 dict:
{"task_name":"task1","task_info":"aaaaa"}
{"task_name":"task2","task_info":"bbbbb"}
{"task_name":"task3","task_info":"ccccc"}
I want to return these dict into one json file.Here is what I did:
data = json.dumps(tasks_info, default=str)
response = make_response(data, 200, {'mimetype': 'application/json'})
response.headers['Content-Disposition'] = "attachment;filename={}.json".format(urllib.parse.quote('result'))
return response
tasks_info is a list that contains 3 dict.
The result file open like that:
[{"task_name":"task1","task_info":"aaaaa"},{"task_name":"task2","task_info":"bbbbb"},{"task_name":"task3","task_info":"ccccc"}]
It is a very long line.
What I want to get looks like
[ // the '[' and ']' is not necessary
{"task_name":"task1","task_info":"aaaaa"},
{"task_name":"task2","task_info":"bbbbb"},
{"task_name":"task3","task_info":"ccccc"}
]
I want every dict show in a distinct line instead of every dict shown in same line.
Are there any way to change the result file look like?
The relevant command that formats this output takes place in json.dumps(). There are a bunch of commands you can give to format your json. A conventional approach is to define the indent level:
import json
tasks_info = [{"task_name":"task1","task_info":"aaaaa"},{"task_name":"task2","task_info":"bbbbb"},{"task_name":"task3","task_info":"ccccc"}]
data = json.dumps(tasks_info, indent=2)
print(data)
Which pretty-prints the json into:
[
{
"task_info": "aaaaa",
"task_name": "task1"
},
{
"task_info": "bbbbb",
"task_name": "task2"
},
{
"task_info": "ccccc",
"task_name": "task3"
}
]
To achieve the exact formatting you desire, you need to define your own formatter class. I used this answer and edited it to fit your formatting requirements. Here is the full code:
import _ctypes
import json
import re
class OneDictPerLine(object):
def __init__(self, value):
self.value = value
def __repr__(self):
if not isinstance(self.value, list):
return repr(self.value)
else: # Sort the representation of any dicts in the list.
reps = ('{{{}}}'.format(', '.join(
('{!r}: {!r}'.format(k, v) for k, v in sorted(v.items()))
)) if isinstance(v, dict)
else
repr(v) for v in self.value)
return '[ \n' + ',\n'.join(reps) + '\n ]'
def di(obj_id):
""" Reverse of id() function. """
# from https://stackoverflow.com/a/15012814/355230
return _ctypes.PyObj_FromPtr(obj_id)
class MyEncoder(json.JSONEncoder):
FORMAT_SPEC = "##{}##"
regex = re.compile(FORMAT_SPEC.format(r"(\d+)"))
def default(self, obj):
return (self.FORMAT_SPEC.format(id(obj)) if isinstance(obj, OneDictPerLine)
else super(MyEncoder, self).default(obj))
def encode(self, obj):
format_spec = self.FORMAT_SPEC # Local var to expedite access.
json_repr = super(MyEncoder, self).encode(obj) # Default JSON repr.
# Replace any marked-up object ids in the JSON repr with the value
# returned from the repr() of the corresponding Python object.
for match in self.regex.finditer(json_repr):
id = int(match.group(1))
# Replace marked-up id with actual Python object repr().
json_repr = json_repr.replace(
'"{}"'.format(format_spec.format(id)), repr(di(id)))
return json_repr
tasks_info = [{"task_name":"task1","task_info":"aaaaa"},{"task_name":"task2","task_info":"bbbbb"},{"task_name":"task3","task_info":"ccccc"}]
data = json.dumps(OneDictPerLine(tasks_info), cls=MyEncoder)
print(data)
#make response and return data ...
output:
[
{'task_info': 'aaaaa', 'task_name': 'task1'},
{'task_info': 'bbbbb', 'task_name': 'task2'},
{'task_info': 'ccccc', 'task_name': 'task3'}
]
In python, I receive JSON data. The data looks like the following stub:
{
"id": 1,
"naam": "4.13",
"ruimte_temperatuur_sensor": {...},
// etc
}
I map this json to an object (note the sensor is mapped already):
ruimte = Ruimte(id=id,
naam=naam,
ruimte_temperatuur_sensor=temperatuur_sensor,
ruimte_humiditeit_sensor=humiditeit_sensor,
ruimte_beweging_sensor=beweging_sensor,
airco_temperatuur_sensor=airco_sensor,
radiator_temperatuur_sensor=radiator_sensor)
The strangest thing happens:
The id field in JSON is an integer, but Python maps it to a tuple. In my debugger, you can clearly see that the id=id maps to an integer, but then all of a sudden my object contains a tuple:
The object's constructor should not cause that:
class Ruimte:
def __init__(self,
id: int,
naam: str,
ruimte_temperatuur_sensor: Sensor,
ruimte_humiditeit_sensor: Sensor,
ruimte_beweging_sensor: Sensor,
airco_temperatuur_sensor: Sensor,
radiator_temperatuur_sensor: Sensor):
self.id = id,
self.naam = naam,
self.ruimte_temperatuur_sensor = ruimte_temperatuur_sensor
self.ruimte_humiditeit_sensor = ruimte_humiditeit_sensor
self.ruimte_beweging_sensor = ruimte_beweging_sensor
self.airco_temperatuur_sensor = airco_temperatuur_sensor
self.radiator_temperatuur_sensor = radiator_temperatuur_sensor
In the sub-objects the id is not parsed to a tuple, for exampe ruimte.airco_temperatuur_sensor.id is an integer:
but that JSON is parsed the same way:
def _parse_json_to_sensor(self, json: dict) -> Sensor:
id = json["id"]
type = SensorType(json["type"])
meet_interval_sec = json["sensorInstelling"]["meetIntervalSec"]
opslaan_interval_sec = json["sensorInstelling"]["opslaanIntervalSec"]
sensor = Sensor(id=id,
type=type,
meet_interval_sec=meet_interval_sec,
opslaan_interval_sec=opslaan_interval_sec)
I'm totally lost on this. What could cause this?
You have commas after the lines where you assign self.id and self.naam. Remove them.
a_string = 'string',
type(a_string)
>>> tuple
The comma in the line:
self.id = id,
leads to the creation of a tuple. See this example:
a = 1
b = 1
c = 1,
print(b)
print(c)
So I'm trying to compare a dict that I have created to a dict response returned by a boto3 call.
The response is a representation of a JSON document and I want to check they are the same.
Boto3 always returned the strings as unicode. Here's the response:
{u'Version': u'2012-10-17', u'Statement': [{u'Action': u'sts:AssumeRole', u'Principal': {u'Service': u'ec2.amazonaws.com'}, u'Effect': u'Allow', u'Sid': u''}]}
I initially created my dict like this:
default_documment = {}
default_documment['Version'] = '2012-10-17'
default_documment['Statement'] = [{}]
default_documment['Statement'][0]['Sid'] = ''
default_documment['Statement'][0]['Effect'] = 'Allow'
default_documment['Statement'][0]['Principal'] = {}
default_documment['Statement'][0]['Principal']['Service'] = 'ec2.amazonaws.com'
default_documment['Statement'][0]['Action'] = 'sts:AssumeRole'
However, when i compare these two dicts with == they are not equal.
So then I tried adding u to all the strings when I create the dict:
# Default document for a new role
default_documment = {}
default_documment[u'Version'] = u'2012-10-17'
default_documment[u'Statement'] = [{}]
default_documment[u'Statement'][0][u'Sid'] = u''
default_documment[u'Statement'][0][u'Effect'] = u'Allow'
default_documment[u'Statement'][0][u'Principal'] = {}
default_documment[u'Statement'][0][u'Principal'][u'Service'] = u'ec2.amazonaws.com'
default_documment[u'Statement'][0][u'Action'] = u'sts:AssumeRole'
This doesn't work either. The dicts are not equally and if i do a print of my dict it doesn't show u'somestring' it just shows 'somestring'.
How can I compare my dict to what boto3 has returned?
Your second attempt works correctly in Python 2.7 and 3.3. Below is just a cut-and-paste of your Boto3 response and your code (with document spelling corrected :)
D = {u'Version': u'2012-10-17', u'Statement': [{u'Action': u'sts:AssumeRole', u'Principal': {u'Service': u'ec2.amazonaws.com'}, u'Effect': u'Allow', u'Sid': u''}]}
default_document = {}
default_document[u'Version'] = u'2012-10-17'
default_document[u'Statement'] = [{}]
default_document[u'Statement'][0][u'Sid'] = u''
default_document[u'Statement'][0][u'Effect'] = u'Allow'
default_document[u'Statement'][0][u'Principal'] = {}
default_document[u'Statement'][0][u'Principal'][u'Service'] = u'ec2.amazonaws.com'
default_document[u'Statement'][0][u'Action'] = u'sts:AssumeRole'
print(D == default_document)
Output:
True
I have some log files that look like many lines of the following:
<tickPrice tickerId=0, field=2, price=201.81, canAutoExecute=1>
<tickSize tickerId=0, field=3, size=25>
<tickSize tickerId=0, field=8, size=534349>
<tickPrice tickerId=0, field=2, price=201.82, canAutoExecute=1>
I need to define a class of type tickPrice or tickSize. I will need to decide which to use before doing the definition.
What would be the Pythonic way to grab these values? In other words, I need an effective way to reverse str() on a class.
The classes are already defined and just contain the presented variables, e.g., tickPrice.tickerId. I'm trying to find a way to extract these values from the text and set the instance attributes to match.
Edit: Answer
This is what I ended up doing-
with open(commandLineOptions.simulationFilename, "r") as simulationFileHandle:
for simulationFileLine in simulationFileHandle:
(date, time, msgString) = simulationFileLine.split("\t")
if ("tickPrice" in msgString):
msgStringCleaned = msgString.translate(None, ''.join("<>,"))
msgList = msgStringCleaned.split(" ")
msg = message.tickPrice()
msg.tickerId = int(msgList[1][9:])
msg.field = int(msgList[2][6:])
msg.price = float(msgList[3][6:])
msg.canAutoExecute = int(msgList[4][15:])
elif ("tickSize" in msgString):
msgStringCleaned = msgString.translate(None, ''.join("<>,"))
msgList = msgStringCleaned.split(" ")
msg = message.tickSize()
msg.tickerId = int(msgList[1][9:])
msg.field = int(msgList[2][6:])
msg.size = int(msgList[3][5:])
else:
print "Unsupported tick message type"
I'm not sure how you want to dynamically create objects in your namespace, but the following will at least dynamically create objects based on your loglines:
Take your line:
line = '<tickPrice tickerId=0, field=2, price=201.81, canAutoExecute=1>'
Remove chars that aren't interesting to us, then split the line into a list:
line = line.translate(None, ''.join('<>,'))
line = line.split(' ')
Name the potential class attributes for convenience:
line_attrs = line[1:]
Then create your object (name, base tuple, dictionary of attrs):
tickPriceObject = type(line[0], (object,), { key:value for key,value in [at.split('=') for at in line_attrs]})()
Prove it works as we'd expect:
print(tickPriceObject.field)
# 2
Approaching the problem with regex, but with the same result as tristan's excellent answer (and stealing his use of the type constructor that I will never be able to remember)
import re
class_instance_re = re.compile(r"""
<(?P<classname>\w[a-zA-Z0-9]*)[ ]
(?P<arguments>
(?:\w[a-zA-Z0-9]*=[0-9.]+[, ]*)+
)>""", re.X)
objects = []
for line in whatever_file:
result = class_instance_re.match(line)
classname = line.group('classname')
arguments = line.group('arguments')
new_obj = type(classname, (object,),
dict([s.split('=') for s in arguments.split(', ')]))
objects.append(new_obj)
I'm trying to serialize the result (a list) of an sqlalchemy query to json.
this is the class:
class Wikilink(Base):
__tablename__='Wikilinks'
__table_args__={'extend_existing':True}
id = Column(Integer,autoincrement=True,primary_key=True)
title = Column(Unicode(350))
user_ip = Column(String(50))
page = Column(String(20))
revision = Column(String(20))
timestamp = Column(String(50))
and I guess my problem is with the __repr__(self): function.
I tried something like:
return '{{0}:{"title":{1}, "Ip":{2}, "page":{3} ,"revision":{4}}}'.format(self.id,self.title.encode('utf-8'),self.user_ip,self.page,self.revision)
or:
return '{"id"={0}, "title"={1}, "Ip"={2}}'.format(self.id,self.title.encode('utf-8'),self.user_ip.encode('utf-8'),self.page,self.revision)
and I got:
TypeError(repr(o) + " is not JSON serializable")
ValueError: Single '}' encountered in format string
I tried:
return '{id=%d, title=%s, Ip=%s}'%(self.id,self.title.encode('utf-8'),self.user_ip.encode('utf-8'))
and I got:
TypeError: {id=8126, title=1 בדצמבר, Ip=147.237.70.106} is not JSON serializable
adding "" around (according to the JSON formatting) like this: "id"="%d", "title"="%s", "Ip"="%s" didn't help either.
I know this is supposed to be dead simple but I just can't get this right
actually bottle is handling the jsonification part automatically, but trying to call json.dumps on the result gives me the same errors.
Instead of trying to convert to json a string, you could define, for example, your own to_dict method that returns the dictionary structure it seems you're trying to create and, after that, generate the json from that structure:
>>> import json
>>> d = {'id':8126, 'title':u'1 בדצמבר', 'ip':'147.237.70.106'}
>>> json.dumps(d)
'{"ip": "147.237.70.106", "id": 8126, "title": "1 \\u05d1\\u05d3\\u05e6\\u05de\\u05d1\\u05e8"}'
I'm not sure I understand what you tried. Couldn't you build the dict and let json.dumps() do the work for you?
Something like:
>>> class Foo:
... id = 1
... title = 'my title'
... to_jsonize = ['id', 'title']
>>>
>>> dct = {name: getattr(Foo,name) for name in Foo.to_jsonize}
>>> import json
>>> json.dumps(dct)
'{"id": 1, "title": "my title"}'