I have a JSON file as follows :
{
"desired":{
"property1":{
"port":"/dev/usbserial",
"rx":{
"watchdoginterval":3600
},
"state":{
"path":"/Users/user1"
},
"enabled":"true",
"active":{
"enabled":"true"
}
},
"property2":{
"signal_interrupt":"USR2",
"signal_description_path":"/tmp/logger.log"
},
"property3":{
"periodmins":40
},
}
}
I am having issues trying to convert this into a string for use with AWS IoT. The function I am using is deviceShadowHandler.shadowUpdate(JSONPayload, customShadowCallback_Update, 5)
Where JSONPayload should be the JSON string.
I have tried :
with open('JSONfile.json' , 'r') as f:
dict = json.load(f)
JSONPayload = str(dict)
but I receive an "Invalid JSON file error".
An attempt to manually create a literal string from the jSON file gets messy with complaints about "EOL while scanning string literal" etc.
What is the best solution to solve this? I am new to JSON and stuff and Python.
Trailing commas are not allowed in JSON.
{
"desired":{
"property1":{
"port":"/dev/usbserial",
"rx":{
"watchdoginterval":3600
},
"state":{
"path":"/Users/user1"
},
"enabled":"true",
"active":{
"enabled":"true"
}
},
"property2":{
"signal_interrupt":"USR2",
"signal_description_path":"/tmp/logger.log"
},
"property3":{
"periodmins":40
} # <- no comma there
}
}
Related
I'm new to Python and I'm quite stuck (I've gone through multiple other stackoverflows and other sites and still can't get this to work).
I've the below json coming out of an API connection
{
"results":[
{
"group":{
"mediaType":"chat",
"queueId":"67d9fb5e-26b2-4db5-b062-bbcfa8d2ca0d"
},
"data":[
{
"interval":"2021-01-14T13:12:19.000Z/2022-01-14T13:12:19.000Z",
"metrics":[
{
"metric":"nOffered",
"qualifier":null,
"stats":{
"max":null,
"min":null,
"count":14,
"count_negative":null,
"count_positive":null,
"sum":null,
"current":null,
"ratio":null,
"numerator":null,
"denominator":null,
"target":null
}
}
],
"views":null
}
]
}
]
}
and what I'm mainly looking to get out of it is (or at least something as close as)
MediaType
QueueId
NOffered
Chat
67d9fb5e-26b2-4db5-b062-bbcfa8d2ca0d
14
Is something like that possible? I've tried multiple things and I either get the whole of this out in one line or just get different errors.
The error you got indicates you missed that some of your values are actually a dictionary within an array.
Assuming you want to flatten your json file to retrieve the following keys: mediaType, queueId, count.
These can be retrieved by the following sample code:
import json
with open(path_to_json_file, 'r') as f:
json_dict = json.load(f)
for result in json_dict.get("results"):
media_type = result.get("group").get("mediaType")
queue_id = result.get("group").get("queueId")
n_offered = result.get("data")[0].get("metrics")[0].get("count")
If your data and metrics keys will have multiple indices you will have to use a for loop to retrieve every count value accordingly.
Assuming that the format of the API response is always the same, have you considered hardcoding the extraction of the data you want?
This should work: With response defined as the API output:
response = {
"results":[
{
"group":{
"mediaType":"chat",
"queueId":"67d9fb5e-26b2-4db5-b062-bbcfa8d2ca0d"
},
"data":[
{
"interval":"2021-01-14T13:12:19.000Z/2022-01-14T13:12:19.000Z",
"metrics":[
{
"metric":"nOffered",
"qualifier":'null',
"stats":{
"max":'null',
"min":'null',
"count":14,
"count_negative":'null',
"count_positive":'null',
"sum":'null',
"current":'null',
"ratio":'null',
"numerator":'null',
"denominator":'null',
"target":'null'
}
}
],
"views":'null'
}
]
}
]
}
You can extract the results as follows:
results = response["results"][0]
{
"mediaType": results["group"]["mediaType"],
"queueId": results["group"]["queueId"],
"nOffered": results["data"][0]["metrics"][0]["stats"]["count"]
}
which gives
{
'mediaType': 'chat',
'queueId': '67d9fb5e-26b2-4db5-b062-bbcfa8d2ca0d',
'nOffered': 14
}
Updated: The XHR response was not correct earlier
I'm failing with flatten my json in a correct way from a XHR-response.
I have just expanded one item below, to make it more readable.
I am using python and I have tried, with incorrect outcome.
u = "URL"
SE_units = requests.get(u,headers=h).json()
dp = pd.json_normalize(SE_units,[SE_units,"Items"])
SE_dp_list.append(dp)
From the XHR-Response below I would like to have the Items-information into a CSV but when i do export.to_CSV I see that it haven't been flattened correctly
{"Content":{
"PaginationCount":12,"FilterValues":null,"Items":
[{
"Id":258370,
"OriginalType":"BostadObjectPage",
"PublishDate":null,
"Title":"02 Skogsvagen",
"Image":
{
"description":null,
"alt":null,
"externalUrl":"/abc.jpg"
},
"StaticMapImage":null,
"Url":"/abcd/",
"HideReadMore":false,
"ProjectData":null,
"ObjectData":
{
"BuildingTypeLabel":"Rad-/Kedje-/Parhus",
"ObjectStatus":"SalesInProgress",
"ObjectStatusLabel":"Till salu",
"ObjectNumber":"02",
"City":"staden",
"RoomInterval":"2-3",
"LivingArea":"101",
"SalesPrice":"2 150 000",
"MonthlyFee":null,
"Elevator":false,
"Balcony":false,
"Terrace":true
},
"FastighetProjectData":null,
"FastighetObjectData":null,
"OfficeData":null
},
{
"Id":258372,
"OriginalType":"BostadObjectPage",
"PublishDate":null,
....."same structure as above"
"OfficeData":null
}],
"NoResultsMessage":null,
"SimplifiedBuildingType":null,
"NextIndex":-1,
"TotalCount":12,
"Heading":null,
"ShowMoreLabel":null,
"DataColumns":null,
"Error":null},
"ObjectSearchData":
{
"BuildingVariantId":"Houses",
"BuildingsFoundLabel":" {count}",
"BuildingTypeIds":[400],
"BuildingsAvailableForSale":12,
"BuildingNoResultsLabel":""
}
}
Expected output format after writing to CSV
I have a problem with a nested json in python script, i need to reproduce the following jq query:
cat inventory.json | jq '.hostvars[] | [.openstack.hostname, .openstack.accessIPv4]'
the json file has a structure like this:
{
"hostvars": {
"096b430e-20f0-4655-bb97-9bb3ab2db73c": {
"openstack": {
"accessIPv4": "192.168.3.6",
"hostname": "vm-1"
}
}
"8fb7b9b7-5ccc-47c8-addf-64563fdd0d4c": {
"openstack": {
"accessIPv4": "192.168.3.7",
"hostname": "vm-2"
}
}
}
}
and the query with jq gives me the correct output:
# cat test.json | jq '.hostvars[] | [.openstack.hostname, .openstack.accessIPv4]'
[
"vm-1",
"192.168.3.6"
]
[
"vm-2",
"192.168.3.7"
]
Now i want reproduce this in python, to handle the individual values in variable but I can't parse the contents of each id, what with jq i do with .hostvars [].
with open('inventory.json', 'r') as inv:
data=inv.read()
obj=json.loads(data)
objh=obj['hostvars'][096b430e-20f0-4655-bb97-9bb3ab2db73c]['openstack']
print(objh)
Calling the id works, but if I replace it with 0 or [] I have a syntax error.
Serializing JSON Data
I think when you are dealing with json python you should use convert to Serializing:
The json module exposes two methods for serializing Python objects into JSON format.
dump() will write Python data to a file-like object. We use this when we want to serialize our Python data to an external JSON file.
dumps() will write Python data to a string in JSON format. This is useful if we want to use the JSON elsewhere in our program, or if we just want to print it to the console to check that it’s correct.
Both the dump() and dumps() methods allow us to specify an optional indent argument. This will change how many spaces is used for indentation, which can make our JSON easier to read.
json_str = json.dumps(data, indent=4)
for exampel:
import json
data={"user":{
"name":"CodeView",
"age":29
}
}
with open("data_file.json","w")as write_file:
json.dump(data,write_file)
json_str=json.dumps(data)
print(json_str)
json_data = {
"hostvars": {
"096b430e-20f0-4655-bb97-9bb3ab2db73c": {
"openstack": {
"accessIPv4": "192.168.3.6",
"hostname": "vm-1"
}
},
"8fb7b9b7-5ccc-47c8-addf-64563fdd0d4c": {
"openstack": {
"accessIPv4": "192.168.3.7",
"hostname": "vm-2"
}
}
}
}
result = [[value['openstack']['hostname'], value['openstack']['accessIPv4']]
for value in json_data['hostvars'].values()]
print(result)
output
[['vm-1', '192.168.3.6'], ['vm-2', '192.168.3.7']]
A simple one, but I've just not yet been able to wrap my head around parsing nested lists and json structures in Python...
Here is the raw message I am trying to parse.
{
"Records": [
{
"messageId": "1b9c0952-3fe3-4ab4-a8ae-26bd5d3445f8",
"receiptHandle": "AQEBy40IsvNDy33dOhn4KB8+7apBecWpSuw5OgL9sw/Nf+tM2esLgqmWjGsd4n0oqB",
"body": "{\n \"Type\" : \"Notification\",\n \"MessageId\" : \"dce5c301-029f-55e1-8cee-959b1ad4e500\",\n \"TopicArn\" : \"arn:aws:sns:ap-southeast-2:062497424678:vid\",\n \"Message\" : \"ChiliChallenge.mp4\",\n \"Timestamp\" : \"2020-01-16T07:51:39.807Z\",\n \"SignatureVersion\" : \"1\",\n \"Signature\" : \"oloRF7SzS8ipWQFZieXDQ==\",\n \"SigningCertURL\" : \"https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-a.pem\",\n \"UnsubscribeURL\" : \"https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:062478:vid\"\n}",
"attributes": {
"ApproximateReceiveCount": "1",
"SentTimestamp": "1579161099897",
"SenderId": "AIDAIY4XD42",
"ApproximateFirstReceiveTimestamp": "1579161099945"
},
"messageAttributes": {},
"md5OfBody": "1f246d643af4ea232d6d4c91f",
"eventSource": "aws:sqs",
"eventSourceARN": "arn:aws:sqs:ap-southeast-2:062497424678:vid",
"awsRegion": "ap-southeast-2"
}
]
}
I am trying to extract the Message in the body section, ending up with a string as "ChiliChallenge.mp4\"
Thanks!
Essentially I just keep getting either TypeError: string indices must be integers or parsing the body but not getting any further into the list without an error.
Here's my attempt:
import json
with open ("event_testing.txt", "r") as myfile:
event=myfile.read().replace('\n', '')
str(event)
event = json.loads(event)
key = event['Records'][0]['body']
print(key)
you can use json.loads to load string
with open ("event_testing.txt", "r") as fp:
event = json.loads(fp.read())
key = json.loads(event['Records'][0]['body'])['Message']
print(key)
'ChiliChallenge.mp4'
Say your message is phrase,
I rebuild your code like:
phrase_2 = phrase["Records"]
print(phrase_2[0]["body"])
Then it works clearly. Because beginning of the Records, it looks like an array so you need to organized it.
i have a following json file
json_data = {
"action":"postRecord",
"data":{
"data":[
{
"info":{
"lid":999,
"cid":1234
},
"info":{
"lid":111,
"cid":"6789"
}
}
]
}
}
i tried the usage of json.load
output = json.load(json_data)
but it returnign a string
i want to access the value of id in a way like output['data']['data']['info']['id']
Isn't your data already in the format you want?
print json_data['data']['data'][0]['info']['cid']
Uh. You're saying that you've got an JSON array as a plain text in a json_data variable?
You should put the JSON data into the json_data variable as a string and use json.loads(json_data).
Since JSON is just text, you should treat it like a string, unless you have opened a JSON file from the disk, then json.load() should work with the file object itself.
In [5]: json_data = '{ "action":"postRecord", "data":{ "data":[ { "info":{ "lid":999, "cid":1234 }, "info":{ "lid":111, "cid":"6789" } } ] } }'
In [6]: output = json.loads(json_data)
In [7]: output['data']['data'][0]['info']['cid']
Out[7]: u'6789'
seems the original json string has problem, there has duplicate info property in one element, so after json.dump to format , the first info will be removed , and the result is the second 6789.
json_data = { "action":"postRecord", "data":{ "data":[ { "info":{ "lid":999, "cid":1234 }, "info":{ "lid":111, "cid":"6789" } } ] } }
data = json.dumps(json_data)
json_to_python = json.loads(data)
print (json_to_python)
print (json_to_python['data']['data'][0]['info']['cid'])
======>
{'action': 'postRecord', 'data': {'data': [{'info1': {'lid': 999, 'cid': 1234}, 'info': {'lid': 111, 'cid': '6789'}}]}}
6789