We Keep Coding

JSONPath issues with Python and jsonpath_ng (Parse error near token ?)

I'm trying to work with jsonpath_ng Python library. For most of the JSONPath filters I usually use it works.
However, I'm struggling with a simple filter clause. It can be summarized in 2 lines.
from jsonpath_ng.ext import parse
jsonpath_expression = parse(f"$.jobs.*.jobSummary.[?(#.storagePolicy.storagePolicyName=='{SPname}')].sizeOfApplication")
My JSON payload is this one:
{
"processinginstructioninfo": {
"attributes": [
{
"name": "WebServer",
"value": "IDM-COMMSERVE"
}
]
},
"totalRecordsWithoutPaging": 161,
"jobs": [
{
"jobSummary": {
"sizeOfApplication": 65552265428,
"vsaParentJobID": 28329591,
"commcellId": 2,
"backupSetName": "defaultBackupSet",
"opType": 59,
"totalFailedFolders": 0,
"totalFailedFiles": 0,
"alertColorLevel": 0,
"jobAttributes": 288232025419153408,
"jobAttributesEx": 67108864,
"isVisible": true,
"localizedStatus": "Completed",
"isAged": false,
"totalNumOfFiles": 0,
"jobId": 28329592,
"jobSubmitErrorCode": 0,
"sizeOfMediaOnDisk": 34199,
"currentPhase": 0,
"status": "Completed",
"lastUpdateTime": 1661877467,
"percentSavings": 99.99995,
"localizedOperationName": "Snap Backup",
"statusColor": "black",
"pendingReason": "",
"errorType": 0,
"backupLevel": 2,
"jobElapsedTime": 59,
"jobStartTime": 1661877408,
"currentPhaseName": "",
"jobType": "Snap Backup",
"isPreemptable": 0,
"backupLevelName": "Incremental",
"attemptStartTime": 0,
"pendingReasonErrorCode": "",
"appTypeName": "Virtual Server",
"percentComplete": 100,
"averageThroughput": 27472.637,
"localizedBackupLevelName": "Incremental",
"currentThroughput": 0,
"subclientName": "default",
"destClientName": "desktop-1058kvf",
"jobEndTime": 1661877467,
"dataSource": {
"dataSourceId": 0
},
"subclient": {
"clientName": "desktop-1058kvf",
"instanceName": "VMInstance",
"backupsetId": 161,
"commCellName": "idm-commserve",
"instanceId": 2,
"subclientId": 235,
"clientId": 71,
"appName": "Virtual Server",
"backupsetName": "defaultBackupSet",
"applicationId": 106,
"subclientName": "default"
},
"storagePolicy": {
"storagePolicyName": "IDM-Metallic-Replica_ReplicationPlan",
"storagePolicyId": 78
},
"destinationClient": {
"clientId": 71,
"clientName": "desktop-1058kvf",
"displayName": "idm-laptop1"
},
"userName": {
"userName": "admin",
"userId": 1
},
"clientGroups": [
{
"clientGroupId": 4,
"clientGroupName": "Laptop Clients"
},
{
"clientGroupId": 46,
"clientGroupName": "Clients For Commserv LiveSync"
},
{
"clientGroupId": 47,
"clientGroupName": "idm-vcsa"
},
{
"clientGroupId": 55,
"clientGroupName": "Laptop plan test clients"
}
]
}
}
]
}
I need to get just the "sizeOfApplication" parameter for every object with a particular "storagePolicyName". That's it. Say, in this case, that the "storagePolicyName" I'm looking values for is "IDM-Metallic-Replica_ReplicationPlan" as an example.
I usually go to My favourite JSONPath site to test the JSONpath I use, and this one
"$.jobs.*.jobSummary.[?(#.storagePolicy.storagePolicyName=='IDM-Metallic-Replica_ReplicationPlan')].sizeOfApplication" works.
But, on Python side, I keep getting "jsonpath_ng.exceptions.JsonPathParserError: Parse error at 1:21 near token ? (?)" errors.
What am I doing wrong?
Thank you!
Mattia

I think the problem here is that jsonpath_ng is being stricter to the JSONPath proposal than the other parsers you have tried.
The first problem is that there shouldn't be a . immediately before a filter condition [?(...)]. So the first step is to remove the . after jobSummary in jobSummary.[?(#storagePolicy....
I made that change to your JSONPath expression, and used jsonpath_ng to run it on your sample data. The parser error had gone, but it returned no matches. So it's still not right.
From reading the JSONPath proposal, it's not clear if you can use a filter operator such as [?(...)] on an object, or only on an array. When used on an array it would return all elements of the array that match the filter. If a JSONPath parser does support a filter on an object, then it seems it returns the object if the filter matches and an empty list of matches otherwise.
I would guess that jsonpath_ng only permits filters on arrays. So let's modify your JSONPath expression to only use filters on arrays. Your JSON has an array in $.jobs, and within each element of this array you want to look within the jobSummary object for a storagePolicy with storagePolicyName=={SPname}. So the following JSONPath expression should return the matching job:
$.jobs[?(#.jobSummary.storagePolicy.storagePolicyName=='{SPname}')]
What we then want to do is to get the value of the sizeOfApplication property within the jobSummary object within each matching job. Note that the matches returned by the above JSONPath expression are elements of the jobs array, not jobSummary objects. We can't just select sizeOfApplication because we're one level further up than we were before. We need to go back into the jobSummary object to get the sizeOfApplication:
$.jobs[?(#.jobSummary.storagePolicy.storagePolicyName=='{SPname}')].jobSummary.sizeOfApplication
I used jsonpath_ng to run this JSONPath expression on your sample data and it gave me the output [65552265428], which seems to be the expected output.

TypeError: string indices must be integers in Python

I want to get key and value from the Json response i got from the currency conveter API.T am getting an error :"of string indices must be integers".Below is the Python code,the data in Json and the error message.
import json
from urllib.request import urlopen
with urlopen ("http://free.currencyconverterapi.com/api/v6/convert?q=ZAR_GBP,ZAR_USD")as response:
source=response.read()
data= json.loads(source)
data=json.dumps(data,indent=2)
print(data)
value= (data['results']['ZAR_GBP']['val'])
print(value)
the Json output is:
{
"query": {
"count": 2
},
"results": {
"ZAR_GBP": {
"id": "ZAR_GBP",
"fr": "ZAR",
"to": "GBP",
"val": 0.056698
},
"ZAR_USD": {
"id": "ZAR_USD",
"val": 0.072289,
"to": "USD",
"fr": "ZAR"
}
}
}
If i want to access the key "val' and its value,this is giving me an error
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-61-eb4254977d14> in <module>()
5 data= json.loads(source)
6 data=json.dumps(data,indent=2)
----> 7 print (data['results']['ZAR_GBP']['val'])
8
TypeError: string indices must be integers

That's because you're converting the json to a string (with json.dumps) and then trying to access said string via keys, which obviously fails, because it's not a json anymore.
It'll work if you remove the data=json.dumps(data,indent=2) line.

Python KeyError JSON

I am trying to get some specific data from a JSON file, but I get the following error:
for item in data['perfil'][0]['regimes']: KeyError: 'regimes'
This is my script, as you can see, I am trying to get Key regimes, because it has specific data, that others don't have:
import json
import matplotlib.pyplot as plt
with open('backupperfil.json') as f:
data = json.load(f)
for item in data['perfil'][0]['regimes']:
print(item['classificacao'])
And this is a small copy of Json File:
{
"perfil":[
{
"data": 1533815887,
"kmh": 0,
"rpm": 0.0,
"pedal": 15.294117647058824
},
{
"data": 1533815888,
"kmh": 0
,"rpm": 0.0
,"pedal": 15.294117647058824
},
{
"data": 1533815889,
"kmh": 0
,"rpm": 0.0
,"pedal": 15.294117647058824
},
{
"kmh": 0
,"rpm": 834.75
,"pedal": 14.117647058823529
},
{
"regimes": [
{
"kmh": 0,
"rpm": 833.75,
"pedal": 14.117647058823529,
"regime_inferior_normal": 318,
"regime_normal": 27,
"regime_agressivo": 1,
"regime_muito_agressivo": 0,
"soma_regimes": 346,
"classificacao": "Regime Inferior a Normal"
}
]
},
{
"kmh": 0,
"rpm": 827.5,
"pedal": 14.117647058823529
}
]
}
How can I get the data inside of Key "regimes"?

In case if you have more than regimes in your json. I'm assuming so , since you gave only a part of your original json
for i in js['perfil']:
if 'regimes' in i:
print(i['regimes'][0]['classificacao'])
This way the the program loops through each jsonobject and checks for the key regimes . If it exists, then it'll be printed

Try this piece of code:-
data['perfil'][4]['regimes'][0]['classificacao']
Make sure the indexes you are using are right.

How to extract parrticular value from nested json values.?

I am new to python.
I have small requirement (i.e) want to extract only one value from the JSON format.
Please do correct me if i am wrong.
JSON input is:
{
"meta": {
"limit": 1,
"next": "/api/v1/ips/?username=sic1&api_key=689db0740ed73c2bf6402a7de0fcf2d7b57111ca&limit=1&objects=&offset=1",
"offset": 0,
"previous": null,
"total_count": 56714
},
"objects": [
{
"_id": "556f4c81dcddec0c41463529",
"bucket_list": [],
"campaign": [
{
"analyst": "prabhu",
"confidence": "medium",
"date": "2015-06-03 14:50:41.440000",
"name": "Combine"
}
],
"created": "2015-06-03 14:50:41.436000",
"ip": "85.26.162.70",
"locations": [],
"modified": "2015-06-18 09:50:51.612000",
"objects": [],
"relationships": [
{
"analyst": "prabhu",
"date": "2015-06-18 09:50:51.369000",
"rel_confidence": "unknown",
"rel_reason": "N/A",
"relationship": "Related_To",
"relationship_date": "2015-06-18 09:50:51.369000",
"type": "Indicator",
"value": "556f4c81dcddec0c4146353a"
}
],
"releasability": [],
"schema_version": 3,
"screenshots": [],
"sectors": [],
"source": [
{
"instances": [
{
"analyst": "prabhu",
"date": "2015-06-03 14:50:41.438000",
"method": "trawl",
"reference": "http://www.openbl.org/lists/base_30days.txt"
}
],
"name": "www.openbl.org"
}
],
"status": "New",
"tickets": [],
"type": "Address - ipv4-addr"
}
]
}
The code i used for getting value only IP's from objects
import requests
from pprint import pprint
import json
url = 'http://127.0.0.1:8080/api/v1/ips/'
params = {'api_key':'xxxxxx','username': 'abcd'}
r = requests.get(url, params=params, verify=False)
parsed = json.loads(r)
print (parsed['objects']['ip'])
The error i am receiving is:
Traceback (most recent call last):
File "testapi.py", line 9, in <module>
parsed = json.loads(r)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer
I just want to get IP's from that JSON input.
Thanks.

You are passing a requests object instead of a str object to json.loads(). You need to change
parsed = json.loads(r)
to
parsed = json.loads(r.text)
Also, parsed['objects'] is a list, you need to access its first element & then get the key ip:
>>> print(parsed['objects'][0]['ip'])

The problem is in this line: parsed = json.loads(r)
You're reciving the json response but insted of feeding json elements to json.loads you're instead feeding it <Response [200]>
>>> r = requests.get('http://www.google.com')
>>> r
<Response [200]>
>>> type(r)
<class 'requests.models.Response'>
(Look closely at the error message. Expected string or buffer Which means you're providing it something that is NOT string or buffer(an object in this case))
This is the reason why str(r) didn't work. Because it just converted <Response 200> to '<Response 200>' which obviously is not json.
change this line to parsed = json.loads(r.text).
>>> type(r.text)
<type 'unicode'>
and then parsed['objects'][0]['ip'] should give you the IP address :)
You can find more about the requests module here

build dynamical JSON/LIST obj in Python

I'm a real Python newbee and I'm having problems creating a JSON/LIST obj.
What I want to end up with is the following JSON to send o an API
{
"request": {
"slice": [
{
"origin": "AMS",
"destination": "SYD",
"date": "2015-06-23"
}
],
"passengers": {
"adultCount": 1,
"infantInLapCount": 0,
"infantInSeatCount": 0,
"childCount": 0,
"seniorCount": 0
},
"solutions": 20,
"refundable": false
}
}
I figured to make a list and then to convert to JSON with the dumps() function. This works. The thing is, I need to change the date field with an iterator to add a day, but I'm stuck on changing this field.
Any advice?
thx!

As your question is a bit vague i can only guess that you're trying to modify the JSON version of your data directly, while you should modify the Python object before converting it into JSON... something like this:
d = {
"request": {
"slice": [
{
"origin": "AMS",
"destination": "SYD",
"date": "2015-06-23"
}
],
"passengers": {
"adultCount": 1,
"infantInLapCount": 0,
"infantInSeatCount": 0,
"childCount": 0,
"seniorCount": 0
},
"solutions": 20,
"refundable": False # note how this is python False, not js false!
}
}
# then you can do:
d["request"]["slice"][0]["date"] = "2015-05-23"
# and finally convert to json:
j = json.dumps(d)
If it happens that you get JSON as a string, you should first convert it into a python object so you can work on it:
# if j is your json string, convert it into a python object
d = json.loads(j)
# then do your modifications as above:
d["request"]["slice"][0]["date"] = "2015-05-23"
# and finally convert back to json:
j = json.dumps(d)