So I keep running into a issue where json is asking to use a int to find pieces of data in a json response. Below is the code that works though the issue is i want to print every 'name' in the json though id have to change the [0] to 1 and then 2 ect. I tried to increment it though that ran into issue too. This could be just me overlooking something but let me know, thanks.
def BaseTesting(TarLink):
PayloadToSend = {
}
HeadersToSend = { # make sure to change your token at least once every 30 mins
'authorization': '',
'user-agent': ''
}
ReqForFriends = requests.post(TarLink, headers=HeadersToSend, data=PayloadToSend).text
LoadedJSONData = json.loads(ReqForFriends)
print(LoadedJSONData['friends'][0]['name'])
BaseTesting(TarLink="")
JSON
{
"friends":[
{
"name":"test1",
"user_id":"1132",
"type":2,
"display":"true",
},
{
"name":"test2",
"user_id":"2341",
"type":1,
"display":"true",
},
{
"name":"test3",
"user_id":"1234",
"type":2,
"display":"true",
},
}
it seems to work properly
LoadedJSONData = {
"friends":[
{
"name":"test1",
"user_id":"1132",
"type":2,
"display":"true",
},
{
"name":"test2",
"user_id":"2341",
"type":1,
"display":"true",
},
{
"name":"test3",
"user_id":"1234",
"type":2,
"display":"true",
},
]
}
for i in range(0,3):
print(LoadedJSONData['friends'][i]['name']) #test1 test2 test3
Related
The query I would like to replicate in DSL is as below:
GET /_search
{
"query":{
"bool":{
"must":[
{
"term":{
"destination":"singapore"
}
},
{
"terms":{
"tag_ids":[
"tag_luxury"
]
}
}
]
}
},
"aggs":{
"max_price":{
"max":{
"field":"price_range_from.SGD"
}
},
"min_price":{
"min":{
"field":"price_range_from.SGD"
}
}
},
"post_filter":{
"range":{
"price_range_from.SGD":{
"gte":0.0,
"lte":100.0
}
}
}
}
The above query
Matches terms - destination and tags_ids
Aggregates to result to find the max price from field price_range_from.SGD
Applies another post_filter to subset the result set within price limits
It works perfectly well in the Elastic/Kibana console.
I replicated the above query in elasticsearch-dsl as below:
es_query = []
es_query.append(Q("term", destination="singapore"))
es_query.append(Q("terms", tag_ids=["tag_luxury"]))
final_query = Q("bool", must=es_query)
es_conn = ElasticSearch.instance().get_client()
dsl_client = DSLSearch(using=es_conn, index=index).get_dsl_client()
dsl_client.query = final_query
dsl_client.aggs.metric("min_price", "min", field="price_range_from.SGD")
dsl_client.aggs.metric("max_price", "max", field="price_range_from.SGD")
q = Q("range", **{"price_range_from.SGD":{"gte": 0.0, "lte": 100.0}})
dsl_client.post_filter(q)
print(dsl_client.to_dict())
response = dsl_client.execute()
print(response.to_dict().get("hits", {}))
Although the aggregations are correct, products beyond the price range are also being returned. There is no error returned but it seems like the post_filter query is not applied.
I dived in the dsl_client object to see whether my query is being captured correctly. I see only the query and aggs but don't see the post_filter part in the object. The query when converted to a dictionary using dsl_client.to_dict() is as below -
{
"query":{
"bool":{
"must":[
{
"term":{
"destination":"singapore"
}
},
{
"terms":{
"tag_ids":[
"tag_luxury"
]
}
}
]
}
},
"aggs":{
"min_price":{
"min":{
"field":"price_range_from.SGD"
}
},
"max_price":{
"max":{
"field":"price_range_from.SGD"
}
}
}
}
Please help. Thanks!
You have to re-assign the dsl_client like:
dsl_client = dsl_client.post_filter(q)
I have a dictionary below:
event = {
"body-json": {},
"params": {
"path": {
"matchphrase": "term"
},
"querystring": {
"dataproduct.keyword": "health"
},
"header": {
"Accept": "application/json"
}
},
"resource-path": "/{matchphrase}"
}
I would like to access the above event dictionary keys & values and frame a new dictionary as follows:
{"query": {"term" : {"dataproduct.keyword": "health"}}}
Here is the code what I tried:
a = event['params']['path']['matchphrase'] #term
b = list(event['params']['querystring'].keys())[0] #dataproduct.keyword
c = list(event['params']['querystring'].values())[0] #health
body=f"{query: {{a} : {{b}: {c}}}}"
print(body)
Am I missing something ?
This should work :
body = {"query":{str(a):{str(b):str(c)}}}
print(body)
The escaping is wrong.
Try this instead:
body = f'{{"query": {{{a!r}: {{{b!r}: {c!r}}}}}}}'
I've also added !r which will return the real representation (repr) of the object (so you don't need to artificially add quotes).
you can create a dictionary and then get a string version of it using json.dumps.
import json
event = {
"body-json": {},
"params": {
"path": {"matchphrase": "term"},
"querystring": {"dataproduct.keyword": "health"},
"header": {"Accept": "application/json"},
},
"resource-path": {"matchphrase}"},
}
a = event["params"]["path"]["matchphrase"] # term
b = list(event["params"]["querystring"].keys())[0] # dataproduct.keyword
c = list(event["params"]["querystring"].values())[0] # health
result = {"query": {a: {b: c}}}
print(json.dumps(result))
Output:
{"query": {"term": {"dataproduct.keyword": "health"}}}
I have a json file called pool.json which contains this:
{
"pools": {
"$poolId": {
"nodes": {
"$nodeId": {
"bcm": {
"address": {
"ip": "10.10.10.10"
},
"password": "ADMIN",
"username": "ADMIN"
}
}
}
}
}
}
This is my Python code:
pool_id = ['123456']
json_pool = json.loads(read_json_file('pool.json'))
for i in pool_id:
json_pool['pools'][i] = json_pool.pop(['pools']['$poolId'])
print('json_pool: %s' % json_pool)
I'm trying to update $poolId with the value in pool_id(I know I've only got one pool_id. I just want to get this piece working before I do anything else). Ive been trying to do this with pop but am having no success when it's nested as in this case. I can get it working when I want to change a top level key. What am I doing wrong?
I think you want to execute json_pool['pools'].pop('$poolId') instead of json_pool.pop(['pools']['$poolId']).
I am trying to scrape some values through a json that looks like:
{
"attributes":{
"531":{
"id":"531",
"code":"taille",
"label":"taille",
"options":[
{
"id":"30",
"label":"40",
"is_in":"0"
},
{
"id":"31",
"label":"41",
"is_in":"1"
}
]
}
},
"template":"Helloworld"
}
My issue is that the number 531 is different in each json file that I am trying to scrape and what I am trying to grab through this json is the label and is_in value
What I have done so far is that I tried to do something like this but I am stuck and dont know how to do if the 531 is changing to something else
getOption = '{
"attributes":{
"531":{
"id":"531",
"code":"taille",
"label":"taille",
"options":[
{
"id":"30",
"label":"40",
"is_in":"0"
},
{
"id":"31",
"label":"41",
"is_in":"1"
}
]
}
},
"template":"Helloworld"
}'
for att, values in getOption.items():
print(values)
So how do I possible scrape the value label and is_in?
I'm not sure if you can have several 531 keys but you can loop through them.
getOption = {
"attributes":{
"531":{
"id":"531",
"code":"taille",
"label":"taille",
"options":[
{
"id":"30",
"label":"40",
"is_in":"0"
},
{
"id":"31",
"label":"41",
"is_in":"1"
}
]
}
},
"template":"Helloworld"
}
attributes = getOption['attributes']
for key in attributes.keys():
for item in attributes[key]['options']:
print(item['label'], item['is_in'])
I am attempting to parse a json response that looks like this:
{
"links": {
"next": "http://www.neowsapp.com/rest/v1/feed?start_date=2015-09-08&end_date=2015-09-09&detailed=false&api_key=xxx",
"prev": "http://www.neowsapp.com/rest/v1/feed?start_date=2015-09-06&end_date=2015-09-07&detailed=false&api_key=xxx",
"self": "http://www.neowsapp.com/rest/v1/feed?start_date=2015-09-07&end_date=2015-09-08&detailed=false&api_key=xxx"
},
"element_count": 22,
"near_earth_objects": {
"2015-09-08": [
{
"links": {
"self": "http://www.neowsapp.com/rest/v1/neo/3726710?api_key=xxx"
},
"id": "3726710",
"neo_reference_id": "3726710",
"name": "(2015 RC)",
"nasa_jpl_url": "http://ssd.jpl.nasa.gov/sbdb.cgi?sstr=3726710",
"absolute_magnitude_h": 24.3,
"estimated_diameter": {
"kilometers": {
"estimated_diameter_min": 0.0366906138,
"estimated_diameter_max": 0.0820427065
},
"meters": {
"estimated_diameter_min": 36.6906137531,
"estimated_diameter_max": 82.0427064882
},
"miles": {
"estimated_diameter_min": 0.0227984834,
"estimated_diameter_max": 0.0509789586
},
"feet": {
"estimated_diameter_min": 120.3760332259,
"estimated_diameter_max": 269.1689931548
}
},
"is_potentially_hazardous_asteroid": false,
"close_approach_data": [
{
"close_approach_date": "2015-09-08",
"close_approach_date_full": "2015-Sep-08 09:45",
"epoch_date_close_approach": 1441705500000,
"relative_velocity": {
"kilometers_per_second": "19.4850295284",
"kilometers_per_hour": "70146.106302123",
"miles_per_hour": "43586.0625520053"
},
"miss_distance": {
"astronomical": "0.0269230459",
"lunar": "10.4730648551",
"kilometers": "4027630.320552233",
"miles": "2502653.4316094954"
},
"orbiting_body": "Earth"
}
],
"is_sentry_object": false
},
}
I am trying to figure out how to parse through to get "miss_distance" dictionary values ? I am unable to wrap my head around it.
Here is what I have been able to do so far:
After I get a Response object from request.get()
response = request.get(url
I convert the response object to json object
data = response.json() #this returns dictionary object
I try to parse the first level of the dictionary:
for i in data:
if i == "near_earth_objects":
dataset1 = data["near_earth_objects"]["2015-09-08"]
#this returns the next object which is of type list
Please someone can explain me :
1. How to decipher this response in the first place.
2. How can I move forward in parsing the response object and get to miss_distance dictionary ?
Please any pointers/help is appreciated.
Thank you
Your data will will have multiple dictionaries for the each date, near earth object, and close approach:
near_earth_objects = data['near_earth_objects']
for date in near_earth_objects:
objects = near_earth_objects[date]
for object in objects:
close_approach_data = object['close_approach_data']
for close_approach in close_approach_data:
print(close_approach['miss_distance'])
The code below gives you a table of date, miss_distances for every object for every date
import json
raw_json = '''
{
"near_earth_objects": {
"2015-09-08": [
{
"close_approach_data": [
{
"miss_distance": {
"astronomical": "0.0269230459",
"lunar": "10.4730648551",
"kilometers": "4027630.320552233",
"miles": "2502653.4316094954"
},
"orbiting_body": "Earth"
}
]
}
]
}
}
'''
if __name__ == "__main__":
parsed = json.loads(raw_json)
# assuming this json includes more than one near_earch_object spread across dates
near_objects = []
for date, near_objs in parsed['near_earth_objects'].items():
for obj in near_objs:
for appr in obj['close_approach_data']:
o = {
'date': date,
'miss_distances': appr['miss_distance']
}
near_objects.append(o)
print(near_objects)
output:
[
{'date': '2015-09-08',
'miss_distances': {
'astronomical': '0.0269230459',
'lunar': '10.4730648551',
'kilometers': '4027630.320552233',
'miles': '2502653.4316094954'
}
}
]