I am unable to create dictionary - python

I am unable to create a dictionary whose structure is like :
"aggs": {
"4": {
"date_histogram": {
"field": "#timestamp",
"interval": "1m",
"time_zone": "Asia/Kolkata",
"min_doc_count": 1,
},
"aggs": {
"5": {
"range": {
"field": "system.core.idle.pct",
"ranges": [{"from": 0, "to": 0.001}, {"from": 0.001, "to": 1}],
"keyed": true,
},
"aggs": {
"2": {"max": {"field": "system.core.system.pct"}},
"3": {"max": {"field": "system.core.idle.pct"}},
},
}
},
}
}
Mostly I am getting the issue in creating aggs:5 inside aggs:4 and then looping it. I need to do it for multiple aggs. the number of agg id can be upto 1000 also.
I am trying to derive it from the dictionary:
"aggs": [
{"id": "1", "enabled": true, "type": "count", "schema": "metric", "params": {}},
{
"id": "2",
"enabled": true,
"type": "max",
"schema": "metric",
"params": {"field": "system.core.system.pct", "customLabel": "system"},
},
{
"id": "3",
"enabled": true,
"type": "max",
"schema": "metric",
"params": {"field": "system.core.idle.pct", "customLabel": "idle"},
},
{
"id": "4",
"enabled": true,
"type": "date_histogram",
"schema": "bucket",
"params": {
"field": "#timestamp",
"interval": "m",
"customInterval": "2h",
"min_doc_count": 1,
"extended_bounds": {},
},
},
{
"id": "5",
"enabled": true,
"type": "range",
"schema": "bucket",
"params": {
"field": "system.core.idle.pct",
"ranges": [{"from": 0, "to": 0.001}, {"from": 0.001, "to": 1}],
},
},
]
Can anyone show the code how to execute it. Basically I am failing in creating nested dictionary.


It looks like you are using JSON data.
You can see this link https://realpython.com/python-json/.
You can use the inbuilt JSON lib to covert the whole thing in a dict
Basically something like this
json_string = """
{
"researcher": {
"name": "Ford Prefect",
"species": "Betelgeusian",
"relatives": [
{
"name": "Zaphod Beeblebrox",
"species": "Betelgeusian"
}
]
}
}
"""
data = json.loads(json_string)


add whole structure in {}, and replace true to True.
{"aggs": {
"4": {
"date_histogram": {
"field": "#timestamp",
"interval": "1m",
"time_zone": "Asia/Kolkata",
"min_doc_count": 1,
},
"aggs": {
"5": {
"range": {
"field": "system.core.idle.pct",
"ranges": [{"from": 0, "to": 0.001}, {"from": 0.001, "to": 1}],
"keyed": True,
},
"aggs": {
"2": {"max": {"field": "system.core.system.pct"}},
"3": {"max": {"field": "system.core.idle.pct"}},
},
}
},
}
}}
{'aggs': {'4': {'date_histogram': {'field': '#timestamp',
'interval': '1m',
'time_zone': 'Asia/Kolkata',
'min_doc_count': 1},
'aggs': {'5': {'range': {'field': 'system.core.idle.pct',
'ranges': [{'from': 0, 'to': 0.001}, {'from': 0.001, 'to': 1}],
'keyed': True},
'aggs': {'2': {'max': {'field': 'system.core.system.pct'}},
'3': {'max': {'field': 'system.core.idle.pct'}}}}}}}}

Related

python parse every loop items in json response

i have json response like this :
r = {
"ID": "0001",
"ST": "87549937737",
"DT": "22062022",
"DATA": {
"0": {
"PC": "100",
"NM": "ALEX",
"AMOUNT": "210,912",
"ACT": "123123456"
},
"1": {
"PC": "100",
"NM": "MARCO",
"AMOUNT": "500,200",
"ACT": "123555456"
},
"2": {
"PC": "100",
"NM": "TIFFANY",
"AMOUNT": "712,111",
"ACT": "123666981"
}
},
"RCMSG": {
"0": {
"SEVERITY": "00"
}
},
"RC": "0000"
}
and i want to parse every loop items in DATA, and my scripts are like this :
for items in r['DATA']:
jdata = {'PRODUCTCODE': items['PC'], 'NAME': items['NM'], 'AMOUNTBALANCE': items['AMOUNT'], 'ACCOUNT': items['ACT']}
return jsonify({'status':'success', 'code':'00', 'data': jdata})
but the response is only parsed '0' in 'DATA, how do i loop every items inside 'DATA' ?
{
"PRODUCTCODE": "100",
"NAME": "ALEX",
"AMOUNTBALANCE": "210,912",
"ACCOUNT": "123123456"
}

In your code, there is a return statement inside the for loop, so it will never traverse all elements in JSON data because it will end its execution at the first item. So, you need to traverse all elements accessing them from the original JSON response:
r = {
"ID": "0001",
"ST": "87549937737",
"DT": "22062022",
"DATA": {
"0": {
"PC": "100",
"NM": "ALEX",
"AMOUNT": "210,912",
"ACT": "123123456"
},
"1": {
"PC": "100",
"NM": "MARCO",
"AMOUNT": "500,200",
"ACT": "123555456"
},
"2": {
"PC": "100",
"NM": "TIFFANY",
"AMOUNT": "712,111",
"ACT": "123666981"
}
},
"RCMSG": {
"0": {
"SEVERITY": "00"
}
},
"RC": "0000"
}
d = []
for i in r['DATA']:
d.append(r['DATA'][i])
print(d)
Output:
[{'PC': '100', 'NM': 'ALEX', 'AMOUNT': '210,912', 'ACT': '123123456'}, {'PC': '100', 'NM': 'MARCO', 'AMOUNT': '500,200', 'ACT': '123555456'}, {'PC': '100', 'NM': 'TIFFANY', 'AMOUNT': '712,111', 'ACT': '123666981'}]

results = []
for k, v in r['DATA'].items():
item = {
"PRODUCTCODE": v["PC"],
"NAME": v["NM"],
"AMOUNTBALANCE": v["AMOUNT"],
"ACCOUNT": v["ACT"],
}
results.append(item)

How to compare two dictionaries and print if one of the values are above zero

I have been trying to work with a JSON object where I have been trying to get values from two different keys. What I want to do is to check if in object 1 contains in object 2 and has the value over 0 then I want to print it out.
get_json = json.dumps({
"attributes": {
"203": {
"id": "203",
"code": "sizefootwear_conf",
"label": "EU",
"options": [{
"id": "6320",
"label": "38",
"products": ["69813"]
},
{
"id": "6351",
"label": "38,5",
"products": ["69817"]
},
{
"id": "6335",
"label": "39",
"products": ["69818"]
},
{
"id": "6354",
"label": "40",
"products": ["69819"]
},
{
"id": "6338",
"label": "40,5",
"products": ["69820"]
},
{
"id": "6357",
"label": "41",
"products": ["69821"]
},
{
"id": "6326",
"label": "42",
"products": ["69822"]
},
{
"id": "6362",
"label": "42,5",
"products": ["69823"]
},
{
"id": "6341",
"label": "43",
"products": ["69824"]
},
{
"id": "6365",
"label": "44",
"products": ["69814"]
},
{
"id": "6344",
"label": "44,5",
"products": ["69815"]
},
{
"id": "6370",
"label": "45,5",
"products": ["69816"]
}
],
"position": "0"
},
"205": {
"id": "205",
"code": "sizefootwearus_conf",
"label": "US",
"options": [{
"id": "6319",
"label": "5,5",
"products": ["69813"]
},
{
"id": "6372",
"label": "6,0",
"products": ["69817"]
},
{
"id": "6334",
"label": "6,5",
"products": ["69818"]
},
{
"id": "6350",
"label": "7,0",
"products": ["69819"]
},
{
"id": "6337",
"label": "7,5",
"products": ["69820"]
},
{
"id": "6353",
"label": "8,0",
"products": ["69821"]
},
{
"id": "6325",
"label": "8,5",
"products": ["69822"]
},
{
"id": "6356",
"label": "9,0",
"products": ["69823"]
},
{
"id": "6340",
"label": "9,5",
"products": ["69824"]
},
{
"id": "6364",
"label": "10,0",
"products": ["69814"]
},
{
"id": "6343",
"label": "10,5",
"products": ["69815"]
},
{
"id": "6328",
"label": "11,5",
"products": ["69816"]
}
],
"position": "1"
},
"204": {
"id": "204",
"code": "sizefootwearuk_conf",
"label": "UK",
"options": [{
"id": "6318",
"label": "5,0",
"products": ["69813"]
},
{
"id": "6352",
"label": "5,5",
"products": ["69817"]
},
{
"id": "6743",
"label": "6,0-EU39",
"products": ["69818"]
},
{
"id": "6744",
"label": "6,0-EU40",
"products": ["69819"]
},
{
"id": "6355",
"label": "6,5",
"products": ["69820"]
},
{
"id": "6336",
"label": "7,0",
"products": ["69821"]
},
{
"id": "6361",
"label": "7,5",
"products": ["69822"]
},
{
"id": "6324",
"label": "8,0",
"products": ["69823"]
},
{
"id": "6363",
"label": "8,5",
"products": ["69824"]
},
{
"id": "6339",
"label": "9,0",
"products": ["69814"]
},
{
"id": "6366",
"label": "9,5",
"products": ["69815"]
},
{
"id": "6369",
"label": "10,5",
"products": ["69816"]
}
],
"position": "2"
}
},
"productStockAlert": {
"entity": "69825",
"map": {
"203": {
"label": "52,5",
"": "",
"6610": "6610",
"6498": "6498",
"6582": "6582",
"6516": "6516",
"6501": "6501",
"6518": "6518",
"6504": "6504",
"6395": "6395",
"6404": "6404",
"6533": "6533",
"6407": "6407",
"6530": "6530",
"6410": "6410",
"6413": "6413",
"6416": "6416",
"6534": "6534",
"6419": "6419",
"6422": "6422",
"6425": "6425",
"6398": "6398",
"6401": "6401",
"6531": "6531",
"6431": "6431",
"6443": "6443",
"6446": "6446",
"6495": "6495",
"6449": "6449",
"6452": "6452",
"6455": "6455",
"6458": "6458",
"6461": "6461",
"6807": "6807",
"6464": "6464",
"6434": "6434",
"6437": "6437",
"6558": "6558",
"6440": "6440",
"6480": "6480",
"6481": "6481",
"6382": "6382",
"6465": "6465",
"6631": "6631",
"6332": "6332",
"6466": "6466",
"6348": "6348",
"6634": "6634",
"6320": "6320",
"6351": "6351",
"6384": "6384",
"6659": "6659",
"6335": "6335",
"6388": "6388",
"6508": "6508",
"6354": "6354",
"6338": "6338",
"6389": "6389",
"6664": "6664",
"6357": "6357",
"6390": "6390",
"6506": "6506",
"6637": "6637",
"6326": "6326",
"6362": "6362",
"6391": "6391",
"6640": "6640",
"6341": "6341",
"6392": "6392",
"6560": "6560",
"6365": "6365",
"6344": "6344",
"6385": "6385",
"6838": "6838",
"6368": "6368",
"6386": "6386",
"6370": "6370",
"6643": "6643",
"6628": "6628",
"6329": "6329",
"6529": "6529",
"6387": "6387",
"6843": "6843",
"6347": "6347",
"6470": "6470",
"6360": "6360",
"6646": "6646",
"6472": "6472",
"6323": "6323",
"6564": "6564",
"6593": "6593",
"6474": "6474",
"6376": "6376",
"6565": "6565",
"6561": "6561",
"6567": "6567",
"6604": "6604",
"6607": "6607"
},
"205": {
"label": "18,0",
"": "",
"6513": "6513",
"6497": "6497",
"6583": "6583",
"6500": "6500",
"6821": "6821",
"6503": "6503",
"6532": "6532",
"6394": "6394",
"6403": "6403",
"6406": "6406",
"6409": "6409",
"6412": "6412",
"6415": "6415",
"6418": "6418",
"6421": "6421",
"6424": "6424",
"6397": "6397",
"6400": "6400",
"6430": "6430",
"6442": "6442",
"6445": "6445",
"6448": "6448",
"6451": "6451",
"6454": "6454",
"6457": "6457",
"6460": "6460",
"6463": "6463",
"6433": "6433",
"6436": "6436",
"6439": "6439",
"6555": "6555",
"6468": "6468",
"6507": "6507",
"6632": "6632",
"6331": "6331",
"6319": "6319",
"6635": "6635",
"6372": "6372",
"6334": "6334",
"6661": "6661",
"6350": "6350",
"6337": "6337",
"6663": "6663",
"6353": "6353",
"6619": "6619",
"6325": "6325",
"6621": "6621",
"6638": "6638",
"6356": "6356",
"6340": "6340",
"6623": "6623",
"6641": "6641",
"6364": "6364",
"6343": "6343",
"6625": "6625",
"6840": "6840",
"6367": "6367",
"6328": "6328",
"6644": "6644",
"6371": "6371",
"6346": "6346",
"6842": "6842",
"6359": "6359",
"6322": "6322",
"6647": "6647",
"6373": "6373",
"6566": "6566",
"6375": "6375",
"6562": "6562",
"6605": "6605",
"6608": "6608"
},
"204": {
"label": "17,0",
"": "",
"6611": "6611",
"6514": "6514",
"6496": "6496",
"6515": "6515",
"6499": "6499",
"6517": "6517",
"6502": "6502",
"6393": "6393",
"6505": "6505",
"6402": "6402",
"6405": "6405",
"6408": "6408",
"6411": "6411",
"6414": "6414",
"6417": "6417",
"6420": "6420",
"6423": "6423",
"6396": "6396",
"6399": "6399",
"6429": "6429",
"6745": "6745",
"6441": "6441",
"6444": "6444",
"6447": "6447",
"6450": "6450",
"6453": "6453",
"6456": "6456",
"6459": "6459",
"6462": "6462",
"6432": "6432",
"6435": "6435",
"6438": "6438",
"6467": "6467",
"6381": "6381",
"6633": "6633",
"6330": "6330",
"6349": "6349",
"6636": "6636",
"6318": "6318",
"6352": "6352",
"6660": "6660",
"6333": "6333",
"6743": "6743",
"6744": "6744",
"6355": "6355",
"6662": "6662",
"6336": "6336",
"6620": "6620",
"6361": "6361",
"6622": "6622",
"6639": "6639",
"6324": "6324",
"6363": "6363",
"6624": "6624",
"6642": "6642",
"6339": "6339",
"6366": "6366",
"6626": "6626",
"6839": "6839",
"6342": "6342",
"6627": "6627",
"6369": "6369",
"6645": "6645",
"6327": "6327",
"6358": "6358",
"6841": "6841",
"6345": "6345",
"6471": "6471",
"6648": "6648",
"6321": "6321",
"6473": "6473",
"6374": "6374",
"6563": "6563",
"6606": "6606",
"6609": "6609"
}
},
"child": {
"6320_6319_6318_": {
"entity": "69813",
"stock_number": 0,
"stock_status": false,
"productId": "69813",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6365_6364_6339_": {
"entity": "69814",
"stock_number": 5,
"stock_status": true,
"productId": "69814",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6344_6343_6366_": {
"entity": "69815",
"stock_number": 3,
"stock_status": true,
"productId": "69815",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6370_6328_6369_": {
"entity": "69816",
"stock_number": 1,
"stock_status": true,
"productId": "69816",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6351_6372_6352_": {
"entity": "69817",
"stock_number": 0,
"stock_status": false,
"productId": "69817",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6335_6334_6743_": {
"entity": "69818",
"stock_number": 0,
"stock_status": false,
"productId": "69818",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6354_6350_6744_": {
"entity": "69819",
"stock_number": 0,
"stock_status": false,
"productId": "69819",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6338_6337_6355_": {
"entity": "69820",
"stock_number": 0,
"stock_status": false,
"productId": "69820",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6357_6353_6336_": {
"entity": "69821",
"stock_number": 3,
"stock_status": true,
"productId": "69821",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6326_6325_6361_": {
"entity": "69822",
"stock_number": 4,
"stock_status": true,
"productId": "69822",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6362_6356_6324_": {
"entity": "69823",
"stock_number": 6,
"stock_status": true,
"productId": "69823",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
},
"6341_6340_6363_": {
"entity": "69824",
"stock_number": 6,
"stock_status": true,
"productId": "69824",
"parent_url": "https://www.bstn.com/eu_en/p/jordan-jordan-why-not-zer0-4-dd4889-006-0250549"
}
}
}
}
)
So what I did is that I created two dicts within a list:
first_loop = []
second_loop = []
total_stock = 0
for idx, sizes in json_value["attributes"].items():
for getId in sizes["options"]:
first_loop.append({getId["label"]: getId["products"][0]})
break
for idx, test in json_value["productStockAlert"]["child"].items():
total_stock += test["stock_number"]
second_loop.append({test["productId"]: test["stock_number"]})
print("first_loop", first_loop)
print("second_loop", second_loop)
print("total_stock", total_stock)
which returns:
first_loop [{'38': '69813'}, {'38,5': '69817'}, {'39': '69818'}, {'40': '69819'}, {'40,5': '69820'}, {'41': '69821'}, {'42': '69822'}, {'42,5': '69823'}, {'43': '69824'}, {'44': '69814'}, {'44,5': '69815'}, {'45,5': '69816'}]
second_loop [{'69813': 0}, {'69814': 5}, {'69815': 3}, {'69816': 1}, {'69817': 0}, {'69818': 0}, {'69819': 0}, {'69820': 0}, {'69821': 3}, {'69822': 4}, {'69823': 6}, {'69824': 6}]
total_stock 28
My issue is how can I compare from first_loop where I check the ID (etc 69816) is in second_loop and has the value above 0, if its above 0 then I want to add it to a append it to a new list etc: 45,5 (1) (Which is the size number from first_loop and the number (value) from second_loop.
Output would end up:
>>> ["41 (3)", "42 (4)", "42,5 (6)", "43 (6)", "44 (5)", "44,5 (3)", "45,5 (1)"]

Basically, you just need to create id-label mapping, id-count mapping and merge them:
id_label_mapping = {o["products"][0]: o["label"] for o in next(iter(json_value["attributes"].values()))["options"]}
id_count_mapping = {o["productId"]: o["stock_number"] for o in json_value["productStockAlert"]["child"].values()}
result = [f"{l} ({id_count_mapping[k]})" for k, l in id_label_mapping.items() if id_count_mapping.get(k)]
In your code you've done 2 major mistakes which makes implementation of last step (merging) much harder.
You're creating list of dicts instead of single dict with different keys;
In first_loop you're using label as key, but in second_loop you're using productId.
If we will fix this 2 gaps, your code will work:
first_loop = {}
second_loop = {}
total_stock = 0
for idx, sizes in json_value["attributes"].items():
for getId in sizes["options"]:
first_loop[getId["products"][0]] = getId["label"]
break
for idx, test in json_value["productStockAlert"]["child"].items():
total_stock += test["stock_number"]
second_loop[test["productId"]] = test["stock_number"]
result = []
for product_id, label in first_loop.items():
count = second_loop.get(product_id)
if count: # filters both None (key doesn't exit) and 0
result.append(f"{label} ({count})")
print("result", result)
print("total_stock", total_stock)

Not sure if it's the most efficient way, but you could:
make dicts not lists, does it need to be a list?
swap the key-value of the first_loop
intersect the sets
get the values from original, print only if > 0
[Code not tested]
first_loop = {}
second_loop = {}
total_stock = 0
for idx, sizes in json_value["attributes"].items():
for getId in sizes["options"]:
first_loop[getId["products"][0]] = getId["label"]
break
for idx, test in json_value["productStockAlert"]["child"].items():
total_stock += test["stock_number"]
second_loop[test["productId"]] = test["stock_number"]
matching = set(first_loop.keys()).intersection(second_loop.keys())
for prod_id in matching:
stock = second_loop.get(prod_id)
if stock > 0:
print(f"{first_loop.get(prod_id) ({stock})")
Lastly, you have a break statement, that will make it run only one time... In which case you do not need a for loop...
It's quite too case specific, but hope it helps...

Elasticsearch bool query sort by date if status is true

I have a JSON file like as follows in an Elasticsearch index. I need to sort data if the advertisement does not expire and status is true, and then sort them as desc. How can I achieve this?
I tried using end_date sort, but it did not work. Also I need to show all expired data which end_date are expired.
advertisement = [
{
"id": 1,
"name": "test",
"status": True,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
},
{
"id": 2,
"name": "test2",
"status": False,
"start_date": "2020-08-09",
"end_date": "2020-08-09",
}]
This is my elastic search method.
def elastic_search(category=None):
client = Elasticsearch(host="localhost", port=9200)
query_all = {
'size': 10000,
'query': {
"bool": {
"filter": [
{
"match": {
"name": "test"
}
}]
},
},
"sort": [
{
"end_date": {
"type": "date",
"order": 'desc'
}
}
]
}
resp = client.search(
index="my-index",
body=query_all
)
return resp
This is my es response
http://localhost:9200/my-index/_search
{
"took":96,
"timed_out":false,
"_shards":{
"total":5,
"successful":5,
"skipped":0,
"failed":0
},
"hits":{
"total":36,
"max_score":1.0,
"hits":[
{
"_index":"my-index",
"_type":"doc",
"_id":"52",
"_score":1.0,
"_source":{
"id": 1,
"name": "test",
"status": True,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
}
},
{
"_index":"my-index",
"_type":"doc",
"_id":"60",
"_score":1.0,
"_source":{
"id": 1,
"name": "English test",
"status": True,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
}
},
{
"_index":"my-index",
"_type":"doc",
"_id":"40",
"_score":1.0,
"_source":{
"id": 1,
"name": "Designw test",
"status": false,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
}
},
{
"_index":"my-index",
"_type":"doc",
"_id":"41",
"_score":1.0,
"_source":{
"id": 1,
"name": "Designw New",
"status": false,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
}
},
{
"_index":"my-index",
"_type":"doc",
"_id":"59",
"_score":1.0,
"_source":{
"id": 1,
"name": "Designw New",
"status": false,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
}
},
{
"_index":"my-index",
"_type":"doc",
"_id":"62",
"_score":1.0,
"_source":{
"id": 1,
"name": "Designw New",
"status": false,
"start_date": "2020-08-09",
"end_date": "2020-09-09",
}
}
]
}
}
This is my mapping http://localhost:9200/my-index/_mapping response.
"my-index":{
"mappings":{
"_doc":{
"properties":{
"address":{
"properties":{
"name":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"start_date":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"end_date":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
},
"id":{
"type":"long"
},
"status":{
"type":"text",
"fields":{
"keyword":{
"type":"keyword",
"ignore_above":256
}
}
}
}
}
}
}
}
}
}

Two things regarding the mapping:
There's an address field there that's not found in your actual documents. Remove it.
Your dates should be mapped correctly using the date datatype.
A correct mapping would look like this:
{
"properties":{
"end_date":{
"type":"date",
"format":"yyyy-MM-dd"
},
"start_date":{
"type":"date",
"format":"yyyy-MM-dd"
},
//...other properties
}
}
Once you get the mapping right, this query looks for all non-expired ads w/ a true status and sorts by the longest running:
{
"query": {
"bool": {
"must": [
{
"range": {
"end_date": {
"gt": "now"
}
}
},
{
"term": {
"status": {
"value": true
}
}
}
]
}
},
"sort": [
{
"end_date": {
"order": "desc"
}
}
]
}
Alternatively, if you're looking for the expired ones, change gt to lt which stands for less-than.

Parsing AWS ATHENA outputs

Relatively new to Python here, coming from a node.js background, having quite a few issues parsing the output I get from get_query_results()
Documentation Link
I have been at this for some hours, i have tried iterating through the ['ResultSetMetadata']['ColumnInfo'] to grab the column names, but i don't know how to tie the ['ResultSet']['Data'] to these items so the code knows which name to apply to each dataValue.
I know i need to select the row headers then add the associated objects to those rows, but the logic on how to do such a thing in python escapes me.
I can see that the first column name always lines up with the first ['Data']['VarCharValue'] so I can get all the values in order, but if I loop through ['ResultSet']['Rows'] how do I isolate the first iteration as the column names to then populate with each other row?
Or is there a better way to do this?
Here is my json.dumps(ATHENAoutput)
{
"ResultSet": {
"Rows": [{
"Data": [{
"VarCharValue": "postcode"
}, {
"VarCharValue": "CountOf"
}]
}, {
"Data": [{
"VarCharValue": "1231"
}, {
"VarCharValue": "2"
}]
}, {
"Data": [{
"VarCharValue": "1166"
}, {
"VarCharValue": "2"
}]
}, {
"Data": [{
"VarCharValue": "3651"
}, {
"VarCharValue": "3"
}]
}, {
"Data": [{
"VarCharValue": "2171"
}, {
"VarCharValue": "2"
}]
}, {
"Data": [{
"VarCharValue": "4697"
}, {
"VarCharValue": "2"
}]
}, {
"Data": [{
"VarCharValue": "4450"
}, {
"VarCharValue": "2"
}]
}, {
"Data": [{
"VarCharValue": "4469"
}, {
"VarCharValue": "1"
}]
}],
"ResultSetMetadata": {
"ColumnInfo": [{
"Scale": 0,
"Name": "postcode",
"Nullable": "UNKNOWN",
"TableName": "",
"Precision": 2147483647,
"Label": "postcode",
"CaseSensitive": true,
"SchemaName": "",
"Type": "varchar",
"CatalogName": "hive"
}, {
"Scale": 0,
"Name": "CountOf",
"Nullable": "UNKNOWN",
"TableName": "",
"Precision": 19,
"Label": "CountOf",
"CaseSensitive": false,
"SchemaName": "",
"Type": "bigint",
"CatalogName": "hive"
}]
}
},
"ResponseMetadata": {
"RetryAttempts": 0,
"HTTPStatusCode": 200,
"RequestId": "18190e7c-901c-40b4-b6ef-10a5013b1a70",
"HTTPHeaders": {
"date": "Mon, 01 Oct 2018 04:51:14 GMT",
"x-amzn-requestid": "18190e7c-901c-40b4-b6ef-10a5013b1a70",
"content-length": "1464",
"content-type": "application/x-amz-json-1.1",
"connection": "keep-alive"
}
}
}
My desired Result is a JSON Array like the following:
[{
"postcode": "2171",
"CountOf": "2"
}, {
"postcode": "4697",
"CountOf": "2"
}, {
"postcode": "1166",
"CountOf": "2"
},
...
]

>>> def get_var_char_values(d):
... return [obj['VarCharValue'] for obj in d['Data']]
...
...
... header, *rows = input_data['ResultSet']['Rows']
... header = get_var_char_values(header)
... result = [dict(zip(header, get_var_char_values(row))) for row in rows]
>>> import json; print(json.dumps(result, indent=2))
[
{
"postcode": "4450",
"CountOf": "2"
},
{
"postcode": "1231",
"CountOf": "2"
},
{
"postcode": "4469",
"CountOf": "1"
},
{
"postcode": "3651",
"CountOf": "3"
},
{
"postcode": "1166",
"CountOf": "2"
},
{
"postcode": "4697",
"CountOf": "2"
},
{
"postcode": "2171",
"CountOf": "2"
}
]

str to dict in python, but maintain the sequence of json attributes

I've tried ast.literal_eval and json.loads but both of these, doesn't maintain the sequence of json attributes when a string is provided. Please see the following example -
String before providing it to json.loads -
{
"type": "array",
"properties": {
"name": {
"type": "string"
},
"i": {
"type": "integer"
},
"strList": {
"type": "array",
"items": {
"type": "string"
}
},
"strMap": {
"type": "object"
},
"p2": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"i": {
"type": "integer"
},
"p3": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"i": {
"type": "integer"
},
"p4": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"i": {
"type": "integer"
}
}
}
}
}
}
}
},
"p3": {
"type": "array",
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"i": {
"type": "integer"
},
"p4": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"i": {
"type": "integer"
}
}
}
}
}
},
"b": {
"type": "boolean",
"required": true
}
},
"classnames": {
"rootNode": {
"classname": "com.agent.Person"
},
"p2": {
"classname": "com.agent.Person2",
"p3": {
"classname": "com.agent.Person3",
"p4": {
"classname": "com.agent.Person4"
}
}
},
"p3": {
"classname": "com.agent.Person3",
"p4": {
"classname": "com.agent.Person4"
}
}
}
}
String after providing it to json.loads -
{
'classnames': {
'p2': {
'classname': 'com.agent.Person2',
'p3': {
'classname': 'com.agent.Person3',
'p4': {
'classname': 'com.agent.Person4'
}
}
},
'p3': {
'classname': 'com.agent.Person3',
'p4': {
'classname': 'com.agent.Person4'
}
},
'rootNode': {
'classname': 'com.agent.Person'
}
},
'properties': {
'b': {
'required': True,
'type': 'boolean'
},
'i': {
'type': 'integer'
},
'name': {
'type': 'string'
},
'p2': {
'items': {
'properties': {
'i': {
'type': 'integer'
},
'name': {
'type': 'string'
},
'p3': {
'properties': {
'i': {
'type': 'integer'
},
'name': {
'type': 'string'
},
'p4': {
'properties': {
'i': {
'type': 'integer'
},
'name': {
'type': 'string'
}
},
'type': 'object'
}
},
'type': 'object'
}
},
'type': 'object'
},
'type': 'array'
},
'p3': {
'items': {
'properties': {
'i': {
'type': 'integer'
},
'name': {
'type': 'string'
},
'p4': {
'properties': {
'i': {
'type': 'integer'
},
'name': {
'type': 'string'
}
},
'type': 'object'
}
},
'type': 'object'
},
'type': 'array'
},
'strList': {
'items': {
'type': 'string'
},
'type': 'array'
},
'strMap': {
'type': 'object'
}
},
'type': 'array'
}
Can anyone please suggest an alternative or something in python which keeps the sequence of attributes as they are and convert the string into the python dictionary?

As tobias_k has said, python dictionaries are unordered, so you'll lose any order information as soon as you load your data into one.
You can, however, load your JSON string into a OrderedDict:
from collections import OrderedDict
import json
json.loads(your_json_string, object_pairs_hook=OrderedDict)
This method is mentioned in the json module documentation

Categories