I am currently trying out something which I am unsure if it is possible.
I am trying to map API values from a JSON string (which has nested values) to a database field but I wish for it to be dynamic.
In the YAML example below, the key would be the database field name and the database field value would be where to obtain the information from the JSON string ("-" delimited for nested values). I am able to read the YAML config but what I don't understand is how to translate it to python code. If it were to be dynamic I have no idea how many [] I would have to put.
YAML: (PYYAML package)
employer: "properties-employer_name"
...
employee_name: "employee"
Python Code: (Python 3.8)
json_data = { properties: {employer_name: "XYZ"}, employee: "Sam" }
employer = json_data["properties"]["employer_name"] # How Do I add [] based on how nested the value is dynamically?
employee = json_data["employee"]
Many thanks!
You could try something like this:
def get_value(data, keys):
# Go over each key and adjust data value to current level
for key in keys:
data = data[key]
return data # Once last key is reached return value
You would get your keys by splitting on '-' if that is how you have it in your yaml so in my example I just saved the value to a string and did it this way:
employer = "properties-employer_name"
keys = employer.split('-') # Gives us ['properties', 'employer_name']
Now we can call our get_value function defined above:
get_value(json_data, keys)
Which returns 'XYZ'
Related
So I am struggling with getting a value from a JSON response. Looking in other post I have managed to write this code but when I try to search for the key (character_id) that I want in the dictionary python says that the key doesn't exist. My solution consists in getting the JSON object from the response, converting it into a string with json.dumps() and the converting it into a dictionary with json.loads(). Then I try to get 'character_id' from the dictionary but it doesn't exist. I am guessing it is related with the format of the dictionary but I have little to none experience in python. The code that makes the query and tries to get the values is this: (dataRequest is a fuction that makes the request and return the response from the api)
characterName = sys.argv[1];
response = dataRequest('http://census.daybreakgames.com/s:888/get/ps2:v2/character/?name.first_lower=' + characterName + '&c:show=character_id')
jsonString = json.dumps(response.json())
print(jsonString)
dic = json.loads(jsonString)
print(dic)
if 'character_id' in dic:
print(dic['character_id'])
The output of the code is:
{"character_list": [{"character_id": "5428662532301799649"}], "returned": 1}
{'character_list': [{'character_id': '5428662532301799649'}], 'returned': 1}
Welcome #Prieto! From what I can see, you probably don't need to serialize/de-serialize the JSON -- response.json() returns a python dictionary object already.
The issue is that you are looking for the 'character_id' key at the top-level of the dictionary, when it seems to be embedded inside another dictionary, that is inside a list. Try something like this:
#...omitted code
for char_obj in dic["character_list"]:
if "character_id" in char_obj:
print(char_obj["character_id"])
if your dic is like {"character_list": [{"character_id": "5428662532301799649"}], "returned": 1}
you get the value of character_id by
print(dic['character_list'][0][character_id])
The problem here is that you're trying to access a dictionary where the key is actually character_list.
What you need to do is to access the character_list value and iterate over or filter the character_id you want.
Like this:
print(jsonString)
dic = json.loads(jsonString)
print(dic)
character_information = dic['character_list'][0] # we access the character list and assume it is the first value
print(character_information["character_id"]) # this is your character id
The way I see it, the only hiccup with the code is this :
if 'character_id' in dic:
print(dic['character_id'])
The problem is that, the JSON file actually consists of actually 2 dictionaries , first is the main one, which has two keys, character_list and returned. There is a second sub-dictionary inside the array, which is the value for the key character_list.
So, what your code should actually look like is something like this:
for i in dic["character_list"]:
print(i["character_id"])
On a side-note, it will help to look at JSON file in this way :
{
"character_list": [
{
"character_id": "5428662532301799649"
}
],
"returned": 1
}
,where, elements enclosed in curly-brackets'{}' imply they are in a dictionary, whereas elements enclosed in curly-brackets'[]' imply they are in a list
I have a json file, which I will read and based on the xyz details will create excel report. Below is the sample json file I will use to extract the information which holds data in format of multiple dictionaries.
Now my requirement is to fetch xyz value one by one and based on it using certain field create a report. Below is the small snippet of the code where I am reading the file and based on key populating results. The data I am referencing after reading it from a file.
def pop_ws(dictionary,ws):
r=1
count=1
for k,v in dictionary.items():
offs=len(v['current'])
ws.cell(row=r+1,column=1).value = k
ws.cell(row=r+1,column=4).value = v['abc']
ws.cell(row=r+1,column=5).value = v['def']
wrk=read_cves(k)
count +=1
if wrk !='SAT':
ws.cell(row=r+1,column=7).value =k
ws.cell(row=r+1,column=8).value =tmp1['public_date']
if 'cvss' in list(tmp1.keys()):
.
.
.
def read_f(data):
with open(dat.json) as f:
wrk = f.read()
I am pretty much stuck on how to code in def read_f(data):, so that it read dat.json and based on value i.e data, fetch details defined as in dictionary structure one by one for all the required data and populate as defined under pop_ws in my code.
The data in def read_f(data): will be a dynamic value and based on it I need to filter the dictionary which have value (stored in data) defined against a key and then extract the whole dictionary into another json file.
Any suggestion on this will be appreciated.
Use json package to load json format data like below:
# Python program to read
# json file
import json
# Opening JSON file
f = open('data.json',)
# returns JSON object as
# a dictionary
data = json.load(f)
# Iterating through the json
# list
for i in data['emp_details']:
print(i)
# Closing file
f.close()
I got this from this link, now you can get dict from the file.
Next you can just filter the dict with specific value like below.
You should use filter() built-in function, with a function that returns True, if the dictionary contains one of the values.
def filter_func(dic, filterdic):
for k,v in filterdic.items():
if k == 'items':
if any(elemv in dic[k] for elemv in v):
return True
elif v == dic[k]:
return True
return False
def filter_cards(deck, filterdic):
return list(filter(lambda dic, filterdic=filterdic: filter_func(dic, filterdic) , deck))
You should use a dictionary as the second element.
filter_cards(deck, {'CVE': 'moderate'})
Hopefully, this could helpful for your situation.
Thanks.
Once you get your json object, you can access each value using the key like so:
print(json_obj["key"]) #prints the json value for that key
In your case
print(wrk["CVE"]) # prints CVE-2020-25624
I have MongoDB collection with many documents each with fields that looks like the one shown in the picture. The problem is with the field "searched". Its values are stored as a string because of which I cannot do a query for values like this {"searched.image_hash":"some_value"}. I use python to store values into MongoDB. In python, The variable "to_search" which is stored as "searched" in mongo is in fact a dictionary. Am not sure why the dictionary in "to_search" variable is stored as string within the mongodb "searched" field. Any suggestion as how to store the dictionary as array of object in mongodb?
The code I used in python is as follows
i have many other keys going into dictionary 'di'
di['account_id'] = acc_num
di['searched']= to_search
di['breakdown_queried'] = breakdown_to_query
di['combination']= [ele for ele in to_search.keys()]
di['ad_ids'] = ad_ids
di['date'] = date.today()
lo_str= ''
di = {k: str(v) for k, v in di.items()}
mongo_obj_remote.client["dev"]["ad_stats_tracker"].delete_one({"_id": {"$in": [di['_id']]}})
di_key_li = ['_id','account_id','date', 'combination','searched', 'ad_ids','breakdown_queried']
mongo_obj_remote.insert_single_document("dev", "ad_stats_tracker", {key: di[key] for key in di_key_li})
If your data in to_search is a python dict (and not a string that looks like one), then the pymongo drivers will store the data as a BSON "object", not a string; affectively this creates a sub-document within the document being stored in the collection.
Looking at your data I would suggest that your to_search is actually a string; the format is not valid for a dict as it contains a set (which pymongo won't be able to store in mongodb anyway).
You can check it with a small amount of debug code just before you insert the data:
print(type(di['searched']))
I am using the python code below to extract some values from an excel spreadsheet and then push them to an html page for further processing. I would like to modify the code below so that I can add additional values against each task, any help
the code below does spit out the following:
{'line items': {'AMS Upgrade': '30667', 'BMS works':
'35722'}}
How can I revise the code below so that I can add 2 more values against each task i.e. AMS Upgrade and BMS works
and get the likes of (note the structure below could be wrong)
{'line items': {'AMS Upgrade': {'30667','100%', '25799'}},{'BMS works':
{'10667','10%', '3572'}} }
Code:
book = xlrd.open_workbook("Example - supporting doc.xls")
first_sheet = book.sheet_by_index(-1)
nested_dict = {}
nested_dict["line items"] = {}
for i in range(21,175):
Line_items = first_sheet.row_slice(rowx=i, start_colx=2, end_colx=8)
if str(Line_items[0].value) and str(Line_items[1].value):
if not Line_items[5].value ==0 :
nested_dict["line items"].update({str(Line_items[0].value) : str(Line_items[1].value)})
print nested_dict
print json.dumps(nested_dict)
*** as requested see excel extract below
In Python, each key of a dict can only be associated with a single value. However that single value can be a dict, list, set, etc that holds many values.
You will need to decide the type to use for the value associated with the 'AMS Upgrade' key, if you want it to hold multiple values like '30667','10%', '222'.
Note: what you have written:
{'30667','100%', '25799'}
Is a set literal in Python.
I'm trying to process a log from Symphony using Pandas, but have some trouble with a malformed JSON which I can't parse.
An example of the log :
'{id:46025,
work_assignment:43313=>43313,
declaration:<p><strong>Bijkomende interventie.</strong></p>\r\n\r\n<p>H </p>\r\n\r\n<p><strong><em>Vaststellingen.</em></strong></p>\r\n\r\n<p><strong><em>CV. </em></strong>De.</p>=><p><strong>Bijkomende interventie.</strong></p>\r\n\r\n<p>He </p>\r\n\r\n<p><strong><em>Vaststellingen.</em></strong></p>\r\n\r\n<p><strong><em>CV. </em></strong>De.</p>,conclusions:<p>H </p>=><p>H </p>}'
What is the best way to process this?
For each part (id/work_assignment/declaration/etc) I would like to retrieve the old and new value (which are separated by "=>").
Use the following code:
def clean(my_log):
my_log.replace("{", "").replace("}", "") # Removes the unneeded { }
my_items = list(my_log.split(",")) # Split at the comma to get the pairs
my_dict = {}
for i in my_items:
key, value = i.split(":") # Split at the colon to separate the key and value
my_dict[key] = value # Add to the dictionary
return my_dict
Function returns a Python dictionary, which can then be converted to JSON using a serializer if needed, or directly used.
Hope I helped :D