We Keep Coding

Create a new dictionary from a nested JSON output after parsing

In python3 I need to get a JSON response from an API call,
and parse it so I will get a dictionary That only contains the data I need.
The final dictionary I ecxpt to get is as follows:
{'Severity Rules': ('cc55c459-eb1a-11e8-9db4-0669bdfa776e', ['cc637182-eb1a-11e8-9db4-0669bdfa776e']), 'auto_collector': ('57e9a4ec-21f7-4e0e-88da-f0f1fda4c9d1', ['0ab2470a-451e-11eb-8856-06364196e782'])}
the JSON response returns the following output:
{
'RuleGroups': [{
'Id': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rules',
'Order': 1,
'Enabled': True,
'Rules': [{
'Id': 'cc637182-eb1a-11e8-9db4-0669bdfa776e',
'Name': 'Severity Rule',
'Description': 'Look for default severity text',
'Enabled': False,
'RuleMatchers': None,
'Rule': '\\b(?P<severity>DEBUG|TRACE|INFO|WARN|ERROR|FATAL|EXCEPTION|[I|i]nfo|[W|w]arn|[E|e]rror|[E|e]xception)\\b',
'SourceField': 'text',
'DestinationField': 'text',
'ReplaceNewVal': '',
'Type': 'extract',
'Order': 21520,
'KeepBlockedLogs': False
}],
'Type': 'user'
}, {
'Id': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c',
'Name': 'auto_collector',
'Order': 4,
'Enabled': True,
'Rules': [{
'Id': '2d6bdc1d-4064-11eb-8856-06364196e782',
'Name': 'auto_collector',
'Description': 'DO NOT CHANGE!! Created via API coralogix-blocker tool',
'Enabled': False,
'RuleMatchers': None,
'Rule': 'AUTODISABLED',
'SourceField': 'subsystemName',
'DestinationField': 'subsystemName',
'ReplaceNewVal': '',
'Type': 'block',
'Order': 1,
'KeepBlockedLogs': False
}],
'Type': 'user'
}]
}
I was able to create a dictionary that contains the name and the RuleGroupsID, like that:
response = requests.get(url,headers=headers)
output = response.json()
outputlist=(output["RuleGroups"])
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
# Create a dictionary of NAME + ID
ruleDic = {}
for key in groupRuleName:
for value in groupRuleID:
ruleDic[key] = value
groupRuleID.remove(value)
break
Which gave me a simple dictionary:
{'Severity Rules': 'cc55c459-eb1a-11e8-9db4-0669bdfa776e', 'Rewrites': 'ddbaa27e-1747-11e9-9db4-0669bdfa776e', 'Extract': '0cb937b6-2354-d23a-5806-4559b1f1e540', 'auto_collector': '4f6fa7c6-d60f-49cd-8c3d-02dcdff6e54c'}
but when I tried to parse it as nested JSON things just didn't work.

In the end, I managed to create a function that returns this dictionary,
I'm doing it by breaking the JSON into 3 lists by the needed elements (which are Name, Id, and Rules from the first nest), and then create another list from the nested JSON ( which listed everything under Rule) which only create a list from the keyword "Id".
Finally creating a dictionary using a zip command on the lists and dictionaries created earlier.
def get_filtered_rules() -> List[dict]:
groupRuleName = [li['Name'] for li in outputlist]
groupRuleID = [li['Id'] for li in outputlist]
ruleIDList = [li['Rules'] for li in outputlist]
ruleIDListClean = []
ruleClean = []
for sublist in ruleIDList:
try:
lstRule = [item['Rule'] for item in sublist]
ruleClean.append(lstRule)
ruleContent=list(zip(groupRuleName, ruleClean))
ruleContentDictionary = dict(ruleContent)
lstID = [item['Id'] for item in sublist]
ruleIDListClean.append(lstID)
# Create a dictionary of NAME + ID + RuleID
ruleDic = dict(zip(groupRuleName, zip(groupRuleID, ruleIDListClean)))
except Exception as e: print(e)
return ruleDic

Conditional list Python

I am opening all the files present in zip, then converting the string to List and thereafter, breaking the list into 5 elements from each line.
My code:
with zipfile.ZipFile(dummy.zip, 'r') as mz:
for f in mz.namelist():
data = mz.read(f).decode('utf-8').replace("\n", '').replace("\r", ',').lstrip().rstrip().split(",") #Removing all spaces and \r from text.
allist = [data[i:i + 5] for i in range(len(data))[::5]] #breaking list into 5 elements each
I am able to do it perfectly if all values are present but if any of the element in missing on particular line then it picks up the value from second line but I want to add NA at place of missing element.
File data:
Dummy.zip = "File1 + File2"
File1:
Yuli, yu#test.com, 0001, 8902
Su, su#test.com, 0002, 8903, Manager
File:2
Zaon, zn#test.com, 100, 9087, Analyst
June, ju#test.com, 278, 6078
Purpose of the code:
I am adding converting this list to dict, so adding keys to each entry as name, email, ext, Empid, Level

You can do it like this:
texts = [b"Yuli, yu#test.com, 0001, 8902\r\nSu, su#test.com, 0002, 8903, Manager",
b"Zaon, zn#test.com, 100, 9087, Analyst\r\nJune, ju#test.com, 278, 6078"]
result =[]
keys = "name,email,ext,Empid,Level".split(",")
# you do:
# with zipfile.ZipFile(dummy.zip, 'r') as mz:
# for f in mz.namelist():
# t = mz.read(f) here instead
# instead
for t in texts:
for line in t.decode("utf-8").split("\n"):
d = {k:"NA" for k in keys} # init all keys with NA
# update with actual values
d.update(zip(keys,(l.strip() for l in line.strip().split(","))))
# add to result list
result.append(d)
print(result)
Output:
[{'name': 'Yuli', 'email': 'yu#test.com', 'ext': '0001', 'Empid': '8902', 'Level': 'NA'},
{'name': 'Su', 'email': 'su#test.com', 'ext': '0002', 'Empid': '8903', 'Level': 'Manager'},
{'name': 'Zaon', 'email': 'zn#test.com', 'ext': '100', 'Empid': '9087', 'Level': 'Analyst'},
{'name': 'June', 'email': 'ju#test.com', 'ext': '278', 'Empid': '6078', 'Level': 'NA'}]

Find item in a list of dictionaries

I have this data
data = [
{
'id': 'abcd738asdwe',
'name': 'John',
'mail': 'test#test.com',
},
{
'id': 'ieow83janx',
'name': 'Jane',
'mail': 'test#foobar.com',
}
]
The id's are unique, it's impossible that multiple dictonaries have the same id.
For example I want to get the item with the id "ieow83janx".
My current solution looks like this:
search_id = 'ieow83janx'
item = [x for x in data if x['id'] == search_id][0]
Do you think that's the be solution or does anyone know an alternative solution?

Since the ids are unique, you can store the items in a dictionary to achieve O(1) lookup.
lookup = {ele['id']: ele for ele in data}
then you can do
user_info = lookup[user_id]
to retrieve it

If you are going to get this kind of operations more than once on this particular object, I would recommend to translate it into a dictionary with id as a key.
data = [
{
'id': 'abcd738asdwe',
'name': 'John',
'mail': 'test#test.com',
},
{
'id': 'ieow83janx',
'name': 'Jane',
'mail': 'test#foobar.com',
}
]
data_dict = {item['id']: item for item in data}
#=> {'ieow83janx': {'mail': 'test#foobar.com', 'id': 'ieow83janx', 'name': 'Jane'}, 'abcd738asdwe': {'mail': 'test#test.com', 'id': 'abcd738asdwe', 'name': 'John'}}
data_dict['ieow83janx']
#=> {'mail': 'test#foobar.com', 'id': 'ieow83janx', 'name': 'Jane'}
In this case, this lookup operation will cost you some constant* O(1) time instead of O(N).

How about the next built-in function (docs):
>>> data = [
... {
... 'id': 'abcd738asdwe',
... 'name': 'John',
... 'mail': 'test#test.com',
... },
... {
... 'id': 'ieow83janx',
... 'name': 'Jane',
... 'mail': 'test#foobar.com',
... }
... ]
>>> search_id = 'ieow83janx'
>>> next(x for x in data if x['id'] == search_id)
{'id': 'ieow83janx', 'name': 'Jane', 'mail': 'test#foobar.com'}
EDIT:
It raises StopIteration if no match is found, which is a beautiful way to handle absence:
>>> search_id = 'does_not_exist'
>>> try:
... next(x for x in data if x['id'] == search_id)
... except StopIteration:
... print('Handled absence!')
...
Handled absence!

Without creating a new dictionary or without writing several lines of code, you can simply use the built-in filter function to get the item lazily, not checking after it finds the match.
next(filter(lambda d: d['id']==search_id, data))
should for just fine.

Would this not achieve your goal?
for i in data:
if i.get('id') == 'ieow83janx':
print(i)
(xenial)vash#localhost:~/python$ python3.7 split.py
{'id': 'ieow83janx', 'name': 'Jane', 'mail': 'test#foobar.com'}
Using comprehension:
[i for i in data if i.get('id') == 'ieow83janx']

if any(item['id']=='ieow83janx' for item in data):
#return item
As any function returns true if iterable (List of dictionaries in your case) has value present.
While using Generator Expression there will not be need of creating internal List. As there will not be duplicate values for the id in List of dictionaries, any will stop the iteration until the condition returns true. i.e the generator expression with any will stop iterating on shortcircuiting. Using List comprehension will create a entire List in the memory where as GE creates the element on the fly which will be better if you are having large items as it uses less memory.

Python: retrieve arbitrary dictionary path and amend data?

Simple Python question, but I'm scratching my head over the answer!
I have an array of strings of arbitrary length called path, like this:
path = ['country', 'city', 'items']
I also have a dictionary, data, and a string, unwanted_property. I know that the dictionary is of arbitrary depth and is dictionaries all the way down, with the exception of the items property, which is always an array.
[CLARIFICATION: The point of this question is that I don't know what the contents of path will be. They could be anything. I also don't know what the dictionary will look like. I need to walk down the dictionary as far as the path indicates, and then delete the unwanted properties from there, without knowing in advance what the path looks like, or how long it will be.]
I want to retrieve the parts of the data object (if any) that matches the path, and then delete the unwanted_property from each.
So in the example above, I would like to retrieve:
data['country']['city']['items']
and then delete unwanted_property from each of the items in the array. I want to amend the original data, not a copy. (CLARIFICATION: By this I mean, I'd like to end up with the original dict, just minus the unwanted properties.)
How can I do this in code?
I've got this far:
path = ['country', 'city', 'items']
data = {
'country': {
'city': {
'items': [
{
'name': '114th Street',
'unwanted_property': 'foo',
},
{
'name': '8th Avenue',
'unwanted_property': 'foo',
},
]
}
}
}
for p in path:
if p == 'items':
data = [i for i in data[p]]
else:
data = data[p]
if isinstance(data, list):
for d in data:
del d['unwanted_property']
else:
del data['unwanted_property']
The problem is that this doesn't amend the original data. It also relies on items always being the last string in the path, which may not always be the case.
CLARIFICATION: I mean that I'd like to end up with:
{
'country': {
'city': {
'items': [
{
'name': '114th Street'
},
{
'name': '8th Avenue'
},
]
}
}
}
Whereas what I have available in data is only [{'name': '114th Street'}, {'name': '8th Avenue'}].
I feel like I need something like XPath for the dictionary.

The problem you are overwriting the original data reference. Change your processing code to
temp = data
for p in path:
temp = temp[p]
if isinstance(temp, list):
for d in temp:
del d['unwanted_property']
else:
del temp['unwanted_property']
In this version, you set temp to point to the same object that data was referring to. temp is not a copy, so any changes you make to it will be visible in the original object. Then you step temp along itself, while data remains a reference to the root dictionary. When you find the path you are looking for, any changes made via temp will be visible in data.
I also removed the line data = [i for i in data[p]]. It creates an unnecessary copy of the list that you never need, since you are not modifying the references stored in the list, just the contents of the references.
The fact that path is not pre-determined (besides the fact that items is going to be a list) means that you may end up getting a KeyError in the first loop if the path does not exist in your dictionary. You can handle that gracefully be doing something more like:
try:
temp = data
for p in path:
temp = temp[p]
except KeyError:
print('Path {} not in data'.format(path))
else:
if isinstance(temp, list):
for d in temp:
del d['unwanted_property']
else:
del temp['unwanted_property']

The problem you are facing is that you are re-assigning the data variable to an undesired value. In the body of your for loop you are setting data to the next level down on the tree, for instance given your example data will have the following values (in order), up to when it leaves the for loop:
data == {'country': {'city': {'items': [{'name': '114th Street', 'unwanted_property': 'foo',}, {'name': '8th Avenue', 'unwanted_property': 'foo',},]}}}
data == {'city': {'items': [{'name': '114th Street', 'unwanted_property': 'foo',}, {'name': '8th Avenue', 'unwanted_property': 'foo',},]}}
data == {'items': [{'name': '114th Street', 'unwanted_property': 'foo',}, {'name': '8th Avenue', 'unwanted_property': 'foo',},]}
data == [{'name': '114th Street', 'unwanted_property': 'foo',}, {'name': '8th Avenue', 'unwanted_property': 'foo',},]
Then when you delete the items from your dictionaries at the end you are left with data being a list of those dictionaries as you have lost the higher parts of the structure. Thus if you make a backup reference for your data you can get the correct output, for example:
path = ['country', 'city', 'items']
data = {
'country': {
'city': {
'items': [
{
'name': '114th Street',
'unwanted_property': 'foo',
},
{
'name': '8th Avenue',
'unwanted_property': 'foo',
},
]
}
}
}
data_ref = data
for p in path:
if p == 'items':
data = [i for i in data[p]]
else:
data = data[p]
if isinstance(data, list):
for d in data:
del d['unwanted_property']
else:
del data['unwanted_property']
data = data_ref

def delKey(your_dict,path):
if len(path) == 1:
for item in your_dict:
del item[path[0]]
return
delKey( your_dict[path[0]],path[1:])
data
{'country': {'city': {'items': [{'name': '114th Street', 'unwanted_property': 'foo'}, {'name': '8th Avenue', 'unwanted_property': 'foo'}]}}}
path
['country', 'city', 'items', 'unwanted_property']
delKey(data,path)
data
{'country': {'city': {'items': [{'name': '114th Street'}, {'name': '8th Avenue'}]}}}

You need to remove the key unwanted_property.
names_list = []
def remove_key_from_items(data):
for d in data:
if d != 'items':
remove_key_from_items(data[d])
else:
for item in data[d]:
unwanted_prop = item.pop('unwanted_property', None)
names_list.append(item)
This will remove the key. The second parameter None is returned if the key unwanted_property does not exist.
EDIT:
You can use pop even without the second parameter. It will raise KeyError if the key does not exist.
EDIT 2: Updated to recursively go into depth of data dict until it finds the items key, where it pops the unwanted_property as desired and append into the names_list list to get the desired output.

Using operator.itemgetter you can compose a function to return the final key's value.
import operator, functools
def compose(*functions):
'''returns a callable composed of the functions
compose(f, g, h, k) -> f(g(h(k())))
'''
def compose2(f, g):
return lambda x: f(g(x))
return functools.reduce(compose2, functions, lambda x: x)
get_items = compose(*[operator.itemgetter(key) for key in path[::-1]])
Then use it like this:
path = ['country', 'city', 'items']
unwanted_property = 'unwanted_property'
for thing in get_items(data):
del thing[unwanted_property]
Of course if the path contains non-existent keys it will throw a KeyError - you probably should account for that:
path = ['country', 'foo', 'items']
get_items = compose(*[operator.itemgetter(key) for key in path[::-1]])
try:
for thing in get_items(data):
del thing[unwanted_property]
except KeyError as e:
print('missing key:', e)

You can try this:
path = ['country', 'city', 'items']
previous_data = data[path[0]]
previous_key = path[0]
for i in path:
previous_data = previous_data[i]
previous_key = i
if isinstance(previous_data, list):
for c, b in enumerate(previous_data):
if "unwanted_property" in b:
del previous_data[c]["unwanted_property"]
current_dict = {}
previous_data_dict = {}
for i, a in enumerate(path):
if i == 0:
current_dict[a] = data[a]
previous_data_dict = data[a]
else:
if a == previous_key:
current_dict[a] = previous_data
else:
current_dict[a] = previous_data_dict[a]
previous_data_dict = previous_data_dict[a]
data = current_dict
print(data)
Output:
{'country': {'city': {'items': [{'name': '114th Street'}, {'name': '8th Avenue'}]}}, 'items': [{'name': '114th Street'}, {'name': '8th Avenue'}], 'city': {'items': [{'name': '114th Street'}, {'name': '8th Avenue'}]}}

How to print the keys and values of a dictionary as table? [duplicate]

This question already has answers here:
List of dicts to/from dict of lists
(14 answers)
Closed 6 years ago.
I have a dictionary that stores a list of items for each key as shown:
name_dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
I want to print the data in the dictionary as table format:
FirstName,MiddleName,LastName # the keys of the dictionary
Elizabeth,H.,Perkins #the values of the keys in one line (the first item in the list)
Scott,T.,Joseph #the values of the keys in new line (the following item in the list)
How to solve this problem?
I have tried doing the suggested solution by Gareth Latty, but that did not work.
with open('C:/Output.csv',"w+") as file:
w = csv.DictWriter(file,name_dict.keys())
w.writeheader()
w.writerow(name_dict)
It outputs the following:
MiddleName,LastName,FirstName
"['H.', 'T.']","['Perkins', 'Joseph']","['Perkins', 'Joseph']"
Any idea how to output the values (the item in the list) of each of the keys in new row?

csv.DictWriter expects a dictionary with the field:single_line pairs for each row which is unfortunately not what you have, you basically need to convert your data structure to be a list of dicts for the single lines:
[{'MiddleName': 'H.', 'FirstName': 'Elizabeth ', 'LastName': 'Perkins'}, {'MiddleName': 'T.', 'FirstName': 'Scott ', 'LastName': 'Joseph'}]
You can convert it with something like this:
import csv
def seperate_entries(dict_of_lists):
iters = [(k,iter(v)) for k,v in dict_of_lists.items()]
try:
while True:
yield {k:next(v) for k,v in iters}
except StopIteration:
return
name_dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
with open('sample.csv',"w+") as file:
w = csv.DictWriter(file,name_dict.keys())
w.writeheader()
w.writerows(seperate_entries(name_dict))

I think you misuse dict. When you have multiple values, you should use a list of dicts and not a dict which the values are lists. Instead of
Dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
You should do:
Dict = [{'FirstName': 'Elizabeth', 'MiddleName': 'H.', 'LastName': 'Perkins'}, {'FirstName': 'Joseph', 'MiddleName': 'T. ', 'LastName': 'Scott'}]
or in a more readable version:
Dict = [
{'FirstName': 'Elizabeth', 'MiddleName': 'H.', 'LastName': 'Perkins'},
{'FirstName': 'Joseph', 'MiddleName': 'T. ', 'LastName': 'Scott' }
]
If you want to print one line (one dictionary of the list), you can do something like that:
def printDict(d):
print d["FirstName"] + "," + d["MiddleName"] + "," + d["LastName"]
And if you want to print each of the elements in the list you have:
def printList(l):
for i in l:
printDict(i)
And just use it like that:
printList(Dict)
With your first (original) Dict, accessing Dict["FirstName"] would return a list, and when printed it would print as:
["Elizabeth", "Joesph"]
But with the second (new way I suggested) Dict, accessing Dict[0]["FirstName"] would return a string, and will print like:
Elizabeth

To access the keys in the Dictionary, you just need to do the following:
middleNames=Dict['Middlename']
firstNames=Dict['FirstName']
lastNames=Dict['LastName']
You now have access to the values stored in the inner list information, this can similarly be accessed by the following:
# Find how many items read (optional)
len(middleNames)
# Then iterate through the items
for mName in middleName:
print mName # this will print each list item e.g.
H.
T.
# for brevity, this is the same as doing this...
middleName[0] etc
Hope this helps.

You need define how many row you have.
Just flat it into rows with all keys in Dict.
import csv
Dict = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
len_row = 2
with open('Output.csv', "w+") as file:
w = csv.DictWriter(file, Dict.keys())
w.writeheader()
for i in range(len_row):
row = {}
for k in Dict.keys():
for v in Dict.values():
row[k] = Dict[k][i]
w.writerow(row)

I'm sure there are more efficient ways of achieving what you want but this lays down a simple outline of what you want and shows how you can achieve it.
names = {'MiddleName': ['H.', 'T.'], 'LastName': ['Perkins', 'Joseph'], 'FirstName': ['Elizabeth ', 'Scott ']}
output = open('output.txt','w')
#NOTE: this will write headers in alphabetical format. You will need to change that so it follows the FirstName, MiddleName, LastName pattern
for key in sorted(names.keys()):
output.write(key + ',')
output.write('\n')
#assuming we always have same amount of middle, first, and last names
for i in range(len(names['FirstName'])):
personFname = ''
personLname = ''
personMname = ''
for key in names.keys():
if key == 'MiddleName':
personMname = names[key][i]
elif key == 'FirstName':
personFname = names[key][i]
elif key == 'LastName':
personLname = names[key][i]
output.write(personFname + ',' + personMname + ',' + personLname)
output.write('\n')
output.close()