We Keep Coding

Drift management of JSON configurations by comparing with dictionary data

I am trying to write a python code for drift management, which compares the application's configuration in JSON with the predefined dictionary of key-value pairs.
Ex: Application configuration in JSON:
{
"location": "us-east-1",
"properties": [
{
"type": "t2.large",
"os": "Linux"
}
],
"sgs": {
"sgid": "x-1234"
}
}
Ex: Dictionary with desired values to compare:
{
"os": "Windows",
"location": "us-east-1"
}
Expected output:
Difference is:
{
"os": "Windows"
}
I have been trying to convert the entire JSON (including sub dicts) into a single dict without sub dicts, and then iterate over it with each values of desired dict. I am able to print all the key, values in line but couldn't convert into a dict.
Is there a better way to do this? Or any references that could help me out here?
import json
def openJsonFile(file):
with open(file) as json_data:
workData = json.load(json_data)
return workData
def recursive_iter(obj):
if isinstance(obj, dict):
for item in obj.items():
yield from recursive_iter(item)
elif any(isinstance(obj, t) for t in (list, tuple)):
for item in obj:
yield from recursive_iter(item)
else:
yield obj
data = openJsonFile('file.json')
for item in recursive_iter(data):
print(item)
Expected output:
{
"location": "us-east-1",
"type": "t2.large",
"os": "Linux"
"sgid": "x-1234"
}

I think this will do what you say you want. I used the dictionary flattening code in this answer with a small modification — I changed it to not concatenate the keys of parent dictionaries with those of the nested ones since that seems to be what you want. This assumes that the keys used in all nested dictionaries are unique from one another, which in my opinion is a weakness of your approach.
You asked for references that could help you: Searching this website for related questions is often a productive way to find solutions to your own problems. This is especially true when what you want to know is something that has probably been asked before (such as how to flatten nested dictionaries).
Also note that I have written the code to closely follow the PEP 8 - Style Guide for Python Code guidelines — which I strongly suggest you read (and start following as well).
import json
desired = {
"os": "Windows",
"location": "us-east-1"
}
def read_json_file(file):
with open(file) as json_data:
return json.load(json_data)
def flatten(d):
out = {}
for key, val in d.items():
if isinstance(val, dict):
val = [val]
if isinstance(val, list):
for subdict in val:
deeper = flatten(subdict).items()
out.update({key2: val2 for key2, val2 in deeper})
else:
out[key] = val
return out
def check_drift(desired, config):
drift = {}
for key, value in desired.items():
if config[key] != value:
drift[key] = value
return drift
if __name__ == '__main__':
from pprint import pprint
config = flatten(read_json_file('config.json'))
print('Current configuration (flattened):')
pprint(config, width=40, sort_dicts=False)
drift = check_drift(desired, config)
print()
print('Drift:')
pprint(drift, width=40, sort_dicts=False)
This is the output it produces:
Current configuration (flattened):
{'location': 'us-east-1',
'type': 't2.large',
'os': 'Linux',
'sgid': 'x-1234'}
Drift:
{'os': 'Windows'}

Querying a nested JSON file in Python without Indexing

I have the below Json file which I need to query to get the values of the keys inside 'validations' in a list
for example the column_values_not_null output will need to be this:
['lu_name', 'transaction_amount']
"validation_file_name": "ctm",
"connection_type": "s3",
"low_threshold": 500000,
"high_threshold": 1000000,
"frequency": "weekly",
"validations": [
{
"columns_to_match_ordered_list" :[
"lu_name",
"site_name",
"transaction_date_time",
"margin",
"transaction_currency_code",
"reversal_indicator_description",
"reversal_amount",
"original_amount"
]
},
{
"column_values_not_null":[
"lu_name",
"transaction_amount"
]
},
{
"column_values_not_duplicate": [
"lu_name",
"response_code_description"
]
}
]
I am able to do the below but I need to do this without using the index value
f = open('test.json')
json_content = json.load(f)
print(json_content['validations'][1]['column_values_not_null'])

Get a list by querying the validations key. The sum( ,[]) are used to flat the list (as required by the condition "without using the index value" if got it right), for details about it with pros and cons see doc.
data = #
def validations(data: dict, key_query: str) -> list:
for k, v in data.items():
if k == 'validations':
return sum(sum([list(d.values()) for d in v if key_query in d], []), [])
print(validations(data, query='column_values_not_null'))
# ['lu_name', 'transaction_amount']

How to extract lists and modify them from different levels of nested dictionary

I want to extract all the lists and modify them from a nested dictionary which has multiple levels and the list can be a value in any of the levels.
For example:
test = {
'Type 1': {
'Type1_mainkey1': {
'Type1_key2': {
'Type1_key2_key1': [
'Type1_list1'
],
'Type1_key2_key2': [
'Type1_list2'
],
'Type1_key2_key2': [
'Type1_list3',
'Type1_list4'
]
}
}
},
'Type 2': {
'Type2_mainkey1': {
'Type2_key2': [
'Type2_list1',
'Type2_list2'
]},
'Type2_key3': {
'Type2_key3_key1': [
'Type2_list3',
'Type2_list4'
]
}
}
}
This is the kind of dictionary that might be present. I was wondering if there is a way of extracting the lists and updating the dictionary.
My function so far:
def find_list(data):
if not any([isinstance(data.get(k), dict) for k in data]):
return data
else:
for dkey in data:
if isinstance(data.get(dkey), dict):
return find_list(data.get(dkey))
else:
continue
And on running this:
out = find_list(test)
My output is:
{'Type1_key2_key1': ['Type1_list1'],
'Type1_key2_key2': ['Type1_list3', 'Type1_list4']}
Whereas, the expected output is all the list items and their keys (so that I can modify the list and update)

In my example below the only thing I am doing differently is collecting the results during each recursive step and adding to previous returns. I added some inline comments to hopefully better explain.
def find_list(data):
d = {} # initialize collection object to hold results
if not any([isinstance(data.get(k), dict) for k in data]):
return data
else:
for dkey in data:
if isinstance(data.get(dkey), dict):
# add results of each recursive call to the collection
d.update(find_list(data.get(dkey)))
return d # return the collection of results
Using your test dictionary with this example gives this output:
{
'Type1_key2_key1': ['Type1_list1'],
'Type1_key2_key2': ['Type1_list3', 'Type1_list4'],
'Type2_key2': ['Type2_list1', 'Type2_list2'],
'Type2_key3_key1': ['Type2_list3', 'Type2_list4']
}

Processing dictionary keys in arbitrary order

I want to transform dictionary into a string. What would be beginner-level question is complicated by few rules that I have to adhere to:
There is a list of known keys that must come out in particular, arbitrary order
Each of known keys is optional, i.e. it may not be present in dictionary
It is guaranteed that at least one of known keys will be present in dictionary
Dictionary may contain additional keys; they must come after known keys and their order is not important
I cannot make assumptions about order in which keys will be added to dictionary
What is the pythonic way of processing some dictionary keys before others?
So far, I have following function:
def format_data(input_data):
data = dict(input_data)
output = []
for key in ["title", "slug", "date", "modified", "category", "tags"]:
if key in data:
output.append("{}: {}".format(key.title(), data[key]))
del data[key]
if data:
for key in data:
output.append("{}: {}".format(key.title(), data[key]))
return "\n".join(output)
data = {
"tags": "one, two",
"slug": "post-title",
"date": "2017-02-01",
"title": "Post Title",
}
print(format_data(data))
data = {
"format": "book",
"title": "Another Post Title",
"date": "2017-02-01",
"slug": "another-post-title",
"custom": "data",
}
print(format_data(data))
Title: Post Title
Slug: post-title
Date: 2017-02-01
Tags: one, two
Title: Another Post Title
Slug: another-post-title
Date: 2017-02-01
Custom: data
Format: book
While this function does provide expected results, it has some issues that makes me think there might be better approach. Namely, output.append() line is duplicated and input data structure is copied to allow it's modification without side-effects.
To sum up, how can I process some keys in particular order and before other keys?

I suggest that you simply run a pair of list comprehensions: one for the desired keys, and one for the rest. Concatenate them in the desired order in bulk, rather than one at a time. This reduces the critical step to a single command to build output.
The first comprehension looks for desired keys in the dict; the second looks for any dict keys not in the "desired" list.
def format_data(input_data):
data = dict(input_data)
key_list = ["title", "slug", "date", "modified", "category", "tags"]
output = ["{}: {}".format(key.title(), data[key]) for key in key_list if key in data] + \
["{}: {}".format(key.title(), data[key]) for key in data if key not in key_list]
return "\n".join(output)

I'd suggest list comprehensions and pop():
def format_data(input_data):
data = dict(input_data)
keys = ["title", "slug", "date", "modified", "category", "tags"]
output = ['{}: {}'.format(key.title(), data.pop(key)) for key in keys if key in data]
output.extend(['{}: {}'.format(key.title(), val) for key, val in data.items()])
return "\n".join(output)
To the concern about deleting during iteration - note that the iteration is over the list of keys, not the dictionary being evaluated, so I wouldn't consider that a red flag.

To completely edit, the below will take a list of primary keys (you can pass them in if you want or set it in a config file) and then it will set those in the beginning of your dictionary.
I think I see what you mean now:
Try this:
from collections import OrderedDict
data = {'aaa': 'bbbb',
'custom': 'data',
'date': '2017-02-01',
'foo': 'bar',
'format': 'book',
'slug': 'another-post-title',
'title': 'Another Post Title'}
def format_data(input_data):
primary_keys = ["title", "slug", "date", "modified", "category", "tags"]
data = OrderedDict((k, input_data.get(k)) for k in primary_keys + input_data.keys())
output = []
for key, value in data.items():
if value:
output.append("{}: {}".format(key.title(), value))
return "\n".join(output)
print(format_data(data))
Title: Another Post Title
Slug: another-post-title
Date: 2017-02-01
Aaa: bbbb
Format: book
Custom: data
Foo: bar

Find the difference between the known keys and the keys in the input dictionary; Use itertools.chain to iterate over both sets of keys; catch KeyErrors for missing keys and just pass. No need to copy the input and no duplication.
import itertools
def format_data(input_data):
known_keys = ["title", "slug", "date", "modified", "category", "tags"]
xtra_keys = set(input_data.keys()).difference(known_keys)
output = []
for key in itertools.chain(known_keys, xtra_keys):
try:
output.append("{}: {}".format(key.title(), data[key]))
except KeyError as e:
pass
return '\n'.join(output)
data = {"tags": "one, two",
"slug": "post-title",
"date": "2017-02-01",
"title": "Post Title",
"foo": "bar"}
>>> print format_data(data)
Title: Post Title
Slug: post-title
Date: 2017-02-01
Tags: one, two
Foo: bar
>>>

Sorting and organizing DNS records

So I have a fairly simple set of data, such as:
['test.sh','api.test.sh','blah.api.test.sh','test.com','api.test.com']
and I need to transform into a hierarchical data structure, I was thinking of doing it with a dictionary:
{ 'name':'test.sh',
'children': { 'name':'api.test.sh',
'children': { 'name':'blah.api.test.sh' }
}
},
{
'name':'test.com',
'children': { 'name':'api.test.com' }
}
And essentially for each high level name I can work my way down and perform the operations I need to do.
My question has more to do with creating a simple way to sort, match, and transform the data. I can think of a few ways to do this but I can't think of anything quite elegant. Also I'm doing this in python.
Thanks

I think this could be what you are looking for:
def sort_dns(l):
to_return = []
# Get top-level domains: the domains that contain the less amount of dots.
count_list = [i.count('.') for i in l]
min_dots = min(count_list)
top_domains = [i for i in l if i.count('.') == min_dots]
# Now for each domain, we find it subdomains.
for domain in top_domains:
sub_domains = [i for i in l if domain in i and i is not domain]
#And untill we aren't at the deepest level, we continue looking for sub domains and repeat the structure
sub_sub_domains = sort_dns(sub_domains) if not len(sub_domains) == 0 else None
to_return.append({'name' : domain, 'childrens' : sub_sub_domains})
return to_return
As you see this function call itself recursively to go infinitely "deep" if needed.
With your example, the result is the following
[
{
'name': 'test.sh',
'childrens': [
{
'name': 'api.test.sh',
'childrens': [
{'name': 'blah.api.test.sh', 'childrens': None}
]
}
]
},
{
'name': 'test.com',
'childrens': [
{'name': 'api.test.com', 'childrens': None}
]
}
]
As you see it handle the case of multiple childrens and no children at all.
Note that if you don't want the 'childrens': None, You can change the function to that:
def sort_dns(l):
to_return = []
# Get top-level domains: the domains that contain the less amount of dots.
count_list = [i.count('.') for i in l]
min_dots = min(count_list)
top_domains = [i for i in l if i.count('.') == min_dots]
# Now for each domain, we find it subdomains.
for domain in top_domains:
sub_domains = [i for i in l if domain in i and i is not domain]
#And untill we aren't at the deepest level, we continue looking for sub domains and repeat the structure
sub_sub_domains = sort_dns(sub_domains) if not len(sub_domains) == 0 else None
if sub_sub_domains:
to_return.append({'name' : domain, 'childrens' : sub_sub_domains})
else:
to_return.append({'name' : domain})
return to_return
Note that this is Python3 Code.
EDIT: I've read roippi Answer and this works great too, His solution is surely the most pythonic. This one's advantage is that it doesn't require any imports. But you should really consider roippi answer as the most elegant.

So, I see a proper approach to this problem happening in three steps: sort, group, format.
First, sorting your inputs to arrange them in logical groups. You can define a quick helper function to define your sort key:
def sorter(netloc):
split = netloc.split('.')
return (split[::-1], -len(split))
And use it thusly:
data = ['test.sh','api.test.sh','blah.api.test.sh','test.com','api.test.com', 'another.com', 'sub.another.com', 'sub.sub.another.com']
#shuffling data, to show that sorting works
import random
random.shuffle(data)
sorted(data, key=sorter)
Out[14]:
['another.com',
'sub.another.com',
'sub.sub.another.com',
'test.com',
'api.test.com',
'test.sh',
'api.test.sh',
'blah.api.test.sh']
Now that everything's in the correct order, do a similar grouping operation with itertools.groupby which groups by the blah.com part of x.y.z.blah.com:
def grouper(netloc):
return ''.join(netloc.split('.')[-2:])
#in-place sort, replicating sorted() call above
data.sort(key=sorter)
from itertools import groupby
[list(g) for k,g in groupby(data, grouper)]
Out[27]:
[['another.com', 'sub.another.com', 'sub.sub.another.com'],
['test.com', 'api.test.com'],
['test.sh', 'api.test.sh', 'blah.api.test.sh']]
Lastly, you need to format these groups into your desired hierarchy. Here is a quick-and-dirty implementation:
def make_hierarchy(groups):
from copy import deepcopy
_groups = deepcopy(groups)
ret = []
for li in _groups:
current = {}
ret.append(current)
while li:
current['name'] = li.pop()
if li:
nxt = {}
current['children'] = nxt
current = nxt
return ret
print(json.dumps(make_hierarchy(grouped), indent=2))
[
{
"children": {
"children": {
"name": "another.com"
},
"name": "sub.another.com"
},
"name": "sub.sub.another.com"
},
{
"children": {
"name": "test.com"
},
"name": "api.test.com"
},
{
"children": {
"children": {
"name": "test.sh"
},
"name": "api.test.sh"
},
"name": "blah.api.test.sh"
}
]
This last implementation depends on a couple of assumptions, namely that there will not be any equivalent-length netlocs in a given group, i.e. sub1.example.com and sub2.example.com will never happen. Obviously you can tweak the implementation as needed.