We Keep Coding

retreive key from nested dict based on value, where key names are unknown

I have the following dict:
{
'foo': {
'name': 'bar',
'options': None,
'type': 'qux'
},
'baz': {
'name': 'grault',
'options': None,
'type': 'plugh'
},
}
The names of the top level keys are unknown at runtime. I am unable to figure out how to get the name of the top level key where the value of type is plugh. I have tried all kinds of iterators, loops, comprehensions etc, but i'm not great with Python. Any pointers would be appreciated.

Try this:
for key, inner_dict in dict_.items():
if inner_dict['type'] == 'plugh':
print(key)
Or if you a one liner to get the first key matching the condition:
key = next(key for key, inner_dict in dict_.items() if inner_dict['type'] == 'plugh')
print(key)
output:
baz

Try iterating over the dict keys and check for the element
for key in d:
if(d[key]['type'] == 'plugh'):
print(key)
baz

You need to iterate over your data like this:
def top_level_key(search_key, data):
for key, value in data.items():
if value['type'] == search_key:
return key
print(top_level_key('plugh', data_dict))

Besides running loop to filter the target, you have another option to use jsonpath, which is quite like xPath
# pip install jsonpath-ng==1.5.2
# python 3.6
from jsonpath_ng.ext import parse
dct = {
'foo': {
'name': 'bar',
'options': None,
'type': 'qux'
},
'baz': {
'name': 'grault',
'options': None,
'type': 'plugh'
},
}
parse_str = '$[?#.type="plugh"]'
jsonpath_expr = parse(parse_str)
jsonpath_results = jsonpath_expr.find(dct)
if len(jsonpath_results) > 0:
result = jsonpath_results[0].value
print(result)
# {'name': 'grault', 'options': None, 'type': 'plugh'}
else:
result = None
Ref: https://pypi.org/project/jsonpath-ng/ to find out more stynax about jsonpath

Get index name of a list made from dictionaries

I want to begin by saying that I am by no mean a python expert so I am sorry if I express myself in an incorrect way.
I am building a script that goes something like this:
from netmiko import ConnectHandler
visw0102 = {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
}
visw0103 = {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}
site1_switches = [visw0102, visw0103]
for switch in site1_switches:
... (rest of the script)
I am trying to get the current index name in the FOR loop by using the enumerate() function to get the index name of the site1_switches list but since that list is made of dictionary items, the dictionary keys are returned:
>>> for index, w in enumerate(switch):
... print(w)
...
device_type
ip
username
password
Is there a way the get the actual index name (VISW010X) instead of values that are in the dictionaries?
Thank you
Edit: Nested dictionary was the answer here, thanks Life is complex
So I was able to get further. Here's the code now.
from netmiko import ConnectHandler
site1_switches = {
'visw0102' : {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
},
'visw0103' : {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}
}
for key, values in site1_switches.items():
device_type = values.get('device_type', {})
ip_address = values.get('ip', {})
username = values.get('username', {})
password = values.get('password', {})
for key in site1_switches.items():
net_connect = ConnectHandler(**dict(key)) <- The ConnectHandler needs a dictionary
Now the problem is that the dictionary key seems to be converted to a tuple but the ConnectHandler module needs a dictionary to proceed.
Here's what I get:
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: dictionary update sequence element #0 has length 8; 2 is required
I would need to find a way to convert the tuple to a dictionary but it seems that dict(key) doesn't work as it puts the tuple in the first dictionary key (or so it seems).
Anyway I can achieve that?
Thanks!

Have you considered using a nested dictionary?
site1_switches = {
'visw0102': {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
},
'visw0103': {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}}
for key, value in site1_switches.items():
print (key)
# output
visw0102
visw0103
Here's another way to accomplish this.
for index, (key, value) in enumerate(site1_switches.items()):
print(index, key, value)
# output
0 visw0102 {'device_type': 'hp_comware', 'ip': '192.168.0.241', 'username': 'admin', 'password': 'password'}
1 visw0103 {'device_type': 'hp_comware', 'ip': '192.168.0.242', 'username': 'admin', 'password': 'password'}
A more complete solution
from netmiko import ConnectHandler
# nested dictionary
site1_switches = {
'visw0102': {
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
},
'visw0103': {
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}}
for key, values in site1_switches.items():
device_type = values.get('device_type', {})
ip_address = values.get('ip', {})
username = values.get('username', {})
password = values.get('password', {})
print (f'{key}', {device_type}, {ip_address}, {username}, {password})
# output
visw0102 {'hp_comware'} {'192.168.0.241'} {'admin'} {'password'}
visw0103 {'hp_comware'} {'192.168.0.242'} {'admin'} {'password'}
print (f'Establishing a connection to {key}')
# output
Establishing a connection to visw0102
# pseudo code based on ConnectHandler parameters
switch_connect = ConnectHandler(device_type=device_type, host=ip_address, username=username, password=password)
# checking that the connection has a prompt
switch_connect.find_prompt()
# What you want to do goes here...
# Example
command_output = switch_connect.send_command('display current-configuration')

Unfortunately, there doesn't seem to be a nice, succinct way of accessing the dictionary's name, but Get name of dictionary provides some possible workarounds:
Nesting your switch dictionaries within an overarching dictionary that maps names to dictionaries is one method.
site1_switches = {
"visw0102": visw0102,
"visw0103": visw0103
}
Another would be to add a "name" key to each dictionary, so that you can access the names of each switch in site1_switches by switch['name']
visw0102 = {
'name': 'visw0102',
'device_type': 'hp_comware',
'ip': '192.168.0.241',
'username': 'admin',
'password': 'password'
}
visw0103 = {
'name': 'visw0103',
'device_type': 'hp_comware',
'ip': '192.168.0.242',
'username': 'admin',
'password': 'password'
}

How to generate python dict from list of file paths

I am using this SSM method to fetch a bunch of sensitive credentials from my AWS parameter store. The path syntax allows me to pull a bunch of related credentials all at once, which is cool, but as you can see the return format is more of a list of dicts. I have stripped away things so that I can grab all my parameters in a format like this:
{
'/a/b/c/d/e': 'xxxxxxxx',
'/a/b/c/d/f': 'xxxxxx',
'/a/b/c/g': 'xxxxxx'
}
But I am having trouble passing that format, into some sort of (im assuming recursive) function that re-formats that dict with path keys into an actual nested dict like this:
{
'a': {
'b': {
'c': {
'd': {
'e': 'xxxxxxx',
'f': 'xxxxxxx'
},
'g': 'xxxxxxx'
}
}
}
}
Anyone done something like this before?

Try this:
d = {
'/a/b/c/d/e': 'xxxxxxxx',
'/a/b/c/d/f': 'xxxxxx',
'/a/b/c/g': 'xxxxxx'
}
output = {}
for key, value in d.items():
directories = key.split('/')[1:]
helper = output
for dire in directories[:-1]:
if dire not in helper:
helper[dire] = {}
helper = helper[dire]
helper[directories[-1]] = value
print(output)
The output was:
{
'a': {
'b': {
'c': {
'd': {
'e': 'xxxxxxxx',
'f': 'xxxxxx'
},
'g': 'xxxxxx'
}
}
}
}

You may inherit dictionary like follwoing
class RecursiveDict(dict):
def __missing__(self, key):
self.__setleaf__(key, RecursiveDict())
return self[key]
def __setleaf__(self,key,value):
super(RecursiveDict, self).__setitem__(key,value)
def __setitem__(self, key,value):
keys = key.strip('/').split('/')
tmp = self
for k in keys[:-1]:
tmp = tmp[k]
tmp.__setleaf__(keys[-1],value)
You may use it as follows
d = RecursiveDict()
d['/a/b/c/d/e'] = 'xxxxxxxx'
d['/a/b/c/d/f'] = 'xxxxxx'
print(d)

This approach doesn't work perfectly with what you're looking for, but if your separator is a period instead of a forward slash, it will work perfectly.
Install unflatten from Pypi (https://pypi.org/project/unflatten/)
pip install unflatten
Code:
from unflatten import unflatten
t = {
'a.b.c.d.e': 'xxxxxxxx',
'a.b.c.d.f': 'xxxxxx',
'a.b.c.g': 'xxxxxx'
}
print(unflatten(t))
Input:
{
'a.b.c.d.e': 'xxxxxxxx',
'a.b.c.d.f': 'xxxxxx',
'a.b.c.g': 'xxxxxx'
}
Output:
{'a': {'b': {'c': {'g': 'xxxxxx', 'd': {'f': 'xxxxxx', 'e': 'xxxxxxxx'}}}}}

I wrote library (ssmenv) that I am using for reading values from parameter store:
from ssmenv import SSMEnv
params = SSMEnv("/service/my-service", "/resource/mysql")
This way you will fetch all keys under "/service/my-service" and "/resource/mysql" namespace as normalised keys e.g. "SERVICE_MY_SERVICE_DEBUG"

How to replace nested python dictionary value from a key as a string format with separated by dots?

a = {
'user': {
'username': 'mic_jack',
'name': {
'first': 'Micheal',
'last': 'Jackson'
},
'email': 'micheal#domain.com',
#...
#... Infinite level of another nested dict
}
}
str_key_1 = 'user.username=john'
str_key_2 = 'user.name.last=henry'
#...
#str_key_n = 'user.level2.level3...leveln=XXX'
Let's consider this 'str_key' string, goes with infinite number of dots/levels.
Expected Output:
a = {
'user': {
'username': 'john', # username, should be replace
'name': {
'first': 'Micheal',
'last': 'henry' # lastname, should be replace
},
'email': 'micheal#domain.com',
...
... # Infinite level of another nested dict
}
}
I'm expecting the answers for applying 'n' Level of nested key string, rather than simply replacing by a['user']['username'] = 'John' statically. Answers must be work for any number of 'dotted' string values.
Thanks in advance!

There are three steps:
Separate the key-value pair string into a fully-qualified key and
value.
Split the key into path components.
Traverse the dictionary to find the relevant value to update.
Here's an example of what the code might look like:
# Split by the delimiter, making sure to split once only
# to prevent splitting when the delimiter appears in the value
key, value = str_key_n.split("=", 1)
# Break the dot-joined key into parts that form a path
key_parts = key.split(".")
# The last part is required to update the dictionary
last_part = key_parts.pop()
# Traverse the dictionary using the parts
current = a
while key_parts:
current = current[key_parts.pop(0)]
# Update the value
current[last_part] = value

I'd go with a recursive function to accomplish this, assuming your key value strings are all valid:
def assign_value(sample_dict, str_keys, value):
access_key = str_keys[0]
if len(str_keys) == 1:
sample_dict[access_key] = value
else:
sample_dict[access_key] = assign_value(sample_dict[access_key], str_keys[1:], value)
return sample_dict
The idea is to traverse your dict until you hit the lowest key and then we assign our new value to that last key;
if __name__ == "__main__":
sample_dict = {
'user': {
'username': 'mic_jack',
'name': {
'first': 'Micheal',
'last': 'Jackson'
},
'email': 'micheal#domain.com'
}
}
str_key_1 = 'user.username=john'
str_keys_1, value_1 = str_key_1.split('=')
sample_dict = assign_value(sample_dict, str_keys_1.split('.'), value_1)
print("result: {} ".format(sample_dict))
str_key_2 = 'user.name.last=henry'
str_keys_2, value_2 = str_key_2.split('=')
sample_dict = assign_value(sample_dict, str_keys_2.split('.'), value_2)
print("result: {}".format(sample_dict))
To use the assign_value you would need to split your original key to the keys and value as seen above;

If you're okay with using exec() and modify your str_key(s), you could do something like:
def get_keys_value(string):
keys, value = string.split("=")
return keys, value
def get_exec_string(dict_name, keys):
exec_string = dict_name
for key in keys.split("."):
exec_string = exec_string + "[" + key + "]"
exec_string = exec_string + "=" + "value"
return exec_string
str_key_1 = "'user'.'username'=john"
str_key_2 = "'user'.'name'.'last'=henry"
str_key_list = [str_key_1, str_key_2]
for str_key in str_key_list:
keys, value = get_keys_value(str_key) # split into key-string and value
exec_string = get_exec_string("a", keys) # extract keys from key-string
exec(exec_string)
print(a)
# prints {'user': {'email': 'micheal#domain.com', 'name': {'last': 'henry', 'first': 'Micheal'}, 'username': 'john'}}

str_key_1 = 'user.username=john'
str_key_2 = 'user.name.last=henry'
a = {
'user': {
'username': 'mic_jack',
'name': {
'first': 'Micheal',
'last': 'Jackson'
},
'email': 'micheal#domain.com',
#...
#... Infinite level of another nested dict
}
}
def MutateDict(key):
strkey, strval = key.split('=')[0], key.split('=')[1]
strkeys = strkey.split('.')
print("strkeys = " ,strkeys)
target = a
k = ""
for k in strkeys:
print(target.keys())
if k in target.keys():
prevTarget = target
target = target[k]
else:
print ("Invalid key specified")
return
prevTarget[k] = strval
MutateDict(str_key_1)
print(a)
MutateDict(str_key_2)
print(a)

How to mask a Python 3 nested dictionary to return a new dictionary with only certain items?

I am consuming several endpoints of an API that is very verbose in the data it returns. I would like to provide a subset of this data to another piece of code elsewhere.
Suppose I am given several dictionaries like this (which I plan to loop through and filter):
asset = {
'id': 1,
'name': 'MY-PC',
'owner': 'me',
'location': 'New York City',
'model': {
'id': 1,
'name': 'Surface',
'manufacturer': {
'id': 1,
'name': 'Microsoft'
}
}
}
I want to create a function that will take that dictionary in, along with a "mask" which will be used to create a new dictionary of only the allowed items. This might be an example mask (though, I can work with whatever format makes the resulting code the most concise):
mask = {
'id': True,
'name': True,
'model': {
'id': True,
'name': True,
'manufacturer': {
'name': True
}
}
}
The function should then return this:
mask = {
'id': 1,
'name': 'MY-PC',
'model': {
'id': 1,
'name': 'Surface',
'manufacturer': {
'name': 'Microsoft'
}
}
}
Is there something already built into Python 3 that would help aid in this? It looks like if I have to do this manually, it's going to get quite ugly quickly. I found itertools.compress, but that seems like it's for lists and won't handle the complexity of dictionaries.

You can recursively build a new dict from the mask by selecting only values corresponding in the main dict:
def prune_dict(dct, mask):
result = {}
for k, v in mask.items():
if isinstance(v, dict):
value = prune_dict(dct[k], v)
if value: # check that dict is non-empty
result[k] = value
elif v:
result[k] = dct[k]
return result
print(prune_dict(asset, mask))
{'id': 1,
'model': {'id': 1, 'manufacturer': {'name': 'Microsoft'}, 'name': 'Surface'},
'name': 'MY-PC'}

This would be a good chance to use recursion, here is some sample code I haven't tested:
def copy(asset, result, mask):
for key_name, value in mask.items():
if value == True:
result[key_name] = asset[key_name]
else:
result[key_name] = x = {}
copy(asset[key_name], x, value)
y = {}
copy(asset, y, mask)

This would probably be a recursive function. Also, for the mask, I recommend this format: mask = ["id", "name", "model.id", "model.name", "model.manufacturer.name"]
Then, you'd first keep only entries that are named in the mask:
def filterstage1(dictionary, mask):
result = {}
for key in dictionary:
if isinstance(dictionary[key], dict):
newmask = [maskname[mask.find(".") + 1:] for maskname in mask if maskname.startswith(key + ".")]
result[k] = filterstage1(dictionary[key], newmask)
elif key in mask:
result[key] = dictionary[key]
return result
Then, depending on whether or not you want to remove sub-dictionaries that were not in the mask and had no subelements, you can include the second stage:
def filterstage2(dictionary, mask):
result = {}
for key in dictionary:
if not (isinstance(dictionary[key], dict) and dictionary[key] == {} and key not in mask):
result[key] = dictionary[key]
Final code: filterstage2(filterstage1(dictionary, mask), mask). You can combine the two stages together if you wish.