Related
Consider this input dictionary:
my_dict = {
'group1':{
'type1': {'val1' : 45, 'val2' : 12, 'val3' : 65},
'type2': {'val5' : 65, 'val6' : 132, 'val7' : 656},
},
'group2':{
'type3': {'val11' : 45, 'val12' : 123, 'val13' : 3},
'type4': {'val51' : 1, 'val61' : 2, 'val71' : 3, },
},
}
I would like to remove the last 'level' (the one that only has numbers), and get something like:
new_dict = {
'group1':{
'type1': ['val1', 'val2', 'val3'],
'type2': ['val5', 'val6', 'val7'],
},
'group2':{
'type3': ['val11', 'val12', 'val13'],
'type4': ['val51', 'val61', 'val71'],
},
}
I am currently doing it by manually looping and so on, but I wonder if there is a way to do it more efficiently.
If your dictionary doesn't have a fixed number of nesting levels, you could write a recursive function to do it:
def stripBottom(d):
if not isinstance(d,dict):
return d
elif not any(isinstance(v,dict) for v in d.values()):
return list(d)
else:
return {k:stripBottom(v) for k,v in d.items()}
Output:
my_dict = {
'group1':{
'type1': {'val1' : 45, 'val2' : 12, 'val3' : 65},
'type2': {'val5' : 65, 'val6' : 132, 'val7' : 656},
},
'group2':{
'type3': {'val11' : 45, 'val12' : 123, 'val13' : 3},
'type4': {'val51' : 1, 'val61' : 2, 'val71' : 3, },
},
}
print(stripBottom(my_dict))
{'group1': {'type1': ['val1', 'val2', 'val3'],
'type2': ['val5', 'val6', 'val7']},
'group2': {'type3': ['val11', 'val12', 'val13'],
'type4': ['val51', 'val61', 'val71']}}
If your dictionary has variable levels of nesting or if you want to drop a specific level, you would need to add a parameter to the function:
def stripBottom(d,level=0):
if not isinstance(d,dict):
return d
elif level == 0:
return list(d)
else:
return {k:stripBottom(v,level-1) for k,v in d.items()}
print(stripBottom(my_dict,1))
{'group1': ['type1', 'type2'], 'group2': ['type3', 'type4']}
You don't give an example of the actual code you are using to change the dict, but likely you will always need 2 levels of looping to iterate over groups, and then types. Something like:
newdict = {}
for k, grp in my_dict.items():
newdict[k] = {}
for typ, val in grp.items():
newdict[k][typ] = list(val)
One option to be more efficient is to not pre-compute these updted values, but instead when you come to use the 'group' data, wrap the output in list at that point. (Though obviously you will still be iterating when the dict is being used, to get these nested dicts).
I have a nested dictionary that looks like below,
{
'product_list.show_date': "May '21",
'product_list.new_users':
{
'product_list.product':
{
'A': None,
'B': 377,
'C': None,
'D': 67,
'E': None,
'F': 1,
'G': None
}
}
}
And I want to clear it out in a way that parent keys are not there. So basically, I want a dictionary that is not nested. Like below,
{
'product_list.show_date': "May '21",
'A': None,
'B': 377,
'C': None,
'D': 67,
'E': None,
'F': 1,
'G': None
}
I am using the recursive function to do this, but it's not 100% correct.
Here's my code,
def clear_nd(d, nested_dict):
for key in nested_dict:
if type(nested_dict[key]) != dict:
d[key] = nested_dict[key]
elif type(nested_dict[key]) == dict:
nested_dict = nested_dict[key]
clear_nd(d, nested_dict)
return d
d = {}
clear_nd(d, nested_dict)
For below example,
nested_dict = {
'product_list.show_date': "May '21",
'product_list.new_users': {
'product_list.product': {
'A': None,
'B': 377,
'C': None,
'D': 67,
'E': None,
'F': 1,
'G': None
},
'prod.product': {
'Alk': None,
'Bay': 377,
'Lent': None,
'R': 67,
'Ter': None,
'Wi': 1,
'e': None
}
},
'duct_list.new_users': {
'pdust.product': {
'H': None,
'y': 377,
'nt': None,
'C': 67,
'sfer': None,
's': 1,
'le': None
}
}
}
Does Pandas or any other library has a way to do this. Structure of the nested dictionary is dynamic so we won't know how deep it is. And Keys will also change, so we won't able to know beforehand what are the keys in the dictionary. Any help will be appreciated. Thanks!!
If you allow the lower level tag labels to take prefixes of higher level tag labels, you can use the Pandas function pandas.json_normalize, which handles nested dict and turn it into a flat table Pandas dataframe.
Then, use pandas.DataFrame.to_dict to turn the Pandas dataframe to a dict. For example,
import pandas as pd
d = {
'product_list.show_date': "May '21",
'product_list.new_users':
{
'product_list.product':
{
'A': None,
'B': 377,
'C': None,
'D': 67,
'E': None,
'F': 1,
'G': None
}
}
}
pd.json_normalize(d).to_dict('records')[0]
Result:
{'product_list.show_date': "May '21",
'product_list.new_users.product_list.product.A': None,
'product_list.new_users.product_list.product.B': 377,
'product_list.new_users.product_list.product.C': None,
'product_list.new_users.product_list.product.D': 67,
'product_list.new_users.product_list.product.E': None,
'product_list.new_users.product_list.product.F': 1,
'product_list.new_users.product_list.product.G': None}
from the given input
lists = ["7ee57f24", "deadbeef"]
I want to get the following output
l1': [
{
'd':
{
'id': '7ee57f24'
}
},
{
'd':
{
'id': 'deadbeed'
}
}
]
I have tried this code
lists = ["7ee57f24", "deadbeef"]
l1 = {"d":[{"id": lis} for lis in lists]}
print(l1)
but it gives me wrong output
{'d': [{'id': '7ee57f24'}, {'id': 'deadbeef'}]}
Use the following:
lists = ["7ee57f24", "deadbeef"]
l1 = [
{"d": {"id": id_}}
for id_ in lists
]
print(l1)
Output:
[{'d': {'id': '7ee57f24'}}, {'d': {'id': 'deadbeef'}}]
I have nested dictionary with this kind of structure:
d = {
"A":{
"Param1":"7",
"Param2":"5",
},
"B":{
"Param1":"1",
"Param2":"2",
},
"C":{
"Param1":"X",
"Param2":"Y",
},
"D":{
"SomeOtherParam1": "a",
"SomeOtherParam2": "3",
}
}
How to get dictionary key by nested key names and parameters? For example: Param1=1 and Param2=2 the output should be B. Is it even possible to do that?
UPDATE
Thanks to #deceze here is exactly what I wanted to achieve:
pexist = next((k for k, v in d.items() if v.get('Param1') and v.get('Param2') if v['Param1'] == '1' and v['Param2'] == '2'), None)
if pexist == None:
print("Does not exist!")
else:
print(pexist)
A pandas module based solution:
import pandas as pd
df = pd.DataFrame({'A': {'Param1': '7', 'Param2': '5'}, 'B': {'Param1': '1', 'Param2': '2'}, 'C': {'Param1': 'X', 'Param2': 'Y'}})
s = (df.loc['Param1'] == '1') & (df.loc['Param2'] == '2')
print(*s[s].keys())
Output:
B
I have a simple data-structure, roughly the following, with arbitrary depth:
# Level 0
{ 'foo': 'bar', 'more':
[ # Level 1
{ 'foo': 'can', 'more': [ # Level 2
{ 'foo': 'haz', 'more': [] }
]
},
{ 'foo': 'baz', 'more': [ # Level 2
{ 'foo': None, 'more': [] }
]
}
]
}
Working on a function to insert at a certain level. If foo value is None, then insert value, else insert a new 'sibling', i.e., a new dict at the same level. Apart from the 0th level, everything is within a list, so should be appendable.
Attempt:
def traverse_to_level(obj, level):
if obj['_level'] == level:
return obj
for _obj in obj['block']:
found = traverse_to_level(_obj, level)
if found is not None:
return found
return None
def set_obj(top_obj, value, level):
obj = traverse_to_level(top_obj, level)
if obj is None:
directive = traverse_to_level(top_obj, level - 1)
assert obj is not None
if obj['foo'] is None:
obj['foo'] = value
elif obj['_level'] < level:
obj['more'].append({'foo': value, 'more': [], '_level': level})
else:
obj = traverse_to_level(top_obj, level - 1)
obj['more'].append({'foo': value, 'more': [], '_level': level})
return obj
What's the right way to traverse a structure like this, and update it in-place by level?
EDIT: Another example, showing more than just 'foo' on each object. To simplify things, if the input startswith # then it's the input should be added to alpha, otherwise beta. If either are already filled, then insert adjacent (i.e., a new sibling {'alpha': None, 'beta': None, 'more': []}).
def make_ab(alpha=None, beta=None, more=None):
return {
'alpha': alpha,
'beta': beta,
'more': more or []
}
def parse(arguments):
level, last_level_change, top_d = 0, -1, make_ab()
for idx, arg in enumerate(arguments):
if arg == '#{':
level += 1
elif arg in frozenset(('#d', '##dog')):
level += 1
last_level_change = idx
elif arg == '#}':
level -= 1
elif idx == last_level_change - 1:
set_obj(top_d, arg, level, 'alpha')
elif idx == last_level_change - 2:
set_obj(top_d, arg, level, 'beta')
else:
set_obj(top_d, arg, level,
'beta' if arg.startswith('##')
else 'alpha')
return top_d
Usage, hinting as to the expected hierarchy through indentation:
actual = parse(
('#d',
'definite',
'##foo_alpha_5', 'nice_beta',
'##name_2', 'amazing_beta',
'##dog',
'##fancy_name', 'fancy_BETA_f',
'##one_AL_PHA_of', 'I_C_U_B_ETA',
'#}',
'#}')
)
And this is the output expected:
{
'alpha': 'definite',
'beta': None,
'more': [
{
'alpha': '##foo_alpha_5',
'beta': 'nice_beta',
'more': []
},
{
'alpha': '##name_2',
'beta': 'amazing_beta',
'more': [
{
'alpha': '##fancy_name',
'beta': 'fancy_BETA_f',
'more': []
},
{
'alpha': '##one_AL_PHA_of',
'beta': 'I_C_U_B_ETA',
'more': []
}
]
}
]
}
How about something like:
def set_obj(top_obj, value, level):
if level == 1:
more = top_obj['more']
hasnt_none = True
for n in more:
if n['foo'] is None:
n['foo'] = value
hasnt_none = False
if hasnt_none:
more.append({'foo': value, 'more': []})
else:
for n in top_obj['more']:
set_obj(n, value, level-1)
? Test:
from pprint import pprint
set_obj(data, 'VALUE', 2)
pprint(data)
Output:
{'foo': 'bar',
'more': [{'foo': 'can',
'more': [{'foo': 'haz', 'more': []}, {'foo': 'VALUE', 'more': []}]},
{'foo': 'baz', 'more': [{'foo': 'VALUE', 'more': []}]}]}
If the more of the level contains one or more {'foo': None, ...} elements, then update the value of these elements. Else, just append {'foo': value, 'more': []} element.
Since you're dealing with nested lists you also need to specify the index at each level in order to form some kind of path. As you indicate in your example there are two different lists at level 2, so at level 1 you need to choose between the first or second dict. For that you can use a path in form of a list or tuple: for example (1, 0) means at level 1 select the second dict and at level 2 select the first dict. This can be written as a function:
from functools import reduce
def get_nested_dict(obj, path):
return reduce(lambda o, n: o['more'][n], path, obj)
Then testing it on the example data:
print(get_nested_dict(test_obj, (0, 0))) # foo: haz
print(get_nested_dict(test_obj, (1, 0))) # foo: None
Now for the insertion we need to get the wrapping list so we can insert a sibling dict in case the 'foo' key is already occupied. For that we can slightly modify the above function:
def get_nested_list(obj, path): # assert len(path) > 0
return reduce(lambda o, n: o[n]['more'], path, obj['more'])
Then we can build the insertion function on top of that:
def insert_into(obj, path, value):
lst = get_nested_list(obj, path[:-1])
try:
dct = lst[path[-1]]
except IndexError:
lst.append({'foo': value, 'more': []}) # create new dict
else:
if dct['foo'] is None:
dct['foo'] = value
else:
lst.append({'foo': value, 'more': []}) # create new dict
We can test it on the example data:
pprint(test_obj, sort_dicts=False)
insert_into(test_obj, (0, 0), 'new')
pprint(test_obj, sort_dicts=False)
insert_into(test_obj, (1, 0), 'new')
pprint(test_obj, sort_dicts=False)
Which prints the following:
{'foo': 'bar',
'more': [{'foo': 'can', 'more': [{'foo': 'haz', 'more': []}]},
{'foo': 'baz', 'more': [{'foo': None, 'more': []}]}]}
{'foo': 'bar',
'more': [{'foo': 'can',
'more': [{'foo': 'haz', 'more': []}, {'foo': 'new', 'more': []}]},
{'foo': 'baz', 'more': [{'foo': None, 'more': []}]}]}
{'foo': 'bar',
'more': [{'foo': 'can',
'more': [{'foo': 'haz', 'more': []}, {'foo': 'new', 'more': []}]},
{'foo': 'baz', 'more': [{'foo': 'new', 'more': []}]}]}