We Keep Coding

Get all the keys of a nested dict

With xmltodict I managed to get my code from xml in a dict and now I want to create an excel.
In this excel the header of a value is going to be all the parents (keys in the dict).
For example:
dict = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
The value male will have the header: adres/roommate/gender.

Here's a way to orgainze the data in the way your question asks:
d = {"name":"Pete", "last-name": "Pencil", "adres":{"street": "example1street", "number":"5", "roommate":{"gender":"male"}}}
print(d)
stack = [('', d)]
headerByValue = {}
while stack:
name, top = stack.pop()
if isinstance(top, dict):
stack += (((name + '/' if name else '') + k, v) for k, v in top.items())
else:
headerByValue[name] = top
print(headerByValue)
Output:
{'adres/roommate/gender': 'male',
'adres/number': '5',
'adres/street': 'example1street',
'last-name': 'Pencil',
'name': 'Pete'}

How to map nested list to flat values

I`m trying to parse a spreadsheet with a header that looks something like this:
My problem is those nested keys below "Контрагент". I decided to parse it like this:
['Дата',
'Номер документа',
'Дебет',
'Кредит',
['Контрагент',
['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа',
'Код дебитора',
'Тип документа']
But now, I don`t really have an idea as how to map it to a flat list of values:
['21.05.2021',
'591324565436',
'0.00',
'526345428.99',
'asdasd',
'234525460140679',
'77130100123412341',
'302328105423534200000000280',
'0445252345234974',
'asdfsadfsd',
'sdfghsfgdhfdghdfgh',
'',
'dfghfgdhfdgh']
Given these variables, I want a function to return following dict:
{
"Дата": "21.05.2021",
"Номер документа": "591324565436",
"Дебет": "0.00",
"Кредит": "526345428.99",
"Контрагент": {
"Наименование": "asdasd",
"ИНН": "234525460140679",
"КПП": "77130100123412341",
"Счет": "302328105423534200000000280",
"БИК": "0445252345234974",
"Наименование банка": "asdfsadfsd"
},
"Назначение платежа": "sdfghsfgdhfdghdfgh",
"Код дебитора": "",
"Тип документа": "dfghfgdhfdgh"
}
I've gone this far without realizing it'd be raising IndexError on the 3rd line:
def map_to_schema(schema, data):
for i, elem in enumerate(data):
key = schema[i]
if isinstance(key, list):
if key[0] not in result:
result[key[0]] = {}
result[key[0]] |= {
key[1][i-len(key)]: elem
}
else:
result[key] = elem
What should I do? Maybe the structure for the schema isn't good enough? I really have no idea...

You could use a dictionary comprehension and an iterator:
headers = ['Дата', 'Номер документа', 'Дебет', 'Кредит',
['Контрагент', ['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа', 'Код дебитора', 'Тип документа']
values = ['21.05.2021', '591324565436', '0.00', '526345428.99', 'asdasd', '234525460140679', '77130100123412341',
'302328105423534200000000280', '0445252345234974', 'asdfsadfsd', 'sdfghsfgdhfdghdfgh', '',
'dfghfgdhfdgh']
it = iter(values)
out = {k[0] if (islist := isinstance(k, list)) else k:
{k2: next(it) for k2 in k[1]} if islist else next(it)
for k in headers}
output:
{'Дата': '21.05.2021',
'Номер документа': '591324565436',
'Дебет': '0.00',
'Кредит': '526345428.99',
'Контрагент': {'Наименование': 'asdasd',
'ИНН': '234525460140679',
'КПП': '77130100123412341',
'Счет': '302328105423534200000000280',
'БИК': '0445252345234974',
'Наименование банка': 'asdfsadfsd'},
'Назначение платежа': 'sdfghsfgdhfdghdfgh',
'Код дебитора': '',
'Тип документа': 'dfghfgdhfdgh'}

Thanks #mozway for this solution! This is essentially the same algorithm, using a for loop.
def map(schema, s_length, row: list):
# If len(row) was less then *true* schema length, it would have thrown StopIteration.
# I ended up just extending row list by delta elements.
if (delta := s_length - len(row)) > 0:
row.extend([""] * delta)
iter_row = iter(row)
result = {}
for key in schema:
if isinstance(key, list):
result[key[0]] = {}
for sub_key in key[1]:
result[key[0]][sub_key] = next(iter_row)
else:
result[key] = next(iter_row)
return result

How to find a specific python dictionary key with a string

How can I match the key name from general_key_array with all_keys dictionary to get "aws." as substring? I added the startswith section but it returns True all the time.
general_keys = dict()
all_keys = {'activity': 'ins','install': 'all','aws.a': 'data', 'aws.b': 'data1', 'aws.c': 'data2'} #read from file
general_key_array = ['install', 'aws.']
for key in general_key_array:
if key.startswith(key) in all_keys:
general_keys[key] = dict(filter(lambda item: key in item[0], all_keys.items()))

You can do it all with a single dictionary comprehension.
general_keys = {key: value for key, value in all_keys.items()
if any(key.startswith(gk) for gk in general_key_array)}

key.startswith(key) is always equal to True because it's tautological that a string starts with itself.
you can add a for loop with the keys of all_keys after the first one you wrote to solve the problem like this
general_keys = dict()
all_keys = {'activity': 'ins','install': 'all','aws.a': 'data', 'aws.b': 'data1', 'aws.c': 'data2'}
general_key_array = ['install', 'aws.']
for key in general_key_array:
for s in all_keys :
if s.startswith(key): # s is part of all_keys, no tautology here
general_keys[key] = dict(filter(lambda item: key in item[0], all_keys.items()))

How to insert data into an array of dictionaries in an order without missing data via regex

This is my code:
I'm trying to use the following code to insert data into an array of dictionaries but unable to insert properly.
Code:
test_list = {'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5', 'module_name-2': 'CH1.FM6', 'module_serial-2': 'PSUXA12345681'}
def parse_subdevice_modules(row):
modules = []
module = {}
for k, v in row.items():
if v:
if re.match("module_name", k):
module['name'] = v
if re.match("module_serial", k):
module['serial'] = v
modules.append(module)
module = {}
return modules
print(parse_subdevice_modules(test_list))
Expected output:
[{'name':'CH1.FM5', serial': 'PSUXA12345680'}, {'name': 'CH1.FM6', 'serial': 'PSUXA12345681'}]
Actual output:
['serial': 'PSUXA12345680'}, {'name': 'CH1.FM6', 'serial': 'PSUXA12345681'}]
Run it here: https://repl.it/repls/WetSteelblueRange
Please note that the order of the data test_list cannot be altered as it comes via an external API so I used regex. Any ideas would be appreciated.

Your code relies on the wrong assumption that keys are ordered and that the serial will always follow the name. The proper solution here is to use a dict (actually a collections.defaultdict to make things easier) to collect and regroup the values you're interested in based on the module number (the final '-N' in the key). Note that you don't need regexps here - Python string already provide the necessary operations for this task:
from collections import defaultdict
def parse_subdevice_modules(row):
modules = defaultdict(dict)
for k, v in row.items():
# first get rid of what we're not interested in
if not v:
continue
if not k.startswith("module_"):
continue
# retrieve the key number (last char) with
# negative string indexing:
key_num = k[-1]
# retrieve the useful part of the key ("name" or "serial")
# by splitting the string:
key_name = k.split("_")[1].split("-")[0]
# and now we just have to store this in our defaultdict
modules[key_num][key_name] = v
# and return only the values.
# NB: in py2.x you don't need the call to `list`,
# you can just return `modules.values()` directly
modules = list(modules.values())
return modules
test_list = {
'profile': '', 'chassis_name': '123', 'supplier_order_num': '',
'device_type': 'mass_storage', 'device_subtype': 'flashblade',
'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5',
'module_name-2': 'CH1.FM6', 'rack_total_pos': '',
'asset_tag': '002000027493', 'module_serial-2': 'PSUXA12345681',
'purchase_order': '0004530869', 'build': 'Test_Build_for_SNOW',
'po_line_num': '00190', 'mac_address': '', 'position': '7',
'model': 'FB-528TB-10X52.8TB', 'manufacturer': 'PureStorage',
'rack': 'Test_Rack_2', 'serial': 'PMPAM1842147D', 'name': 'FB02'
}
print(parse_subdevice_modules(test_list))

You can do somthing like this also.
test_list = {'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5', 'module_name-2': 'CH1.FM6',
'module_serial-2': 'PSUXA12345681'}
def parse_subdevice_modules(row):
modules_list = []
for key, value in row.items():
if not value or key.startswith('module_name'):
continue
if key.startswith('module_serial'):
module_name_key = f'module_name-{key.split("-")[-1]}'
modules_list.append({'serial': value, 'name': row[module_name_key]})
return modules_list
print(parse_subdevice_modules(test_list))
Output:
[{'serial': 'PSUXA12345680', 'name': 'CH1.FM5'}, {'serial': 'PSUXA12345681', 'name': 'CH1.FM6'}]

You would need to check if module contains 2 elements and append it to modules:
test_list = {'module_serial-1': 'PSUXA12345680', 'module_name-1': 'CH1.FM5', 'module_name-2': 'CH1.FM6', 'module_serial-2': 'PSUXA12345681'}
def parse_subdevice_modules(row):
modules = []
module = {}
for k, v in row.items():
if v:
if k.startswith('module_name'):
module['name'] = v
elif k.startswith("module_serial"):
module['serial'] = v
if len(module) == 2:
modules.append(module)
module = {}
return modules
print(parse_subdevice_modules(test_list))
Returns:
[{'serial': 'PSUXA12345680'}, {'name': 'CH1.FM5'}, {'name': 'CH1.FM6'}, {'serial': 'PSUXA12345681'}]

Merge duplicate entries in array of dict

I'm struggling with a recursive merge problem.
Let's say I have:
a=[{'name':"bob",
'age':10,
'email':"bob#bla",
'profile':{'id':1, 'role':"admin"}},
{'name':"bob",
'age':10,
'email':"other mail",
'profile':{'id':2, 'role':"dba"},
'home':"/home/bob"
}]
and I need something to recursively merge entries. If value for an existing given key on the same level is different it appends the value to an array.
b = merge(a)
print b
{'name':"bob",
'age':10,
'email':["bob#bla","other mail"],
'profile':{'id':[1,2], 'role'=["admin", "dba"], 'home':"/home/bob"}
I wrote this code:
def merge(items):
merged = {}
for item in items:
for key in item.keys():
if key in merged.keys():
if item[key] != merged[key]:
if not isinstance(merged[key], list):
merged[key] = [merged[key]]
if item[key] not in merged[key]:
merged[key].append(item[key])
else:
merged[key] = item[key]
return merged
The output is:
{'age': 10,
'email': ['bob#bla', 'other mail'],
'home': '/home/bob',
'name': 'bob',
'profile': [{'id': 1, 'role': 'admin'}, {'id': 2, 'role': 'dba'}]}
Which is not what I want.
I can't figure out how to deal with recursion.
Thanks :)

As you iterate over each dictionary in the arguments, then each key and value in each dictionary, you want the following rules:
If there is nothing against that key in the output, add the new key and value to the output;
If there is a value for that key, and it's the same as the new value, do nothing;
If there is a value for that key, and it's a list, append the new value to the list;
If there is a value for that key, and it's a dictionary, recursively merge the new value with the existing dictionary;
If there is a value for that key, and it's neither a list nor a dictionary, make the value in the output a list of the current value and the new value.
In code:
def merge(*dicts):
"""Recursively merge the argument dictionaries."""
out = {}
for dct in dicts:
for key, val in dct.items():
try:
out[key].append(val) # 3.
except AttributeError:
if out[key] == val:
pass # 2.
elif isinstance(out[key], dict):
out[key] = merge(out[key], val) # 4.
else:
out[key] = [out[key], val] # 5.
except KeyError:
out[key] = val # 1.
return out
In use:
>>> import pprint
>>> pprint.pprint(merge(*a))
{'age': 10,
'email': ['bob#bla', 'other mail'],
'home': '/home/bob',
'name': 'bob',
'profile': {'id': [1, 2], 'role': ['admin', 'dba']}}