We Keep Coding

How to map nested list to flat values

I`m trying to parse a spreadsheet with a header that looks something like this:
My problem is those nested keys below "Контрагент". I decided to parse it like this:
['Дата',
'Номер документа',
'Дебет',
'Кредит',
['Контрагент',
['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа',
'Код дебитора',
'Тип документа']
But now, I don`t really have an idea as how to map it to a flat list of values:
['21.05.2021',
'591324565436',
'0.00',
'526345428.99',
'asdasd',
'234525460140679',
'77130100123412341',
'302328105423534200000000280',
'0445252345234974',
'asdfsadfsd',
'sdfghsfgdhfdghdfgh',
'',
'dfghfgdhfdgh']
Given these variables, I want a function to return following dict:
{
"Дата": "21.05.2021",
"Номер документа": "591324565436",
"Дебет": "0.00",
"Кредит": "526345428.99",
"Контрагент": {
"Наименование": "asdasd",
"ИНН": "234525460140679",
"КПП": "77130100123412341",
"Счет": "302328105423534200000000280",
"БИК": "0445252345234974",
"Наименование банка": "asdfsadfsd"
},
"Назначение платежа": "sdfghsfgdhfdghdfgh",
"Код дебитора": "",
"Тип документа": "dfghfgdhfdgh"
}
I've gone this far without realizing it'd be raising IndexError on the 3rd line:
def map_to_schema(schema, data):
for i, elem in enumerate(data):
key = schema[i]
if isinstance(key, list):
if key[0] not in result:
result[key[0]] = {}
result[key[0]] |= {
key[1][i-len(key)]: elem
}
else:
result[key] = elem
What should I do? Maybe the structure for the schema isn't good enough? I really have no idea...

You could use a dictionary comprehension and an iterator:
headers = ['Дата', 'Номер документа', 'Дебет', 'Кредит',
['Контрагент', ['Наименование', 'ИНН', 'КПП', 'Счет', 'БИК', 'Наименование банка']],
'Назначение платежа', 'Код дебитора', 'Тип документа']
values = ['21.05.2021', '591324565436', '0.00', '526345428.99', 'asdasd', '234525460140679', '77130100123412341',
'302328105423534200000000280', '0445252345234974', 'asdfsadfsd', 'sdfghsfgdhfdghdfgh', '',
'dfghfgdhfdgh']
it = iter(values)
out = {k[0] if (islist := isinstance(k, list)) else k:
{k2: next(it) for k2 in k[1]} if islist else next(it)
for k in headers}
output:
{'Дата': '21.05.2021',
'Номер документа': '591324565436',
'Дебет': '0.00',
'Кредит': '526345428.99',
'Контрагент': {'Наименование': 'asdasd',
'ИНН': '234525460140679',
'КПП': '77130100123412341',
'Счет': '302328105423534200000000280',
'БИК': '0445252345234974',
'Наименование банка': 'asdfsadfsd'},
'Назначение платежа': 'sdfghsfgdhfdghdfgh',
'Код дебитора': '',
'Тип документа': 'dfghfgdhfdgh'}

Thanks #mozway for this solution! This is essentially the same algorithm, using a for loop.
def map(schema, s_length, row: list):
# If len(row) was less then *true* schema length, it would have thrown StopIteration.
# I ended up just extending row list by delta elements.
if (delta := s_length - len(row)) > 0:
row.extend([""] * delta)
iter_row = iter(row)
result = {}
for key in schema:
if isinstance(key, list):
result[key[0]] = {}
for sub_key in key[1]:
result[key[0]][sub_key] = next(iter_row)
else:
result[key] = next(iter_row)
return result

How to find a specific python dictionary key with a string

How can I match the key name from general_key_array with all_keys dictionary to get "aws." as substring? I added the startswith section but it returns True all the time.
general_keys = dict()
all_keys = {'activity': 'ins','install': 'all','aws.a': 'data', 'aws.b': 'data1', 'aws.c': 'data2'} #read from file
general_key_array = ['install', 'aws.']
for key in general_key_array:
if key.startswith(key) in all_keys:
general_keys[key] = dict(filter(lambda item: key in item[0], all_keys.items()))

You can do it all with a single dictionary comprehension.
general_keys = {key: value for key, value in all_keys.items()
if any(key.startswith(gk) for gk in general_key_array)}

key.startswith(key) is always equal to True because it's tautological that a string starts with itself.
you can add a for loop with the keys of all_keys after the first one you wrote to solve the problem like this
general_keys = dict()
all_keys = {'activity': 'ins','install': 'all','aws.a': 'data', 'aws.b': 'data1', 'aws.c': 'data2'}
general_key_array = ['install', 'aws.']
for key in general_key_array:
for s in all_keys :
if s.startswith(key): # s is part of all_keys, no tautology here
general_keys[key] = dict(filter(lambda item: key in item[0], all_keys.items()))

Create new dictionary with specific keys from old dictionary

I want to make a new dictionary that prints a new object containing uuid, name, website, and email address for all rows of my dict that have values for all four of these attributes.
I thought I did this for email, name, and website below in my code but I noticed sometimes name or email wont print (because they have missing values), how do I drop those? Also, uuid is outside of the nested dictionary, how do I add that in the new dictionary too?
I attached my code and an element from my code below.
new2 = {}
for i in range (0, len(json_file)):
try:
check = json_file[i]['payload']
new = {k: v for k, v in check.items() if v is not None}
new2 = {k: new[k] for k in new.keys() & {'name', 'website', 'email'}}
print(new2)
except:
continue
Dictionary sample:
{
"payload":{
"existence_full":1,
"geo_virtual":"[\"56.9459720|-2.1971226|20|within_50m|4\"]",
"latitude":"56.945972",
"locality":"Stonehaven",
"_records_touched":"{\"crawl\":8,\"lssi\":0,\"polygon_centroid\":0,\"geocoder\":0,\"user_submission\":0,\"tdc\":0,\"gov\":0}",
"address":"The Lodge, Dunottar",
"email":"dunnottarcastle#btconnect.com",
"existence_ml":0.5694238217658721,
"domain_aggregate":"",
"name":"Dunnottar Castle",
"search_tags":[
"Dunnottar Castle Aberdeenshire",
"Dunotter Castle"
],
"admin_region":"Scotland",
"existence":1,
"category_labels":[
[
"Landmarks",
"Buildings and Structures"
]
],
"post_town":"Stonehaven",
"region":"Kincardineshire",
"review_count":"719",
"geocode_level":"within_50m",
"tel":"01569 762173",
"placerank":65,
"longitude":"-2.197123",
"placerank_ml":37.27916073464469,
"fax":"01330 860325",
"category_ids_text_search":"",
"website":"http://www.dunnottarcastle.co.uk",
"status":"1",
"geocode_confidence":"20",
"postcode":"AB39 2TL",
"category_ids":[
108
],
"country":"gb",
"_geocode_quality":"4"
},
"uuid":"3867aaf3-12ab-434f-b12b-5d627b3359c3"
}

Try using the dict.get() method:
def new_dict(input_dict, keys, fallback='payload'):
ret = dict()
for key in keys:
val = input_dict.get(key) or input_dict[fallback].get(key)
if val:
ret.update({key:val})
if len(ret) == 4: # or you could do: if set(ret.keys()) == set(keys):
print(ret)
for dicto in json_file:
new_dict(dicto, ['name','website','email','uuid'])
{'name': 'Dunnottar Castle', 'website': 'http://www.dunnottarcastle.co.uk', 'email': 'dunnottarcastle#btconnect.com', 'uuid': '3867aaf3-12ab-434f-b12b-5d627b3359c3'}

Dynamically changing key value in dictionary

I am checking the key in dictionary, if it contains space remove it.
def query_combination(sentence,mydict):
for key in mydict.keys():
if key == 'key':
pass
else:
print 'key is : ',key
if " " in key:
temp = key
key = key.replace(' ',"")
print 'new key : ',key
sentence = sentence.replace(temp ,key)
print 'new sentence : ',sentence
print mydict
mydict = {'films': {'match': ['Space', 'Movie', 'six', 'two', 'one']}, u'Popeye Doyle': {'score': 100, 'match': [u'People', 'heaven', 'released']}}
sentence ='What films featured the character Popeye Doyle'
combination = query_combination(sentence,mydict)
I could not dynamically change the new key value to mydict. Any suggestion much appreciable

If you get a string out of the dictionary, and then change it and make a new string, the dictionary won't know about it; you can add a new entry to the dictionary and remove the old one:
if " " in key:
newkey = key.replace(' ',"")
mydict[newkey] = mydict[key]
del mydict[key]
print 'new key : ', newkey

You could try this
def query_combination(sentence,mydict):
for key in mydict.iterkeys():
if " " in key:
temp = key
mydict[key.replace(" ","")] = mydict[key] # create new key
del mydict[key] # delete old key
sentence = sentence.replace(temp ,key)
Another solution in one line would be
mydict[key.replace(" ","")] = mydict.pop(key)

key = key.replace(' ',"") does not affect the actual key in the dictionary, it is changing a copy of that key. You need to add the value to the dictionary with the new key and remove the old key. Here's one way to do it:
def query_combination(sentence, mydict):
for old_key, new_key in [(key, key.replace(' ', '')) for key in mydict if ' ' in key]:
mydict[new_key] = mydict.pop(old_key)
sentence = sentence.replace(old_key, new_key)
Note, however, that you are replacing the key in the string sentence, but sentence is local to function query_combination(), so the outer scope sentence is unaffected by the replacement. I am not sure if that was what you hoped your code would do, but if it was you could simply return the revised sentence from the function, or include it as an item in the dictionary.
Given that sentence is not actually updated by your function, you can simplify the whole function to a mere dictionary comprehension:
>>> mydict = {'films': {'match': ['Space', 'Movie', 'six', 'two', 'one']}, u'Popeye Doyle': {'score': 100, 'match': [u'People', 'heaven', 'released']}}
>>> mydict = {key.replace(' ', '') : value for key, value in mydict.items()}
>>> mydict
{'films': {'match': ['Space', 'Movie', 'six', 'two', 'one']}, u'PopeyeDoyle': {'score': 100, 'match': [u'People', 'heaven', 'released']}}

Merge duplicate entries in array of dict

I'm struggling with a recursive merge problem.
Let's say I have:
a=[{'name':"bob",
'age':10,
'email':"bob#bla",
'profile':{'id':1, 'role':"admin"}},
{'name':"bob",
'age':10,
'email':"other mail",
'profile':{'id':2, 'role':"dba"},
'home':"/home/bob"
}]
and I need something to recursively merge entries. If value for an existing given key on the same level is different it appends the value to an array.
b = merge(a)
print b
{'name':"bob",
'age':10,
'email':["bob#bla","other mail"],
'profile':{'id':[1,2], 'role'=["admin", "dba"], 'home':"/home/bob"}
I wrote this code:
def merge(items):
merged = {}
for item in items:
for key in item.keys():
if key in merged.keys():
if item[key] != merged[key]:
if not isinstance(merged[key], list):
merged[key] = [merged[key]]
if item[key] not in merged[key]:
merged[key].append(item[key])
else:
merged[key] = item[key]
return merged
The output is:
{'age': 10,
'email': ['bob#bla', 'other mail'],
'home': '/home/bob',
'name': 'bob',
'profile': [{'id': 1, 'role': 'admin'}, {'id': 2, 'role': 'dba'}]}
Which is not what I want.
I can't figure out how to deal with recursion.
Thanks :)

As you iterate over each dictionary in the arguments, then each key and value in each dictionary, you want the following rules:
If there is nothing against that key in the output, add the new key and value to the output;
If there is a value for that key, and it's the same as the new value, do nothing;
If there is a value for that key, and it's a list, append the new value to the list;
If there is a value for that key, and it's a dictionary, recursively merge the new value with the existing dictionary;
If there is a value for that key, and it's neither a list nor a dictionary, make the value in the output a list of the current value and the new value.
In code:
def merge(*dicts):
"""Recursively merge the argument dictionaries."""
out = {}
for dct in dicts:
for key, val in dct.items():
try:
out[key].append(val) # 3.
except AttributeError:
if out[key] == val:
pass # 2.
elif isinstance(out[key], dict):
out[key] = merge(out[key], val) # 4.
else:
out[key] = [out[key], val] # 5.
except KeyError:
out[key] = val # 1.
return out
In use:
>>> import pprint
>>> pprint.pprint(merge(*a))
{'age': 10,
'email': ['bob#bla', 'other mail'],
'home': '/home/bob',
'name': 'bob',
'profile': {'id': [1, 2], 'role': ['admin', 'dba']}}