We Keep Coding

Combine dictionaries with the same value for a specific key

I have a group of dictionaries of 2 patterns like {"Id": 1, "title":"example"} and {"Id": 1, "location":"city"}. I want to combine these 2 together to get {"Id": 1, "title":"example", "location":"city"}, for all the dictionaries with Ids that match. In this case the group is of 200 items of 100 titles and 100 locations all with Ids from 0-99. I want to return a list of 100 combined dictionaries.
May be like the following:
def ResultHandler(extractedResult: list):
jsonObj = {}
jsonList = []
for result in extractedResult:
for key, val in result.items():
#this works if its hardcoded val to a number...
if key == "Id" and val == 1:
jsonObj.update(result)
jsonList.append(jsonObj)
return jsonList

Group the dicts by ID. Then merge each group.
from collections import defaultdict
def merge_dicts(dicts):
grouped = defaultdict(list)
for d in dicts:
grouped[d['id']].append(d)
merged = []
for ds in grouped.values():
m = {}
for d in ds:
m |= d # If Python < 3.9 : m = {**m, **d}
merged.append(m)
return merged

A more functional (but slightly less efficient) approach:
from itertools import groupby
from functools import reduce
from operator import itemgetter
new_data = []
for _, g in groupby(sorted(data, key=itemgetter("Id")), key=itemgetter("Id")):
new_data.append(reduce(lambda d1, d2: {**d1, **d2}, g))

This function has a nested loop. The outer loop iterates through the list of dictionaries. The inner loop
iterates through the list of dictionaries again to check if the id of the current dictionary is
already in the list of dictionaries. If it is not, it appends the dictionary to the list of
dictionaries. If it is, it updates the dictionary in the list of dictionaries with the contents of
the current dictionary.
lst = [
{"id": 1, "fname": "John"},
{"id": 2, "name": "Bob"},
{"id": 1, "lname": "Mary"},
]
def combine_dicts(lst):
res = []
for d in lst:
if d.get("id") not in [x.get("id") for x in res]:
res.append(d)
else:
for r in res:
if r.get("id") == d.get("id"):
r.update(d)
return res
print(combine_dicts(last))
# output: [{'id': 1, 'fname': 'John', 'lname': 'Mary'}, {'id': 2, 'name': 'Bob'}]

The following code should work:
def resultHandler(extractedResult):
jsonList = []
for i in range(len(extractedResult) // 2):
jsonList.append({"Id": i})
for i in range(len(extractedResult)):
for j in range(len(jsonList)):
if jsonList[j]["Id"] == extractedResult[i]["Id"]:
if "title" in extractedResult[i]:
jsonList[j]["title"] = extractedResult[i]["title"];
else:
jsonList[j]["location"] = extractedResult[i]["location"];
return jsonList;
extractedResult = [{"Id": 0, "title":"example1"}, {"Id": 1, "title":"example2"}, {"Id": 0, "location":"example3"}, {"Id": 1, "location":"example4"}]
jsonList = resultHandler(extractedResult)
print(jsonList)
Output:
[{'Id': 0, 'title': 'example1', 'location': 'example3'}, {'Id': 1, 'title': 'example2', 'location': 'example4'}]
This code works by first filling up jsonList with Id values from 0 to half of the length of extractedResult (so the number of IDs).
Then, for every dictionary in extractedResult, we find the dictionary in jsonList with the matching ID. If that dictionary of extractedResult contains a key, "title", then we create that value for that dictionary in jsonList. The same applied for "location".
I hope this helps answer your question! Please let me know if you need any further clarification or details :)

This code will solve your problem in linear time i.e., O(n) where n is the order of growth of the length of your dictionary. It will consider only those Id which has both title and location and will ignore the rest.
from collections import Counter
data = [{"Id": 1, "title":"example1"},
{"Id": 2, "title":"example2"},
{"Id": 3, "title":"example3"},
{"Id": 4, "title":"example4"},
{"Id": 1, "location":"city1"},
{"Id": 2, "location":"city2"},
{"Id": 4, "location":"city4"},
{"Id": 5, "location":"city5"}]
paired_ids = set([key for key, val in dict(Counter([item["Id"] for item in data])).items() if val == 2]) # O(n)
def combine_dict(data):
result = {key: [] for key in paired_ids} # O(m), m: number of paired ids (m <= n/2)
for item in data: # O(n)
items = list(item.items())
id, tl, val = items[0][1], items[1][0], items[1][1]
if id in paired_ids: # O(1), as paired_ids is a set lookup takes O(1)
result[id].append({tl: val})
return [{"Id": id, "title": lst[0]["title"], "location": lst[1]["location"]} for id, lst in result.items()] # O(n)
print(*combine_dict(data), sep="\n")
Output:
{'Id': 1, 'title': 'example1', 'location': 'city1'}
{'Id': 2, 'title': 'example2', 'location': 'city2'}
{'Id': 4, 'title': 'example4', 'location': 'city4'}

I want to create function that keep appending new data to nested dictionary

I want to store new data(Key, value) to the nested dictionary.
But I don't have any clue how to fix it.
def add(a,b,c,d,container)
container = {} # as database
data ={}
data[a] = [{"first": b, "second": c, "third": d}]
for e in data:
if date[date] not in calendar:
container[date[a]] = {}
container[date[a]].update([{"first": b, "second": c, "third": d})
add(name, 1, 2, hi, container)
add(name1, 2, 1, hi, container)
I see the following output:
{name: [{"first": 1, "second": 2, "third":hi }]}
{name1: [{"first": 2, "second": 1, "third":hi }]}
I expect the output as:
{name: [{"first": 1, "second": 2, "third":hi }], name1: [{"first": 2, "second": 1, "third":hi }]}
Please help me out!

You are creating a local dictionary in your add function. Did you see that?
def add(a,b,c,d,container)
container = {} # it's a local new dict
# ...
Instead you should create the dictionary outside of the function, otherwise you'll always get a new dictionary containing only one key. For example:
container = {}
def add(a, b, c, d):
container[a] = b
container[c] = d

i am creating a global_dict which is main dictionary where storing of all the
dict element is taken place.
for your given problem, if you want to add the data in a dict continously and that dict have same structure, then better:
is to create a default dict , and in that dict update the values and finally add that new small dict to the main dict using update method.
def funcion(a,b,c,d, container):
new_dic={a:[{'first':b,"Second":c,"third":d}]}
container.update(new_dic)
funcion('name', 1, 2, 'hi')
funcion('name1', 2, 1,'hi')
print(container)
"""
output
{'name': [{'first': 1, 'Second': 2, 'third': 'hi'}],
'name1': [{'first': 2, 'Second': 1, 'third': 'hi'}]
}
"

I recently made a module for this and posted it on github, it doesnt use recursion so therefore it has absolutely no limits to it. It allows you to edit, add, and remove from a nested dictionary using a keypath. Here it is on stackoverflow(this will answer your question):
How can you add entries, and retrieve, alter, or remove values from specific keys in any nested dictionary without recursion?
and here it is on github:
https://github.com/kthewhispers/Nested-Dictionary-Tools-Python/tree/master/src

How to sum a list of dicts

What is the most Pythonic way to take a list of dicts and sum up all the values for matching keys from every row in the list?
I did this but I suspect a comprehension is more Pythonic:
from collections import defaultdict
demandresult = defaultdict(int) # new blank dict to store results
for d in demandlist:
for k,v in d.iteritems():
demandresult[k] = demandresult[k] + v
In Python - sum values in dictionary the question involved the same key all the time, but in my case, the key in each row might be a new key never encountered before.

I think that your method is quite pythonic. Comprehensions are nice but they shouldn't really be overdone, and they can lead to really messy one-liners, like the one below :).
If you insist on a dict comp:
demand_list = [{u'2018-04-29': 1, u'2018-04-30': 1, u'2018-05-01': 1},
{u'2018-04-21': 1},
{u'2018-04-18': 1, u'2018-04-19': 1, u'2018-04-17' : 1}]
d = {key:sum(i[key] for i in demand_list if key in i)
for key in set(a for l in demand_list for a in l.keys())}
print(d)
>>>{'2018-04-21': 1, '2018-04-17': 1, '2018-04-29': 1, '2018-04-30': 1, '2018-04-19': 1, '2018-04-18': 1, '2018-05-01': 1}

Here is another one-liner (ab-)using collections.ChainMap to get the combined keys:
>>> from collections import ChainMap
>>> {k: sum(d.get(k, 0) for d in demand_list) for k in ChainMap(*demand_list)}
{'2018-04-17': 1, '2018-04-21': 1, '2018-05-01': 1, '2018-04-30': 1, '2018-04-19': 1, '2018-04-29': 1, '2018-04-18': 1}
This is easily the slowest of the methods proposed here.

The only thing that seemed unclear in your code was the double-for-loop. It may be clearer to collapse the demandlist into a flat iterable—then the loopant presents the logic as simply as possible. Consider:
demandlist = [{
u'2018-04-29': 1,
u'2018-04-30': 1,
u'2018-05-01': 1
}, {
u'2018-04-21': 1
}, {
u'2018-04-18': 1,
u'2018-04-19': 1,
u'2018-04-17': 1
}]
import itertools as it
from collections import defaultdict
demandresult = defaultdict(int)
for k, v in it.chain.from_iterable(map(lambda d: d.items(), demandlist)):
demandresult[k] = demandresult[k] + v
(With this, print(demandresult) prints defaultdict(<class 'int'>, {'2018-04-29': 1, '2018-04-30': 1, '2018-05-01': 1, '2018-04-21': 1, '2018-04-18': 1, '2018-04-19': 1, '2018-04-17': 1}).)
Imagining myself reading this for the first time (or a few months later), I can see myself thinking, "Ok, I'm collapsing demandlist into a key-val iterable, I don't particularly care how, and then summing values of matching keys."
It's unfortunate that I need that map there to ensure the final iterable has key-val pairs… it.chain.from_iterable(demandlist) is a key-only iterable, so I need to call items on each dict.
Note that unlike many of the answers proposed, this implementation (like yours!) minimizes the number of scans over the data to just one—performance win (and I try to pick up as many easy performance wins as I can).

I suppose you want to return a list of summed values of each dictionary.
list_of_dict = [
{'a':1, 'b':2, 'c':3},
{'d':4, 'e':5, 'f':6}
]
sum_of_each_row = [sum(v for v in d.values()) for d in list_of_dict] # [6,15]
If you want to return the total sum, just simply wrap sum() to "sum_of_each_row".
EDIT:
The main problem is that you don't have a default value for each of the keys, so you can make use of the method dict.setdefault() to set the default value when there's a new key.
list_of_dict = [
{'a':1, 'b':1},
{'b':1, 'c':1},
{'a':2}
]
d = {}
d = {k:d[k]+v if k in d.keys() else d.setdefault(k,v)
for row in list_of_dict for k,v in row.items()} # {'a':3, 'b':2, 'c':1}

Denormalize/flatten list of nested objects into dot separated key value pairs

It would have simpler if my nested objects were dictionaries, but these are list of dictionaries.
Example:
all_objs1 = [{
'a': 1,
'b': [{'ba': 2, 'bb': 3}, {'ba': 21, 'bb': 31}],
'c': 4
}, {
'a': 11,
'b': [{'ba': 22, 'bb': 33, 'bc': [{'h': 1, 'e': 2}]}],
'c': 44
}]
I expect output in following format:
[
{'a': 1, 'b.ba': 2, 'b.bb': 3, 'c': 4},
{'a': 1, 'b.ba': 21, 'b.bb': 31, 'c': 4},
{'a': 11, 'b.ba': 22, 'b.bb': 33, 'bc.h': 1, 'bc.e': 2, 'c': 44},
]
Basically, number of flattened objects generated will be equal to (obj * depth)
With my current code:
def flatten(obj, flattened_obj, last_key=''):
for k,v in obj.iteritems():
if not isinstance(v, list):
flattened_obj.update({last_key+k : v})
else:
last_key += k + '.'
for nest_obj in v:
flatten(nest_obj, flattened_obj, last_key)
last_key = remove_last_key(last_key)
def remove_last_key(key_path):
second_dot = key_path[:-1].rfind('.')
if second_dot > 0:
return key_path[:second_dot+1]
return key_path
Output:
[
{'a': 1, 'b.bb': 31, 'c': 4, 'b.ba': 21},
{'a': 11, 'b.bc.e': 2, 'c': 44, 'b.bc.h': 1, 'b.bb': 33, 'b.ba': 22}
]
I am able to flatten the object (not accurate though), but I am not able to create a new object at each nested object.
I can not use pandas library as my app is deployed on app engine.

code.py:
from itertools import product
from pprint import pprint as pp
all_objs = [{
"a": 1,
"b": [{"ba": 2, "bb": 3}, {"ba": 21, "bb": 31}],
"c": 4,
#"d": [{"da": 2}, {"da": 5}],
}, {
"a": 11,
"b": [{"ba": 22, "bb": 33, "bc": [{"h": 1, "e": 2}]}],
"c": 44,
}]
def flatten_dict(obj, parent_key=None):
base_dict = dict()
complex_items = list()
very_complex_items = list()
for key, val in obj.items():
new_key = ".".join((parent_key, key)) if parent_key is not None else key
if isinstance(val, list):
if len(val) > 1:
very_complex_items.append((key, val))
else:
complex_items.append((key, val))
else:
base_dict[new_key] = val
if not complex_items and not very_complex_items:
return [base_dict]
base_dicts = list()
partial_dicts = list()
for key, val in complex_items:
partial_dicts.append(flatten_dict(val[0], parent_key=new_key))
for product_tuple in product(*tuple(partial_dicts)):
new_base_dict = base_dict.copy()
for new_dict in product_tuple:
new_base_dict.update(new_dict)
base_dicts.append(new_base_dict)
if not very_complex_items:
return base_dicts
ret = list()
very_complex_keys = [item[0] for item in very_complex_items]
very_complex_vals = tuple([item[1] for item in very_complex_items])
for product_tuple in product(*very_complex_vals):
for base_dict in base_dicts:
new_dict = base_dict.copy()
new_items = zip(very_complex_keys, product_tuple)
for key, val in new_items:
new_key = ".".join((parent_key, key)) if parent_key is not None else key
new_dict.update(flatten_dict(val, parent_key=new_key)[0])
ret.append(new_dict)
return ret
def main():
flatten = list()
for obj in all_objs:
flatten.extend(flatten_dict(obj))
pp(flatten)
if __name__ == "__main__":
main()
Notes:
As expected, recursion is used
It's general, it also works for the case that I mentioned in my 2nd comment (for one input dict having more than one key with a value consisting of a list with more than one element), that can be tested by decommenting the "d" key in all_objs. Also, theoretically it should support any depth
flatten_dict: takes an input dictionary and outputs a list of dictionaries (as the input dictionary might yield more than one output dictionary):
Every key having a "simple" (not list) value, goes into the output dictionar(y/ies) unchanged
At this point, a base output dictionary is complete (if the input dictionary will generate more than output dictionary, all will have the base dictionary keys/values, if it only generates one output dictionary, then that will be the base one)
Next, the keys with "problematic" values - that may generate more than output dictionary - (if any) are processed:
Keys having a list with a single element ("problematic") - each might generate more than one output dictionary:
Each of the values will be flattened (might yield more than one output dictionary); the corresponding key will be used in the process
Then, the cartesian product will be computed on all the flatten dictionary lists (for current input, there will only be one list with one element)
Now, each product item needs to be in a distinct output dictionary, so the base dictionary is duplicated and updated with the keys / values of every element in the product item (for current input, there will be only one element per product item)
The new dictionary is appended to a list
At this point a list of base dictionaries (might only be one) is complete, if no values consisting of lists with more than one element are present, this is the return list, else everything below has to be done for each base dictionary in the list
Keys having a list with a more elements ("very problematic") - each will generate more than one output dictionaries:
First, the cartesian product will be computed against all the values (lists with more than one element). In the current case, since since it's only one such list, each product item will only contain an element from that list
Then, for each product item element, its key will need to be established based on the lists order (for the current input, the product item will only contain one element, and also, there will only be one key)
Again, each product item needs to be in a distinct output dictionary, so the base dictionary is duplicated and updated with the keys / values, of the flattened product item
The new dictionary is appended to the output dictionaries list
Works with Python 3 and Python 2
Might be slow (especially for big input objects), as performance was not the goal. Also since it was built bottom-up (adding functionality as new cases were handled), it is pretty twisted (RO: intortocheated :) ), there might be a simpler implementation that I missed.
Output:
c:\Work\Dev\StackOverflow\q046341856>c:\Work\Dev\VEnvs\py35x64_test\Scripts\python.exe code.py
[{'a': 1, 'b.ba': 2, 'b.bb': 3, 'c': 4},
{'a': 1, 'b.ba': 21, 'b.bb': 31, 'c': 4},
{'a': 11, 'b.ba': 22, 'b.bb': 33, 'b.bc.e': 2, 'b.bc.h': 1, 'c': 44}]
#EDIT0:
Made it more general (although it's not visible for the current input): values containing only one element can yield more than output dictionary (when flattened), addressed that case (before I was only considering the 1st output dictionary, simply ignoring the rest)
Corrected a logical error that was masked out tuple unpacking combined with cartesian product: if not complex_items ... part
#EDIT1:
Modified the code to match a requirement change: the key in the flattened dictionary must have the full nesting path in the input dictionary

use this code to get your desired output. It generates output based on recursive call.
import json
from copy import deepcopy
def flatten(final_list, all_obj, temp_dct, last_key):
for dct in all_obj:
deep_temp_dct = deepcopy(temp_dct)
for k, v in dct.items():
if isinstance(v, list):
final_list, deep_temp_dct = flatten(final_list, v, deep_temp_dct, k)
else:
prefix = ""
if last_key : prefix = last_key + "."
key = prefix+ k
deep_temp_dct[key] = v
if deep_temp_dct not in final_list:
final_list.append(deep_temp_dct)
return final_list, deep_temp_dct
final_list, _ = flatten([], all_objs1, {}, "")
print json.dumps(final_list, indent =4 )
let me know if it works for you.

Creating multiple dictionaries based on other dictionary values in python

I have a list that contains many dictionaries. Each dictionary represents a change that has occurred within my application. The "change" dictionary has the following entries:
userid: The user ID for a user
ctype: A reference to a change type in my application
score: A score
The ctype can be one of about 12 different strings to include "deletion", "new", "edit" and others. Here is an example of one of the "change" dictionaries:
{'userid':2, 'score':10, 'ctype':'edit'}
My question is, how can I create a dictionary that will aggregate all of the change types for each user within this large list of dictionaries? I would like to add the score from each change dictionary to create a total score and add each ctype instance together to get a count of each instance. The goal is to have a list of dictionaries with each dictionary looking like this:
{'userid':2, 'score':325, 'deletion':2, 'new':4, 'edit':9}
I have been trying to work this out but I am pretty new to python and I wasn't sure how to count the actual change types. The other part that gets me is how to refer to a dictionary based on 'userid'. If someone can present an answer I am sure that all of this will become very apparent to me. I appreciate any and all help.

The key thing to agregate data here is to have a dictionary where each key is the userid, and each entry is the data relevant to that userid.
final_data = {}
for entry in data:
userid = entry["userid"]
if userid not in final_data:
final_data[userid] = {"userid": userid, "score": 0}
final_data[userid]["score"] += entry["score"]
if not entry["ctype"] in final_data[userid]:
final_data[userid][entry["ctype"]] = 1
else:
final_data[userid][entry["ctype"]] += 1
If you want the result as a list of dictionaries, just use final_data.values()

Could you have
(Mock up not real python.)
{userid : {score : 1, ctype : ''}}
You can nest dict's as values in python dictionaries.

It could look like so:
change_types = ['deletion', 'new', 'edit', ...]
user_changes = {}
for change in change_list:
userid = change['userid']
if not userid in user_changes:
aggregate = {}
aggregate['score'] = 0
for c in change_types:
aggregate[c] = 0
aggregate['userid'] = userid
user_changes[userid] = aggregate
else:
aggregate = user_changes[userid]
change_type = change['ctype']
aggregate[change_type] = aggregate[change_type] + 1
aggregate['score'] = aggregate['score'] + change['score']
Actually making a class for the aggregates would be a good idea.

To index the dictionaries with respect to userid, you can use a dictionary of dictionaries:
from collections import defaultdict
dict1 = {'userid': 1, 'score': 10, 'ctype': 'edit'}
dict2 = {'userid': 2, 'score': 13, 'ctype': 'other'}
dict3 = {'userid': 1, 'score': 1, 'ctype': 'edit'}
list_of_dicts = [dict1, dict2, dict3]
user_dict = defaultdict(lambda: defaultdict(int))
for d in list_of_dicts:
userid = d['userid']
user_dict[userid]['score'] += d['score']
user_dict[userid][d['ctype']] += 1
# user_dict is now
# defaultdict(<function <lambda> at 0x02A7DF30>,
# {1: defaultdict(<type 'int'>, {'edit': 2, 'score': 11}),
# 2: defaultdict(<type 'int'>, {'score': 13, 'other': 1})})
In the example, I used a defaultdict to avoid checking at every iteration if the key d['ctype'] exists.