We Keep Coding

Python: consolidate list of lists

I want to consolidate a list of lists (of dicts), but I have honestly no idea how to get it done.
The list looks like this:
l1 = [
[
{'id': 1, 'category': 5}, {'id': 3, 'category': 7}
],
[
{'id': 1, 'category': 5}, {'id': 4, 'category': 8}, {'id': 6, 'category': 9}
],
[
{'id': 6, 'category': 9}, {'id': 9, 'category': 16}
],
[
{'id': 2, 'category': 4}, {'id': 5, 'category': 17}
]
]
If one of the dicts from l1[0] is also present in l1[1], I want to concatenate the two lists and delete l1[0]. Afterwards I want to check if there are values from l1[1] also present in l1[2].
So my desired output would eventually look like this:
new_list = [
[
{'id': 1, 'category': 5}, {'id': 3, 'category': 7}, {'id': 4, 'category': 8}, {'id': 6, 'category': 9}, {'id': 9, 'category': 16}
],
[
{'id': 2, 'category': 4}, {'id': 5, 'category': 17}
]
]
Any idea how it can be done?
I tried it with 3 different for loops, but it wouldnt work, because I change the length of the list and by doing so I provoke an index-out-of-range error (apart from that it would be an ugly solution anyway):
for list in l1:
for dictionary in list:
for index in range(0, len(l1), 1):
if dictionary in l1[index]:
dictionary in l1[index].append(list)
dictionary.remove(list)
Can I apply some map or list_comprehension here?
Thanks a lot for any help!

IIUC, the following algorithm works.
Initialize result to empty
For each sublist in l1:
if sublist and last item in result overlap
append into last list of result without overlapping items
otherwise
append sublist at end of result
Code
# Helper functions
def append(list1, list2):
' append list1 and list2 (without duplicating elements) '
return list1 + [d for d in list2 if not d in list1]
def is_intersect(list1, list2):
' True if list1 and list2 have an element in common '
return any(d in list2 for d in list1) or any(d in list1 for d in list2)
# Generate desired result
result = [] # resulting list
for sublist in l1:
if not result or not is_intersect(sublist, result[-1]):
result.append(sublist)
else:
# Intersection with last list, so append to last list in result
result[-1] = append(result[-1], sublist)
print(result)
Output
[[{'id': 1, 'category': 5},
{'id': 3, 'category': 7},
{'id': 4, 'category': 8},
{'id': 6, 'category': 9},
{'id': 9, 'category': 16}],
[{'id': 2, 'category': 4}, {'id': 5, 'category': 17}]]
​

maybe you can try to append the elements into a new list. by doing so, the original list will remain the same and index-out-of-range error wouldn't be raised.
new_list = []
for list in l1:
inner_list = []
for ...
if dictionary in l1[index]:
inner_list.append(list)
...
new_list.append(inner_list)

itertools.permutations for a set of dicts

before I ask my question I wanted to let y'all know that I am a beginner coder and I appreciate all inputs so thanks in advance for your help.
Assume I have the following object:
set: [
{'id': 1
'type': a
},
{'id': 2
'type': a
},
{'id': 3
'type': b
},
{'id': 4
'type': b
},
{'id': 5
'type': a
}
]
How could I use itertools.permutations to calculate (and eventually make copies of) all combinations of the 'type' key? i.e. have a set in which ids 3 and 5 have type b, the rest a, a set in which ids 2 and 5 have b and the rest a, etc.
Furthermore, is it possible calculate combinations given certain conditions (for example, ids 1 and 2 have to always be type a for all combos)?
Thank you all in advance

Building off of what John already has, we can constrain the permutations as you described by removing the constrained types from the types list before calculating permutations. Then you could insert the types back into each permutation as defined by the constraint. The following example script would return the permutations with the constraint: "ids 1 and 2 have to always be type a for all combos". Feel free to remove the print statements.
from itertools import permutations
# these define which types must be located at the given id (index + 1)
constraints = [
{'id': 1, 'type': "a"},
{'id': 2, 'type': "a"}
]
# this is the list of types to permute
typeset = [
{'id': 1, 'type': "a"},
{'id': 2, 'type': "a"},
{'id': 3, 'type': "b"},
{'id': 4, 'type': "b"},
{'id': 5, 'type': "a"}
]
# we create lists of types to permute and remove the ones that are constrained
types_to_permute = [d["type"] for d in typeset]
constraint_types = [d["type"] for d in constraints]
for constraint_type in constraint_types:
types_to_permute.remove(constraint_type)
print(types_to_permute)
# we calculate the permutations for the unconstrained types
permutations = list(permutations(types_to_permute, r=None))
permutations = [list(permutation) for permutation in permutations]
print(permutations)
# we insert the constrained types in their given locations
for permutation in permutations:
for constraint in constraints:
permutation.insert(constraint["id"] - 1, constraint["type"])
print(permutations)
# we reconstruct the permutations in the orignal format (list of dictionaries)
final_permutations = []
for permutation in permutations:
final_permutation = []
for index, type_object in enumerate(permutation):
final_permutation.append({"id": index + 1, "type": type_object})
final_permutations.append(final_permutation)
print(final_permutations)
The final permutations would be:
[
[
{'id': 1, 'type': 'a'},
{'id': 2, 'type': 'a'},
{'id': 3, 'type': 'b'},
{'id': 4, 'type': 'b'},
{'id': 5, 'type': 'a'}],
[
{'id': 1, 'type': 'a'},
{'id': 2, 'type': 'a'},
{'id': 3, 'type': 'b'},
{'id': 4, 'type': 'a'},
{'id': 5, 'type': 'b'}],
[
{'id': 1, 'type': 'a'},
{'id': 2, 'type': 'a'},
{'id': 3, 'type': 'b'},
{'id': 4, 'type': 'b'},
{'id': 5, 'type': 'a'}],
[
{'id': 1, 'type': 'a'},
{'id': 2, 'type': 'a'},
{'id': 3, 'type': 'b'},
{'id': 4, 'type': 'a'},
{'id': 5, 'type': 'b'}],
[
{'id': 1, 'type': 'a'},
{'id': 2, 'type': 'a'},
{'id': 3, 'type': 'a'},
{'id': 4, 'type': 'b'},
{'id': 5, 'type': 'b'}],
[
{'id': 1, 'type': 'a'},
{'id': 2, 'type': 'a'},
{'id': 3, 'type': 'a'},
{'id': 4, 'type': 'b'},
{'id': 5, 'type': 'b'}]]

You could use a list comprehension to pull out all the "type" values, then you can either push them through a set (or just leave it in the list) to compute the permutations.
from itertools import permutations
typeset = [...] # what you named `set` in the question
types = [d["type"] for d in typeset]
# Optionally, push `types` through a `set` to get only the unique `type` values
# types = set(types)
combos = list(permutations(types, r=None))
In case you haven't looked, here's a link to the docs on permutations. r is entirely optional, but it would let you compute permutations of length r instead of len(types).
Alternatively, let's consider the case where you want the permutation of your List[Dict]:
from itertools import permutations
typeset = [...] # as before
types = range(len(typeset))
combos = list(permutations(types, r=None))
You can now retrieve all permutations of typeset based on the indices stored in combos. However, depending on the size of your typeset, I wouldn't recommend it as that could consume quite a bit of memory.

How to convert key to value in dictionary type?

I have a question about the convert key.
First, I have this type of word count in Data Frame.
[Example]
dict = {'forest': 10, 'station': 3, 'office': 7, 'park': 2}
I want to get this result.
[Result]
result = {'name': 'forest', 'value': 10,
'name': 'station', 'value': 3,
'name': 'office', 'value': 7,
'name': 'park', 'value': 2}
Please check this issue.

As Rakesh said:
dict cannot have duplicate keys
The closest way to achieve what you want is to build something like that
my_dict = {'forest': 10, 'station': 3, 'office': 7, 'park': 2}
result = list(map(lambda x: {'name': x[0], 'value': x[1]}, my_dict.items()))
You will get
result = [
{'name': 'forest', 'value': 10},
{'name': 'station', 'value': 3},
{'name': 'office', 'value': 7},
{'name': 'park', 'value': 2},
]

As Rakesh said, You can't have duplicate values in the dictionary
You can simply try this.
dict = {'forest': 10, 'station': 3, 'office': 7, 'park': 2}
result = {}
count = 0;
for key in dict:
result[count] = {'name':key, 'value': dict[key]}
count = count + 1;
print(result)

Combining multiple lists of dictionaries

I have several lists of dictionaries, where each dictionary contains a unique id value that is common among all lists. I'd like to combine them into a single list of dicts, where each dict is joined on that id value.
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
desired_output = [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]
I tried doing something like the answer found at https://stackoverflow.com/a/42018660/7564393, but I'm getting very confused since I have more than 2 lists. Should I try using a defaultdict approach? More importantly, I am NOT always going to know the other values, only that the id value is present in all dicts.

You can use itertools.groupby():
from itertools import groupby
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
desired_output = []
for _, values in groupby(sorted([*list1, *list2, *list3], key=lambda x: x['id']), key=lambda x: x['id']):
temp = {}
for d in values:
temp.update(d)
desired_output.append(temp)
Result:
[{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]

list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
# combine all lists
d = {} # id -> dict
for l in [list1, list2, list3]:
for list_d in l:
if 'id' not in list_d: continue
id = list_d['id']
if id not in d:
d[id] = list_d
else:
d[id].update(list_d)
# dicts with same id are grouped together since id is used as key
res = [v for v in d.values()]
print(res)

You can first build a dict of dicts, then turn it into a list:
from itertools import chain
from collections import defaultdict
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
dict_out = defaultdict(dict)
for d in chain(list1, list2, list3):
dict_out[d['id']].update(d)
out = list(dict_out.values())
print(out)
# [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]
itertools.chain allows you to iterate on all the dicts contained in the 3 lists. We build a dict dict_out having the id as key, and the corresponding dict being built as value. This way, we can easily update the already built part with the small dict of our current iteration.

Here, I have presented a functional approach without using itertools (which is excellent in rapid development work).
This solution will work for any number of lists as the function takes variable number of arguments and also let user to specify the type of return output (list/dict).
By default it returns list as you want that otherwise it returns dictionary in case if you pass as_list = False.
I preferred dictionary to solve this because its fast and search complexity is also less.
Just have a look at the below get_packed_list() function.
get_packed_list()
def get_packed_list(*dicts_lists, as_list=True):
output = {}
for dicts_list in dicts_lists:
for dictionary in dicts_list:
_id = dictionary.pop("id") # id() is in-built function so preferred _id
if _id not in output:
# Create new id
output[_id] = {"id": _id}
for key in dictionary:
output[_id][key] = dictionary[key]
dictionary["id"] = _id # push back the 'id' after work (call by reference mechanism)
if as_list:
return [output[key] for key in output]
return output # dictionary
Test
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
output = get_packed_list(list1, list2, list3)
print(output)
# [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]
output = get_packed_list(list1, list2, list3, as_list=False)
print(output)
# {1: {'id': 1, 'value': 20, 'sum': 10, 'total': 30}, 2: {'id': 2, 'value': 21, 'sum': 11, 'total': 32}}

list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
print(list1+list2+list3)

list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
result = []
for i in range(0,len(list1)):
final_dict = dict(list(list1[i].items()) + list(list2[i].items()) + list(list3[i].items()))
result.append(final_dict)
print(result)
output : [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]

Ordering a Django queryset based on other list with ids and scores

I'm a bit mentally stuck at something, that seems really simple at first glance.
I'm grabbing a list of ids to be selected and scores to sort them based on.
My current solution is the following:
ids = [1, 2, 3, 4, 5]
items = Item.objects.filter(pk__in=ids)
Now I need to add a score based ordering somehow so I'll build the following list:
scores = [
{'id': 1, 'score': 15},
{'id': 2, 'score': 7},
{'id': 3, 'score': 17},
{'id': 4, 'score': 11},
{'id': 5, 'score': 9},
]
ids = [score['id'] for score in scores]
items = Item.objects.filter(pk__in=ids)
So far so good - but how do I actually add the scores as some sort of aggregate and sort the queryset based on them?

Sort the scores list, and fetch the queryset using in_bulk().
scores = [
{'id': 1, 'score': 15},
{'id': 2, 'score': 7},
{'id': 3, 'score': 17},
{'id': 4, 'score': 11},
{'id': 5, 'score': 9},
]
sorted_scores = sorted(scores) # use reverse=True for descending order
ids = [score['id'] for score in scores]
items = Item.objects.in_bulk(ids)
Then generate a list of the items in the order you want:
items_in_order = [items[x] for x in ids]