We Keep Coding

Python: consolidate list of lists

I want to consolidate a list of lists (of dicts), but I have honestly no idea how to get it done.
The list looks like this:
l1 = [
[
{'id': 1, 'category': 5}, {'id': 3, 'category': 7}
],
[
{'id': 1, 'category': 5}, {'id': 4, 'category': 8}, {'id': 6, 'category': 9}
],
[
{'id': 6, 'category': 9}, {'id': 9, 'category': 16}
],
[
{'id': 2, 'category': 4}, {'id': 5, 'category': 17}
]
]
If one of the dicts from l1[0] is also present in l1[1], I want to concatenate the two lists and delete l1[0]. Afterwards I want to check if there are values from l1[1] also present in l1[2].
So my desired output would eventually look like this:
new_list = [
[
{'id': 1, 'category': 5}, {'id': 3, 'category': 7}, {'id': 4, 'category': 8}, {'id': 6, 'category': 9}, {'id': 9, 'category': 16}
],
[
{'id': 2, 'category': 4}, {'id': 5, 'category': 17}
]
]
Any idea how it can be done?
I tried it with 3 different for loops, but it wouldnt work, because I change the length of the list and by doing so I provoke an index-out-of-range error (apart from that it would be an ugly solution anyway):
for list in l1:
for dictionary in list:
for index in range(0, len(l1), 1):
if dictionary in l1[index]:
dictionary in l1[index].append(list)
dictionary.remove(list)
Can I apply some map or list_comprehension here?
Thanks a lot for any help!

IIUC, the following algorithm works.
Initialize result to empty
For each sublist in l1:
if sublist and last item in result overlap
append into last list of result without overlapping items
otherwise
append sublist at end of result
Code
# Helper functions
def append(list1, list2):
' append list1 and list2 (without duplicating elements) '
return list1 + [d for d in list2 if not d in list1]
def is_intersect(list1, list2):
' True if list1 and list2 have an element in common '
return any(d in list2 for d in list1) or any(d in list1 for d in list2)
# Generate desired result
result = [] # resulting list
for sublist in l1:
if not result or not is_intersect(sublist, result[-1]):
result.append(sublist)
else:
# Intersection with last list, so append to last list in result
result[-1] = append(result[-1], sublist)
print(result)
Output
[[{'id': 1, 'category': 5},
{'id': 3, 'category': 7},
{'id': 4, 'category': 8},
{'id': 6, 'category': 9},
{'id': 9, 'category': 16}],
[{'id': 2, 'category': 4}, {'id': 5, 'category': 17}]]
​

maybe you can try to append the elements into a new list. by doing so, the original list will remain the same and index-out-of-range error wouldn't be raised.
new_list = []
for list in l1:
inner_list = []
for ...
if dictionary in l1[index]:
inner_list.append(list)
...
new_list.append(inner_list)

How to convert list of dict into two lists?

For example:
persons = [{'id': 1, 'name': 'john'}, {'id': 2, 'name': 'mary'}, {'id': 3, 'name': 'tom'}]
I want to get two lists from it:
ids = [1, 2, 3]
names = ['john', 'mary', 'tom']
What I did:
names = [d['name'] for d in persons]
ids = [d['id'] for d in persons]
Is there a better way to do it?

I'd stick with using list comprehension or use #Woodford technique
ids,name = [dcts['id'] for dcts in persons],[dcts['name'] for dcts in persons]
output
[1, 2, 3]
['john', 'mary', 'tom']

What you did works fine. Another way to handle this (not necessarily better, depending on your needs) is to store your data in a more efficient dictionary and pull the names/ids out of it when you need them:
>>> persons = [{'id': 1, 'name': 'john'}, {'id': 2, 'name': 'mary'}, {'id': 3, 'name': 'tom'}]
>>> p2 = {x['id']: x['name'] for x in persons}
>>> p2
{1: 'john', 2: 'mary', 3: 'tom'}
>>> list(p2.keys())
[1, 2, 3]
>>> list(p2.values())
['john', 'mary', 'tom']

You can do it with pandas in a vectorized fashion:
import pandas as pd
persons = [{'id': 1, 'name': 'john'}, {'id': 2, 'name': 'mary'}, {'id': 3, 'name': 'tom'}]
df = pd.DataFrame(persons)
id_list = df.id.tolist() #[1, 2, 3]
name_list = df.name.tolist() #['john', 'mary', 'tom']

An alternative, inspired by this question, is
ids, names = zip(*map(lambda x: x.values(), persons))
that return tuples. If you need lists
ids, names = map(list, zip(*map(lambda x: x.values(), persons)))
It is a little bit slower on my laptop using python3.9 than the accepted answer but it might be useful.

It sounds like you're trying to iterate through the values of your list while unpacking your dictionaries:
persons = [{'id': 1, 'name': 'john'}, {'id': 2, 'name': 'mary'}, {'id': 3, 'name': 'tom'}]
for x in persons:
id, name = x.values()
ids.append(id)
names.append(name)

Combining multiple lists of dictionaries

I have several lists of dictionaries, where each dictionary contains a unique id value that is common among all lists. I'd like to combine them into a single list of dicts, where each dict is joined on that id value.
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
desired_output = [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]
I tried doing something like the answer found at https://stackoverflow.com/a/42018660/7564393, but I'm getting very confused since I have more than 2 lists. Should I try using a defaultdict approach? More importantly, I am NOT always going to know the other values, only that the id value is present in all dicts.

You can use itertools.groupby():
from itertools import groupby
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
desired_output = []
for _, values in groupby(sorted([*list1, *list2, *list3], key=lambda x: x['id']), key=lambda x: x['id']):
temp = {}
for d in values:
temp.update(d)
desired_output.append(temp)
Result:
[{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]

list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
# combine all lists
d = {} # id -> dict
for l in [list1, list2, list3]:
for list_d in l:
if 'id' not in list_d: continue
id = list_d['id']
if id not in d:
d[id] = list_d
else:
d[id].update(list_d)
# dicts with same id are grouped together since id is used as key
res = [v for v in d.values()]
print(res)

You can first build a dict of dicts, then turn it into a list:
from itertools import chain
from collections import defaultdict
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
dict_out = defaultdict(dict)
for d in chain(list1, list2, list3):
dict_out[d['id']].update(d)
out = list(dict_out.values())
print(out)
# [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]
itertools.chain allows you to iterate on all the dicts contained in the 3 lists. We build a dict dict_out having the id as key, and the corresponding dict being built as value. This way, we can easily update the already built part with the small dict of our current iteration.

Here, I have presented a functional approach without using itertools (which is excellent in rapid development work).
This solution will work for any number of lists as the function takes variable number of arguments and also let user to specify the type of return output (list/dict).
By default it returns list as you want that otherwise it returns dictionary in case if you pass as_list = False.
I preferred dictionary to solve this because its fast and search complexity is also less.
Just have a look at the below get_packed_list() function.
get_packed_list()
def get_packed_list(*dicts_lists, as_list=True):
output = {}
for dicts_list in dicts_lists:
for dictionary in dicts_list:
_id = dictionary.pop("id") # id() is in-built function so preferred _id
if _id not in output:
# Create new id
output[_id] = {"id": _id}
for key in dictionary:
output[_id][key] = dictionary[key]
dictionary["id"] = _id # push back the 'id' after work (call by reference mechanism)
if as_list:
return [output[key] for key in output]
return output # dictionary
Test
list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
output = get_packed_list(list1, list2, list3)
print(output)
# [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]
output = get_packed_list(list1, list2, list3, as_list=False)
print(output)
# {1: {'id': 1, 'value': 20, 'sum': 10, 'total': 30}, 2: {'id': 2, 'value': 21, 'sum': 11, 'total': 32}}

list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
print(list1+list2+list3)

list1 = [{'id': 1, 'value': 20}, {'id': 2, 'value': 21}]
list2 = [{'id': 1, 'sum': 10}, {'id': 2, 'sum': 11}]
list3 = [{'id': 1, 'total': 30}, {'id': 2, 'total': 32}]
result = []
for i in range(0,len(list1)):
final_dict = dict(list(list1[i].items()) + list(list2[i].items()) + list(list3[i].items()))
result.append(final_dict)
print(result)
output : [{'id': 1, 'value': 20, 'sum': 10, 'total': 30}, {'id': 2, 'value': 21, 'sum': 11, 'total': 32}]

Assign keys to the tuple items

l1 = [1, 2, 3]
l2 = ['foo', 'bar', 'test']
z1 = zip(l1,l2)
list(z1)
[(1, 'foo'), (2, 'bar'), (3, 'test')]
That's a sample of my code. Now I want to map (??) each value of the tuples to either id, or name.
So I can get a result like :
[('id':1, 'name':'foo'), ('id':2, 'name':'bar'), ('id':3, 'name':'test')]
What I did is :
>>> result = []
>>> for i in zip(l1,l2):
... d['id'] = i[0]
... d['name'] = i[1]
... result.append(d)
>>> result
[{'id': 3, 'name': 'test'}, {'id': 3, 'name': 'test'}, {'id': 3, 'name': 'test'}]
but 1st) it doesn't works and 2nd) not pythonic at all as far as I can tell...
I don't understand why the loop above doesn't works. If I do :
>>> for i in zip(l1,l2):
... print(i[0], i[1])
...
1 foo
2 bar
3 test
it iterates every item without problem and the append() I used above with the list shouldn't cause any problems...

You can write that for loop pythonically using a dict comprehension:
[{'id': id, 'name': name} for id,name in zip(l1, l2)]
Depending on how complex this dict actually is, and what you want to do with it, you might want to consider using a simple class instead.

Id d is your dictionary what you are doing is replacing the values of the keys(id & name) of the dictionary d and your are appending it to a list.
To create a list of dictionary you can use a list comprehension like
l1 = [1, 2, 3]
l2 = ['foo', 'bar', 'test']
result = [{'id': i[0], 'name':i[1]} for i in zip(l1,l2)]

Your solution doesn't work because you are "overriding" the dict at each step :
>>> for i in zip(l1,l2):
... d['id']=i[0]
... d['name']=i[1]
... r.append(d)
... print r
...
[{'id': 1, 'name': 'foo'}]
[{'id': 2, 'name': 'bar'}, {'id': 2, 'name': 'bar'}]
[{'id': 3, 'name': 'test'}, {'id': 3, 'name': 'test'}, {'id': 3, 'name': 'test'}]
One way to go is to "reset" your dict at the beginning of each step:
>>> for i in zip(l1,l2):
... d={}
... d['id']=i[0]
... d['name']=i[1]
... r.append(d)
... print r
...
[{'id': 1, 'name': 'foo'}]
[{'id': 1, 'name': 'foo'}, {'id': 2, 'name': 'bar'}]
[{'id': 1, 'name': 'foo'}, {'id': 2, 'name': 'bar'}, {'id': 3, 'name': 'test'}]
But as others pointed out, [{'id': id, 'name': name} for id, name in zip(l1, l2)] is more pythonic.
And I don't know if you could use it, but you could build your dict from your zip:
>>> dict(zip(l1, l2))
{1: 'foo', 2: 'bar', 3: 'test'}

Iterating over list of dictionaries

I have a list -myList - where each element is a dictionary. I wish to iterate over this list but I am only interesting in one attribute - 'age' - in each dictionary each time. I am also interested in keeping count of the number of iterations.
I do:
for i, entry in enumerate(myList):
print i;
print entry['age'];
But was wondering is there something more pythonic. Any tips?

You could use a generator to only grab ages.
# Get a dictionary
myList = [{'age':x} for x in range(1,10)]
# Enumerate ages
for i, age in enumerate(d['age'] for d in myList):
print i,age
And, yeah, don't use semicolons.

Very simple way, list of dictionary iterate
>>> my_list
[{'age': 0, 'name': 'A'}, {'age': 1, 'name': 'B'}, {'age': 2, 'name': 'C'}, {'age': 3, 'name': 'D'}, {'age': 4, 'name': 'E'}, {'age': 5, 'name': 'F'}]
>>> ages = [li['age'] for li in my_list]
>>> ages
[0, 1, 2, 3, 4, 5]

For printing, probably what you're doing is just about right. But if you want to store the values, you could use a list comprehension:
>>> d_list = [dict((('age', x), ('foo', 1))) for x in range(10)]
>>> d_list
[{'age': 0, 'foo': 1}, {'age': 1, 'foo': 1}, {'age': 2, 'foo': 1}, {'age': 3, 'foo': 1}, {'age': 4, 'foo': 1}, {'age': 5, 'foo': 1}, {'age': 6, 'foo': 1}, {'age': 7, 'foo': 1}, {'age': 8, 'foo': 1}, {'age': 9, 'foo': 1}]
>>> ages = [d['age'] for d in d_list]
>>> ages
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> len(ages)
10

The semicolons at the end of lines aren't necessary in Python (though you can use them if you want to put multiple statements on the same line). So it would be more pythonic to omit them.
But the actual iteration strategy is easy to follow and pretty explicit about what you're doing. There are other ways to do it. But an explicit for-loop is perfectly pythonic.
(Niklas B.'s answer will not do precisely what you're doing: if you want to do something like that, the format string should be "{0}\n{1}".)