Let's say I have a list of dictionaries:
[
{'id': 1, 'name': 'john', 'age': 34},
{'id': 1, 'name': 'john', 'age': 34},
{'id': 2, 'name': 'hanna', 'age': 30},
]
How can I obtain a list of unique dictionaries (removing the duplicates)?
[
{'id': 1, 'name': 'john', 'age': 34},
{'id': 2, 'name': 'hanna', 'age': 30},
]
So make a temporary dict with the key being the id. This filters out the duplicates.
The values() of the dict will be the list
In Python2.7
>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... ]
>>> {v['id']:v for v in L}.values()
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]
In Python3
>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... ]
>>> list({v['id']:v for v in L}.values())
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]
In Python2.5/2.6
>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... ]
>>> dict((v['id'],v) for v in L).values()
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]
The usual way to find just the common elements in a set is to use Python's set class. Just add all the elements to the set, then convert the set to a list, and bam the duplicates are gone.
The problem, of course, is that a set() can only contain hashable entries, and a dict is not hashable.
If I had this problem, my solution would be to convert each dict into a string that represents the dict, then add all the strings to a set() then read out the string values as a list() and convert back to dict.
A good representation of a dict in string form is JSON format. And Python has a built-in module for JSON (called json of course).
The remaining problem is that the elements in a dict are not ordered, and when Python converts the dict to a JSON string, you might get two JSON strings that represent equivalent dictionaries but are not identical strings. The easy solution is to pass the argument sort_keys=True when you call json.dumps().
EDIT: This solution was assuming that a given dict could have any part different. If we can assume that every dict with the same "id" value will match every other dict with the same "id" value, then this is overkill; #gnibbler's solution would be faster and easier.
EDIT: Now there is a comment from André Lima explicitly saying that if the ID is a duplicate, it's safe to assume that the whole dict is a duplicate. So this answer is overkill and I recommend #gnibbler's answer.
In case the dictionaries are only uniquely identified by all items (ID is not available) you can use the answer using JSON. The following is an alternative that does not use JSON, and will work as long as all dictionary values are immutable
[dict(s) for s in set(frozenset(d.items()) for d in L)]
Here's a reasonably compact solution, though I suspect not particularly efficient (to put it mildly):
>>> ds = [{'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30}
... ]
>>> map(dict, set(tuple(sorted(d.items())) for d in ds))
[{'age': 30, 'id': 2, 'name': 'hanna'}, {'age': 34, 'id': 1, 'name': 'john'}]
You can use numpy library (works for Python2.x only):
import numpy as np
list_of_unique_dicts=list(np.unique(np.array(list_of_dicts)))
To get it worked with Python 3.x (and recent versions of numpy), you need to convert array of dicts to numpy array of strings, e.g.
list_of_unique_dicts=list(np.unique(np.array(list_of_dicts).astype(str)))
a = [
{'id':1,'name':'john', 'age':34},
{'id':1,'name':'john', 'age':34},
{'id':2,'name':'hanna', 'age':30},
]
b = {x['id']:x for x in a}.values()
print(b)
outputs:
[{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]
Since the id is sufficient for detecting duplicates, and the id is hashable: run 'em through a dictionary that has the id as the key. The value for each key is the original dictionary.
deduped_dicts = dict((item["id"], item) for item in list_of_dicts).values()
In Python 3, values() doesn't return a list; you'll need to wrap the whole right-hand-side of that expression in list(), and you can write the meat of the expression more economically as a dict comprehension:
deduped_dicts = list({item["id"]: item for item in list_of_dicts}.values())
Note that the result likely will not be in the same order as the original. If that's a requirement, you could use a Collections.OrderedDict instead of a dict.
As an aside, it may make a good deal of sense to just keep the data in a dictionary that uses the id as key to begin with.
We can do with pandas
import pandas as pd
yourdict=pd.DataFrame(L).drop_duplicates().to_dict('r')
Out[293]: [{'age': 34, 'id': 1, 'name': 'john'}, {'age': 30, 'id': 2, 'name': 'hanna'}]
Notice slightly different from the accept answer.
drop_duplicates will check all column in pandas , if all same then the row will be dropped .
For example :
If we change the 2nd dict name from john to peter
L=[
{'id': 1, 'name': 'john', 'age': 34},
{'id': 1, 'name': 'peter', 'age': 34},
{'id': 2, 'name': 'hanna', 'age': 30},
]
pd.DataFrame(L).drop_duplicates().to_dict('r')
Out[295]:
[{'age': 34, 'id': 1, 'name': 'john'},
{'age': 34, 'id': 1, 'name': 'peter'},# here will still keeping the dict in the out put
{'age': 30, 'id': 2, 'name': 'hanna'}]
There are a lot of answers here, so let me add another:
import json
from typing import List
def dedup_dicts(items: List[dict]):
dedupped = [ json.loads(i) for i in set(json.dumps(item, sort_keys=True) for item in items)]
return dedupped
items = [
{'id': 1, 'name': 'john', 'age': 34},
{'id': 1, 'name': 'john', 'age': 34},
{'id': 2, 'name': 'hanna', 'age': 30},
]
dedup_dicts(items)
I have summarized my favorites to try out:
https://repl.it/#SmaMa/Python-List-of-unique-dictionaries
# ----------------------------------------------
# Setup
# ----------------------------------------------
myList = [
{"id":"1", "lala": "value_1"},
{"id": "2", "lala": "value_2"},
{"id": "2", "lala": "value_2"},
{"id": "3", "lala": "value_3"}
]
print("myList:", myList)
# -----------------------------------------------
# Option 1 if objects has an unique identifier
# -----------------------------------------------
myUniqueList = list({myObject['id']:myObject for myObject in myList}.values())
print("myUniqueList:", myUniqueList)
# -----------------------------------------------
# Option 2 if uniquely identified by whole object
# -----------------------------------------------
myUniqueSet = [dict(s) for s in set(frozenset(myObject.items()) for myObject in myList)]
print("myUniqueSet:", myUniqueSet)
# -----------------------------------------------
# Option 3 for hashable objects (not dicts)
# -----------------------------------------------
myHashableObjects = list(set(["1", "2", "2", "3"]))
print("myHashAbleList:", myHashableObjects)
In python 3, simple trick, but based on unique field (id):
data = [ {'id': 1}, {'id': 1}]
list({ item['id'] : item for item in data}.values())
I don't know if you only want the id of your dicts in the list to be unique, but if the goal is to have a set of dict where the unicity is on all keys' values.. you should use tuples key like this in your comprehension :
>>> L=[
... {'id':1,'name':'john', 'age':34},
... {'id':1,'name':'john', 'age':34},
... {'id':2,'name':'hanna', 'age':30},
... {'id':2,'name':'hanna', 'age':50}
... ]
>>> len(L)
4
>>> L=list({(v['id'], v['age'], v['name']):v for v in L}.values())
>>>L
[{'id': 1, 'name': 'john', 'age': 34}, {'id': 2, 'name': 'hanna', 'age': 30}, {'id': 2, 'name': 'hanna', 'age': 50}]
>>>len(L)
3
Hope it helps you or another person having the concern....
Expanding on John La Rooy (Python - List of unique dictionaries) answer, making it a bit more flexible:
def dedup_dict_list(list_of_dicts: list, columns: list) -> list:
return list({''.join(row[column] for column in columns): row
for row in list_of_dicts}.values())
Calling Function:
sorted_list_of_dicts = dedup_dict_list(
unsorted_list_of_dicts, ['id', 'name'])
If there is not a unique id in the dictionaries, then I'd keep it simple and define a function as follows:
def unique(sequence):
result = []
for item in sequence:
if item not in result:
result.append(item)
return result
The advantage with this approach, is that you can reuse this function for any comparable objects. It makes your code very readable, works in all modern versions of Python, preserves the order in the dictionaries, and is fast too compared to its alternatives.
>>> L = [
... {'id': 1, 'name': 'john', 'age': 34},
... {'id': 1, 'name': 'john', 'age': 34},
... {'id': 2, 'name': 'hanna', 'age': 30},
... ]
>>> unique(L)
[{'id': 1, 'name': 'john', 'age': 34}, {'id': 2, 'name': 'hanna', 'age': 30}]
In python 3.6+ (what I've tested), just use:
import json
#Toy example, but will also work for your case
myListOfDicts = [{'a':1,'b':2},{'a':1,'b':2},{'a':1,'b':3}]
#Start by sorting each dictionary by keys
myListOfDictsSorted = [sorted(d.items()) for d in myListOfDicts]
#Using json methods with set() to get unique dict
myListOfUniqueDicts = list(map(json.loads,set(map(json.dumps, myListOfDictsSorted))))
print(myListOfUniqueDicts)
Explanation: we're mapping the json.dumps to encode the dictionaries as json objects, which are immutable. set can then be used to produce an iterable of unique immutables. Finally, we convert back to our dictionary representation using json.loads. Note that initially, one must sort by keys to arrange the dictionaries in a unique form. This is valid for Python 3.6+ since dictionaries are ordered by default.
Well all the answers mentioned here are good, but in some answers one can face error if the dictionary items have nested list or dictionary, so I propose simple answer
a = [str(i) for i in a]
a = list(set(a))
a = [eval(i) for i in a]
Objects can fit into sets. You can work with objects instead of dicts and if needed after all set insertions convert back to a list of dicts. Example
class Person:
def __init__(self, id, age, name):
self.id = id
self.age = age
self.name = name
my_set = {Person(id=2, age=3, name='Jhon')}
my_set.add(Person(id=3, age=34, name='Guy'))
my_set.add({Person(id=2, age=3, name='Jhon')})
# if needed convert to list of dicts
list_of_dict = [{'id': obj.id,
'name': obj.name,
'age': obj.age} for obj in my_set]
A quick-and-dirty solution is just by generating a new list.
sortedlist = []
for item in listwhichneedssorting:
if item not in sortedlist:
sortedlist.append(item)
Let me add mine.
sort target dict so that {'a' : 1, 'b': 2} and {'b': 2, 'a': 1} are not treated differently
make it as json
deduplicate via set (as set does not apply to dicts)
again, turn it into dict via json.loads
import json
[json.loads(i) for i in set([json.dumps(i) for i in [dict(sorted(i.items())) for i in target_dict]])]
There may be more elegant solutions, but I thought it might be nice to add a more verbose solution to make it easier to follow. This assumes there is not a unique key, you have a simple k,v structure, and that you are using a version of python that guarantees list order. This would work for the original post.
data_set = [
{'id': 1, 'name': 'john', 'age': 34},
{'id': 1, 'name': 'john', 'age': 34},
{'id': 2, 'name': 'hanna', 'age': 30},
]
# list of keys
keys = [k for k in data_set[0]]
# Create a List of Lists of the values from the data Set
data_set_list = [[v for v in v.values()] for v in data_set]
# Dedupe
new_data_set = []
for lst in data_set_list:
# Check if list exists in new data set
if lst in new_data_set:
print(lst)
continue
# Add list to new data set
new_data_set.append(lst)
# Create dicts
new_data_set = [dict(zip(keys,lst)) for lst in new_data_set]
print(new_data_set)
Pretty straightforward option:
L = [
{'id':1,'name':'john', 'age':34},
{'id':1,'name':'john', 'age':34},
{'id':2,'name':'hanna', 'age':30},
]
D = dict()
for l in L: D[l['id']] = l
output = list(D.values())
print output
Heres an implementation with little memory overhead at the cost of not being as compact as the rest.
values = [ {'id':2,'name':'hanna', 'age':30},
{'id':1,'name':'john', 'age':34},
{'id':1,'name':'john', 'age':34},
{'id':2,'name':'hanna', 'age':30},
{'id':1,'name':'john', 'age':34},]
count = {}
index = 0
while index < len(values):
if values[index]['id'] in count:
del values[index]
else:
count[values[index]['id']] = 1
index += 1
output:
[{'age': 30, 'id': 2, 'name': 'hanna'}, {'age': 34, 'id': 1, 'name': 'john'}]
This is the solution I found:
usedID = []
x = [
{'id':1,'name':'john', 'age':34},
{'id':1,'name':'john', 'age':34},
{'id':2,'name':'hanna', 'age':30},
]
for each in x:
if each['id'] in usedID:
x.remove(each)
else:
usedID.append(each['id'])
print x
Basically you check if the ID is present in the list, if it is, delete the dictionary, if not, append the ID to the list
Related
I am trying to create a dictionary with keys from x and values from a list of tuples 'users'. I need output in two formats: dict of lists & list of dict.
#input
users = [(2, "jhon", "jhon#company.com"),
(3, "mike", "mike#company.com"),
(4, "sam", "sam#company.com")]
x = ("id", "name", "email")
#expected output:
dict_of_lists = {'id': [2, 3, 4], 'name': ['jhon', 'mike', 'sam'], 'email': ['jhon#company.com', 'mike#company.com', 'sam#company.com']}
list_of_dicts = [{'id': 2, 'name': 'jhon', 'email': 'jhon#company.com'}, {'id': 3, 'name': 'mike', 'email': 'mike#company.com'}, {'id': 4, 'name': 'sam', 'email': 'same#company.com'}]
My code to generate dict of lists:
Dict = {}
def Add(key, value):
Dict[key] = value
ListId=[]
ListName=[]
ListEmail=[]
for i in range(len(users)):
ListId.append(users[i][0])
ListName.append(users[i][1])
ListEmail.append(users[i][2])
Add(x[0],ListId)
Add(x[1],ListName)
Add(x[2],ListEmail)
print(Dict)
My code to generate list of dict:
res = []
for i in range(len(users)):
Dict = {x[0] : users[i][0] , x[1] : users[i][1] , x[2]: users[i][2]}
res.append(Dict)
print(res)
I am looking for any other efficient solution to this question. Also, I have hardcoded index value from tuple x & list of tuple 'users' to access tuple value. How can I make code dynamic such that when a new key is added to x & value to users, it gets added to my output?
I have checked the web to find an answer but couldn't find anything similar to my question.
This is my first StackOverflow question. In case you have any suggestions for my question to write or ask in a better way then do let me know.
For the first one:
dict_of_lists = dict(zip(x, map(list, zip(*users))))
# or if tuples are fine
# dict_of_lists = dict(zip(x, zip(*users)))
output:
{'id': [2, 3, 4],
'name': ['jhon', 'mike', 'sam'],
'email': ['jhon#company.com', 'mike#company.com', 'sam#company.com']}
For the second:
list_of_dicts = [dict(zip(x,l)) for l in users]
output:
[{'id': 2, 'name': 'jhon', 'email': 'jhon#company.com'},
{'id': 3, 'name': 'mike', 'email': 'mike#company.com'},
{'id': 4, 'name': 'sam', 'email': 'sam#company.com'}]
Use list comprehension and dict comprehension:
list_of_dicts = [dict(zip(x,u)) for u in users]
dict_of_lists = {k: [u[i] for u in users] for i, k in enumerate(x)}
I have a List like this:
oldL = [
{"id":1, "name":"aaa"},
{"id":2, "name":"bbb"},
{"id":3, "name":"ccc"},
]
and I want to convert to this:
newL = [
{"id":1, "name":"aaa", "age":16},
{"id":2, "name":"bbb", "age":16},
{"id":3, "name":"ccc", "age":16}
]
By using "for structure" like this:
li = []
for sd in oldL:
sd.update({"age":16})
li.append(sd)
print(li)
I can get the right result, But I want to use the Comprehensions, How to get it? Thanks a lot.
I had test the case:
newL=[ sd.update({"age": 16}) for sd in oldL]
but I got [None, None, None], because the return value of update() is None.
If I test the case like this:
detailL =[ (sd.update({"age": 16}), sd) for sd in schedule_detailL]
the result is :
[(None, {'id': 1, 'name': 'aaa', 'schedule_id': 1}), (None, {'id': 2, 'name': 'bbb', 'schedule_id': 1}), (None, {'id': 3, 'name': 'ccc', 'schedule_id': 1})]
In Python 3 you can use the ** dictionary unpacking operator to unpack the original dictionary and then create a new one with the addition of the age key:
oldL = [
{"id":1, "name":"aaa"},
{"id":2, "name":"bbb"},
{"id":3, "name":"ccc"},
]
newL = [ { **d, "age": 16 } for d in oldL ]
print(newL)
Output:
[
{'id': 1, 'name': 'aaa', 'age': 16},
{'id': 2, 'name': 'bbb', 'age': 16},
{'id': 3, 'name': 'ccc', 'age': 16}
]
Note that this creates new dictionaries; it does not update the old ones. To update the old ones requires the use of update in a loop e.g.
for d in oldL:
d.update({"age": 16})
print(oldL)
Output:
[
{'id': 1, 'name': 'aaa', 'age': 16},
{'id': 2, 'name': 'bbb', 'age': 16},
{'id': 3, 'name': 'ccc', 'age': 16}
]
The update method of a dict updates that dict in-place, which is the reason you get None as the return value.
Basically, your comprehension works, you just updated it in-place.
However, using comprehensions for operations like this is usually unadvised.
If you instead want to create an actual new list of dicts, rather than updating the old one, you could do this:
add_dct = {'age': 16}
newL = [{**sd, **add_dct} for sd in oldL]
I have an extension question to remove dictionary from list, except that I have a list of dictionary keys I would like to remove from the list.
So, I have something like:
a=[{'id': 1, 'name': 'paul'},{'id': 2, 'name': 'john'},{'id': 3, 'name': 'john2'},{'id': 4, 'name': 'johnasc'}]
now, I have a del_list, like so:
del_id_list=[2,4]
what is the efficient way (assuming list a is LARGE) to delete dictionaries with id from del_list from a?
one way, recreating the list using a list comprehension (and declaring del_id_list as a set for faster lookup):
a=[{'id': 1, 'name': 'paul'},{'id': 2, 'name': 'john'},{'id': 3, 'name': 'john2'},{'id': 4, 'name': 'johnasc'}]
del_id_list={2,4}
new_a = [d for d in a if d['id'] not in del_id_list]
result:
[{'id': 1, 'name': 'paul'}, {'id': 3, 'name': 'john2'}]
Get acquainted with filter:
result = filter(lambda x: x['id'] not in del_id_list,a)
EDIT:
Regarding the del_id_list itself, if it's long you may want to consider the complexity of the in statement. Maybe a set and even a dict (with arbitrary value) would be better. Check this.
EDIT 2:
As #Jean correctly points out, this is a iteration sequence in Py3. Just add list:
result = list(filter(lambda x: x['id'] not in del_id_list,a))
sourceList = [{'id': 1, 'name': 'paul'},{'id': 2, 'name': 'john'},{'id': 3, 'name': 'john2'},{'id': 4, 'name': 'johnasc'}]
del_id_list = [2,4]
for itemDict in sourceList:
if itemDict['id'] in del_id_list:
sourceList.remove(itemDict)
print(sourceList)
Result -> [{'name': 'paul', 'id': 1}, {'name': 'john2', 'id': 3}]
I have here a very tricky task here.I want to compare x number of lists in list of lists and that lists contain dictionaries.So i want to compare the dictionaries in these lists based on the 'name' key in the dictionaries if it match it should pass if not it should copy the whole dictionary to the lists that don't have it with editing the 'balance' key vlaue to '0'.
For example let's assume we have list of lists like this :
list_of_lists=[[{'name': u'Profit','balance': 10},{'name': u'Income','balance': 30},{'name': u'NotIncome','balance': 15}],[{'name': u'Profit','balance': 20},{'name': u'Income','balance': 10}]]
So the result should be :
list_of_lists=[[{'name': u'Profit','balance': 10},{'name': u'Income','balance': 30},{'name': u'NotIncome','balance': 15}],[{'name': u'Profit','balance': 20},{'name': u'Income','balance': 10},{'name': u'NotIncome','balance': 0}]]
Here is my code but i can't get it work with 2 lists or more(I don't know the number of lists in the list (maybe 2,3 or 4 etc...) :
for line in lines:
for d1, d2 in zip(line[0], line[1]):
for key, value in d1.items():
if value != d2[key]:
print key, value, d2[key]
You could first create a set containing all the names and then iterate the sublists one by one adding the missing dicts:
import pprint
l = [
[
{'name': u'Profit','balance': 10},
{'name': u'Income','balance': 30},
{'name': u'NotIncome','balance': 15}
],
[
{'name': u'Profit','balance': 20},
{'name': u'Income','balance': 10}
],
[]
]
all_names = {d['name'] for x in l for d in x}
for sub_list in l:
for name in (all_names - {d['name'] for d in sub_list}):
sub_list.append({'name': name, 'balance': 0})
pprint.pprint(l)
Output:
[[{'balance': 10, 'name': u'Profit'},
{'balance': 30, 'name': u'Income'},
{'balance': 15, 'name': u'NotIncome'}],
[{'balance': 20, 'name': u'Profit'},
{'balance': 10, 'name': u'Income'},
{'balance': 0, 'name': u'NotIncome'}],
[{'balance': 0, 'name': u'Profit'},
{'balance': 0, 'name': u'Income'},
{'balance': 0, 'name': u'NotIncome'}]]
That said you should consider converting sublists to dicts where keys are names and values are balances in order to ease the processing.
How would I generate a list of values of a certain field of objects in a list?
Given the list of objects:
[ {name: "Joe", group: 1}, {name: "Kirk", group: 2}, {name: "Bob", group: 1}]
I want to generate list of the name field values:
["Joe", "Kirk", "Bob"]
The built-in filter() function seems to come close, but it will return the entire objects themselves.
I'd like a clean, one line solution such as:
filterLikeFunc(function(obj){return obj.name}, mylist)
Sorry, I know that's c syntax.
Just replace filter built-in function with map built-in function.
And use get function which will not give you key error in the absence of that particular key to get value for name key.
data = [{'name': "Joe", 'group': 1}, {'name': "Kirk", 'group': 2}, {'name': "Bob", 'group': 1}]
print map(lambda x: x.get('name'), data)
In Python 3.x
print(list(map(lambda x: x.get('name'), data)))
Results:
['Joe', 'Kirk', 'Bob']
Using List Comprehension:
print [each.get('name') for each in data]
Using a list comprehension approach you get:
objects = [{'group': 1, 'name': 'Joe'}, {'group': 2, 'name': 'Kirk'}, {'group': 1, 'name': 'Bob'}]
names = [i["name"] for i in objects]
For a good intro to list comprehensions, see https://docs.python.org/2/tutorial/datastructures.html
Just iterate over your list of dicts and pick out the name value and put them in a list.
x = [ {'name': "Joe", 'group': 1}, {'name': "Kirk", 'group': 2}, {'name': "Bob", 'group': 1}]
y = [y['name'] for y in x]
print(y)