Related
Say, I have a string
s = 'AAABBBCAB'
How can I efficiently calculate the prefix sum of frequencies of each character in the string, i.e.:
psum = [{'A': 1}, {'A': 2}, {'A': 3}, {'A': 3, 'B': 1}, {'A': 3, 'B': 2}, {'A': 3, 'B': 3}, {'A': 3, 'B': 3, 'C': 1}, {'A': 4, 'B': 3, 'C': 1}, {'A': 4, 'B': 4, 'C': 1}]
You can do it in one line using itertools.accumulate and collections.Counter:
from collections import Counter
from itertools import accumulate
s = 'AAABBBCAB'
psum = list(accumulate(map(Counter, s)))
This gives you a list of Counter objects. Now, to get frequencies for any substring of s in O(1) time, you can simply subtract counters, e.g.:
>>> psum[6] - psum[1] # get frequencies for s[2:7]
Counter({'B': 3, 'A': 1, 'C': 1})
this is an option:
from collections import Counter
c = Counter()
s = 'AAABBBCAB'
psum = []
for char in s:
c.update(char)
psum.append(dict(c))
# [{'A': 1}, {'A': 2}, {'A': 3}, {'A': 3, 'B': 1}, {'A': 3, 'B': 2},
# {'A': 3, 'B': 3}, {'A': 3, 'B': 3, 'C': 1}, {'A': 4, 'B': 3, 'C': 1},
# {'A': 4, 'B': 4, 'C': 1}]
i use collections.Counter in order to keep a 'running sum' and add (a copy of the result) to the list psum. this way i iterate once only over the string s.
if you prefer to have collections.Counter objects in your result, you could change the last line to
psum.append(c.copy())
in order to get
[Counter({'A': 1}), Counter({'A': 2}), ...
Counter({'A': 4, 'B': 4, 'C': 1})]
the same result could also be achieved with this (using accumulate was first proposed in Eugene Yarmash's answer; i just avoid map in favour of a generator expression):
from itertools import accumulate
from collections import Counter
s = "AAABBBCAB"
psum = list(accumulate(Counter(char) for char in s))
just for completeness (as there is no 'pure dict' answer here yet). if you do not want to use Counter or defaultdict you could use this as well:
c = {}
s = 'AAABBBCAB'
psum = []
for char in s:
c[char] = c.get(char, 0) + 1
psum.append(c.copy())
although defaultdict is usually more performant than dict.get(key, default).
You actually don't even need a counter for this, just a defaultdict would suffice!
from collections import defaultdict
c = defaultdict(int)
s = 'AAABBBCAB'
psum = []
#iterate through the character
for char in s:
#Update count for each character
c[char] +=1
#Add the updated dictionary to the output list
psum.append(dict(c))
print(psum)
The output looks like
[{'A': 1}, {'A': 2}, {'A': 3}, {'A': 3, 'B': 1},
{'A': 3, 'B': 2}, {'A': 3, 'B': 3},
{'A': 3, 'B': 3, 'C': 1}, {'A': 4, 'B': 3, 'C': 1},
{'A': 4, 'B': 4, 'C': 1}]
Simplest would be to use the Counter object from collections.
from collections import Counter
s = 'AAABBBCAB'
[ dict(Counter(s[:i]) for i in range(1,len(s))]
Yields:
[{'A': 1}, {'A': 2}, {'A': 3}, {'A': 3, 'B': 1}, {'A': 3, 'B': 2},
{'A': 3, 'B': 3}, {'A': 3, 'B': 3, 'C': 1}, {'A': 4, 'B': 3, 'C': 1}]
In Python 3.8 you can use a list comprehension with an assignment expression (aka "the walrus operator"):
>>> from collections import Counter
>>> s = 'AAABBBCAB'
>>> c = Counter()
>>> [c := c + Counter(x) for x in s]
[Counter({'A': 1}), Counter({'A': 2}), Counter({'A': 3}), Counter({'A': 3, 'B': 1}), Counter({'A': 3, 'B': 2}), Counter({'A': 3, 'B': 3}), Counter({'A': 3, 'B': 3, 'C': 1}), Counter({'A': 4, 'B': 3, 'C': 1}), Counter({'A': 4, 'B': 4, 'C': 1})]
here is my list of dict:
l = [{'a': 2, 'c': 1, 'b': 3},
{'a': 2, 'c': 3, 'b': 1},
{'a': 1, 'c': 2, 'b': 3},
{'a': 1, 'c': 3, 'b': 2},
{'a': 2, 'c': 5, 'b': 3}]
and now I want to sort the list by keys and orders provided by the user. for instance:
keys = ['a', 'c', 'b']
orders = [1, -1, 1]
I tried to using lambda in sort()method but it failed in a weird way :
>>> l.sort(key=lambda x: (order * x[key] for (key, order) in zip(keys, orders)))
>>> l
[{'a': 2, 'c': 5, 'b': 3},
{'a': 1, 'c': 3, 'b': 2},
{'a': 1, 'c': 2, 'b': 3},
{'a': 2, 'c': 3, 'b': 1},
{'a': 2, 'c': 1, 'b': 3}]
Anyone know how to solve this?
You were almost there; your lambda produces generator expressions and those happen to be ordered by their memory address (in Python 2) and produce a TypeError: '<' not supported between instances of 'generator' and 'generator' exception in Python 3.
Use a list comprehension instead:
l.sort(key=lambda x: [order * x[key] for (key, order) in zip(keys, orders)])
Demo:
>>> l = [{'a': 1, 'c': 2, 'b': 3},
... {'a': 1, 'c': 3, 'b': 2},
... {'a': 2, 'c': 1, 'b': 3},
... {'a': 2, 'c': 5, 'b': 3},
... {'a': 2, 'c': 3, 'b': 1}]
>>> keys = ['a', 'c', 'b']
>>> orders = [1, -1, 1]
>>> l.sort(key=lambda x: [order * x[key] for (key, order) in zip(keys, orders)])
>>> from pprint import pprint
>>> pprint(l)
[{'a': 1, 'b': 2, 'c': 3},
{'a': 1, 'b': 3, 'c': 2},
{'a': 2, 'b': 3, 'c': 5},
{'a': 2, 'b': 1, 'c': 3},
{'a': 2, 'b': 3, 'c': 1}]
This may be a duplicate but the closest I could find was Comparing 2 lists consisting of dictionaries with unique keys in python which did not work for me.
So I have two lists of dictionaries.
y = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
How do I compare these two lists so my compare results in the intersection of the two lists. I can't convert it to set since it says unhashable type (dict)
Your question and it's title seem at odds with each other.
The intersection of the 2 lists would be the common elements of both list. The question title requests the elements that are not in both lists. Which is it that you want?
For the intersection, it is not very efficient (being O(n^2) in time), but this list comprehension will do it:
>>> a = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
>>> b = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
>>> [d for d in a if d in b]
[{'a': 1, 'b': 2, 'c': 3}]
y1 = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y2 = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
print [x for x in y1 if x in y2] # prints [{'a': 1, 'c': 3, 'b': 2}]
A dict (or list) is not hashable, however, a tuple is. You can convert the list of dicts to a set of tuples. Perform the intersection and then convert back
the code to convert to a set-of-tuples
y_tupleset = set(tuple(sorted(d.items())) for d in y)
the code to convert back the intersected set-of-tuples to a list-of-dicts
y_dictlist = [dict(it) for it in list(y_tupleset)]
Thus, the full code would be:
y0 = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y1 = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
y0_tupleset = set(tuple(sorted(d.items())) for d in y0)
y1_tupleset = set(tuple(sorted(d.items())) for d in y1)
y_inter = y0_tupleset.intersection(y1_tupleset)
y_inter_dictlist = [dict(it) for it in list(y_inter)]
print(y_inter_dictlist)
# prints the following line
[{'a': 1, 'c': 3, 'b': 2}]
edit: d.items() is valid on python3, for python2, it should be replaced with d.iteritems()
Pick your poison:
y1 = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y2 = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
y3 = [{'a': 1, 'b': 2, 'c': 3}, {'a': 4, 'b': 2, 'c': 6}]
# Returns a list of keys that are in both dictionaries
def intersect_keys(d1, d2):
return [k for k in d1 if k in d2]
# Returns a list of values that are in both dictionaries
def intersect_vals(d1, d2):
return [v for v in d1.itervalues() if v in d2.itervalues()]
# Returns a list of (key,value) pairs that are in both dictionaries
def intersect_pairs(d1, d2):
return [(k,v) for (k,v) in d1.iteritems() if k in d2 and d2[k] == v]
print(intersect_keys(*y1)) # ['a', 'c', 'b']
print(intersect_vals(*y1)) # [3]
print(intersect_pairs(*y1)) # []
print(intersect_keys(*y2)) # ['a', 'c', 'b']
print(intersect_vals(*y2)) # []
print(intersect_pairs(*y2)) # []
print(intersect_keys(*y3)) # ['a', 'c', 'b']
print(intersect_vals(*y3)) # [2]
print(intersect_pairs(*y3)) # [('b', 2)]
Note: the examples compare the two elements of the y* list, which was how I interpreted your question. You could of course use something like:
print(intersect_pairs(y1[0], y2[0]))
To compute the intersection the first dictionary in the y1 and y2 lists.
I know that I can set a key-value pair by using
dict[key] = value
but I have a very long list of dicts of the type
dict = [{a:1, b:2, c:3, d:4},
{a:2, b:3, c:4, d:5},
{a:5, b:7, c:3, d:9}]
and I'd like to do something along the lines of
dict = map(lambda x: x['d'] <- x['d'] -1, dict)
how would I go about this? (This is a very simplified example so I'm not really trying to just subtract a number from all items by a particular key)
expected output would be in this case and not the general case I'm looking for
[{a:1, b:2, c:3, d:3},
{a:2, b:3, c:4, d:4},
{a:5, b:7, c:3, d:8}]
EDIT: 2
I believe the following does not work - so any similar solution would be helpful:
dict = map(lambda x: x.update(d, x[d] - 1), dict)
dicts = [{'a':1, 'b':2, 'c':3, 'd':4},
{'a':2, 'b':3, 'c':4, 'd':5},
{'a':5, 'b':7, 'c':3, 'd':9}]
for d in dicts:
d['d'] -= 1
Output:
In [94]: dicts
Out[94]:
[{'d': 3, 'b': 2, 'c': 3, 'a': 1},
{'d': 4, 'b': 3, 'c': 4, 'a': 2},
{'d': 8, 'b': 7, 'c': 3, 'a': 5}]
how about this: as exactly you said
>>> dicts = [{'a':1, 'b':2, 'c':3, 'd':4},
{'a':2, 'b':3, 'c':4, 'd':5},
{'a':5, 'b':7, 'c':3, 'd':9}]
>>> map(lambda x:x.update([('d',x['d']-1)]),dicts)
[None, None, None]
>>> dicts
[{'a': 1, 'c': 3, 'b': 2, 'd': 3}, {'a': 2, 'c': 4, 'b': 3, 'd': 4}, {'a': 5, 'c': 3, 'b': 7, 'd': 8}]
update will update the dictionary with (key,value) pair. Returns None
map is a way of transforming an iterable to a list by performing the same operation on every item from the iterable. I don't think that's what you want to do here, and it has confused you.
On the face of it (although you haven't mentioned what the real operation is that you want to perform) a simple for is all that is necessary:
dict_list = [
{'a': 1, 'b': 2, 'c': 3, 'd': 4},
{'a': 2, 'b': 3, 'c': 4, 'd': 5},
{'a': 5, 'b': 7, 'c': 3, 'd': 9},
]
for d in dict_list:
d['d'] -= 1
print(d)
output
{'a': 1, 'b': 2, 'c': 3, 'd': 3}
{'a': 2, 'b': 3, 'c': 4, 'd': 4}
{'a': 5, 'b': 7, 'c': 3, 'd': 8}
Using dict.__setitem__ and temporary list (or any other collection typer) trick:
>>> dicts = [{'a':1, 'b':2, 'c':3, 'd':4},
... {'a':2, 'b':3, 'c':4, 'd':5},
... {'a':5, 'b':7, 'c':3, 'd':9}]
>>> map(lambda d: [d.__setitem__('d', d['d'] - 1), d][1], dicts)
[{'a': 1, 'c': 3, 'b': 2, 'd': 3},
{'a': 2, 'c': 4, 'b': 3, 'd': 4},
{'a': 5, 'c': 3, 'b': 7, 'd': 8}]
Using simple for loop is moe recommended way. Especially there's a side effect in the function.
BTW, don't use dict as a variable name. It will shadows builtin function/type dict.
How about this:
my_dict = {k: f(v) for k, v in my_dict.iteritems()}
where f is whatever function you want.
This question already has an answer here:
Group list of dictionaries to list of list of dictionaries with same property value
(1 answer)
Closed 8 years ago.
Consider a list of dicts:
items = [
{'a': 1, 'b': 9, 'c': 8},
{'a': 1, 'b': 5, 'c': 4},
{'a': 2, 'b': 3, 'c': 1},
{'a': 2, 'b': 7, 'c': 9},
{'a': 3, 'b': 8, 'c': 2}
]
Is there a pythonic way to extract and group these items by their a field, such that:
result = {
1 : [{'b': 9, 'c': 8}, {'b': 5, 'c': 4}]
2 : [{'b': 3, 'c': 1}, {'b': 7, 'c': 9}]
3 : [{'b': 8, 'c': 2}]
}
References to any similar Pythonic constructs are appreciated.
Use itertools.groupby:
>>> from itertools import groupby
>>> from operator import itemgetter
>>> {k: list(g) for k, g in groupby(items, itemgetter('a'))}
{1: [{'a': 1, 'c': 8, 'b': 9},
{'a': 1, 'c': 4, 'b': 5}],
2: [{'a': 2, 'c': 1, 'b': 3},
{'a': 2, 'c': 9, 'b': 7}],
3: [{'a': 3, 'c': 2, 'b': 8}]}
If item are not in sorted order then you can either sort them and then use groupby or you can use collections.OrderedDict(if order matters) or collections.defaultdict to do it in O(N) time:
>>> from collections import OrderedDict
>>> d = OrderedDict()
>>> for item in items:
... d.setdefault(item['a'], []).append(item)
...
>>> dict(d.items())
{1: [{'a': 1, 'c': 8, 'b': 9},
{'a': 1, 'c': 4, 'b': 5}],
2: [{'a': 2, 'c': 1, 'b': 3},
{'a': 2, 'c': 9, 'b': 7}],
3: [{'a': 3, 'c': 2, 'b': 8}]}
Update:
I see that you only want the those keys to be returned that we didn't use for grouping, for that you'll need to do something like this:
>>> group_keys = {'a'}
>>> {k:[{k:d[k] for k in d.viewkeys() - group_keys} for d in g]
for k, g in groupby(items, itemgetter(*group_keys))}
{1: [{'c': 8, 'b': 9},
{'c': 4, 'b': 5}],
2: [{'c': 1, 'b': 3},
{'c': 9, 'b': 7}],
3: [{'c': 2, 'b': 8}]}
Note: This code assumes the the data is already sorted. If it is not, we have to sort it manually
from itertools import groupby
print {key:list(grp) for key, grp in groupby(items, key=lambda x:x["a"])}
Output
{1: [{'a': 1, 'b': 9, 'c': 8}, {'a': 1, 'b': 5, 'c': 4}],
2: [{'a': 2, 'b': 3, 'c': 1}, {'a': 2, 'b': 7, 'c': 9}],
3: [{'a': 3, 'b': 8, 'c': 2}]}
To get the result in the same format you asked for,
from itertools import groupby
from operator import itemgetter
a_getter, getter, keys = itemgetter("a"), itemgetter("b", "c"), ("b", "c")
def recon_dicts(items):
return dict(zip(keys, getter(items)))
{key: map(recon_dicts, grp) for key, grp in groupby(items, key=a_getter)}
Output
{1: [{'c': 8, 'b': 9}, {'c': 4, 'b': 5}],
2: [{'c': 1, 'b': 3}, {'c': 9, 'b': 7}],
3: [{'c': 2, 'b': 8}]}
If the data is not sorted already, you can either use the defaultdict method in this answer, or you can use sorted function to sort based on a, like this
{key: map(recon_dicts, grp)
for key, grp in groupby(sorted(items, key=a_getter), key=a_getter)}
References:
operator.itemgetter
itertools.groupby
zip, map, dict, sorted