I have sets of values that I want to apply as parameters to a function:
params = {
'a': [1, 2, 3],
'b': [5, 6, 7],
'x': [None, 'eleven', 'f'],
# et cetera
}
I want to run myfunc() with all possible combinations, so myfunc(a=1, b=5, x=None ...), myfunc(a=2, b=5, x=None ...) ... myfunc(a=3, b=7, x='f' ...). Is there something (for example in itertools) that can help? I thought about using itertools.product() but that doesn't keep the names of the parameters and just gives me tuples of the combinations.
You can use itertools.product to get all combinations of arguments:
>>> import itertools
>>> for xs in itertools.product([1,2], [5,6], ['eleven', 'f']):
... print(xs)
...
(1, 5, 'eleven')
(1, 5, 'f')
(1, 6, 'eleven')
(1, 6, 'f')
(2, 5, 'eleven')
(2, 5, 'f')
(2, 6, 'eleven')
(2, 6, 'f')
With Argument list unpacking, you can call myfunc with all combinations of keyword arguments:
params = {
'a': [1, 2, 3],
'b': [5, 6, 7],
'x': [None, 'eleven', 'f'],
}
def myfunc(**args):
print(args)
import itertools
keys = list(params)
for values in itertools.product(*map(params.get, keys)):
myfunc(**dict(zip(keys, values)))
output:
{'a': 1, 'x': None, 'b': 5}
{'a': 1, 'x': None, 'b': 6}
{'a': 1, 'x': None, 'b': 7}
{'a': 1, 'x': 'eleven', 'b': 5}
{'a': 1, 'x': 'eleven', 'b': 6}
{'a': 1, 'x': 'eleven', 'b': 7}
{'a': 1, 'x': 'f', 'b': 5}
...
Ordering of .keys and .values are guaranteed across all Python versions (unless dict is altered which does not happen here), so this might be a bit trivial:
from itertools import product
for vals in product(*params.values()):
myfunc(**dict(zip(params, vals)))
You can find the gurantee in the docs:
If keys, values and items views are iterated over with no intervening
modifications to the dictionary, the order of items will directly
correspond.
Demo:
for vals in product(*params.values()):
print(dict(zip(params, vals)))
{'a': 1, 'x': None, 'b': 5}
{'a': 1, 'x': None, 'b': 6}
{'a': 1, 'x': None, 'b': 7}
{'a': 1, 'x': 'eleven', 'b': 5}
{'a': 1, 'x': 'eleven', 'b': 6}
{'a': 1, 'x': 'eleven', 'b': 7}
{'a': 1, 'x': 'f', 'b': 5}
{'a': 1, 'x': 'f', 'b': 6}
{'a': 1, 'x': 'f', 'b': 7}
...
I developed combu which is that solution.
Install combu
pip install combu
Use combu
# Case 1: Directly call
import combu
for res, param in combu.execute(myfunc, params):
print(res, params)
# Case 2: Use class
from combu import Combu
comb = Combu(myfunc)
for res, param in comb.execute(params):
print(res, params)
Related
I have a dict looks like this :
my_dict = {
"a":[1, 2, 3],
"b":[10],
"c":[4, 5],
"d":[11]
}
And I would like to obtain a list containig all combinations keeping keys and value like this:
result = [
{"a":1, "b":10, "c":4, "d":11},
{"a":1, "b":10, "c":5, "d":11},
{"a":2, "b":10, "c":4, "d":11},
{"a":2, "b":10, "c":5, "d":11},
{"a":3, "b":10, "c":4, "d":11},
{"a":3, "b":10, "c":5, "d":11}
]
Do someone have a solution for this ?
Is there any existing solution to do this, or how should I proceed to do it myself ?
Thank you.
A task for itertools.product:
>>> from itertools import product
>>> for dict_items in product(*[product([k],v) for k, v in my_dict.items()]):
... print(dict(dict_items))
{'a': 1, 'b': 10, 'c': 4, 'd': 11}
{'a': 1, 'b': 10, 'c': 5, 'd': 11}
{'a': 2, 'b': 10, 'c': 4, 'd': 11}
{'a': 2, 'b': 10, 'c': 5, 'd': 11}
{'a': 3, 'b': 10, 'c': 4, 'd': 11}
{'a': 3, 'b': 10, 'c': 5, 'd': 11}
Small explanation:
The inner product(...) will expand the dict to a list such as [[(k1, v11), (k1, v12), ...], [(k2, v21), (k2, v22), ...], ...].
The outer product(...) will reassemble the items lists by choosing one tuple from each list.
dict(...) will create a dictionary from a sequence of (k1, v#), (k2, v#), ... tuples.
Try:
def permute(d):
k = d.keys()
perms = itertools.product(*d.values())
return [dict(zip(k, v)) for v in perms]
Example usage:
>>> d = {'a': [1, 2, 3], 'b': [10], 'c': [4, 5], 'd': [11]}
>>> pprint(permute(d))
[{'a': 1, 'b': 10, 'c': 4, 'd': 11},
{'a': 1, 'b': 10, 'c': 5, 'd': 11},
{'a': 2, 'b': 10, 'c': 4, 'd': 11},
{'a': 2, 'b': 10, 'c': 5, 'd': 11},
{'a': 3, 'b': 10, 'c': 4, 'd': 11},
{'a': 3, 'b': 10, 'c': 5, 'd': 11}]
Assuming that you are only interested in my_dict having 4 keys, it is simple enough to use nested for loops:
my_dict = {
"a": [1, 2, 3],
"b": [10],
"c": [4, 5],
"d": [11]
}
result = []
for a_val in my_dict['a']:
for b_val in my_dict['b']:
for c_val in my_dict['c']:
for d_val in my_dict['d']:
result.append({'a': a_val, 'b': b_val, 'c': c_val, 'd': d_val})
print(result)
This gives the expected result.
You can use:
from itertools import product
allNames = sorted(my_dict)
values= list(product(*(my_dict[Name] for Name in allNames)))
d = list(dict(zip(['a','b','c','d'],i)) for i in values)
Output:
[{'a': 1, 'c': 4, 'b': 10, 'd': 11},
{'a': 1, 'c': 5, 'b': 10, 'd': 11},
{'a': 2, 'c': 4, 'b': 10, 'd': 11},
{'a': 2, 'c': 5, 'b': 10, 'd': 11},
{'a': 3, 'c': 4, 'b': 10, 'd': 11},
{'a': 3, 'c': 5, 'b': 10, 'd': 11}]
itertools.product produces the combinations of a list of iterators.
dict.values() gets the list needed.
For each combination, zip up the dict.keys() with the combination.
Use a list comprehension to collect them up:
from itertools import product
from pprint import pprint
my_dict = {
"a":[1, 2, 3],
"b":[10],
"c":[4, 5],
"d":[11]
}
result = [dict(zip(my_dict,i)) for i in product(*my_dict.values())]
pprint(result)
Output:
[{'a': 1, 'b': 10, 'c': 4, 'd': 11},
{'a': 1, 'b': 10, 'c': 5, 'd': 11},
{'a': 2, 'b': 10, 'c': 4, 'd': 11},
{'a': 2, 'b': 10, 'c': 5, 'd': 11},
{'a': 3, 'b': 10, 'c': 4, 'd': 11},
{'a': 3, 'b': 10, 'c': 5, 'd': 11}]
here is my list of dict:
l = [{'a': 2, 'c': 1, 'b': 3},
{'a': 2, 'c': 3, 'b': 1},
{'a': 1, 'c': 2, 'b': 3},
{'a': 1, 'c': 3, 'b': 2},
{'a': 2, 'c': 5, 'b': 3}]
and now I want to sort the list by keys and orders provided by the user. for instance:
keys = ['a', 'c', 'b']
orders = [1, -1, 1]
I tried to using lambda in sort()method but it failed in a weird way :
>>> l.sort(key=lambda x: (order * x[key] for (key, order) in zip(keys, orders)))
>>> l
[{'a': 2, 'c': 5, 'b': 3},
{'a': 1, 'c': 3, 'b': 2},
{'a': 1, 'c': 2, 'b': 3},
{'a': 2, 'c': 3, 'b': 1},
{'a': 2, 'c': 1, 'b': 3}]
Anyone know how to solve this?
You were almost there; your lambda produces generator expressions and those happen to be ordered by their memory address (in Python 2) and produce a TypeError: '<' not supported between instances of 'generator' and 'generator' exception in Python 3.
Use a list comprehension instead:
l.sort(key=lambda x: [order * x[key] for (key, order) in zip(keys, orders)])
Demo:
>>> l = [{'a': 1, 'c': 2, 'b': 3},
... {'a': 1, 'c': 3, 'b': 2},
... {'a': 2, 'c': 1, 'b': 3},
... {'a': 2, 'c': 5, 'b': 3},
... {'a': 2, 'c': 3, 'b': 1}]
>>> keys = ['a', 'c', 'b']
>>> orders = [1, -1, 1]
>>> l.sort(key=lambda x: [order * x[key] for (key, order) in zip(keys, orders)])
>>> from pprint import pprint
>>> pprint(l)
[{'a': 1, 'b': 2, 'c': 3},
{'a': 1, 'b': 3, 'c': 2},
{'a': 2, 'b': 3, 'c': 5},
{'a': 2, 'b': 1, 'c': 3},
{'a': 2, 'b': 3, 'c': 1}]
This may be a duplicate but the closest I could find was Comparing 2 lists consisting of dictionaries with unique keys in python which did not work for me.
So I have two lists of dictionaries.
y = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
How do I compare these two lists so my compare results in the intersection of the two lists. I can't convert it to set since it says unhashable type (dict)
Your question and it's title seem at odds with each other.
The intersection of the 2 lists would be the common elements of both list. The question title requests the elements that are not in both lists. Which is it that you want?
For the intersection, it is not very efficient (being O(n^2) in time), but this list comprehension will do it:
>>> a = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
>>> b = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
>>> [d for d in a if d in b]
[{'a': 1, 'b': 2, 'c': 3}]
y1 = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y2 = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
print [x for x in y1 if x in y2] # prints [{'a': 1, 'c': 3, 'b': 2}]
A dict (or list) is not hashable, however, a tuple is. You can convert the list of dicts to a set of tuples. Perform the intersection and then convert back
the code to convert to a set-of-tuples
y_tupleset = set(tuple(sorted(d.items())) for d in y)
the code to convert back the intersected set-of-tuples to a list-of-dicts
y_dictlist = [dict(it) for it in list(y_tupleset)]
Thus, the full code would be:
y0 = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y1 = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
y0_tupleset = set(tuple(sorted(d.items())) for d in y0)
y1_tupleset = set(tuple(sorted(d.items())) for d in y1)
y_inter = y0_tupleset.intersection(y1_tupleset)
y_inter_dictlist = [dict(it) for it in list(y_inter)]
print(y_inter_dictlist)
# prints the following line
[{'a': 1, 'c': 3, 'b': 2}]
edit: d.items() is valid on python3, for python2, it should be replaced with d.iteritems()
Pick your poison:
y1 = [{'a': 3, 'b': 4, 'c': 5}, {'a': 1, 'b': 2, 'c': 3}]
y2 = [{'a': 4, 'b': 5, 'c': 6}, {'a': 1, 'b': 2, 'c': 3}]
y3 = [{'a': 1, 'b': 2, 'c': 3}, {'a': 4, 'b': 2, 'c': 6}]
# Returns a list of keys that are in both dictionaries
def intersect_keys(d1, d2):
return [k for k in d1 if k in d2]
# Returns a list of values that are in both dictionaries
def intersect_vals(d1, d2):
return [v for v in d1.itervalues() if v in d2.itervalues()]
# Returns a list of (key,value) pairs that are in both dictionaries
def intersect_pairs(d1, d2):
return [(k,v) for (k,v) in d1.iteritems() if k in d2 and d2[k] == v]
print(intersect_keys(*y1)) # ['a', 'c', 'b']
print(intersect_vals(*y1)) # [3]
print(intersect_pairs(*y1)) # []
print(intersect_keys(*y2)) # ['a', 'c', 'b']
print(intersect_vals(*y2)) # []
print(intersect_pairs(*y2)) # []
print(intersect_keys(*y3)) # ['a', 'c', 'b']
print(intersect_vals(*y3)) # [2]
print(intersect_pairs(*y3)) # [('b', 2)]
Note: the examples compare the two elements of the y* list, which was how I interpreted your question. You could of course use something like:
print(intersect_pairs(y1[0], y2[0]))
To compute the intersection the first dictionary in the y1 and y2 lists.
This question already has an answer here:
Group list of dictionaries to list of list of dictionaries with same property value
(1 answer)
Closed 8 years ago.
Consider a list of dicts:
items = [
{'a': 1, 'b': 9, 'c': 8},
{'a': 1, 'b': 5, 'c': 4},
{'a': 2, 'b': 3, 'c': 1},
{'a': 2, 'b': 7, 'c': 9},
{'a': 3, 'b': 8, 'c': 2}
]
Is there a pythonic way to extract and group these items by their a field, such that:
result = {
1 : [{'b': 9, 'c': 8}, {'b': 5, 'c': 4}]
2 : [{'b': 3, 'c': 1}, {'b': 7, 'c': 9}]
3 : [{'b': 8, 'c': 2}]
}
References to any similar Pythonic constructs are appreciated.
Use itertools.groupby:
>>> from itertools import groupby
>>> from operator import itemgetter
>>> {k: list(g) for k, g in groupby(items, itemgetter('a'))}
{1: [{'a': 1, 'c': 8, 'b': 9},
{'a': 1, 'c': 4, 'b': 5}],
2: [{'a': 2, 'c': 1, 'b': 3},
{'a': 2, 'c': 9, 'b': 7}],
3: [{'a': 3, 'c': 2, 'b': 8}]}
If item are not in sorted order then you can either sort them and then use groupby or you can use collections.OrderedDict(if order matters) or collections.defaultdict to do it in O(N) time:
>>> from collections import OrderedDict
>>> d = OrderedDict()
>>> for item in items:
... d.setdefault(item['a'], []).append(item)
...
>>> dict(d.items())
{1: [{'a': 1, 'c': 8, 'b': 9},
{'a': 1, 'c': 4, 'b': 5}],
2: [{'a': 2, 'c': 1, 'b': 3},
{'a': 2, 'c': 9, 'b': 7}],
3: [{'a': 3, 'c': 2, 'b': 8}]}
Update:
I see that you only want the those keys to be returned that we didn't use for grouping, for that you'll need to do something like this:
>>> group_keys = {'a'}
>>> {k:[{k:d[k] for k in d.viewkeys() - group_keys} for d in g]
for k, g in groupby(items, itemgetter(*group_keys))}
{1: [{'c': 8, 'b': 9},
{'c': 4, 'b': 5}],
2: [{'c': 1, 'b': 3},
{'c': 9, 'b': 7}],
3: [{'c': 2, 'b': 8}]}
Note: This code assumes the the data is already sorted. If it is not, we have to sort it manually
from itertools import groupby
print {key:list(grp) for key, grp in groupby(items, key=lambda x:x["a"])}
Output
{1: [{'a': 1, 'b': 9, 'c': 8}, {'a': 1, 'b': 5, 'c': 4}],
2: [{'a': 2, 'b': 3, 'c': 1}, {'a': 2, 'b': 7, 'c': 9}],
3: [{'a': 3, 'b': 8, 'c': 2}]}
To get the result in the same format you asked for,
from itertools import groupby
from operator import itemgetter
a_getter, getter, keys = itemgetter("a"), itemgetter("b", "c"), ("b", "c")
def recon_dicts(items):
return dict(zip(keys, getter(items)))
{key: map(recon_dicts, grp) for key, grp in groupby(items, key=a_getter)}
Output
{1: [{'c': 8, 'b': 9}, {'c': 4, 'b': 5}],
2: [{'c': 1, 'b': 3}, {'c': 9, 'b': 7}],
3: [{'c': 2, 'b': 8}]}
If the data is not sorted already, you can either use the defaultdict method in this answer, or you can use sorted function to sort based on a, like this
{key: map(recon_dicts, grp)
for key, grp in groupby(sorted(items, key=a_getter), key=a_getter)}
References:
operator.itemgetter
itertools.groupby
zip, map, dict, sorted
I wanted to create a dictionary of dictionaries in Python:
Suppose I already have a list which contains the keys:
keys = ['a', 'b', 'c', 'd', 'e']
value = [1, 2, 3, 4, 5]
Suppose I have a data field with numeric values (20 of them)
I want to define a dictionary which stores 4 different dictionaries with the given to a corresponding value
for i in range(0, 3)
for j in range(0, 4)
dictionary[i] = { 'keys[j]' : value[j] }
So basically, it should be like:
dictionary[0] = {'a' : 1, 'b' : 2, 'c' : 3, 'd': 4, 'e':5}
dictionary[1] = {'a' : 1, 'b' : 2, 'c' : 3, 'd': 4, 'e':5}
dictionary[2] = {'a' : 1, 'b' : 2, 'c' : 3, 'd': 4, 'e':5}
dictionary[3] = {'a' : 1, 'b' : 2, 'c' : 3, 'd': 4, 'e':5}
What is the best way to achieve this?
Use a list comprehension and dict(zip(keys,value)) will return the dict for you.
>>> keys = ['a', 'b', 'c', 'd', 'e']
>>> value = [1, 2, 3, 4, 5]
>>> dictionary = [dict(zip(keys,value)) for _ in xrange(4)]
>>> from pprint import pprint
>>> pprint(dictionary)
[{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}]
If you want a dict of dicts then use a dict comprehension:
>>> keys = ['a', 'b', 'c', 'd', 'e']
>>> value = [1, 2, 3, 4, 5]
>>> dictionary = {i: dict(zip(keys,value)) for i in xrange(4)}
>>> pprint(dictionary)
{0: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
1: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
2: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5},
3: {'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5}}
An alternative that only zips once...:
from itertools import repeat
map(dict, repeat(zip(keys,values), 4))
Or, maybe, just use dict.copyand construct the dict once:
[d.copy() for d in repeat(dict(zip(keys, values)), 4)]
for a list of dictionaries:
dictionary = [dict(zip(keys,value)) for i in xrange(4)]
If you really wanted a dictionary of dictionaries like you said:
dictionary = dict((i,dict(zip(keys,value))) for i in xrange(4))
I suppose you could use pop or other dict calls which you could not from a list
BTW: if this is really a data/number crunching application, I'd suggest moving on to numpy and/or pandas as great modules.
Edit re: OP comments,
if you want indicies for the type of data you are talking about:
# dict keys must be tuples and not lists
[(i,j) for i in xrange(4) for j in range(3)]
# same can come from itertools.product
from itertools import product
list(product(xrange4, xrange 3))