I have a dictionary:
>>> print(dict)
{'gs': ['bags', 'begs', 'bogs'],
'le': ['Cole', 'Dale', 'Dole'],
'll': ['Ball', 'Bell', 'Bill']}
For every single key I want to pick only one word (randomly) from its list.
The output would be like:
{'gs': begs, 'le': 'Cole', 'll': 'Bill'}
and so on.
I have tried loads of things but none has given me a word for every key of the dictionary. Is there a simple way of doing that?
just use random.choice on the values of the dictionary, rebuilding a dict comprehension with only 1 name as value
import random
d = {'gs': ['bags', 'begs', 'bogs'],
'le': ['Cole', 'Dale', 'Dole'],
'll': ['Ball', 'Bell', 'Bill']}
result = {k:random.choice(v) for k,v in d.items()}
one output:
{'gs': 'bogs', 'le': 'Dale', 'll': 'Bell'}
I think one easy way would be to pick a random number between 0 and the length of each list and then picking the item that corresponds to that index! Just iterate through the dictionary's keys (using list(yourdictionary)), get each list, find its length, choose the random number and finally get the element.
I have some data stored in a list that I would like to group based on a value.
For example, if my data is
data = [(1, 'a'), (2, 'x'), (1, 'b')]
and I want to group it by the first value in each tuple to get
result = [(1, 'ab'), (2, 'x')]
how would I go about it?
More generally, what's the recommended way to group data in python? Is there a recipe that can help me?
The go-to data structure to use for all kinds of grouping is the dict. The idea is to use something that uniquely identifies a group as the dict's keys, and store all values that belong to the same group under the same key.
As an example, your data could be stored in a dict like this:
{1: ['a', 'b'],
2: ['x']}
The integer that you're using to group the values is used as the dict key, and the values are aggregated in a list.
The reason why we're using a dict is because it can map keys to values in constant O(1) time. This makes the grouping process very efficient and also very easy. The general structure of the code will always be the same for all kinds of grouping tasks: You iterate over your data and gradually fill a dict with grouped values. Using a defaultdict instead of a regular dict makes the whole process even easier, because we don't have to worry about initializing the dict with empty lists.
import collections
groupdict = collections.defaultdict(list)
for value in data:
group = value[0]
value = value[1]
groupdict[group].append(value)
# result:
# {1: ['a', 'b'],
# 2: ['x']}
Once the data is grouped, all that's left is to convert the dict to your desired output format:
result = [(key, ''.join(values)) for key, values in groupdict.items()]
# result: [(1, 'ab'), (2, 'x')]
The Grouping Recipe
The following section will provide recipes for different kinds of inputs and outputs, and show how to group by various things. The basis for everything is the following snippet:
import collections
groupdict = collections.defaultdict(list)
for value in data: # input
group = ??? # group identifier
value = ??? # value to add to the group
groupdict[group].append(value)
result = groupdict # output
Each of the commented lines can/has to be customized depending on your use case.
Input
The format of your input data dictates how you iterate over it.
In this section, we're customizing the for value in data: line of the recipe.
A list of values
More often than not, all the values are stored in a flat list:
data = [value1, value2, value3, ...]
In this case we simply iterate over the list with a for loop:
for value in data:
Multiple lists
If you have multiple lists with each list holding the value of a different attribute like
firstnames = [firstname1, firstname2, ...]
middlenames = [middlename1, middlename2, ...]
lastnames = [lastname1, lastname2, ...]
use the zip function to iterate over all lists simultaneously:
for value in zip(firstnames, middlenames, lastnames):
This will make value a tuple of (firstname, middlename, lastname).
Multiple dicts or a list of dicts
If you want to combine multiple dicts like
dict1 = {'a': 1, 'b': 2}
dict2 = {'b': 5}
First put them all in a list:
dicts = [dict1, dict2]
And then use two nested loops to iterate over all (key, value) pairs:
for dict_ in dicts:
for value in dict_.items():
In this case, the value variable will take the form of a 2-element tuple like ('a', 1) or ('b', 2).
Grouping
Here we'll cover various ways to extract group identifiers from your data.
In this section, we're customizing the group = ??? line of the recipe.
Grouping by a list/tuple/dict element
If your values are lists or tuples like (attr1, attr2, attr3, ...) and you want to group them by the nth element:
group = value[n]
The syntax is the same for dicts, so if you have values like {'firstname': 'foo', 'lastname': 'bar'} and you want to group by the first name:
group = value['firstname']
Grouping by an attribute
If your values are objects like datetime.date(2018, 5, 27) and you want to group them by an attribute, like year:
group = value.year
Grouping by a key function
Sometimes you have a function that returns a value's group when it's called. For example, you could use the len function to group values by their length:
group = len(value)
Grouping by multiple values
If you wish to group your data by more than a single value, you can use a tuple as the group identifier. For example, to group strings by their first letter and their length:
group = (value[0], len(value))
Grouping by something unhashable
Because dict keys must be hashable, you will run into problems if you try to group by something that can't be hashed. In such a case, you have to find a way to convert the unhashable value to a hashable representation.
sets: Convert sets to frozensets, which are hashable:
group = frozenset(group)
dicts: Dicts can be represented as sorted (key, value) tuples:
group = tuple(sorted(group.items()))
Modifying the aggregated values
Sometimes you will want to modify the values you're grouping. For example, if you're grouping tuples like (1, 'a') and (1, 'b') by the first element, you might want to remove the first element from each tuple to get a result like {1: ['a', 'b']} rather than {1: [(1, 'a'), (1, 'b')]}.
In this section, we're customizing the value = ??? line of the recipe.
No change
If you don't want to change the value in any way, simple delete the value = ??? line from your code.
Keeping only a single list/tuple/dict element
If your values are lists like [1, 'a'] and you only want to keep the 'a':
value = value[1]
Or if they're dicts like {'firstname': 'foo', 'lastname': 'bar'} and you only want to keep the first name:
value = value['firstname']
Removing the first list/tuple element
If your values are lists like [1, 'a', 'foo'] and [1, 'b', 'bar'] and you want to discard the first element of each tuple to get a group like [['a', 'foo], ['b', 'bar']], use the slicing syntax:
value = value[1:]
Removing/Keeping arbitrary list/tuple/dict elements
If your values are lists like ['foo', 'bar', 'baz'] or dicts like {'firstname': 'foo', 'middlename': 'bar', 'lastname': 'baz'} and you want delete or keep only some of these elements, start by creating a set of elements you want to keep or delete. For example:
indices_to_keep = {0, 2}
keys_to_delete = {'firstname', 'middlename'}
Then choose the appropriate snippet from this list:
To keep list elements: value = [val for i, val in enumerate(value) if i in indices_to_keep]
To delete list elements: value = [val for i, val in enumerate(value) if i not in indices_to_delete]
To keep dict elements: value = {key: val for key, val in value.items() if key in keys_to_keep]
To delete dict elements: value = {key: val for key, val in value.items() if key not in keys_to_delete]
Output
Once the grouping is complete, we have a defaultdict filled with lists. But the desired result isn't always a (default)dict.
In this section, we're customizing the result = groupdict line of the recipe.
A regular dict
To convert the defaultdict to a regular dict, simply call the dict constructor on it:
result = dict(groupdict)
A list of (group, value) pairs
To get a result like [(group1, value1), (group1, value2), (group2, value3)] from the dict {group1: [value1, value2], group2: [value3]}, use a list comprehension:
result = [(group, value) for group, values in groupdict.items()
for value in values]
A nested list of just values
To get a result like [[value1, value2], [value3]] from the dict {group1: [value1, value2], group2: [value3]}, use dict.values:
result = list(groupdict.values())
A flat list of just values
To get a result like [value1, value2, value3] from the dict {group1: [value1, value2], group2: [value3]}, flatten the dict with a list comprehension:
result = [value for values in groupdict.values() for value in values]
Flattening iterable values
If your values are lists or other iterables like
groupdict = {group1: [[list1_value1, list1_value2], [list2_value1]]}
and you want a flattened result like
result = {group1: [list1_value1, list1_value2, list2_value1]}
you have two options:
Flatten the lists with a dict comprehension:
result = {group: [x for iterable in values for x in iterable]
for group, values in groupdict.items()}
Avoid creating a list of iterables in the first place, by using list.extend instead of list.append. In other words, change
groupdict[group].append(value)
to
groupdict[group].extend(value)
And then just set result = groupdict.
A sorted list
Dicts are unordered data structures. If you iterate over a dict, you never know in which order its elements will be listed. If you don't care about the order, you can use the recipes shown above. But if you do care about the order, you have to sort the output accordingly.
I'll use the following dict to demonstrate how to sort your output in various ways:
groupdict = {'abc': [1], 'xy': [2, 5]}
Keep in mind that this is a bit of a meta-recipe that may need to be combined with other parts of this answer to get exactly the output you want. The general idea is to sort the dictionary keys before using them to extract the values from the dict:
groups = sorted(groupdict.keys())
# groups = ['abc', 'xy']
Keep in mind that sorted accepts a key function in case you want to customize the sort order. For example, if the dict keys are strings and you want to sort them by length:
groups = sorted(groupdict.keys(), key=len)
# groups = ['xy', 'abc']
Once you've sorted the keys, use them to extract the values from the dict in the correct order:
# groups = ['abc', 'xy']
result = [groupdict[group] for group in groups]
# result = [[1], [2, 5]]
Remember that this can be combined with other parts of this answer to get different kinds of output. For example, if you want to keep the group identifiers:
# groups = ['abc', 'xy']
result = [(group, groupdict[group]) for group in groups]
# result = [('abc', [1]), ('xy', [2, 5])]
For your convenience, here are some commonly used sort orders:
Sort by number of values per group:
groups = sorted(groudict.keys(), key=lambda group: len(groupdict[group]))
result = [groupdict[group] for group in groups]
# result = [[2, 5], [1]]
Counting the number of values in each group
To count the number of elements associated with each group, use the len function:
result = {group: len(values) for group, values in groupdict.items()}
If you want to count the number of distinct elements, use set to eliminate duplicates:
result = {group: len(set(values)) for group, values in groupdict.items()}
An example
To demonstrate how to piece together a working solution from this recipe, let's try to turn an input of
data = [["A",0], ["B",1], ["C",0], ["D",2], ["E",2]]
into
result = [["A", "C"], ["B"], ["D", "E"]]
In other words, we're grouping lists by their 2nd element.
The first two lines of the recipe are always the same, so let's start by copying those:
import collections
groupdict = collections.defaultdict(list)
Now we have to find out how to loop over the input. Since our input is a simple list of values, a normal for loop will suffice:
for value in data:
Next we have to extract the group identifier from the value. We're grouping by the 2nd list element, so we use indexing:
group = value[1]
The next step is to transform the value. Since we only want to keep the first element of each list, we once again use list indexing:
value = value[0]
Finally, we have to figure out how to turn the dict we generated into a list. What we want is a list of values, without the groups. We consult the Output section of the recipe to find the appropriate dict flattening snippet:
result = list(groupdict.values())
Et voilà:
data = [["A",0], ["B",1], ["C",0], ["D",2], ["E",2]]
import collections
groupdict = collections.defaultdict(list)
for value in data:
group = value[1]
value = value[0]
groupdict[group].append(value)
result = list(groupdict.values())
# result: [["A", "C"], ["B"], ["D", "E"]]
itertools.groupby
There is a general purpose recipe in itertools and it's groupby().
A schema of this recipe can be given in this form:
[(k, aggregate(g)) for k, g in groupby(sorted(data, key=extractKey), extractKey)]
The two relevant parts to change in the recipe are:
define the grouping key (extractKey): in this case getting the first item of the tuple:
lambda x: x[0]
aggregate grouped results (if needed) (aggregate): g contains all the matching tuples for each key k (e.g. (1, 'a'), (1, 'b') for key 1, and (2, 'x') for key 2), we want to take only the second item of the tuple and concatenate all of those in one string:
''.join(x[1] for x in g)
Example:
from itertools import groupby
extractKey = lambda x: x[0]
aggregate = lambda g: ''.join(x[1] for x in g)
[(k, aggregate(g)) for k, g in groupby(sorted(data, key=extractKey), extractKey)]
# [(1, 'ab'), (2, 'x')]
Sometimes, extractKey, aggregate, or both can be inlined into a one-liner (we omit sort key too, as that's redundant for this example):
[(k, ''.join(x[1] for x in g)) for k, g in groupby(sorted(data), lambda x: x[0])]
# [(1, 'ab'), (2, 'x')]
Pros and cons
Comparing this recipe with the recipe using defaultdict there are pros and cons in both cases.
groupby() tends to be slower (about twice as slower in my tests) than the defaultdict recipe.
On the other hand, groupby() has advantages in the memory constrained case where the values are being produced on the fly; you can process the groups in a streaming fashion, without storing them; defaultdict will require the memory to store all of them.
Pandas groupby
This isn't a recipe as such, but an intuitive and flexible way to group data using a function. In this case, the function is str.join.
import pandas as pd
data = [(1, 'a'), (2, 'x'), (1, 'b')]
# create dataframe from list of tuples
df = pd.DataFrame(data)
# group by first item and apply str.join
grp = df.groupby(0)[1].apply(''.join)
# create list of tuples from index and value
res = list(zip(grp.index, grp))
print(res)
[(1, 'ab'), (2, 'x')]
Advantages
Fits nicely into workflows that only require list output at the end of a sequence of vectorisable steps.
Easily adaptable by changing ''.join to list or other reducing function.
Disadvantages
Overkill for an isolated task: requires list -> pd.DataFrame -> list conversion.
Introduces dependency on a 3rd party library.
Multiple-parse list comprehension
This is inefficient compared to the dict and groupby solutions.
However, for small lists where performance is not a concern, you can perform a list comprehension which parses the list for each unique identifier.
res = [(i, ''.join([j[1] for j in data if j[0] == i]))
for i in set(list(zip(*data))[0])]
[(1, 'ab'), (2, 'x')]
The solution can be split into 2 parts:
set(list(zip(*data))[0]) extracts the unique set of identifiers which we iterate via a for loop within the list comprehension.
(i, ''.join([j[1] for j in data if j[0] == i])) applies the logic we require for the desired output.
I have a dictionary where the keys are strings and the values are integers. I have to return a list which has the keys with smaller values first. In case of a tie, the key which comes first lexicographically is to be returned.
For example, consider a dictionary which maps names to weights.
d={'Tony':78,'Steve':88,'Bruce':68,'Thor':78,'Bob':50}
Then the resultant list should be:
['Bob','Bruce','Thor','Tony','Steve']
Since Thor comes before Tony, lexicographically.
How do I accomplish this?
If you want to break the tie by name, first sort by value then key:
from operator import itemgetter
print(list(map(itemgetter(0), sorted(d.items(), key=itemgetter(1,0)))))
['Bob', 'Bruce', 'Thor', 'Tony', 'Steve']
Or a regular list comp:
print([k for k,_ in sorted(d.items(), key=itemgetter(1, 0))])
Fairly easy:
result = sorted(d, key=lambda k: (d[k], k))
If the dict is huge, it might be worth avoiding the per-key dict lookup by iterating the items directly and using itemgetter to flip keys and values for sorting, then remove values in the result:
from operator import itemgetter
results = list(map(itemgetter(0), sorted(d.items(), key=itemgetter(1, 0))))
I have two python dictionaries, and I want to return combinations of the dictionary keys if the keys do not have any values in common with each other.
Sort of like:
for i in 'a':
for j in 'b':
for i.items():
if any.i.items() in any.j.items()
continue
else show me someway to pair all combinations together
Lets say my code is as shown below:
a={'dog':['hungry','fun'],'cat':['ugly','old'],'fish':[1,2,'hungry',4]}
b={'raccoon':['ugly',1,3,], 'bird':[18,'hungry'],'cat':['orange','short']}
This would return
dog+raccoon, dog+cat, cat+bird, cat+cat, fish+cat
because none of these potential pairings have dictionary values in common. If anyone could show me how I would do this I would appreciate it.
You can use sets:
from itertools import product
pairs = set()
for (ak, av), (bk, bv) in product(a.items(), b.items()):
if not set(av).intersection(set(bv)):
pairs.add("%s+%s" % tuple(sorted([ak, bk])))
print pairs
Produces:
set(['bird+cat', 'cat+fish', 'cat+cat', 'cat+dog', 'dog+raccoon'])
This can be more efficient if you have sets instead of lists in the first place.
May be this can get it:
a={'dog':['hungry','fun'],'cat':['ugly','old'],'fish':[1,2,'hungry',4]}
b={'raccoon':['ugly',1,3,], 'bird':[18,'hungry'],'cat':['orange','short']}
result = []
for b_key, b_value in b.iteritems():
for a_key, a_value in a.iteritems():
if not any(x in a_value for x in b_value):
result.append((a_key, b_key))
print(result)
[('dog', 'raccoon'), ('cat', 'bird'), ('fish', 'cat'), ('dog', 'cat'), ('cat', 'cat')]
def no_shared(a,b):
res = []
pairs = [(x, y) for x in a for y in b]
for x,y in pairs:
if set(a[x]) & set(b[y]):
continue
res.append((x, y))
return res
Basically test for any intersection between the lists (as sets) and then skip adding it if they have any intersection.
Also, if the list values are always going to be unique for that entry in that dict (seems so from the example) then storing it initially as a set saves you the conversion.