Is there a better way to sort a list by a nested tuple values than writing an itemgetter alternative that extracts the nested tuple value:
def deep_get(*idx):
def g(t):
for i in idx: t = t[i]
return t
return g
>>> l = [((2,1), 1),((1,3), 1),((3,6), 1),((4,5), 2)]
>>> sorted(l, key=deep_get(0,0))
[((1, 3), 1), ((2, 1), 1), ((3, 6), 1), ((4, 5), 2)]
>>> sorted(l, key=deep_get(0,1))
[((2, 1), 1), ((1, 3), 1), ((4, 5), 2), ((3, 6), 1)]
I thought about using compose, but that's not in the standard library:
sorted(l, key=compose(itemgetter(1), itemgetter(0))
Is there something I missed in the libs that would make this code nicer?
The implementation should work reasonably with 100k items.
Context: I would like to sort a dictionary of items that are a histogram. The keys are a tuples (a,b) and the value is the count. In the end the items should be sorted by count descending, a and b. An alternative is to flatten the tuple and use the itemgetter directly but this way a lot of tuples will be generated.
Yes, you could just use a key=lambda x: x[0][1]
Your approach is quite good, given the data structure that you have.
Another approach would be to use another structure.
If you want speed, the de-factor standard NumPy is the way to go. Its job is to efficiently handle large arrays. It even has some nice sorting routines for arrays like yours. Here is how you would write your sort over the counts, and then over (a, b):
>>> arr = numpy.array([((2,1), 1),((1,3), 1),((3,6), 1),((4,5), 2)],
dtype=[('pos', [('a', int), ('b', int)]), ('count', int)])
>>> print numpy.sort(arr, order=['count', 'pos'])
[((1, 3), 1) ((2, 1), 1) ((3, 6), 1) ((4, 5), 2)]
This is very fast (it's implemented in C).
If you want to stick with standard Python, a list containing (count, a, b) tuples would automatically get sorted in the way you want by Python (which uses lexicographic order on tuples).
I compared two similar solutions. The first one uses a simple lambda:
def sort_one(d):
result = d.items()
result.sort(key=lambda x: (-x[1], x[0]))
return result
Note the minus on x[1], because you want the sort to be descending on count.
The second one takes advantage of the fact that sort in Python is stable. First, we sort by (a, b) (ascending). Then we sort by count, descending:
def sort_two(d):
result = d.items()
result.sort()
result.sort(key=itemgetter(1), reverse=True)
return result
The first one is 10-20% faster (both on small and large datasets), and both complete under 0.5sec on my Q6600 (one core used) for 100k items. So avoiding the creation of tuples doesn't seem to help much.
This might be a little faster version of your approach:
l = [((2,1), 1), ((1,3), 1), ((3,6), 1), ((4,5), 2)]
def deep_get(*idx):
def g(t):
return reduce(lambda t, i: t[i], idx, t)
return g
>>> sorted(l, key=deep_get(0,1))
[((2, 1), 1), ((1, 3), 1), ((4, 5), 2), ((3, 6), 1)]
Which could be shortened to:
def deep_get(*idx):
return lambda t: reduce(lambda t, i: t[i], idx, t)
or even just simply written-out:
sorted(l, key=lambda t: reduce(lambda t, i: t[i], (0,1), t))
Related
I have a list of tuple in python and I would like to sort it first the decreasing order of value(int) and if it matches the increasing order of word(str)
data = [(1, u'day'), (2, u'is'), (2, u'lunny'), (4, u'the')]
data.sort(key = lambda x: (x[0], x[1]), reverse=True)
The above sorts by decreasing order of value but fails to handle sorting by increasing order of str(2nd value in the tuple).
Does anyone have suggestion for a workaround on how to fix this?
You could leave reverse=False but just negate the first value
>>> data.sort(key=lambda x: (-x[0], x[1]))
>>> data
[(4, 'the'), (2, 'is'), (2, 'lunny'), (1, 'day')]
You could also run sort twice, taking advantage of it being stable (see doc). Maybe you find this to be more explicit:
>>> data = [(1, u'day'), (2, u'is'), (2, u'lunny'), (4, u'the')]
>>> data.sort(key = lambda x: x[1])
>>> data.sort(key = lambda x: x[0], reverse=True)
>>> data
[(4, 'the'), (2, 'is'), (2, 'lunny'), (1, 'day')]
When sorting by x[0], the previous order (ascending by x[1]) is preserved.
Mark Lutz in his book "Learning Python" gives an example:
>>> [(x,y) for x in range(5) if x%2==0 for y in range(5) if y%2==1]
[(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
>>>
a bit later he remarks that 'a map and filter equivalent' of this is possible though complex and nested.
The closest one I ended up with is the following:
>>> list(map(lambda x:list(map(lambda y:(y,x),filter(lambda x:x%2==0,range(5)))), filter(lambda x:x%2==1,range(5))))
[[(0, 1), (2, 1), (4, 1)], [(0, 3), (2, 3), (4, 3)]]
>>>
The order of tuples is different and nested list had to be introduced. I'm curious what would be the equivalent.
A note to append to #Kasramvd's explanation.
Readability is important in Python. It's one of the features of the language. Many will consider the list comprehension the only readable way.
Sometimes, however, especially when you are working with multiple iterations of conditions, it is clearer to separate your criteria from logic. In this case, using the functional method may be preferable.
from itertools import product
def even_and_odd(vals):
return (vals[0] % 2 == 0) and (vals[1] %2 == 1)
n = range(5)
res = list(filter(even_and_odd, product(n, n)))
One important point that you have to notice is that your nested list comprehension is of O(n2) order. Meaning that it's looping over a product of two ranges. If you want to use map and filter you have to create all the combinations. You can do that after or before filtering but what ever you do you can't have all those combinations with those two functions, unless you change the ranges and/or modify something else.
One completely functional approach is to use itertools.product() and filter as following:
In [16]: from itertools import product
In [17]: list(filter(lambda x: x[0]%2==0 and x[1]%2==1, product(range(5), range(5))))
Out[17]: [(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
Also note that using a nested list comprehension with two iterations is basically more readable than multiple map/filter functions. And regarding the performance using built-in funcitons is faster than list comprehension when your function are merely built-in so that you can assure all of them are performing at C level. When you break teh chain with something like a lambda function which is Python/higher lever operation your code won't be faster than a list comprehension.
I think the only confusing part in the expression [(x, y) for x in range(5) if x % 2 == 0 for y in range(5) if y % 2 == 1] is that there an implicit flatten operation is hidden.
Let's consider the simplified version of the expression first:
def even(x):
return x % 2 == 0
def odd(x):
return not even(x)
c = map(lambda x: map(lambda y: [x, y],
filter(odd, range(5))),
filter(even, range(5)))
print(c)
# i.e. for each even X we have a list of odd Ys:
# [
# [[0, 1], [0, 3]],
# [[2, 1], [2, 3]],
# [[4, 1], [4, 3]]
# ]
However, we need pretty the same but flattened list [(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)].
From the official python docs we can grab the example of flatten function:
from itertools import chain
flattened = list(chain.from_iterable(c)) # we need list() here to unroll an iterator
print(flattened)
Which is basically an equivalent for the following list comprehension expression:
flattened = [x for sublist in c for x in sublist]
print(flattened)
# ... which is basically an equivalent to:
# result = []
# for sublist in c:
# for x in sublist:
# result.append(x)
Range support step argument, so I come up with this solution using itertools.chain.from_iterable to flatten inner list:
from itertools import chain
list(chain.from_iterable(
map(
lambda x:
list(map(lambda y: (x, y), range(1, 5, 2))),
range(0, 5, 2)
)
))
Output:
Out[415]: [(0, 1), (0, 3), (2, 1), (2, 3), (4, 1), (4, 3)]
I have a tuple list to_order such as:
to_order = [(0, 1), (1, 3), (2, 2), (3,2)]
And a list which gives the order to apply to the second element of each tuple of to_order:
order = [2, 1, 3]
So I am looking for a way to get this output:
ordered_list = [(2, 2), (3,2), (0, 1), (1, 3)]
Any ideas?
You can provide a key that will check the index (of the second element) in order and sort based on it:
to_order = [(0, 1), (1, 3), (2, 2), (3,2)]
order = [2, 1, 3]
print(sorted(to_order, key=lambda item: order.index(item[1]))) # [(2, 2), (3, 2), (0, 1), (1, 3)]
EDIT
Since, a discussion on time complexities was start... here ya go, the following algorithm runs in O(n+m), using Eric's input example:
N = 5
to_order = [(randrange(N), randrange(N)) for _ in range(10*N)]
order = list(set(pair[1] for pair in to_order))
shuffle(order)
def eric_sort(to_order, order):
bins = {}
for pair in to_order:
bins.setdefault(pair[1], []).append(pair)
return [pair for i in order for pair in bins[i]]
def alfasin_new_sort(to_order, order):
arr = [[] for i in range(len(order))]
d = {k:v for v, k in enumerate(order)}
for item in to_order:
arr[d[item[1]]].append(item)
return [item for sublist in arr for item in sublist]
from timeit import timeit
print("eric_sort", timeit("eric_sort(to_order, order)", setup=setup, number=1000))
print("alfasin_new_sort", timeit("alfasin_new_sort(to_order, order)", setup=setup, number=1000))
OUTPUT:
eric_sort 59.282021682999584
alfasin_new_sort 44.28244407700004
Algorithm
You can distribute the tuples in a dict of lists according to the second element and iterate over order indices to get the sorted list:
from collections import defaultdict
to_order = [(0, 1), (1, 3), (2, 2), (3, 2)]
order = [2, 1, 3]
bins = defaultdict(list)
for pair in to_order:
bins[pair[1]].append(pair)
print(bins)
# defaultdict(<class 'list'>, {1: [(0, 1)], 3: [(1, 3)], 2: [(2, 2), (3, 2)]})
print([pair for i in order for pair in bins[i]])
# [(2, 2), (3, 2), (0, 1), (1, 3)]
sort or index aren't needed and the output is stable.
This algorithm is similar to the mapping mentioned in the supposed duplicate. This linked answer only works if to_order and order have the same lengths, which isn't the case in OP's question.
Performance
This algorithm iterates twice over each element of to_order. The complexity is O(n). #alfasin's first algorithm is much slower (O(n * m * log n)), but his second one is also O(n).
Here's a list with 10000 random pairs between 0 and 1000. We extract the unique second elements and shuffle them in order to define order:
from random import randrange, shuffle
from collections import defaultdict
from timeit import timeit
from itertools import chain
N = 1000
to_order = [(randrange(N), randrange(N)) for _ in range(10*N)]
order = list(set(pair[1] for pair in to_order))
shuffle(order)
def eric(to_order, order):
bins = defaultdict(list)
for pair in to_order:
bins[pair[1]].append(pair)
return list(chain.from_iterable(bins[i] for i in order))
def alfasin1(to_order, order):
arr = [[] for i in range(len(order))]
d = {k:v for v, k in enumerate(order)}
for item in to_order:
arr[d[item[1]]].append(item)
return [item for sublist in arr for item in sublist]
def alfasin2(to_order, order):
return sorted(to_order, key=lambda item: order.index(item[1]))
print(eric(to_order, order) == alfasin1(to_order, order))
# True
print(eric(to_order, order) == alfasin2(to_order, order))
# True
print("eric", timeit("eric(to_order, order)", globals=globals(), number=100))
# eric 0.3117517130003762
print("alfasin1", timeit("alfasin1(to_order, order)", globals=globals(), number=100))
# alfasin1 0.36100843100030033
print("alfasin2", timeit("alfasin2(to_order, order)", globals=globals(), number=100))
# alfasin2 15.031453827000405
Another solution:
[item for key in order for item in filter(lambda x: x[1] == key, to_order)]
This solution works off of order first, filtering to_order for each key in order.
Equivalent:
ordered = []
for key in order:
for item in filter(lambda x: x[1] == key, to_order):
ordered.append(item)
Shorter, but I'm not aware of a way to do this with list comprehension:
ordered = []
for key in order:
ordered.extend(filter(lambda x: x[1] == key, to_order))
Note: This will not throw a ValueError if to_order contains a tuple x where x[1] is not in order.
I personally prefer the list objects sort function rather than the built-in sort which generates a new list rather than changing the list in place.
to_order = [(0, 1), (1, 3), (2, 2), (3,2)]
order = [2, 1, 3]
to_order.sort(key=lambda x: order.index(x[1]))
print(to_order)
>[(2, 2), (3, 2), (0, 1), (1, 3)]
A little explanation on the way: The key parameter of the sort method basically preprocesses the list and ranks all the values based on a measure. In our case order.index() looks at the first occurrence of the currently processed item and returns its position.
x = [1,2,3,4,5,3,3,5]
print x.index(5)
>4
It is easy to implement a regular double sort:
pairs = [(1, 2), (2, 1), (1, 3), (2, 4), (3, 1)]
sorted(pairs,key=lambda x: (x[0],x[1]))
# out: [(1, 2), (1, 3), (2, 1), (2, 4), (3, 1)]
I am interested how to do it with the second elements in the reverse order. This can be easily implemented by grouping the pairs by the first item at first and then adding the the sorted second items together. I have implemented this both using itertools.groupby and defaultdict. Still, it remains far more complex, than the regular double sort, so i wonder, if there is a neat trick to do it in a more concise way.
double_sort(pairs)
# out: [(1, 3), (1, 2), (2, 4), (2, 1), (3, 1)]
PS! I know how to do it with numpy.argsort and would mostly like to see a standard lib approach.
This will work for numbers and similar data types
sorted(pairs, key=lambda x: (x[0], -x[1]))
This will work for all comparable types only in Python 2 :(
sorted(pairs, lambda x, y: cmp(x[0], y[0]) or cmp(y[1], x[1]))
I have a list
l=[(1,2),(1,6),(3,4),(3,6),(1,4),(4,3)]
I want to return a list that contains lists by the first number in each tuple.
Something like this:
[[2,4,6],[4,6],[3]]
To make a program that iterates on list and writing a whole function that does it is easy.
I want to find a oneliner - python way of doing it.
Any ideas?
>>> from itertools import groupby
>>> from operator import itemgetter
>>> L = [(1,2), (1,6), (3,4), (3,6), (1,4), (4,3)]
>>> [[y for x, y in v] for k, v in groupby(sorted(L), itemgetter(0))]
[[2, 4, 6], [4, 6], [3]]
Explanation
This works by using itertools.groupby. groupby finds consecutive groups in an iterable, returning an iterator through key, group pairs.
The argument given to groupby is a key function, itemgetter(0) which is called for each tuple, returning the first item as the key to groupby.
groupby groups elements in their original order so if you want to group by the first number in the list, it must first be sorted so groupby can go through the first numbers in ascending order and actually group them.
>>> sorted(L)
[(1, 2), (1, 4), (1, 6), (3, 4), (3, 6), (4, 3)]
There is the sorted list where you can clearly see the groups that will be created if you look back to the final output. Now you can use groupby to show the key, group pairs.
[(1, <itertools._grouper object at 0x02BB7ED0>), (3, <itertools._grouper object at 0x02BB7CF0>), (4, <itertools._grouper object at 0x02BB7E30>)]
Here are the sorted items grouped by the first number. groupby returns the group for each key as an iterator, this is great and very efficient but for this example we will just convert it to a list to make sure it's working properly.
>>> [(k, list(v)) for k,v in groupby(sorted(L), itemgetter(0))]
[(1, [(1, 2), (1, 4), (1, 6)]), (3, [(3, 4), (3, 6)]), (4, [(4, 3)])]
That is almost the right thing but the required output shows only the 2nd number in the groups in each list. So the following achieves the desired result.
[[y for x, y in v] for k, v in groupby(sorted(L), itemgetter(0))]
l = [(1, 2), (1, 6), (3, 4), (3, 6), (1, 4), (4, 3)]
d = {}
for (k, v) in l:
d.setdefault(k, []).append(v)
print d.values()
I know it's not a one liner, but perhaps it's easier to read than a one liner.