If i had to reduce over a pair of values, how do i write the lambda expression for the same.
testr = [('r1', (1, 1)), ('r1', (1, 5)),('r2', (1, 1)),('r3', (1, 1))]
Desired output is
('r1', (2, 6)),('r2', (1, 1)),('r3', (1, 1))
Reduce it by Key:
.reduceByKey(lambda a, b: (a[0]+b[0], a[1]+b[1]))
You can make it more general purpose for arbitrary length tuples with zip:
.reduceByKey(lambda a, b: tuple(x+y for x,y in zip(a,b)))
it is not clear for me how reduce can use to reduce with lambda to reduce list tuples with different keys. My solution is can reduce list of tuples, but it uses function, which is perhaps too troublesome to do in pure lambda, if not impossible.
def reduce_tuple_list(tl):
import operator as op
import functools as fun
import itertools as it
# sort the list for groupby
tl = sorted(tl,key=op.itemgetter(0))
# this function with reduce lists with the same key
def reduce_with_same_key(tl):
def add_tuple(t1,t2):
k1, tl1 = t1
k2, tl2 = t2
if k1 == k2:
l1,r1 = tl1
l2,r2 = tl2
l = l1+l2
r = r1+r2
return k1,(l,r)
else:
return t1,t2
return tuple(fun.reduce(add_tuple, tl))
# group by keys
groups = []
for k, g in it.groupby(tl, key=op.itemgetter(0)):
groups.append(list(g))
new_list = []
# we need to add only lists whose length is greater than one
for el in groups:
if len(el) > 1: # reduce
new_list.append(reduce_with_same_key(el))
else: # single tuple without another one with the same key
new_list.append(el[0])
return new_list
testr = [('r1', (1, 1)), ('r3', (11, 71)), ('r1', (1, 5)),('r2', (1, 1)),('r3', (1, 1))]
>>> reduce_tuple_list(testr)
[('r1', (2, 6)), ('r2', (1, 1)), ('r3', (12, 72))]
you can use combineByKey method
testr = sc.parallelize((('r1', (1, 1)), ('r1', (1, 5)),('r2', (1, 1)),('r3', (1, 1))))
testr.combineByKey(lambda x:x,lambda x,y:(x[0]+y[0],x[1]+y[1]),lambda x,y:(x[0]+x[1],y[0]+y[1])).collect()
Related
Let's say I have the following array:
a = [4,2,3,1,4]
Then I sort it:
b = sorted(A) = [1,2,3,4,4]
How could I have a list that map where each number was, ex:
position(b,a) = [3,1,2,0,4]
to clarify this list contains the positions not values)
(ps' also taking in account that first 4 was in position 0)
b = sorted(enumerate(a), key=lambda i: i[1])
This results is a list of tuples, the first item of which is the original index and second of which is the value:
[(3, 1), (1, 2), (2, 3), (0, 4), (4, 4)]
def position(a):
return sorted(range(len(a)), key=lambda k: a[k])
I have a list of tuples: [(2, Operation.SUBSTITUTED), (1, Operation.DELETED), (2, Operation.INSERTED)]
I would like to sort this list in 2 ways:
First by its 1st value by ascending value, i.e. 1, 2, 3... etc
Second by its 2nd value by reverse alphabetical order, i.e. Operation.SUBSTITITUTED, Operation.INSERTED, Operation, DELETED
So the above list should be sorted as:
[(1, Operation.DELETED), (2, Operation.SUBSTITUTED), (2, Operation.INSERTED)]
How do I go about sort this list?
Since sorting is guaranteed to be stable, you can do this in 2 steps:
lst = [(2, 'Operation.SUBSTITUTED'), (1, 'Operation.DELETED'), (2, 'Operation.INSERTED')]
res_int = sorted(lst, key=lambda x: x[1], reverse=True)
res = sorted(res_int, key=lambda x: x[0])
print(res)
# [(1, 'Operation.DELETED'), (2, 'Operation.SUBSTITUTED'), (2, 'Operation.INSERTED')]
In this particular case, because the order of comparison can be easily inverted for integers, you can sort in one time using negative value for integer key & reverse:
lst = [(2, 'Operation.SUBSTITUTED'), (1, 'Operation.DELETED'), (2, 'Operation.INSERTED')]
res = sorted(lst, key=lambda x: (-x[0],x[1]), reverse=True)
result:
[(1, 'Operation.DELETED'), (2, 'Operation.SUBSTITUTED'), (2, 'Operation.INSERTED')]
negating the integer key cancels the "reverse" aspect, only kept for the second string criterion.
You can use this:
from operator import itemgetter
d = [(1, 'DELETED'), (2, 'INSERTED'), (2, 'SUBSTITUTED')]
d.sort(key=itemgetter(1),reverse=True)
d.sort(key=itemgetter(0))
print(d)
Another way using itemgetter from operator module:
from operator import itemgetter
lst = [(2, 'Operation.SUBSTITUTED'), (1, 'Operation.DELETED'), (2, 'Operation.INSERTED')]
inter = sorted(lst, key=itemgetter(1), reverse=True)
sorted_lst = sorted(inter, key=itemgetter(0))
print(sorted_lst)
# [(1, 'Operation.DELETED'), (2, 'Operation.SUBSTITUTED'), (2, 'Operation.INSERTED')]
I have a tuple list to_order such as:
to_order = [(0, 1), (1, 3), (2, 2), (3,2)]
And a list which gives the order to apply to the second element of each tuple of to_order:
order = [2, 1, 3]
So I am looking for a way to get this output:
ordered_list = [(2, 2), (3,2), (0, 1), (1, 3)]
Any ideas?
You can provide a key that will check the index (of the second element) in order and sort based on it:
to_order = [(0, 1), (1, 3), (2, 2), (3,2)]
order = [2, 1, 3]
print(sorted(to_order, key=lambda item: order.index(item[1]))) # [(2, 2), (3, 2), (0, 1), (1, 3)]
EDIT
Since, a discussion on time complexities was start... here ya go, the following algorithm runs in O(n+m), using Eric's input example:
N = 5
to_order = [(randrange(N), randrange(N)) for _ in range(10*N)]
order = list(set(pair[1] for pair in to_order))
shuffle(order)
def eric_sort(to_order, order):
bins = {}
for pair in to_order:
bins.setdefault(pair[1], []).append(pair)
return [pair for i in order for pair in bins[i]]
def alfasin_new_sort(to_order, order):
arr = [[] for i in range(len(order))]
d = {k:v for v, k in enumerate(order)}
for item in to_order:
arr[d[item[1]]].append(item)
return [item for sublist in arr for item in sublist]
from timeit import timeit
print("eric_sort", timeit("eric_sort(to_order, order)", setup=setup, number=1000))
print("alfasin_new_sort", timeit("alfasin_new_sort(to_order, order)", setup=setup, number=1000))
OUTPUT:
eric_sort 59.282021682999584
alfasin_new_sort 44.28244407700004
Algorithm
You can distribute the tuples in a dict of lists according to the second element and iterate over order indices to get the sorted list:
from collections import defaultdict
to_order = [(0, 1), (1, 3), (2, 2), (3, 2)]
order = [2, 1, 3]
bins = defaultdict(list)
for pair in to_order:
bins[pair[1]].append(pair)
print(bins)
# defaultdict(<class 'list'>, {1: [(0, 1)], 3: [(1, 3)], 2: [(2, 2), (3, 2)]})
print([pair for i in order for pair in bins[i]])
# [(2, 2), (3, 2), (0, 1), (1, 3)]
sort or index aren't needed and the output is stable.
This algorithm is similar to the mapping mentioned in the supposed duplicate. This linked answer only works if to_order and order have the same lengths, which isn't the case in OP's question.
Performance
This algorithm iterates twice over each element of to_order. The complexity is O(n). #alfasin's first algorithm is much slower (O(n * m * log n)), but his second one is also O(n).
Here's a list with 10000 random pairs between 0 and 1000. We extract the unique second elements and shuffle them in order to define order:
from random import randrange, shuffle
from collections import defaultdict
from timeit import timeit
from itertools import chain
N = 1000
to_order = [(randrange(N), randrange(N)) for _ in range(10*N)]
order = list(set(pair[1] for pair in to_order))
shuffle(order)
def eric(to_order, order):
bins = defaultdict(list)
for pair in to_order:
bins[pair[1]].append(pair)
return list(chain.from_iterable(bins[i] for i in order))
def alfasin1(to_order, order):
arr = [[] for i in range(len(order))]
d = {k:v for v, k in enumerate(order)}
for item in to_order:
arr[d[item[1]]].append(item)
return [item for sublist in arr for item in sublist]
def alfasin2(to_order, order):
return sorted(to_order, key=lambda item: order.index(item[1]))
print(eric(to_order, order) == alfasin1(to_order, order))
# True
print(eric(to_order, order) == alfasin2(to_order, order))
# True
print("eric", timeit("eric(to_order, order)", globals=globals(), number=100))
# eric 0.3117517130003762
print("alfasin1", timeit("alfasin1(to_order, order)", globals=globals(), number=100))
# alfasin1 0.36100843100030033
print("alfasin2", timeit("alfasin2(to_order, order)", globals=globals(), number=100))
# alfasin2 15.031453827000405
Another solution:
[item for key in order for item in filter(lambda x: x[1] == key, to_order)]
This solution works off of order first, filtering to_order for each key in order.
Equivalent:
ordered = []
for key in order:
for item in filter(lambda x: x[1] == key, to_order):
ordered.append(item)
Shorter, but I'm not aware of a way to do this with list comprehension:
ordered = []
for key in order:
ordered.extend(filter(lambda x: x[1] == key, to_order))
Note: This will not throw a ValueError if to_order contains a tuple x where x[1] is not in order.
I personally prefer the list objects sort function rather than the built-in sort which generates a new list rather than changing the list in place.
to_order = [(0, 1), (1, 3), (2, 2), (3,2)]
order = [2, 1, 3]
to_order.sort(key=lambda x: order.index(x[1]))
print(to_order)
>[(2, 2), (3, 2), (0, 1), (1, 3)]
A little explanation on the way: The key parameter of the sort method basically preprocesses the list and ranks all the values based on a measure. In our case order.index() looks at the first occurrence of the currently processed item and returns its position.
x = [1,2,3,4,5,3,3,5]
print x.index(5)
>4
Is there a better way to sort a list by a nested tuple values than writing an itemgetter alternative that extracts the nested tuple value:
def deep_get(*idx):
def g(t):
for i in idx: t = t[i]
return t
return g
>>> l = [((2,1), 1),((1,3), 1),((3,6), 1),((4,5), 2)]
>>> sorted(l, key=deep_get(0,0))
[((1, 3), 1), ((2, 1), 1), ((3, 6), 1), ((4, 5), 2)]
>>> sorted(l, key=deep_get(0,1))
[((2, 1), 1), ((1, 3), 1), ((4, 5), 2), ((3, 6), 1)]
I thought about using compose, but that's not in the standard library:
sorted(l, key=compose(itemgetter(1), itemgetter(0))
Is there something I missed in the libs that would make this code nicer?
The implementation should work reasonably with 100k items.
Context: I would like to sort a dictionary of items that are a histogram. The keys are a tuples (a,b) and the value is the count. In the end the items should be sorted by count descending, a and b. An alternative is to flatten the tuple and use the itemgetter directly but this way a lot of tuples will be generated.
Yes, you could just use a key=lambda x: x[0][1]
Your approach is quite good, given the data structure that you have.
Another approach would be to use another structure.
If you want speed, the de-factor standard NumPy is the way to go. Its job is to efficiently handle large arrays. It even has some nice sorting routines for arrays like yours. Here is how you would write your sort over the counts, and then over (a, b):
>>> arr = numpy.array([((2,1), 1),((1,3), 1),((3,6), 1),((4,5), 2)],
dtype=[('pos', [('a', int), ('b', int)]), ('count', int)])
>>> print numpy.sort(arr, order=['count', 'pos'])
[((1, 3), 1) ((2, 1), 1) ((3, 6), 1) ((4, 5), 2)]
This is very fast (it's implemented in C).
If you want to stick with standard Python, a list containing (count, a, b) tuples would automatically get sorted in the way you want by Python (which uses lexicographic order on tuples).
I compared two similar solutions. The first one uses a simple lambda:
def sort_one(d):
result = d.items()
result.sort(key=lambda x: (-x[1], x[0]))
return result
Note the minus on x[1], because you want the sort to be descending on count.
The second one takes advantage of the fact that sort in Python is stable. First, we sort by (a, b) (ascending). Then we sort by count, descending:
def sort_two(d):
result = d.items()
result.sort()
result.sort(key=itemgetter(1), reverse=True)
return result
The first one is 10-20% faster (both on small and large datasets), and both complete under 0.5sec on my Q6600 (one core used) for 100k items. So avoiding the creation of tuples doesn't seem to help much.
This might be a little faster version of your approach:
l = [((2,1), 1), ((1,3), 1), ((3,6), 1), ((4,5), 2)]
def deep_get(*idx):
def g(t):
return reduce(lambda t, i: t[i], idx, t)
return g
>>> sorted(l, key=deep_get(0,1))
[((2, 1), 1), ((1, 3), 1), ((4, 5), 2), ((3, 6), 1)]
Which could be shortened to:
def deep_get(*idx):
return lambda t: reduce(lambda t, i: t[i], idx, t)
or even just simply written-out:
sorted(l, key=lambda t: reduce(lambda t, i: t[i], (0,1), t))
I need to sort a list of tuples by first item in descending order, and then by second item in ascending order.
To do this, I have implemented the following function, but I think it could be faster.
>>> compare = lambda a, b: -cmp(b[1], a[1]) if b[0] == a[0] else cmp(b[0], a[0])
>>> sorted([(0, 2), (0, 1), (1, 0), (1, 2)], cmp=compare)
[(1, 0), (1, 2), (0, 1), (0, 2)]
Is it possible to optimize it? See the comparison against the built-in function:
>>> timeit.Timer(stmt='sorted([(int(random.getrandbits(4)),int(random.getrandbits(4))) for x in xrange(10)], cmp=compare)', setup='import random; compare=compare = lambda a, b: -cmp(b[1], a[1]) if b[0] == a[0] else cmp(b[0], a[0])').timeit(100000)
4.0584850867917339
>>> timeit.Timer(stmt='sorted([(int(random.getrandbits(4)),int(random.getrandbits(4))) for x in xrange(10)])', setup='import random').timeit(100000)
2.6582965153393161
For me, it is a little faster to use a key instead of a comparison function, and arguably also easier to read:
sorted([(0, 2), (0, 1), (1, 0), (1, 2)], key = lambda x:(-x[0], x[1]))
This requires Python 2.4 or newer.
How does this stack up for you?
compare = lambda a, b: cmp(b[0], a[0]) and cmp(a[1],b[1])