We Keep Coding

Better ways to find pairs that sum to N

Is there a faster way to write this, the function takes a list and a value to find the pairs of numeric values in that list that sum to N without duplicates I tried to make it faster by using sets instead of using the list itself (however I used count() which I know is is linear time) any suggestions I know there is probably a way
def pairsum_n(list1, value):
set1 = set(list1)
solution = {(min(i, value - i) , max(i, value - i)) for i in set1 if value - i in set1}
solution.remove((value/2,value/2)) if list1.count(value/2) < 2 else None
return solution
"""
Example: value = 10, list1 = [1,2,3,4,5,6,7,8,9]
pairsum_n = { (1,9), (2,8), (3,7), (4,6) }
Example: value = 10, list2 = [5,6,7,5,7,5,3]
pairsum_n = { (5,5), (3,7) }
"""

Your approach is quite good, it just needs a few tweaks to make it more efficient. itertools is convenient, but it's not really suitable for this task because it produces so many unwanted pairs. It's ok if the input list is small, but it's too slow if the input list is large.
We can avoid producing duplicates by looping over the numbers in order, stopping when i >= value/2, after using a set to get rid of dupes.
def pairsum_n(list1, value):
set1 = set(list1)
list1 = sorted(set1)
solution = []
maxi = value / 2
for i in list1:
if i >= maxi:
break
j = value - i
if j in set1:
solution.append((i, j))
return solution
Note that the original list1 is not modified. The assignment in this function creates a new local list1. If you do actually want (value/2, value/2) in the output, just change the break condition.
Here's a slightly more compact version.
def pairsum_n(list1, value):
set1 = set(list1)
solution = []
for i in sorted(set1):
j = value - i
if i >= j:
break
if j in set1:
solution.append((i, j))
return solution
It's possible to condense this further, eg using itertools.takewhile, but it will be harder to read and there won't be any improvement in efficiency.

Try this, running time O(nlogn):
v = [1, 2, 3, 4, 5, 6, 7, 8, 9]
l = 0
r = len(v)-1
def myFunc(v, value):
ans = []
% this block search for the pair (value//2, value//2)
if value % 2 == 0:
c = [i for i in v if i == value // 2]
if len(c) >= 2:
ans.append((c[0], c[1]))
v = list(set(v))
l = 0
r = len(v)-1
v.sort()
while l<len(v) and r >= 0 and l < r:
if v[l] + v[r] == value:
ans.append((v[l], v[r]))
l += 1
r -= 1
elif v[l] + v[r] < value:
l += 1
else:
r -= 1
return list(set(ans))
It is called the Two pointers technique and it works as follows. First of all, sort the array. This imposes a minimum running time of O(nlogn). Then set two pointers, one pointing at the start of the array l and other pointing at its last element r (pointers name are for left and right).
Now, look at the list. If the sum of the values returned at position l and r is lower than the value we are looking for, then we need to increment l. If it's greater, we need to decrement r.
If v[l] + v[r] == value than we can increment/decrement both l or r since in any case we want to skip the combination of values (v[l], v[r]) as we don't want duplicates.

Timings: this is actually slower then the other 2 solutions. Due to the amount of combinations produced but not actually needed it gets worse the bigger the lists are.
You can use itertools.combinations to produce the 2-tuple-combinations for you.
Put them into a set if they match your value, then return as set/list:
from itertools import combinations
def pairsum_n(list1, value):
"""Returns the unique list of pairs of combinations of numbers from
list1 that sum up `value`. Reorders the values to (min_value,max_value)."""
result = set()
for n in combinations(list1, 2):
if sum(n) == value:
result.add( (min(n),max(n)) )
return list(result)
# more ugly one-liner:
# return list(set(((min(n),max(n)) for n in combinations(list1,2) if sum(n)==value)))
data = [1,2,3,4,5,6,6,5,4,3,2,1]
print(pairsum_n(data,7))
Output:
[(1, 6), (2, 5), (3, 4)]
Fun little thing, with some sorting overhead you can get all at once:
def pairsum_n2(data, count_nums=2):
"""Generate a dict with all count_nums-tuples from data. Key into the
dict is the sum of all tuple-values."""
d = {}
for n in (tuple(sorted(p)) for p in combinations(data,count_nums)):
d.setdefault(sum(n),set()).add(n)
return d
get_all = pairsum_n2(data,2) # 2 == number of numbers to combine
for k in get_all:
print(k," -> ", get_all[k])
Output:
3 -> {(1, 2)}
4 -> {(1, 3), (2, 2)}
5 -> {(2, 3), (1, 4)}
6 -> {(1, 5), (2, 4), (3, 3)}
7 -> {(3, 4), (2, 5), (1, 6)}
2 -> {(1, 1)}
8 -> {(2, 6), (4, 4), (3, 5)}
9 -> {(4, 5), (3, 6)}
10 -> {(5, 5), (4, 6)}
11 -> {(5, 6)}
12 -> {(6, 6)}
And then just access the one you need via:
print(get_all.get(7,"Not possible")) # {(3, 4), (2, 5), (1, 6)}
print(get_all.get(17,"Not possible")) # Not possible

Have another solution, it's alot faster then the one I just wrote, not as fast as #PM 2Ring's answer:
def pairsum_n(list1, value):
set1 = set(list1)
if list1.count(value/2) < 2:
set1.remove(value/2)
return set((min(x, value - x) , max(x, value - x)) for x in filterfalse(lambda x: (value - x) not in set1, set1))

Converting List to Tupled [duplicate]

This question already has answers here:
How do I split a list into equally-sized chunks?
(66 answers)
Closed 1 year ago.
How do I make a for loop or a list comprehension so that every iteration gives me two elements?
l = [1,2,3,4,5,6]
for i,k in ???:
print str(i), '+', str(k), '=', str(i+k)
Output:
1+2=3
3+4=7
5+6=11

You need a pairwise() (or grouped()) implementation.
def pairwise(iterable):
"s -> (s0, s1), (s2, s3), (s4, s5), ..."
a = iter(iterable)
return zip(a, a)
for x, y in pairwise(l):
print("%d + %d = %d" % (x, y, x + y))
Or, more generally:
def grouped(iterable, n):
"s -> (s0,s1,s2,...sn-1), (sn,sn+1,sn+2,...s2n-1), (s2n,s2n+1,s2n+2,...s3n-1), ..."
return zip(*[iter(iterable)]*n)
for x, y in grouped(l, 2):
print("%d + %d = %d" % (x, y, x + y))
In Python 2, you should import izip as a replacement for Python 3's built-in zip() function.
All credit to martineau for his answer to my question, I have found this to be very efficient as it only iterates once over the list and does not create any unnecessary lists in the process.
N.B: This should not be confused with the pairwise recipe in Python's own itertools documentation, which yields s -> (s0, s1), (s1, s2), (s2, s3), ..., as pointed out by #lazyr in the comments.
Little addition for those who would like to do type checking with mypy on Python 3:
from typing import Iterable, Tuple, TypeVar
T = TypeVar("T")
def grouped(iterable: Iterable[T], n=2) -> Iterable[Tuple[T, ...]]:
"""s -> (s0,s1,s2,...sn-1), (sn,sn+1,sn+2,...s2n-1), ..."""
return zip(*[iter(iterable)] * n)

Well you need tuple of 2 elements, so
data = [1,2,3,4,5,6]
for i,k in zip(data[0::2], data[1::2]):
print str(i), '+', str(k), '=', str(i+k)
Where:
data[0::2] means create subset collection of elements that (index % 2 == 0)
zip(x,y) creates a tuple collection from x and y collections same index elements.

>>> l = [1,2,3,4,5,6]
>>> zip(l,l[1:])
[(1, 2), (2, 3), (3, 4), (4, 5), (5, 6)]
>>> zip(l,l[1:])[::2]
[(1, 2), (3, 4), (5, 6)]
>>> [a+b for a,b in zip(l,l[1:])[::2]]
[3, 7, 11]
>>> ["%d + %d = %d" % (a,b,a+b) for a,b in zip(l,l[1:])[::2]]
['1 + 2 = 3', '3 + 4 = 7', '5 + 6 = 11']

A simple solution.
l = [1, 2, 3, 4, 5, 6]
for i in range(0, len(l), 2):
print str(l[i]), '+', str(l[i + 1]), '=', str(l[i] + l[i + 1])

While all the answers using zip are correct, I find that implementing the functionality yourself leads to more readable code:
def pairwise(it):
it = iter(it)
while True:
try:
yield next(it), next(it)
except StopIteration:
# no more elements in the iterator
return
The it = iter(it) part ensures that it is actually an iterator, not just an iterable. If it already is an iterator, this line is a no-op.
Usage:
for a, b in pairwise([0, 1, 2, 3, 4, 5]):
print(a + b)

I hope this will be even more elegant way of doing it.
a = [1,2,3,4,5,6]
zip(a[::2], a[1::2])
[(1, 2), (3, 4), (5, 6)]

In case you're interested in the performance, I did a small benchmark (using my library simple_benchmark) to compare the performance of the solutions and I included a function from one of my packages: iteration_utilities.grouper
from iteration_utilities import grouper
import matplotlib as mpl
from simple_benchmark import BenchmarkBuilder
bench = BenchmarkBuilder()
#bench.add_function()
def Johnsyweb(l):
def pairwise(iterable):
"s -> (s0, s1), (s2, s3), (s4, s5), ..."
a = iter(iterable)
return zip(a, a)
for x, y in pairwise(l):
pass
#bench.add_function()
def Margus(data):
for i, k in zip(data[0::2], data[1::2]):
pass
#bench.add_function()
def pyanon(l):
list(zip(l,l[1:]))[::2]
#bench.add_function()
def taskinoor(l):
for i in range(0, len(l), 2):
l[i], l[i+1]
#bench.add_function()
def mic_e(it):
def pairwise(it):
it = iter(it)
while True:
try:
yield next(it), next(it)
except StopIteration:
return
for a, b in pairwise(it):
pass
#bench.add_function()
def MSeifert(it):
for item1, item2 in grouper(it, 2):
pass
bench.use_random_lists_as_arguments(sizes=[2**i for i in range(1, 20)])
benchmark_result = bench.run()
mpl.rcParams['figure.figsize'] = (8, 10)
benchmark_result.plot_both(relative_to=MSeifert)
So if you want the fastest solution without external dependencies you probably should just use the approach given by Johnysweb (at the time of writing it's the most upvoted and accepted answer).
If you don't mind the additional dependency then the grouper from iteration_utilities will probably be a bit faster.
Additional thoughts
Some of the approaches have some restrictions, that haven't been discussed here.
For example a few solutions only work for sequences (that is lists, strings, etc.), for example Margus/pyanon/taskinoor solutions which uses indexing while other solutions work on any iterable (that is sequences and generators, iterators) like Johnysweb/mic_e/my solutions.
Then Johnysweb also provided a solution that works for other sizes than 2 while the other answers don't (okay, the iteration_utilities.grouper also allows setting the number of elements to "group").
Then there is also the question about what should happen if there is an odd number of elements in the list. Should the remaining item be dismissed? Should the list be padded to make it even sized? Should the remaining item be returned as single? The other answer don't address this point directly, however if I haven't overlooked anything they all follow the approach that the remaining item should be dismissed (except for taskinoors answer - that will actually raise an Exception).
With grouper you can decide what you want to do:
>>> from iteration_utilities import grouper
>>> list(grouper([1, 2, 3], 2)) # as single
[(1, 2), (3,)]
>>> list(grouper([1, 2, 3], 2, truncate=True)) # ignored
[(1, 2)]
>>> list(grouper([1, 2, 3], 2, fillvalue=None)) # padded
[(1, 2), (3, None)]

Use the zip and iter commands together:
I find this solution using iter to be quite elegant:
it = iter(l)
list(zip(it, it))
# [(1, 2), (3, 4), (5, 6)]
Which I found in the Python 3 zip documentation.
it = iter(l)
print(*(f'{u} + {v} = {u+v}' for u, v in zip(it, it)), sep='\n')
# 1 + 2 = 3
# 3 + 4 = 7
# 5 + 6 = 11
To generalise to N elements at a time:
N = 2
list(zip(*([iter(l)] * N)))
# [(1, 2), (3, 4), (5, 6)]

for (i, k) in zip(l[::2], l[1::2]):
print i, "+", k, "=", i+k
zip(*iterable) returns a tuple with the next element of each iterable.
l[::2] returns the 1st, the 3rd, the 5th, etc. element of the list: the first colon indicates that the slice starts at the beginning because there's no number behind it, the second colon is only needed if you want a 'step in the slice' (in this case 2).
l[1::2] does the same thing but starts in the second element of the lists so it returns the 2nd, the 4th, 6th, etc. element of the original list.

With unpacking:
l = [1,2,3,4,5,6]
while l:
i, k, *l = l
print(f'{i}+{k}={i+k}')
Note: this will consume l, leaving it empty afterward.

There are many ways to do that. For example:
lst = [1,2,3,4,5,6]
[(lst[i], lst[i+1]) for i,_ in enumerate(lst[:-1])]
>>>[(1, 2), (2, 3), (3, 4), (4, 5), (5, 6)]
list(zip(*[iter(lst)]*2))
>>>[(1, 2), (3, 4), (5, 6)]

you can use more_itertools package.
import more_itertools
lst = range(1, 7)
for i, j in more_itertools.chunked(lst, 2):
print(f'{i} + {j} = {i+j}')

For anyone it might help, here is a solution to a similar problem but with overlapping pairs (instead of mutually exclusive pairs).
From the Python itertools documentation:
from itertools import izip
def pairwise(iterable):
"s -> (s0,s1), (s1,s2), (s2, s3), ..."
a, b = tee(iterable)
next(b, None)
return izip(a, b)
Or, more generally:
from itertools import izip
def groupwise(iterable, n=2):
"s -> (s0,s1,...,sn-1), (s1,s2,...,sn), (s2,s3,...,sn+1), ..."
t = tee(iterable, n)
for i in range(1, n):
for j in range(0, i):
next(t[i], None)
return izip(*t)

The title of this question is misleading, you seem to be looking for consecutive pairs, but if you want to iterate over the set of all possible pairs than this will work :
for i,v in enumerate(items[:-1]):
for u in items[i+1:]:

A simplistic approach:
[(a[i],a[i+1]) for i in range(0,len(a),2)]
this is useful if your array is a and you want to iterate on it by pairs.
To iterate on triplets or more just change the "range" step command, for example:
[(a[i],a[i+1],a[i+2]) for i in range(0,len(a),3)]
(you have to deal with excess values if your array length and the step do not fit)

The polished Python3 solution is given in one of the itertools recipes:
import itertools
def grouper(iterable, n, fillvalue=None):
"Collect data into fixed-length chunks or blocks"
# grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx"
args = [iter(iterable)] * n
return itertools.zip_longest(*args, fillvalue=fillvalue)

Another try at cleaner solution
def grouped(itr, n=2):
itr = iter(itr)
end = object()
while True:
vals = tuple(next(itr, end) for _ in range(n))
if vals[-1] is end:
return
yield vals
For more customization options
from collections.abc import Sized
def grouped(itr, n=2, /, truncate=True, fillvalue=None, strict=False, nofill=False):
if strict:
if isinstance(itr, Sized):
if len(itr) % n != 0:
raise ValueError(f"{len(itr)=} is not divisible by {n=}")
itr = iter(itr)
end = object()
while True:
vals = tuple(next(itr, end) for _ in range(n))
if vals[-1] is end:
if vals[0] is end:
return
if strict:
raise ValueError("found extra stuff in iterable")
if nofill:
yield tuple(v for v in vals if v is not end)
return
if truncate:
return
yield tuple(v if v is not end else fillvalue for v in vals)
return
yield vals

Thought that this is a good place to share my generalization of this for n>2, which is just a sliding window over an iterable:
def sliding_window(iterable, n):
its = [ itertools.islice(iter, i, None)
for i, iter
in enumerate(itertools.tee(iterable, n)) ]
return itertools.izip(*its)

I need to divide a list by a number and fixed like this.
l = [1,2,3,4,5,6]
def divideByN(data, n):
return [data[i*n : (i+1)*n] for i in range(len(data)//n)]
>>> print(divideByN(l,2))
[[1, 2], [3, 4], [5, 6]]
>>> print(divideByN(l,3))
[[1, 2, 3], [4, 5, 6]]

Using typing so you can verify data using mypy static analysis tool:
from typing import Iterator, Any, Iterable, TypeVar, Tuple
T_ = TypeVar('T_')
Pairs_Iter = Iterator[Tuple[T_, T_]]
def legs(iterable: Iterator[T_]) -> Pairs_Iter:
begin = next(iterable)
for end in iterable:
yield begin, end
begin = end

Here we can have alt_elem method which can fit in your for loop.
def alt_elem(list, index=2):
for i, elem in enumerate(list, start=1):
if not i % index:
yield tuple(list[i-index:i])
a = range(10)
for index in [2, 3, 4]:
print("With index: {0}".format(index))
for i in alt_elem(a, index):
print(i)
Output:
With index: 2
(0, 1)
(2, 3)
(4, 5)
(6, 7)
(8, 9)
With index: 3
(0, 1, 2)
(3, 4, 5)
(6, 7, 8)
With index: 4
(0, 1, 2, 3)
(4, 5, 6, 7)
Note: Above solution might not be efficient considering operations performed in func.

This is simple solution, which is using range function to pick alternative elements from a list of elements.
Note: This is only valid for an even numbered list.
a_list = [1, 2, 3, 4, 5, 6]
empty_list = []
for i in range(0, len(a_list), 2):
empty_list.append(a_list[i] + a_list[i + 1])
print(empty_list)
# [3, 7, 11]

Compute all ways to bin a series of integers into N bins, where each bin only contains contiguous numbers

I want find all possible ways to map a series of (contiguous) integers M = {0,1,2,...,m} to another series of integers N = {0,1,2,...,n} where m > n, subject to the constraint that only contiguous integers in M map to the same integer in N.
The following piece of python code comes close (start corresponds to the first element in M, stop-1 corresponds to the last element in M, and nbins corresponds to |N|):
import itertools
def find_bins(start, stop, nbins):
if (nbins > 1):
return list(list(itertools.product([range(start, ii)], find_bins(ii, stop, nbins-1))) for ii in range(start+1, stop-nbins+2))
else:
return [range(start, stop)]
E.g
In [20]: find_bins(start=0, stop=5, nbins=3)
Out[20]:
[[([0], [([1], [2, 3, 4])]),
([0], [([1, 2], [3, 4])]),
([0], [([1, 2, 3], [4])])],
[([0, 1], [([2], [3, 4])]),
([0, 1], [([2, 3], [4])])],
[([0, 1, 2], [([3], [4])])]]
However, as you can see the output is nested, and for the life of me, I cant find a way to properly amend the code without breaking it.
The desired output would look like this:
In [20]: find_bins(start=0, stop=5, nbins=3)
Out[20]:
[[(0), (1), (2, 3, 4)],
[(0), (1, 2), (3, 4)],
[(0), (1, 2, 3), (4)],
[(0, 1), (2), (3, 4)],
[(0, 1), (2, 3), (4)],
[(0, 1, 2), (3), (4)]]

I suggest a different approach: a partitioning into n non-empty bins is uniquely determined by the n-1 distinct indices marking the boundaries between the bins, where the first marker is after the first element, and the final marker before the last element. itertools.combinations() can be used directly to generate all such index tuples, and then it's just a matter of using them as slice indices. Like so:
def find_nbins(start, stop, nbins):
from itertools import combinations
base = range(start, stop)
nbase = len(base)
for ixs in combinations(range(1, stop - start), nbins - 1):
yield [tuple(base[lo: hi])
for lo, hi in zip((0,) + ixs, ixs + (nbase,))]
Then, e.g.,
for x in find_nbins(0, 5, 3):
print(x)
displays:
[(0,), (1,), (2, 3, 4)]
[(0,), (1, 2), (3, 4)]
[(0,), (1, 2, 3), (4,)]
[(0, 1), (2,), (3, 4)]
[(0, 1), (2, 3), (4,)]
[(0, 1, 2), (3,), (4,)]
EDIT: Making it into 2 problems
Just noting that there's a more general underlying problem here: generating the ways to break an arbitrary sequence into n non-empty bins. Then the specific question here is applying that to the sequence range(start, stop). I believe viewing it that way makes the code easier to understand, so here it is:
def gbins(seq, nbins):
from itertools import combinations
base = tuple(seq)
nbase = len(base)
for ixs in combinations(range(1, nbase), nbins - 1):
yield [base[lo: hi]
for lo, hi in zip((0,) + ixs, ixs + (nbase,))]
def find_nbins(start, stop, nbins):
return gbins(range(start, stop), nbins)

This does what I want; I will gladly accept simpler, more elegant solutions:
def _split(start, stop, nbins):
if (nbins > 1):
out = []
for ii in range(start+1, stop-nbins+2):
iterator = itertools.product([range(start, ii)], _split(ii, stop, nbins-1))
for item in iterator:
out.append(item)
return out
else:
return [range(start, stop)]
def _unpack(nested):
unpacked = []
if isinstance(nested, (list, tuple)):
for item in nested:
if isinstance(item, tuple):
for subitem in item:
unpacked.extend(_unpack(subitem))
elif isinstance(item, list):
unpacked.append([_unpack(subitem) for subitem in item])
elif isinstance(item, int):
unpacked.append([item])
return unpacked
else: # integer
return nested
def find_nbins(start, stop, nbins):
nested = _split(start, stop, nbins)
unpacked = [_unpack(item) for item in nested]
return unpacked

how to split an iterable in constant-size chunks [duplicate]

This question already has answers here:
How do I split a list into equally-sized chunks?
(66 answers)
Closed 8 months ago.
I am surprised I could not find a "batch" function that would take as input an iterable and return an iterable of iterables.
For example:
for i in batch(range(0,10), 1): print i
[0]
[1]
...
[9]
or:
for i in batch(range(0,10), 3): print i
[0,1,2]
[3,4,5]
[6,7,8]
[9]
Now, I wrote what I thought was a pretty simple generator:
def batch(iterable, n = 1):
current_batch = []
for item in iterable:
current_batch.append(item)
if len(current_batch) == n:
yield current_batch
current_batch = []
if current_batch:
yield current_batch
But the above does not give me what I would have expected:
for x in batch(range(0,10),3): print x
[0]
[0, 1]
[0, 1, 2]
[3]
[3, 4]
[3, 4, 5]
[6]
[6, 7]
[6, 7, 8]
[9]
So, I have missed something and this probably shows my complete lack of understanding of python generators. Anyone would care to point me in the right direction ?
[Edit: I eventually realized that the above behavior happens only when I run this within ipython rather than python itself]

This is probably more efficient (faster)
def batch(iterable, n=1):
l = len(iterable)
for ndx in range(0, l, n):
yield iterable[ndx:min(ndx + n, l)]
for x in batch(range(0, 10), 3):
print x
Example using list
data = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10] # list of data
for x in batch(data, 3):
print(x)
# Output
[0, 1, 2]
[3, 4, 5]
[6, 7, 8]
[9, 10]
It avoids building new lists.

The recipes in the itertools module provide two ways to do this depending on how you want to handle a final odd-sized lot (keep it, pad it with a fillvalue, ignore it, or raise an exception):
from itertools import islice, zip_longest
def batched(iterable, n):
"Batch data into lists of length n. The last batch may be shorter."
# batched('ABCDEFG', 3) --> ABC DEF G
it = iter(iterable)
while True:
batch = list(islice(it, n))
if not batch:
return
yield batch
def grouper(iterable, n, *, incomplete='fill', fillvalue=None):
"Collect data into non-overlapping fixed-length chunks or blocks"
# grouper('ABCDEFG', 3, fillvalue='x') --> ABC DEF Gxx
# grouper('ABCDEFG', 3, incomplete='strict') --> ABC DEF ValueError
# grouper('ABCDEFG', 3, incomplete='ignore') --> ABC DEF
args = [iter(iterable)] * n
if incomplete == 'fill':
return zip_longest(*args, fillvalue=fillvalue)
if incomplete == 'strict':
return zip(*args, strict=True)
if incomplete == 'ignore':
return zip(*args)
else:
raise ValueError('Expected fill, strict, or ignore')

More-itertools includes two functions that do what you need:
chunked(iterable, n) returns an iterable of lists, each of length n (except the last one, which may be shorter);
ichunked(iterable, n) is similar, but returns an iterable of iterables instead.

As others have noted, the code you have given does exactly what you want. For another approach using itertools.islice you could see an example of following recipe:
from itertools import islice, chain
def batch(iterable, size):
sourceiter = iter(iterable)
while True:
batchiter = islice(sourceiter, size)
yield chain([batchiter.next()], batchiter)

Solution for Python 3.8 if you are working with iterables that don't define a len function, and get exhausted:
from itertools import islice
def batcher(iterable, batch_size):
iterator = iter(iterable)
while batch := list(islice(iterator, batch_size)):
yield batch
Example usage:
def my_gen():
yield from range(10)
for batch in batcher(my_gen(), 3):
print(batch)
>>> [0, 1, 2]
>>> [3, 4, 5]
>>> [6, 7, 8]
>>> [9]
Could of course be implemented without the walrus operator as well.

This is a very short code snippet I know that does not use len and works under both Python 2 and 3 (not my creation):
def chunks(iterable, size):
from itertools import chain, islice
iterator = iter(iterable)
for first in iterator:
yield list(chain([first], islice(iterator, size - 1)))

Weird, seems to work fine for me in Python 2.x
>>> def batch(iterable, n = 1):
... current_batch = []
... for item in iterable:
... current_batch.append(item)
... if len(current_batch) == n:
... yield current_batch
... current_batch = []
... if current_batch:
... yield current_batch
...
>>> for x in batch(range(0, 10), 3):
... print x
...
[0, 1, 2]
[3, 4, 5]
[6, 7, 8]
[9]

A workable version without new features in python 3.8, adapted from #Atra Azami's answer.
import itertools
def batch_generator(iterable, batch_size=1):
iterable = iter(iterable)
while True:
batch = list(itertools.islice(iterable, batch_size))
if len(batch) > 0:
yield batch
else:
break
for x in batch_generator(range(0, 10), 3):
print(x)
Output:
[0, 1, 2]
[3, 4, 5]
[6, 7, 8]
[9]

def batch(iterable, n):
iterable=iter(iterable)
while True:
chunk=[]
for i in range(n):
try:
chunk.append(next(iterable))
except StopIteration:
yield chunk
return
yield chunk
list(batch(range(10), 3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]

Moving as much into CPython as possible, by leveraging islice and iter(callable) behavior:
from itertools import islice
def chunked(generator, size):
"""Read parts of the generator, pause each time after a chunk"""
# islice returns results until 'size',
# make_chunk gets repeatedly called by iter(callable).
gen = iter(generator)
make_chunk = lambda: list(islice(gen, size))
return iter(make_chunk, [])
Inspired by more-itertools, and shortened to the essence of that code.

This is what I use in my project. It handles iterables or lists as efficiently as possible.
def chunker(iterable, size):
if not hasattr(iterable, "__len__"):
# generators don't have len, so fall back to slower
# method that works with generators
for chunk in chunker_gen(iterable, size):
yield chunk
return
it = iter(iterable)
for i in range(0, len(iterable), size):
yield [k for k in islice(it, size)]
def chunker_gen(generator, size):
iterator = iter(generator)
for first in iterator:
def chunk():
yield first
for more in islice(iterator, size - 1):
yield more
yield [k for k in chunk()]

I like this one,
def batch(x, bs):
return [x[i:i+bs] for i in range(0, len(x), bs)]
This returns a list of batches of size bs, you can make it a generator by using a generator expression (i for i in iterable) of course.

Here is an approach using reduce function.
Oneliner:
from functools import reduce
reduce(lambda cumulator,item: cumulator[-1].append(item) or cumulator if len(cumulator[-1]) < batch_size else cumulator + [[item]], input_array, [[]])
Or more readable version:
from functools import reduce
def batch(input_list, batch_size):
def reducer(cumulator, item):
if len(cumulator[-1]) < batch_size:
cumulator[-1].append(item)
return cumulator
else:
cumulator.append([item])
return cumulator
return reduce(reducer, input_list, [[]])
Test:
>>> batch([1,2,3,4,5,6,7], 3)
[[1, 2, 3], [4, 5, 6], [7]]
>>> batch(a, 8)
[[1, 2, 3, 4, 5, 6, 7]]
>>> batch([1,2,3,None,4], 3)
[[1, 2, 3], [None, 4]]

This would work for any iterable.
from itertools import zip_longest, filterfalse
def batch_iterable(iterable, batch_size=2):
args = [iter(iterable)] * batch_size
return (tuple(filterfalse(lambda x: x is None, group)) for group in zip_longest(fillvalue=None, *args))
It would work like this:
>>>list(batch_iterable(range(0,5)), 2)
[(0, 1), (2, 3), (4,)]
PS: It would not work if iterable has None values.

You can just group iterable items by their batch index.
def batch(items: Iterable, batch_size: int) -> Iterable[Iterable]:
# enumerate items and group them by batch index
enumerated_item_groups = itertools.groupby(enumerate(items), lambda t: t[0] // batch_size)
# extract items from enumeration tuples
item_batches = ((t[1] for t in enumerated_items) for key, enumerated_items in enumerated_item_groups)
return item_batches
It is often the case when you want to collect inner iterables so here is more advanced version.
def batch_advanced(items: Iterable, batch_size: int, batches_mapper: Callable[[Iterable], Any] = None) -> Iterable[Iterable]:
enumerated_item_groups = itertools.groupby(enumerate(items), lambda t: t[0] // batch_size)
if batches_mapper:
item_batches = (batches_mapper(t[1] for t in enumerated_items) for key, enumerated_items in enumerated_item_groups)
else:
item_batches = ((t[1] for t in enumerated_items) for key, enumerated_items in enumerated_item_groups)
return item_batches
Examples:
print(list(batch_advanced([1, 9, 3, 5, 2, 4, 2], 4, tuple)))
# [(1, 9, 3, 5), (2, 4, 2)]
print(list(batch_advanced([1, 9, 3, 5, 2, 4, 2], 4, list)))
# [[1, 9, 3, 5], [2, 4, 2]]

Related functionality you may need:
def batch(size, i):
""" Get the i'th batch of the given size """
return slice(size* i, size* i + size)
Usage:
>>> [1,2,3,4,5,6,7,8,9,10][batch(3, 1)]
>>> [4, 5, 6]
It gets the i'th batch from the sequence and it can work with other data structures as well, like pandas dataframes (df.iloc[batch(100,0)]) or numpy array (array[batch(100,0)]).

from itertools import *
class SENTINEL: pass
def batch(iterable, n):
return (tuple(filterfalse(lambda x: x is SENTINEL, group)) for group in zip_longest(fillvalue=SENTINEL, *[iter(iterable)] * n))
print(list(range(10), 3)))
# outputs: [(0, 1, 2), (3, 4, 5), (6, 7, 8), (9,)]
print(list(batch([None]*10, 3)))
# outputs: [(None, None, None), (None, None, None), (None, None, None), (None,)]

I use
def batchify(arr, batch_size):
num_batches = math.ceil(len(arr) / batch_size)
return [arr[i*batch_size:(i+1)*batch_size] for i in range(num_batches)]

Keep taking (at most) n elements until it runs out.
def chop(n, iterable):
iterator = iter(iterable)
while chunk := list(take(n, iterator)):
yield chunk
def take(n, iterable):
iterator = iter(iterable)
for i in range(n):
try:
yield next(iterator)
except StopIteration:
return

This code has the following features:
Can take lists or generators (no len()) as input
Does not require imports of other packages
No padding added to last batch
def batch_generator(items, batch_size):
itemid=0 # Keeps track of current position in items generator/list
batch = [] # Empty batch
for item in items:
batch.append(item) # Append items to batch
if len(batch)==batch_size:
yield batch
itemid += batch_size # Increment the position in items
batch = []
yield batch # yield last bit

Using Python's list index() method on a list of tuples or objects?

Python's list type has an index() method that takes one parameter and returns the index of the first item in the list matching the parameter. For instance:
>>> some_list = ["apple", "pear", "banana", "grape"]
>>> some_list.index("pear")
1
>>> some_list.index("grape")
3
Is there a graceful (idiomatic) way to extend this to lists of complex objects, like tuples? Ideally, I'd like to be able to do something like this:
>>> tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
>>> some_list.getIndexOfTuple(1, 7)
1
>>> some_list.getIndexOfTuple(0, "kumquat")
2
getIndexOfTuple() is just a hypothetical method that accepts a sub-index and a value, and then returns the index of the list item with the given value at that sub-index. I hope
Is there some way to achieve that general result, using list comprehensions or lambas or something "in-line" like that? I think I could write my own class and method, but I don't want to reinvent the wheel if Python already has a way to do it.

How about this?
>>> tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
>>> [x for x, y in enumerate(tuple_list) if y[1] == 7]
[1]
>>> [x for x, y in enumerate(tuple_list) if y[0] == 'kumquat']
[2]
As pointed out in the comments, this would get all matches. To just get the first one, you can do:
>>> [y[0] for y in tuple_list].index('kumquat')
2
There is a good discussion in the comments as to the speed difference between all the solutions posted. I may be a little biased but I would personally stick to a one-liner as the speed we're talking about is pretty insignificant versus creating functions and importing modules for this problem, but if you are planning on doing this to a very large amount of elements you might want to look at the other answers provided, as they are faster than what I provided.

Those list comprehensions are messy after a while.
I like this Pythonic approach:
from operator import itemgetter
tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
def collect(l, index):
return map(itemgetter(index), l)
# And now you can write this:
collect(tuple_list,0).index("cherry") # = 1
collect(tuple_list,1).index("3") # = 2
If you need your code to be all super performant:
# Stops iterating through the list as soon as it finds the value
def getIndexOfTuple(l, index, value):
for pos,t in enumerate(l):
if t[index] == value:
return pos
# Matches behavior of list.index
raise ValueError("list.index(x): x not in list")
getIndexOfTuple(tuple_list, 0, "cherry") # = 1

One possibility is to use the itemgetter function from the operator module:
import operator
f = operator.itemgetter(0)
print map(f, tuple_list).index("cherry") # yields 1
The call to itemgetter returns a function that will do the equivalent of foo[0] for anything passed to it. Using map, you then apply that function to each tuple, extracting the info into a new list, on which you then call index as normal.
map(f, tuple_list)
is equivalent to:
[f(tuple_list[0]), f(tuple_list[1]), ...etc]
which in turn is equivalent to:
[tuple_list[0][0], tuple_list[1][0], tuple_list[2][0]]
which gives:
["pineapple", "cherry", ...etc]

You can do this with a list comprehension and index()
tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
[x[0] for x in tuple_list].index("kumquat")
2
[x[1] for x in tuple_list].index(7)
1

Inspired by this question, I found this quite elegant:
>>> tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
>>> next(i for i, t in enumerate(tuple_list) if t[1] == 7)
1
>>> next(i for i, t in enumerate(tuple_list) if t[0] == "kumquat")
2

I would place this as a comment to Triptych, but I can't comment yet due to lack of rating:
Using the enumerator method to match on sub-indices in a list of tuples.
e.g.
li = [(1,2,3,4), (11,22,33,44), (111,222,333,444), ('a','b','c','d'),
('aa','bb','cc','dd'), ('aaa','bbb','ccc','ddd')]
# want pos of item having [22,44] in positions 1 and 3:
def getIndexOfTupleWithIndices(li, indices, vals):
# if index is a tuple of subindices to match against:
for pos,k in enumerate(li):
match = True
for i in indices:
if k[i] != vals[i]:
match = False
break;
if (match):
return pos
# Matches behavior of list.index
raise ValueError("list.index(x): x not in list")
idx = [1,3]
vals = [22,44]
print getIndexOfTupleWithIndices(li,idx,vals) # = 1
idx = [0,1]
vals = ['a','b']
print getIndexOfTupleWithIndices(li,idx,vals) # = 3
idx = [2,1]
vals = ['cc','bb']
print getIndexOfTupleWithIndices(li,idx,vals) # = 4

ok, it might be a mistake in vals(j), the correction is:
def getIndex(li,indices,vals):
for pos,k in enumerate(lista):
match = True
for i in indices:
if k[i] != vals[indices.index(i)]:
match = False
break
if(match):
return pos

z = list(zip(*tuple_list))
z[1][z[0].index('persimon')]

tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
def eachtuple(tupple, pos1, val):
for e in tupple:
if e == val:
return True
for e in tuple_list:
if eachtuple(e, 1, 7) is True:
print tuple_list.index(e)
for e in tuple_list:
if eachtuple(e, 0, "kumquat") is True:
print tuple_list.index(e)

Python's list.index(x) returns index of the first occurrence of x in the list. So we can pass objects returned by list compression to get their index.
>>> tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
>>> [tuple_list.index(t) for t in tuple_list if t[1] == 7]
[1]
>>> [tuple_list.index(t) for t in tuple_list if t[0] == 'kumquat']
[2]
With the same line, we can also get the list of index in case there are multiple matched elements.
>>> tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11), ("banana", 7)]
>>> [tuple_list.index(t) for t in tuple_list if t[1] == 7]
[1, 4]

I guess the following is not the best way to do it (speed and elegance concerns) but well, it could help :
from collections import OrderedDict as od
t = [('pineapple', 5), ('cherry', 7), ('kumquat', 3), ('plum', 11)]
list(od(t).keys()).index('kumquat')
2
list(od(t).values()).index(7)
7
# bonus :
od(t)['kumquat']
3
list of tuples with 2 members can be converted to ordered dict directly, data structures are actually the same, so we can use dict method on the fly.

This is also possible using Lambda expressions:
l = [('rana', 1, 1), ('pato', 1, 1), ('perro', 1, 1)]
map(lambda x:x[0], l).index("pato") # returns 1
Edit to add examples:
l=[['rana', 1, 1], ['pato', 2, 1], ['perro', 1, 1], ['pato', 2, 2], ['pato', 2, 2]]
extract all items by condition:
filter(lambda x:x[0]=="pato", l) #[['pato', 2, 1], ['pato', 2, 2], ['pato', 2, 2]]
extract all items by condition with index:
>>> filter(lambda x:x[1][0]=="pato", enumerate(l))
[(1, ['pato', 2, 1]), (3, ['pato', 2, 2]), (4, ['pato', 2, 2])]
>>> map(lambda x:x[1],_)
[['pato', 2, 1], ['pato', 2, 2], ['pato', 2, 2]]
Note: The _ variable only works in the interactive interpreter. More generally, one must explicitly assign _, i.e. _=filter(lambda x:x[1][0]=="pato", enumerate(l)).

I came up with a quick and dirty approach using max and lambda.
>>> tuple_list = [("pineapple", 5), ("cherry", 7), ("kumquat", 3), ("plum", 11)]
>>> target = 7
>>> max(range(len(tuple_list)), key=lambda i: tuple_list[i][1] == target)
1
There is a caveat though that if the list does not contain the target, the returned index will be 0, which could be misleading.
>>> target = -1
>>> max(range(len(tuple_list)), key=lambda i: tuple_list[i][1] == target)
0