I very often run into the need to split a sequence into the two subsequences of elements that satisfy and don't satisfy a given predicate (preserving the original relative ordering).
This hypothetical "splitter" function would look something like this in action:
>>> data = map(str, range(14))
>>> pred = lambda i: int(i) % 3 == 2
>>> splitter(data, pred)
[('2', '5', '8', '11'), ('0', '1', '3', '4', '6', '7', '9', '10', '12', '13')]
My question is:
does Python already have a standard/built-in way to do this?
This functionality is certainly not difficult to code (see Addendum below), but for a number of reasons, I'd prefer to use a standard/built-in method than a self-rolled one.
Thanks!
Addendum:
The best standard function I've found so far for handling this task in Python is itertools.groupby. To use it for this particular task however, it is necessary to call the predicate function twice for each list member, which I find annoyingly silly:
>>> import itertools as it
>>> [tuple(v[1]) for v in it.groupby(sorted(data, key=pred), key=pred)]
[('0', '1', '3', '4', '6', '7', '9', '10', '12', '13'), ('2', '5', '8', '11')]
(The last output above differs from the desired one shown earlier in that the subsequence of elements that satisfy the predicate comes last rather than first, but this is very minor, and very easy to fix if needed.)
One can avoid the redundant calls to the predicate (by doing, basically, an "inline memoization"), but my best stab at this gets pretty elaborate, a far cry from the simplicity of splitter(data, pred):
>>> first = lambda t: t[0]
>>> [zip(*i[1])[1] for i in it.groupby(sorted(((pred(x), x) for x in data),
... key=first), key=first)]
[('0', '1', '3', '4', '6', '7', '9', '10', '12', '13'), ('2', '5', '8', '11')]
BTW, if you don't care about preserving the original ordering, sorted's default sort order gets the job done (so the key parameter may be omitted from the sorted call):
>>> [zip(*i[1])[1] for i in it.groupby(sorted(((pred(x), x) for x in data)),
... key=first)]
[('0', '1', '3', '4', '6', '7', '9', '10', '12', '13'), ('2', '5', '8', '11')]
I know you said you didn't want to write your own function, but I can't imagine why. Your solutions involve writing your own code, you just aren't modularizing them into functions.
This does exactly what you want, is understandable, and only evaluates the predicate once per element:
def splitter(data, pred):
yes, no = [], []
for d in data:
if pred(d):
yes.append(d)
else:
no.append(d)
return [yes, no]
If you want it to be more compact (for some reason):
def splitter(data, pred):
yes, no = [], []
for d in data:
(yes if pred(d) else no).append(d)
return [yes, no]
Partitioning is one of those itertools recipes that does just that. It uses tee() to make sure it's iterating the collection in one pass despite the multiple iterators, the builtin filter() function to grab items that satisfies the predicate as well as filterfalse() to get the opposite effect of the filter. This is as close as you're going to get at a standard/builtin method.
def partition(pred, iterable):
'Use a predicate to partition entries into false entries and true entries'
# partition(is_odd, range(10)) --> 0 2 4 6 8 and 1 3 5 7 9
t1, t2 = tee(iterable)
return filterfalse(pred, t1), filter(pred, t2)
In more_itertools there is a function called partition, which does exactly what topicstarter asked for.
from more_itertools import partition
numbers = [1, 2, 3, 4, 5, 6, 7]
predicate = lambda x: x % 2 == 0
predicate_false, predicate_true = partition(predicate, numbers)
print(list(predicate_false), list(predicate_true))
The result is [1, 3, 5, 7] [2, 4, 6].
If you don't care about efficiency, I think groupby (or any "putting data into n bins" functions) has some nice correspondence,
by_bins_iter = itertools.groupby(sorted(data, key=pred), key=pred)
by_bins = dict((k, tuple(v)) for k, v in by_bins_iter)
You can then get to your solution by,
return by_bins.get(True, ()), by_bins.get(False, ())
A slight variation of one of the OP's implementations and another commenter's implementation above using groupby:
groups = defaultdict(list, { k : list(ks) for k, ks in groupby(items, f) })
groups[True] == the matching items, or [] if none returned True
groups[False] == the non-matching items, or [] if none returned False
Sadly, as you point out, groupby requires that the items be sorted by the predicate first, so if that's not guaranteed, you need this:
groups = defaultdict(list, { k : list(ks) for k, ks in groupby(sorted(items, key=f), f) })
Quite a mouthful, but it is a single expression that partitions a list by a predicate using only built-in functions.
I don't think you can just use sorted without the key parameter, because groupby creates a new group when it hits a new value from the key function. So sorted will only work if the items sort naturally by the predicate provided.
As a slightly more general solution to partitioning, consider grouping. Consider the following function, inspired by clojure's group-by function.
You give it a collection of items to group, and a function that will be used to group them. Here's the code:
def group_by(seq, f):
groupings = {}
for item in seq:
res = f(item)
if res in groupings:
groupings[res].append(item)
else:
groupings[res] = [item]
return groupings
For the OP's original case:
y = group_by(range(14), lambda i: int(i) % 3 == 2)
{False: [0, 1, 3, 4, 6, 7, 9, 10, 12, 13], True: [2, 5, 8, 11]}
A more general case of grouping elements in a sequence by string length:
x = group_by(["x","xx","yy","zzz","z","7654321"], len)
{1: ['x', 'z'], 2: ['xx', 'yy'], 3: ['zzz'], 7: ['7654321']}
This can be extended to many cases, and is a core functionality of functional languages. It works great with the dynamically typed python, as the keys in the resulting map can be any type. Enjoy!
Reducing the iterable into the 2 partitions using functools.reduce you can get rid of the key function:
import functools
functools.reduce(
lambda tf, x: (tf[0], [*tf[1], x]) if pred(x) else ([*tf[0], x], tf[1]),
data,
([], []),
)
>>> (['0', '1', '3', '4', '6', '7', '9', '10', '12', '13'], ['2', '5', '8', '11'])
Related
Given a list which contains both strings and None values, in which some of the strings have embedded newlines, I wish to split the strings with newlines into multiple strings and return a flattened list.
I've written code to do this using a generator function, but the code is rather bulky and I'm wondering if it's possible to do it more concisely using a list comprehension or a function from the itertools module. itertools.chain doesn't seem to be able to decline to iterate any non-iterable elements.
def expand_newlines(lines):
r"""Split strings with newlines into multiple strings.
>>> l = ["1\n2\n3", None, "4\n5\n6"]
>>> list(expand_newlines(l))
['1', '2', '3', None, '4', '5', '6']
"""
for line in lines:
if line is None:
yield line
else:
for l in line.split('\n'):
yield l
You can use yield from.
def expand(lines):
for line in lines:
if isinstance(line,str):
yield from line.split('\n')
elif line is None:
yield line
list(expand(l))
#['1', '2', '3', None, '4', '5', '6']
Here's a single line, but I think #Ch3steR's solution is more readable.
from itertools import chain
list(chain.from_iterable(i.splitlines() if i is not None and '\n' in i else [i]
for i in lines))
You could use itertools.chain if you did the following
import itertools
def expand_newlines(lines):
return itertools.chain.from_iterable(x.split("\n") if x else [None]
for x in lines)
Using more_itertools.collapse to flatten nested lists:
Given
import more_itertools as mit
lst = ["1\n2\n3", None, "7\n8\n9"]
Demo
list(mit.collapse([x.split("\n") if x else x for x in lst ]))
# ['1', '2', '3', None, '7', '8', '9']
more_itertools is a third-party package. Install via > pip install more_itertools.
If you might modify list inplace then you might do:
lst = ["1\n2\n3", None, "4\n5\n6"]
for i in range(len(lst))[::-1]:
if isinstance(lst[i], str):
lst[i:i+1] = lst[i].split("\n")
print(lst) # ['1', '2', '3', None, '4', '5', '6']
this solution utilize fact that you might not only get python's list slices, but also assign to them. It moves from right to left, as otherwise I would need to keep count of additional items, which would make it harder.
Similar to #blueteeth's answer but more concise by way of inverting the logic:
import itertools
chainfi = itertools.chain.from_iterable
def expand_newlines(lines):
r"""Split strings with newlines into multiple strings.
>>> l = ["1\n2\n3", None, "4\n5\n6"]
>>> list(expand_newlines(l))
['1', '2', '3', None, '4', '5', '6']
"""
return chainfi([None] if l is None else l.split('\n') for l in lines)
None is the special case so that's what we should be checking for.
This is concise enough that I wouldn't even bother writing a function for it—I just kept it in the function to confirm it works via doctest.
I have a list of data that is in the structure of name and then score like this:
['Danny', '8', 'John', '5', 'Sandra', 10]
What I require to do in the simplest way possible is sort the data by lowest to highest score for example like this:
['John', '5', 'Danny', '8', 'Sandra', 10]
You should create pairings which will make your life a lot easier:
l = ['Danny', '8', 'John', '5', 'Sandra', '10']
it = iter(l)
srt = sorted(zip(it, it), key=lambda x: int(x[1]))
Which will give you:
[('John', '5'), ('Danny', '8'), ('Sandra', '10')]
it = iter(l) creates an iterator, then zip(it, it) basically calls (next(it), next(it)) each iteration so you create pairs of tuples in the format (user, score), then we sort by the second element of each tuple which is the score, casting to int.
You may be as well to cast to int and then sortif you plan on using the data, you could also create a flat list from the sorted data but I think that would be a bad idea.
The best data structure for your problem is Dictionary.
In your situatiton you need to map between names and scores.
dict = {'Danny':'8', 'John':'5', 'Sandra':'10'}
sorted_dict = ((k, dict[k]) for k in sorted(dict, key=dict.get, reverse=False))
for k, v in genexp:
... k, v
('John', '5')
('Danny', '8')
('Sandra', 10)
Assume I'd like to convert a list of strings to integer, but it cannot be done for all elements.
I know this works:
a = ['2.0','3.0','4.0','5.0','Cherry']
b = []
for k in a:
try:
int(k)
b.append(int(k))
except:
pass
print b
> [2, 3, 4, 5]
But is there also a shorter way of doing this? I thought about something like:
b = [try int(k) for k in a]
This may sound like a silly question since I do have a working solution, but I have often been shown shorter ways of doing the same thing and always appreciated this kind of help. I am using Python 2.7
Thanks!
Edit: sorry, I was also talking about floating point. I just changed my example data
There is no way to use try/except clauses inside List Comprehensions but this could help:
a = ['2','3','4','5','Cherry']
print [int(x) for x in a if x.isdigit()]
Output:
['2', '3', '4', '5']
Update (as the question was updated):
This could help but I don't know how good/accurate is to use it:
a = ['2.0','3.0','4.0','5.0', '44545.45', 'Cherry']
[float(x) for x in a if x.split('.')[0].isdigit() and x.split('.')[1].isdigit()]
Output:
[2.0, 3.0, 4.0, 5.0, 44545.45]
Try this one.
def is_number(k):
try:
float(k)
except ValueError:
return False
return True
[int(float(k)) for k in a if is_number(k)]
If you want to compress try-except into one line then i think answer is NO and it is answered at here.
I would go with mix of regex and isinstance check-It captures all number types e.g. float, long, int and complex-
>>>a=['2', '3', '4', '5', 'Cherry', '23.3', '-3']
>>>filter(bool,[i if isinstance(eval(i),(int, long, float, complex)) else None for i in filter(lambda x: re.findall(r'[-+]?\d+[\.]?\d*',x),a)])
>>>['2', '3', '4', '5', '23.3', '-3']
If you want to capture only floats-
>>>filter (bool,[i if isinstance(eval(i),float) else None for i in filter(lambda x: re.findall(r'[-+]?\d+[\.]?\d*',x),a)])
>>>['23.3']
If you want to capture only int-
>>>filter (bool,[i if isinstance(eval(i),int) else None for i in filter(lambda x: re.findall(r'[-+]?\d+[\.]?\d*',x),a)])
>>>['2', '3', '4', '5', '-3']
If i have a few lists
I am able to combine list 1 with list 2 however, I have not managed to succeed to combine the other lists.
def alternator():
iets = []
for i in range(len(list2)):
something += [list1[i]]
something +=[list2[i]]
result = something
result_weaver(result)
def result(x):
list31 = list3
if len(list3) < len(x) :
while len(list31) != len(x):
list31 += '-'
I decided to add '-' in order to make sure the lengths of both lists were equal, so the for loop could go to work.
Does anyone have better ideas on how to program this ?
Use itertools.zip_longest() here:
try:
from itertools import zip_longest
except ImportError:
# Python 2
from itertools import izip_longest as zip_longest
def alternate(list1, list2):
return [v for v in sum(zip_longest(list1, list2), ()) if v is not None]
The zip_longest() call adds None placeholders (similar to your own attempt to add - characters), which we need to remove again from the sum() output after zipping.
Demo:
>>> alternate(list1, list2)
['1', '5', '2', '6', '3', '7', '8']
>>> alternate(alternate(list1, list2), list3)
['1', '9', '5', '2', '6', '3', '7', '8']
Sorry for the very basic question, but this is actually a 2-part question:
Given a list, I need to replace the values of '?' with 'i' and the 'x' with an integer, 10. The list does not always have the same number of elements, so I need a loop that permits me to do this.
a = ['1', '7', '?', '8', '5', 'x']
How do I grab the index of where the value is equal to '?'. It'd be nice if this show me how I could grab all the index and values in a list as well.
Write a function for it and use map() to call it on every element:
def _replaceitem(x):
if x == '?':
return 'i'
elif x == 'x':
return 10
else:
return x
a = map(_replaceitem, a)
Note that this creates a new list. If the list is too big or you don't want this for some other reason, you can use for i in xrange(len(a)): and then update a[i] if necessary.
To get (index, value) pairs from a list, use enumerate(a) which returns an iterator yielding such pairs.
To get the first index where the list contains a given value, use a.index('?').
For 1:
for i in range(len(a)):
if a[i] == '?':
a[i] = 'i'
elif a[i] == 'x':
a[i] = 10
For 2, what do you mean by "key"? If you mean index:
index = a.index('?')
Only because no one's mentioned it yet, here's my favourite non-for-loop idiom for performing replacements like this:
>>> a = ['1', '7', '?', '8', '5', 'x']
>>> reps = {'?': 'i', 'x': 10}
>>> b = [reps.get(x,x) for x in a]
>>> b
['1', '7', 'i', '8', '5', 10]
The .get() method is incredibly useful, and scales up better than an if/elif chain.
Start by reading the Built-in Types section of the Library Reference. I think that you are looking for list.index.
it is function called 'index':
>>> a = ['1', '7', '?', '8', '5', 'x']
>>> a.index('?')
2
You can now use lambda
# replace occurrences of ?
a = map(lambda x: i if x == '?' else x, a)
# replace occurrences of x
a = list(map(lambda x: 10 if x == 'x' else x, a))
a = ['1', '7', '?', '8', '5', 'x']
for index, item in enumerate(a):
if item == "?":
a[index] = "i"
elif item == "x":
a[index = 10
print a