Related
i am refreshing my python (2.7) and i am discovering iterators and generators.
As i understood, they are an efficient way of navigating over values without consuming too much memory.
So the following code do some kind of logical indexing on a list:
removing the values of a list L that triggers a False conditional statement represented here by the function f.
I am not satisfied with my code because I feel this code is not optimal for three reasons:
I read somewhere that it is better to use a for loop than a while loop.
However, in the usual for i in range(10), i can't modify the value of 'i' because it seems that the iteration doesn't care.
Logical indexing is pretty strong in matrix-oriented languages, and there should be a way to do the same in python (by hand granted, but maybe better than my code).
Third reason is just that i want to use generator/iterator on this example to help me understand.
Third reason is just that i want to use generator/iterator on this example to help me understand.
TL;DR : Is this code a good pythonic way to do logical indexing ?
#f string -> bool
def f(s):
return 'c' in s
L=['','a','ab','abc','abcd','abcde','abde'] #example
length=len(L)
i=0
while i < length:
if not f(L[i]): #f is a conditional statement (input string output bool)
del L[i]
length-=1 #cut and push leftwise
else:
i+=1
print 'Updated list is :', L
print length
This code has a few problems, but the main one is that you must never modify a list you're iterating over. Rather, you create a new list from the elements that match your condition. This can be done simply in a for loop:
newlist = []
for item in L:
if f(item):
newlist.append(item)
which can be shortened to a simple list comprehension:
newlist = [item for item in L if f(item)]
It looks like filter() is what you're after:
newlist = filter(lambda x: not f(x), L)
filter() filters (...) an iterable and only keeps the items for which a predicate returns True. In your case f(..) is not quite the predicate but not f(...).
Simpler:
def f(s):
return 'c' not in s
newlist = filter(f, L)
See: https://docs.python.org/2/library/functions.html#filter
Never modify a list with del, pop or other methods that mutate the length of the list while iterating over it. Read this for more information.
The "pythonic" way to filter a list is to use reassignment and either a list comprehension or the built-in filter function:
List comprehension:
>>> [item for item in L if f(item)]
['abc', 'abcd', 'abcde']
i want to use generator/iterator on this example to help me understand
The for item in L part is implicitly making use of the iterator protocol. Python lists are iterable, and iter(somelist) returns an iterator .
>>> from collections import Iterable, Iterator
>>> isinstance([], Iterable)
True
>>> isinstance([], Iterator)
False
>>> isinstance(iter([]), Iterator)
True
__iter__ is not only being called when using a traditional for-loop, but also when you use a list comprehension:
>>> class mylist(list):
... def __iter__(self):
... print('iter has been called')
... return super(mylist, self).__iter__()
...
>>> m = mylist([1,2,3])
>>> [x for x in m]
iter has been called
[1, 2, 3]
Filtering:
>>> filter(f, L)
['abc', 'abcd', 'abcde']
In Python3, use list(filter(f, L)) to get a list.
Of course, to filter a list, Python needs to iterate over it, too:
>>> filter(None, mylist())
iter has been called
[]
"The python way" to do it would be to use a generator expression:
# list comprehension
L = [l for l in L if f(l)]
# alternative generator comprehension
L = (l for l in L if f(l))
It depends on your context if a list or a generator is "better" (see e.g. this so question). Because your source data is coming from a list, there is no real benefit of using a generator here.
For simply deleting elements, especially if the original list is no longer needed, just iterate backwards:
Python 2.x:
for i in xrange(len(L) - 1, -1, -1):
if not f(L[i]):
del L[i]
Python 3.x:
for i in range(len(L) - 1, -1, -1):
if not f(L[i]):
del L[i]
By iterating from the end, the "next" index does not change after deletion and a for loop is possible. Note that you should use the xrange generator in Python 2, or the range generator in Python 3, to save memory*.
In cases where you must iterate forward, use your given solution above.
*Note that Python 2's xrange will break if there are >= 2 ** 32 - 1 elements. Python 3's range, as well as the less efficient Python 2's range do not have this limitation.
This question already has answers here:
Python for-in loop preceded by a variable [duplicate]
(5 answers)
Closed 8 years ago.
I am reading an article about python removing duplicate element in a list.
there is a function defined as:
def f8(seq): # Dave Kirby
# Order preserving
seen = set()
return [x for x in seq if x not in seen and not seen.add(x)]
However, i don't really understand the syntax for
[x for x in seq if x not in seen and not seen.add(x)]
what is this syntax ? how do I read it?
thank you.
Firstly list comprehensions are usually easy to read, here is a simple example:
[x for x in seq if x != 2]
translates to:
result = []
for x in seq:
if x != 2:
result.append(x)
The reason why you can't read this code is because it is not readable and hacky code as I stated in this question:
def f8(seq):
seen = set()
return [x for x in seq if x not in seen and not seen.add(x)]
translates to:
def f8(seq):
seen = set()
result = []
for x in seq:
if x not in seen and not seen.add(x): # not seen.add(...) always True
result.append(x)
and relies on the fact that set.add is an in-place method that always returns None so not None evaluates to True.
>>> s = set()
>>> y = s.add(1) # methods usually return None
>>> print s, y
set([1]) None
The reason why the code has been written this way is to sneakily take advantage of Python's list comprehension speed optimizations.
Python methods will usually return None if they modify the data structure (pop is one of the exceptions)
I also noted that the current accepted way of doing this (2.7+) which is more readable and doesn't utilize a hack is as follows:
>>> from collections import OrderedDict
>>> items = [1, 2, 0, 1, 3, 2]
>>> list(OrderedDict.fromkeys(items))
[1, 2, 0, 3]
Dictionary keys must be unique, therefore the duplicates are filtered out.
It is called a list comprehension, they provide a syntactically more compact and more efficient way of writing a normal for-loop based solution.
def f8(seq): # Dave Kirby
# Order preserving
seen = set()
return [x for x in seq if x not in seen and not seen.add(x)]
The above list comprehension is roughly equivalent to:
def f8(seq):
seen = set()
lis =[]
for x in seq:
if x not in seen:
lis.append(x)
seen.add(x)
return lis
The construct is called a list comprehension
[x for x in seq if some_condition]. In this case the condition is that x isn't already in the resulting list. You can't inspect the result of a list comprehension while you are running it, so it keeps track of the items that are in there using a set called seen
This condition here is a bit tricky because it relies on a side-effect
not in seen and not seen.add(x)
seen.add() always returns None. If the item is in seen,
not in seen is False, so the and shortcircuits.
If the item is not in seen,
not in seen is True and not seen.add(x) is also True, so the item is included, and as a side-effect, it is added to the seen set
While, this type of thing can be fun, it's not a particularly clear way to express the intent.
I think the less tricky way is much more readable
def f8(seq):
seen = set()
result = []
for x in seq:
if x not in seen:
result.append(x)
seen.add(x)
return result
I am using generators to perform searches in lists like this simple example:
>>> a = [1,2,3,4]
>>> (i for i, v in enumerate(a) if v == 4).next()
3
(Just to frame the example a bit, I am using very much longer lists compared to the one above, and the entries are a little bit more complicated than int. I do it this way so the entire lists won't be traversed each time I search them)
Now if I would instead change that to i == 666, it would return a StopIteration because it can't find any 666 entry in a.
How can I make it return None instead? I could of course wrap it in a try ... except clause, but is there a more pythonic way to do it?
If you are using Python 2.6+ you should use the next built-in function, not the next method (which was replaced with __next__ in 3.x). The next built-in takes an optional default argument to return if the iterator is exhausted, instead of raising StopIteration:
next((i for i, v in enumerate(a) if i == 666), None)
You can chain the generator with (None,):
from itertools import chain
a = [1,2,3,4]
print chain((i for i, v in enumerate(a) if v == 6), (None,)).next()
but I think a.index(2) will not traverse the full list, when 2 is found, the search is finished. you can test this:
>>> timeit.timeit("a.index(0)", "a=range(10)")
0.19335955439601094
>>> timeit.timeit("a.index(99)", "a=range(100)")
2.1938486138533335
I'm looking for a concise and functional style way to apply a function to one element of a tuple and return the new tuple, in Python.
For example, for the following input:
inp = ("hello", "my", "friend")
I would like to be able to get the following output:
out = ("hello", "MY", "friend")
I came up with two solutions which I'm not satisfied with.
One uses a higher-order function.
def apply_at(arr, func, i):
return arr[0:i] + [func(arr[i])] + arr[i+1:]
apply_at(inp, lambda x: x.upper(), 1)
One uses list comprehensions (this one assumes the length of the tuple is known).
[(a,b.upper(),c) for a,b,c in [inp]][0]
Is there a better way? Thanks!
Here is a version that works on any iterable and returns a generator:
>>> inp = ("hello", "my", "friend")
>>> def apply_nth(fn, n, iterable):
... return (fn(x) if i==n else x for (i,x) in enumerate(iterable))
...
>>> tuple(apply_nth(str.upper, 1, inp))
('hello', 'MY', 'friend')
You can extend this so that instead of one position you can give it a list of positions:
>>> def apply_at(fn, pos_lst, iterable):
... pos_lst = set(pos_lst)
... return (fn(x) if i in pos_lst else x for (i,x) in enumerate(iterable))
...
>>> ''.join(apply_at(str.upper, [2,4,6,8], "abcdefghijklmno"))
'abCdEfGhIjklmno'
I commented in support of your first snippet, but here are a couple other ways for the record:
(lambda (a,b,c): [a,b.upper(),c])(inp)
(Won't work in Python 3.x.) And:
[inp[0], inp[1].upper(), inp[2]]
>>> inp = "hello", "my", "friend"
>>> index = 1
>>> inp[:index] + ( str.upper(inp[index]),) + inp[index + 1:]
('hello', 'MY', 'friend')
Seems simple, the only thing you may need to know is that to make a single element tuple, do (elt,)
Maybe some' like this?
>>>inp = ("hello", "my", "friend")
>>>out = tuple([i == 1 and x.upper() or x for (x,i) in zip(t,range(len(t)))])
>>> out
('hello', 'MY', 'friend')
Note: rather than (x,i) in zip(t, range(len(t))) I should have thought of using the enumerate function : (i,x) in enumerate(t)
Making it a bit more general:
Rather than hard-coding the 1, we can place it in a variable.
Also, by using a tuple for that purpose, we can apply the function to elements at multiple indexes.
>>>inp = ("hello", "my", "friend")
>>>ix = (0,2)
>>>out = tuple([i in ix and x.upper() or x for (i, x) in enumerate(t)])
>>> out
('HELLO', 'my', 'FRIEND')
Also, we can "replace" the zip()/enumerate() by map(), in something like
out = tuple(map(lambda x,i : i == 1 and x.upper() or x, inp, range(len(inp)) ) )
Edit: (addressing comment about specifying the function to apply):
Could be something as simple as:
>>> f = str.upper # or whatever function taking a single argument
>>> out = tuple(map(lambda x,i : i == 1 and f(x) or x, inp, range(len(inp)) ) )
Since we're talking about applying any function, we should mention the small caveat with the condition and if_true or if_false construct which is not exactly a substitute for the if/else ternary operator found in other languages. The limitation is that the function cannot return a value which is equivalent to False (None, 0, 0.0, '' for example). A suggestion to avoid this problem, is, with Python 2.5 and up, to use the true if-else ternary operator, as shown in Dave Kirby's answer (note the when_true if condition else when_false syntax of this operator)
I don't understand if you want to apply a certain function to every element in the tuple that passes some test, or if you would like it to apply the function to any element present at a certain index of the tuple. So I have coded both algorithms:
This is the algorithm (coded in Python) that I would use to solve this problem in a functional language like scheme:
This function will identify the element identifiable by id and apply func to it and return a list with that element changed to the output of func. It will do this for every element identifiable as id:
def doSomethingTo(tup, id):
return tuple(doSomethingToHelper(list(tup), id))
def doSomethingToHelper(L, id):
if len(L) == 0:
return L
elif L[0] == id:
return [func(L[0])] + doSomethingToHelper(L[1:], id)
else:
return [L[0]] + doSomethingToHelper(L[1:], id)
This algorithm will find the element at the index of the tuple and apply func to it, and stick it back into its original index in the tuple
def doSomethingAt(tup, i):
return tuple(doSomethingAtHelper(list(tup), i, 0))
def doSomethingAtHelper(L, index, i):
if len(L) == 0:
return L
elif i == index:
return [func(L[0])] + L[1:]
else:
return [L[0]] + doSomethingAtHelper(L[1:], index, i+1)
i also like the answer that Dave Kirby gave. however, as a public service announcement, i'd like to say that this is not a typical use case for tuples -- these are data structures that originated in Python as a means to move data (parameters, arguments) to and from functions... they were not meant for the programmer to use as general array-like data structures in applications -- this is why lists exist. naturally, if you're needing the read-only/immutable feature of tuples, that is a fair argument, but given the OP question, this should've been done with lists instead -- note how there is extra code to either pull the tuple apart and put the resulting one together and/or the need to temporarily convert to a list and back.
I'm wondering if there's a reason that there's no first(iterable) in the Python built-in functions, somewhat similar to any(iterable) and all(iterable) (it may be tucked in a stdlib module somewhere, but I don't see it in itertools). first would perform a short-circuit generator evaluation so that unnecessary (and a potentially infinite number of) operations can be avoided; i.e.
def identity(item):
return item
def first(iterable, predicate=identity):
for item in iterable:
if predicate(item):
return item
raise ValueError('No satisfactory value found')
This way you can express things like:
denominators = (2, 3, 4, 5)
lcd = first(i for i in itertools.count(1)
if all(i % denominators == 0 for denominator in denominators))
Clearly you can't do list(generator)[0] in that case, since the generator doesn't terminate.
Or if you have a bunch of regexes to match against (useful when they all have the same groupdict interface):
match = first(regex.match(big_text) for regex in regexes)
You save a lot of unnecessary processing by avoiding list(generator)[0] and short-circuiting on a positive match.
In Python 2, if you have an iterator, you can just call its next method. Something like:
>>> (5*x for x in xrange(2,4)).next()
10
In Python 3, you can use the next built-in with an iterator:
>>> next(5*x for x in range(2,4))
10
There's a Pypi package called “first” that does this:
>>> from first import first
>>> first([0, None, False, [], (), 42])
42
Here's how you would use to return the first odd number, for example:
>> first([2, 14, 7, 41, 53], key=lambda x: x % 2 == 1)
7
If you just want to return the first element from the iterator regardless of whether is true or not, do this:
>>> first([0, None, False, [], (), 42], key=lambda x: True)
0
It's a very small package: it only contains this function, it has no dependencies, and it works on Python 2 and 3. It's a single file, so you don't even have to install it to use it.
In fact, here's almost the entire source code (from version 2.0.1, by Hynek Schlawack, released under the MIT licence):
def first(iterable, default=None, key=None):
if key is None:
for el in iterable:
if el:
return el
else:
for el in iterable:
if key(el):
return el
return default
I asked a similar question recently (it got marked as a duplicate of this question by now). My concern also was that I'd liked to use built-ins only to solve the problem of finding the first true value of a generator. My own solution then was this:
x = next((v for v in (f(x) for x in a) if v), False)
For the example of finding the first regexp match (not the first matching pattern!) this would look like this:
patterns = [ r'\d+', r'\s+', r'\w+', r'.*' ]
text = 'abc'
firstMatch = next(
(match for match in
(re.match(pattern, text) for pattern in patterns)
if match),
False)
It does not evaluate the predicate twice (as you would have to do if just the pattern was returned) and it does not use hacks like locals in comprehensions.
But it has two generators nested where the logic would dictate to use just one. So a better solution would be nice.
There is a "slice" iterator in itertools. It emulates the slice operations that we're familiar with in python. What you're looking for is something similar to this:
myList = [0,1,2,3,4,5]
firstValue = myList[:1]
The equivalent using itertools for iterators:
from itertools import islice
def MyGenFunc():
for i in range(5):
yield i
mygen = MyGenFunc()
firstValue = islice(mygen, 0, 1)
print firstValue
There's some ambiguity in your question. Your definition of first and the regex example imply that there is a boolean test. But the denominators example explicitly has an if clause; so it's only a coincidence that each integer happens to be true.
It looks like the combination of next and itertools.ifilter will give you what you want.
match = next(itertools.ifilter(None, (regex.match(big_text) for regex in regexes)))
Haskell makes use of what you just described, as the function take (or as the partial function take 1, technically). Python Cookbook has generator-wrappers written that perform the same functionality as take, takeWhile, and drop in Haskell.
But as to why that's not a built-in, your guess is as good as mine.