Related
i am refreshing my python (2.7) and i am discovering iterators and generators.
As i understood, they are an efficient way of navigating over values without consuming too much memory.
So the following code do some kind of logical indexing on a list:
removing the values of a list L that triggers a False conditional statement represented here by the function f.
I am not satisfied with my code because I feel this code is not optimal for three reasons:
I read somewhere that it is better to use a for loop than a while loop.
However, in the usual for i in range(10), i can't modify the value of 'i' because it seems that the iteration doesn't care.
Logical indexing is pretty strong in matrix-oriented languages, and there should be a way to do the same in python (by hand granted, but maybe better than my code).
Third reason is just that i want to use generator/iterator on this example to help me understand.
Third reason is just that i want to use generator/iterator on this example to help me understand.
TL;DR : Is this code a good pythonic way to do logical indexing ?
#f string -> bool
def f(s):
return 'c' in s
L=['','a','ab','abc','abcd','abcde','abde'] #example
length=len(L)
i=0
while i < length:
if not f(L[i]): #f is a conditional statement (input string output bool)
del L[i]
length-=1 #cut and push leftwise
else:
i+=1
print 'Updated list is :', L
print length
This code has a few problems, but the main one is that you must never modify a list you're iterating over. Rather, you create a new list from the elements that match your condition. This can be done simply in a for loop:
newlist = []
for item in L:
if f(item):
newlist.append(item)
which can be shortened to a simple list comprehension:
newlist = [item for item in L if f(item)]
It looks like filter() is what you're after:
newlist = filter(lambda x: not f(x), L)
filter() filters (...) an iterable and only keeps the items for which a predicate returns True. In your case f(..) is not quite the predicate but not f(...).
Simpler:
def f(s):
return 'c' not in s
newlist = filter(f, L)
See: https://docs.python.org/2/library/functions.html#filter
Never modify a list with del, pop or other methods that mutate the length of the list while iterating over it. Read this for more information.
The "pythonic" way to filter a list is to use reassignment and either a list comprehension or the built-in filter function:
List comprehension:
>>> [item for item in L if f(item)]
['abc', 'abcd', 'abcde']
i want to use generator/iterator on this example to help me understand
The for item in L part is implicitly making use of the iterator protocol. Python lists are iterable, and iter(somelist) returns an iterator .
>>> from collections import Iterable, Iterator
>>> isinstance([], Iterable)
True
>>> isinstance([], Iterator)
False
>>> isinstance(iter([]), Iterator)
True
__iter__ is not only being called when using a traditional for-loop, but also when you use a list comprehension:
>>> class mylist(list):
... def __iter__(self):
... print('iter has been called')
... return super(mylist, self).__iter__()
...
>>> m = mylist([1,2,3])
>>> [x for x in m]
iter has been called
[1, 2, 3]
Filtering:
>>> filter(f, L)
['abc', 'abcd', 'abcde']
In Python3, use list(filter(f, L)) to get a list.
Of course, to filter a list, Python needs to iterate over it, too:
>>> filter(None, mylist())
iter has been called
[]
"The python way" to do it would be to use a generator expression:
# list comprehension
L = [l for l in L if f(l)]
# alternative generator comprehension
L = (l for l in L if f(l))
It depends on your context if a list or a generator is "better" (see e.g. this so question). Because your source data is coming from a list, there is no real benefit of using a generator here.
For simply deleting elements, especially if the original list is no longer needed, just iterate backwards:
Python 2.x:
for i in xrange(len(L) - 1, -1, -1):
if not f(L[i]):
del L[i]
Python 3.x:
for i in range(len(L) - 1, -1, -1):
if not f(L[i]):
del L[i]
By iterating from the end, the "next" index does not change after deletion and a for loop is possible. Note that you should use the xrange generator in Python 2, or the range generator in Python 3, to save memory*.
In cases where you must iterate forward, use your given solution above.
*Note that Python 2's xrange will break if there are >= 2 ** 32 - 1 elements. Python 3's range, as well as the less efficient Python 2's range do not have this limitation.
primes = [2,3,5,7..] (prime numbers)
map(lambda x:print(x),primes)
It does not print anything.
Why is that?
I've tried
sys.stdout.write(x)
too, but doesn't work either.
Since lambda x: print(x) is a syntax error in Python < 3, I'm assuming Python 3. That means map returns a generator, meaning to get map to actually call the function on every element of a list, you need to iterate through the resultant generator.
Fortunately, this can be done easily:
list(map(lambda x:print(x),primes))
Oh, and you can get rid of the lambda too, if you like:
list(map(print,primes))
But, at that point you are better off with letting print handle it:
print(*primes, sep='\n')
NOTE: I said earlier that '\n'.join would be a good idea. That is only true for a list of str's.
This works for me:
>>> from __future__ import print_function
>>> map(lambda x: print(x), primes)
2
3
5
7
17: [None, None, None, None]
Are you using Python 2.x where print is a statement, not a function?
Alternatively, you can unpack it by putting * before map(...) like the following
[*map(...)]
or
{*map(...)}
Choose the output you desire, a list or a dictionary.
Another reason why you could be seeing this is that you're not evaluating the results of the map function. It returns a generator (an iterable) that evaluates your function lazily and not an actual list.
primes = [2,3,5,7]
map(print, primes) # no output, because it returns a generator
primes = [2,3,5,7]
for i in map(print, primes):
pass # prints 2,3,5,7
Alternately, you can do list(map(print, primes)) which will also force the generator to be evaluated and call the print function on each member of your list.
I am trying to use print inside lambda. Something like that:
lambda x: print x
I understand, that in Python 2.7 print is not a function. So, basically, my question is: Is there a pretty way to use print as function in Python 2.7?
You can import print_function from the __future__ and use it as a function like this
from __future__ import print_function
map(print, [1, 2, 3])
# 1
# 2
# 3
The question is about Python 2, but I ended up here from Google trying to use the print function inside a lambda in Python 3. I'm adding this answer for context for others that come here for the same.
If you only want to see the code that works and not how I arrived there, skip to the last code sample at the bottom. I wanted to clearly document what didn't work for learning purposes.
Desired result
Let's suppose you want to define a lambda print_list that prints each item of a list with a newline in between.
lst = [1, 2, 3]
print_list = lambda lst: ...
The desired output is:
1
2
3
And there should be no unused return value.
Attempt 1 - A map doesn't evaluate the print function in Python 3
To start, here's what doesn't work well in Python 3:
map(print, lst)
However, the output is somewhat counterintuitively not printed lines, because the map call in Python 3 returns an iterator instead of an evaluated list.
Output:
n/a
Return value:
<map at 0x111b3a6a0>
Attempt 2 - Evaluate the map iterator
You can realize the printing by passing the map result to list(...), which produces the ideal output, but has the side effect of returning a list of nulls (as evaluated in the REPL).
list(map(print, lst))
Output:
1
2
3
Return value:
[None, None, None]
You could workaround this by using the underscore throwaway variable convention:
_ = list(map(print, lst))
A similar approach is calling print inside a list comprehension:
[print(i) for i in lst]
I don't love these approaches because they both still generate an unused return value.
Attempt 3 - Apply the unpacking operator to the map iterator
Like this:
[*map(print, [1, 2, 3])]
(This still returns a list of nulls which is non-ideal.)
In the comments above #thefourtheye suggests using a one-line for loop:
for item in [1, 2, 3]: print(item)
This works fine for most cases and avoids the side effect. Attempting to put this in a lambda throws a SyntaxError. I tried wrapping it in parens without success; though there is probably a way to achieve this, I haven't figured it out.
(SOLUTION!) Attempt 4 - Apply the unpacking operator inside of the print call
The answer I arrived at is to explode the list inside the print call alongside using the separator arg:
print(*lst, sep='\n')
Output:
1
2
3
This produces the intended result without a return value.
Finally, let's wrap it up in a lambda to use as desired:
print_list = lambda lst: print(*lst, sep='\n')
print_list([1, 2, 3])
This was the best solution for my use case in Python 3.
Related questions
Why map(print, a_list) doesn't work?
Print doesnt print when it's in map, Python
If you don't want to import from __future__ you can just make the lambda write to the standard output:
>>>import sys
>>>l = lambda x : sys.stdout.write(x)
>>>l('hi')
'hi'
I guess there is another scenario people may be interested in: "print out the intermediate step value of the lambda function variables"
For instance, say I want to find out the charset of a collection of char list:
In [5]: instances = [["C","O","c","1","c","c","c","c","c","1","O","C","C","N","C"],
...: ["C","C","O","C","(","=","O",")","C","C","(","=","O",")","c"],
...: ["C","N","1","C","C","N","(","C","c","2","c","c","c","(","N"],
...: ["C","l","c","1","c","c","c","2","c","(","N","C","C","C","["],
...: ["C","C","c","1","c","c","c","(","N","C","(","=","S",")","N"]]
one way of doing this is to use reduce:
def build_charset(instances):
return list(functools.reduce((lambda x, y: set(y) | x), instances, set()))
In this function, reduce takes a lambda function with two variables x, y, which at the beginning I thought it would be like x -> instance, and y -> set(). But its results give a different story, so I want to print their value on the fly. lambda function, however, only take a single expression, while the print would introduce another one.
Inspired by set(y) | x, I tried this one and it worked:
lambda x, y: print(x, y) or set(y) | x
Note that print() is of NoneType, so you cannot do and, xor these kinds of operation that would change the original value. But or works just fine in my case.
Hope this would be helpful to those who also want to see what's going on during the procedure.
I created a line that appends an object to a list in the following manner
>>> foo = list()
>>> def sum(a, b):
... c = a+b; return c
...
>>> bar_list = [9,8,7,6,5,4,3,2,1,0]
>>> [foo.append(sum(i,x)) for i, x in enumerate(bar_list)]
[None, None, None, None, None, None, None, None, None, None]
>>> foo
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]
>>>
The line
[foo.append(sum(i,x)) for i, x in enumerate(bar_list)]
would give a pylint W1060 Expression is assigned to nothing, but since I am already using the foo list to append the values I don't need to assing the List Comprehension line to something.
My questions is more of a matter of programming correctness
Should I drop list comprehension and just use a simple for expression?
>>> for i, x in enumerate(bar_list):
... foo.append(sum(i,x))
or is there a correct way to use both list comprehension an assign to nothing?
Answer
Thank you #user2387370, #kindall and #Martijn Pieters. For the rest of the comments I use append because I'm not using a list(), I'm not using i+x because this is just a simplified example.
I left it as the following:
histogramsCtr = hist_impl.HistogramsContainer()
for index, tupl in enumerate(local_ranges_per_histogram_list):
histogramsCtr.append(doSubHistogramData(index, tupl))
return histogramsCtr
Yes, this is bad style. A list comprehension is to build a list. You're building a list full of Nones and then throwing it away. Your actual desired result is a side effect of this effort.
Why not define foo using the list comprehension in the first place?
foo = [sum(i,x) for i, x in enumerate(bar_list)]
If it is not to be a list but some other container class, as you mentioned in a comment on another answer, write that class to accept an iterable in its constructor (or, if it's not your code, subclass it to do so), then pass it a generator expression:
foo = MyContainer(sum(i, x) for i, x in enumerate(bar_list))
If foo already has some value and you wish to append new items:
foo.extend(sum(i,x) for i, x in enumerate(bar_list))
If you really want to use append() and don't want to use a for loop for some reason then you can use this construction; the generator expression will at least avoid wasting memory and CPU cycles on a list you don't want:
any(foo.append(sum(i, x)) for i, x in enumerate(bar_list))
But this is a good deal less clear than a regular for loop, and there's still some extra work being done: any is testing the return value of foo.append() on each iteration. You can write a function to consume the iterator and eliminate that check; the fastest way uses a zero-length collections.deque:
from collections import deque
do = deque([], maxlen=0).extend
do(foo.append(sum(i, x)) for i, x in enumerate(bar_list))
This is actually fairly readable, but I believe it's not actually any faster than any() and requires an extra import. However, either do() or any() is a little faster than a for loop, if that is a concern.
I think it's generally frowned upon to use list comprehensions just for side-effects, so I would say a for loop is better in this case.
But in any case, couldn't you just do foo = [sum(i,x) for i, x in enumerate(bar_list)]?
You should definitely drop the list comprehension. End of.
You are confusing anyone reading your code. You are building a list for the side-effects.
You are paying CPU cycles and memory for building a list you are discarding again.
In your simplified case, you are overlooking the fact you could have used a list comprehension directly:
[sum(i,x) for i, x in enumerate(bar_list)]
I think in Python 3 I'll be able to do:
first, *rest = l
which is exactly what I want, but I'm using 2.6. For now I'm doing:
first = l[0]
rest = l[1:]
This is fine, but I was just wondering if there's something more elegant.
first, rest = l[0], l[1:]
Basically the same, except that it's a oneliner. Tuple assigment rocks.
This is a bit longer and less obvious, but generalized for all iterables (instead of being restricted to sliceables):
i = iter(l)
first = next(i) # i.next() in older versions
rest = list(i)
You can do
first = l.pop(0)
and then l will be the rest. It modifies your original list, though, so maybe it’s not what you want.
Yet another one, working with python 2.7. Just use an intermediate function. Logical as the new behavior mimics what happened for functions parameters passing.
li = [1, 2, 3]
first, rest = (lambda x, *y: (x, y))(*li)
If l is string typeI would suggest:
first, remainder = l.split(None, maxsplit=1)