The following code is for illustrative purposes only.
def get_messages_from_redis():
for item in self.pubsub.listen():
yield (item['channel'], item['data']) # how to ignore the first yield?
I know the following way can ignore the first yield value:
g = get_messages_from_redis()
next(g)
But how to ignore this in get_messages_from_redis()?
(counter can be used to control whether to yield, but is there a better way?)
Iterate inside your function before yielding. I'm not sure what your iterable is exactly, but here's a generic example assuming a list.
def get_messages_from_redis():
for item in self.pubsub.listen()[1:]:
yield item['channel'], item['data']
For a more universal solution, you could create an iterator of your iterable, iterate over the first one, then loop and yield from there. Note: This is mostly for broader coverage, I'm not sure what negative consequences this might have with certain iterables.
def iter_skip_first(i):
iterable = iter(i)
next(iterable)
for i in iterable:
yield i
li = [1, 2, 3, 4]
d = {"one": 1, "two": 2, "three": 3, "four": 4}
print(*iter_skip_first(li))
print(*iter_skip_first(d))
So i've been learning about generators and came across this code
def flatten(a):
for i in a:
try:
yield from flatten(i)
except TypeError:
yield i
a = list(flatten([1, 2, 2, 3, [1, 2, 68,[[98, 85 ,97],67]]]))
From what i understand right now the equivalent of calling a function back inside of the function itself is done with yield from flatten(i) in this case. I tried changing the code to yield flatten(i) but then a lot of weird things happen, instead of flattening my list the code returns a list of generator objects. Is there a reason for that? Is there something i am missing about generator functions?
I have generator like
def not_nones(some_iterable):
for item in some_iterable:
if item is not None:
yield item
But since "flat is better than nested", I would like to do this in one line, like:
def not_nones(some_iterable):
for item in some_iterable:
yield item if item is not None else None
But this will actually make None an item of the generator.
Is it possible to yield nothing in a one-liner anyway?
You could just return a generator expression:
def not_nones(iterable):
return (item for item in iterable if item is not None)
Or for a real one-liner:
not_nones = lambda it: (i for i in it if i is not None)
which at this point is getting more into code-golf territory.
But really, there's not much wrong with your current code; it does what it needs to do, in a reasonable way. Your code is what I would have written in this situation.
You could use itertools.ifilter(). Given the right predicate function it provides exactly the functionality you are implementing here.
Example:
import itertools
# make up data
l = [1, None, 2, None, 3]
# predicate function
not_none = lambda x: x is not None
# filter out None values
not_nones = itertools.ifilter(not_none, l)
print list(not_nones) # prints [1, 2, 3]
For reference:
https://docs.python.org/2/library/itertools.html#itertools.ifilter
I want to get the only Non-None element from this list:
L = [None, [None,None], [None, <__main__.Car object at 0x02A11550>], [None, None, None], None]
I've tried
L = [x for x in L if x is not None]
But the result is
[[None, None], [None, <__main__.Car object at 0x02A11550>], [None, None, None]]
Deleting only the None that are not inside of any list.
Is there any way to clean the whole list? so the output is
<__main__.Car object at 0x02A11550>
def flatten(lst):
for element in lst:
if hasattr(element,"__iter__"):
yield from flatten(element)
elif not element is None:
yield element
new_list = flatten(L)
I'll break this down for you, first starting with generators. The yield keyword is sister to return, but with much different functionality. Both are used to bring values out of a function into its calling scope, but yield allows you to jump back into the function afterwards! As an example, below is a generator that accepts a list full of numbers and produces the square for each number in the list.
def example_generator(number_list):
for number in number_list:
yield number**2
>>> gen = example_generator([1,2,3])
>>> type(gen)
<class 'generator'>
>>> next(gen) # next() is used to get the next value from an iterator
1
>>> next(gen)
4
>>> next(gen)
9
>>> next(gen)
Traceback (most recent call last):
File "<pyshell#6>", line 1, in <module>
next(gen)
StopIteration
Generators are one-time use, however. As you can see, after I reached the end of the generator, it threw an exception StopIteration. If I built it again and ran through it with a loop, then tried to run through it AGAIN...
>>> gen = example_generator([1,2,3]) # remember this is a new generator, we JUST made it
>>> for item in gen:
... print(item)
1
4
9
>>> for item in gen:
... print(item)
>>>
It doesn't do anything the second time. The generator is exhausted. That's the downside -- the upside is that it's generally much faster and more memory-efficient to use generators instead of lists.
yield also allows you to use another keyword: from. That's what I did there in case of a nested list (hasattr(element,"__iter__") just means that the element has an attribute .__iter__, which means it can be iterated upon using something like a for loop). You give yield from another generator, and it yields each element from THAT generator in turn. For example:
def flatten_lite(lst):
for element in lst:
if type(element) is list: # more readable, IMO
yield from flatten_lite(element)
else:
yield element
a = flatten_lite([1,2,3,[4,5,6,[7],8],9])
Here's what it does in turn:
for element in [1,2,3,[4,5,6,[7],8],9]:
# element == 1
if element is of type list: # it's not, skip this
else: yield element # which is 1
:: NEXT ITERATION ::
# element == 2, same as before
:: NEXT ITERATION ::
# element == 3, same as before
:: NEXT ITERATION ::
# element == [4,5,6,[7],8]
if element is of type list: # it is!!
yield from flatten_lite([4,5,6,[7],8])
:: STOP EXECUTION UNTIL WE GET A VALUE FROM THAT NEW GENERATOR ::
>>> NEW GENERATOR
for element in [4,5,6,[7],8]:
# element is 4
yield 4
:: THE OUTER GENERATOR YIELDS 4 ::
:: NEXT ITERATION ::
# element is 5
yield 5
:: THE OUTER GENERATOR YIELDS 4 ::
:: NEXT ITERATION ::
# element is 6
yield 6
:: THE OUTER GENERATOR YIELDS 4 ::
:: NEXT ITERATION ::
# element is [7]
if element is of type list # [7] is a list!
yield from flatten_lite([7])
:: STOP EXECUTION UNTIL WE GET A VALUE FROM THAT NEW GENERATOR ::
# etc etc
So basically the code above says (in pseudocode):
flatten is a function that accepts parameter: lst
for each element in lst:
if element can be iterated on:
yield every element in turn from the generator created
by this function called on the element instead of the
main list
if it's not, and isn't None:
yield element
When you call it, it builds a generator that can be iterated upon. To make it into a formal list, you'll have to do list(flatten(L)), but in most cases you don't need that.
Is that any clearer?
Another slightly more modular approach:
def flatten(l):
""" http://stackoverflow.com/questions/2158395/flatten-an-irregular-list-of-lists-in-python/2158532#2158532 """
for el in l:
if isinstance(el, collections.Iterable) and not isinstance(el, basestring):
for sub in flatten(el):
yield sub
else:
yield el
filter(None,flatten(L)) #wrap with `list` in python 3.x
A generic flatten function is something you should keep in your toolbox, since (so far) it's not something you can find in the standard library, and it comes up occasionally.
Just for fun, how about:
from itertools import chain, ifilterfalse
result = list(ifilterfalse(lambda x: x is None, chain(*[x for x in L if x is not None])))
This will return a list with only the Car element present. It will generalize to return a list with any non-None element.
In Python 3.x I think you swap ifilterfalse for filterfalse and it works the same.
chain() is designed to flatten a list of lists for iteration. ifilterfalse can work directly on the chain returned. ifilterfalse gets rid of elements that match the predicate specified by the lambda function.
Note that if you have strings in L, chain() will essentially break the strings up into individual elements. If that is a problem for you, see this other SO post.
Another implementation which avoids the problem of non-iterables at the base level:
result = list(ifilterfalse(lambda x: x is None, chain(*[x if hasattr(x, '__iter__') else [x] for x in L if x is not None])))
I'm told this may not work in Python 3 because of how str is implemented there. Anyway I'm only posting these ideas so that you are aware of functionality already available in the Python standard library under itertools. Have fun learning Python!
I am using generators to perform searches in lists like this simple example:
>>> a = [1,2,3,4]
>>> (i for i, v in enumerate(a) if v == 4).next()
3
(Just to frame the example a bit, I am using very much longer lists compared to the one above, and the entries are a little bit more complicated than int. I do it this way so the entire lists won't be traversed each time I search them)
Now if I would instead change that to i == 666, it would return a StopIteration because it can't find any 666 entry in a.
How can I make it return None instead? I could of course wrap it in a try ... except clause, but is there a more pythonic way to do it?
If you are using Python 2.6+ you should use the next built-in function, not the next method (which was replaced with __next__ in 3.x). The next built-in takes an optional default argument to return if the iterator is exhausted, instead of raising StopIteration:
next((i for i, v in enumerate(a) if i == 666), None)
You can chain the generator with (None,):
from itertools import chain
a = [1,2,3,4]
print chain((i for i, v in enumerate(a) if v == 6), (None,)).next()
but I think a.index(2) will not traverse the full list, when 2 is found, the search is finished. you can test this:
>>> timeit.timeit("a.index(0)", "a=range(10)")
0.19335955439601094
>>> timeit.timeit("a.index(99)", "a=range(100)")
2.1938486138533335