I am experiencing a bit of confusion with how to place a for loop at the end of a line in python, for instance
for i in x:
print i
produces the expected result but if I run
print i for i in x
I get a syntax error. Could someone explain a little more about how one goes about putting your for loops at the end of a line like this.
In earlier versions of Python, the idea of list comprehensions was introduced to neaten up these two patterns of code:
# Go through each item in a list and run a test on them
# build a new list containing only items that pass the test
results = []
for item in somelist:
if sometest(item):
results.add(item)
and
# build a new list by changing every item in a list in the
# same way
results = []
for item in somelist:
results.add(2 * item)
by adding a new syntax that includes doing all three things in one - changing the items and/or testing them to only include some in the result, and creating the list of the results:
results = [2 * item for item in somelist if sometest(item)]
# results is a list
This feature uses the [] syntax that indicates "list" in Python, and it builds a list right away in memory.
Sometimes you don't want or need the entire list built in memory right away, so later versions of Python introduced generator expressions - the same idea, but they save memory by quickly returning a generator which you can iterate over as if it was a list, and it builds the list as and when you use it.
But they can't have the same syntax and be a direct swap out because they behave slightly differently, so they use () instead of [], e.g.:
somelist = [1,2,3,4,5]
results = (2 * item for item in somelist if sometest(item))
# results is a generator
If you try and call a function with this, you get two layers of parentheses:
function((2 * item for item in somelist))
Which looks silly, so you can leave one out:
function(2 * item for item in somelist)
Which appears to be a standalone backwards for loop on its own, but it actually isn't.
So with parentheses you can write this:
>>> print (item for item in [1,2,3])
<generator object <genexpr> at 0x7fe31b8663c0>
Which is the closest thing to what you wrote that is valid syntax in Python 2.x, but it doesn't do what you expect, and ^^ is what it does and why.
Or this:
>>> print [item for item in [1,2,3]]
[1,2,3]
Which generates a list and prints the list, the other close thing to what you wrote, but still not what you expected.
-- (there really isn't much point in me posting now a bunch other answers have appeared while I was writing this, but eh).
expression for i in x
doesn't mean anything by itself in Python. You can have
(expression for i in x)
[expression for i in x]
But you can't do
[print i for i in x]
because print isn't an expression, it's a statement.
First of all:
this is considered bad style—you're essentially just abusing the list comprehension syntax to get what's effectively a different notation for imperative for loops.
this only works in Python 3 where print is a function by default, or when you do from __future__ import print_function in Python 2.x
However, if you insist on putting the for ... part after the print i part, you can do:
[print(i) for i in x]
(but I'm writing that example for purely "academic" purposes)
P.S. if you let us know what you want to do, we might be able to provide a suitable overall solution, but if you're just asking for the sake of it, then that's it.
You can't. You've probably seen a generator expression, which takes the form x for x in iter. a for loop is slightly different, though you can definitely see for x in iter inside the genexp.
In Python3, you can do:
print(*(i for i in x))
And as #wim points out in the comments, you can make it more "for loopy" by doing
print(*(i for i in x), sep='\n')
You can of course do arbitrary changes since this is a genexp, so e.g. i**2 for i in x will give you the square of each item in x, i for i in x if i%2 will give all odd numbers in x, etc.
This will create a generator of each item in x, then pass each one in turn (using the * assignment the same way *args is built and etc) as separate arguments to print
The only thing I can think of that is equivalent in python 2.x would be this:
print '\n'.join(str(i) for i in x)
But don't do that. The for loop as you had it is much clearer.
If you are just fooling around in interactive interpreter and want to one-liner it for some reason, this will work:
for i in x: print i
But it's violating pep8 so I wouldn't write a line like that into a script.
Related
This question already has answers here:
Generator expressions vs. list comprehensions
(13 answers)
Closed 3 years ago.
I'm some code in my project but I came across one problem which I solved but I'm not getting how it works. When I change the type of Brackets used in code,value in year is different.
when I use square brackets in line 2 at start and end of statement after =
import datetime
years=[x for x in range(2015,datetime.datetime.now().year)]
when I print(years) it gives output [2015,2016,2017,2018]
but when I use round brackets in line 2 like this
years=(x for x in range(1940,datetime.datetime.now().year))
when I print it ,it gives output <generator object <genexpr> at 0x041DB630>
I don't understand why this happens ,can anyone please explain. Thanks
These are two different, though related, constructs.
[x for x in range(2015,datetime.datetime.now().year)]
is known as a list comprehension, whereas
(x for x in range(2015,datetime.datetime.now().year))
is known as a generator expression.
Read more at https://djangostars.com/blog/list-comprehensions-and-generator-expressions/
Generators are functions that can be paused and resumed on the fly, returning an object that can be iterated over. Unlike lists, they are lazy and thus produce items one at a time and only when asked. So they are much more memory efficient when dealing with large datasets.
Just like list comprehensions, generators can also be written in the same manner except they return a generator object rather than a list:
>>> my_list = ['a', 'b', 'c', 'd']
>>> gen_obj = (x for x in my_list)
>>> for val in gen_obj:
... print(val)
...
a
b
c
d
Here are the explanations:
With round brackets it's called a generator expression, where you would have to do list(..) to make it a list and tuple(..) to make it a tuple and so on... more on the documentation
Generator iterators are created by the yield keyword. The real difference between them and ordinary functions is that yield unlike return is both exit and entry point for the function’s body. That means, after each yield call not only the generator returns something but also remembers its state. Calling the next() method brings control back to the generator starting after the last executed yield statement. Each yield statement is executed only once, in the order it appears in the code. After all the yield statements have been executed iteration ends.
With square brackets it's called a list comprehension, where it would give a list, since square brackets are for lists, more on the documentation
A list comprehension follows the form of the mathematical set-builder notation (set comprehension) as distinct from the use of map() and filter() functions.
What you are trying is comprehension and it works by looping or iterating over items and assigning them into a container.
Below is the list comprehension using square brackets:
[thing for thing in things]
But what you have tried is using parentheses which is generator comprehension not tuple comprehension, as parentheses have been kept reserved for generator comprehension, hence:
(thing for thing in things)
will result in a generator iterator, not a tuple. To get tuple iterator use as done below:
tuple(thing for thing in things)
You are creating a generator expression in the 2nd instance.
You would need to wrap it in list() or tuple() to get an iterable output.
While in the 1st instance your generating a list.
You can readmore about the issue Getting <generator object <genexpr>
This is probably simple, but, I cannot find it for some reason. For example:
def fnc(dt):
print dt
return;
#[(fnc(y)) for y in range(5) for x in range(5)]
for x in range(0, 5):
fnc(x)
for y in range(0, 5):
fnc(y)
I would like the commented out line to have similar behaviour with the double nested loop bellow it. Is this possible? I give up, I cannot find it! Thanks for any input.
You have to used nested list comprehensions to achieve the same result:
[(fnc(x),[fnc(y) for y in range(5)]) for x in range(5)]
I used a tuple (fnc(x), [...]) to output x before performing the list comprehension for y.
P.S.: Don't actually use this. Stick to your loops.
You don't need a list comprehension here. List comprehensions are for building lists not for side effects, as you have in your for loop. Any solution that provides the same result using a list comp. (like the one below) will produce a useless list of Nones:
[fnc(y) for x in range(5) if fnc(x) or 1 for y in range(5)]
The code is unpythonic and unreadable. You should never use it. fnc(x) is always evaluated while evaluating the if, and the branch is always taken because it is short-circuited with a truthy value using or, so that the nested loop always executes ∀ x.
The Pythonic way is to use a vanilla for like you've done.
What you can do is probably technically possible (I'm thinking of a class with an overridden iterator that calls func() in the iteration, although I'm not sure if this is actually implementable).
The implementation, however, would be an aberration.
List comprehensions are intended as a fast way to filter, combine and/or process data in a list to generate another one. You should think of them as a way to quickly apply a function to all the data in the list, appending each time the function results to the output list.
This is why there's no syntax to, say, do assignments or external function calls in comprehensions. If you need to call a function in the inner loop before processing the data, you're better off with the nested loop approach (which is also much more readable than anything equivalent hacked to work in a comprehension)
Consider the following example code. The example is given just to highlight the different functionality between map and itertools.imap. What I really want to do cannot be
done with a list comprehension because in my real problem I am not creating a list but filling a larger numpy array with small arrays. So in response to the code below, please do not suggest: [f(x) for x in range(3)]
Example:
g = []
def f(x):
g.append(x)
The results I get with map and itertools.map:
map(f, range(3)) # Results in g = [0,1,2]
itertools.imap(f, range(3)) # Does not change g
I want the g to change as done by the map function.
However, I have heard that map will (or does) behave like
itertools.imap in Python 3. Even though I am using Python 2.7 I want to learn the correct way of using an iterator version of the map. How do I use itertools.imap to achieve
the same outcome as I get with map?
I can do:
b = itertools.imap(f, range(3))
list(b) # This gives g = [0,1,2] if starting with an empty g.
Is this the correct way, or is there a better way?
Thanks in advance.
itertools functions return generators; they only operate when iterated over. So itertools.imap(f, range(3)) won't actually do anything until you run it to completion e.g. with list.
Per http://docs.python.org/2/library/itertools.html#recipes, the most efficient way to consume an iterator is with a zero-length deque:
collections.deque(itertools.imap(f, range(3)), maxlen=0)
However, if you're calling a function for its side effects you should be using an imperative as opposed to functional syntax:
for i in range(3):
f(i)
You are using map() incorrectly. It should not be used for the side effects of the function call, but to transform an iterable.
You can use list(itertools.imap(f, range(3))) to get the behavior you want, but I would argue that you should change your entire approach to use a normal for loop:
for i in range(3):
f(i)
This makes it clear that the return values from the f() calls are not being used.
The difference is that map in Python 2.x is the equivalent of list(map(...)) in Python 3. That's it - nothing special...
I was writing a python function that looked something like this
def foo(some_list):
for i in range(0, len(some_list)):
bar(some_list[i], i)
so that it was called with
x = [0, 1, 2, 3, ... ]
foo(x)
I had assumed that index access of lists was O(1), but was surprised to find that for large lists this was significantly slower than I expected.
My question, then, is how are python lists are implemented, and what is the runtime complexity of the following
Indexing: list[x]
Popping from the end: list.pop()
Popping from the beginning: list.pop(0)
Extending the list: list.append(x)
For extra credit, splicing or arbitrary pops.
there is a very detailed table on python wiki which answers your question.
However, in your particular example you should use enumerate to get an index of an iterable within a loop. like so:
for i, item in enumerate(some_seq):
bar(item, i)
The answer is "undefined". The Python language doesn't define the underlying implementation. Here are some links to a mailing list thread you might be interested in.
It is true that Python's lists have
been implemented as contiguous
vectors in the C implementations of
Python so far.
I'm not saying that the O()
behaviours of these things should be
kept a secret or anything. But you
need to interpret them in the context
of how Python works generally.
Also, the more Pythonic way of writing your loop would be this:
def foo(some_list):
for item in some_list:
bar(item)
Lists are indeed O(1) to index - they are implemented as a vector with proportional overallocation, so perform much as you'd expect. The likely reason you were finding this code slower than you expected is the call to "range(0, len(some_list))".
range() creates a new list of the specified size, so if some_list has 1,000,000 items, you will create a new million item list up front. This behaviour changes in python3 (range is an iterator), to which the python2 equivalent is xrange, or even better for your case, enumerate
if you need index and value then use enumerate:
for idx, item in enumerate(range(10, 100, 10)):
print idx, item
Python list actually nothing but arrays. Thus,
indexing takes O(1)
for pop and append again it should be O(1) as per the docs
Check out following link for details:
http://dustycodes.wordpress.com/2012/03/31/pythons-data-structures-complexity-analysis/
What Python's user-made list-comprehension construction is the most useful?
I have created the following two quantifiers, which I use to do different verification operations:
def every(f, L): return not (False in [f(x) for x in L])
def some(f, L): return True in [f(x) for x in L]
an optimized versions (requres Python 2.5+) was proposed below:
def every(f, L): return all(f(x) for x in L)
def some(f, L): return any(f(x) for x in L)
So, how it works?
"""For all x in [1,4,9] there exists such y from [1,2,3] that x = y**2"""
answer = every([1,4,9], lambda x: some([1,2,3], lambda y: y**2 == x))
Using such operations, you can easily do smart verifications, like:
"""There exists at least one bot in a room which has a life below 30%"""
answer = some(bots_in_this_room, lambda x: x.life < 0.3)
and so on, you can answer even very complicated questions using such quantifiers. Of course, there is no infinite lists in Python (hey, it's not Haskell :) ), but Python's lists comprehensions are very practical.
Do you have your own favourite lists-comprehension constructions?
PS: I wonder, why most people tend not to answer questions but criticize presented examples? The question is about favourite list-comprehension construction actually.
anyand all are part of standard Python from 2.5. There's no need to make your own versions of these. Also the official version of any and all short-circuit the evaluation if possible, giving a performance improvement. Your versions always iterate over the entire list.
If you want a version that accepts a predicate, use something like this that leverages the existing any and all functions:
def anyWithPredicate(predicate, l): return any(predicate(x) for x in l)
def allWithPredicate(predicate, l): return all(predicate(x) for x in l)
I don't particularly see the need for these functions though, as it doesn't really save much typing.
Also, hiding existing standard Python functions with your own functions that have the same name but different behaviour is a bad practice.
There aren't all that many cases where a list comprehension (LC for short) will be substantially more useful than the equivalent generator expression (GE for short, i.e., using round parentheses instead of square brackets, to generate one item at a time rather than "all in bulk at the start").
Sometimes you can get a little extra speed by "investing" the extra memory to hold the list all at once, depending on vagaries of optimization and garbage collection on one or another version of Python, but that hardly amounts to substantial extra usefulness of LC vs GE.
Essentially, to get substantial extra use out of the LC as compared to the GE, you need use cases which intrinsically require "more than one pass" on the sequence. In such cases, a GE would require you to generate the sequence once per pass, while, with an LC, you can generate the sequence once, then perform multiple passes on it (paying the generation cost only once). Multiple generation may also be problematic if the GE / LC are based on an underlying iterator that's not trivially restartable (e.g., a "file" that's actually a Unix pipe).
For example, say you are reading a non-empty open text file f which has a bunch of (textual representations of) numbers separated by whitespace (including newlines here and there, empty lines, etc). You could transform it into a sequence of numbers with either a GE:
G = (float(s) for line in f for s in line.split())
or a LC:
L = [float(s) for line in f for s in line.split()]
Which one is better? Depends on what you're doing with it (i.e, the use case!). If all you want is, say, the sum, sum(G) and sum(L) will do just as well. If you want the average, sum(L)/len(L) is fine for the list, but won't work for the generator -- given the difficulty in "restarting f", to avoid an intermediate list you'll have to do something like:
tot = 0.0
for i, x in enumerate(G): tot += x
return tot/(i+1)
nowhere as snappy, fast, concise and elegant as return sum(L)/len(L).
Remember that sorted(G) does return a list (inevitably), so L.sort() (which is in-place) is the rough equivalent in this case -- sorted(L) would be supererogatory (as now you have two lists). So when sorting is needed a generator may often be preferred simply due to conciseness.
All in all, since L is identically equivalent to list(G), it's hard to get very excited about the ability to express it via punctuation (square brackets instead of round parentheses) instead of a single, short, pronounceable and obvious word like list;-). And that's all a LC is -- punctuation-based syntax shortcut for list(some_genexp)...!
This solution shadows builtins which is generally a bad idea. However the usage feels fairly pythonic, and it preserves the original functionality.
Note there are several ways to potentially optimize this based on testing, including, moving the imports out into the module level and changing f's default into None and testing for it instead of using a default lambda as I did.
def any(l, f=lambda x: x):
from __builtin__ import any as _any
return _any(f(x) for x in l)
def all(l, f=lambda x: x):
from __builtin__ import all as _all
return _all(f(x) for x in l)
Just putting that out there for consideration and to see what people think of doing something so potentially dirty.
For your information, the documentation for module itertools in python 3.x lists some pretty nice generator functions.