I wrote two function f and g with same functionality
def f(l, count):
if count > 1:
for i in f(l, count-1):
yield i + 1
else:
yield from l
for i in f(range(100000),900):
pass
print('f')
and
def g(l, count):
if count > 1:
tmp = []
for i in g(l, count-1):
tmp.append(i+1)
return tmp
else:
return l
for i in g(range(100000),900):
pass
print('f')
and i
I think f shuold be faster but g is faster when in run it
time for g
real 0m5.977s
user 0m5.956s
sys 0m0.020s
time for f
real 0m7.389s
user 0m7.376s
sys 0m0.012s
There are a couple of big differences between a solution that yields a result and one that computes the complete result.
The yield keeps returning the next result until exhausted while the complete calculation is always done fully so if you had a test that might terminate your calculation early, (often the case), the yield method will only be called enough times to meet that criteria - this often results in faster code.
The yield result only consumes enough memory to hold the generator and a single result at any moment in time - the full calculation consumes enough memory to hold all of the results at once. When you get to really large data sets that can make the difference between something that runs regardless of the size and something that crashes.
So yield is slightly more expensive, per operation, but much more reliable and often faster in cases where you don't exhaust the results.
I have no idea what your h and g functions are doing but remember this,
Yield is a keyword that is used like return, except the function will return a generator and that is the reason it takes time.
There is a wonderful explanation about what yeild does. Check this answer on stackoverflow.
Related
I just saw an example of Fibonacci series using a generator from a webpage.
This is the example I saw
def fibonacci(n):
curr = 1
prev = 0
counter = 0
while counter < n:
yield curr
prev, curr = curr, prev + curr
counter += 1
The user can input the limit 'n'.
My question is what's the use of using generator here?
how generator benefits the user here?
What advantage he gets instead of just using a normal print message here to print the series?
I am agreeing if the user has to store the value in a list and process it later, yield can do generate number on fly.
What could be the use of yield in this case?
By yielding the value, the caller of the function has access to the intermediate results, making this memory efficient.
For example, the following would give you all results up to n as well, but requires that you create a list first to hold those values:
def fibonacci(n):
results = []
curr = 1
prev = 0
counter = 0
while counter < n:
results.append(curr)
prev, curr = curr, prev + curr
counter += 1
return results
If my code then asked for fibonacci(10 ** 9), that would not only take a long time to produce, but would also require a significant amount of memory.
The generator option gives the caller access to the results immediately, which means they can use just that one intermediate result in another part of the program.)
This makes the generator function far more versatile and flexible. You could sum the fibonacci results, without having to rewrite the generator, for example:
sum_of_fibonacci_10 = sum(fibonacci(10))
This is still memory efficient, the summing takes place as the results are produced, at no point do all 10 results have to exist in memory. Just the current intermediate value, and the sum total up to now, are required.
If you wanted to just print the values instead, you can do so in a loop:
for next_value in fibonacci(10):
print(next_value)
and the code is still just as efficient, and we still did not have to change the generator function.
So generators let you maintain and share intermediate state, saving you memory. This can be done with a class too, but a generator is much more readable. The following would do the same, but is not nearly as easy to follow:
class FibonacciGenerator:
def __init__(self, n):
self.n = n
self.curr = 1
self.prev = 0
self.counter = 0
def __iter__(self):
return self
def __next__(self):
if self.counter >= self.n:
raise StopIteration
self.prev, self.curr = self.curr, self.prev + self.curr
self.counter += 1
return self.prev
The advantage of a generator instead of a function is that it can keep its state.
If you had used a recursive function, calculating the 5th fibonacci number would be equivalent to calculating the 1st, then calculating the 1st and 2nd, then calculating the 1st, 2nd and 3rd, and so on until the 5th.
You can see how this is the problem for large numbers; there's a lot of resources being wasted.
With a generator, you calculate the 1st, and remember what it is, so that you don't have to calculate it again in order to find the second. Unlike a function that stops when it reaches its return, the generator can keep state. So it calculates the 1st number, and when requested, calculates the 2nd, then the 3rd, without wasting processing power because it doesn't need to recalculate the whole thing each time; it remembers its previous state.
With regards to print vs yield, obviously you can't use the printed result if you want to use the fibonacci numbers you calculated for something else; it's just output on the screen. If you'd like to generate a graph, for example, print wouldn't do, you'd have to pass values to your plotting function.
Using a generator allows you to generate the 'tip' of the sequence one at a time, at your leisure. You can retrieve and print the result, but you don't need to.
A plain ol' loop in a function would run until the breaking condition was met - or forever - and without any special bells and whistles, would block the execution of your code until the loop terminated.
This means an unbounded loop would only stop once you killed the interpreter, and soon enough you'd have to kill it because it would start eating resources like crazy computing very high sums.
Yield pauses between calls and returns the value rather than just displaying it unlike print(). This means you can retrieve nth fibonacci number, do some other stuff for five minutes, then retrieve n+1th number.
I am trying to write a program that finds the smallest pallindrome cube. My code:
def cubepal():
i=3
if str(i**3)==str(i**3)[::-1]:
return i**3
else:
i=i+1
cubepal()
I am pretty sure that the first part is correct, but the last three lines make the program run in an endless loop. In the end it only says the problem is on line
if str(i**3)==str(i**3)[::-1]:
I don't see why there is a problem. Could somebody help?
The reason is that you don't scope your variables correctly.
You call cubepal and cubepal initializes i=3 now when you perform the recursive call, i is a local variable equal to 3.
def cubepal(i):
if str(i**3)==str(i**3)[::-1]:
return i**3
else:
return cubepal(i+1)
and call it with cubepal(3).
Although for such cases, one better not uses recursion: if it is expected that i will be very large (this is not the case here), but it could be, for a memory-inefficient Python interpeter, result in a call-stack that scales with the value of the result.
A better way to handle this is I guess a while loop.
def cubepal():
i = 3
while str(i**3) != str(i**3)[::-1]:
i = i+1
return i**3
This will in general be more efficient as well, because calling a function leads to some overhead regarding bookkeeping and the call-stack.
You probably meant to do this:
def cubepal(i):
if str(i**3)==str(i**3)[::-1]:
return i**3
else:
return cubepal(i+1)
print(cubepal(3))
which yields "343".
Of course, there is no point to do it this way in Python since Python lacks TCO.
My code is short and has lesser number of iterations than the other but still it gets a time limit exceeded error while the other code is accepted on codechef.com. The codes are solution to the "Ambiguous permutation" problem on codechef.com
Why is this code:
def inverse(data,x):
temp=[]
for i in range(x):
temp.append(0)
for i in range(x):
temp[data[i]-1]=i+1
return temp
def main():
while 1:
x=input()
if not x:
return
data=raw_input().split(' ')
for i in range(len(data)):
data[i]=int(data[i])
temp = inverse(data,x)
if temp==data:
print 'ambiguous'
else:
print 'not ambiguous'
if __name__ == "__main__":
main()
faster than this code:
while True:
output=[]
n= input()
if n==0: break
caseList= raw_input().split()
amb=1
for i in range(1, n+1):
output.append(str(caseList.index(str(i))+1))
if output==caseList:
print ("ambiguous")
else:
print ("not ambiguous")
Please help me out...
I take it your second code isn't inside a function.
Access to local variables in a function is significantly faster than accessing global variables. That means code run in global scope is always likely to be slower.
When such a consistent time difference shows up, we are not talking about something that is 2, or 3 times slower - if one method passes, and the other always times out, it is a huge time difference- normally tied to algorithm complexity.
Although the first code has plenty of room to get better (usage of xrange instead of range, for example), in the final array, the result is update by random access, straight by the index caculated by data[i] - 1- while in the second snippet, you use caseList.index(str(i)) to retrieve each index. The "index" lisr method performs a linear search on the list, starting at the beginning. It may seem little, but when it is doen for each element in the list, your algorithm which was O(N) becomes O(N²) - which expains the dramatic speed down in this second form.
It looks like you're using strings where the other code is using ints, which will slow you down somewhat.
I have a function in python whose output is a generator :
def main_function(x):
r = get_range()
for i in range(r):
yield x+i
I want to refactor the code (I've simplified the use case but actual computation might be complex & longer. Please see EDIT below). Based on my understanding, this is what I should do to keep the functionality unchanged :
(a) Same interface as original code
def sub_function(x,r):
for i in range(r):
yield x+i
def main_function(x):
r = get_range()
return sub_function(x,r)
As compared to other approaches where :
(b) This would return a generator of a generator (Are there any advantages of this approach ?)
def sub_function(x,r):
for i in range(r):
yield x+i
def main_function(x):
r = get_range()
yield sub_function(x,r)
(c) This would defeat the purpose of a generator (Is that correct?)
def sub_function(x,r):
return [x+i for i in range(r)]
def main_function(x):
r = get_range()
for i in sub_function(x,r):
yield(i)
EDIT : Comments point out that the right answer is use case dependent. I want to add that my use case is parsing an XML file to extract fields and write them to a database. This part is delegated to sub_function(). I also asked this question for a general understanding of the usage of nested yield for refactoring code.
You're right; the initial example and a) do the same thing since both return a generator.
b) is different: It returns a generator which yields a single element (which is another generator). To use that, you need two loops (one over the outer and one over the inner generator).
There are no advantages per se but sometimes, it can be useful to build nested generators.
c) could be worse but I'm pretty sure that the [x for x in y] is actually implemented as a generator, too. So it's a bit more expensive but not that much.
For example, files, in Python, are iterable - they iterate over the lines in the file. I want to count the number of lines.
One quick way is to do this:
lines = len(list(open(fname)))
However, this loads the whole file into memory (at once). This rather defeats the purpose of an iterator (which only needs to keep the current line in memory).
This doesn't work:
lines = len(line for line in open(fname))
as generators don't have a length.
Is there any way to do this short of defining a count function?
def count(i):
c = 0
for el in i: c += 1
return c
To clarify, I understand that the whole file will have to be read! I just don't want it in memory all at once
Short of iterating through the iterable and counting the number of iterations, no. That's what makes it an iterable and not a list. This isn't really even a python-specific problem. Look at the classic linked-list data structure. Finding the length is an O(n) operation that involves iterating the whole list to find the number of elements.
As mcrute mentioned above, you can probably reduce your function to:
def count_iterable(i):
return sum(1 for e in i)
Of course, if you're defining your own iterable object you can always implement __len__ yourself and keep an element count somewhere.
If you need a count of lines you can do this, I don't know of any better way to do it:
line_count = sum(1 for line in open("yourfile.txt"))
The cardinality package provides an efficient count() function and some related functions to count and check the size of any iterable: http://cardinality.readthedocs.org/
import cardinality
it = some_iterable(...)
print(cardinality.count(it))
Internally it uses enumerate() and collections.deque() to move all the actual looping and counting logic to the C level, resulting in a considerable speedup over for loops in Python.
I've used this redefinition for some time now:
def len(thingy):
try:
return thingy.__len__()
except AttributeError:
return sum(1 for item in iter(thingy))
It turns out there is an implemented solution for this common problem. Consider using the ilen() function from more_itertools.
more_itertools.ilen(iterable)
An example of printing a number of lines in a file (we use the with statement to safely handle closing files):
# Example
import more_itertools
with open("foo.py", "r+") as f:
print(more_itertools.ilen(f))
# Output: 433
This example returns the same result as solutions presented earlier for totaling lines in a file:
# Equivalent code
with open("foo.py", "r+") as f:
print(sum(1 for line in f))
# Output: 433
Absolutely not, for the simple reason that iterables are not guaranteed to be finite.
Consider this perfectly legal generator function:
def forever():
while True:
yield "I will run forever"
Attempting to calculate the length of this function with len([x for x in forever()]) will clearly not work.
As you noted, much of the purpose of iterators/generators is to be able to work on a large dataset without loading it all into memory. The fact that you can't get an immediate length should be considered a tradeoff.
Because apparently the duplication wasn't noticed at the time, I'll post an extract from my answer to the duplicate here as well:
There is a way to perform meaningfully faster than sum(1 for i in it) when the iterable may be long (and not meaningfully slower when the iterable is short), while maintaining fixed memory overhead behavior (unlike len(list(it))) to avoid swap thrashing and reallocation overhead for larger inputs.
# On Python 2 only, get zip that lazily generates results instead of returning list
from future_builtins import zip
from collections import deque
from itertools import count
def ilen(it):
# Make a stateful counting iterator
cnt = count()
# zip it with the input iterator, then drain until input exhausted at C level
deque(zip(it, cnt), 0) # cnt must be second zip arg to avoid advancing too far
# Since count 0 based, the next value is the count
return next(cnt)
Like len(list(it)), ilen(it) performs the loop in C code on CPython (deque, count and zip are all implemented in C); avoiding byte code execution per loop is usually the key to performance in CPython.
Rather than repeat all the performance numbers here, I'll just point you to my answer with the full perf details.
For filtering, this variation can be used:
sum(is_good(item) for item in iterable)
which can be naturally read as "count good items" and is shorter and simpler (although perhaps less idiomatic) than:
sum(1 for item in iterable if is_good(item)))
Note: The fact that True evaluates to 1 in numeric contexts is specified in the docs
(https://docs.python.org/3.6/library/stdtypes.html#boolean-values), so this coercion is not a hack (as opposed to some other languages like C/C++).
We'll, if you think about it, how do you propose you find the number of lines in a file without reading the whole file for newlines? Sure, you can find the size of the file, and if you can gurantee that the length of a line is x, you can get the number of lines in a file. But unless you have some kind of constraint, I fail to see how this can work at all. Also, since iterables can be infinitely long...
I did a test between the two common procedures in some code of mine, which finds how many graphs on n vertices there are, to see which method of counting elements of a generated list goes faster. Sage has a generator graphs(n) which generates all graphs on n vertices. I created two functions which obtain the length of a list obtained by an iterator in two different ways and timed each of them (averaging over 100 test runs) using the time.time() function. The functions were as follows:
def test_code_list(n):
l = graphs(n)
return len(list(l))
and
def test_code_sum(n):
S = sum(1 for _ in graphs(n))
return S
Now I time each method
import time
t0 = time.time()
for i in range(100):
test_code_list(5)
t1 = time.time()
avg_time = (t1-t0)/10
print 'average list method time = %s' % avg_time
t0 = time.time()
for i in range(100):
test_code_sum(5)
t1 = time.time()
avg_time = (t1-t0)/100
print "average sum method time = %s" % avg_time
average list method time = 0.0391882109642
average sum method time = 0.0418473792076
So computing the number of graphs on n=5 vertices this way, the list method is slightly faster (although 100 test runs isn't a great sample size). But when I increased the length of the list being computed by trying graphs on n=7 vertices (i.e. changing graphs(5) to graphs(7)), the result was this:
average list method time = 4.14753051996
average sum method time = 3.96504004002
In this case the sum method was slightly faster. All in all, the two methods are approximately the same speed but the difference MIGHT depend on the length of your list (it might also just be that I only averaged over 100 test runs, which isn't very high -- would have taken forever otherwise).