Why does Python create a new frame for each recursive function call in object oriented programming?
I have tried to search for answers on the internet but could not find any specific reason or justification for it. I
In some cases, Python could absolutely get away with reusing a stack frame for recursive function calls:
def factorial(n, a=1):
if n == 0:
return a
else:
return factorial(n - 1, n * a)
But often every call needs its own stack frame, since there's some state that's unique to each iteration. Let's say that instead of returning the values immediately, we wanted to print them out:
def factorial2(n, depth=0):
if n == 0:
value = 1
else:
value = n * factorial2(n-1, depth+1)
print(f"Depth: {depth}, Value: {value}")
return value
If we call factorial2(3), then by the time we're at the deepest function call, there are four different depth and value variables in different stack frames. Python needs to use these values later, so it can't throw away the stack frames in the meantime.
Languages like Scheme still create new stack frames for recursive functions in the general case, but they can avoid it in the special case of tail-call recursion. In the first factorial, the recursion is the very last thing that happens before the function returns, so a language like Scheme would know it could re-use the stack frame.
Python could implement this optimization, but Guido van Rossum has opposed it, arguing that it would make debugging harder and encourage non-Pythonic code. You can read these blog articles for his full thought process:
http://neopythonic.blogspot.com/2009/04/tail-recursion-elimination.html
http://neopythonic.blogspot.com/2009/04/final-words-on-tail-calls.html
Related
The code I already have is for a bot that receives a mathematical expression and calculates it. Right now I have it doing multiply, divide, subtract and add. The problem though is I want to build support for parentheses and parentheses inside parentheses. For that to happen, I need to run the code I wrote for the expressions without parentheses for the expression inside the parentheses first. I was going to check for "(" and append the expression inside it to a list until it reaches a ")" unless it reaches another "(" first in which case I would create a list inside a list. I would subtract, multiply and divide and then the numbers that are left I just add together.
So is it possible to call a definition/function from within itself?
Yes, this is a fundamental programming technique called recursion, and it is often used in exactly the kind of parsing scenarios you describe.
Just be sure you have a base case, so that the recursion ends when you reach the bottom layer and you don't end up calling yourself infinitely.
(Also note the easter egg when you Google recursion: "Did you mean recursion?")
Yes, as #Daniel Roseman said, this is a fundamental programming technique called recursion.
Recursion should be used instead of iteration when you want to produce a cleaner solution compared to an iterative version. However, recursion is generally more expensive than iteration because it requires winding, or pushing new stack frames onto the call stack each time a recursive function is invoked -- these operations take up time and stack space, which can lead to an error called stack overflow if the stack frames consume all of the memory allocated for the call stack.
Here is an example of it in Python
def recur_factorial(n):
"""Function to return the factorial of a number using recursion"""
if n == 1:
return n
else:
return n*recur_factorial(n-1)
For more detail, visit the github gist that was used for this answer
yes it's possible in "python recursion"
and the best describe is: "A physical world example would be to place two parallel mirrors facing each other. Any object in between them would be reflected recursively"
Problem: Return the number of nodes in the linked list.
I am learning recursion recently. I know how to use iteration to solve this problem but I am stick by the recursion way. The following is my code and it always return 1 instead of the real count of the linked list. I cannot figure out the problem and hope someone can help me. How can I fix the problem?
def numberOfNodes(head):
total_node = 0
return __helper(total_node, head)
def __helper(total_node, head):
if not head:
return total_node += 1
__helper(total_node, head.next)
return total_node
Recursion is a poor choice for this sort of thing (adds overhead and risks blowing the call stack for no good reason), but if you do use it for educational purposes, it's easiest to pass the total up the call stack, not down:
def linked_list_len(head):
return linked_list_len(head.next) + 1 if head else 0
Basically, add 1 per frame where the head node exists, then call the function again with the next node. The base case is when head is None.
In some languages that offer tail call optimization, you can avoid the + 1 work that happens to the variable returned by the child recursive call per frame. This allows the compiler or interpreter to convert recursion to a loop, avoiding stack overflows. The code would look like (similar to your approach, with the difference that the + 1 is added in the recursive call):
def linked_list_len(head, total=0):
return linked_list_len(head.next, total + 1) if head else total
But Python doesn't support TCO so you may as well write it the simpler way shown above.
In a Python tutorial, I've learned that
Like functions, generators can be recursively programmed. The following
example is a generator to create all the permutations of a given list of items.
def permutations(items):
n = len(items)
if n==0: yield []
else:
for i in range(len(items)):
for cc in permutations(items[:i]+items[i+1:]):
yield [items[i]]+cc
for p in permutations(['r','e','d']): print(''.join(p))
for p in permutations(list("game")): print(''.join(p) + ", ", end="")
I cannot figure out how it generates the results. The recursive things and 'yield' really confused me. Could someone explain the whole process clearly?
There are 2 parts to this --- recursion and generator. Here's the non-generator version that just uses recursion:
def permutations2(items):
n = len(items)
if n==0: return [[]]
else:
l = []
for i in range(len(items)):
for cc in permutations2(items[:i]+items[i+1:]):
l.append([items[i]]+cc)
return l
l.append([item[i]]+cc) roughly translates to the permutation of these items include an entry where item[i] is the first item, and permutation of the rest of the items.
The generator part yield one of the permutations instead of return the entire list of permutations.
When you call a function that returns, it disappears after having produced its result.
When you ask a generator for its next element, it produces it (yields it), and pauses -- yields (the control back) to you. When asked again for the next element, it will resume its operations, and run normally until hitting a yield statement. Then it will again produce a value and pause.
Thus calling a generator with some argument causes creation of actual memory entity, an object, capable of running, remembering its state and arguments, and producing values when asked.
Different calls to the same generator produce different actual objects in memory. The definition is a recipe for the creation of that object. After the recipe is defined, when it is called it can call any other recipe it needs -- or the same one -- to create new memory objects it needs, to produce the values for it.
This is a general answer, not Python-specific.
Thanks for the answers. It really helps me to clear my mind and now I want to share some useful resources about recursion and generator I found on the internet, which is also very friendly to the beginners.
To understand generator in python. The link below is really readable and easy to understand.
What does the "yield" keyword do in Python?
To understand recursion, "https://www.youtube.com/watch?v=MyzFdthuUcA". This youtube video gives a "patented" 4 steps method to writing any recursive method/function. That is very clear and practicable. The channel also has several videos to show people how does the recursion works and how to trace it.
I hope it can help someone like me.
I’d like to be pointed toward a reference that could better explain recursion when a function employs multiple recursive calls. I think I get how Python handles memory when a function employs a single instance of recursion. I can use print statements to track where the data is at any given point while the function processes the data. I can then walk each of those steps back to see how the resultant return value was achieved.
Once multiple instances of recursion are firing off during a single function call I am no longer sure how the data is actually being processed. The previously illuminating method of well-placed print statements reveals a process that looks quantum, or at least more like voodoo.
To illustrate my quandary here are two basic examples: the Fibonacci and Hanoi towers problems.
def getFib(n):
if n == 1 or n == 2:
return 1
return getFib(n-1) + getFib(n-2)
The Fibonacci example features two inline calls. Is getFib(n-1) resolved all the way through the stack first, then getFib(n-2) resolved similarly, each of the resultants being put into new stacks, and those stacks added together line by line, with those sums being totaled for the result?
def hanoi(n, s, t, b):
assert n > 0
if n ==1:
print 'move ', s, ' to ', t
else:
hanoi(n-1,s,b,t)
hanoi(1,s,t,b)
hanoi(n-1,b,t,s)
Hanoi presents a different problem, in that the function calls are in successive lines. When the function gets to the first call, does it resolve it to n=1, then move to the second call which is already n=1, then to the third until n=1?
Again, just looking for reference material that can help me get smart on what’s going on under the hood here. I’m sure it’s likely a bit much to explain in this setting.
http://www.pythontutor.com/visualize.html
There's even a Hanoi link there so you can follow the flow of code.
This is a link to the hanoi code that they show on their site, but it may have to be adapated to visualize your exact code.
http://www.pythontutor.com/visualize.html#code=%23+move+a+stack+of+n+disks+from+stack+a+to+stack+b,%0A%23+using+tmp+as+a+temporary+stack%0Adef+TowerOfHanoi(n,+a,+b,+tmp)%3A%0A++++if+n+%3D%3D+1%3A%0A++++++++b.append(a.pop())%0A++++else%3A%0A++++++++TowerOfHanoi(n-1,+a,+tmp,+b)%0A++++++++b.append(a.pop())%0A++++++++TowerOfHanoi(n-1,+tmp,+b,+a)%0A++++++++%0Astack1+%3D+%5B4,3,2,1%5D%0Astack2+%3D+%5B%5D%0Astack3+%3D+%5B%5D%0A++++++%0A%23+transfer+stack1+to+stack3+using+Tower+of+Hanoi+rules%0ATowerOfHanoi(len(stack1),+stack1,+stack3,+stack2)&mode=display&cumulative=false&heapPrimitives=false&drawParentPointers=false&textReferences=false&showOnlyOutputs=false&py=2&curInstr=0
I've just run into a tricky issue. The following code is supposed to split words into chunks of length numOfChar. The function calls itself, which makes it impossible to have the resulting list (res) inside the function. But if I keep it outside as a global variable, then every subsequent call of the function with different input values leads to a wrong result because res doesn't get cleared.
Can anyone help me out?
Here's the code
(in case you are interested, this is problem 7-23 from PySchools.com):
res = []
def splitWord(word, numOfChar):
if len(word) > 0:
res.append(word[:numOfChar])
splitWord(word[numOfChar:], numOfChar)
return res
print splitWord('google', 2)
print splitWord('google', 3)
print splitWord('apple', 1)
print splitWord('apple', 4)
A pure recursive function should not modify the global state, this counts as a side effect.
Instead of appending-and-recursion, try this:
def splitWord(word, numOfChar):
if len(word) > 0:
return [word[:numOfChar]] + splitWord(word[numOfChar:], numOfChar)
else:
return []
Here, you chop the word into pieces one piece at a time, on every call while going down, and then rebuild the pieces into a list while going up.
This is a common pattern called tail recursion.
P.S. As #e-satis notes, recursion is not an efficient way to do this in Python. See also #e-satis's answer for a more elaborate example of tail recursion, along with a more Pythonic way to solve the problem using generators.
Recursion is completely unnecessary here:
def splitWord(word, numOfChar):
return [word[i:i+numOfChar] for i in xrange(0, len(word), numOfChar)]
If you insist on a recursive solution, it is a good idea to avoid global variables (they make it really tricky to reason about what's going on). Here is one way to do it:
def splitWord(word, numOfChar):
if len(word) > 0:
return [word[:numOfChar]] + splitWord(word[numOfChar:], numOfChar)
else:
return []
To elaborate on #Helgi answer, here is a more performant recursive implémentation. It updates the list instead of summing two lists (which results in the creation of a new object every time).
This pattern forces you to pass a list object as third parameter.
def split_word(word, num_of_chars, tail):
if len(word) > 0:
tail.append(word[:num_of_chars])
return split_word(word[num_of_chars:], num_of_chars, tail)
return tail
res = split_word('fdjskqmfjqdsklmfjm', 3, [])
Another advantage of this form, is that it allows tail recursion optimisation. It's useless in Python because it's not a language that performs such optimisation, but if you translate this code into Erlang or Lisp, you will get it for free.
Remember, in Python you are limited by the recursion stack, and there is no way out of it. This is why recursion is not the preferred method.
You would most likely use generators, using yield and itertools (a module to manipulate generators). Here is a very good example of a function that can split any iterable in chunks:
from itertools import chain, islice
def chunk(seq, chunksize, process=iter):
it = iter(seq)
while True:
yield process(chain([it.next()], islice(it, chunksize - 1)))
Now it's a bit complicated if you start learning Python, so I'm not expecting you to fully get it now, but it's good that you can see this and know it exists. You'll come back to it later (we all did, Python iteration tools are overwhelming at first).
The benefits of this approach are:
It can chunk ANY iterable, not just strings, but also lists, dictionaries, tuples, streams, files, sets, queryset, you name it...
It accepts iterables of any length, and even one with an unknown length (think bytes stream here).
It eats very few memory, as the best thing with generators is that they generate the values on the fly, one by one, and they don't store the previous results before computing the next.
It returns chunks of any nature, meaning you can have a chunks of x letters, lists of x items, or even generators spitting out x items (which is the default).
It returns a generator, and therefor can be use in a flow of other generators. Piping data from one generator to the other, bash style, is a wonderful Python ability.
To get the same result that with your function, you would do:
In [17]: list(chunk('fdjskqmfjqdsklmfjm', 3, ''.join))
Out[17]: ['fdj', 'skq', 'mfj', 'qds', 'klm', 'fjm']