We Keep Coding

Intermediate results from recursion

I have a problem where I need to produce something which is naturally computed recursively, but where I also need to be able to interrogate the intermediate steps in the recursion if needed.
I know I can do this by passing and mutating a list or similar structure. However, this looks ugly to me and I'm sure there must be a neater way, e.g. using generators. What I would ideally love to be able to do is something like:
intermediate_results = [f(x) for x in range(T)]
final_result = intermediate_results[T-1]
in an efficient way. While my solution is not performance critical, I can't justify the massive amount of redundant effort in that first line. It looks to me like a generator would be perfect for this except for the fact that f is fundamentally much more suited to recursion in my case (which at least in my mind is the complete opposite of a generator, but maybe I'm just not thinking far enough outside of the box).
Is there a neat Pythonic way of doing something like this that I just don't know about, or do I just need to just capitulate and pollute my function f by passing it an intermediate_results list which I then mutate as a side-effect?

I have a generic solution for you using a decorator. We create a Memoize class which stores the results of previous times the function is executed (including in recursive calls). If the arguments given have already been seen, the cached versions are used to quickly lookup the result.
The custom class has the benefit over an lru_cache in that you can see the results.
from functools import wraps
class Memoize:
def __init__(self):
self.store = {}
def save(self, fun):
#wraps(fun)
def wrapper(*args):
if args not in self.store:
self.store[args] = fun(*args)
return self.store[args]
return wrapper
m = Memoize()
#m.save
def fibo(n):
if n <= 0: return 0
elif n == 1: return 1
else: return fibo(n-1) + fibo(n-2)
Then after running different things you can see what the cache contains. When you run future function calls, m.store will be used as a lookup so calculation doesn't need to be redone.
>>> f(8)
21
>>> m.store
{(1,): 1,
(0,): 0,
(2,): 1,
(3,): 2,
(4,): 3,
(5,): 5,
(6,): 8,
(7,): 13,
(8,): 21}
You could modify the save function to use the name of the function and the args as the key, so that multiple function results can be stored in the same Memoize class.

You can use your existing solution that makes many "redundant" calls to f, but employ the use of function caching to save the results to previous calls to f.
In other words, when f(x1) is called, it's input arguments and corresponding return values are saved, and the next time it is called, the result is simply pulled from the cache
see functools.lru_cache for the standard library solution to this
ie:
from functools import lru_cache
#lru_cache
intermediate_results = [f(x) for x in range(T)]
final_result = intermediate_results[T-1]
Note, however, f must be a pure function (no side-effects, 1-to-1 mapping) for this to work properly

Having considered your comments, I'll now try to give another perspective on the problem.
So, let's consider a concrete example:
def f(x):
a = 2
return g(x) + a if x != 0 else 0
def g(x):
b = 1
return h(x) - b
def h(x):
c = 1/2
return f(x-1)*(1+c)
I
First of all, it should be mentioned that (in our particular case) the algorithm has form of: f(x) = p(f(x - 1)) for some p. It follows that f(x) = p^x(f(0)) = p^x(0). That means we should just apply p to 0 x times to get the desired result, which can be done in an iterative process, so this can be written without recursion. Though I believe that your real case is much harder. Moreover, it would be too boring and uninformative to stop here)
II
Generally speaking, we can divide all possible solutions into two groups: the ones that require refactoring (i.e. rewriting functions f, g, h) and the ones that do not. I have little to offer from the latter one (and I don't think anyone can). Consider the following, however:
def fk(x, k):
a = 2
return k(gk(x, k) + a if x != 0 else 0)
def gk(x, k):
b = 1
return k(hk(x, k) - b)
def hk(x, k):
c = 1/2
return k(fk(x-1, k)*(1+c))
def printret(x):
print(x)
return x
f(4, printret) # see what happens
Inspired by continuation-passing style, but that's totally not it.
What's the point? It's something between your idea of passing a list to write down all the computations and memoizing. This k carries additional behavior with it, such as printing or writing to list (you can make a function that writes to some list, why not?). But if you look carefully you'll see that it lefts inner code of these functions practically untouched (only input and output to function are affected), so one can produce a decorator associated with a function like printret that does essentially the same thing for f, g, h.
Pros: no need to modify code, much more flexible than passing a list, no additional work (like in memoizing).
Cons: Impure (printing or modifying sth), not so flexible as we would like.
III
Now let's see how modifying function bodies can help. Don't be afraid of what's written below, take your time and play with that thing a little.
class Logger:
def __init__(self, lst, cur_val):
self.lst = lst
self.cur_val = cur_val
def bind(self, f):
res = f(self.cur_val)
return Logger([self.cur_val] + res.lst + self.lst, res.cur_val)
def __repr__(self):
return "Logger( " + repr({'value' : self.cur_val,'lst' : self.lst}) + " )"
def unit(x):
return Logger([], x)
# you can also play with lala
def lala(x):
if x <= 0:
return unit(1)
else:
return lala(x - 1).bind(lambda y: unit(2*y))
def f(x):
a = 2
if x == 0:
return unit(0)
else:
return g(x).bind(lambda y: unit(y + a))
def g(x):
b = 1
return h(x).bind(lambda y: unit(y - b))
def h(x):
c = 1/2
return f(x-1).bind(lambda y: unit(y*(1+c)))
f(4) # see for yourself
Logger is called a monad. I'm not very familiar with this concept myself, but I guess I'm doing everything right) f, g, h are functions that take a number and return a Logger instance. Logger's bind takes in a function (like f) and returns Logger with new value (computed by f) and updated 'logs'. The key point - as I see it - is the ability to do whatever we want with collected functions in the order the resulting value was calculated.
Afterword
I'm not at all some kind of 'guru' of functional programming, I believe I'm missing a lot of things here. But what I've understood is that functional programming is about inversing the flow of the program. That's why, for instance, I totally agree with your opinion about generators being opposed to functional programming. When we use generator gen in, say, function func, we yield values one by one to func and func does sth with them in e.g. a loop. The functional approach would be to make gen a function taking func as a parameter and make func perform computations on 'yielded' values. It's like gen and func exchanged their places. So the flow is inversed! And there are plenty of other ways of inversing the flow. Monads are one of them.

itertools islice gets a generator, start value and stop value. it will give you the elements between the start value and stop value as a generator. if islice is not clear you can check the docs here https://docs.python.org/3/library/itertools.html
intermediate_result = map(f, range(T))
final_result = next(itertools.islice(intermediate_result, start=T-1, stop=T))

Function that returns an accumulator in Python

I am reading Hackers and Painters and am confused by a problem mentioned by the author to illustrate the power of different programming languages.
The problem is:
We want to write a function that generates accumulators—a function that takes a number n, and returns a function that takes another number i and returns n incremented by i. (That’s incremented by, not plus. An accumulator has to accumulate.)
The author mentions several solutions with different programming languages. For example, Common Lisp:
(defun foo (n)
(lambda (i) (incf n i)))
and JavaScript:
function foo(n) { return function (i) { return n += i } }
However, when it comes to Python, the following codes do not work:
def foo(n):
s = n
def bar(i):
s += i
return s
return bar
f = foo(0)
f(1) # UnboundLocalError: local variable 's' referenced before assignment
A simple modification will make it work:
def foo(n):
s = [n]
def bar(i):
s[0] += i
return s[0]
return bar
I am new to Python. Why doesn the first solution not work while the second one does? The author mentions lexical variables but I still don't get it.

s += i is just sugar for s = s + i.*
This means you assign a new value to the variable s (instead of mutating it in place). When you assign to a variable, Python assumes it is local to the function. However, before assigning it needs to evaluate s + i, but s is local and still unassigned -> Error.
In the second case s[0] += i you never assign to s directly, but only ever access an item from s. So Python can clearly see that it is not a local variable and goes looking for it in the outer scope.
Finally, a nicer alternative (in Python 3) is to explicitly tell it that s is not a local variable:
def foo(n):
s = n
def bar(i):
nonlocal s
s += i
return s
return bar
(There is actually no need for s - you could simply use n instead inside bar.)
*The situation is slightly more complex, but the important issue is that computation and assignment are performed in two separate steps.

An infinite generator is one implementation. You can call __next__ on a generator instance to extract successive results iteratively.
def incrementer(n, i):
while True:
n += i
yield n
g = incrementer(2, 5)
print(g.__next__()) # 7
print(g.__next__()) # 12
print(g.__next__()) # 17
If you need a flexible incrementer, one possibility is an object-oriented approach:
class Inc(object):
def __init__(self, n=0):
self.n = n
def incrementer(self, i):
self.n += i
return self.n
g = Inc(2)
g.incrementer(5) # 7
g.incrementer(3) # 10
g.incrementer(7) # 17

In Python if we use a variable and pass it to a function then it will be Call by Value whatever changes you make to the variable it will not be reflected to the original variable.
But when you use a list instead of a variable then the changes that you make to the list in the functions are reflected in the original List outside the function so this is called call by reference.
And this is the reason for the second option does work and the first option doesn't.

How do I call a list or object defined in another instance into another instance?

class Sieve:
def __init__(self, digit):
self.digit = []
numbers = [True]*digit
if digit <= -1:
raise RuntimeError("Cannot use negative values.")
numbers[0] = False
numbers[1] = False
def findPrimes(self):
for i in range(len(self.digit)):
if numbers[i]:
j = (i+i)
while (j < len(numbers)):
numbers[j] = False
j += i
print(numbers)
This is my first post on stackoverflow and after trying for a long time figuring out what to do and searching the internet, I'm afraid I have to ask here.
My program is supposed to find all prime numbers given any number in the initial class. I was trying to check my findPrimes(), but I was having a difficult time trying to get the values to print. I keep getting the error "numbers is not defined". Am I supposed to define it outside of findPrimes? Or is there a specific way I can call numbers from init to findPrimes? Thank you!

According to your post, your functions are unindented, so they're defined outside of your class.
Remember python relies heavily on indenting due to the lack of things like braces.
Also, your variable are local to your functions, and aren't properties of your class. You need to add them to your class by prepending them with self. (like you did with self.digit).
so self.numbers = ... instead of numbers = ...

Please explain Python for loops to a C++/C#/Java programmer

I have had experience in Java/C#/C++ and for loops or pretty much if not exactly done the same. Now I'm learning Python through Codecademy. I find it poor the way it trys to explain for loops to me. The code they give you is
my_list = [1,9,3,8,5,7]
for number in my_list:
# Your code here
print 2 * number
Is this saying for every number in my_list ... print 2 * number.
That somewhat makes sense to me if that's true but I don't get number and how that works. It's not even a variable declared earlier. Are you declaring a variable withing the for loop? And how does Python know that number is accessing the values within my_list and multiplying them by 2? Also, how do for loops work with things other than lists because I've looked at other Python code that contains for loops and they make no sense. Could you please find some way to explain the way these are similar to something like C# for loops or just explain Python for loops in general.

Yes, number is a newly defined variable. Python does not require variables to be declared before using them. And the understanding of the loop iteration is correct.
This is the same sytnax Borne-style shells use (such as bash).
The logic of the for loop is this: assign the named variable the next value in the list, iterate, repeat.
correction
As for other non-list values, they should translate into a sequence in python. Try this:
val="1 2 3"
for number in val:
print number
Note this prints "1", " ", "2", " ", "3".
Here's a useful reference: http://www.tutorialspoint.com/python/python_for_loop.htm.

The quick answer, to relate to C#, is that a Python for loop is roughly equivalent to a C# foreach loop. C++ sort of has similar facilities (BOOST_FOREACH for example, or the for syntax in C++11), but C does not have an equivalent.
There is no equivalent in Python of the C-style for (initial; condition; increment) style loop.
Python for loops can iterate over more than just lists; they can iterate over anything that is iterable. See for example What makes something iterable in python.

Python doesn't need variables to be declared it can be declared itself at the time of initialization
While and do while are similar to those languages but for loop is quite different in python
you can use it for list similar to for each
but for another purpose like to run from 1 to 10 you can use,
for number in range(10):
print number

Python for loops should be pretty similar to C# foreach loops. It steps through my_list and at each step you can use "number" to reverence to that element in the list.
If you want to access list indices as well as list elements while you are iterating, the usual idiom is to use g the "enumerate function:
for (i, x) in enumerate(my_list):
print "the", i, "number in the list is", x
The foreach loop should should be similar to the following desugared code:
my_iterator = iter(my_list)
while True:
try:
number = iter.next()
#Your code here
print 2*number
except StopIteration:
break

Pretty similar to this Java loop: Java for loop syntax: "for (T obj : objects)"
In python there's no need to declare variable type, that's why number has no type.

In Python you don't need to declare variables. In this case the number variable is defined by using it in the loop.
As for the loop construct itself, it's similar to the C++11 range-based for loop:
std::vector<int> my_list = { 1, 9, 3, 8, 5, 7 };
for (auto& number : my_list)
std::cout << 2 * number << '\n';
This can of course be implemented pre-C++11 using std::for_each with a suitable functor object (which may of course be a C++11 lambda expression).
Python does not have an equivalent to the normal C-style for loop.

I will try to explain the python for loop to you in much basic way as possible:
Let's say we have a list:
a = [1, 2, 3, 4, 5]
Before we jump into the for loop let me tell you we don't have to initialize the variable type in python while declaring variable.
int a, str a is not required.
Let's go to for loop now.
for i in a:
print 2*i
Now, what does it do?
The loop will start from the first element so,
i is replaced by 1 and it is multiplied by 2 and displayed. After it's done with 1 it will jump to 2.
Regarding your another question:
Python knows its variable type in it's execution:
>>> a = ['a', 'b', 'c']
>>> for i in a:
... print 2*i
...
aa
bb
cc
>>>

Python uses protocols (duck-typing with specially named methods, pre and post-fixed with double underscores). The equivalent in Java would be an interface or an abstract base class.
In this case, anything in Python which implements the iterator protocol can be used in a for loop:
class TheStandardProtocol(object):
def __init__(self):
self.i = 0
def __iter__(self):
return self
def __next__(self):
self.i += 1
if self.i > 15: raise StopIteration()
return self.i
# In Python 2 `next` is the only protocol method without double underscores
next = __next__
class TheListProtocol(object):
"""A less common option, but still valid"""
def __getitem__(self, index):
if index > 15: raise IndexError()
return index
We can then use instances of either class in a for loop and everything will work correctly:
standard = TheStandardProtocol()
for i in standard: # `__iter__` invoked to get the iterator
# `__next__` invoked and its return value bound to `i`
# until the underlying iterator returned by `__iter__`
# raises a StopIteration exception
print i
# prints 1 to 15
list_protocol = TheListProtocol()
for x in list_protocol: # Python creates an iterator for us
# `__getitem__` is invoked with ascending integers
# and the return value bound to `x`
# until the instance raises an IndexError
print x
# prints 0 to 15
The equivalent in Java is the Iterable and Iterator interface:
class MyIterator implements Iterable<Integer>, Iterator<Integer> {
private Integer i = 0;
public Iterator<Integer> iterator() {
return this;
}
public boolean hasNext() {
return i < 16;
}
public Integer next() {
return i++;
}
}
// Elsewhere
MyIterator anIterator = new MyIterator();
for(Integer x: anIterator) {
System.out.println(x.toString());
}

Pythonic enumeration of while loop

Python has an elegant way of automatically generating a counter variable in for loops: the enumerate function. This saves the need of initializing and incrementing a counter variable. Counter variables are also ugly because they are often useless once the loop is finished, yet their scope is not the scope of the loop, so they occupy the namespace without need (although I am not sure whether enumerate actually solves this).
My question is, whether there is a similar pythonic solution for while loops. enumerate won't work for while loops since enumerate returns an iterator. Ideally, the solution should be "pythonic" and not require function definitions.
For example:
x=0
c=0
while x<10:
x=int(raw_input())
print x,c
c+=1
In this case we would want to avoid initializing and incrementing c.
Clarification:
This can be done with an endless for loop with manual termination as some have suggested, but I am looking for a solution that makes the code clearer, and I don't think that solution makes the code clearer in this case.

Improvement (in readability, I'd say) to Ignacio's answer:
x = 0
for c in itertools.takewhile(lambda c: x < 10, itertools.count()):
x = int(raw_input())
print x, c
Advantages:
Only the while loop condition is in the loop header, not the side-effect raw_input.
The loop condition can depend on any condition that a normal while loop could. It's not necessary to "import" the variables referenced into the takewhile, as they are already visible in the lambda scope. Additionally it can depend on the count if you want, though not in this case.
Simplified: enumerate no longer appears at all.

Again with the itertools...
import itertools
for c, x in enumerate(
itertools.takewhile(lambda v: v < 10,
(int(raw_input()) for z in itertools.count())
)
):
print c, x

If you want zero initialization before the while loop, you can use a Singleton with a counter:
class Singleton(object):
_instance = None
def __new__(cls, *args, **kwargs):
if not cls._instance:
cls._instance = super(Singleton, cls).__new__(
cls, *args, **kwargs)
cls.count=0
else:
cls.count+=1
return cls._instance
Then there will only be one instance of Singleton and each additional instance just adds one:
>>> Singleton().count # initial instance
0
>>> Singleton().count
1
>>> Singleton().count
2
>>> Singleton().count
3
Then your while loop becomes:
while Singleton():
x=int(raw_input('x: '))
if x>10: break
print 'While loop executed',Singleton().count,'times'
Entering 1,2,3,11 it prints:
x: 1
x: 2
x: 3
x: 11
While loop executed 4 times
If you do not mind a single line initialization before the while loop, you can just subclass an interator:
import collections
class WhileEnum(collections.Iterator):
def __init__(self,stop=None):
self.stop=stop
self.count=0
def next(self): # '__next__' on Py 3, 'next' on Py 2
if self.stop is not None:
self.remaining=self.stop-self.count
if self.count>=self.stop: return False
self.count+=1
return True
def __call__(self):
return self.next()
Then your while loop becomes:
enu=WhileEnum()
while enu():
i=int(raw_input('x: '))
if i>10: break
print enu.count
I think the second is the far better approach. You can have multiple enumerators and you can also set a limit on how many loops to go:
limited_enum=WhileEnum(5)

I don't think it's possible to do what you want in the exact way you want it. If I understand right, you want a while loop that increments a counter each time through, without actually exposing a visible counter outside the scope of the loop. I think the way to do this would be to rewrite your while loop as a nonterminating for loop, and check the end condition manually. For your example code:
import itertools
x = 0
for c in itertools.count():
x = int(raw_input())
print x, c
if x >= 10:
break
The problem is that fundamentally you're doing iteration, with the counter. If you don't want to expose that counter, it needs to come from the loop construct. Without defining a new function, you're stuck with a standard loop and an explicit check.
On the other hand, you could probably also define a generator for this. You'd still be iterating, but you could at least wrap the check up in the loop construct.