I want to improve the performance of a recursive calculation of values
n = 100
def T(n,k):
q = 0
if n == 0 and k == 0:
return(1)
q = 1
if k>n or n<0:
return(0)
q = 1
if q != 1:
return(T(n-1,k-1)+n*T(n-1,k))
for i in range(n):
for n in range(i+1):
print(T(i,n))
print("*********")
However, I've only found ways to do this with functions that only take 1 argument, like this:
def mem(f):
memory = {}
def inner_function(x):
if x not in memory:
memory[x] = f(x)
return memory[x]
else:
return memory[x]
return inner_function
#mem
def fibonacci(n):
if n == 1 or n == 0:
return 1
else:
return fibonacci(n-1) + fibonacci(n-2)
I was thinking of doing a 2d array, but I don't know yet (assuming it's possible) how the idea of doing so, with lists of lists, would help.
You can use functools.lru_cache for this
from functools import lru_cache
#lru_cache(maxsize=32)
def T(n,k):
q = 0
if n == 0 and k == 0:
return(1)
q = 1
if k>n or n<0:
return(0)
q = 1
if q != 1:
return(T(n-1,k-1)+n*T(n-1,k))
You can use this decorator to memoize function calls, with this function in particular this will save up to the maxsize most recent calls.
Note in this particular case, the vast majority of the time is actually spent writing to console because of your print statements. If you remove this (but still leave your T(i,n) invocation) your code will complete almost instantaneously.
You can easily extend your mem decorator to work with variable *args parameters, looking them up in the memory. This would work with **kwargs, too, you'd have to convert them to a hashable type though, e.g. frozenset of tuples. And of course all parameters must be hashable for this.
def mem(f):
memory = {}
def inner_function(*args):
if args not in memory:
memory[args] = f(*args)
return memory[args]
return inner_function
Tested with your T function, works fine. In practice, however, you might still want to use functools.lru_cache.
Related
I am writing a recursive function to make permutations of digits from 0 to n. The program will return the th permutation that is obtained. It all works well but I had to use the cheap trick of defining count as a list, that is count=[0]. In this way I am using the properties of lists in order to properly update the variable count[0] at each iteration.
Ideally, what I would like to do is to define count as an integer number instead. However, this does not work because count is then updated only locally, within the scope of the function at the time it is called.
What is the proper way to count the iterations in a recursive function like this?
Below I show the code. It works, but I hate the way I am using count here.
import numpy as np
N=10
available=np.ones(N)
def permutations(array, count=[0], n=N, start=0, end=N, th=100):
if count[0]<th:
for i in range(n):
if available[i]:
array[start]=i
if end-start>1:
available[i]=0
permutations(array, count, n, start+1, end)
available[i]=1
else:
count[0]+=1
break
if count[0]==th:
a=''.join(str(i) for i in array)
return a
def main():
array=[0 for _ in range(N)]
count=[0]
print(permutations(array, count, N, start=0, end=N))
if __name__=="__main__":
main()
Not necessarily ideal but to answer the question, one could use a global variable as follows:
import numpy as np
N = 10
available = np.ones(N)
count = 0
def permutations(array, n=N, start=0, end=N, th=100):
global count
if count < th:
for i in range(n):
if available[i]:
array[start] = i
if end-start > 1:
available[i] = 0
permutations(array, n, start+1, end)
available[i] = 1
else:
count += 1
break
if count == th:
return ''.join(str(i) for i in array)
def main():
array = [0 for _ in range(N)]
print(permutations(array, N, start=0, end=N))
print(f'{count=}')
if __name__ == "__main__":
main()
Output:
0123495786
count=100
Different ways to update a variable from other scopes... and each with its own advantages and disadvantages (performance, access to the variable, ...):
with global approach (as Pingu did),
with nonlocal
and with function's attribute.
The example under consideration, the factorial function, is merely illustrative but it can be easily readapted to your case.
def fact_global(n):
global counter
counter += 1
if n == 1:
return 1
return n*fact_global(n-1)
def fact_nonlocal(n):
counter = 0
def __fact(n):
nonlocal counter
counter += 1
if n == 1:
return 1
return n*__fact(n-1)
return __fact(n)
def fact_attr(n):
fact_attr.counter = 0
def __fact(n):
fact_attr.counter += 1
if n == 1:
return 1
return n*__fact(n-1)
return __fact(n)
n = 10
# case: global
counter = 0
fact_global(n)
print('global', counter)
# case: nonlocal
fact_nonlocal(n)
import inspect
fr = inspect.currentframe()
print('nonlocal', fr.f_locals['counter']) # not recommended, just for curiosity!
# case: function's attribute
fact_attr(n)
print('attr', fact_attr.counter)
Retrieving the value of the variable under investigation is quite straightforward with a global-setting and with function's attribute but not trivial (and not recommended) with nonlocal (inspect is more a debugging tool).
Here a partial result of the workbench:
n=860
fact_nonlocal(n): 2.60644 ± 0.00586
fact_global(n): 2.74698 ± 0.02283
fact_attr(n): 3.01219 ± 0.00539
The results are of the same order of magnitude (due to limitations of the host only tested with a maximum of n=860 so not reliable for an asymptotic conclusion), in this case it seems that it doesn't really matter which one you choose but it is more important to focus on how you are going to access to the variable later in the program.
Below, I came up with artificially passing the reference var n2 from f2() to g2(x) instead of the global var n in f() and the nested g(). Any other better ways to replace global vars in this case?
from random import randint
# global value var
def f():
global n
n=0
def g():
global n
if randint(1,100)>50: n+=1
for _ in range(100): g()
print(n)
# local reference var
def f2():
n2=[0]
for _ in range(100): g2(n2)
print(n2[0])
def g2(x):
if randint(1,100)>50: x[0]+=1
Short answer: You are trying to pass by reference an immutable value (integer) and want to update it. Wrapping that in a tiny class, or list, or dict like you're doing is the way to go. But there are other ways if you are able to slightly modify your code.
Longer answer: (Note: This might not be a direct answer to your question.)
I understand this is an artificial example. But think about your real problem --Does g2() need to know that there is a variable that is supposed to update as a part of its invocation? Is there a way that the responsibility of updating a variable belongs to that which defines it? How about f2() is the one that defines the variables and also updates it? That way you can limit all the changes to that variable to a very small perimeter (f2()).
My approach in that case would be something like:
def f2():
n2 = 0
for _ in range(100):
n2 += g2()
print(n2)
def g2():
return 1 if randint(1,100)>50 else 0
From working with functional languages, and from trying to write reproducible tests, I've generally tried to adopt a rule that a function should declare all of its inputs as parameters, and produce all of its outputs as return values: to the maximum extent possible a function should never have side effects. Using the global keyword probably indicates you're breaking this rule.
For example, your "g" function takes the current state variable and either increments it or doesn't. You don't need a global for that (and you don't need it to be a nested function either):
from random import randint
def g(n):
"""Returns either n or n+1, with 50% probability."""
if randint(1,100)>50:
return n+1
else:
return n
Then the iterating function can call that a bunch of times:
def f():
"""Produces a number between 0 and 100 with $DISTRIBUTION."""
n = 0
for _ in range(100):
n = g(n)
return n
And finally at the top level:
if __name__ == '__main__':
print(f())
Since we're never totally sure about our code, we can write some tests.
def test_f():
n = f()
assert n >= 0 and n < 100
def test_g():
n = g(0)
assert n == 0 or n == 1
def test_g_dist():
count = 100
ns = [g(0) for _ in range(count)]
assert(all(n == 0 or n == 1) for n in ns)
zeros = len([n for n in ns if n == 0])
ones = len([n for n in ns if n == 1])
assert zeros + ones == count
# won't always pass; probability of failure left as an exercise
assert zeros > 0.45 * count and zeros < 0.55 * count
Notice that I can call f() and g(n) as many times as I want and they'll never interfere with anything else. Running my own unit tests at startup time would be a little unusual, but I'm free to if that's what I want to do.
I am a novice at python, but was surprised at how slow this recursive call took to execute:
def daH(m:int):
if m == 1:
return int(1)
else:
if m <= .5 * (daH(m-1) * (daH(m-1) +1)):
return int(daH(m-1))
else:
return int(daH(m-1) + 1)
print(daH(10)) # prints 4
print(daH(11)) # prints 5
print(daH(15)) # prints 5
print(daH(16)) # prints 6
print(daH(106)) # prints ??? (gave up waiting)
I ran it on IDLE, python 3.6. I added the INT stuff but it did not help. I had no problems running the standard factorial recursion and printing factorial(106).
Can this attempt at recursion be salvaged?
You are computing daH(m-1) 3 times, making the algorithm slower than necessary. Instead, calculate it just once and bind the result to a local variable. (Also, not necessary to cast to int)
def daH(m:int):
if m == 1:
return 1
else:
r = daH(m-1)
if m <= .5 * r * (r + 1):
return r
else:
return r + 1
Calling the function three times instead of once may not seem like much, but remember that those calls will stack exponentially! You call it three times, and each of those again call it three times, and so on. This results in a complexity of O(3m), which even for m=15 results in about 15 million recursive calls, as opposed to the 15 that are actually necessary.
Since daH is a mathematical function and always produces the same output for a given input, you can cache the values and use them.
Sample output
$ time python use_cache.py
4
5
5
6
15
real 0m0.194s
user 0m0.144s
sys 0m0.047s
$
Caching decorator
$ cat use_cache.py
def use_cache_when_possible(func):
cache = {}
def foo(*args):
if args not in cache:
cache[args] = func(*args)
return cache[args]
return foo
#use_cache_when_possible
def daH(m):
if m == 1:
return int(1)
else:
if m <= .5 * (daH(m-1) * (daH(m-1) +1)):
return int(daH(m-1))
else:
return int(daH(m-1) + 1)
print(daH(10))
print(daH(11))
print(daH(15))
print(daH(16))
print(daH(106))
$
You are calling daH unnecessary number of times.
Your code make 3 calls to daH(m-1) for every call daH(m), which makes exponential complexity, which scales badly, i.e performs badly on larger values of m.
You can save the result of call todaH and use it in places where you called daH with same input, instead of calling the function multiple times.
def daH(m:int):
if m == 1:
return 1
else:
ret_val = daH(m-1)
if m <= .5 * (ret_val * (ret_val +1)):
return ret_val
else:
return ret_val + 1
Here is the code I currently have.
def fibonacci(n):
if n == 1:
return 1
elif n == 2:
return 1
else:
value = fibonacci(n - 1) + fibonacci(n - 2)
return value
This currently takes quite some time to calculate values greater than n = 30. Is there a more computationally efficient method to accomplish this?
Adding a value cache to trade some memory for a reduced processing time can be a useful method. A purely recursive program will attempt to calculate values over and over again, however this takes time for larger values. If the values do not change, then storing them can be helpful. It is important to note, however, that should values be volatile you might need a different approach.
fibonacci_value_cache = {}
def fibonacci(n):
if n == 1:
return 1
elif n == 2:
return 1
elif n in fibonacci_value_cache:
return fibonacci_value_cache[n]
else:
fibonacci_value_cache[n] = fibonacci(n - 1) + fibonacci(n - 2)
return fibonacci_value_cache[n]
n = 100
print("Fib " + str(n) + ": " + str(fibonacci(n)))
Here, we check if the value is in the dictionary and return it if it is, otherwise we calculate it and add it to the dictionary. This means that we are make better use of the processor by not calculating the same value multiple times.
There's a recipe for a decorator that uses as an example exactly what you want. It's named Memoize in the PythonDecoratorLibrary.
It may seem like overkill, but having the memoized decorator around could be useful for other future tasks. That said, here it is in its entirety (although I changed the print at the end):
import collections
import functools
class memoized(object):
'''Decorator. Caches a function's return value each time it is called.
If called later with the same arguments, the cached value is returned
(not reevaluated).
'''
def __init__(self, func):
self.func = func
self.cache = {}
def __call__(self, *args):
if not isinstance(args, collections.Hashable):
# uncacheable. a list, for instance.
# better to not cache than blow up.
return self.func(*args)
if args in self.cache:
return self.cache[args]
else:
value = self.func(*args)
self.cache[args] = value
return value
def __repr__(self):
'''Return the function's docstring.'''
return self.func.__doc__
def __get__(self, obj, objtype):
'''Support instance methods.'''
return functools.partial(self.__call__, obj)
#memoized
def fibonacci(n):
"Return the nth fibonacci number."
if n in (0, 1):
return n
return fibonacci(n-1) + fibonacci(n-2)
print(fibonacci(12))
Using idea of Dynamic Programming, and store the intermediate results to save computational cost, it could be very efficient. The code below cost less than 0.02s for n=10000 on my laptop.
def fib(n): # return Fibonacci series up to n
result = []
a, b = 0, 1
for i in range(n):
result.append(b)
a, b = b, a + b
return result
No need for caching/memoization. Here's a Python 3 implementation that expresses the Fibonacci sequence as powers of a matrix, then does efficient exponentiation via halving and squaring. The result is O(log n) in both time and storage.
def matrix_fib(n):
if n == 1:
return [0,1]
else:
f = matrix_fib(n // 2)
c = f[0] * f[0] + f[1] * f[1]
d = f[1] * (f[1] + 2 * f[0])
return [c,d] if (n & 1) == 0 else [d,c+d]
def fib(n):
return n if n == 0 else matrix_fib(n)[1]
print(fib(1000000))
On my laptop this coughs up the value of the millionth Fibonacci number in a little over half a second, and the bulk of that is probably in the big integer arithmetic and formatting of the output—the result is ridiculously large. You don't need to worry about stack overflow, though. The call stack depth for this is only log2(1000000) = 20.
My problem is that I want:
and then I want it to get evaluted in the calc_energy function 360 times
and then store the result so I could count the year average, I want to calculate the energy average..
Should I call the function instead of return in def_decide(v)???
Could someone help me get in on the right track?
def calc_v(latitud):
for t in range(0,360):
v=(23.5*math.sin(math.pi*(t-80)/180)+90-latitud)/90
decide_v(v):
def decide_v(v):
if 0<v<1:
return(v**2)
if v>=1:
return(1)
if v<=0:
return(0)
def calc_energy(v):
W=self.area*random(0,1)*self.sundigit*v
return W
def sort(self,W):
W+=100/360
You can make a generator from calc_v and then use it as you would use a list of values (notice yield instead of return):
def calc_v(latitude):
for t in range(0,360):
v=(23.5*math.sin(math.pi*(t-80)/180)+90-latitude)/90
yield v
def decide_v(v):
if 0<v<1:
return v**2
elif v>=1:
return 1
else:
return 0
for v in calc_v(latitude):
print decide_v(v)
You can use recursion which is a function calling itself. Below you can see the function factorial is called within itself. http://www.python-course.eu/recursive_functions.php
def factorial(n):
print("factorial has been called with n = " + str(n))
if n == 1:
return 1
else:
res = n * factorial(n-1)
print("intermediate result for ", n, " * factorial(" ,n-1, "): ",res)
return res
I get the feeling that you don't really want to know how to loop a function. I think you just want to call your function in the example that you gave.
def calc_v(latitud):
for t in range(0,366):
v=(23.5*math.sin(math.pi*(t-80)/180)+90-latitud)/90
decide_v(v) # return v
def decide_v(v):
if 0<v<1:
print(v**2)
if v>=1:
print(1)
if v<=0:
print(0)
When you use the return statement in a function then you will leave the loop and the function, so your for loop will only run through t=0 then it will break out of the function. I think you just want to call your function and give it the variable v.