I need to add a single decorator - my_decorator - to multiple functions. Instead of adding #my_decorator above each function definition, I'd like to add it to all the functions in an isolated code block, so I have everything in one place.
Is there a clean, Pythonic way to do this? Obviously I can do the following, but it gets a bit repetitive. If I ever change the decorator name, I'll need to update it in every single line.
function1 = my_decorator(function1)
function2 = my_decorator(function2)
function3 = my_decorator(function3)
My second thought was to put the functions in a list and iterate through it, but neither of the below approaches worked. They modify the value at each list index, while leaving the original functions unchanged, so I would have to call, for example, function_list[0]() to get the function which has the decorator added.
# approach one - a list comprehension
function_list = [function1, function2, function3]
function_list = [my_decorator(function) for function in function_list]
# approach two - a for loop
function_list = [function1, function2, function3]
for index in range(len(function_list)):
function_list[index] = my_decorator(function_list[index])
I finally found an approach that works, but using exec() doesn't seem very Pythonic, so I'm wondering if there's a better way.
function_list = ['function1', 'function2', 'function3']
for index in range(len(function_list)):
exec("{0} = my_decorator({0})".format(function_list[index]))
Here is the code I was testing this out on, which would add a decorator that prints "Hello" before the function is called:
def my_decorator(function):
def wrapper(*args, **kwargs):
print("Hello!", end=' ')
return function(*args, **kwargs)
return wrapper
def ask_about_day(name):
print("How is your day going, {0}?".format(name))
def comment_about_weather():
print("It's lovely outside today, isn't it?")
# assign decorator using exec() - possibly unpythonic
function_list = ['ask_about_day', 'comment_about_weather']
for index in range(len(function_list)):
exec("{0} = my_decorator({0})".format(function_list[index]))
def main():
name = input("What is your name? ").capitalize()
if name:
ask_about_day(name)
else:
comment_about_weather()
main()
This is almost certainly a really bad idea.
What you're trying to do is replace arbitrary globals, by name. The right way to do that is to call globals and modify the resulting dict.
For example:
g = globals()
for func in function1, function2, function3:
g[func.__name__] = my_decorator(func)
Of course this looks ugly and hacky, but that's because what you're doing is inherently ugly and hacky, so Python isn't going to go out of its way to make it look nice so you can pretend otherwise.
Also, notice that however you do this, you're forced to repeat the names of all of your functions at least once. Is that really worth it to avoid repeating the name of your decorator the same number of times? It's at least as much typing, and a lot easier to get silently wrong.
After reading the following, I think I understand the value of wrapping even the simplest of scripts in a main() function.
Should I use a main() method in a simple Python script?
Why use def main()?
Should I define all my functions inside or outside main()?
Is there a right or wrong way? What are the advantages and disadvantages of both approaches?
I would discourage defining functions inside of main(), especially if you have multiple files in your Python script. Any function B defined inside of function A cannot be used by anything outside of function A, severely limiting its usability. Functions defined inside main() cannot be imported elsewhere, for example.
Defining functions inside of main() lets you easily override other functions you may have written elsewhere that have the same name, but the instances where this is really useful are few and far between, and you should not be doing it as a general practice. Overall there are many more reasons for defining functions outside of main() than there are for defining them inside, and if you're learning Python that's definitely how you should handle it.
If you define a function inside of the main function, you won't be able to use it from the outside. Here is an example:
def outer():
print "outer function"
def main():
def inner():
print "inner function"
inner()
if __name__ == "__main__":
main() # outputs: "inner function"
outer() # outputs: "outer function"
inner() # fails!
So main is just an entry point not usually a good place for lots of methods. Best practice would be to separate functionality among classes. It also makes it easier to manage in the future if changes need to be made. Hope this helps.
Hello and good question.
Just by "being use to" main() functions, many Python programmer with former experience in C++, Java, and C# love to use def name = "main" for of some sort. While "old school" programmers like this, Python is a very versitile, free form programming (actually it is a scripting language) what it does not need a main function.
If you were to make a basic calculator program, you can just make a function called "calc", and Python will be happy. I have made a GUI Vector Calculator w/o a "main" function, so there is Zero need.
def calc(x, y):
sum = x + y
return sum
I often do interactive work in Python that involves some expensive operations that I don't want to repeat often. I'm generally running whatever Python file I'm working on frequently.
If I write:
import functools32
#functools32.lru_cache()
def square(x):
print "Squaring", x
return x*x
I get this behavior:
>>> square(10)
Squaring 10
100
>>> square(10)
100
>>> runfile(...)
>>> square(10)
Squaring 10
100
That is, rerunning the file clears the cache. This works:
try:
safe_square
except NameError:
#functools32.lru_cache()
def safe_square(x):
print "Squaring", x
return x*x
but when the function is long it feels strange to have its definition inside a try block. I can do this instead:
def _square(x):
print "Squaring", x
return x*x
try:
safe_square_2
except NameError:
safe_square_2 = functools32.lru_cache()(_square)
but it feels pretty contrived (for example, in calling the decorator without an '#' sign)
Is there a simple way to handle this, something like:
#non_resetting_lru_cache()
def square(x):
print "Squaring", x
return x*x
?
Writing a script to be executed repeatedly in the same session is an odd thing to do.
I can see why you'd want to do it, but it's still odd, and I don't think it's unreasonable for the code to expose that oddness by looking a little odd, and having a comment explaining it.
However, you've made things uglier than necessary.
First, you can just do this:
#functools32.lru_cache()
def _square(x):
print "Squaring", x
return x*x
try:
safe_square_2
except NameError:
safe_square_2 = _square
There is no harm in attaching a cache to the new _square definition. It won't waste any time, or more than a few bytes of storage, and, most importantly, it won't affect the cache on the previous _square definition. That's the whole point of closures.
There is a potential problem here with recursive functions. It's already inherent in the way you're working, and the cache doesn't add to it in any way, but you might only notice it because of the cache, so I'll explain it and show how to fix it. Consider this function:
#lru_cache()
def _fact(n):
if n < 2:
return 1
return _fact(n-1) * n
When you re-exec the script, even if you have a reference to the old _fact, it's going to end up calling the new _fact, because it's accessing _fact as a global name. It has nothing to do with the #lru_cache; remove that, and the old function will still end up calling the new _fact.
But if you're using the renaming trick above, you can just call the renamed version:
#lru_cache()
def _fact(n):
if n < 2:
return 1
return fact(n-1) * n
Now the old _fact will call fact, which is still the old _fact. Again, this works identically with or without the cache decorator.
Beyond that initial trick, you can factor that whole pattern out into a simple decorator. I'll explain step by step below, or see this blog post.
Anyway, even with the less-ugly version, it's still a bit ugly and verbose. And if you're doing this dozens of times, my "well, it should look a bit ugly" justification will wear thin pretty fast. So, you'll want to handle this the same way you always factor out ugliness: wrap it in a function.
You can't really pass names around as objects in Python. And you don't want to use a hideous frame hack just to deal with this. So you'll have to pass the names around as strings. ike this:
globals().setdefault('fact', _fact)
The globals function just returns the current scope's global dictionary. Which is a dict, which means it has the setdefault method, which means this will set the global name fact to the value _fact if it didn't already have a value, but do nothing if it did. Which is exactly what you wanted. (You could also use setattr on the current module, but I think this way emphasizes that the script is meant to be (repeatedly) executed in someone else's scope, not used as a module.)
So, here that is wrapped up in a function:
def new_bind(name, value):
globals().setdefault(name, value)
… which you can turn that into a decorator almost trivially:
def new_bind(name):
def wrap(func):
globals().setdefault(name, func)
return func
return wrap
Which you can use like this:
#new_bind('foo')
def _foo():
print(1)
But wait, there's more! The func that new_bind gets is going to have a __name__, right? If you stick to a naming convention, like that the "private" name must be the "public" name with a _ prefixed, we can do this:
def new_bind(func):
assert func.__name__[0] == '_'
globals().setdefault(func.__name__[1:], func)
return func
And you can see where this is going:
#new_bind
#lru_cache()
def _square(x):
print "Squaring", x
return x*x
There is one minor problem: if you use any other decorators that don't wrap the function properly, they will break your naming convention. So… just don't do that. :)
And I think this works exactly the way you want in every edge case. In particular, if you've edited the source and want to force the new definition with a new cache, you just del square before rerunning the file, and it works.
And of course if you want to merge those two decorators into one, it's trivial to do so, and call it non_resetting_lru_cache.
However, I'd keep them separate. I think it's more obvious what they do. And if you ever want to wrap another decorator around #lru_cache, you're probably still going to want #new_bind to be the outermost decorator, right?
What if you want to put new_bind into a module that you can import? Then it's not going to work, because it will be referring to the globals of that module, not the one you're currently writing.
You can fix that by explicitly passing your globals dict, or your module object, or your module name as an argument, like #new_bind(__name__), so it can find your globals instead of its. But that's ugly and repetitive.
You can also fix it with an ugly frame hack. At least in CPython, sys._getframe() can be used to get your caller's frame, and frame objects have a reference to their globals namespace, so:
def new_bind(func):
assert func.__name__[0] == '_'
g = sys._getframe(1).f_globals
g.setdefault(func.__name__[1:], func)
return func
Notice the big box in the docs that tells you this is an "implementation detail" that may only apply to CPython and is "for internal and specialized purposes only". Take this seriously. Whenever someone has a cool idea for the stdlib or builtins that could be implemented in pure Python, but only by using _getframe, it's generally treated almost the same as an idea that can't be implemented in pure Python at all. But if you know what you're doing, and you want to use this, and you only care about present-day versions of CPython, it will work.
There is no persistent_lru_cache in the stdlib. But you can build one pretty easily.
The functools source is linked directly from the docs, because this is one of those modules that's as useful as sample code as it is for using it directly.
As you can see, the cache is just a dict. If you replace that with, say, a shelf, it will become persistent automatically:
def persistent_lru_cache(filename, maxsize=128, typed=False):
"""new docstring explaining what dbpath does"""
# same code as before up to here
def decorating_function(user_function):
cache = shelve.open(filename)
# same code as before from here on.
Of course that only works if your arguments are strings. And it could be a little slow.
So, you might want to instead keep it as an in-memory dict, and just write code that pickles it to a file atexit, and restores it from a file if present at startup:
def decorating_function(user_function):
# ...
try:
with open(filename, 'rb') as f:
cache = pickle.load(f)
except:
cache = {}
def cache_save():
with lock:
with open(filename, 'wb') as f:
pickle.dump(cache, f)
atexit.register(cache_save)
# …
wrapper.cache_save = cache_save
wrapper.cache_filename = filename
Or, if you want it to write every N new values (so you don't lose the whole cache on, say, an _exit or a segfault or someone pulling the cord), add this to the second and third versions of wrapper, right after the misses += 1:
if misses % N == 0:
cache_save()
See here for a working version of everything up to this point (using save_every as the "N" argument, and defaulting to 1, which you probably don't want in real life).
If you want to be really clever, maybe copy the cache and save that in a background thread.
You might want to extend the cache_info to include something like number of cache writes, number of misses since last cache write, number of entries in the cache at startup, …
And there are probably other ways to improve this.
From a quick test, with save_every=1, this makes the cache on both get_pep and fib (from the functools docs) persistent, with no measurable slowdown to get_pep and a very small slowdown to fib the first time (note that fib(100) has 100097 hits vs. 101 misses…), and of course a large speedup to get_pep (but not fib) when you re-run it. So, just what you'd expect.
I can't say I won't just use #abarnert's "ugly frame hack", but here is the version that requires you to pass in the calling module's globals dict. I think it's worth posting given that decorator functions with arguments are tricky and meaningfully different from those without arguments.
def create_if_not_exists_2(my_globals):
def wrap(func):
if "_" != func.__name__[0]:
raise Exception("Function names used in cine must begin with'_'")
my_globals.setdefault(func.__name__[1:], func)
def wrapped(*args):
func(*args)
return wrapped
return wrap
Which you can then use in a different module like this:
from functools32 import lru_cache
from cine import create_if_not_exists_2
#create_if_not_exists_2(globals())
#lru_cache()
def _square(x):
print "Squaring", x
return x*x
assert "_square" in globals()
assert "square" in globals()
I've gained enough familiarity with decorators during this process that I was comfortable taking a swing at solving the problem another way:
from functools32 import lru_cache
try:
my_cine
except NameError:
class my_cine(object):
_reg_funcs = {}
#classmethod
def func_key (cls, f):
try:
name = f.func_name
except AttributeError:
name = f.__name__
return (f.__module__, name)
def __init__(self, f):
k = self.func_key(f)
self._f = self._reg_funcs.setdefault(k, f)
def __call__(self, *args, **kwargs):
return self._f(*args, **kwargs)
if __name__ == "__main__":
#my_cine
#lru_cache()
def fact_my_cine(n):
print "In fact_my_cine for", n
if n < 2:
return 1
return fact_my_cine(n-1) * n
x = fact_my_cine(10)
print "The answer is", x
#abarnert, if you are still watching, I'd be curious to hear your assessment of the downsides of this method. I know of two:
You have to know in advance what attributes to look in for a name to associate with the function. My first stab at it only looked at func_name which failed when passed an lru_cache object.
Resetting a function is painful: del my_cine._reg_funcs[('__main__', 'fact_my_cine')], and the swing I took at adding a __delitem__ was unsuccessful.
I learned that functions should perform a well-defined action and should be named according to what they do. For example, a function that adds its two arguments could be written:
def adder(arg1, arg2):
return arg1 + arg2
Here, adder has a well-defined role and its name says exactly what it does. Now, if a game loop was implemented as a function:
def mainloop(fps):
while True:
# DO STUFF
tick(fps)
would it still be right to say that the function is doing something and has a well-defined role? If so, what exactly are the advantages of implementing the loop as a function instead of leaving it in the global scope?
Due to an implementation detail, the same code in a function can run faster than when it is in the global namespace.
It would still be good form to have your loop outside of a function, there is nothing wrong with that. For one of my own pygame projects, I had to do that because otherwise there was scoping issues. An advantage of putting it into a function is that you could potentially access it from another python file:
from game import mainloop
mainloop(65)
Does python have an equivalent to Tcl's uplevel command? For those who don't know, the "uplevel" command lets you run code in the context of the caller. Here's how it might look in python:
def foo():
answer = 0
print "answer is", answer # should print 0
bar()
print "answer is", answer # should print 42
def bar():
uplevel("answer = 42")
It's more than just setting variables, however, so I'm not looking for a solution that merely alters a dictionary. I want to be able to execute any code.
In general, what you ask is not possible (with the results you no doubt expect). E.g., imagine the "any code" is x = 23. Will this add a new variable x to your caller's set of local variables, assuming you do find a black-magical way to execute this code "in the caller"? No it won't -- the crucial optimization performed by the Python compiler is to define once and for all, when def executes, the exact set of local variables (all the barenames that get assigned, or otherwise bound, in the function's body), and turn every access and setting to those barenames into very fast indexing into the stackframe. (You could systematically defeat that crucial optimization e.g. by having an exec '' at the start of every possible caller -- and see your system's performance crash through the floor in consequence).
Except for assigning to the caller's local barenames, exec thecode in thelocals, theglobals may do roughly what you want, and the inspect module lets you get the locals and globals of the caller in a semi-reasonable way (in as far as deep black magic -- which would make me go postal on any coworker suggesting it be perpetrated in production code -- can ever be honored with the undeserved praise of calling it "semi-reasonable", that is;-).
But you do specify "I want to be able to execute any code." and the only solution to that unambiguous specification (and thanks for being so precise, as it makes answering easier!) is: then, use a different programming language.
Is the third party library written in Python? If yes, you could rewrite and rebind the function "foo" at runtime with your own implementation. Like so:
import third_party
original_foo = third_party.foo
def my_foo(*args, **kwds):
# do your magic...
original_foo(*args, **kwds)
third_party.foo = my_foo
I guess monkey-patching is slighly better than rewriting frame locals. ;)