We Keep Coding

Understanding Python's Callback(ExitStack) example

I am trying to understand the examples for replacing the use of try-finally and flag variables in Python's documentation
According to the documentation instead of:
cleanup_needed = True
try:
result = perform_operation()
if result:
cleanup_needed = False
finally:
if cleanup_needed:
cleanup_resources()
we could use a small ExitStack-based helper class Callback like this (I added the perform_operation and cleanup_resources function):
from contextlib import ExitStack
class Callback(ExitStack):
def __init__(self, callback, /, *args, **kwds):
super(Callback, self).__init__()
self.callback(callback, *args, **kwds)
def cancel(self):
self.pop_all()
def perform_operation():
return False
def cleanup_resources():
print("Cleaning up resources")
with Callback(cleanup_resources) as cb:
result = perform_operation()
if result:
cb.cancel()
I think the code simulates the exceptional case, where the perform_operation() did not run smoothly and a cleanup is needed (perform_operation() returned False). The Callback class magically takes care of the running the cleanup_resources() function (I can't quite understand why, by the way).
Then I simulated the normal case, where everything runs smoothly and no cleanup is needed, I changed the code to make perform_operation() return True instead. In this case, however, the cleanup_resources function also runs and the code errors out:
$ python minimal.py
Cleaning up resources
Traceback (most recent call last):
File "minimal.py", line 26, in <module>
cb.cancel()
File "minimal.py", line 12, in cancel
self.pop_all()
File "C:\ProgramData\Anaconda3\envs\claw\lib\contextlib.py", line 390, in pop_all
new_stack = type(self)()
TypeError: __init__() missing 1 required positional argument: 'callback'
Can you explain what exactly is going on here and how this whole ExitStack and callack stuff works?

missing 1 required positional argument in click CLI

I've written a CLI with click originally as a module and it worked fine. But since my project got bigger I now need to have attributes the CLI can work with, so I tried to turn it into a class, but I'm running into an error doing it. My code is like the following:
import click
import click_repl
import os
from prompt_toolkit.history import FileHistory
class CLI:
def __init__(self):
pass
#click.group(invoke_without_command=True)
#click.pass_context
def cli(self, ctx):
if ctx.invoked_subcommand is None:
ctx.invoke(self.repl)
#cli.command()
def foo(self):
print("foo")
#cli.command()
def repl(self):
prompt_kwargs = {
'history': FileHistory(os.path.expanduser('~/.repl_history'))
}
click_repl.repl(click.get_current_context(), prompt_kwargs)
def main(self):
while True:
try:
self.cli(obj={})
except SystemExit:
pass
if __name__ == "__main__":
foo = CLI()
foo.main()
Without all the selfs and the class CLI: the CLI is working as expected, but as a class it runs into an error: TypeError: cli() missing 1 required positional argument: 'ctx' I don't understand why this happens. As far as I know calling self.cli() should pass self automatically, thus obj={} should be passed as ctx.obj, so it shouldn't make any difference to cli if it's wrapped in a class or not.
Can someone explain to me, why this happens and more important, how I can fix it?
In case it's relevant here is the complete error stack trace:
Traceback (most recent call last):
File "C:/Users/user/.PyCharmCE2018.2/config/scratches/exec.py", line
37, in <module>
foo.main()
File "C:/Users/user/.PyCharmCE2018.2/config/scratches/exec.py", line
30, in main
self.cli(obj={})
File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site- packages\click\core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\click\core.py", line 717, in main
rv = self.invoke(ctx)
File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\click\core.py", line 1114, in invoke
return Command.invoke(self, ctx)
File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\click\core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\click\core.py", line 555, in invoke
return callback(*args, **kwargs)
File "C:\Users\user\AppData\Local\Programs\Python\Python37\lib\site-packages\click\decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
TypeError: cli() missing 1 required positional argument: 'ctx'
EDIT: The problem seems to be the pass_context call. Usually pass_context would provide the current context as first parameter to the function, so that obj={} would be passed to the context instance. But since I wrapped the click-group into a class, the first spot is taken by the self-reference, so that the current context can't be passed to the function. Any ideas of how to work around that?
I tried changing def cli() the following way:
#click.group(invoke_without_command=True)
def cli(self):
ctx = click.get_current_context()
ctx.obj = {}
if ctx.invoked_subcommand is None:
ctx.invoke(self.repl)
So I don't pass the context by call avoiding a conflict with self, but if I try to run this with self.cli() error TypeError: cli() missing 1 required positional argument: 'self' happens.
Calling it with self.cli(self) runs into TypeError: 'CLI' object is not iterable

I am afraid the click library is not designed to work as a class. Click makes use of decorators. Don't take decorators too lightly. Decorators literally take your function as argument and return a different function.
For example:
#cli.command()
def foo(self):
Is something in line of
foo = cli.command()(foo)
So, I am afraid that click has not support to decorate functions bound to classes, but can only decorate functions that are unbound. So, basically the solution to your answer is, don't use a class.
You might be wondering how to organize your code now. Most languages present you the class as an unit of organization.
Python however goes one step further and gives you modules as well. Basically a file is a module and within this file everything you put in there is automatically associated with that file as a module.
So, just name a file cli.py and create your attributes as global variables. This might give you other problems, since you cannot alter global variables in a function scope, but you can use a class to contain your variables instead.
class Variables:
pass
variables = Variables()
variables.something = "Something"
def f():
variables.something = "Nothing"

Name of a Python function in a stack trace

In both Python2 and Python3, in the stack trace the __name__ of a function is not used, the original name (the one that is specified after def) is used instead.
Consider the example:
import traceback
def a():
return b()
def b():
return c()
def c():
print("\n".join(line.strip() for line in traceback.format_stack()))
a.__name__ = 'A'
b.__name__ = 'B'
c.__name__ = 'C'
a();
The output is:
File "test.py", line 16, in <module>
a();
File "test.py", line 4, in a
return b()
File "test.py", line 7, in b
return c()
File "test.py", line 10, in c
print("\n".join(line.strip() for line in traceback.format_stack()))
Why so? How do I change the name that is used in the stack trace? Where is the __name__ attribute used then?

So, basically every function has three things that can be considered being name of the function:
The original name of the code block
It's stored in the f.__code__.co_name (where f is the function object). If you use def orig_name to create function, orig_name is that name. For lambas it's <lambda>.
This attribute is readonly and can't be changed. So the only way to create function with the custom name in runtime I'm aware of is exec:
exec("""def {name}():
print '{name}'
""".format(name='any')) in globals()
any() # prints 'any'
(There is also more low-level way to do this that was mentioned in a comment to the question.)
The immutability of co_name actually makes sense: with that you can be sure that the name you see in the debugger (or just stack trace) is exactly the same you see in the source code (along with the filename and line number).
The __name__ attribute of the function object
It's also aliased to func_name.
You can modify it (orig_name.__name__ = 'updated name') and you surely do on a daily basis: #functools.wraps copies the __name__ of the decorated function to the new one.
__name__ is used by tools like pydoc, that's why you need #functools.wraps: so you don't see the technical details of every decorator in your documentation. Look at the example:
from functools import wraps
def decorator1(f):
def decorated(*args, **kwargs):
print 'start1'
f(*args, **kwargs)
return decorated
def decorator2(f):
#wraps(f)
def decorated(*args, **kwargs):
print 'start2'
f(*args, **kwargs)
return decorated
#decorator1
def test1():
print 'test1'
#decorator2
def test2():
print 'test2'
Here is the pydoc output:
FUNCTIONS
decorator1(f)
decorator2(f)
test1 = decorated(*args, **kwargs)
test2(*args, **kwargs)
With wraps there is no sign of decorated in the documentation.
Name of the reference
One more thing that can be called function name (though it hardly is) is the name of a variable or an attribute where reference to that function is stored.
If you create function with def name, the name attribute will be added to the current scope. In case of lambda you should assign the result to some variable: name = lambda: None.
Obviously you can create more than one reference to the same function and all that references can have different names.
The only way all that three things are connected to each other is the def foo statement that creates function object with both __name__ and __code__.co_name equal to foo and assign it to the foo attribute of the current scope. But they are not bound in any way and can be different from each other:
import traceback
def make_function():
def orig_name():
"""Docstring here
"""
traceback.print_stack()
return orig_name
globals()['name_in_module'] = make_function()
name_in_module.__name__ = 'updated name'
name_in_module()
Output:
File "my.py", line 13, in <module>
name_in_module()
File "my.py", line 7, in orig_name
traceback.print_stack()
Pydoc:
FUNCTIONS
make_function()
name_in_module = updated name()
Docstring here
I thank other people for comments and answers, they helped me to organize my thoughts and knowledge.

Tried to explore the CPython implementation, definitely not an expert. As pointed out in the comments, when the stack entry of f is printed, the attribute f.__code__.co_name is used. Also, f.__name__ is initially set to f.__code__.co_name, but when you modify the former, the latter is not modified accordingly.
Therefore, I tried to modify that directly, but it is not possible:
>>> f.__code__.co_name = 'g'
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: readonly attribute
>>>
Why are there two ways to say a function's name? Well, according to the documentation, __name__ is defined for "class, function, method, descriptor, or generator instance", so in the case of functions it maps to that attribute, for other objects it will map to something else.

global name 'display' is not found

I'm currently designing a code for a troubleshooter. My problem is that when I press "other", there should be a data entry form label, which there is. But when i type something into it and press continue, it shows an error that says "
Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Python33\lib\tkinter\__init__.py", line 1482, in __call__
return self.func(*args)
File"R:\ICT\ControlledAssessment_15_17\Computing\Unit3\RiddhiSharma\FinalTroubleshooter.py", line 15, in click
display.insert(END, key)
NameError: global name 'display' is not defined
I think the problem in my actual code is here:
def click():
display.insert(END, key)
output.delete(0.0, END)
try:
entered_text=entry.get()
definition = my_glossary[entered_text]
except:
definition = "There is no entry for this word."
output.insert(END, definition)

Apparently display is not part of the tkinter module or it is not defined altogether. So when you try to use it, it raises an error because you have not defined display as a variable, function, class etc. So define display is a function that takes two arguments, END and key.
When defining display. You could do something like this:
display = 123
You could also make it a ENTRY, or whatever display is supposed to be. In your case, display doesn't have to be equal to 123. It could be equal to a class:
display = Display()
Just define Display() first... but display can be anything you want, as long you define it and use it correctly.

#PythonMaster thanks a lot for your help. To solve my problem I replaced the display function and wrote the code out like this:
def click():
entered_text = entry.get()
output.delete(0.0, END)
try:
definition = my_glossary[entered_text]
except:
definition = "There is no entry for this word."
output.insert(END, definition)

Is it possible to create a dynamic localized scope in Python?

I have a scenario where I'm dynamically running functions at run-time and need to keep track of a "localized" scope. In the example below, "startScope" and "endScope" would actually be creating levels of "nesting" (in reality, the stuff contained in this localized scope isn't print statements...it's function calls that send data elsewhere and the nesting is tracked there. startScope / endScope just set control flags that are used to start / end the current nesting depth).
This all works fine for tracking the nested data, however, exceptions are another matter. Ideally, an exception would result in "falling out" of the current localized scope and not end the entire function (myFunction in the example below).
def startScope():
#Increment our control object's (not included in this example) nesting depth
control.incrementNestingDepth()
def endScope():
#Decrement our control object's (not included in this example) nesting depth
control.decrementNestingDepth()
def myFunction():
print "A"
print "B"
startScope()
print "C"
raise Exception
print "D"
print "This print statement and the previous one won't get printed"
endScope()
print "E"
def main():
try:
myFunction()
except:
print "Error!"
Running this would (theoretically) output the following:
>>> main()
A
B
C
Error!
E
>>>
I'm quite certain this isn't possible as I've written it above - I just wanted to paint a picture of the sort of end-result I'm trying to achieve.
Is something like this possible in Python?
Edit: A more relevant (albeit lengthy) example of how this is actually being used:
class Log(object):
"""
Log class
"""
def __init__(self):
#DataModel is defined elsewhere and contains a bunch of data structures / handles nested data / etc...
self.model = DataModel()
def Warning(self, text):
self.model.put("warning", text)
def ToDo(self, text):
self.model.put("todo", text)
def Info(self, text):
self.model.put("info", text)
def StartAdvanced(self):
self.model.put("startadvanced")
def EndAdvanced(self):
self.model.put("endadvanced")
def AddDataPoint(self, data):
self.model.put("data", data)
def StartTest(self):
self.model.put("starttest")
def EndTest(self):
self.model.put("endtest")
def Error(self, text):
self.model.put("error", text)
#myScript.py
from Logger import Log
def test_alpha():
"""
Crazy contrived example
In this example, there are 2 levels of nesting...everything up to StartAdvanced(),
and after EndAdvanced() is included in the top level...everything between the two is
contained in a separate level.
"""
Log.Warning("Better be careful here!")
Log.AddDataPoint(fancyMath()[0])
data = getSerialData()
if data:
Log.Info("Got data, let's continue with an advanced test...")
Log.StartAdvanced()
#NOTE: If something breaks in one of the following methods, then GOTO (***)
operateOnData(data)
doSomethingCrazy(data)
Log.ToDo("Fill in some more stuff here later...")
Log.AddDataPoint(data)
Log.EndAdvanced()
#(***) Ideally, we would resume here if an exception is raised in the above localized scope
Log.Info("All done! Log some data and wrap everything up!")
Log.AddDataPoint({"data": "blah"})
#Done
#framework.py
import inspect
from Logger import Log
class Framework(object):
def __init__(self):
print "Framework init!"
self.tests = []
def loadTests(self, file):
"""
Simplifying this for the sake of clarity
"""
for test in file:
self.tests.append(test)
def runTests(self):
"""
Simplifying this for the sake of clarity
"""
#test_alpha() as well as any other user tests will be run here
for test in self.tests:
Log.StartTest()
try:
test()
except Exception,e :
Log.Error(str(e))
Log.EndTest()
#End

You can achieve a similar effect with a context manager using a with statement. Here I use the contextlib.contextmanager decorator:
#contextlib.contextmanager
def swallower():
try:
yield
except ZeroDivisionError:
print("We stopped zero division error")
def foo():
print("This error will be trapped")
with swallower():
print("Here comes error")
1/0
print("This will never be reached")
print("Merrily on our way")
with swallower():
print("This error will propagate")
nonexistentName
print("This won't be reached")
>>> foo()
This error will be trapped
Here comes error
We stopped zero division error
Merrily on our way
This error will propagate
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
foo()
File "<pyshell#3>", line 10, in foo
nonexistentName
NameError: global name 'nonexistentName' is not defined
It cannot be done with an ordinary function call as in your example. In your example, the function startScope returns before the rest of the body of myFunction executes, so startScope can't have any effect on it. To handle exceptions, you need some kind of explicit structure (either a with statement or a regular try/except) inside myFunction; there's no way to make a simple function call magically intercept exceptions that are raised in its caller.
You should read up on context managers as they seem to fit what you're trying to do. The __enter__ and __exit__ methods of the context manager would correspond to your startScope and endScope. Whether it will do exactly what you want depends on exactly what you want those "manager" functions to do, but you will probably have more luck doing it with a context manager than trying to do it with simple function calls.