Say I have an flag --debug/--no-debug defined for the base command. This flag will affect the behavior of many operations in my program. Right now I find myself passing this flag as function parameters all over the place, which doesn't seem elegant. Especially when I need to access this flag in a deep call stack, I'll have to add this parameter to every single function on the stack.
I can instead create a global variable is_debug and set its value at the beginning of the command function that receives the value of this flag. But this doesn't seem elegant to me either.
Is there a better way to make some option values globally accessible using the Click library?
There are two ways to do so, depending on your needs. Both of them end up using the click Context.
Personally, I'm a fan of Option 2 because then I don't have to modify function signatures (and I rarely write multi-threaded programs). It also sounds more like what you're looking for.
Option 1: Pass the Context to the function
Use the click.pass_context decorator to pass the click context to the function.
Docs:
Usage: https://click.palletsprojects.com/en/7.x/commands/#nested-handling-and-contexts
API: https://click.palletsprojects.com/en/7.x/api/#click.pass_context
# test1.py
import click
#click.pass_context
def some_func(ctx, bar):
foo = ctx.params["foo"]
print(f"The value of foo is: {foo}")
#click.command()
#click.option("--foo")
#click.option("--bar")
def main(foo, bar):
some_func(bar)
if __name__ == "__main__":
main()
$ python test1.py --foo 1 --bar "bbb"
The value of foo is: 1
Option 2: click.get_current_context()
Pull the context directly from the current thread via click.get_current_context(). Available starting in Click 5.0.
Docs:
Usage: https://click.palletsprojects.com/en/7.x/advanced/#global-context-access
API: https://click.palletsprojects.com/en/7.x/api/#click.get_current_context
Note: This only works if you're in the current thread (the same thread as what was used to set up the click commands originally).
# test2.py
import click
def some_func(bar):
c = click.get_current_context()
foo = c.params["foo"]
print(f"The value of foo is: {foo}")
#click.command()
#click.option("--foo")
#click.option("--bar")
def main(foo, bar):
some_func(bar)
if __name__ == "__main__":
main()
$ python test2.py --foo 1 --bar "bbb"
The value of foo is: 1
To build on top of the Option 2 given by #dthor, I wanted to make this more seamless, so I combined it with the trick to modify global scope of a function and came up with the below decorator:
def with_click_params(func):
#functools.wraps(func)
def wrapper(*args, **kwargs):
g = func.__globals__
sentinel = object()
ctx = click.get_current_context()
oldvalues = {}
for param in ctx.params:
oldvalues[param] = g.get(param, sentinel)
g[param] = ctx.params[param]
try:
return func(*args, **kwargs)
finally:
for param in ctx.params:
if oldvalues[param] is sentinel:
del g[param]
else:
g[param] = oldvalues[param]
return wrapper
You would use it like this (borrowing sample from #dthor's answer):
#with_click_params
def some_func():
print(f"The value of foo is: {foo}")
print(f"The value of bar is: {bar}")
#click.command()
#click.option("--foo")
#click.option("--bar")
def main(foo, bar):
some_func()
if __name__ == "__main__":
main()
Here is it in action:
$ python test2.py --foo 1 --bar "bbb"
The value of foo is: 1
The value of bar is: bbb
Caveats:
Function can only be called from a click originated call stack, but this is a conscious choice (i.e., you would make assumptions on the variable injection). The click unit testing guide should be useful here.
The function is no longer thread safe.
It is also possible to be explicit on the names of the params to inject:
def with_click_params(*params):
def wrapper(func):
#functools.wraps(func)
def inner_wrapper(*args, **kwargs):
g = func.__globals__
sentinel = object()
ctx = click.get_current_context()
oldvalues = {}
for param in params:
oldvalues[param] = g.get(param, sentinel)
g[param] = ctx.params[param]
try:
return func(*args, **kwargs)
finally:
for param in params:
if oldvalues[param] is sentinel:
del g[param]
else:
g[param] = oldvalue
return inner_wrapper
return wrapper
#with_click_params("foo")
def some_func():
print(f"The value of foo is: {foo}")
#click.command()
#click.option("--foo")
#click.option("--bar")
def main(foo, bar):
some_func()
if __name__ == "__main__":
main()
Related
I'm trying to use returned data from one function into multiple other functions. But I don't want the first function to run each time; which is happening in my case.
#Function lab
def func_a():
print('running function a')
data = 'test'
return data
def func_b():
print(func_a())
def func_c():
print(func_a())
def func_d():
print(func_a())
if __name__ == '__main__':
func_a()
func_b()
func_c()
func_d()
Each time that whole function_a runs. But I just want the returned data from "func_a" in other functions.
IIUC, you could alleviate this with a simple class.
I hold the state of the class which runs func_a in a variable called output. I can then reference this output variable once the class has finished running as much as I like in all other functions without having to re-run func_a.
Hope this helps!
class FunctionA:
def __init__(self):
self.output = None
def run_function(self):
print('running function a')
data = 'test'
self.output = data
def func_b():
print(func_a.output)
def func_c():
print(func_a.output)
def func_d():
print(func_a.output)
if __name__ == '__main__':
func_a = FunctionA()
func_a.run_function()
func_b()
func_c()
func_d()
>>> running function a
>>> test
>>> test
>>> test
Your func_a does two things. To make this clear, let's call it, print_and_return_data.
There are several ways to to break apart the two things print_and_return_data does. One way is to split up the two behaviors into smaller sub-methods:
def print_and_return_data():
print('running function a') # keeping the old print behavior
data = 'test'
return data
into:
def print_run():
print('running function a') # keeping the old print behavior
def return_data():
return 'test'
def print_and_return_data():
print_run()
return return_data()
So that other functions only use what they need:
def func_b():
print(return_data())
Another way is to change print_and_return_data to behave differently the first time it's called from the following times it's called (I don't recommend this because functions changing based on how many times it's been called can be confusing):
context = {'has_printed_before': False}
def print_and_return_data():
if not context['has_printed_before']:
print('running function a')
context['has_printed_before'] = True
data = 'test'
return data
def func_b():
print(print_and_return_data())
if __name__ == '__main__':
func_a() # prints
func_b() # won't print
One way to avoid "functions behaving differently when they're called" is to pass the variation (the "context") in as an argument:
def return_data(also_print=False):
if also_print:
print('running function a')
data = 'test'
return data
def func_b():
print(return_data())
if __name__ == '__main__':
func_a(also_print=True) # prints
func_b() # won't print
I am using the python unittest module for testing a file that takes a command line argument. The argument is a file name which is then passed into a function like so:
file_name = str(sys.argv[1])
file = open(file_name)
result = main_loop(file)
print(result)
My test is set up like so:
class testMainFile(unittest.TestCase):
def test_main_loop(self):
file = open('file_name.json')
result = main_file.main_loop(file)
self.assertEqual(result, 'Expected Result')
if __name__ == 'main':
unittest.main()
When I run the test I get an "IndexError: list index out of range".
I tried passing the argument when running the test but to no avail. How do I run my test without error?
I think you have couple of options here. Firstly go to documentation and checkout patch because i think you can get away with
from unittest.mock import patch
#patch('sys.argv', ['mock.py', 'test-value'])
def test_main_loop(self):
Options for fun:
One would be simply to override the sys.argv next to your call
def test_main_loop(self):
file = open('file_name.json')
+ orginal_argv = sys.argv
+ sys.argv = ['mock argv', 'my-test-value']
result = main_file.main_loop(file)
+ sys.argv = orginal_argv
self.assertEqual(result, 'Expected Result')
Second would be to create a simple wrapper for your function
def set_sys_argv(func: Callable):
sys.argv = ['mock.py', 'my_test_value']
def wrapper(*args, **kwargs):
func()
return wrapper
and use it with test function
#set_sys_argv
def test_main_loop(self):
We can improve it slightly and make it more generic making a decorator that accepts the values to mock
def set_sys_argv(*argv):
sys.argv = argv
def _decorator(func: Callable):
def wrapper(*args, **kwargs):
func()
return wrapper
return _decorator
and use it similarly to patch
#set_sys_argv('mock.py', 'test-value')
def test_main_loop(self):
Third would be to create a context manager, likewise:
class ReplaceSysArgv(list):
def __enter__(self):
self._argv = sys.argv
sys.argv = ['mock', 'my-test-value']
return self
def __exit__(self, *args):
sys.argv = self._argv
and use it with your code
def test_main_loop(self):
file = open('file_name.json')
with ReplaceSysArgv():
result = main_file.main_loop(file)
self.assertEqual(result, 'Expected Result')
you have to push the arguments onto sys.argv before retrieving them (if your code is pulling from command-line arguments - it's unclear to me where in your test you're using the command-line arguments but I digress)
so something like first doing
import sys
sys.argv = ['mock_filename.py', 'json_file.json']
#... continue with rest of program / test.
I want to achieve a pseudo-db-like transaction using context manager.
Take for example:
class Transactor:
def a(): pass
def b(d, b): pass
def c(i): pass
#contextmanager
def get_session(self):
txs = []
yield self # accumulate method calls
for tx in tx:
tx() # somehow pass the arguments
def main():
t = Transactor()
with t.get_session() as session:
session.a() # inserts `a` into `txs`
... more code ...
session.c(value) # inserts `c` and `(value)` into `txs`
session.b(value1, value2) # inserts `b` and `(value1, value2)` into `txs`
... more code ...
# non-transator related code
f = open('file.txt') # If this throws an exception,
# break out of the context manager,
# and discard previous transactor calls.
... more code ...
session.a() # inserts `a` into `txs`
session.b(x, y) # inserts `b` and `(x, y)` into `txs`
# Now is outside of context manager.
# The following calls should execute immediately
t.a()
t.b(x, y)
t.c(k)
If something goes wrong such as an exception, discard txs (rollback). If it makes it to the end of the context, execute each instruction in order of insertion and pass in the appropriate arguments.
How can to trap the method call for later execution?
And one extra caveat:
If get_session is not called, I want to execute the instructions immediately.
It's not pretty, but to follow the structure you're looking for you'd have to build a temporary transaction class that holds your function queues and execute it after the context manager exits. You'll need to use functools.partial, but there are some restrictions though:
All the queued up calls must be methods based on your "session" instance. Anything else gets executed right away.
I don't know how you want to handle non-callable session attributes, so for now I assume it'll just retrieve the value.
Having said that, here's my take on it:
from functools import partial
class TempTrans:
# pass in the object instance to mimic
def __init__(self, obj):
self._queue = []
# iterate through the attributes and methods within the object and its class
for attr, val in type(obj).__dict__.items() ^ obj.__dict__.items():
if not attr.startswith('_'):
if callable(val):
setattr(self, attr, partial(self._add, getattr(obj, attr)))
else:
# placeholder to handle non-callable attributes
setattr(self, attr, val)
# function to add to queue
def _add(self, func, *args, **kwargs):
self._queue.append(partial(func, *args, **kwargs))
# function to execute the queue
def _execute(self):
_remove = []
# iterate through the queue to call the functions.
# I suggest catching errors here in case your functions falls through
for func in self._queue:
try:
func()
_remove.append(func)
except Exception as e:
print('some error occured')
break
# remove the functions that were successfully ran
for func in _remove:
self._queue.remove(func)
Now onto the context manager (it will be outside your class, you can place it in as a class method if you wish):
#contextmanager
def temp_session(obj):
t = TempTrans(obj)
try:
yield t
t._execute()
print('Transactions successfully ran')
except:
print('Encountered errors, queue was not executed')
finally:
print(t._queue) # debug to see what's left of the queue
Usage:
f = Foo()
with temp_session(f) as session:
session.a('hello')
session.b(1, 2, 3)
# a hello
# b 1 2 3
# Transactions successfully ran
# []
with temp_session(f) as session:
session.a('hello')
session.b(1, 2, 3)
session.attrdoesnotexist # expect an error
# Encountered errors, queue was not executed
# [
# functools.partial(<bound method Foo.a of <__main__.Foo object at 0x0417D3B0>>, 'hello'),
# functools.partial(<bound method Foo.b of <__main__.Foo object at 0x0417D3B0>>, 1, 2, 3)
# ]
This solution was a bit contrived because of the way you wanted it structured, but if you didn't need a context manager and doesn't need the session to look like a direct function call, it's trivial to just use partial:
my_queue = []
# some session
my_queue.append(partial(f, a))
my_queue.append(partial(f, b))
for func in my_queue:
func()
So I'm playing with the sys.settrace function. In Python 3.7, a new opcode event is added
'opcode'
The interpreter is about to execute a new opcode (see dis for
opcode details). The local trace function is called; arg is None; the
return value specifies the new local trace function. Per-opcode events
are not emitted by default: they must be explicitly requested by
setting f_trace_opcodes to True on the frame.
I was able to get those opcode events, however there seems to be no further information——I don't even know what the opcode is and what it's doing.
Does this event only brings finer granularity, without giving any extra information?
If you want further information, you will need to examine the frame argument that is passed into the tracing function. In particular, you want to look at frame.f_lasti to find the last instruction executed and frame.f_code.co_code to access that instruction. Using those two together will give you the actual opcode. If you want the mnemonic, then you will want to use dis.opname; but if you simply what to match it with another opcode, then you can use dis.opmap instead. The following example is contrived, but it demonstrates some of what is possible using the hints just provided:
#! /usr/bin/env python3
import dis
import sys
def main():
dis.dis(add)
sys.settrace(get_trace(False, get_callback(celebrate)))
total = add(1, 2)
print(f'total = {total}')
sys.settrace(None)
total = add(3, 4)
print(f'total = {total}')
print('Done')
def get_trace(trace_lines=True, opcode_callback=None):
trace_opcodes = callable(opcode_callback)
# noinspection PyUnusedLocal
def trace(frame, event, arg):
frame.f_trace_lines = trace_lines
frame.f_trace_opcodes = trace_opcodes
if trace_opcodes and event == 'opcode':
opcode = frame.f_code.co_code[frame.f_lasti]
opname = dis.opname[opcode]
opcode_callback(frame, opcode, opname)
return trace
return trace
def get_callback(return_handler=None):
handle_return = callable(return_handler)
def echo_opcode(frame, opcode, opname):
print(f'# {opname} ({opcode}) #')
if handle_return and opcode == dis.opmap['RETURN_VALUE']:
return_handler(frame)
return echo_opcode
# noinspection PyUnusedLocal
def celebrate(frame):
print('/-------------------\\')
print('| We are returning! |')
print('\\-------------------/')
def add(a, b):
return a + b
if __name__ == '__main__':
main()
I use Boto to access Amazon S3. And for file uploading I can assign a callback function. The problem is that I cannot access the needed variables from that callback function until I make them global. In another hand, if I make them global, they are global for all other Celery tasks, too (until I restart Celery), as the file uploading is executed from a Celery task.
Here is a function that uploads a JSON file with information about video conversion progress.
def upload_json():
global current_frame
global path_to_progress_file
global bucket
json_file = Key(bucket)
json_file.key = path_to_progress_file
json_file.set_contents_from_string('{"progress": "%s"}' % current_frame,
cb=json_upload_callback, num_cb=2, policy="public-read")
And here are 2 callback functions for uploading frames generated by ffmpeg during the video conversion and a JSON file with the progress information.
# Callback functions that are called by get_contents_to_filename.
# The first argument is representing the number of bytes that have
# been successfully transmitted from S3 and the second is representing
# the total number of bytes that need to be transmitted.
def frame_upload_callback(transmitted, to_transmit):
if transmitted == to_transmit:
upload_json()
def json_upload_callback(transmitted, to_transmit):
global uploading_frame
if transmitted == to_transmit:
print "Frame uploading finished"
uploading_frame = False
Theoretically, I could pass the uploading_frame variable to the upload_json function, but it wouldn’t get to json_upload_callback as it’s executed by Boto.
In fact, I could write something like this.
In [1]: def make_function(message):
...: def function():
...: print message
...: return function
...:
In [2]: hello_function = make_function("hello")
In [3]: hello_function
Out[3]: <function function at 0x19f4c08>
In [4]: hello_function()
hello
Which, however, doesn’t let you edit the value from the function, just lets you read the value.
def myfunc():
stuff = 17
def lfun(arg):
print "got arg", arg, "and stuff is", stuff
return lfun
my_function = myfunc()
my_function("hello")
This works.
def myfunc():
stuff = 17
def lfun(arg):
print "got arg", arg, "and stuff is", stuff
stuff += 1
return lfun
my_function = myfunc()
my_function("hello")
And this gives an UnboundLocalError: local variable 'stuff' referenced before assignment.
Thanks.
In Python 2.x closed over variables are read-only (not for the Python VM, but just because of the syntax that doesn't allow writing to a non local and non global variable).
You can however use a closure over a mutable value... i.e.
def myfunc():
stuff = [17] # <<---- this is a mutable object
def lfun(arg):
print "got arg", arg, "and stuff[0] is", stuff[0]
stuff[0] += 1
return lfun
my_function = myfunc()
my_function("hello")
my_function("hello")
If you are instead using Python 3.x the keyword nonlocal can be used to specify that a variable used in read/write in a closure is not a local but should be captured from the enclosing scope:
def myfunc():
stuff = 17
def lfun(arg):
nonlocal stuff
print "got arg", arg, "and stuff is", stuff
stuff += 1
return lfun
my_function = myfunc()
my_function("hello")
my_function("hello")
You could create a partial function via functools.partial. This is a way to call a function with some variables pre-baked into the call. However, to make that work you'd need to pass a mutable value - eg a list or dict - into the function, rather than just a bool.
from functools import partial
def callback(arg1, arg2, arg3):
arg1[:] = [False]
print arg1, arg2, arg3
local_var = [True]
partial_func = partial(callback, local_var)
partial_func(2, 1)
print local_var # prints [False]
A simple way to do these things is to use a local function
def myfunc():
stuff = 17
def lfun(arg):
print "got arg", arg, "and stuff is", stuff
stuff += 1
def register_callback(lfun)
This will create a new function every time you call myfunc, and it will be able to use the local "stuff" copy.