I need to create an algorithm which is executed as a sequence of functions chosen at run-time, a bit like the strategy pattern. I want to avoid creating many single method classes (100+), but use simple functions, and assemble them using a decorator.
actions = []
def act(specific_fn):
actions.append(specific_fn)
return specific_fn
#act
def specific_fn1():
print("specific_fn1")
#act
def specific_fn2():
print("specific_fn2")
def execute_strategy():
[f() for f in actions]
I have a couple of questions:
How can I modify the decorator function act to take the list actions as a parameter, so that it adds the decorated function into the list?
How do I use specific_fnX defined in another file? Currently, I just import the file inside the calling function - but that seems odd. Any other options?
Also, any other ideas on implementing this pattern?
This is a pretty good tutorial on using decorators, both with and without arguments. Others may know of others.
The trick is to remember that a decorator is a function call, but when applied to a function definition, it magically replaces the function with the returned result. If you want #my_decorator(my_list) to be a decorator, then my_decorator(my_list) must either return a function or an object with a __call__ method, and that function is then called on called on specific_fn1.
So yes, you need a function that returns a function that returns a function. Looking at some examples in a tutorial will make this clearer.
As to your second question, you can just call my_decorator(my_list)(specific_fn1) without using a decorator, and then ignorning the result.
Related
Problem
I have a function make_pipeline that accepts an arbitrary number of functions, which it then calls to perform sequential data transformation. The resulting call chain performs transformations on a pandas.DataFrame. Some, but not all functions that it may call need to operate on a sub-array of the DataFrame. I have written multiple selector functions. However at present each member-function of the chain has to be explicitly be given the user-selected selector/filter function. This is VERY error-prone and accessibility is very important as the end-code is addressed to non-specialists (possibly with no Python/programming knowledge), so it must be "batteries-included". This entire project is written in a functional style (that's what's always worked for me).
Sample Code
filter_func = simple_filter()
# The API looks like this
make_pipeline(
load_data("somepath", header = [1,0]),
transform1(arg1,arg2),
transform2(arg1,arg2, data_filter = filter_func),# This function needs access to user-defined filter function
transform3(arg1,arg2,, data_filter = filter_func),# This function needs access to user-defined filter function
transform4(arg1,arg2),
)
Expected API
filter_func = simple_filter()
# The API looks like this
make_pipeline(
load_data("somepath", header = [1,0]),
transform1(arg1,arg2),
transform2(arg1,arg2),
transform3(arg1,arg2),
transform4(arg1,arg2),
)
Attempted
I thought that if the data_filter alias is available in the caller's namespace, it also becomes available (something similar to a closure) to all functions it calls. This seems to happen with some toy examples but wont work in the case (UnboundError).
What's a good way to make a function defined in one place available to certain interested functions in the call chain? I'm trying to avoid global.
Notes/Clarification
I've had problems with OOP and mutable states in the past, and functional programming has worked quite well. Hence I've set a goal for myself to NOT use classes (to the extent that Python enables me to anyways). So no classes.
I should have probably clarified this initially: In the pipeline the output of all functions is a DataFrame and the input of all functions (except load data obviously) is a DataFrame. The functions are decorated with a wrapper that calls functools.partial because we want the user to supply the args to each function but not execute it. The actual execution is done be a forloop in make_pipeline.
Each function accepts df:pandas.DataFrame plus all arguements that are specific to that function. The statement seen above transform1(arg1,arg2,...) actually calls the decorated transform1 witch returns functools.partial(transform, arg1,arg2,...) which is now has a signature like transform(df:pandas.DataFrame).
load_dataframe is just a convenience function to load the initial dataframe so that all other functions can begin operating on it. It just felt more intuitive to users to have it part of the chain rather that a separate call
The problem is this: I need a way for a filter function to be initialized (called) in only on place, such that every function in the call chain that needs access to the filter function, gets it without it being explicitly passed as argument to said function. If you're wondering why this is the case, it's because I feel that end users will find it unintuitive and arbitrary. Some functions need it, some don't. I'm also pretty certain that they will make all kinds of errors like passing different filters, forgetting it sometimes etc.
(Update) I've also tried inspect.signature() in make_pipeline to check if each function accepts a data_filter argument and pass it on. However, this raises an incorrect function signature error so some unclear reason (likely because of the decorators/partial calls). If signature could the return the non-partial function signature, this would solve the issue, but I couldn't find much info in the docs
Turns out it was pretty easy. The solution is inspect.signature.
def make_pipeline(*args, data_filter:Optional[Callable[...,Any]] = None)
d = args[0]
for arg in args[1:]:
if "data_filter" in inspect.signature(arg):
d = arg(d, data_filter = data_filter)
else:
d= arg(d)
Leaving this here mostly for reference because I think this is a mini design pattern. I've also seen an function._closure_ on unrelated subject. That may also work, but will likely be more complicated.
I've only recently learned about decorators, and despite reading nearly every search result I can find about this question, I cannot figure this out. All I want to do is define some function "calc(x,y)", and wrap its result with a series of external functions, without changing anything inside of my function, nor its calls in the script, such as:
#tan
#sqrt
def calc(x,y):
return (x+y)
### calc(x,y) = tan(sqrt(calc(x,y))
### Goal is to have every call of calc in the script automatically nest like that.
After reading about decorators for almost 10 hours yesterday, I got the strong impression this is what they were used for. I do understand that there are various ways to modify how the functions are passed to one another, but I can't find any obvious guide on how to achieve this. I read that maybe functools wraps can be used for this purpose, but I cannot figure that out either.
Most of the desire here is to be able to quickly and easily test how different functions modify the results of others, without having to tediously wrap functions between parenthesis... That is, to avoid having to mess with parenthesis at all, having my modifier test functions defined on their own lines.
A decorator is simply a function that takes a function and returns another function.
def tan(f):
import math
def g(x,y):
return math.tan(f(x,y))
return g
If you are incrementally designing a function that could have variable number of outputs, what is the best way to design that function? E.g.
def function(input):
return output1, output2, ...
or
def function(input):
return dict(output1=...)
In both cases, you need a bunch of if statements to sort through and utilize the outputs; the difference is where the if statements are used (within function or outside of the function over the dictionary). I am not sure what principle to use to decide on what to do.
If you need to return multiple things, it means the function is either complex and you should break it down, or you need an object with attributes that is "processed" in the function. A dict is a standard object, but you can also create your own, depends if you want to go more the OOP way or the functional/procedural way
I've been learning and experimenting with decorators. I understand what they do: they allow you to write modular code by allowing you to add functionality to existing functions without changing them.
I found a great thread which really helped me learn how to do it by explaining all the ins and outs here: How to make a chain of function decorators?
But what is missing from this thread (and from other resources I've looked at) is WHY do we need the decorator syntax? Why do I need nested functions to create a decorator? Why can't I just take an existing function, write another one with some extra functionality, and feed the first into the 2nd to make it do something else?
Is it just because this is the convention? What am I missing? I assume my inexperience is showing here.
I will try to keep it simple and explain it with examples.
One of the things that I often do is measure the time taken by API's that I build and then publish them on AWS.
Since it is a very common use case I have created a decorator for it.
def log_latency():
def actual_decorator(f):
#wraps(f)
def wrapped_f(*args, **kwargs):
t0 = time()
r = f(*args, **kwargs)
t1 = time()
async_daemon_execute(public_metric, t1 - t0, log_key)
return r
return wrapped_f
return actual_decorator
Now if there is any method that I want to measure the latency, I just put annotate it with the required decorator.
#log_latency()
def batch_job_api(param):
pass
Suppose you want to write a secure API which only works if you send a header with a particular value then you can use a decorator for it.
def secure(f):
#wraps(f)
def wrapper(*args, **kw):
try:
token = request.headers.get("My_Secret_Token")
if not token or token != "My_Secret_Text":
raise AccessDenied("Required headers missing")
return f(*args, **kw)
return wrapper
Now just write
#secure
def my_secure_api():
pass
I have also been using the above syntax for API specific exceptions, and if a method needs a database interaction instead of acquiring a connection I use #session decorator which tells that this method will use a database connection and you don't need to handle one yourself.
I could have obviously avoided it, by writing a function that checks for the header or prints time taken by API on AWS, but that just looks a bit ugly and nonintuitive.
There is no convention for this, at least I am not aware of them but it definitely makes the code more readable and easier to manage.
Most of the IDE's also have different color syntax for annotation which makes it easier to understand and organize code.
So, I would it may just be because of inexperience; if you start using them you will automatically start knowing where to use them.
From PEP 318 -- Decorators for Functions and Methods (with my own added emphasis):
Motivation
The current method of applying a transformation to a
function or method places the actual transformation after the function
body. For large functions this separates a key component of the
function's behavior from the definition of the rest of the function's
external interface. For example:
def foo(self):
perform method operation
foo = classmethod(foo)
This becomes less readable with longer methods. It also seems less than pythonic to name
the function three times for what is conceptually a single
declaration. A solution to this problem is to move the transformation
of the method closer to the method's own declaration. The intent of
the new syntax is to replace
def foo(cls):
pass
foo = synchronized(lock)(foo)
foo = classmethod(foo)
with an alternative that places the decoration in the function's declaration:
#classmethod
#synchronized(lock)
def foo(cls):
pass
I came across a neatly-written article which explains decorators in Python, why we should leverage them, how to use them and more.
Some of the benefits of using decorators in Python include:
Allows us to extend functions without actually modifying them.
Helps us stick with the Open-Closed Principle.
It helps reduce code duplication -- we can factorize out repeated code into a decorator.
Decorators help keep a project organized and easy to maintain.
Reference: https://tomisin.dev/blog/understanding-decorators-in-python
A great example of why decorators are useful is the numba package.
It provides a decorator to speed up Python functions:
#jit
def my_slow_function():
# ...
The #jit decorator does some very amazing and complex operations - it compiles the whole function (well, parts of it) to machine code. If Python did not have decorators you would have to write the function in machine code yourself... or more seriously, the syntax would look like this:
def my_slow_function():
# ...
my_slow_function = jit(my_slow_function)
The purpose of the decorator syntax is to make the second example nicer. It's just syntactic sugar so you don't have to type the function name three times.
Let me start by saying what I would like to do. I want to create a lazy wrapper for a variable, as in I record all the method calls and operator calls and evaluate them later when I specify the variable to call it on.
As such, I want to be able to intercept all the method calls and operator calls and special methods so that I can work on them. However, __getattr__ doesn't intercept operator calls or __str__ and such, so I want to know if there is a generic way to overload all method calls, or should I just dynamically create a class and duplicate the code for all of it (which I already did, but is ugly).
It can be done, but yes, it becomes "ugly" - I wrote a lazy decorator once, that turns any function into a "lazily computed function".
Basically, I found out that the only moment an object's value is actually used in Python is when one of the special "dunder" methods s called. For example, when you have a number, it's value is only used when you are either using it in another operation, or converting it to a string for IO (which also uses a "dunder" method)
So, my wrapper anotates the parameters to a function call, and returns an special object,
which has potentially all of the "dunder" methods. Just when one of those methods is called, the original function is called - and its return value is then cached for further usage.
The implementation is here:
https://bitbucket.org/jsbueno/metapython/src/510a7d125b24/lazy_decorator.py
Sorry for the text and most of the presentation being in Portuguese.