I have several string processing functions like:
def func1(s):
return re.sub(r'\s', "", s)
def func2(s):
return f"[{s}]"
...
I want to combine them into one pipeline function: my_pipeline(), so that I can use it as an argument, for example:
class Record:
def __init__(self, s):
self.name = s
def apply_func(self, func):
return func(self.name)
rec = Record(" hell o")
output = rec.apply_func(my_pipeline)
# output = "[hello]"
The goal is to use my_pipeline as an argument, otherwise I need to call these functions one by one.
Thank you.
You can write a simple factory function or class to build a pipeline function:
>>> def pipeline(*functions):
... def _pipeline(arg):
... result = arg
... for func in functions:
... result = func(result)
... return result
... return _pipeline
...
>>> rec = Record(" hell o")
>>> rec.apply_func(pipeline(func1, func2))
'[hello]'
This is a more refined version written with reference to this using functools.reduce:
>>> from functools import reduce
>>> def pipeline(*functions):
... return lambda initial: reduce(lambda arg, func: func(arg), functions, initial)
I didn't test it, but according to my intuition, each loop will call the function one more time at the python level, so the performance may not be as good as the loop implementation.
You can just create a function which calls these functions:
def my_pipeline(s):
return func1(func2(s))
Using a list of functions (so you can assemble these elsewhere):
def func1(s):
return re.sub(r'\s', "", s)
def func2(s):
return f"[{s}]"
def func3(s):
return s + 'tada '
def callfuncs(s, pipeline):
f0 = s
pipeline.reverse()
for f in pipeline:
f0 = f(f0)
return f0
class Record:
def __init__(self, s):
self.name = s
def apply_func(self, pipeline):
return callfuncs(s.name, pipeline)
# calling order func1(func2(func3(s)))
my_pipeline = [func1, func2, func3]
rec = Record(" hell o")
output = rec.apply_func(my_pipeline)
Related
I have the following simplified code:
class States:
def __init__(self):
pass
def state1(self):
a = 2*10
return a
def state2(self):
a = 50/10
return a
class Results:
def __init__(self):
pass
def result(self):
states = States()
x = []
for i in [state1,state2]:
state_result = states.i()
x.append(state_result)
return x
I want to loop through every function in the class "States". Of course
for i in [state1,state2]
will return "name 'state1' is not defined", but I hope it gives an idea what I try to achieve.
You can use dir() to get the name of the functions of a class. You can then use getattr() to call the function.
class States:
def __init__(self):
pass
def state1(self):
a = 2*10
return a
def state2(self):
a = 50/10
return a
state = States()
for func in dir(States):
if func.startswith('__'):
continue
print(func)
print(getattr(state, func)())
Will output
state1
20
state2
5.0
You can do this tho:
def result(self):
states = States()
x = []
for i in [states.state1,states.state2]: # changed state1 to states.state1 and so on
state_result = i()
x.append(state_result)
return x
I think you can use lambda. Here, i made a simple example for you.
def foo(text):
print(text)
a = [lambda: foo("hey"), lambda: foo("boo")]
for i in a:
i()
Result:
hey
boo
In your case, you should come over with this:
for i in [lambda: state1(), lambda:state2()]:
state_result = i()
x.append(state_result)
But if you ask my opinion, it's important to inform you that calling functions through a list is not a healthy way. A software languge usually has a solution for many cases; but in this case, i think your point of view is wrong. Doing work by messing with built-in techniques and trying to find some secret tricks is is not a suggested thing.
The clean way to do this is to "register" your state methods. SOmething like this:
class States():
states = []
def register_state(cache):
def inner(fn):
cache.append(fn)
return inner
#register_state(states)
def state1(self):
a = 2*10
return a
#register_state(states)
def state2(self):
a = 50/10
return a
Then your Results class can do
class Results:
def __init__(self):
pass
def result(self):
states = States()
x = []
for i in states.states:
state_result = i(states)
x.append(state_result)
return x
You can get the members of class States via the class' dict as:
States.__dict__
Which'll give you all the attributes and function of your class as:
{'__module__': '__main__', '__init__': <function States.__init__ at 0x00000183066F0A60>, 'state1': <function States.state1 at 0x00000183066F0AF0>, 'state2': <function States.state2 at 0x000001830 ...
You can filter this into a list comprehension dict to not include dunders as:
[funcname for funcname in States.__dict__ if not (str.startswith('__') and str.endswith('__'))]
This will return you a list of member functions as:
['state1', 'state2']
Then create an object of States as:
states = States()
get the whole calculation done as:
for funcname in [funcname for funcname in States.__dict__ if not (funcname.startswith('__') and funcname.endswith('__'))]:
x.append(States.__dict__[funcname](states))
Better yet, make it a comprehension as:
[States.__dict__[funcname](states) for funcname in States.__dict__ if not (funcname.startswith('__') and funcname.endswith('__'))]
Your answer after applying this approach is: [20, 5.0]
or get the dict of functionName and returnValues as a comprehension:
{funcname: States.__dict__[funcname](states) for funcname in States.__dict__ if not (funcname.startswith('__') and funcname.endswith('__'))}
Which'll give you an output as:
{'state1': 20, 'state2': 5.0}
I am trying to use functional programming to create a dictionary containing a key and a function to execute:
myDict={}
myItems=("P1","P2","P3",...."Pn")
def myMain(key):
def ExecP1():
pass
def ExecP2():
pass
def ExecP3():
pass
...
def ExecPn():
pass
Now, I have seen a code used to find the defined functions in a module, and I need to do something like this:
for myitem in myItems:
myDict[myitem] = ??? #to dynamically find the corresponding function
So my question is, How do I make a list of all the Exec functions and then assign them to the desired item using the a dictionary? so at the end I will have myDict["P1"]() #this will call ExecP1()
My real problem is that I have tons of those items and I making a library that will handle them so the final user only needs to call myMain("P1")
I think using the inspect module, but I am not so sure how to do it.
My reason to avoid:
def ExecPn():
pass
myDict["Pn"]=ExecPn
is that I have to protect code as I am using it to provide a scripting feature within my application.
Simplify, simplify, simplify:
def p1(args):
whatever
def p2(more args):
whatever
myDict = {
"P1": p1,
"P2": p2,
...
"Pn": pn
}
def myMain(name):
myDict[name]()
That's all you need.
You might consider the use of dict.get with a callable default if name refers to an invalid function—
def myMain(name):
myDict.get(name, lambda: 'Invalid')()
(Picked this neat trick up from Martijn Pieters)
Simplify, simplify, simplify + DRY:
tasks = {}
task = lambda f: tasks.setdefault(f.__name__, f)
#task
def p1():
whatever
#task
def p2():
whatever
def my_main(key):
tasks[key]()
Not proud of it, but:
def myMain(key):
def ExecP1():
pass
def ExecP2():
pass
def ExecP3():
pass
def ExecPn():
pass
locals()['Exec' + key]()
I do however recommend that you put those in a module/class whatever, this is truly horrible.
If you are willing to add a decorator for each function, you can define a decorator which adds each function to a dictionary:
def myMain(key):
tasks = {}
def task(task_fn):
tasks[task_fn.__name__] = task_fn
#task
def ExecP1():
print(1)
#task
def ExecP2():
print(2)
#task
def ExecP3():
print(3)
#task
def ExecPn():
print(4)
tasks['Exec' + key]()
Another option is to place all the functions under a class (or in a different module) and use getattr:
def myMain(key):
class Tasks:
def ExecP1():
print(1)
def ExecP2():
print(2)
def ExecP3():
print(3)
def ExecPn():
print(4)
task = getattr(Tasks, 'Exec' + key)
task()
# index dictionary by list of key names
def fn1():
print "One"
def fn2():
print "Two"
def fn3():
print "Three"
fndict = {"A": fn1, "B": fn2, "C": fn3}
keynames = ["A", "B", "C"]
fndict[keynames[1]]()
# keynames[1] = "B", so output of this code is
# Two
You can just use
myDict = {
"P1": (lambda x: function1()),
"P2": (lambda x: function2()),
...,
"Pn": (lambda x: functionn())}
myItems = ["P1", "P2", ..., "Pn"]
for item in myItems:
myDict[item]()
This will call methods from dictionary
This is python switch statement with function calling
Create few modules as per the your requirement.
If want to pass arguments then pass.
Create a dictionary, which will call these modules as per requirement.
def function_1(arg):
print("In function_1")
def function_2(arg):
print("In function_2")
def function_3(fileName):
print("In function_3")
f_title,f_course1,f_course2 = fileName.split('_')
return(f_title,f_course1,f_course2)
def createDictionary():
dict = {
1 : function_1,
2 : function_2,
3 : function_3,
}
return dict
dictionary = createDictionary()
dictionary[3](Argument)#pass any key value to call the method
#!/usr/bin/python
def thing_a(arg=None):
print 'thing_a', arg
def thing_b(arg=None):
print 'thing_b', arg
ghetto_switch_statement = {
'do_thing_a': thing_a,
'do_thing_b': thing_b
}
ghetto_switch_statement['do_thing_a']("It's lovely being an A")
ghetto_switch_statement['do_thing_b']("Being a B isn't too shabby either")
print "Available methods are: ", ghetto_switch_statement.keys()
Often classes are used to enclose methods and following is the extension for answers above with default method in case the method is not found.
class P:
def p1(self):
print('Start')
def p2(self):
print('Help')
def ps(self):
print('Settings')
def d(self):
print('Default function')
myDict = {
"start": p1,
"help": p2,
"settings": ps
}
def call_it(self):
name = 'start'
f = lambda self, x : self.myDict.get(x, lambda x : self.d())(self)
f(self, name)
p = P()
p.call_it()
class CallByName():
def method1(self):
pass
def method2(self):
pass
def method3(self):
pass
def get_method(self, method_name):
method = getattr(self, method_name)
return method()
callbyname = CallByName()
method1 = callbyname.get_method(method_name)
```
def p1( ):
print("in p1")
def p2():
print("in p2")
myDict={
"P1": p1,
"P2": p2
}
name=input("enter P1 or P2")
myDictname
You are wasting your time:
You are about to write a lot of useless code and introduce new bugs.
To execute the function, your user will need to know the P1 name anyway.
Etc., etc., etc.
Just put all your functions in the .py file:
# my_module.py
def f1():
pass
def f2():
pass
def f3():
pass
And use them like this:
import my_module
my_module.f1()
my_module.f2()
my_module.f3()
or:
from my_module import f1
from my_module import f2
from my_module import f3
f1()
f2()
f3()
This should be enough for starters.
What is a good design pattern to implement templated object generation (not sure that's the name) in python?
By that, I mean having a function such as:
from typing import TypeVar
T = TypeVar('T')
def mk_templated_obj_factory(template: T) -> Callable[..., T]:
"""Returns a f(**kwargs) function that returns an object of type T created by a template of the same type."""
Python has templated strings. Something like `"this {is} a {template}".format' would be how one could achieve the above. If we want to get a "proper" function that has a signature (useful for a user so they know what arguments they need to provide!), we could do this:
from inspect import signature, Signature, Parameter
from operator import itemgetter
from typing import Callable
f = "hello {name} how are you {verb}?".format
def templated_string_func(template: str) -> Callable:
"""A function making templated strings. Like template.format, but with a signature"""
f = partial(str.format, template)
names = filter(None, map(itemgetter(1), string.Formatter().parse(template)))
params = [Parameter(name=name, kind=Parameter.KEYWORD_ONLY) for name in names]
f.__signature__ = Signature(params)
return f
f = templated_string_func("hello {name} how are you {verb}?")
assert f(name='Christian', verb='doing') == 'hello Christian how are you doing?'
assert str(signature(f)) == '(*, name, verb)'
But would if we want to make dict factories? Something having this behavior:
g = templated_dict_func(template={'hello': '$name', 'how are you': ['$verb', 2]})
assert g(name='Christian', verb='doing') == {'hello': '$name', 'how are you': ['doing', 2]}
What about other types of objects?
It seems like something that would have a solid design pattern...
I would recommend using decorators to register your template function generating functions in a dictionary that maps from types to the functions that handle them. The dictionary is needed in order to be able to template objects of any type in an extensible way, without writing all the templating logic in a single big function, but instead adding handling logic for new types as needed.
The core code is in the Templater class, just grouped here for organisation:
class Templater:
templater_registry: dict[type, Callable[[Any], TemplateFunc]] = {}
#classmethod
def register(cls, handles_type: type):
def decorator(f):
cls.templater_registry[handles_type] = f
return f
return decorator
...
Where TemplateFunc is defined as Generator[str, None, Callable[..., T]], a generator that yields strs and returns a function that returns some type T. This is chosen so that the template handlers can yield the names of their keyword arguments and then return their template function. The Templater.template_func method uses a something of type TemplateFunc to generate a function with the correct signature.
The register decorator presented above is written such that:
#Templater.register(dict)
def templated_dict_func(template: dict[K, V]):
pass
is equivalent to:
def templated_dict_func(template: dict[K, V]):
pass
Templater.templater_registry[dict] = templated_dict_func
The code for templating any type is fairly self-explainatory:
class Templater:
...
#classmethod
def template_func_generator(cls, template: T) -> TemplateFunc[T]:
# if it is a type that can be a template
if type(template) in cls.templater_registry:
# then return the template handler
template_factory = cls.templater_registry[type(template)]
return template_factory(template)
else:
# else: an empty generator that returns a function that returns the template unchanged,
# since we don't know how to handle it
def just_return():
return lambda: template
yield # this yield is needed to tell python that this is a generator
return just_return()
The code for templating strings is fairly unchanged, except that the argument names are yielded instead of put in the function signature:
#Templater.register(str)
def templated_string_func(template: str) -> TemplateFunc[str]:
"""A function making templated strings. Like template.format, but with a signature"""
f = partial(str.format, template)
yield from filter(None, map(itemgetter(1), string.Formatter().parse(template)))
return f
The list template function could look like this:
#Templater.register(list)
def templated_list_func(template: list[T]) -> TemplateFunc[list[T]]:
entries = []
for item in template:
item_template_func = yield from Templater.template_func_generator(item)
entries.append(item_template_func)
def template_func(**kwargs):
return [
item_template_func(**kwargs)
for item_template_func in entries
]
return template_func
Although, if you cannot guarantee that every template function can handle extra arguments, you need to track which arguments belong to which elements and only pass the necessary ones. I use the get_generator_return utility function (defined later on) to capture both the yielded values and the return value of the recursive calls.
#Templater.register(list)
def templated_list_func(template: list[T]) -> TemplateFunc[list[T]]:
entries = []
for item in template:
params, item_template_func = get_generator_return(Templater.template_func_generator(item))
params = tuple(params)
yield from params
entries.append((item_template_func, params))
def template_func(**kwargs):
return [
item_template_func(**{arg: kwargs[arg] for arg in args})
for item_template_func, args in entries
]
return template_func
The dict handler is implemented similarly. This system could be extended to support all kinds of different objects, including arbitrary dataclasses and more, but I leave that as an exercise for the reader!
Here is the entire working example:
import string
from functools import partial
from inspect import Signature, Parameter
from operator import itemgetter
from typing import Callable, Any, TypeVar, Generator, Tuple, Dict, List
from collections import namedtuple
T = TypeVar('T')
U = TypeVar('U')
def get_generator_return(gen: Generator[T, Any, U]) -> Tuple[Generator[T, Any, U], U]:
return_value = None
def inner():
nonlocal return_value
return_value = yield from gen
gen_items = list(inner())
def new_gen():
yield from gen_items
return return_value
return new_gen(), return_value
# TemplateFunc: TypeAlias = Generator[str, None, Callable[..., T]]
TemplateFunc = Generator[str, None, Callable[..., T]]
class Templater:
templater_registry: Dict[type, Callable[[Any], TemplateFunc]] = {}
#classmethod
def register(cls, handles_type: type):
def decorator(f):
cls.templater_registry[handles_type] = f
return f
return decorator
#classmethod
def template_func_generator(cls, template: T) -> TemplateFunc[T]:
if type(template) in cls.templater_registry:
template_factory = cls.templater_registry[type(template)]
return template_factory(template)
else:
# an empty generator that returns a function that returns the template unchanged,
# since we don't know how to handle it
def just_return():
return lambda: template
yield # this yield is needed to tell python that this is a generator
return just_return()
#classmethod
def template_func(cls, template: T) -> Callable[..., T]:
gen = cls.template_func_generator(template)
params, f = get_generator_return(gen)
f.__signature__ = Signature(Parameter(name=param, kind=Parameter.KEYWORD_ONLY) for param in params)
return f
#Templater.register(str)
def templated_string_func(template: str) -> TemplateFunc[str]:
"""A function making templated strings. Like template.format, but with a signature"""
f = partial(str.format, template)
yield from filter(None, map(itemgetter(1), string.Formatter().parse(template)))
return f
K = TypeVar('K')
V = TypeVar('V')
#Templater.register(dict)
def templated_dict_func(template: Dict[K, V]) -> TemplateFunc[Dict[K, V]]:
DictEntryInfo = namedtuple('DictEntryInfo', ['key_func', 'value_func', 'key_args', 'value_args'])
entries: list[DictEntryInfo] = []
for key, value in template.items():
key_params, key_template_func = get_generator_return(Templater.template_func_generator(key))
value_params, value_template_func = get_generator_return(Templater.template_func_generator(value))
key_params = tuple(key_params)
value_params = tuple(value_params)
yield from key_params
yield from value_params
entries.append(DictEntryInfo(key_template_func, value_template_func, key_params, value_params))
def template_func(**kwargs):
return {
entry_info.key_func(**{arg: kwargs[arg] for arg in entry_info.key_args}):
entry_info.value_func(**{arg: kwargs[arg] for arg in entry_info.value_args})
for entry_info in entries
}
return template_func
#Templater.register(list)
def templated_list_func(template: List[T]) -> TemplateFunc[List[T]]:
entries = []
for item in template:
params, item_template_func = get_generator_return(Templater.template_func_generator(item))
params = tuple(params)
yield from params
entries.append((item_template_func, params))
def template_func(**kwargs):
return [
item_template_func(**{arg: kwargs[arg] for arg in args})
for item_template_func, args in entries
]
return template_func
g = Templater.template_func(template={'hello': '{name}', 'how are you': ['{verb}', 2]})
assert g(name='Christian', verb='doing') == {'hello': 'Christian', 'how are you': ['doing', 2]}
print(g.__signature__)
I already have a class poly(), and a method get_function()
class poly():
def __init__(self,n):
func = ''
var = []
for i in range(n + 1):
func += ('k'+str(i)) + ' * '+ 'x ** ' + str(i) + ' + '
var.append('k'+str(i))
func = func[:-3]
self.func_str = func
self.var = var
siglist = var.copy()
siglist.append('x')
self.siglist = tuple(siglist)
def get_function(self, *args):
return eval(self.func_str)
Now what I want to do is to pass self.siglist to signature of get_function for future usage(scipy.optimize.curve_fit needs __signature__ to do curve fitting)
I ran
pol = poly(2)
inspect.signature(pol.get_function)
It shows that the signature of that function is <Signature (*args)>
But I want to change signature from *args to k0, k1, x(Stored in tuple siglist)
What I found in Python Cookbook is:
from functools import wraps
import inspect
def optional_debug(func):
#wraps(func)
def wrapper(*args, debug=False, **kwargs):
return func(*args, **kwargs)
sig = inspect.signature(func)
parms = list(sig.parameters.values())
# what is inspect.Parameter.KEYWORD_ONLY do ?
parms.append(inspect.Parameter('debug', inspect.Parameter.KEYWORD_ONLY, default=False))
wrapper.__signature__ = sig.replace(parameters=parms)
return wrapper
#optional_debug
def test(input):
pass
print(inspect.signature(test))
This function is able to change the signature of a function, the result is:
(input, *, debug=False)
How to pass self.siglist to edit the signature if I put the decorator outside the class, and why is there a * in the signature after using a decorator to change it?
________________To edit the __signature__ if a function is not in a class___________
def make_sig(*names):
parms = [Parameter(name, Parameter.POSITIONAL_OR_KEYWORD)
for name in names]
return Signature(parms)
def test():
pass
ls = ('a','b')
test.__signature__ = make_sig(*ls)
inspect.signature(test)
get:
<Signature (a, b)>
But What about it inside a class?
I have these two functions:
def swap_joker1(afile):
idx = afile.index(27)
if idx == len(afile) - 1:
afile[idx],afile[0]=afile[0],afile[idx]
else:
afile[idx],afile[idx+1]=afile[idx+1],afile[idx]
return afile
def swap_joker2(afile):
idx = afile1.index(28)
afile[idx],afile[idx+1],afile[idx+2]=afile[idx+1],afile[idx+2],afile[idx]
return afile
how can I compose them together. and become a new function called cipher_functions?
Well, you can create your own cute function composition function:
import functools
def compose(*functions):
return functools.reduce(lambda f, g: lambda x: f(g(x)), functions)
def foo(var):
return var // 2
def bar(var):
return var + 1
if __name__ == '__main__':
# Apply bar, then foo
composition = compose(bar, foo)
print composition(6)
You can apply the functions in whatever way you like, and in as many ways as you like. This answer was made possible, with the help of this website.
A few methods:
cipher_functions = lambda afile:swap_joker2(swap_joker1(afile))
def cipher_functions(afile):
return swap_joker2(swap_joker1(afile))
import functional #third party, not maintained. Alternatives exist
cipher_functions = functional.compose(swap_joker1, swap_joker2)
It would be great if Python offered a composition operator. Unfortunately, you need to do it yourself.
def cipher_functions(afile):
# This is f(g(x)); swap for g(f(x)) if necessary
return swap_joker1(swap_joker2(afile))
You can define a composition function easily:
def compose(f, g):
return lambda *args, **kwargs: f(g(*args, **kwargs))
cipher_functions = compose(swap_joker1, swap_joker2)
Your swap_joker1 function returns either afile or None, which make composition a bit hard.
Assuming it was actually:
def swap_joker1(afile):
idx = afile.index(27)
if idx == len(afile) - 1:
afile[idx],afile[0]=afile[0],afile[idx]
else:
afile[idx],afile[idx+1]=afile[idx+1],afile[idx]
return afile
Which returns afile unconditionally, you can simply write the composition as:
def cipher_functions(afile):
return swap_joker2(swap_joker1(afile))
And more generally:
def compose(f, g):
"Returns x -> (f o g)(x) = f(g(x))"
def fog(*args, **kwargs):
return f(g(*args, **kwargs))
return fog
cipher_functions = compose(swap_joker2, swap_joker1)