I recently had following code in mind and wondered what was wrong with it. Previously I used the .get method of dictionaries with success, but now i wanted to pass arguments too and this is where i noticed a somewhat weird behavior:
def string_encoder(nmstr):
return nmstr.encode('UTF-8')
def int_adder(nr_int):
return int(nr_int) + int(nr_int)
def selector(fun, val):
return {'str_en': string_encoder(val),
'nr_add': int_adder(val)}.get(fun, string_encoder(val))
selector('str_en', 'Test') -> ValueError
selector('str_en', 1) -> AttributeError
The above code will never run.
To inspect the issue i supplied a small piece of code:
def p1(pstr):
print('p1: ', pstr)
return pstr
def p2(pstr):
print('p2: ', pstr)
return pstr
def selector_2(fun, val):
return {'p1': p1(val),
'p2': p2(val)}.get(fun, p2(val))
selector_2('p1', 'Test')
Out[]: p1: Test
p2: Test
p2: Test
'Test'
I would expect the following .get('p1', 'test') to output 'p1: test' test.
But as it appears to me, every argument is evaluated, even if it is not selected. So my question is: Why is every argument evaluated with the .get method, or how can this behavior be explained?
dict creation is eager, as is argument evaluation. So before get even runs, you've called string_encoder twice, and int_adder once (and since the behaviors are largely orthogonal, you'll get an error for anything but a numeric str like "123").
You need to avoid calling the function until you know which one to call (and ideally, only call that function once).
The simplest solution is to have the dict and get call contain the functions themselves, rather than the result of calling them; you'll end up with whichever function wins, and you can then call that function. For example:
def selector(fun, val):
# Removed (val) from all mentions of functions
return {'str_en': string_encoder,
'nr_add': int_adder}.get(fun, string_encoder)(val) # <- But used it to call resulting function
Given string_encoder is your default, you could remove 'str_en' handling entirely to simplify to:
return {'nr_add': int_adder}.get(fun, string_encoder)(val)
which leads to the realization that you're not really getting anything out of the dict. dicts have cheap lookup, but you're rebuilding the dict every call, so you didn't save a thing. Given that you really only have two behaviors:
Call int_adder if fun is 'nr_add'
Otherwise, call string_encoder
the correct solution is just an if check which is more efficient, and easier to read:
def selector(fun, val):
if fun == 'nr_add':
return int_adder(val)
return string_encoder(val)
# Or if you love one-liners:
return int_adder(val) if fun == 'nr_add' else string_encoder(val)
If your real code has a lot of entries in the dict, not just two, one of which is unnecessary, then you can use a dict for performance, but build it once at global scope and reference it in the function so you're not rebuilding it every call (which loses all performance benefits of dict), e.g.:
# Built only once at global scope
_selector_lookup_table = {
'str_en': string_encoder,
'nr_add': int_adder,
'foo': some_other_func,
...
'baz': yet_another_func,
}
def selector(fun, val):
# Reused in function for each call
return _selector_lookup_table.get(fun, default_func)(val)
If you want to avoid evaluation of functions and only chooses the function, do this instead for your second block (the syntax will also work for your first block):
def selector_2(fun, val):
return {'p1': p1,
'p2': p2}.get(fun)(val)
Related
I have a function that that currently returns two values, an int and a string, for example:
def myfunc():
return 0, 'stringA'
This function is already in use in a lot of code, but I'd need to improve it so it returns three values, an int and two strings, for example:
def myfunc():
return 0, 'stringA', 'stringB'
Of course, I'd like to keep compatibility with existing code, so returning the values like the above modified function will lead to a ValueError.
One solution would be to wrap the improved function into another function with the old name, so we call the initial function in existing code, and the new function in new code, for example:
def newmyfunc():
return 0, 'A', 'B'
def myfunc():
result1, result2, _ = newmyfunc()
return result1, result2
As far as this solution works, I don't really find it elegant.
Is there a better way to achieve this goal?
Something like a polymorphic function which could return two or three values without having to modify existing code that uses the function?
First up, answering a question you didn't ask, but which may help in the future or for other folks:
When I find that I'm returning multiple items from a single function, and especially when the list of items returned starts to grow, I often find it useful to return either a dict or an object rather than a tuple. The reason is that as the returned-item list grows, it becomes harder to keep track of which item's at which index. If the group of returned items are going to be used separately and aren't closely-related other than both coming from the same function, I prefer a dict. If the returned items are being used together in multiple locations (e.g. user name, password, host & port), wrap them all in an object (instantiate a custom class), and just pass that around. YMMV, and it sounds like you're trying to avoid refactoring the code, so:
The simplest solution to your actual question is to add a keyword argument to your function, set a default on that argument, and use it to decide which version of the arguments to return:
def myfunc(return_length=2):
if return_length == 2:
return 0, 'stringA'
elif return_length == 3:
return 0, 'stringA', 'stringB'
else:
raise ValueError(f'Unexpected number of return arguments {return_length}')
Old code continues to call the function as-is, and new code explicitly calls my_func(return_length=3). At such point as all the old code gets deprecated, you can change the default value to 3 and/or throw an error when it's set to 2.
An example with decorators: the body of the involved functions stays untouched, the "modification"-part is delegated to an external function, the decorator.
Assumed "ground functions" take no arguments.
def dec(f_reference):
return lambda f_extra_feature: lambda:(*f_reference(), f_extra_feature())
def myfunc():
return 0, 'stringA'
def xxxfunc():
return 'XXX'
myfunc = dec(f_reference=myfunc)(f_extra_feature=xxxfunc)
print(myfunc)
#(0, 'stringA', 'XXX')
Depending on the needs the second parameter, f_extra_feature, can be made implicit.
A less flexible decoration could be done with the syntactic sugar notation
# order of argument is changed!
def dec2(f_extra_feature):
return lambda f_reference: lambda:(*f_reference(), f_extra_feature())
def xxxfunc():
return 'XXX'
#dec2(f_extra_feature=xxxfunc)
def myfunc():
return 0, 'stringA'
print(myfunc())
#(0, 'stringA', 'XXX')
EDIT:
def newmyfunc():
return 0, 'A', 'B'
def replacer(f):
return lambda f_target: lambda: f()[slice(0, 2)]
#replacer(newmyfunc)
def myfunc():
return 0, 'stringA'
# new body of the function, execute newmyfunc
print(myfunc())
I'm creating a function that takes in a callback function as an argument. I want to be able to use it like this:
def callback1(result, found_index):
# do stuffs
def callback2(result):
# do same stuffs even though it's missing the found_index parameter
somefunct(callback1)
somefunct(callback2)
# somefunct calls the callback function like this:
def somefunct(callback):
# do stuffs, and assign result and found_index
callback(result, found_index) # should not throw error
For context, I am somewhat trying to replicate how javascript's callback functions work for the .forEach function on arrays. You can make a function that takes in only the array item on that specific iteration, or the array item and index, or even the array item, index, and original array:
let some_array = ["apple", "orange", "banana"];
function callback1(value, index) {
console.log(`Item at index ${index}: ${value}`);
}
function callback2(value) {
console.log(`Value: ${value}`);
}
some_array.forEach(callback1); // runs with no errors
some_array.forEach(callback2); // runs with no errors
Furthermore, I don't want the callback function to force the * operator, but also allow them to use it if needed. Thank you, wonderful people of python.
(Posting this separately since it's fundamentally different to my other answer.)
If you need to pass a lot of values to some callbacks, without requiring other callbacks to declare a lot of unused parameters, a neat solution is to encapsulate all of those values in a single object. You can use collections.namedtuple to define a value type with named attributes, and then the callback can take one parameter and decide which attributes to use.
from collections import namedtuple
SomeFunctionResult = namedtuple('SomeFunctionResult', 'foo bar baz qux quz')
def some_function(callback):
result = SomeFunctionResult('foo', 'bar', 'baz', 'qux', 'quz')
callback(result)
Example:
>>> some_function(lambda r: print(r.foo, r.bar))
foo bar
>>> some_function(lambda r: print(r.baz, r.qux, r.quz))
baz qux quz
The downside is that this makes some_function less usable with existing functions which might expect to receive foo directly, rather than an object with a foo attribute. In that case, you have to write some_function(lambda r: blah(r.foo)) which is not as neat as some_function(blah).
The simplest approach would be to unify the signatures of your callbacks. Let's say you defined your forEach function as follows
def forEach(iterable, callback):
for index, elem in enumerate(iterable):
callback(elem, index)
You could then define Python analogs of the callack1 and callback2 Javascript functions as
def callback1(value, index):
print(f"Item at index {index}: {value}")
def callback2(value, _index):
print(f"Value: {value})
Rather than performing any complicated parameter-count-reasoning, exception handling, or dynamic dispatch within forEach, we delegate the decision of how to handle the value and index arguments to the callbacks themselves. If you need to adapt a single-parameter callback to work with forEach, you could simply use a wrapper lambda that discards the second argument:
forEach(some_iterable, lambda value, _index: callback(value))
However, at this point, you just have an obfuscated for loop, which would be much more cleanly expressed as
for elem in some_iterable:
callback(elem)
In this case, it is easier to ask for forgiveness than permission.
def some_function(callback):
result = 'foo'
found_index = 5
try:
callback(result, found_index)
except TypeError:
callback(result)
Example:
>>> some_function(print)
foo 5
>>> some_function(lambda x: print(x))
foo
this is the modified python code snippet you have provided that produces error , this works with no problem , you just have to unify the callback arguments number and type for each callback function called within the main function and define somefunc before calling it .
def callback1(result, found_index):
# do stuffs
result="overridden result in callback 1"
found_index ="overridden found_index in callback 1"
print(result,found_index)
def callback2(result,found_index):
# do same stuffs even though it's missing the found_index parameter
result="overridden result in callback 2"
print(result,found_index)
# somefunct calls the callback function like this:
def somefunct(callback):
# do stuffs, and assign result and found_index
result = "overridden result in somefunct"
found_index = "overridden index in somefunct"
callback(result, found_index) # NOW it should not throw error as the callback is fed with the 2 arguments used in callback1 and ignored in callback2
somefunct(callback1)
somefunct(callback2)
use optional arguments and check how much elemnts returned, sort of switch case:
https://linux.die.net/diveintopython/html/power_of_introspection/optional_arguments.html
def cons(a, b):
def pair(f):
return f(a, b)
return pair
def car(f):
def left(a, b):
return a
return f(left)
def cdr(f):
def right(a, b):
return b
return f(right)
Found this python code on git.
Just want to know what is f(a,b) in cons definition is, and how does it work?
(Not a function I guess)
cons is a function, that takes two arguments, and returns a function that takes another function, which will consume these two arguments.
For example, consider the following function:
def add(a, b):
return a + b
This is just a function that adds the two inputs, so, for instance, add(2, 5) == 7
As this function takes two arguments, we can use cons to call this function:
func_caller = cons(2, 5) # cons receives two arguments and returns a function, which we call func_caller
result = func_caller(add) # func_caller receives a function, that will process these two arguments
print(result) # result is the actual result of doing add(2, 5), i.e. 7
This technique is useful for wrapping functions and executing stuff, before and after calling the appropriate functions.
For example, we can modify our cons function to actually print the values before and after calling add:
def add(a, b):
print('Adding {} and {}'.format(a, b))
return a + b
def cons(a, b):
print('Received arguments {} and {}'.format(a, b))
def pair(f):
print('Calling {} with {} and {}'.format(f, a, b))
result = f(a, b)
print('Got {}'.format(result))
return result
return pair
With this update, we get the following outputs:
func_caller = cons(2, 5)
# prints "Received arguments 2 and 5" from inside cons
result = func_caller(add)
# prints "Calling add with 2 and 5" from inside pair
# prints "Adding 2 and 5" from inside add
# prints "Got 7" from inside pair
This isn't going to make any sense to you until you know what cons, car, and cdr mean.
In Lisp, lists are stored as a very simple form of linked list. A list is either nil (like None) for an empty list, or it's a pair of a value and another list. The cons function takes a value and a list and returns you another list just by making a pair:
def cons(head, rest):
return (head, rest)
And the car and cdr functions (they stand for "Contents of Address|Data Register", because those are the assembly language instructions used to implement them on a particular 1950s computer, but that isn't very helpful) return the first or second value from a pair:
def car(lst):
return lst[0]
def cdr(lst):
return lst[1]
So, you can make a list:
lst = cons(1, cons(2, cons(3, None)))
… and you can get the second value from it:
print(car(cdr(lst))
… and you can even write functions to get the nth value:
def nth(lst, n):
if n == 0:
return car(lst)
return nth(cdr(lst), n-1)
… or print out the whole list:
def printlist(lst):
if lst:
print(car(lst), end=' ')
printlist(cdr(lst))
If you understand how these work, the next step is to try them on those weird definitions you found.
They still do the same thing. So, the question is: How? And the bigger question is: What's the point?
Well, there's no practical point to using these weird functions; the real point is to show you that everything in computer science can be written with just functions, no built-in data structures like tuples (or even integers; that just takes a different trick).
The key is higher-order functions: functions that take functions as values and/or return other functions. You actually use these all the time: map, sort with a key, decorators, partial… they’re only confusing when they’re really simple:
def car(f):
def left(a, b):
return a
return f(left)
This takes a function, and calls it on a function that returns the first of its two arguments.
And cdr is similar.
It's hard to see how you'd use either of these, until you see cons:
def cons(a, b):
def pair(f):
return f(a, b)
return pair
This takes two things and returns a function that takes another function and applies it to those two things.
So, what do we get from cons(3, None)? We get a function that takes a function, and applies it to the arguments 3 and None:
def pair3(f):
return f(3, None)
And if we call cons(2, cons(3, None))?
def pair23(f):
return f(2, pair3)
And what happens if you call car on that function? Trace through it:
def left(a, b):
return a
return pair23(left)
That pair23(left) does this:
return left(2, pair3)
And left is dead simple:
return 2
So, we got the first element of (2, cons(3, None)).
What if you call cdr?
def right(a, b):
return a
return pair23(right)
That pair23(right) does this:
return right(2, pair3)
… and right is dead simple, so it just returns pair3.
You can work out that if we call car(cdr(pair23)), we're going to get the 3 out of it.
And now you can write lst = cons(1, cons(2, cons(3, None))), write the recursive nth and printlist functions above, and trace through how they work on lst.
I mentioned above that you can even get rid of integers. How do you do that? Read about Church numerals. You define zero and successor functions. Then you can define one as successor(zero) and two as successor(one). You can even recursively define add so that add(x, zero) is x but add(x, successor(y)) is successor(add(x, y)), and go on to define mul, etc.
You also need a special function you can use as a value for nil.
Anyway, once you've done that, using all of the other definitions above, you can do lst = cons(zero(cons(one, cons(two, cons(three, nil)))), and nth(lst, two) will give you back one. (Of course writing printlist will be a bit trickier…)
Obviously, this is all going to be a lot slower than just using tuples and integers and so on. But theoretically, it’s interesting.
Consider this: we could write a tiny dialect of Python that has only three kinds of statements—def, return, and expression statements—and only three kinds of expressions—literals, identifiers, and function calls—and it could do everything normal Python does. (In fact, you could get rid of statements altogether just by having a function-defining expression, which Python already has.) That tiny language would be a pain to use, but it would a lot easier to write a program to reason about programs in that tiny language. And we even know how to translate code using tuples, loops, etc. into code in this tiny subset language, which means we can write a program that reasons about that real Python code.
In fact, with a couple more tricks (curried functions and/or static function types, and lazy evaluation), the compiler/interpreter could do that kind of reasoning on the fly and optimize our code for us. It’s easy to tell programmatically that car(cdr(cons(2, cons(3, None)) is going to return 3 without having to actually evaluate most of those function calls, so we can just skip evaluating them and substitute 3 for the whole expression.
Of course this breaks down if any function can have side effects. You obviously can’t just substitute None for print(3) and get the same results. So instead, you need some clever trick where IO is handled by some magic object that evaluates functions to figure out what it should read and write, and then the whole rest of the program, the part that users write, becomes pure and can be optimized however you want. With a couple more abstractions, we can even make IO something that doesn’t have to be magical to do that.
And then you can build a standard library that gives you back all those things we gave up, written in terms of defining and calling functions, so it’s actually usable—but under the covers it’s all just reducing pure function calls, which is simple enough for a computer to optimize. And then you’ve basically written Haskell.
I'm brand-new to decorators and closures, I'm trying to practice with a simple example. When executed it raises an error of:
NameError: name 'congratulate' is not defined
What do I need to change?
"""
A recursive function to check if a string is a palindrome.
"""
#congratulate
def palindrome(phrase):
characters = [char.lower() for char in phrase if char.isalpha()]
chars_len = len(characters)
out1 = characters[0]
out2 = characters[-1]
if chars_len <= 2:
return out1 == out2
else:
if out1 == out2:
return palindrome(characters[1:-1])
else:
return False
def congratulate(func):
if func:
print('Congratulations, it\'s a palindrome!')
if __name__ == '__main__':
print(palindrome('Rats live on no evil star'))
"""
A recursive function to check if a string is a palindrome.
"""
def congratulate(func):
def wrapper(*argv, **kargs):
result = func(*argv, **kargs)
if result:
print('Congratulations, it\'s a palindrome!')
return result
return wrapper
#congratulate
def palindrome(phrase):
characters = [char.lower() for char in phrase if char.isalpha()]
chars_len = len(characters)
out1 = characters[0]
out2 = characters[-1]
if chars_len <= 2:
return out1 == out2
else:
if out1 == out2:
return palindrome(characters[1:-1])
else:
return False
if __name__ == '__main__':
print(palindrome('Rats live on no evil star'))
the essence of understanding decorator is
#f
def g(args)
=>
f(g)(args)
I know I'm late to the party, but I want to expand.
As noted, the NameError in this case is caused by the fact that you use a name before you actually create one. Moving congratulate() to the top remedies this.
Appart from the NameError you have two implicit Logic Errors relating to Decorator/Function Functionality:
First Issue:
Your if clause in congratulate always evaluates to True; you aren't exactly congratulating when a string is a palindrome.
This is caused by the fact that function objects always evaluate to True, so a condition of the form if func: will always execute:
def f():
pass
if f:
print("I'm true!")
# Prints: I'm true!
This is thankfully trivial and can easily be fixed by actually calling the function if func("test string"):
Second Issue:
The second issue here is less trivial and probably caused by the fact that decorators can be comfusing. You aren't actually using
congratulate() the way decorators are supposed to be used.
A decorator is a callable that returns a callable (callables are things like functions, classes overloaded on __call__). What your 'decorator' is doing here is simply accepting a function object, evaluating if the object is True and then printing congratulations.
Worst part? It is also implicitly rebinding the name palindrome to None.
Again, you can see this indirect effect (+1 for rhyming) in this next snippet:
def decor(f):
if f: print("Decorating can be tricky")
#decor
def f():
print("Do I even Exist afterwards?")
# When executed, this prints:
Decorating can be tricky
Cool, our function f has been decorated, but, look what happens when we try calling our function f:
f()
TypeError Traceback (most recent call last)
<ipython-input-31-0ec059b9bfe1> in <module>()
----> 1 f()
TypeError: 'NoneType' object is not callable
Yes, our function object f has now been assigned to None, the return value of our decor function.
This happens because as pointed out, the #syntax is directly equivalent to the following:
#decor
def f(): pass
# similar to
f = decor(f) # we re-assign the name f!
Because of this we must make sure the return value of a decorator is an object that can afterwards be called again, ergo, a callable object.
So what do you do? One option you might consider would be simply returning the function you passed:
def congratulate(func):
if func("A test Phrase!"):
print('Congratulations, it\'s a palindrome!')
return func
This will guarantee that after the decorator runs on your palindrome() function, the name palindrome is still going to map to a callable object.
The problem? This turns out to be a one-time ride. When Python encounters your decorator and your function, it's going to execute congratulate once and as a result only going to execute your if clause once.
But you need it to run this if every time your function is called! What can you do in order to accomplish this? Return a function that executes the decorated function (so called nested function decorators).
By doing this you create a new function for the name palindrome and this function contains your original function which you make sure is executed each time palindrome() is called.
def congratulate(func): # grabs your decorated function
# a new function that uses the original decorated function
def newFunc():
# Use the function
if func("Test string"):
print('Congratulations, it\'s a palindrome!')
# Return the function that uses the original function
return newFunc
newFunc is now a function that issues calls to your original function.
The decoration process now assigns the palindrome name to the newFunc object (notice how we returned it with return newFunc.
As a result, each time you execute a call of the form palindrome() this is tranlated to newFunc() which in turn calls func() in its body. (If you're still with me I commend you).
What's the final issue here? We've hard-coded the parameters for func. As is, everytime you call palindrome() function newFunc() will call your original function func with a call signature of func("Test String"), which is not what we want, we need to be able to pass parameters.
What's the solution? Thankfully, this is simple: Pass an argument to newFunc() which will then pass the argument to func():
def congratulate(func): # grabs your decorated function
# a new function that uses the original decorated function
# we pass the required argument <phrase>
def newFunc(phrase):
# Use the function
# we use the argument <phrase>
if func(phrase):
print('Congratulations, it\'s a palindrome!')
# Return the function that uses the original function
return newFunc
Now, everytime you call palindrome('Rats live on no evil star') this will translate to a call of newFunc('Rats live on no evil star') which will then transfer that call to your func as func('Rats live on no evil star') in the if clause.
After execution, this works wonderfully and get's you the result you wanted:
palindrome('Rats live on no evil star')
Congratulations, it's a palindrome!
I hope you enjoy reading, I believe I'm done (for now)!
Move the congratulate() function above the function it's decorating (palindrome).
Consider this Python segment:
def someTestFunction():
if someTest:
return value1
elif someOtherTest:
return value2
elif yetSomeOtherTest:
return value3
return None
def SomeCallingFunction():
a = someTestFunction()
if a != None:
return a
... normal execution continues
Now, the question: the three-line segment in the beginning of SomeCallingFunction to get the value of the test function and bail out if it's not None, is repeated very often in many other functions. Three lines is too long. I want to shorten it to one. How do I do that?
I can freely restructure this code and the contents of someTestFunction however needed. I thought of using exceptions, but those don't seem to help in cutting down the calling code length.
(I've read a bit about Python decorators, but haven't used them. Would this be the place? How would it work?)
If you want to use a decorator, it would look like this:
def testDecorator(f):
def _testDecorator():
a = someTestFunction()
if a is None:
return f()
else: return a
return _testDecorator
#testDecorator
def SomeCallingFunction():
... normal execution
When the module is first imported, it runs testDecorator, passing it your original SomeCallingFunction as a parameter. A new function is returned, and that gets bound to the SomeCallingFunction name. Now, whenever you call SomeCallingFunction, it runs that other function, which does the check, and returns either a, or the result of the original SomeCallingFunction.
I often use a hash table in place of a series of elifs:
def someTestFunction(decorated_test):
options = {
'val1': return_val_1,
'val2': return_val_2
}
return options[decorated_test]
You can set up options as a defaultdict(None) to default to None if a key isn't found.
If you can't get your tests in that form, then a series of if statements might actually be the best thing to do.
One small thing you can do to shorten your code is to use this:
if a: return a
There may be other ways to shorten your code, but these are the ones I can come up with on the spot.
I think this would do it:
UPDATE Fixed!
Sorry for yesterday, I rushed and didn't test the code!
def test_decorator( test_func ):
def tester( normal_function ):
def tester_inner():
a = test_func()
if a is not None:
return a
return normal_function()
return tester_inner
return tester
#usage:
#test_decorator( my_test_function )
def my_normal_function():
#.... normal execution continue ...
It's similar to DNS's answer but allows you to specify which test function you want to use