Is there a way to both transform and filter in a single list comprehension, i.e.:
def transform(el):
if some_condition(el):
return None
return complex_logic(el)
def main():
transformed = [transform(el) for el in some_list if transform(el) != None]
but avoid calling transform twice? I.e. assign it to a variable, something like (in pseudo-Python):
def main():
transformed = [transformed for el in some_list let transformed = transform(el) if transformed != None]
Since Python 3.8 you can use walrus operator :=:
def main():
return [res for el in some_list if (res := transform(el)) is not None]
This way the result of calling to the transform function is stored in res then you can use it in the expression part of your list comprehension.
Replace let transformed = transform(el) with for transformed in [transform(el)].
I would approach the solution from simple, over idiomatic to readable:
Simple loop with temp var
The simple but verbose for-loop can be used to "cache" the transformation result in a temporary variable t:
def transform(el):
if some_condition(el):
return None
return complex_logic(el)
def main():
transformed_list = []
for el in some_list:
t = transform(el) # invoked once
if t is not None: # equivalent to `if transform(el) != None`
transformed_list.append(t)
Embedded List Comprehension
Like Kelly Bundy suggests, embed the list-comprehensions:
transformation of elements
filter for non-null
See also Temporary variable within list comprehension
Decouple condition from transformation
Command–query separation (CQS) can be applied
to have a simplifying effect on a program, making its states (via queries) and state changes (via commands) more comprehensible.
Assume the 2 given functions (some_condition and complex_logic) were separately defined because each implements a single-responsibility (SRP). Then it would be consequential to take advantage of this separation and reuse the 2 steps in suitable composition:
Query: by filter first using the condition function as predicate
Command: afterwards to transform by complex logic
This way the pipeline or stream might even become more readable:
transformed = [complex_logic(el) for el in filter(some_condition, some_list)]
Finally this is close to what Samwise advised in his comment: Now the SRP is followed.
I have a function I'm using to test in an if/then.
The issue is that I'm executing the function BOTH in the if conditional, and then again after the if statement because the function returns two items.
This just seems wasteful and I'm trying to think of ways to improve this. Here's a really basic version of what I'm trying to avoid: "True" is returned to allow the condition to pass, but then then "coolstuff()" is executed again to get more information from the function.
"coolstuff()" could possibly return false, so I can't use the returned string "stuff" as the test.
def coolstuff():
return True, "stuff"
if coolstuff()[0]:
coolthing = coolstuff()[1]
print coolthing
There's gotta be a better way to do this, no? My brain is melting a little as I try to hash it out.
I basically want to do something like this (invalid) syntax:
def coolstuff():
return True, "stuff"
if a, b == coolstuff() and a:
print b
Just collect both results into variables
a, b = fn()
if a:
# work with b
def coolstuff():
if valid:
return "stuff"
return None
data = coolstuff()
if data:
print(data)
Call the function and capture the entire returned value:
x = coolstuff()
Now you have access to both parts of the returned value, in x[0] and x[1].
Store it:
state, coolvar = coolstuff()
if state:
do_whatever(coolvar)
If in newer Python, you could use the dreaded walrus (but I prefer ti7's approach of just assigning in a separate line):
if (x := coolstuff())[0]:
print(x[1])
I'm tinkering with reinforcement learning, trying to implement simple environment using functional programming.
I have functions step: State, Action -> State and action_space: State -> [Action, ...].
I'm looking for functional programming way of implementing iterative composition of function step over initial State
s = State()
e = [s]
while action_space(s):
a = best(action_space(s))
s = step(s, a)
e.append(s)
So what I think I need is way of creating comprehension [f(x) and c(x), f(f(x)) and c(f(x)), f(f(f(x))) and c(f(f(x)), ...], but without unnecessary evaluations.
Thanks in advance.
Comprehension is not possible for this scenario, but you could create a generator. Before going into that, I would also suggest that you define your best and step functions in such a way that they return something falsy (like False or None) whenever the last (actions) argument is falsy. It will simplify your code.
Then you can define this generator:
def unfold(s):
while s:
yield s
s = step(s, best(action_space(s)))
This has the advantage of being lazy. You can use it like this:
e = [s for s in unfold(State())]
Or just:
e = list(unfold(State()))
A more functional programming approach could be this recursive function:
def unfold(s):
return [s] + unfold(step(s, best(action_space(s)))) if s else []
e = unfold(State())
If you find the need of a named function annoying, then you could go for this trick:
e = (lambda unfold:lambda s:unfold(unfold,s))(
lambda unfold, s: [s] + unfold(unfold, step(s, best(action_space(s)))) if s else []
)(State())
This is quite obscure though. The first function is called immediately with a another function as argument (the one on the second line): the actual unfold function, which gets is name by the parameter name of the first function.
Then this first function returns another function (lambda s:unfold(unfold,s)) that accepts the state argument, and calls unfold with it. We also invoke that function immediately, passing it the initial value State().
Note that unfold is also passed itself down the recursion tree so to keep it within scope.
Conventional way of dealing with optional list arguments is the following:
def func(list_of_vals = None):
if list_of_vals is None:
list_of_vals = []
...
I wounder if the following (shorter) version has any pitfalls? why nobody do that? is it considered more obscure?
list_of_vals = list_of_vals or []
The pattern if arg is None: is usual, it is familiar to Python developers and has no weird edge cases. My recommendation is just stick with the convention.
Your proposal using or logically deviates when bool(list_of_vals) == False but list_of_vals is not None, so I would recommend against doing that.
Another possible option is to "duck type" using an empty tuple:
def a(vals=()):
...
Because tuples are immutable, this has none of the pitfalls of the mutable default list. There are many use-cases where you only need to the input container to be indexable and iterable, so vals can happily remain as a tuple.
There are two cases, you want that the list can be altered outside the function, then the second variant prevents calling a with:
some_list = []
a(some_list)
print(some_list)
If you want to prevent alternation of the list parameter, you should make a copy inside a
def a(l=()):
l = list(l)
I am always annoyed by this fact:
$ cat foo.py
def foo(flag):
if flag:
return (1,2)
else:
return None
first, second = foo(True)
first, second = foo(False)
$ python foo.py
Traceback (most recent call last):
File "foo.py", line 8, in <module>
first, second = foo(False)
TypeError: 'NoneType' object is not iterable
The fact is that in order to correctly unpack without troubles I have either to catch the TypeError or to have something like
values = foo(False)
if values is not None:
first, second = values
Which is kind of annoying. Is there a trick to improve this situation (e.g. to so set both first and second to None without having foo returning (None, None)) or a suggestion about the best design strategy for cases like the one I present ? *variables maybe ?
Well, you could do...
first,second = foo(True) or (None,None)
first,second = foo(False) or (None,None)
but as far as I know there's no simpler way to expand None to fill in the entirety of a tuple.
I don't see what is wrong with returning (None,None). It is much cleaner than the solutions suggested here which involve far more changes in your code.
It also doesn't make sense that you want None to automagically be split into 2 variables.
I think there is a problem of abstraction.
A function should maintain some level of abstraction, that helps in reducing complexity of the code.
In this case, either the function is not maintaining the right abstraction, either the caller is not respecting it.
The function could have been something like get_point2d(); in this case, the level of the abstraction is on the tuple, and therefore returning None would be a good way to signal some particular case (e.g. non-existing entity). The error in this case would be to expect two items, while actually the only thing you know is that the function returns one object (with information related to a 2d point).
But it could also have been something like get_two_values_from_db(); in this case the abstraction would be broken by returning None, because the function (as the name suggest) should return two values and not one!
Either way, the main goal of using a function - reducing complexity - is, at least partially, lost.
Note that this issue would not appear clearly with the original name; that's also why it is always important to give good names to function and methods.
I don't think there's a trick. You can simplify your calling code to:
values = foo(False)
if values:
first, second = values
or even:
values = foo(False)
first, second = values or (first_default, second_default)
where first_default and second_default are values you'd give to first and second as defaults.
How about this:
$ cat foo.py
def foo(flag):
if flag:
return (1,2)
else:
return (None,)*2
first, second = foo(True)
first, second = foo(False)
Edit: Just to be clear, the only change is to replace return None with return (None,)*2. I am extremely surprised that no one else has thought of this. (Or if they have, I would like to know why they didn't use it.)
You should be careful with the x or y style of solution. They work, but they're a bit broader than your original specification. Essentially, what if foo(True) returns an empty tuple ()? As long as you know that it's OK to treat that as (None, None), you're good with the solutions provided.
If this were a common scenario, I'd probably write a utility function like:
# needs a better name! :)
def to_tup(t):
return t if t is not None else (None, None)
first, second = to_tup(foo(True))
first, second = to_tup(foo(False))
def foo(flag):
return ((1,2) if flag else (None, None))
OK, I would just return (None, None), but as long as we are in whacko-land (heh), here is a way using a subclass of tuple. In the else case, you don't return None, but instead return an empty container, which seems to be in the spirit of things. The container's "iterator" unpacks None values when empty. Demonstrates the iterator protocol anyway...
Tested using v2.5.2:
class Tuple(tuple):
def __iter__(self):
if self:
# If Tuple has contents, return normal tuple iterator...
return super(Tuple, self).__iter__()
else:
# Else return a bogus iterator that returns None twice...
class Nonerizer(object):
def __init__(self):
self.x=0
def __iter__(self):
return self
def next(self):
if self.x < 2:
self.x += 1
return None
else:
raise StopIteration
return Nonerizer()
def foo(flag):
if flag:
return Tuple((1,2))
else:
return Tuple() # It's not None, but it's an empty container.
first, second = foo(True)
print first, second
first, second = foo(False)
print first, second
Output is the desired:
1 2
None None
Over 10 years later, if you want to use default values I don't think there is a better way than the one already provided:
first, second = foo(False) or (first_default, second_default)
However, if you want to skip the case when None is returned, starting from Python 3.8 you can use the walrus operator (ie. assignment expressions) - also note the simplified foo:
def foo(flag):
return (1, 2) if flag else None
if values := Foo(False):
(first, second) = values
You could use an else branch to assign default values that's worse than the previous or option.
Sadly, the walrus operator does not support unparenthesized tuples so it is just a one line gain compared to:
values = foo(False)
if values:
first, second = values
One mechanism you can use to avoid the problem entirely when you have control of the method foo is to change the prototype to allow giving a default. This works if you are wrapping state but can't guarantee that a particular tuple value exists.
# self.r is for example, a redis cache
# This method is like foo -
# it has trouble when you unpack a json serialized tuple
def __getitem__(self, key):
val = self.r.get(key)
if val is None:
return None
return json.loads(val)
# But this method allows the caller to
# specify their own default value whether it be
# (None, None) or an entire object
def get(self, key, default):
val = self.r.get(key)
if val is None:
return default
return json.loads(val)
I found a solution for this problem:
Return None or return an object.
However you don't want to have to write a class just to return an object. For this, you can use a named tuple
Like this:
from collections import namedtuple
def foo(flag):
if flag:
return None
else:
MyResult = namedtuple('MyResult',['a','b','c']
return MyResult._make([1,2,3])
And then:
result = foo(True) # result = True
result = foo(False) # result = MyResult(a=1, b=2, c=3)
And you have access to the results like this:
print result.a # 1
print result.b # 2
print result.c # 3