I am learning concepts of functional programming and trying out problem exercises.
An exercise,
Reverse a list using map/reduce.
My solution:
lists = [ 1, 2, 3, 5 ,6]
def tree_reverse(lists):
return reduce(lambda a, x: a + [a.insert(0,x)], lists, [])
print tree_reverse(lists)
Output:
[6, 5, 3, 2, 1, None, None, None, None, None]
I don't understand why there are Nones equal to number of elements in list.
EDIT: Extending question for case of nested lists.
lists = [ 1 ,2 ,3 , [5,6,7], 8, [9, 0, 1 ,4 ]]
At first, i don't know Python good enough, but as you marked the question as "functional-programming" and you stated you want to learn in general about functional programming i want to give some general explanation of your problem.
At first, you should rethink what the map and reduce operation do and what they expect. Let's start with the map function.
A map function for a list (in functional programming you don't have only one map, you have one map for every generic type) executes a function on every single argument. That function can operate on that single argument and transform it. The result of this transformation is turned into a new list. One important note is that the function you pass to map only ever sees one element, so you cannot use map itself to transform a list as a whole. You only can transform every single element of a list. But changing the order needs the idea of having at least two elements of a list, and a map only ever gets a single element. This is the reason why the reverse logic cannot be in map and has to be part of the reduce operation.
The reduce function on the other hand expects a function that takes two things of the same type, and returns a single item of the same type. For example the reduce function could take the "+" function. "+" is a function that takes two int and returns a new int. The signature is basically int + int = int. The essence of reduce is to take two things and turn it into a single new item. If you for example had something more complex like a class "Foo", then the function you have to provide to reduce need to have the signature Foo * Foo -> Foo. Meaning take a Foo as first and second argument, and return a single new Foo.
It is also important to note how reduce will work with that function. Let's assume you have the List [1,2,3,4,5] on top let's assume you have an accumulator function that takes two int that just adds them together. What you can think of what will happen is the following.
The reduce function takes the first two arguments of your list (1 and 2 in this case)
Passes those to the accumulator function (1 + 2 = 3)
And replaces the two arguments in the list with the result. [3,3,4,5]
Go back to 1) and repeat that process until your List only have a single item
Return that single item
So what you can imagine what happens is the following
[1,2,3,4,5] -> 1 + 2 = 3
[3,3,4,5] -> 3 + 3 = 6
[6,4,5] -> 6 + 4 = 10
[10,5] -> 10 + 5 = 15
[15] -> Return just "15". Not a "[15]"
Actually the process internal usually works a little bit different. But this is the process you can imagine what happens. It is important to note that you never modify a list. You always create new lists in that process. Another important note. At the end of the reduce function you have a list of an single item. But the reduce function does not return the whole list. It returns just the single element. So you get 15 an int as a result. You don't get a list with a single item containing 15. Reduce will just return that single element. Whatever it is. One other way to think about it. You always get exactly the type back of your accumulator function. If you pass a accumulator function that takes two int adds them and returns a new int. Your reduce function will also return an int. If you pass an accumulator function that takes two Foo classes and returns a new Foo. Your reduce function will also return a Foo as its result. The return type of reduce is always the same as the types of your accumulator function.
Now let's take all those pieces and put them together. The goal is to reverse a list. The first important thing is. Your result type will be a list. That also means that the function you pass into reduce also have to return a list. But as the input are always the same as the output. You now have to provide a accumulator function that takes two lists, and returns a single new list.
But now let's step back. What happens if you use your list "[1,2,3,4,5]" directly as the input to reduce? The short answer is, it will not work. reduce will take two argument of your list. But what you have is a list of int. But your accumulator function expects two lists not two int. To solve that problem, what you now can do is try to convert every single element of the list into its own list. So how do we transform every single element of a list? Right! With map! So what you have to do is the following. You first map your list into a list of lists.
[1,2,3,4,5] -> [[1],[2],[3],[4],[5]]
Now your reduce function gets the first two elements of your first list. It means you now have an accumulator function that gets [1] and [2] as its argument. Two separate lists. But your accumulator function has to return a single new list. If unclear, just remind what the accumulator function takes and return.
int, int -> int
float,float -> float
Foo,Foo -> Foo
list,list -> list
So what you now have to do is to combine those two lists into a single new list. Or in other words you have to append/concat those two lists. I don't knew exactly how you concat two lists in Python let's assume here the operation would be "++". So if you concat just the first argument and the second argument, you will not get what you want.
[[1],[2],[3],[4],[5]] -> [1] ++ [2] = [1,2]
[[1,2],[3],[4],[5]] -> [1,2] ++ [3] = [1,2,3]
[[1,2,3],[4],[5]] -> [1,2,3] ++ [4] = [1,2,3,4]
[[1,2,3,4],[5]] -> [1,2,3,4] ++ [5] = [1,2,3,4,5]
[[1,2,3,4,5]] -> Return first element [1,2,3,4,5]
So what you have to do is to concat the second argument with the first one.
[[1],[2],[3],[4],[5]] -> [2] ++ [1] = [2,1]
[[2,1],[3],[4],[5]] -> [3] ++ [2,1] = [3,2,1]
[[3,2,1],[4],[5]] -> [4] ++ [3,2,1] = [4,3,2,1]
[[4,3,2,1],[5]] -> [5] ++ [4,3,2,1] = [5,4,3,2,1]
[[5,4,3,2,1]] -> Return first element [5,4,3,2,1]
And now what you get is your reversed list. So what you have todo
map every element to a list. So you get a list of list
use reduce to concat every list of list into a new list in reversed order.
For example, in F# the whole code would look like this.
let list = [1;2;3;4;5]
let reversed =
list
|> List.map (fun x -> [x]) // map every element to a list
|> List.reduce (fun xs ys -> List.append ys xs) // Note ys ++ xs - reversed order for appending
printfn "%A" reversed
// prints: [5;4;3;2;1]
I think you should be able to translate this to Python. As a further notice. Doing "map" and then a "concat" operation (without reversing) is also named "bind" in functional programming.
Your reduce operation is appending a None to the end of the list a because a.insert(0,x) will return None.
So you are modifying the list a in-place, but appending a None value at the same time.
If you rewrite your reduce operation to use a for loop, it'd look like:
def reverse_list(lists):
r = []
for x in lists:
val = r.insert(0,x)
r = r + [val] # val (the return value of r.insert) is None
return r
In case of nested list. I am posting answer for reference.
lists = [ 1 ,2 ,3 , [5,6,7], 8, [9, 0, 1 ,4 ]]
def tree_reverse(lists):
return reduce(lambda a, x: ( map(tree_reverse, [x]) if isinstance(x,list) else [x] ) + a , lists , [] )
print tree_reverse(lists)
Output :
[[4, 1, 0, 9], 8, [7, 6, 5], 3, 2, 1]
Functional programming languages will often use recursion to implement iteration.
A recursive tree_reverse function might look like
def tree_reverse(lists):
return tree_reverse(lists[1:]) + [lists[0]] if lists else []
If you look at Haskell it might be implemented like this
reverse' :: [a] -> [a]
reverse' [] = []
reverse' (x:xs) = reverse' xs ++ [x]
Where the x is the head of the list (first element) and xs is the tail (the list minus the head) and you can see that reverse' is called recursively with those elements reversed and the reversed list is built up iteratively.
Related
Why do these two operations (append() resp. +) give different results?
>>> c = [1, 2, 3]
>>> c
[1, 2, 3]
>>> c += c
>>> c
[1, 2, 3, 1, 2, 3]
>>> c = [1, 2, 3]
>>> c.append(c)
>>> c
[1, 2, 3, [...]]
>>>
In the last case there's actually an infinite recursion. c[-1] and c are the same. Why is it different with the + operation?
To explain "why":
The + operation adds the array elements to the original array. The array.append operation inserts the array (or any object) into the end of the original array, which results in a reference to self in that spot (hence the infinite recursion in your case with lists, though with arrays, you'd receive a type error).
The difference here is that the + operation acts specific when you add an array (it's overloaded like others, see this chapter on sequences) by concatenating the element. The append-method however does literally what you ask: append the object on the right-hand side that you give it (the array or any other object), instead of taking its elements.
An alternative
Use extend() if you want to use a function that acts similar to the + operator (as others have shown here as well). It's not wise to do the opposite: to try to mimic append with the + operator for lists (see my earlier link on why). More on lists below:
Lists
[edit] Several commenters have suggested that the question is about lists and not about arrays. The question has changed, though I should've included this earlier.
Most of the above about arrays also applies to lists:
The + operator concatenates two lists together. The operator will return a new list object.
List.append does not append one list with another, but appends a single object (which here is a list) at the end of your current list. Adding c to itself, therefore, leads to infinite recursion.
As with arrays, you can use List.extend to add extend a list with another list (or iterable). This will change your current list in situ, as opposed to +, which returns a new list.
Little history
For fun, a little history: the birth of the array module in Python in February 1993. it might surprise you, but arrays were added way after sequences and lists came into existence.
The concatenation operator + is a binary infix operator which, when applied to lists, returns a new list containing all the elements of each of its two operands. The list.append() method is a mutator on list which appends its single object argument (in your specific example the list c) to the subject list. In your example this results in c appending a reference to itself (hence the infinite recursion).
An alternative to '+' concatenation
The list.extend() method is also a mutator method which concatenates its sequence argument with the subject list. Specifically, it appends each of the elements of sequence in iteration order.
An aside
Being an operator, + returns the result of the expression as a new value. Being a non-chaining mutator method, list.extend() modifies the subject list in-place and returns nothing.
Arrays
I've added this due to the potential confusion which the Abel's answer above may cause by mixing the discussion of lists, sequences and arrays.
Arrays were added to Python after sequences and lists, as a more efficient way of storing arrays of integral data types. Do not confuse arrays with lists. They are not the same.
From the array docs:
Arrays are sequence types and behave very much like lists, except that the type of objects stored in them is constrained. The type is specified at object creation time by using a type code, which is a single character.
append is appending an element to a list. if you want to extend the list with the new list you need to use extend.
>>> c = [1, 2, 3]
>>> c.extend(c)
>>> c
[1, 2, 3, 1, 2, 3]
Python lists are heterogeneous that is the elements in the same list can be any type of object. The expression: c.append(c) appends the object c what ever it may be to the list. In the case it makes the list itself a member of the list.
The expression c += c adds two lists together and assigns the result to the variable c. The overloaded + operator is defined on lists to create a new list whose contents are the elements in the first list and the elements in the second list.
So these are really just different expressions used to do different things by design.
The method you're looking for is extend(). From the Python documentation:
list.append(x)
Add an item to the end of the list; equivalent to a[len(a):] = [x].
list.extend(L)
Extend the list by appending all the items in the given list; equivalent to a[len(a):] = L.
list.insert(i, x)
Insert an item at a given position. The first argument is the index of the element before which to insert, so a.insert(0, x) inserts at the front of the list, and a.insert(len(a), x) is equivalent to a.append(x).
you should use extend()
>>> c=[1,2,3]
>>> c.extend(c)
>>> c
[1, 2, 3, 1, 2, 3]
other info: append vs. extend
See the documentation:
list.append(x)
Add an item to the end of the list; equivalent to a[len(a):] = [x].
list.extend(L)
- Extend the list by appending all the items in the given list;
equivalent to a[len(a):] = L.
c.append(c) "appends" c to itself as an element. Since a list is a reference type, this creates a recursive data structure.
c += c is equivalent to extend(c), which appends the elements of c to c.
The following python tutorial says that:
List comprehension is a complete substitute for the lambda function as well as the functions map(), filter() and reduce().
http://python-course.eu/python3_list_comprehension.php
However, it does not mention an example how a list comprehension can substitute a reduce() and I can't think of an example how it should be possible.
Can please someone explain how to achieve a reduce-like functionality with list comprehension or confirm that it isn't possible?
Ideally, list comprehension is to create a new list. Quoting official documentation,
List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.
whereas reduce is used to reduce an iterable to a single value. Quoting functools.reduce,
Apply function of two arguments cumulatively to the items of sequence, from left to right, so as to reduce the sequence to a single value.
So, list comprehension cannot be used as a drop-in replacement for reduce.
I was surprised at first to find that Guido van Rossum, creator of Python, was against reduce. His reasoning was that beyond summing, multiplying, and-ing, and or-ing, using reduce yields an unreadable solution that is better suited by a function which iterates through and updates an accumulator. His article on the matter is here. So no, there isn't a list comprehension alternative to reduce, instead the "pythonic" way is to implement an accumulating function the old fashioned way:
Instead of:
out = reduce((lambda x,y: x*y),[1,2,3])
Use:
def prod(myList):
out = 1
for el in myList:
out *= el
return out
Of course nothing stops you from continuing to use reduce (python 2) or functools.reduce (python 3)
List comprehensions are supposed to return lists. If your reduce is supposed to return a list, then yes, you can replace it with a list comprehension.
But this is no obstacle to providing "reduce-like functionality". Python lists can contain any object. If you'll accept your result contained in a single-item list, then there is a [...][0] list comprehension form that can replace any reduce() whatsoever.
This should be obvious, but that form is
[x for x in [reduce(function, sequence, initial)]][0]
for some binary function and and some iterable sequence and some initial value. Or, if you want the initial from the first of the iterable,
[x for x in [reduce(function, sequence)]][0]
Arguably, the above is cheating, and also pointless, since you could just use reduce without the comprehension. So let's try it without reduce.
[stack.append(function(stack.pop(), e)) or stack[0]
for stack in ([initial],)
for e in sequence][-1]
This produces a list of all the intermediate values, and we want the last one. [-1] is just as easy as [0]. We need an accumulator to reduce, but can't use assignment statements in a comprehension, hence the stack (which is just a list), but we could have used many other data structures here. The .append() always returns None, so we use or stack[0] to put the value so far in the resulting list.
It's a little more difficult without initial,
[stack.append(function(stack.pop(), e)) or stack[0]
for it in [iter(sequence)]
for stack in [[next(it)]]
for e in it][-1]
Really, you might as well use a for statement at this point.
But this takes up memory for the list of intermediate values. For a very long sequence, that might be a problem. But we can avoid that too by using generator expressions.
Doing this is tricky, so let's start with an easier example and work up to it.
stack = [initial]
[stack.append(function(stack.pop(), e)) for e in sequence]
stack.pop() # returns the answer
It computes the answer, but also creates a useless list of Nones. We can avoid that by converting it to a generator expression inside a list comprehension.
stack = [initial]
[_ for _s in (stack.append(function(stack.pop(), e)) or ()
for e in sequence)
for _ in _s]
stack.pop()
The list comprehension exhausts the generator that updates the stack, but returns an empty list itself. This is possible because the inner loop always has zero iterations, because _s is always an empty tuple.
We can move the stack.pop() inside if the last _s has one element. It doesn't matter what that element is though. So we chain on a [None] as the final _s.
from itertools import chain
stack = [initial]
[stack.pop()
for _s in chain((stack.append(function(stack.pop(), e)) or ()
for e in sequence),
[[None]])
for _ in _s][0]
Again, we have a single-item list comprehension. We can also implement chain as a generator expression. And you've already seen how to move the stack variable inside using a single-item list.
[stack.pop()
for stack in [[initial]]
for _s in (
x
for xs in [
(stack.append(function(stack.pop(), e)) or ()
for e in sequence),
[[None]],
]
for x in xs)
for _ in _s][0]
And we can also get the initial from the sequence for the two-argument reduce.
[stack.pop()
for it in [iter(sequence)]
for stack in [[next(it)]]
for _s in (
x
for xs in [
(stack.append(function(stack.pop(), e)) or ()
for e in it),
[[None]],
]
for x in xs)
for _ in _s][0]
This is insane. But it works. So yes, it's possible to get "reduce-like functionality" with comprehensions. That doesn't mean you should. Seven fors is too hard!
You could accomplish something like a reduce with a comprehension by using a couple of helper functions that I've named last and cofold:
>>> last(r(a+b) for a, b, r in cofold(range(10)))
45
This is functionally equivalent to
>>> reduce(lambda a, b: a+b, range(10))
45
Note that unlike reduce() the comprehension didn't use a lambda.
The trick is to use a generator with a callback to "return" the result of the operator. cofold is the corecursive dual of the reduce (or fold) function.
_sentinel = object()
def cofold(it, initial=_sentinel):
if initial is _sentinel:
it = iter(it)
accumulator = next(it)
else:
accumulator = initial
def callback(result):
nonlocal accumulator
accumulator = result
return result
for element in it:
yield accumulator, element, callback
Here's cofold in a list comprehension.
>>> [r(a+b) for a, b, r in cofold(range(10))]
[1, 3, 6, 10, 15, 21, 28, 36, 45]
The elements represent each step in the dual reduction. The last one is our answer. The last function is trivial.
def last(it):
for e in it:
pass
return e
Unlike reduce, cofold is a lazy generator, so it can safely act on infinite iterables when used in a generator expression.
>>> from itertools import islice, count
>>> lazy_results = (r(a+b) for a, b, r in cofold(count()))
>>> [*islice(lazy_results, 0, 9)]
[1, 3, 6, 10, 15, 21, 28, 36, 45]
>>> next(lazy_results)
55
>>> next(lazy_results)
66
I'm from a C++ background so this problem seems a little absurd to me:
Let's suppose I have a function:
def scale(data, factor):
for val in data:
val *= factor
This doesn't work as intended, if I pass a list, it changes nothing, but
def scale(data, factor):
for index, val in enumerate(data):
data[index] *= factor
and lst = [val * factor for val in lst] works properly.
How does Python handle argument passing? How do I know if the actual reference, or alias is passed?
if you want to mutate the list, you need to reference the elements. This version uses map (it could be written using list comprehensions)
def scale(data, factor):
return map(lambda x : x*factor, data)
a lambda function is an anonymous function.
>>> (lambda x : x + 1) (5)
6
The x takes the place of the variable in this case 5 + 1
So in this case, we traverse the list applying the function f(x) -> x * factor to every element of the list. The original list is not mutated, but instead we return a new version.
In python basic data types are passed by value - for example int, str, bool etc are passed by value
Derived data types like classes, enum, list, dict are passed by reference.
In your example, the problem is how you use the for loop - not the function argument. If you do:
for val in lst:
val += 1
The values inside lst won't get updated because the val is not the same as lst[0], lst[1] and so on IF val is of the basic data types. So, even here, the val is copied by value.
Second, In your example with enumerate:
But when you loop over the enumerated list, you are using data[index] - which modifies the element in the actual list.
And finally, In your example with the generator:
lst = [val * factor for val in lst] - here the generator loops over every element and creates a new list which is again stored in lst. This is something like a = a + 2 but extended to lists.
This behaviour is so because the basic data types are passed by value and the derived data types like lists are passed by reference consider this
>>> x = 24
>>> x + 1
25
>>> x
24
but on the otherhand with a list
>>> y = [1, 2, 3, 4]
>>> y.remove(2)
>>> y
[1,3,4]
so you should always be careful to reassign values back when performing operations on them in the case of the basic data ypes and also be careful with datatypes that are passed by reference because you could accidentally modify a variable without knowing
Like this:
> (my-append (list 1 2) 3)
'(1 2 3)
I know append in racket is actually to concatenate two list.
And cons just add an element to the head of a list instead of tail
Does anyone have ideas about this?
In Pyton, the append() method modifies the list in-place:
lst = [1, 2]
lst.append(3)
lst
=> [1, 2, 3]
Racket's lists are immutable by default, the closest thing to an in-place append() requires you to use mutable lists, and you have to pack the element to be appended in a list of its own:
(require scheme/mpair)
(define lst (mlist 1 2))
(mappend! lst (mlist 3))
lst
=> (mlist 1 2 3)
Notice that using immutable lists with the append procedure will produce a new list, leaving the original list untouched - so it wouldn't be "like Python":
(define lst (list 1 2))
(append lst (list 3)) ; this returns the new list '(1 2 3)
lst
=> (list 1 2)
In fact, Scheme's append procedure behaves just like Python's + operation between lists:
lst = [1, 2]
lst + [3] # this returns the new list [1, 2, 3]
lst
=> [1, 2]
Come to think of it, Python's append() might have a misleading name - in most functional programming languages, the append operation is always defined between two lists, whereas in Python it's an operation between a list and an element. Maybe add() would have been a better name, like Java's add() operation of the List interface.
If you're in Racket, you probably want to look into the growable vector library (data/gvector). This provides a container type that supports many of the features you're used to with Python's growable lists.
Example:
#lang racket
(require data/gvector)
(define lst (gvector 1 2))
(gvector-add! lst 3)
(for ([elt lst]) (printf "I see: ~s\n" elt))
Otherwise, your question ends up reducing to: how do I make immutable linked lists work like mutable sequentially-allocated arrays? And that's not going to work well: data types are different for different reasons.
I don't know what you mean with "append like python", but if all you want is to insert a new element at the end of the list, you may use this:
(define (my-append lst el)
(append lst (list el)))
Without any heavy libraries such as numpy, I want to uniformly handle a single list or multi-dimensional list in my code. For example, the function sum_up(list_or_matrix) should
return 6 for argument [1, 2, 3] and return 9 for [[1, 2, 3], [1, 2, 0]].
My question is:
1. Can I code in a way without explicitly detecting the dimension of my input such as by isinstance(arg[0], (tuple, list))?
2. If I have to do so, is there any elegant way of detecting the dimension of a list (of list of list ...), e.g. recursively?
As many users suggested you can always use dict instead of list for any-dimensinal collection. Dictionaries are accepting tuples as arguments as they are hashable. So you can easy fill-up your collection like
>>> m = {}
>>> m[1] = 1
>>> m[1,2] = 12
>>> m[1,2,"three",4.5] = 12345
>>> sum(m.values()) #better use m.itervalues() in python 2.*
12358
You can solve this problem using recursion, like this:
#!/usr/bin/env python
def sum(seq_or_elem):
if hasattr(seq_or_elem, '__iter__'):
# We were passed a sequence so iterate over it, summing the elements.
total = 0
for i in seq_or_elem:
total += sum(i)
return total
else:
# We were passed an atomic element, the sum is identical to the passed value.
return seq_or_elem
Test:
>>> print(sum([1, 2, [3, [4]], [], 5]))
15
Well I dont see a way if you are planning to use a single function to sum up your list like sum_up(list_or_matrix).
If you are having a list of lists I would only imagine you need to loop through the list to find out if its a 1-D or a 2-D list. Anyway whats wrong in looping?
def sum_up(matrix):
2DMatrix = False
for item in matrix:
if type(item) == list:
2DMatrix = True
if(2DMatrix):
//sum 2d matrix
else:
//sum 1d matrix
A simple way to sum up a matrix is as follow:
def sum_up(matrix):
if isinstance(matrix, (tuple, list)):
return sum(matrix)
else:
return sum((sum(x) for x in matrix))
The 'else' branch uses list comprehension, a powerful and quick tool.
You could sum recursively until you have a scalar value:
def flatten(x):
if isinstance(x, list):
return sum(map(flatten, x))
return x
Note: you can use collections.Iterable (or another base class) instead of list, depending on what you want to flatten.