I have a function that calls subprocess.check_call() twice. I want to test all their possible outputs. I want to be able to set the first check_call() to return 1 and the second to return 0 and to do so for all possible combinations. The below is what I have so far. I am not sure how to adjust the expected return value
#patch('subprocess.check_call')
def test_hdfs_dir_func(mock_check_call):
for p, d in list(itertools.product([1, 0], repeat=2)):
if p or d:
You can assign the side_effect of your mock to an iterable and that will return the next value in the iterable each time it's called. In this case, you could do something like this:
import copy
import itertools
import subprocess
from unittest.mock import patch
#patch('subprocess.check_call')
def test_hdfs_dir_func(mock_check_call):
return_values = itertools.product([0, 1], repeat=2)
# Flatten the list; only one return value per call
mock_check_call.side_effect = itertools.chain.from_iterable(copy.copy(return_values))
for p, d in return_values:
assert p == subprocess.check_call()
assert d == subprocess.check_call()
Note a few things:
I don't have your original functions so I put my own calls to check_call in the loop.
I'm using copy on the original itertools.product return value because if I don't, it uses the original iterator. This exhausts that original iterator when what we want is 2 separate lists: one for the mock's side_effect and one for you to loop through in your test.
You can do other neat stuff with side_effect, not just raise. As shown above, you can change the return value for multiple calls: https://docs.python.org/3/library/unittest.mock-examples.html#side-effect-functions-and-iterables
Not only that, but you can see from the link above that you can also give it a function pointer. That allows you to do even more complex logic when keeping track of multiple mock calls.
Related
def cons(a, b):
def pair(f):
return f(a, b)
return pair
def car(f):
def left(a, b):
return a
return f(left)
def cdr(f):
def right(a, b):
return b
return f(right)
Found this python code on git.
Just want to know what is f(a,b) in cons definition is, and how does it work?
(Not a function I guess)
cons is a function, that takes two arguments, and returns a function that takes another function, which will consume these two arguments.
For example, consider the following function:
def add(a, b):
return a + b
This is just a function that adds the two inputs, so, for instance, add(2, 5) == 7
As this function takes two arguments, we can use cons to call this function:
func_caller = cons(2, 5) # cons receives two arguments and returns a function, which we call func_caller
result = func_caller(add) # func_caller receives a function, that will process these two arguments
print(result) # result is the actual result of doing add(2, 5), i.e. 7
This technique is useful for wrapping functions and executing stuff, before and after calling the appropriate functions.
For example, we can modify our cons function to actually print the values before and after calling add:
def add(a, b):
print('Adding {} and {}'.format(a, b))
return a + b
def cons(a, b):
print('Received arguments {} and {}'.format(a, b))
def pair(f):
print('Calling {} with {} and {}'.format(f, a, b))
result = f(a, b)
print('Got {}'.format(result))
return result
return pair
With this update, we get the following outputs:
func_caller = cons(2, 5)
# prints "Received arguments 2 and 5" from inside cons
result = func_caller(add)
# prints "Calling add with 2 and 5" from inside pair
# prints "Adding 2 and 5" from inside add
# prints "Got 7" from inside pair
This isn't going to make any sense to you until you know what cons, car, and cdr mean.
In Lisp, lists are stored as a very simple form of linked list. A list is either nil (like None) for an empty list, or it's a pair of a value and another list. The cons function takes a value and a list and returns you another list just by making a pair:
def cons(head, rest):
return (head, rest)
And the car and cdr functions (they stand for "Contents of Address|Data Register", because those are the assembly language instructions used to implement them on a particular 1950s computer, but that isn't very helpful) return the first or second value from a pair:
def car(lst):
return lst[0]
def cdr(lst):
return lst[1]
So, you can make a list:
lst = cons(1, cons(2, cons(3, None)))
… and you can get the second value from it:
print(car(cdr(lst))
… and you can even write functions to get the nth value:
def nth(lst, n):
if n == 0:
return car(lst)
return nth(cdr(lst), n-1)
… or print out the whole list:
def printlist(lst):
if lst:
print(car(lst), end=' ')
printlist(cdr(lst))
If you understand how these work, the next step is to try them on those weird definitions you found.
They still do the same thing. So, the question is: How? And the bigger question is: What's the point?
Well, there's no practical point to using these weird functions; the real point is to show you that everything in computer science can be written with just functions, no built-in data structures like tuples (or even integers; that just takes a different trick).
The key is higher-order functions: functions that take functions as values and/or return other functions. You actually use these all the time: map, sort with a key, decorators, partial… they’re only confusing when they’re really simple:
def car(f):
def left(a, b):
return a
return f(left)
This takes a function, and calls it on a function that returns the first of its two arguments.
And cdr is similar.
It's hard to see how you'd use either of these, until you see cons:
def cons(a, b):
def pair(f):
return f(a, b)
return pair
This takes two things and returns a function that takes another function and applies it to those two things.
So, what do we get from cons(3, None)? We get a function that takes a function, and applies it to the arguments 3 and None:
def pair3(f):
return f(3, None)
And if we call cons(2, cons(3, None))?
def pair23(f):
return f(2, pair3)
And what happens if you call car on that function? Trace through it:
def left(a, b):
return a
return pair23(left)
That pair23(left) does this:
return left(2, pair3)
And left is dead simple:
return 2
So, we got the first element of (2, cons(3, None)).
What if you call cdr?
def right(a, b):
return a
return pair23(right)
That pair23(right) does this:
return right(2, pair3)
… and right is dead simple, so it just returns pair3.
You can work out that if we call car(cdr(pair23)), we're going to get the 3 out of it.
And now you can write lst = cons(1, cons(2, cons(3, None))), write the recursive nth and printlist functions above, and trace through how they work on lst.
I mentioned above that you can even get rid of integers. How do you do that? Read about Church numerals. You define zero and successor functions. Then you can define one as successor(zero) and two as successor(one). You can even recursively define add so that add(x, zero) is x but add(x, successor(y)) is successor(add(x, y)), and go on to define mul, etc.
You also need a special function you can use as a value for nil.
Anyway, once you've done that, using all of the other definitions above, you can do lst = cons(zero(cons(one, cons(two, cons(three, nil)))), and nth(lst, two) will give you back one. (Of course writing printlist will be a bit trickier…)
Obviously, this is all going to be a lot slower than just using tuples and integers and so on. But theoretically, it’s interesting.
Consider this: we could write a tiny dialect of Python that has only three kinds of statements—def, return, and expression statements—and only three kinds of expressions—literals, identifiers, and function calls—and it could do everything normal Python does. (In fact, you could get rid of statements altogether just by having a function-defining expression, which Python already has.) That tiny language would be a pain to use, but it would a lot easier to write a program to reason about programs in that tiny language. And we even know how to translate code using tuples, loops, etc. into code in this tiny subset language, which means we can write a program that reasons about that real Python code.
In fact, with a couple more tricks (curried functions and/or static function types, and lazy evaluation), the compiler/interpreter could do that kind of reasoning on the fly and optimize our code for us. It’s easy to tell programmatically that car(cdr(cons(2, cons(3, None)) is going to return 3 without having to actually evaluate most of those function calls, so we can just skip evaluating them and substitute 3 for the whole expression.
Of course this breaks down if any function can have side effects. You obviously can’t just substitute None for print(3) and get the same results. So instead, you need some clever trick where IO is handled by some magic object that evaluates functions to figure out what it should read and write, and then the whole rest of the program, the part that users write, becomes pure and can be optimized however you want. With a couple more abstractions, we can even make IO something that doesn’t have to be magical to do that.
And then you can build a standard library that gives you back all those things we gave up, written in terms of defining and calling functions, so it’s actually usable—but under the covers it’s all just reducing pure function calls, which is simple enough for a computer to optimize. And then you’ve basically written Haskell.
I have the following code to stop the iterator on a certain value, save the state until the value and return both the saved state and the original state. I am using takewhile from itertools to get the values till the given break_point, and then I use chain on the saved iterator until the break_point and the initial iterator to merge them:
from itertools import takewhile, chain
def iter_break(iterator_input, break_point):
new_iter = list(takewhile(lambda x: x <= break_point-1, iterator_input))
return chain(iter(new_iter), iterator_input)
import unittest
class TestEqual(unittest.TestCase):
def test_iters(self):
it = iter(range(20))
old_it = iter_break(it, 10)
self.assertEqual(list(it), list(old_it))
if __name__ == '__main__':
unittest.main()
The problem is, in the end the returned iterator and the full iterator I am returning are not similar since the returned one misses one value, and it misses the value that is equal to the break point itself. Please help.
it is not just missing the breakpoint value, it's missing all the values before it because it's just an iterator created with iter and not a list, so iter_sample uses up values as it iterates over them with takewhile. This includes the breakpoint itself because takewhile needs to see that value in order to know that the condition is no longer satisfied.
I've written some code to the effect of
def get_stuff(input):
for a, b in itertools.zip_longest(input, input):
# do some processing
yield a, b
which currently is called as get_stuff(sys.stdin) but in the future may change to get_stuff(some_file_I_opened). The reason I use zip_longest is to chunk the reading into pairs (ie: read two lines at a time). I'd like to mock input in my testing but I can't seem to get the mock to play nicely with itertools.zip_longest.
My mocking code right now:
def test_get_rover_information_happy_case(self):
self.input = sys.stdin
self.input = MagicMock()
self.input.__iter__.return_value = ["5 5 N", "LRMR"]
a, b = get_stuff(self.input)
What happens with this mocking code is that get_stuff is called twice. The first time it's called a and b both have the value "5 5 N" and the second time they both have the value LRMR.
How can I mock input such that two values are read at a time?
Just make the return_value an iterator over the list, rather than return the list object itself:
self.input.__iter__.return_value = iter(["5 5 N", "LRMR"])
Otherwise, each time iter(input) is called, the same list object is returned, and any iteration over it will start from index 0. By returning an iterator object, each iteration will pick up where the last one left off, which will simulate sys.stdin the way you want.
I was study iter(), in its official document, it says i can do iter(v,w) , so that iter() will call v until it return the value w, then it stops.
But I tried for half hour, and still can't work out a function that can return multiple result.
Here is my code, I expect it to return 1,2,3,4,5:
def x():
for i in range(10):
return i
a = iter(x, 5)
a.next()
I know that when I return i, that I was actually quit the function.
Maybe it's impossible to return result for multiple times for a function.
But how should I use a function to make that iter(x,5) work properly?
iter() calls the function each time. Your function returns the same value on each call (the first number in the range(10) list).
You could change the function to use a global to illustrate how iter() with two arguments works:
i = 0
def f():
global i
i += 1
return i
for x in iter(f, 5):
print x
Now each time f() is called, a new number is returned. You could use a default argument, or an instance with state and a method on that instance, too. As long as the function returns something different when called more than once it'll fit the iter(a, b) usecase.
iter() with two arguments is most often called with a method, where the state of an instance changes with each call. The .readline() method on a file object, for example:
for line in iter(fileobject.readline, ''):
which would work exactly like iterating over the fileobject iterable directly, except it wouldn't use the internal file iteration buffer. That could sometimes be a requirement (see the file.next() method for more information on the file iteration buffer).
You can of course pass in a lambda function too:
for chunk in iter(lambda: fileobject.read(2048), ''):
Now we are reading the file object is chunks of up to 2048 bytes instead of line by line.
After #Martjin Pieters's answer, I've got the idea.
And this is the piece of code I wrote which can use iter(v,w) correctly:
import random
def x():
return random.randrange(1,10)
a = iter(x,5)
while True:
print a.next()
In this code, a.next() will return the value a get from x(), until x() returns 5, then it will stop.
You could also use a generator function, via the yield keyword. Example:
def x():
for i in range(10):
yield i
What is an idiomatic way to create an infinite iterator from a function? For example
from itertools import islice
import random
rand_characters = to_iterator( random.randint(0,256) )
print ' '.join( islice( rand_characters, 100))
would produce 100 random numbers
You want an iterator which continuously yields values until you stop asking it for new ones? Simply use
it = iter(function, sentinel)
which calls function() for each iteration step until the result == sentinel.
So choose a sentinel which can never be returned by your wanted function, such as None, in your case.
rand_iter = lambda start, end: iter(random.randint(start, end), None)
rand_bytes = rand_iter(0, 256)
If you want to monitor some state on your machine, you could do
iter_mystate = iter(getstate, None)
which, in turn, infinitely calls getstate() for each iteration step.
But beware of functions returning None as a valid value! In this case, you should choose a sentinel which is guaranteed to be unique, maybe an object created for exactly this job:
iter_mystate = iter(getstate, object())
Every time I see iter with 2 arguments, I need to scratch my head an look up the documentation to figure out exactly what is going on. Simply because of that, I would probably roll my own:
def call_forever(callback):
while True:
yield callback()
Or, as stated in the comments by Jon Clements, you could use the itertools.repeatfunc recipe which allows you to pass arguments to the function as well:
import itertools as it
def repeatfunc(func, times=None, *args):
"""
Repeat calls to func with specified arguments.
Example: repeatfunc(random.random)
"""
if times is None:
return it.starmap(func, it.repeat(args))
return it.starmap(func, it.repeat(args, times))
Although I think that the function signature def repeatfunc(func,times=None,*args) is a little awkward. I'd prefer to pass a tuple as args (it seems more explicit to me, and "explicit is better than implicit"):
import itertools as it
def repeatfunc(func, args=(),times=None):
"""
Repeat calls to func with specified arguments.
Example: repeatfunc(random.random)
"""
if times is None:
return it.starmap(func, it.repeat(args))
return it.starmap(func, it.repeat(args, times))
which allows it to be called like:
repeatfunc(func,(arg1,arg2,...,argN),times=4) #repeat 4 times
repeatfunc(func,(arg1,arg2,...)) #repeat infinitely
instead of the vanilla version from itertools:
repeatfunc(func,4,arg1,arg2,...) #repeat 4 times
repeatfunc(func,None,arg1,arg2,...) #repeat infinitely