Background
I want to test my code which depends on random module.
The problematic PR is https://github.com/Axelrod-Python/Axelrod/pull/202 and code is here https://github.com/Axelrod-Python/Axelrod/blob/master/axelrod/strategies/qlearner.py
The problem
Since random module produces pseudo-random numbers, I always set random.seed(X) to known value X. This works for consecutive test runs. However, Python 3 seems to give different numbers than Python 2 when using random.choice([D, C])
Following snippet:
import random
random.seed(1)
for i in range(10):
print(random.choice(['C', 'D']), end=', ')
gives different result for Python 2 and 3
$ python2 test.py
C, D, D, C, C, C, D, D, C, C
$ python3 test.py
C, C, D, C, D, D, D, D, C, C
However, random.random method works the same on 2.x and 3.x:
import random
random.seed(1)
for i in range(10):
print(random.random())
$ python3 test.py
0.13436424411240122
0.8474337369372327
0.763774618976614
0.2550690257394217
0.49543508709194095
0.4494910647887381
0.651592972722763
0.7887233511355132
0.0938595867742349
0.02834747652200631
$ python2 test.py
0.134364244112
0.847433736937
0.763774618977
0.255069025739
0.495435087092
0.449491064789
0.651592972723
0.788723351136
0.0938595867742
0.028347476522
Workaround
I can mock the output of random.choice, which works well for simple test cases. However, for fairly complicated test cases, I'm not able to mock output, because I simply don't know how it should look like.
The question
Have I done something wrong when calling random.choice method?
There is a completely different implementation of random.choice in each version.
Python 2.7:
def choice(self, seq):
"""Choose a random element from a non-empty sequence."""
return seq[int(self.random() * len(seq))] # raises IndexError if seq is empty
https://hg.python.org/cpython/file/2.7/Lib/random.py
Python 3.4:
def choice(self, seq):
"""Choose a random element from a non-empty sequence."""
try:
i = self._randbelow(len(seq))
except ValueError:
raise IndexError('Cannot choose from an empty sequence')
return seq[i]
https://hg.python.org/cpython/file/3.4/Lib/random.py
The _randbelow method may call random() more than once, or may call getrandbits which has a different underlying call to _urandom.
According to https://docs.python.org/2/library/random.html, the RNG was changed in Python 2.4 and may use operating system resources. Based on this and the other answer to this question, it's not reasonable to expect Random to give the same result on two different versions of Python, two different operating systems, or even two different computers. For all any of us knows, the next version of Python could implement a Random function that uses the system's microphone to generate a random sequence.
Short version: you should never depend on a RNG to give a deterministic result. If you need a known sequence to satisfy a unit test, you need to either redesign your method or your unit test.
One way you might do this is to split your method into two parts: one part generates the random number. The second part consumes the value and acts on it. You would then write two unit tests: one to test coverage of the generated values, and a separate one to test the output of your method based on specific inputs.
Another way might be to change your method to output not just the result but the random number that created that result. You can modify your unit test to compare the two and pass or fail the test based on the expected output of known pairs.
Or perhaps your unit test can be modified to simply run the test n number of times and look for a spread that confirms some sort of randomness.
I've got the exact same problem and I'm disappointed at the number of responses that point to error on your part when, seeding the random function is expected to produce reliably consistent results across versions of Python, machine and operating system.
What seems to work is, painfully, to have your own random class and override the relevant methods with the logic from Python 2.7.
from random import Random
class MyRandom(Random):
def sample(self, population, k):
(code from Python 2.7.6 random module updated for Python 3 syntax)
def choice...
my_random = MyRandom(0)
my_random.sample(['Apples', 'Bananas', 'Carrots'])
The randomness functions themselves are different so having the same seed result in the same numbers doesn't fix many random functions that refuse to return the same results. While there are reasons for the newer random functions, those reasons are moot on existing code bases already dependant on the older functions.
Anyway, I hope this can help anyone else fighting this issue.
Related
I am using two different methods of trying to generate a bootstrap sample
np.random.seed(335)
y=np.random.normal(0,1,5)
b=np.empty(len(y)) #initializes an empty vector
for j in range(len(y)):
a = np.random.randint(1,len(y)) #Draws a random integer from 1 to n, where n is our sample size
b[j] = y[a-1] #indicies in python start at zero, the worst part of Python in my opinion
c = np.random.choice(y, size=5)
print(b)
print(c)
and for my output I get different results
[1.04749432 1.71963433 1.71963433 1.71963433 1.71963433]
[-0.25224454 -0.25224454 0.46604474 1.71963433 0.46604474]
I think the answer has something to do with the random number generator, but I'm confused as to the exact reason.
This comes down to the use of different algorithms for randomized selection. There are numerous equivalent ways to select items at random with replacement using a pseudorandom generator (or to generate random variates from any other distribution). In particular, the algorithm for numpy.random.choice need not make use of numpy.random.randint in theory. What matters is that these equivalent ways should produce the same distribution of random variates. In the case of NumPy, look at NumPy's source code.
Another, less important, reason for different results is that the two different selection procedures (randint and choice) produce pseudorandom numbers themselves, which can differ from each other because the selection procedures didn't begin with the same seed (more precisely, the same sequence of pseudorandom numbers). If we set the seed to the same value before beginning each procedure:
np.random.seed(335)
y=np.random.normal(0,1,5)
b=np.empty(len(y))
np.random.seed(999999) # Seed selection procedure 1
for j in range(len(y)):
a = np.random.randint(1,len(y))
b[j] = y[a-1]
np.random.seed(999999) # Seed selection procedure 2
c = np.random.choice(y, size=5)
print(b)
print(c)
then each procedure will begin with the same pseudorandom numbers. But even so, the two procedures may use different algorithms for random selection, and these differences may still lead to different results.
(However, numpy.random.* functions, such as randint and choice, have become legacy functions as of NumPy 1.17, and their algorithms are expected to remain as they are for backward compatibility reasons. That version didn't deprecate any numpy.random.* functions, however, so they are still available for the time being. See also this question. In newer applications you should make use of the new system introduced in version 1.17, including numpy.random.Generator, if you have that version or later. One advantage of the new system is that the application relies less on global state.)
I am designing python assignments for a class. I define functions, write docstrings and then I implement them. Afterward, I'd like to remove all my implementations of the functions and replace only the code (not the doc-strings, function names, and arguments) with a raise NotImplementedError.
Is there any tool (e.g. IDE) which removes all the code for me automatically, so that I don't have to replace the implemented function by myself? I was thinking about writing a small script, but I thought I might ask here before I do this ...
If anyone has written something similar or knows of a quick way how to this, I would appreciate this a lot.
Here's a minimal example of what I'd like to achieve:
test.py
def add(a,b):
"""
Adds two numbers
"""
return a+b
def multiply(a,b):
"""
Multiplies two numbers
"""
return a*b
should become in an automated fashion (and of course for much larger files):
test.py
def add(a,b):
"""
Adds two numbers
"""
raise NotImplementedEror
def multiply(a,b):
"""
Multiplies two numbers
"""
raise NotImplementedEror
I don't know of a tool to do specifically this, but Python provides great AST manipulation tools within its own standard library via the ast module. You'll need a third party module to "unparse" the result after transformation back into regular Python code, and after a quick search I found this one seems to do the trick, although there do seem to be many others.
Here's a bit of sample code to get you in the right direction. Obviously, you'll need to tweak this to get the exact behavior you want, especially if you want to provide classes instead of just top-level functions (as I've written nothing to handle that use case). But with a bit of Python knowledge, it should be possible to automate.
Also, this is Python 3 (which, as of the start of 2020, is the only supported Python version). If you're still on Python 2, it may require some modifications.
import ast
import astunparse
# Read our file using the built-in Python AST module.
with open('filename.py') as f:
data = ast.parse(f.read(), 'filename.py')
# Loop through all declarations.
for decl in data.body:
# Only modify functions
if isinstance(decl, ast.FunctionDef):
# The docstring is the first statement of the body. So we don't
# want to change it. Instead, replace the rest of the body with
# our pre-built "raise" call. Note that I figured out what "raise"
# looked like in AST form by running
#
# ast.dump(ast.parse("raise NotImplementedError()"))
#
decl.body[1:] = [ast.Raise(ast.Call(ast.Name('NotImplementedError'), [], []), None)]
# Use astunparse to pretty print the result as Python code.
print(astunparse.unparse(data))
It's definitely possible to automate, if you're willing to take the time to do it. If you're planning to do this for several assignments, or even over several semesters, you may consider making a script for it. But if you're just doing it once, it may be more worth your time to just do it by hand.
I am looking for a Python function that will mimic the behavior of rand() (and srand()) in c with the following requirements:
I can provide the same epoch time into the Python equivalent of srand() to seed the function
The equivalent of rand()%256 should result in the same char value as in c if both were provided the same seed.
So far, I have considered both the random library and numpy's random library. Instead of providing a random number from 0 to 32767 as C does though both yield a floating point number from 0 to 1 on their random functions. When attempting random.randint(0,32767), I yielded different results than when in my C function.
TL;DR
Is there an existing function/libary in Python that follows the same random sequence as C?
You can accomplish this with CDLL from https://docs.python.org/3/library/ctypes.html
from ctypes import CDLL
libc = CDLL("libc.so.6")
libc.srand(42)
print(libc.rand() % 32768)
You can't make a Python version of rand and srand functions "follo[w] the same random sequence" of C's rand and srand because the C standard doesn't specify exactly what that sequence is, even if the seed is given. Notably:
rand uses an unspecified pseudorandom number algorithm, and that algorithm can differ between C implementations, including versions of the same standard library.
rand returns values no greater than RAND_MAX, and RAND_MAX can differ between C implementations.
In general, the best way to "sync" PRNGs between two programs in different languages is to implement the same PRNG in both languages. This may be viable assuming you're not using pseudorandom numbers for information security purposes. See also this question: How to sync a PRNG between C#/Unity and Python?
You can use random.seed(). The following example should print the same sequence every time it runs:
import random
random.seed(42)
for _ in range(10):
print(random.randint(0, 32768))
Update: Just saw the last comment by the OP. No, this code won't give you the same sequence as the C code, because of reasons given in other comments. Two different C implementations won't agree either.
Scenario
Let test be the module we run as __main__. This module contains one global variable named primes, which is initialized in the module with the following assignment.
primes = []
The module also contains a function named pi, which alters this global variable:
def pi(n):
global primes
"""Some code that modifies the global 'primes' variable"""
I then want to time said function using the builtin timeit module. I want to use the timeit.repeat function and get the minimum value of the timing, as a way of improving the measurement's accuracy (instead of measuring just one time, which may be subject to slow-down due to unrelated processes).
print(min(timeit.repeat('test.pi(50000)',
setup="import test",
number=1, repeat=10)) * 1000)
The problem is that the pi function behaves differently depending on the value of primes: I expected that, for each repetition, the import test statement in the setup parameter would re-run the primes = [] statement in the test, thus 'resetting' primes so that the code being executed would be identical for each repetition. But, instead, the value of primes that resulted from the previous execution is used, so I had to add the statement test.primes = [] to the setup parameter:
print(min(timeit.repeat('test.pi(50000)',
setup="import test \n" + "test.primes = []",
number=1, repeat=10)) * 1000)
Question
This leads me to the question: is there a direct way (i.e. in one statement) to 'reset' the values of all the global variables to what they were when they were first assigned in the module?
In this specific scenario adding that one statement to manually 'reset' primes works fine, but consider a case in which there are a lot of global variables, and you want to 'reset' all of them.
Side quest-ion
Why doesn't the statement import test re-run the initial primes = [] assignment?
Let's start with your side question, because it turns out that it's actually central to everything:
Why doesn't the statement import test re-run the initial primes = [] assignment?"
Because, as explained in the docs on the import system and the import statement, what import test does is, loosely, this pseudocode:
if 'test' not in sys.modules:
find, load (compiling if needed), and exec the module
sys.modules['test'] = result
test = sys['test.modules']
OK, but why does it do that?
If you have two modules that both import the same module, they expect to see the same globals. And remember that types, functions, etc. defined at the top level of a function are all globals. For example, if sortedlist.py imports collections.abc to class SortedList(collections.abc.Sequence):, and scraper.py imports collections.abc to isinstance(something, collections.abc.Sequence), you'd want a SortedList to pass that test—but it won't if those are two completely independent types because they came from two different module objects that happen to have the same name,
If you have 12 modules that all import pandas as pd, you'd be running all the Pandas initialization code 12 times. Except that some of your modules also probably import each other, so they'd each be run multiple times, and import Pandas each time. How long do you think it would take to run all the Pandas initialization 60 times?
So, reusing existing modules is almost always what you want.
And when you don't, that's usually a sign that there's something wrong with your design (which may well be the case here).
But "almost always" isn't "always". So there are ways around it. None of them are usually a good idea for live code, but for things like unit tests and benchmarking, there are three basic options that are all fine, as long as the tradeoffs are the ones you want:
del sys.modules['test']. This is obviously pretty hacky, but it actually does exactly what you want here. Any existing references to the old module are completely untouched, but the next time anyone does import test, they're going to get a brand-new test module.
importlib.reload(test). This sounds great, but it may on the one hand be overkill (notice that it forces the module source to be recompiled, which you don't need), while on the other it may not be sufficient (it doesn't actually reset the globals—if your code does primes = [] at the top level, that line gets executed, so who cares, but if your code instead does, say, globals().setdefault('primes', []) inside the pi function, you care).
Instead of import test, manually do all the steps up through executing the module (see the examples in the importlib docs), but don't store it in sys.modules['test'] or in test, just store it in a local variable you discard after each test. This is probably the cleanest, although it does mean 6 lines of code instead of 1.
I'm still pretty new to programming and just learning how to unittest. I need to test a function that returns a random value. I've so far found answers suggesting the use of a specific seed value so that the 'random' sequence is constant and can be compared. This is what I've got so far:
This is the function I want to test:
import random
def roll():
'''Returns a random number in the range 1 to 6, inclusive.'''
return random.randint(1, 6)
And this is my unittest:
class Tests(unittest.TestCase):
def test_random_roll(self):
random.seed(900)
seq = random.randint(1, 6)
self.assertEqual(roll(), seq)
How do I set the corresponding seed value for the PRNG in the function so that it can be tested without writing it into the function itself? Or is this completely the wrong way to go about testing a random number generator?
Thanks
The other answers are correct as far as they go. Here I'm answering the deeper question of how to test a random number generator:
Your provided function is not really a random number generator, as its entire implementation depends on a provided random number generator. In other words, you are trusting that Python provides you with a sensible random generator. For most purposes, this is a good thing to do. If you are writing cryptographic primitives, you might want to do something else, and at that point you would want some really robust test strategies (but they will never be enough).
Testing a function returns a specific sequence of numbers tells you virtually nothing about the correctness of your function in terms of "producing random numbers". A predefined sequence of numbers is the opposite of a random sequence.
So, what do you actually want to test? For 'roll' function, I think you'd like to test:
That given 'enough' rolls it produces all the numbers between 1 and 6, preferably in 'approximately' equal proportions.
That it doesn't produce anything else.
The problem with 1. is that your function is defined to be a random sequence, so there is always a non-zero chance that any hard limits you put in to define 'enough' or 'approximately equal' will occasionally fail. You could do some calculations to pick some limits that would make sure your test is unlikely to fail more than e.g. 1 in a billion times, or you could slap a random.seed() call that will mean it will never fail if it passes once (unless the underlying implementation from Python changes).
Item 2. could be 'tested' more easily - generate some large 'N' number of items, check that all are within expected outcome.
For all of this, however, I'd ask what value the unit tests actually are. You literally cannot write a test to check whether something is 'random' or not. To see whether the function has a reasonable source of randomness and uses it correctly, tests are useless - you have to inspect the code. Once you have done that, it's clear that your function is correct (providing Python provides a decent random number generator).
In short, this is one of those cases where unit tests provide extremely little value. I would probably just write one test (item 2 above), and leave it at that.
By seeding the prng with a known seed, you know which sequence it will produce, so you can test for this sequence:
class Tests(unittest.TestCase):
def test_random_roll(self):
random.seed(900)
self.assertEqual(roll(), 6)
self.assertEqual(roll(), 2)
self.assertEqual(roll(), 5)