We Keep Coding

How to mock a Python method that is called more than once inside a loop using Pytest

Let's say I have a Python method:
def good_method(self) -> None:
txt = "Some text"
response = self.bad_method(txt)
resources = response["resources"]
print (resources)
while resources:
response = self.bad_method(txt)
resources = response["resources"]
print (resources)
Now let's say I want to write a unit test for it. The bad_method() returns a dictionary and it could get called over and over in the while loop. I have been trying to Mock the bad_method() so it returns a nested dictionary, so the while loop runs once. This is the code:
from unittest.mock import MagicMock
def test_good_method():
dic = {"resources": {"resources": "values"}}
def side_effect():
return dic
self.bad_method() = MagicMock(side_effect=side_effect())
self.good_method()
I expected to first get a {"resources": "values"} printed out, and then a values. But the only thing I get is a resources. What am I doing wrong? How can I achieve what I expect?

def test_good_method():
thing = MyClassContainingGoodAndBadMethod()
thing.bad_method = MagicMock(return_value={"a": "b"})
thing.good_method()
assert thing.bad_method.call_count == 10 # or however many times it is supposed to be called

How to call functions and variables from a dictionary or a json file

I am trying to print a string, call functions and variables in a line.
such as [Hello! %(job), %(name)s, (function_name)]->[Hello! student, John, This is the function.]
json01.json
{
"test" : "Hello! %(job), %(name)s, (function_name)"
}
test01.py
import json
a = 'test'
name = 'John'
job = 'student'
def function_name(message):
print(message)
with open('json01.json') as json_file:
json_dict = json.load(json_file)
if a in json_dict:
print(json_dict[a] %locals())
#This works if there's only variables in the value
#but I don't know how to call functions when the value is not only function's name but also variables..
Is there any simple way to print whatever they are in the value?
or is there another way to do this work?
Sorry for the poor explanation and Thank you!

You can define your own function replacer using regular expressions. I defined an example syntax here as: Hello, !(function_name) where function_name is the name of the function that is called.
Using regular expressions we find all occurences of a function call and try to
evaluate them one by one. If successfull, we replace the function's name with the return value.
import re
def runfunctions(string):
# find all functions defined with our syntax
funcs = re.findall(r'!\((.*?)\)', string)
result = string
# iterate through found functions
for func in funcs:
try:
# try to evaluate with globals()[func]() and replace function call
# with return value
result = re.sub(r'!\(' + func + r'\)', globals()[func](), result)
except (KeyError, TypeError) as e:
# if func is not callable or does not exist catch error
print("Error while evaluating functions in string:", e)
# return final result
return result
Note: I used globals instead of locals as otherwise the function is not found.
You can use it like this:
if a in json_dict:
replaced_vars = json_dict[a] % locals()
replaced_funcs = runfunctions(replaced_vars)
print(replaced_funcs)

Python function asserting value over pre declared value

I have a function (func.py). Structure of which look like this:
database = 'VENUS'
def first_function():
print("do some thing")
def second_function():
print("call third function)
third_function()
def third_function(db = database):
print("do some other thing")
I need to import this function and used the inner defined function. But, I want to use a different key for database. Basically, I want to overwrite database = 'VENUS' and use database = 'MARS' while second function call the third function. is there any way to do this?

Just provide the database name as argument
first_function("MARS")
second_function("MARS")

So the problem here, if I understood correctly, is that the default argument for func.third_function is defined at import time. It doesn't matter if you later modify the func.database variable, since the change will not reflect on the default argument of func.third_function.
One (admittedly hacky) solution is to inject a variable using a closure over the imported function. Example:
file.py:
x = 1
def print_x(xvalue = x)
print(xvalue)
Python console:
>>> import file
>>> file.print_x()
1
>>> file.x = 10
>>> file.print_x() # does not work (as you're probably aware)
1
>>> def inject_var(func_to_inject, var):
def f(*args, **kwargs):
return func_to_inject(var, *args, **kwargs)
return f
>>> file.print_x = inject_var(file.print_x, 10)
>>> file.print_x() # works
10
So using the inject_var as written above, you could probably do:
func.third_function = inject_var(func.third_function, "MARS")

Python best module import practise

I have a python module mymodule.py:
def auth():
'''Authorize and generate a JSON file'''
return j
j = auth()
def get_value(key):
'''Takes the key and return value from JSON'''
value = j[key]
return value
I have a program where I use this module myprogram.py:
import mymodule
keys = [1,2,3,4,5]
def simple_program(keys):
# mymodule.auth() should I place it here?
for key in keys:
value = mymodule.get_value(key)
return value
So the goal is to call mymodule.auth() once, every time I run simple_program to refresh the JSON file. I don't know how to achieve this. Because myprogram.py is also a module and I call simple_program() from another .py file. So where do I place mymodule.auth()? Is it ok to place mymodule.auth() inside simple_program?

The instant you import mymodule the code below runs
j = auth()
which is why when you call mymodule.get_value() it works. This causes J to be a singleton in the global space. Everytime you import this, auth() will run again. This could be bad.
What you could do is this:
def auth():
'''Authorize and generate a JSON file'''
return j
j = None
def get_value(key):
global j
'''Takes the key and return value from JSON'''
if not j:
j = auth()
value = j[key]
return value
Now you just need to run get_value() and everything should work fine. No need to execute auth() again.

Your exact use case is a little vague (e.g. simple_program is not the main program but smth like a subroutine? and it is called several times from another py file?), but it seems to me like you should get familiar with classes. I would suggest to implement auth() as a class, e.g. like this:
class MyJson(object):
def __init__(self):
self._json = ... # do authorization and generation here and save the result as member
def get_value(self, key):
value = self._json[key]
return value
Now import and create an object of that class wherever you need it for the first time
from mymodule import MyJson
# ...
my_json = MyJson()
If you only need it to be initialized once, do that in your main program and pass the my_json object as parameter to simple_program (which should possibly also be a class). And then use it like
value = my_json.get_value(key)

python: generating methods for a convenience class programatically

So I've written a module that contains a bunch of functions to easily interact with a subprocess. This subprocess has a whole bunch of settings that let you change how it formats and behaves. I realized that it'd be nice to have a convenience class that you could use as a handler to store the settings you prefer to use and pass them on to the module level functions. here's the example code I'm doing testing with:
import inspect
class MyHandler(object):
def __init__(self):
self.format_string='class format string'
self.database='class database'
self.mode = "class mode"
def rename(self, *args, **kwargs):
self._pass_to_function(rename, *args, **kwargs)
def _pass_to_function(self, function, *overrided_args, **overrided_kwargs):
# get the function's remaining arguments with the inspect module
functon_kwargs = inspect.getargspec(function)[0][len(overrided_args):]
handler_vars = vars(self)
kwargs_to_pass = {}
for arg in functon_kwargs:
if arg in handler_vars:
kwargs_to_pass[arg] = handler_vars[arg]
for arg in overrided_kwargs:
kwargs_to_pass[arg] = overrided_kwargs[arg]
return function(*overrided_args, **kwargs_to_pass)
def rename(targets, format_string=None, database=None, mode=None,
not_in_class='None'):
print 'targets = {}'.format(targets)
print 'format_string = {}'.format(format_string)
print 'database = {}'.format(database)
print 'mode = {}'.format(mode)
print 'not_in_class = {}\n'.format(not_in_class)
return
The thing I like about this solution is that it uses the attributes stored in the class, but you can easily override them by simply adding them to the method call if you want a one-off with a different setting. To do this I have the _pass_to_function as a kind of wrapper function to parse and fill in the needed settings and overrides. Here's how it looks:
>>> import argstest
>>> argstest.rename('some_file.avi', database='some database')
targets = some_file.avi
format_string = None
database = some database
mode = None
not_in_class = None
>>> tst = argstest.MyHandler()
>>> tst.rename('some_file.avi')
targets = some_file.avi
format_string = class format string
database = class database
mode = class mode
not_in_class = None
>>> tst.rename('some_file.avi', 'one off format string', not_in_class=True)
targets = some_file.avi
format_string = one off format string
database = class database
mode = class mode
not_in_class = True
Now in my real module I have dozens of module-level functions that I want to access from the handler class. Ideally they would generate automatically based on the functions in the module. Seeing as how all the methods are only going to be passing everything to _pass_to_function I get the sense that this shouldn't be very difficult but I'm having a lot of trouble figuring out exactly how.
I've read about using type to generate a meta-class, but I don't see how I would use it in this situation. Am I not seeing how I could use type? Should I use some sort of module level script that adds the functions with setattr? Is what I was doing the better/clearer way to do things?
Any and all advice would be appreciated.

Okay, I think I've answered my own question for now. This is how the module looks:
import inspect
import sys
from types import MethodType
class MyHandler(object):
def __init__(self):
self.format_string = 'class format string'
self.database = 'class database'
self.mode = "class mode"
self._populate_methods()
def _populate_methods(self):
to_add = inspect.getmembers(sys.modules[__name__], inspect.isfunction)
to_add = [x[0] for x in to_add if not x[0].startswith('_')]
for func_name in to_add:
func = getattr(sys.modules[__name__], func_name) # strings to functions
self._add_function_as_method(func_name, func)
def _add_function_as_method(self, func_name, func):
def f(self, *args, **kwargs): # the template for the method we'll add
return self._pass_to_function(func, *args, **kwargs)
setattr(MyHandler, func_name, MethodType(f, None, MyHandler))
def _pass_to_function(self, function, *overrided_args, **overrided_kwargs):
functon_kwargs = inspect.getargspec(function)[0][len(overrided_args):]
handler_vars = vars(self)
kwargs_to_pass = {}
for arg in functon_kwargs:
if arg in handler_vars:
kwargs_to_pass[arg] = handler_vars[arg]
for arg in overrided_kwargs:
kwargs_to_pass[arg] = overrided_kwargs[arg]
return function(*overrided_args, **kwargs_to_pass)
def rename(targets, format_string=None, database=None, mode=None,
not_in_class='None'):
print 'targets = {}'.format(targets)
print 'format_string = {}'.format(format_string)
print 'database = {}'.format(database)
print 'mode = {}'.format(mode)
print 'not_in_class = {}\n'.format(not_in_class)
return
def something_else():
print "this function should become a method"
def _not_a_member():
print "this function should not become a method"
I've added the _populate_methods and the _add_function_as_method member functions. the _populate_methods function gets the name of all "public" functions in the module, de-references them to their function and passes each one though _add_function_as_method. All this method does is use an internal function to capture arguments and sent them to _pass_to_function, and set that function as a method using setattr.
phew
so it works, but I'm still wondering if there isn't a clearer or more straight forward way to get this done. I'd be very grateful if anyone could chime in.