We Keep Coding

Mocking os.path.exists and os.makedirs returning AssertionError

I have a function like below.
# in retrieve_data.py
import os
def create_output_csv_file_path_and_name(output_folder='outputs') -> str:
"""
Creates an output folder in the project root if it doesn't already exist.
Then returns the path and name of the output CSV file, which will be used
to write the data.
"""
if not os.path.exists(output_folder):
os.makedirs(output_folder)
logging.info(f"New folder created for output file: " f"{output_folder}")
return os.path.join(output_folder, 'results.csv')
I also created a unit test file like below.
# in test_retrieve_data.py
class OutputCSVFilePathAndNameCreationTest(unittest.TestCase):
#patch('path.to.retrieve_data.os.path.exists')
#patch('path.to.retrieve_data.os.makedirs')
def test_create_output_csv_file_path_and_name_calls_exists_and_makedirs_once_when_output_folder_is_not_created_yet(
self,
os_path_exists_mock,
os_makedirs_mock
):
os_path_exists_mock.return_value = False
retrieve_cradle_profile_details.create_output_csv_file_path_and_name()
os_path_exists_mock.assert_called_once()
os_makedirs_mock.assert_called_once()
But when I run the above unit test, I get the following error.
def assert_called_once(self):
"""assert that the mock was called only once.
"""
if not self.call_count == 1:
msg = ("Expected '%s' to have been called once. Called %s times.%s"
% (self._mock_name or 'mock',
self.call_count,
self._calls_repr()))
raise AssertionError(msg)
AssertionError: Expected 'makedirs' to have been called once. Called 0 times.
I tried poking around with pdb.set_trace() in create_output_csv_file_path_and_name method and I'm sure it is receiving a mocked object for os.path.exists(), but the code never go pasts that os.path.exists(output_folder) check (output_folder was already created in the program folder but I do not use it for unit testing purpose and want to keep it alone). What could I possibly be doing wrong here to mock os.path.exists() and os.makedirs()? Thank you in advance for your answers!

You have the arguments to your test function reversed. When you have stacked decorators, like:
#patch("retrieve_data.os.path.exists")
#patch("retrieve_data.os.makedirs")
def test_create_output_csv_file_path_...():
They apply bottom to top, so you need to write:
#patch("retrieve_data.os.path.exists")
#patch("retrieve_data.os.makedirs")
def test_create_output_csv_file_path_and_name_calls_exists_and_makedirs_once_when_output_folder_is_not_created_yet(
self, os_makedirs_mock, os_path_exists_mock
):
With this change, if I have this in retrieve_data.py:
import os
import logging
def create_output_csv_file_path_and_name(output_folder='outputs') -> str:
"""
Creates an output folder in the project root if it doesn't already exist.
Then returns the path and name of the output CSV file, which will be used
to write the data.
"""
if not os.path.exists(output_folder):
os.makedirs(output_folder)
logging.info(f"New folder created for output file: " f"{output_folder}")
return os.path.join(output_folder, 'results.csv')
And this is test_retrieve_data.py:
import unittest
from unittest.mock import patch
import retrieve_data
class OutputCSVFilePathAndNameCreationTest(unittest.TestCase):
#patch("retrieve_data.os.path.exists")
#patch("retrieve_data.os.makedirs")
def test_create_output_csv_file_path_and_name_calls_exists_and_makedirs_once_when_output_folder_is_not_created_yet(
self, os_makedirs_mock, os_path_exists_mock
):
os_path_exists_mock.return_value = False
retrieve_data.create_output_csv_file_path_and_name()
os_path_exists_mock.assert_called_once()
os_makedirs_mock.assert_called_once()
Then the tests run successfully:
$ python -m unittest -v
test_create_output_csv_file_path_and_name_calls_exists_and_makedirs_once_when_output_folder_is_not_created_yet (test_retrieve_data.OutputCSVFilePathAndNameCreationTest.test_create_output_csv_file_path_and_name_calls_exists_and_makedirs_once_when_output_folder_is_not_created_yet) ... ok
----------------------------------------------------------------------
Ran 1 test in 0.001s
OK
Update I wanted to leave a comment on the diagnostics I performed here, because I didn't initially spot the reversed arguments, either, but the problem became immediately apparent when I added a breakpoint() the beginning of the test and printed out the values of the mocks:
(Pdb) p os_path_exists_mock
<MagicMock name='makedirs' id='140113966613456'>
(Pdb) p os_makedirs_mock
<MagicMock name='exists' id='140113966621072'>
The fact that the names were swapped made the underlying problem easy to spot.

Mocking os.environ with python unittests

I am trying to test a class that handles for me the working directory based on a given parameter. To do so, we are using a class variable to map them.
When a specific value is passed, the path is retrieved from the environment variables (See baz in the example below). This is the specific case that I'm trying to test.
I'm using Python 3.8.13 and unittest.
I'm trying to avoid:
I don't want to mock the WorkingDirectory.map dictionary because I want to make sure we are fetching from the environ with that particular variable (BAZ_PATH).
Unless is the only solution, I would like to avoid editing the values during the test, i.e I would prefer not to do something like: os.environ["baz"] = DUMMY_BAZ_PATH
What I've tried
I tried mocking up the environ as a dictionary as suggested in other publications, but I can't make it work for some reason.
# working_directory.py
import os
class WorkingDirectory:
map = {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH"),
}
def __init__(self, env: str):
self.env = env
self.path = self.map[self.env]
#property
def data_dir(self):
return os.path.join(self.path, "data")
# Other similar methods...
Test file:
# test.py
import os
import unittest
from unittest import mock
from working_directory import WorkingDirectory
DUMMY_BAZ_PATH = "path/to/baz"
class TestWorkingDirectory(unittest.TestCase):
#mock.patch.dict(os.environ, {"BAZ_PATH": DUMMY_BAZ_PATH})
def test_controlled_baz(self):
wd = WorkingDirectory("baz")
self.assertEqual(wd.path, DUMMY_BAZ_PATH)
Error
As shown in the error, os.environ doesn't seem to be properly patched as it returns Null.
======================================================================
FAIL: test_controlled_baz (test_directory_structure_utils.TestWorkingDirectory)
----------------------------------------------------------------------
Traceback (most recent call last):
File "~/.pyenv/versions/3.8.13/lib/python3.8/unittest/mock.py", line 1756, in _inner
return f(*args, **kw)
File "~/Projects/dummy_project/tests/unit/test_directory_structure_utils.py", line 127, in test_controlled_baz
self.assertEqual(wd.path, DUMMY_BAZ_PATH)
AssertionError: None != 'path/to/baz'
----------------------------------------------------------------------
Ran 136 tests in 0.325s
FAILED (failures=1, skipped=5)
This seems to be because the BAZ_PATH doesn't exist actually. However, I would expect this to be OK since is being patched.
When, in the mapping dictionary, "baz": os.environ.get("BAZ_PATH"), I repalce BAZ_PATH for a variable that actually exist in my environment, i.e HOME, it returns the actual value of HOME instead of the DUMMY_BAZ_PATH, which lead me to think that I'm definetely doing something wrong patching
AssertionError: '/Users/cestla' != 'path/to/baz'
Expected result
Well, obviously, I am expecting the test_controlled_baz passes succesfully.

So the problem is that you added map as a static variable.
Your patch works correctly as you can see here:
patch actually works
The problem is that when it runs it's already too late because the map variable was already calculated (before the patch).
If you want you can move it to the init function and it will function correctly:
class WorkingDirectory:
def __init__(self, env: str):
self.map = {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH")
}
self.env = env
self.path = self.map[self.env]
If for some reason you wish to keep it static, you have to also patch the object itself.
writing something like this will do the trick:
class TestWorkingDirectory(unittest.TestCase):
#mock.patch.dict(os.environ, {"BAZ_PATH": DUMMY_BAZ_PATH})
def test_controlled_baz(self):
with mock.patch.object(WorkingDirectory, "map", {
"foo": "path/to/foo",
"bar": "path/to/bar",
"baz": os.environ.get("BAZ_PATH")
}):
wd = WorkingDirectory("baz")
self.assertEqual(wd.path, DUMMY_BAZ_PATH)

That's not directly answer to your question but a valid answer either way imo:
Don't try to patch that (it's possible, but harder and cumbersome).
Use config file for your project.
e.g. use pyproject.toml and inside configure the pytest extension:
[tool.pytest.ini_options]
env=[
"SOME_VAR_FOR_TESTS=some_value_for_that_var"
]

Generic Function Calls With Keyword Arguments

Working in Python 3.8.
I prefer to have one return per function as it makes debugging easier.
Current code which I can live with:
if foo:
return redirect(url_for('what.what'))
else:
return render_template('who.who', form=my_form)
Desired code though:
if foo:
ret_func = redirect
ref_func_args = url_for("what.what")
else:
ret_func = render_template
ret_func_args = ('who.html', form=my_form) # syntax help needed here
return ret_func(ret_func_args) # and probably some syntax help here also
Python understandably doesn't like the ret_func_args = ('who.html', form=my_form) and in particular form=my_form.
render_template is expecting a keyword argument named 'form'.
What is the proper syntax for ret_func_args = ('who.html', form=my_form) and perhaps the ret_func(ret_func_args) line? The ret_func(ret_func_args) does work if foo is true but can not figure out how to pass named parameters when foo is false in this case.

Python mock: AssertionError: Expected and actual call not same

I am new to unittest.mock library and unable to solve the issue I am experiencing.
I have a class called ‘function.py’ in the below folder structure
src
_ init.py
function.py
tests
init.py
test_function.py
In test_function.py I have some code like this:
import unittest
from unittest import mock
from ..src.function import get_subscriptions
from ..src import function
class TestCheckOrder(unittest.TestCase):
#mock.patch.object(function, 'table')
def test_get_subscriptions_success(self, mocked_table):
mocked_table.query.return_value = []
user_id = "test_user"
status = True
get_subscriptions(user_id, status)
mocked_table.query.assert_called_with(
KeyConditionExpression=conditions.Key('user_id').eq(user_id),
FilterExpression=conditions.Attr('status').eq(int(status)))
In function.py:
import boto3
from boto3.dynamodb import conditions
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table("Subscriptions")
def get_subscriptions(user_id, active=True):
results = table.query(
KeyConditionExpression=conditions.Key(
'user_id').eq(user_id),
FilterExpression=conditions.Attr('status').eq(int(active))
)
return results['Items']
If I run this I get the following exception:
**AssertionError: Expected call: query(FilterExpression=<boto3.dynamodb.conditions.Equals object at 0x1116011d0>, KeyConditionExpression=<boto3.dynamodb.conditions.Equals object at 0x111601160>)
Actual call: query(FilterExpression=<boto3.dynamodb.conditions.Equals object at 0x1116010f0>, KeyConditionExpression=<boto3.dynamodb.conditions.Equals object at 0x111601080>)**
Thanks in advance for helping me out.

The issue is that when you're calling assert_called_with in your test, you're creating new instances of conditions.Key and conditions.Attr. And as these instances are different from one we had in the actual call, there's a mismatch(check the hex ids shown in the traceback).
Instead of this you can fetch the kwargs from the function call itself and test their properties:
name, args, kwargs = mocked_table.query.mock_calls[0]
assert kwargs['KeyConditionExpression'].get_expression()['values'][1] == user_id
assert kwargs['FilterExpression'].get_expression()['values'][1] == int(status)

Python Mock not correctly setting return value

I am attempting to build unit tests and have been using mock, However upon using two patch statements, I was not able to set the proper return values.
#patch('pulleffect.lib.google.gcal_helper.validate_and_refresh_creds')
#patch('pulleffect.lib.google.gcal_helper.get_google_creds')
def test_get_calendar_list_for_gcalhelper_without_credentials(self,
mock_get_google_creds,
mock_validate_and_refresh_creds):
mock_validate_and_refresh_creds = "redirect"
mock_get_google_creds = "credentials"
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
however the assert fails and instead of the expected string redirect I instead get
<MagicMock name = "validate_and_refresh_creds() id = 14054613955344>
I was wondering what is necessary to have redirect returned instead. I have not encountered this issue when only patching a single method.

I was able to fix the issue of
<MagicMock name = "foo()" id = number>
incorrectly appearing by replacing my earlier code with:
from mock import MagicMock
def test_get_calendar_list_for_gcalhelper_without_credentials(self):
rtn = { "redirect": "/gcal/authenticate"}
pulleffect.lib.google.gcal_helper.validate_and_refresh_creds = MagicMock(name = "sup", return_value = rtn)
pulleffect.lib.google.gcal_helper.get_google_creds = MagicMock(name = "sup2", return_value = "redirect")
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
this allowed the return values to be properly set.

mock_get_google_creds and mock_validate_and_refresh_creds created with patch decorator are ordinary mock objects (Mock or MagicMock). Direct assignment is not the correct way to set return values. Use return_value attribute:
mock_validate_and_refresh_creds.return_value = "redirect"
Also you can set it during patching:
patch takes arbitrary keyword arguments. These will be passed to the
Mock (or new_callable) on construction.
#patch('pulleffect.lib.google.gcal_helper.get_google_creds', return_value="redirect")

I recommend you to use this solution. You should move your functions to helper class and instead static methods user class methods, because it's possible to mock class in this way.
class GCallHelper(object):
#classmethond
def validate_and_refresh(cls):
...
return result
def test_get_calendar_list_for_gcalhelper_without_credentials(self):
with patch('pulleffect.lib.google.gcal_helper') as mocked_gcal:
mocked_gcal.return_value.validate_and_refresh_creds.return_value = 'redirect'
mocked_gcal.return_value.get_google_creds.return_value = 'credentials'
credentials = pulleffect.lib.google.gcal_helper.get_calendar_list("name","widget")
assert b'redirect' in credentials
p.s. And you forgot 'return_value' in your example.