first of all, sorry if this has been answered before but I have been looking all over the place and could not find anything.
I have a function that does this (in summary):
def func(path):
with open(path) as f:
json.load(f)
I want to unit test it and what I would like is that when I use open as context manager, f is the name of the argument I passed to open. The issue I am having is that I cannot mock the enter method in the proper module, it breaks everytime. This is whay I am doing so far:
def setUp(self):
# Setup the content of the config files for the tests
json.load = MagicMock(side_effect=file_content)
# Opening a file returns the name of the file
builtins.open = MagicMock(side_effect=lambda x: x) #1
builtins.open.__enter__ = MagicMock(side_effect=builtins.open.return_value) #2
the MagicMock number 1 does what I expect it to do, when I run open("test") it returns "test". Nonetheless I cannot mock enter to return whatever I passed to open, it always fails with AttributeError: __enter__. I have tried also doing
builtins.open.return_value.__enter__ = MagicMock...
with no luck. Can anyone think on a way that this can be achieved? I have seen how to mock magic methods and I have thought I was comfortable with unittest mocking but in this case I cannot find a fitting solution. Thanks!
I got it!
I had to define the side_effect dinamically but seems to be working fine. This is what I did:
def setUp(self):
# Setup the content of the config files for the tests
json.load = MagicMock(side_effect=file_content)
# Opening a file returns the name of the file
def get_mock_context(filename):
mock_context = MagicMock()
mock_context.__enter__.return_value = filename
mock_context.__exit__.return_value = False
return mock_context
builtins.open = MagicMock(side_effect=get_mock_context)
In this way, the returned value is a mock that has the __enter__ and __exit__ methods returning exactly the filename I passed to the call of open.
Related
I am trying to figure out how to know if a method of class is being called inside a method.
following is the code for the unit test:
# test_unittes.py file
def test_purge_s3_files(mocker):
args = Args()
mock_s3fs = mocker.patch('s3fs.S3FileSystem')
segment_obj = segments.Segmentation()
segment_obj.purge_s3_files('sample')
mock_s3fs.bulk_delete.assert_called()
inside the purge_s3_file method bulk_delete is called but when asserting it says that the method was expected to be called and it is not called!
mocker = <pytest_mock.plugin.MockerFixture object at 0x7fac28d57208>
def test_purge_s3_files(mocker):
args = Args()
mock_s3fs = mocker.patch('s3fs.S3FileSystem')
segment_obj = segments.Segmentation(environment='qa',
verbose=True,
args=args)
segment_obj.purge_s3_files('sample')
> mock_s3fs.bulk_delete.assert_called()
E AssertionError: Expected 'bulk_delete' to have been called.
I don't know how to test this and how to assert if the method is called!
Below you can find the method being testing:
# segments.py file
import s3fs
def purge_s3_files(self, prefix=None):
bucket = 'sample_bucket'
files = []
fs = s3fs.S3FileSystem()
if fs.exists(f'{bucket}/{prefix}'):
files.extend(fs.ls(f'{bucket}/{prefix}'))
else:
print(f'Directory {bucket}/{prefix} does not exist in s3.')
print(f'Purging S3 files from {bucket}/{prefix}.')
print(*files, sep='\n')
fs.bulk_delete(files)
The problem you are facing is that the mock you are setting up is mocking out the class, and you are not using the instance to use and check your mocks. In short, this should fix your problem (there might be another issue explained further below):
m = mocker.patch('s3fs.S3FileSystem')
mock_s3fs = m.return_value # (or mock_s3())
There might be a second problem in how you are not referencing the right path to what you want to mock.
Depending on what your project root is considered (considering your comment here) your mock would need to be referenced accordingly:
mock('app.segments.s3fs.S3FileSystem')
The rule of thumb is that you always want to mock where you are testing.
If you are able to use your debugger (or output to your console) you will (hopefully :)) see that your expected call count will be inside the return_value of your mock object. Here is a snippet from my debugger using your code:
You will see the call_count attribute set to 1. Pointing back to what I mentioned at the beginning of the answer, by making that change, you will now be able to use the intended mock_s3fs.bulk_delete_assert_called().
Putting it together, your working test with modification runs as expected (note, you should also set up the expected behaviour and assert the other fs methods you are calling in there):
def test_purge_s3_files(mocker):
m = mocker.patch("app.segments.s3fs.S3FileSystem")
mock_s3fs = m.return_value # (or m())
segment_obj = segments.Segmentation(environment='qa',
verbose=True,
args=args)
segment_obj.purge_s3_files('sample')
mock_s3fs.bulk_delete.assert_called()
Python mock testing depends on where the mock is being used. So you have the mock the function calls where it is imported.
Eg.
app/r_executor.py
def r_execute(file):
# do something
But the actual function call happens in another namespace ->
analyse/news.py
from app.r_executor import r_execute
def analyse():
r_execute(file)
To mock this I should use
mocker.patch('analyse.news.r_execute')
# not mocker.patch('app.r_executor.r_execute')
How can I mock the path (".test/locations.yml), because it does not exist in this project where I run my test. It exists in the CI environment.
I test my function get_matches_mr and then it says path location file not found
Do you have any idea?
Code
def read_location_file():
locations_file_path = os.path.join(".test/location.yml")
if not os.path.isfile(locations_file_path):
raise RuntimeError("Location file not found: " + locations_file_path)
with open(locations_file_path, "r") as infile:
location_file = yaml.safe_load(infile.read())
test_locations= location_file["paths"]
return test_locations
def get_matches_mr(self):
merge_request = MergeRequest()
locations = self.read_location_file()
data_locations= merge_request.get_matches(locations)
return data_locations
Like suggested in the comment, I would also say the best way to test such a scenario is to mock read_location_file. Because mocking the file system methods like os.path.join would mean that you limit the test to a certain implementation, which is a bad practice. The unit test suite should not know about the implementation detail, but only about the interfaces to be tested. Usually, in test driven development you write the test before the logic is implemented. This way you would not even know os.path.join is used.
The following code shows how to mock the read_location_file method. Assuming the class containing your two methods is called ClassToBeTested (replace with your actual class name).
import os.path
from class_to_test import ClassToBeTested
def test_function_to_test(tmpdir, monkeypatch):
def mockreturn():
return [
os.path.join(tmpdir, "sample/path/a"),
os.path.join(tmpdir, "sample/path/b"),
os.path.join(tmpdir, "sample/path/c"),
]
monkeypatch.setattr(ClassToBeTested, 'read_location_file', mockreturn)
c = ClassToBeTested()
assert c.get_matches_mr()
Note: I use the fixtures tmpdir and monkeypatch, which are both built-ins of pytest:
See this answer to find some info about tmpdir (in the linked answer I explained tmp_path, but it provides the same concept as tmpdir; the difference is tmp_path returns a pathlib.Path object, and tmpdir returns a py.path.local object).
monkeypatch is a pytest fixture that provides methods for mocking/patching of objects.
Split your function into two parts:
Finding and opening the correct file.
Reading and parsing the opened file.
Your function only does the second part; the call can be responsible for the first part.
def read_location_file(infile):
location_file = yaml.safe_load(infile.read())
test_locations= location_file["paths"]
return test_locations
Your test code can then use something like io.StringIO to verify that your function can parse it correctly.
def test_read_location():
assert read_location_file(io.StringIO("...")) == ...
Your production code will handle opening the file:
with open(location_file_path) as f:
locations = read_location_file(f)
What would be an elegant a way of unit testing a method like this ?
from feedgen.feed import FeedGenerator
def generate_feed(file_list):
fg = FeedGenerator()
for f in file_list:
fe = fg.add_entry()
fg.rss_str(pretty=True)
fg.rss_file('podcast.xml')
Obviously the method doesn't return any value, but writes an xml file instead . Is there an elegant way of unit testing this ? Or should I look into refactoring my method to make it more unit testing friendly and assert against something ?
The standard way of doing this would be to use patch and mock to mock the FeedGenerator class.
If you pass the FeedGenerator in as an argument, you don't need to patch:
def generate_feed(file_list: List[File], fg: FeedGenerator) -> None:
# do stuff to fg based on file_list
Now your test can do:
file_list = [File("foo"), File("bar")] # or whatever?
fg = MagicMock(spec=FeedGenerator)
generate_feed(file_list, fg)
assert fg.add_entry.call_count == len(file_list)
# etc
rss_file can take either a file name or a file-like object as its argument. Rewrite generate_feed to accept an argument that gets passed to rss_file.
def generate_feed(file_list, fobj="podcast.xml"):
fg = FeedGenetaror()
for f in file_list:
fe = fg.add_entry()
fg.rss_str(pretty=True)
fg.rss_file(fobj)
In production, you can write
with open("podcast.xml", "w") as f:
generate_feed(file_list, f)
or simply
generate_feed(file_list)
For testing, you can use a StringIO object.
f = io.StringIO()
generate_feed(file_list, f)
Then you can test the contents of f (via f.getvalue()) as if it were the XML file you generated.
You could test to see if the file is actually created, in the correct path, and delete it after that (because it's only a test) if that's applicable to you. You can even read the file to see if it's set-up properly.
You could also have that function return True of False upon success or failure in writing the file, and test for those (for example, if you know there are edge cases that SHOULD fail).
Another useful test would be to just check that the method doesn't raise an error under various conditions.
I am trying to unit test a method that reads the lines from a file and process it.
with open([file_name], 'r') as file_list:
for line in file_list:
# Do stuff
I tried several ways described on another questions but none of them seems to work for this case. I don't quite understand how python uses the file object as an iterable on the lines, it internally use file_list.readlines() ?
This way didn't work:
with mock.patch('[module_name].open') as mocked_open: # also tried with __builtin__ instead of module_name
mocked_open.return_value = 'line1\nline2'
I got an
AttributeError: __exit__
Maybe because the with statement have this special attribute to close the file?
This code makes file_list a MagicMock. How do I store data on this MagicMock to iterate over it ?
with mock.patch("__builtin__.open", mock.mock_open(read_data="data")) as mock_file:
Best regards
The return value of mock_open (until Python 3.7.1) doesn't provide a working __iter__ method, which may make it unsuitable for testing code that iterates over an open file object.
Instead, I recommend refactoring your code to take an already opened file-like object. That is, instead of
def some_method(file_name):
with open([file_name], 'r') as file_list:
for line in file_list:
# Do stuff
...
some_method(file_name)
write it as
def some_method(file_obj):
for line in file_obj:
# Do stuff
...
with open(file_name, 'r') as file_obj:
some_method(file_obj)
This turns a function that has to perform IO into a pure(r) function that simply iterates over any file-like object. To test it, you don't need to mock open or hit the file system in any way; just create a StringIO object to use as the argument:
def test_it(self):
f = StringIO.StringIO("line1\nline2\n")
some_method(f)
(If you still feel the need to write and test a wrapper like
def some_wrapper(file_name):
with open(file_name, 'r') as file_obj:
some_method(file_obj)
note that you don't need the mocked open to do anything in particular. You test some_method separately, so the only thing you need to do to test some_wrapper is verify that the return value of open is passed to some_method. open, in this case, can be a plain old mock with no special behavior.)
I have a class with some #staticmethod's that are procedures, thus they do not return anything / their return type is None.
If they fail during their execution, they throw an Exception.
I want to unittest this class, but I am struggling with designing positive tests.
For negative tests this task is easy:
assertRaises(ValueError, my_static_method(*args))
assertRaises(MyCustomException, my_static_method(*args))
...but how do I create positive tests? Should I redesign my procedures to always return True after execution, so that I can use assertTrue on them?
Without seeing the actual code it is hard to guess, however I will make some assumptions:
The logic in the static methods is deterministic.
After doing some calculation on the input value there is a result
and some operation is done with this result.
python3.4 (mock has evolved and moved over the last few versions)
In order to test code one has to check that at least in the end it produces the expected results. If there is no return value then the result is usually stored or send somewhere. In this case we can check that the method that stores or sends the result is called with the expected arguments.
This can be done with the tools available in the mock package that has become part of the unittest package.
e.g. the following static method in my_package/my_module.py:
import uuid
class MyClass:
#staticmethod
def my_procedure(value):
if isinstance(value, str):
prefix = 'string'
else:
prefix = 'other'
with open('/tmp/%s_%s' % (prefix, uuid.uuid4()), 'w') as f:
f.write(value)
In the unit test I will check the following:
open has been called.
The expected file name has been calculated.
openhas been called in write mode.
The write() method of the file handle has been called with the expected argument.
Unittest:
import unittest
from unittest.mock import patch
from my_package.my_module import MyClass
class MyClassTest(unittest.TestCase):
#patch('my_package.my_module.open', create=True)
def test_my_procedure(self, open_mock):
write_mock = open_mock.return_value.write
MyClass.my_procedure('test')
self.assertTrue(open_mock.call_count, 1)
file_name, mode = open_mock.call_args[0]
self.assertTrue(file_name.startswith('/tmp/string_'))
self.assertEqual(mode, 'w')
self.assertTrue(write_mock.called_once_with('test'))
If your methods do something, then I'm sure there should be a logic there. Let's consider this dummy example:
cool = None
def my_static_method(something):
try:
cool = int(something)
except ValueError:
# logs here
for negative test we have:
assertRaises(ValueError, my_static_method(*args))
and for possitive test we can check cool:
assertIsNotNone(cool)
So you're checking if invoking my_static_method affects on cool.