I have a python method does the following:
list the files under a directory using os.listdir(/test)
regex match some of the files under the directory, put the files in a list
read the contents out of files in the list, do some aggregation stuff.
Obviously, the only interesting part for me to test in my case is 2, 3, so 1 is definitely something I want to mock against. I started doing patch file creation/deletion under /test folder in my setUp() and tearDown(). But colleague told me it's not good idea to do I/O in unitest.
so what's the best way to mock build in os.listdir() in my unitest? or what's the alternatives?
Is there anything I can do to achieve something like:
setUp() {
#mock a few files eg.test1.txt, test2.txt, test3.txt under directory /test
#without physically creating them using I/O
}
tearDown() {
#whatever cleanup required
}
What about using the Mock module?
>>> import os
>>> from mock import MagicMock
>>> os.listdir = MagicMock(return_value=['file1.txt', 'file2.txt', 'file3.txt'])
>>> os.listdir('./test')
['file1.txt', 'file2.txt', 'file3.txt']
If you don't want to mokey-patch (ie. break) os, then you could use mock_os or the likes.
Read about starting and stopping:
http://docs.python.org/dev/py3k/library/unittest.mock.html#patch-methods-start-and-stop
And:
http://docs.python.org/dev/py3k/library/unittest.mock.html#quick-guide
I find that the Mock Module is the way to go for both listing files and reading mocked data. These can of course be combined in one test but I have separated these out in a working file for clarity.
import unittest
from mock import patch, mock_open
import os
class Test(unittest.TestCase):
#patch.object(os, 'listdir')
def test_listdir(self, mock_listdir):
expected = ['file1.txt', 'file2.txt']
mock_listdir.return_value = expected
self.assertEquals(expected, Code().get_folder("~"))
def test_file_mock(self):
expected_string = "Some File Contents"
mocked_file_object = mock_open(read_data=expected_string)
with patch('__main__.open', mocked_file_object, create=True) as mocked_open:
self.assertEquals(expected_string, Code().get_file_as_string('any'))
class Code(object):
def get_folder(self, folder):
return os.listdir(folder)
def get_file_as_string(self, afile):
with open(afile, 'r') as handle:
return handle.read()
if __name__ == '__main__':
unittest.main()
Related
I want to create a tool called unifile for saving and opening files
like this unifile.open.yaml("file.yaml").
This is my structure:
unifile
|
├-open
| └--__init__.py
|
└-save
└--__init__.py
Code that call my module:
import unifile
a = unifile.open.yaml("file.yaml")
open/init.py
import yaml
class open():
def yml(self, file_path):
try:
with open(file_path, "r", encoding="utf-8") as yaml_conf:
yaml_file = yaml.safe_load(yaml_conf)
return yaml_file
except OSError:
print("Can't load yaml")
1 error if I import unifile always say:
module unifile has no atribute open
2 error in __init__.py I can't open file
[pylint] Context manager 'open' doesn't implement enter and exit. [not-context-manager]
here adding solution to ur problem, make your project structure like this.
add unifile/__init__.py file in the unifile itself not in other modules.
then unifile/open/_open.py file content
import yaml
class Open():
def __init__(self):
pass
def yml(self, file_path):
try:
with open(file_path, "r", encoding="utf-8") as yaml_conf:
yaml_file = yaml.safe_load(yaml_conf)
return yaml_file
except OSError:
print("Can't load yaml")
content of the unifile/__init__.py file
from .open._open import Open
in terminal run the program like this
Also, It is better to create a object element first then proceed ahead.
Two issues, two answers.
First, you should add an init file in unifile. With this, Python will understand that unifile is a package with a sub package.
Second, open is a built-in function and you overwrite it by calling your class open. Change your class name and it should work.
You are getting this error because unifile is not a package. There isn't any init.py file at the top level same as open and save. You also cannot call open.yml directly, because open is a class in package open, so either you will have to import open from open, create its instance and then call iml on that instance.
from open import open
a = open().yml('file.yml')
You are getting this error, because you are trying to override an existing keyword in Python open which you should strictly prohibit doing. So you should name your class anything except a reserved keyword.
I am working through an online python course and have an issue that I am having trouble working through. I have a executable directory with the following layout
reader/
|--__main__.py
|--reader
|--__init__.py
|--reader.py
|--compressed
|--gzipped.py
|--bzipped.py
|--__init__.py
When I do not have these modules in the top directory, I can import just fine and use all of the functionality. When I put them in the top level directory and run the executable directory from the command line with
python reader test.gz
I am getting the following error
AttributeError: module 'reader' has not attribute 'Reader'
The code for main.py is
import sys
import reader
r = reader.Reader(sys.argv[1])
try:
print(r.read())
finally:
r.close()
The code for reader.py is
import os
from reader.compressed import gzipped, bzipped
extension_map = {
'.bz2': bzipped.opener,
'.gz': gzipped.opener,
}
class Reader:
def __init__(self, filename):
extension = os.path.splitext(filename)[1]
opener = extension_map.get(extension, open)
self.f = opener(filename, 'rt')
def close(self):
self.f.close()
def read(self):
return self.f.read()
I can provide the rest of the files if needed. I am using the current distribution of Anaconda. Any help or explanations would be appreciated.
It seems to me that because of your __init__.py file the folder reader is seen as a python module. Thereore your main.py code tries to find the class Reader at the folder level, but you need to look for Reader within reader.py. Can you try changing
r = reader.Reader(sys.argv[1])
to
r = reader.reader.Reader(sys.argv[1])
Also, you have a lot of modules/files/modules called reader, e.g. the class Reader is in reader/reader/reader.py. This I would try to avoid as it can lead to confusion.
When you have multiple modules/modules that are named identically you can often get the namespace into a recursive loop. Also, the Reader class is capitalized.
Like mentioned above, try python reader.reader test.gz
While you are troubleshooting try the tab-complete feature to see if your modules are being loaded properly.
In my python package I have a configuration module that reads a yaml file (when creating the instance) at an explicit location, i.e. something like
class YamlConfig(object):
def __init__(self):
filename = os.path.join(os.path.expanduser('~'), '.hanzo\\config.yml')
with open(filename) as fs:
self.cfg = yaml.load(fs.read())
Now what should I do when writing my unit test if I don't want to use the explicitly specified file? Instead I want to create a temporary config.yml to be used for testing.
I could simply allow for a specified filename in __init__(), but I strongly prefer forcing the filename location. I.e. like this
class YamlConfig(object):
def __init__(self, filename=os.path.join(os.path.expanduser('~'), '.hanzo\\config.yml')):
with open(filename) as fs:
self.cfg = yaml.load(fs.read())
Is there other ways to solve my issue? I guess it might be possible using mock right way? Also feel free to give any comments about upside and downside.
You need to run the test inside the mock context:
import unittest.mock as umock
with umock.patch('__main__.open', umock.mock_open(read_data='yaml data')):
# Run your test here and open the file
with open('filename') as f:
You might need to use library yaml to generate yaml output for read_data
I did this for one of my projects. Here's what my function looked like (simplified):
def parse_template(cve: str) -> Dict:
"""Parse yaml templates and return relevant information
Args:
cve: CVE id to identify respective template
Returns:
Dict: return CVE name
"""
year = cve.split("-")[1]
file_path = f"/cves/{year}/{cve}.yaml"
with open(file_path, "r") as f:
data = yaml.load(f, Loader=yaml.SafeLoader)
return {
"name": data.get("name", "None")
}
To write tests for this, we first create a python fixture which patches the "open" function which is responsible for opening my yaml file.
#pytest.fixture
def mock_template(mocker: MockFixture):
"""Mock yaml file"""
sample_dict={
"name":"Example cve name"
}
mocked_yaml = mocker.mock_open(read_data=yaml.dump(sample_dict))
mocker.patch("builtins.open", mocked_yaml)
Then we just use this fixture to validate our function with something like:
def test_parse_template(mock_template:pytest.fixture) -> None:
"""Tests for opening yaml template and extract data"""
result=parse_template(cve="CVE-2022-1337")
assert "name" in result.keys() # or any other validation
my function reads from a file, and a doctest needs to be written in a way independent of an absolute path. What's the best way of wrting a doctest? Writing a temp file is expensive and not failproof.
You could have a parameter that takes the path, marked with underscore to state it's for internal use only. The argument should then default to the absolute path in non-test mode. Named temporary files are the solution, and using a with statement should be failproof.
#!/usr/bin/env python3
import doctest
import json
import tempfile
def read_config(_file_path='/etc/myservice.conf'):
"""
>>> with tempfile.NamedTemporaryFile() as tmpfile:
... tmpfile.write(b'{"myconfig": "myvalue"}') and True
... tmpfile.flush()
... read_config(_file_path=tmpfile.name)
True
{'myconfig': 'myvalue'}
"""
with open(_file_path, 'r') as f:
return json.load(f)
# Self-test
if doctest.testmod()[0]:
exit(1)
For Python 2.x the doctest would be different:
#!/usr/bin/env python2
import doctest
import json
import tempfile
def read_config(_file_path='/etc/myservice.conf'):
"""
>>> with tempfile.NamedTemporaryFile() as tmpfile:
... tmpfile.write(b'{"myconfig": "myvalue"}') and True
... tmpfile.flush()
... read_config(_file_path=tmpfile.name)
{u'myconfig': u'myvalue'}
"""
with open(_file_path, 'r') as f:
return json.load(f)
# Self-test
if doctest.testmod()[0]:
exit(1)
Your doctest could use module StringIO to provide a file object from a string.
I'm trying to build tests for some models that have a FileField. The model looks like this:
class SolutionFile(models.Model):
'''
A file from a solution.
'''
solution = models.ForeignKey(Solution)
file = models.FileField(upload_to=make_solution_file_path)
I have encountered two problems:
When saving data to a fixture using ./manage.py dumpdata, the file contents are not saved, only the file name is saved into the fixture. While I find this to be the expected behavior as the file contents are not saved into the database, I'd like to somehow include this information in the fixture for tests.
I have a test case for uploading a file that looks like this:
def test_post_solution_file(self):
import tempfile
import os
filename = tempfile.mkstemp()[1]
f = open(filename, 'w')
f.write('These are the file contents')
f.close()
f = open(filename, 'r')
post_data = {'file': f}
response = self.client.post(self.solution.get_absolute_url()+'add_solution_file/', post_data,
follow=True)
f.close()
os.remove(filename)
self.assertTemplateUsed(response, 'tests/solution_detail.html')
self.assertContains(response, os.path.basename(filename))
While this test works just fine, it leaves the uploaded file in the media directory after finishing. Of course, the deletion could be taken care of in tearDown(), but I was wondering if Django had another way of dealing with this.
One solution I was thinking of was using a different media folder for tests which must be kept synced with the test fixtures. Is there any way to specify another media directory in settings.py when tests are being run? And can I include some sort of hook to dumpdata so that it syncs the files in the media folders?
So, is there a more Pythonic or Django-specific way of dealing with unit tests involving files?
Django provides a great way to write tests on FileFields without mucking about in the real filesystem - use a SimpleUploadedFile.
from django.core.files.uploadedfile import SimpleUploadedFile
my_model.file_field = SimpleUploadedFile('best_file_eva.txt', b'these are the contents of the txt file')
It's one of django's magical features-that-don't-show-up-in-the-docs :). However it is referred to here.
You can override the MEDIA_ROOT setting for your tests using the #override_settings() decorator as documented:
from django.test import override_settings
#override_settings(MEDIA_ROOT='/tmp/django_test')
def test_post_solution_file(self):
# your code here
I've written unit tests for an entire gallery app before, and what worked well for me was using the python tempfile and shutil modules to create copies of the test files in temporary directories and then delete them all afterwards.
The following example is not working/complete, but should get you on the right path:
import os, shutil, tempfile
PATH_TEMP = tempfile.mkdtemp(dir=os.path.join(MY_PATH, 'temp'))
def make_objects():
filenames = os.listdir(TEST_FILES_DIR)
if not os.access(PATH_TEMP, os.F_OK):
os.makedirs(PATH_TEMP)
for filename in filenames:
name, extension = os.path.splitext(filename)
new = os.path.join(PATH_TEMP, filename)
shutil.copyfile(os.path.join(TEST_FILES_DIR, filename), new)
#Do something with the files/FileField here
def remove_objects():
shutil.rmtree(PATH_TEMP)
I run those methods in the setUp() and tearDown() methods of my unit tests and it works great! You've got a clean copy of your files to test your filefield that are reusable and predictable.
with pytest and pytest-django, I use this in conftest.py file:
import tempfile
import shutil
from pytest_django.lazy_django import skip_if_no_django
from pytest_django.fixtures import SettingsWrapper
#pytest.fixture(scope='session')
##pytest.yield_fixture()
def settings():
"""A Django settings object which restores changes after the testrun"""
skip_if_no_django()
wrapper = SettingsWrapper()
yield wrapper
wrapper.finalize()
#pytest.fixture(autouse=True, scope='session')
def media_root(settings):
tmp_dir = tempfile.mkdtemp()
settings.MEDIA_ROOT = tmp_dir
yield settings.MEDIA_ROOT
shutil.rmtree(tmp_dir)
#pytest.fixture(scope='session')
def django_db_setup(media_root, django_db_setup):
print('inject_after')
might be helpful:
https://dev.funkwhale.audio/funkwhale/funkwhale/blob/de777764da0c0e9fe66d0bb76317679be964588b/api/tests/conftest.py
https://framagit.org/ideascube/ideascube/blob/master/conftest.py
https://stackoverflow.com/a/56177770/5305401
This is what I did for my test. After uploading the file it should end up in the photo property of my organization model object:
import tempfile
filename = tempfile.mkstemp()[1]
f = open(filename, 'w')
f.write('These are the file contents')
f.close()
f = open(filename, 'r')
post_data = {'file': f}
response = self.client.post("/org/%d/photo" % new_org_data["id"], post_data)
f.close()
self.assertEqual(response.status_code, 200)
## Check the file
## org is where the file should end up
org = models.Organization.objects.get(pk=new_org_data["id"])
self.assertEqual("These are the file contents", org.photo.file.read())
## Remove the file
import os
os.remove(org.photo.path)