I'm trying to build tests for some models that have a FileField. The model looks like this:
class SolutionFile(models.Model):
'''
A file from a solution.
'''
solution = models.ForeignKey(Solution)
file = models.FileField(upload_to=make_solution_file_path)
I have encountered two problems:
When saving data to a fixture using ./manage.py dumpdata, the file contents are not saved, only the file name is saved into the fixture. While I find this to be the expected behavior as the file contents are not saved into the database, I'd like to somehow include this information in the fixture for tests.
I have a test case for uploading a file that looks like this:
def test_post_solution_file(self):
import tempfile
import os
filename = tempfile.mkstemp()[1]
f = open(filename, 'w')
f.write('These are the file contents')
f.close()
f = open(filename, 'r')
post_data = {'file': f}
response = self.client.post(self.solution.get_absolute_url()+'add_solution_file/', post_data,
follow=True)
f.close()
os.remove(filename)
self.assertTemplateUsed(response, 'tests/solution_detail.html')
self.assertContains(response, os.path.basename(filename))
While this test works just fine, it leaves the uploaded file in the media directory after finishing. Of course, the deletion could be taken care of in tearDown(), but I was wondering if Django had another way of dealing with this.
One solution I was thinking of was using a different media folder for tests which must be kept synced with the test fixtures. Is there any way to specify another media directory in settings.py when tests are being run? And can I include some sort of hook to dumpdata so that it syncs the files in the media folders?
So, is there a more Pythonic or Django-specific way of dealing with unit tests involving files?
Django provides a great way to write tests on FileFields without mucking about in the real filesystem - use a SimpleUploadedFile.
from django.core.files.uploadedfile import SimpleUploadedFile
my_model.file_field = SimpleUploadedFile('best_file_eva.txt', b'these are the contents of the txt file')
It's one of django's magical features-that-don't-show-up-in-the-docs :). However it is referred to here.
You can override the MEDIA_ROOT setting for your tests using the #override_settings() decorator as documented:
from django.test import override_settings
#override_settings(MEDIA_ROOT='/tmp/django_test')
def test_post_solution_file(self):
# your code here
I've written unit tests for an entire gallery app before, and what worked well for me was using the python tempfile and shutil modules to create copies of the test files in temporary directories and then delete them all afterwards.
The following example is not working/complete, but should get you on the right path:
import os, shutil, tempfile
PATH_TEMP = tempfile.mkdtemp(dir=os.path.join(MY_PATH, 'temp'))
def make_objects():
filenames = os.listdir(TEST_FILES_DIR)
if not os.access(PATH_TEMP, os.F_OK):
os.makedirs(PATH_TEMP)
for filename in filenames:
name, extension = os.path.splitext(filename)
new = os.path.join(PATH_TEMP, filename)
shutil.copyfile(os.path.join(TEST_FILES_DIR, filename), new)
#Do something with the files/FileField here
def remove_objects():
shutil.rmtree(PATH_TEMP)
I run those methods in the setUp() and tearDown() methods of my unit tests and it works great! You've got a clean copy of your files to test your filefield that are reusable and predictable.
with pytest and pytest-django, I use this in conftest.py file:
import tempfile
import shutil
from pytest_django.lazy_django import skip_if_no_django
from pytest_django.fixtures import SettingsWrapper
#pytest.fixture(scope='session')
##pytest.yield_fixture()
def settings():
"""A Django settings object which restores changes after the testrun"""
skip_if_no_django()
wrapper = SettingsWrapper()
yield wrapper
wrapper.finalize()
#pytest.fixture(autouse=True, scope='session')
def media_root(settings):
tmp_dir = tempfile.mkdtemp()
settings.MEDIA_ROOT = tmp_dir
yield settings.MEDIA_ROOT
shutil.rmtree(tmp_dir)
#pytest.fixture(scope='session')
def django_db_setup(media_root, django_db_setup):
print('inject_after')
might be helpful:
https://dev.funkwhale.audio/funkwhale/funkwhale/blob/de777764da0c0e9fe66d0bb76317679be964588b/api/tests/conftest.py
https://framagit.org/ideascube/ideascube/blob/master/conftest.py
https://stackoverflow.com/a/56177770/5305401
This is what I did for my test. After uploading the file it should end up in the photo property of my organization model object:
import tempfile
filename = tempfile.mkstemp()[1]
f = open(filename, 'w')
f.write('These are the file contents')
f.close()
f = open(filename, 'r')
post_data = {'file': f}
response = self.client.post("/org/%d/photo" % new_org_data["id"], post_data)
f.close()
self.assertEqual(response.status_code, 200)
## Check the file
## org is where the file should end up
org = models.Organization.objects.get(pk=new_org_data["id"])
self.assertEqual("These are the file contents", org.photo.file.read())
## Remove the file
import os
os.remove(org.photo.path)
Related
I use django-storages and have user related content stored in folders on S3. Now I want users to have the ability to download all their files at once, preferably in a zip file. All the previous posts related to this are old or not working for me.
The closest to working code I have so far:
from io import BytesIO
import zipfile
from django.conf import settings
from ..models import Something
from django.core.files.storage import default_storage
class DownloadIncomeTaxFiles(View):
def get(self, request, id):
itr = Something.objects.get(id=id)
files = itr.attachments
zfname = 'somezip.zip'
b = BytesIO()
with zipfile.ZipFile(b, 'w') as zf:
for current_file in files:
try:
fh = default_storage.open(current_file.file.name, "r")
zf.writestr(fh.name, bytes(fh.read()))
except Exception as e:
print(e)
response = HttpResponse(zf, content_type="application/x-zip-compressed")
response['Content-Disposition'] = 'attachment; filename={}'.format(zfname)
return response
This creates what looks like a zipfile but the only content it has is '<zipfile.ZipFile [closed]>'
I got many different results, mainly with errors like zipfile expecting string or bytes content while a FieldFile is provided. At this point I'm completely stuck.
The problem was that I needed to revert back to the beginning of the file by adding
zf.seek(0)
just before returning the file in the HttpResponse.
I want to mock a a filesystemcall that is creating an file. But I got the problem, that I am using flask to create the output and flask also needs to read the teamplate from filesystem. So I am running in an error while rendering the output with flask.
Is there a good way to mock just one file instead of all filesystem calls?
def func_to_test(self, data_for_html):
template_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), 'templates'))
app = flask.Flask('my app', template_folder=template_dir)
with app.app_context():
rendered = render_template('index.html', data=data_for_html)
with open(self.fileName, **self.options_file) as html_file:
html_file.write(rendered)
def test_func(self, data):
fake_file_path = "fake/file/path/filename"
m = mock_open()
with patch('builtins.open', mock_open()) as m:
data_writer = FlaskObject(fileName=fake_file_path)
data_writer.write(data)
Split the function to test so you can test each part in isolation:
def _generate_content(self, data):
template_dir = os.path.abspath(os.path.join(os.path.dirname(__file__), 'templates'))
app = flask.Flask('my app', template_folder=template_dir)
with app.app_context():
return render_template('index.html', data=data_for_html)
def _write_content(self, content):
with open(self.fileName, **self.options_file) as html_file:
html_file.write(content)
def func_to_test(self, data_for_html):
rendered = self._generate_content(data_for_html)
self._write_content(rendered)
and then you can mock those two methods and test that func_to_test calls them both with expected values.
Instead of mocking open you could create a temporary file that you write to instead using tempfile.
def test_func(self, data):
with tempfile.NamedTemporaryFile() as f:
data_writer = FlaskObject(fileName=f.name)
data_writer.write(data)
This will not work on windows, if you wish for it to work on windows you would have to create the temp file with delete=False, close the file and then delete the file after the test
I am trying to add some validation for user uploaded files. This requires running through a custom script I made called "sumpin", which only takes a filepath as a variable and sends back JSON data that will verify. Everything inside my script is working independently, putting it together where the error occurs.
Since this is file validation, I decided to expand my file_extension validator that was already working.
models.py
from allauthdemo.fileuploadapp.slic3rcheck import sumpin
def user_directory_path_files(instance, filename):
return os.path.join('uploads', str(instance.objectid), filename)
def validate_file_extension(value):
ext = os.path.splitext(value.name)[1]
valid_extensions = ['.stl','.STL']
if not ext in valid_extensions:
raise ValidationError(u'Please upload a .stl file type only')
data = sumpin(value.path)
print (data)
class subfiles(models.Model):
STL = models.FileField(_('STL Upload'),
upload_to=user_directory_path_files, validators=[validate_file_extension])
The error that I get is that the path (value.path) is not valid.
This is the incorrect path because the upload_to tag must change this at a later point. This may be obvious, but I also need to have the file at the filepath location when my script is called. So essentially my questions are...
How can pass the "upload_to" path into my validator to run through my custom script?
Is there a better method to deal with uploaded files, like in the main class with a "save" or "clean" function?
I've found my own answer, but I'll post it here in case someone runs across this issue in the future.
I was incorrect, a validator wouldn't actually download the file. I need to use a file upload handler, which is shown below.
import os
from django.core.files.storage import default_storage
from allauthdemo.fileuploadapp.slic3rcheck import sumpin
def handle_uploaded_file(f):
with open(default_storage.path('tmp/'+f.name), 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
data = sumpin(default_storage.path('tmp/'+f.name))
os.remove(default_storage.path('tmp/'+f.name))
return data
then I call this inside my views.py.
from allauthdemo.fileuploadapp.uploadhandler import handle_uploaded_file
#login_required
def STLupload(request):
# Handle file upload
if request.method == 'POST':
formA = ObjectUp(request.POST, request.FILES)
if formA is_valid():
data = handle_uploaded_file(request.FILES['STL'])
This will return whatever I called to return within handle_upload_file, which worked perfect for my issues. Hopefully someone will find this useful the future.
I have a python method does the following:
list the files under a directory using os.listdir(/test)
regex match some of the files under the directory, put the files in a list
read the contents out of files in the list, do some aggregation stuff.
Obviously, the only interesting part for me to test in my case is 2, 3, so 1 is definitely something I want to mock against. I started doing patch file creation/deletion under /test folder in my setUp() and tearDown(). But colleague told me it's not good idea to do I/O in unitest.
so what's the best way to mock build in os.listdir() in my unitest? or what's the alternatives?
Is there anything I can do to achieve something like:
setUp() {
#mock a few files eg.test1.txt, test2.txt, test3.txt under directory /test
#without physically creating them using I/O
}
tearDown() {
#whatever cleanup required
}
What about using the Mock module?
>>> import os
>>> from mock import MagicMock
>>> os.listdir = MagicMock(return_value=['file1.txt', 'file2.txt', 'file3.txt'])
>>> os.listdir('./test')
['file1.txt', 'file2.txt', 'file3.txt']
If you don't want to mokey-patch (ie. break) os, then you could use mock_os or the likes.
Read about starting and stopping:
http://docs.python.org/dev/py3k/library/unittest.mock.html#patch-methods-start-and-stop
And:
http://docs.python.org/dev/py3k/library/unittest.mock.html#quick-guide
I find that the Mock Module is the way to go for both listing files and reading mocked data. These can of course be combined in one test but I have separated these out in a working file for clarity.
import unittest
from mock import patch, mock_open
import os
class Test(unittest.TestCase):
#patch.object(os, 'listdir')
def test_listdir(self, mock_listdir):
expected = ['file1.txt', 'file2.txt']
mock_listdir.return_value = expected
self.assertEquals(expected, Code().get_folder("~"))
def test_file_mock(self):
expected_string = "Some File Contents"
mocked_file_object = mock_open(read_data=expected_string)
with patch('__main__.open', mocked_file_object, create=True) as mocked_open:
self.assertEquals(expected_string, Code().get_file_as_string('any'))
class Code(object):
def get_folder(self, folder):
return os.listdir(folder)
def get_file_as_string(self, afile):
with open(afile, 'r') as handle:
return handle.read()
if __name__ == '__main__':
unittest.main()
I'm having problems with the standard Django FileField and tempfile.TemporaryFile. Whenever I try to save a FileField with the TemporaryFile, I get the "Unable to determine the file's size" error.
For example, given a model named Model, a filefield named FileField, and a temporaryfile named TempFile:
Model.FileField.save('foobar', django.core.files.File(TempFile), save=True)
This will give me the aforementioned error. Any thoughts?
I had this problem with tempfile.TemporaryFile. When I switched to tempfile.NamedTemporaryFile it went away. I believe that TemporaryFile just simulates being a file (on some operating system at least), whereas NamedTemporaryFile really is a file.
I was having the same problem and was able to solve it for my case. This is the code that django uses to determine the size of a file:
def _get_size(self):
if not hasattr(self, '_size'):
if hasattr(self.file, 'size'):
self._size = self.file.size
elif os.path.exists(self.file.name):
self._size = os.path.getsize(self.file.name)
else:
raise AttributeError("Unable to determine the file's size.")
return self._size
Therefore, django will raise an AttributeError if the file does not exist on disk (or have a size attribute already defined). Since the TemporaryFile class attempts to create a file in memory instead of actually on disk, this _get_size method doesn't work. In order to get it to work, I had to do something like this:
import tempfile, os
# Use tempfile.mkstemp, since it will actually create the file on disk.
(temp_filedescriptor, temp_filepath) = tempfile.mkstemp()
# Close the open file using the file descriptor, since file objects
# returned by os.fdopen don't work, either
os.close(temp_filedescriptor)
# Open the file on disk
temp_file = open(temp_filepath, "w+b")
# Do operations on your file here . . .
modelObj.fileField.save("filename.txt", File(temp_file))
temp_file.close()
# Remove the created file from disk.
os.remove(temp_filepath)
Alternatively (and preferably), if you can calculate the size of the temporary file you're creating, you could set a size attribute on the TemporaryFile object directly. Due to the libraries I was using, this was not a possibility for me.
I had this issue on Heroku even with tempfile.NamedTemporaryFile and was quite disappointed ...
I solved it using Steven's tips by setting arbitrary size manually (yes, dirty, but work for me):
from django.core.files import File
from django.core.files.temp import NamedTemporaryFile
img_temp = NamedTemporaryFile()
# Do your stuffs ...
img_temp.flush()
img_temp.size = 1024
media.thumbnail.save('dummy', File(img_temp))
Thanks !
I know this is a bit old but I've managed to save a base64 file (without having the actual file saved on the disk) by using the ContentFile class provided by Django.
According to the docs:
The ContentFile class inherits from File, but unlike File it operates on string content (bytes also supported), rather than an actual file.
The snippet below receives a base64 string, extract it's data and file extension and save it to an ImageField using the ContentFile class
import uuid
from django.core.files.base import ContentFile
def convert_b64data(b64data, filename):
file_format, imgstr = b64data.split(';base64,')
ext = file_format.split('/')[-1]
return {
'obj': base64.b64decode(imgstr),
'extension': ext,
}
b64data = request.data['b64file']
filename = str(uuid.uuid4())
file_data = convert_b64data(b64data, filename)
file_path = 'media/{}/{}.{}'.format(
user.code,
filename,
file_data['extension']
)
user.banner.save(file_path, ContentFile(file_data['obj']))
In newer versions of Django (I checked on 3.2), you just may need to wrap the file in a ContentFile.
from django.core.files.base import ContentFile
Model.FileField.save('foobar', ContentFile(file))
https://docs.djangoproject.com/en/3.2/ref/files/file/