I'm trying to test a method that requires the use of json.load in Python 3.6.
And after several attempts, I tried running the test "normally" (with the usual unittest.main() from the CLI), and in the iPython REPL.
Having the following function (simplified for purpose of the example)
def load_metadata(name):
with open("{}.json".format(name)) as fh:
return json.load(fh)
with the following test:
class test_loading_metadata(unittest2.TestCase):
#patch('builtins.open', new_callable=mock_open(read_data='{"disabled":True}'))
def test_load_metadata_with_disabled(self, filemock):
result = load_metadata("john")
self.assertEqual(result,{"disabled":True})
filemock.assert_called_with("john.json")
The result of the execution of the test file, yields a heart breaking:
TypeError: the JSON object must be str, bytes or bytearray, not 'MagicMock'
While executing the same thing in the command line, gives a successful result.
I tried in several ways (patching with with, as decorator), but the only thing that I can think of, is the unittest library itself, and whatever it might be doing to interfere with mock and patch.
Also checked versions of python in the virtualenv and ipython, the versions of json library.
I would like to know why what looks like the same code, works in one place
and doesn't work in the other.
Or at least a pointer in the right direction to understand why this could be happening.
json.load() simply calls fh.read(), but fh is not a mock_open() object. It's a mock_open()() object, because new_callable is called before patching to create the replacement object:
>>> from unittest.mock import patch, mock_open
>>> with patch('builtins.open', new_callable=mock_open(read_data='{"disabled":True}')) as filemock:
... with open("john.json") as fh:
... print(fh.read())
...
<MagicMock name='open()().__enter__().read()' id='4420799600'>
Don't use new_callable, you don't want your mock_open() object to be called! Just pass it in as the new argument to #patch() (this is also the second positional argument, so you can leave off the new= here):
#patch('builtins.open', mock_open(read_data='{"disabled":True}'))
def test_load_metadata_with_disabled(self, filemock):
at which point you can call .read() on it when used as an open() function:
>>> with patch('builtins.open', mock_open(read_data='{"disabled":True}')) as filemock:
... with open("john.json") as fh:
... print(fh.read())
...
{"disabled":True}
The new argument is the object that'll replace the original when patching. If left to the default, new_callable() is used instead. You don't want new_callable() here.
Related
I have a function
def execute(process: subprocess.Popen):
if not isinstance(process, subprocess.Popen):
raise ValueError('expected: subprocessPopen, found: %s' % type(process))
data = io.TextIOWrapper(process.stdout, encoding='utf-8')
func1(data_input)
I want to unit test it, where I want data as a dictionary which is to be passed to func1 as an argument but not able to find a way out.
Have a look at the spec argument to the Mock object, in order to make it pass the isinstance() test:
https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock
Have a look at the patch decorator for how to replace the process.stdout member with, e.g., a io.BytesIO object:
https://docs.python.org/3/library/unittest.mock.html#unittest.mock.patch
You also might want to consider the examples given here:
https://docs.python.org/3/library/unittest.mock.html#quick-guide
import mock
def test_run_func():
subprocess.Popen = mock.MagicMock(return_value=expected_return_value)
# The rest of your test...
Something like should work.
I'm using python 2.7 and delving into TDD. I'm trying to test a simple function that uses the csv module and returns a csv.reader object. I want to test that the correct type of object is being returned with the assertIsInstance test however I'm having trouble figuring out how to make this work.
#!/usr/bin/python
import os, csv
def importCSV(fileName):
'''importCSV brings in the CSV transaction file to be analyzed'''
try:
if not(os.path.exists("data")):
os.makedirs("data")
except(IOError):
return "Couldn't create data directory!"
try:
fileFullName = os.path.join("data", fileName)
return csv.reader(file(fileFullName))
except(IOError):
return "File not found!"
The test currently looks like this....
#!/usr/bin/python
from finaImport import finaImport
import unittest, os, csv
class testImport(unittest.TestCase):
'''Tests for importing a CSV file'''
def testImportCSV(self):
''' Test a good file and make sure importCSV returns a csv reader object '''
readerObject = finaImport.importCSV("toe")
self.assertTrue(str(type(readerObject))), "_csv.reader")
I really don't think wrapping "toe" in a str and type function is correct. When I try something like...
self.assertIsInstance(finaImport.importCSV("toe"), csv.reader)
It returns an error like...
TypeError: isinstance() arg2 must be a class, type, or tuple of classes and types
Help???
self.assertTrue(str(type(readerObject)), "_csv.reader")
I don't think that your first test (above) is so bad (I fixed a small typo there; you had an extra closing parenthesis). It checks that the type name is exactly "_csv.reader". On the other hand, the underscore in "_csv" tells you that this object is internal to the csv module. In general, you shouldn't be concerned about that.
Your attempt at the assertIsInstance test is flawed in that csv.reader is a function object. If you try it in the REPL, you see:
>>> import csv
>>> csv.reader
<built-in function reader>
Often, we care less about the type of an object and more about whether it implements a certain interface. In this case, the help for csv.reader says:
>>> help(csv.reader)
... The returned object is an iterator. ...
So, you could do the following test (instead or in addition to your other one):
self.assertIsInstance(readerObject, collections.Iterator)
You'll need a import collections for that, of course. And, you might want to test that the iterator returns lists of strings, or something like this. That would allow you to use something else under the hood later and the test would still pass.
I am trying to unit test a piece of code:
def _parse_results(self, file_name):
results_file = open(file_name)
results_data = list(csv.reader(results_file))
index = len(results_data[1])-1
results_file.close()
return float(results_data[1][index])
by using mock_open like so:
#mock.patch('path.open', mock.mock_open(read_data='test, test2, test3, test4'))
def test_parse_results(self):
cut = my_class(emulate=True)
self.assertEqual(VAL, cut._parse_results('file'))
The problem I am running into is that I do not get any data when running csv.reader. If I run results_file.readlines() I get 'test, test2, test3, test4' which means that mock_open is working properly. But when I run csv.reader(results_file) I lose all the data.
This is because mock_open doesn't implement every feature that a file has, and notably not some of the ones that csv needs.
mock_open implements the methods read(), readline() and readlines(), and works both as a function and when called as a context manager (https://docs.python.org/3/library/unittest.mock.html#mock-open), whereas csv.reader works with…
any object which supports the iterator protocol and returns a string each time its __next__() method is called — file objects and list objects are both suitable
— https://docs.python.org/3/library/csv.html#csv.reader
Note that mock_open doesn't implement the __next__() method, and doesn't raise StopIteration when the end is reached, so it won't work with csv.reader.
The solution, as #Emily points out in her answer, is to turn the file into a list of its lines. This is possible because mock_open implements readlines(), and the resulting list is suitable for reading into csv.reader as the documentation says.
This really got me too, and was a nightmare to pinpoint. To use your example code, this works
results_data = list(csv.reader(results_file.read()))
and this works
results_data = list(csv.reader(results_file.readlines()))
but this doesn't work
results_data = list(csv.reader(results_file))
using Python 3.4.
It seems counter to the documented interface of csv.reader so maybe an expert can elaborate on why.
It used to be in Python (2.6) that one could ask:
isinstance(f, file)
but in Python 3.0 file was removed.
What is the proper method for checking to see if a variable is a file now? The What'sNew docs don't mention this...
def read_a_file(f)
try:
contents = f.read()
except AttributeError:
# f is not a file
substitute whatever methods you plan to use for read. This is optimal if you expect that you will get passed a file like object more than 98% of the time. If you expect that you will be passed a non file like object more often than 2% of the time, then the correct thing to do is:
def read_a_file(f):
if hasattr(f, 'read'):
contents = f.read()
else:
# f is not a file
This is exactly what you would do if you did have access to a file class to test against. (and FWIW, I too have file on 2.6) Note that this code works in 3.x as well.
In python3 you could refer to io instead of file and write
import io
isinstance(f, io.IOBase)
Typically, you don't need to check an object type, you could use duck-typing instead i.e., just call f.read() directly and allow the possible exceptions to propagate -- it is either a bug in your code or a bug in the caller code e.g., json.load() raises AttributeError if you give it an object that has no read attribute.
If you need to distinguish between several acceptable input types; you could use hasattr/getattr:
def read(file_or_filename):
readfile = getattr(file_or_filename, 'read', None)
if readfile is not None: # got file
return readfile()
with open(file_or_filename) as file: # got filename
return file.read()
If you want to support a case when file_of_filename may have read attribute that is set to None then you could use try/except over file_or_filename.read -- note: no parens, the call is not made -- e.g., ElementTree._get_writer().
If you want to check certain guarantees e.g., that only one single system call is made (io.RawIOBase.read(n) for n > 0) or there are no short writes (io.BufferedIOBase.write()) or whether read/write methods accept text data (io.TextIOBase) then you could use isinstance() function with ABCs defined in io module e.g., look at how saxutils._gettextwriter() is implemented.
Works for me on python 2.6... Are you in a strange environment where builtins aren't imported by default, or where somebody has done del file, or something?
(You may read this question for some background)
I would like to have a gracefully-degrading way to pickle objects in Python.
When pickling an object, let's call it the main object, sometimes the Pickler raises an exception because it can't pickle a certain sub-object of the main object. For example, an error I've been getting a lot is "can’t pickle module objects." That is because I am referencing a module from the main object.
I know I can write up a little something to replace that module with a facade that would contain the module's attributes, but that would have its own issues(1).
So what I would like is a pickling function that automatically replaces modules (and any other hard-to-pickle objects) with facades that contain their attributes. That may not produce a perfect pickling, but in many cases it would be sufficient.
Is there anything like this? Does anyone have an idea how to approach this?
(1) One issue would be that the module may be referencing other modules from within it.
You can decide and implement how any previously-unpicklable type gets pickled and unpickled: see standard library module copy_reg (renamed to copyreg in Python 3.*).
Essentially, you need to provide a function which, given an instance of the type, reduces it to a tuple -- with the same protocol as the reduce special method (except that the reduce special method takes no arguments, since when provided it's called directly on the object, while the function you provide will take the object as the only argument).
Typically, the tuple you return has 2 items: a callable, and a tuple of arguments to pass to it. The callable must be registered as a "safe constructor" or equivalently have an attribute __safe_for_unpickling__ with a true value. Those items will be pickled, and at unpickling time the callable will be called with the given arguments and must return the unpicked object.
For example, suppose that you want to just pickle modules by name, so that unpickling them just means re-importing them (i.e. suppose for simplicity that you don't care about dynamically modified modules, nested packages, etc, just plain top-level modules). Then:
>>> import sys, pickle, copy_reg
>>> def savemodule(module):
... return __import__, (module.__name__,)
...
>>> copy_reg.pickle(type(sys), savemodule)
>>> s = pickle.dumps(sys)
>>> s
"c__builtin__\n__import__\np0\n(S'sys'\np1\ntp2\nRp3\n."
>>> z = pickle.loads(s)
>>> z
<module 'sys' (built-in)>
I'm using the old-fashioned ASCII form of pickle so that s, the string containing the pickle, is easy to examine: it instructs unpickling to call the built-in import function, with the string sys as its sole argument. And z shows that this does indeed give us back the built-in sys module as the result of the unpickling, as desired.
Now, you'll have to make things a bit more complex than just __import__ (you'll have to deal with saving and restoring dynamic changes, navigate a nested namespace, etc), and thus you'll have to also call copy_reg.constructor (passing as argument your own function that performs this work) before you copy_reg the module-saving function that returns your other function (and, if in a separate run, also before you unpickle those pickles you made using said function). But I hope this simple cases helps to show that there's really nothing much to it that's at all "intrinsically" complicated!-)
How about the following, which is a wrapper you can use to wrap some modules (maybe any module) in something that's pickle-able. You could then subclass the Pickler object to check if the target object is a module, and if so, wrap it. Does this accomplish what you desire?
class PickleableModuleWrapper(object):
def __init__(self, module):
# make a copy of the module's namespace in this instance
self.__dict__ = dict(module.__dict__)
# remove anything that's going to give us trouble during pickling
self.remove_unpickleable_attributes()
def remove_unpickleable_attributes(self):
for name, value in self.__dict__.items():
try:
pickle.dumps(value)
except Exception:
del self.__dict__[name]
import pickle
p = pickle.dumps(PickleableModuleWrapper(pickle))
wrapped_mod = pickle.loads(p)
Hmmm, something like this?
import sys
attribList = dir(someobject)
for attrib in attribList:
if(type(attrib) == type(sys)): #is a module
#put in a facade, either recursively list the module and do the same thing, or just put in something like str('modulename_module')
else:
#proceed with normal pickle
Obviously, this would go into an extension of the pickle class with a reimplemented dump method...