We Keep Coding

Unit testing tornado applications: How to improve the display of error messages

I am using unittest to test a tornado app having several handlers, one of which raises an exception. If I run the following test code with python test.py:
# test.py
import unittest
import tornado.web
import tornado.testing
class MainHandler(tornado.web.RequestHandler):
def get(self):
self.write('Hello World') # handler works correctly
class HandlerWithError(tornado.web.RequestHandler):
def get(self):
raise Exception('Boom') # handler raises an exception
self.write('Hello World')
def make_app():
return tornado.web.Application([
(r'/main/', MainHandler),
(r'/error/', HandlerWithError),
])
class TornadoTestCase(tornado.testing.AsyncHTTPTestCase):
def get_app(self):
return make_app()
def test_main_handler(self):
response = self.fetch('/main/')
self.assertEqual(response.code, 200) # test should pass
def test_handler_with_error(self):
response = self.fetch('/error/')
self.assertEqual(response.code, 200) # test should fail with error
if __name__ == '__main__':
unittest.main()
the test output looks like:
ERROR:tornado.application:Uncaught exception GET /error/ (127.0.0.1)
HTTPServerRequest(protocol='http', host='localhost:36590', method='GET', uri='/error/', version='HTTP/1.1', remote_ip='127.0.0.1', headers={'Connection': 'close', 'Host': 'localhost:3
6590', 'Accept-Encoding': 'gzip'})
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/web.py", line 1332, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "test.py", line 13, in get
raise Exception('Boom') # handler raises an exception
Exception: Boom
ERROR:tornado.access:500 GET /error/ (127.0.0.1) 19.16ms
F.
======================================================================
FAIL: test_handler_with_error (__main__.TornadoTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/tornado/testing.py", line 118, in __call__
result = self.orig_method(*args, **kwargs)
File "test.py", line 33, in test_handler_with_error
self.assertEqual(response.code, 200) # test should fail with error
AssertionError: 500 != 200
----------------------------------------------------------------------
Ran 2 tests in 0.034s
FAILED (failures=1)
However, I would expect unittest to report an Error for the second test, instead of a failing assertion. Moreover, the fact that the traceback for the 'Boom' exception appears before the unittest test report and does not include a reference to the failing test function makes it difficult to find the source of the exception.
Any suggestions how to handle this situation?
Thanks in advance!
EDIT
What I find unexpected is the fact that test_handler_with_error actually arrives at making the assertEqual assertion, instead of throwing the error. For example, the following code does not execute the self.assertEqualstatement, and consequently reports an ERROR instead of a FAIL in the test output:
# simple_test.py
import unittest
def foo():
raise Exception('Boom')
return 'bar'
class SimpleTestCase(unittest.TestCase):
def test_failing_function(self):
result = foo()
self.assertEqual(result, 'bar')
if __name__ == '__main__':
unittest.main()

You can disable logging and only the test reports will appear:
logging.disable(logging.CRITICAL)
You can put that for example in
created TestCase subclass
test runner
More info How can I disable logging while running unit tests in Python Django?
Keep in mind that CI/CD systems actually use normalized report e.g. junit and then present it in more readable/elegant way - more info:
Python script to generate JUnit report from another testing result
How to output coverage XML with nosetests?

This is expected behavior. Your test itself asserts that the return code is HTTP 200, and since this is a formal assert that is false, the outcome is a "failure" instead of an "error". You can suppress logs as mentioned in kwaranuk's answer, but then you lose the information about what actually caused the HTTP 500 error.
Why does your code reach the assert, instead of throwing? It's because your test code does not call HandlerWithError.get. Your test code begins an asynchronous HTTP GET operation with an HTTP client provided by the AsyncHTTPTestCase class. (Check the source code of that class for details.) The event loop runs until HandlerWithError.get receives the request over a localhost socket, and responds on that socket with an HTTP 500. When HandlerWithError.get fails, it doesn't raise an exception into your test function, any more than a failure at Google.com would raise an exception: it merely results in an HTTP 500.
Welcome to the world of async! There's no easy way to neatly associate the assertion error and the traceback from HandlerWithError.get().

Mocking fails in Django unittest of utils.py

I am trying to write a unittest for a function named search_ldap(), which searches an LDAP server given a particular username. Here is the function definition in utils.py (note: I'm using Python 3):
from ldap3 import Server, Connection
def search_ldap(username):
result = ()
baseDN = "o=Universiteit van Tilburg,c=NL"
searchFilter = '(uid={})'.format(username)
attributes = ['givenName', 'cn', 'employeeNumber', 'mail']
try:
server = Server('ldap.example.com', use_ssl=True)
conn = Connection(server, auto_bind=True)
conn.search(baseDN, searchFilter, attributes=attributes)
for a in attributes:
result += (conn.response[0]['attributes'][a][0], )
except Exception:
raise LDAPError('Error in LDAP query')
return result
Of course I don't want to actually connect to ldap.example.com during testing, so I've decided to use Python's mock object library to mock the Server() and Connection() classes in my unittests. Here is the test code:
from unittest import mock
from django.test import TestCase
class LdapTest(TestCase):
#mock.patch('ldap3.Server')
#mock.patch('ldap3.Connection')
def test_search_ldap(self, mockConnection, mockServer):
from .utils import search_ldap
search_ldap('username')
self.assertTrue(mockServer.called)
self.assertTrue(mockConnection.called)
This test simply asserts that the mocked Server and Connection objects are instantiated. However, thay don't, because when I run the tests with ./manage.py test I receive the following error:
Creating test database for alias 'default'...
F.
======================================================================
FAIL: test_search_ldap (uvt_user.tests.LdapTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.4/unittest/mock.py", line 1142, in patched
return func(*args, **keywargs)
File "/home/jj/projects/autodidact/uvt_user/tests.py", line 28, in test_search_ldap
self.assertTrue(mockServer.called)
AssertionError: False is not true
----------------------------------------------------------------------
Ran 2 tests in 0.030s
FAILED (failures=1)
Destroying test database for alias 'default'...
Why are my tests failing? How can I successfully mock ldap3's Server and Connection classes?

To mock class you should provide it's fake implementation with required methods. For example:
class FakeServer:
def call():
pass
class LdapTest(TestCase):
#mock.patch('ldap3.Server', FakeServer)
def test_search_ldap(self):
<do you checks here>

With patch() it is important that you patch objects in the namespace where they are looked up. This is explained in the Where to patch section of the documentation. There is a fundamental difference between doing
from ldap3 import Server
server = Server()
and
import ldap3
server = ldap3.Server()
In the first case (also the case in the original question), the name "Server" belongs to the current module. In the second case, the name "Server" belongs to the ldap3 module where it is defined. The following Django unittest patches the correct "Server" and "Connection" names and should work as intended:
from unittest import mock
from django.test import TestCase
class LdapTest(TestCase):
#mock.patch('appname.utils.Server')
#mock.patch('appname.utils.Connection')
def test_search_ldap(self, mockConnection, mockServer):
from .utils import search_ldap
search_ldap('username')
self.assertTrue(mockServer.called)
self.assertTrue(mockConnection.called)

How to test that tornado read_message got nothing to read

I have a Tornado chat and I'm doing some tests, most client messages generate a reply from the server, but others must not generate any reply.
I managed to do it with this code, waiting for the read timeout to occur, there is a better way to do it?
import json
import tornado
from tornado.httpclient import HTTPRequest
from tornado.web import Application
from tornado.websocket import websocket_connect
from tornado.testing import AsyncHTTPTestCase, gen_test
class RealtimeHandler(tornado.websocket.WebSocketHandler):
def on_message(self, message):
if message != 'Hi':
self.write_message('Hi there')
return
class ChatTestCase(AsyncHTTPTestCase):
def get_app(self):
return Application([
('/rt', RealtimeHandler),
])
#gen_test
def test_no_reply(self):
request = HTTPRequest('ws://127.0.0.1:%d/rt' % self.get_http_port())
ws = yield websocket_connect(request)
ws.write_message('Hi')
with self.assertRaises(tornado.ioloop.TimeoutError):
response = yield ws.read_message()
Also there is a problem when test ends
======================================================================
ERROR: test_no_reply (myproj.tests.ChatTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/ubuntu/my_env/local/lib/python2.7/site-packages/tornado/testing.py", line 120, in __call__
result = self.orig_method(*args, **kwargs)
File "/home/ubuntu/my_env/local/lib/python2.7/site-packages/tornado/testing.py", line 506, in post_coroutine
self._test_generator.throw(e)
StopIteration

In general, it's difficult to test for a negative: how long do you wait before you conclude that the thing you're testing for will never happen? It's better to rearrange things so that the test can be expressed in positive terms. That's difficult to do in this toy example, but consider the following handler:
class RealtimeHandler(tornado.websocket.WebSocketHandler):
def on_message(self, message):
if int(message) % 2 == 1:
self.write_message('%s is odd' % message)
In this case you could test it by sending the messages 1, 2, and 3, and asserting that you get two responses, "1 is odd" and "3 is odd".
The StopIteration failure you see is slightly surprising to me: I would not expect a timeout to be catchable within the #gen_test method, so doing so may have unexpected results, but I wouldn't have expected it to turn into a StopIteration. In any case, it's better to restructure the test so that you don't have to rely on timeouts. And if you do need a timeout, use gen.with_timeout so you can control the timeout from inside the test instead of relying on the one from outside in #gen_test.

Just to illustrate the #Ben Darnell answer.
from tornado import gen
class ChatTestCase(AsyncHTTPTestCase):
def get_app(self):
return Application([
('/rt', RealtimeHandler),
])
#gen_test
def test_no_reply(self):
request = HTTPRequest('ws://127.0.0.1:%d/rt' % self.get_http_port())
ws = yield websocket_connect(request)
ws.write_message('Hi')
with self.assertRaises(gen.TimeoutError):
response = yield gen.with_timeout(datetime.timedelta(seconds=4), ws.read_message()

Function is not called using mock.patch

I am trying to test the Class BluetoothClient which connects to a BluetoothSocket. To avoid using real sockets I just want to test that the connect() method from the socket is called with the right parameters. Using mock.patch to replace the imported bluetooth module in my bluetooth_control module doesn't work out like expected.
As I see it, the connect() method is called but the assertion tells me otherwise.
Code:
Unit Under Test (bluetooth_control.py):
import bluetooth
class BluetoothClient(object):
def __init__(self):
self.address="98:D3:31:B2:EF:32"
self.port=1
def establishConnection(self):
self.createSocket()
self.connect()
def createSocket(self):
self.sock=bluetooth.BluetoothSocket( bluetooth.RFCOMM )
def connect(self):
print "connect: sock="+str(self.sock)
self.sock.connect((self.address, self.port))
Test (bluetooth_control_test.py):
import unittest
import mock
import bluetooth_control
import bluetooth
class TestShelf(unittest.TestCase):
def setUp(self):
unittest.TestCase.setUp(self)
self.bc = bluetooth_control.BluetoothClient()
print "setUp"
def tearDown(self):
self.shelf = None
print "tearDown"
#mock.patch('bluetooth_control.bluetooth')
def testEstablishConnection(self,mock_bluetooth):
self.bc.establishConnection()
print "testEstablishConnection sock="+str(self.bc.sock)
mock_bluetooth.connect().assert_called_with(self.bc.sock,("98:D3:31:B2:EF:32",1))
if __name__ == "__main__":
unittest.main()
Output:
setUp
connect: sock=<MagicMock name='bluetooth.BluetoothSocket()' id='140433322111504'>
testEstablishConnection sock=<MagicMock name='bluetooth.BluetoothSocket()' id='140433322111504'>
FtearDown
======================================================================
FAIL: testEstablishConnection (__main__.TestShelf)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/mock.py", line 1201, in patched
return func(*args, **keywargs)
File "bluetooth_control_test.py", line 21, in testEstablishConnection
mock_bluetooth.connect().assert_called_with(self.bc.sock,("98:D3:31:B2:EF:32",1))
File "/usr/lib/python2.7/site-packages/mock.py", line 831, in assert_called_with
raise AssertionError('Expected call: %s\nNot called' % (expected,))
AssertionError: Expected call: mock(<MagicMock name='bluetooth.BluetoothSocket()' id='140433322111504'>, ('98:D3:31:B2:EF:32', 1))
Not called
----------------------------------------------------------------------
Ran 1 test in 0.003s
FAILED (failures=1)

After looking at the problem again two days later I found the stupid mistakes I made. I had to patch the actual method and remove the falsely added brackets at the assertion.
I am not going to delete this question so that maybe it will help someone to avoid these mistakes.
Test
import unittest
import mock
import bluetooth_control
import bluetooth
class TestShelf(unittest.TestCase):
def setUp(self):
unittest.TestCase.setUp(self)
self.bc = bluetooth_control.BluetoothClient()
print "setUp"
def tearDown(self):
self.shelf = None
print "tearDown"
#mock.patch('bluetooth_control.bluetooth.BluetoothSocket.connect')
def testEstablishConnection(self,mock_connect):
self.bc.establishConnection()
print "testEstablishConnection sock="+str(self.bc.sock)
mock_connect.assert_called_with(("98:D3:31:B2:EF:32",1))
if __name__ == "__main__":
unittest.main()

Python unittest testing MongoDB randomly fails

Having a weird problem with Python's unittest and PyMongo. The test randomly succeeds or fails:
import unittest
from pymongo import Connection
from tractor import Tractor
class TestTractor(unittest.TestCase):
def setUp(self):
self.tractor = Tractor(1)
self.mongo = Connection()
self.db = self.mongo.tractor
self.db.classes.remove({'name': {'$regex':'^test_'}})
self.action_class_id = self.db.classes.insert({'name': 'test_action',
'metaclass': 'action'})
self.object_class_id = self.db.classes.insert({'name': 'test_object',
'metaclass': 'object'})
def tearDown(self):
self.tractor = None
def test_create_class(self):
cid1 = self.tractor.create_action_class('test_create_action_class')
cid2 = self.tractor.create_object_class('test_create_object_class')
self.assertNotEqual(cid1, None)
self.assertNotEqual(cid2, None)
action_obj = self.db.classes.find_one({'_id': cid1})
object_obj = self.db.classes.find_one({'_id': cid2})
self.assertNotEqual(cid1, cid2)
self.assertEqual(action_obj['_id'], cid1)
self.assertEqual(object_obj['_id'], cid2)
self.assertEqual(action_obj['name'], 'test_create_action_class')
self.assertEqual(object_obj['name'], 'test_create_object_class')
Class being tested:
from pymongo import Connection
from pymongo.objectid import ObjectId
class Tractor(object):
def __init__(self, uid):
self.uid = uid
self.mongo = Connection()
self.db = self.mongo.tractor
# Classes
def create_action_class(self, name):
return self.db.classes.insert({'name': name,
'attributes': [],
'metaclass': 'action'})
def create_object_class(self, name):
return self.db.classes.insert({'name': name,
'attributes': [],
'metaclass': 'object'})
Random behavior:
silver#aregh-6930-lnx ~/projects/traction/tractor $ python -m unittest discover
......ssEssssssssss
======================================================================
ERROR: test_create_class (tests.test_tractor.TestTractor)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/silver/projects/traction/tractor/tests/test_tractor.py", line 64, in test_create_class
self.assertEqual(action_obj['_id'], cid1)
TypeError: 'NoneType' object is not subscriptable
----------------------------------------------------------------------
Ran 19 tests in 0.023s
FAILED (errors=1, skipped=12)
...
silver#aregh-6930-lnx ~/projects/traction/tractor $ python -m unittest discover
......ss.ssssssssss
----------------------------------------------------------------------
Ran 19 tests in 0.015s
OK (skipped=12)
These two results randomly happen for the same test as I rerun the test without changing anything neither in the class nor in the test.
All of this runs on my machine and I know for sure that while running the test, nobody else tinkers neither with MongoDB nor with the code.
What gives?

I strongly suspect the problem here is that you are not using "safe" mode for your writes.
By default MongoDB uses "fire and forget" mode. This means that the insert command is sent to the server, but the driver doesn't check for any server responses.
When you switch to "safe" mode, the driver will send the insert command and it will then send a second command getLastError. This second command will return when the server has actually committed the write.
Again, by default you are running in "fire and forget" mode, so there is indeed a potential race condition here. For unit tests you will need to run with "safe" mode on.
The function signature for insert is defined here. However, you should also be able to make the change at the Connection level so that each connection to the DB uses "safe" mode by default.