The Issue:
Dangling mock objects in global modules (mocked members of imported
modules that never get restored) have been causing various transient
failures in the unit test suite.
The main issues posed by dangling mock objects include:
- Such object references propagate across the entire test suite. Any
caller may be hit by a non-functional or worse crippled module member
because some other (potentially totally unrelated) test case failed to
restore it.
- Dangling mock references shared across different test modules may
lead to unexpected results/behavior in multi-threaded environments. One
example could be a test case failing because a mock got called multiple
times from unrelated modules.
Such issues are likely to exhibit transient random behavior depending
on the runtime environment making them difficult to debug.
This contribution is aiming to provide a simple transparent detection
layer that would help us prevent (most of) such issues.
Solution Strategy:
We extend the 'testtools.TestCase' class currently used as the base
class of Trove unit tests with functions that attempts to detect leaked
mock objects in imported modules. The new 'trove_testtools.TestCase'
implementation basically retrieves all loaded modules and scans their
members for mock objects before and after each test case.
An outline of the procedure follows:
1. Override the setUp() method and add a cleanup call (executed after
the tearDown()) to collect mock references before and after a test case.
2. Subtract the two sets after each test case and mark it as failed if
there are any new mock references left over.
Code Impact:
The existing test classes replace 'testtools.TestCase' base class with
'trove_testtools.TestCase'.
Documentation Impact:
Added a short document on recommended mocking strategies in unit tests
to the Trove Developer Documentation.
Known Limitations:
The current implementation has a configurable recursion depth which is
the number of nested levels to examine when searching for mocks.
Higher setting will potentially uncover more dangling objects,
at the cost of increased scanning time.
We set it to 2 by default to get better coverage.
This setting will increase test runtime.
Recommended Mocking Patterns:
Mock Guide: https://docs.python.org/3/library/unittest.mock.html
- Mocking a class or object shared across multiple test cases.
Use the patcher pattern in conjunction with the setUp() and tearDown()
methods [ see section 26.4.3.5. of Mock Guide ].
def setUp(self):
super(CouchbaseBackupTests, self).setUp()
self.exe_timeout_patch = patch.object(utils, 'execute_with_timeout')
def test_case(self):
# This line can be moved to the setUp() method if the mock object
# is not needed.
mock_object = self.exe_timeout_patch.start()
def tearDown(self):
super(CouchbaseBackupTests, self).tearDown()
self.exe_timeout_patch.stop()
Note also: patch.stopall()
Stop all active patches. Only stops patches started with start.
- Mocking a class or object for a single entire test case.
Use the decorator pattern.
@patch.object(utils, 'execute_with_timeout')
@patch.object(os, 'popen')
def test_case(self, popen_mock, execute_with_timeout_mock):
pass
@patch.multiple(utils, execute_with_timeout=DEFAULT,
generate_random_password=MagicMock(return_value=1))
def test_case(self, generate_random_password, execute_with_timeout):
pass
- Mocking a class or object for a smaller scope within one test case.
Use the context manager pattern.
def test_case(self):
# Some code using real implementation of 'generate_random_password'.
with patch.object(utils, 'generate_random_password') as pwd_mock:
# Using the mocked implementation of 'generate_random_password'.
# Again code using the actual implementation of the method.
def test_case(self):
with patch.multiple(utils, execute_with_timeout_mock=DEFAULT,
generate_random_password=MagicMock(
return_value=1)) as mocks:
password_mock = mocks['generate_random_password']
execute_mock = mocks['execute_with_timeout_mock']
Change-Id: Ia487fada249aa903410a1a3fb3f717d6e0d581d2
Closes-Bug: 1447833
4899f49401