Errors in code fail silently in unit tests

[Tihomir Trifonov]

Looking at a recent review (https://review.openstack.org/#/c/100656/), I've discovered that the unit test swallows some code errors and returns no error. The reason is that checking for Action.allowed() is enclosed in try:except, but the logged exception is not displayed in the unit test results. Here is the code:

https://github.com/openstack/horizon/blob/master/horizon/tables/base.py#L1140

When commenting the try:except block, the unit test failed, and here is the fix that was needed to bring tests back to passed:
http://paste.openstack.org/show/84431/

Let's see how to change unittest verbosity and make the error messages appear in unit tests result even if a test is passed. This is a must for at least LOG.exception messages.

[Gary W. Smith (gary-w-smith)]

If errors are occuring undetected in unit tests, then it is important to address this problem. But I disagree with the approach of logging all exceptions, even when the test passes; this causes false positives to appear in the unit test logs, such as those addressed by https://review.openstack.org/#/c/115368/.

A better approach would be to provide a way for the test to specify which log messages are expected, and to suppress those, yet permit all other errors to be displayed. In other words, something akin to https://docs.python.org/2/library/unittest.html#unittest.TestCase.assertRaisesRegexp, but where the regexp matches (and suppresses) the log message rather than the text of the exception itself.

Does that sound like a reasonable alternative?

[Gary W. Smith (gary-w-smith)]

As suggested in the review comments for https://review.openstack.org/#/c/115368, it would be appropriate to implement this via a context manager or your suggestion there of a decorator like:

@test.suppress_log(logging.CRITICAL, ("class1", "class2"))
    def test_method(self):