Test still passes even with tests failure

[Lin Hua Cheng]

Tests seems still to pass even with test failures.

There are two tests failure on the code, and test run still report it passes:

https://bugs.launchpad.net/horizon/+bug/1517670
https://bugs.launchpad.net/horizon/+bug/1517653

[Lin Hua Cheng (lin-hua-cheng)]

david-lyle suggested that the impact of those calls are not verified by the test, so they aren't failing.

We should fix the asserts on those tests.

[Rob Cresswell (robcresswell-deactivatedaccount)]

@Lin: Going to close this, as we don't need a bug to track other bugs IMO.

[Lin Hua Cheng (lin-hua-cheng)]

@Rob: This isn't to track the other two bugs, but to track that the test are still passing even with some errors.

[Lin Hua Cheng (lin-hua-cheng)]

I like us to investigate if there a way to make the test fail if some mock fails.

[Lin Hua Cheng (lin-hua-cheng)]

another patch sneaked in test failures: https://bugs.launchpad.net/horizon/+bug/1522814

[Matthias Runge (mrunge)]

another one:
https://bugs.launchpad.net/horizon/+bug/1527575

[David Lyle (david-lyle)]

The last example Matthias is posting is a logged error that is handled in the d-o-a code. It's not actually a failure but shows up because it should be marked a warning. That was changed. But I don't think reporting failure based on that would be beneficial.

[Richard Jones (r1chardj0n3s)]

This bug report is too vague to be useful. It should identify specific failures that can be addressed.

[David Lyle (david-lyle)]

I'm taking this bug to mean that tests can pass with unmocked api calls. I have discovered that our code for catching such things is still looking at httplib2, which the clients all moved away from. Working on a fix.

[Lin Hua Cheng (lin-hua-cheng)]

David, your assumption is correct.

https://review.openstack.org/#/c/301862/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/301862
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=037d1c0927b64848ea0934990ba6538d9d7a8133
Submitter: Jenkins
Branch: master

commit 037d1c0927b64848ea0934990ba6538d9d7a8133
Author: David Lyle <email address hidden>
Date: Wed Mar 16 10:25:26 2016 -0600

    removing httplib2 test dependency

    Once upon a time, the python-*client libraries were primarily built to
    use httplib2. They have subsequently shift to using requests and thus
    urllib3. The horizon test helpers code was maintaining a reference to
    httplib2 as it intercepted errant library calls that were not mocked.

    httplib2 is not actively maintained and OpenStack is moving to remove it
    as a dependency. See
    http://lists.openstack.org/pipermail/openstack-dev/2016-March/089225.html
    for more details.

    This patch removed the httplib2 dependency. Upon removing the
    dependency it exposed a missed update from httplib2 to urllib3. A
    function that was intended to catch unmocked calls was only listening
    for httplib2 connections. This patch updates that failsafe to work with
    urllib3. Upon doing so, it pointed out many, many missing mocks and in
    turn, many broken tests that appeared to work because of API call
    failures. This patch adds the missing mocks and fixes the broken tests.

    The new failsafe prints the stack trace when an outside connection is
    attempted. Additionally, to fix the fact that a missed mock used to
    allow tests to potentially pass, as documented by bug 1517704, a test
    failure is now forced on tests where a missing mock is detected.

    Closes-Bug: #1517704
    Implements blueprint: remove-httplib2-dep
    Change-Id: Iaabdf03966c14c82e0c58a3b1ab1a6755c05adcb

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/horizon 10.0.0.0b1 development milestone.