Test still passes even with tests failure

Bug #1517704 reported by Lin Hua Cheng on 2015-11-19
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Dashboard (Horizon)
High
David Lyle

Bug Description

Tests seems still to pass even with test failures.

There are two tests failure on the code, and test run still report it passes:

https://bugs.launchpad.net/horizon/+bug/1517670
https://bugs.launchpad.net/horizon/+bug/1517653

Lin Hua Cheng (lin-hua-cheng) wrote :

david-lyle suggested that the impact of those calls are not verified by the test, so they aren't failing.

We should fix the asserts on those tests.

Rob Cresswell (robcresswell) wrote :

@Lin: Going to close this, as we don't need a bug to track other bugs IMO.

Changed in horizon:
status: New → Invalid
Lin Hua Cheng (lin-hua-cheng) wrote :

@Rob: This isn't to track the other two bugs, but to track that the test are still passing even with some errors.

Lin Hua Cheng (lin-hua-cheng) wrote :

I like us to investigate if there a way to make the test fail if some mock fails.

Changed in horizon:
status: Invalid → New
Lin Hua Cheng (lin-hua-cheng) wrote :

another patch sneaked in test failures: https://bugs.launchpad.net/horizon/+bug/1522814

Changed in horizon:
importance: Undecided → High
milestone: none → mitaka-2
importance: High → Critical
Changed in horizon:
milestone: mitaka-2 → mitaka-3
David Lyle (david-lyle) wrote :

The last example Matthias is posting is a logged error that is handled in the d-o-a code. It's not actually a failure but shows up because it should be marked a warning. That was changed. But I don't think reporting failure based on that would be beneficial.

David Lyle (david-lyle) on 2016-03-04
Changed in horizon:
importance: Critical → High
Richard Jones (r1chardj0n3s) wrote :

This bug report is too vague to be useful. It should identify specific failures that can be addressed.

Changed in horizon:
status: New → Incomplete
David Lyle (david-lyle) wrote :

I'm taking this bug to mean that tests can pass with unmocked api calls. I have discovered that our code for catching such things is still looking at httplib2, which the clients all moved away from. Working on a fix.

Changed in horizon:
assignee: nobody → David Lyle (david-lyle)
status: Incomplete → Confirmed
status: Confirmed → Triaged
milestone: mitaka-3 → mitaka-rc1
David Lyle (david-lyle) on 2016-03-16
Changed in horizon:
milestone: mitaka-rc1 → newton-1
Changed in horizon:
status: Triaged → In Progress
Changed in horizon:
assignee: David Lyle (david-lyle) → Timur Sufiev (tsufiev-x)
Timur Sufiev (tsufiev-x) on 2016-04-27
Changed in horizon:
assignee: Timur Sufiev (tsufiev-x) → David Lyle (david-lyle)
Lin Hua Cheng (lin-hua-cheng) wrote :

David, your assumption is correct.

https://review.openstack.org/#/c/301862/

Reviewed: https://review.openstack.org/301862
Committed: https://git.openstack.org/cgit/openstack/horizon/commit/?id=037d1c0927b64848ea0934990ba6538d9d7a8133
Submitter: Jenkins
Branch: master

commit 037d1c0927b64848ea0934990ba6538d9d7a8133
Author: David Lyle <email address hidden>
Date: Wed Mar 16 10:25:26 2016 -0600

    removing httplib2 test dependency

    Once upon a time, the python-*client libraries were primarily built to
    use httplib2. They have subsequently shift to using requests and thus
    urllib3. The horizon test helpers code was maintaining a reference to
    httplib2 as it intercepted errant library calls that were not mocked.

    httplib2 is not actively maintained and OpenStack is moving to remove it
    as a dependency. See
    http://lists.openstack.org/pipermail/openstack-dev/2016-March/089225.html
    for more details.

    This patch removed the httplib2 dependency. Upon removing the
    dependency it exposed a missed update from httplib2 to urllib3. A
    function that was intended to catch unmocked calls was only listening
    for httplib2 connections. This patch updates that failsafe to work with
    urllib3. Upon doing so, it pointed out many, many missing mocks and in
    turn, many broken tests that appeared to work because of API call
    failures. This patch adds the missing mocks and fixes the broken tests.

    The new failsafe prints the stack trace when an outside connection is
    attempted. Additionally, to fix the fact that a missed mock used to
    allow tests to potentially pass, as documented by bug 1517704, a test
    failure is now forced on tests where a missing mock is detected.

    Closes-Bug: #1517704
    Implements blueprint: remove-httplib2-dep
    Change-Id: Iaabdf03966c14c82e0c58a3b1ab1a6755c05adcb

Changed in horizon:
status: In Progress → Fix Released

This issue was fixed in the openstack/horizon 10.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers