Sporadic share type cleanup failure in tests in ShareIpRulesForNFSNegativeTest

[Goutham Pacha Ravi]

Share type failures are occurring randomly on the CI:

Dummy Driver failure:
http://logs.openstack.org/77/428077/3/check/gate-manila-tempest-minimal-dsvm-dummy-ubuntu-xenial/eee927a/console.html

2017-02-02 20:28:27.592648 | 2017-02-02 20:28:27.592 | ==============================
2017-02-02 20:28:27.594770 | 2017-02-02 20:28:27.594 | Failed 1 tests - output below:
2017-02-02 20:28:27.596459 | 2017-02-02 20:28:27.596 | ==============================
2017-02-02 20:28:27.598547 | 2017-02-02 20:28:27.598 |
2017-02-02 20:28:27.600783 | 2017-02-02 20:28:27.600 | tearDownClass (manila_tempest_tests.tests.api.test_rules_negative.ShareIpRulesForNFSNegativeTest)
2017-02-02 20:28:27.602993 | 2017-02-02 20:28:27.602 | -------------------------------------------------------------------------------------------------
2017-02-02 20:28:27.605258 | 2017-02-02 20:28:27.604 |
2017-02-02 20:28:27.607614 | 2017-02-02 20:28:27.607 | Captured traceback:
2017-02-02 20:28:27.610262 | 2017-02-02 20:28:27.609 | ~~~~~~~~~~~~~~~~~~~
2017-02-02 20:28:27.612734 | 2017-02-02 20:28:27.612 | Traceback (most recent call last):
2017-02-02 20:28:27.615194 | 2017-02-02 20:28:27.614 | File "tempest/test.py", line 311, in tearDownClass
2017-02-02 20:28:27.617565 | 2017-02-02 20:28:27.617 | six.reraise(etype, value, trace)
2017-02-02 20:28:27.619953 | 2017-02-02 20:28:27.619 | File "tempest/test.py", line 294, in tearDownClass
2017-02-02 20:28:27.622183 | 2017-02-02 20:28:27.621 | teardown()
2017-02-02 20:28:27.624300 | 2017-02-02 20:28:27.623 | File "/opt/stack/new/manila/manila_tempest_tests/tests/api/base.py", line 278, in resource_cleanup
2017-02-02 20:28:27.626489 | 2017-02-02 20:28:27.626 | cls.clear_resources(cls.class_resources)
2017-02-02 20:28:27.628426 | 2017-02-02 20:28:27.628 | File "/opt/stack/new/manila/manila_tempest_tests/tests/api/base.py", line 884, in clear_resources
2017-02-02 20:28:27.630769 | 2017-02-02 20:28:27.630 | client.delete_share_type(res_id)
2017-02-02 20:28:27.633144 | 2017-02-02 20:28:27.632 | File "/opt/stack/new/manila/manila_tempest_tests/services/share/v2/json/shares_client.py", line 814, in delete_share_type
2017-02-02 20:28:27.634430 | 2017-02-02 20:28:27.634 | resp, body = self.delete("types/%s" % share_type_id, version=version)
2017-02-02 20:28:27.636178 | 2017-02-02 20:28:27.635 | File "/opt/stack/new/manila/manila_tempest_tests/services/share/v2/json/shares_client.py", line 75, in delete
2017-02-02 20:28:27.637759 | 2017-02-02 20:28:27.637 | body=body)
2017-02-02 20:28:27.639618 | 2017-02-02 20:28:27.639 | File "tempest/lib/common/rest_client.py", line 307, in delete
2017-02-02 20:28:27.641557 | 2017-02-02 20:28:27.641 | return self.request('DELETE', url, extra_headers, headers, body)
2017-02-02 20:28:27.643179 | 2017-02-02 20:28:27.642 | File "tempest/lib/common/rest_client.py", line 665, in request
2017-02-02 20:28:27.645504 | 2017-02-02 20:28:27.645 | resp, resp_body)
2017-02-02 20:28:27.647390 | 2017-02-02 20:28:27.646 | File "tempest/lib/common/rest_client.py", line 768, in _error_checker
2017-02-02 20:28:27.649163 | 2017-02-02 20:28:27.648 | raise exceptions.BadRequest(resp_body, resp=resp)
2017-02-02 20:28:27.651156 | 2017-02-02 20:28:27.650 | tempest.lib.exceptions.BadRequest: Bad request
2017-02-02 20:28:27.653464 | 2017-02-02 20:28:27.652 | Details: {u'message': u'Target share type is still in use.', u'code': 400}

Drilling down into the tempest log:

We're trying to cleanup the share when it's still in "creating" state:

    Response - Headers: {'x-openstack-manila-api-version': '2.32', 'connection': 'close', 'date': 'Thu, 02 Feb 2017 20:20:21 GMT', 'content-type': 'application/json; charset=UTF-8', 'content-length': '121', 'status': '403', 'content-location': 'http://149.202.163.122:8786/v2/e8808f5ac36a471da87da86a6813f7f2/shares/a0f972a6-013d-4e24-b97b-065f995037c8', 'x-compute-request-id': 'req-de11b8b0-c36e-40b9-a8a5-53d428bbf99d', 'vary': 'X-OpenStack-Manila-API-Version'}
        Body: {"forbidden": {"message": "Invalid share: Share status must be one of ('available', 'error', 'inactive').", "code": 403}} _log_request_full tempest/lib/common/rest_client.py:431
2017-02-02 20:20:21.830 18443 ERROR manila_tempest_tests.tests.api.base [-] Suppressed cleanup error in Manila:
Traceback (most recent call last):
  File "/opt/stack/new/manila/manila_tempest_tests/tests/api/base.py", line 872, in clear_resources
    client.delete_share(res_id)
  File "/opt/stack/new/manila/manila_tempest_tests/services/share/v2/json/shares_client.py", line 285, in delete_share
    resp, body = self.delete(uri, version=version)
  File "/opt/stack/new/manila/manila_tempest_tests/services/share/v2/json/shares_client.py", line 75, in delete
    body=body)
  File "tempest/lib/common/rest_client.py", line 307, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "tempest/lib/common/rest_client.py", line 665, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 758, in _error_checker
    raise exceptions.Forbidden(resp_body, resp=resp)
Forbidden: Forbidden
Details: {u'message': u"Invalid share: Share status must be one of ('available', 'error', 'inactive').", u'code': 403}

This isn't the intent of the test "test_add_access_rule_on_share_with_no_host"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/428771

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/428771
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=ba3088c4ee2415c44673eb0b82e45ed0aa5e7df2
Submitter: Jenkins
Branch: master

commit ba3088c4ee2415c44673eb0b82e45ed0aa5e7df2
Author: Goutham Pacha Ravi <email address hidden>
Date: Thu Feb 2 18:10:35 2017 -0500

    Fix wrong access-rule negative test

    The test test_add_access_rule_on_share_with_no_host in
    manila_tempest_tests.tests.api.test_rules_negative\
           .ShareIpRulesForNFSNegativeTest
    expects a share to fail to schedule, and then validates that an
    access rule cannot be added. However, the share may
    still be 'creating' when the access change is being
    attempted.

    While the test passes (access rules cannot be added when
    the share status isn't 'available'), the intent of that particular
    test is to ensure access rules cannot be added when the share has no
    host.

    This fixes the cleanup error that happens in consequence,
    because we don't allow deleting shares that are in
    'creating' state.

    In the spirit of true black box testing, we should ensure that the
    share fails to schedule before attempting this test.

    Change-Id: Id150e0ef546ddfd40d97d9b1a54fd8ba23028720
    Closes-Bug: #1661421

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 4.0.0.0rc1 release candidate.