API test listing is empty if plugin not configured or auth_url not available

[Cliff Parsons]

If the heat-tempest-plugin is installed, and the command "tempest run --list-tests" is issued, the Heat API (Gabbi) tests will not be listed if the auth_url is unreachable/mis-provisioned, or if the key authorization-related variables are not provisioned in the tempest.conf file. The API tests should be listed regardless of whether or not the plugin can authenticate using the auth service (keystone).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/544025

[Rabi Mishra (rabi)]

Gabbi does not allow for building/discovering tests without a live host or an wsgi intercept[1]

Though you can bypass that by passing dummy host/url, it would fail with undecipherable errors when running a discovered test (without the tempest.conf being configured for heat_plugin).

ex.
AssertionError: unable to replace $ENVIRON in $ENVIRON['OS_TOKEN'], data unavailable: 'OS_TOKEN'

Therefore I'm not sure if passing placeholders to gabbi is a good idea.

[1] https://github.com/cdent/gabbi/blob/master/gabbi/driver.py#L86

[Cliff Parsons (cliffhparsons)]

Thank you, Rabi, for your feedback!

I see that point of view clearly, however, in the solution that I coded, I have included warning logs for the cases where the auth information was not properly included in the configuration file. These warning logs print out the detailed exception information that was caught from Keystone, and should be enough to figure out what the problem is. So while I agree that the error you point out above is fairly undecipherable, the log information that comes with it should help figure out the problem.

[Cliff Parsons (cliffhparsons)]

After some discussion with the core Heat group members, Zane Bitter coded up a fix within Gabbi for this issue. https://github.com/cdent/gabbi/commit/01993966c179791186977e27c64b9e525a566408

This is working great, so it is acceptable for our purposes, but we would like to get the requirements.txt file updated so that this new version of Gabbi (1.42.0) will get installed automatically when heat-tempest-plugin is installed. I will create a new commit for that change.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/561709

[OpenStack Infra (hudson-openstack)]

Change abandoned by Cliff Parsons (<email address hidden>) on branch: master
Review: https://review.openstack.org/544025
Reason: Abandonding this commit because a fix has been made to the Gabbi module: https://github.com/cdent/gabbi/commit/01993966c179791186977e27c64b9e525a566408#diff-aa0920de5c2971110a3663b2f9d3e513

[OpenStack Infra (hudson-openstack)]

Change abandoned by Cliff Parsons (<email address hidden>) on branch: master
Review: https://review.openstack.org/561709
Reason: No longer can use this version of gabbi. and also including the gabbi version update as part of https://review.openstack.org/#/c/544025/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/544025
Committed: https://git.openstack.org/cgit/openstack/heat-tempest-plugin/commit/?id=a08034e3d94d37db02d093f0f70150450a338e42
Submitter: Zuul
Branch: master

commit a08034e3d94d37db02d093f0f70150450a338e42
Author: Cliff Parsons <email address hidden>
Date: Tue Feb 13 11:20:28 2018 -0600

    Force API test listing even if auth fails

    If the heat-tempest-plugin is installed, and the command
    "tempest run --list-tests" is issued, the Heat API (Gabbi) tests
    will not be listed if the auth_url is unreachable/mis-provisioned,
    or if the key authorization-related variables are not provisioned
    in the tempest.conf file. The API tests should be listed regardless
    of whether or not the plugin can authenticate using keystone.

    In the event that the plugin is not configured (or not configured
    correctly), or if keystone is not reachable, this commit combined
    with the following gabbi fix:

    https://github.com/cdent/gabbi/commit/8354d545471f5a2dcdff7e5c05d298b2afd8e6ae

    ensures that Gabbi build_tests() returns the list of api tests,
    even when keystone cannot be reached due to whatever reason above.

    Note that even while running the tests, if the authorization
    information is not present or is incorrect, the user will be able
    to see the warning logs with the keystoneauth exceptions present.
    However, the exceptions will not prevent a test listing from being
    given to the user.

    Change-Id: I3921f0df245d3745b95d3a312c41b9f9b15c2ba9
    Closes-Bug: #1749218

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat-tempest-plugin 0.2.0 release.