a broken configure_tempest.sh can lead to silent failures

[aeva black]

Within runs of devstack-gate, devstack/tools/configure_tempest.sh was not finding a valid FLAVOR id for the last few weeks, because of bug:

https://bugs.launchpad.net/devstack/+bug/1037347

This should have been noticed immediately, and that patch should never have landed, but tempest is suppressing errors that happen during init, and converting them to nose.SkipTest messages, which leads Jenkins to still report the test as a success! This seems to be coming from here:
https://github.com/openstack/tempest/blob/master/tempest/tests/compute/__init__.py#L62

To see the output of such a suppressed error, look at any gate-tempest-devstack-vm console log since the above patch landed. For example:
https://jenkins.openstack.org/job/gate-tempest-devstack-vm/4857/console

I think tempest should not be suppressing errors here in a way that makes Jenkins think things pass when they don't. What's the reason for it?

[Matthew Treinish (treinish)]

There have been some devstack tempest changes that fixed the underlying issue causing the failures. But, I agree just changing that skip test to an error on a broken configuration is probably the better way to handle it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/19541
Committed: http://github.com/openstack/tempest/commit/80a8d4e887924c54f2d1e8078e5705d462d256f8
Submitter: Jenkins
Branch: master

commit 80a8d4e887924c54f2d1e8078e5705d462d256f8
Author: Frederic Lepied <email address hidden>
Date: Sat Jan 12 12:43:52 2013 +0100

    Don't ignore exceptions.

    - Raise an exception when there is a configuration error.
    - Let exceptions raise themself.

    Fixes: bug #1037354
    Change-Id: I7e064bf95d6dbce7a52a010411672c237e77afae