tempest.api.compute.servers.test_servers.ServerShowV263Test only passing when image has already been cached

[Lee Yarwood]

tempest.api.compute.servers.test_servers.ServerShowV263Test aims to test the optional trusted_image_certificates parameter introduced into n-api in microversion 2.63:

Add trusted_image_certificates to REST API
https://review.opendev.org/#/c/486204/

Add new schema for Nova microversion 2.63
https://review.opendev.org/#/c/526485/

However this test currently relies on other tests having cached the image used to create the test instance before it is run. Otherwise the underlying logic within Nova [1] to validate the image using the provided certificates will be executed and fail as the image is missing image properties such as img_signature_hash_method etc [2].

[1] https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L402-L460
[2] https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2019-05-31_07_06_11_779

This behaviour can be easily reproduced in devstack by cleaning out the local image cache and running tempest.api.compute.servers.test_servers.ServerShowV263Test in isolation:

$ rm -rf ~/data/nova/instances/_base/*
$ tempest run --regex tempest.api.compute.servers.test_servers.ServerShowV263Test
[..]
==============================
Failed 1 tests - output below:
==============================

tempest.api.compute.servers.test_servers.ServerShowV263Test.test_show_update_rebuild_list_server[id-71b8e3d5-11d2-494f-b917-b094a4afed3c]
-------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "tempest/api/compute/servers/test_servers.py", line 194, in test_show_update_rebuild_list_server
        wait_until='ACTIVE')
      File "tempest/api/compute/base.py", line 246, in create_test_server
        **kwargs)
      File "tempest/common/compute.py", line 265, in create_test_server
        server['id'])
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
        self.force_reraise()
      File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
        six.reraise(self.type_, self.value, self.tb)
      File "tempest/common/compute.py", line 236, in create_test_server
        clients.servers_client, server['id'], wait_until)
      File "tempest/common/waiters.py", line 76, in wait_for_server_status
        server_id=server_id)
    tempest.exceptions.BuildErrorException: Server 254fb4a1-74a8-4681-ad67-a287bb3cde66 failed to build and is in ERROR status
    Details: {u'message': u'Build of instance 254fb4a1-74a8-4681-ad67-a287bb3cde66 aborted: Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_hash_method.', u'code': 500, u'created': u'2019-06-06T08:38:11Z'}

Running all of the test_servers tests together however typically allows this to pass:

$ rm -rf ~/data/nova/instances/_base/*
$ tempest run --regex tempest.api.compute.servers.test_servers
[..]
======
Totals
======
Ran: 59 tests in 150.0000 sec.
 - Passed: 58
 - Skipped: 1
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 522.7800 sec.

I'm going to tag nova into this bug report as well as I'm not sure that the current behaviour of n-api is correct. At present we attempt to verify the image when the optional trusted_image_certificates parameter is provided, regardless of CONF.glance.verify_glance_signatures being enabled within nova.conf:

https://github.com/openstack/nova/blob/6009cdaa47a10f6d06a31b68fdc6a263bced13ad/nova/image/glance.py#L414

While documented that behaviour doesn't seem right and I believe led to some incorrect assumptions being made by the initial writer of this test.

https://docs.openstack.org/nova/latest/user/certificate-validation.html

"Certificate validation will only be performed if image signature validation is enabled. However, the presence of trusted certificate IDs overrides the enable_certificate_validation and verify_glance_signatures settings. In other words, if a list of trusted certificate IDs is provided to the instance create or rebuild commands, signature verification and certificate validation will be performed, regardless of their settings in the Nova configurations. See Using Signature Verification for details."

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/663596

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663596
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=7f7c232b29fe4e3f1cb446a810112db43d7dccdc
Submitter: Zuul
Branch: master

commit 7f7c232b29fe4e3f1cb446a810112db43d7dccdc
Author: Lee Yarwood <email address hidden>
Date: Thu Jun 6 10:57:19 2019 +0100

    compute: Make image certificate validation a configurable feature

    tempest.api.compute.servers.test_servers.ServerShowV263Test would
    previously always run and provide the optional
    trusted_image_certificates parameter to validate the 2.63 API
    microversion. However if the test image had not been cached locally this
    would actually cause the underlying certificate validation logic within
    Nova to be called resulting in the test failing as the test image would
    not have the required img_signature_* properties set and the provided
    trusted certificates are just test strings.

    This change makes this specific test optional (not running by default)
    now by introducing two required compute configurables. These allow a
    signed image with the required img_signature_* properties set along with
    a valid list of trusted image certificates to also be used during the
    test.

    Ica1d32ca66d6f89692e9cc90e415bf569332f58d is currently attempting to
    wire up various image certificate and signature validation tests under
    the barbican-tempest-plugin.

    Change-Id: Ib346d383c430d5151d9aafa6e856dd0a7cae8a23
    Closes-bug: 1831866

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 21.0.0 release.