[rocky][fs020] test_resize_server_revert_with_volume_attached and test_show_update_rebuild_list_server failing after moving to 2 compute nodes with Virtual Interface creation failed

[chandan kumar]

Earlier all the fs020 promotion jobs are using 1 compute nodes and
https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky/631cdc8/logs/tempest.html.gz
test_resize_server_revert_with_volume_attached -> skipped
test_show_update_rebuild_list_server -> passing

recently we have updated the fs020 jobs to use 2 compute nodes then
https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/tempest.html.gz
and both the tests started failing
test_resize_server_revert_with_volume_attached-
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_server_actions.py", line 66, in tearDown
    self.server_check_teardown()
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/base.py", line 206, in server_check_teardown
    cls.server_id)
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 124, in wait_for_server_termination
    raise lib_exc.DeleteErrorException(resource_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a failed to delete and is in ERROR status
}}}

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/lib/decorators.py", line 112, in wrapper
    raise exc
tempest.exceptions.BuildErrorException: Server 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a failed to build and is in ERROR status
Details: {u'message': u'Virtual Interface creation failed', u'code': 500, u'created': u'2019-05-31T07:23:24Z'}

test_show_update_rebuild_list_server
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tempest/api/compute/servers/test_servers.py", line 236, in test_show_update_rebuild_list_server
    server['id'], 'ACTIVE')
  File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server 8d66eb24-8be3-4838-85c0-c12b89606add failed to build and is in ERROR status
Details: {u'message': u'Signature verification for the image failed: Required image properties for signature verification do not exist. Cannot verify signature. Missing property: img_signature_hash_method.', u'code': 500, u'created': u'2019-05-31T07:06:11Z'}

From nova compute logs on compute 1 it appears both are failures are interrelated:
https://logs.rdoproject.org/openstack-periodic-24hr/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky/207e8c1/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz?level=ERROR#_2019-05-31_07_23_24_654

2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [req-da2e5b88-baa4-4f69-ab6b-4c8f65d0fdc3 740ed86c929746909aa4941dc3fe7386 a2c98268aeef429dac06d5c54e0f0afb - default default] [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] Setting instance vm_state to ERROR: VirtualInterfaceCreateException: Virtual Interface creation failed
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] Traceback (most recent call last):
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8037, in _error_out_instance_on_exception
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] yield
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4179, in finish_revert_resize
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] block_device_info, power_on)
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8491, in finish_revert_migration
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] power_on=power_on)
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5662, in _create_domain_and_network
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] raise exception.VirtualInterfaceCreateException()
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a] VirtualInterfaceCreateException: Virtual Interface creation failed
2019-05-31 07:23:24.654 8 ERROR nova.compute.manager [instance: 6bcd6b53-ad34-4a99-99d6-8c3fba2f837a]
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server [req-da2e5b88-baa4-4f69-ab6b-4c8f65d0fdc3 740ed86c929746909aa4941dc3fe7386 a2c98268aeef429dac06d5c54e0f0afb - default default] Exception during message handling: VirtualInterfaceCreateException: Virtual Interface creation failed
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 79, in wrapped
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server function_name, call_dict, binary, tb)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 69, in wrapped
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 158, in decorated_function
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 1141, in decorated_function
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 146, in decorated_function
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server kwargs['instance'], e, sys.exc_info())
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4179, in finish_revert_resize
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server block_device_info, power_on)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 8491, in finish_revert_migration
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server power_on=power_on)
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5662, in _create_domain_and_network
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server raise exception.VirtualInterfaceCreateException()
2019-05-31 07:23:25.060 8 ERROR oslo_messaging.rpc.server VirtualInterfaceCreateException: Virtual Interface creation failed

[chandan kumar (chkumar246)]

Image signature verification is used with nova, barbican and glance, it is failing while server goes to rebuild_spawning then it goes to error state.

[Ade Lee (alee-3)]

Gotta get a glance/nova person to look at this. I have no idea why the above test is even checking for signatures

-- in the nova.conf file:
   verify_glance_signatures is set to False

-- there is no barbican in the setup

[Marios Andreou (marios-b)]

not a promotion blocker since this periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-rocky isn't in promotion criteria afaics https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/rocky.ini replaced by periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky

remove promotion blocker?

[chandan kumar (chkumar246)]

The failure is coming in https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config/CentOS-7/rocky.ini#L25 periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-rocky job. It is coming after we switched to 2 compute nodes.

[wes hayutin (weshayutin)]

FYI.. 2 compute nodes should be the new standard job in promotion criteria across releases.

[Abhishek Kekane (abhishek-kekane)]

AFAIK, the test uses trusted_image_certificates while creating the server which is used to define a list of trusted certificate IDs that can be used during image signature verification and certificate validation. This parameter is optional in server create requests if allowed by policy, and is not supported for volume-backed instances.

As signature verification from nova requires either trusted_certificates or verify_glance_signatures set to True in nova.conf [1] and trusted_certificates are set in above failing tests [2] it does not have any thing to do with glance (glance validates signature only while uploading the data to image).

[1] https://github.com/openstack/nova/blame/master/nova/image/glance.py#L414
[2] https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_servers.py#L190

[Matthew Booth (mbooth-9)]

The VIF creation failure is a known issue. It presumably just started failing because the test requires 2 computes to run:

    @decorators.idempotent_id('c03aab19-adb1-44f5-917d-c419577e9e68')
    @testtools.skipUnless(CONF.compute_feature_enabled.resize,
                          'Resize not available.')
    def test_resize_server_revert(self):

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/663525

[Lee Yarwood (lyarwood)]

I've created https://bugs.launchpad.net/nova/+bug/1831866 to track the tempest.api.compute.servers.test_servers.ServerShowV263Test race.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663525
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=366634808a86e67a7ee51a048e3adcd4550f4cd3
Submitter: Zuul
Branch: master

commit 366634808a86e67a7ee51a048e3adcd4550f4cd3
Author: Chandan Kumar (raukadah) <email address hidden>
Date: Thu Jun 6 12:36:43 2019 +0530

    Add resize_server_revert and rebuild_list_server to skip list

    Since The VIF creation failure is a known issue as per the bug
    https://bugzilla.redhat.com/show_bug.cgi?id=1678681

    tempest.api.compute.servers.test_servers.ServerShowV263Test.
    test_show_update_rebuild_list_server test is also failing due
    to https://bugs.launchpad.net/nova/+bug/1831866

    Adding it to skip list to unblock promotion pipeline.

    Related-Bug: #1831223

    Change-Id: I75696049a2776fbe8693b2aaef71f2342b52b387
    Signed-off-by: Chandan Kumar (raukadah) <email address hidden>

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/663869

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/699394
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=f1b00f047569913eb33fe67c6cf714ca0e8b4373
Submitter: Zuul
Branch: master

commit f1b00f047569913eb33fe67c6cf714ca0e8b4373
Author: Soniya Vyas <email address hidden>
Date: Fri Jan 3 15:40:56 2020 +0530

    [Rocky] Removed passing tests from skiplist

    Class neutron_tempest_plugin.scenario has maximum
    number of tests passed. Hence, it is removed and
    only failing testsis kept.

    In addition to above, added two more tests from
    class 'TestSecurityGroupsBasicOps' and a test from
    class 'ServersOnMultiNodesTest'

    Related-bug: #1737940
    Related-bug: #1753209
    Related-bug: #1843259
    Related-bug: #1793482
    Related-bug: #1831223
    Related-bug: #1857365

    Signed-off by: Soniya Vyas<email address hidden>
    Change-Id: I8239bb694187d7f912163742e836a7362cdb1483