Gate tempest dsvm neutron dvr test fails

[Daniel Alvarez]

The following tests are failing in the neutron gate:

tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_active_server [6.911205s] ... FAILED
(tempest.api.compute.servers.test_server_addresses_negative.ServerAddressesNegativeTestJSON) [0.000000s] ... FAILED
tempest.api.compute.servers.test_delete_server.DeleteServersTestJSON.test_delete_server_while_in_attached_volume [3.348451s] ... FAILED
tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_duplicate_network_nics [8.901531s] ... FAILED

I spotted this message in logs [1]:
"Connection to the hypervisor is broken on host: ubuntu-xenial-osic-cloud1-disk-6162583"

Tracebacks:

2016-12-14 10:34:36.039551 | Captured traceback:
2016-12-14 10:34:36.039562 | ~~~~~~~~~~~~~~~~~~~
2016-12-14 10:34:36.039577 | Traceback (most recent call last):
2016-12-14 10:34:36.039611 | File "tempest/api/compute/servers/test_delete_server.py", line 49, in test_delete_active_server
2016-12-14 10:34:36.039634 | waiters.wait_for_server_termination(self.client, server['id'])
2016-12-14 10:34:36.039658 | File "tempest/common/waiters.py", line 111, in wait_for_server_termination
2016-12-14 10:34:36.039693 | raise exceptions.BuildErrorException(server_id=server_id)
2016-12-14 10:34:36.039728 | tempest.exceptions.BuildErrorException: Server e127e6ff-c7bb-43a2-bbe9-c2683ffdf018 failed to build and is in ERROR status

2016-12-14 10:34:36.043578 | Captured traceback:
2016-12-14 10:34:36.043588 | ~~~~~~~~~~~~~~~~~~~
2016-12-14 10:34:36.043602 | Traceback (most recent call last):
2016-12-14 10:34:36.043619 | File "tempest/test.py", line 100, in wrapper
2016-12-14 10:34:36.043636 | return f(self, *func_args, **func_kwargs)
2016-12-14 10:34:36.043668 | File "tempest/api/compute/servers/test_delete_server.py", line 110, in test_delete_server_while_in_attached_volume
2016-12-14 10:34:36.043687 | server = self.create_test_server(wait_until='ACTIVE')
2016-12-14 10:34:36.043709 | File "tempest/api/compute/base.py", line 232, in create_test_server
2016-12-14 10:34:36.043718 | **kwargs)
2016-12-14 10:34:36.043752 | File "tempest/common/compute.py", line 167, in create_test_server
2016-12-14 10:34:36.043763 | % server['id'])
2016-12-14 10:34:36.043798 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-14 10:34:36.043810 | self.force_reraise()
2016-12-14 10:34:36.043846 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-14 10:34:36.043864 | six.reraise(self.type_, self.value, self.tb)
2016-12-14 10:34:36.043886 | File "tempest/common/compute.py", line 149, in create_test_server
2016-12-14 10:34:36.043905 | clients.servers_client, server['id'], wait_until)
2016-12-14 10:34:36.043927 | File "tempest/common/waiters.py", line 75, in wait_for_server_status
2016-12-14 10:34:36.043939 | server_id=server_id)
2016-12-14 10:34:36.044257 | tempest.exceptions.BuildErrorException: Server b1472499-6bdc-41fb-98ca-9d1f9ef578ed failed to build and is in ERROR status
2016-12-14 10:34:36.044301 | Details: {u'code': 500, u'created': u'2016-12-14T10:05:22Z', u'message': u'No valid host was found. There are not enough hosts available.'}

2016-12-14 10:34:36.039827 | Captured traceback:
2016-12-14 10:34:36.039838 | ~~~~~~~~~~~~~~~~~~~
2016-12-14 10:34:36.039852 | Traceback (most recent call last):
2016-12-14 10:34:36.039870 | File "tempest/test.py", line 241, in setUpClass
2016-12-14 10:34:36.039885 | six.reraise(etype, value, trace)
2016-12-14 10:34:36.039903 | File "tempest/test.py", line 234, in setUpClass
2016-12-14 10:34:36.039915 | cls.resource_setup()
2016-12-14 10:34:36.039944 | File "tempest/api/compute/servers/test_server_addresses_negative.py", line 36, in resource_setup
2016-12-14 10:34:36.039972 | cls.server = cls.create_test_server(wait_until='ACTIVE')
2016-12-14 10:34:36.039995 | File "tempest/api/compute/base.py", line 232, in create_test_server
2016-12-14 10:34:36.040005 | **kwargs)
2016-12-14 10:34:36.040027 | File "tempest/common/compute.py", line 167, in create_test_server
2016-12-14 10:34:36.040038 | % server['id'])
2016-12-14 10:34:36.040083 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-12-14 10:34:36.040107 | self.force_reraise()
2016-12-14 10:34:36.040143 | File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-12-14 10:34:36.040161 | six.reraise(self.type_, self.value, self.tb)
2016-12-14 10:34:36.040183 | File "tempest/common/compute.py", line 149, in create_test_server
2016-12-14 10:34:36.040202 | clients.servers_client, server['id'], wait_until)
2016-12-14 10:34:36.040224 | File "tempest/common/waiters.py", line 75, in wait_for_server_status
2016-12-14 10:34:36.040237 | server_id=server_id)
2016-12-14 10:34:36.040271 | tempest.exceptions.BuildErrorException: Server b053e199-6492-4e5d-a0b7-900e3df43b6d failed to build and is in ERROR status
2016-12-14 10:34:36.040329 | Details: {u'code': 500, u'created': u'2016-12-14T10:05:21Z', u'message': u'Build of instance b053e199-6492-4e5d-a0b7-900e3df43b6d aborted: Connection to the hypervisor is broken on host: ubuntu-xenial-osic-cloud1-disk-6162583'}

[1] http://logs.openstack.org/40/402140/14/check/gate-tempest-dsvm-neutron-dvr-ubuntu-xenial/9a353ab/console.html

[Brian Haley (brian-haley)]

I think this is related to the libvirtd issues we've been seeing on xenial, https://bugs.launchpad.net/nova/+bug/1643911

I don't see the exact footprint in the syslog for this job, but the "Connection to the hypervisor is broken on host" is telling.

Actually, syslog does have this:

Dec 14 10:05:19 ubuntu-xenial-osic-cloud1-disk-6162583
kernel: traps: libvirtd[16302] general protection ip:7f7560066355 sp:7ffea18bcf60 error:0 in libc-2.23.so[7f755ffe5000+1bf000]

libvirtd seems to have restarted right afterwards, but it does make this suspect.

[Andrea Frittoli (andrea-frittoli)]

This also looks like another incarnation of https://bugs.launchpad.net/tempest/+bug/1646779, which is a Tempest bug.