tempest_neutron_plugin tests failing

[Arx Cruz]

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-1ctlr_2comp-featureset020-master/0fad0c0/logs/undercloud/var/log/tempest/stestr_results.html.gz

This is being consistent.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 102, in test_two_vms_fips
    servers, server_ssh_clients, fips = self._create_vms(hyper, avail_zone)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 73, in _create_vms
    availability_zone=az))
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/base.py", line 146, in create_server
    self.wait_for_server_active(server['server'], client=client)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/base.py", line 473, in wait_for_server_active
    server, constants.SERVER_STATUS_ACTIVE, client)
  File "/usr/lib/python3.6/site-packages/neutron_tempest_plugin/scenario/base.py", line 463, in wait_for_server_status
    waiters.wait_for_server_status(client, server['id'], status, **kwargs)
  File "/usr/lib/python3.6/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
    server_id=server_id)
tempest.exceptions.BuildErrorException: Server c6daed4a-9d15-4156-8fdf-8de96fbf26a0 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2020-11-24T03:49:27Z', 'message': 'No valid host was found. No such host - host: overcloud-novacompute-0.novalocal node: None ', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 1512, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python3.6/site-packages/nova/conductor/manager.py", line 884, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python3.6/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/client.py", line 179, in call\n transport_options=self.transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/transport.py", line 128, in _send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 654, in send\n transport_options=transport_options)\n File "/usr/lib/python3.6/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in _send\n raise result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. No such host - host: overcloud-novacompute-0.novalocal node: None \nTraceback (most recent call last):\n\n File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/manager.py", line 156, in select_destinations\n enable_pinning_translate=True)\n\n File "/usr/lib/python3.6/site-packages/nova/scheduler/utils.py", line 664, in resources_from_request_spec\n raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. No such host - host: overcloud-novacompute-0.novalocal node: None \n\n'}

[Slawek Kaplonski (slaweq)]

I looked at that issue today and from what I see the problem may be in nova.
Neutron-tempest-plugin test is looking for hosts by doing "GET /v2.1/os-hypervisors":

2020-11-24 03:49:25.197 11 INFO nova.api.openstack.requestlog [req-6547b2f2-d7b2-4e50-a46e-da8fdfd07790 a78d55547796476bbb5861930700d622 11c8bec3c80b4c819c0913f919f736c7 - default default] 172.17.0.91 "GET /v2.1/os-hypervisors" status: 200 len: 231 microversion: 2.1 time: 0.051316

And as You can see in tempest log, it get hostname "overcloud-novacompute-0.novalocal":

Response - Headers: {'date': 'Tue, 24 Nov 2020 03:49:25 GMT', 'server': 'Apache', 'content-length': '231', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version,Accept-Encoding', 'x-openstack-request-id': 'req-6547b2f2-d7b2-4e50-a46e-da8fdfd07790', 'x-compute-request-id': 'req-6547b2f2-d7b2-4e50-a46e-da8fdfd07790', 'connection': 'close', 'content-type': 'application/json', 'status': '200', 'content-location': 'http://10.0.0.5:8774/v2.1/os-hypervisors'}
            Body: b'{"hypervisors": [{"id": 1, "hypervisor_hostname": "overcloud-novacompute-0.novalocal", "state": "up", "status": "enabled"}, {"id": 2, "hypervisor_hostname": "overcloud-novacompute-1.novalocal", "state": "up", "status": "enabled"}]}'

But in the nova-compute's config "host" is set as: "host=overcloud-novacompute-0.localdomain" and that is also visible in nova api, like for example:

2020-11-24 03:43:13.300 11 INFO nova.api.openstack.compute.server_external_events [req-9e816b17-df2d-40cf-bba2-6b9c810f67bc 1f0a7f8a71504d5da979687cd98bb9af c4da28f50a9542a9b64e6432f41c9807 - default default] Creating event network-changed:04f2f874-d32b-457a-bed6-e0b98d4d9d0f for instance 9db686b1-dd63-4b5b-b472-24569f70b268 on overcloud-novacompute-0.localdomain

So the question to nova team is - should we change our list_hypervisors() call to something else? And why those are different hostnames in various places in Nova?

[Rafael Folco (rafaelfolco)]

skiplist https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/763985

[Oliver Walsh (owalsh)]

Root cause is dhcp_domain in nova.conf is not being set. Fix proposed in
https://review.opendev.org/c/openstack/puppet-tripleo/+/762415
https://review.opendev.org/c/openstack/puppet-nova/+/765131

[Oliver Walsh (owalsh)]

^H^H^H

Fix proposed in:
https://review.opendev.org/c/openstack/tripleo-heat-templates/+/765132
https://review.opendev.org/c/openstack/puppet-nova/+/765131

[Marios Andreou (marios-b)]

after the patch at https://review.opendev.org/c/openstack/tripleo-heat-templates/+/765132 merges we need to revert the skip https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/763985

[Marios Andreou (marios-b)]

per comment #5 revert posted at https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/765940

[wes hayutin (weshayutin)]

https://review.opendev.org/c/openstack/tripleo-heat-templates/+/765132

revert of skip list https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/766431 needs a rebase

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 14.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 13.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.4.3 release.