neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin is failing on master promotions

[Ronelle Landy]

neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin started failing in master promotions on 06/12.

2019-06-11 17:58:54 | {0} neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin.test_two_vms_fips [6.444351s] ... FAILED
2019-06-11 17:58:54 |
2019-06-11 17:58:54 | Captured traceback:
2019-06-11 17:58:54 | ~~~~~~~~~~~~~~~~~~~
2019-06-11 17:58:54 | Traceback (most recent call last):
2019-06-11 17:58:54 | File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 102, in test_two_vms_fips
2019-06-11 17:58:54 | server_ssh_clients, fips = self._create_vms(hyper, avail_zone)
2019-06-11 17:58:54 | File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 77, in _create_vms
2019-06-11 17:58:54 | const.SERVER_STATUS_ACTIVE)
2019-06-11 17:58:54 | File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
2019-06-11 17:58:54 | server_id=server_id)
2019-06-11 17:58:54 | tempest.exceptions.BuildErrorException: Server 813f7cd9-e7ba-4e08-bb51-b72fde6a8824 failed to build and is in ERROR status
2019-06-11 17:58:54 | Details: {u'message': u'No valid host was found. No such host - host: overcloud-novacompute-1.novalocal node: None ', u'code': 500, u'details': u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1301, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 807, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call\n retry=self.retry)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 127, in _send\n retry=retry)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in send\n call_monitor_timeout, retry=retry)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 635, in _send\n raise result\n', u'created': u'2019-06-11T17:58:50Z'}
2019-06-11 17:58:54 |
2

The full tempest log is below:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9d25dfa/logs/undercloud/home/zuul/tempest.log.txt.gz

Note that the two other dns failures are known.

This failure is possibly fallout from https://review.opendev.org/#/c/660741/ - Add nova::metadata::dhcp_domain

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/664966

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/665091

[Martin Schuppert (mschuppert)]

The issue is in puppet-nova. enabled_apis [1] won't get set when nova_metadata_wsgi_enabled == true, which is the case in tripleo. But the enabled_apis list get also modified for this case, so we need to move setting the config outside the conditional at [3].

[1] https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L401
[2] https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L353
[3] https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L379

[OpenStack Infra (hudson-openstack)]

Change abandoned by Martin Schuppert (<email address hidden>) on branch: master
Review: https://review.opendev.org/665091
Reason: issue is in puppet-nova

[Ronelle Landy (rlandy)]

Considering ykarel's workaround https://review.opendev.org/#/c/664966/ as a mid-term solution to unblock promotions if needed

[Ronelle Landy (rlandy)]

https://review.opendev.org/#/c/664966/ was w+ - reminder to remove this workaround when the bug is fixed

[wes hayutin (weshayutin)]

I guess the patch is now on

https://review.opendev.org/#/c/665131/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/664966
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=12aaf4b54c7dfe2e5e3044882af78bb340a5d63c
Submitter: Zuul
Branch: master

commit 12aaf4b54c7dfe2e5e3044882af78bb340a5d63c
Author: yatinkarel <email address hidden>
Date: Wed Jun 12 21:27:35 2019 +0530

    Also set nova::network::neutron::dhcp_domain

    https://review.opendev.org/#/c/659394 set dhcp_domain
    config with metadata manifest, but still nova hosts have dhcp_domain
    set to default i.e. novalocal. Until it's clear why api/dhcp_domain
    set in nova_metadata volume is not entertained by nova, let's set
    nova::network::neutron::dhcp_domain also to ''.

    Partial-Bug: #1832537
    Change-Id: Ica532e7fe2c4578ec273855cb69179864f4746b4

[Martin Schuppert (mschuppert)]

Just a FYI, I have not managed to finish this due to other priority issues and had to re prioritize since we have a workaround. I am out next week on PTO and will return to this afterwards.

Sorry for that.

[wes hayutin (weshayutin)]

FYI.. https://bugzilla.redhat.com/show_bug.cgi?id=1723451 used to track a long term fix

[Martin Schuppert (mschuppert)]

While the nova::api has an issue with setting the enabled_apis, this is not the root cause for this issue. I was wrong there.

In general we use the config_drive. The compute driver/ironic creates the config drive with the metadata and adds the instance information [1]. For this metadata added to the config drive the [api]dhcp_domain is used to get and add the hostname [2].

* [api]/dhcp_domain commented out (restart nova_compute)

[root@overcloud-novacompute-0 ~]# hostname
overcloud-novacompute-0.novalocal

[root@overcloud-novacompute-0 ~]# mount /dev/vda1 /mnt
mount: /dev/vda1 is write-protected, mounting read-only

[root@overcloud-novacompute-0 ~]# cat /mnt/openstack/latest/meta_data.json | python -m json.tool
{
...
    "hostname": "overcloud-novacompute-0.novalocal",
...

* [api]/dhcp_domain=

[root@overcloud-novacompute-0 ~]# mount /dev/vda1 /mnt/
[root@overcloud-novacompute-0 ~]# cat /mnt/openstack/latest/meta_data.json | python -m json.tool
{
...
    "hostname": "overcloud-novacompute-0",
...

On the undercloud the nova_compute container used for the ironic driver uses the same nova.conf
/var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf as the api service. For this nova.conf we don't use the nova::metadata class which would set the [api]dhcp_domain setting with the nova::metadata::dhcp_domain key we have in the template.

In the puppet-tripleo nova/api.pp we include ::nova::network::neutron [3] which explains why we get the expected hostname domain when nova::network::neutron::dhcp_domain is set via the workaround patch.

So we'd need to include ::nova::metadata also for the compute services.

[1] https://github.com/openstack/nova/blob/master/nova/virt/configdrive.py#L71
[2] https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L517
[3] https://github.com/openstack/puppet-tripleo/blob/master/manifests/profile/base/nova/api.pp#L128

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/668949

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/669208

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/669208
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=e677aeacd706674762751f8b530ad08d6d925910
Submitter: Zuul
Branch: master

commit e677aeacd706674762751f8b530ad08d6d925910
Author: Martin Schuppert <email address hidden>
Date: Thu Jul 4 18:22:14 2019 +0200

    Set nova::keystone auth details hiera on compute

    For config drive we need to set the [api]/dhcp_domain
    in the computes. Therefore ::nova::metadata is included
    via https://review.opendev.org/668949 to the
    tripleo::profile::base::nova::compute class. We'd need
    to add this informaton as the ::nova::metadata includes
    ::nova::keystone::authtoken.

    Change-Id: Ib5d7cdd8efcb40d313a55b6ab516e7f15e619187
    Related-Bug: #1832537

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/668949
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=3273d29163aed22b0ec5385738d06b6083aa9b03
Submitter: Zuul
Branch: master

commit 3273d29163aed22b0ec5385738d06b6083aa9b03
Author: Martin Schuppert <email address hidden>
Date: Wed Jul 3 11:46:20 2019 +0200

    Include nova::metadata to compute profile

    The old DEFAULT/dhcp_domain setting was moved to api/dhcp_domain.
    nova::network::neutron::dhcp_domain will be removed later in the cycle.
    We need include nova::metadata which sets the new [api]/dhcp_domain
    as this is used by the virt driver to generate the config drive.

    Closes-Bug: 1832537

    Depends-On: https://review.opendev.org/668663
    Depends-On: https://review.opendev.org/668666
    Depends-On: https://review.opendev.org/665131
    Depends-On: https://review.opendev.org/669066
    Depends-On: https://review.opendev.org/669208

    Change-Id: I5da1e67684f8317eec8c499c3534977e00a63098

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/762415

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762415
Committed: https://opendev.org/openstack/puppet-tripleo/commit/287b1b6ce77fe143d781ccd1b900c3bdbe6ada87
Submitter: Zuul
Branch: master

commit 287b1b6ce77fe143d781ccd1b900c3bdbe6ada87
Author: Oliver Walsh <email address hidden>
Date: Wed Nov 11 20:42:14 2020 +0000

    Stop including nova::metadata on computes

    I5da1e67684f8317eec8c499c3534977e00a63098 include nova::metadata on computes
    however this includes nova::db which we must not do on computes.
    Push the fix up to puppet-nova and nova.

    Depends-On: If6a26527a737a7184ebddd5b4bc346d64827e9e3
    Depends-On: I47b95ad46e2d4e5b1f370a2f840826e87da2d703
    Change-Id: I07caa3185427b48e6e7d60965fa3e6157457018c
    Related-bug: #1871482
    Related-bug: #1903908
    Related-bug: #1832537