neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin is failing on master promotions

Bug #1832537 reported by Ronelle Landy on 2019-06-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Martin Schuppert

Bug Description

neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin started failing in master promotions on 06/12.

2019-06-11 17:58:54 | {0} neutron_tempest_plugin.scenario.admin.test_floatingip.FloatingIpTestCasesAdmin.test_two_vms_fips [6.444351s] ... FAILED
2019-06-11 17:58:54 |
2019-06-11 17:58:54 | Captured traceback:
2019-06-11 17:58:54 | ~~~~~~~~~~~~~~~~~~~
2019-06-11 17:58:54 | Traceback (most recent call last):
2019-06-11 17:58:54 | File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 102, in test_two_vms_fips
2019-06-11 17:58:54 | server_ssh_clients, fips = self._create_vms(hyper, avail_zone)
2019-06-11 17:58:54 | File "/usr/lib/python2.7/site-packages/neutron_tempest_plugin/scenario/admin/test_floatingip.py", line 77, in _create_vms
2019-06-11 17:58:54 | const.SERVER_STATUS_ACTIVE)
2019-06-11 17:58:54 | File "/usr/lib/python2.7/site-packages/tempest/common/waiters.py", line 76, in wait_for_server_status
2019-06-11 17:58:54 | server_id=server_id)
2019-06-11 17:58:54 | tempest.exceptions.BuildErrorException: Server 813f7cd9-e7ba-4e08-bb51-b72fde6a8824 failed to build and is in ERROR status
2019-06-11 17:58:54 | Details: {u'message': u'No valid host was found. No such host - host: overcloud-novacompute-1.novalocal node: None ', u'code': 500, u'details': u' File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1301, in schedule_and_build_instances\n instance_uuids, return_alternates=True)\n File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 807, in _schedule_instances\n return_alternates=return_alternates)\n File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations\n instance_uuids, return_objects, return_alternates)\n File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 178, in call\n retry=self.retry)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 127, in _send\n retry=retry)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 644, in send\n call_monitor_timeout, retry=retry)\n File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 635, in _send\n raise result\n', u'created': u'2019-06-11T17:58:50Z'}
2019-06-11 17:58:54 |
2

The full tempest log is below:
http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-1ctlr_2comp-featureset020-master/9d25dfa/logs/undercloud/home/zuul/tempest.log.txt.gz

Note that the two other dns failures are known.

This failure is possibly fallout from https://review.opendev.org/#/c/660741/ - Add nova::metadata::dhcp_domain

Ronelle Landy (rlandy) on 2019-06-12
Changed in tripleo:
milestone: none → train-2
importance: Undecided → High
status: New → Triaged
tags: added: promotion-blocker
Changed in tripleo:
importance: High → Critical
tags: added: tempest

Fix proposed to branch: master
Review: https://review.opendev.org/664966

Changed in tripleo:
assignee: nobody → yatin (yatinkarel)
status: Triaged → In Progress

Fix proposed to branch: master
Review: https://review.opendev.org/665091

Changed in tripleo:
assignee: yatin (yatinkarel) → Martin Schuppert (mschuppert)
Martin Schuppert (mschuppert) wrote :

The issue is in puppet-nova. enabled_apis [1] won't get set when nova_metadata_wsgi_enabled == true, which is the case in tripleo. But the enabled_apis list get also modified for this case, so we need to move setting the config outside the conditional at [3].

[1] https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L401
[2] https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L353
[3] https://github.com/openstack/puppet-nova/blob/master/manifests/api.pp#L379

Change abandoned by Martin Schuppert (<email address hidden>) on branch: master
Review: https://review.opendev.org/665091
Reason: issue is in puppet-nova

Ronelle Landy (rlandy) wrote :

Considering ykarel's workaround https://review.opendev.org/#/c/664966/ as a mid-term solution to unblock promotions if needed

Ronelle Landy (rlandy) wrote :

https://review.opendev.org/#/c/664966/ was w+ - reminder to remove this workaround when the bug is fixed

wes hayutin (weshayutin) wrote :

I guess the patch is now on

https://review.opendev.org/#/c/665131/

Reviewed: https://review.opendev.org/664966
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=12aaf4b54c7dfe2e5e3044882af78bb340a5d63c
Submitter: Zuul
Branch: master

commit 12aaf4b54c7dfe2e5e3044882af78bb340a5d63c
Author: yatinkarel <email address hidden>
Date: Wed Jun 12 21:27:35 2019 +0530

    Also set nova::network::neutron::dhcp_domain

    https://review.opendev.org/#/c/659394 set dhcp_domain
    config with metadata manifest, but still nova hosts have dhcp_domain
    set to default i.e. novalocal. Until it's clear why api/dhcp_domain
    set in nova_metadata volume is not entertained by nova, let's set
    nova::network::neutron::dhcp_domain also to ''.

    Partial-Bug: #1832537
    Change-Id: Ica532e7fe2c4578ec273855cb69179864f4746b4

Martin Schuppert (mschuppert) wrote :

Just a FYI, I have not managed to finish this due to other priority issues and had to re prioritize since we have a workaround. I am out next week on PTO and will return to this afterwards.

Sorry for that.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers