validation scripts should ping default gateways before controllers

Bug #1793598 reported by Bob Fournier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Undecided
Dan Sneddon

Bug Description

From downstream bug report:

While deploying an Overcloud that is utilizing IPv6 addressing, we sometimes see where the pinging to the controllers fails as part of the deployment validation which then results in a failed deploy. Through troubleshooting, we've determined that pings to the gateways do not fail and when pinging the gateway, it appears to "fix" the ability to ping to the Controller Nodes.

Within our team we've tried to figure out why this would be the case and one of our team members pointed out that IPV6 does not have ARP, it has neighbor solicitation and router solicitation, and sometimes neighbor solicitation isn't on or doesn't work, in which case you -have- to establish connectivity to the router before any other host communication works. He said this only applies to non eui64 (mac address embedded in ipv6 address) setups.

We're not sure if that's what is happening here, however, we do know if we continually run an Ansible Playbook that connects tot he overcloud nodes and has them ping their default gateways during deployment, the deployment succeeds 100% of the time.

In looking at the all_nodes.sh validation script:

/usr/share/openstack-tripleo-heat-templates/validation-scripts/all-nodes.sh

We noticed that it is performing 2 tests, ping_controller_ips and ping_default_gateways, in that order. We believe flipping these around will correct this issue.

This BZ is to have those 2 tests swapped so the ping of the gateways happens before pinging the controllers.

Bob Fournier (bfournie)
Changed in tripleo:
assignee: nobody → Bob Fournier (bfournie)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/604229

Changed in tripleo:
status: New → In Progress
Changed in tripleo:
assignee: Bob Fournier (bfournie) → Dan Sneddon (dsneddon)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/604229
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ae3379e07011813feee2621a21f8ea66c6d0e059
Submitter: Zuul
Branch: master

commit ae3379e07011813feee2621a21f8ea66c6d0e059
Author: Bob Fournier <email address hidden>
Date: Thu Sep 20 17:00:14 2018 -0400

    Ping default gateways before controllers

    It may be necessary, in certain situations when using IPv6,
    to ping the default gateways first before attempting to
    ping other hosts in the validation script. This swaps the
    order of the pings between the controllers and default gateways.

    Change-Id: I14b7fcffb0609aee3f0f53150496461396348322
    Closes-Bug: 1793598

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/607173

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.openstack.org/607173
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=244b7bf10712ec478240ee2172bfaf506d65b8c3
Submitter: Zuul
Branch: stable/rocky

commit 244b7bf10712ec478240ee2172bfaf506d65b8c3
Author: Bob Fournier <email address hidden>
Date: Thu Sep 20 17:00:14 2018 -0400

    Ping default gateways before controllers

    It may be necessary, in certain situations when using IPv6,
    to ping the default gateways first before attempting to
    ping other hosts in the validation script. This swaps the
    order of the pings between the controllers and default gateways.

    Change-Id: I14b7fcffb0609aee3f0f53150496461396348322
    Closes-Bug: 1793598
    (cherry picked from commit ae3379e07011813feee2621a21f8ea66c6d0e059)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/607367

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.openstack.org/607367
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=036dc358aea59efd07616ad29bdf3e6272601d50
Submitter: Zuul
Branch: stable/queens

commit 036dc358aea59efd07616ad29bdf3e6272601d50
Author: Bob Fournier <email address hidden>
Date: Thu Sep 20 17:00:14 2018 -0400

    Ping default gateways before controllers

    It may be necessary, in certain situations when using IPv6,
    to ping the default gateways first before attempting to
    ping other hosts in the validation script. This swaps the
    order of the pings between the controllers and default gateways.

    Change-Id: I14b7fcffb0609aee3f0f53150496461396348322
    Closes-Bug: 1793598
    (cherry picked from commit ae3379e07011813feee2621a21f8ea66c6d0e059)
    (cherry picked from commit 244b7bf10712ec478240ee2172bfaf506d65b8c3)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.0.0

This issue was fixed in the openstack/tripleo-heat-templates 10.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.1.0

This issue was fixed in the openstack/tripleo-heat-templates 8.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 9.1.0

This issue was fixed in the openstack/tripleo-heat-templates 9.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.