upgrades jobs are failing to ping GW (192.0.2.1) after update

[Dan Prince]

We are seeing repeated ping failures today on the upgrades jobs:

Apr 6 10:32:27 localhost os-collect-config: Trying to ping fd00:fd00:fd00:4000::12 for local network fd00:fd00:fd00:4000::/64...SUCCESS
Apr 6 10:32:27 localhost os-collect-config: Trying to ping default gateway 192.0.2.1...FAILURE
Apr 6 10:32:27 localhost os-collect-config: 192.0.2.1 is not pingable.
Apr 6 10:32:27 localhost os-collect-config: [2016-04-06 10:32:27,321] (heat-config) [DEBUG]
Apr 6 10:32:27 localhost os-collect-config: [2016-04-06 10:32:27,321] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/105fef55-7065-4059-8f94-832b16bdff98. [1]

----

This is a bit weird since subsequent calls in the same os-refresh-config run seem to be able to notify heat on the same 192.0.2.1 IP address.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/302386

[Dan Prince (dan-prince)]

Okay. Just spotted the issue. When using IPv6 (which the upgrades job uses) the $ping variable set set to ping6. This command now gets reused in the ping_default_gateway function due to:

https://review.openstack.org/#/c/298919/

We need to revert that commit or use the alternate version I just coded above. (which is correct).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/302393

[Dan Prince (dan-prince)]

To clarify: 'ping6 192.0.2.1' will always fail. This is what was happening.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302393
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8557ab7123b4550d23c191ad0247e42e132419a3
Submitter: Jenkins
Branch: master

commit 8557ab7123b4550d23c191ad0247e42e132419a3
Author: Dan Prince <email address hidden>
Date: Wed Apr 6 14:17:23 2016 -0400

    Revert "Ping retry"

    This reverts commit 570c690bfb118e0cf130b7dbed7992676519ed9b.

    This patch broke the ping_gateway_function when using IPv6
    network isolation.

    Change-Id: I57850a527804f2e753270fd9063d119d41a83b17
    Closes-bug: #1567011

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302386
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1c8ad9f41d9ceb59a6ffc7f0e991189901649e29
Submitter: Jenkins
Branch: master

commit 1c8ad9f41d9ceb59a6ffc7f0e991189901649e29
Author: Dan Prince <email address hidden>
Date: Wed Apr 6 14:06:14 2016 -0400

    Add ping_retry function

    This patch cleans up the retry logic so that it is all
    consolidated in the same function.

    Change-Id: I003afe4963e2af8d55f11a4ae4867e563a22134f
    Related-bug: #1567011

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/404835

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/404835
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1ca306ac99307433eea39c1f10700dff76dd49cf
Submitter: Jenkins
Branch: stable/mitaka

commit 1ca306ac99307433eea39c1f10700dff76dd49cf
Author: Dan Prince <email address hidden>
Date: Wed Apr 6 14:06:14 2016 -0400

    Add ping_retry function

    This patch cleans up the retry logic so that it is all
    consolidated in the same function.

    Change-Id: I003afe4963e2af8d55f11a4ae4867e563a22134f
    Related-bug: #1567011
    (cherry picked from commit 1c8ad9f41d9ceb59a6ffc7f0e991189901649e29)