upgrades jobs are failing to ping GW (192.0.2.1) after update

Bug #1567011 reported by Dan Prince
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Dan Prince

Bug Description

We are seeing repeated ping failures today on the upgrades jobs:

Apr 6 10:32:27 localhost os-collect-config: Trying to ping fd00:fd00:fd00:4000::12 for local network fd00:fd00:fd00:4000::/64...SUCCESS
Apr 6 10:32:27 localhost os-collect-config: Trying to ping default gateway 192.0.2.1...FAILURE
Apr 6 10:32:27 localhost os-collect-config: 192.0.2.1 is not pingable.
Apr 6 10:32:27 localhost os-collect-config: [2016-04-06 10:32:27,321] (heat-config) [DEBUG]
Apr 6 10:32:27 localhost os-collect-config: [2016-04-06 10:32:27,321] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-script/105fef55-7065-4059-8f94-832b16bdff98. [1]

----

This is a bit weird since subsequent calls in the same os-refresh-config run seem to be able to notify heat on the same 192.0.2.1 IP address.

Dan Prince (dan-prince)
Changed in tripleo:
assignee: nobody → Dan Prince (dan-prince)
status: New → In Progress
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/302386

Revision history for this message
Dan Prince (dan-prince) wrote :

Okay. Just spotted the issue. When using IPv6 (which the upgrades job uses) the $ping variable set set to ping6. This command now gets reused in the ping_default_gateway function due to:

https://review.openstack.org/#/c/298919/

We need to revert that commit or use the alternate version I just coded above. (which is correct).

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/302393

Revision history for this message
Dan Prince (dan-prince) wrote :

To clarify: 'ping6 192.0.2.1' will always fail. This is what was happening.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/302393
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=8557ab7123b4550d23c191ad0247e42e132419a3
Submitter: Jenkins
Branch: master

commit 8557ab7123b4550d23c191ad0247e42e132419a3
Author: Dan Prince <email address hidden>
Date: Wed Apr 6 14:17:23 2016 -0400

    Revert "Ping retry"

    This reverts commit 570c690bfb118e0cf130b7dbed7992676519ed9b.

    This patch broke the ping_gateway_function when using IPv6
    network isolation.

    Change-Id: I57850a527804f2e753270fd9063d119d41a83b17
    Closes-bug: #1567011

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/302386
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1c8ad9f41d9ceb59a6ffc7f0e991189901649e29
Submitter: Jenkins
Branch: master

commit 1c8ad9f41d9ceb59a6ffc7f0e991189901649e29
Author: Dan Prince <email address hidden>
Date: Wed Apr 6 14:06:14 2016 -0400

    Add ping_retry function

    This patch cleans up the retry logic so that it is all
    consolidated in the same function.

    Change-Id: I003afe4963e2af8d55f11a4ae4867e563a22134f
    Related-bug: #1567011

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/tripleo-heat-templates 5.0.0.0b1

This issue was fixed in the openstack/tripleo-heat-templates 5.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-heat-templates (stable/mitaka)

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/404835

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-heat-templates (stable/mitaka)

Reviewed: https://review.openstack.org/404835
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1ca306ac99307433eea39c1f10700dff76dd49cf
Submitter: Jenkins
Branch: stable/mitaka

commit 1ca306ac99307433eea39c1f10700dff76dd49cf
Author: Dan Prince <email address hidden>
Date: Wed Apr 6 14:06:14 2016 -0400

    Add ping_retry function

    This patch cleans up the retry logic so that it is all
    consolidated in the same function.

    Change-Id: I003afe4963e2af8d55f11a4ae4867e563a22134f
    Related-bug: #1567011
    (cherry picked from commit 1c8ad9f41d9ceb59a6ffc7f0e991189901649e29)

tags: added: in-stable-mitaka
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers