Ovs::Bridge[datacentre:br-ex]/Vs_bridge[br-ex]/ensure: change from absent to present failed: Execution of '/usr/sbin/ip link set dev br-ex up' returned 1: Cannot find device

[Matt Young]

This appears to be blocking promotions for both ocata and pike:

https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/74fb0a2/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-ocata/3c08e7f/undercloud/home/jenkins/failed_deployment_list.log.txt.gz

OC deploy is failing with the following error(s)

---

"Error: Execution of '/usr/sbin/ip link set dev br-ex up' returned 1: Cannot find device \"br-ex\""
"Error: /Stage[main]/Neutron::Agents::Ml2::Ovs/Neutron::Plugins::Ovs::Bridge[datacentre:br-ex]/Vs_bridge[br-ex]/ensure: change from absent to present failed: Execution of '/usr/sbin/ip link set dev br-ex up' returned 1: Cannot find device

---

Root cause (beyond b-ex not existing) has not yet been done.

[Matt Young (halcyondude)]

Capturing IRC convo here...this could potentially be related to

https://bugs.launchpad.net/tripleo/+bug/1752441

Which is a pingtest failing because of OVS startup issues. That bug is a bit different in that the OC deploy is actually working (or is reported as such) while this bug is OC deploy failing entirely, as OVS appears to not be happy

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/549838

[Alex Schultz (alex-schultz)]

Yea it's the same root cause. iptables is being restarted which causes openvswitch kernel module to be unloaded which triggers random other failures related to the interfaces.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549838
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=bb5013920ac658c99d9ae2ab7f81847b274aa177
Submitter: Zuul
Branch: master

commit bb5013920ac658c99d9ae2ab7f81847b274aa177
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 11:06:52 2018 -0700

    Reload iptables instead of restart

    Due to bz#1520534, restarting iptables may cause unrelated kernel
    modules to be unloaded. In order to not trigger this condition we should
    reload iptables from the configuration rather than restart the whole
    process.

    Change-Id: Ifc625eb51f6cc2a0a4cf4f83ac7a4978db641d75
    Closes-Bug: #1752441
    Closes-Bug: #1753492

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/549939

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/549940

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/549941

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/549942

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549939
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=9868c0395b50a6c5e2c21c765bd1cb5345ab0ea4
Submitter: Zuul
Branch: stable/queens

commit 9868c0395b50a6c5e2c21c765bd1cb5345ab0ea4
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 11:06:52 2018 -0700

    Reload iptables instead of restart

    Due to bz#1520534, restarting iptables may cause unrelated kernel
    modules to be unloaded. In order to not trigger this condition we should
    reload iptables from the configuration rather than restart the whole
    process.

    Change-Id: Ifc625eb51f6cc2a0a4cf4f83ac7a4978db641d75
    Closes-Bug: #1752441
    Closes-Bug: #1753492
    (cherry picked from commit bb5013920ac658c99d9ae2ab7f81847b274aa177)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549941
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8dee1557874272126dbb8bc41affbd7ec0063097
Submitter: Zuul
Branch: stable/ocata

commit 8dee1557874272126dbb8bc41affbd7ec0063097
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 11:06:52 2018 -0700

    Reload iptables instead of restart

    Due to bz#1520534, restarting iptables may cause unrelated kernel
    modules to be unloaded. In order to not trigger this condition we should
    reload iptables from the configuration rather than restart the whole
    process.

    Change-Id: Ifc625eb51f6cc2a0a4cf4f83ac7a4978db641d75
    Closes-Bug: #1752441
    Closes-Bug: #1753492
    (cherry picked from commit bb5013920ac658c99d9ae2ab7f81847b274aa177)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549942
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=fbab9255944d1dac59fd9e1f271f69b24fd90cf1
Submitter: Zuul
Branch: stable/newton

commit fbab9255944d1dac59fd9e1f271f69b24fd90cf1
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 11:06:52 2018 -0700

    Reload iptables instead of restart

    Due to bz#1520534, restarting iptables may cause unrelated kernel
    modules to be unloaded. In order to not trigger this condition we should
    reload iptables from the configuration rather than restart the whole
    process.

    Change-Id: Ifc625eb51f6cc2a0a4cf4f83ac7a4978db641d75
    Closes-Bug: #1752441
    Closes-Bug: #1753492
    (cherry picked from commit bb5013920ac658c99d9ae2ab7f81847b274aa177)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549940
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=58e99c7f0af25ed2aed7d46b7d219abcabd6c1d2
Submitter: Zuul
Branch: stable/pike

commit 58e99c7f0af25ed2aed7d46b7d219abcabd6c1d2
Author: Alex Schultz <email address hidden>
Date: Mon Mar 5 11:06:52 2018 -0700

    Reload iptables instead of restart

    Due to bz#1520534, restarting iptables may cause unrelated kernel
    modules to be unloaded. In order to not trigger this condition we should
    reload iptables from the configuration rather than restart the whole
    process.

    Change-Id: Ifc625eb51f6cc2a0a4cf4f83ac7a4978db641d75
    Closes-Bug: #1752441
    Closes-Bug: #1753492
    (cherry picked from commit bb5013920ac658c99d9ae2ab7f81847b274aa177)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 7.4.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 6.5.10 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 5.6.8 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 8.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 9.0.0 release.