IPv6 flaps on OVS restart due to ndisc_notify settings on RHEL/CentOS

[Dan Sneddon]

On RHEL/CentOS, the IPv6 stack is configured by default to not send Neighbor Discovery Notifications. This has the result that when OVS restarts or an internal port flaps, neighbors are not made aware of the new MAC address.

By enabling the sysctl parameter net.ipv6.conf.all.ndisc_notify, a gratuitous neighbor discovery will be sent from the host, which will update the MAC tables of all hosts attached to the network.

In order to enable this behavior by default in TripleO, we should add the following default sysctl parameter in deployment/kernel/kernel-baremetal-puppet.yaml:

net.ipv6.conf.all.ndisc_notify:
  value: 1

[Dan Sneddon (dsneddon)]

This change should be backported, as it fixes a bug and carries no risk.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/657429

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657429
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=4b113a7a1287f30e0e57abf4e3c74870d4720534
Submitter: Zuul
Branch: master

commit 4b113a7a1287f30e0e57abf4e3c74870d4720534
Author: Dan Sneddon <email address hidden>
Date: Mon May 6 11:40:41 2019 -0700

    Enable ndisc_notify sysctl setting to notify of MAC changes

    On RHEL/CentOS systems, ndisc_notify is disabled by default. When
    OVS restarts or an internal port flaps, the MAC address may change.
    Without ndisc_notify, neighbor hosts on the same network will not
    know about the MAC change, and will lose connectivity until the MAC
    timer expires.

    This change sets net.ipv6.conf.all.ndisc_notify to 1, which will
    cause a gratuitous neighbor discovery packet which will update MAC
    address tables on neighboring hosts.

    Change-Id: I89062275541d7c6bb6fb725f5283ba59feb38a94
    Closes-bug: 1827927

[Bogdan Dobrelya (bogdando)]

See https://<email address hidden>/arp-stale-cache-and-a-rather-peculiar-arp-notify-behavior-in-linux-9c5105d07a49 for the related IPv4 issue. It seems that backporting the fix would not provide a complete fix via usual minor updates meanings, but instead we'd need some additional orchestration for the system interfaces getting restarted in order to apply new sysctl values.

Also, I believe we should also set arp_notify=1 for the system configuration (IPv4 vs IPv6) consistency sakes.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/657605

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/657686

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/657690

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/657691

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657686
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=f3df90f2c0afa352d4dcf14463a9364c9080afe6
Submitter: Zuul
Branch: master

commit f3df90f2c0afa352d4dcf14463a9364c9080afe6
Author: Nate Johnston <email address hidden>
Date: Tue May 7 16:45:20 2019 -0400

    Set arp_notify to match ndisc_notify

    A previous change [1] enabled ndisc_notify, which is the IPv6 equivalent
    for IPv4's arp_notify. Enable the latter to keep them consistent with
    each other.

    Change-Id: I15e8348585a9c0ee824a4c123677992010980b9e
    Related-Bug: #1827927

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657605
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b15532ae089a1da6e03e672a7b776e4d8fc934d6
Submitter: Zuul
Branch: stable/stein

commit b15532ae089a1da6e03e672a7b776e4d8fc934d6
Author: Dan Sneddon <email address hidden>
Date: Mon May 6 11:40:41 2019 -0700

    Enable ndisc_notify sysctl setting to notify of MAC changes

    On RHEL/CentOS systems, ndisc_notify is disabled by default. When
    OVS restarts or an internal port flaps, the MAC address may change.
    Without ndisc_notify, neighbor hosts on the same network will not
    know about the MAC change, and will lose connectivity until the MAC
    timer expires.

    This change sets net.ipv6.conf.all.ndisc_notify to 1, which will
    cause a gratuitous neighbor discovery packet which will update MAC
    address tables on neighboring hosts.

    Change-Id: I89062275541d7c6bb6fb725f5283ba59feb38a94
    Closes-bug: 1827927
    (cherry picked from commit 4b113a7a1287f30e0e57abf4e3c74870d4720534)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657690
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=884a0ad2082c4cbefc870028228cfd27881dad90
Submitter: Zuul
Branch: stable/rocky

commit 884a0ad2082c4cbefc870028228cfd27881dad90
Author: Dan Sneddon <email address hidden>
Date: Mon May 6 11:40:41 2019 -0700

    Enable ndisc_notify sysctl setting to notify of MAC changes

    On RHEL/CentOS systems, ndisc_notify is disabled by default. When
    OVS restarts or an internal port flaps, the MAC address may change.
    Without ndisc_notify, neighbor hosts on the same network will not
    know about the MAC change, and will lose connectivity until the MAC
    timer expires.

    This change sets net.ipv6.conf.all.ndisc_notify to 1, which will
    cause a gratuitous neighbor discovery packet which will update MAC
    address tables on neighboring hosts.

    Change-Id: I89062275541d7c6bb6fb725f5283ba59feb38a94
    Closes-bug: 1827927
    (cherry picked from commit 4b113a7a1287f30e0e57abf4e3c74870d4720534)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657691
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=3ff4bdcd35f352f292ec5f46013107ca48be552c
Submitter: Zuul
Branch: stable/queens

commit 3ff4bdcd35f352f292ec5f46013107ca48be552c
Author: Dan Sneddon <email address hidden>
Date: Mon May 6 11:40:41 2019 -0700

    Enable ndisc_notify sysctl setting to notify of MAC changes

    On RHEL/CentOS systems, ndisc_notify is disabled by default. When
    OVS restarts or an internal port flaps, the MAC address may change.
    Without ndisc_notify, neighbor hosts on the same network will not
    know about the MAC change, and will lose connectivity until the MAC
    timer expires.

    This change sets net.ipv6.conf.all.ndisc_notify to 1, which will
    cause a gratuitous neighbor discovery packet which will update MAC
    address tables on neighboring hosts.

    Change-Id: I89062275541d7c6bb6fb725f5283ba59feb38a94
    Closes-bug: 1827927
    (cherry picked from commit 4b113a7a1287f30e0e57abf4e3c74870d4720534)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/660209

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660209
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=1616da335b48f47dfbfec62a840444b3e674b74f
Submitter: Zuul
Branch: stable/stein

commit 1616da335b48f47dfbfec62a840444b3e674b74f
Author: Nate Johnston <email address hidden>
Date: Tue May 7 16:45:20 2019 -0400

    Set arp_notify to match ndisc_notify

    A previous change [1] enabled ndisc_notify, which is the IPv6 equivalent
    for IPv4's arp_notify. Enable the latter to keep them consistent with
    each other.

    Change-Id: I15e8348585a9c0ee824a4c123677992010980b9e
    Related-Bug: #1827927
    (cherry picked from commit f3df90f2c0afa352d4dcf14463a9364c9080afe6)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/660849

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/660850

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660849
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=debdaa4b375c317198e8399ff989486a5a490d78
Submitter: Zuul
Branch: stable/rocky

commit debdaa4b375c317198e8399ff989486a5a490d78
Author: Nate Johnston <email address hidden>
Date: Tue May 7 16:45:20 2019 -0400

    Set arp_notify to match ndisc_notify

    A previous change [1] enabled ndisc_notify, which is the IPv6 equivalent
    for IPv4's arp_notify. Enable the latter to keep them consistent with
    each other.

    Change-Id: I15e8348585a9c0ee824a4c123677992010980b9e
    Related-Bug: #1827927
    (cherry picked from commit f3df90f2c0afa352d4dcf14463a9364c9080afe6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/660850
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c47de732a52890f8e3a7a42447dcf3efc3ca4cdd
Submitter: Zuul
Branch: stable/queens

commit c47de732a52890f8e3a7a42447dcf3efc3ca4cdd
Author: Nate Johnston <email address hidden>
Date: Tue May 7 16:45:20 2019 -0400

    Set arp_notify to match ndisc_notify

    A previous change [1] enabled ndisc_notify, which is the IPv6 equivalent
    for IPv4's arp_notify. Enable the latter to keep them consistent with
    each other.

    Change-Id: I15e8348585a9c0ee824a4c123677992010980b9e
    Related-Bug: #1827927
    (cherry picked from commit f3df90f2c0afa352d4dcf14463a9364c9080afe6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.