Accidental centralized floating IP removal

[Oleg Bondarev]

release: Pike.

When a DVR router with centralized floating IPs is rescheduled from down l3 agent to another,
following traces are seen on the destination agent:

2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib [-] Failed sending gratuitous ARP to 10.13.250.14 on qg-af0de258-a8 in namespace snat-afe70a67-a007-4bcf-93ac-099aad63411c: Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address
: ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib Traceback (most recent call last):
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 1097, in _arping
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib ip_wrapper.netns.execute(arping_cmd, extra_ok_codes=[1])
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 916, in execute
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib log_fail_as_error=log_fail_as_error, **kwargs)
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 151, in execute
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib raise ProcessExecutionError(msg, returncode=returncode)
2019-02-21 14:55:59,150.150 24730 ERROR neutron.agent.linux.ip_lib ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: bind: Cannot assign requested address

Earlier in logs following is seen:

2019-02-22 07:52:14,894.894 9528 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'snat-afe70a67-a007-4bcf-93ac-099aad63411c', 'ip', '-4', 'addr', 'del', '10.13.250.14/32', 'dev', 'qg-af0de258-a8'] execute_rootwrap_daemon /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:108
2019-02-22 07:52:14,922.922 9528 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'snat-afe70a67-a007-4bcf-93ac-099aad63411c', 'conntrack', '-D', '-d', '10.13.250.14'] execute_rootwrap_daemon /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:108

So centralized floating ip is deleted from gw device for some reason.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/638648

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638648
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1ee18775a99492f445f1fb29e2d4639cbdff613d
Submitter: Zuul
Branch: master

commit 1ee18775a99492f445f1fb29e2d4639cbdff613d
Author: Oleg Bondarev <email address hidden>
Date: Fri Feb 22 16:00:30 2019 +0400

    DVR edge router: avoid accidental centralized floating IP remove

    Need to pass centralized floating IPs as preserve_ips to
    _external_gateway_added during DVR router update.
    Otherwise IP addresses will be deleted from gw device in certain case.
    The case is when a router with active centralized floating IPs is
    being scheduled to a new dvr_snat L3 agent (rescheduled from a down one).
    Please see corresponding traces in the bug description.

    Change-Id: Iaeb9fbed73144df6fcd9092c665ed19986e85f4d
    Closes-bug: #1817306

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/640676

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/640677

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640677
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a906ace3efa50a23517f2a3387dc1b99c489fca0
Submitter: Zuul
Branch: stable/pike

commit a906ace3efa50a23517f2a3387dc1b99c489fca0
Author: Oleg Bondarev <email address hidden>
Date: Fri Feb 22 16:00:30 2019 +0400

    DVR edge router: avoid accidental centralized floating IP remove

    Need to pass centralized floating IPs as preserve_ips to
    _external_gateway_added during DVR router update.
    Otherwise IP addresses will be deleted from gw device in certain case.
    The case is when a router with active centralized floating IPs is
    being scheduled to a new dvr_snat L3 agent (rescheduled from a down one).
    Please see corresponding traces in the bug description.

    Change-Id: Iaeb9fbed73144df6fcd9092c665ed19986e85f4d
    Closes-bug: #1817306
    (cherry picked from commit 1ee18775a99492f445f1fb29e2d4639cbdff613d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640642
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c620b3c91efcf38c039981f3153c02ed3bf67de3
Submitter: Zuul
Branch: stable/rocky

commit c620b3c91efcf38c039981f3153c02ed3bf67de3
Author: Oleg Bondarev <email address hidden>
Date: Fri Feb 22 16:00:30 2019 +0400

    DVR edge router: avoid accidental centralized floating IP remove

    Need to pass centralized floating IPs as preserve_ips to
    _external_gateway_added during DVR router update.
    Otherwise IP addresses will be deleted from gw device in certain case.
    The case is when a router with active centralized floating IPs is
    being scheduled to a new dvr_snat L3 agent (rescheduled from a down one).
    Please see corresponding traces in the bug description.

    Change-Id: Iaeb9fbed73144df6fcd9092c665ed19986e85f4d
    Closes-bug: #1817306
    (cherry picked from commit 1ee18775a99492f445f1fb29e2d4639cbdff613d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640676
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bf8a2c879ed02efd8776ee38a1bad404c2c27df0
Submitter: Zuul
Branch: stable/queens

commit bf8a2c879ed02efd8776ee38a1bad404c2c27df0
Author: Oleg Bondarev <email address hidden>
Date: Fri Feb 22 16:00:30 2019 +0400

    DVR edge router: avoid accidental centralized floating IP remove

    Need to pass centralized floating IPs as preserve_ips to
    _external_gateway_added during DVR router update.
    Otherwise IP addresses will be deleted from gw device in certain case.
    The case is when a router with active centralized floating IPs is
    being scheduled to a new dvr_snat L3 agent (rescheduled from a down one).
    Please see corresponding traces in the bug description.

    Change-Id: Iaeb9fbed73144df6fcd9092c665ed19986e85f4d
    Closes-bug: #1817306
    (cherry picked from commit 1ee18775a99492f445f1fb29e2d4639cbdff613d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.