DVR-HA: Removing a router from an agent, does not clear the namespaces on the agent

[Swaminathan Vasudevan]

Removing an active or a standby ha-router from an agent, does not clear the router namespace and the Snat namespaces.
This leads to sometimes having two Active HA routers and two 'ha-interface' in the snat namespace for DVR routers.
This can be very easily reproduced.

1. Create a HA-DVR router. ( minimum two node setup with 'dvr_snat' agent mode)
2. Attach interface to the router
3. Attach gateway to the router.
4. Now check the l3-agent-list-hosting-router for router.
5. Then remove the router from one of the agent ( l3-agent-router-remove )
6. Expected result is router namespace and snat namespace to be removed. ( But it is removed).
7. At the minimum we should clear the HA interfaces for that agent so that the HA router does not get into Active mode again.

This bug might have been introduced by this patch.
https://review.openstack.org/#/c/522362/7

This bug is seen since Ocata/Pike and probably also in master branch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/638566

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/638566
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d9e0bab6acf6a608ed5bf32524c7d362f052a3ac
Submitter: Zuul
Branch: master

commit d9e0bab6acf6a608ed5bf32524c7d362f052a3ac
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Feb 21 17:14:03 2019 -0800

    DVR-HA: Unbinding a HA router from agent does not clear HA interface

    Removing an active or a standby HA router from an agent that has a
    valid DVR serviceable port (such as DHCP), does not remove the
    HA interface associated with the Router in the SNAT namespace.

    When we try to add the HA router back to the agent, then it
    adds more than one HA interface to the SNAT Namespace causing
    more problems and we sometimes also see multiple active routers.

    This bug might have been introduced by this patch [1].

    Fix the problem by just adding the router namespaces without HA
    interfaces when there is no HA and re-insert the HA interfaces
    when HA router is bound to the agent into the namespace.

    [1] https://review.openstack.org/#/c/522362/
    Closes-Bug: #1816698

    Change-Id: Ie625abcb73f8185bb2bee06dcd26a01d8af0b0d1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/642304

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/642348

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/642357

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/642363

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/642304
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3c66b1c45394ddb4b6bca757f3ce9d76caeeba85
Submitter: Zuul
Branch: stable/rocky

commit 3c66b1c45394ddb4b6bca757f3ce9d76caeeba85
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Feb 21 17:14:03 2019 -0800

    DVR-HA: Unbinding a HA router from agent does not clear HA interface

    Removing an active or a standby HA router from an agent that has a
    valid DVR serviceable port (such as DHCP), does not remove the
    HA interface associated with the Router in the SNAT namespace.

    When we try to add the HA router back to the agent, then it
    adds more than one HA interface to the SNAT Namespace causing
    more problems and we sometimes also see multiple active routers.

    This bug might have been introduced by this patch [1].

    Fix the problem by just adding the router namespaces without HA
    interfaces when there is no HA and re-insert the HA interfaces
    when HA router is bound to the agent into the namespace.

    [1] https://review.openstack.org/#/c/522362/
    Closes-Bug: #1816698

    Change-Id: Ie625abcb73f8185bb2bee06dcd26a01d8af0b0d1
    (cherry picked from commit d9e0bab6acf6a608ed5bf32524c7d362f052a3ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/642348
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2e033b3b0d57268856eb453802148129ebd0c954
Submitter: Zuul
Branch: stable/queens

commit 2e033b3b0d57268856eb453802148129ebd0c954
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Feb 21 17:14:03 2019 -0800

    DVR-HA: Unbinding a HA router from agent does not clear HA interface

    Removing an active or a standby HA router from an agent that has a
    valid DVR serviceable port (such as DHCP), does not remove the
    HA interface associated with the Router in the SNAT namespace.

    When we try to add the HA router back to the agent, then it
    adds more than one HA interface to the SNAT Namespace causing
    more problems and we sometimes also see multiple active routers.

    This bug might have been introduced by this patch [1].

    Fix the problem by just adding the router namespaces without HA
    interfaces when there is no HA and re-insert the HA interfaces
    when HA router is bound to the agent into the namespace.

    [1] https://review.openstack.org/#/c/522362/
    Closes-Bug: #1816698

    Change-Id: Ie625abcb73f8185bb2bee06dcd26a01d8af0b0d1
    (cherry picked from commit d9e0bab6acf6a608ed5bf32524c7d362f052a3ac)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/642363
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d145f978665cc2c3311a47f39b4ce5ad3712adff
Submitter: Zuul
Branch: stable/ocata

commit d145f978665cc2c3311a47f39b4ce5ad3712adff
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Feb 21 17:14:03 2019 -0800

    DVR-HA: Unbinding a HA router from agent does not clear HA interface

    Removing an active or a standby HA router from an agent that has a
    valid DVR serviceable port (such as DHCP), does not remove the
    HA interface associated with the Router in the SNAT namespace.

    When we try to add the HA router back to the agent, then it
    adds more than one HA interface to the SNAT Namespace causing
    more problems and we sometimes also see multiple active routers.

    This bug might have been introduced by this patch [1].

    Fix the problem by just adding the router namespaces without HA
    interfaces when there is no HA and re-insert the HA interfaces
    when HA router is bound to the agent into the namespace.

    [1] https://review.openstack.org/#/c/522362/
    Conflicts:
              neutron/agent/l3/agent.py
    Closes-Bug: #1816698

    Change-Id: Ie625abcb73f8185bb2bee06dcd26a01d8af0b0d1
    (cherry picked from commit d9e0bab6acf6a608ed5bf32524c7d362f052a3ac)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/642357
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4b49404aee88562f034104cccb3af9453ff45e81
Submitter: Zuul
Branch: stable/pike

commit 4b49404aee88562f034104cccb3af9453ff45e81
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Feb 21 17:14:03 2019 -0800

    DVR-HA: Unbinding a HA router from agent does not clear HA interface

    Removing an active or a standby HA router from an agent that has a
    valid DVR serviceable port (such as DHCP), does not remove the
    HA interface associated with the Router in the SNAT namespace.

    When we try to add the HA router back to the agent, then it
    adds more than one HA interface to the SNAT Namespace causing
    more problems and we sometimes also see multiple active routers.

    This bug might have been introduced by this patch [1].

    Fix the problem by just adding the router namespaces without HA
    interfaces when there is no HA and re-insert the HA interfaces
    when HA router is bound to the agent into the namespace.

    [1] https://review.openstack.org/#/c/522362/
    Conflicts:
              neutron/agent/l3/agent.py
    Closes-Bug: #1816698

    Change-Id: Ie625abcb73f8185bb2bee06dcd26a01d8af0b0d1
    (cherry picked from commit d9e0bab6acf6a608ed5bf32524c7d362f052a3ac)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron ocata-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron pike-eol release.