HA network remains even if there is no more HA router

[Sylvain Afchain]

Currently when the last HA router of a tenant is deleted the HA network belonging to this tenant is not removed. This is the case in the rollback of a router creation and in the delete_router itself.

[Carl Baldwin (carl-baldwin)]

There is some relevant discussion in this patch in PS 74 [1] and PS 75 [2]

[1] https://review.openstack.org/#/c/64553/74/neutron/db/l3_hamode_db.py
[2] https://review.openstack.org/#/c/64553/75/neutron/db/l3_hamode_db.py

[Assaf Muller (amuller)]

Removed assignee. Eugene, if you'd like to have a crack at this, please take ownership again.

[Yair Fried (yfried)]

This causes resource leakage in Tempest. HA networks aren't cleaned, and, when using limited VNI range, or VLAN range, Tests will start failing once limit is reached.
Suggestion:
Introduce a cleaner to the tenant (not router!) cleanup code:

1. search for the HA network of this tenant (neutron net-list --tenant-id " ") **note that the network has empty tenant-id but the tenant it belongs to is part of the network name.
2. if exist, log a warning.
3. if CONF.network.cleanup_l3_ha: delete the network

This has to be logged and has to be turned off by default.

[yong sheng gong (gongysh)]

agree with Yair Fried,
We should not try to delete the HA network when the last HA router is deleted, it risks race condition with other HA router creation.

We can provide a tool which to remove a tenant's resources.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/207395

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/207395
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/207395
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=853f7d7a74a2281b0cd34c337362e73635e501a2
Submitter: Jenkins
Branch: master

commit 853f7d7a74a2281b0cd34c337362e73635e501a2
Author: sridhargaddam <email address hidden>
Date: Thu Jul 30 10:54:39 2015 +0000

    Delete HA network when last HA router is deleted

    Currently when the last HA router of a tenant is deleted the HA network
    belonging to this tenant is not removed. While running tempest aganist an
    OpenStack setup where tenant VLANs (with small VLAN range) is used we hit
    the limits are tempest tests start to fail as we cannot create new networks.
    This patch addresses this issue by deleting the HA network when the last HA
    router is deleted for the tenant.

    Closes-Bug: #1367157

    Co-Authored-By: Ann Kamyshnikova<email address hidden>

    Change-Id: I1d50b973aed4148857ac3d2bbee0d38e2e199783

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/250266

[Ihar Hrachyshka (ihar-hrachyshka)]

Removed Kilo tag since it does not seem especially critical, networks are still available for explicit clean up.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/250266
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=478012a397029077e8444fb2872a78988d492f74
Submitter: Jenkins
Branch: stable/liberty

commit 478012a397029077e8444fb2872a78988d492f74
Author: sridhargaddam <email address hidden>
Date: Thu Jul 30 10:54:39 2015 +0000

    Delete HA network when last HA router is deleted

    Currently when the last HA router of a tenant is deleted the HA network
    belonging to this tenant is not removed. While running tempest aganist an
    OpenStack setup where tenant VLANs (with small VLAN range) is used we hit
    the limits are tempest tests start to fail as we cannot create new networks.
    This patch addresses this issue by deleting the HA network when the last HA
    router is deleted for the tenant.

    Closes-Bug: #1367157

    Co-Authored-By: Ann Kamyshnikova<email address hidden>

    Change-Id: I1d50b973aed4148857ac3d2bbee0d38e2e199783
    (cherry picked from commit 853f7d7a74a2281b0cd34c337362e73635e501a2)

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/254586

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254586
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f54cba053556a43d51ccd895cdf8232c51210299
Submitter: Jenkins
Branch: master

commit f54cba053556a43d51ccd895cdf8232c51210299
Author: LIU Yulong <email address hidden>
Date: Tue Dec 8 14:13:44 2015 +0800

    Catch known exceptions during deleting last HA router

    In some scenarios, for instance rally test create_and_delete_routers,
    it will get some exceptions, such as the network in use exception,
    during the router deleting api call, but actually the router has
    been deleted. There has race between HA router create and delete,
    if set more api and rpc worker race raises exception more frequently.
    Because the inconsistent error message was not useful for user,
    this patch will catch those know exceptions ObjectDeletedError,
    NetworkInUse when user delete last HA router.

    At the same time, when user create the first HA router, but because
    of the failure of HA network creation, the router will be deleted,
    then the deleting HA network will raise AttributeError, this patch
    also move HA network deleting procedure under ha_network exist check
    block.

    Change-Id: I8cda00c1e7caffc4dfb20a817a11c60736855bb5
    Closes-Bug: #1523780
    Related-Bug: #1367157

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/259580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/259580
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ff021e7e4c7cc28bf4524313d85f521d6ed4c3eb
Submitter: Jenkins
Branch: stable/liberty

commit ff021e7e4c7cc28bf4524313d85f521d6ed4c3eb
Author: LIU Yulong <email address hidden>
Date: Tue Dec 8 14:13:44 2015 +0800

    Catch known exceptions during deleting last HA router

    In some scenarios, for instance rally test create_and_delete_routers,
    it will get some exceptions, such as the network in use exception,
    during the router deleting api call, but actually the router has
    been deleted. There has race between HA router create and delete,
    if set more api and rpc worker race raises exception more frequently.
    Because the inconsistent error message was not useful for user,
    this patch will catch those know exceptions ObjectDeletedError,
    NetworkInUse when user delete last HA router.

    At the same time, when user create the first HA router, but because
    of the failure of HA network creation, the router will be deleted,
    then the deleting HA network will raise AttributeError, this patch
    also move HA network deleting procedure under ha_network exist check
    block.

    Change-Id: I8cda00c1e7caffc4dfb20a817a11c60736855bb5
    Closes-Bug: #1523780
    Related-Bug: #1367157
    (cherry picked from commit f54cba053556a43d51ccd895cdf8232c51210299)

[David Paterson (davpat2112)]

From what I am seeing the HA networks are still being left behind and I verified the patched code in : https://review.openstack.org/#/c/259580/ is present. After running tempest there are many HA networks left behind.

[Ken'ichi Ohmichi (oomichi)]

This bug seems to be fixed on Neutron side, so it is nice to close the tempest bug at this time.