unable to delete lbaasv2 health monitor if its listener deleted

[alex kang]

problem is in Kilo neutron-lbaas branch.

monitor is attached a pool.
When pool and listener were deleted, not error reported that there is a health-monitor associated to pool.

If all lbaas resoruces except health-monitor were deleted, health monitor can not be deleted.

See the following procedure to reproduce this issue:

$ neutron lbaas-loadbalancer-create --name=v-lb2 lb2-v-1574810802
$ neutron lbaas-listener-create --protocol=HTTP --protocol-port=80 --name=v-lb2-1 --loadbalancer=v-lb2
$ neutron lbaas-pool-create --lb-algorithm=ROUND_ROBIN --protocol=HTTP --name=v-lb2-pool --listener=v-lb2-1
$ neutron lbaas-member-create --subnet lb2-v-1574810802 --address 10.199.88.3 --protocol-port=80 v-lb2-pool
$ neutron lbaas-member-create --subnet lb2-v-1574810802 --address 10.199.88.4 --protocol-port=80 v-lb2-pool
$ neutron lbaas-healthmonitor-create --max-retries=3 --delay=3 --timeout=10 --type=HTTP --pool=v-lb2-pool
$ neutron lbaas-member-delete 4d2977fc-5600-4dbf-8af2-35c017c8f4a0 v-lb2-pool
$ neutron lbaas-member-delete 2f60a49b-add1-43d6-97d8-4e53a925b25f v-lb2-pool
$ neutron lbaas-pool-delete v-lb2-pool
$ neutron lbaas-listener-delete v-lb2-1
$ neutron lbaashealthmonitor-delete 044f98a5-755d-498d-a38e-18eb8ca13884

neutron log seems point to lbaas resources were gone.
In this specific issue, we should just remove the health monitor from system.

2016-04-10 16:57:38.220 INFO neutron.wsgi [req-7e697943-e70d-4ac8-a840-b1edf441806a Venus Venus] 10.34.57.68 - - [10/Apr/2016 16:57:38] "GET /v2.0/lbaas/healthmonitors.json?fields=id&id=044f98a5-755d-498d-a38e-18eb8ca13884 HTTP/1.1" 200 444 0.112257
2016-04-10 16:57:38.252 ERROR neutron.api.v2.resource [req-aaeae392-33b2-427c-96a0-918782882c9a Venus Venus] delete failed
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource Traceback (most recent call last):
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 83, in resource
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource result = method(request=request, **args)
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 490, in delete
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource obj_deleter(request.context, id, **kwargs)
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource File "/opt/stack/neutron-lbaas/neutron_lbaas/services/loadbalancer/plugin.py", line 906, in delete_healthmonitor
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource constants.PENDING_DELETE)
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource File "/opt/stack/neutron-lbaas/neutron_lbaas/db/loadbalancer/loadbalancer_dbv2.py", line 163, in test_and_set_status
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource db_lb_child.root_loadbalancer.id)
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource File "/opt/stack/neutron-lbaas/neutron_lbaas/db/loadbalancer/models.py", line 115, in root_loadbalancer
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource return self.pool.listener.loadbalancer
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource AttributeError: 'NoneType' object has no attribute 'listener'
2016-04-10 16:57:38.252 4158 TRACE neutron.api.v2.resource
2016-04-10 16:57:38.253 INFO neutron.wsgi [req-aaeae392-33b2-427c-96a0-918782882c9a Venus Venus] 10.34.57.68 - - [10/Apr/2016 16:57:38] "DELETE /v2.0/lbaas/healthmonitors/044f98a5-755d-498d-a38e-18eb8ca13884.json HTTP/1.1" 500 383 0.030720

[Brandon Logan (brandon-logan)]

Looks like the lbaasv2 API needs to fail the pool delete request if the column neutron.lbaas_pools.healthmonitor_id is populated for that pool. It obviously should return back to the user why and what healthmonitor needs to be deleted first.

[Zhou Zhihong (shuihushen)]

This bug is same to https://bugs.launchpad.net/neutron/+bug/1526964

[Evgeny Fedoruk (evgenyf)]

Why should pool's HM be handled differently from pool's members?
The members are deleted cascade when pool is deleted.

[Kobi Samoray (ksamoray)]

Evgeny - the driver can cascade the deletion, yet the plugin should delete the healthmonitor from the DB.
As of now it doesn't - not with Octavia, at least.

[Evgeny Fedoruk (evgenyf)]

I wonder what is the right approach here, I tested the cascade deletion of HM when pool is deleted, just alike members, and it solves the problem.
But, I think (and I may be mistaken) there is an intention to share HM in feature. If HM will be shared among pools, pool deletion should not delete the HM (like it does now) and HM creation should not depend on any pool.

If HM is not shared, it should be deleted cascade like members.

What do you think?

[Brandon Logan (brandon-logan)]

if pool deletion did not cascade delete the HM in previous versions, then we shouldn't start doing it now because that will break backwards compatibility.

[Evgeny Fedoruk (evgenyf)]

Can you please elaborate? As I understand, we delete pool members cascade because members are not shared among pools. The HM is not shared also.

The difference is that member has a pool_id as a FK and this is a one-to-many relationship, and HM has no FK to a pool, I do not know what is the reason for that.

When pool with HM is deleted, HM just remains stuck in DB table and can not be re-used in any way with the current API. If we will start deleting it, how backwards compatibility will be broken?

[Brandon Logan (brandon-logan)]

what I mean is if we changed the API to cascade delete a pool's health monitor if that pool is deleted, then that has changed the behavior of the API in a backwards incompatible way. I'm just saying we can't do a cascade delete like behavior from the API perspective. What we do need is to prevent a pool from being deleted if it has a health monitor associated with it.

[Evgeny Fedoruk (evgenyf)]

Sorry, but I still don't understand.
As for now, API allows to delete pool with HM. HM remains in DB and can not be re-used, so actually it's garbage. When we will delete it from DB - we will just clean-up garbage. User experience will not be changed.

In other hand, if we will prevent deleting pool with HM, here is where we change API behavior.

I may miss your point...

[Evgeny Fedoruk (evgenyf)]

Ok, after discussing the issue with Brandon Logan, the solution seems clear.
Since HM is a first-level resource and not a sub resource of the pool (like member),
deleting it cascade along with a pool deletion does not follow consistency of neutron's API.

So, allowing pool deletion when there is HM associated with it, is initially a bug.
That's what should be fixed. Plugin should prevent pool deletion if pool has HM.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/324380

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/324380
Committed: https://git.openstack.org/cgit/openstack/neutron-lbaas/commit/?id=4effc9b96ef6317cefb25768dd60bd0f5f4abac5
Submitter: Jenkins
Branch: master

commit 4effc9b96ef6317cefb25768dd60bd0f5f4abac5
Author: Evgeny Fedoruk <email address hidden>
Date: Thu Jun 2 01:29:42 2016 -0700

    Preventing pool deletion if pool has healthmonitor

    If HM is associated to pool, HM should be deleted prior
    to pool deletion.
    Trying to delete pool with HM will fail with EntityInUse exception.
    This is to preserve common neutron's API concept to delete cascade
    resources' subresources only.

    Change-Id: I1bfc4d8d8ec7e83b1de11c8fb3e66282bfd06806
    Closes-Bug: 1571097

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/329073

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/329073
Committed: https://git.openstack.org/cgit/openstack/neutron-lbaas/commit/?id=db3b921a174b0104d3858d57fde2ac0fb77e2c57
Submitter: Jenkins
Branch: stable/mitaka

commit db3b921a174b0104d3858d57fde2ac0fb77e2c57
Author: Evgeny Fedoruk <email address hidden>
Date: Thu Jun 2 01:29:42 2016 -0700

    Preventing pool deletion if pool has healthmonitor

    If HM is associated to pool, HM should be deleted prior
    to pool deletion.
    Trying to delete pool with HM will fail with EntityInUse exception.
    This is to preserve common neutron's API concept to delete cascade
    resources' subresources only.

    Change-Id: I1bfc4d8d8ec7e83b1de11c8fb3e66282bfd06806
    Closes-Bug: 1571097
    (cherry picked from commit 4effc9b96ef6317cefb25768dd60bd0f5f4abac5)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron-lbaas 9.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron-lbaas 8.2.0 release.