Node check cannot properly handle resource deletion

[yangyide]

My senin version is master.

My cluster bind health_policy_poll policy, when I delete vm that belong this cluster, node cannot auto recover through policy.

Below is error log. I hope node can auto recover, because that is health policy meaning.

2017-04-07 15:04:32.331 21107 INFO senlin.engine.service [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Checking cluster '25018444-961b-4301-a86a-52452ec6a718'.
2017-04-07 15:04:33.738 21107 INFO senlin.engine.service [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Cluster check action queued: 21dd55d7-eb13-43be-a30d-14039f27fa15.
2017-04-07 15:04:35.103 21107 INFO senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] test_cluster [25018444] CLUSTER_CHECK - start: None
2017-04-07 15:04:36.205 21107 INFO senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] node-25018444-232 [81187992] NODE_CHECK - start: None
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] ResourceNotFound: No Server found for 42377008-aca4-4bb0-829e-5afcf9c93a61, Instance 42377008-aca4-4bb0-829e-5afcf9c93a61 could not be found.
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk Traceback (most recent call last):
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/opt/senlin/senlin/drivers/openstack/sdk.py", line 96, in invoke_with_catch
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return func(driver, *args, **kwargs)
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/opt/senlin/senlin/drivers/openstack/nova_v2.py", line 50, in server_get
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return self.conn.compute.servers(details=details, **query)
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/compute/v2/_proxy.py", line 371, in get_server
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return self._get(_server.Server, server)
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 37, in check
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return method(self, expected, actual, *args, **kwargs)
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 225, in _get
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk http_status=e.http_status, cause=e.cause)
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk ResourceNotFound: ResourceNotFound: No Server found for 42377008-aca4-4bb0-829e-5afcf9c93a61, Instance 42377008-aca4-4bb0-829e-5afcf9c93a61 could not be found.
2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk
2017-04-07 15:04:36.705 21107 ERROR senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] node-25018444-232 [81187992] NODE_CHECK - error: Node check failed.
2017-04-07 15:04:37.082 21107 WARNING senlin.engine.health_manager [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Cluster check action failed

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/455503

[Qiming Teng (tengqim)]

Please describe how you deleted the VM.

[yangyide (yangyide01)]

I directly use nova delete vm-uuid, that is to say ，if other people delete my servers, but if my cluster binding
health policy, should auto recover my nodes.

[yangyide (yangyide01)]

I think for senlin/profiles/os/nova/server.py do_check, if checked vm exist nova instances table, function should

return true that vm statue is active, else return false even if this vm was deleted. if not exist nova instances table

,function raise exception.

[Qiming Teng (tengqim)]

to clarify, senlin never peeks into nova (or any other services' db table). We don't care how data are organized and stored by the corresponding service. We do, however, trust a service's public APIs.

In this context, a GET to nova servers/<server_id> will return 404 if the server is deleted. The response will be captured by openstacksdk and the latter raises a NotFound exception. Senlin does an exception type translation to make code better managed.

There is no such thing as a nova server is gone but Senlin is misunderstanding it.

The problem you described could be a bug how senlin treats a ResourceNotFound exception when it does a node check. If the node is not found, we should still treat it as a node failure and continue to do a recover.

With that said, the patch proposed is an incorrect solution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/456187

[yangyide (yangyide01)]

I dont know why raise exception in senlin/profiles/os/nova/server.py do_check function, Can we return false when happen
exception?

[Qiming Teng (tengqim)]

The reason we raise exceptions instead of returning False is that:

1. An exception can carry more contextual information about the error that occurred.

2. The 'False' return value may and may not convey the complete message. Sometimes it causes confusion.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/456187
Committed: https://git.openstack.org/cgit/openstack/senlin/commit/?id=616d4ed2778ac9cb9f171010e5355a0e4b599f34
Submitter: Jenkins
Branch: master

commit 616d4ed2778ac9cb9f171010e5355a0e4b599f34
Author: YiDe Yang <[e-mail]>
Date: Wed Apr 12 19:41:10 2017 +0800

    Improve check_object for health_policy_poll recover

    check_object return false when happen exception,
    for recover nodes which cluster policy binding health_policy_poll.

    Closes-Bug: #1680758

    Change-Id: I34d189e0a8f29191deb3b1722de66bd51bd471fb

[OpenStack Infra (hudson-openstack)]

Change abandoned by yangyide ([e-mail]) on branch: master
Review: https://review.openstack.org/455503

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/senlin 4.0.0.0b2 development milestone.