senlin

  1. Bug #1680758
  2. Activity log

Activity log for bug #1680758

Date Who What changed Old value New value Message
2017-04-07 09:00:38 yangyide bug added bug
2017-04-07 09:01:17 yangyide attachment added 0001-Improve-node-check-for-health_policy_poll.patch https://bugs.launchpad.net/senlin/+bug/1680758/+attachment/4857137/+files/0001-Improve-node-check-for-health_policy_poll.patch
2017-04-07 09:01:29 yangyide senlin: assignee yangyide (yangyide01)
2017-04-07 09:02:38 yangyide description My senin version is master. My cluster bind health_policy_poll policy, when I delete vm that belong this cluster, node cannot auto recover through policy. Below is error log. I hope node can auto recover, because that is health policy meaning. 2017-04-07 15:04:32.331 21107 INFO senlin.engine.service [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Checking cluster '25018444-961b-4301-a86a-52452ec6a718'. 2017-04-07 15:04:33.738 21107 INFO senlin.engine.service [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Cluster check action queued: 21dd55d7-eb13-43be-a30d-14039f27fa15. 2017-04-07 15:04:35.103 21107 INFO senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] test_cluster [25018444] CLUSTER_CHECK - start: None 2017-04-07 15:04:36.205 21107 INFO senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] node-25018444-232 [81187992] NODE_CHECK - start: None 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] ResourceNotFound: No Server found for 42377008-aca4-4bb0-829e-5afcf9c93a61, Instance 42377008-aca4-4bb0-829e-5afcf9c93a61 could not be found. 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk Traceback (most recent call last): 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/opt/senlin/senlin/drivers/openstack/sdk.py", line 96, in invoke_with_catch 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return func(driver, *args, **kwargs) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/opt/senlin/senlin/drivers/openstack/nova_v2.py", line 50, in server_get 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return self.conn.compute.servers(details=details, **query) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/compute/v2/_proxy.py", line 371, in get_server 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return self._get(_server.Server, server) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 37, in check 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return method(self, expected, actual, *args, **kwargs) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 225, in _get 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk http_status=e.http_status, cause=e.cause) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk ResourceNotFound: ResourceNotFound: No Server found for 42377008-aca4-4bb0-829e-5afcf9c93a61, Instance 42377008-aca4-4bb0-829e-5afcf9c93a61 could not be found. 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk 2017-04-07 15:04:36.705 21107 ERROR senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] node-25018444-232 [81187992] NODE_CHECK - error: Node check failed. 2017-04-07 15:04:37.082 21107 WARNING senlin.engine.health_manager [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Cluster check action failed My senin version is master. My cluster bind health_policy_poll policy, when I delete vm that belong this cluster, node cannot auto recover through policy. Below is error log. I hope node can auto recover, because that is health policy meaning. 2017-04-07 15:04:32.331 21107 INFO senlin.engine.service [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Checking cluster '25018444-961b-4301-a86a-52452ec6a718'. 2017-04-07 15:04:33.738 21107 INFO senlin.engine.service [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Cluster check action queued: 21dd55d7-eb13-43be-a30d-14039f27fa15. 2017-04-07 15:04:35.103 21107 INFO senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] test_cluster [25018444] CLUSTER_CHECK - start: None 2017-04-07 15:04:36.205 21107 INFO senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] node-25018444-232 [81187992] NODE_CHECK - start: None 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] ResourceNotFound: No Server found for 42377008-aca4-4bb0-829e-5afcf9c93a61, Instance 42377008-aca4-4bb0-829e-5afcf9c93a61 could not be found. 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk Traceback (most recent call last): 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/opt/senlin/senlin/drivers/openstack/sdk.py", line 96, in invoke_with_catch 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return func(driver, *args, **kwargs) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/opt/senlin/senlin/drivers/openstack/nova_v2.py", line 50, in server_get 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return self.conn.compute.servers(details=details, **query) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/compute/v2/_proxy.py", line 371, in get_server 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return self._get(_server.Server, server) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 37, in check 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk return method(self, expected, actual, *args, **kwargs) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 225, in _get 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk http_status=e.http_status, cause=e.cause) 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk ResourceNotFound: ResourceNotFound: No Server found for 42377008-aca4-4bb0-829e-5afcf9c93a61, Instance 42377008-aca4-4bb0-829e-5afcf9c93a61 could not be found. 2017-04-07 15:04:36.578 21107 ERROR senlin.drivers.openstack.sdk 2017-04-07 15:04:36.705 21107 ERROR senlin.engine.event [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] node-25018444-232 [81187992] NODE_CHECK - error: Node check failed. 2017-04-07 15:04:37.082 21107 WARNING senlin.engine.health_manager [req-9801a74c-e95d-4ed3-9748-ff8feabbfdc5 - - - - -] Cluster check action failed
2017-04-11 03:52:24 OpenStack Infra senlin: status New In Progress
2017-04-14 01:20:11 Qiming Teng senlin: importance Undecided High
2017-04-14 01:20:15 Qiming Teng senlin: milestone pike-2
2017-04-14 01:21:13 Qiming Teng summary health_policy_poll cannot recover node which vm was deleted Node check cannot properly handle resource deletion
2017-04-14 08:04:53 OpenStack Infra senlin: status In Progress Fix Released