scheduler fails to handle empty capabilities

[Huang Zhiteng]

On one of the old stable release, we encountered a bug in scheduler where it failed to handle empty capabilities reported by faulty driver. So even only one of the backend didn't behave well, scheduler would raise exception every time it handles scheduling request. The latest scheduler in trunk still have this issue.

2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/patterns/linear_flow.py", line 172, in run_it
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp result = runner(context, *args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/utils.py", line 260, in __call__
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp self.result = self.task(*args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/decorators.py", line 148, in wrapper
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/decorators.py", line 264, in wrapper
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/decorators.py", line 199, in wrapper
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/decorators.py", line 234, in wrapper
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/taskflow/decorators.py", line 177, in wrapper
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1715, in schedule_create_volume
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp _error_out_volume(context, db, volume_id, reason=e)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/volume/flows/create_volume/__init__.py", line 1700, in schedule_create_volume
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 65, in schedule_create_volume
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 236, in _schedule
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp filter_properties)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/scheduler/filter_scheduler.py", line 219, in _get_weighted_candidates
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp hosts = self.host_manager.get_all_host_states(elevated)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/scheduler/host_manager.py", line 286, in get_all_host_states
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp host_state.update_from_volume_capability(capabilities)
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp File "/opt/openstack/cinder/2013.2.3.r6/lib/python2.7/site-packages/cinder/scheduler/host_manager.py", line 129, in update_from_volume_capability
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp self.total_capacity_gb = capability['total_capacity_gb']
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp KeyError: 'total_capacity_gb'
2014-12-01 18:43:14.335 6384 TRACE cinder.openstack.common.rpc.amqp

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139584

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/139584
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=6ac5054ceb398b55a03d311ff868e5c93a65dcbb
Submitter: Jenkins
Branch: master

commit 6ac5054ceb398b55a03d311ff868e5c93a65dcbb
Author: Zhiteng Huang <email address hidden>
Date: Fri Dec 5 18:01:53 2014 +0800

    Allow HostState to handle empty capabilities

    In some cases, where the capabilities reported from driver is
    emtpy, scheduler would raise exception thus unable to correctly
    handle scheduling request even when other backend candidates are
    able to fulfill the request. This change does some modifications
    to HostState so that it is able to handle empty capabilities.

    Change-Id: Icb6522852d427dea3fbd1ec4b4cc94a184d3669d
    Fixes-bug: #1398875