Scheduler selects stopped volume manager

[Ollie Leahy]

Sometimes if a volume manager goes down we see the scheduler continuing to select that volume manager for volume creation. Resulting in the volume getting stuck in the creating state until the volume manager comes back up.

In the following log snippet the service volmanager0002 is reported to be down, but the scheduler selects it to create a volume.

2013-07-04 08:04:29.075 11585 WARNING cinder.scheduler.host_manager [req-42dafc8c-ad60-46df-8f89-60291b2f5ed5 53940994267304 46514221745886] volume service is down or disabled. (host: cr-aw1rde1-az1-volmanager0002)
2013-07-04 08:04:29.076 11585 DEBUG cinder.scheduler.filter_scheduler [req-42dafc8c-ad60-46df-8f89-60291b2f5ed5 53940994267304 46514221745886] Filtered [host 'cr-aw1rde1-az1-volmanager0001': free_capacity_gb: 99999999997, host 'cr-aw1rde1-az1-volmanager0002': free_capacity_gb: 99999999998] _schedule /usr/lib/python2.7/dist-packages/cinder/scheduler/filter_scheduler.py:212
2013-07-04 08:04:29.076 11585 DEBUG cinder.scheduler.filter_scheduler [req-42dafc8c-ad60-46df-8f89-60291b2f5ed5 53940994267304 46514221745886] Choosing WeighedHost [host: cr-aw1rde1-az1-volmanager0002, weight: 99999999998.0] _schedule /usr/lib/python2.7/dist-packages/cinder/scheduler/filter_scheduler.py:218
2013-07-04 08:04:29.096 11585 DEBUG cinder.openstack.common.rpc.amqp [req-42dafc8c-ad60-46df-8f89-60291b2f5ed5 53940994267304 46514221745886] Making asynchronous cast on cinder-volume.cr-aw1rde1-az1-volmanager0002... cast /usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py:623
2013-07-04 08:04:29.097 11585 DEBUG cinder.openstack.common.rpc.amqp [req-42dafc8c-ad60-46df-8f89-60291b2f5ed5 53940994267304 46514221745886] UNIQUE_ID is c12390b33f804e70899098c0805d3083. _add_unique_id /usr/lib/python2.7/dist-packages/cinder/openstack/common/rpc/amqp.py:337

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/35682

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/35682
Committed: http://github.com/openstack/cinder/commit/3283ce500493d4908e91e002dc82b81e4468b26e
Submitter: Jenkins
Branch: master

commit 3283ce500493d4908e91e002dc82b81e4468b26e
Author: Ollie Leahy <email address hidden>
Date: Thu Jul 4 13:58:08 2013 +0000

    Scheduler should not select down volume managers

    Currently volume managers that are not disabled, but are not running
    may be selected by the scheduler for volume creation operations.

    Change-Id: Ia06a3da8b31eef918d55aaaf71f863683c721026
    Fixes: bug #1197741