failed to attach volumes to instances after configuration change & services restart

[Yogev Rabl]

Description of problem:
The attachment of volumes failed with the errors that are available in the log file attached. Prior to the error I was running 8 active instances, made a configuration change - increased the number of workers in the Cinder, Nova & Glance services, then restarted the services.

Ran the command:
# nova volume-attach 6aac6fb6-ef22-48b0-b6ac-99bc94787422 57edbc5c-8a1f-49f2-b8bf-280ab857222d auto

+----------+--------------------------------------+
| Property | Value |
+----------+--------------------------------------+
| device | /dev/vdc |
| id | 57edbc5c-8a1f-49f2-b8bf-280ab857222d |
| serverId | 6aac6fb6-ef22-48b0-b6ac-99bc94787422 |
| volumeId | 57edbc5c-8a1f-49f2-b8bf-280ab857222d |
+----------+--------------------------------------+

cinder list output:
 +--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+
| 57edbc5c-8a1f-49f2-b8bf-280ab857222d | available | dust-bowl | 100 | None | false | |
| 731a118d-7bd6-4538-a3b2-60543179281e | available | bowl-the-dust | 100 | None | false | |
+--------------------------------------+-----------+---------------+------+-------------+----------+-------------+

Version-Release number of selected component (if applicable):
python-cinder-2014.1-7.el7ost.noarch
openstack-nova-network-2014.1-7.el7ost.noarch
python-novaclient-2.17.0-2.el7ost.noarch
openstack-cinder-2014.1-7.el7ost.noarch
openstack-nova-common-2014.1-7.el7ost.noarch
python-cinderclient-1.0.9-1.el7ost.noarch
openstack-nova-compute-2014.1-7.el7ost.noarch
openstack-nova-conductor-2014.1-7.el7ost.noarch
openstack-nova-scheduler-2014.1-7.el7ost.noarch
openstack-nova-api-2014.1-7.el7ost.noarch
openstack-nova-cert-2014.1-7.el7ost.noarch
openstack-nova-novncproxy-2014.1-7.el7ost.noarch
python-nova-2014.1-7.el7ost.noarch
openstack-nova-console-2014.1-7.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Launch instances
2. Increase the number of workers for the Cinder, Nova & Glance
3. Create a volume
4. Attach the volume to the instance.

Actual results:
The attachment process fail.

Expected results:
The volume should be attached to the instance.

[Yogev Rabl (yrabl)]

As a workaround, with new instances the volume are been attached.

[Nikola Đipanov (ndipanov)]

Worth noting this is reported against Icehouse.

I failed to reproduce it on a fresh install, so marking as incomplete.