cinder volume stuck in 'error ' state in sm-provisioned cluster

[wenqing liang]

SM-provisioned ML-2744 kilo cluster

root@cmbu-ceph-perf1:/var/log/cinder# ceph -s
    cluster 683fa351-e502-4fdb-af2d-73def45c28b1
     health HEALTH_OK
     monmap e2: 4 mons at {cmbu-ceph-perf1=5.0.0.1:6789/0,cmbu-ceph-perf2=5.0.0.2:6789/0,cmbu-ceph-perf3=5.0.0.3:6789/0,cmbu-ceph-perf4=5.0.0.4:6789/0}
            election epoch 8, quorum 0,1,2,3 cmbu-ceph-perf1,cmbu-ceph-perf2,cmbu-ceph-perf3,cmbu-ceph-perf4
     osdmap e60: 12 osds: 12 up, 12 in
      pgmap v138: 832 pgs, 3 pools, 696 MB data, 94 objects
            1829 MB used, 8639 GB / 8640 GB avail
                 832 active+clean

root@cmbu-ceph-perf1:~# cinder create --image-id fe457f51-95b3-4322-b196-31bfbdc98664 --display-name MFC_VOLUME 10
+---------------------------------------+--------------------------------------+
| Property | Value |
+---------------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| consistencygroup_id | None |
| created_at | 2016-10-04T18:23:19.000000 |
| description | None |
| encrypted | False |
| id | 17c2df4b-1da0-440a-b707-8fbee8a43f2c |
| metadata | {} |
| multiattach | False |
| name | MFC_VOLUME |
| os-vol-host-attr:host | None |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | c2b91069fdc34b3b8ead9be2ed4de8a8 |
| os-volume-replication:driver_data | None |
| os-volume-replication:extended_status | None |
| replication_status | disabled |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | creating |
| user_id | 4fcdbcc763cf44b79f7fedaac4062aa3 |
| volume_type | None |
+---------------------------------------+--------------------------------------+
root@cmbu-ceph-perf1:~#
root@cmbu-ceph-perf1:/var/log/cinder# cinder list
+--------------------------------------+--------+------------+------+----------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+------------+------+----------------+----------+-------------+
| 17c2df4b-1da0-440a-b707-8fbee8a43f2c | error | MFC_VOLUME | 10 | None | false | |
| e25b3779-254b-4b43-a141-1a0eb8210ae6 | error | test | 10 | ocs-block-disk | false | |
+--------------------------------------+--------+------------+------+----------------+----------+-------------+
root@cmbu-ceph-perf1:/var/log/cinder# ls -ltr
total 1856
-rw-r--r-- 1 cinder cinder 568633 Oct 4 11:29 cinder-scheduler.log
-rw-r--r-- 1 cinder cinder 152494 Oct 4 11:32 cinder-manage.log
-rw-r--r-- 1 cinder cinder 269253 Oct 4 11:33 cinder-volume.log
-rw-r--r-- 1 cinder cinder 897652 Oct 4 11:33 cinder-api.log
root@cmbu-ceph-perf1:/var/log/cinder# tail cinder-volume.log
2016-10-04 11:31:15.264 19779 INFO cinder.volume.manager [req-ccf4e9bf-c816-49f4-a47b-09b0f20b07f5 - - - - -] Updating volume status
2016-10-04 11:31:15.265 19779 WARNING cinder.volume.manager [req-ccf4e9bf-c816-49f4-a47b-09b0f20b07f5 - - - - -] Unable to update stats, RBDDriver -1.1.0 (config name rbd-disk) driver is uninitialized.
2016-10-04 11:32:15.264 19779 DEBUG cinder.openstack.common.periodic_task [req-a4f0c5fb-47b8-4ea0-b455-1d906d260359 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/cinder/openstack/common/periodic_task.py:219
2016-10-04 11:32:15.265 19779 DEBUG cinder.openstack.common.periodic_task [req-a4f0c5fb-47b8-4ea0-b455-1d906d260359 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/cinder/openstack/common/periodic_task.py:219
2016-10-04 11:32:15.265 19779 INFO cinder.volume.manager [req-a4f0c5fb-47b8-4ea0-b455-1d906d260359 - - - - -] Updating volume status
2016-10-04 11:32:15.266 19779 WARNING cinder.volume.manager [req-a4f0c5fb-47b8-4ea0-b455-1d906d260359 - - - - -] Unable to update stats, RBDDriver -1.1.0 (config name rbd-disk) driver is uninitialized.
2016-10-04 11:33:15.265 19779 DEBUG cinder.openstack.common.periodic_task [req-5f3e582e-91a6-465d-864d-aa9677fe07f7 - - - - -] Running periodic task VolumeManager._publish_service_capabilities run_periodic_tasks /usr/lib/python2.7/dist-packages/cinder/openstack/common/periodic_task.py:219
2016-10-04 11:33:15.265 19779 DEBUG cinder.openstack.common.periodic_task [req-5f3e582e-91a6-465d-864d-aa9677fe07f7 - - - - -] Running periodic task VolumeManager._report_driver_status run_periodic_tasks /usr/lib/python2.7/dist-packages/cinder/openstack/common/periodic_task.py:219
2016-10-04 11:33:15.266 19779 INFO cinder.volume.manager [req-5f3e582e-91a6-465d-864d-aa9677fe07f7 - - - - -] Updating volume status
2016-10-04 11:33:15.267 19779 WARNING cinder.volume.manager [req-5f3e582e-91a6-465d-864d-aa9677fe07f7 - - - - -] Unable to update stats, RBDDriver -1.1.0 (config name rbd-disk) driver is uninitialized.
root@cmbu-ceph-perf1:/var/log/cinder#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24708
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24709
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/24708
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24708
Committed: http://github.org/Juniper/contrail-puppet/commit/e0eb85a779cccd9f3b051ed31baeb9d24b3cec3a
Submitter: Zuul
Branch: master

commit e0eb85a779cccd9f3b051ed31baeb9d24b3cec3a
Author: Dheeraj Gautam <email address hidden>
Date: Wed Oct 5 18:59:11 2016 -0700

SM-Storage: remove rabbit-host from storgae.

Closes-Bug: #1630373

1. remove rabbit_host config from storage. as this would be taken care
openstack role
2. rectified the ipaddress for keystone configuration

PATCH 2:
no changes needed for contrail_openstack_ip. this is already calculated by
contrail/storage.pp

Change-Id: I94f2810d6002fa5f3eb236afeee562b147a58477

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/24709
Committed: http://github.org/Juniper/contrail-puppet/commit/403db1f2129013ea59e529faf8265143fa1cf806
Submitter: Zuul
Branch: R3.1

commit 403db1f2129013ea59e529faf8265143fa1cf806
Author: Dheeraj Gautam <email address hidden>
Date: Wed Oct 5 18:59:11 2016 -0700

SM-Storage: remove rabbit-host from storgae.

Closes-Bug: #1630373

1. remove rabbit_host config from storage. as this would be taken care
openstack role
2. rectified the ipaddress for keystone configuration

PATCH 2:
no changes needed for contrail_openstack_ip. this is already calculated by
contrail/storage.pp

Change-Id: I94f2810d6002fa5f3eb236afeee562b147a58477