Cinder external ceph volumes don't work unless nova ceph backend is enabled

[Will Szumski]

In a cloud with external ceph [1] configured as the backend for Cinder but not nova, we see problems when creating volume-backed instances. The following errors are seen in nova logs:

2018-06-21 11:29:01.454 24 ERROR nova.scheduler.utils [req-504318b7-666c-44ae-b3b8-e57388be604e aa63d0eb27bd41fbb8a7de48556a972e c90400f
36cdb434984a3c436d34630b9 - default default] [instance: 41e3cbe2-19a2-48ca-aeef-26f387d499f0] Error from last host: svn4-dr06-u7 (node s
vn4-dr06-u7): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1840,
in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.
py", line 2117, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build
of instance 41e3cbe2-19a2-48ca-aeef-26f387d499f0 was re-scheduled: Secret not found: no secret with matching uuid '12345-6789-1011-12
13-1456789'\n"]

and:

(openstack) 1 stack@eod-seed:~/will/kayobe-env/src/kayobe$ openstack server show test -c fault -f json
{
  "fault": {
    "message": "Build of instance f1bceb3a-4e8a-4cc6-b775-f5fe1ec8901e aborted: Volume attachment 2e6ee797-8e27-4b7e-858f-59ec37fa998b could not be found.",
    "code": 500,
    "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 1840, in _do_build_and_run_instance\n filter_properties, request_spec)\n File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2062, in _build_and_run_instance\n bdms=block_device_mapping)\n File \"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n self.force_reraise()\n File \"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n six.reraise(self.type_, self.value, self.tb)\n File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2014, in _build_and_run_instance\n block_device_mapping) as resources:\n File \"/usr/lib64/python2.7/contextlib.py\", line 17, in __enter__\n return self.gen.next()\n File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 2221, in _build_resources\n reason=e.format_message())\n",
    "created": "2018-06-21T10:12:45Z"
  }

if you haven't configured the nova ceph backend. It should be possible to use cinder with nova without enabling this.

Here is a snippet of globals.yml configuration that leads to this issue:

enable_ceph: "no"
cinder_backend_ceph: "yes"
nova_backend_ceph: "no"

[1] https://docs.openstack.org/kolla-ansible/latest/reference/external-ceph-guide.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577239
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=19b1e9c08dcb706ad7e396db6e03d3726f35747c
Submitter: Zuul
Branch: master

commit 19b1e9c08dcb706ad7e396db6e03d3726f35747c
Author: Will Szumski <email address hidden>
Date: Thu Jun 21 17:36:30 2018 +0100

    Allow nova to use cephfs cinder volumes without nova cephfs backend

    It is not necessary to enable the nova cephfs backend to make use
    of cinder cephfs volumes.

    Change-Id: I35c3d2e49769962e5c47f585d91d1efd492a53d6
    Closes-Bug: #1778107

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 7.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633017
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=00e7a8342b42099eac107a31f1fde9d958b15ad8
Submitter: Zuul
Branch: stable/queens

commit 00e7a8342b42099eac107a31f1fde9d958b15ad8
Author: Will Szumski <email address hidden>
Date: Thu Jun 21 17:36:30 2018 +0100

    Allow nova to use ceph cinder volumes without nova rbd backend

    It is not necessary to enable the nova rbd backend to make use of cinder
    ceph volumes.

    Note: the commit message was changed from the cherry picked commit to
    reflect that this refers to Ceph RBD, not CephFS.

    Change-Id: I35c3d2e49769962e5c47f585d91d1efd492a53d6
    Closes-Bug: #1778107
    (cherry picked from commit 19b1e9c08dcb706ad7e396db6e03d3726f35747c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 6.2.0 release.