Nova failed to mount a RBD volume without extra ceph.conf

[xiaoxi,chen]

When mounting a RBD volume to an instance, nova-volume/cinder doesn't pass Ceph monitor(MON) addresses to libvirt.Whithout the MON address, libvirt failed to tal with MON and then cannot plug in the disk into volume.
This bug also affect cinder.

Here is the error log from nova-compute:

2012-11-12 12:12:36 ERROR nova.compute.manager [req-0bb59b29-5d0a-4e1d-8f90-b3ceffb41ddb 94baac25958a4400a95535b95fad6012 4b2e53cca0
3245d0ac58041f1b6489d1] [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] Failed to attach volume a69b76be-ce01-4ede-bfac-8230d5e4c48
f at /dev/vdb
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] Traceback (most recent call last):
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] File "/usr/lib/python2.7/dist-pack
ages/nova/compute/manager.py", line 1971, in _attach_volume
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] mountpoint)
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] File "/usr/lib/python2.7/dist-pack
ages/nova/exception.py", line 117, in wrapped
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] temp_level, payload)
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] File "/usr/lib/python2.7/contextli
b.py", line 24, in __exit__
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] self.gen.next()
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] File "/usr/lib/python2.7/dist-pack
ages/nova/exception.py", line 92, in wrapped
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] return f(*args, **kw)
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] File "/usr/lib/python2.7/dist-pack
ages/nova/virt/libvirt/driver.py", line 649, in attach_volume
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] raise exception.DeviceIsBusy(dev
ice=mount_device)
2012-11-12 12:12:36 TRACE nova.compute.manager [instance: 5b919282-6e93-4b16-9bb5-9cb95d25bc20] DeviceIsBusy: The supplied device (v
db) is busy.

When looking into libvirt's log,the following error presents:

unable to find any monitors in conf. please specify monitors via -m monaddr or -c ceph.conf

Here is the XML generated by libvirt,
<disk type="network" device="disk">
  <driver name="qemu" type="raw" cache="none"/>
  <source protocol="rbd" name="nova/volume-a69b76be-ce01-4ede-bfac-8230d5e4c48f"/>
  <target bus="virtio" dev="vdb"/>
  <serial>a69b76be-ce01-4ede-bfac-8230d5e4c48f</serial>
</disk>
But from the link (http://libvirt.org/storage.html#StorageBackendRBD), we can know that monitor's ips can be specified in XML rather than a extra ceph.conf
     <pool type="rbd">
        <name>myrbdpool</name>
        <source>
          <name>rbdpool</name>
            <host name='1.2.3.4' port='6789'/>
            <host name='my.ceph.monitor' port='6789'/>
            <host name='third.ceph.monitor' port='6789'/>
            <auth username='admin' type='ceph'>
              <secret uuid='2ec115d7-3a88-3ceb-bc12-0ac909a6fd87'/>
            </auth>
        </source>
      </pool>

[Josh Durgin (jdurgin)]

This is how it's designed to work right now. See my response to the question https://answers.launchpad.net/nova/+question/201366.

As I noted there, this should be changed in the future, but that's more of a cleanup/feature than a bug.
Note that http://ceph.com/docs/master/rbd/rbd-openstack/#configure-openstack-ceph-clients includes adding a ceph.conf with
monitor addresses to the compute host.

[xiaoxi,chen (xiaoxichen)]

But as long as ceph.conf can only exist one,it seems hard to have several ceph clusters as backends.
In some scenarios, service provider would like to have ceph cluster A with SSDs but ceph cluster B with SATA disks, to meet different criteria .Something like Amazon's 'standard EBS' and 'Provisioned EBS'.

[Josh Durgin (jdurgin)]

You can use different pools within a single ceph cluster to do that. Pools may share any number of osds, from 0, some, to all, depending on your crush rules (see http://ceph.com/docs/master/cluster-ops/crush-map/). So you could have one pool with ssds, and another with sata disks. Currently this would require two instances of nova-volume or cinder-volume, one for each pool.

[xiaoxi,chen (xiaoxichen)]

Well,we have to admit that there exist some scenarios which require several ceph clusters as backends-although Ceph is power enough to properly combine 2 clusters into 1 by clush rules ,but it require much more effort than just starting several cinder-volume instance and config proper monitor addresses for them.

I am not sure if sheepdog face the same situation since sheepdog need gateway address for data accessing.As a framework,Nova and Cinder should work together to support passing more informations from cinder configuration to libvirt.

[Josh Durgin (jdurgin)]

It's certainly useful, and passing the full connection info from cinder makes sense in the future, I just wouldn't call it a high priority bug. There's a blueprint and a patch for partial support on the nova side https://blueprints.launchpad.net/nova/+spec/better-libvirt-network-volume-support

[Davanum Srinivas (DIMS) (dims-v)]

Can we please reduce the importance for Nova to Medium as well?

[Razique Mahroua (razique)]

I had the same issue, thank you Josh for your answer - at the moment - if people are still facing the issue, they need to create a /etc/ceph/ceph.conf on every compute node that contains the mon addresses - enven if not any ceph component is installed on the nodes

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/30790

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/30791

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/30791
Committed: http://github.com/openstack/cinder/commit/483b84e42b90f2ffe0a09f5e38b85eb64cf8f7d9
Submitter: Jenkins
Branch: master

commit 483b84e42b90f2ffe0a09f5e38b85eb64cf8f7d9
Author: Josh Durgin <email address hidden>
Date: Fri May 24 18:18:30 2013 -0700

    rbd: send ceph monitor addresses with connection info

    Previously we relied on a ceph configuration file on the compute host
    for this information. Sending the info directly from cinder makes more
    complex setups with multiple ceph clusters talking to the same compute
    hosts possible.

    Refresh the monitor addresses for each initialize_connection() call,
    since monitors may be added or removed while cinder-volume is
    running.

    Fixes: bug 1077817
    Signed-off-by: Josh Durgin <email address hidden>

    Change-Id: I34a1fa16ce1f4524ba25832faf3129303e755100

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/30790
Committed: http://github.com/openstack/nova/commit/4d99a3486911badc39be4e089bd4106c03a8a671
Submitter: Jenkins
Branch: master

commit 4d99a3486911badc39be4e089bd4106c03a8a671
Author: Josh Durgin <email address hidden>
Date: Fri May 24 17:31:17 2013 -0700

    libvirt: improve the specification of network disks

    Allow zero or more hosts, and make the name optional, since only a
    host is needed to use nbd. This allows cinder to fully specify the
    location of a sheepdog or rbd volume without depending on extra
    configuration files, and lays the groundwork for supporting nbd.

    Rename the internal attribute source_host to source_name to reflect
    its usage in the libvirt xml, and add source_hosts and source_ports
    attributes to store the new information.

    Fixes: bug 1077817
    blueprint better-libvirt-network-volume-support
    Signed-off-by: Josh Durgin <email address hidden>

    Change-Id: I8ac431751692e52ba0786768cea996388962922d