Cannot change volume type using ceph with alternative (non-default) cluster name

[Eric]

Steps to reproduce:

1. Configure a ceph cluster using an alternative cluster name (e.g. 'my-ceph') instead of the default (ceph)
2. Configure two ceph backends using this cluster to use in cinder
3. Add the appropriate volume type(s)
4. Create a volume
5. Change the volume type of the volume

Expected result:

Migration of the volume

Actual result:

Exception in cinder-volume: Error: error calling conf_read_file: errno EINVAL

Cause:

The os-brick RBDConnector initiates a RBDClient only specifying user and pool. This causes the client to default to configuration file "/etc/ceph/ceph.conf"

The cinder rbd volume driver specifies all required information for a connection (while to me it seems to be tailored to nova/libvirt) but it is not used by os-brick. If the data specified by the volume driver cannot be used by the os-brick RBDClient, either the ceph cluster name or full path to the appropriate ceph configfile should be added to the connection data which in turn should be used by the os-brick connector.

PS.

I wouldn't mind writing a patch, but since I'm relatively new to cinder I'm not aware of the bigger picture. Because of that I would need to know what the desired fix should be. Either adapting os-brick to use the already available connection info or appending either the cluster name or config full path to the connection info.

[Danny Al-Gaaf (danny-al-gaaf)]

If I have not overseen something cinder doesn't use os_brick connector for rbd but uses it own implementation. Can you please provide your cinder configuration and commands to reproduce?

[Eric (ericvanblokland)]

I'm using Horizon to issue the commands required for migration.

I've attached the stacktrace (sorry for not including it initially) which clearly shows the volume manager using os-brick. Are you sure this could be caused by configuration?

If so let me know and I'll post the configuration, as I would need to remove some potentially sensitive information.

[Eric (ericvanblokland)]

I've just noticed the cinder-backup process has the same issue.

[jcat (jcat)]

We have the same issue for cinder-backup on mitaka.

We have two different ceph clusters, one for cinder-volume, and one for cinder-backup, each with a different ${cluster_name}.conf.

We get a different error than "conf_read_file", but only because we still have a ceph.conf in place with some default client settings.

Trace attached.

[Sean McGinnis (sean-mcginnis)]

Unassigning due to no activity for > 6 months.

[Xiaojun Liao (wwba)]

Hi, Eric:

I have followed the steps in my newton envirenment, but didn't reproduce it. I configure two ceph backends like this:
[rbd1]
volume_backend_name=rbd1
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=rbd1
rbd_max_clone_depth=5
rbd_user=cinder
rbd_flatten_volume_from_snapshot=False
rbd_ceph_conf=/etc/ceph/ceph.conf

[rbd2]
volume_backend_name=rbd2
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=rbd2
rbd_max_clone_depth=5
rbd_user=cinder
rbd_flatten_volume_from_snapshot=False
rbd_cluster_name = mycluster
rbd_ceph_conf=/etc/ceph/mycluster.conf

did you configure the 'rbd_cluster_name' option for you non-default cluster?

[Xiaojun Liao (wwba)]

For two ceph backends, it seems retype works well in newton.