cinder backend by ceph HA setup

[gabriel staicu]

Setup:

openstack havana installed on ubuntu precise on 10 servers.
Cinder has as a backend a ceph cluster. Here is a an example of the cinder.conf:

[DEFAULT]
debug=true
verbose=true
rabbit_ha_queues=True
rabbit_hosts=10.6.5.8:5672,10.6.5.10:5672
rabbit_userid=guest

sql_connection=mysql://cinder:cinder_pass@10.6.5.11/cinder
sql_idle_timeout=30
sql_max_retries=-1
sql_retry_interval=5

osapi_volume_listen = 10.6.5.10
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
volume_name_template = volume-%s
auth_strategy = keystone
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=volumes
glance_api_version=2
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot=False

service_down_time=5
report_interval=1
glance_host=10.6.5.11

Here are the agents running:
root@opstck10:~# cinder service-list
+------------------+----------+------+---------+-------+----------------------------+
| Binary | Host | Zone | Status | State | Updated_at |
+------------------+----------+------+---------+-------+----------------------------+
| cinder-scheduler | opstck08 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-scheduler | opstck09 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-scheduler | opstck10 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-volume | opstck01 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-volume | opstck03 | nova | enabled | up | 2014-02-14T16:23:42.000000 |
| cinder-volume | opstck04 | nova | enabled | up | 2014-02-14T16:23:42.000000 |
| cinder-volume | opstck05 | nova | enabled | up | 2014-02-14T16:23:42.000000 |
| cinder-volume | opstck06 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-volume | opstck08 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-volume | opstck09 | nova | enabled | up | 2014-02-14T16:23:41.000000 |
| cinder-volume | opstck10 | nova | enabled | up | 2014-02-14T16:23:42.000000 |
+------------------+----------+------+---------+-------+----------------------------+

The problem that I am seeing is that I cannot configure cinder to be truly HA.
For example I have this volume:

cinder show 2f5be8c9-941e-49cb-8eb8-f6def3ca8af9
+--------------------------------+--------------------------------------+
| Property | Value |
+--------------------------------+--------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | false |
| created_at | 2014-02-11T15:33:58.000000 |
| display_description | |
| display_name | cirros-0.3.1 |
| id | 2f5be8c9-941e-49cb-8eb8-f6def3ca8af9 |
| metadata | {u'readonly': u'False'} |
| os-vol-host-attr:host | opstck01 |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 8c25ff44225f4e78ab3f526d99c1b7e1 |
| size | 1 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| volume_type | None |
+--------------------------------+--------------------------------------+

As can be seen the parent host of this volume is opstck01. But in reality this volume is not on the hdd of the opstck01. It is on a ceph cluster:
root@opstck10:~# rbd -p volumes ls |grep 2f5be8c9-941e-49cb-8eb8-f6def3ca8af9
volume-2f5be8c9-941e-49cb-8eb8-f6def3ca8af9

If cinder-volume on opstck01 is down it cannot be spawned a new instance from that volume even it is accessible.

[gabriel staicu (gabriel-staicu)]

The solution for HA cinder is the parameter <host> in /etc/cinder/cinder.conf on the machines that are running cinder-volume service.

If you set this parameter to the same string on all the hosts that are running cinder-volume service than all the hosts will be parent to any volume in the ceph cluster, so any host can create a volume from an already existing volume.

So I am suggesting to close the bug as designed. I will log a bug against the documentation.

Thanks,
Gabriel

[Simon Effenberg (launchpaq)]

It is still valid if you want to use the multi_backend feature (enabled_backends=a,b,c) and one backend is a shared backend (Ceph or other) which needs a "host=GENERICNAME" but another backend is only local so it needs a unique identifier.

So like invalidating other parameters in the [DEFAULT] section when enabling multiple backends also the "host" parameter should be only used within the backend section.

Something like:

[DEFAULT]
...
enabled_backends=rbd1,lvm1

[rbd1]
host = rbd-cluster1
..

[lvm1]
host = HOSTNAME_OF_THE_HOST

[Sachi King (nakato)]

As far as I can tell, what you're proposing Simon is already possible.
Can you verify this or provide more details?

[John Haller (john-haller)]

See https://review.openstack.org/#/c/95022/ which introduced per-backend hostname in May 2014

[Jon Bernard (jbernard)]

John Haller's patch link does address this issue, perhaps this bug can be closed?

[Simon Effenberg (launchpaq)]

I didn't try it out by now but the patch seems to address it.. so closing the bug is fine.