feature: enable ceph rbd support

[bugproxy]

This is a request to support ceph-rbd as a backing store for cinder-deployed volumes and nova ephemeral disks as part of the compute proxy charm for KVM for z.

When an openstack juju bundle gets deployed with a ceph cluster as a backing store for cinder(-ceph), the ceph-rbd client on KVM for z should be configured to communicate with this cinder-ceph, along with the necessary configurations in libvirt and nova to allow attachments of the ceph-rbd volumes/disks to the nova instances running on KVM for z.

---uname output---
Linux zfwcec082 4.4.0-40.60.el7_2.kvmibm1_1_3.2.s390x #1 SMP Tue Oct 18 14:41:51 EDT 2016 s390x s390x s390x GNU/Linux
Machine Type = 2847-743

---Steps to Reproduce---
 1. Deploy KVM on z via juju nova-compute-proxy charm (https://jujucharms.com/u/openstack-charmers-next/nova-compute-proxy/) along with ceph and cinder-ceph on other (Ubuntu) systems.
2. create a volume on ceph backend and attach to a nova instance running on KVM for z.

[James Page (james-page)]

Scope of work:

add requires: mon relation to nova-compute-proxy charm
template + rendering on change
push /etc/ceph/ceph.conf and /etc/ceph/client.nova-compute-proxy.keyring to the remote machines
ensure that ceph-common/librbd etc.. are enabled for libvirt/qemu on KVM for zSeries.

[Dmitrii Shcherbakov (dmitriis)]

There is support in qemu and the linux kernel:

[root@s4lp3 ~]# cat /etc/kvmibm-release
KVM for IBM z Systems release 1.1.3-beta4.3 (Z)

[root@s4lp3 ~]# ldd /usr/bin/qemu-system-s390x | grep rbd
 librbd.so.1 => /lib64/librbd.so.1 (0x000003ff8c580000)

[root@s4lp3 ~]# repoquery --requires --recursive --resolve qemu-kvm | grep rbd
librbd1-1:10.2.2-2.el7_2.kvmibm1_1_3.2.s390x

[root@s4lp3 ~]# qemu-kvm --drive format=? |& grep rbd
Supported formats: dmg ssh rbd tftp ftps ftp https http iscsi luks sheepdog nbd null-aio null-co host_cdrom host_device file blkreplay blkverify blkdebug parallels quorum vhdx qed qcow2 vvfat vpc bochs cloop vmdk vdi qcow raw

[root@s4lp3 ~]# stat /lib/modules/4.4.0-40.60.el7_2.kvmibm1_1_3.2.s390x/kernel/drivers/block/rbd.ko | grep file
  Size: 198462 Blocks: 392 IO Block: 4096 regular file

However, there is no support in libvirt:

[root@s4lp3 ~]# virsh --version=long
Virsh command line tool of libvirt 2.3.0
See web site at http://libvirt.org/

Compiled with support for:
 Hypervisors: QEMU/KVM LXC ESX Test
 Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
 Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM
 Miscellaneous: Daemon Nodedev SELinux Secrets Debug DTrace Readline Modular

On a system with RBD compiled it should be listed:
Compiled with support for:
 Hypervisors: QEMU/KVM LXC UML Xen LibXL OpenVZ VMWare VirtualBox ESX Test
 Networking: Remote Network Bridging Interface netcf Nwfilter VirtualPort
 Storage: Dir Disk Filesystem SCSI Multipath iSCSI LVM RBD Sheepdog ZFS
 Miscellaneous: Daemon Nodedev AppArmor Secrets Debug Readline Modular

Sources (just to make sure):
https://libvirt.org/git/?p=libvirt.git;a=blob;f=tools/virsh.c;h=7eb51ab7dc40c93202cb101fc2c3b7bfd6979734;hb=HEAD#l639

[cargonza (cargonza)]

we're currently investigating this issue based on jamespage's notes. I've allocated an engineer's time to investigate the effort and needed work. We will have more information by the end of the week.

[James Page (james-page)]

Lack of RBD support in libvirt will block use of ceph with nova-compute/libvirt/kvm.

Marking z Series bug as Incomplete pending enablement of this feature in the libvirt rpm's.

[Vance Morris (vmorris)]

Could you give more information on why this is necessary?

I'm using a z/KVM compute node now with a ceph backend and I'm able to create instances that both boot from volume and also boot from image.

I can give more information on my configuration and environment if needed.

[root@zs95kf ~]# cat /etc/os-release
NAME="KVM for IBM z Systems"
VERSION="1.1.3-rc.3 (Z)"
ID="kvmibm"
ID_LIKE="rhel fedora"
VERSION_ID="1.1.3-rc.3"
PRETTY_NAME="KVM for IBM z Systems 1.1.3-rc.3 (Z)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:ibm:kvmibm:1.1.3-rc.3"
BUILD_ID="20161216"

[root@zs95kf ~]# yum list | grep ceph
ceph.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-base.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-common.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-fuse.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-mds.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-mon.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-osd.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
ceph-selinux.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
libcephfs1.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher
python-cephfs.s390x 1:10.2.3-4.el7_2.kvmibm1_1_3.1 @frobisher

[root@zs95kf ~]# cat /etc/ceph/ceph.conf
[global]
fsid = 7e60958b-2adf-4313-ba47-95f2c725d886
mon_initial_members = xcephu27, xcephu29, xcephu31
mon_host = 10.20.92.121,10.20.92.122,10.20.92.123
public_network = 10.20.92.0/24
cluster_network = 10.20.93.0/24
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx
[mon]
mon_clock_drift_allowed = .500
[client]
    rbd cache = true
    rbd cache writethrough until flush = true
    admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
    log file = /var/log/qemu/qemu-guest-$pid.log
    rbd concurrent management ops = 20

[root@zs95kf ~]# cat /etc/nova/nova.conf
...
[libvirt]
disk_cachemodes = block=none
live_migration_tunnelled = True
live_migration_uri = qemu+tcp://stack@%s/system
virt_type = kvm
inject_password = false
inject_key = false
inject_partition = -2
use_usb_tablet = False
cpu_mode = none
images_type = rbd
images_rbd_pool = u14-vms
images_rbd_ceph_conf = /etc/ceph/ceph.conf
disk_cachemodes="network=writeback"
rbd_user = cinder
rbd_secret_uuid = <redacted>

[root@zs95kf ~]# virsh dumpxml instance-228
....
    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <auth username='cinder'>
        <secret type='ceph' uuid='053c94bd-d581-426c-9748-b1f11b63fdc6'/>
      </auth>
      <source protocol='rbd' name='u14-volumes/volume-84972b12-fea7-4330-a6e8-00c63d8b1a52'>
        <host name='10.20.92.121' port='6789'/>
        <host name='10.20.92.122' port='6789'/>
        <host name='10.20.92.123' port='6789'/>
      </source>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <serial>84972b12-fea7-4330-a6e8-00c63d8b1a52</serial>
      <alias name='virtio-disk0'/>
      <address type='ccw' cssid='0xfe...

[James Page (james-page)]

Hmm - I was under the impression that use of rbd devices in libvirt was dependent on the feature being enabled - maybe something is missing on our install.

[Heinz-Werner Seeck (heinz-werner-seeck)]

Will be fixed by final release of KVM for IBM z 1.1.2 Fixpack 4.

[Vance Morris (vmorris)]

I'm guessing that the RBD bits in libvirt enable one to define ceph pools there. Are there other functions that are enabled that might be needed by openstack?

I'm just curious, it looks like it'll be compiled in soon.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2017-09-07 06:31 EDT-------
IBM Bugzilla status-> closed; not requested anymore