[Impact]
Attempting to map an rbd image using the 4.4 LTS (Xenial) kernel client from a Jewel or Luminous Ceph cluster with optimal tunables fails due to feature set mismatch.
The Jewel release of Ceph introduced a new set of CRUSH tunables. These tunables were first introduced in the 4.5 Linux kernel and are thus not available in the 16.04 LTS 4.4 Linux Kernel. Attempting to map RBD images as block devices will fail due to not being able to understand these new tunables:
(from kern.log)
Oct 30 21:19:05 ceph-7 kernel: [ 815.674075] Key type ceph registered
Oct 30 21:19:05 ceph-7 kernel: [ 815.676862] libceph: loaded (mon/osd proto 15/24)
Oct 30 21:19:05 ceph-7 kernel: [ 815.678970] rbd: loaded (major 251)
Oct 30 21:19:05 ceph-7 kernel: [ 815.689556] libceph: mon0 10.5.0.19:6789 feature set mismatch, my 106b84a842a42 < server's 40106b84a842a42, missing 400000000000000
Oct 30 21:19:05 ceph-7 kernel: [ 815.692897] libceph: mon0 10.5.0.19:6789 missing required protocol features
Support for the new CRUSH tunables were added in upstream kernel 4.5 in http://www.spinics.net/lists/ceph-devel/msg28421.html.
[Test Case]
1. Deploy a Jewel or Luminous Ceph cluster.
2. Create rbd image suitable for the kernel client:
$ rbd create --pool rbd --image-feature layering --size 1G test
3. Map the rbd image to the local server:
$ rbd map --pool rbd test
[Regression Potential]
Minimal. Code is limited to kernel rbd driver and new code should primarily affect clients connecting to clusters with the new tunables options.
[Additional Info]
A workaround is to change the crush tunables configured for the Ceph cluster to a legacy version (hammer or lower) via:
$ ceph osd crush tunables hammer
However, changing the tunables to hammer fails to allow the cluster to take advantage of newer placement strategies which reduces the amount of data movement throughout the cluster.
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Oct 31 01:23 seq
crw-rw---- 1 root audio 116, 33 Oct 31 01:23 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 16.04
Ec2AMI: ami-00000001
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: OpenStack Foundation OpenStack Nova
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-98-generic root=UUID=d7006b2f-ace6-464d-8b21-17180b3ed360 ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 4.4.0-98.121-generic 4.4.90
RelatedPackageVersions:
linux-restricted-modules-4.4.0-98-generic N/A
linux-backports-modules-4.4.0-98-generic N/A
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial ec2-images
Uname: Linux 4.4.0-98-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
_MarkForUpload: True
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.1-1ubuntu1~cloud0
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-zesty
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.1-1ubuntu1~cloud0:bd04/01/2014:svnOpenStackFoundation:pnOpenStackNova:pvr15.0.6:cvnQEMU:ct1:cvrpc-i440fx-zesty:
dmi.product.name: OpenStack Nova
dmi.product.version: 15.0.6
dmi.sys.vendor: OpenStack Foundation
This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:
apport-collect 1728739
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.