Unable to attach rados block device to instances

[James Page]

Attaching rbd devices to libvirt/qemu managed instances currently fails with:

2015-09-15 09:47:09.369+0000: 31908: error : qemuMonitorJSONCheckError:381 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'

I *think* that rbd support may have been accidentally dropped from qemu (qemu-system-x86 no longer depends on it).

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: qemu-system-x86 1:2.3+dfsg-5ubuntu5
ProcVersionSignature: Ubuntu 4.2.0-7.7-generic 4.2.0
Uname: Linux 4.2.0-7-generic x86_64
ApportVersion: 2.18.1-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Sep 15 10:55:16 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-11-25 (293 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20141124)
KvmCmdLine:
 COMMAND STAT EUID RUID PID PPID %CPU COMMAND
 kvm-irqfd-clean S< 0 0 375 2 0.0 [kvm-irqfd-clean]
MachineType: LENOVO 2324CTO
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-7-generic.efi.signed root=UUID=d03faa3b-5b36-44a5-b02e-deccef3ebcbb ro default_hugepagesz=1GB hugepagesz=1G hugepages=1 iommu=pt intel_iommu=on
SourcePackage: qemu
UpgradeStatus: Upgraded to wily on 2015-05-26 (111 days ago)
dmi.bios.date: 01/14/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: G2ET91WW (2.51 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2324CTO
dmi.board.vendor: LENOVO
dmi.board.version: Win8 Pro DPK TPG
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG2ET91WW(2.51):bd01/14/2013:svnLENOVO:pn2324CTO:pvrThinkPadX230:rvnLENOVO:rn2324CTO:rvrWin8ProDPKTPG:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2324CTO
dmi.product.version: ThinkPad X230
dmi.sys.vendor: LENOVO

[Ryan Harper (raharper)]

we're definitely missing librbd linking in wily which was present before. Looking at the build scripts, it it appears for linux-amd64 we emit the enable-rbd, so the next question is why we didn't actually compile and link against it.

[Ryan Harper (raharper)]

Looks like the rbd driver has been broken out into a separate package[1]

qemu-block-extra

If you install that does that fix your issue?

1. qemu (1:2.2+dfsg-6exp) unstable; urgency=medium

   Since Debian release 2.2+dfsg-6exp, a new package named qemu-block-extra
   has been created and some less frequently used block backends has been
   split out of main qemu-system binaries and from qemu-img binary to
   this new package. The backends which has been split are:
     curl
     iscsi
     rbd (ceph/rados)
     ssh
   If you use any of these, please install qemu-block-extra package in
   addition to qemu-system-* or qemu-utils package, because without it
   these block backends wont work anymore.

[Ryan Harper (raharper)]

Have qemu-system-common and qemu-utils depend on qemu-block-extra otherwise the packaging change changes expected behavior; namely before the packaging change users could install qemu-system-x86 (and others arches) and expect to use curl or rbd as a qemu block backend. After the packaging change users now are suggested to install qemu-block-extra but it's not immediately clear that qemu itself has missing features now due to the missing libraries. This breaks charms and other scripts which have expected qemu to have specific features present. Resolve this by having a dependency on the extra package which contains the block function.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "qemu_system_common_depend_on_qemu_block_extra.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[James Page (james-page)]

I've tried installing qemu-block-extras, but I'm still seeing the same error; I have tried restarting the instance (stop/start to ensure a new qemu process is created) and restarting libvirt but no-go right now.

[Ryan Harper (raharper)]

Do you have the libvirt log and the qemu command line used to run the guest?

[Ryan Harper (raharper)]

I think there is something else going on; clearly qemu-block-extra is needed if you're attempting to connect to an rbd device. For example we can attempt to connect to an rbd pool with path data/wily. I don't actually have an rbd pool up. If qemu doesn't have block_rbd from qemu-block-extra it will fail differently than if we do but don't have a valid pool

# without qemu-block-extra:

(kriek) ~ % qemu-system-x86_64 -m 1024 -drive format=raw,file=rbd:data/wily

(process:10825): GLib-WARNING **: /build/glib2.0-hcw3A1/glib2.0-2.45.7/./glib/gmem.c:482: custom memory allocation vtable not supported
qemu-system-x86_64: -drive format=raw,file=rbd:data/wily: Unknown protocol 'rbd'

# with qemu-block-extra installed:

(kriek) ~ % sudo apt-get install qemu-block-extra
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  dh-apparmor g++-4.9 grub-pc-bin libboost-iostreams1.55.0 libboost-system1.55.0 libboost-thread1.55.0 libicu52 libmirprotobuf0 libstdc++-4.9-dev linux-headers-3.19.0-20 linux-headers-3.19.0-20-generic
  linux-image-3.19.0-20-generic linux-image-extra-3.19.0-20-generic python-support
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  qemu-block-extra
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 0 B/18.7 kB of archives.
After this operation, 113 kB of additional disk space will be used.
Selecting previously unselected package qemu-block-extra:amd64.
(Reading database ... 172606 files and directories currently installed.)
Preparing to unpack .../qemu-block-extra_1%3a2.3+dfsg-5ubuntu5_amd64.deb ...
Unpacking qemu-block-extra:amd64 (1:2.3+dfsg-5ubuntu5) ...
Setting up qemu-block-extra:amd64 (1:2.3+dfsg-5ubuntu5) ...
(kriek) ~ % qemu-system-x86_64 -m 1024 -drive format=raw,file=rbd:data/wily

(process:10890): GLib-WARNING **: /build/glib2.0-hcw3A1/glib2.0-2.45.7/./glib/gmem.c:482: custom memory allocation vtable not supported
no monitors specified to connect to.
qemu-system-x86_64: -drive format=raw,file=rbd:data/wily: error connecting

[Ryan Harper (raharper)]

Starting the debugging.

First, we must have qemu-block-extra. After adding that we can check that qemu itself supports rbd with:

qemu-system-x86_64 -drive format=? 2>&1 | grep rbd

Next, we see libvirt barfing on the missing block device it tried to add and spits out the disk it was going to use:

2015-09-16 14:31:45.853+0000: 7856: error : qemuMonitorJSONCheckError:381 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2015-09-16 14:31:45.857+0000: 7856: error : qemuMonitorTextDriveDel:2599 : operation failed: deleting file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none drive failed: 2015-09-16T14:31:45.857818Z Device 'file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none' not found^M

Attempted to run qemu (now with rbd support) with the above -drive line:

# qemu-system-x86_64 -monitor stdio -S -nographic -drive file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O
1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none
qemu-system-x86_64: -drive file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx;none:mon_host=10.5.29.172:6789;10.5.29
.173:6789;10.5.29.174:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none: conf option 6789;10.5.29.173:6789;10.5.29.174:6789 has no value
root@juju-devel3-machine-14:/var/log/libvirt/qemu# qemu-system-x86_64 -monitor stdio -S -nographic -drive file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O
1S3bUpOHaAFBqq1SA==:auth_supported=cephx;none:mon_host=10.5.29.172:6789;10.5.29.173:6789;10.5.29.174:6789;,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none
WARNING: Image format was not specified for 'rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: cannot use stdio by multiple character devices
none:mon_host=10.5.29.172:6789: command not found
10.5.29.173:6789: command not f...

[Ryan Harper (raharper)]

it appears that qemu wants additional escaping of colons when specifying the port in mon_host options.

mon_host=10.5.29.172\:6789 fails to parse

mon_host=10.5.29.172\\:6789 parses correct.

Looking at why that changed (or if libvirt stopped adding additional escaping...).

[Ryan Harper (raharper)]

Looks like parsing is all fine. I recreated the error by enabling debug and seeing the qemu hmp command passed, then I launched my own instance and surfaced the HMP monitor, when I repeated the drive_add command it complained, unknown protocol rbd; this is the result of qemu being launched by libvirt and libvirt's apparmor policy not allowing qemu to mmap the needed block libraries.

[ 9606.350763] type=1400 audit(1442421386.001:26): apparmor="DENIED" operation="file_mmap" profile="libvirt-1b1ecffb-bddc-4825-8adb-3dce71fbcc57" name="/usr/lib/x86_64-linux-gnu/qemu/block-curl.so" pid=23428 comm="qemu-system-x86" requested_mask="m" denied_mask="m" fsuid=107 ouid=0
[ 9606.350902] type=1400 audit(1442421386.001:27): apparmor="DENIED" operation="file_mmap" profile="libvirt-1b1ecffb-bddc-4825-8adb-3dce71fbcc57" name="/usr/lib/x86_64-linux-gnu/qemu/block-rbd.so" pid=23428 comm="qemu-system-x86" requested_mask="m" denied_mask="m" fsuid=107 ouid=0

[Ryan Harper (raharper)]

OK, updating the libvirt-qemu apparmor abstraction resolves this issue:

# diff -u libvirt-qemu.orig libvirt-qemu
--- libvirt-qemu.orig 2015-09-16 18:14:46.013741000 +0000
+++ libvirt-qemu 2015-09-16 18:14:34.001741000 +0000
@@ -144,6 +144,10 @@

   # for rbd
   /etc/ceph/ceph.conf r,
+ /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so rm,
+
+ # for curl
+ /usr/lib/x86_64-linux-gnu/qemu/block-curl.so rm,

   # for access to hugepages
   owner "/run/hugepages/kvm/libvirt/qemu/**" rw,

now, we can attach an rbd volume:

root@juju-devel3-machine-14:~# virsh list
 Id Name State
----------------------------------------------------
 2 instance-00000001 running

root@juju-devel3-machine-14:~# virsh qemu-monitor-command --hmp instance-00000001 'info block'
drive-virtio-disk0: /var/lib/nova/instances/1b1ecffb-bddc-4825-8adb-3dce71fbcc57/disk (qcow2)
    Cache mode: writeback, direct
    Backing file: /var/lib/nova/instances/_base/0d0b68a8b7de02d81bdb0b644132349a8663ed1a (chain depth: 1)

drive-virtio-disk1: rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789 (raw)
    Cache mode: writeback, direct

[Ryan Harper (raharper)]

This bug requires a fix to libvirt's apparmor profile as well.

[Ryan Harper (raharper)]

debdiff includes an updated apparmor profile for libvirt-qemu for libraries provided by qemu-block-extra

[Martin Pitt (pitti)]

Can you please clarify the status here for sponsoring? qemu_system_common_depend_on_qemu_block_extra.debdiff seems a bit harsh -- the whole point of splitting out a separate binary is that you *don't* have to have it installed all the time, no? So maybe a Recommends: or Suggests:? But AFAIUI this isn't actually the root of the problem, but it's the apparmor violation in the second patch (libvirt_qemu_block_extra_apparmor_update.debdiff )? I'm going to upload the latter as that looks straightforward.

[James Page (james-page)]

Martin

Re the qemu changes; the issue will be for upgrades - for example, in vivid, the qemu binaries include rbd support - however an upgrade and reboot to wily with the situation as it is now (or even with Recommends/Suggests) would result in instances which cannot access block devices that they could before.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.3+dfsg-5ubuntu6

---------------
qemu (1:2.3+dfsg-5ubuntu6) wily; urgency=medium

  * Make qemu-system-common and qemu-utils depend on qemu-block-extra
    to fix errors with missing block backends. (LP: #1495895)
  * Cherry pick fixes for vmdk stream-optimized subformat (LP: #1006655)
  * Apply fix for memory corruption during live-migration in tcg mode
    (LP: #1493049)
  * Apply tracing patch to remove use of custom vtable in newer glibc
    (LP: #1491972)

 -- Ryan Harper <email address hidden> Tue, 15 Sep 2015 09:37:23 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.2.16-2ubuntu10

---------------
libvirt (1.2.16-2ubuntu10) wily; urgency=medium

  * Add qemu-block-extra libraries to libvirt apparmor profile (LP: #1495895)

 -- Ryan Harper <email address hidden> Wed, 16 Sep 2015 13:20:48 -0500