Unable to attach rados block device to instances

Bug #1495895 reported by James Page
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
High
Unassigned
libvirt (Ubuntu)
Fix Released
High
Unassigned
qemu (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Attaching rbd devices to libvirt/qemu managed instances currently fails with:

2015-09-15 09:47:09.369+0000: 31908: error : qemuMonitorJSONCheckError:381 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'

I *think* that rbd support may have been accidentally dropped from qemu (qemu-system-x86 no longer depends on it).

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: qemu-system-x86 1:2.3+dfsg-5ubuntu5
ProcVersionSignature: Ubuntu 4.2.0-7.7-generic 4.2.0
Uname: Linux 4.2.0-7-generic x86_64
ApportVersion: 2.18.1-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Sep 15 10:55:16 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-11-25 (293 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20141124)
KvmCmdLine:
 COMMAND STAT EUID RUID PID PPID %CPU COMMAND
 kvm-irqfd-clean S< 0 0 375 2 0.0 [kvm-irqfd-clean]
MachineType: LENOVO 2324CTO
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.2.0-7-generic.efi.signed root=UUID=d03faa3b-5b36-44a5-b02e-deccef3ebcbb ro default_hugepagesz=1GB hugepagesz=1G hugepages=1 iommu=pt intel_iommu=on
SourcePackage: qemu
UpgradeStatus: Upgraded to wily on 2015-05-26 (111 days ago)
dmi.bios.date: 01/14/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: G2ET91WW (2.51 )
dmi.board.asset.tag: Not Available
dmi.board.name: 2324CTO
dmi.board.vendor: LENOVO
dmi.board.version: Win8 Pro DPK TPG
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvrG2ET91WW(2.51):bd01/14/2013:svnLENOVO:pn2324CTO:pvrThinkPadX230:rvnLENOVO:rn2324CTO:rvrWin8ProDPKTPG:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 2324CTO
dmi.product.version: ThinkPad X230
dmi.sys.vendor: LENOVO

Revision history for this message
James Page (james-page) wrote :
James Page (james-page)
description: updated
Revision history for this message
Ryan Harper (raharper) wrote :

we're definitely missing librbd linking in wily which was present before. Looking at the build scripts, it it appears for linux-amd64 we emit the enable-rbd, so the next question is why we didn't actually compile and link against it.

Changed in qemu (Ubuntu):
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Ryan Harper (raharper) wrote :

Looks like the rbd driver has been broken out into a separate package[1]

qemu-block-extra

If you install that does that fix your issue?

1. qemu (1:2.2+dfsg-6exp) unstable; urgency=medium

   Since Debian release 2.2+dfsg-6exp, a new package named qemu-block-extra
   has been created and some less frequently used block backends has been
   split out of main qemu-system binaries and from qemu-img binary to
   this new package. The backends which has been split are:
     curl
     iscsi
     rbd (ceph/rados)
     ssh
   If you use any of these, please install qemu-block-extra package in
   addition to qemu-system-* or qemu-utils package, because without it
   these block backends wont work anymore.

Changed in qemu (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Ryan Harper (raharper) wrote :

Have qemu-system-common and qemu-utils depend on qemu-block-extra otherwise the packaging change changes expected behavior; namely before the packaging change users could install qemu-system-x86 (and others arches) and expect to use curl or rbd as a qemu block backend. After the packaging change users now are suggested to install qemu-block-extra but it's not immediately clear that qemu itself has missing features now due to the missing libraries. This breaks charms and other scripts which have expected qemu to have specific features present. Resolve this by having a dependency on the extra package which contains the block function.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "qemu_system_common_depend_on_qemu_block_extra.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
James Page (james-page) wrote :

I've tried installing qemu-block-extras, but I'm still seeing the same error; I have tried restarting the instance (stop/start to ensure a new qemu process is created) and restarting libvirt but no-go right now.

Changed in qemu (Ubuntu):
status: Incomplete → New
Revision history for this message
Ryan Harper (raharper) wrote :

Do you have the libvirt log and the qemu command line used to run the guest?

Changed in qemu (Ubuntu):
status: New → Incomplete
Revision history for this message
Ryan Harper (raharper) wrote :

I think there is something else going on; clearly qemu-block-extra is needed if you're attempting to connect to an rbd device. For example we can attempt to connect to an rbd pool with path data/wily. I don't actually have an rbd pool up. If qemu doesn't have block_rbd from qemu-block-extra it will fail differently than if we do but don't have a valid pool

# without qemu-block-extra:

(kriek) ~ % qemu-system-x86_64 -m 1024 -drive format=raw,file=rbd:data/wily

(process:10825): GLib-WARNING **: /build/glib2.0-hcw3A1/glib2.0-2.45.7/./glib/gmem.c:482: custom memory allocation vtable not supported
qemu-system-x86_64: -drive format=raw,file=rbd:data/wily: Unknown protocol 'rbd'

# with qemu-block-extra installed:

(kriek) ~ % sudo apt-get install qemu-block-extra
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  dh-apparmor g++-4.9 grub-pc-bin libboost-iostreams1.55.0 libboost-system1.55.0 libboost-thread1.55.0 libicu52 libmirprotobuf0 libstdc++-4.9-dev linux-headers-3.19.0-20 linux-headers-3.19.0-20-generic
  linux-image-3.19.0-20-generic linux-image-extra-3.19.0-20-generic python-support
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
  qemu-block-extra
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 0 B/18.7 kB of archives.
After this operation, 113 kB of additional disk space will be used.
Selecting previously unselected package qemu-block-extra:amd64.
(Reading database ... 172606 files and directories currently installed.)
Preparing to unpack .../qemu-block-extra_1%3a2.3+dfsg-5ubuntu5_amd64.deb ...
Unpacking qemu-block-extra:amd64 (1:2.3+dfsg-5ubuntu5) ...
Setting up qemu-block-extra:amd64 (1:2.3+dfsg-5ubuntu5) ...
(kriek) ~ % qemu-system-x86_64 -m 1024 -drive format=raw,file=rbd:data/wily

(process:10890): GLib-WARNING **: /build/glib2.0-hcw3A1/glib2.0-2.45.7/./glib/gmem.c:482: custom memory allocation vtable not supported
no monitors specified to connect to.
qemu-system-x86_64: -drive format=raw,file=rbd:data/wily: error connecting

Revision history for this message
Ryan Harper (raharper) wrote :
Download full text (3.4 KiB)

Starting the debugging.

First, we must have qemu-block-extra. After adding that we can check that qemu itself supports rbd with:

qemu-system-x86_64 -drive format=? 2>&1 | grep rbd

Next, we see libvirt barfing on the missing block device it tried to add and spits out the disk it was going to use:

2015-09-16 14:31:45.853+0000: 7856: error : qemuMonitorJSONCheckError:381 : internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
2015-09-16 14:31:45.857+0000: 7856: error : qemuMonitorTextDriveDel:2599 : operation failed: deleting file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none drive failed: 2015-09-16T14:31:45.857818Z Device 'file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none' not found^M

Attempted to run qemu (now with rbd support) with the above -drive line:

# qemu-system-x86_64 -monitor stdio -S -nographic -drive file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O
1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none
qemu-system-x86_64: -drive file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx;none:mon_host=10.5.29.172:6789;10.5.29
.173:6789;10.5.29.174:6789,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none: conf option 6789;10.5.29.173:6789;10.5.29.174:6789 has no value
root@juju-devel3-machine-14:/var/log/libvirt/qemu# qemu-system-x86_64 -monitor stdio -S -nographic -drive file=rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O
1S3bUpOHaAFBqq1SA==:auth_supported=cephx;none:mon_host=10.5.29.172:6789;10.5.29.173:6789;10.5.29.174:6789;,if=none,id=drive-virtio-disk1,format=raw,serial=da89c042-0628-4630-9d95-4624452d346c,cache=none
WARNING: Image format was not specified for 'rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx' and probing guessed raw.
         Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
         Specify the 'raw' format explicitly to remove the restrictions.
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: cannot use stdio by multiple character devices
none:mon_host=10.5.29.172:6789: command not found
10.5.29.173:6789: command not f...

Read more...

Revision history for this message
Ryan Harper (raharper) wrote :

it appears that qemu wants additional escaping of colons when specifying the port in mon_host options.

mon_host=10.5.29.172\:6789 fails to parse

mon_host=10.5.29.172\\:6789 parses correct.

Looking at why that changed (or if libvirt stopped adding additional escaping...).

Revision history for this message
Ryan Harper (raharper) wrote :

Looks like parsing is all fine. I recreated the error by enabling debug and seeing the qemu hmp command passed, then I launched my own instance and surfaced the HMP monitor, when I repeated the drive_add command it complained, unknown protocol rbd; this is the result of qemu being launched by libvirt and libvirt's apparmor policy not allowing qemu to mmap the needed block libraries.

[ 9606.350763] type=1400 audit(1442421386.001:26): apparmor="DENIED" operation="file_mmap" profile="libvirt-1b1ecffb-bddc-4825-8adb-3dce71fbcc57" name="/usr/lib/x86_64-linux-gnu/qemu/block-curl.so" pid=23428 comm="qemu-system-x86" requested_mask="m" denied_mask="m" fsuid=107 ouid=0
[ 9606.350902] type=1400 audit(1442421386.001:27): apparmor="DENIED" operation="file_mmap" profile="libvirt-1b1ecffb-bddc-4825-8adb-3dce71fbcc57" name="/usr/lib/x86_64-linux-gnu/qemu/block-rbd.so" pid=23428 comm="qemu-system-x86" requested_mask="m" denied_mask="m" fsuid=107 ouid=0

Revision history for this message
Ryan Harper (raharper) wrote :

OK, updating the libvirt-qemu apparmor abstraction resolves this issue:

# diff -u libvirt-qemu.orig libvirt-qemu
--- libvirt-qemu.orig 2015-09-16 18:14:46.013741000 +0000
+++ libvirt-qemu 2015-09-16 18:14:34.001741000 +0000
@@ -144,6 +144,10 @@

   # for rbd
   /etc/ceph/ceph.conf r,
+ /usr/lib/x86_64-linux-gnu/qemu/block-rbd.so rm,
+
+ # for curl
+ /usr/lib/x86_64-linux-gnu/qemu/block-curl.so rm,

   # for access to hugepages
   owner "/run/hugepages/kvm/libvirt/qemu/**" rw,

now, we can attach an rbd volume:

root@juju-devel3-machine-14:~# virsh list
 Id Name State
----------------------------------------------------
 2 instance-00000001 running

root@juju-devel3-machine-14:~# virsh qemu-monitor-command --hmp instance-00000001 'info block'
drive-virtio-disk0: /var/lib/nova/instances/1b1ecffb-bddc-4825-8adb-3dce71fbcc57/disk (qcow2)
    Cache mode: writeback, direct
    Backing file: /var/lib/nova/instances/_base/0d0b68a8b7de02d81bdb0b644132349a8663ed1a (chain depth: 1)

drive-virtio-disk1: rbd:cinder-ceph/volume-da89c042-0628-4630-9d95-4624452d346c:id=nova-compute:key=AQDUdflVYhZ4FhAAmT/7O1S3bUpOHaAFBqq1SA==:auth_supported=cephx\;none:mon_host=10.5.29.172\:6789\;10.5.29.173\:6789\;10.5.29.174\:6789 (raw)
    Cache mode: writeback, direct

Revision history for this message
Ryan Harper (raharper) wrote :

This bug requires a fix to libvirt's apparmor profile as well.

Changed in libvirt (Ubuntu):
importance: Undecided → High
status: New → Confirmed
Changed in qemu (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Ryan Harper (raharper) wrote :

debdiff includes an updated apparmor profile for libvirt-qemu for libraries provided by qemu-block-extra

Revision history for this message
Martin Pitt (pitti) wrote :

Can you please clarify the status here for sponsoring? qemu_system_common_depend_on_qemu_block_extra.debdiff seems a bit harsh -- the whole point of splitting out a separate binary is that you *don't* have to have it installed all the time, no? So maybe a Recommends: or Suggests:? But AFAIUI this isn't actually the root of the problem, but it's the apparmor violation in the second patch (libvirt_qemu_block_extra_apparmor_update.debdiff )? I'm going to upload the latter as that looks straightforward.

Changed in libvirt (Ubuntu):
status: Confirmed → Fix Committed
Revision history for this message
James Page (james-page) wrote :

Martin

Re the qemu changes; the issue will be for upgrades - for example, in vivid, the qemu binaries include rbd support - however an upgrade and reboot to wily with the situation as it is now (or even with Recommends/Suggests) would result in instances which cannot access block devices that they could before.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.3+dfsg-5ubuntu6

---------------
qemu (1:2.3+dfsg-5ubuntu6) wily; urgency=medium

  * Make qemu-system-common and qemu-utils depend on qemu-block-extra
    to fix errors with missing block backends. (LP: #1495895)
  * Cherry pick fixes for vmdk stream-optimized subformat (LP: #1006655)
  * Apply fix for memory corruption during live-migration in tcg mode
    (LP: #1493049)
  * Apply tracing patch to remove use of custom vtable in newer glibc
    (LP: #1491972)

 -- Ryan Harper <email address hidden> Tue, 15 Sep 2015 09:37:23 -0500

Changed in qemu (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 1.2.16-2ubuntu10

---------------
libvirt (1.2.16-2ubuntu10) wily; urgency=medium

  * Add qemu-block-extra libraries to libvirt apparmor profile (LP: #1495895)

 -- Ryan Harper <email address hidden> Wed, 16 Sep 2015 13:20:48 -0500

Changed in libvirt (Ubuntu):
status: Fix Committed → Fix Released
Changed in cloud-archive:
status: New → Fix Released
importance: Undecided → High
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers