Unable to attach rados block device to instances
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Ubuntu Cloud Archive |
High
|
Unassigned | ||
| | libvirt (Ubuntu) |
High
|
Unassigned | ||
| | qemu (Ubuntu) |
High
|
Unassigned | ||
Bug Description
Attaching rbd devices to libvirt/qemu managed instances currently fails with:
2015-09-15 09:47:09.369+0000: 31908: error : qemuMonitorJSON
I *think* that rbd support may have been accidentally dropped from qemu (qemu-system-x86 no longer depends on it).
ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: qemu-system-x86 1:2.3+dfsg-5ubuntu5
ProcVersionSign
Uname: Linux 4.2.0-7-generic x86_64
ApportVersion: 2.18.1-0ubuntu1
Architecture: amd64
CurrentDesktop: Unity
Date: Tue Sep 15 10:55:16 2015
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-11-25 (293 days ago)
InstallationMedia: Ubuntu 15.04 "Vivid Vervet" - Alpha amd64 (20141124)
KvmCmdLine:
COMMAND STAT EUID RUID PID PPID %CPU COMMAND
kvm-irqfd-clean S< 0 0 375 2 0.0 [kvm-irqfd-clean]
MachineType: LENOVO 2324CTO
ProcKernelCmdLine: BOOT_IMAGE=
SourcePackage: qemu
UpgradeStatus: Upgraded to wily on 2015-05-26 (111 days ago)
dmi.bios.date: 01/14/2013
dmi.bios.vendor: LENOVO
dmi.bios.version: G2ET91WW (2.51 )
dmi.board.
dmi.board.name: 2324CTO
dmi.board.vendor: LENOVO
dmi.board.version: Win8 Pro DPK TPG
dmi.chassis.
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.
dmi.modalias: dmi:bvnLENOVO:
dmi.product.name: 2324CTO
dmi.product.
dmi.sys.vendor: LENOVO
| James Page (james-page) wrote : | #1 |
| description: | updated |
| Ryan Harper (raharper) wrote : | #2 |
| Changed in qemu (Ubuntu): | |
| importance: | Undecided → High |
| status: | New → Confirmed |
| Ryan Harper (raharper) wrote : | #3 |
Looks like the rbd driver has been broken out into a separate package[1]
qemu-block-extra
If you install that does that fix your issue?
1. qemu (1:2.2+dfsg-6exp) unstable; urgency=medium
Since Debian release 2.2+dfsg-6exp, a new package named qemu-block-extra
has been created and some less frequently used block backends has been
split out of main qemu-system binaries and from qemu-img binary to
this new package. The backends which has been split are:
curl
iscsi
rbd (ceph/rados)
ssh
If you use any of these, please install qemu-block-extra package in
addition to qemu-system-* or qemu-utils package, because without it
these block backends wont work anymore.
| Changed in qemu (Ubuntu): | |
| status: | Confirmed → Incomplete |
| Ryan Harper (raharper) wrote : | #4 |
Have qemu-system-common and qemu-utils depend on qemu-block-extra otherwise the packaging change changes expected behavior; namely before the packaging change users could install qemu-system-x86 (and others arches) and expect to use curl or rbd as a qemu block backend. After the packaging change users now are suggested to install qemu-block-extra but it's not immediately clear that qemu itself has missing features now due to the missing libraries. This breaks charms and other scripts which have expected qemu to have specific features present. Resolve this by having a dependency on the extra package which contains the block function.
The attachment "qemu_system_
[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]
| tags: | added: patch |
| James Page (james-page) wrote : | #6 |
I've tried installing qemu-block-extras, but I'm still seeing the same error; I have tried restarting the instance (stop/start to ensure a new qemu process is created) and restarting libvirt but no-go right now.
| Changed in qemu (Ubuntu): | |
| status: | Incomplete → New |
| Ryan Harper (raharper) wrote : | #7 |
Do you have the libvirt log and the qemu command line used to run the guest?
| Changed in qemu (Ubuntu): | |
| status: | New → Incomplete |
| Ryan Harper (raharper) wrote : | #8 |
I think there is something else going on; clearly qemu-block-extra is needed if you're attempting to connect to an rbd device. For example we can attempt to connect to an rbd pool with path data/wily. I don't actually have an rbd pool up. If qemu doesn't have block_rbd from qemu-block-extra it will fail differently than if we do but don't have a valid pool
# without qemu-block-extra:
(kriek) ~ % qemu-system-x86_64 -m 1024 -drive format=
(process:10825): GLib-WARNING **: /build/
qemu-system-x86_64: -drive format=
# with qemu-block-extra installed:
(kriek) ~ % sudo apt-get install qemu-block-extra
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
dh-apparmor g++-4.9 grub-pc-bin libboost-
linux-
Use 'apt-get autoremove' to remove them.
The following NEW packages will be installed:
qemu-block-extra
0 upgraded, 1 newly installed, 0 to remove and 15 not upgraded.
Need to get 0 B/18.7 kB of archives.
After this operation, 113 kB of additional disk space will be used.
Selecting previously unselected package qemu-block-
(Reading database ... 172606 files and directories currently installed.)
Preparing to unpack .../qemu-
Unpacking qemu-block-
Setting up qemu-block-
(kriek) ~ % qemu-system-x86_64 -m 1024 -drive format=
(process:10890): GLib-WARNING **: /build/
no monitors specified to connect to.
qemu-system-x86_64: -drive format=
| Ryan Harper (raharper) wrote : | #9 |
Starting the debugging.
First, we must have qemu-block-extra. After adding that we can check that qemu itself supports rbd with:
qemu-system-x86_64 -drive format=? 2>&1 | grep rbd
Next, we see libvirt barfing on the missing block device it tried to add and spits out the disk it was going to use:
2015-09-16 14:31:45.853+0000: 7856: error : qemuMonitorJSON
2015-09-16 14:31:45.857+0000: 7856: error : qemuMonitorText
Attempted to run qemu (now with rbd support) with the above -drive line:
# qemu-system-x86_64 -monitor stdio -S -nographic -drive file=rbd:
1S3bUpOHaAFBqq1
qemu-system-x86_64: -drive file=rbd:
.173:6789;
root@juju-
1S3bUpOHaAFBqq1
WARNING: Image format was not specified for 'rbd:cinder-
Specify the 'raw' format explicitly to remove the restrictions.
QEMU 2.3.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: cannot use stdio by multiple character devices
none:mon_
10.5.29.173:6789: command not f...
| Ryan Harper (raharper) wrote : | #10 |
it appears that qemu wants additional escaping of colons when specifying the port in mon_host options.
mon_host=
mon_host=
Looking at why that changed (or if libvirt stopped adding additional escaping...).
| Ryan Harper (raharper) wrote : | #11 |
Looks like parsing is all fine. I recreated the error by enabling debug and seeing the qemu hmp command passed, then I launched my own instance and surfaced the HMP monitor, when I repeated the drive_add command it complained, unknown protocol rbd; this is the result of qemu being launched by libvirt and libvirt's apparmor policy not allowing qemu to mmap the needed block libraries.
[ 9606.350763] type=1400 audit(144242138
[ 9606.350902] type=1400 audit(144242138
| Ryan Harper (raharper) wrote : | #12 |
OK, updating the libvirt-qemu apparmor abstraction resolves this issue:
# diff -u libvirt-qemu.orig libvirt-qemu
--- libvirt-qemu.orig 2015-09-16 18:14:46.013741000 +0000
+++ libvirt-qemu 2015-09-16 18:14:34.001741000 +0000
@@ -144,6 +144,10 @@
# for rbd
/etc/
+ /usr/lib/
+
+ # for curl
+ /usr/lib/
# for access to hugepages
owner "/run/hugepages
now, we can attach an rbd volume:
root@juju-
Id Name State
-------
2 instance-00000001 running
root@juju-
drive-virtio-disk0: /var/lib/
Cache mode: writeback, direct
Backing file: /var/lib/
drive-virtio-disk1: rbd:cinder-
Cache mode: writeback, direct
| Ryan Harper (raharper) wrote : | #13 |
This bug requires a fix to libvirt's apparmor profile as well.
| Changed in libvirt (Ubuntu): | |
| importance: | Undecided → High |
| status: | New → Confirmed |
| Changed in qemu (Ubuntu): | |
| status: | Incomplete → Confirmed |
| Ryan Harper (raharper) wrote : | #14 |
debdiff includes an updated apparmor profile for libvirt-qemu for libraries provided by qemu-block-extra
| Martin Pitt (pitti) wrote : | #15 |
Can you please clarify the status here for sponsoring? qemu_system_
| Changed in libvirt (Ubuntu): | |
| status: | Confirmed → Fix Committed |
| James Page (james-page) wrote : | #16 |
Martin
Re the qemu changes; the issue will be for upgrades - for example, in vivid, the qemu binaries include rbd support - however an upgrade and reboot to wily with the situation as it is now (or even with Recommends/
| Launchpad Janitor (janitor) wrote : | #17 |
This bug was fixed in the package qemu - 1:2.3+dfsg-5ubuntu6
---------------
qemu (1:2.3+
* Make qemu-system-common and qemu-utils depend on qemu-block-extra
to fix errors with missing block backends. (LP: #1495895)
* Cherry pick fixes for vmdk stream-optimized subformat (LP: #1006655)
* Apply fix for memory corruption during live-migration in tcg mode
(LP: #1493049)
* Apply tracing patch to remove use of custom vtable in newer glibc
(LP: #1491972)
-- Ryan Harper <email address hidden> Tue, 15 Sep 2015 09:37:23 -0500
| Changed in qemu (Ubuntu): | |
| status: | Confirmed → Fix Released |
| Launchpad Janitor (janitor) wrote : | #18 |
This bug was fixed in the package libvirt - 1.2.16-2ubuntu10
---------------
libvirt (1.2.16-2ubuntu10) wily; urgency=medium
* Add qemu-block-extra libraries to libvirt apparmor profile (LP: #1495895)
-- Ryan Harper <email address hidden> Wed, 16 Sep 2015 13:20:48 -0500
| Changed in libvirt (Ubuntu): | |
| status: | Fix Committed → Fix Released |
| Changed in cloud-archive: | |
| status: | New → Fix Released |
| importance: | Undecided → High |
we're definitely missing librbd linking in wily which was present before. Looking at the build scripts, it it appears for linux-amd64 we emit the enable-rbd, so the next question is why we didn't actually compile and link against it.