attaching rbd fails because apparmor forbids access to ceph.conf

Bug #1026404 reported by Paul Collins
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Medium
Serge Hallyn
Precise
Medium
Unassigned

Bug Description

==============================
SRU Justification:
1. Impact: ceph volumes cannot be used in libvirt-qemu
2. Development fix: update apparmor policy to allow qemu under libvirt access to ceph.conf
3. Stable fix: same as development fix
4. test case: see comment #4
5. Regression potential: if ceph.conf was deemed to have sensitive information, qemu under libvirt could now read that. No functionality regression should result from simply allowing read access to a configuration file.
==============================

I've been doing a little work with openstack, using ceph as a backend for nova-volume.

When I attempt to attach an RBD volume to a running instance, it fails (with some delightfully unhelpful errors on the nova side). The following is logged in the instance's libvirt log file

unable to find any monitors in conf. please specify monitors via -m monaddr or -c ceph.conf

and in dmesg we find

type=1400 audit(1342656681.070:52): apparmor="DENIED" operation="open" parent=1 profile="libvirt-732fc0e8-5a8b-46d0-9689-f0c3e8f619f2" name="/etc/ceph/ceph.conf" pid=27601 comm="kvm" requested_mask="r" denied_mask="r" fsuid=110 ouid=0

Adding the following to /etc/apparmor.d/abstractions/libvirt-qemu seems to be sufficient, at least for my no-cephx (i.e., no authentication and therefore no key material) testing environment. For a cephx environment things are more complex, because a keyring file will also need to be read by the qemu process.

=== modified file 'apparmor.d/abstractions/libvirt-qemu'
--- apparmor.d/abstractions/libvirt-qemu 2012-07-18 23:37:13 +0000
+++ apparmor.d/abstractions/libvirt-qemu 2012-07-19 00:10:18 +0000
@@ -123,3 +123,6 @@
   /etc/pki/CA/* r,
   /etc/pki/libvirt/ r,
   /etc/pki/libvirt/** r,
+
+ # for rbd
+ /etc/ceph/ceph.conf r,

Paul Collins (pjdc)
summary: - attaching rbd fails because apparmor forbids access to ceph.conf, etc.
+ attaching rbd fails because apparmor forbids access to ceph.conf
Revision history for this message
James Page (james-page) wrote :

Hi Paul

Makes sense - I can see why this blocks.

Marking Triaged - this does not really need Confirmation.

Changed in libvirt (Ubuntu):
status: New → Triaged
importance: Undecided → High
milestone: none → quantal-alpha-3
Changed in libvirt (Ubuntu Precise):
status: New → Triaged
importance: Undecided → Medium
Changed in libvirt (Ubuntu):
importance: High → Medium
Changed in libvirt (Ubuntu Precise):
milestone: none → precise-updates
Changed in libvirt (Ubuntu):
assignee: nobody → Serge Hallyn (serge-hallyn)
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.9.13-0ubuntu3

---------------
libvirt (0.9.13-0ubuntu3) quantal; urgency=low

  * debian/apparmor/libvirt-qemu: add ceph.conf (LP: #1026404)
  * debian/patches: re-add 9002-better_default_uri_virsh.patch (LP: #1026515)
 -- Serge Hallyn <email address hidden> Thu, 19 Jul 2012 07:58:39 -0500

Changed in libvirt (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

A proposed test case (for SRU). Paul, does this make sense?

1. apt-get install ceph ceph-common ceph-fs-common ceph-mds qemu-kvm libvirt-bin
2. cat > /etc/ceph/ceph.conf << EOF
[osd]
 osd journal size = 1000
 filestore xattr use omap = true

[mon.a]
 host = localhost
 mon addr = 127.0.0.1:6789

[osd.0]
 host = localhost

[osd.1]
 host = localhost

[mds.a]
 host = localhost

EOF
3. qemu-img create cdboot.img 1G
4. wget http://people.canonical.com/~serge/cdboot-ceph.img
5. virsh define cdboot
6. virsh start cdboot

If ceph.conf could not be read, this will immediately exit with errors including:

unable to find any monitors in conf. please specify monitors via -m monaddr or -c ceph.conf

If ceph.conf could be read, then 'virsh start' will hang for awhile and (though it won't succeed without ceph configured) will NOT show the error

unable to find any monitors in conf. please specify monitors via -m monaddr or -c ceph.conf

(Paul, if you can show in a few steps how to configure a working ceph install that'd be even better)

Revision history for this message
Paul Collins (pjdc) wrote :

Below is a mostly paste'n'go Ceph recipe — apt clears the input buffer, so it can't be pasted in its entirety.

I've tested it on a blank VM and it seems to work. It could be a little tidier, e.g. currently logs will end up in /var/log/ceph.

I'm unsure about the test case as you have it above. The failure with Openstack occurs when we try to attach a volume to a running instance, not when the instance is started, so the test case probably needs to simulate that more closely. With a working Ceph handy the test case can now attach a volume and then perhaps poll "virsh dumpxml" for a while to ensure it shows up.

I seem to recall "virsh attach-disk" being a sufficiently incomplete mapping to "virsh attach-device" that it can't be used for RBDs. The test case will probably have to construct a suitable XML fragment and feed it to "virsh attach-device". Something like this should work:

    <disk type='network' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source protocol='rbd' name='rbd/blorp'/>
      <target dev='vdz' bus='virtio'/>
    </disk>

# begin recipe

sudo su -

apt-get update && apt-get -y install ceph

# paste from here to end

export CEPH_ROOT_TEMP=$(mktemp -d /tmp/ceph-root.XXXXXXXX)

export MKCEPHFS_TEMP=$(mktemp -d /tmp/mkcephfs.XXXXXXXX)

mkdir /etc/ceph

cat > /etc/ceph/ceph.conf << EOF
[osd]
 osd journal size = 1024
 filestore xattr use omap = true
 osd data = ${CEPH_ROOT_TEMP}/\$name
 osd journal = ${CEPH_ROOT_TEMP}/journal.\$name

[mon.${HOSTNAME}]
 host = ${HOSTNAME}
 mon addr = 127.0.0.1:6789
 mon data = ${CEPH_ROOT_TEMP}/\$name
[osd.0]
 host = ${HOSTNAME}
[osd.1]
 host = ${HOSTNAME}
EOF

mkdir ${CEPH_ROOT_TEMP}/osd.{0,1}

mkcephfs -c /etc/ceph/ceph.conf -d $MKCEPHFS_TEMP --prepare-monmap

mkcephfs -d $MKCEPHFS_TEMP --init-local-daemons osd

mkcephfs -d $MKCEPHFS_TEMP --prepare-mon

mkcephfs -d $MKCEPHFS_TEMP --init-local-daemons mon

/etc/init.d/ceph start

ceph -s

rbd create --size 1024 blorp

rbd info blorp

# end

description: updated
Changed in libvirt (Ubuntu Precise):
status: Triaged → In Progress
Revision history for this message
Scott Kitterman (kitterman) wrote : Please test proposed package

Hello Paul, or anyone else affected,

Accepted into precise-proposed. The package will build now and be available in a few hours in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libvirt (Ubuntu Precise):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Paul Collins (pjdc) wrote :

Looks good.

tags: added: verification-done
removed: verification-needed
Revision history for this message
Colin Watson (cjwatson) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.9.8-2ubuntu17.3

---------------
libvirt (0.9.8-2ubuntu17.3) precise-proposed; urgency=low

  * debian/apparmor/libvirt-qemu: add ceph.conf (LP: #1026404)
 -- Serge Hallyn <email address hidden> Wed, 25 Jul 2012 07:02:47 -0500

Changed in libvirt (Ubuntu Precise):
status: Fix Committed → Fix Released
James Page (james-page)
Changed in libvirt (Ubuntu Precise):
milestone: precise-updates → ubuntu-12.04.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers