Apparmor denies qemu access to /tmp directory

[Takenori MATSUMOTO]

I find the following messages in syslog.

Sep 1 07:12:38 cn1 kernel: [510962.435074] type=1400 audit(1409523158.899:794): apparmor="DENIED" operation="open" profile="libvirt-62ecd5a8-9b3b-4320-92f0-e54ef8cdf851" name="/tmp/" pid=11254 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=0
Sep 1 07:12:38 cn1 kernel: [510962.435085] type=1400 audit(1409523158.899:795): apparmor="DENIED" operation="open" profile="libvirt-62ecd5a8-9b3b-4320-92f0-e54ef8cdf851" name="/var/tmp/" pid=11254 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=0

Is this OK to just ignore this messages? Or if this is bug, is there solution or workaround?

Seeing nova-compute.log, at the time, nova-compute just tried to create an instance.

2014-09-01 07:12:38.096 3245 INFO nova.virt.libvirt.driver [req-fd1d0575-0c7a-4977-a9bd-605824da7612 c771d7600a4445d98ef365311dab7b51 f468ee3dda1142c587cf44d31031eca9] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Creating image
2014-09-01 07:12:38.212 3245 INFO nova.virt.libvirt.firewall [req-fd1d0575-0c7a-4977-a9bd-605824da7612 c771d7600a4445d98ef365311dab7b51 f468ee3dda1142c587cf44d31031eca9] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Called setup_basic_filtering in nwfilter
2014-09-01 07:12:38.212 3245 INFO nova.virt.libvirt.firewall [req-fd1d0575-0c7a-4977-a9bd-605824da7612 c771d7600a4445d98ef365311dab7b51 f468ee3dda1142c587cf44d31031eca9] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Ensuring static filters
2014-09-01 07:12:39.093 3245 INFO nova.compute.manager [-] Lifecycle event 0 on VM 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851
2014-09-01 07:12:39.163 3245 INFO nova.virt.libvirt.driver [-] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Instance spawned successfully.
2014-09-01 07:12:39.191 3245 INFO nova.compute.manager [-] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] During sync_power_state the instance has a pending task (spawning). Skip.

A volume was attached to the instance after that, then the instance was destroyed.

2014-09-01 07:14:09.597 3245 AUDIT nova.compute.manager [req-7120818d-61a7-4f78-8a22-f186034f3da1 c771d7600a4445d98ef365311dab7b51 f468ee3dda1142c587cf44d31031eca9] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Attaching volume 4531347c-71fe-4571-985b-a4b9ae3ab70e to /dev/vdb

2014-09-01 07:20:11.584 3245 AUDIT nova.compute.manager [req-da17d4bd-1d23-4ead-888f-da5e66dde656 c771d7600a4445d98ef365311dab7b51 f468ee3dda1142c587cf44d31031eca9] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Detach volume 4531347c-71fe-4571-985b-a4b9ae3ab70e from mountpoint /dev/vdb

2014-09-01 07:21:14.313 3245 INFO nova.compute.manager [-] Lifecycle event 1 on VM 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851
2014-09-01 07:21:14.317 3245 INFO nova.virt.libvirt.driver [-] [instance: 62ecd5a8-9b3b-4320-92f0-e54ef8cdf851] Instance destroyed successfully.

It seems this messages cause no problem. But I don't know whether this DENIED error from apparmor can be ignored.
In another running test, we saw the message with an error of accessing ceph file.

Aug 4 06:36:02 cn1 kernel: [1058887.801002] type=1400 audit(1407101762.890:2098): apparmor="DENIED" operation="open" profile="libvirt-53a1d479-2299-4069-a691-72f3f5ae7a6e" name="/tmp/" pid=8336 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=110 ouid=0
Aug 4 06:36:02 cn1 kernel: [1058887.801012] type=1400 audit(1407101762.890:2099): apparmor="DENIED" operation="open" profile="libvirt-53a1d479-2299-4069-a691-72f3f5ae7a6e" name="/var/tmp/" pid=8336 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=110 ouid=0
Aug 4 06:37:31 cn1 kernel: [1058976.654848] type=1400 audit(1407101851.714:2102): apparmor="DENIED" operation="open" profile="libvirt-53a1d479-2299-4069-a691-72f3f5ae7a6e" name="/etc/ceph/ceph.client.cinder.keyring" pid=8336 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=110 ouid=108

It could be possibility that ceph library linked to qemu-system-x86 tried to access forbidden files.

- Software versions
ii nova-common 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - common files
ii nova-compute 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2014.1.2-0ubuntu1.1 all OpenStack Compute - compute node libvirt support

ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.2 amd64 QEMU full system emulation binaries (x86)
ii ceph-common 0.80.1-0ubuntu1.1 amd64 common utilities to mount and interact with a ceph storage cluster

[Dave Chiluk (chiluk)]

I accidentally opened a similar but slightly different bug to this one at
https://bugs.launchpad.net/charms/+source/ceph/+bug/1403648

These messages can be safely ignored, although we are pursuing a way to clean them up.

[Serge Hallyn (serge-hallyn)]

So should this be marked a dup of 1403648 or not? That is, will the proposed fix in that bug's comments also clean up this bug?

Marking low priority since "these messages can be safely ignored".

[Dave Chiluk (chiluk)]

@Takenori
These messages can be worked around by editing /etc/apparmor.d/abstractions/libvirt-qemu as such
__________________________________________________________
--- libvirt-qemu.orig 2014-12-19 20:13:31.162926539 +0000
+++ libvirt-qemu 2014-12-19 20:20:32.226276231 +0000
@@ -130,6 +130,7 @@

   # for rbd
   /etc/ceph/ceph.conf r,
+ /var/lib/charm/ceph/ceph.conf r,

   # for access to hugepages
   owner "/run/hugepages/kvm/libvirt/qemu/**" rw,
@@ -146,3 +147,8 @@
   # for ppc device-tree access
   @{PROC}/device-tree/ r,
   @{PROC}/device-tree/** r,
+
+ # deny qemu access to /tmp
+ deny /tmp/ r,
+ deny /var/tmp/ r,
+
----------------------------------------------------------------------------------

You are correct in assuming that qemu using librbd to talk to the ceph cluster which in turn invokes reads on /etc/ceph/ceph.conf. However I'm not sure why your specific setup is attempting to access your cinder keyring. If I understand correctly once the ceph.conf is read it should use the secret contained in the libvirt xml definition for the VM rather than the global keyrings. Can you please verify that your libvirt xml for related guests has a line like
<secret type='ceph' uuid='514c9fca-8cbe-11e2-9c52-3bc8c7819472'/>
albeit with a different uuid?

Also can you then check /etc/libvirt/secrets/<uuid.xml> and check where it's pointing?

Recent revisions of the charms have the secret file pointing at <name>client.nova-compute secret</name> on nova-compute nodes rather than the cinder key. Which really only should get used on the cinder node.

Additionally I was unable to reproduce the error similar to "Aug 4 06:37:31 cn1 kernel: [1058976.654848] type=1400 audit(1407101851.714:2102): apparmor="DENIED" operation="open" profile="libvirt-53a1d479-2299-4069-a691-72f3f5ae7a6e" name="/etc/ceph/ceph.client.cinder.keyring" pid=8336 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=110 ouid=108" .

[Helio Loureiro (helioloureiro)]

Issue back on Xenial, 16.04.

I commented the denials statements and included an extra "/tmp rw," to ensure it could access properly /tmp directory.

[Nathan Rennie-Waldock (nathan-renniewaldock)]

In normal use, this doesn't seem to be an issue. However, access to /tmp is required to enable qemu's samba server.

[Paul Donohue (s-launchpad-paulsd-com)]

To use qemu's samba server without allowing access to all of /tmp, you can add the following lines to /etc/apparmor.d/abstractions/libvirt-qemu somewhere above the deny rule:

owner /tmp/qemu-smb.*/ rw,
owner /tmp/qemu-smb.*/* rw,

It would be nice if this could be included in the official version of that file...

[Christian Ehrhardt  (paelzer)]

FYI Broken the smb related discussion into new bug 1786159

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 4.6.0-2ubuntu1

---------------
libvirt (4.6.0-2ubuntu1) cosmic; urgency=medium

  * Merged with Debian unstable (LP: #1786957).
    Among many other new features and fixes this includes fixes
    for (LP: #1754871), Remaining changes:
    - Disable libssh2 support (universe dependency)
    - Disable firewalld support (universe dependency)
    - Set qemu-group to kvm (for compat with older ubuntu)
    - Additional apport package-hook
    - Autostart default bridged network (As upstream does, but not Debian).
      In addition to just enabling it our solution provides:
      + do not autostart if subnet is already taken (e.g. in guests).
      + iterate some alternative subnets before giving up
    - d/p/ubuntu/Allow-libvirt-group-to-access-the-socket.patch: This is
      the group based access to libvirt functions as it was used in Ubuntu
      for quite long.
      + d/p/ubuntu/daemon-augeas-fix-expected.patch fix some related tests
        due to the group access change.
      + d/libvirt-daemon-system.postinst: add users in sudo to the libvirt
        group.
    - ubuntu/parallel-shutdown.patch: set parallel shutdown by default.
    - d/p/ubuntu/enable-kvm-spice.patch: compat with older Ubuntu qemu/kvm
      which provided a separate kvm-spice.
    - Xen related
      - d/p/ubuntu/ubuntu-libxl-qemu-path.patch: this change was split. The
        section that adapts the path of the emulator to the Debian/Ubuntu
        packaging is kept.
      - d/p/ubuntu/ubuntu-libxl-Fix-up-VRAM-to-minimum-requirements.patch: auto
        set VRAM to minimum requirements
      - d/p/ubuntu/xen-default-uri.patch: set default URI on xen hosts
      - Add libxl log directory
      - libvirt-uri.sh: Automatically switch default libvirt URI for users on
        Xen dom0 via user profile (was missing on changelogs before)
    - d/p/ubuntu/apibuild-skip-libvirt-common.h: drop libvirt-common.h from
      included_files to avoid build failures due to duplicate definitions.
    - Update README.Debian with Ubuntu changes
    - Convert libvirt0, libnss_libvirt and libvirt-dev to multi-arch.
    - Enable some additional features on ppc64el and s390x (for arch parity)
      + systemtap, zfs, numa and numad on s390x.
      + systemtap on ppc64el.
    - d/t/control, d/t/smoke-qemu-session: fixup smoke-qemu-session by making
      vmlinuz available and accessible (Debian bug 848314)
    - d/t/control, d/t/smoke-lxc: fix up lxc smoke test isolation
    - Add dnsmasq configuration to work with system wide dnsmasq (drop >18.04,
      no more UCA onto Xenial then which has global dnsmasq by default).
    - d/p/ubuntu/ubuntu_machine_type.patch: accept ubuntu types as pci440fx
    - Further upstreamed apparmor Delta, especially any new one
      Our former delta is split into logical pieces and is either Ubuntu only
      or is part of a continuous upstreaming effort.
      Listing related remaining changes in debian/patches/ubuntu-aa/:
      + 0001-apparmor-Allow-pygrub-to-run-on-Debian-Ubuntu.patch: apparmor:
        Allow pygrub to run on Debian/Ubuntu
      + 0003-apparmor-libvirt-qemu-Allow-read-access-to-overcommi.patch:
        ...