apparmor denies access to vms for libvirt

[Sergey Kolekonov]

After a compute node is rebooted, apparmor starts to block access to vms for libvirt daemon.

Example logs:

/var/log/kern.log:
<5>Jul 9 13:06:54 node-4 kernel: [ 18.127069] type=1400 audit(1436447214.575:19): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=3724 comm="libvirtd" requested_mask="trace" denied_mask="trace" peer="unconfined"
<5>Jul 9 13:06:54 node-4 kernel: [ 18.127303] type=1400 audit(1436447214.575:20): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=3724 comm="libvirtd" requested_mask="trace" denied_mask="trace" peer="unconfined"

/var/log/libvirt/libvirtd.log:
2015-07-09 13:09:09.983+0000: 3728: error : virProcessKillPainfully:373 : Failed to terminate process 5613 with SIGTERM: Permission denied

  release: "7.0"
  openstack_version: "2014.2.2-7.0"
  api: "1.0"
  build_number: "14"
  build_id: "2015-07-08_12-22-41"
  nailgun_sha: "976baf842242a5f97c95bdc3e20328fa0558bf69"
  python-fuelclient_sha: "018c53561baa87397e50368c7f48674829c322c0"
  astute_sha: "9cbb8ae5adbe6e758b24b3c1021aac1b662344e8"
  fuel-library_sha: "6b0c909ad800e9d17bc7436affcc814db3e05cd3"
  fuel-ostf_sha: "e3ad92b0e4a9301ffe0969a3bc5d6073966a27b4"
  fuelmain_sha: "185d21d4d42233d5abbc57728e94c8f70114a49e"

[Aleksander Mogylchenko (amogylchenko)]

We ship custom libvirt (not from Ubuntu but from Debian), which does not have apparmor rules at all:
https://review.fuel-infra.org/gitweb?p=packages/trusty/libvirt.git;a=commit;h=abedecff4bede482906227a5cedbb17cf1b8302d

They should be added to the package, ensuring the presence of this particular fix:
https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1298611

[Alexei Sheplyakov (asheplyakov)]

> We ship custom libvirt (not from Ubuntu but from Debian), which does not have apparmor rules at all:
> They should be added to the package

I think switching off apparmor (via the kerne command line) is a better option.
This way we can consume updated directly from Debian Jessie without having to rebase the apparmor support patch.

[Aleksander Mogylchenko (amogylchenko)]

well, then the question is really this: do we need apparmor?

[Sergey Kolekonov (skolekonov)]

Such a strategy looks a bit strange. Does it make sense to use Ubuntu and then disable built-in features to support Debian packages?

[Roman Podoliaka (rpodolyaka)]

I agree with Sergey on this: libvirt/apparmor pair worked correctly in 6.1 and I don't see why we should change that just for the sake of bringing a new libvirt package from Debian.

Left as is, this causes multiple issues like https://bugs.launchpad.net/fuel/+bug/1481772 as well as breaking simple things like stopping of VMs.

So, IMHO, we should either use proper apparmor profiles for Debian libvirt package or roll back to the Trusty version of libvirt.

[Aleksander Mogylchenko (amogylchenko)]

The main purpose of having update package (not the one from Ubuntu upstream) was to introduce features not available there yet. So you should pick one of two: either stable package from Ubuntu Trusty (with less features), or more cutting-edge package from Debian.

[Roman Podoliaka (rpodolyaka)]

Then, I'm afraid, we should say no to features and use the version from Trusty (unless you are going to do a back port of 'apparmor-ready' libvirt package from Ubuntu Vivid or newer).

[Bogdan Dobrelya (bogdando)]

Raised to critical due to many duplicates

[Aleksander Mogylchenko (amogylchenko)]

Could you please test if this package fixes the problem:
http://perestroika-repo-tst.infra.mirantis.net/review/LP-1473421/mos-repos/ubuntu/7.0

[Aleksander Mogylchenko (amogylchenko)]

Assigned to the reporter to do the test.

[Roman Podoliaka (rpodolyaka)]

Alexander, I upgraded the libvirt packages from the repository you provided and now both `nova stop' and `nova delete' succeed. But it's was just a `smoke` testing, not a thorough one for sure.

libvirt packages versions: http://paste.openstack.org/show/412873/

NOTE: apparently a node reboot was required for apparmor profiles to be applied properly.

[Sergey Kolekonov (skolekonov)]

I've checked these packages on my environment briefly, everything looks ok

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/10342
Submitter: Michael Semenov <email address hidden>
Branch: 7.0

Commit: 8594c4ab91acf3f00814426f3494e6701bdf7666
Author: Aleksandr Mogylchenko <email address hidden>
Date: Thu Aug 13 09:45:05 2015

Update libvirt-bin package to fix apparmor denials.

Patch source:
http://www.redhat.com/archives/libvir-list/2014-October/msg00011.html

Update of auto* files was required since Trusty comes with newer automake,
causing the build to fail.

Change-Id: Icf264fe6f0ccc68e6a85caab60a46f565c5ff04b
Closes-Bug: #1473421

[Aleksander Mogylchenko (amogylchenko)]

My previous analysis was not complete: libvirt-bin package from Debian does have apparmor support, although limited (because apparmor support in Debian is limited). I've updated our package with the patch from upstream fixing this problem:
http://www.redhat.com/archives/libvir-list/2014-October/msg00011.html

According to tests the problem is gone. Patch limits our ability to receive updated from upstream Debian (because we use modified package), but in the other hand all changes are in one single patch.