Loading of libvirt profile fails in artful

[Christian Ehrhardt ]

Hi,
we have artful images since a few days - yay!
Unfortunately they run into issues.

I first thought this would be related to our slightly uncommon KVM-in-LXD setup but I can reproduce in nested KVM as well.
For simplicity I'll not mention the -in-lXD logs here as they are more noisy and less common.
I'm rather convinced if I'd have artful on bare metal it would show there as well but couldn't prove yet.

First of all aa-status looks sane to me:
$ sudo aa-status
apparmor module is loaded.
15 profiles are loaded.
15 profiles are in enforce mode.
   /sbin/dhclient
   /usr/bin/lxc-start
   /usr/lib/NetworkManager/nm-dhcp-client.action
   /usr/lib/NetworkManager/nm-dhcp-helper
   /usr/lib/connman/scripts/dhclient-script
   /usr/lib/snapd/snap-confine
   /usr/lib/snapd/snap-confine//mount-namespace-capture-helper
   /usr/sbin/libvirtd
   /usr/sbin/libvirtd//qemu_bridge_helper
   /usr/sbin/tcpdump
   lxc-container-default
   lxc-container-default-cgns
   lxc-container-default-with-mounting
   lxc-container-default-with-nesting
   virt-aa-helper
0 profiles are in complain mode.
2 processes have profiles defined.
2 processes are in enforce mode.
   /sbin/dhclient (850)
   /usr/sbin/libvirtd (3635)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

But on actually creating a guest I get an apparmor related issue:
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
$ uvt-kvm create --password=ubuntu x-on-a-test release=xenial label=daily
uvt-kvm: error: libvirt: internal error: Process exited prior to exec: libvirt: error : unable to set AppArmor profile 'libvirt-38a056b4-d6b6-4bd7-a61a-add6d9b68bb0' for '/usr/bin/kvm-spice': No such file or directory

Along that I see this error about change_profile:
apparmor="DENIED" operation="change_profile" info="label not found" error=-2 profile="/usr/sbin/libvirtd" name="libvirt-38a056b4-d6b6-4bd7-a61a-add6d9b68bb0" pid=4492 comm="libvirtd"

The same on a zesty system loads fine and dmesg holds a working reload.
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
$ uvt-kvm create --password=ubuntu x-on-z-test release=xenial label=daily
(working fine)

I cleared dmesg and started the guest on both to get all apparmor messages that are related:
Good case (zesty)
apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-87503137-d2ad-4bd1-bc37-f2b7e4b468d5" pid=6099 comm="apparmor_parser"
apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-87503137-d2ad-4bd1-bc37-f2b7e4b468d5" pid=6138 comm="apparmor_parser

Bad case (artful)
apparmor="DENIED" operation="change_profile" info="label not found" error=-2 profile="/usr/sbin/libvirtd" name="libvirt-c5ef4b2e-9fcc-42a6-9fc0-651c4ed698f1" pid=4618 comm="libvirtd"

I didn't see load/replace in artful so far.

[John Johansen (jjohansen)]

There is a bug in the /etc/apparmor.d/abstractions/libvirt-qemu file on line 183

  /sys/devices/system/cpu/cpu*/online r

is missing the the trailing ,
it should be
  /sys/devices/system/cpu/cpu*/online r,

this prevents libvirt from loading the vm profile. Unfortunately it does not report the error and only fails/reports the error when it attempts to transition to the profile that failed being loaded.

Once the abstraction is fixed nested kvm works as expected for me.

I have not tried this with an lxd container yet

[Christian Ehrhardt  (paelzer)]

On Thu, Apr 27, 2017 at 12:55 PM, John Johansen <<email address hidden>
> wrote:

> /sys/devices/system/cpu/cpu*/online r
>

Thanks, yes that is a recent fix .. umm error.
Fixed soon - thanks.

--
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

[Christian Ehrhardt  (paelzer)]

Thanks John for spotting the issue!
I was asked why I usually try to test too much, that is the reason :-/
Rushing it once and failing instantly.

But hey our regular jenkins based testing found it - at least.
Still that means probably overall 1-2 days of broken guests :-/.

Now that this is known to be my fault I'll mark the apparmor bit as incomplete.

Fix is already building for artful.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 2.5.0-3ubuntu7

---------------
libvirt (2.5.0-3ubuntu7) artful; urgency=medium

  * debian/patches/ubuntu/apparmor-ppcwrapper.patch: update to add missing
    colon (LP: #1686621).

 -- Christian Ehrhardt <email address hidden> Thu, 27 Apr 2017 13:16:05 +0200

[Christian Ehrhardt  (paelzer)]

For the internet search graveyard I beg a pardon, yes it is a comma not a colon - the guy that named ";" a semi-colon will forever be hated by me as it is in some sense also a semi-comma.
Anyway not worth or able to fix the changelogs back in time, but then also not so important.