libvirt fails with failure to open mount namespace

[Thadeu Lima de Souza Cascardo]

[Impact]

 * Libvirt will no more be able to start guests with newer kernels
   (>=4.18)

 * We brought a fix upstream that we want to backport to potentially
   affected releases (B+C)

[Test Case]

Note: This can but does not have to be tested in nested virtualization (Since we need to change the kernel it is probably the preferred way to do it in nested, so this is what I describe)
1. Get a KVM guest with the Release to test (Cosmic as an example) and give it a size to be able to host a nested guest
  $ sudo apt install uvtool-libvirt
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --cpu 8 --memory 4096 --password ubuntu c-nested-kvm-4.18 arch=amd64 release=cosmic label=daily
  $ uvt-kvm ssh c-nested-kvm-4.18
2. In the guest set up another guest
  $ uvt-kvm ssh c-nested-kvm-4.18
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --password ubuntu c-nested-guest arch=amd64 release=cosmic label=daily
  $ sudo apt install uvtool-libvirt
3. The above will work, now get a kernel >=4.18 e.g. the mainline kernels at
   https://wiki.ubuntu.com/Kernel/MainlineBuilds
   $ wget (link of the day)
   $ sudo dpkg -i *.deb
4. reboot into the 4.18 kernel
   $ sudo reboot
5. Start the guest again
   $ virsh start c-nested-guest

Without fix this will fail like:
  error: Failed to start domain c-nested-guest
  error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

And an associated Dmesg/Apparmor entry:
[ 519.529760] audit: type=1400 audit(1535364820.976:27): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=1692 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-4ae5dbbe-641c-4934-ae7c-956bcc8b9e2f"

6. In the guest update from PPA/Proposed that you want to test to bring in the libvirt package with the updated rules (Example with the PPA).
  $ sudo add-apt-repository ppa:ci-train-ppa-service/3381
  $ sudo apt upgrade

7. Start the guest again and it will now work
  $ virsh start c-nested-guest

[Regression Potential]

 * The change is "only" opening up the apparmor profile for libvirt a
   little bit for the access now being reported as read in addition.
   That should in itself be rather regression free, but in addition it has
   an ack of Jamie Strandboge to be sure from Securities POV (you can
   follow the upstreaming thread at to check:
     https://www.redhat.com/archives/libvir-list/2018-August/msg01597.html

[Other Info]

 * This is no issue "yet" but would be one when the Cosmic Kernel becomes available as HWE kernel. We'd want to have the apparmor profile in libvirt ready at the time so that we never have users see this error in the Field on a Released Ubuntu version.

---

$ virsh start cosmic-i386
error: Failed to start domain cosmic-i386
error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

Happens with 4.18 only, 4.17 is fine.

27013 openat(AT_FDCWD, "/proc/27012/ns/mnt", O_RDONLY) = -1 EACCES (Permission denied)

Opening a self or parent mount namespace works fine as tested with cat /proc/self/ns/mnt and cat /proc/self/$$/mnt. Still investigating.

[Thadeu Lima de Souza Cascardo (cascardo)]

So, investigated this into an apparmor difference between 4.17 and 4.18. This is due to commit 338d0be437ef10e247a35aed83dbab182cf406a2 ("apparmor: fix ptrace read check").

libvirtd here is using only trace, and not read. The patch below for libvirtd apparmor policy fixes it for me.

--- /etc/apparmor.d/usr.sbin.libvirtd 2018-08-23 14:52:04.574252908 -0300
+++ ../usr.sbin.libvirtd 2018-08-23 14:51:46.773728841 -0300
@@ -50,10 +50,10 @@
   # for --p2p migrations
   unix (send, receive) type=stream addr=none peer=(label=unconfined addr=none),

- ptrace (trace) peer=unconfined,
- ptrace (trace) peer=/usr/sbin/libvirtd,
- ptrace (trace) peer=/usr/sbin/dnsmasq,
- ptrace (trace) peer=libvirt-*,
+ ptrace (read,trace) peer=unconfined,
+ ptrace (read,trace) peer=/usr/sbin/libvirtd,
+ ptrace (read,trace) peer=/usr/sbin/dnsmasq,
+ ptrace (read,trace) peer=libvirt-*,

   signal (send) peer=/usr/sbin/dnsmasq,
   signal (read, send) peer=libvirt-*,

[Thadeu Lima de Souza Cascardo (cascardo)]

ago 23 14:40:18 calabresa audit[5997]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=5997 comm="libvirtd" requested_mask="read" denied_mask="read" peer="unconfined"
ago 22 18:00:12 calabresa audit[4409]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=4409 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-14e92a75-7668-4b97-8f92-322fc1b9c78a"
ago 23 09:18:06 calabresa audit[2507]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=2507 comm="libvirtd" requested_mask="read" denied_mask="read" peer="unconfined"

Here are some messages with the apparmor denial.

[Christian Ehrhardt  (paelzer)]

Tested, confirmed wrapped in a libvirt upstream patch and submitted at https://www.redhat.com/archives/libvir-list/2018-August/msg01532.html

Lets give the people a few days to read and ack as well then we can push it to cosmic.

It came to my mind that due to HWE-Kernels we eventually also want this in Bionic.
So lets add a task right away to not forget.

[Christian Ehrhardt  (paelzer)]

Upstream accepted: https://libvirt.org/git/?p=libvirt.git;a=commit;h=8741b9435108b1f0d87670e44e1ed75f806b7791

[Christian Ehrhardt  (paelzer)]

Prepared in PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3381

[Christian Ehrhardt  (paelzer)]

Tested against Kernel 4.18 from mainline builds:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18/linux-image-unsigned-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb

For the latter SRU I summarized the testing procedure:
Note: This can but does not have to be tested in nested virtualization (Since we need to change the kernel it is probably the preferred way to do it in nested, so this is what I describe)
1. Get a KVM guest with the Release to test (Cosmic as an example) and give it a size to be able to host a nested guest
  $ sudo apt install uvtool-libvirt
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --cpu 8 --memory 4096 --password ubuntu c-nested-kvm-4.18 arch=amd64 release=cosmic label=daily
  $ uvt-kvm ssh c-nested-kvm-4.18
2. In the guest set up another guest
  $ uvt-kvm ssh c-nested-kvm-4.18
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --password ubuntu c-nested-guest arch=amd64 release=cosmic label=daily
  $ sudo apt install uvtool-libvirt
3. The above will work, now get a kernel >=4.18 e.g. the mainline kernels at
   https://wiki.ubuntu.com/Kernel/MainlineBuilds
   $ wget (link of the day)
   $ sudo dpkg -i *.deb
4. reboot into the 4.18 kernel
   $ sudo reboot
5. Start the guest again
   $ virsh start c-nested-guest

Without fix this will fail like:
  error: Failed to start domain c-nested-guest
  error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

And an associated Dmesg/Apparmor entry:
[ 519.529760] audit: type=1400 audit(1535364820.976:27): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=1692 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-4ae5dbbe-641c-4934-ae7c-956bcc8b9e2f"

6. In the guest update from PPA/Proposed that you want to test to bring in the libvirt package with the updated rules (Example with the PPA).
  $ sudo add-apt-repository ppa:ci-train-ppa-service/3381
  $ sudo apt upgrade

7. Start the guest again and it will now work
  $ virsh start c-nested-guest

[Christian Ehrhardt  (paelzer)]

Tested from PPA, also passed through some more checks and all look fine - but that was expected given the change being "just" opening up apparmor a little bit.

Uploading to Cosmic ...

[Christian Ehrhardt  (paelzer)]

FYI cosmic is slightly stalled by glibc migration, other than that it looks good to me at the moment.

[Christian Ehrhardt  (paelzer)]

PPA for the Bionic SRU at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3386
Note: This not only waits for Cosmic to complete, but also for the former SRU to complete as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 4.6.0-2ubuntu3

---------------
libvirt (4.6.0-2ubuntu3) cosmic; urgency=medium

  * d/p/ubuntu-aa/lp-1788603-fix-ptrace-rules-with-kernel-4.18.patch:
    avoid issues with newer kernels >=4.18 (LP: #1788603)

 -- Christian Ehrhardt <email address hidden> Mon, 27 Aug 2018 10:57:57 +0200

[Christian Ehrhardt  (paelzer)]

Ok, glibc was unblocked and we can take a look at pushing this for Bionic as well.

[Christian Ehrhardt  (paelzer)]

Had another set of regression tests as I bundled another fix.
All good still.

Pushing for the SRU teams consideration ...

[Andy Whitcroft (apw)]

Hello Thadeu, or anyone else affected,

Accepted libvirt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/4.0.0-1ubuntu8.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Christian Ehrhardt  (paelzer)]

Started a guest with Bionic and Kernel 4.18 - working due to the updated apparmor profile.
We already had general regression tests on the identical PPA content, not redoing the same.

Setting verified.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 4.0.0-1ubuntu8.5

---------------
libvirt (4.0.0-1ubuntu8.5) bionic; urgency=medium

  * d/p/ubuntu-aa/lp-1788603-fix-ptrace-rules-with-kernel-4.18.patch:
    avoid issues with newer kernels >=4.18 (LP: #1788603)
  * d/p/ubuntu/lp-1789659-don-t-check-for-parallel-iteration-in-hash.patch:
    remove broken and redundant check for parallel iteration in hash functions
    (LP: #1789659)

 -- Christian Ehrhardt <email address hidden> Tue, 28 Aug 2018 07:26:19 +0200