libvirt fails with failure to open mount namespace

Bug #1788603 reported by Thadeu Lima de Souza Cascardo on 2018-08-23
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Debian)
Fix Released
Unknown
libvirt (Ubuntu)
Critical
Unassigned
Bionic
Undecided
Unassigned
Cosmic
Critical
Unassigned
linux (Ubuntu)
High
Thadeu Lima de Souza Cascardo
Bionic
Undecided
Unassigned
Cosmic
High
Thadeu Lima de Souza Cascardo

Bug Description

[Impact]

 * Libvirt will no more be able to start guests with newer kernels
   (>=4.18)

 * We brought a fix upstream that we want to backport to potentially
   affected releases (B+C)

[Test Case]

Note: This can but does not have to be tested in nested virtualization (Since we need to change the kernel it is probably the preferred way to do it in nested, so this is what I describe)
1. Get a KVM guest with the Release to test (Cosmic as an example) and give it a size to be able to host a nested guest
  $ sudo apt install uvtool-libvirt
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --cpu 8 --memory 4096 --password ubuntu c-nested-kvm-4.18 arch=amd64 release=cosmic label=daily
  $ uvt-kvm ssh c-nested-kvm-4.18
2. In the guest set up another guest
  $ uvt-kvm ssh c-nested-kvm-4.18
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --password ubuntu c-nested-guest arch=amd64 release=cosmic label=daily
  $ sudo apt install uvtool-libvirt
3. The above will work, now get a kernel >=4.18 e.g. the mainline kernels at
   https://wiki.ubuntu.com/Kernel/MainlineBuilds
   $ wget (link of the day)
   $ sudo dpkg -i *.deb
4. reboot into the 4.18 kernel
   $ sudo reboot
5. Start the guest again
   $ virsh start c-nested-guest

Without fix this will fail like:
  error: Failed to start domain c-nested-guest
  error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

And an associated Dmesg/Apparmor entry:
[ 519.529760] audit: type=1400 audit(1535364820.976:27): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=1692 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-4ae5dbbe-641c-4934-ae7c-956bcc8b9e2f"

6. In the guest update from PPA/Proposed that you want to test to bring in the libvirt package with the updated rules (Example with the PPA).
  $ sudo add-apt-repository ppa:ci-train-ppa-service/3381
  $ sudo apt upgrade

7. Start the guest again and it will now work
  $ virsh start c-nested-guest

[Regression Potential]

 * The change is "only" opening up the apparmor profile for libvirt a
   little bit for the access now being reported as read in addition.
   That should in itself be rather regression free, but in addition it has
   an ack of Jamie Strandboge to be sure from Securities POV (you can
   follow the upstreaming thread at to check:
     https://www.redhat.com/archives/libvir-list/2018-August/msg01597.html

[Other Info]

 * This is no issue "yet" but would be one when the Cosmic Kernel becomes available as HWE kernel. We'd want to have the apparmor profile in libvirt ready at the time so that we never have users see this error in the Field on a Released Ubuntu version.

---

$ virsh start cosmic-i386
error: Failed to start domain cosmic-i386
error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

Happens with 4.18 only, 4.17 is fine.

27013 openat(AT_FDCWD, "/proc/27012/ns/mnt", O_RDONLY) = -1 EACCES (Permission denied)

Opening a self or parent mount namespace works fine as tested with cat /proc/self/ns/mnt and cat /proc/self/$$/mnt. Still investigating.

Changed in linux (Ubuntu):
assignee: nobody → Thadeu Lima de Souza Cascardo (cascardo)
importance: Undecided → High
status: New → Confirmed
status: Confirmed → In Progress

So, investigated this into an apparmor difference between 4.17 and 4.18. This is due to commit 338d0be437ef10e247a35aed83dbab182cf406a2 ("apparmor: fix ptrace read check").

libvirtd here is using only trace, and not read. The patch below for libvirtd apparmor policy fixes it for me.

--- /etc/apparmor.d/usr.sbin.libvirtd 2018-08-23 14:52:04.574252908 -0300
+++ ../usr.sbin.libvirtd 2018-08-23 14:51:46.773728841 -0300
@@ -50,10 +50,10 @@
   # for --p2p migrations
   unix (send, receive) type=stream addr=none peer=(label=unconfined addr=none),

- ptrace (trace) peer=unconfined,
- ptrace (trace) peer=/usr/sbin/libvirtd,
- ptrace (trace) peer=/usr/sbin/dnsmasq,
- ptrace (trace) peer=libvirt-*,
+ ptrace (read,trace) peer=unconfined,
+ ptrace (read,trace) peer=/usr/sbin/libvirtd,
+ ptrace (read,trace) peer=/usr/sbin/dnsmasq,
+ ptrace (read,trace) peer=libvirt-*,

   signal (send) peer=/usr/sbin/dnsmasq,
   signal (read, send) peer=libvirt-*,

Changed in libvirt (Ubuntu Cosmic):
importance: Undecided → Critical

ago 23 14:40:18 calabresa audit[5997]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=5997 comm="libvirtd" requested_mask="read" denied_mask="read" peer="unconfined"
ago 22 18:00:12 calabresa audit[4409]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=4409 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-14e92a75-7668-4b97-8f92-322fc1b9c78a"
ago 23 09:18:06 calabresa audit[2507]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=2507 comm="libvirtd" requested_mask="read" denied_mask="read" peer="unconfined"

Here are some messages with the apparmor denial.

Tested, confirmed wrapped in a libvirt upstream patch and submitted at https://www.redhat.com/archives/libvir-list/2018-August/msg01532.html

Lets give the people a few days to read and ack as well then we can push it to cosmic.

It came to my mind that due to HWE-Kernels we eventually also want this in Bionic.
So lets add a task right away to not forget.

Changed in libvirt (Ubuntu Bionic):
status: New → Confirmed
Changed in libvirt (Ubuntu Cosmic):
status: New → In Progress

Tested against Kernel 4.18 from mainline builds:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18/linux-image-unsigned-4.18.0-041800-generic_4.18.0-041800.201808122131_amd64.deb

For the latter SRU I summarized the testing procedure:
Note: This can but does not have to be tested in nested virtualization (Since we need to change the kernel it is probably the preferred way to do it in nested, so this is what I describe)
1. Get a KVM guest with the Release to test (Cosmic as an example) and give it a size to be able to host a nested guest
  $ sudo apt install uvtool-libvirt
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --cpu 8 --memory 4096 --password ubuntu c-nested-kvm-4.18 arch=amd64 release=cosmic label=daily
  $ uvt-kvm ssh c-nested-kvm-4.18
2. In the guest set up another guest
  $ uvt-kvm ssh c-nested-kvm-4.18
  $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 release=cosmic label=daily
  $ uvt-kvm create --password ubuntu c-nested-guest arch=amd64 release=cosmic label=daily
  $ sudo apt install uvtool-libvirt
3. The above will work, now get a kernel >=4.18 e.g. the mainline kernels at
   https://wiki.ubuntu.com/Kernel/MainlineBuilds
   $ wget (link of the day)
   $ sudo dpkg -i *.deb
4. reboot into the 4.18 kernel
   $ sudo reboot
5. Start the guest again
   $ virsh start c-nested-guest

Without fix this will fail like:
  error: Failed to start domain c-nested-guest
  error: internal error: child reported: Kernel does not provide mount namespace: Permission denied

And an associated Dmesg/Apparmor entry:
[ 519.529760] audit: type=1400 audit(1535364820.976:27): apparmor="DENIED" operation="ptrace" profile="/usr/sbin/libvirtd" pid=1692 comm="libvirtd" requested_mask="read" denied_mask="read" peer="libvirt-4ae5dbbe-641c-4934-ae7c-956bcc8b9e2f"

6. In the guest update from PPA/Proposed that you want to test to bring in the libvirt package with the updated rules (Example with the PPA).
  $ sudo add-apt-repository ppa:ci-train-ppa-service/3381
  $ sudo apt upgrade

7. Start the guest again and it will now work
  $ virsh start c-nested-guest

Tested from PPA, also passed through some more checks and all look fine - but that was expected given the change being "just" opening up apparmor a little bit.

Uploading to Cosmic ...

description: updated

FYI cosmic is slightly stalled by glibc migration, other than that it looks good to me at the moment.

PPA for the Bionic SRU at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3386
Note: This not only waits for Cosmic to complete, but also for the former SRU to complete as well.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 4.6.0-2ubuntu3

---------------
libvirt (4.6.0-2ubuntu3) cosmic; urgency=medium

  * d/p/ubuntu-aa/lp-1788603-fix-ptrace-rules-with-kernel-4.18.patch:
    avoid issues with newer kernels >=4.18 (LP: #1788603)

 -- Christian Ehrhardt <email address hidden> Mon, 27 Aug 2018 10:57:57 +0200

Changed in libvirt (Ubuntu Cosmic):
status: In Progress → Fix Released

Ok, glibc was unblocked and we can take a look at pushing this for Bionic as well.

Had another set of regression tests as I bundled another fix.
All good still.

Pushing for the SRU teams consideration ...

Hello Thadeu, or anyone else affected,

Accepted libvirt into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/4.0.0-1ubuntu8.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libvirt (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-bionic

Started a guest with Bionic and Kernel 4.18 - working due to the updated apparmor profile.
We already had general regression tests on the identical PPA content, not redoing the same.

Setting verified.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux (Ubuntu Bionic):
status: New → Confirmed

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 4.0.0-1ubuntu8.5

---------------
libvirt (4.0.0-1ubuntu8.5) bionic; urgency=medium

  * d/p/ubuntu-aa/lp-1788603-fix-ptrace-rules-with-kernel-4.18.patch:
    avoid issues with newer kernels >=4.18 (LP: #1788603)
  * d/p/ubuntu/lp-1789659-don-t-check-for-parallel-iteration-in-hash.patch:
    remove broken and redundant check for parallel iteration in hash functions
    (LP: #1789659)

 -- Christian Ehrhardt <email address hidden> Tue, 28 Aug 2018 07:26:19 +0200

Changed in libvirt (Ubuntu Bionic):
status: Fix Committed → Fix Released
Changed in libvirt (Debian):
status: Unknown → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.