libvirt-qemu apparmor profiles misses several important entries

Bug #1680384 reported by ChristianEhrhardt on 2017-04-06
18
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
High
ChristianEhrhardt
Trusty
Undecided
Unassigned
Xenial
High
ChristianEhrhardt
Yakkety
High
ChristianEhrhardt
Zesty
High
ChristianEhrhardt

Bug Description

[Impact]

 * Attaching of virtual functions (SR-IOV) not possible without manually
   modification of apparmor rules
 * libvirt/qemu guest logfiles can not state the program that sent the
   shutdown signal
 * on ppc64el when starting on SMT enabled it failes (expected) but does
   not present the hint about SMT being the reasons that was added in
   >=zesty
 * various apparmor denials in the log hide other issues
 * Fix by opening up apparmor rules as needed for these use-cases

[Test Case]

 * The rules covere a set of use cases, outlining them all here. Some only
   apply to >=Zesty, so I mark each subtest with release names.
 * All start with "Spawn a kvm guest e.g. via uvtool-libvirt"
 * A) >=Xenial - shutdown message
   - A1 - start the guest
   - A2 - stop the guest
   - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log
   - A-check: without the fix the log ends with <unknown process>
              With the fix it shall contain the name of the process
              stopping it instead of. So libvirt shutdown will show
              libvirtd, kill from bash shows bash, ...)
 * B) >=Xenial
   - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your
     system, one example is outlined here:
      # Prep VFs and attach
      $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs
      $ sudo modprobe vfio-pci
      $ lspci -n -s 0005:01:01.3
        0005:01:01.3 0200: 10df:e228 (rev 10)
      $ cat VF-5.1.1.3.xml
        <hostdev mode='subsystem' type='pci' managed='yes'>
          <source>
            <address domain='0x0005' bus='0x01' slot='0x01'
             function='0x3'/>
          </source>
        </hostdev>
    - B2 until bug 1679704 is fixed you need to manually increase the
      locked memory limit of the qemu process before going on
    - B3 attach the device
      $ virsh attach-device z-test VF-5.1.1.3.xml
    - B-check: without the fix it will fail for a set of apparmor denials
               with the fix the attaching will succeed
 * C) >=Zesty
   - C1 - on a ppc64el system start and stop a guest
   - C-check: without the fix apparmor will hold many denies for ppc64_cpu
              and/or accessing /sys/devices/system/cpu/...
              With the fix not only the denies are gone, but also if you
              run on a system with SMT enabled (kvm will fail there) it
              will report to you in the log about "Error: You must disable
              SMT ..."

[Regression Potential]

 * We are only further "opening up" the current apparmor profiles. So we
   should not end up affecting existing use cases by new blocks.
 * There is a very slight chance that due to accesses being allowed follow
   on actions are triggered which people are not used to like "the VF hot
   attach actually working". So if one relied on it not working it will be
   different for them.
 * Further I made some brainstorming on the potential effects but the only
   other one that came to my mind being a regression of some sort is that
   it now correctly states what send the shutdown signal. So when qemu did
   not end by itself (guest stops) but is stopped (shutdown) the old
   message was:
     terminating on signal 15 from pid 22938 (<unknown process>)
   And now will be like:
     terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd)
   For the hopefully super unlikely case that someone e.g. grepped for
   unknown here it might be a regression.
   And making that message more useful (along with getting rid of the
   associated aa deny is part of the intended fix).

[Other Info]

 * Tested manually and via regression tests in advance on bileto tickets.
   So all are green there now, there were two unrelated in some dep8 tests
   due to what seems transient network timeouts (worked on retry) - if
   showing up in proposed as well we will likely have to retrigger the
   tests until good. Latest ticket for these to look at sniff builds:
     => https://bileto.ubuntu.com/#/ticket/2738
 * Upload will be directly, since in some of the releases I will need
   dpdk-genversion -v which bileto does not support (and it doesn't
   support publishing selective releases) yet I wanted to show the extra
   tests and checks made and therefore using bileto was beneficial even if
   eventually uploading "old style"

-----

By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction.

Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there.

None of these are per Guest, so add to /etc/apparmor.d/abstractions/libvirt-qemu

The missing entries as identified are:
#1
For virtual functions the generic vfio interface
/etc/apparmor.d/abstractions/libvirt-qemu like:
  # allow guest access to the generic base vfio interface (LP: #1678322)
  /dev/vfio/vfio rw,

I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed.

#2
Binaries for ppc64el based checks in the kvm wrapper script.
That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions.

apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0
apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0

#3
Qemu tries to read who killed it to report on shutdown
[ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0

The following should be rather save
/proc/*/cmdline r,

That will make the <unkown process> in logs like the following more readable:
"qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)"

Changed in libvirt (Ubuntu):
status: New → Triaged
assignee: nobody → ChristianEhrhardt (paelzer)
importance: Undecided → High
ChristianEhrhardt (paelzer) wrote :

Prepared a Test PPA at
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2708/

Once tested on the issues and regressions we should shove that into zesty asap to meet the Freeze deadlines (or make it a very early update).

ChristianEhrhardt (paelzer) wrote :

Final Freeze today and this is not blocking install, so assume to reroll for AA once open and then SRU back.
But Tests on the PPA are fine for now.

ChristianEhrhardt (paelzer) wrote :

There is a section the ppc changes should go to:
# kvm.powerpc executes this
[...]

Also while verifying I found that the changes are in the patch but where lost in the magic of the profiles are generated on build.
But since we need to reroll for AA anyway I'll tackle that once I have a chance to push somewhere.

ChristianEhrhardt (paelzer) wrote :

The shutdown and vfio changes are actually already good, but the ppc kvm wrapper needs more.
Not only rearrange as outlined above, but also ppc64_cpu accesses more down the road, so we need to add this and check for more:

[20760.368049] audit: type=1400 audit(1491488425.671:93): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/sys/devices/system/cpu/subcores_per_core" pid=50462 comm="ppc64_cpu" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0
[20760.369318] audit: type=1400 audit(1491488425.675:94): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/sys/devices/system/cpu/cpu0/online" pid=50462 comm="ppc64_cpu" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0

------- Comment From <email address hidden> 2017-04-13 21:08 EDT-------
Please, reverse mirror LP1680384 (libvirt-qemu apparmor profiles misses several important entries).

tags: added: architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704
ChristianEhrhardt (paelzer) wrote :

Somewhat refurbished way to solve this in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2724

Once built I'm testing these in combination with the special preview kernel JJ provided to get the setrlimit issue solved.

On the actual integration we will have to see if this will still be a zesty upload or if AA will be open and we SRU from there.

ChristianEhrhardt (paelzer) wrote :

Updated the ppa with an extended fix to cover follow on access of ppc64_cpu in the apparmor profile as well now.

Since we wait on the kernel/apparmor fix to be complete (bug 1679704) I pushed it to the ppa that I mentioned before.
Now I wait on a ppc64el test kernel now by the other bug, as we need them together to fix the original issue and test this end-to-end.

Regression Testing on the changes was fine, but since we only "allow more" that was expected.

Changed in libvirt (Ubuntu):
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 2.5.0-3ubuntu6

---------------
libvirt (2.5.0-3ubuntu6) artful; urgency=medium

  * Add missing apparmor profile entries (LP: #1680384)
    - debian/patches/ubuntu/apparmor-vfio.patch: apparmor: add /dev/vfio
      for vf (hot) attach
    - debian/patches/ubuntu/apparmor-ppcwrapper.patch: apparmor: allow
      extra tools executed by kvm.powerpc
    - debian/patches/ubuntu/apparmor-shutdown.patch: apparmor: allow to
      parse cmdline of the pid that send the shutdown signal

 -- Christian Ehrhardt <email address hidden> Tue, 25 Apr 2017 14:10:06 +0200

Changed in libvirt (Ubuntu):
status: In Progress → Fix Released
Changed in libvirt (Ubuntu Xenial):
status: New → Triaged
Changed in libvirt (Ubuntu Yakkety):
status: New → Triaged
Changed in libvirt (Ubuntu Zesty):
status: New → Triaged
Changed in libvirt (Ubuntu Xenial):
importance: Undecided → High
Changed in libvirt (Ubuntu Yakkety):
importance: Undecided → High
Changed in libvirt (Ubuntu Zesty):
importance: Undecided → High
Changed in libvirt (Ubuntu Xenial):
assignee: nobody → ChristianEhrhardt (paelzer)
Changed in libvirt (Ubuntu Yakkety):
assignee: nobody → ChristianEhrhardt (paelzer)
Changed in libvirt (Ubuntu Zesty):
assignee: nobody → ChristianEhrhardt (paelzer)
description: updated
description: updated
ChristianEhrhardt (paelzer) wrote :

- Added and filled associated SRU Template
- checked once more the dep8 tests on the test bileto ppa
- checked regression tests
- added tasks for affected releases

With that in place uploaded to the unapproved queue of these releases

Please do note that Trusty is a bit old for the SRU as-is, I'll need to revaluate it once the others passed.
Reasoning: a) no complains so far about that release
           b) I'll need to re-use my test system and heavily modify it as it usually isn't working
              on trusty.
           c) not stalling the fixes for those we can verify more easily

Changed in libvirt (Ubuntu Xenial):
status: Triaged → In Progress
Changed in libvirt (Ubuntu Yakkety):
status: Triaged → Fix Committed
Changed in libvirt (Ubuntu Xenial):
status: In Progress → Fix Committed
Changed in libvirt (Ubuntu Zesty):
status: Triaged → Fix Committed
Changed in libvirt (Ubuntu Trusty):
status: New → Incomplete

Hello ChristianEhrhardt, or anyone else affected,

Accepted libvirt into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/2.5.0-3ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Brian Murray (brian-murray) wrote :

Hello ChristianEhrhardt, or anyone else affected,

Accepted libvirt into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/2.1.0-1ubuntu9.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Brian Murray (brian-murray) wrote :

Hello ChristianEhrhardt, or anyone else affected,

Accepted libvirt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

ChristianEhrhardt (paelzer) wrote :

Verification is ok for Yakkety and Zesty.
On Xenial there is an issue (nothing that breaks, but it doesn't fix it due to the apparmor packaging structure there being different).

I already have a follow on patch ready and test it now.
Hopefully I can reject/replace the one in x-p rather soon.

Changed in libvirt (Ubuntu Xenial):
status: Fix Committed → In Progress
tags: added: verification-done-yakkety verification-done-zesty verification-needed-xenial
removed: verification-needed
ChristianEhrhardt (paelzer) wrote :

Ok, with the fixup for Xenial (https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2769/+packages) it is also good there now.
Now contacting the release Team to help me replace that in proposed and do a final test from there.

Andy Whitcroft (apw) wrote :

Hello ChristianEhrhardt, or anyone else affected,

Accepted libvirt into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.3.1-1ubuntu10.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in libvirt (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed
ChristianEhrhardt (paelzer) wrote :

With so much back and forth going on I double checked that this is against the right package from proposed.

$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 1.3.1-1ubuntu10.10
  Candidate: 1.3.1-1ubuntu10.10
  Version table:
 *** 1.3.1-1ubuntu10.10 500
        500 http://ports.ubuntu.com//ubuntu-ports xenial-proposed/main ppc64el Packages
        100 /var/lib/dpkg/status

It is, and with that the vfio denial is now gone in xenial as well.
Thanks everybody - setting verification-done and wait its grace period everything in -proposed shall have.

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 2.1.0-1ubuntu9.5

---------------
libvirt (2.1.0-1ubuntu9.5) yakkety; urgency=medium

  * Add missing apparmor profile entries (LP: #1680384)
    - debian/patches/ubuntu/apparmor-vfio.patch: apparmor: add /dev/vfio
      for vf (hot) attach
    - debian/patches/ubuntu/apparmor-shutdown.patch: apparmor: allow to
      parse cmdline of the pid that send the shutdown signal

 -- Christian Ehrhardt <email address hidden> Wed, 26 Apr 2017 13:46:42 +0200

Changed in libvirt (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 2.5.0-3ubuntu5.1

---------------
libvirt (2.5.0-3ubuntu5.1) zesty; urgency=medium

  * Add missing apparmor profile entries (LP: #1680384)
    - debian/patches/ubuntu/apparmor-vfio.patch: apparmor: add /dev/vfio
      for vf (hot) attach
    - debian/patches/ubuntu/apparmor-ppcwrapper.patch: apparmor: allow
      extra tools executed by kvm.powerpc
    - debian/patches/ubuntu/apparmor-shutdown.patch: apparmor: allow to
      parse cmdline of the pid that send the shutdown signal

 -- Christian Ehrhardt <email address hidden> Wed, 26 Apr 2017 13:47:27 +0200

Changed in libvirt (Ubuntu Zesty):
status: Fix Committed → Fix Released
ChristianEhrhardt (paelzer) wrote :

FYI - Xenial was re-rolled so it waits a few days longer to be the right time in -proposed without reports. The SRU Team will likely pick it up in a few days.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 1.3.1-1ubuntu10.10

---------------
libvirt (1.3.1-1ubuntu10.10) xenial; urgency=medium

  * Fix bad SRU backport to match apparmor structure of libvirt
    in Xenial (LP: #1680384)
    - drop d/p/ubuntu/apparmor-shutdown.patch as the libvirt code here doesn't
      trigger it.
    - drop d/p/ubuntu/apparmor-vfio.patch as xenial still uses profiles from
      debian/apparmor/
    - apply content of apparmor-vfio.patch to debian/apparmor/libvirt-qemu
      to fix the actual issue.

libvirt (1.3.1-1ubuntu10.9) xenial; urgency=medium

  * Add missing apparmor profile entries (LP: #1680384)
    - debian/patches/ubuntu/apparmor-vfio.patch: apparmor: add /dev/vfio
      for vf (hot) attach
    - debian/patches/ubuntu/apparmor-shutdown.patch: apparmor: allow to
      parse cmdline of the pid that send the shutdown signal

 -- Christian Ehrhardt <email address hidden> Tue, 16 May 2017 12:38:02 +0200

Changed in libvirt (Ubuntu Xenial):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers