Activity log for bug #1680384

Date Who What changed Old value New value Message
2017-04-06 09:33:14 Christian Ehrhardt  bug added bug
2017-04-06 09:33:19 Christian Ehrhardt  libvirt (Ubuntu): status New Triaged
2017-04-06 09:33:21 Christian Ehrhardt  libvirt (Ubuntu): assignee ChristianEhrhardt (paelzer)
2017-04-06 09:33:22 Christian Ehrhardt  libvirt (Ubuntu): importance Undecided High
2017-04-06 09:34:27 Christian Ehrhardt  bug added subscriber Ubuntu Server Team
2017-04-14 04:59:23 bugproxy tags architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704
2017-04-20 06:12:30 Christian Ehrhardt  libvirt (Ubuntu): status Triaged In Progress
2017-04-26 15:42:24 Launchpad Janitor libvirt (Ubuntu): status In Progress Fix Released
2017-05-09 09:22:33 Christian Ehrhardt  nominated for series Ubuntu Zesty
2017-05-09 09:22:33 Christian Ehrhardt  bug task added libvirt (Ubuntu Zesty)
2017-05-09 09:22:33 Christian Ehrhardt  nominated for series Ubuntu Yakkety
2017-05-09 09:22:33 Christian Ehrhardt  bug task added libvirt (Ubuntu Yakkety)
2017-05-09 09:22:33 Christian Ehrhardt  nominated for series Ubuntu Xenial
2017-05-09 09:22:33 Christian Ehrhardt  bug task added libvirt (Ubuntu Xenial)
2017-05-09 09:22:41 Christian Ehrhardt  libvirt (Ubuntu Xenial): status New Triaged
2017-05-09 09:22:42 Christian Ehrhardt  libvirt (Ubuntu Yakkety): status New Triaged
2017-05-09 09:22:44 Christian Ehrhardt  libvirt (Ubuntu Zesty): status New Triaged
2017-05-09 09:22:46 Christian Ehrhardt  libvirt (Ubuntu Xenial): importance Undecided High
2017-05-09 09:22:48 Christian Ehrhardt  libvirt (Ubuntu Yakkety): importance Undecided High
2017-05-09 09:22:50 Christian Ehrhardt  libvirt (Ubuntu Zesty): importance Undecided High
2017-05-09 09:22:51 Christian Ehrhardt  libvirt (Ubuntu Xenial): assignee ChristianEhrhardt (paelzer)
2017-05-09 09:22:52 Christian Ehrhardt  libvirt (Ubuntu Yakkety): assignee ChristianEhrhardt (paelzer)
2017-05-09 09:22:54 Christian Ehrhardt  libvirt (Ubuntu Zesty): assignee ChristianEhrhardt (paelzer)
2017-05-09 09:54:01 Christian Ehrhardt  description By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction. Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there. None of these are per Guest, so add to /etc/apparmor.d/abstractions/libvirt-qemu The missing entries as identified are: #1 For virtual functions the generic vfio interface /etc/apparmor.d/abstractions/libvirt-qemu like: # allow guest access to the generic base vfio interface (LP: #1678322) /dev/vfio/vfio rw, I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed. #2 Binaries for ppc64el based checks in the kvm wrapper script. That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions. apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 #3 Qemu tries to read who killed it to report on shutdown [ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 The following should be rather save /proc/*/cmdline r, That will make the <unkown process> in logs like the following more readable: "qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)" [Impact] * Attaching of virtual functions (SR-IOV) not possible without manually modification of apparmor rules * libvirt/qemu guest logfiles can not state the program that sent the shutdown signal * on ppc64el when starting on SMT enabled it failes (expected) but does not present the hint about SMT being the reasons that was added in >=zesty * various apparmor denials in the log hide other issues * Fix by opening up apparmor rules as needed for these use-cases [Test Case] * The rules covere a set of use cases, outlining them all here. Some only apply to >=Zesty, so I mark each subtest with release names. * All start with "Spawn a kvm guest e.g. via uvtool-libvirt" * A) >=Xenial - shutdown message - A1 - start the guest - A2 - stop the guest - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log - A-check: without the fix the log ends with <unknown process> With the fix it shall contain the name of the process stopping it instead of. So libvirt shutdown will show libvirtd, kill from bash shows bash, ...) * B) >=Xenial - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your system, one example is outlined here: # Prep VFs and attach $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs $ sudo modprobe vfio-pci $ lspci -n -s 0005:01:01.3 0005:01:01.3 0200: 10df:e228 (rev 10) $ cat VF-5.1.1.3.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0005' bus='0x01' slot='0x01' function='0x3'/> </source> </hostdev> - B2 until bug 1679704 is fixed you need to manually increase the locked memory limit of the qemu process before going on - B3 attach the device $ virsh attach-device z-test VF-5.1.1.3.xml - B-check: without the fix it will fail for a set of apparmor denials with the fix the attaching will succeed * C) >=Zesty - C1 - on a ppc64el system start and stop a guest - C-check: without the fix apparmor will hold many denies for ppc64_cpu and/or accessing /sys/devices/system/cpu/... With the fix not only the denies are gone, but also if you run on a system with SMT enabled (kvm will fail there) it will report to you in the log about "Error: You must disable SMT ..." [Regression Potential] * We are only further "opening up" the current apparmor profiles. So we should not end up affecting existing use cases by new blocks. * There is a very slight chance that due to accesses being allowed follow on actions are triggered which people are not used to like "the VF hot attach actually working". So if one relied on it not working it will be different for them. * Further I made some brainstorming on the potential effects but the only other one that came to my mind being a regression of some sort is that it now correctly states what send the shutdown signal. So when qemu did not end by itself (guest stops) but is stopped (shutdown) the old message was: terminating on signal 15 from pid 22938 (<unknown process>) And now will be like: terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd) For the hopefully super unlikely case that someone e.g. grepped for unknown here it might be a regression. And making that message more useful (along with getting rid of the associated aa deny is part of the intended fix). [Other Info] * Tested manually and via regression tests in advance on bileto tickets. So far almost all green, unrelated issues in some dep8 due to what seems transient network timeouts - if showing up in proposed as well we will likely have to retrigger the tests until good. Latest ticket for this is https://bileto.ubuntu.com/#/ticket/2738 * Upload will be directly, since in some of the releases I will need dpdk-genversion -v which bileto does not support (and it doesn't support publishing selective releases) yet I wanted to show the extra tests and checks made and therefore using bileto was beneficial even if eventually uploading "old style" ----- By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction. Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there. None of these are per Guest, so add to /etc/apparmor.d/abstractions/libvirt-qemu The missing entries as identified are: #1 For virtual functions the generic vfio interface /etc/apparmor.d/abstractions/libvirt-qemu like:   # allow guest access to the generic base vfio interface (LP: #1678322)   /dev/vfio/vfio rw, I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed. #2 Binaries for ppc64el based checks in the kvm wrapper script. That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions. apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 #3 Qemu tries to read who killed it to report on shutdown [ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 The following should be rather save /proc/*/cmdline r, That will make the <unkown process> in logs like the following more readable: "qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)"
2017-05-09 09:55:59 Christian Ehrhardt  description [Impact] * Attaching of virtual functions (SR-IOV) not possible without manually modification of apparmor rules * libvirt/qemu guest logfiles can not state the program that sent the shutdown signal * on ppc64el when starting on SMT enabled it failes (expected) but does not present the hint about SMT being the reasons that was added in >=zesty * various apparmor denials in the log hide other issues * Fix by opening up apparmor rules as needed for these use-cases [Test Case] * The rules covere a set of use cases, outlining them all here. Some only apply to >=Zesty, so I mark each subtest with release names. * All start with "Spawn a kvm guest e.g. via uvtool-libvirt" * A) >=Xenial - shutdown message - A1 - start the guest - A2 - stop the guest - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log - A-check: without the fix the log ends with <unknown process> With the fix it shall contain the name of the process stopping it instead of. So libvirt shutdown will show libvirtd, kill from bash shows bash, ...) * B) >=Xenial - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your system, one example is outlined here: # Prep VFs and attach $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs $ sudo modprobe vfio-pci $ lspci -n -s 0005:01:01.3 0005:01:01.3 0200: 10df:e228 (rev 10) $ cat VF-5.1.1.3.xml <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0005' bus='0x01' slot='0x01' function='0x3'/> </source> </hostdev> - B2 until bug 1679704 is fixed you need to manually increase the locked memory limit of the qemu process before going on - B3 attach the device $ virsh attach-device z-test VF-5.1.1.3.xml - B-check: without the fix it will fail for a set of apparmor denials with the fix the attaching will succeed * C) >=Zesty - C1 - on a ppc64el system start and stop a guest - C-check: without the fix apparmor will hold many denies for ppc64_cpu and/or accessing /sys/devices/system/cpu/... With the fix not only the denies are gone, but also if you run on a system with SMT enabled (kvm will fail there) it will report to you in the log about "Error: You must disable SMT ..." [Regression Potential] * We are only further "opening up" the current apparmor profiles. So we should not end up affecting existing use cases by new blocks. * There is a very slight chance that due to accesses being allowed follow on actions are triggered which people are not used to like "the VF hot attach actually working". So if one relied on it not working it will be different for them. * Further I made some brainstorming on the potential effects but the only other one that came to my mind being a regression of some sort is that it now correctly states what send the shutdown signal. So when qemu did not end by itself (guest stops) but is stopped (shutdown) the old message was: terminating on signal 15 from pid 22938 (<unknown process>) And now will be like: terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd) For the hopefully super unlikely case that someone e.g. grepped for unknown here it might be a regression. And making that message more useful (along with getting rid of the associated aa deny is part of the intended fix). [Other Info] * Tested manually and via regression tests in advance on bileto tickets. So far almost all green, unrelated issues in some dep8 due to what seems transient network timeouts - if showing up in proposed as well we will likely have to retrigger the tests until good. Latest ticket for this is https://bileto.ubuntu.com/#/ticket/2738 * Upload will be directly, since in some of the releases I will need dpdk-genversion -v which bileto does not support (and it doesn't support publishing selective releases) yet I wanted to show the extra tests and checks made and therefore using bileto was beneficial even if eventually uploading "old style" ----- By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction. Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there. None of these are per Guest, so add to /etc/apparmor.d/abstractions/libvirt-qemu The missing entries as identified are: #1 For virtual functions the generic vfio interface /etc/apparmor.d/abstractions/libvirt-qemu like:   # allow guest access to the generic base vfio interface (LP: #1678322)   /dev/vfio/vfio rw, I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed. #2 Binaries for ppc64el based checks in the kvm wrapper script. That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions. apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 #3 Qemu tries to read who killed it to report on shutdown [ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 The following should be rather save /proc/*/cmdline r, That will make the <unkown process> in logs like the following more readable: "qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)" [Impact]  * Attaching of virtual functions (SR-IOV) not possible without manually    modification of apparmor rules  * libvirt/qemu guest logfiles can not state the program that sent the    shutdown signal  * on ppc64el when starting on SMT enabled it failes (expected) but does    not present the hint about SMT being the reasons that was added in    >=zesty  * various apparmor denials in the log hide other issues  * Fix by opening up apparmor rules as needed for these use-cases [Test Case]  * The rules covere a set of use cases, outlining them all here. Some only    apply to >=Zesty, so I mark each subtest with release names.  * All start with "Spawn a kvm guest e.g. via uvtool-libvirt"  * A) >=Xenial - shutdown message    - A1 - start the guest    - A2 - stop the guest    - A3 - check the log in /var/log/libvirt/qemu/<guestname>.log    - A-check: without the fix the log ends with <unknown process>               With the fix it shall contain the name of the process               stopping it instead of. So libvirt shutdown will show               libvirtd, kill from bash shows bash, ...)  * B) >=Xenial    - B1 - prepare a hotplug xml and a SR-IOV device - that depends on your      system, one example is outlined here:       # Prep VFs and attach       $ echo 4 | sudo tee /sys/bus/pci/devices/0005\:01\:00.0/sriov_numvfs       $ sudo modprobe vfio-pci       $ lspci -n -s 0005:01:01.3         0005:01:01.3 0200: 10df:e228 (rev 10)       $ cat VF-5.1.1.3.xml         <hostdev mode='subsystem' type='pci' managed='yes'>           <source>             <address domain='0x0005' bus='0x01' slot='0x01'              function='0x3'/>           </source>         </hostdev>     - B2 until bug 1679704 is fixed you need to manually increase the       locked memory limit of the qemu process before going on     - B3 attach the device       $ virsh attach-device z-test VF-5.1.1.3.xml     - B-check: without the fix it will fail for a set of apparmor denials                with the fix the attaching will succeed  * C) >=Zesty    - C1 - on a ppc64el system start and stop a guest    - C-check: without the fix apparmor will hold many denies for ppc64_cpu               and/or accessing /sys/devices/system/cpu/...               With the fix not only the denies are gone, but also if you               run on a system with SMT enabled (kvm will fail there) it               will report to you in the log about "Error: You must disable               SMT ..." [Regression Potential]  * We are only further "opening up" the current apparmor profiles. So we    should not end up affecting existing use cases by new blocks.  * There is a very slight chance that due to accesses being allowed follow    on actions are triggered which people are not used to like "the VF hot    attach actually working". So if one relied on it not working it will be    different for them.  * Further I made some brainstorming on the potential effects but the only    other one that came to my mind being a regression of some sort is that    it now correctly states what send the shutdown signal. So when qemu did    not end by itself (guest stops) but is stopped (shutdown) the old    message was:      terminating on signal 15 from pid 22938 (<unknown process>)    And now will be like:      terminating on signal 15 from pid 7714 (/usr/sbin/libvirtd)    For the hopefully super unlikely case that someone e.g. grepped for    unknown here it might be a regression.    And making that message more useful (along with getting rid of the    associated aa deny is part of the intended fix). [Other Info]  * Tested manually and via regression tests in advance on bileto tickets.    So all are green there now, there were two unrelated in some dep8 tests due to what seems transient network timeouts (worked on retry) - if showing up in proposed as well we will likely have to retrigger the tests until good. Latest ticket for these to look at sniff builds: => https://bileto.ubuntu.com/#/ticket/2738  * Upload will be directly, since in some of the releases I will need    dpdk-genversion -v which bileto does not support (and it doesn't    support publishing selective releases) yet I wanted to show the extra    tests and checks made and therefore using bileto was beneficial even if    eventually uploading "old style" ----- By debugging various bugs I've found several entries missing in the libvirt-qemu apparmor abstraction. Those issues - now that they are understood - are taken out of the original bugs to focus on the remaining debugging there. None of these are per Guest, so add to /etc/apparmor.d/abstractions/libvirt-qemu The missing entries as identified are: #1 For virtual functions the generic vfio interface /etc/apparmor.d/abstractions/libvirt-qemu like:   # allow guest access to the generic base vfio interface (LP: #1678322)   /dev/vfio/vfio rw, I checked with the security team and also according to https://www.kernel.org/doc/Documentation/vfio.txt this base interface should be save to be allowed. #2 Binaries for ppc64el based checks in the kvm wrapper script. That wrapper extension about smt awareness was tested and works fine as-is, but not when driven via libvirt due to the profile restrictions. apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/usr/sbin/ppc64_cpu" pid=9539 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 apparmor="DENIED" operation="exec" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/bin/grep" pid=9541 comm="kvm" requested_mask="x" denied_mask="x" fsuid=64055 ouid=0 #3 Qemu tries to read who killed it to report on shutdown [ 518.615420] audit: type=1400 audit(1491467132.255:21): apparmor="DENIED" operation="open" profile="libvirt-88b15add-b290-431d-9e49-fa771588f2f5" name="/proc/7989/cmdline" pid=9531 comm="qemu-system-ppc" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0 The following should be rather save /proc/*/cmdline r, That will make the <unkown process> in logs like the following more readable: "qemu-system-ppc64: terminating on signal 15 from pid 10924 (<unknown process>)"
2017-05-09 10:12:14 Christian Ehrhardt  libvirt (Ubuntu Xenial): status Triaged In Progress
2017-05-09 10:12:18 Christian Ehrhardt  libvirt (Ubuntu Yakkety): status Triaged Fix Committed
2017-05-09 10:12:20 Christian Ehrhardt  libvirt (Ubuntu Xenial): status In Progress Fix Committed
2017-05-09 10:12:22 Christian Ehrhardt  libvirt (Ubuntu Zesty): status Triaged Fix Committed
2017-05-09 10:12:26 Christian Ehrhardt  nominated for series Ubuntu Trusty
2017-05-09 10:12:26 Christian Ehrhardt  bug task added libvirt (Ubuntu Trusty)
2017-05-09 10:12:37 Christian Ehrhardt  libvirt (Ubuntu Trusty): status New Incomplete
2017-05-11 19:00:31 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2017-05-11 19:00:36 Brian Murray bug added subscriber SRU Verification
2017-05-11 19:00:42 Brian Murray tags architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-needed
2017-05-16 10:58:07 Christian Ehrhardt  libvirt (Ubuntu Xenial): status Fix Committed In Progress
2017-05-16 11:00:04 Christian Ehrhardt  tags architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-needed architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-done-yakkety verification-done-zesty verification-needed-xenial
2017-05-16 11:41:39 Andy Whitcroft libvirt (Ubuntu Xenial): status In Progress Fix Committed
2017-05-16 11:41:45 Andy Whitcroft tags architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-done-yakkety verification-done-zesty verification-needed-xenial architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-done-yakkety verification-done-zesty verification-needed verification-needed-xenial
2017-05-16 12:32:30 Christian Ehrhardt  tags architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-done-yakkety verification-done-zesty verification-needed verification-needed-xenial architecture-ppc64le bugnameltc-153458 severity-high targetmilestone-inin1704 verification-done verification-done-xenial verification-done-yakkety verification-done-zesty
2017-05-22 15:02:05 Łukasz Zemczak removed subscriber Ubuntu Stable Release Updates Team
2017-05-22 15:02:40 Launchpad Janitor libvirt (Ubuntu Yakkety): status Fix Committed Fix Released
2017-05-22 15:12:08 Launchpad Janitor libvirt (Ubuntu Zesty): status Fix Committed Fix Released
2017-05-24 01:42:33 Launchpad Janitor libvirt (Ubuntu Xenial): status Fix Committed Fix Released
2017-12-06 22:12:12 Martin Pitt bug watch added https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=878203
2017-12-06 22:12:12 Martin Pitt bug task added libvirt (Debian)
2017-12-06 22:44:44 Bug Watch Updater libvirt (Debian): status Unknown Confirmed
2018-01-23 12:27:22 Bug Watch Updater libvirt (Debian): status Confirmed Fix Released