apparmor profile for libvirtd/libvirt-daemon needs fixing

Bug #1881969 reported by Robert Dinse
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
libvirt (Debian)
Fix Released
Unknown
libvirt (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Incomplete
Undecided
Unassigned
Hirsute
Incomplete
Undecided
Unassigned
Impish
Fix Released
Undecided
Unassigned

Bug Description

Libvirtd is trying to use a capability being denied it by apparmor.

[474656.842239] audit: type=1400 audit(1591211959.677:101): apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444 comm="libvirtd" capability=17 capname="sys_rawio"

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: libvirt-daemon 6.0.0-0ubuntu8.1
Uname: Linux 5.6.0 x86_64
ApportVersion: 2.20.11-0ubuntu27.2
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: MATE
Date: Wed Jun 3 14:01:30 2020
InstallationDate: Installed on 2017-05-27 (1103 days ago)
InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
SourcePackage: libvirt
UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: [modified]
modified.conffile..etc.libvirt.nwfilter.qemu-announce-self.xml: [modified]
modified.conffile..etc.libvirt.qemu.networks.default.xml: [modified]
mtime.conffile..etc.libvirt.nwfilter.allow-arp.xml: 2017-05-27T04:38:59.454073
mtime.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: 2017-05-27T04:38:58.894071
mtime.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: 2017-05-27T04:38:58.990072
mtime.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: 2017-05-27T04:38:59.714073
mtime.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: 2017-05-27T04:38:59.522073
mtime.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: 2018-10-27T01:48:21.872648
mtime.conffile..etc.libvirt.nwfilter.clean-traffic.xml: 2017-05-27T04:38:59.582073
mtime.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: 2017-05-27T04:38:58.942071
mtime.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: 2017-05-27T04:38:59.870074
mtime.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: 2017-05-27T04:38:59.818074
mtime.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: 2017-05-27T04:38:59.110072
mtime.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: 2017-05-27T04:38:59.178072
mtime.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: 2017-05-27T04:38:59.774074
mtime.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: 2017-05-27T04:38:59.254072
mtime.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: 2017-05-27T04:38:59.394073
mtime.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: 2017-05-27T04:38:59.646073
mtime.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: 2017-05-27T04:38:59.050072
mtime.conffile..etc.libvirt.nwfilter.qemu-announce-self.xml: 2017-05-27T04:38:59.322073
mtime.conffile..etc.libvirt.qemu.networks.default.xml: 2017-05-27T04:38:58.478070

Revision history for this message
Robert Dinse (nanook) wrote :
Revision history for this message
Paride Legovini (paride) wrote :

Thanks Robert for this bug report. Looks like this has to be fixed in the libvirt-daemon-system binary package.

For reference I linked this report to an analogous Debian bug, even if src:libvirt is not synced from Debian.

Changed in libvirt (Ubuntu):
status: New → Triaged
tags: added: server-next
Changed in libvirt (Debian):
status: Unknown → Incomplete
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I'd agree and work on adding the rule upstream and into Ubuntu, but what I need to to do is help to understand "why this triggers for you".

I run libvirt locally and in many tests, but so far have never seen this apparmor denial.
Although if it is a non fatal bug it is easier to miss ...

The linked Debian bug (thanks paride) has a bit more details how to trigger.
But the bug also is almost a year old and no one else has hit this, ... that is odd.

I've set up a L1 guest with an extra disk as scsi disk
 44 <disk type='file' device='disk'>
 45 <driver name='qemu' type='qcow2'/>
 46 <source file='/var/lib/uvtool/libvirt/images/testguest-scsi-ephem-00.qcow'/>
 47 <target dev='sda' bus='scsi'/>
 48 <address type='drive' controller='0' bus='0' target='0' unit='0'/>
 49 </disk>
...
100 <controller type='scsi' index='0' model='virtio-scsi'>
101 <address type='pci' domain='0x0000' bus='0x0a' slot='0x01' function='0x0'/>
102 </controller>

In the guest that appears as scsi disk, here from lshw:
  *-scsi
       description: SCSI storage controller
       product: Virtio SCSI
       vendor: Red Hat, Inc.
       physical id: 1
       bus info: pci@0000:07:01.0
       version: 00
       width: 64 bits
       clock: 33MHz
       capabilities: scsi msix bus_master cap_list
       configuration: driver=virtio-pci latency=0
       resources: irq:23 ioport:c000(size=64) memory:fc000000-fc000fff memory:fe000000-fe003fff
  *-disk
       description: SCSI Disk
       product: QEMU HARDDISK
       vendor: QEMU
       physical id: 0.0.0
       bus info: scsi@0:0.0.0
       logical name: /dev/sda
       version: 2.5+
       size: 4GiB (4294MB)
       capabilities: 5400rpm
       configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512
  *-sata
       description: SATA controller
       product: 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
       vendor: Intel Corporation
       physical id: 1f.2
       bus info: pci@0000:00:1f.2
       version: 02
       width: 32 bits
       clock: 33MHz
       capabilities: sata msi ahci_1.0 bus_master cap_list
       configuration: driver=ahci latency=0
       resources: irq:41 ioport:d060(size=32) memory:fd41b000-fd41bfff

Using that to define another guest:
        <disk type='block' device='disk'>
                <driver name='qemu' type='raw'/>
                <source dev='/dev/sda'/>
                <target dev='sda' bus='scsi'/>
        </disk>
        <controller type='scsi' index='0' model='virtio-scsi'/>

But with that the guest starts fine and no apparmor denial shows up.
Could you help by outlining how you configure your host and guest so that this issue triggers.

Only then we have a use case that we can tie to the new apparmor rule to allow this.

Changed in libvirt (Ubuntu):
status: Triaged → Incomplete
Revision history for this message
Robert Dinse (nanook) wrote : Re: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs fixing
Download full text (8.0 KiB)

      I am simply using virt-manager to create virtual machines, I've got
maybe half a dozen on a box, and then after a reboot they are set to start
automatically and dmesg will give me those messages. This is under Ubuntu
20.04, I can not swear they were not happening earlier but I did not notice
them.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Fri, 5 Jun 2020, Christian Ehrhardt  wrote:

> Date: Fri, 05 Jun 2020 04:23:39 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> I'd agree and work on adding the rule upstream and into Ubuntu, but what
> I need to to do is help to understand "why this triggers for you".
>
> I run libvirt locally and in many tests, but so far have never seen this apparmor denial.
> Although if it is a non fatal bug it is easier to miss ...
>
> The linked Debian bug (thanks paride) has a bit more details how to trigger.
> But the bug also is almost a year old and no one else has hit this, ... that is odd.
>
> I've set up a L1 guest with an extra disk as scsi disk
> 44 <disk type='file' device='disk'>
> 45 <driver name='qemu' type='qcow2'/>
> 46 <source file='/var/lib/uvtool/libvirt/images/testguest-scsi-ephem-00.qcow'/>
> 47 <target dev='sda' bus='scsi'/>
> 48 <address type='drive' controller='0' bus='0' target='0' unit='0'/>
> 49 </disk>
> ...
> 100 <controller type='scsi' index='0' model='virtio-scsi'>
> 101 <address type='pci' domain='0x0000' bus='0x0a' slot='0x01' function='0x0'/>
> 102 </controller>
>
>
> In the guest that appears as scsi disk, here from lshw:
> *-scsi
> description: SCSI storage controller
> product: Virtio SCSI
> vendor: Red Hat, Inc.
> physical id: 1
> bus info: pci@0000:07:01.0
> version: 00
> width: 64 bits
> clock: 33MHz
> capabilities: scsi msix bus_master cap_list
> configuration: driver=virtio-pci latency=0
> resources: irq:23 ioport:c000(size=64) memory:fc000000-fc000fff memory:fe000000-fe003fff
> *-disk
> description: SCSI Disk
> product: QEMU HARDDISK
> vendor: QEMU
> physical id: 0.0.0
> bus info: scsi@0:0.0.0
> logical name: /dev/sda
> version: 2.5+
> size: 4GiB (4294MB)
> capabilities: 5400rpm
> configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512
> *-sata
> description: SATA controller
> product: 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
> vendor: Intel Corporation
> physical id: 1f.2
> bus info: pci@0000:00:1f.2
> version: 02
> width: 32 bits
> clock: 33MHz
> capabilities: sata msi ahci_1.0 bus_master cap_list
> configuration: driver=ahci latency=0
> resources: irq:41 ioport:d060(size=32) memory:fd41b000-fd41bf...

Read more...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hmm,
virt-manager can still set up a lot of different guest configurations.
I've been using virt-manager guests as well and they don't show this.

You said you see these messages after a reboot on auto-start.
Can you try to un-break this a bit.

For example:
a) disable auto-starting the guests, does the libvirtd daemon still trigger the denial at reboot?
b) if (a) didn't trigger it, then does it happen once you start the guests?
c) did you made any changes to /etc/libvirt/*?
d) if (b) is true does it happen for all the guests?
e) since the other bug report mentions scsi disks, does your host or guest setup use scsi (or other less common disks)?
f) if you find a particular guest that triggers it, could you share the guest xml definition?
...

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (5.7 KiB)

      I will be rebooting one of the physical hosts in a little more than an
hour, I'll disable auto-start and try it.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Fri, 5 Jun 2020, Christian Ehrhardt  wrote:

> Date: Fri, 05 Jun 2020 05:34:26 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> Hmm,
> virt-manager can still set up a lot of different guest configurations.
> I've been using virt-manager guests as well and they don't show this.
>
> You said you see these messages after a reboot on auto-start.
> Can you try to un-break this a bit.
>
> For example:
> a) disable auto-starting the guests, does the libvirtd daemon still trigger the denial at reboot?
> b) if (a) didn't trigger it, then does it happen once you start the guests?
> c) did you made any changes to /etc/libvirt/*?
> d) if (b) is true does it happen for all the guests?
> e) since the other bug report mentions scsi disks, does your host or guest setup use scsi (or other less common disks)?
> f) if you find a particular guest that triggers it, could you share the guest xml definition?
> ...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1881969
>
> Title:
> apparmor profile for libvirtd/libvirt-daemon needs fixing
>
> Status in libvirt package in Ubuntu:
> Incomplete
> Status in libvirt package in Debian:
> Incomplete
>
> Bug description:
> Libvirtd is trying to use a capability being denied it by apparmor.
>
> [474656.842239] audit: type=1400 audit(1591211959.677:101):
> apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444
> comm="libvirtd" capability=17 capname="sys_rawio"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: libvirt-daemon 6.0.0-0ubuntu8.1
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27.2
> Architecture: amd64
> CasperMD5CheckResult: skip
> CurrentDesktop: MATE
> Date: Wed Jun 3 14:01:30 2020
> InstallationDate: Installed on 2017-05-27 (1103 days ago)
> InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
> SourcePackage: libvirt
> UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
> modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
> modified.conffile.....

Read more...

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (7.3 KiB)

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Fri, 5 Jun 2020, Christian Ehrhardt  wrote:

> Date: Fri, 05 Jun 2020 05:34:26 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> Hmm,
> virt-manager can still set up a lot of different guest configurations.
> I've been using virt-manager guests as well and they don't show this.
>
> You said you see these messages after a reboot on auto-start.
> Can you try to un-break this a bit.
>
> For example:
> a) disable auto-starting the guests, does the libvirtd daemon still trigger the denial at reboot?

      All guests are disabled, and I verified none of them were started after
rebooting the physical host, still on the physical host I got:

audit: type=1400 audit(1591342993.302:33): apparmor="DENIED"
operation="capable" profile="libvirtd" pid=2140 comm="libvirtd" capability=17
capname="sys_rawio"
[ 120.450546] audit: type=1400 audit(1591342993.312:34): apparmor="DENIED"
operation="capable" profile="libvirtd" pid=2140 comm="libvirtd" capability=10
capname="net_bind_service"
[ 120.491737] audit: type=1400 audit(1591342993.352:35): apparmor="DENIED"
operation="capable" profile="libvirtd" pid=2140 comm="libvirtd" capability=10
capname="net_bind_service"
[ 120.947173] audit: type=1400 audit(1591342993.802:36): apparmor="DENIED"
operation="capable" profile="libvirtd" pid=2140 comm="libvirtd" capability=10
capname="net_bind_service"
[ 121.039929] audit: type=1400 audit(1591342993.902:37): apparmor="DENIED"
operation="capable" profile="libvirtd" pid=2140 comm="libvirtd" capability=17
capname="sys_rawio"

> b) if (a) didn't trigger it, then does it happen once you start the guests?

      After starting guests, I got two additional messages for each guest:

[ 379.820259] audit: type=1400 audit(1591343252.681:51): apparmor="STATUS"
operation="profile_replace" profile="unconfined"
name="libvirt-deead97e-a18c-4643-a345-c8f43fe2d64d" pid=34595
comm="apparmor_parser"
[ 379.823200] audit: type=1400 audit(1591343252.681:52): apparmor="DENIED"
operation="capable" profile="libvirtd" pid=2140 comm="libvirtd" capability=10
capname="net_bind_service"

> c) did you made any changes to /etc/libvirt/*?

      No

> d) if (b) is true does it happen for all the guests?

      Yes

> e) since the other bug report mentions scsi disks, does your host or guest setup use scsi (or other less common disks)?

      I am using a RAID10 array of SCSI disks (WD Black 4GB x 4) for the
partition /var/lib/libvirt is on and a WD black 2GB for root partition.

> f) if you find a particular guest that triggers it, could you share the guest xml definition?

      They all trigger it regardless of configuration or guest OS.

> ...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> htt...

Read more...

Revision history for this message
Paride Legovini (paride) wrote :

Christian: do you think it's worth trying to emulate an actual hardware controller instead of using virtio-scsi in your nested VM test setup? Maybe sys_rawio is not used with virtio-scsi.

Robert: I think sharing the XML definition of a VM triggering the problem would still be useful. You can easily dump it with:

  $ virsh list --all # show all the domains
  $ virsh dumpxml <domain name>

One question, just to be sure: does the sys_rawio denial prevent the VM from running, or do you see the error but the VM still runs?

Thanks!

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (5.5 KiB)

      Since it occurred without starting ANY virtual server, the XML definition
would seem to me to be irrelevant.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Fri, 5 Jun 2020, Paride Legovini wrote:

> Date: Fri, 05 Jun 2020 12:16:54 -0000
> From: Paride Legovini <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> Christian: do you think it's worth trying to emulate an actual hardware
> controller instead of using virtio-scsi in your nested VM test setup?
> Maybe sys_rawio is not used with virtio-scsi.
>
> Robert: I think sharing the XML definition of a VM triggering the
> problem would still be useful. You can easily dump it with:
>
> $ virsh list --all # show all the domains
> $ virsh dumpxml <domain name>
>
> One question, just to be sure: does the sys_rawio denial prevent the VM
> from running, or do you see the error but the VM still runs?
>
> Thanks!
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1881969
>
> Title:
> apparmor profile for libvirtd/libvirt-daemon needs fixing
>
> Status in libvirt package in Ubuntu:
> Incomplete
> Status in libvirt package in Debian:
> Incomplete
>
> Bug description:
> Libvirtd is trying to use a capability being denied it by apparmor.
>
> [474656.842239] audit: type=1400 audit(1591211959.677:101):
> apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444
> comm="libvirtd" capability=17 capname="sys_rawio"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: libvirt-daemon 6.0.0-0ubuntu8.1
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27.2
> Architecture: amd64
> CasperMD5CheckResult: skip
> CurrentDesktop: MATE
> Date: Wed Jun 3 14:01:30 2020
> InstallationDate: Installed on 2017-05-27 (1103 days ago)
> InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
> SourcePackage: libvirt
> UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
> modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
> modified.conffile..etc.lib...

Read more...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

First of all thanks Robert.

Out of the results we have learned that this is really only related to libvirt (re)starting and not to anything with the guests.

Two capabilities that are hit are:
  capname="sys_rawio"
  capname="net_bind_service"

I have a system with /var on LVM on SCSI (Fcp) disks.
It does not hit either of the above apparmor issues.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI rurther related links (also talk about scsi disks being related):
- https://github.com/cockpit-project/cockpit/issues/12250
- https://github.com/cockpit-project/cockpit/pull/12545

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I was using this real SCSI disks to check Paride's theory that virtio-scsi might be special.

I was using /var on it as well as passing a guest disk with it on either the multipath or the pure /dev/sda1 device.

Still nothing triggered the denials that we got reported.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hmm,
the only other occurrence of net_bind_service was way back when the profiles were not generated correctly. But since you hit this on re-load the profile is loaded for sure.

@Robert - just to be sure - do you have the up-to-date package and conffiles?

ubuntu@s1lp05:~$ dpkg -S /etc/apparmor.d/usr.sbin.libvirtd
libvirt-daemon-system: /etc/apparmor.d/usr.sbin.libvirtd

ubuntu@s1lp05:~$ md5sum /etc/apparmor.d/usr.sbin.libvirtd
0900e11903140bcab48786c8ec744a06 /etc/apparmor.d/usr.sbin.libvirtd

$ apt-cache policy libvirt-daemon
libvirt-daemon:
  Installed: 6.0.0-0ubuntu8
...

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (5.2 KiB)

      It may or may not be relevant but I am using a 5.7.0 kernel compiled
from mainline kernel.org source.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Mon, 8 Jun 2020, Christian Ehrhardt  wrote:

> Date: Mon, 08 Jun 2020 07:00:31 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> I was using this real SCSI disks to check Paride's theory that virtio-
> scsi might be special.
>
> I was using /var on it as well as passing a guest disk with it on either
> the multipath or the pure /dev/sda1 device.
>
> Still nothing triggered the denials that we got reported.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1881969
>
> Title:
> apparmor profile for libvirtd/libvirt-daemon needs fixing
>
> Status in libvirt package in Ubuntu:
> Incomplete
> Status in libvirt package in Debian:
> Incomplete
>
> Bug description:
> Libvirtd is trying to use a capability being denied it by apparmor.
>
> [474656.842239] audit: type=1400 audit(1591211959.677:101):
> apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444
> comm="libvirtd" capability=17 capname="sys_rawio"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: libvirt-daemon 6.0.0-0ubuntu8.1
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27.2
> Architecture: amd64
> CasperMD5CheckResult: skip
> CurrentDesktop: MATE
> Date: Wed Jun 3 14:01:30 2020
> InstallationDate: Installed on 2017-05-27 (1103 days ago)
> InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
> SourcePackage: libvirt
> UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
> modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: [modifi...

Read more...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hmm, interesting.
any chance to reboot into a normal Focal 5.4 kernel for a try if that is part of the cause?

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (5.0 KiB)

      No these servers are in service providing services to people so I can't
just reboot at my leisure.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Mon, 8 Jun 2020, Christian Ehrhardt  wrote:

> Date: Mon, 08 Jun 2020 08:42:02 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> Hmm, interesting.
> any chance to reboot into a normal Focal 5.4 kernel for a try if that is part of the cause?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1881969
>
> Title:
> apparmor profile for libvirtd/libvirt-daemon needs fixing
>
> Status in libvirt package in Ubuntu:
> Incomplete
> Status in libvirt package in Debian:
> Incomplete
>
> Bug description:
> Libvirtd is trying to use a capability being denied it by apparmor.
>
> [474656.842239] audit: type=1400 audit(1591211959.677:101):
> apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444
> comm="libvirtd" capability=17 capname="sys_rawio"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: libvirt-daemon 6.0.0-0ubuntu8.1
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27.2
> Architecture: amd64
> CasperMD5CheckResult: skip
> CurrentDesktop: MATE
> Date: Wed Jun 3 14:01:30 2020
> InstallationDate: Installed on 2017-05-27 (1103 days ago)
> InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
> SourcePackage: libvirt
> UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
> modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: [modified]
> m...

Read more...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

On Mon, Jun 8, 2020 at 11:01 AM Robert Dinse <email address hidden>
wrote:

> No these servers are in service providing services to people so I can't
> just reboot at my leisure.
>

Too bad, it is the most likely case atm to further identify the root cause.
If at a maintenance window somewhen you can give it a try let us know what
the outcome is.

FYI - I have asked the security Team if these particular new denials ring
any bell in regard to the new upstream kernel version and/or Ubuntu Delta
on our kernel.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I already got a reply, it didn't ring a bell.
Never the less this is a test worth a try once you are able to do it.
The apparmor maintainer will poke at it a bit more and let me know.

P.S. since we had much earlier reports of the same denial it surely doesn't have to be the new kernel.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

@Paride - do you have other scsi adapters we could use. I have tried both ends - the virtual scsi in KVM and the multi-channel FCP on s390x. But if you have something more common like an x86 box with a local scsi adapter we could try - then let me know.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

@Robert - you said you see that since an upgrade to 20.04 - to what extend could you try other older libvirt versions?

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (5.4 KiB)

      Yes I will do but that could be several weeks.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Mon, 8 Jun 2020, Christian Ehrhardt  wrote:

> Date: Mon, 08 Jun 2020 09:04:00 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: Re: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon
> needs fixing
>
> On Mon, Jun 8, 2020 at 11:01 AM Robert Dinse <email address hidden>
> wrote:
>
>> No these servers are in service providing services to people so I can't
>> just reboot at my leisure.
>>
>
> Too bad, it is the most likely case atm to further identify the root cause.
> If at a maintenance window somewhen you can give it a try let us know what
> the outcome is.
>
> FYI - I have asked the security Team if these particular new denials ring
> any bell in regard to the new upstream kernel version and/or Ubuntu Delta
> on our kernel.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1881969
>
> Title:
> apparmor profile for libvirtd/libvirt-daemon needs fixing
>
> Status in libvirt package in Ubuntu:
> Incomplete
> Status in libvirt package in Debian:
> Incomplete
>
> Bug description:
> Libvirtd is trying to use a capability being denied it by apparmor.
>
> [474656.842239] audit: type=1400 audit(1591211959.677:101):
> apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444
> comm="libvirtd" capability=17 capname="sys_rawio"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: libvirt-daemon 6.0.0-0ubuntu8.1
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27.2
> Architecture: amd64
> CasperMD5CheckResult: skip
> CurrentDesktop: MATE
> Date: Wed Jun 3 14:01:30 2020
> InstallationDate: Installed on 2017-05-27 (1103 days ago)
> InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
> SourcePackage: libvirt
> UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
> modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
> modified.conffil...

Read more...

Revision history for this message
Robert Dinse (nanook) wrote :
Download full text (5.0 KiB)

      I could try on my workstation but main drive is an SSD rather than
SATA.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Mon, 8 Jun 2020, Christian Ehrhardt  wrote:

> Date: Mon, 08 Jun 2020 09:10:46 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1881969] Re: apparmor profile for libvirtd/libvirt-daemon needs
> fixing
>
> @Robert - you said you see that since an upgrade to 20.04 - to what
> extend could you try other older libvirt versions?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1881969
>
> Title:
> apparmor profile for libvirtd/libvirt-daemon needs fixing
>
> Status in libvirt package in Ubuntu:
> Incomplete
> Status in libvirt package in Debian:
> Incomplete
>
> Bug description:
> Libvirtd is trying to use a capability being denied it by apparmor.
>
> [474656.842239] audit: type=1400 audit(1591211959.677:101):
> apparmor="DENIED" operation="capable" profile="libvirtd" pid=3393444
> comm="libvirtd" capability=17 capname="sys_rawio"
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: libvirt-daemon 6.0.0-0ubuntu8.1
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27.2
> Architecture: amd64
> CasperMD5CheckResult: skip
> CurrentDesktop: MATE
> Date: Wed Jun 3 14:01:30 2020
> InstallationDate: Installed on 2017-05-27 (1103 days ago)
> InstallationMedia: Ubuntu-MATE 17.04 "Zesty Zapus" - Release amd64 (20170412)
> SourcePackage: libvirt
> UpgradeStatus: Upgraded to focal on 2020-04-26 (38 days ago)
> modified.conffile..etc.libvirt.nwfilter.allow-arp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp-server.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-dhcp.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-incoming-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.allow-ipv4.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic-gateway.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.clean-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-arp-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-multicast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-ip-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-mac-broadcast.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-mac-spoofing.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-other-l2-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.no-other-rarp-traffic.xml: [modified]
> modified.conffile..etc.libvirt.nwfilter.qemu-announce-self-rarp.xml: [modified]
> modified.conffile....

Read more...

tags: removed: server-next
Revision history for this message
costinel (costinel) wrote (last edit ):

what is the fix right now?

I get this message only once per kernel lifetime, when libvirtd starts for the first time, without starting any vm guest.

If I stop and restart libvirtd, it never shows again.

happens both with 20.04 lts kernel and hwe kernel (5.4 and 5.11). this is a clean install of 20.04lts & no customisations to libvirtd

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Costinel and Robert,
sorry that we all had dropped the ball on this for a while - it was mostly by the lack of being able to reproduce it anywhere else that stalled this.

... [imagine a long useless trip trying to re-trigger it, but details of that would not help] ...

Much better I found that it was already found, discussed and eventually fixed [1].
This is in libvirt 7.5 and later and thereby released in Ubuntu 21.04 and later.

Since we didn't yet find a functional issue if that is lacking (other than the stray log entries), I'd keep it at that (fixed in later relases, can be worked around via config files in older releases).

If either of you (or anyone else) spots a functional issue that gets resolved by adding those we can re-consider an SRU. In that case please speak up here what use-case is blocked by not having those allowed. If it outweights the risk of adding that we can SRU this changed default config.

If not, for anyone interested, the workaround for an admin would be to add the rules [1] added in your local override `/etc/apparmor.d/local/usr.sbin.libvirtd`.

---

P.S. Along that I also found why (among other reasons) that those messages might sometimes be relates to a scsi setup. I've found that (the above is the rule for the daemon) also guests might run into trouble with that if using <disk type='block' device='lun' rawio='yes'> as outlined in [2]. That might be a real issue, but would be another bug and needs upstream implementation in virt-aa-helper and libvirtd to add the rule accordingly for that guest configuration.
But OTOH since using that feature also includes "domain have to be executed with root privileges" and therefore isn't very safe we would not want to make that easier.
So for the few who want that very special and less safe subcase, consider adding the rule to your local guest override in /etc/apparmor.d/local/abstractions/libvirt-qemu

---

P.P.S and the message we see on service start might despite all the confusion be capability checking on service startup. While I didn't see why it would happen exactly *sometimes* for an undefined subset of sometimes. I can see that the probing of features when the daemon starts will include sys_rawio since it is meant to eable/drop that for lxc based guests.

There are also pool-capabilities, but while it would seem reasonable I couldn't quickly find sys_rawio there.

But if the usage for sys_rawio is really only unsafe guest disks and lxc caps, then (again) I think this isn't something we want to make much easier to use as it is less safe (and for lxc usage LXD is very much recommended over libvirt anyway).

[1]: https://gitlab.com/libvirt/libvirt/-/commit/4f2811eb816ed1da215b86778dfcf483917666a1
[2]: https://gitlab.com/libvirt/libvirt/-/commit/397e6a705b98a89c0cc6ca74db9648cf8697e214
[3]:

Changed in libvirt (Ubuntu):
status: Incomplete → Fix Released
Changed in libvirt (Ubuntu Impish):
status: New → Fix Released
Changed in libvirt (Ubuntu Hirsute):
status: New → Incomplete
Changed in libvirt (Ubuntu Focal):
status: New → Incomplete
Revision history for this message
Robert Dinse (nanook) wrote :

It's been a year and a half since I submitted this bug, I was on kernel 5.6.0 (mainstream from kernel.org) and now I'm on 5.13.19 and this error is no longer showing at boot time.

Changed in libvirt (Debian):
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.