Apparmor denies /usr/libexec/virt-aa-helper access to ovmf files even when profiles allows it

[José Pekkarinen]

Hi,

This is a follow-up bug I'm pursuing with the gentoo community in case upstream can give some guidance. The original bug is:

https://bugs.gentoo.org/683976

In summary, apparmor denies access to the files under /usr/share/edk2-ovmf/ though the profiles are set to allow at least access to the relevant files to virt-aa-helper. This prevent ovmf domains to start at all. Some of the relevant outputs will follow for the eyes that doesn't want to follow the link:

# virsh start os-1
error: Failed to start domain os-1
error: internal error: cannot load AppArmor profile 'libvirt-34c41008-ab91-483b-959c-81a7a12ae9be'

os section of the domain:

  <os>
    <type arch='x86_64' machine='pc-i440fx-2.12'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2-ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/os-1_VARS.fd</nvram>
    <boot dev='network'/>
    <boot dev='hd'/>
  </os>

# ls /usr/share/edk2-ovmf/
OVMF_CODE.fd OVMF.fd OVMF_VARS.fd

# ls /var/lib/libvirt/qemu/nvram/os-1_VARS.fd
/var/lib/libvirt/qemu/nvram/os-1_VARS.fd

From libvirt.log:

2019-04-21 14:51:09.481+0000: 9347: error : virCommandWait:2636 : internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/virt-aa-helper -c -u libvirt-34c41008-ab91-483b-959c-81a7a12ae9be) unexpected exit status 1: 2019-04-21 14:51:09.480+0000: 27054: info : libvirt version: 5.2.0
2019-04-21 14:51:09.480+0000: 27054: info : hostname: mole
2019-04-21 14:51:09.480+0000: 27054: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vda': Permission denied
2019-04-21 14:51:09.480+0000: 27054: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vdb': Permission denied
virt-aa-helper: error: /usr/share/edk2-ovmf/OVMF_CODE.fd
virt-aa-helper: error: skipped restricted file
virt-aa-helper: error: invalid VM definition

2019-04-21 14:51:09.481+0000: 9347: error : AppArmorGenSecurityLabel:469 : internal error: cannot load AppArmor profile 'libvirt-34c41008-ab91-483b-959c-81a7a12ae9be'

# apparmor_parser -d usr.libexec.virt-aa-helper | grep ovmf
Mode: r:r Name: (/usr/share/edk2-ovmf/OVMF_{CODE,VARS}.fd)

# apparmor_parser -d usr.libexec.virt-aa-helper
----- Debugging built structures -----
Name: virt-aa-helper
Profile Mode: Enforce
Capabilities: dac_override dac_read_search
Network: inet inet6
--- Entries ---
Mode: r:r Name: (/**.[iI][sS][oO])
Mode: r:r Name: (/**.img)
Mode: r:r Name: (/**.qcow{,2})
Mode: r:r Name: (/**.qed)
Mode: r:r Name: (/**.raw)
Mode: r:r Name: (/**.vmdk)
Mode: r:r Name: (/**/disk{,.*})
Mode: r:r Name: (/dev/dri/{,*})
Mode: rwa:rwa Name: (/dev/full)
Mode: wa:wa Name: (/dev/log)
Mode: rwa:rwa Name: (/dev/null)
Mode: r:r Name: (/dev/random)
Mode: r:r Name: (/dev/urandom)
Mode: rwa:rwa Name: (/dev/zero)
Mode: r:r Name: (/etc/apparmor.d/libvirt/*)
Mode: rwa:rwa Name: (/etc/apparmor.d/libvirt/libvirt-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*-[0-9a-f]*)
Mode: r:r Name: (/etc/bindresvport.blacklist)
Mode: rm:rm Name: (/etc/ld.so.cache)
Mode: r:r Name: (/etc/ld.so.conf)
Mode: r:r Name: (/etc/ld.so.conf.d/{,*.conf})
Mode: r:r Name: (/etc/ld.so.preload)
Mode: r:r Name: (/etc/libnl-3/classid)
Mode: r:r Name: (/etc/locale.alias)
Mode: r:r Name: (/etc/locale/**)
Mode: r:r Name: (/etc/localtime)
Mode: rwalkmx: Name: (/home//.ecryptfs/*/.Private/**)
        link: (/**)
Mode: rm:rm Name: (/opt/*-linux-uclibc/lib/ld-uClibc*so*)
Mode: r:r Name: (/proc//[0-9]*/net/psched)
Mode: r: Name: (/proc//[0-9]*/status)
Mode: r:r Name: (/proc//cpuinfo)
Mode: r:r Name: (/proc//filesystems)
Mode: r:r Name: (/proc//meminfo)
Mode: r:r Name: (/proc//stat)
Mode: r:r Name: (/proc//sys/crypto/*)
Mode: r:r Name: (/proc//sys/kernel/cap_last_cap)
Mode: r:r Name: (/proc//sys/kernel/ngroups_max)
Mode: r:r Name: (/proc//sys/kernel/version)
Mode: r:r Name: (/proc//sys/vm/overcommit_memory)
Mode: r:r Name: (/proc//{[1-9],[1-9][0-9],[1-9][0-9][0-9],[1-9][0-9][0-9][0-9],[1-9][0-9][0-9][0-9][0-9],[1-9][0-9][0-9][0-9][0-9][0-9],[1-4][0-9][0-9][0-9][0-9][0-9][0-9]}/{maps,auxv,status})
Mode: wa:wa Name: (/run/systemd/journal/dev-log)
Mode: wa:wa Name: (/run/systemd/journal/socket)
Mode: rwa:rwa Name: (/run/systemd/journal/stdout)
Mode: r:r Name: (/sys//devices/system/cpu/)
Mode: r:r Name: (/sys//devices/system/cpu/online)
Mode: r:r Name: (/sys/bus/usb/devices/)
Mode: r:r Name: (/sys/devices/)
Mode: r:r Name: (/sys/devices/**)
Mode: rm:rm Name: (/usr/lib/*-linux-gnu*/gconv/*.so)
Mode: rm:rm Name: (/usr/lib/*-linux-gnu*/gconv/gconv-modules*)
Mode: rm:rm Name: (/usr/libexec/virt-aa-helper)
Mode: rm:rm Name: (/usr/lib{,32,64}/gconv/*.so)
Mode: rm:rm Name: (/usr/lib{,32,64}/gconv/gconv-modules*)
Mode: rm:rm Name: (/usr/lib{,32,64}/locale/**)
Mode: r:r Name: (/usr/share/**/locale/**)
Mode: r:r Name: (/usr/share/X11/locale/**)
Mode: r:r Name: (/usr/share/common-licenses/**)
Mode: r:r Name: (/usr/share/edk2-ovmf/OVMF_{CODE,VARS}.fd)
Mode: r:r Name: (/usr/share/locale-bundle/**)
Mode: r:r Name: (/usr/share/locale-langpack/**)
Mode: r:r Name: (/usr/share/locale/**)
Mode: r:r Name: (/usr/share/zoneinfo/)
Mode: r:r Name: (/usr/share/zoneinfo/**)
Mode: r:r Name: (/var/lib/libvirt/images/)
Mode: r:r Name: (/var/lib/libvirt/images/**)
Mode: r:r Name: (/var/lib/nova/instances/_base/*)
Mode: r:r Name: (/{,var/}run/libvirt/**/[sv]d[a-z])
Mode: r:r Name: (/{media,mnt,opt,srv}/**)
Mode: r:r Name: (/{usr/,}lib/*-linux-gnu*/**)
Mode: rm:rm Name: (/{usr/,}lib/*-linux-gnu*/**/lib*.so*)
Mode: rm:rm Name: (/{usr/,}lib/*-linux-gnu*/ld{,32,64}-*.so)
Mode: rm:rm Name: (/{usr/,}lib/*-linux-gnu*/lib*.so*)
Mode: rm:rm Name: (/{usr/,}lib/i386-linux-gnu/tls/i686/{cmov,nosegneg}/ld-*.so)
Mode: rm:rm Name: (/{usr/,}lib/i386-linux-gnu/tls/i686/{cmov,nosegneg}/lib*.so*)
Mode: rm:rm Name: (/{usr/,}lib/tls/i686/{cmov,nosegneg}/ld-*.so)
Mode: rm:rm Name: (/{usr/,}lib/tls/i686/{cmov,nosegneg}/lib*.so*)
Mode: r:r Name: (/{usr/,}lib{,32,64}/**)
Mode: rm:rm Name: (/{usr/,}lib{,32,64}/**/lib*.so*)
Mode: rm:rm Name: (/{usr/,}lib{,32,64}/ld{,32,64}-*.so)
Mode: rm:rm Name: (/{usr/,}lib{,32,64}/lib*.so*)
Mode: x:x Name: (/{usr/,}sbin/apparmor_parser)
Mode: r:r Name: ({/home//*,/root}/)
Mode: r:r Name: ({/home//*,/root}/**)
Mode: rwalkmx: Name: ({/home//*,/root}/.Private/**)
        link: (/**)
Mode: r:r Name: (/dev/dasd*)
Mode: r:r Name: (/dev/dm-*)
Mode: r:r Name: (/dev/drbd[0-9]*)
Mode: r:r Name: (/dev/mapper/)
Mode: r:r Name: (/dev/mapper/*)
Mode: r:r Name: (/dev/nvme*)
Mode: r:r Name: (/dev/sd*)
Mode: r:r Name: (/dev/vd*)
Mode: r:r Name: (/dev/zd[0-9]*)
Mode: r:r Name: (/proc//[0-9]*/mounts)
Mode: rwalkm:rwalkm Name: ({/home//*,/root}/.*)
        link: (/**)
Mode: rwa:rwa Name: ({/home//*,/root}/.*/)
Mode: rwalkm:rwalkm Name: ({/home//*,/root}/.*/**)
        link: (/**)
Mode: rwa:rwa Name: ({/home//*,/root}/bin/)
Mode: rwalkm:rwalkm Name: ({/home//*,/root}/bin/**)
        link: (/**)
ptrace (readby ),
ptrace (tracedby ),
ptrace (read ) virt-aa-helper,
signal (receive ) unconfined,
signal virt-aa-helper,
signal set=(exists),
unix () peer=( label="virt-aa-helper"),
unix () peer=( label="unconfined"),
unix (),
unix (),

And from further investigation, setting the profile in complain
mode doesn't allow the vms to start either:

# aa-complain usr.libexec.virt-aa-helper
Setting /etc/apparmor.d/usr.libexec.virt-aa-helper to complain mode.
# apparmor_parser -d /etc/apparmor.d/usr.libexec.virt-aa-helper
----- Debugging built structures -----
Name: virt-aa-helper
Profile Mode: Complain
Capabilities: dac_override dac_read_search
Network: inet inet6
...
# virsh start os-1
error: Failed to start domain os-1
error: internal error: cannot load AppArmor profile 'libvirt-34c41008-ab91-483b-959c-81a7a12ae9be'

And the error in the libvirtd.log:

2019-04-21 19:09:41.910+0000: 9349: error : virCommandWait:2636 : internal error: Child process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/libexec/virt-aa-helper -c -u libvirt-34c41008-ab91-483b-959c-81a7a12ae9be) unexpected exit status 1: 2019-04-21 19:09:41.909+0000: 41822: info : libvirt version: 5.2.0
2019-04-21 19:09:41.909+0000: 41822: info : hostname: mole
2019-04-21 19:09:41.909+0000: 41822: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vda': Permission denied
2019-04-21 19:09:41.909+0000: 41822: error : virStorageFileBackendFileRead:129 : Failed to open file '/dev/mole-vg0/os-1-vdb': Permission denied
virt-aa-helper: error: /usr/share/edk2-ovmf/OVMF_CODE.fd
virt-aa-helper: error: skipped restricted file
virt-aa-helper: error: invalid VM definition

This is based in vanilla kernel 5.0.8, apparmor 2.13.2.

Thanks!

José.

[José Pekkarinen (koalinux)]

For completeness, this is an output of what happens when I disable apparmor
from the system, the same domain is used:

# virsh start os-1
Domain os-1 started

From libvirtd.log:

2019-04-21 20:59:39.777+0000: 8775: info : libvirt version: 5.2.0
2019-04-21 20:59:39.777+0000: 8775: info : hostname: mole
2019-04-21 20:59:39.777+0000: 8775: warning : qemuDomainObjTaint:8050 : Domain id=1 name='mirror' uuid=6f017442-432a-4c94-b4dd-3247f524e075 is tainted: high-privileges
2019-04-21 20:59:39.842+0000: 8775: warning : qemuDomainObjTaint:8050 : Domain id=1 name='mirror' uuid=6f017442-432a-4c94-b4dd-3247f524e075 is tainted: host-cpu
2019-04-21 20:59:49.385+0000: 8774: warning : qemuDomainObjTaint:8050 : Domain id=2 name='os-1' uuid=34c41008-ab91-483b-959c-81a7a12ae9be is tainted: high-privileges
2019-04-21 20:59:49.385+0000: 8774: warning : qemuDomainObjTaint:8050 : Domain id=2 name='os-1' uuid=34c41008-ab91-483b-959c-81a7a12ae9be is tainted: host-cpu

Ignore the mirror domain as it's not ovmf related, and it works
either with or without apparmor.

[Jamie Strandboge (jdstrand)]

This isn't an AppArmor bug, but rather, potentially a libvirt bug. With this error:

error: internal error: cannot load AppArmor profile 'libvirt-34c41008-ab91-483b-959c-81a7a12ae9be'

what are the contents of /etc/apparmor.d/libvirt/libvirt-34c41008-ab91-483b-959c-81a7a12ae9be*, /etc/apparmor.d/abstractions/libvirt-qemu and /etc/apparmor.d/libvirt/TEMPLATE.qemu?

[Launchpad Janitor (janitor)]

[Expired for AppArmor because there has been no activity for 60 days.]

[Joaquin Menchaca (darkn3rd)]

This is still an issue.

The workaround:

```bash
sudo systemctl disable apparmor
```